在对老系统进行saas改造的时候,在项目初期使用了动态数据源的方式处理业务。
动态数据源
SaaS是Software-as-a-service(软件即服务) 它是一种通过Internet提供软件的模式,厂商将应用软件统一部署在自己的服务器
①独立性:每个租户的系统相互独立。
②平台性:所有租户归平台统一管理。
③隔离性:每个租户的数据相互隔离。
需求
方案
独立数据库、共享数据库、共享架构、OLTP、OLAP
根据客户需求可以选择隔离数据库。
也可以选择公用数据库。
采用OLTP方案进行数据同步。
采用OLAP方案进行冷数据同步。
技术方案
spring boot 2.4.5
mybatis plus
AbstractRoutingDataSource
Spring 官网提供的切换数据源的抽象方法,基于查找键将getConnection()调用路由到各种目标 DataSource 之一的抽象DataSource实现。后者通常(但不一定)通过一些线程绑定的事务上下文来确定。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 @Override public Connection getConnection () throws SQLException { return determineTargetDataSource().getConnection(); } @Override public Connection getConnection (String username, String password) throws SQLException { return determineTargetDataSource().getConnection(username, password); } @Nullable protected abstract Object determineCurrentLookupKey () ; protected DataSource determineTargetDataSource () { Assert.notNull(this .resolvedDataSources, "DataSource router not initialized" ); Object lookupKey = determineCurrentLookupKey(); DataSource dataSource = this .resolvedDataSources.get(lookupKey); if (dataSource == null && (this .lenientFallback || lookupKey == null )) { dataSource = this .resolvedDefaultDataSource; } if (dataSource == null ) { throw new IllegalStateException ("Cannot determine target DataSource for lookup key [" + lookupKey + "]" ); } return dataSource; }
启动效果 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 [main] [SpringApplication.java : 679] The following profiles are active: dev [main] [Bootstrap.java : 68] UT026010: Buffer pool was not set on WebSocketDeploymentInfo, the default pool will be used [main] [ServletContextImpl.java : 371] Initializing Spring embedded WebApplicationContext [main] [ServletWebServerApplicationContext.java : 289] Root WebApplicationContext: initialization completed in 1022 ms [main] [HikariDataSource.java : 110] HikariPool-1 - Starting... [main] [HikariDataSource.java : 123] HikariPool-1 - Start completed. [main] [DynamicRoutingDataSource.java : 33] setDynamicRoutingDataSource ... [main] [DynamicRoutingDataSourceConfig.java : 82] initDynamicDataSource ... [main] [BaseJdbcLogger.java : 137] ==> Preparing: SELECT id,tenant_id,tenant_name,datasource_url,datasource_username,datasource_password,datasource_driver,system_account,system_password,system_project,is_enable,create_time,update_time FROM tenant_info [main] [BaseJdbcLogger.java : 137] ==> Parameters: [main] [BaseJdbcLogger.java : 137] <== Total: 3 [main] [DynamicRoutingDataSourceConfig.java : 98] init dataSource OTQyNTkyMjA3ODI5ODYwMzUy [main] [HikariDataSource.java : 80] HikariPool-2 - Starting... [main] [HikariDataSource.java : 82] HikariPool-2 - Start completed. [main] [DynamicRoutingDataSourceConfig.java : 98] init dataSource OTQyNTkyMjA3OTA5NTUyMTI4 [main] [HikariDataSource.java : 80] HikariPool-3 - Starting... [main] [HikariDataSource.java : 82] HikariPool-3 - Start completed. [main] [DynamicRoutingDataSourceConfig.java : 98] init dataSource OTQyNTkyMjA3OTU5ODgzNzc2 [main] [HikariDataSource.java : 80] HikariPool-4 - Starting... [main] [HikariDataSource.java : 82] HikariPool-4 - Start completed. [main] [DynamicRoutingDataSource.java : 33] setDynamicRoutingDataSource ... [main] [ExecutorConfigurationSupport.java : 181] Initializing ExecutorService 'applicationTaskExecutor' [main] [Undertow.java : 120] starting server: Undertow - 2.2.7.Final [main] [Xnio.java : 95] XNIO version 3.8.0.Final [main] [NioXnio.java : 59] XNIO NIO Implementation Version 3.8.0.Final [main] [Version.java : 52] JBoss Threads version 3.1.0.Final [main] [UndertowWebServer.java : 133] Undertow started on port(s) 9033 (http) [main] [StartupInfoLogger.java : 61] Started AppApplication in 2.921 seconds (JVM running for 3.593)
首先初始化默认的数据源HikariPool-1。
通过查询租户表,再次初始化3个数据源DataSourceHikariPool-2\HikariPool-3\HikariPool-4
。
测试用例
1 2 3 4 5 6 7 8 9 10 11 12 ➜ Downloads curl http://127.0.0.1:9033 {"code" :"200" ,"all-db" :["942592207909552128" ,"942592207829860352" ,"942592207959883776" ],"db" :"master" }% ➜ Downloads curl http://127.0.0.1:9033/942592207909552128 {"code" :"200" ,"key" :"942592207909552128" ,"db" :["docker_dynamic_data_1" ]}% ➜ Downloads curl http://127.0.0.1:9033 {"code" :"200" ,"all-db" :["942592207909552128" ,"942592207829860352" ,"942592207959883776" ],"db" :"942592207909552128" }% ➜ Downloads curl http://127.0.0.1:9033/1 {"code" :"200" ,"key" :"master" ,"db" :["docker_dynamic_data" ]}%
DynamicDataSourceContextHolder
数据源上下文管理工具,用于维护当前线程数据源key。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 public class DynamicDataSourceContextHolder { private static class SingletonHolder { private static final DynamicDataSourceContextHolder INSTANCE = new DynamicDataSourceContextHolder (); } private DynamicDataSourceContextHolder () { } public static final DynamicDataSourceContextHolder getInstance () { return SingletonHolder.INSTANCE; } private static final ThreadLocal<String> contextHolder = new ThreadLocal <String>() { @Override protected String initialValue () { return DynamicDataSourceConstant.MASTER; } }; private static Set<Object> dataSourceKeys = Collections.synchronizedSet(new HashSet <>()); public void setDataSourceKey (String key) { if (containDataSourceKey(key)) { if (!ObjectUtils.isEmpty(key)) { contextHolder.set(key); } } } public String getDataSourceKey () { return contextHolder.get(); } public void clearDataSourceKey () { contextHolder.remove(); } public boolean containDataSourceKey (String key) { return dataSourceKeys.contains(key); } public boolean addDataSourceKeys (Collection<? extends Object> keys) { return dataSourceKeys.addAll(keys); } public Set<Object> all () { return dataSourceKeys; } }
DynamicRoutingDataSource
AbstractRoutingDataSource实现类,用于初始化数据源相关信息。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 public class DynamicRoutingDataSource extends AbstractRoutingDataSource { private static final Logger logger = LoggerFactory.getLogger(DynamicRoutingDataSource.class); @Override public DataSource determineTargetDataSource () { return super .determineTargetDataSource(); } public void setDynamicRoutingDataSource (DataSource defaultTargetDataSource, Map<Object, Object> targetDataSources) { super .setDefaultTargetDataSource(defaultTargetDataSource); super .setTargetDataSources(targetDataSources); super .afterPropertiesSet(); DynamicDataSourceContextHolder.getInstance().addDataSourceKeys(targetDataSources.keySet()); logger.info("setDynamicRoutingDataSource ... " ); } public boolean toggleDataSource (String key) { if (DynamicDataSourceContextHolder.getInstance().containDataSourceKey(key)) { String concurrentDataBase = DynamicDataSourceContextHolder.getInstance().getDataSourceKey(); DynamicDataSourceContextHolder.getInstance().setDataSourceKey(key); determineTargetDataSource(); logger.info("toggleDataSource {} -> {}" ,concurrentDataBase ,key); return true ; } return false ; } @Override protected Object determineCurrentLookupKey () { return DynamicDataSourceContextHolder.getInstance().getDataSourceKey(); } }
DynamicRoutingDataSourceConfig
初始化相关bean。这里需要将DynamicRoutingDataSource放到MybatisSqlSessionFactoryBean、PlatformTransactionManager中。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 @Configuration public class DynamicRoutingDataSourceConfig { private static final Logger logger = LoggerFactory.getLogger(DynamicRoutingDataSourceConfig.class); @Autowired ApplicationContext applicationContext; @Resource TenantInfoDao tenantInfoDao; @Autowired HikariDataSource hikariDataSource; @Bean(DynamicDataSourceConstant.MASTER) @ConfigurationProperties(prefix = "spring.datasource.hikari") public DataSource master () { return DataSourceBuilder.create().build(); } @Bean @ConditionalOnBean(name = {DynamicDataSourceConstant.MASTER}) public DynamicRoutingDataSource dynamicRoutingDataSource (@Qualifier(DynamicDataSourceConstant.MASTER) DataSource dataSource) { List<String> dataBasesList = new JdbcTemplate (dataSource).queryForList("SELECT DATABASE()" , String.class); if (CollectionUtils.isEmpty(dataBasesList)) { ClassPathResource classPathResource = new ClassPathResource ("create.sql" ); try { ScriptUtils.executeSqlScript(dataSource.getConnection(), classPathResource); } catch (SQLException e) { e.printStackTrace(); } } DynamicRoutingDataSource dynamicRoutingDataSource = new DynamicRoutingDataSource (); dynamicRoutingDataSource.setDynamicRoutingDataSource(dataSource, new HashMap <>()); return dynamicRoutingDataSource; } @PostConstruct @ConditionalOnBean(DynamicRoutingDataSource.class) public void initDynamicDataSource () throws Exception { logger.info("initDynamicDataSource ..." ); DynamicRoutingDataSource dynamicRoutingDataSource = applicationContext.getBean(DynamicRoutingDataSource.class); HikariDataSource master = (HikariDataSource) applicationContext.getBean(DynamicDataSourceConstant.MASTER); Map<Object, Object> dataSourceMap = new HashMap <>(); List<TenantInfo> tenantInfoList = tenantInfoDao.selectList(null ); tenantInfoList.stream().forEach(i -> { logger.info(" init dataSource {}" , i.getTenantName()); HikariConfig hikariConfig = new HikariConfig (); hikariConfig.setDriverClassName(i.getDatasourceDriver()); hikariConfig.setJdbcUrl(i.getDatasourceUrl()); hikariConfig.setUsername(i.getDatasourceUsername()); hikariConfig.setPassword(i.getDatasourcePassword()); hikariConfig.setMaximumPoolSize(20 ); hikariConfig.setMinimumIdle(2 ); hikariConfig.setIdleTimeout(600000 ); hikariConfig.setConnectionTimeout(30000 ); hikariConfig.setMaxLifetime(1800000 ); DataSource dataSource = new HikariDataSource (hikariConfig); dataSourceMap.put(i.getTenantId(), dataSource); } ); dynamicRoutingDataSource.setDynamicRoutingDataSource(master, dataSourceMap); } @Bean public MybatisSqlSessionFactoryBean sqlSessionFactoryBean (@Qualifier("dynamicRoutingDataSource") DynamicRoutingDataSource dynamicRoutingDataSource ) throws Exception { MybatisSqlSessionFactoryBean sessionFactory = new MybatisSqlSessionFactoryBean (); sessionFactory.setDataSource(dynamicRoutingDataSource); sessionFactory.setMapperLocations(new PathMatchingResourcePatternResolver ().getResources("classpath:mapper/*.xml" )); sessionFactory.setTypeAliasesPackage("cn.z201.dynamic.persistence.entity" ); MybatisConfiguration config = new MybatisConfiguration (); config.setMapUnderscoreToCamelCase(true ); config.setLogImpl(Slf4jImpl.class); sessionFactory.setConfiguration(config); return sessionFactory; } @Bean public PlatformTransactionManager transactionManager (@Qualifier("dynamicRoutingDataSource") DynamicRoutingDataSource dynamicRoutingDataSource) { return new DataSourceTransactionManager (dynamicRoutingDataSource); } @Bean public DynamicJdbcTemplateManager dynamicJdbcTemplateManager (@Qualifier("dynamicRoutingDataSource") DynamicRoutingDataSource dynamicRoutingDataSource) { return new DynamicJdbcTemplateManager (dynamicRoutingDataSource); } }
单元测试
测试前初始化一个默认数据库、租户表信息。
测试流程
插入租户表。插入测试的租户信息。
循环切换租户数据源。并输出对应数据源连接的db信息。
SQL 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 CREATE DATABASE IF NOT EXISTS `docker_dynamic_data` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; DROP TABLE IF EXISTS docker_dynamic_data.`tenant_info`;CREATE TABLE docker_dynamic_data.`tenant_info`( `id` BIGINT (20 ) NOT NULL AUTO_INCREMENT COMMENT '主键id' , `is_enable` bit(1 ) NOT NULL DEFAULT b'1' COMMENT '数据是否有效 1 有效 0 无效' , `create_time` bigint (20 ) unsigned NOT NULL COMMENT '创建时间' , `update_time` bigint (20 ) unsigned NOT NULL COMMENT '更新时间' , `tenant_id` VARCHAR (255 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL COMMENT '租户id' , `tenant_name` VARCHAR (255 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL COMMENT '租户名称' , `datasource_url` VARCHAR (255 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL COMMENT '数据源url' , `datasource_username` VARCHAR (255 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL COMMENT '数据源用户名' , `datasource_password` VARCHAR (255 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL COMMENT '数据源密码' , `datasource_driver` VARCHAR (255 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL COMMENT '数据源驱动' , `system_account` VARCHAR (255 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL COMMENT '系统账号' , `system_password` VARCHAR (255 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL COMMENT '账号密码' , `system_project` VARCHAR (255 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL COMMENT '系统PROJECT' , PRIMARY KEY (`id`) USING BTREE ) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci ROW_FORMAT = Dynamic ; CREATE UNIQUE INDEX tenant_id_unique on docker_dynamic_data.`tenant_info` (tenant_id);CREATE UNIQUE INDEX tenant_name_unique on docker_dynamic_data.`tenant_info` (tenant_name);
AppApplicationTest 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 @Slf4j @ExtendWith(SpringExtension.class) @SpringBootTest(classes = AppApplication.class, webEnvironment = SpringBootTest.WebEnvironment.NONE) @AutoConfigureMockMvc @TestMethodOrder(MethodOrderer.OrderAnnotation.class) public class AppApplicationTest { @Autowired private ApplicationContext applicationContext; @Resource private TenantInfoDao tenantInfoDao; @Autowired private DynamicJdbcTemplateManager jdbcTemplate; @BeforeEach public void before () { log.info("before" ); List<TenantInfo> tenantInfoList = tenantInfoDao.selectList(null ); if (CollectionUtils.isEmpty(tenantInfoList)) { TenantInfo tenantInfo = new TenantInfo (); int i = 3 ; for (int j = 0 ; j < i; j++) { String sql = "CREATE DATABASE IF NOT EXISTS `docker_dynamic_data_${db}` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" .replace("${db}" , String.valueOf(j)); log.info("sql {} \n" , sql); tenantInfo.setTenantId(String.valueOf(SnowflakeTool.getInstance().nextId())); tenantInfo.setTenantName(Base64.getEncoder().encodeToString(tenantInfo.getTenantId().getBytes())); tenantInfo.setDatasourceUsername("root" ); tenantInfo.setDatasourcePassword("123456" ); tenantInfo.setDatasourceDriver("com.mysql.cj.jdbc.Driver" ); tenantInfo.setDatasourceUrl(DynamicRoutingDataSourceTool.buildDataBase("docker_dynamic_data_" + j)); tenantInfoDao.insert(tenantInfo); } } } @AfterEach public void after () { log.info("after" ); } @Test public void setUp () { log.info("bean Count {}" , applicationContext.getBeanDefinitionCount()); lookupCollectionType(applicationContext); DynamicRoutingDataSource dynamicRoutingDataSource = applicationContext.getBean(DynamicRoutingDataSource.class); Set<Object> dataSourceKeys = DynamicDataSourceContextHolder.getInstance().all(); log.info("dataSourceKey All {}" , dataSourceKeys); for (Object dataSourceKey : dataSourceKeys) { dynamicRoutingDataSource.toggleDataSource(dataSourceKey.toString()); concurrentDataBase(); } } private void concurrentDataBase () { List<String> dataBasesList = jdbcTemplate.dynamicJdbcTemplate().queryForList("SELECT DATABASE();" , String.class); log.info("concurrentDataBase {}" , dataBasesList); } private void lookupCollectionType (BeanFactory beanFactory) { if (beanFactory instanceof ListableBeanFactory) { ListableBeanFactory listableBeanFactory = (ListableBeanFactory) beanFactory; Map<String, DataSource> beansMap = listableBeanFactory.getBeansOfType(DataSource.class); beansMap.forEach((key, value) -> { System.out.println("单一类型集合查找 " + key + " " + value); }); } } }
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 [main] [AppApplicationTest.java : 53 ] before [main] [BaseJdbcLogger.java : 137 ] ==> Preparing: SELECT id,tenant_id,tenant_name,datasource_url,datasource_username,datasource_password,datasource_driver,system_account,system_password,system_project,is_enable,create_time,update_time FROM tenant_info [main] [BaseJdbcLogger.java : 137 ] ==> Parameters: [main] [BaseJdbcLogger.java : 137 ] <== Total: 3 [main] [AppApplicationTest.java : 80 ] bean Count 96 单一类型集合查找 master HikariDataSource (HikariPool-1 ) 单一类型集合查找 dynamicRoutingDataSource cn.z201.dynamic.dynamic.DynamicRoutingDataSource@63cd2cd2 [main] [AppApplicationTest.java : 84 ] dataSourceKey All [942592207909552128 , 942592207829860352 , 942592207959883776 ] [main] [DynamicRoutingDataSource.java : 46 ] toggleDataSource master -> 942592207909552128 [main] [AppApplicationTest.java : 95 ] concurrentDataBase [docker_dynamic_data_1] [main] [DynamicRoutingDataSource.java : 46 ] toggleDataSource 942592207909552128 -> 942592207829860352 [main] [AppApplicationTest.java : 95 ] concurrentDataBase [docker_dynamic_data_0] [main] [DynamicRoutingDataSource.java : 46 ] toggleDataSource 942592207829860352 -> 942592207959883776 [main] [AppApplicationTest.java : 95 ] concurrentDataBase [docker_dynamic_data_2] [main] [AppApplicationTest.java : 75 ] after
END