服务拆分之基础设施拆分

Infrastructure unbundling of services

背景: 因历史原因, 前期多个服务共用一个 rds 实例和一个 Redis 实例, 在实际使用中经常会因某一个服务异常导致 rds 或 Redis 负载异常, 进而影响其他服务造成雪崩. 故进行基础资源拆分来隔离风险.

Background : Due to historical reasons, multiple services share one RDS instance and one Redis instance in the early stage. In actual use, it is common to cause RDS or Redis load abnormalities due to one service exception, thus affecting other services and causing avalanches. Therefore, base resource splitting is performed to isolate risks.

本次拆分基于 AWS 平台

The split is based on AWS

创建原实例的只读副本实例

Create a read-only copy instance of the original instance

原理

principle
AWS console-> rds console-> databases -> select rds-> action-> creat read replica

将 k8s 上 pod 副本数调为 0

Set the number of pod replicas on K8S to 0
kubectl edit deployments test -n test
set spec.replicas: 0

验证只读实例和原实例记录行数是否相同

Verify that the read-only instance and the original instance record the same number of rows

不建议用 schema 统计 有误差

It is not recommended to use schema statistics with errors

information_schema.tables 对于 InnoDB 表, table_rows 行计数仅是大概估计值, 不准确.

information_schema.tables For InnoDB tables, table_rows rows count is only approximate and not accurate.

用下面 sql 生成查询库下所有表的记录行数:

Use the following SQL to generate the number of record rows for all tables under the query library:
select concat(
    'select  count(*) from',
    TABLE_SCHEMA,
    '.',
    TABLE_name,
    'union all'
) from information_schema.tables
where TABLE_SCHEMA='test_db';

生成类似如下的查询 sql, 在原实例和只读实例进行查询

generate query SQL similar to the following for the original and read-only instances
select count(*) from table1
union all
select count(*) from table2
union all
select count(*) from table3
...

将只读 rds 实例提升为正常 rds 实例

Promote a read-only RDS instance to a normal RDS instance

待只读实例和源实例一致后将只读实例提升为正常实例

Promote a read-only instance to a normal instance after it is consistent with the source instance
AWS console-> rds console-> databases -> select rds-> action-> promote

创建 Redis 备份, 并恢复一个新的 Redis

Create a Redis backup and restore a new Redis
AWS console->Redis->action->backup
AWS console->Redis->resotre

修改配置中心连接信息

Example Modify the connection information of the configuration center
datasource.host
Redis.host
...

将 k8s pod 副本数恢复之前数量

Restore the number of k8S pod copies to previous number
kubectl edit deployments test -n test
set spec.replicas: x

将 AWS 拆出来的资源 rds Redis 导入到现有 terraform 中

Import the resource RDS Redis from AWS into the existing Terraform

参考如下

Refer to the following

Terraform 反向导出

总结

to summarize

本次拆分可以保证数据 0 损失, 因进行了 k8s pod 副本数调整, 会对对拆分的服务根据实际情况会有部分时间不可用, 建议在服务访问量低时进行此操作

This split can ensure zero data loss. Because the number of K8S POD copies is adjusted, the split service may be unavailable for some time according to the actual situation. You are advised to perform this operation when the service traffic is low

来源: https://www.qcloud.com/developer/article/1924903