在数据驱动的时代,数据库间的数据同步至关重要。无论是构建数据仓库、实现读写分离、灾备恢复,还是微服务间的数据共享,增量同步与全量同步都是核心策略。理解它们的原理、差异和适用场景,是设计健壮数据流的关键。
查看 mysqld
sh which mysqld
查看 my.cnf 文件路径
sh/usr/sbin/mysqld --verbose --help |grep -A 1 'Default options'
对于自建 MySQL , 需要先开启 Binlog 写入功能,配置 binlog-format 为 ROW 模式,my.cnf 中配置如下
sh[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义,不要和 canal 的 slaveId 重复
授权 canal 链接 MySQL 账号具有作为 MySQL slave 的权限, 如果已有账户可直接 grant
shCREATE USER canal IDENTIFIED BY 'canal';
grant system_user on *.* to "root"; //MySQL8版本中新增了一个system_user帐户类型,由于root用户没有SYSTEM_USER权限,导致错误出现。
GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
ALTER USER 'canal'@'%' IDENTIFIED WITH mysql_native_password BY 'canal';
FLUSH PRIVILEGES;
shsystemctl restart mysqld.service
下载: 访问 release 页面 , 选择需要的包下载, 如以 v1.1.5版本为例
shwget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz
解压缩
shmkdir canal
tar -zxvf canal.deployer-1.1.5.tar.gz -C ./canal
结构:
参考全量导出数据 MASTER_LOG_POS
进入conf目录,创建目录msb-edu ,并将example中instance.properties拷贝到对应的msb-edu目录下
shvi conf/ msb-edu/instance.properties
调整 instance.properties 配置
properties## mysql serverId canal.instance.mysql.slaveId = 1234 # enable gtid use true/false canal.instance.gtidon=false #position info,需要改成自己的数据库信息 canal.instance.master.address = 127.0.0.1:3306 canal.instance.master.journal.name = #指定binlog的消费位点,这个位点可以再数据库导出数据的文件中获取 canal.instance.master.position = canal.instance.master.timestamp = #canal.instance.standby.address = #canal.instance.standby.journal.name = #canal.instance.standby.position = #canal.instance.standby.timestamp = #username/password,需要改成自己的数据库信息 canal.instance.dbUsername = canal canal.instance.dbPassword = canal canal.instance.defaultDatabaseName = canal.instance.connectionCharset = UTF-8 #table regex canal.instance.filter.regex = .\*\\\\..\*
canal.instance.connectionCharset 代表数据库的编码方式对应到 java 中的编码类型,比如 UTF-8,GBK , ISO-8859-1
如果系统是1个 cpu,需要将 canal.instance.parser.parallel 设置为 false
shsh bin/startup.sh
shtail -100f logs/canal/canal.log
shtail -100f logs/ msb-edu/ msb-edu.log
shsh bin/stop.sh
sh#开放指定端口
firewall-cmd --zone=public --add-port=11111/tcp --permanent
#关闭指定端口
firewall-cmd --zone=public --remove-port=3306/tcp --permanent
#重启防火墙
firewall-cmd --reload
引入依赖
xml<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>top.javatool</groupId>
<artifactId>canal-spring-boot-starter</artifactId>
<version>1.2.1-RELEASE</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<version>1.1.22</version>
</dependency>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>sharding-jdbc-spring-boot-starter</artifactId>
<version>4.1.1</version>
</dependency>
<dependency>
<groupId>com.baomidou</groupId>
<artifactId>mybatis-plus-boot-starter</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<!-- 代码自动生成器依赖-->
<dependency>
<groupId>com.baomidou</groupId>
<artifactId>mybatis-plus-generator</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.velocity</groupId>
<artifactId>velocity-engine-core</artifactId>
<version>2.3</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
generate创建代码
java@SpringBootApplication
public class CodeGeneratorApplication {
public static void main(String[] args) {
// 代码生成器配置
AutoGenerator generator = new AutoGenerator();
// 配置数据源
generator.setDataSource(new DataSourceConfig()
.setUrl("jdbc:mysql://localhost:3306/msb-edu")
.setDriverName("com.mysql.cj.jdbc.Driver")
.setUsername("root")
.setPassword("root"));
// 配置包名
generator.setPackageInfo(new PackageConfig()
.setParent("com.msb.transmit")
.setEntity("entity")
.setMapper("mapper")
.setService("service")
.setController("controller"));
// 配置策略
generator.setStrategy(new StrategyConfig()
.setNaming(NamingStrategy.underline_to_camel)
.setColumnNaming(NamingStrategy.underline_to_camel)
.setInclude("edu_order")
.setEntityLombokModel(true)
.setRestControllerStyle(true)
.setControllerMappingHyphenStyle(true));
// 执行生成代码
generator.execute();
}
}
引入配置
javacanal.server=192.168.139.101:11111
canal.destination=msb-edu
spring.shardingsphere.datasource.names=m0,m1,m2,m3
spring.shardingsphere.datasource.m0.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m0.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m0.url=jdbc:mysql://192.168.139.102:3306/msb-edu-0?serverTimezone=Asia/Shanghai
spring.shardingsphere.datasource.m0.username=root
spring.shardingsphere.datasource.m0.password=123456
spring.shardingsphere.datasource.m1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m1.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m1.url=jdbc:mysql://192.168.139.102:3306/msb-edu-1?serverTimezone=Asia/Shanghai
spring.shardingsphere.datasource.m1.username=root
spring.shardingsphere.datasource.m1.password=123456
spring.shardingsphere.datasource.m2.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m2.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m2.url=jdbc:mysql://192.168.139.102:3306/msb-edu-2?serverTimezone=Asia/Shanghai
spring.shardingsphere.datasource.m2.username=root
spring.shardingsphere.datasource.m2.password=123456
spring.shardingsphere.datasource.m3.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m3.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m3.url=jdbc:mysql://192.168.139.102:3306/msb-edu-3?serverTimezone=Asia/Shanghai
spring.shardingsphere.datasource.m3.username=root
spring.shardingsphere.datasource.m3.password=123456
#真实表 分布 groovy语法 m1.edu_order_ 表示一个常量 $->{1..2} 标识一个计算符 整体标识 m1.edu_order_1 m1.edu_order_2
spring.shardingsphere.sharding.tables.edu_order.actual-data-nodes=m$->{0..3}.edu_order_$->{0..7}
#分表策略
spring.shardingsphere.sharding.tables.edu_order.table-strategy.inline.sharding-column=user_id
spring.shardingsphere.sharding.tables.edu_order.table-strategy.inline.algorithm-expression=edu_order_$->{user_id % 8}
#分库策略
spring.shardingsphere.sharding.tables.edu_order.database-strategy.inline.sharding-column=user_id
spring.shardingsphere.sharding.tables.edu_order.database-strategy.inline.algorithm-expression=m$->{(int)(user_id / 8)%4}
canal同步数据
java@Slf4j
@Component
@CanalTable(value = "edu_order")
public class EduOrderHandler implements EntryHandler<EduOrderCanal> {
@Autowired
private EduOrderMapper eduOrderMapper;
@Override
public void insert(EduOrderCanal eduOrderCanal) {
log.info("新增数据:{}", eduOrderCanal);
EduOrder eduOrder = covertEduOrder(eduOrderCanal);
eduOrderMapper.insert(eduOrder);
}
private EduOrder covertEduOrder(EduOrderCanal eduOrderCanal) {
EduOrder eduOrder = new EduOrder();
BeanUtils.copyProperties(eduOrderCanal, eduOrder);
eduOrder.setCouponAmount(eduOrderCanal.getCoupon_amount());
eduOrder.setTotalAmount(eduOrderCanal.getTotal_amount());
eduOrder.setCreateTime(eduOrderCanal.getCreate_time());
eduOrder.setUpdateTime(eduOrderCanal.getUpdate_time());
eduOrder.setIsDeleted(eduOrderCanal.getIs_deleted());
eduOrder.setPayAmount(eduOrderCanal.getPay_amount());
eduOrder.setSourceType(eduOrderCanal.getSource_type());
eduOrder.setUserId(new Long(eduOrderCanal.getUser_id()));
eduOrder.setId(new Long(eduOrderCanal.getId()));
return eduOrder;
}
@Override
public void update(EduOrderCanal before, EduOrderCanal after) {
log.info("更新数据: before:{} after:{}", before, after);
eduOrderMapper.updateById(covertEduOrder(after));
}
}
同步实体
java@Data
public class EduOrder implements Serializable {
private static final long serialVersionUID = 1L;
/**
* 订单id
*/
private Long id;
/**
* 订单标题
*/
private String title;
/**
* 用户编号
*/
private Long userId;
/**
* 订单总金额
*/
private BigDecimal totalAmount;
/**
* 优惠金额
*/
private BigDecimal couponAmount;
/**
* 实付金额
*/
private BigDecimal payAmount;
/**
* 订单来源(1:pc,2:app,3:小程序)
*/
private Integer sourceType;
/**
* 订单状态(0:免费,1:待支付,2:已支付,3:取消,4:已退款)
*/
private Integer status;
/**
* 是否删除(0-:否,1:是)
*/
private Integer isDeleted;
/**
* 更新时间
*/
private String updateTime;
/**
* 下单时间
*/
private String createTime;
}
java@Data
public class EduOrderCanal implements Serializable {
private static final long serialVersionUID = 1L;
/**
* 订单id
*/
private String id;
/**
* 订单标题
*/
private String title;
/**
* 用户编号
*/
private String user_id;
/**
* 订单总金额
*/
private BigDecimal total_amount;
/**
* 优惠金额
*/
private BigDecimal coupon_amount;
/**
* 实付金额
*/
private BigDecimal pay_amount;
/**
* 订单来源(1:pc,2:app,3:小程序)
*/
private Integer source_type;
/**
* 订单状态(0:免费,1:待支付,2:已支付,3:取消,4:已退款)
*/
private Integer status;
/**
* 是否删除(0-:否,1:是)
*/
private Integer is_deleted;
/**
* 更新时间
*/
private String update_time;
/**
* 下单时间
*/
private String create_time;
}
全量导出
shmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --triggers --routines --events >/opt/edu-order.sql
数据传输
shellscp -r /opt/edu-order.sql root@192.168.139.102:/opt/
导入对应的主库表中
mysqlmysql> use msb-edu; Database changed mysql> source /opt/edu-order.sql
每个库中有8张表,每个表中暂时存放的全数据。
根据规则:
库msb-edu-0:
表 | 数据规则 |
---|---|
edu_order_0 | 求余 0 |
edu_order-1 | 求余 1 |
edu_order-2 | 求余 2 |
edu_order-3 | 求余 3 |
edu_order-4 | 求余 4 |
edu_order-5 | 求余 5 |
edu_order-6 | 求余 6 |
edu_order-7 | 求余 7 |
sqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 0" --triggers --routines --events >/opt/edu-order-0.sql
sed -i 's/edu_order/edu_order_0/g' /opt/edu-order-0.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 1" --triggers --routines --events >/opt/edu-order-1.sql sed -i 's/edu_order/edu_order_1/g' /opt/edu-order-1.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 2" --triggers --routines --events >/opt/edu-order-2.sql sed -i 's/edu_order/edu_order_2/g' /opt/edu-order-2.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 3" --triggers --routines --events >/opt/edu-order-3.sql sed -i 's/edu_order/edu_order_3/g' /opt/edu-order-3.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 4" --triggers --routines --events >/opt/edu-order-4.sql sed -i 's/edu_order/edu_order_4/g' /opt/edu-order-4.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 5" --triggers --routines --events >/opt/edu-order-5.sql sed -i 's/edu_order/edu_order_5/g' /opt/edu-order-5.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 6" --triggers --routines --events >/opt/edu-order-6.sql sed -i 's/edu_order/edu_order_6/g' /opt/edu-order-6.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 7" --triggers --routines --events >/opt/edu-order-7.sql sed -i 's/edu_order/edu_order_7/g' /opt/edu-order-7.sql
登录mysql
mysqlmysql -uroot -p
导入数据
mysqlmysql> use msb-edu-0; Database changed mysql> source /opt/edu-order-0.sql mysql> source /opt/edu-order-1.sql mysql> source /opt/edu-order-2.sql mysql> source /opt/edu-order-3.sql mysql> source /opt/edu-order-4.sql mysql> source /opt/edu-order-5.sql mysql> source /opt/edu-order-6.sql mysql> source /opt/edu-order-7.sql
库msb-edu-1:
表 | 数据规则 |
---|---|
edu_order-0 | 求余 8 |
edu_order-1 | 求余 9 |
edu_order-2 | 求余 10 |
edu_order-3 | 求余 11 |
edu_order-4 | 求余 12 |
edu_order-5 | 求余 13 |
edu_order-6 | 求余 14 |
edu_order-7 | 求余 15 |
sqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 8" --triggers --routines --events >/opt/edu-order-0.sql
sed -i 's/edu_order/edu_order_0/g' /opt/edu-order-0.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 9" --triggers --routines --events >/opt/edu-order-1.sql sed -i 's/edu_order/edu_order_1/g' /opt/edu-order-1.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 10" --triggers --routines --events >/opt/edu-order-2.sql sed -i 's/edu_order/edu_order_2/g' /opt/edu-order-2.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 11" --triggers --routines --events >/opt/edu-order-3.sql sed -i 's/edu_order/edu_order_3/g' /opt/edu-order-3.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 12" --triggers --routines --events >/opt/edu-order-4.sql sed -i 's/edu_order/edu_order_4/g' /opt/edu-order-4.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 13" --triggers --routines --events >/opt/edu-order-5.sql sed -i 's/edu_order/edu_order_5/g' /opt/edu-order-5.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 14" --triggers --routines --events >/opt/edu-order-6.sql sed -i 's/edu_order/edu_order_6/g' /opt/edu-order-6.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 15" --triggers --routines --events >/opt/edu-order-7.sql sed -i 's/edu_order/edu_order_7/g' /opt/edu-order-7.sql
登录mysql
mysqlmysql -uroot -p
导入数据
mysqlmysql> use msb-edu-1; Database changed mysql> source /opt/edu-order-0.sql mysql> source /opt/edu-order-1.sql mysql> source /opt/edu-order-2.sql mysql> source /opt/edu-order-3.sql mysql> source /opt/edu-order-4.sql mysql> source /opt/edu-order-5.sql mysql> source /opt/edu-order-6.sql mysql> source /opt/edu-order-7.sql
库msb-edu-2:
表 | 数据规则 |
---|---|
edu_order-0 | 求余 16 |
edu_order-1 | 求余 17 |
edu_order-2 | 求余 18 |
edu_order-3 | 求余 19 |
edu_order-4 | 求余 20 |
edu_order-5 | 求余 21 |
edu_order-6 | 求余 22 |
edu_order-7 | 求余 23 |
sqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 16" --triggers --routines --events >/opt/edu-order-0.sql
sed -i 's/edu_order/edu_order_0/g' /opt/edu-order-0.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 17" --triggers --routines --events >/opt/edu-order-1.sql sed -i 's/edu_order/edu_order_1/g' /opt/edu-order-1.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 18" --triggers --routines --events >/opt/edu-order-2.sql sed -i 's/edu_order/edu_order_2/g' /opt/edu-order-2.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 19" --triggers --routines --events >/opt/edu-order-3.sql sed -i 's/edu_order/edu_order_3/g' /opt/edu-order-3.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 20" --triggers --routines --events >/opt/edu-order-4.sql sed -i 's/edu_order/edu_order_4/g' /opt/edu-order-4.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 21" --triggers --routines --events >/opt/edu-order-5.sql sed -i 's/edu_order/edu_order_5/g' /opt/edu-order-5.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 22" --triggers --routines --events >/opt/edu-order-6.sql sed -i 's/edu_order/edu_order_6/g' /opt/edu-order-6.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 23" --triggers --routines --events >/opt/edu-order-7.sql sed -i 's/edu_order/edu_order_7/g' /opt/edu-order-7.sql
登录mysql
mysqlmysql -uroot -p
导入数据
mysqlmysql> use msb-edu-2; Database changed mysql> source /opt/edu-order-0.sql mysql> source /opt/edu-order-1.sql mysql> source /opt/edu-order-2.sql mysql> source /opt/edu-order-3.sql mysql> source /opt/edu-order-4.sql mysql> source /opt/edu-order-5.sql mysql> source /opt/edu-order-6.sql mysql> source /opt/edu-order-7.sql
库msb-edu-3:
表 | 数据规则 |
---|---|
edu_order-0 | 求余 24 |
edu_order-1 | 求余 25 |
edu_order-2 | 求余 26 |
edu_order-3 | 求余 27 |
edu_order-4 | 求余 28 |
edu_order-5 | 求余 29 |
edu_order-6 | 求余 30 |
edu_order-7 | 求余 31 |
sqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 24" --triggers --routines --events >/opt/edu-order-0.sql
sed -i 's/edu_order/edu_order_0/g' /opt/edu-order-0.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 25" --triggers --routines --events >/opt/edu-order-1.sql sed -i 's/edu_order/edu_order_1/g' /opt/edu-order-1.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 26" --triggers --routines --events >/opt/edu-order-2.sql sed -i 's/edu_order/edu_order_2/g' /opt/edu-order-2.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 27" --triggers --routines --events >/opt/edu-order-3.sql sed -i 's/edu_order/edu_order_3/g' /opt/edu-order-3.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 28" --triggers --routines --events >/opt/edu-order-4.sql sed -i 's/edu_order/edu_order_4/g' /opt/edu-order-4.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 29" --triggers --routines --events >/opt/edu-order-5.sql sed -i 's/edu_order/edu_order_5/g' /opt/edu-order-5.sql
mysqlmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 30" --triggers --routines --events >/opt/edu-order-6.sql sed -i 's/edu_order/edu_order_6/g' /opt/edu-order-6.sql
shmysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 31" --triggers --routines --events >/opt/edu-order-7.sql
sed -i 's/edu_order/edu_order_7/g' /opt/edu-order-7.sql
sql-- 查询某个库中某个表数据量
SELECT TABLE_NAME, TABLE_ROWS
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA ='your_database_name';
本文作者:柳始恭
本文链接:
版权声明:本博客所有文章除特别声明外,均采用 BY-NC-SA 许可协议。转载请注明出处!