2025-06-17
数据库
0

目录

增量同步
修改 Mysql 配置
安装 canal
配置修改
启动
查看 server 日志
查看 instance 的日志
关闭
5.3 创建项目
全量同步

在数据驱动的时代,数据库间的数据同步至关重要。无论是构建数据仓库、实现读写分离、灾备恢复,还是微服务间的数据共享,增量同步与全量同步都是核心策略。理解它们的原理、差异和适用场景,是设计健壮数据流的关键。

增量同步

修改 Mysql 配置

查看 mysqld

sh
which mysqld

查看 my.cnf 文件路径

sh
/usr/sbin/mysqld --verbose --help |grep -A 1 'Default options'

image.png

对于自建 MySQL , 需要先开启 Binlog 写入功能,配置 binlog-format 为 ROW 模式,my.cnf 中配置如下

sh
[mysqld] log-bin=mysql-bin # 开启 binlog binlog-format=ROW # 选择 ROW 模式 server_id=1 # 配置 MySQL replaction 需要定义,不要和 canal 的 slaveId 重复

授权 canal 链接 MySQL 账号具有作为 MySQL slave 的权限, 如果已有账户可直接 grant

sh
CREATE USER canal IDENTIFIED BY 'canal'; grant system_user on *.* to "root"; //MySQL8版本中新增了一个system_user帐户类型,由于root用户没有SYSTEM_USER权限,导致错误出现。 GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ; ALTER USER 'canal'@'%' IDENTIFIED WITH mysql_native_password BY 'canal'; FLUSH PRIVILEGES;
  • 重启我们的mysql
sh
systemctl restart mysqld.service

安装 canal

下载: 访问 release 页面 , 选择需要的包下载, 如以 v1.1.5版本为例

sh
wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz

解压缩

sh
mkdir canal tar -zxvf canal.deployer-1.1.5.tar.gz -C ./canal

结构:

image.png

配置修改

参考全量导出数据 MASTER_LOG_POS

image.png

进入conf目录,创建目录msb-edu ,并将example中instance.properties拷贝到对应的msb-edu目录下

image.png

sh
vi conf/ msb-edu/instance.properties

调整 instance.properties 配置

properties
## mysql serverId canal.instance.mysql.slaveId = 1234 # enable gtid use true/false canal.instance.gtidon=false #position info,需要改成自己的数据库信息 canal.instance.master.address = 127.0.0.1:3306 canal.instance.master.journal.name = #指定binlog的消费位点,这个位点可以再数据库导出数据的文件中获取 canal.instance.master.position = canal.instance.master.timestamp = #canal.instance.standby.address = #canal.instance.standby.journal.name = #canal.instance.standby.position = #canal.instance.standby.timestamp = #username/password,需要改成自己的数据库信息 canal.instance.dbUsername = canal canal.instance.dbPassword = canal canal.instance.defaultDatabaseName = canal.instance.connectionCharset = UTF-8 #table regex canal.instance.filter.regex = .\*\\\\..\*

canal.instance.connectionCharset 代表数据库的编码方式对应到 java 中的编码类型,比如 UTF-8,GBK , ISO-8859-1

如果系统是1个 cpu,需要将 canal.instance.parser.parallel 设置为 false

启动

sh
sh bin/startup.sh

查看 server 日志

sh
tail -100f logs/canal/canal.log

image.png

查看 instance 的日志

sh
tail -100f logs/ msb-edu/ msb-edu.log

image.png

关闭

sh
sh bin/stop.sh
sh
#开放指定端口 firewall-cmd --zone=public --add-port=11111/tcp --permanent #关闭指定端口 firewall-cmd --zone=public --remove-port=3306/tcp --permanent #重启防火墙 firewall-cmd --reload

5.3 创建项目

引入依赖

xml
<dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>top.javatool</groupId> <artifactId>canal-spring-boot-starter</artifactId> <version>1.2.1-RELEASE</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>druid</artifactId> <version>1.1.22</version> </dependency> <dependency> <groupId>org.apache.shardingsphere</groupId> <artifactId>sharding-jdbc-spring-boot-starter</artifactId> <version>4.1.1</version> </dependency> <dependency> <groupId>com.baomidou</groupId> <artifactId>mybatis-plus-boot-starter</artifactId> <version>3.2.0</version> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <scope>runtime</scope> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency> <!-- 代码自动生成器依赖--> <dependency> <groupId>com.baomidou</groupId> <artifactId>mybatis-plus-generator</artifactId> <version>3.2.0</version> </dependency> <dependency> <groupId>org.apache.velocity</groupId> <artifactId>velocity-engine-core</artifactId> <version>2.3</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies>

generate创建代码

java
@SpringBootApplication public class CodeGeneratorApplication { public static void main(String[] args) { // 代码生成器配置 AutoGenerator generator = new AutoGenerator(); // 配置数据源 generator.setDataSource(new DataSourceConfig() .setUrl("jdbc:mysql://localhost:3306/msb-edu") .setDriverName("com.mysql.cj.jdbc.Driver") .setUsername("root") .setPassword("root")); // 配置包名 generator.setPackageInfo(new PackageConfig() .setParent("com.msb.transmit") .setEntity("entity") .setMapper("mapper") .setService("service") .setController("controller")); // 配置策略 generator.setStrategy(new StrategyConfig() .setNaming(NamingStrategy.underline_to_camel) .setColumnNaming(NamingStrategy.underline_to_camel) .setInclude("edu_order") .setEntityLombokModel(true) .setRestControllerStyle(true) .setControllerMappingHyphenStyle(true)); // 执行生成代码 generator.execute(); } }

引入配置

java
canal.server=192.168.139.101:11111 canal.destination=msb-edu spring.shardingsphere.datasource.names=m0,m1,m2,m3 spring.shardingsphere.datasource.m0.type=com.alibaba.druid.pool.DruidDataSource spring.shardingsphere.datasource.m0.driver-class-name=com.mysql.cj.jdbc.Driver spring.shardingsphere.datasource.m0.url=jdbc:mysql://192.168.139.102:3306/msb-edu-0?serverTimezone=Asia/Shanghai spring.shardingsphere.datasource.m0.username=root spring.shardingsphere.datasource.m0.password=123456 spring.shardingsphere.datasource.m1.type=com.alibaba.druid.pool.DruidDataSource spring.shardingsphere.datasource.m1.driver-class-name=com.mysql.cj.jdbc.Driver spring.shardingsphere.datasource.m1.url=jdbc:mysql://192.168.139.102:3306/msb-edu-1?serverTimezone=Asia/Shanghai spring.shardingsphere.datasource.m1.username=root spring.shardingsphere.datasource.m1.password=123456 spring.shardingsphere.datasource.m2.type=com.alibaba.druid.pool.DruidDataSource spring.shardingsphere.datasource.m2.driver-class-name=com.mysql.cj.jdbc.Driver spring.shardingsphere.datasource.m2.url=jdbc:mysql://192.168.139.102:3306/msb-edu-2?serverTimezone=Asia/Shanghai spring.shardingsphere.datasource.m2.username=root spring.shardingsphere.datasource.m2.password=123456 spring.shardingsphere.datasource.m3.type=com.alibaba.druid.pool.DruidDataSource spring.shardingsphere.datasource.m3.driver-class-name=com.mysql.cj.jdbc.Driver spring.shardingsphere.datasource.m3.url=jdbc:mysql://192.168.139.102:3306/msb-edu-3?serverTimezone=Asia/Shanghai spring.shardingsphere.datasource.m3.username=root spring.shardingsphere.datasource.m3.password=123456 #真实表 分布 groovy语法 m1.edu_order_ 表示一个常量 $->{1..2} 标识一个计算符 整体标识 m1.edu_order_1 m1.edu_order_2 spring.shardingsphere.sharding.tables.edu_order.actual-data-nodes=m$->{0..3}.edu_order_$->{0..7} #分表策略 spring.shardingsphere.sharding.tables.edu_order.table-strategy.inline.sharding-column=user_id spring.shardingsphere.sharding.tables.edu_order.table-strategy.inline.algorithm-expression=edu_order_$->{user_id % 8} #分库策略 spring.shardingsphere.sharding.tables.edu_order.database-strategy.inline.sharding-column=user_id spring.shardingsphere.sharding.tables.edu_order.database-strategy.inline.algorithm-expression=m$->{(int)(user_id / 8)%4}

canal同步数据

java
@Slf4j @Component @CanalTable(value = "edu_order") public class EduOrderHandler implements EntryHandler<EduOrderCanal> { @Autowired private EduOrderMapper eduOrderMapper; @Override public void insert(EduOrderCanal eduOrderCanal) { log.info("新增数据:{}", eduOrderCanal); EduOrder eduOrder = covertEduOrder(eduOrderCanal); eduOrderMapper.insert(eduOrder); } private EduOrder covertEduOrder(EduOrderCanal eduOrderCanal) { EduOrder eduOrder = new EduOrder(); BeanUtils.copyProperties(eduOrderCanal, eduOrder); eduOrder.setCouponAmount(eduOrderCanal.getCoupon_amount()); eduOrder.setTotalAmount(eduOrderCanal.getTotal_amount()); eduOrder.setCreateTime(eduOrderCanal.getCreate_time()); eduOrder.setUpdateTime(eduOrderCanal.getUpdate_time()); eduOrder.setIsDeleted(eduOrderCanal.getIs_deleted()); eduOrder.setPayAmount(eduOrderCanal.getPay_amount()); eduOrder.setSourceType(eduOrderCanal.getSource_type()); eduOrder.setUserId(new Long(eduOrderCanal.getUser_id())); eduOrder.setId(new Long(eduOrderCanal.getId())); return eduOrder; } @Override public void update(EduOrderCanal before, EduOrderCanal after) { log.info("更新数据: before:{} after:{}", before, after); eduOrderMapper.updateById(covertEduOrder(after)); } }

同步实体

java
@Data public class EduOrder implements Serializable { private static final long serialVersionUID = 1L; /** * 订单id */ private Long id; /** * 订单标题 */ private String title; /** * 用户编号 */ private Long userId; /** * 订单总金额 */ private BigDecimal totalAmount; /** * 优惠金额 */ private BigDecimal couponAmount; /** * 实付金额 */ private BigDecimal payAmount; /** * 订单来源(1:pc,2:app,3:小程序) */ private Integer sourceType; /** * 订单状态(0:免费,1:待支付,2:已支付,3:取消,4:已退款) */ private Integer status; /** * 是否删除(0-:否,1:是) */ private Integer isDeleted; /** * 更新时间 */ private String updateTime; /** * 下单时间 */ private String createTime; }
java
@Data public class EduOrderCanal implements Serializable { private static final long serialVersionUID = 1L; /** * 订单id */ private String id; /** * 订单标题 */ private String title; /** * 用户编号 */ private String user_id; /** * 订单总金额 */ private BigDecimal total_amount; /** * 优惠金额 */ private BigDecimal coupon_amount; /** * 实付金额 */ private BigDecimal pay_amount; /** * 订单来源(1:pc,2:app,3:小程序) */ private Integer source_type; /** * 订单状态(0:免费,1:待支付,2:已支付,3:取消,4:已退款) */ private Integer status; /** * 是否删除(0-:否,1:是) */ private Integer is_deleted; /** * 更新时间 */ private String update_time; /** * 下单时间 */ private String create_time; }

全量同步

全量导出

sh
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --triggers --routines --events >/opt/edu-order.sql

image.png

数据传输

shell
scp -r /opt/edu-order.sql root@192.168.139.102:/opt/

导入对应的主库表中

mysql
mysql> use msb-edu; Database changed mysql> source /opt/edu-order.sql

每个库中有8张表,每个表中暂时存放的全数据。

根据规则:

库msb-edu-0:

数据规则
edu_order_0求余 0
edu_order-1求余 1
edu_order-2求余 2
edu_order-3求余 3
edu_order-4求余 4
edu_order-5求余 5
edu_order-6求余 6
edu_order-7求余 7
sql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 0" --triggers --routines --events >/opt/edu-order-0.sql sed -i 's/edu_order/edu_order_0/g' /opt/edu-order-0.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 1" --triggers --routines --events >/opt/edu-order-1.sql sed -i 's/edu_order/edu_order_1/g' /opt/edu-order-1.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 2" --triggers --routines --events >/opt/edu-order-2.sql sed -i 's/edu_order/edu_order_2/g' /opt/edu-order-2.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 3" --triggers --routines --events >/opt/edu-order-3.sql sed -i 's/edu_order/edu_order_3/g' /opt/edu-order-3.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 4" --triggers --routines --events >/opt/edu-order-4.sql sed -i 's/edu_order/edu_order_4/g' /opt/edu-order-4.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 5" --triggers --routines --events >/opt/edu-order-5.sql sed -i 's/edu_order/edu_order_5/g' /opt/edu-order-5.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 6" --triggers --routines --events >/opt/edu-order-6.sql sed -i 's/edu_order/edu_order_6/g' /opt/edu-order-6.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 7" --triggers --routines --events >/opt/edu-order-7.sql sed -i 's/edu_order/edu_order_7/g' /opt/edu-order-7.sql

登录mysql

mysql
mysql -uroot -p

导入数据

mysql
mysql> use msb-edu-0; Database changed mysql> source /opt/edu-order-0.sql mysql> source /opt/edu-order-1.sql mysql> source /opt/edu-order-2.sql mysql> source /opt/edu-order-3.sql mysql> source /opt/edu-order-4.sql mysql> source /opt/edu-order-5.sql mysql> source /opt/edu-order-6.sql mysql> source /opt/edu-order-7.sql

库msb-edu-1:

数据规则
edu_order-0求余 8
edu_order-1求余 9
edu_order-2求余 10
edu_order-3求余 11
edu_order-4求余 12
edu_order-5求余 13
edu_order-6求余 14
edu_order-7求余 15
sql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 8" --triggers --routines --events >/opt/edu-order-0.sql sed -i 's/edu_order/edu_order_0/g' /opt/edu-order-0.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 9" --triggers --routines --events >/opt/edu-order-1.sql sed -i 's/edu_order/edu_order_1/g' /opt/edu-order-1.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 10" --triggers --routines --events >/opt/edu-order-2.sql sed -i 's/edu_order/edu_order_2/g' /opt/edu-order-2.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 11" --triggers --routines --events >/opt/edu-order-3.sql sed -i 's/edu_order/edu_order_3/g' /opt/edu-order-3.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 12" --triggers --routines --events >/opt/edu-order-4.sql sed -i 's/edu_order/edu_order_4/g' /opt/edu-order-4.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 13" --triggers --routines --events >/opt/edu-order-5.sql sed -i 's/edu_order/edu_order_5/g' /opt/edu-order-5.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 14" --triggers --routines --events >/opt/edu-order-6.sql sed -i 's/edu_order/edu_order_6/g' /opt/edu-order-6.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 15" --triggers --routines --events >/opt/edu-order-7.sql sed -i 's/edu_order/edu_order_7/g' /opt/edu-order-7.sql

登录mysql

mysql
mysql -uroot -p

导入数据

mysql
mysql> use msb-edu-1; Database changed mysql> source /opt/edu-order-0.sql mysql> source /opt/edu-order-1.sql mysql> source /opt/edu-order-2.sql mysql> source /opt/edu-order-3.sql mysql> source /opt/edu-order-4.sql mysql> source /opt/edu-order-5.sql mysql> source /opt/edu-order-6.sql mysql> source /opt/edu-order-7.sql

库msb-edu-2:

数据规则
edu_order-0求余 16
edu_order-1求余 17
edu_order-2求余 18
edu_order-3求余 19
edu_order-4求余 20
edu_order-5求余 21
edu_order-6求余 22
edu_order-7求余 23
sql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 16" --triggers --routines --events >/opt/edu-order-0.sql sed -i 's/edu_order/edu_order_0/g' /opt/edu-order-0.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 17" --triggers --routines --events >/opt/edu-order-1.sql sed -i 's/edu_order/edu_order_1/g' /opt/edu-order-1.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 18" --triggers --routines --events >/opt/edu-order-2.sql sed -i 's/edu_order/edu_order_2/g' /opt/edu-order-2.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 19" --triggers --routines --events >/opt/edu-order-3.sql sed -i 's/edu_order/edu_order_3/g' /opt/edu-order-3.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 20" --triggers --routines --events >/opt/edu-order-4.sql sed -i 's/edu_order/edu_order_4/g' /opt/edu-order-4.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 21" --triggers --routines --events >/opt/edu-order-5.sql sed -i 's/edu_order/edu_order_5/g' /opt/edu-order-5.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 22" --triggers --routines --events >/opt/edu-order-6.sql sed -i 's/edu_order/edu_order_6/g' /opt/edu-order-6.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 23" --triggers --routines --events >/opt/edu-order-7.sql sed -i 's/edu_order/edu_order_7/g' /opt/edu-order-7.sql

登录mysql

mysql
mysql -uroot -p

导入数据

mysql
mysql> use msb-edu-2; Database changed mysql> source /opt/edu-order-0.sql mysql> source /opt/edu-order-1.sql mysql> source /opt/edu-order-2.sql mysql> source /opt/edu-order-3.sql mysql> source /opt/edu-order-4.sql mysql> source /opt/edu-order-5.sql mysql> source /opt/edu-order-6.sql mysql> source /opt/edu-order-7.sql

库msb-edu-3:

数据规则
edu_order-0求余 24
edu_order-1求余 25
edu_order-2求余 26
edu_order-3求余 27
edu_order-4求余 28
edu_order-5求余 29
edu_order-6求余 30
edu_order-7求余 31
sql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 24" --triggers --routines --events >/opt/edu-order-0.sql sed -i 's/edu_order/edu_order_0/g' /opt/edu-order-0.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 25" --triggers --routines --events >/opt/edu-order-1.sql sed -i 's/edu_order/edu_order_1/g' /opt/edu-order-1.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 26" --triggers --routines --events >/opt/edu-order-2.sql sed -i 's/edu_order/edu_order_2/g' /opt/edu-order-2.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 27" --triggers --routines --events >/opt/edu-order-3.sql sed -i 's/edu_order/edu_order_3/g' /opt/edu-order-3.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 28" --triggers --routines --events >/opt/edu-order-4.sql sed -i 's/edu_order/edu_order_4/g' /opt/edu-order-4.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 29" --triggers --routines --events >/opt/edu-order-5.sql sed -i 's/edu_order/edu_order_5/g' /opt/edu-order-5.sql
mysql
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 30" --triggers --routines --events >/opt/edu-order-6.sql sed -i 's/edu_order/edu_order_6/g' /opt/edu-order-6.sql
sh
mysqldump -uroot -p123456 --single-transaction --master-data=2 --databases msb-edu --tables edu_order --where="user_id % 32 = 31" --triggers --routines --events >/opt/edu-order-7.sql sed -i 's/edu_order/edu_order_7/g' /opt/edu-order-7.sql
sql
-- 查询某个库中某个表数据量 SELECT TABLE_NAME, TABLE_ROWS FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA ='your_database_name';

本文作者:柳始恭

本文链接:

版权声明:本博客所有文章除特别声明外,均采用 BY-NC-SA 许可协议。转载请注明出处!