2025年数据采集工具之maxwell

科技前沿 • 2025-01-19 07:12 • 阅读 59

大家好，我是讯享网，很高兴认识大家。

1、安装maxwell

网站

https://maxwells-daemon.io/changelog/

选择v1.29.2

（v130.0以后的全是jdk11 ，就不是jdk8了）

选择maxwell-1.29.2.tar.gz点一下就下载了

安装就是把安装包扔到虚拟机的指定位置

tar -zxvf maxwell-1.29.2.tar.gz -C /usr/local/

讯享网

我没配置环境变量，其实也可去配置环境变量

2、mysql的环境配置

1)修改mysql的配置

讯享网 vim /etc/my.cnf [mysqld] server_id=1 log-bin=mysql-bin binlog_format=row binlog-do-db=test_maxwell

binlog-do-db就是监听的数据库

systemctl restart mysqld

讯享网mysql -uroot -p'@Mmforu45'

 show variables like '%binlog%'; #查看 binlog_format | ROW

查看真实的binlog

讯享网

讯享网 cd /var/lib/mysql ll mysqlbin.000001 mysqlbin.index

2)初始化元数据库

mysql -uroot -p'@Mmforu45'

讯享网mysql> CREATE DATABASE maxwell; mysql> set global validate_password_length=4; mysql> set global validate_password_policy=0; mysql> GRANT ALL ON maxwell.* TO 'maxwell'@'%' IDENTIFIED BY '@Mmforu45'; mysql> GRANT SELECT ,REPLICATION SLAVE , REPLICATION CLIENT ON *.* TO maxwell@'%'; mysql> flush privileges;

3、Maxwell的启动方式

3.1第一种启动方式

启动一下maxwell，但是maxwell和mysql一定是在同一个电脑里面！

cd /usr/local/maxwell-1.29.2

讯享网bin/maxwell --user='maxwell' --password='@Mmforu45' --host='qianfeng03' --producer=stdout

因为maxwell是前台打印，所以就关闭就直接ctrl+c

3.2 第二种启动方式

就是修改配置，然后启动配置

cd /usr/local/maxwell-1.29.2 cp config.properties.example config.properties

修改

讯享网# tl;dr config log_level=info producer=stdout kafka.bootstrap.servers=localhost:9092 # mysql login info host=qianfeng03 user=maxwell password=@Mmforu45

启动

bin/maxwell --config ./config.properties

打印在控制台的操作

讯享网bin/maxwell --config ./config.properties

在mysql中

4、案例实操

4.1 案例1

mysql -uroot -p'@Mmforu45'

讯享网use test_maxwell; create table test01( id tinyint, name varchar(24) ); insert into test01 value(1,'aaaa'),(2,'bbbbb'),(3,'cccc');

回看一下前台打印日志

[root@qianfeng03 maxwell-1.29.2]# bin/maxwell --config ./config.properties Using kafka version: 1.0.0 23:25:22,490 INFO Maxwell - Starting Maxwell. maxMemory:  bufferMemory Usage: 0.25 23:25:22,538 INFO Maxwell - Maxwell v1.29.2 is booting (StdoutProducer), starti ng at Position[BinlogPosition[mysql-bin.000001:77135], lastHeartbeat= 62] 23:25:22,632 INFO MysqlSavedSchema - Restoring schema id 1 (last modified at Po sition[BinlogPosition[mysql-bin.000001:5598], lastHeartbeat=0]) 23:25:22,730 INFO BinlogConnectorReplicator - Setting initial binlog pos to: my sql-bin.000001:77135 23:25:22,740 INFO BinaryLogClient - Connected to qianfeng03:3306 at mysql-bin.0 00001/77135 (sid:6379, cid:68) 23:25:22,740 INFO BinlogConnectorReplicator - Binlog connected. 23:32:59,024 INFO AbstractSchemaStore - storing schema @Position[BinlogPosition [mysql-bin.000001:], lastHeartbeat=24] after applying "create t able test01( id tinyint, name varchar(24) )" to test_maxwell, new schema id is 2 { 
   "database":"test_maxwell","table":"test01","type":"insert","ts":,"xid ":2227,"xoffset":0,"data":{ 
   "id":1,"name":"aaaa"}} { 
   "database":"test_maxwell","table":"test01","type":"insert","ts":,