0、集群环境介绍
10.20.201.51 namenoderesourcemanagerhmasterspark
10.20.201.52 namenoderesourcemanagerhmasterspark
10.20.201.53 datanodenodemanagerhregionserversparkzookeeperkafkakafkaconnect
10.20.201.54 datanodenodemanagerhregionserversparkzookeeperkafkakafkaconnect
10.20.201.55 datanodenodemanagerhregionserversparkzookeeperkafkakafkaconnect
10.20.201.56 mysql hive
192.168.112.201 flumekafkaconnect
1.zookeeper集诚心的龙猫/龙猫
2.kafka集诚心的龙猫/龙猫
/BDA PPS/Kafka/bin/Kafka-server-start.sh/BDA PPS/Kafka/config/server.properties
/BDA PPS/Kafka/bin/Kafka-server-stop.sh
主题操作
/BDA PPS/Kafka/bin/Kafka-topics.sh-list-zookeeperhdfsaha 33602181，hdfsbha 33602181，hdfsbha:2181，hdfscha
/BDA PPS/Kafka/bin/Kafka-topics.sh-describe-zookeeperhdfsaha 33602181、hdfsbha 33602181、hdfsbha:2181、hdfsbha
/BDA PPS/Kafka/bin/Kafka-topics.sh-delete-zookeeperhdfsaha 33602181、hdfsbha:2181、hdfsbha:2181、hdfsccc
/BDA PPS/Kafka/bin/Kafka-topics.sh-create-zookeeperhdfsaha 33602181、hdfsbha:2181、hdfsbha:2181、hdfscc
/BDA PPS/Kafka/bin/Kafka-console-productor.sh-bootstrap-serverhdfsaha 33609092，HDFSaha33609092，hdfsaha:9092
/BDA PPS/Kafka/bin/Kafka-console-consumer.sh-bootstrap-serverhdfsaha :9092、hdfsaha:9092、hdf Saha :060092
3 .数据收集(kafka生产方192.168.112.201 )。
3.1 flume jdbc插件收集mysql数据
flume启动命令
/BDA PPS/flume/bin/flume-ng agentconf/BDA PPS/flume/conf /conf-file/BDA PPS/flume/conf/flume-kaf kaf
3.2 Kafka连接JDBC收集mysql数据
cd /bdapps/confluent-4.0.0
nohup./bin/connect-standalone./etc/Kafka/connect-standalone.properties./etc/Kafka-connect-JDBC/cone
4 .数据消费
4.1 kafka connect诚心诚意龙猫/p
/BDA PPS/confluent/bin/connect-distributed./etc/Kafka/connect-distributed.properties
连接操作示例
Curl localhost :8083/connectors|jq
Curl localhost :8083/connector-plugins|jq
Curl localhost :8083/connectors/test-MySQL-sink|jq
curl localhost :8083/connectors/test-MySQL-sink/config|jq
Curl localhost :8083/connectors/test-MySQL-sink/status|jq
Curl localhost :8083/connectors/local-file-sink/tasks|jq
Curl localhost :8083/connectors/test-MySQL-sink/tasks/0/status|jq
4.2火花流