将设为首页浏览此站
开启辅助访问 天气与日历 收藏本站联系我们切换到窄版

易陆发现论坛

 找回密码
 开始注册
查看: 42|回复: 2
收起左侧

Kafka环境搭建及使用教程

[复制链接]
发表于 2023-1-3 10:58:57 | 显示全部楼层 |阅读模式

马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。

您需要 登录 才可以下载或查看,没有帐号?开始注册

x
kafka里自带了一个zookeeper,可以不需要再去下载zookeeper。在启动kafka前,需要先启动zookeeper。
# B1 `5 P" A  t5 E9 q' \$ W( t+ `, B2 }# |2 G, o& ]; ?9 p; M
(1)首先去修改config目录下的zookeeper.properties, y8 i( H. `2 Z( H" x: {8 v
(2)修改config目录下的server.properties,修改log.dirs和zookeeper.connect。log.dirs是日志存放文件夹,zookeeper.connect是zookeeper连接地址(端口和clientPort保持一致)。" U/ d; N6 h% G) I; n3 P2 c
8 a2 E% ]9 ]; r# V" V" q
至此,kafka单机版环境已搭建完毕。(如果对里面的参数没修改的话,可不修改,直接用默认的)
" X3 ~* C; B( p( w3 V& ]4 Q0 Q0 m& N* G

) m" D% \: z2 S
; R2 O3 A$ L: `  g( }: c  okafka使用教程1 M- F. f+ ]3 _' g/ I, d
(1)启动kafka自带的zookeeper(加了-daemon,可以让这个进程挂在后台进行,否则会发现是个阻塞进程,只能再开一个终端进行其他操作)
( x2 H1 Q3 r& u( p2 W9 h/ Hbin/zookeeper-server-start.sh -daemon config/zookeeper.properties! D' C9 j5 t3 H, v  ?& R
; [$ W- o' ~, d1 w$ y2 G5 _
(2)启动kafka
& ^; d' T2 ~: Q6 ybin/kafka-server-start.sh -daemon config/server.properties
" G+ {! S4 Z' j0 d0 j
2 e) _  X* u3 `" A(3)创建一个topic8 J" j2 z4 t6 y" K) `* Y4 _
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test/ L! q2 L$ M' s  d
2 k1 \6 ?8 z8 o. z6 f
* }. C1 U5 T" @" \% {# Q5 ^
(4)查看topic列表
2 o# o5 Q, v9 m* s8 A- Obin/kafka-topics.sh -list -zookeeper localhost:2181
2 v2 J  R2 r+ h9 Z# W0 Q) t4 f8 f2 T! P
(5)创建生产者进程  (9092是默认的broker-list端口). _7 I3 _/ a% W$ R- u# {0 S
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
* t1 f6 z9 b( [1 F; U! x% M$ n$ V2 f: h9 E
(6)创建消费者进程8 z# {: |2 _3 t: u
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning8 {* m  Z6 D! r+ {+ I

- T5 t5 ~4 l0 J(7)外网访问 vi config/server.properties
/ Y* ~: @" G( v+ s4 X! eadvertised.listeners=PLAINTEXT://192.168.8.65:90926 V- G8 J/ |" d/ C' a+ F
 楼主| 发表于 2023-1-3 11:28:57 | 显示全部楼层
cat config/zookeeper.properties
5 J" k3 N6 ~* y; f# Licensed to the Apache Software Foundation (ASF) under one or more
* T. C( _  l: V# contributor license agreements.  See the NOTICE file distributed with" o( V, k8 m5 r" i3 X' t4 ~" [
# this work for additional information regarding copyright ownership.7 c* G; U+ h! U1 V$ y. P# j  V
# The ASF licenses this file to You under the Apache License, Version 2.0  _& u* \) M2 u: q6 k9 a& J
# (the "License"); you may not use this file except in compliance with
' F2 `+ x+ B/ f) y& i# the License.  You may obtain a copy of the License at/ D4 u' F9 u# P, j$ R) o
# 0 |4 }  I8 j7 k4 O
#    http://www.apache.org/licenses/LICENSE-2.0
2 l9 Q) K# o7 C+ F7 t, g! A/ B# : b, J  W" ~9 j' v! R
# Unless required by applicable law or agreed to in writing, software' g& L7 ~. X6 A2 }
# distributed under the License is distributed on an "AS IS" BASIS,& J, y. Z* H/ e& w# A
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9 {0 ?' H; \" y1 r. p& h# See the License for the specific language governing permissions and
; d1 Q5 G- U, D% E/ o% l0 Y7 b  Y# limitations under the License.6 c/ h4 M5 R9 Q& J+ A
# the directory where the snapshot is stored.: X0 o- [6 S1 Q/ l# g& |
dataDir=/mnt/kafka_2.13-3.3.1/data/zookeeper
8 i0 T- |! E, x, ?: e7 ~1 }: ?# the port at which the clients will connect
  P/ E" Y$ H9 T9 _7 B7 A( O  Y' d) \: AclientPort=21814 N3 n2 v8 O, [; p
# disable the per-ip limit on the number of connections since this is a non-production config
$ Y9 ]6 I) R7 |5 U: \maxClientCnxns=0
9 p) R$ {7 x+ q  _  q; u# Disable the adminserver by default to avoid port conflicts.9 R% |2 V2 u0 q: @8 l3 {
# Set the port to something non-conflicting if choosing to enable this
. b: z6 K" b9 x- E$ }admin.enableServer=false2 _6 a8 s1 v; f! ^( x) e0 ~
# admin.serverPort=8080( a/ @8 Z0 g  {5 d. B- \
& A0 h4 G1 n3 a$ a# W) r4 }8 D
  o7 B. K  ]% X% d9 g) D* a6 `/ c
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
& G) W( @8 z5 I& q! W: `: \9 q
 楼主| 发表于 2023-1-3 11:40:15 | 显示全部楼层
[root@ceph2 kafka_2.13-3.3.1]# egrep -v "^#|^$" config/server.properties # T) M" z6 r9 V0 N& P/ N. C! B
broker.id=0+ S0 e% v' s" e) ?. u0 C
listeners=PLAINTEXT://192.168.0.233:90929 [! l& @& z8 i3 z+ b; [0 s
num.network.threads=34 q/ ~. T, c- a* I. t
num.io.threads=85 [1 ]2 D6 Z% O3 c/ [  n( C: U: M
socket.send.buffer.bytes=102400
- f3 P( h4 Y- I; p: z0 v( r' _socket.receive.buffer.bytes=102400
8 x, @  K& Y8 _5 B8 @$ y( Dsocket.request.max.bytes=104857600
$ X9 d! l& l% _, l3 l, j* O( _, [log.dirs=/var/log/kafka-logs8 R3 `1 n" s7 z) k& [6 k
num.partitions=13 b' E( P* G& l
num.recovery.threads.per.data.dir=12 x% Q0 Q" B' c7 y
offsets.topic.replication.factor=1
! n9 n5 y. N+ ]( x7 l' i9 ~$ T  s3 ]; Rtransaction.state.log.replication.factor=1
% X- c. t! S) B$ |2 X9 Htransaction.state.log.min.isr=1
. ?2 e' N8 V4 I* e* ilog.retention.hours=1689 o* |+ s8 }& H# Z
log.retention.check.interval.ms=300000, r4 V9 d$ R' R6 k: x' C+ ?  M
zookeeper.connect=localhost:2181
1 u& h- |. e, szookeeper.connection.timeout.ms=18000# Y% H+ t0 ~; |$ W8 x: L6 K! ]
group.initial.rebalance.delay.ms=0' K+ Y5 m7 v4 T: g6 j( F) r
2 m0 a3 s- [; o4 H5 I  I: u

4 c) e1 b' ^7 K; x1 P8 X& a[root@ceph2 kafka_2.13-3.3.1]# nohup bin/kafka-server-start.sh config/server.properties &, B( ^2 n7 D7 M
; H, E* X7 z* Z

5 Q$ F9 q- v, h4 A! U: C% P+ d8 \% X: \$ J9 B9 `! \3 _' o
netstat  -ntlp |grep java* w" ?. ]! t5 X8 @" V# r
tcp6       0      0 192.168.0.233:9092      :::*                    LISTEN      21222/java         
* @% p/ G5 K9 Y- c! {' ~; Stcp6       0      0 :::2181                 :::*                    LISTEN      20780/java         
) ?- j1 n+ N) e8 Rtcp6       0      0 :::41467                :::*                    LISTEN      21222/java         
+ l, ~9 A: T" @" \( Itcp6       0      0 :::40571                :::*                    LISTEN      20780/java          * o. y5 X/ W# m- Z% x0 x( [

, u1 T! ]0 O, j8 u' ]
您需要登录后才可以回帖 登录 | 开始注册

本版积分规则

关闭

站长推荐上一条 /4 下一条

如有购买积分卡请联系497906712

QQ|返回首页|Archiver|手机版|小黑屋|易陆发现 点击这里给我发消息

GMT+8, 2023-2-9 15:02 , Processed in 0.042241 second(s), 22 queries .

Powered by LR.LINUX.cloud bbs168x X3.2 Licensed

© 2012-2022 Comsenz Inc.

快速回复 返回顶部 返回列表