将设为首页浏览此站
开启辅助访问 天气与日历 收藏本站联系我们切换到窄版

易陆发现论坛

 找回密码
 开始注册
查看: 71|回复: 2
收起左侧

pgs not deep-scrubbed in time异常处理

[复制链接]
发表于 2022-12-20 17:00:15 | 显示全部楼层 |阅读模式

马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。

您需要 登录 才可以下载或查看,没有帐号?开始注册

x
发现出现异常warn信息,虽然不影响整个集群正常使用,但强迫症患者还是忍不了,下面是过程。查看具体报错信息
# Q8 G' Q4 D# h' g; ^5 K+ `HEALTH_WARN 2 pgs not deep-scrubbed in time+ \+ a. N+ s! L% F& T! h
PG_NOT_DEEP_SCRUBBED 2 pgs not deep-scrubbed in time
; {* R$ c. u* N" u- |3 g( Q    pg 18.41 not deep-scrubbed since 2022-12-07 20:15:50.550606
# h! N* W/ D  R9 \. `3 [0 [8 \6 ]    pg 5.16d not deep-scrubbed since 2022-12-07 22:21:58.141071
8 ]6 Z' c6 }% `1 M* J6 [- F& K* ]/ s+ \
* s" N+ M  x) G9 q
[root@controller1 ~]# ceph pg deep-scrub 18.41
3 B. j8 w. }) A2 Q* g0 Yinstructing pg 18.41 on osd.6 to deep-scrub
* r3 E" ?) f3 c& A' C4 \+ U[root@controller1 ~]# ceph pg deep-scrub 5.16
  U9 D: n, A5 a- R' e, yinstructing pg 5.16 on osd.13 to deep-scrub
. F/ k; w1 N9 ~) x5 J
9 H- F8 S& i( A: q
* }+ G2 s- B; S3 p
8 b6 A) U- q' r3 c[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
+ \. Q( u& B- y* w% O    "osd_deep_scrub_interval": "604800.000000",
+ ~/ O1 v: k. k/ O/ t( D[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800' |1 j; b0 C: i& r( a  D' o5 y
[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval
! u( v1 z; N9 `2 J( E: P5 @6 A    "osd_deep_scrub_interval": "3628800.000000",
* {5 S( M% `; B' p6 U[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 36288008 @0 \# M  u" y/ v# h* D( |
恢复正常后,再改回来:
# q8 Q" t. N  {* q  h& v3 o/ f: K[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval8 b' r  F7 u7 R
    "osd_deep_scrub_interval": "604800.000000",
7 K$ k6 o; a4 O' t4 `7 A: d[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800
5 K  t; w0 q; k( }: T7 g. R( e[root@controller1 ~]# ceph daemon osd.6 config show |grep osd_deep_scrub_interval3 }) A' l$ X" W4 B) Z8 c5 U
    "osd_deep_scrub_interval": "3628800.000000",
5 f- \& y3 _# R; u5 Q7 L  a( z[root@controller1 ~]# ceph config set global osd_deep_scrub_interval 3628800^C3 G" p$ |' T$ n# d# Z' |  Q$ G

$ O' o- [1 b7 k[root@controller1 ~]# + ~2 k' Q+ b' s, n: h& R+ \
[root@controller1 ~]# ceph -s+ U/ ]) j' I  {& Z
  cluster:
6 Y5 ?/ l$ ]+ i8 v7 O7 O) p    id:     9d22e36a-2bdd-4d2d-8394-48af75ead777
9 H& d0 P* T- P' F    health: HEALTH_OK
$ G) \( s( S2 D2 g+ G
% H0 y) S! |3 [# k, c* X. a  services:
% }) X% Q- Q2 f+ Y4 M    mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 5M)
7 n/ Y! \% `2 K+ g+ \: y3 E. P    mgr: ceph1(active, since 19M), standbys: ceph2,ceph3* H+ z9 B/ Y7 T) `, E% e
    osd: 40 osds: 40 up (since 3w), 40 in (since 12M)
0 O9 M7 r& ^; O    rgw: 3 daemons active (host09, host10, host11)$ ?. T' m4 W) X% e4 B% H& q$ L

+ Y3 y2 ^8 O$ r5 L  task status:) K" m& a% ]. O" f! Z- ]

- @9 l5 K# g+ q1 _3 y  data:
9 x: X+ w9 Y, R& r    pools:   16 pools, 3072 pgs
8 d& |3 P5 j, Q& N9 m    objects: 4.20M objects, 16 TiB
% ~/ q' `" ]4 L. ^    usage:   40 TiB used, 107 TiB / 148 TiB avail
0 w5 r# q. m3 F8 F% V1 O    pgs:     3067 active+clean
# V* Y2 ~7 d5 H             5    active+clean+scrubbing+deep
  o) a. c6 B+ M3 }
. A, _8 ~: l, f( g  io:2 ]# y% b" ?, J) g
    client:   403 KiB/s rd, 9.5 MiB/s wr, 514 op/s rd, 466 op/s wr1 O. E4 B- L3 w$ A7 c  A

0 U( ^9 ]7 Z8 P/ K0 W% I7 f% j+ A8 m[root@ ~]# ceph config set global osd_deep_scrub_interval 604800
% m7 {2 @: T! d% ~1 e& ~[root@ ~]# - ^8 x* X% f! f. U' F& }9 q! C

4 q! w7 O0 k' l; n7 W5 M/ W6 Y+ ^" H. L9 @: U" [  t
systemctl restart ceph-osd@6.service% V# V! L: j6 B% I# R2 d

! U2 S6 v5 G$ S8 w3 L1 ~8 t4 f$ q# F" l5 A$ O' d3 f

0 }6 o6 R7 m! o' J
 楼主| 发表于 2022-12-20 17:00:16 | 显示全部楼层
根源, B2 Y9 f6 H2 y) U9 a0 J
RHCS 4 has introduced 2 new parameters on scrubbing interval warning, and default values are  ]  s4 B/ e7 ?
Raw# F7 K0 {- u% j
"mon_warn_pg_not_deep_scrubbed_ratio": "0.750000",
& w% f6 ~6 |8 W, ^; D"mon_warn_pg_not_scrubbed_ratio": "0.500000",
% F: s/ j! `- e' NThese ratios are based on the following parameters, and default values are
3 D$ @9 W6 D) Q# q, u% URaw  L, h1 ~! k: i* X4 H: n7 d( L
"osd_scrub_max_interval": "604800.000000",
: I  N$ O2 C1 z- S3 T"osd_deep_scrub_interval": "604800.000000"* n4 ^& @! D. l! K# j+ P8 s9 U
When pgs are not scrubbed / deep scrubbed for the configured ratio of the interval, warnings of "pgs not scrubbed / deep-scrubbed in time" will show in Ceph status.
1 r5 A2 C+ i  j/ R
' k  N+ S; x2 F- v9 ?8 O4 [When setting "mon_warn_pg_not_deep_scrubbed_ratio" or "mon_warn_pg_not_scrubbed_ratio" values to 0, warnings will be disabled. Please evaluate the cluster's past usage carefully before setting.
- |9 P3 U* @+ ~* ~4 m6 |5 G# ~, c( V  X
Sometimes, we might get a false warning even though the osd_deep_scrub_interval has been increased. This happens when the scrubbing parameters are not applied globally because these settings are used by both OSDs and MONs.( x6 ]" h  T5 k, l5 K2 H# b( @8 |
The OSDs use them to determine when to run scrub, and the MONs/MGRs use them to check if they need to show the warning.
6 P  _8 G( B6 t/ @' ~! NYou can set it globally as:4 J0 {8 G* @6 {; \- c+ }

7 D0 Z1 f. D, X! A0 _+ oRaw
3 d) x" m2 E) T* U9 Q) r# ceph config set global osd_deep_scrub_interval 3628800
3 r1 H  J7 t' b- w" o5 f2 `/ i诊断步骤
4 p- _+ y# F0 M* M/ T: f) L* ~. fChecking scrubbing related configurations by admin socket
+ o4 @: E9 c5 O( J& R( ^Raw
3 i" t) ^' F& }) }( ^: f2 K, nceph --admin-daemon /var/run/ceph/<admin_socket_name>.asok config show | grep scrub
 楼主| 发表于 2022-12-20 17:00:17 | 显示全部楼层
You can set the deep scrub period to 2 week, to stretch the deep scrub window. Insted of8 d  `8 x  S- U) A/ `

  e9 H9 O# w3 `* n6 E8 _, i osd_deep_scrub_interval = 604800# b  `- Y. i' u2 k. `4 w
use:1 s7 D7 s+ @2 z7 y7 U1 b6 f

6 r# P2 v3 F( _6 K& [. G4 } osd_deep_scrub_interval = 1209600
您需要登录后才可以回帖 登录 | 开始注册

本版积分规则

关闭

站长推荐上一条 /4 下一条

如有购买积分卡请联系497906712

QQ|返回首页|Archiver|手机版|小黑屋|易陆发现 点击这里给我发消息

GMT+8, 2023-2-9 14:53 , Processed in 0.042441 second(s), 21 queries .

Powered by LR.LINUX.cloud bbs168x X3.2 Licensed

© 2012-2022 Comsenz Inc.

快速回复 返回顶部 返回列表