we have 3 Kafka brokers machines with version 2.7
each Kafka have 8 disks as the following
/dev/sdh 1.7T 1.2T 511G 70% /var/data/topic-logs8
/dev/sde 1.7T 891G 786G 54% /var/data/topic-logs5
/dev/sdc 1.7T 1.1T 551G 68% /var/data/topic-data3
/dev/sdf 1.7T 1.5T 161G 91% /var/data/topic-data6
/dev/sdg 1.7T 1017G 660G 61% /var/data/topic-data7
/dev/sdd 1.7T 1.7T 480K 100% /var/data/topic-data4
/dev/sda 1.9T 1.6T 311G 84% /var/data/topic-data1
as we can see above one of the disks became 100%
since one of the disk is 100% , then all replica data on that disk isn’t reached
therefore we get the following behavior:
kafka-topics.sh --zookeeper zoo_server:2181 --describe
Topic: mnior_mnol_frontend Partition: 85 Leader: none Replicas: 1012,1010,1011 Isr: 1011
Topic: mnior_mnol_frontend Partition: 86 Leader: 1010 Replicas: 1010,1011,1012 Isr: 1010,1011,1012
Topic: mnior_mnol_frontend Partition: 87 Leader: 1011 Replicas: 1011,1010,1012 Isr: 1010,1011,1012
Topic: mnior_mnol_frontend Partition: 88 Leader: 1012 Replicas: 1012,1011,1010 Isr: 1010,1012
Topic: mnior_mnol_frontend Partition: 89 Leader: 1011 Replicas: 1010,1012,1011 Isr: 1011
Topic: mnior_mnol_frontend Partition: 90 Leader: 1011 Replicas: 1011,1012,1010 Isr: 1010,1011,1012
Topic: mnior_mnol_frontend Partition: 91 Leader: 1012 Replicas: 1012,1010,1011 Isr: 1010,1011,1012
Topic: mnior_mnol_frontend Partition: 92 Leader: 1010 Replicas: 1010,1011,1012 Isr: 1010,1011,1012
Topic: mnior_mnol_frontend Partition: 93 Leader: 1011 Replicas: 1011,1010,1012 Isr: 1011
topic name – mnior_mnol_frontend
kafka brokers id’s – 1010,1011,1012
Isr – we have replica 3 , and we can see that Isr is missing due disk faulty
as we can see above and due disk failure , we have here missing replica and also some none leaders
actually , we was thinking that Kafka itself should create the missing replica on the other good ( that are less then 100% ) disks , but this inst happens
so I want to understand – if this is limitation of Kafka?
or maybe we can resolve the issue by Kafka settings ( for example by Kafka server.properties )