k8s集群突然报错

目录

一.问题分析

1.报错原因

2.排错步骤

二.恢复

1.备份etcd与

2.恢复


一.问题分析

1.报错原因

       服务器非正常关机(意外掉电、强制拔电)后 etcd 数据损坏。

我这报错与预先操作:

      其中一个网络插件flannel直接丢失,重启集群所有docker与k8s,发现k8s组件启动不了(etcd,apiserver,controller-manager,scheduler),在把组件单独重启,发现还是有两个组件报错(etcd,apiserver),查看apiserver日志发现出现报错Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused,2379是etcd的端口,那么apiserver是由于etcd无法连接而启动不了。接着查看etcd日志发现报错mvcc: cannot unmarshal event: proto: wrong wireType = 0 for field Key此报错是由于服务器非正常关机(意外掉电,强制拔电)后 etcd数据损坏导致的

2.排错步骤

# 1.查看报错
~]# kubectl get pods -n kube-system -o wide
NAME        READY STATUS  RESTARTS  AGE   IP           NODE         NOMINATED   NODE     READINESS GATES
flannel-xxx 1/1   Running 6         145d  node2的ip    node2的ip         
flannel-xxx 1/1   Running 7         145d  node3的ip    node3的ip         

## 注:我这是三节点,现在只有两个
#在node1上查看容器,发现容器也没有了
~]# docker ps -a | grep flannel   
~]# docker ps | grep flannel

# 2.备份日志(在所有节点执行)
~]# systemctl status docker.service  -l  > dockers.err.txt
~]# systemctl status kubelet.service  -l  > kubelet.err.txt
~]# journalctl -xefu docker > docker.err.txt 

# 3.重启kubelet和docker(在所有节点执行)
~]# systemctl restart docker      
~]# systemctl restart kubelet     
~]# kubectl get pods -n kube-system -o wide
NAME        READY STATUS  RESTARTS  AGE   IP           NODE         NOMINATED   NODE     READINESS GATES
flannel-xxx 1/1   Running 6       145d  node1的ip    node1的ip         
flannel-xxx 1/1   Running 6       145d  node2的ip    node2的ip         
flannel-xxx 1/1   Running 7       145d  node3的ip    node3的ip         
kube-apiserver-xx 0/1  Running 23 34d   node{1..3}的ip node3{1..3}的ip      
kube-controller-manager-xx 0/1 Running 2 145d node{1..3}的ip node{1..3}的ip    
kube-scheduler-xx 0/1  Running 2  145d        node{1..3}的ip node{1..3}的ip    
etcd-xxx   0/1    Running 10      145d        node{1..3}的ip node{1..3}的ip    
# 注:此时发现网络插件恢复了,但是etcd,apiserver,controller-manager,schedule这四个组件报错了

# 3.单独在重启组件
~]# kubectl get pods -n kube-system | grep -v "NAME" | grep "0/1" | awk '{print $1}'| xargs -i kubectl delete pods -n kube-system {}
# 测试发现还是有两个报错
~]# kubectl get pods -n kube-system -o wide
NAME        READY STATUS  RESTARTS  AGE   IP           NODE         NOMINATED   NODE     READINESS GATES
kube-apiserver-xx 0/1  Running 23 34d   node{1..3}的ip node3{1..3}的ip      
etcd-xxx   0/1    Running 10      145d        node{1..3}的ip node{1..3}的ip    

# 4.查看日志
~]# docker logs -f apiserver的CONTAINER ID
xxxx
Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused
~]# docker logs -f etcd的CONTAINER ID
xxxx
mvcc: cannot unmarshal event: proto: wrong wireType = 0 for field Key

二.恢复

1.备份etcd与

#1.不知道路径可以查找etcd数据目录
~]# find / -type d -name member
~] cd /var/lib/etcd/member
~]# mv * /root/member.back
# 注:在故障节点执行,删除或者移走备份都可以

2.恢复

#1.重启etcd
~]# docker ps -a|grep etcd
~]# docker rm -f etcd的ID        # 所有节点执行
~]# systemctl restart kubelet    # 重启kubelet

#2.查看
~]# kubectl get pods
# 注:此时没有报错了

你可能感兴趣的:(小技巧,容器,服务器,java,前端)