The ETCD is the core Kubernetes service that contains all resources that have been created
It is started by the kubelet as a static pod on the control node
Losing the ETCD means loosing all the k8s data. That’s why we need a backup
To backup the ETCD, root access is required to run the etcdctl tool
Use: sudo apt install ectd-client
Note: etcdctl use the wrong API version by default, fix this by using
sudo ETCDCTL_API=3 ectdctl … snapshot save
To set the right version of API set the environment variable ETCDCTL_API command
export ETCDCTL_API=3
To use etcdctl, you need to specify the etcd API service endpoint, as well as cacert, cert and key to be used
To get all of the data about etcd data locations, and certs etc
Use : ps aux | grep etcd
touk@k8smaster:~$ ps aux | grep etcd
root 2103 2.9 1.9 11214464 78708 ? Ssl 00:53 4:43 etcd --advertise-client-urls=https://192.168.1.13:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --experimental-initial-corrupt-check=true --experimental-watch-progress-notify-interval=5s --initial-advertise-peer-urls=https://192.168.1.13:2380 --initial-cluster=k8smaster=https://192.168.1.13:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://192.168.1.13:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://192.168.1.13:2380 --name=k8smaster --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
root 2282 5.0 9.4 1170304 375232 ? Ssl 00:55 8:02 kube-apiserver --advertise-address=192.168.1.13 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
touk 91861 0.0 0.0 11588 716 pts/0 S+ 03:35 0:00 grep --color=auto etcd
Start Interacting with ETCDCTL:
Make sure you can get the keys out of the API
sudo ETCDCTL_API=3 etcdctl --endpoints=localhost:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key get / --prefix --keys-only
Now we are ready to make a backup
sudo ETCDCTL_API=3 etcdctl --endpoints=localhost:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcdbackup.db
Snapshot saved at /tmp/etcdbackup.db
You can view a nice result using this command:
sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status /tmp/etcdbackup.db
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| d8571afe | 784570 | 972 | 15 MB |
+----------+----------+------------+------------+
The Recommended approach is to restore it another directory rather then the default
To start using it, core kubernetes services must be stopped, after which the etcd can be reconfigured to use the new directory
To stop the core services, temporarily move
sudo cd /etc/kubernetes/manifests/
sudo mv * ..
Now lets restore it !
ETCDCTL_API=3 etcdctl snapshot restore --data-dir /var/lib/etcd-from-backup /tmp/etcdbackup.db
Once the etcd pod has stopped, reconfigure the etcd to use the non-default etcd path vi changing the hostPath in it’s manifest file:
Since we moved the manifest file we will findit in /etc/kubernetes
Modify etcd.yaml to point to the new directory of backup
sudo vim /etc/kubernetes/etcd.yaml
hostPath:
path: /var/lib/etcd-from-backup
sudo ETCDCTL_API=3 etcdctl snapshot restore --data-dir /var/lib/etcd-from-backup etcdbackup.db
sudo mv ../*.yaml .