Etcd restore
Hi,
This question is related to etcd backup and restore.
I've set up a kubernetes cluster using stacked etc topology using kubeadm. Dual control-plane nodes and dual worker-nodes. I end up with the static pod manifests in /etc/kubernetes/manifests
and the control-plane services are running as pods. Only kubelet is running as a systemd service.
I've created a snapshot of my etcd databases on both control-plane nodes.
I simulate a data failure:
1. Stop API servers by removing the manifest files.
2. Delete /var/lib/etcd/
on both control-plane nodes.
Now I want to do a complete restore of etcd database.
I'm doing it using etcdctl on both control-plane nodes.
It seems to work and I get the output:
2022-07-01 09:05:14.228623 I | mvcc: restore compact to 42333 2022-07-01 09:05:14.233684 I | etcdserver/membership: added member a874c87fd42044f [https://127.0.0.1:2380] to cluster c9be114fc2da2776
kubectl get nodes is showing all nodes ready. However, when I do a single change, such as scaling a deployment. Something gets really wrong. Suddenly the API servers disagree on node health, and the replication change is not performed.
Am I doing this wrong?
Best Answer
-
@oleksazhel Thank you for your willingness to help.
I think my error was to do separate etcd snapshot. One for each node. Then I restored each node from its own snapshot.
I read in etcd documentation that "all that is needed is a single snapshot “db” file" and "all members should restore using the same snapshot."
https://etcd.io/docs/v3.5/op-guide/recovery/Doing like this made it work:
- Make one snapshot from a control-plane node.
- Stop etcd, api-server, kube-scheduler, kube-controller on both nodes.
- Delete etcd data-dir on both nodes to simulate data loss.
- Restore etcd data-dir on first node using
etcdctl snapshot restore
- Copy snapshot to second node and restore using
etcdctl snapshot restore
- Start etcd and verify
etcdctl endpoint health
,etcdctl endpoint status
,etcdctl member list
- Start api-server, kube-controller and kube-scheduler
- Restart kubelet
1
Answers
-
@pnts Could you provide output of
ETCDCTL_API=3 etcdctl -w table \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ member list
and
ETCDCTL_API=3 etcdctl -w table \ --endpoints <CP1_IP_ADDRESS>:2379,<CP2_IP_ADDRESS>:2379 \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ endpoint status
0
Categories
- All Categories
- 112 LFX Mentorship
- 112 LFX Mentorship: Linux Kernel
- 603 Linux Foundation IT Professional Programs
- 315 Cloud Engineer IT Professional Program
- 139 Advanced Cloud Engineer IT Professional Program
- 52 DevOps Engineer IT Professional Program
- 66 Cloud Native Developer IT Professional Program
- 3 Express Training Courses
- 3 Express Courses - Discussion Forum
- 5.3K Training Courses
- 17 LFC110 Class Forum - Discontinued
- 8 LFC131 Class Forum
- 30 LFD102 Class Forum
- 178 LFD103 Class Forum
- LFD110 Class Forum
- 24 LFD121 Class Forum
- LFD133 Class Forum
- 2 LFD137 Class Forum
- 62 LFD201 Class Forum
- 2 LFD210 Class Forum
- 1 LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 127 LFD232 Class Forum - Discontinued
- LFD233 Class Forum
- LFD237 Class Forum
- 22 LFD254 Class Forum
- 643 LFD259 Class Forum
- 107 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 4 LFD273 Class Forum
- 1 LFS101 Class Forum
- LFS112 Class Forum
- LFS116 Class Forum
- LFS118 Class Forum
- LFS144 Class Forum
- 2 LFS145 Class Forum
- LFS151 Class Forum
- LFS157 Class Forum
- LFS158 Class Forum
- LFS162 Class Forum
- LFS167 Class Forum
- 28 LFS200 Class Forum
- 735 LFS201 Class Forum - Discontinued
- 1 LFS201-JP クラス フォーラム
- 13 LFS203 Class Forum
- 98 LFS207 Class Forum
- 299 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 36 LFS243 Class Forum
- 12 LFS244 Class Forum
- LFS245 Class Forum
- 41 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 141 LFS253 Class Forum
- LFS254 Class Forum
- LFS255 Class Forum
- 2 LFS256 Class Forum
- LFS257 Class Forum
- 1.2K LFS258 Class Forum
- 9 LFS258-JP クラス フォーラム
- 109 LFS260 Class Forum
- 144 LFS261 Class Forum
- 39 LFS262 Class Forum
- 82 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 20 LFS267 Class Forum
- 18 LFS268 Class Forum
- 26 LFS269 Class Forum
- 198 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS274 Class Forum
- 3 LFS281 Class Forum
- 254 LFW211 Class Forum
- 173 LFW212 Class Forum
- 9 SKF100 Class Forum
- SKF200 Class Forum
- 781 Hardware
- 198 Drivers
- 68 I/O Devices
- 37 Monitors
- 95 Multimedia
- 174 Networking
- 87 Printers & Scanners
- 83 Storage
- 742 Linux Distributions
- 80 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 23 openSUSE
- 143 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 347 Ubuntu
- 450 Linux System Administration
- 31 Cloud Computing
- 69 Command Line/Scripting
- Github systems admin projects
- 89 Linux Security
- 76 Network Management
- 101 System Management
- 46 Web Management
- 51 Mobile Computing
- 18 Android
- 23 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 355 Off Topic
- 109 Introductions
- 167 Small Talk
- 18 Study Material
- 504 Programming and Development
- 283 Kernel Development
- 203 Software Development
- 844 Software
- 210 Applications
- 180 Command Line
- 3 Compiling/Installing
- 107 Games
- 308 Installation
- 51 All In Program
- 51 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)