See below how to defragment the etcd database of one node. In multi-node installations, repeat the procedure sequentially in all the nodes.

Defragmenting a live etcd member blocks the system from reading and writing data while rebuilding states. This operation can take several seconds, during which the whole cluster node is unavailable.

To defragment the etcd database in one node

  1. Get the current etcd revision. 

    rev=$(sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/client.crt --key /var/lib/rancher/k3s/server/tls/etcd/client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')

  2. Compact away all old etcd revisions. 

    sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/client.crt --key /var/lib/rancher/k3s/server/tls/etcd/client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt compact $rev

  3. Defragment etcd on the node. 

    sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/client.crt --key /var/lib/rancher/k3s/server/tls/etcd/client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt defrag

  4. Disarm the etcd database space alarm. 

    sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/client.crt --key /var/lib/rancher/k3s/server/tls/etcd/client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt alarm disarm