Frequently Asked Question

Moving Containerd Storage
Last Updated 6 months ago

To move the containerd data directory from /var/lib/containerd to /home/var/lib/containerd, the safest approach is:

  1. stop any services using containerd
  2. create the new directory
  3. copy the data with rsync
  4. update /etc/containerd/config.toml
  5. restart services and verify
  6. remove the old directory only after everything is working

This procedure applies to most Linux systems using systemd.

Important checks before starting

  • The destination should be on a local Linux filesystem such as ext4 or xfs.
  • Avoid using network filesystems such as NFS, CIFS or sshfs for containerd storage.
  • Ensure /home has enough free space for the existing containerd data.

Check available space:

df -h /var/lib/containerd /home
du -sh /var/lib/containerd

1. Stop containerd and any dependent services

On a standard host using only containerd:

systemctl stop containerd

On a Kubernetes node, stop kubelet first so it does not immediately try to use containerd again:

systemctl stop kubelet
systemctl stop containerd

Confirm the services have stopped:

systemctl status containerd --no-pager

On Kubernetes nodes:

systemctl status kubelet containerd --no-pager

2. Create the new destination directory

Create /home/var/lib/containerd with appropriate ownership and permissions:

install -d -m 0711 -o root -g root /home/var/lib/containerd

3. Copy the existing data with rsync

Use rsync so permissions, ownership, links and extended attributes are preserved:

rsync -aHAX --numeric-ids --info=progress2 /var/lib/containerd/ /home/var/lib/containerd/

Notes:

  • The trailing slash on /var/lib/containerd/ is important.
  • This copies the contents of the old directory into the new one.

After the copy completes, compare the sizes:

du -sh /var/lib/containerd /home/var/lib/containerd

4. Back up and edit /etc/containerd/config.toml

First, back up the current configuration if it exists:

cp -a /etc/containerd/config.toml /etc/containerd/config.toml.bak.$(date +%F-%H%M%S)

If /etc/containerd/config.toml does not already exist, generate a default one:

containerd config default > /etc/containerd/config.toml

Open the configuration file:

nano /etc/containerd/config.toml

Find the top-level root setting and change it from:

root = "/var/lib/containerd"

to:

root = "/home/var/lib/containerd"

The state setting should normally remain unchanged:

state = "/run/containerd"

The relevant section should look like this:

root = "/home/var/lib/containerd"
state = "/run/containerd"

If preferred, the root line can be changed non-interactively with:

sed -i 's#^root = "/var/lib/containerd"#root = "/home/var/lib/containerd"#' /etc/containerd/config.toml

Verify the change:

grep -E '^(root|state) =' /etc/containerd/config.toml

5. Apply SELinux context if SELinux is enabled

This is especially important when moving the data under /home, because SELinux may label it as normal home-directory content rather than container storage.

Check whether SELinux is enforcing:

sestatus

If SELinux is enabled, mirror the existing context from /var/lib/containerd to the new location and relabel it:

semanage fcontext -a -e /var/lib/containerd /home/var/lib/containerd
restorecon -R -v /home/var/lib/containerd

If the semanage command is not installed, install the relevant package for the distribution first. On many RHEL-based systems this is provided by:

dnf install -y policycoreutils-python-utils

6. Start containerd again

On a standard host:

systemctl start containerd

On a Kubernetes node:

systemctl start containerd
systemctl start kubelet

Check service status:

systemctl status containerd --no-pager -l

On Kubernetes nodes:

systemctl status containerd kubelet --no-pager -l

Review recent logs if the service does not start cleanly:

journalctl -u containerd -n 50 --no-pager

On Kubernetes nodes:

journalctl -u kubelet -n 50 --no-pager

7. Verify containerd is using the new location

Basic checks:

grep '^root =' /etc/containerd/config.toml
du -sh /home/var/lib/containerd

If ctr is available, confirm that objects are visible:

ctr containers list

On Kubernetes nodes, the k8s.io namespace is usually the relevant one:

ctr -n k8s.io containers list
ctr -n k8s.io images list

If crictl is installed, this is also useful on Kubernetes nodes:

crictl ps -a
crictl images

At this stage, workloads should be running normally and new data activity should appear under /home/var/lib/containerd.

8. Remove the old source directory

Do not remove /var/lib/containerd until containerd has restarted successfully and any containers or pods are confirmed healthy.

A cautious approach is to rename the old directory first:

mv /var/lib/containerd /var/lib/containerd.old

If everything continues to run correctly, remove it:

rm -rf /var/lib/containerd.old

If the original directory was left in place during testing and is no longer needed, remove it directly:

rm -rf /var/lib/containerd

Summary of the main command sequence

For a typical Kubernetes node, the full sequence is:

systemctl stop kubelet
systemctl stop containerd
install -d -m 0711 -o root -g root /home/var/lib/containerd
rsync -aHAX --numeric-ids --info=progress2 /var/lib/containerd/ /home/var/lib/containerd/
cp -a /etc/containerd/config.toml /etc/containerd/config.toml.bak.$(date +%F-%H%M%S)
sed -i 's#^root = "/var/lib/containerd"#root = "/home/var/lib/containerd"#' /etc/containerd/config.toml
semanage fcontext -a -e /var/lib/containerd /home/var/lib/containerd
restorecon -R -v /home/var/lib/containerd
systemctl start containerd
systemctl start kubelet
systemctl status containerd kubelet --no-pager -l

Then, once verified:

rm -rf /var/lib/containerd

Rollback method

If containerd fails to start after the move, revert by stopping services, restoring the old root path in /etc/containerd/config.toml, and starting the services again.

Typical rollback sequence:

systemctl stop kubelet
systemctl stop containerd
sed -i 's#^root = "/home/var/lib/containerd"#root = "/var/lib/containerd"#' /etc/containerd/config.toml
systemctl start containerd
systemctl start kubelet

If required, restore the original configuration backup:

cp -a /etc/containerd/config.toml.bak.YYYY-MM-DD-HHMMSS /etc/containerd/config.toml

Using the backup-and-verify approach before deleting the old directory provides the safest migration path.

This FAQ was generated and/or edited by GAIN, GENs Artificial Intelligence Network and should not be considered 100% accurate. Always check facts and do your research, things change all the time. If you are unsure about any information provided, please raise a support ticket for clarification.

Powered by GEN UK CLEAN GREEN ENERGY

Loading ...