Frequently Asked Question

Moving Containerd Storage
Last Updated 2 hours ago

To move the containerd data directory from /var/lib/containerd to /home/var/lib/containerd, the safest approach is:

  1. stop any services using containerd
  2. create the new directory
  3. copy the data with rsync
  4. update /etc/containerd/config.toml
  5. restart services and verify
  6. remove the old directory only after everything is working

This procedure applies to most Linux systems using systemd.

Important checks before starting

  • The destination should be on a local Linux filesystem such as ext4 or xfs.
  • Avoid using network filesystems such as NFS, CIFS or sshfs for containerd storage.
  • Ensure /home has enough free space for the existing containerd data.

Check available space:

df -h /var/lib/containerd /home
du -sh /var/lib/containerd

1. Stop containerd and any dependent services

On a standard host using only containerd:

systemctl stop containerd

On a Kubernetes node, stop kubelet first so it does not immediately try to use containerd again:

systemctl stop kubelet
systemctl stop containerd

Confirm the services have stopped:

systemctl status containerd --no-pager

On Kubernetes nodes:

systemctl status kubelet containerd --no-pager

2. Create the new destination directory

Create /home/var/lib/containerd with appropriate ownership and permissions:

install -d -m 0711 -o root -g root /home/var/lib/containerd

3. Copy the existing data with rsync

Use rsync so permissions, ownership, links and extended attributes are preserved:

rsync -aHAX --numeric-ids --info=progress2 /var/lib/containerd/ /home/var/lib/containerd/

Notes:

  • The trailing slash on /var/lib/containerd/ is important.
  • This copies the contents of the old directory into the new one.

After the copy completes, compare the sizes:

du -sh /var/lib/containerd /home/var/lib/containerd

4. Back up and edit /etc/containerd/config.toml

First, back up the current configuration if it exists:

cp -a /etc/containerd/config.toml /etc/containerd/config.toml.bak.$(date +%F-%H%M%S)

If /etc/containerd/config.toml does not already exist, generate a default one:

containerd config default > /etc/containerd/config.toml

Open the configuration file:

nano /etc/containerd/config.toml

Find the top-level root setting and change it from:

root = "/var/lib/containerd"

to:

root = "/home/var/lib/containerd"

The state setting should normally remain unchanged:

state = "/run/containerd"

The relevant section should look like this:

root = "/home/var/lib/containerd"
state = "/run/containerd"

If preferred, the root line can be changed non-interactively with:

sed -i 's#^root = "/var/lib/containerd"#root = "/home/var/lib/containerd"#' /etc/containerd/config.toml

Verify the change:

grep -E '^(root|state) =' /etc/containerd/config.toml

5. Apply SELinux context if SELinux is enabled

This is especially important when moving the data under /home, because SELinux may label it as normal home-directory content rather than container storage.

Check whether SELinux is enforcing:

sestatus

If SELinux is enabled, mirror the existing context from /var/lib/containerd to the new location and relabel it:

semanage fcontext -a -e /var/lib/containerd /home/var/lib/containerd
restorecon -R -v /home/var/lib/containerd

If the semanage command is not installed, install the relevant package for the distribution first. On many RHEL-based systems this is provided by:

dnf install -y policycoreutils-python-utils

6. Start containerd again

On a standard host:

systemctl start containerd

On a Kubernetes node:

systemctl start containerd
systemctl start kubelet

Check service status:

systemctl status containerd --no-pager -l

On Kubernetes nodes:

systemctl status containerd kubelet --no-pager -l

Review recent logs if the service does not start cleanly:

journalctl -u containerd -n 50 --no-pager

On Kubernetes nodes:

journalctl -u kubelet -n 50 --no-pager

7. Verify containerd is using the new location

Basic checks:

grep '^root =' /etc/containerd/config.toml
du -sh /home/var/lib/containerd

If ctr is available, confirm that objects are visible:

ctr containers list

On Kubernetes nodes, the k8s.io namespace is usually the relevant one:

ctr -n k8s.io containers list
ctr -n k8s.io images list

If crictl is installed, this is also useful on Kubernetes nodes:

crictl ps -a
crictl images

At this stage, workloads should be running normally and new data activity should appear under /home/var/lib/containerd.

8. Remove the old source directory

Do not remove /var/lib/containerd until containerd has restarted successfully and any containers or pods are confirmed healthy.

A cautious approach is to rename the old directory first:

mv /var/lib/containerd /var/lib/containerd.old

If everything continues to run correctly, remove it:

rm -rf /var/lib/containerd.old

If the original directory was left in place during testing and is no longer needed, remove it directly:

rm -rf /var/lib/containerd

Summary of the main command sequence

For a typical Kubernetes node, the full sequence is:

systemctl stop kubelet
systemctl stop containerd
install -d -m 0711 -o root -g root /home/var/lib/containerd
rsync -aHAX --numeric-ids --info=progress2 /var/lib/containerd/ /home/var/lib/containerd/
cp -a /etc/containerd/config.toml /etc/containerd/config.toml.bak.$(date +%F-%H%M%S)
sed -i 's#^root = "/var/lib/containerd"#root = "/home/var/lib/containerd"#' /etc/containerd/config.toml
semanage fcontext -a -e /var/lib/containerd /home/var/lib/containerd
restorecon -R -v /home/var/lib/containerd
systemctl start containerd
systemctl start kubelet
systemctl status containerd kubelet --no-pager -l

Then, once verified:

rm -rf /var/lib/containerd

Rollback method

If containerd fails to start after the move, revert by stopping services, restoring the old root path in /etc/containerd/config.toml, and starting the services again.

Typical rollback sequence:

systemctl stop kubelet
systemctl stop containerd
sed -i 's#^root = "/home/var/lib/containerd"#root = "/var/lib/containerd"#' /etc/containerd/config.toml
systemctl start containerd
systemctl start kubelet

If required, restore the original configuration backup:

cp -a /etc/containerd/config.toml.bak.YYYY-MM-DD-HHMMSS /etc/containerd/config.toml

Using the backup-and-verify approach before deleting the old directory provides the safest migration path.

This FAQ was generated and/or edited by GAIN, GENs Artificial Intelligence Network and should not be considered 100% accurate. Always check facts and do your research, things change all the time. If you are unsure about any information provided, please raise a support ticket for clarification.

This website relies on temporary cookies to function, but no personal data is ever stored in the cookies.
OK
Powered by GEN UK CLEAN GREEN ENERGY

Loading ...