Frequently Asked Question

HelpDesk Debug Ceph
Last Updated 3 hours ago

When raising a Ceph support ticket, attach a standard diagnostic capture taken as close as possible to the time of the issue. This provides an immediate snapshot of cluster health, quorum state, MON/MGR status, OSD state, pools, RADOS, CephFS and local service status.

General guidance

  • Run the commands from a host that has working Ceph CLI access, typically an admin node or a monitor.
  • Use a user with sufficient privileges to query the cluster.
  • Capture the output exactly as returned, including warnings and errors.
  • If the issue is transient, collect the information while the problem is occurring if possible.
  • If the cluster uses a non-default configuration or keyring, include the appropriate options in each command.

A useful approach is to save all command output into a single text file and attach that file to the ticket. Most good terminal tools can do this by default. 

Minimum information to include in the ticket

Also include the following non-command information in the ticket body:

  • Time the issue started
  • Whether the issue is ongoing or intermittent
  • Symptoms observed by users or applications
  • Recent changes, for example:
  • node reboot
  • package updates
  • network changes
  • disk replacement
  • OSD maintenance
  • monitor or manager failover
  • Impacted services:
  • RBD
  • CephFS
  • RGW
  • backups
  • virtual machines

Core Ceph commands to attach

Run all of the following and attach the output.

1. Cluster health and overall status

ceph -s
ceph health
ceph health detail
ceph df
ceph osd df
ceph osd perf

These show:

  • overall health summary
  • detailed health warnings and errors
  • cluster usage
  • OSD utilisation balance
  • OSD latency figures

2. Quorum and monitor status

ceph quorum_status -f json-pretty
ceph mon stat
ceph mon dump
ceph mon metadata

These show:

  • which monitors are in quorum
  • monitor election and membership details
  • monitor addresses and ranks
  • monitor daemon metadata

3. Manager status and metadata

ceph mgr stat
ceph mgr dump
ceph mgr metadata

These show:

  • active and standby managers
  • manager module and service state
  • manager daemon metadata

4. OSD state and topology

ceph osd stat
ceph osd tree
ceph osd dump
ceph osd crush tree
ceph osd crush rule dump
ceph osd pool ls detail

These show:

  • OSD up/in counts
  • OSD hierarchy by host and crush location
  • OSD map and flags
  • CRUSH topology and rules
  • pool configuration details

5. Placement groups and recovery state

ceph pg stat
ceph pg dump pgs_brief
ceph osd blocked-by

These show:

  • PG state summary
  • individual PG states
  • any blocked OSD relationships

If the cluster is large and pg dump output is very large, it is still useful, but compress the output before attaching if needed.

6. RADOS and pool checks

rados df
rados lspools

These show:

  • per-pool object and space usage
  • list of pools visible to the client

7. CephFS status (if applicable)

If CephFS is in use, include:

ceph fs status
ceph fs ls
ceph fs dump
ceph mds stat
ceph mds metadata

These show:

  • filesystem health and active ranks
  • filesystem definitions
  • MDS map and failover information
  • MDS daemon metadata

8. RBD status (if applicable)

If RBD is in use, include:

rbd ls -p <pool-name>
rbd du -p <pool-name>

Use the affected RBD pool or pools. This helps confirm image visibility and space usage.

Service status from each Ceph node

For the node where the issue is observed, and ideally for all Ceph nodes, attach service state output.

Systems using systemd

systemctl status ceph.target
systemctl status 'ceph*.service'
systemctl --type=service --state=failed | grep ceph
systemctl list-units --type=service | grep ceph

These show:

  • whether the Ceph target is active
  • individual Ceph service states
  • failed Ceph services
  • loaded Ceph-related units

If using cephadm or containerised Ceph

If the cluster is managed with cephadm, also include:

ceph orch status
ceph orch ps
ceph orch host ls

These show:

  • orchestrator health
  • daemon placement and runtime state
  • known hosts in the cephadm inventory

Optional but strongly recommended commands

These are often very useful for diagnosis and should be included where practical.

Cluster versions and report

ceph versions
ceph report

Notes:

  • ceph report is extremely useful because it contains a broad cluster snapshot.
  • The output can be large, so save it to a file and compress it if necessary.

Configuration snapshot

ceph config dump
ceph config get mon
ceph config get mgr

If daemon-specific settings are relevant, collect them for affected daemons as well.

Recent crash information

ceph crash ls
ceph crash info <crash-id>

Include crash details for any recent entries around the time of the issue.

Suggested collection method

The following example creates a timestamped text file containing the most useful outputs.

TS=$(date +%F-%H%M%S)
OUT="ceph-helpdesk-${TS}.txt"

{
  echo "===== DATE ====="
  date

  echo "===== HOSTNAME ====="
  hostname -f

  echo "===== CEPH STATUS ====="
  ceph -s
  ceph health
  ceph health detail
  ceph df
  ceph osd df
  ceph osd perf

  echo "===== QUORUM / MON ====="
  ceph quorum_status -f json-pretty
  ceph mon stat
  ceph mon dump
  ceph mon metadata

  echo "===== MGR ====="
  ceph mgr stat
  ceph mgr dump
  ceph mgr metadata

  echo "===== OSD ====="
  ceph osd stat
  ceph osd tree
  ceph osd dump
  ceph osd crush tree
  ceph osd crush rule dump
  ceph osd pool ls detail

  echo "===== PG ====="
  ceph pg stat
  ceph pg dump pgs_brief
  ceph osd blocked-by

  echo "===== RADOS ====="
  rados df
  rados lspools

  echo "===== CEPHFS ====="
  ceph fs status
  ceph fs ls
  ceph fs dump
  ceph mds stat
  ceph mds metadata

  echo "===== ORCHESTRATOR ====="
  ceph orch status
  ceph orch ps
  ceph orch host ls

  echo "===== VERSIONS / REPORT ====="
  ceph versions
  ceph report

  echo "===== CONFIG ====="
  ceph config dump

  echo "===== CRASH ====="
  ceph crash ls

  echo "===== SYSTEMD ====="
  systemctl status ceph.target
  systemctl status 'ceph*.service'
  systemctl --type=service --state=failed | grep ceph
  systemctl list-units --type=service | grep ceph
} > "${OUT}" 2>&1

Compress the file before attaching:

gzip "${OUT}"

If the issue relates to one specific area

For monitor quorum issues

Ensure these are present:

ceph quorum_status -f json-pretty
ceph mon stat
ceph mon dump
systemctl status 'ceph-mon*.service'

For manager failover or dashboard issues

Ensure these are present:

ceph mgr stat
ceph mgr dump
ceph mgr metadata
systemctl status 'ceph-mgr*.service'

For OSD down, full or recovery issues

Ensure these are present:

ceph osd stat
ceph osd tree
ceph osd dump
ceph osd df
ceph osd perf
ceph pg stat
ceph pg dump pgs_brief
systemctl status 'ceph-osd*.service'

For CephFS issues

Ensure these are present:

ceph fs status
ceph fs dump
ceph mds stat
ceph mds metadata
systemctl status 'ceph-mds*.service'

For client, pool or object access issues

Ensure these are present:

ceph osd pool ls detail
rados lspools
rados df

If one pool is affected, also include relevant pool-specific client commands where available.

Important notes

  • Some commands may return errors if a subsystem is not in use, for example CephFS or cephadm. Include that output anyway.
  • Some outputs contain hostnames, IP addresses, pool names and crash metadata. Review internal data handling requirements before sharing externally.
  • Large outputs such as ceph report, ceph pg dump pgs_brief and ceph config dump should be attached as files rather than pasted into the ticket body.
  • If the cluster is unhealthy enough that some commands hang, attach the output from the commands that do complete and note which commands did not return.

This website relies on temporary cookies to function, but no personal data is ever stored in the cookies.
OK
Powered by GEN UK CLEAN GREEN ENERGY

Loading ...