Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.shiftlabs.dev/llms.txt

Use this file to discover all available pages before exploring further.

Ceph nodes are the physical or virtual servers that run Ceph daemons and provide storage services. This page allows you to view node status, manage node membership, and perform maintenance operations on your Ceph cluster nodes.

​
Key Concepts

Node

A server running Ceph daemons (monitors, OSDs, managers) that participates in the cluster.

Maintenance Mode

A state where a node is temporarily taken offline for maintenance while the cluster remains operational.

OSD

Object Storage Daemon - the service that stores data on physical disks attached to a node.

Labels

Metadata tags assigned to nodes for service placement and organization.

​
Required Permissions

ActionPermission
View Nodesiam:project:infrastructure:ceph:read
Add Nodeiam:project:infrastructure:ceph:write
Remove Nodeiam:project:infrastructure:ceph:write
Enter Maintenanceiam:project:infrastructure:ceph:execute
Exit Maintenanceiam:project:infrastructure:ceph:execute

​
Node Status

StatusDescription
OnlineNode is healthy and participating in the cluster
OfflineNode is not responding or disconnected from the cluster
MaintenanceNode is in maintenance mode (daemons stopped, no data migration)

​
How to View Nodes

1

Select Cluster

Choose a Ceph cluster from the cluster dropdown. The first ready cluster is selected by default.
2

View Node List

The list shows all nodes in the cluster with their status, address, roles, and OSD count.
3

Filter and Search

Use the search box to find nodes by hostname, address, or role.
4

Review Statistics

Check the summary cards for:
  • Total Nodes: All nodes in the cluster
  • Online: Healthy and operational nodes
  • Maintenance: Nodes in maintenance mode
  • Offline: Unreachable or disconnected nodes

​
How to View Node Details

1

Click Node Name

Click on a node hostname in the list to open the detail page.
2

Review Dashboard

The detail page shows comprehensive node information:
  • Node Status: Current state (online, offline, maintenance)
  • OSDs: Number of Object Storage Daemons on this node
  • IP Address: Network address of the node
  • Services/Roles: Ceph services running on the node
3

View Hardware Information

Review hardware specifications:
  • CPU: Model, cores, threads
  • Memory: Total and available RAM
  • Architecture: CPU architecture type
4

View System Information

Check operating system details:
  • OS: Operating system and version
  • Kernel: Linux kernel version
  • Uptime: How long the node has been running
  • Security: SELinux/security settings
5

View Storage

Review attached storage:
  • HDD: Count and total capacity of hard disk drives
  • Flash/SSD: Count and total capacity of solid-state drives
  • Physical Disks: Detailed list of all attached disks
6

View Network Interfaces

Check network configuration:
  • Interface Name: Network interface identifier
  • Status: Operational state (up/down)
  • IPv4 Address: Assigned IP address
  • MTU: Maximum transmission unit size
  • Type: Interface type (ethernet, bond, etc.)

​
How to Add Nodes to a Cluster

Adding nodes to an existing Ceph cluster involves OS preparation, SSH trust setup, and registration with the Ceph orchestrator.
1

Register Node Settings

First, create a node entry in the system with connection details:
  • Hostname
  • IP address
  • SSH credentials
  • Node role
2

Click Add Node

Click the Add Node button in the page header.
3

Select Nodes

Choose which registered nodes to add to the cluster.
4

Start Add Operation

Click Add to begin the node addition process. The operation runs in the background and includes:
  1. Pre-addition validation and SSH connectivity check
  2. Operating system preparation
  3. Network configuration (/etc/hosts updates)
  4. SSH trust establishment from primary monitor
  5. Registration with Ceph orchestrator
5

Monitor Progress

Track the operation progress in the operation logs panel.
Adding nodes is a multi-phase operation that prepares the OS, establishes SSH trust, and registers the node with the Ceph orchestrator. Progress is tracked and logged throughout the process.

​
How to Remove Nodes from a Cluster

Removing nodes safely drains services and removes the node from the cluster.
1

Select Nodes

Select one or more nodes to remove using the checkboxes.
2

Click Remove Node(s)

Click the Remove Node(s) button. A wizard opens to guide you through the process.
3

Review Impact

The wizard shows which nodes will be removed and their current services.
4

Confirm Removal

Confirm the removal to start the operation. The process includes:
  1. Pre-removal validation
  2. Removal from CRUSH map
  3. Draining the host (stopping all daemons)
  4. Removal from orchestrator
  5. Database record update
5

Monitor Progress

Track the operation progress in the operation logs panel.
Removing a node will stop all Ceph services on that node. Ensure the cluster has sufficient capacity and redundancy before removing nodes. Data will be rebalanced to remaining nodes.

​
How to Enter Maintenance Mode

Maintenance mode allows you to perform hardware or software maintenance on a node without affecting cluster health.
1

Find the Node

Locate the node in the list.
2

Click Actions Menu

Click the actions menu (three dots) on the node row.
3

Select Enter Maintenance

Choose Enter Maintenance from the menu. For nodes with issues, use Force Enter Maintenance.
4

Confirm

Confirm the operation. The node will stop accepting new requests and drain existing operations.
Maintenance mode gracefully stops Ceph daemons on the node. The cluster will not attempt to rebalance data while the node is in maintenance mode, making this ideal for short-term maintenance.

​
How to Exit Maintenance Mode

1

Find the Node

Locate the node in maintenance mode.
2

Click Actions Menu

Click the actions menu on the node row.
3

Select Exit Maintenance

Choose Exit Maintenance from the menu.
4

Confirm

Confirm the operation. Ceph daemons will restart and the node will rejoin the cluster.

​
Node Configuration Fields

FieldDescription
HostnameThe node’s hostname as known to Ceph
AddressIP address for cluster communication
StatusCurrent operational state (online, offline, maintenance)
RolesCeph services/labels assigned to the node (mon, osd, mgr, etc.)
OSD CountNumber of OSDs running on this node
Ceph VersionVersion of Ceph software on the node

​
Detail Page Sections

​
Node Information Card

Shows basic node identity:
  • Hostname
  • IP address
  • Ceph version
  • Active services/daemons

​
Hardware Card

Displays hardware specifications:
  • CPU model, core count, thread count
  • Total and available memory
  • System architecture
  • Hardware vendor and model

​
System Card

Shows operating system details:
  • OS distribution and version
  • Kernel version
  • System uptime
  • Security configuration (SELinux)
  • FQDN

​
Storage Card

Summarizes attached storage:
  • HDD count and total capacity
  • Flash/SSD count and total capacity

​
Network Interfaces Card

Lists all network interfaces with:
  • Interface name
  • Operational status
  • IPv4 address
  • MTU setting
  • Interface type
  • Network driver

​
Physical Disks Card

Shows detailed disk information:
  • HDD List: All rotational drives with vendor, model, serial, size
  • Flash List: All solid-state drives with vendor, model, serial, size

​
Troubleshooting

  • Verify the node is powered on and network is connected
  • Check SSH connectivity to the node
  • Verify firewall rules allow Ceph ports (3300, 6789, 6800-7300)
  • Check if Ceph daemons are running on the node
  • Verify SSH credentials are correct
  • Ensure the node meets minimum requirements (disk space, memory, CPU)
  • Check that the node’s hostname is resolvable
  • Verify network connectivity between nodes
  • Check operation logs for specific errors
  • Ensure the cluster has enough capacity to handle data redistribution
  • Check if the node has the only copy of any data
  • Verify the primary monitor is accessible
  • Check operation logs for specific errors
  • Check if there are active operations on the node
  • Verify cluster health allows maintenance
  • Use force maintenance if regular maintenance fails
  • Ensure sufficient cluster redundancy
  • Verify the Ceph orchestrator is running
  • Check if there are pending operations on the node
  • Try the exit maintenance operation again
  • Check cluster logs for errors
  • The node may not have the facts/inventory data collected
  • Verify the Ceph orchestrator has access to the node
  • Check if the node has required packages installed
  • Try refreshing the page after a few minutes

​
FAQ

Maintenance mode is temporary - the node remains part of the cluster but stops serving requests. No data migration occurs, making it quick to enter and exit. Use this for short-term maintenance.Removing a node permanently removes it from the cluster. Data is migrated to other nodes, and the node must be re-added if you want to use it again. Use this when decommissioning hardware.
Use force maintenance when:
  • Regular maintenance mode fails
  • The node is having issues preventing graceful shutdown
  • You need to take the node offline urgently
Force maintenance may not wait for operations to complete cleanly.
Adding a node typically takes several minutes, depending on:
  • Network speed between nodes
  • OS preparation requirements
  • Number of existing nodes in the cluster
The operation logs show progress through each phase.
Yes. Select multiple nodes using the checkboxes and click Remove Node(s). The removal wizard will process each node sequentially to ensure cluster stability.
Labels control which Ceph services run on each node:
  • mon: Monitor daemon
  • osd: Object Storage Daemon
  • mgr: Manager daemon
  • mds: Metadata Server (for CephFS)
  • rgw: RADOS Gateway (for object storage)
The orchestrator uses labels to place services appropriately.
When a node is removed:
  1. OSDs on the node are marked out
  2. Data is redistributed to remaining OSDs
  3. The node is removed from CRUSH map
  4. The cluster rebalances to maintain redundancy
Ensure the cluster has sufficient capacity before removing nodes.
A node may show 0 OSDs if:
  • The node only runs monitor or manager services
  • Disks haven’t been added as OSDs yet
  • The node was recently added and OSDs haven’t been deployed
Use the OSDs page to add disks as OSDs if needed.
Yes. You can re-add a previously removed node by:
  1. Ensuring the node entry exists in the system
  2. Using the Add Node operation
  3. Optionally adding OSDs back if needed
Note that any data previously on the node’s OSDs will have been redistributed.