> ## Documentation Index
> Fetch the complete documentation index at: https://docs.shiftlabs.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Nodes

> Manage Ceph cluster nodes including hardware, services, and maintenance operations

Ceph nodes are the physical or virtual servers that run Ceph daemons and provide storage services. This page allows you to view node status, manage node membership, and perform maintenance operations on your Ceph cluster nodes.

## Key Concepts

<CardGroup cols={2}>
  <Card title="Node" icon="server">
    A server running Ceph daemons (monitors, OSDs, managers) that participates in the cluster.
  </Card>

  <Card title="Maintenance Mode" icon="wrench">
    A state where a node is temporarily taken offline for maintenance while the cluster remains operational.
  </Card>

  <Card title="OSD" icon="hard-drive">
    Object Storage Daemon - the service that stores data on physical disks attached to a node.
  </Card>

  <Card title="Labels" icon="tags">
    Metadata tags assigned to nodes for service placement and organization.
  </Card>
</CardGroup>

## Required Permissions

| Action            | Permission                                |
| ----------------- | ----------------------------------------- |
| View Nodes        | `iam:project:infrastructure:ceph:read`    |
| Add Node          | `iam:project:infrastructure:ceph:write`   |
| Remove Node       | `iam:project:infrastructure:ceph:write`   |
| Enter Maintenance | `iam:project:infrastructure:ceph:execute` |
| Exit Maintenance  | `iam:project:infrastructure:ceph:execute` |

## Node Status

| Status          | Description                                                      |
| --------------- | ---------------------------------------------------------------- |
| **Online**      | Node is healthy and participating in the cluster                 |
| **Offline**     | Node is not responding or disconnected from the cluster          |
| **Maintenance** | Node is in maintenance mode (daemons stopped, no data migration) |

## How to View Nodes

<Steps>
  <Step title="Select Cluster">
    Choose a Ceph cluster from the cluster dropdown. The first ready cluster is selected by default.
  </Step>

  <Step title="View Node List">
    The list shows all nodes in the cluster with their status, address, roles, and OSD count.
  </Step>

  <Step title="Filter and Search">
    Use the search box to find nodes by hostname, address, or role.
  </Step>

  <Step title="Review Statistics">
    Check the summary cards for:

    * **Total Nodes**: All nodes in the cluster
    * **Online**: Healthy and operational nodes
    * **Maintenance**: Nodes in maintenance mode
    * **Offline**: Unreachable or disconnected nodes
  </Step>
</Steps>

## How to View Node Details

<Steps>
  <Step title="Click Node Name">
    Click on a node hostname in the list to open the detail page.
  </Step>

  <Step title="Review Dashboard">
    The detail page shows comprehensive node information:

    * **Node Status**: Current state (online, offline, maintenance)
    * **OSDs**: Number of Object Storage Daemons on this node
    * **IP Address**: Network address of the node
    * **Services/Roles**: Ceph services running on the node
  </Step>

  <Step title="View Hardware Information">
    Review hardware specifications:

    * **CPU**: Model, cores, threads
    * **Memory**: Total and available RAM
    * **Architecture**: CPU architecture type
  </Step>

  <Step title="View System Information">
    Check operating system details:

    * **OS**: Operating system and version
    * **Kernel**: Linux kernel version
    * **Uptime**: How long the node has been running
    * **Security**: SELinux/security settings
  </Step>

  <Step title="View Storage">
    Review attached storage:

    * **HDD**: Count and total capacity of hard disk drives
    * **Flash/SSD**: Count and total capacity of solid-state drives
    * **Physical Disks**: Detailed list of all attached disks
  </Step>

  <Step title="View Network Interfaces">
    Check network configuration:

    * **Interface Name**: Network interface identifier
    * **Status**: Operational state (up/down)
    * **IPv4 Address**: Assigned IP address
    * **MTU**: Maximum transmission unit size
    * **Type**: Interface type (ethernet, bond, etc.)
  </Step>
</Steps>

## How to Add Nodes to a Cluster

Adding nodes to an existing Ceph cluster involves OS preparation, SSH trust setup, and registration with the Ceph orchestrator.

<Steps>
  <Step title="Register Node Settings">
    First, create a node entry in the system with connection details:

    * Hostname
    * IP address
    * SSH credentials
    * Node role
  </Step>

  <Step title="Click Add Node">
    Click the **Add Node** button in the page header.
  </Step>

  <Step title="Select Nodes">
    Choose which registered nodes to add to the cluster.
  </Step>

  <Step title="Start Add Operation">
    Click **Add** to begin the node addition process. The operation runs in the background and includes:

    1. Pre-addition validation and SSH connectivity check
    2. Operating system preparation
    3. Network configuration (/etc/hosts updates)
    4. SSH trust establishment from primary monitor
    5. Registration with Ceph orchestrator
  </Step>

  <Step title="Monitor Progress">
    Track the operation progress in the operation logs panel.
  </Step>
</Steps>

<Info>
  Adding nodes is a multi-phase operation that prepares the OS, establishes SSH trust, and registers the node with the Ceph orchestrator. Progress is tracked and logged throughout the process.
</Info>

## How to Remove Nodes from a Cluster

Removing nodes safely drains services and removes the node from the cluster.

<Steps>
  <Step title="Select Nodes">
    Select one or more nodes to remove using the checkboxes.
  </Step>

  <Step title="Click Remove Node(s)">
    Click the **Remove Node(s)** button. A wizard opens to guide you through the process.
  </Step>

  <Step title="Review Impact">
    The wizard shows which nodes will be removed and their current services.
  </Step>

  <Step title="Confirm Removal">
    Confirm the removal to start the operation. The process includes:

    1. Pre-removal validation
    2. Removal from CRUSH map
    3. Draining the host (stopping all daemons)
    4. Removal from orchestrator
    5. Database record update
  </Step>

  <Step title="Monitor Progress">
    Track the operation progress in the operation logs panel.
  </Step>
</Steps>

<Warning>
  Removing a node will stop all Ceph services on that node. Ensure the cluster has sufficient capacity and redundancy before removing nodes. Data will be rebalanced to remaining nodes.
</Warning>

## How to Enter Maintenance Mode

Maintenance mode allows you to perform hardware or software maintenance on a node without affecting cluster health.

<Steps>
  <Step title="Find the Node">
    Locate the node in the list.
  </Step>

  <Step title="Click Actions Menu">
    Click the actions menu (three dots) on the node row.
  </Step>

  <Step title="Select Enter Maintenance">
    Choose **Enter Maintenance** from the menu. For nodes with issues, use **Force Enter Maintenance**.
  </Step>

  <Step title="Confirm">
    Confirm the operation. The node will stop accepting new requests and drain existing operations.
  </Step>
</Steps>

<Info>
  Maintenance mode gracefully stops Ceph daemons on the node. The cluster will not attempt to rebalance data while the node is in maintenance mode, making this ideal for short-term maintenance.
</Info>

## How to Exit Maintenance Mode

<Steps>
  <Step title="Find the Node">
    Locate the node in maintenance mode.
  </Step>

  <Step title="Click Actions Menu">
    Click the actions menu on the node row.
  </Step>

  <Step title="Select Exit Maintenance">
    Choose **Exit Maintenance** from the menu.
  </Step>

  <Step title="Confirm">
    Confirm the operation. Ceph daemons will restart and the node will rejoin the cluster.
  </Step>
</Steps>

## Node Configuration Fields

| Field            | Description                                                     |
| ---------------- | --------------------------------------------------------------- |
| **Hostname**     | The node's hostname as known to Ceph                            |
| **Address**      | IP address for cluster communication                            |
| **Status**       | Current operational state (online, offline, maintenance)        |
| **Roles**        | Ceph services/labels assigned to the node (mon, osd, mgr, etc.) |
| **OSD Count**    | Number of OSDs running on this node                             |
| **Ceph Version** | Version of Ceph software on the node                            |

## Detail Page Sections

### Node Information Card

Shows basic node identity:

* Hostname
* IP address
* Ceph version
* Active services/daemons

### Hardware Card

Displays hardware specifications:

* CPU model, core count, thread count
* Total and available memory
* System architecture
* Hardware vendor and model

### System Card

Shows operating system details:

* OS distribution and version
* Kernel version
* System uptime
* Security configuration (SELinux)
* FQDN

### Storage Card

Summarizes attached storage:

* HDD count and total capacity
* Flash/SSD count and total capacity

### Network Interfaces Card

Lists all network interfaces with:

* Interface name
* Operational status
* IPv4 address
* MTU setting
* Interface type
* Network driver

### Physical Disks Card

Shows detailed disk information:

* **HDD List**: All rotational drives with vendor, model, serial, size
* **Flash List**: All solid-state drives with vendor, model, serial, size

## Troubleshooting

<AccordionGroup>
  <Accordion title="Node shows offline status">
    * Verify the node is powered on and network is connected
    * Check SSH connectivity to the node
    * Verify firewall rules allow Ceph ports (3300, 6789, 6800-7300)
    * Check if Ceph daemons are running on the node
  </Accordion>

  <Accordion title="Cannot add node to cluster">
    * Verify SSH credentials are correct
    * Ensure the node meets minimum requirements (disk space, memory, CPU)
    * Check that the node's hostname is resolvable
    * Verify network connectivity between nodes
    * Check operation logs for specific errors
  </Accordion>

  <Accordion title="Node removal fails">
    * Ensure the cluster has enough capacity to handle data redistribution
    * Check if the node has the only copy of any data
    * Verify the primary monitor is accessible
    * Check operation logs for specific errors
  </Accordion>

  <Accordion title="Cannot enter maintenance mode">
    * Check if there are active operations on the node
    * Verify cluster health allows maintenance
    * Use force maintenance if regular maintenance fails
    * Ensure sufficient cluster redundancy
  </Accordion>

  <Accordion title="Node stuck in maintenance mode">
    * Verify the Ceph orchestrator is running
    * Check if there are pending operations on the node
    * Try the exit maintenance operation again
    * Check cluster logs for errors
  </Accordion>

  <Accordion title="Hardware information not showing">
    * The node may not have the facts/inventory data collected
    * Verify the Ceph orchestrator has access to the node
    * Check if the node has required packages installed
    * Try refreshing the page after a few minutes
  </Accordion>
</AccordionGroup>

## FAQ

<AccordionGroup>
  <Accordion title="What is the difference between maintenance mode and removing a node?">
    **Maintenance mode** is temporary - the node remains part of the cluster but stops serving requests. No data migration occurs, making it quick to enter and exit. Use this for short-term maintenance.

    **Removing a node** permanently removes it from the cluster. Data is migrated to other nodes, and the node must be re-added if you want to use it again. Use this when decommissioning hardware.
  </Accordion>

  <Accordion title="When should I use Force Enter Maintenance?">
    Use force maintenance when:

    * Regular maintenance mode fails
    * The node is having issues preventing graceful shutdown
    * You need to take the node offline urgently

    Force maintenance may not wait for operations to complete cleanly.
  </Accordion>

  <Accordion title="How long does adding a node take?">
    Adding a node typically takes several minutes, depending on:

    * Network speed between nodes
    * OS preparation requirements
    * Number of existing nodes in the cluster

    The operation logs show progress through each phase.
  </Accordion>

  <Accordion title="Can I remove multiple nodes at once?">
    Yes. Select multiple nodes using the checkboxes and click **Remove Node(s)**. The removal wizard will process each node sequentially to ensure cluster stability.
  </Accordion>

  <Accordion title="What are node labels/roles used for?">
    Labels control which Ceph services run on each node:

    * **mon**: Monitor daemon
    * **osd**: Object Storage Daemon
    * **mgr**: Manager daemon
    * **mds**: Metadata Server (for CephFS)
    * **rgw**: RADOS Gateway (for object storage)

    The orchestrator uses labels to place services appropriately.
  </Accordion>

  <Accordion title="What happens to data when a node is removed?">
    When a node is removed:

    1. OSDs on the node are marked out
    2. Data is redistributed to remaining OSDs
    3. The node is removed from CRUSH map
    4. The cluster rebalances to maintain redundancy

    Ensure the cluster has sufficient capacity before removing nodes.
  </Accordion>

  <Accordion title="Why does the node show 0 OSDs?">
    A node may show 0 OSDs if:

    * The node only runs monitor or manager services
    * Disks haven't been added as OSDs yet
    * The node was recently added and OSDs haven't been deployed

    Use the OSDs page to add disks as OSDs if needed.
  </Accordion>

  <Accordion title="Can I add a node that was previously removed?">
    Yes. You can re-add a previously removed node by:

    1. Ensuring the node entry exists in the system
    2. Using the Add Node operation
    3. Optionally adding OSDs back if needed

    Note that any data previously on the node's OSDs will have been redistributed.
  </Accordion>
</AccordionGroup>