> ## Documentation Index
> Fetch the complete documentation index at: https://docs.shiftlabs.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# OSDs

> Manage Ceph Object Storage Daemons for distributed block storage

Object Storage Daemons (OSDs) are the core storage components of a Ceph cluster. Each OSD manages a physical disk and handles data replication, recovery, and rebalancing. This page allows you to view OSD status, add new OSDs, and safely remove existing ones.

## Key Concepts

<CardGroup cols={2}>
  <Card title="OSD" icon="hard-drive">
    Object Storage Daemon - a service that stores data on a physical disk and handles replication.
  </Card>

  <Card title="Device Class" icon="microchip">
    The type of storage device: HDD (rotational), SSD (solid-state), or NVMe (high-speed SSD).
  </Card>

  <Card title="Up/Down" icon="power-off">
    Whether the OSD process is running (Up) or stopped (Down).
  </Card>

  <Card title="In/Out" icon="arrows-left-right">
    Whether the OSD is participating in data placement (In) or excluded from it (Out).
  </Card>
</CardGroup>

## Required Permissions

| Action       | Permission                                |
| ------------ | ----------------------------------------- |
| View OSDs    | `iam:project:infrastructure:ceph:read`    |
| Add OSD      | `iam:project:infrastructure:ceph:write`   |
| Mark In/Out  | `iam:project:infrastructure:ceph:write`   |
| Reweight OSD | `iam:project:infrastructure:ceph:write`   |
| Scrub OSD    | `iam:project:infrastructure:ceph:execute` |
| Remove OSD   | `iam:project:infrastructure:ceph:delete`  |

## OSD Status

### Up/Down Status

| Status   | Description                             |
| -------- | --------------------------------------- |
| **Up**   | OSD daemon is running and responsive    |
| **Down** | OSD daemon is stopped or not responding |

### In/Out Status

| Status  | Description                                             |
| ------- | ------------------------------------------------------- |
| **In**  | OSD participates in data placement and receives data    |
| **Out** | OSD is excluded from data placement; data migrates away |

<Info>
  An OSD can be **Up** but **Out** - this means it's running but not receiving new data. This is commonly used during maintenance or before removal.
</Info>

## Device Classes

| Class    | Description                              |
| -------- | ---------------------------------------- |
| **HDD**  | Traditional rotational hard disk drive   |
| **SSD**  | Solid-state drive with faster random I/O |
| **NVMe** | High-performance NVMe solid-state drive  |

<Tip>
  Device classes are used by CRUSH rules to place data on specific storage types. For example, you can configure pools to use only SSDs for high-performance workloads.
</Tip>

## How to View OSDs

<Steps>
  <Step title="Select Cluster">
    Choose a Ceph cluster from the cluster dropdown. The first ready cluster is selected by default.
  </Step>

  <Step title="View OSD List">
    The table shows all OSDs with their status, host, device class, placement groups, and utilization.
  </Step>

  <Step title="Filter and Search">
    Use the search box to find OSDs by ID, hostname, or device class. Filter by status (Up, Down, In, Out).
  </Step>

  <Step title="Review Statistics">
    Check the summary cards for:

    * **Total OSDs**: All OSDs in the cluster
    * **Up**: Running OSD daemons
    * **Down**: Stopped or unresponsive OSDs
    * **In Cluster**: OSDs actively storing data
    * **Total Capacity**: Combined storage across all OSDs
    * **Avg Usage**: Average utilization percentage
  </Step>
</Steps>

## How to Add an OSD

Adding an OSD creates a new storage daemon on an available disk.

<Steps>
  <Step title="Click Add OSD">
    Click the **Add OSD** button in the page header.
  </Step>

  <Step title="View Available Disks">
    A drawer opens showing all available (unused) disks across cluster nodes.
  </Step>

  <Step title="Select Disks">
    Check the disks you want to use as OSDs. Each disk shows:

    * Device path (e.g., `/dev/sdb`)
    * Host where the disk is located
    * Disk size
    * Device type (HDD, SSD, NVMe)
  </Step>

  <Step title="Add OSDs">
    Click **Add OSDs** to create OSD daemons on the selected disks.
  </Step>

  <Step title="Monitor Progress">
    The operation runs in the background. The new OSDs will appear in the list once created.
  </Step>
</Steps>

<Info>
  Only available disks that don't already have OSDs are shown. If no disks appear, all disks may already be in use or there may be no OSD-role nodes in the cluster.
</Info>

## How to Mark an OSD Out

Marking an OSD "out" removes it from data placement, causing data to migrate to other OSDs.

<Steps>
  <Step title="Find the OSD">
    Locate the OSD in the list.
  </Step>

  <Step title="Open Actions Menu">
    Click the actions menu (dropdown arrow) on the OSD row.
  </Step>

  <Step title="Click Mark Out">
    Select **Mark Out** from the menu.
  </Step>

  <Step title="Wait for Migration">
    Data will begin migrating to other OSDs. Monitor cluster health until all placement groups are active+clean.
  </Step>
</Steps>

<Info>
  Mark Out is the first step in safely removing an OSD. It triggers data migration without stopping the OSD daemon.
</Info>

## How to Mark an OSD In

Marking an OSD "in" adds it back to data placement.

<Steps>
  <Step title="Find the OSD">
    Locate the OSD that is currently marked out.
  </Step>

  <Step title="Open Actions Menu">
    Click the actions menu on the OSD row.
  </Step>

  <Step title="Click Mark In">
    Select **Mark In** from the menu.
  </Step>

  <Step title="Wait for Rebalancing">
    Data will begin distributing to this OSD. The cluster will rebalance automatically.
  </Step>
</Steps>

## How to Scrub an OSD

Scrubbing verifies data integrity by comparing object replicas across OSDs.

<Steps>
  <Step title="Find the OSD">
    Locate the OSD to scrub.
  </Step>

  <Step title="Open Actions Menu">
    Click the actions menu on the OSD row.
  </Step>

  <Step title="Select Scrub Type">
    Choose either:

    * **Scrub**: Light verification of object metadata
    * **Deep Scrub**: Full verification including data checksums
  </Step>

  <Step title="Monitor Progress">
    The scrub runs in the background. Check cluster logs for results.
  </Step>
</Steps>

<Warning>
  Deep scrub is I/O intensive and may impact performance. Schedule deep scrubs during low-usage periods.
</Warning>

## How to Remove an OSD

Removing an OSD is a multi-step process to ensure data safety. A guided wizard walks you through the process.

<Steps>
  <Step title="Select OSD(s)">
    Select one or more OSDs using checkboxes, or click **Remove OSD** from the actions menu.
  </Step>

  <Step title="Click Remove">
    Click the **Remove** button to open the removal wizard.
  </Step>

  <Step title="Pre-flight Check">
    Review the selected OSDs and the amount of data that will need to migrate.
  </Step>

  <Step title="Safety Check">
    The system checks if removal is safe:

    * **Safe to Destroy**: All data has sufficient replicas elsewhere
    * **Not Safe**: Removal would cause data loss (requires force removal)
  </Step>

  <Step title="Data Migration">
    Click **Start Migration** to mark OSDs out and begin data migration. Wait for all placement groups to reach active+clean state.
  </Step>

  <Step title="Confirm Removal">
    Type **REMOVE** to confirm you want to permanently remove the OSDs.
  </Step>

  <Step title="Cleanup Commands">
    After removal, the wizard provides `ceph orch device zap` commands to clean the disks for reuse.
  </Step>
</Steps>

<Warning>
  **DESTRUCTIVE OPERATION**: Removing an OSD is permanent. Ensure data migration is complete before confirming removal. Skipping migration may cause data loss.
</Warning>

## OSD Table Fields

| Field      | Description                                    |
| ---------- | ---------------------------------------------- |
| **OSD**    | OSD identifier (e.g., osd.0, osd.1)            |
| **Host**   | Node where the OSD is running                  |
| **Class**  | Device class (HDD, SSD, NVMe)                  |
| **Status** | Up/Down and In/Out status badges               |
| **PGs**    | Number of placement groups on this OSD         |
| **Size**   | Total capacity of the OSD                      |
| **Usage**  | Current utilization percentage with visual bar |

## Removal Wizard Steps

### Step 1: Pre-flight Check

Shows selected OSDs with:

* OSD ID and hostname
* Device class
* Data to migrate
* Placement group count

### Step 2: Safety Check

Verifies if OSDs can be safely destroyed:

* Checks replica counts for all affected placement groups
* Shows "Safe to Destroy" or "Not Safe to Destroy"
* Option to force removal if unsafe (not recommended)

### Step 3: Data Migration

* Marks OSDs as "out"
* Monitors data migration progress
* Shows when all placement groups are active+clean
* Option to skip waiting (may cause data loss)

### Step 4: Confirm Removal

* Requires typing REMOVE to confirm
* Option to force removal (skips safety checks)
* Executes OSD deletion

### Step 5: Cleanup

* Shows completion status
* Provides zap commands for disk cleanup
* Commands can be copied to clipboard

## Troubleshooting

<AccordionGroup>
  <Accordion title="OSD shows Down status">
    * Check if the OSD host is accessible
    * Verify the OSD daemon is running: `systemctl status ceph-osd@<id>`
    * Check for disk failures or hardware issues
    * Review OSD logs for errors
  </Accordion>

  <Accordion title="No available disks for new OSD">
    * All disks may already have OSDs
    * Ensure the node has the OSD role assigned
    * Check if disks are properly detected by the system
    * Verify disks aren't mounted or in use by other services
  </Accordion>

  <Accordion title="OSD utilization is very high">
    * Consider adding more OSDs to the cluster
    * Check if other OSDs are down or out
    * Verify CRUSH rules are distributing data evenly
    * Consider reweighting OSDs to balance load
  </Accordion>

  <Accordion title="Data migration is slow">
    * This is normal for large amounts of data
    * Check network bandwidth between nodes
    * Verify no backfill/recovery throttling is set too low
    * Monitor `ceph status` for recovery progress
  </Accordion>

  <Accordion title="Cannot remove OSD - not safe to destroy">
    * Some placement groups don't have enough replicas
    * Wait for recovery to complete
    * Check if other OSDs are down
    * Use force removal only if you accept potential data loss
  </Accordion>

  <Accordion title="OSD removal stuck">
    * Check cluster health for blocking issues
    * Verify network connectivity
    * Check if the OSD daemon is still running
    * Review operation logs for specific errors
  </Accordion>

  <Accordion title="Scrub taking too long">
    * Deep scrub is I/O intensive on large OSDs
    * Check OSD performance and disk health
    * Consider adjusting scrub scheduling options
    * Large OSDs with many objects take longer
  </Accordion>
</AccordionGroup>

## FAQ

<AccordionGroup>
  <Accordion title="What is the difference between Up/Down and In/Out?">
    **Up/Down** indicates whether the OSD daemon process is running.

    **In/Out** indicates whether the OSD participates in data placement.

    An OSD can be:

    * **Up + In**: Normal operation, storing and serving data
    * **Up + Out**: Running but not receiving data (draining)
    * **Down + In**: Not running but expected to return (temporary failure)
    * **Down + Out**: Not running and excluded from placement
  </Accordion>

  <Accordion title="When should I mark an OSD out vs remove it?">
    **Mark Out** when:

    * Performing temporary maintenance
    * The OSD will return to service
    * You want to drain data without removing the OSD

    **Remove** when:

    * Decommissioning a disk permanently
    * Replacing failed hardware
    * The OSD will not return to service
  </Accordion>

  <Accordion title="What happens when I add a new OSD?">
    When you add an OSD:

    1. The Ceph orchestrator deploys an OSD daemon on the disk
    2. The OSD is added to the CRUSH map
    3. The cluster begins rebalancing data to include the new OSD
    4. Data gradually distributes across all OSDs
  </Accordion>

  <Accordion title="How long does OSD removal take?">
    Removal time depends on:

    * Amount of data on the OSD
    * Network speed between nodes
    * Number of remaining OSDs
    * Current cluster load

    Small OSDs may complete in minutes; large OSDs can take hours.
  </Accordion>

  <Accordion title="What is the 'safe to destroy' check?">
    This check verifies that removing the OSD won't cause data loss:

    * Checks all placement groups on the OSD
    * Ensures each PG has sufficient replicas on other OSDs
    * If any PG would lose its last copy, removal is blocked

    Force removal bypasses this check but may cause data loss.
  </Accordion>

  <Accordion title="What are the zap commands for?">
    After removing an OSD, the disk may still have Ceph metadata. The `ceph orch device zap` command:

    * Removes all Ceph data from the disk
    * Clears partition tables and LVM data
    * Makes the disk available for reuse

    This is optional but recommended before reusing the disk.
  </Accordion>

  <Accordion title="Should I use force removal?">
    **Avoid force removal** unless you:

    * Understand the risk of data loss
    * Have backups of critical data
    * Are removing already-failed OSDs
    * Accept that some data may be lost

    Normal removal with data migration is always safer.
  </Accordion>

  <Accordion title="What is OSD reweight?">
    Reweight adjusts how much data an OSD receives (0.0 to 1.0):

    * **1.0**: Full weight, normal data distribution
    * **0.5**: Half weight, receives half the normal data
    * **0.0**: No data (equivalent to marking out)

    Use reweight to gradually drain an OSD or balance uneven utilization.
  </Accordion>

  <Accordion title="When should I run a deep scrub?">
    Deep scrub verifies data integrity at the bit level:

    * Run periodically for data verification
    * After suspected disk issues
    * When data corruption is suspected

    Deep scrubs are scheduled automatically but can be run manually.
  </Accordion>
</AccordionGroup>