> ## Documentation Index
> Fetch the complete documentation index at: https://docs.shiftlabs.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade

> Upgrade Ceph clusters to newer versions with controlled rollout

The Upgrade page allows you to upgrade your Ceph cluster to newer versions. Upgrades are performed in a controlled manner, updating one daemon at a time to minimize disruption. You can monitor progress, pause, resume, or stop upgrades as needed.

## Key Concepts

<CardGroup cols={2}>
  <Card title="Rolling Upgrade" icon="rotate">
    Ceph upgrades one daemon at a time, maintaining cluster availability throughout the process.
  </Card>

  <Card title="Target Version" icon="bullseye">
    The Ceph version or container image you are upgrading to.
  </Card>

  <Card title="Daemon" icon="gear">
    A Ceph service process (mon, mgr, osd, mds, rgw) that gets upgraded individually.
  </Card>

  <Card title="Upgrade Blockers" icon="hand">
    Conditions that prevent an upgrade from starting, such as cluster health errors.
  </Card>
</CardGroup>

## Required Permissions

| Action              | Permission                              |
| ------------------- | --------------------------------------- |
| View Upgrade Status | `iam:project:infrastructure:ceph:read`  |
| Start Upgrade       | `iam:project:infrastructure:ceph:write` |
| Pause Upgrade       | `iam:project:infrastructure:ceph:write` |
| Resume Upgrade      | `iam:project:infrastructure:ceph:write` |
| Stop Upgrade        | `iam:project:infrastructure:ceph:write` |
| View History        | `iam:project:infrastructure:ceph:read`  |

## Upgrade Status

| Status          | Description                                     |
| --------------- | ----------------------------------------------- |
| **Up to Date**  | Cluster is running the latest available version |
| **Available**   | Newer version is available for upgrade          |
| **In Progress** | Upgrade is currently running                    |
| **Paused**      | Upgrade is paused and can be resumed            |

## How to Check Available Upgrades

<Steps>
  <Step title="Select Cluster">
    Choose a Ceph cluster from the cluster dropdown.
  </Step>

  <Step title="View Current Status">
    Review the statistics cards showing:

    * **Current Version**: The version currently running
    * **Target**: Available upgrade target or "Up to Date"
    * **Progress**: Percentage complete (if upgrading)
    * **Daemons**: Number of daemons in the cluster
    * **Health**: Current cluster health status
  </Step>

  <Step title="Check for Blockers">
    If blockers exist, they will be displayed with warnings. Common blockers include:

    * **HEALTH\_ERR**: Cluster has critical health errors
    * Active maintenance operations
    * Insufficient redundancy
  </Step>
</Steps>

## How to Start an Upgrade

<Steps>
  <Step title="Click Start Upgrade">
    Click the **Start Upgrade** button to open the upgrade wizard.
  </Step>

  <Step title="Pre-flight Check">
    The wizard checks cluster health and readiness:

    * Cluster health status (HEALTH\_OK, HEALTH\_WARN, HEALTH\_ERR)
    * Critical blockers are highlighted

    <Warning>
      Critical blockers (like HEALTH\_ERR) must be resolved before upgrading. Non-critical warnings allow proceeding with caution.
    </Warning>
  </Step>

  <Step title="Select Version">
    Choose your upgrade target:

    * **Available Versions**: Pre-defined versions from the registry
    * **Custom Image**: Specify a container image URL directly

    The recommended version is highlighted if available.
  </Step>

  <Step title="Confirm Upgrade">
    Type **UPGRADE** to confirm you want to proceed. This prevents accidental upgrades.
  </Step>

  <Step title="Monitor Progress">
    The wizard shows real-time progress:

    * Overall progress percentage
    * Current daemon being upgraded
    * Daemon upgrade status (pending, upgrading, upgraded)
  </Step>
</Steps>

<Warning>
  **IMPORTANT**: Upgrades can take significant time depending on cluster size. Do not interrupt power or network during the upgrade process.
</Warning>

## How to Use Custom Image Upgrades

Custom image upgrades allow you to specify an exact container image URL, useful for:

* Testing pre-release versions
* Using custom-built images
* Air-gapped environments with private registries

<Steps>
  <Step title="Start Upgrade Wizard">
    Click **Start Upgrade** and complete the pre-flight check.
  </Step>

  <Step title="Select Custom Image">
    Toggle the **Use Custom Image** option in the version selection step.
  </Step>

  <Step title="Enter Image URL">
    Enter the full container image URL, for example:

    * `quay.io/ceph/ceph:v18.2.4`
    * `registry.example.com/ceph/ceph:v18.2.4`
  </Step>

  <Step title="Confirm and Start">
    Complete the confirmation step to begin the upgrade.
  </Step>
</Steps>

## How to Pause an Upgrade

Pausing stops the upgrade at the current daemon while maintaining cluster stability.

<Steps>
  <Step title="Locate Pause Button">
    During an active upgrade, the **Pause** button appears in the upgrade controls.
  </Step>

  <Step title="Click Pause">
    Click **Pause** to stop the upgrade process. The current daemon will complete before pausing.
  </Step>

  <Step title="Verify Paused State">
    The status changes to "Paused" and the cluster remains in a mixed-version state.
  </Step>
</Steps>

<Info>
  Pausing is useful when you need to investigate issues or perform other maintenance. The cluster remains operational during the pause.
</Info>

## How to Resume an Upgrade

<Steps>
  <Step title="Verify Paused State">
    Ensure the upgrade shows as "Paused" in the status.
  </Step>

  <Step title="Click Resume">
    Click the **Resume** button to continue the upgrade from where it stopped.
  </Step>

  <Step title="Monitor Progress">
    The upgrade continues with the next daemon in sequence.
  </Step>
</Steps>

## How to Stop an Upgrade

Stopping an upgrade cancels the process entirely.

<Steps>
  <Step title="Click Stop">
    Click the **Stop** button during an active or paused upgrade.
  </Step>

  <Step title="Confirm Stop">
    Confirm that you want to stop the upgrade.
  </Step>
</Steps>

<Warning>
  Stopping an upgrade leaves the cluster in a mixed-version state. While Ceph supports mixed versions, it's recommended to either complete the upgrade or roll back manually.
</Warning>

## How to View Upgrade History

<Steps>
  <Step title="Navigate to History Tab">
    Click the **History** tab on the Upgrade page.
  </Step>

  <Step title="Review Past Upgrades">
    The history table shows:

    * From/To versions
    * Status (success, failed, cancelled)
    * Start and completion times
    * Duration
    * Number of daemons upgraded
  </Step>

  <Step title="Filter Results">
    Use search and pagination to find specific upgrade records.
  </Step>
</Steps>

## Statistics Cards

### Current Version

The Ceph version currently running on the cluster (e.g., 18.2.2 reef).

### Target/Available

Shows either:

* **Up to Date**: No newer version available
* **Available**: A newer version is available to upgrade to
* **Target**: The version being upgraded to (during upgrade)

### Progress

Percentage of daemons that have been upgraded. Only shown during active upgrades.

### Daemons

Total number of Ceph daemons that will be upgraded.

### Health

Current cluster health status:

* **HEALTH\_OK**: All healthy
* **HEALTH\_WARN**: Warnings present (upgrade can proceed with caution)
* **HEALTH\_ERR**: Critical issues (upgrade blocked)

## Upgrade Order

Ceph upgrades daemons in a specific order to maintain cluster stability:

1. **Managers (mgr)** - Updated first as they coordinate the upgrade
2. **Monitors (mon)** - Quorum is maintained throughout
3. **OSDs** - Updated one at a time to preserve data availability
4. **Metadata Servers (mds)** - For CephFS clusters
5. **RADOS Gateways (rgw)** - For object storage clusters

## Troubleshooting

<AccordionGroup>
  <Accordion title="Upgrade shows 'Upgrade Blocked'">
    * Check cluster health with `ceph health detail`
    * Resolve any HEALTH\_ERR conditions
    * Ensure all OSDs are up and in
    * Verify no other operations are in progress
  </Accordion>

  <Accordion title="Upgrade is very slow">
    * Each daemon restart takes time
    * Large OSDs may take longer to restart
    * Check for recovery/backfill operations
    * Network bandwidth affects image download speed
  </Accordion>

  <Accordion title="Upgrade stuck on a daemon">
    * Check the daemon's host is accessible
    * Verify the container image is available
    * Review daemon logs on the host
    * Consider pausing and investigating
  </Accordion>

  <Accordion title="Cluster unhealthy after partial upgrade">
    * Mixed versions are supported but not ideal
    * Resume the upgrade to complete it
    * If issues persist, check version compatibility
    * Contact support for rollback procedures
  </Accordion>

  <Accordion title="Custom image not pulling">
    * Verify the image URL is correct
    * Check registry authentication
    * Ensure nodes can reach the registry
    * Test with `podman pull <image>` on a node
  </Accordion>

  <Accordion title="Cannot find available versions">
    * Versions are managed by the system administrator
    * Check if versions are marked as active in settings
    * Verify your cluster's current version name matches version groups
  </Accordion>
</AccordionGroup>

## FAQ

<AccordionGroup>
  <Accordion title="How long does an upgrade take?">
    Upgrade duration depends on:

    * Number of daemons in the cluster
    * OSD sizes (larger OSDs take longer to restart)
    * Network speed for image downloads
    * Any recovery operations that occur

    Small clusters may complete in under an hour; large clusters can take several hours.
  </Accordion>

  <Accordion title="Is the cluster available during upgrade?">
    Yes. Ceph's rolling upgrade process maintains cluster availability:

    * Only one daemon is upgraded at a time
    * Monitor quorum is preserved
    * Data redundancy protects against OSD restarts

    Clients may experience brief latency spikes during daemon restarts.
  </Accordion>

  <Accordion title="What if I need to roll back?">
    Ceph doesn't support automatic rollback. Options include:

    * Restore from backup (if available)
    * Manually reinstall previous version (complex)
    * Complete the upgrade and address issues

    Prevention is best: test upgrades in a staging environment first.
  </Accordion>

  <Accordion title="Can I skip versions?">
    Generally, you should upgrade to the next major version only. For example:

    * Quincy → Reef (supported)
    * Pacific → Reef (not recommended, upgrade to Quincy first)

    Check Ceph's official upgrade documentation for supported paths.
  </Accordion>

  <Accordion title="What is the recommended version?">
    The recommended version is determined by:

    * Latest stable release in your version series
    * System administrator configuration
    * Known compatibility with your environment

    It's highlighted in the version selection list.
  </Accordion>

  <Accordion title="Should I upgrade if cluster shows HEALTH_WARN?">
    It depends on the warning:

    * Minor warnings (clock skew, nearfull): Usually safe to proceed
    * Data-related warnings (degraded PGs): Resolve first
    * OSD warnings (down OSDs): Fix before upgrading

    Review each warning to understand its impact.
  </Accordion>

  <Accordion title="What happens if power fails during upgrade?">
    Ceph is resilient to interruptions:

    * Completed daemons retain their new version
    * In-progress daemon may need manual recovery
    * The upgrade can be resumed once power is restored
    * Data is protected by replication
  </Accordion>

  <Accordion title="How do I know the upgrade completed successfully?">
    Signs of successful upgrade:

    * Progress shows 100%
    * Current Version matches Target Version
    * Cluster health returns to HEALTH\_OK
    * All daemons show the new version

    Check the History tab for the official completion record.
  </Accordion>
</AccordionGroup>