> ## Documentation Index
> Fetch the complete documentation index at: https://docs.shiftlabs.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Schema Registry

> Manage Avro, JSON, and Protobuf schemas for Kafka topics

Schema Registry provides centralized schema management for Kafka messages. It stores and validates schemas, ensuring producers and consumers agree on message formats and preventing incompatible changes.

## Key Concepts

<CardGroup cols={2}>
  <Card title="Schema" icon="file-code">
    A definition of message structure. Supports Avro, JSON Schema, and Protobuf formats.
  </Card>

  <Card title="Subject" icon="tag">
    A named collection of schema versions. Typically named after the topic with `-key` or `-value` suffix.
  </Card>

  <Card title="Compatibility" icon="check-double">
    Rules that determine what schema changes are allowed. Prevents breaking changes.
  </Card>

  <Card title="Version" icon="code-branch">
    Each schema update creates a new version. All versions are retained for compatibility checking.
  </Card>
</CardGroup>

## Required Permissions

| Action                | Permission                                |
| --------------------- | ----------------------------------------- |
| View schemas          | `iam:project:infrastructure:kafka:read`   |
| Create/Update schemas | `iam:project:infrastructure:kafka:write`  |
| Delete schemas        | `iam:project:infrastructure:kafka:delete` |

## Schema Types

| Type         | Description                                 | Use Case                                |
| ------------ | ------------------------------------------- | --------------------------------------- |
| **Avro**     | Binary format with schema evolution support | High-performance, compact serialization |
| **JSON**     | JSON Schema for validation                  | Human-readable, web-friendly            |
| **Protobuf** | Google's Protocol Buffers                   | Cross-language, efficient serialization |

## Subject Naming Convention

Subjects follow a naming pattern based on the topic name:

| Pattern         | Example        | Use                       |
| --------------- | -------------- | ------------------------- |
| `{topic}-key`   | `orders-key`   | Schema for message keys   |
| `{topic}-value` | `orders-value` | Schema for message values |

<Info>
  Subject names must be 1-255 characters. Use descriptive names that match your topic naming convention.
</Info>

## How to Create a Schema

<Steps>
  <Step title="Select Kafka Cluster">
    Choose the target Kafka cluster from the dropdown.
  </Step>

  <Step title="Click Create Schema">
    Click the **Create Schema** button.
  </Step>

  <Step title="Enter Subject Name">
    Provide a subject name following the naming convention (e.g., `orders-value`).
  </Step>

  <Step title="Select Schema Type">
    Choose the schema format:

    * **Avro**: For compact binary serialization
    * **JSON**: For JSON Schema validation
    * **Protobuf**: For Protocol Buffers
  </Step>

  <Step title="Define Schema">
    Enter the schema definition in the code editor. Examples are provided for each type.
  </Step>

  <Step title="Set Compatibility (Optional)">
    Choose a compatibility mode or leave as default (BACKWARD).
  </Step>

  <Step title="Create">
    Click **Create Schema** to register the schema.
  </Step>
</Steps>

### Example Schemas

**Avro Schema**

```json theme={null}
{
  "type": "record",
  "name": "User",
  "namespace": "com.example",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}
```

**JSON Schema**

```json theme={null}
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "id": {"type": "string"},
    "name": {"type": "string"},
    "email": {"type": "string"}
  },
  "required": ["id", "name"]
}
```

**Protobuf Schema**

```protobuf theme={null}
syntax = "proto3";
package com.example;

message User {
  string id = 1;
  string name = 2;
  optional string email = 3;
}
```

## How to View Schema Details

<Steps>
  <Step title="Find the Schema">
    Locate the schema in the list using search or type filter.
  </Step>

  <Step title="Click to Open">
    Click on the schema row to open the detail page.
  </Step>

  <Step title="Explore Tabs">
    The detail page has three tabs:

    * **Overview**: Current schema definition and metadata
    * **Versions**: All registered versions of this schema
    * **Compatibility**: Current compatibility settings
  </Step>
</Steps>

## How to Register a New Version

<Steps>
  <Step title="Open Schema Details">
    Navigate to the schema detail page.
  </Step>

  <Step title="Click Register Version">
    Click the **Register New Version** button.
  </Step>

  <Step title="Update Schema">
    Modify the schema definition. Changes must comply with the compatibility mode.
  </Step>

  <Step title="Submit">
    Click **Register** to add the new version.
  </Step>
</Steps>

<Warning>
  New versions must pass compatibility checks. If the schema violates compatibility rules, registration will fail.
</Warning>

## Compatibility Modes

Compatibility modes control what schema changes are allowed:

| Mode                     | Description                                                     |
| ------------------------ | --------------------------------------------------------------- |
| **BACKWARD**             | New schema can read data written by the previous version        |
| **BACKWARD\_TRANSITIVE** | New schema can read data written by all previous versions       |
| **FORWARD**              | Previous schema can read data written by the new version        |
| **FORWARD\_TRANSITIVE**  | All previous schemas can read data written by the new version   |
| **FULL**                 | Both backward and forward compatible with the previous version  |
| **FULL\_TRANSITIVE**     | Both backward and forward compatible with all previous versions |
| **NONE**                 | No compatibility checking (not recommended for production)      |

### Choosing a Compatibility Mode

| Scenario                            | Recommended Mode   |
| ----------------------------------- | ------------------ |
| Consumers updated before producers  | BACKWARD           |
| Producers updated before consumers  | FORWARD            |
| Unknown update order                | FULL               |
| Schema migrations with coordination | NONE (temporarily) |

<Info>
  **BACKWARD** is the default and most common mode. It ensures new consumers can read messages from older producers.
</Info>

### Compatible Changes by Mode

**BACKWARD Compatible Changes:**

* Adding optional fields with defaults
* Removing fields

**FORWARD Compatible Changes:**

* Adding fields
* Removing optional fields with defaults

**Breaking Changes (Not Compatible):**

* Changing field types
* Renaming fields
* Removing required fields (BACKWARD)
* Adding required fields (FORWARD)

## How to Change Compatibility

<Steps>
  <Step title="Open Schema Details">
    Navigate to the schema detail page.
  </Step>

  <Step title="Go to Compatibility Tab">
    Select the **Compatibility** tab.
  </Step>

  <Step title="Select New Mode">
    Choose the desired compatibility mode from the dropdown.
  </Step>

  <Step title="Update">
    Click **Update Compatibility** to apply the change.
  </Step>
</Steps>

<Warning>
  Changing compatibility mode only affects future versions. Existing versions retain their original compatibility relationships.
</Warning>

## How to Delete a Schema

<Steps>
  <Step title="Find the Schema">
    Locate the schema in the list.
  </Step>

  <Step title="Click Delete">
    Click the delete button or open the detail page and click **Delete Schema**.
  </Step>

  <Step title="Confirm">
    Confirm the deletion. This performs a soft delete by default.
  </Step>
</Steps>

<Warning>
  Deleting a schema removes all versions. Ensure no producers or consumers depend on this schema before deletion.
</Warning>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Schema registration failed - compatibility error" icon="circle-question">
    * Your schema change violates the compatibility mode
    * Review the error message for specific field issues
    * Consider adding default values to new fields
    * Temporarily set compatibility to NONE if migration is coordinated
  </Accordion>

  <Accordion title="Schema validation error" icon="circle-question">
    * Check JSON syntax (missing commas, brackets)
    * Verify Avro type declarations are valid
    * Ensure Protobuf syntax matches the declared version
    * Use external validators to test schema before registration
  </Accordion>

  <Accordion title="Cannot find schema" icon="circle-question">
    * Verify you're connected to the correct Kafka cluster
    * Check the subject name spelling
    * Schema may have been deleted
    * Refresh the page or clear filters
  </Accordion>

  <Accordion title="Producer/Consumer failing with schema error" icon="circle-question">
    * Verify the schema ID exists in the registry
    * Check that the client is configured to use schema registry
    * Ensure network connectivity to schema registry endpoint
    * Verify authentication credentials if required
  </Accordion>

  <Accordion title="Cannot delete schema" icon="circle-question">
    * You need delete permission
    * Check if the schema is referenced by other subjects
    * Verify the schema registry is healthy
  </Accordion>
</AccordionGroup>

## FAQ

<AccordionGroup>
  <Accordion title="What happens when I delete a schema?">
    Soft delete removes the schema from normal queries but retains data for potential recovery. Hard delete permanently removes all versions. Producers and consumers using the schema will fail after deletion.
  </Accordion>

  <Accordion title="Can I have multiple schemas for one topic?">
    Yes. Typically you have two subjects per topic: one for the message key (`topic-key`) and one for the value (`topic-value`). Each can have different schemas and evolve independently.
  </Accordion>

  <Accordion title="How do schema IDs work?">
    Each schema version gets a unique numeric ID. Producers embed this ID in messages. Consumers use the ID to fetch the correct schema for deserialization. IDs are global across the registry.
  </Accordion>

  <Accordion title="What's the difference between version and ID?">
    **Version** is sequential within a subject (1, 2, 3...). **ID** is globally unique across all subjects. The same schema content in different subjects gets different IDs.
  </Accordion>

  <Accordion title="Should I use Avro, JSON, or Protobuf?">
    **Avro**: Best for Kafka-native workflows, compact format, excellent schema evolution. **JSON Schema**: Good for web APIs, human-readable, widely understood. **Protobuf**: Best for cross-language systems, very efficient, strong typing.
  </Accordion>

  <Accordion title="Can I change schema type after creation?">
    No. The schema type (Avro, JSON, Protobuf) cannot be changed for an existing subject. Create a new subject with the desired type and migrate producers/consumers.
  </Accordion>

  <Accordion title="How do I handle breaking changes?">
    For breaking changes: (1) Create a new subject with the new schema, (2) Update consumers to handle both formats, (3) Migrate producers to the new subject, (4) Deprecate the old subject when migration is complete.
  </Accordion>

  <Accordion title="What's the recommended workflow for schema changes?">
    1. Test the new schema locally
    2. Register in development environment
    3. Update consumers first (for BACKWARD compatibility)
    4. Update producers
    5. Promote to production following the same order
  </Accordion>
</AccordionGroup>

## Best Practices

### Schema Design

* Include optional fields with defaults for future extensibility
* Use meaningful field names that describe the data
* Add documentation/comments to complex schemas
* Version your schemas in source control

### Compatibility Strategy

* Use BACKWARD compatibility for most use cases
* Use FULL\_TRANSITIVE for critical data pipelines
* Avoid NONE in production except during coordinated migrations
* Test compatibility changes in non-production first

### Naming Conventions

Follow consistent naming patterns:

```
{domain}.{entity}.{event-type}

Examples:
orders.payment.completed-value
users.profile.updated-value
inventory.stock.adjusted-value
```

### Evolution Guidelines

* Always add new fields as optional with defaults
* Never remove required fields
* Never change field types
* Use union types (Avro) or oneOf (JSON) for flexible fields
* Document breaking changes and migration plans