Schema Registry

Schema Registry provides centralized schema management for Kafka messages. It stores and validates schemas, ensuring producers and consumers agree on message formats and preventing incompatible changes.

Key Concepts

Schema

A definition of message structure. Supports Avro, JSON Schema, and Protobuf formats.

Subject

A named collection of schema versions. Typically named after the topic with -key or -value suffix.

Compatibility

Rules that determine what schema changes are allowed. Prevents breaking changes.

Version

Each schema update creates a new version. All versions are retained for compatibility checking.

Required Permissions

Action	Permission
View schemas	`iam:project:infrastructure:kafka:read`
Create/Update schemas	`iam:project:infrastructure:kafka:write`
Delete schemas	`iam:project:infrastructure:kafka:delete`

Schema Types

Type	Description	Use Case
Avro	Binary format with schema evolution support	High-performance, compact serialization
JSON	JSON Schema for validation	Human-readable, web-friendly
Protobuf	Google’s Protocol Buffers	Cross-language, efficient serialization

Subject Naming Convention

Subjects follow a naming pattern based on the topic name:

Pattern	Example	Use
`{topic}-key`	`orders-key`	Schema for message keys
`{topic}-value`	`orders-value`	Schema for message values

Subject names must be 1-255 characters. Use descriptive names that match your topic naming convention.

How to Create a Schema

Select Kafka Cluster

Choose the target Kafka cluster from the dropdown.

Click Create Schema

Click the Create Schema button.

Enter Subject Name

Provide a subject name following the naming convention (e.g., orders-value).

Select Schema Type

Choose the schema format:

Avro: For compact binary serialization
JSON: For JSON Schema validation
Protobuf: For Protocol Buffers

Define Schema

Enter the schema definition in the code editor. Examples are provided for each type.

Set Compatibility (Optional)

Choose a compatibility mode or leave as default (BACKWARD).

Create

Click Create Schema to register the schema.

Example Schemas

Avro Schema

{
  "type": "record",
  "name": "User",
  "namespace": "com.example",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}

JSON Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "id": {"type": "string"},
    "name": {"type": "string"},
    "email": {"type": "string"}
  },
  "required": ["id", "name"]
}

Protobuf Schema

syntax = "proto3";
package com.example;

message User {
  string id = 1;
  string name = 2;
  optional string email = 3;
}

How to View Schema Details

Find the Schema

Locate the schema in the list using search or type filter.

Click to Open

Click on the schema row to open the detail page.

Explore Tabs

The detail page has three tabs:

Overview: Current schema definition and metadata
Versions: All registered versions of this schema
Compatibility: Current compatibility settings

How to Register a New Version

Open Schema Details

Navigate to the schema detail page.

Click Register Version

Click the Register New Version button.

Update Schema

Modify the schema definition. Changes must comply with the compatibility mode.

Submit

Click Register to add the new version.

New versions must pass compatibility checks. If the schema violates compatibility rules, registration will fail.

Compatibility Modes

Compatibility modes control what schema changes are allowed:

Mode	Description
BACKWARD	New schema can read data written by the previous version
BACKWARD_TRANSITIVE	New schema can read data written by all previous versions
FORWARD	Previous schema can read data written by the new version
FORWARD_TRANSITIVE	All previous schemas can read data written by the new version
FULL	Both backward and forward compatible with the previous version
FULL_TRANSITIVE	Both backward and forward compatible with all previous versions
NONE	No compatibility checking (not recommended for production)

Choosing a Compatibility Mode

Scenario	Recommended Mode
Consumers updated before producers	BACKWARD
Producers updated before consumers	FORWARD
Unknown update order	FULL
Schema migrations with coordination	NONE (temporarily)

BACKWARD is the default and most common mode. It ensures new consumers can read messages from older producers.

Compatible Changes by Mode

BACKWARD Compatible Changes:

Adding optional fields with defaults
Removing fields

FORWARD Compatible Changes:

Adding fields
Removing optional fields with defaults

Breaking Changes (Not Compatible):

Changing field types
Renaming fields
Removing required fields (BACKWARD)
Adding required fields (FORWARD)

How to Change Compatibility

Open Schema Details

Navigate to the schema detail page.

Go to Compatibility Tab

Select the Compatibility tab.

Select New Mode

Choose the desired compatibility mode from the dropdown.

Update

Click Update Compatibility to apply the change.

Changing compatibility mode only affects future versions. Existing versions retain their original compatibility relationships.

How to Delete a Schema

Find the Schema

Locate the schema in the list.

Click Delete

Click the delete button or open the detail page and click Delete Schema.

Confirm

Confirm the deletion. This performs a soft delete by default.

Deleting a schema removes all versions. Ensure no producers or consumers depend on this schema before deletion.

Troubleshooting

Schema registration failed - compatibility error

Your schema change violates the compatibility mode
Review the error message for specific field issues
Consider adding default values to new fields
Temporarily set compatibility to NONE if migration is coordinated

Schema validation error

Check JSON syntax (missing commas, brackets)
Verify Avro type declarations are valid
Ensure Protobuf syntax matches the declared version
Use external validators to test schema before registration

Cannot find schema

Verify you’re connected to the correct Kafka cluster
Check the subject name spelling
Schema may have been deleted
Refresh the page or clear filters

Producer/Consumer failing with schema error

Verify the schema ID exists in the registry
Check that the client is configured to use schema registry
Ensure network connectivity to schema registry endpoint
Verify authentication credentials if required

Cannot delete schema

You need delete permission
Check if the schema is referenced by other subjects
Verify the schema registry is healthy

FAQ

What happens when I delete a schema?

Soft delete removes the schema from normal queries but retains data for potential recovery. Hard delete permanently removes all versions. Producers and consumers using the schema will fail after deletion.

Can I have multiple schemas for one topic?

Yes. Typically you have two subjects per topic: one for the message key (topic-key) and one for the value (topic-value). Each can have different schemas and evolve independently.

How do schema IDs work?

Each schema version gets a unique numeric ID. Producers embed this ID in messages. Consumers use the ID to fetch the correct schema for deserialization. IDs are global across the registry.

What's the difference between version and ID?

Version is sequential within a subject (1, 2, 3…). ID is globally unique across all subjects. The same schema content in different subjects gets different IDs.

Should I use Avro, JSON, or Protobuf?

Avro: Best for Kafka-native workflows, compact format, excellent schema evolution. JSON Schema: Good for web APIs, human-readable, widely understood. Protobuf: Best for cross-language systems, very efficient, strong typing.

Can I change schema type after creation?

No. The schema type (Avro, JSON, Protobuf) cannot be changed for an existing subject. Create a new subject with the desired type and migrate producers/consumers.

How do I handle breaking changes?

For breaking changes: (1) Create a new subject with the new schema, (2) Update consumers to handle both formats, (3) Migrate producers to the new subject, (4) Deprecate the old subject when migration is complete.

What's the recommended workflow for schema changes?

Test the new schema locally
Register in development environment
Update consumers first (for BACKWARD compatibility)
Update producers
Promote to production following the same order

Best Practices

Schema Design

Include optional fields with defaults for future extensibility
Use meaningful field names that describe the data
Add documentation/comments to complex schemas
Version your schemas in source control

Compatibility Strategy

Use BACKWARD compatibility for most use cases
Use FULL_TRANSITIVE for critical data pipelines
Avoid NONE in production except during coordinated migrations
Test compatibility changes in non-production first

Naming Conventions

Follow consistent naming patterns:

{domain}.{entity}.{event-type}

Examples:
orders.payment.completed-value
users.profile.updated-value
inventory.stock.adjusted-value

Evolution Guidelines

Always add new fields as optional with defaults
Never remove required fields
Never change field types
Use union types (Avro) or oneOf (JSON) for flexible fields
Document breaking changes and migration plans

Getting Started

Infrastructure

Platform Services

CI/CD & Deployments

Pipeline & Helm

Performance Testing

Security

Mesh Networking

Access Management

Audit & Compliance

Settings

Key Concepts

Schema

Subject

Compatibility

Version

Required Permissions

Schema Types

Subject Naming Convention

How to Create a Schema

Example Schemas

How to View Schema Details

How to Register a New Version

Compatibility Modes

Choosing a Compatibility Mode

Compatible Changes by Mode

How to Change Compatibility

How to Delete a Schema

Troubleshooting

FAQ

Best Practices

Schema Design

Compatibility Strategy

Naming Conventions

Evolution Guidelines

​Key Concepts

Schema

Subject

Compatibility

Version

​Required Permissions

​Schema Types

​Subject Naming Convention

​How to Create a Schema

​Example Schemas

​How to View Schema Details

​How to Register a New Version

​Compatibility Modes

​Choosing a Compatibility Mode

​Compatible Changes by Mode

​How to Change Compatibility

​How to Delete a Schema

​Troubleshooting

​FAQ

​Best Practices

​Schema Design

​Compatibility Strategy

​Naming Conventions

​Evolution Guidelines

Key Concepts

Required Permissions

Schema Types

Subject Naming Convention

How to Create a Schema

Example Schemas

How to View Schema Details

How to Register a New Version

Compatibility Modes

Choosing a Compatibility Mode

Compatible Changes by Mode

How to Change Compatibility

How to Delete a Schema

Troubleshooting

FAQ

Best Practices

Schema Design

Compatibility Strategy

Naming Conventions

Evolution Guidelines