Key Concepts
StatefulSet
A controller that manages pods with stable identities and persistent storage.
Stable Identity
Each pod gets a persistent hostname (e.g., mysql-0, mysql-1) that survives restarts.
Headless Service
A service that provides DNS entries for each pod without load balancing.
VolumeClaimTemplate
Template for creating PersistentVolumeClaims for each pod automatically.
Required Permissions
| Action | Permission |
|---|---|
| View statefulsets | iam:project:infrastructure:kubernetes:read |
| Scale statefulset | iam:project:infrastructure:kubernetes:write |
| Restart statefulset | iam:project:infrastructure:kubernetes:write |
| Create statefulset | iam:project:infrastructure:kubernetes:write |
| Edit statefulset | iam:project:infrastructure:kubernetes:write |
| Delete statefulset | iam:project:infrastructure:kubernetes:delete |
StatefulSet Status Values
| Status | Description |
|---|---|
| Running | All replicas are ready |
| Updating | Rolling update in progress |
| NotReady | Some replicas are not ready |
| Scaled to 0 | StatefulSet has zero replicas |
| Unknown | Status cannot be determined |
StatefulSet Metrics
| Metric | Description |
|---|---|
| Replicas | Desired number of pods |
| Ready | Pods that are ready (ready/desired) |
| Current | Pods with current spec version |
| Updated | Pods updated to latest template |
| Available | Pods available for service |
How to View StatefulSets
How to View StatefulSet Details
How to Scale a StatefulSet
Scaling behavior:
- Scale up: New pods are created in order (pod-0, pod-1, pod-2…)
- Scale down: Pods are terminated in reverse order (highest ordinal first)
- Each pod must be Running and Ready before the next is created/terminated
How to Restart a StatefulSet
Restart triggers a rolling restart of all pods.Restart performs an ordered rolling restart. By default, pods are updated one at a time in reverse ordinal order (highest to lowest).
How to Create a StatefulSet
Write YAML
Enter the StatefulSet manifest in YAML format. Include:
serviceNamepointing to a headless servicevolumeClaimTemplatesif persistent storage is needed
How to Edit a StatefulSet
Modify Spec
Edit the StatefulSet specification. Common changes:
- Container image (triggers rolling update)
- Resource requests/limits
- Environment variables
- Replica count
How to Delete a StatefulSet
Update Strategies
StatefulSets support two update strategies:| Strategy | Description |
|---|---|
| RollingUpdate | Updates pods in reverse ordinal order (default). Configurable via partition. |
| OnDelete | Only updates pods when they are manually deleted. |
- Pods with ordinal >= partition are updated
- Pods with ordinal < partition are not updated
- Useful for canary deployments
Pod Management Policy
| Policy | Description |
|---|---|
| OrderedReady | Create/delete pods one at a time in order (default) |
| Parallel | Create/delete all pods simultaneously |
OrderedReady is recommended for most stateful applications. Use Parallel only when pods don’t depend on each other.
Troubleshooting
StatefulSet stuck in Updating
StatefulSet stuck in Updating
- A pod may be failing readiness checks
- Check pod events and logs for errors
- Verify PVC is bound and storage is available
- Check if previous pod terminated cleanly
Pods not starting in order
Pods not starting in order
- Previous pod must be Running and Ready first
- Check pod events for scheduling issues
- Verify PVC binding if using persistent storage
- Check headless service exists and selectors match
PVC not being created
PVC not being created
- Verify
volumeClaimTemplatesis correctly defined - Check StorageClass exists and is default or specified
- Verify storage provisioner is working
- Check for quota limits on PVCs
Data lost after pod restart
Data lost after pod restart
- Ensure PVC is correctly mounted in pod spec
- Verify data is written to the mounted path, not container filesystem
- Check PVC still exists and is bound
Cannot connect to pods via headless service
Cannot connect to pods via headless service
- Verify headless service has
clusterIP: None - Check service selector matches pod labels
- Use full DNS name:
pod-0.service-name.namespace.svc.cluster.local - Verify pods are Ready
Scale down not deleting pods
Scale down not deleting pods
- Pods are deleted in reverse order, one at a time
- Each pod must terminate before the next begins
- Check for pods stuck in Terminating state
- Verify no finalizers blocking deletion
FAQ
What's the difference between StatefulSet and Deployment?
What's the difference between StatefulSet and Deployment?
Deployments are for stateless applications where pods are interchangeable. StatefulSets provide stable identities, ordered operations, and persistent storage for stateful applications like databases.
Why do StatefulSets need a headless service?
Why do StatefulSets need a headless service?
The headless service provides DNS entries for each pod (e.g.,
mysql-0.mysql.default.svc.cluster.local). This allows clients to connect to specific pods by name, which is essential for stateful applications.Are PVCs deleted when I delete a StatefulSet?
Are PVCs deleted when I delete a StatefulSet?
No. PersistentVolumeClaims are preserved to prevent data loss. You must manually delete PVCs if you want to free the storage.
How do I perform a canary update?
How do I perform a canary update?
Use the
partition field in RollingUpdate strategy. Set partition to N to only update pods with ordinal >= N, leaving lower ordinals on the old version for testing.Can I scale a StatefulSet to zero?
Can I scale a StatefulSet to zero?
Yes. Scaling to zero terminates all pods but preserves PVCs. When you scale back up, pods reattach to their original PVCs with data intact.
Why are pods created one at a time?
Why are pods created one at a time?
OrderedReady policy ensures each pod is Running and Ready before creating the next. This is crucial for applications like databases where initialization order matters.
How do I update the volumeClaimTemplate?
How do I update the volumeClaimTemplate?
VolumeClaimTemplates are immutable. To change storage configuration, you must delete the StatefulSet (with orphan policy to keep pods), update the manifest, and recreate it.
What happens to my data if a node fails?
What happens to my data if a node fails?
If using dynamically provisioned storage, the PVC remains bound. When the pod is rescheduled to another node, it reattaches to the same PV (if the storage supports it) or waits for the node to recover.