Key Concepts
Upgrade Check
Pre-flight validation that verifies version availability and node readiness.
Impact Analysis
Assessment of cluster health, affected workloads, and risk level before upgrade.
Rolling Upgrade
Sequential upgrade of nodes with cordon/drain to maintain availability.
Selective Upgrade
Option to upgrade specific nodes instead of the entire cluster.
Required Permissions
| Action | Permission |
|---|---|
| View upgrade impact | iam:project:infrastructure:kubernetes:read |
| Run upgrade check | iam:project:infrastructure:kubernetes:read |
| Execute upgrade | iam:project:infrastructure:kubernetes:write |
Version Upgrade Rules
Kubernetes enforces strict version upgrade paths:| Rule | Description |
|---|---|
| No major version jumps | Cannot upgrade across major versions (e.g., 1.x to 2.x) |
| One minor at a time | Cannot skip minor versions (e.g., 1.28 → 1.30 not allowed) |
| No downgrades | Cannot downgrade to a lower minor version |
| Patch upgrades | Can upgrade to any higher patch version within same minor |
v1.28.5→v1.28.10(patch upgrade)v1.28.5→v1.29.0(minor upgrade)
v1.28.5→v1.30.0(skips 1.29)v1.29.0→v1.28.5(downgrade)
Upgradeable Components
| Component | Description |
|---|---|
| Kubernetes | kubeadm, kubelet, kubectl |
| containerd | Container runtime |
| CNI Plugin | Network plugin (Calico, Flannel, Cilium) |
| Metrics Server | Resource metrics provider |
How to Set Target Versions
Configure Target Versions
Set target versions for:
- Kubernetes version
- containerd version (optional)
- CNI plugin version (optional)
- Metrics server version (optional)
How to Run Upgrade Check
Upgrade check validates prerequisites before upgrade.Upgrade check must pass before you can proceed with the upgrade.
How to View Upgrade Impact
Impact analysis assesses risk and identifies potential issues.View Impact Analysis
The impact analysis shows:
- Current vs target version comparison
- Cluster health status
- Node readiness details
- Affected workload count
- Risk level assessment
- Recommendations
Pre-Upgrade Requirements
The upgrade will be blocked if these conditions are not met:| Requirement | Details |
|---|---|
| Master node count | Standalone: exactly 1 master. Multinode: 1 or odd number (3, 5, 7) |
| First master | One master must be marked as “first master” |
| SSH connectivity | All nodes must be accessible via SSH |
| Nodes ready | All nodes must be in Ready state |
| etcd healthy | etcd pods must be running |
| Control plane healthy | API server, controller-manager, scheduler must be running |
Risk Levels
| Level | Criteria | Recommendation |
|---|---|---|
| Critical | Errors found in validation | Do not proceed until resolved |
| High | Cluster unhealthy or critical workloads affected | Proceed with caution |
| Medium | Multiple warnings or >50 affected pods | Plan maintenance window |
| Low | No significant issues | Safe to proceed |
How to Upgrade a Cluster
Upgrade Process
The platform follows this sequence:- Master nodes first - Control plane upgraded before workers
- For each node:
- Cordon (mark unschedulable)
- Drain (evict pods)
- Upgrade kubeadm, kubelet, kubectl
- Upgrade containerd (if configured)
- Uncordon (allow scheduling)
- Wait for node Ready
The first master node runs
kubeadm upgrade apply. Subsequent nodes run kubeadm upgrade node.How to Upgrade Selective Nodes
Upgrade specific nodes instead of the entire cluster.Configure Options
Set upgrade options:
- Batch size - Number of nodes to upgrade in parallel
- Node order - Masters first, workers first, or as listed
Troubleshooting
Upgrade check fails with version not found
Upgrade check fails with version not found
- Verify the target version exists in Kubernetes apt repository
- Check that the version format is correct (e.g.,
1.29.0notv1.29) - Ensure nodes have internet access to download packages
Cannot upgrade: invalid version path
Cannot upgrade: invalid version path
- You cannot skip minor versions - upgrade one minor version at a time
- Example: From 1.27.x, upgrade to 1.28.x first, then 1.29.x
Upgrade blocked due to node not ready
Upgrade blocked due to node not ready
- Check kubelet status on the affected node
- Verify network connectivity to control plane
- Review node conditions for pressure (memory, disk, PID)
Upgrade fails during drain
Upgrade fails during drain
- Some pods may have PodDisruptionBudgets preventing eviction
- Pods with local storage may not drain without force
- Review drain logs for specific pod failures
Node not coming back after upgrade
Node not coming back after upgrade
- SSH to the node and check kubelet status
- Verify kubelet and containerd are running
- Check kubelet logs for certificate or connectivity errors
- Ensure firewall allows required ports
Cluster unreachable after control plane upgrade
Cluster unreachable after control plane upgrade
- Wait for API server to restart (may take a few minutes)
- Check API server pod status in kube-system
- Verify kubeconfig credentials are still valid
- Check etcd health
FAQ
How long does an upgrade take?
How long does an upgrade take?
Depends on cluster size. Each node takes several minutes for drain, upgrade, and restart. A 5-node cluster typically completes in 15-30 minutes.
Is there downtime during upgrade?
Is there downtime during upgrade?
Applications with multiple replicas spread across nodes experience minimal disruption. Single-replica workloads will have brief downtime when their node is drained.
Can I rollback an upgrade?
Can I rollback an upgrade?
Kubernetes does not support direct rollback. Restore from etcd backup if critical issues occur. Always backup before upgrading.
Should I upgrade containerd with Kubernetes?
Should I upgrade containerd with Kubernetes?
Not required for every K8s upgrade. Check containerd compatibility matrix for your target K8s version.
What about CNI plugin upgrades?
What about CNI plugin upgrades?
CNI plugins are upgraded separately. The platform can upgrade Calico, Flannel, or Cilium alongside Kubernetes.
How do I handle a failed mid-upgrade?
How do I handle a failed mid-upgrade?
The cluster may be in a mixed-version state. Resume the upgrade for remaining nodes or manually upgrade via SSH.
Can I upgrade a single-master cluster?
Can I upgrade a single-master cluster?
Yes, but there will be control plane downtime during the master upgrade. Consider adding masters for high availability before upgrading production clusters.
Best Practices
- Test in non-production first - Upgrade staging clusters before production
- Create etcd backup - Always backup before upgrading
- Check release notes - Review Kubernetes changelog for breaking changes
- Plan maintenance window - Schedule upgrades during low-traffic periods
- Monitor after upgrade - Verify application health after completion
- Keep versions current - Don’t fall too far behind; smaller jumps are safer