k8s-troubleshooting

Current Versions & Troubleshooting Tools (January 2026)

| Platform | Version | Key Changes | |----------|---------|-------------| | Kubernetes | 1.31.x | Sidecar containers GA, Pod lifecycle improvements | | OpenShift | 4.17.x | OVN-Kubernetes default, enhanced web terminal | | EKS | 1.31 | Pod Identity, Auto Mode, Karpenter 1.x | | AKS | 1.31 | Cilium CNI, Workload Identity GA | | GKE | 1.31 | Autopilot improvements, Gateway API GA |

Troubleshooting Tools & CLIs

| Tool | Version | Install | Purpose | |------|---------|---------|--------| | kubectl | 1.31.x | brew install kubectl | Cluster operations | | oc | 4.17.x | brew install openshift-cli | OpenShift operations | | k9s | 0.32.x | brew install k9s | Terminal UI | | stern | 1.30.x | brew install stern | Multi-pod log tailing | | kubectx/kubens | 0.9.x | brew install kubectx | Context/namespace switching | | krew | 0.4.x | kubectl plugin manager | Plugin ecosystem | | kubectl-node-shell | - | kubectl krew install node-shell | Node access | | kubectl-neat | - | kubectl krew install neat | Clean YAML output | | kubectl-tree | - | kubectl krew install tree | Resource hierarchy |

# Essential CLI tool installation
brew install kubectl kubectx k9s stern

# Install krew (kubectl plugin manager)
(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/arm64/arm64/')" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-${OS}_${ARCH}.tar.gz" &&
  tar zxvf krew-${OS}_${ARCH}.tar.gz &&
  KREW=./krew-${OS}_${ARCH} && "$KREW" install krew
)

# Install useful kubectl plugins
kubectl krew install ctx ns neat tree node-shell images lineage

# Multi-pod log streaming with stern
stern -n ${NAMESPACE} ${POD_PREFIX}
stern -A -l app=${APP_NAME} --since 1h

# Interactive cluster navigation with k9s
k9s -n ${NAMESPACE}
k9s --context ${CONTEXT}

Command Usage Convention

IMPORTANT: This skill uses kubectl as the primary command in all examples. When working with:

• OpenShift/ARO clusters: Replace all kubectl commands with oc
• Standard Kubernetes clusters (AKS, EKS, GKE, etc.): Use kubectl as shown

The agent will automatically detect the cluster type and use the appropriate command.

Systematic approach to diagnosing and resolving cluster issues through event analysis, log interpretation, and root cause identification.

Proactive Cluster Health Analysis (Popeye-Style)

Cluster Scoring Framework

Popeye uses a health scoring system (0-100) to assess cluster health. Critical issues reduce the score significantly:

• BOOM (Critical): -50 points - Security vulnerabilities, resource exhaustion, failed services
• WARN (Warning): -20 points - Configuration inefficiencies, best practice violations
• INFO (Informational): -5 points - Non-critical issues, optimization opportunities

Quick Cluster Health Assessment

#!/bin/bash
# Comprehensive cluster health check based on Popeye patterns
echo "=== POPEYE-STYLE CLUSTER HEALTH ASSESSMENT ==="

# 1. Node Health Check
echo "### NODE HEALTH (Critical Weight: 1.0) ###"
kubectl get nodes -o wide | grep -E "NotReady|Unknown" && echo "BOOM: Unhealthy nodes detected!" || echo "✓ All nodes healthy"

# 2. Pod Issues Check
echo -e "\n### POD HEALTH (Critical Weight: 1.0) ###"
POD_ISSUES=$(kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers | wc -l)
if [ $POD_ISSUES -gt 0 ]; then
    echo "WARN: $POD_ISSUES pods not running"
    kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
else
    echo "✓ All pods running"
fi

# 3. Security Issues Check
echo -e "\n### SECURITY ASSESSMENT (Critical Weight: 1.0) ###"
# Check for privileged containers
PRIVILEGED=$(kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].securityContext.privileged == true) | "\(.metadata.namespace)/\(.metadata.name)"' | wc -l)
if [ $PRIVILEGED -gt 0 ]; then
    echo "BOOM: $PRIVILEGED privileged containers detected (Security Risk!)"
else
    echo "✓ No privileged containers found"
fi

# Check for containers running as root
ROOT_CONTAINERS=$(kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].securityContext.runAsUser == 0) | "\(.metadata.namespace)/\(.metadata.name)"' | wc -l)
if [ $ROOT_CONTAINERS -gt 0 ]; then
    echo "WARN: $ROOT_CONTAINERS containers running as root"
else
    echo "✓ No containers running as root"
fi

# 4. Resource Configuration Check
echo -e "\n### RESOURCE CONFIGURATION (Warning Weight: 0.8) ###"
NO_LIMITS=$(kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].resources.limits == null) | "\(.metadata.namespace)/\(.metadata.name)"' | wc -l)
if [ $NO_LIMITS -gt 0 ]; then
    echo "WARN: $NO_LIMITS containers without resource limits"
else
    echo "✓ All containers have resource limits"
fi

# 5. Storage Issues Check
echo -e "\n### STORAGE HEALTH (Warning Weight: 0.5) ###"
PENDING_PVC=$(kubectl get pvc -A --field-selector=status.phase!=Bound --no-headers | wc -l)
if [ $PENDING_PVC -gt 0 ]; then
    echo "WARN: $PENDING_PVC PVCs not bound"
    kubectl get pvc -A --field-selector=status.phase!=Bound
else
    echo "✓ All PVCs bound"
fi

# 6. Network Issues Check
echo -e "\n### NETWORKING (Warning Weight: 0.5) ###"
# Check services without endpoints
EMPTY_ENDPOINTS=$(kubectl get svc -A -o json | jq -r '.items[] | select(.spec.clusterIP != "None") | select(.status.loadBalancer.ingress == null) | "\(.metadata.namespace)/\(.metadata.name)"' | wc -l)
if [ $EMPTY_ENDPOINTS -gt 0 ]; then
    echo "WARN: $EMPTY_ENDPOINTS services may have endpoint issues"
else
    echo "✓ Services appear healthy"
fi

# OpenShift specific checks
if command -v oc &> /dev/null; then
    echo -e "\n### OPENSHIFT CLUSTER OPERATORS (Critical Weight: 1.0) ###"
    DEGRADED=$(oc get clusteroperators --no-headers | grep -c -E "False.*True|False.*False")
    if [ $DEGRADED -gt 0 ]; then
        echo "BOOM: $DEGRADED cluster operators degraded/unavailable"
        oc get clusteroperators | grep -E "False.*True|False.*False"
    else
        echo "✓ All cluster operators healthy"
    fi
fi

Security Vulnerability Detection

Container Security Analysis

# Security Context Validation
echo "=== CONTAINER SECURITY ANALYSIS ==="

# 1. Privileged Containers (Critical)
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{range .spec.containers[*]}{.name}{"\t"}{.securityContext.privileged}{"\n"}{end}{end}' | grep "true" && echo "BOOM: Privileged containers found!" || echo "✓ No privileged containers"

# 2. Host Namespace Access (Critical)
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.hostNetwork}{"\t"}{.spec.hostPID}{"\t"}{.spec.hostIPC}{"\n"}' | grep -E "true.*true|true$|true\s" && echo "BOOM: Host namespace access detected!" || echo "✓ No host namespace access"

# 3. Capabilities Check (Warning)
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].securityContext.capabilities.add != null) | "\(.metadata.namespace)/\(.metadata.name): \(.spec.containers[].securityContext.capabilities.add[])"'

# 4. Read-Only Root Filesystem (Warning)
READONLY_ISSUES=$(kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].securityContext.readOnlyRootFilesystem == false) | "\(.metadata.namespace)/\(.metadata.name)"' | wc -l)
echo "INFO: $READONLY_ISSUES containers without read-only root filesystem"

RBAC Security Analysis

echo "=== RBAC SECURITY ANALYSIS ==="

# Check for overly permissive roles
kubectl get clusterroles -o json | jq -r '.items[] | select(.rules[].verbs[] == "*") | "\(.metadata.name): Wildcard permissions detected"'

# Check service account permissions
kubectl get serviceaccounts -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name)"'

Performance Issues Detection

Resource Utilization Analysis

echo "=== PERFORMANCE ANALYSIS ==="

# Find pods approaching memory limits
kubectl top pods -A --no-headers | awk '{print $4}' | sed 's/Mi//' | while read mem; do
    if [ "$mem" -gt 900 ]; then
        echo "WARN: Pod using high memory: ${mem}Mi"
    fi
done

# CPU throttling detection
kubectl top pods -A --no-headers | awk '{print $3}' | sed 's/m//' | while read cpu; do
    if [ "$cpu" -gt 900 ]; then
        echo "WARN: Pod using high CPU: ${cpu}m"
    fi
done

Configuration Best Practices Validation

Deployment Health Checks

echo "=== DEPLOYMENT BEST PRACTICES ==="

# Check for liveness/readiness probes
NO_PROBES=$(kubectl get deployments -A -o json | jq -r '.items[] | select(.spec.template.spec.containers[].livenessProbe == null or .spec.template.spec.containers[].readinessProbe == null) | "\(.metadata.namespace)/\(.metadata.name)"' | wc -l)
echo "INFO: $NO_PROBES deployments missing health probes"

# Check for pod disruption budgets
PDB_COUNT=$(kubectl get pdb -A --no-headers | wc -l)
DEPLOY_COUNT=$(kubectl get deployments -A --no-headers | wc -l)
echo "INFO: $PDB_COUNT pod disruption budgets for $DEPLOY_COUNT deployments"

# Rolling update strategy
NO_STRATEGY=$(kubectl get deployments -A -o json | jq -r '.items[] | select(.spec.strategy.type != "RollingUpdate") | "\(.metadata.namespace)/\(.metadata.name)"' | wc -l)
echo "INFO: $NO_STRATEGY deployments not using RollingUpdate"

Troubleshooting Workflow

1. Identify Scope: Pod, Node, Namespace, or Cluster-wide issue?
2. Gather Context: Events, logs, resource status, recent changes
3. Analyze Symptoms: Match patterns to known issues
4. Determine Root Cause: Follow diagnostic tree
5. Remediate: Apply fix and verify resolution
6. Document: Record findings for future reference

Quick Diagnostic Commands

# Pod status overview
kubectl get pods -n ${NAMESPACE} -o wide

# Recent events (sorted by time)
kubectl get events -n ${NAMESPACE} --sort-by='.lastTimestamp'

# Pod details and events
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}

# Container logs (current)
kubectl logs ${POD_NAME} -n ${NAMESPACE} -c ${CONTAINER}

# Container logs (previous crashed instance)
kubectl logs ${POD_NAME} -n ${NAMESPACE} -c ${CONTAINER} --previous

# Node status
kubectl get nodes -o wide
kubectl describe node ${NODE_NAME}

# Resource usage
kubectl top pods -n ${NAMESPACE}
kubectl top nodes

# OpenShift specific
oc get events -n ${NAMESPACE}
oc adm top pods -n ${NAMESPACE}
oc get clusteroperators
oc adm node-logs ${NODE_NAME} -u kubelet

Cloud Provider Specific Commands

# EKS Troubleshooting (AWS)
aws eks describe-cluster --name ${CLUSTER} --query 'cluster.status'
aws eks describe-addon --cluster-name ${CLUSTER} --addon-name vpc-cni --query 'addon.status'
eksctl utils describe-stacks --cluster ${CLUSTER}
exportctl get nodegroup --cluster ${CLUSTER}

# EKS Pod Identity issues
aws eks describe-pod-identity-association --cluster-name ${CLUSTER} --association-id ${ASSOC_ID}

# EKS CloudWatch Logs Insights query for control plane logs
aws logs filter-log-events --log-group-name /aws/eks/${CLUSTER}/cluster \
  --filter-pattern "ERROR" --start-time ${TIMESTAMP}

# AKS Troubleshooting (Azure)
az aks show --resource-group ${RG} --name ${CLUSTER} --query provisioningState
az aks get-credentials --resource-group ${RG} --name ${CLUSTER} --admin
az aks browse --resource-group ${RG} --name ${CLUSTER}  # Opens dashboard

# AKS diagnostic logs
az aks kollect --resource-group ${RG} --name ${CLUSTER} --storage-account ${STORAGE}
az aks check-network outbound --resource-group ${RG} --name ${CLUSTER}

# AKS Workload Identity issues
az aks show --resource-group ${RG} --name ${CLUSTER} --query 'oidcIssuerProfile'

# GKE Troubleshooting (Google Cloud)
gcloud container clusters describe ${CLUSTER} --region ${REGION} --format='value(status)'
gcloud container clusters get-credentials ${CLUSTER} --region ${REGION}

# GKE operations and errors
gcloud container operations list --filter="targetLink:${CLUSTER}" --sort-by="~startTime" --limit=10
gcloud container operations describe ${OPERATION_ID} --region ${REGION}

# GKE Workload Identity issues
gcloud iam service-accounts get-iam-policy ${GSA_EMAIL}

# GKE node pool issues
gcloud container node-pools describe ${POOL} --cluster ${CLUSTER} --region ${REGION}

# ARO Troubleshooting (Azure Red Hat OpenShift)
az aro show --resource-group ${RG} --name ${CLUSTER} --query provisioningState
az aro list-credentials --resource-group ${RG} --name ${CLUSTER}
az aro show --resource-group ${RG} --name ${CLUSTER} --query 'networkProfile'

# ROSA Troubleshooting (Red Hat OpenShift on AWS)
rosa describe cluster --cluster ${CLUSTER}
rosa logs install --cluster ${CLUSTER}
rosa logs uninstall --cluster ${CLUSTER}
rosa list machinepools --cluster ${CLUSTER}

Pod Status Interpretation

Pod Phase States

| Phase | Meaning | Action | |-------|---------|--------| | Pending | Not scheduled or pulling images | Check events, node resources, PVC status | | Running | At least one container running | Check container statuses if issues | | Succeeded | All containers completed successfully | Normal for Jobs | | Failed | All containers terminated, at least one failed | Check logs, exit codes | | Unknown | Cannot determine state | Node communication issue |

Container State Analysis

Waiting States

| Reason | Cause | Resolution | |--------|-------|------------| | ContainerCreating | Setting up container | Check events for errors, volume mounts | | ImagePullBackOff | Cannot pull image | Verify image name, registry access, credentials | | ErrImagePull | Image pull failed | Check image exists, network, ImagePullSecrets | | CreateContainerConfigError | Config error | Check ConfigMaps, Secrets exist and mounted correctly | | InvalidImageName | Malformed image reference | Fix image name in spec | | CrashLoopBackOff | Container repeatedly crashing | Check logs --previous, fix application |

Terminated States

| Reason | Exit Code | Cause | Resolution | |--------|-----------|-------|------------| | OOMKilled | 137 | Memory limit exceeded | Increase memory limit, fix memory leak | | Error | 1 | Application error | Check logs for stack trace | | Error | 126 | Command not executable | Fix command/entrypoint permissions | | Error | 127 | Command not found | Fix command path, verify image contents | | Error | 128 | Invalid exit code | Application bug | | Error | 130 | SIGINT (Ctrl+C) | Normal if manual termination | | Error | 137 | SIGKILL | OOM or forced termination | | Error | 143 | SIGTERM | Graceful shutdown requested | | Completed | 0 | Normal exit | Expected for Jobs/init containers |

Event Analysis

Event Types and Severity

Type: Normal   → Informational, typically no action needed
Type: Warning  → Potential issue, investigate

Critical Events to Monitor

Pod Scheduling Events

| Event Reason | Meaning | Resolution | |--------------|---------|------------| | FailedScheduling | Cannot place pod | Check node resources, taints, affinity | | Unschedulable | No suitable node | Add nodes, adjust requirements | | NodeNotReady | Target node unavailable | Check node status | | TaintManagerEviction | Pod evicted due to taint | Check node taints, add tolerations |

FailedScheduling Analysis:

# Common messages and fixes:
"Insufficient cpu"           → Reduce requests or add capacity
"Insufficient memory"        → Reduce requests or add capacity  
"node(s) had taint"          → Add toleration or remove taint
"node(s) didn't match selector" → Fix nodeSelector/affinity
"persistentvolumeclaim not found" → Create PVC or fix name
"0/3 nodes available"        → All nodes have issues, check each

Image Events

| Event Reason | Meaning | Resolution | |--------------|---------|------------| | Pulling | Downloading image | Normal, wait | | Pulled | Image downloaded | Normal | | Failed | Pull failed | Check image name, registry, auth | | BackOff | Repeated pull failures | Fix underlying issue | | ErrImageNeverPull | Image not local with Never policy | Change imagePullPolicy or pre-pull |

ImagePullBackOff Diagnosis:

# Check image name is correct
kubectl get pod ${POD} -o jsonpath='{.spec.containers[*].image}'

# Verify ImagePullSecrets
kubectl get pod ${POD} -o jsonpath='{.spec.imagePullSecrets}'
kubectl get secret ${SECRET} -n ${NAMESPACE}

# Test registry access
kubectl run test --image=${IMAGE} --restart=Never --rm -it -- /bin/sh

Volume Events

| Event Reason | Meaning | Resolution | |--------------|---------|------------| | FailedMount | Cannot mount volume | Check PVC, storage class, permissions | | FailedAttachVolume | Cannot attach volume | Check cloud provider, volume exists | | VolumeResizeFailed | Cannot expand volume | Check storage class allows expansion | | ProvisioningFailed | Cannot create volume | Check storage class, quotas |

PVC Pending Diagnosis:

# Check PVC status and events
kubectl describe pvc ${PVC_NAME} -n ${NAMESPACE}

# Verify StorageClass exists and is default
kubectl get storageclass

# Check for available PVs (if not dynamic provisioning)
kubectl get pv

# OpenShift: Check storage operator
oc get clusteroperator storage

Container Events

| Event Reason | Meaning | Resolution | |--------------|---------|------------| | Created | Container created | Normal | | Started | Container started | Normal | | Killing | Container being stopped | Normal during updates/scale-down | | Unhealthy | Probe failed | Fix probe or application | | ProbeWarning | Probe returned warning | Check probe configuration | | BackOff | Container crashing | Check logs, fix application |

Event Patterns

Flapping Pod (Repeated restarts)

Events:
  Warning  BackOff    Container is in waiting state due to CrashLoopBackOff
  Normal   Pulled     Container image already present
  Normal   Created    Created container
  Normal   Started    Started container
  Warning  BackOff    Back-off restarting failed container

Diagnosis: Check kubectl logs --previous, application is crashing on startup.

Resource Starvation

Events:
  Warning  FailedScheduling  0/3 nodes are available: 3 Insufficient cpu

Diagnosis: Cluster needs more capacity or pod requests are too high.

Probe Failures

Events:
  Warning  Unhealthy  Liveness probe failed: HTTP probe failed with statuscode: 503
  Normal   Killing    Container failed liveness probe, will be restarted

Diagnosis: Application not responding, check if startup is slow (use startupProbe) or app is unhealthy.

Log Analysis Patterns

Common Error Patterns

Application Startup Failures

# Java
java.lang.OutOfMemoryError: Java heap space
→ Increase memory limit, tune JVM heap (-Xmx)

java.net.ConnectException: Connection refused
→ Dependency not ready, add init container or retry logic

# Python
ModuleNotFoundError: No module named 'xxx'
→ Missing dependency, fix requirements.txt/Dockerfile

# Node.js
Error: Cannot find module 'xxx'
→ Missing dependency, fix package.json or node_modules

# General
ECONNREFUSED, Connection refused
→ Service dependency not available

ENOTFOUND, getaddrinfo ENOTFOUND
→ DNS resolution failed, check service name

Database Connection Issues

# PostgreSQL
FATAL: password authentication failed
→ Wrong credentials, check Secret values

connection refused
→ Database not running or wrong host/port

too many connections
→ Connection pool exhaustion, configure pool size

# MySQL
Access denied for user
→ Wrong credentials or missing grants

Can't connect to MySQL server
→ Database not running or network issue

# MongoDB
MongoNetworkError
→ Connection string wrong or network issue

Memory Issues

# Container OOMKilled
Last State: Terminated
Reason: OOMKilled
Exit Code: 137

→ Solutions:
1. Increase memory limit
2. Profile application memory usage
3. Fix memory leaks
4. For JVM: Set -Xmx < container limit (leave ~25% headroom)

Permission Issues

# File system
Permission denied
mkdir: cannot create directory: Permission denied
→ Check securityContext, runAsUser, fsGroup

# OpenShift SCC
Error: container has runAsNonRoot and image has non-numeric user
→ Add runAsUser to securityContext

pods "xxx" is forbidden: unable to validate against any security context constraint
→ Create appropriate SCC or use service account with SCC access

Log Analysis Commands

# Search for errors in logs
kubectl logs ${POD} -n ${NS} | grep -iE "(error|exception|fatal|panic)"

# Follow logs in real-time
kubectl logs -f ${POD} -n ${NS}

# Logs from all containers in pod
kubectl logs ${POD} -n ${NS} --all-containers

# Logs from multiple pods (by label)
kubectl logs -l app=${APP_NAME} -n ${NS} --all-containers

# Logs with timestamps
kubectl logs ${POD} -n ${NS} --timestamps

# Logs from last hour
kubectl logs ${POD} -n ${NS} --since=1h

# Logs from last 100 lines
kubectl logs ${POD} -n ${NS} --tail=100

# OpenShift: Node-level logs
oc adm node-logs ${NODE} -u kubelet
oc adm node-logs ${NODE} -u crio
oc adm node-logs ${NODE} --path=journal

Node Troubleshooting

Node Conditions

| Condition | Status | Meaning | |-----------|--------|---------| | Ready | True | Node healthy | | Ready | False | Kubelet not healthy | | Ready | Unknown | No heartbeat from node | | MemoryPressure | True | Low memory | | DiskPressure | True | Low disk space | | PIDPressure | True | Too many processes | | NetworkUnavailable | True | Network not configured |

Node NotReady Diagnosis

# Check node status
kubectl describe node ${NODE_NAME}

# Check kubelet status (SSH to node or via oc adm)
systemctl status kubelet
journalctl -u kubelet -f

# Check container runtime
systemctl status crio  # or containerd/docker
journalctl -u crio -f

# Check node resources
df -h
free -m
top

# OpenShift: Machine status
oc get machines -n openshift-machine-api
oc describe machine ${MACHINE} -n openshift-machine-api

Node Resource Pressure

# Check resource allocation vs capacity
kubectl describe node ${NODE} | grep -A 10 "Allocated resources"

# Find pods using most resources
kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory

# Evict pods from node (drain)
kubectl drain ${NODE} --ignore-daemonsets --delete-emptydir-data

Networking Troubleshooting

DNS Issues

# Test DNS resolution from a debug pod
kubectl run dns-test --image=busybox:1.28 --rm -it --restart=Never -- nslookup ${SERVICE_NAME}
kubectl run dns-test --image=busybox:1.28 --rm -it --restart=Never -- nslookup ${SERVICE_NAME}.${NAMESPACE}.svc.cluster.local

# Ch

<!-- Content truncated for initial SEO render. Open the source file tab for the full file. -->