A pod dying and a system recovering are not the same thing. That gap is where most of the interesting failures actually live, and it's what pushed me to build fault-sentinel, a lightweight, Kubernetes-native chaos engineering CLI written in Go.
Kubernetes gives you self-healing by default: a container crashes, the kubelet restarts it; a node degrades, the scheduler moves work elsewhere. What that default behavior doesn't tell you is whether anything using the system noticed. A pod's status can read Running again in seconds while in-flight requests during the gap were silently dropped. Recovery, as reported by kubectl get pods, and recovery, as experienced by whatever depends on that pod, are two different claims.
fault-sentinel exists to test the second one directly, by deliberately breaking things and watching what actually happens, rather than trusting that a clean deployment means a resilient one.
Established chaos engineering platforms do this more comprehensively than a personal CLI ever could. I built fault-sentinel anyway, for a specific reason: I wanted to understand the mechanics from the inside, not just apply someone else's custom resource and trust the abstraction. Writing the pod-eviction logic directly against client-go means I know exactly what a graceful deletion does at the API level, not just that a CRD exists for it.
This is a personal learning project, not a replacement for production-grade chaos tooling, and it doesn't try to be.
Architecture
graph TD
CLI[fault-sentinel CLI] -->|client-go| KubeAPI[Kubernetes API Server]
KubeAPI -->|Pod Termination| PodTerm[Target Workload]
KubeAPI -->|SPDY Exec| Exec[Container Exec Session]
Exec -->|tc netem| NetLat[Network Delay Injection]
CLI -->|/metrics| Prom[Prometheus]
Prom -->|Dashboards| Grafana[Grafana]
The CLI runs standalone, no persistent daemon, no custom resource definitions. It only holds cluster access for the duration of an active experiment, not as a standing permission.
The three experiments
Pod termination: selects a pod by label selector and deletes it via the CoreV1 API directly, rather than shelling out to kubectl delete, to watch how fast the ReplicaSet controller notices and replaces it.
Network latency injection: uses Linux Traffic Control (tc netem) inside the target container's network namespace to inject artificial delay, with automatic cleanup once the experiment window ends. This requires the NET_ADMIN Linux capability on the target container, which is itself a real trade-off: you're loosening a workload's security posture slightly, purely so it can be a valid subject for chaos testing.
CPU stress: generates bounded compute load across a configurable number of worker goroutines, to observe how a workload behaves under compute pressure.
Proving it, not just building it
I ran each experiment against a live cluster and checked the actual outcome, not just that the command executed without error:
Killing a pod by label selector and confirming the ReplicaSet controller replaced it, watched via kubectl get pods -w
Injecting a 250ms delay into a running pod's network namespace and confirming both the delay and its removal afterward
Running CPU stress and confirming it stayed bounded within the container's own resource limits, rather than affecting the node
The CI pipeline runs golangci-lint, unit tests with Go's race detector, a multi-stage Docker build, and a Trivy vulnerability scan on every push. The race detector matters specifically here: a chaos tool with its own concurrency bugs while testing other systems' concurrency behavior would defeat the entire point.
Observability
fault-sentinel exposes Prometheus metrics on :8080/metrics during every run: chaos_experiments_total (by type and status), chaos_injected_faults_total (by target and fault type), and chaos_experiment_duration_seconds. These let you correlate the exact window of an injected failure against whatever else you're watching, latency, error rates, endpoint convergence, in Grafana.
Security model
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fault-sentinel-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create", "get"]
A chaos tool is, by definition, something that deletes pods and manipulates network behavior. That's real trust to grant a piece of software, so RBAC is scoped as narrowly as the tool can function on: list and delete on pods, nothing cluster-wide, nothing touching secrets or unrelated resources.
What this project didn't try to be
This isn't a replacement for Chaos Mesh or Litmus, and I'm not claiming it should run against a production cluster. It has no scheduled or recurring experiments, no web UI, no multi-cluster orchestration. What it does have is a small, complete surface area I understand fully, because I built every part of it myself, which was the actual point.
The lesson that stuck with me most: recovery isn't binary. A pod coming back after being killed isn't the same as the system recovering. You have to watch what happens to in-flight requests and error rates during the gap itself, not just confirm the pod count went back to normal afterward. That's the difference between "it healed" and "it healed in a way that didn't hurt anyone using it."
Repo: github.com/aashiruu/fault-sentinel
Author: Okikiola Ashiru — Cloud & DevOps Engineer
