Skip to content

Use BATS to Automate Kubernetes Testing

BATS Testing
The Bash Automated Testing System brings familiar scripting to Kubernetes validation.

Have you wanted to test things that you know how to run in your bash scripts or terminal, but aren't quite sure how to make it happen? The Bash Automated Testing System (BATS) makes that dream a reality!

BATS lets you create automated, repeatable tests using familiar bash commands. Your tests work equally well in your terminal and CI/CD pipelines, giving you confidence to promote Kubernetes resources to production.

This post covers practical examples of testing Kubernetes clusters, from basic API connectivity to complex application health checks and resource validation.

Why BATS for Kubernetes Testing?

Traditional Kubernetes testing often involves complex frameworks that require learning new languages or tools. BATS takes a different approach - if you can write a bash script, you can write a BATS test.

The BATS Advantage

  • ✅ Familiar Syntax: Uses bash commands you already know
  • ✅ CI/CD Ready: Runs anywhere bash runs
  • ✅ Fast Execution: Lightweight and quick to execute
  • ✅ Clear Output: Human-readable test results
  • ✅ Easy Integration: Works with any CI/CD pipeline
  • ✅ No Dependencies: Just needs bash and BATS core

Real-World Use Cases

BATS is perfect for: - Cluster smoke tests after deployments - Application health verification before promoting to production - Infrastructure validation in CI/CD pipelines - Configuration drift detection in existing clusters - Multi-environment consistency checks

Setting Up BATS for Kubernetes Testing

Installing BATS

# Option 1: Using Git (recommended for CI/CD)
git clone https://github.com/bats-core/bats-core.git
cd bats-core
sudo./install.sh /usr/local

# Option 2: Using package manager
# macOS
brew install bats-core

# Ubuntu/Debian
sudo apt-get install bats

# Verify installation
bats --version

Project Structure for Kubernetes Testing

k8s-bats-tests/
├── tests/
│ ├── cluster/
│ │ ├── api-server.bats
│ │ ├── nodes.bats
│ │ └── networking.bats
│ ├── applications/
│ │ ├── nginx.bats
│ │ ├── database.bats
│ │ └── monitoring.bats
│ └── helpers/
│ ├── k8s-helpers.bash
│ └── test-helpers.bash
├── config/
│ └── test-config.yaml
└── run-tests.sh

Basic Cluster Health Tests

Let's start with fundamental tests that validate your cluster is functioning properly.

API Server Connectivity

# tests/cluster/api-server.bats
#!/usr/bin/env bats

load '../helpers/k8s-helpers'

@test "kubectl can connect to API server" {
 run kubectl cluster-info
 [ "$status" -eq 0 ]
 [[ "$output" == *"Kubernetes control plane"* ]]
}

@test "API server is responding to health checks" {
 run kubectl get --raw='/healthz'
 [ "$status" -eq 0 ]
 [[ "$output" == "ok" ]]
}

@test "API server version is accessible" {
 run kubectl version --client=false --output=json
 [ "$status" -eq 0 ]
 [[ "$output" == *"serverVersion"* ]]
}

Node Health Validation

# tests/cluster/nodes.bats
#!/usr/bin/env bats

load '../helpers/k8s-helpers'

@test "all nodes are in Ready state" {
 run kubectl get nodes --no-headers
 [ "$status" -eq 0 ]

 # Check each line doesn't contain NotReady
 while IFS= read -r line; do
 [[! "$line" == *"NotReady"* ]]
 done <<< "$output"
}

@test "minimum number of nodes are present" {
 run kubectl get nodes --no-headers
 [ "$status" -eq 0 ]

 node_count=$(echo "$output" | wc -l)
 minimum_nodes=${MINIMUM_NODES:-2}
 [ "$node_count" -ge "$minimum_nodes" ]
}

@test "nodes have sufficient resources" {
 run kubectl top nodes --no-headers
 [ "$status" -eq 0 ]

 # Check CPU usage is below 80%
 while read -r line; do
 cpu_usage=$(echo "$line" | awk '{print $3}' | sed 's/%//')
 [ "$cpu_usage" -lt 80 ]
 done <<< "$output"
}

Core System Pods

# tests/cluster/system-pods.bats
#!/usr/bin/env bats

load '../helpers/k8s-helpers'

@test "kube-system pods are running" {
 run kubectl get pods -n kube-system --field-selector=status.phase!=Running --no-headers
 [ "$status" -eq 0 ]
 [ -z "$output" ] # No non-running pods
}

@test "CoreDNS is functional" {
 run kubectl get deployment coredns -n kube-system -o jsonpath='{.status.readyReplicas}'
 [ "$status" -eq 0 ]
 [ "$output" -gt 0 ]
}

@test "kube-proxy is running on all nodes" {
 expected_nodes=$(kubectl get nodes --no-headers | wc -l)
 actual_pods=$(kubectl get pods -n kube-system -l k8s-app=kube-proxy --no-headers | wc -l)
 [ "$expected_nodes" -eq "$actual_pods" ]
}

Application-Specific Testing

Web Application Health Checks

# tests/applications/nginx.bats
#!/usr/bin/env bats

load '../helpers/k8s-helpers'

setup() {
 NAMESPACE=${NAMESPACE:-default}
 APP_NAME=${APP_NAME:-nginx}
}

@test "nginx deployment is ready" {
 run kubectl get deployment "$APP_NAME" -n "$NAMESPACE"
 [ "$status" -eq 0 ]
 [[ "$output" == *"1/1"* ]]
}

@test "nginx pods are running and ready" {
 run kubectl get pods -l app="$APP_NAME" -n "$NAMESPACE" --field-selector=status.phase=Running --no-headers
 [ "$status" -eq 0 ]
 [ -n "$output" ] # At least one pod running

 # Check all pods are ready (ready/total format)
 while read -r line; do
 ready_status=$(echo "$line" | awk '{print $2}')
 ready_count=$(echo "$ready_status" | cut -d'/' -f1)
 total_count=$(echo "$ready_status" | cut -d'/' -f2)
 [ "$ready_count" -eq "$total_count" ]
 done <<< "$output"
}

@test "nginx service is accessible" {
 run kubectl get service "$APP_NAME" -n "$NAMESPACE"
 [ "$status" -eq 0 ]

 # Port forward and test HTTP response
 kubectl port-forward service/"$APP_NAME" 8080:80 -n "$NAMESPACE" &
 PF_PID=$!
 sleep 2

 run curl -s -o /dev/null -w "%{http_code}" http://localhost:8080
 kill $PF_PID 2>/dev/null

 [ "$status" -eq 0 ]
 [ "$output" = "200" ]
}

@test "nginx has proper resource limits" {
 run kubectl get pod -l app="$APP_NAME" -n "$NAMESPACE" -o jsonpath='{.items[0].spec.containers[0].resources.limits.memory}'
 [ "$status" -eq 0 ]
 [ -n "$output" ] # Memory limit is set
}

Database Connectivity Tests

# tests/applications/database.bats
#!/usr/bin/env bats

load '../helpers/k8s-helpers'

setup() {
 NAMESPACE=${NAMESPACE:-default}
 DB_NAME=${DB_NAME:-postgres}
}

@test "database pod is running" {
 run kubectl get pods -l app="$DB_NAME" -n "$NAMESPACE" --field-selector=status.phase=Running
 [ "$status" -eq 0 ]
 [[ "$output" == *"Running"* ]]
}

@test "database is accepting connections" {
 run kubectl exec deployment/"$DB_NAME" -n "$NAMESPACE" -- pg_isready -h localhost
 [ "$status" -eq 0 ]
 [[ "$output" == *"accepting connections"* ]]
}

@test "database has persistent storage" {
 run kubectl get pvc -l app="$DB_NAME" -n "$NAMESPACE"
 [ "$status" -eq 0 ]
 [[ "$output" == *"Bound"* ]]
}

Helper Functions and Utilities

Create reusable helper functions to avoid code duplication and improve maintainability:

# tests/helpers/k8s-helpers.bash

# Wait for deployment to be ready with retry logic
wait_for_deployment() {
 local deployment="$1"
 local namespace="${2:-default}"
 local timeout="${3:-300}"
 local max_retries="${4:-3}"

 for i in $(seq 1 $max_retries); do
 if kubectl wait --for=condition=available \
 --timeout="${timeout}s" \
 deployment/"$deployment" -n "$namespace"; then
 return 0
 fi
 echo "Attempt $i failed, retrying..."
 sleep 10
 done
 return 1
}

# Check if pod has specific label
pod_has_label() {
 local pod="$1"
 local label="$2"
 local namespace="${3:-default}"

 kubectl get pod "$pod" -n "$namespace" \
 -o jsonpath="{.metadata.labels.$label}" | grep -q.
}

# Get pod resource usage
get_pod_cpu_usage() {
 local pod="$1"
 local namespace="${2:-default}"

 kubectl top pod "$pod" -n "$namespace" --no-headers | awk '{print $2}'
}

# Test service endpoint
test_service_endpoint() {
 local service="$1"
 local port="$2"
 local namespace="${3:-default}"
 local path="${4:-/}"

 kubectl run curl-test --image=curlimages/curl:latest --rm -i --restart=Never \
 -- curl -s -o /dev/null -w "%{http_code}" \
 "http://$service.$namespace.svc.cluster.local:$port$path"
}

Configuration-Driven Tests

# tests/applications/configurable.bats
#!/usr/bin/env bats

load '../helpers/k8s-helpers'
load '../helpers/test-config'

@test "applications defined in config are healthy" {
 for app in "${APPLICATIONS[@]}"; do
 echo "Testing application: $app"
 run kubectl get deployment "$app" -n "$NAMESPACE"
 [ "$status" -eq 0 ]
 [[ "$output" == *"1/1"* ]] || [[ "$output" == *"2/2"* ]]
 done
}

@test "required secrets exist" {
 for secret in "${REQUIRED_SECRETS[@]}"; do
 echo "Checking secret: $secret"
 run kubectl get secret "$secret" -n "$NAMESPACE"
 [ "$status" -eq 0 ]
 done
}

Load Testing with BATS

# tests/performance/load-test.bats
#!/usr/bin/env bats

@test "application handles concurrent requests" {
 # Start multiple background jobs
 for i in {1..10}; do
 (
 for j in {1..5}; do
 curl -s -o /dev/null -w "%{http_code}\\n" \
 http://my-app.default.svc.cluster.local
 done
 ) &
 done

 # Wait for all background jobs
 wait

 # Check application is still responsive
 run curl -s -o /dev/null -w "%{http_code}" \
 http://my-app.default.svc.cluster.local
 [ "$output" = "200" ]
}

CI/CD Integration

GitLab CI Integration

#.gitlab-ci.yml
stages:
 - test
 - deploy
 - validate

k8s-smoke-tests:
 stage: validate
 image:
 name: bitnami/kubectl:latest
 entrypoint: ['']
 before_script:
 - apt-get update && apt-get install -y git curl jq
 - git clone https://github.com/bats-core/bats-core.git
 - cd bats-core &&./install.sh /usr/local && cd..
 - kubectl config use-context $KUBE_CONTEXT
 script:
 - bats --formatter junit tests/cluster/ > cluster-results.xml
 - bats --formatter junit tests/applications/ > app-results.xml
 artifacts:
 reports:
 junit:
 - "*-results.xml"
 expire_in: 30 days
 only:
 - main
 when: manual

Test Runner Script

#!/bin/bash
# run-tests.sh

set -e

NAMESPACE=${NAMESPACE:-default}
ENVIRONMENT=${ENVIRONMENT:-development}
TEST_SUITE=${TEST_SUITE:-all}

echo "🧪 Running Kubernetes tests for environment: $ENVIRONMENT"
echo "📋 Namespace: $NAMESPACE"
echo "🎯 Test suite: $TEST_SUITE"

# Export variables for tests
export NAMESPACE
export ENVIRONMENT

case $TEST_SUITE in
 "cluster")
 echo "🏗️ Running cluster tests..."
 bats tests/cluster/;;
 "apps")
 echo "🚀 Running application tests..."
 bats tests/applications/;;
 "all")
 echo "🎯 Running all tests..."
 bats tests/cluster/
 bats tests/applications/;;
 *)
 echo "❌ Unknown test suite: $TEST_SUITE"
 echo "Available options: cluster, apps, all"
 exit 1;;
esac

echo "✅ All tests completed successfully!"

Advanced Testing Scenarios

Multi-Environment Testing

# tests/environments/staging-vs-prod.bats
#!/usr/bin/env bats

@test "staging and production have same application versions" {
 staging_version=$(kubectl get deployment myapp -n staging -o jsonpath='{.spec.template.spec.containers[0].image}')
 prod_version=$(kubectl get deployment myapp -n production -o jsonpath='{.spec.template.spec.containers[0].image}')

 # In practice, you might want different logic here
 echo "Staging: $staging_version"
 echo "Production: $prod_version"

 # Both should be valid images
 [[ "$staging_version" == *":"* ]]
 [[ "$prod_version" == *":"* ]]
}

@test "production has higher resource limits than staging" {
 staging_memory=$(kubectl get deployment myapp -n staging -o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}')
 prod_memory=$(kubectl get deployment myapp -n production -o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}')

 # Convert to numbers for comparison (simplified)
 staging_mb=$(echo "$staging_memory" | sed 's/Mi//')
 prod_mb=$(echo "$prod_memory" | sed 's/Mi//')

 [ "$prod_mb" -gt "$staging_mb" ]
}

Security Testing

# tests/security/rbac.bats
#!/usr/bin/env bats

@test "default service account cannot create deployments" {
 run kubectl auth can-i create deployments --as=system:serviceaccount:default:default
 [ "$status" -eq 0 ]
 [[ "$output" == "no" ]]
}

@test "application pods run as non-root" {
 run kubectl get pods -l app=myapp -o jsonpath='{.items[*].spec.securityContext.runAsNonRoot}'
 [ "$status" -eq 0 ]
 [[ "$output" == *"true"* ]]
}

@test "network policies are enforced" {
 run kubectl get networkpolicies
 [ "$status" -eq 0 ]
 [ -n "$output" ] # At least one network policy exists
}

@test "pods do not run with privileged security context" {
 run kubectl get pods --all-namespaces -o jsonpath='{.items[*].spec.securityContext.privileged}'
 [ "$status" -eq 0 ]
 [[ "$output"!= *"true"* ]] # No pods should be privileged
}

Best Practices and Tips

Test Organization

  1. Group by Scope: Separate cluster-level tests from application tests
  2. Use Descriptive Names: Test names should clearly indicate what they validate
  3. Leverage Helpers: Create reusable functions for common operations
  4. Environment-Specific: Use variables for environment-specific values

Performance Considerations

# Parallel test execution
bats --jobs 4 tests/

# Fail fast on first error
bats --tap tests/ | tee results.tap

# Only run specific tests
bats tests/cluster/api-server.bats

Debugging Tests

# Run with verbose output
bats --verbose-run tests/

# Show all output (including successful tests)
bats --show-output-of-passing-tests tests/

# Debug specific test
bats --filter "nginx deployment is ready" tests/applications/

Complete Application Stack Testing

Here's a comprehensive example that tests a complete web application stack:

# tests/complete-stack.bats
#!/usr/bin/env bats

load 'helpers/k8s-helpers'

setup() {
 NAMESPACE="web-app"
 APP_NAME="todo-app"
 DB_NAME="postgres"
}

@test "database is ready" {
 wait_for_deployment "$DB_NAME" "$NAMESPACE"

 run kubectl exec deployment/"$DB_NAME" -n "$NAMESPACE" -- pg_isready
 [ "$status" -eq 0 ]
}

@test "backend API is ready" {
 wait_for_deployment "$APP_NAME-backend" "$NAMESPACE"

 # Test API health endpoint
 response=$(test_service_endpoint "$APP_NAME-backend" "8080" "$NAMESPACE" "/health")
 [ "$response" = "200" ]
}

@test "frontend is serving content" {
 wait_for_deployment "$APP_NAME-frontend" "$NAMESPACE"

 response=$(test_service_endpoint "$APP_NAME-frontend" "80" "$NAMESPACE")
 [ "$response" = "200" ]
}

@test "database connection from API works" {
 run kubectl exec deployment/"$APP_NAME-backend" -n "$NAMESPACE" -- \
 curl -s http://localhost:8080/api/status
 [ "$status" -eq 0 ]
 [[ "$output" == *"database_connected\":true"* ]]
}

@test "complete request flow works" {
 # Create a test todo item via API
 run kubectl run curl-test --image=curlimages/curl:latest --rm -i --restart=Never \
 -- curl -X POST -H "Content-Type: application/json" \
 -d '{"title":"BATS test item","completed":false}' \
 "http://$APP_NAME-backend.$NAMESPACE.svc.cluster.local:8080/api/todos"

 [ "$status" -eq 0 ]
 [[ "$output" == *"BATS test item"* ]]
}

Conclusion

BATS provides a powerful, familiar way to test Kubernetes clusters and applications. By leveraging bash commands you already know, you can create comprehensive test suites that validate everything from cluster health to application functionality.

The key benefits of this approach:

  • ✅ Simplicity: No need to learn new testing frameworks
  • ✅ Flexibility: Test anything you can script
  • ✅ Integration: Works seamlessly with any CI/CD pipeline
  • ✅ Reliability: Catch issues before they affect users
  • ✅ Documentation: Tests serve as executable documentation

Next Steps

  1. Start Small: Begin with basic cluster health checks
  2. Add Gradually: Expand to application-specific tests
  3. Automate: Integrate into your CI/CD pipelines
  4. Iterate: Continuously improve based on failures and feedback
  5. Share: Create a library of common tests for your team

With BATS, you can build confidence in your Kubernetes deployments through comprehensive, automated testing that speaks your language - bash!

Pro Tip

Combine BATS with your GitLab runners from our GitLab Runner EKS series for a powerful testing pipeline that validates your applications automatically!

What will you test first with BATS? Start with the basic cluster tests and let me know how it works for your environment!