Health Check Endpoint Design: /healthz Best Practices

A /healthz endpoint that always returns 200 is worse than no health endpoint at all - it gives orchestrators and monitoring tools false confidence right up until a full outage. Good health check design distinguishes between "the process is running," "the process can serve traffic," and "the process's dependencies are actually healthy," and exposes each as a distinct signal.

Liveness vs Readiness vs Deep Health

These three checks answer different questions and should not be conflated into one endpoint.

Conflating these causes real incidents. If your liveness probe checks the database and the database has a brief blip, Kubernetes will restart every pod simultaneously - turning a transient DB issue into a full outage.

Endpoint Structure

A common and effective pattern is three separate routes:

GET /healthz/live      -> liveness: process responds, no dependency checks
GET /healthz/ready     -> readiness: can serve traffic (config loaded, warm cache, etc.)
GET /healthz/deep      -> deep check: DB, cache, downstream APIs

Liveness: Keep It Trivial

curl -s -o /dev/null -w "%{http_code}\n" https://service.internal/healthz/live

The handler should do nothing but confirm the HTTP server is accepting connections and the event loop isn't blocked. No database calls, no network calls to other services.

@app.get("/healthz/live")
def live():
    return {"status": "ok"}, 200

Readiness: Check What's Needed to Serve Traffic

Readiness can check local state - is startup complete, are required config values loaded, is the connection pool initialized - without necessarily reaching out over the network.

@app.get("/healthz/ready")
def ready():
    if not app_state.startup_complete:
        return {"status": "starting"}, 503
    return {"status": "ready"}, 200

Deep Health: Check Dependencies, With Timeouts

@app.get("/healthz/deep")
def deep():
    checks = {}
    checks["database"] = check_db(timeout=1.0)
    checks["cache"] = check_redis(timeout=0.5)
    checks["queue"] = check_queue(timeout=1.0)

    healthy = all(v["ok"] for v in checks.values())
    status_code = 200 if healthy else 503
    return {"status": "ok" if healthy else "degraded", "checks": checks}, status_code

Always set aggressive timeouts on dependency checks. A health check that hangs waiting on a slow database defeats the purpose - it should fail fast and report 503 rather than block.

Response Format

A structured JSON body is far more useful than a bare 200/503. A reasonable shape:

{
  "status": "degraded",
  "version": "3.14.2",
  "uptime_seconds": 48213,
  "checks": {
    "database": { "ok": true, "latency_ms": 4 },
    "cache": { "ok": false, "latency_ms": 502, "error": "timeout" },
    "queue": { "ok": true, "latency_ms": 12 }
  }
}

This lets both humans and automated monitoring assert on individual dependency status, not just an overall boolean, which is critical for root-causing an incident quickly.

Caching Health Check Results

Deep checks against a database on every single request from every monitoring probe and every orchestrator can itself become load. Mitigate with a short cache:

• Cache deep check results for 2–5 seconds
• Serve cached results to concurrent callers within that window
• Never cache liveness - it must reflect the current instant

_cache = {"result": None, "expires_at": 0}

def get_deep_health():
    now = time.time()
    if _cache["result"] and now < _cache["expires_at"]:
        return _cache["result"]
    result = run_deep_checks()
    _cache["result"] = result
    _cache["expires_at"] = now + 3
    return result

Security: Don't Leak Internals

Health endpoints are often unauthenticated by necessity (load balancers and orchestrators frequently can't attach credentials). That means they're also publicly reachable, and a naive implementation leaks information:

• Don't return full stack traces or internal hostnames in error details
• Don't expose dependency connection strings or internal IPs, even in error messages
• Consider a lightweight public /healthz (status only) and a more detailed /healthz/deep restricted to internal networks or an auth token

# Public: minimal info
curl https://example.com/healthz
{"status":"ok"}

# Internal only, behind auth or network restriction
curl -H "Authorization: Bearer $INTERNAL_TOKEN" \
  https://example.com/healthz/deep

Checklist: Health Endpoint Design

• Liveness endpoint does zero dependency checks
• Readiness endpoint reflects startup and local state only
• Deep health endpoint checks dependencies with strict timeouts
• Each check returns a structured status, not just a boolean
• Deep results are cached for a few seconds to avoid self-inflicted load
• Detailed dependency info is restricted from public access
• Orchestrator liveness/readiness probes point to the correct distinct endpoints
• External monitoring checks /healthz/deep on a schedule separate from orchestrator probes

Monitoring Your Health Endpoints Externally

Internal orchestrator probes only tell you about instances the orchestrator can already see. External uptime monitoring against your public health endpoint - from outside your network, on a schedule - catches DNS issues, edge/CDN misconfigurations, and network partitions that an internal probe never will. Tools like ServerInspector can assert on both the status code and the JSON body of /healthz/deep, alerting when a specific dependency degrades rather than just when the whole service goes down.

FAQ

Should liveness and readiness ever be the same endpoint?

No. Liveness failures trigger a restart; readiness failures just pull the instance from load balancer rotation. Merging them means a slow dependency can cause unnecessary, cascading restarts across your fleet.

What HTTP status code should a degraded deep health check return?

503 Service Unavailable is the standard choice - it signals "temporarily unable to serve" without implying the process itself is broken, which keeps orchestrators from taking overly aggressive action like restarting.

How often should external monitoring hit my health endpoint?

Every 30–60 seconds for liveness/basic checks is typical; deep health checks with dependency validation can run every 1–5 minutes since they carry more load and are meant to catch slower-developing degradation.

Can I put authentication on my health check?

Yes for detailed/deep endpoints - restrict them to internal networks or require a token. Keep a minimal public endpoint (status only, no details) for load balancers and external monitors that can't attach credentials.

Do I need a health check for a stateless service with no dependencies?

Yes, at minimum a liveness check confirming the process responds. Even stateless services benefit from readiness checks during startup and rolling deploys to avoid routing traffic before the instance is fully warmed up.