Production Engineering Readiness Checklist

Before promoting any service or architecture change to production, verify the following baseline criteria.

1. Reliability & Traffic Management

• Timeout Budgets: Every downstream HTTP/gRPC call defines an explicit, non-infinite timeout.
• Retry Policies: Retries are restricted to idempotent endpoints and incorporate exponential backoff with full random jitter.
• Circuit Breakers: Unhealthy external dependencies fail fast rather than stalling request worker threads.
• Health Probes: Liveness and readiness probes verify genuine internal health without putting excessive load on databases.
• Rate Limiting: Public ingress endpoints implement token-bucket or sliding-window rate limiting to resist denial-of-service degradation.

2. Observability & Telemetry

• Structured Logging: All logs are emitted in structured JSON format with standard fields (timestamp, level, trace_id, message).
• Distributed Tracing: Incoming request contexts propagate W3C traceparent headers through all internal boundaries.
• Golden Signals Metrics: Dashboards actively track Latency, Traffic, Errors, and Saturation.
• Actionable Alerts: Alerts are configured only for user-impacting symptoms, each paired with an explicit runbook URL.

3. Security & Access Control

• Secret Management: No API keys, database credentials, or certificates exist in source control or container layers.
• Least Privilege: Application runtime processes run under non-root system users.
• TLS Everywhere: All internal and external network traffic is encrypted via TLS 1.3.
• Dependency Audits: Automated vulnerability scans (npm audit, trivy, snyk) run in the CI pipeline with zero critical/high blockers.

4. Rollback & Disaster Recovery

• Automated Rollback: Deployment mechanism supports single-click or metric-triggered rollback to the preceding immutable build.
• Database Migrations: Schema alterations are backward-compatible (expand-and-contract pattern) so older code runs safely alongside new migrations.
• Documented Runbooks: The on-call engineer has a documented procedure for diagnosing degraded states and restarting services safely.