A detailed backend reference
Errors are not exceptional, they are inevitable. Database queries fail, external APIs time out, users send bad data, and business logic hits edge cases nobody predicted. This chapter builds the mindset and concrete toolkit for detecting, handling, and recovering from every class of backend error, before it silently costs you money or trust.
01
The Fault-Tolerant Mindset
“The question is not whether errors will happen, it is how you will handle them when they do.”
Every backend engineer must internalise a simple truth: your system will fail. Not might. Will. The sources are everywhere:
- Database queries will occasionally fail or time out.
- External APIs (payments, email, auth) will go down.
- Users will send malformed, missing, or malicious data.
- Business logic will hit edge cases no one thought of during design.
A fault-tolerant system is not one that never breaks. It is one that breaks predictably, recovers gracefully, and tells you exactly what happened. Achieving that requires a deliberate mindset shift from “I’ll handle errors later” to “I’ll design for failure from day one.”
02
The Five Classes of Backend Errors
Backend errors can be grouped into five broad categories. Each has a different origin, detection strategy, and fix.
(1) Logic Errors
App runs but produces wrong results. Hardest to detect. Can silently drain money for weeks.
(2) Database Errors
Connection failures, deadlocks, constraint violations, malformed SQL. Can bring the whole app down.
(3) External Service Errors
Third-party APIs (payment, email, auth) time out, rate-limit, or go offline. You have no control.
(4) Input Validation Errors
Users send bad, missing, or out-of-range data. Easiest to handle, if your validation layer is robust.
(5) Configuration Errors
Missing env vars, wrong credentials on deploy. Surface at startup, not at runtime, if you do it right.
03
Logic Errors: The Silent Killers
Logic errors are the most dangerous class because your application keeps running, it just does the wrong thing. No crash, no stack trace, no 500 response. Just quietly wrong results accumulating over time.
Classic Example
An e-commerce platform applies a discount twice due to a bug in the promotion engine. The result: negative shipping costs. The app runs perfectly. Every order ships. The company loses money on every transaction. This goes unnoticed for weeks because no monitoring alert fires on “negative shipping cost.”
Common Root Causes
- Misunderstood requirements: notes from a sprint meeting were ambiguous; you implemented what you thought was asked, not what was intended.
- Incorrect algorithms: a complex discount or pricing formula has an off-by-one error or a wrong operator (
*instead of+). - Unhandled edge cases: a user who has never purchased before triggers a “past-purchase-based” discount path that wasn’t designed for zero-purchase users.
Prevention Strategies
- Write unit tests for every business rule, especially discount, pricing, and permission logic.
- Add business metric monitoring (e.g., alert if average order value drops by >20% in one hour).
- Use property-based testing (Go:
gopter, Python:hypothesis) to auto-generate edge-case inputs. - Require peer review for all payment and auth-related code changes.
04
Database Errors
Most backend applications are meaningless without their database. A database error of any kind means your app cannot serve real data, which usually means a broken UI or cascading failures across services.
(1) Connection Errors
Your backend cannot reach the database server. Possible causes:
- Network partition or DNS failure between app server and DB server.
- Database server is overloaded or down.
- Connection pool exhausted: all pooled TCP connections are in use; new requests queue up or fail immediately.
(2) Constraint Violation Errors
You are trying to perform an operation that violates a database-level rule:
| Constraint Type | Trigger | Appropriate Response |
|---|---|---|
| Unique | Insert a duplicate email / username | HTTP 409 Conflict or 400, “Email already in use” |
| Foreign Key | Reference a row that doesn’t exist | HTTP 404, “Author ID not found” / 400 |
| Not Null | Missing required column value | HTTP 400, “Field X is required” |
| Check | Value fails a custom rule (e.g. price > 0) | HTTP 400, domain-specific message |
(3) Query / Syntax Errors
Malformed SQL, a table name typo, referencing a column that was renamed, or a missing join condition. These are usually caught in development but can slip through if raw SQL strings are built dynamically.
(4) Deadlocks
A deadlock occurs when two (or more) transactions each hold a lock that the other needs:
Postgres detects deadlocks automatically and kills one transaction with error code 40P01. Your application must retry that transaction. Prevention: always acquire locks in a consistent order across all code paths.
05
External Service Errors
Modern SaaS backends depend on a constellation of third-party services, payment processors (Stripe), email (Resend, SendGrid), object storage (S3), auth (Clerk, Auth0), AI (OpenAI). Every one of these is a point of failure outside your control.
(1) Network Failures
The internet between your server and the external API is unreliable. You will encounter: connection timeouts, DNS resolution failures, network partitions, and TLS handshake errors. Set explicit timeouts on every outgoing HTTP call, never let a slow third-party API block your goroutine / thread indefinitely.
(2) Rate Limiting: HTTP 429
Every serious external API enforces rate limits to prevent abuse. If your app hammers an API (due to a bug, a traffic spike, or a loop error), you will receive HTTP 429 Too Many Requests.
The standard mitigation is Exponential Backoff with Jitter:
(3) Service Outage / Downtime
Major cloud providers (AWS, GCP) and popular SaaS services go down occasionally. Your app needs a strategy for when a critical dependency is completely unavailable:
- Fallback: if Redis cache is down, fall back to direct DB reads for non-critical data.
- Graceful degradation: disable the affected feature (e.g., “AI suggestions temporarily unavailable”) rather than crashing the whole app.
- Circuit breaker pattern: after N consecutive failures, stop sending requests to the broken service and return a cached/default response immediately. Re-try the service after a cool-down period.
06
Input Validation Errors
These are the easiest errors to handle because you define the rules. Your validation layer is the first line of defence: catch bad data at the entry point, before it reaches your database or business logic.
Types of Validation
| Type | What It Checks | Example |
|---|---|---|
| Format | Shape/pattern of the value | Email regex, ISO date, E.164 phone |
| Range | Numeric bounds, string length, array size | Price: 0-99999, name: 2-100 chars |
| Required | Mandatory field present | user_id must not be null |
| Business Rule | Domain-specific constraint | Booking end_date > start_date |
| Referential | Related entity actually exists | category_id exists in categories table |
Always validate at both layers: frontend (UX) and backend (security). Never trust client-side validation alone. The backend is the authoritative gate.
07
Configuration Errors
Configuration errors happen at the boundary between environments, dev -> staging -> production. A missing OPENAI_API_KEY, a wrong database URL, or a forgotten secret can silently break specific features while the rest of the app appears healthy.
Fail Fast at Startup: Not at Runtime
The golden rule: validate all required environment variables before the server starts accepting traffic. If any are missing or corrupt, crash immediately with a clear error message.
Bad: Runtime Failure
- App starts successfully
- First user hits the AI image endpoint
- OpenAI call fails, key is missing
- User gets a mysterious 500 error
- Old deployment is already stopped
- Site is down until manually fixed
Good: Startup Failure
- New deployment starts
- Config validation runs immediately
- Missing key detected -> process exits with clear message
- Blue-green: old deployment still running
- Zero downtime, ops team fixes and redeploys
Config Validation at Boot
check every required variable before the port is bound
package config
import (
"fmt"
"os"
"strings"
)
type Config struct {
DatabaseURL string
OpenAIKey string
JWTSecret string
ResendAPIKey string
}
// MustLoad panics if any required variable is missing.
// Call this once in main() before http.ListenAndServe.
func MustLoad() Config {
required := []string{
"DATABASE_URL",
"OPENAI_API_KEY",
"JWT_SECRET",
"RESEND_API_KEY",
}
var missing []string
for _, key := range required {
if os.Getenv(key) == "" {
missing = append(missing, key)
}
}
if len(missing) > 0 {
// Crash immediately, loud and clear
panic(fmt.Sprintf("[FATAL] missing required env vars: %s",
strings.Join(missing, ", ")))
}
return Config{
DatabaseURL: os.Getenv("DATABASE_URL"),
OpenAIKey: os.Getenv("OPENAI_API_KEY"),
JWTSecret: os.Getenv("JWT_SECRET"),
ResendAPIKey: os.Getenv("RESEND_API_KEY"),
}
}
// main.go
func main() {
cfg := config.MustLoad() // panics here if config invalid
server := newServer(cfg)
log.Fatal(server.ListenAndServe())
}import os
import sys
from dataclasses import dataclass
REQUIRED = [
"DATABASE_URL",
"OPENAI_API_KEY",
"JWT_SECRET",
"RESEND_API_KEY",
]
@dataclass(frozen=True)
class Config:
database_url: str
openai_key: str
jwt_secret: str
resend_api_key: str
def must_load() -> Config:
"""Exit if any required variable is missing.
Call this once at startup, before the server binds a port.
"""
missing = [key for key in REQUIRED if not os.getenv(key)]
if missing:
# Crash immediately, loud and clear
sys.exit(f"[FATAL] missing required env vars: {', '.join(missing)}")
return Config(
database_url=os.environ["DATABASE_URL"],
openai_key=os.environ["OPENAI_API_KEY"],
jwt_secret=os.environ["JWT_SECRET"],
resend_api_key=os.environ["RESEND_API_KEY"],
)
# main.py
config = must_load() # exits here if config invalidconst REQUIRED = [
'DATABASE_URL',
'OPENAI_API_KEY',
'JWT_SECRET',
'RESEND_API_KEY',
]
// Exits if any required variable is missing.
// Call this once at startup, before app.listen().
export function mustLoad() {
const missing = REQUIRED.filter((key) => !process.env[key])
if (missing.length > 0) {
// Crash immediately, loud and clear
console.error(`[FATAL] missing required env vars: ${missing.join(', ')}`)
process.exit(1)
}
return {
databaseUrl: process.env.DATABASE_URL,
openaiKey: process.env.OPENAI_API_KEY,
jwtSecret: process.env.JWT_SECRET,
resendApiKey: process.env.RESEND_API_KEY,
}
}
// server.js
const config = mustLoad() // exits here if config invalid
app.listen(8080)const REQUIRED = ['DATABASE_URL', 'OPENAI_API_KEY', 'JWT_SECRET', 'RESEND_API_KEY'] as const
type RequiredVar = (typeof REQUIRED)[number]
// The return type is the real payoff: every field is `string`, never
// `string | undefined`. Because this function has already proved the
// variable is present, nothing downstream needs a second null check,
// and `process.env.X!` never has to appear anywhere else in the codebase.
export type Config = Record<RequiredVar, string>
export function mustLoad(): Config {
const missing = REQUIRED.filter((key) => !process.env[key])
if (missing.length > 0) {
console.error(`[FATAL] missing required env vars: ${missing.join(', ')}`)
process.exit(1) // crash immediately, loud and clear
}
return Object.fromEntries(
REQUIRED.map((key) => [key, process.env[key] as string]),
) as Config
}
export const config = mustLoad() // exits at import if config invalid@Component
public class Config {
private static final List<String> REQUIRED = List.of(
"DATABASE_URL",
"OPENAI_API_KEY",
"JWT_SECRET",
"RESEND_API_KEY");
// @PostConstruct runs while the context is starting, so a missing
// variable fails the application BEFORE the port is ever bound. That
// is the whole point of failing fast: an app that refused to start is
// obvious, an app that started half-configured is not.
@PostConstruct
void validate() {
List<String> missing = REQUIRED.stream()
.filter(key -> System.getenv(key) == null || System.getenv(key).isBlank())
.toList();
if (!missing.isEmpty()) {
// Crash immediately, loud and clear
throw new IllegalStateException(
"[FATAL] missing required env vars: " + String.join(", ", missing));
}
}
}
// Spring's own spelling of the same guarantee is a placeholder with no
// default, which fails startup and names the property:
// app.database-url: ${DATABASE_URL}
// See chapter 14 for the fuller version with per-field validation.08
Proactive Error Detection: Health Checks
“The best error handling starts before the error happens.”
Health checks continuously verify that your system is working, not just that it is running. There is a critical difference:
What to Check
- Database: run a lightweight representative query. Track query time; if it jumps from 50ms to 4s, something is wrong before users notice.
- External services: payment processors: run periodic test transactions; email: send to an internal address; auth: generate and validate a test token.
- Configuration: verify all required env vars are loaded and non-empty at startup.
- Cache warmup: ensure critical caches (session store, product catalogue) are populated before serving traffic.
Deep Health Check Endpoint
ping every dependency, report per-check status, answer 503 when degraded
type HealthStatus struct {
Status string `json:"status"`
Checks map[string]string `json:"checks"`
}
func healthHandler(db *pgxpool.Pool, rdb *redis.Client) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
checks := map[string]string{}
overall := "ok"
// DB check
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()
if err := db.Ping(ctx); err != nil {
checks["database"] = "unhealthy: " + err.Error()
overall = "degraded"
} else {
checks["database"] = "ok"
}
// Redis check
if err := rdb.Ping(r.Context()).Err(); err != nil {
checks["cache"] = "unhealthy: " + err.Error()
overall = "degraded"
} else {
checks["cache"] = "ok"
}
status := http.StatusOK
if overall != "ok" { status = http.StatusServiceUnavailable }
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
json.NewEncoder(w).Encode(HealthStatus{Status: overall, Checks: checks})
}
}import asyncio
from fastapi import Response
from sqlalchemy import text
@app.get("/ready")
async def health(response: Response):
checks: dict[str, str] = {}
overall = "ok"
# DB check, with its own timeout
try:
await asyncio.wait_for(db.execute(text("SELECT 1")), timeout=2.0)
checks["database"] = "ok"
except Exception as e:
checks["database"] = f"unhealthy: {e}"
overall = "degraded"
# Redis check
try:
await redis.ping()
checks["cache"] = "ok"
except Exception as e:
checks["cache"] = f"unhealthy: {e}"
overall = "degraded"
response.status_code = 200 if overall == "ok" else 503
return {"status": overall, "checks": checks}app.get('/ready', async (req, res) => {
const checks = {}
let overall = 'ok'
// DB check, with its OWN timeout. A health check that can hang is
// worse than no health check: the orchestrator's probe times out, the
// pod is killed, and nothing in the logs explains why.
try {
await withTimeout(pool.query('SELECT 1'), 2000)
checks.database = 'ok'
} catch (err) {
checks.database = `unhealthy: ${err.message}`
overall = 'degraded'
}
// Redis check
try {
await redis.ping()
checks.cache = 'ok'
} catch (err) {
checks.cache = `unhealthy: ${err.message}`
overall = 'degraded'
}
res.status(overall === 'ok' ? 200 : 503).json({ status: overall, checks })
})import type { Request, Response } from 'express'
interface HealthStatus {
status: 'ok' | 'degraded'
checks: Record<string, string>
}
// Describing a check as DATA rather than as another try/catch block is
// what keeps this endpoint honest as dependencies are added: one entry
// per dependency, all of them run concurrently, and nobody can add a
// fifth check that forgets its timeout.
const CHECKS: Array<{ name: string; probe: () => Promise<unknown> }> = [
{ name: 'database', probe: () => pool.query('SELECT 1') },
{ name: 'cache', probe: () => redis.ping() },
]
export async function health(_req: Request, res: Response): Promise<void> {
const results = await Promise.all(
CHECKS.map(async ({ name, probe }) => {
try {
await withTimeout(probe(), 2000)
return [name, 'ok'] as const
} catch (err) {
return [name, `unhealthy: ${(err as Error).message}`] as const
}
}),
)
const checks = Object.fromEntries(results)
const overall: HealthStatus['status'] =
results.every(([, v]) => v === 'ok') ? 'ok' : 'degraded'
res.status(overall === 'ok' ? 200 : 503).json({ status: overall, checks })
}// Spring Boot Actuator already exposes /actuator/health, and it already
// aggregates a DataSource and a Redis check with no code at all:
// management.endpoint.health.show-details: always
// management.endpoint.health.probes.enabled: true
// Anything it does not know about is one bean each.
@Component
public class CacheHealthIndicator implements HealthIndicator {
private final StringRedisTemplate redis;
CacheHealthIndicator(StringRedisTemplate redis) {
this.redis = redis;
}
@Override
public Health health() {
try {
redis.getConnectionFactory().getConnection().ping();
return Health.up().build();
} catch (Exception e) {
// One DOWN makes the aggregate DOWN, and /actuator/health then
// answers 503, which is what the orchestrator actually watches.
return Health.down(e).withDetail("cache", "unhealthy").build();
}
}
}
// Liveness and readiness are separate GROUPS, and conflating them is the
// classic mistake: a database outage should fail readiness (stop routing
// traffic here) but NOT liveness, because restarting the pod will not
// bring the database back, it will only add a crash loop to the incident.
// management.endpoint.health.group.readiness.include: db,cache
// management.endpoint.health.group.liveness.include: ping09
Monitoring & Observability
Health checks tell you something is broken right now. Monitoring tells you something is about to break, and gives you the context to understand why something broke after the fact.
What to Track
| Category | Metrics to Monitor | Why |
|---|---|---|
| HTTP Layer | 4xx / 5xx rate, p50/p95/p99 latency | Surface user-facing issues immediately |
| Database | Query duration, connection pool usage, deadlock count | Detect slow queries before timeout |
| External Services | Call success rate, latency, 429 count | Know when a dependency is degrading |
| Business Metrics | Successful transactions/min, failed payments, sign-up rate | Catch logic errors invisible to error rates |
| Infrastructure | CPU, memory, disk I/O, network throughput | Resource exhaustion precedes crashes |
Structured Logging (JSON)
Plain-text logs are hard to query at scale. Use structured JSON logs so log aggregation tools (Grafana Loki, Datadog, ELK) can parse, filter, and alert on them programmatically.
one queryable object per event, never an interpolated sentence
// Go's standard library has had structured logging since 1.21: log/slog.
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
// Good, structured, queryable, no sensitive data
logger.Error("payment_failed",
"user_id", userID,
"order_id", orderID,
"amount_cents", 4999,
"provider", "stripe",
"error_code", "card_declined",
"correlation_id", correlationID,
)
// Bad: one opaque sentence. No aggregator can filter on "all declined
// cards for this provider", because there are no fields to filter on.
logger.Error(fmt.Sprintf("payment failed for %s on order %s", userID, orderID))import structlog
log = structlog.get_logger()
# Good, structured, queryable, no sensitive data
log.error(
"payment_failed",
user_id="u_9a3f", # ID, not email
correlation_id="req_abc123",
provider="stripe",
error_code="card_declined",
amount_cents=4999,
)
# BAD, never log PII or secrets
# log.error("payment_failed", email="alice@example.com", card="4242...")import pino from 'pino'
const log = pino({
// Redaction belongs HERE, not at the call sites. One list, applied to
// every log line, is the only version of this that survives contact
// with a growing codebase.
redact: ['*.password', '*.card', 'req.headers.authorization'],
})
// Good, structured, queryable, no sensitive data.
// pino's convention: the object first, the message second.
log.error({
user_id: 'u_9a3f', // ID, not email
correlation_id: 'req_abc123',
provider: 'stripe',
error_code: 'card_declined',
amount_cents: 4999,
}, 'payment_failed')
// BAD, never log PII or secrets
// log.error({ email: 'alice@example.com', card: '4242...' }, 'payment_failed')import pino from 'pino'
// Naming the event shape is worth it for exactly one reason: the field
// names become a contract. A dashboard querying `error_code` keeps
// working because a rename now breaks the build rather than silently
// emitting a field nothing is watching.
interface PaymentFailed {
user_id: string
correlation_id: string
provider: 'stripe' | 'adyen'
error_code: string
amount_cents: number
}
const log = pino({ redact: ['*.password', '*.card', 'req.headers.authorization'] })
export function logPaymentFailed(event: PaymentFailed): void {
log.error(event, 'payment_failed')
}
// And because `email` is not a field of PaymentFailed, the PII mistake
// below does not compile:
// logPaymentFailed({ ...event, email: user.email })private static final Logger log = LoggerFactory.getLogger(PaymentService.class);
// SLF4J's message is a template, not a formatted string. The two look
// alike and are not: the placeholders are only filled in if the level is
// enabled, and structured appenders can keep the arguments separate.
log.atError()
.setMessage("payment_failed")
.addKeyValue("user_id", "u_9a3f") // ID, not email
.addKeyValue("correlation_id", "req_abc123")
.addKeyValue("provider", "stripe")
.addKeyValue("error_code", "card_declined")
.addKeyValue("amount_cents", 4999)
.log();
// With net.logstash.logback.encoder.LogstashEncoder, every addKeyValue
// becomes a top-level JSON field that Loki or ELK can filter on.
// BAD, an interpolated sentence: no fields, nothing to query, and the
// string is built even when ERROR is disabled.
// log.error("payment failed for " + user.getEmail() + " on " + orderId);10
Recovery Strategies
Recoverable vs Non-Recoverable
Recoverable Errors
- Transient network glitch to email API
- Database connection pool temporarily exhausted
- Rate limit 429 from external service
Strategy: Retry with exponential backoff. Queue the work. Don’t give up immediately.
Non-Recoverable Errors
- Redis cluster completely down
- Payment processor offline for hours
- Corrupt data in the DB
Strategy: Graceful degradation. Fallback. Disable the feature. Protect core functionality.
Exponential Backoff
double the wait each attempt, add jitter, and never retry a permanent failure
func sendEmailWithRetry(to, subject, body string) error {
maxRetries := 5
baseDelay := 1 * time.Second
for attempt := 0; attempt < maxRetries; attempt++ {
err := emailClient.Send(to, subject, body)
if err == nil {
return nil // success
}
if !isRetryable(err) {
return fmt.Errorf("permanent failure: %w", err)
}
// Exponential backoff: 1s, 2s, 4s, 8s, 16s
wait := baseDelay * time.Duration(1<<attempt)
// Add jitter (+/-20%) to prevent thundering herd
jitter := time.Duration(rand.Int63n(int64(wait / 5)))
time.Sleep(wait + jitter)
log.Warn("email send failed, retrying",
"attempt", attempt+1,
"wait_ms", wait.Milliseconds(),
"error", err)
}
return fmt.Errorf("all %d retries exhausted", maxRetries)
}
func isRetryable(err error) bool {
// Retry on 429, 503, network errors; not on 400, 401, 422
var httpErr *HTTPError
if errors.As(err, &httpErr) {
return httpErr.StatusCode == 429 || httpErr.StatusCode >= 500
}
return true // network errors are always retryable
}import random
import time
def send_email_with_retry(to: str, subject: str, body: str) -> None:
max_retries = 5
base_delay = 1.0 # seconds
for attempt in range(max_retries):
try:
email_client.send(to, subject, body)
return # success
except Exception as err:
if not is_retryable(err):
raise RuntimeError("permanent failure") from err
# Exponential backoff: 1s, 2s, 4s, 8s, 16s
wait = base_delay * (2 ** attempt)
# Add jitter (+/-20%) to prevent thundering herd
jitter = random.uniform(0, wait / 5)
time.sleep(wait + jitter)
log.warning(
"email_send_failed_retrying",
attempt=attempt + 1,
wait_ms=int(wait * 1000),
error=str(err),
)
raise RuntimeError(f"all {max_retries} retries exhausted")
def is_retryable(err: Exception) -> bool:
# Retry on 429, 503, network errors; not on 400, 401, 422
if isinstance(err, HTTPError):
return err.status_code == 429 or err.status_code >= 500
return True # network errors are always retryableconst sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
export async function sendEmailWithRetry(to, subject, body) {
const maxRetries = 5
const baseDelay = 1000 // ms
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
await emailClient.send(to, subject, body)
return // success
} catch (err) {
if (!isRetryable(err)) {
throw new Error('permanent failure', { cause: err })
}
// Exponential backoff: 1s, 2s, 4s, 8s, 16s
const wait = baseDelay * 2 ** attempt
// Add jitter (+/-20%) to prevent thundering herd
const jitter = Math.random() * (wait / 5)
await sleep(wait + jitter)
log.warn({ attempt: attempt + 1, wait_ms: wait, err },
'email send failed, retrying')
}
}
throw new Error(`all ${maxRetries} retries exhausted`)
}
function isRetryable(err) {
// Retry on 429, 503, network errors; not on 400, 401, 422
if (err.statusCode) return err.statusCode === 429 || err.statusCode >= 500
return true // network errors are always retryable
}// Generic, because "retry with backoff" is never needed exactly once.
// The type parameter keeps the wrapped call's return value intact, so
// adding retries to a call site changes nothing about how it is used.
export async function withRetry<T>(
work: () => Promise<T>,
opts: { maxRetries?: number; baseDelayMs?: number; signal?: AbortSignal } = {},
): Promise<T> {
const { maxRetries = 5, baseDelayMs = 1000, signal } = opts
for (let attempt = 0; attempt < maxRetries; attempt++) {
// The AbortSignal matters more than it looks. Without it, a client
// that hung up three seconds ago still costs you thirty-one seconds
// of retries against an API you are paying per call for.
signal?.throwIfAborted()
try {
return await work()
} catch (err) {
if (!isRetryable(err)) {
throw new Error('permanent failure', { cause: err })
}
const wait = baseDelayMs * 2 ** attempt // 1s, 2s, 4s, 8s, 16s
const jitter = Math.random() * (wait / 5) // +/-20%, no thundering herd
await sleep(wait + jitter, signal)
log.warn({ attempt: attempt + 1, wait_ms: wait, err }, 'retrying')
}
}
throw new Error(`all ${maxRetries} retries exhausted`)
}
function isRetryable(err: unknown): boolean {
// Retry on 429, 503, network errors; not on 400, 401, 422
if (err instanceof HTTPError) {
return err.statusCode === 429 || err.statusCode >= 500
}
return true // network errors are always retryable
}
// Usage: the result is still a Message, cache or no retries.
const sent: Message = await withRetry(() => emailClient.send(to, subject, body))// Spring Retry collapses the whole loop into an annotation, and its
// defaults are exactly the ones above: multiplier 2, plus a random
// factor that supplies the jitter.
@Service
public class EmailService {
@Retryable(
retryFor = RetryableException.class, // 429, 5xx, network
noRetryFor = PermanentException.class, // 400, 401, 422: never retry
maxAttempts = 5,
backoff = @Backoff(delay = 1000, multiplier = 2, random = true))
public void sendEmail(String to, String subject, String body) {
emailClient.send(to, subject, body); // 1s, 2s, 4s, 8s, 16s (+jitter)
}
// Called once every attempt is exhausted. Without a @Recover method
// the final exception simply propagates, which is often right, but
// that should be a decision rather than something you discover.
@Recover
void recover(RetryableException e, String to, String subject, String body) {
log.error("all retries exhausted to={}", to, e);
deadLetter.park(to, subject, body);
}
}
// Resilience4j is the other standard choice, and its advantage is that
// the retry composes with a circuit breaker: a provider that is fully
// down stops being called at all, instead of being retried five times
// by every single request that arrives.Automatic vs Manual Recovery
- Automatic: restart crashed processes (systemd, Kubernetes restart policy), clean up corrupted caches, switch to backup systems. Design carefully, automatic recovery can sometimes amplify a problem.
- Manual: data corruption, payment discrepancies, security incidents. These require human judgment. Document the runbook. Test it. Know who is on-call.
11
Global Error Handler: The Final Safety Net
The global error handler is a single middleware that sits at the outermost layer of your application, catches every error that bubbles up from any layer, and converts it into a properly formatted HTTP response.
Two Major Advantages
- No forgotten error conditions: every unhandled error falls through to the global handler’s default case (
500 + "something went wrong"). Nothing silently swallowed. - Zero redundancy: database error handling logic lives in one file, not scattered across 40 repository methods. Change the unique-violation message once, it applies everywhere.
12
Global Error Handler Implementation
The shape is the same in every language and splits in two: a typed application error that carries the status code and a safe, user-facing message, and a single place at the edge that turns anything thrown or returned into that envelope. Go has no exceptions, so its errors travel as return values; the others let an exception fly and catch it at the boundary. Either way, exactly one place decides what the client sees.
The typed application error
a status code, a safe message, and the original error kept for logs only
package apperr
import "net/http"
// AppError is the canonical error type for this application.
type AppError struct {
Code int // HTTP status code
Message string // Safe, user-facing message
Details any // Optional: field-level errors for 400s
Err error // Original error, for logging only, NEVER sent to client
}
func (e *AppError) Error() string { return e.Message }
// Constructors
func NotFound(resource string) *AppError {
return &AppError{Code: http.StatusNotFound, Message: resource + " not found"}
}
func Conflict(msg string) *AppError {
return &AppError{Code: http.StatusConflict, Message: msg}
}
func BadRequest(msg string, details any) *AppError {
return &AppError{Code: http.StatusBadRequest, Message: msg, Details: details}
}
func Internal(err error) *AppError {
return &AppError{
Code: http.StatusInternalServerError,
Message: "something went wrong", // NEVER expose err.Error() here
Err: err,
}
}class AppError(Exception):
"""The canonical error type for this application."""
def __init__(self, status: int, message: str, details=None, cause=None):
super().__init__(message)
self.status = status # HTTP status code
self.message = message # Safe, user-facing message
self.details = details # Optional: field-level errors for 400s
self.cause = cause # Original error, for logging only
class NotFoundError(AppError):
def __init__(self, resource: str):
super().__init__(404, f"{resource} not found")
class ConflictError(AppError):
def __init__(self, msg: str):
super().__init__(409, msg)
class BadRequestError(AppError):
def __init__(self, msg: str, details=None):
super().__init__(400, msg, details)
class InternalError(AppError):
def __init__(self, cause: Exception):
# NEVER put str(cause) in `message`, it reaches the client
super().__init__(500, "something went wrong", cause=cause)// The canonical error type for this application.
export class AppError extends Error {
constructor(code, message, details, cause) {
super(message, { cause }) // `cause` is for logging only, NEVER sent out
this.name = 'AppError'
this.code = code // HTTP status code
this.details = details // Optional: field-level errors for 400s
}
}
// Constructors
export const notFound = (resource) => new AppError(404, `${resource} not found`)
export const conflict = (msg) => new AppError(409, msg)
export const badRequest = (msg, details) => new AppError(400, msg, details)
export const internal = (cause) =>
new AppError(500, 'something went wrong', undefined, cause)
// ^ NEVER cause.message here: it leaks the stack, the
// SQL, and often the schema straight to the client.interface FieldError {
field: string
issue: string
}
export class AppError extends Error {
constructor(
readonly code: number, // HTTP status code
override readonly message: string, // safe, user-facing
readonly details?: FieldError[], // field-level errors for 400s
override readonly cause?: unknown, // original error, LOGGING ONLY
) {
super(message, { cause })
this.name = 'AppError'
}
}
// The separation the types are enforcing is the security one from sec 14:
// `message` and `details` are the only fields the handler is allowed to
// serialise, and `cause` is deliberately typed `unknown` so nobody can
// casually interpolate it into a response without narrowing it first.
export const notFound = (resource: string): AppError =>
new AppError(404, `${resource} not found`)
export const conflict = (msg: string): AppError => new AppError(409, msg)
export const badRequest = (msg: string, details?: FieldError[]): AppError =>
new AppError(400, msg, details)
export const internal = (cause: unknown): AppError =>
new AppError(500, 'something went wrong', undefined, cause)// The canonical error type for this application.
public class AppException extends RuntimeException {
private final int code; // HTTP status code
private final Object details; // Optional: field-level errors for 400s
public AppException(int code, String message, Object details, Throwable cause) {
super(message, cause); // cause is for logging only
this.code = code;
this.details = details;
}
public int code() { return code; }
public Object details() { return details; }
// Factories, so a status code is never typed as a bare int at a call site
public static AppException notFound(String resource) {
return new AppException(404, resource + " not found", null, null);
}
public static AppException conflict(String msg) {
return new AppException(409, msg, null, null);
}
public static AppException badRequest(String msg, Object details) {
return new AppException(400, msg, details, null);
}
public static AppException internal(Throwable cause) {
// NEVER cause.getMessage() as the message: it reaches the client
return new AppException(500, "something went wrong", null, cause);
}
}
// Extending RuntimeException, not Exception, is deliberate: a checked
// exception would force every layer in between to declare or wrap it,
// which is exactly the ceremony that makes people swallow errors.The handler at the edge
one place that maps any failure to the envelope, and logs the rest
package middleware
import (
"encoding/json"
"errors"
"log/slog"
"net/http"
"github.com/jackc/pgx/v5/pgconn"
apperr "yourapp/errors"
)
type ErrorResponse struct {
Code int `json:"code"`
Message string `json:"message"`
Details any `json:"details,omitempty"`
}
// GlobalErrorHandler wraps a handler that returns an error.
func GlobalErrorHandler(next func(http.ResponseWriter, *http.Request) error) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
err := next(w, r)
if err == nil { return }
var appErr *apperr.AppError
switch {
// Already wrapped as AppError
case errors.As(err, &appErr):
if appErr.Err != nil {
slog.Error("app error", "err", appErr.Err)
}
// Postgres unique constraint violation -> 409
case isPgError(err, "23505"):
appErr = apperr.Conflict("resource already exists")
// Postgres foreign key violation -> 404
case isPgError(err, "23503"):
appErr = apperr.NotFound("referenced resource")
// pgx no-rows -> 404
case errors.Is(err, pgx.ErrNoRows):
appErr = apperr.NotFound("resource")
// Everything else -> 500 (never leak internal error)
default:
slog.Error("unhandled error", "err", err)
appErr = apperr.Internal(err)
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(appErr.Code)
json.NewEncoder(w).Encode(ErrorResponse{
Code: appErr.Code,
Message: appErr.Message,
Details: appErr.Details,
})
}
}
func isPgError(err error, code string) bool {
var pgErr *pgconn.PgError
return errors.As(err, &pgErr) && pgErr.Code == code
}from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from psycopg2 import errors as pg_errors
import logging
app = FastAPI()
logger = logging.getLogger("app")
# --- Global exception handlers ---
@app.exception_handler(AppError)
async def app_error_handler(request: Request, exc: AppError):
return JSONResponse(
status_code=exc.status,
content={"code": exc.status, "message": exc.message, "details": exc.details}
)
@app.exception_handler(pg_errors.UniqueViolation)
async def unique_violation_handler(request: Request, exc):
logger.warning("unique_violation", extra={"path": request.url.path})
return JSONResponse(status_code=409,
content={"code": 409, "message": "resource already exists"})
@app.exception_handler(pg_errors.ForeignKeyViolation)
async def fk_violation_handler(request: Request, exc):
return JSONResponse(status_code=404,
content={"code": 404, "message": "referenced resource not found"})
@app.exception_handler(Exception)
async def unhandled_error_handler(request: Request, exc: Exception):
# Log the real error internally, never expose it
logger.error("unhandled_exception", exc_info=exc,
extra={"path": request.url.path})
return JSONResponse(status_code=500,
content={"code": 500, "message": "something went wrong"})import { AppError, conflict, notFound, internal } from '../errors.js'
// Express identifies an error handler by its FOUR parameters. Drop `next`
// and it silently becomes an ordinary middleware that is never called on
// an error, which is the classic way this safety net stops working.
export function globalErrorHandler(err, req, res, next) {
let appErr
if (err instanceof AppError) {
// Already wrapped
if (err.cause) log.error({ err: err.cause }, 'app error')
appErr = err
} else if (err.code === '23505') {
// Postgres unique constraint violation -> 409
appErr = conflict('resource already exists')
} else if (err.code === '23503') {
// Postgres foreign key violation -> 404
appErr = notFound('referenced resource')
} else {
// Everything else -> 500 (never leak internal error)
log.error({ err, path: req.path }, 'unhandled error')
appErr = internal(err)
}
res.status(appErr.code).json({
code: appErr.code,
message: appErr.message,
...(appErr.details && { details: appErr.details }),
})
}
// Registered LAST, after every route, because Express runs the chain in
// order and an error handler can only catch what is upstream of it.
app.use(globalErrorHandler)
// An async handler that rejects does NOT reach here on Express 4; wrap
// it, or use Express 5, which forwards rejected promises for you.import type { NextFunction, Request, Response } from 'express'
import { AppError, conflict, notFound, internal } from '../errors'
// Postgres surfaces its errors as a `code` string on an untyped object,
// so narrowing it is the honest first step rather than a cast.
function pgCode(err: unknown): string | undefined {
return typeof err === 'object' && err !== null && 'code' in err
? String((err as { code: unknown }).code)
: undefined
}
export function globalErrorHandler(
err: unknown, // `unknown`, not `Error`: anything at all can be thrown
req: Request,
res: Response,
_next: NextFunction,
): void {
let appErr: AppError
if (err instanceof AppError) {
if (err.cause) log.error({ err: err.cause }, 'app error')
appErr = err
} else {
switch (pgCode(err)) {
case '23505': appErr = conflict('resource already exists'); break // unique violation
case '23503': appErr = notFound('referenced resource'); break // FK violation
default:
log.error({ err, path: req.path }, 'unhandled error')
appErr = internal(err) // never leak the original message
}
}
res.status(appErr.code).json({
code: appErr.code,
message: appErr.message,
...(appErr.details && { details: appErr.details }),
})
}// @RestControllerAdvice IS the global handler: one class, applied to
// every controller, with a method per exception type. Spring picks the
// most specific match, so the Exception method is the final net.
@RestControllerAdvice
public class GlobalExceptionHandler {
private static final Logger log = LoggerFactory.getLogger(GlobalExceptionHandler.class);
record ErrorResponse(int code, String message, Object details) {}
// Already wrapped as AppException
@ExceptionHandler(AppException.class)
ResponseEntity<ErrorResponse> onApp(AppException e) {
if (e.getCause() != null) {
log.error("app error", e.getCause()); // the real error, internally
}
return ResponseEntity.status(e.code())
.body(new ErrorResponse(e.code(), e.getMessage(), e.details()));
}
// Postgres unique constraint violation -> 409
@ExceptionHandler(DuplicateKeyException.class)
ResponseEntity<ErrorResponse> onDuplicate() {
return ResponseEntity.status(409)
.body(new ErrorResponse(409, "resource already exists", null));
}
// Foreign key / referenced row missing -> 404
@ExceptionHandler(DataIntegrityViolationException.class)
ResponseEntity<ErrorResponse> onIntegrity() {
return ResponseEntity.status(404)
.body(new ErrorResponse(404, "referenced resource not found", null));
}
// Everything else -> 500, and the real error goes to the log ONLY.
@ExceptionHandler(Exception.class)
ResponseEntity<ErrorResponse> onUnhandled(Exception e, HttpServletRequest req) {
log.error("unhandled error path={}", req.getRequestURI(), e);
return ResponseEntity.status(500)
.body(new ErrorResponse(500, "something went wrong", null));
}
}
// Turn OFF the default error attributes, or Spring's own /error page can
// hand back the exception message and a stack trace, undoing all of this:
// server.error.include-message: never
// server.error.include-stacktrace: never
// server.error.include-binding-errors: never14
Security: What to Expose, What to Hide
(1) Never Leak Internal Details
Database error messages from Postgres contain table names, column names, index names, and constraint names. If you forward a raw pgconn.PgError message directly to the client, an attacker learns your schema and can craft more targeted SQL injection attempts.
| What You Got | What to Send to Client |
|---|---|
duplicate key value violates unique constraint "users_email_key" | "Email already in use" |
relation "usres" does not exist (typo) | "Something went wrong" |
deadlock detected on relation 42816 | "Something went wrong, please retry" |
stack trace: panic at server.go:142 | "Internal server error" |
(2) Vague Auth Errors (On Purpose)
Login endpoints are the most attacked surface in any application. If you return specific messages like “no user with this email exists” vs “password is incorrect”, an attacker can enumerate valid emails through a simple loop.
(3) Safe Logging Practices
Logs are often shipped to third-party aggregation services (Datadog, Grafana Cloud, ELK). In major data breaches, leaked log files exposed millions of records, because engineers had carelessly logged sensitive fields.
- Never log: passwords, API keys, credit card numbers, SSNs, full email addresses, session tokens.
- Log instead: user ID (not email), correlation/request ID, operation name, error code.
- Use a log scrubbing library (Go:
slogwith a custom handler; Python:structlogprocessors; Node: pino’sredact; Java: a LogbackRegexReplacement) to automatically redact known sensitive fields.
log the identifier, never the identity
// UNSAFE, never do this
slog.Error("login_failed",
"email", user.Email, // PII leak
"password", req.Password, // catastrophic
"api_key", cfg.OpenAIKey, // secret leak
)
// SAFE, IDs and correlation only
slog.Error("login_failed",
"user_id", user.ID,
"correlation_id", r.Header.Get("X-Request-ID"),
"reason", "invalid_credentials", // generic code, not DB message
)# UNSAFE, never do this
log.error("login_failed",
email=user.email, # PII leak
password=req.password, # catastrophic
api_key=cfg.openai_key, # secret leak
)
# SAFE, IDs and correlation only
log.error("login_failed",
user_id=user.id,
correlation_id=request.headers.get("X-Request-ID"),
reason="invalid_credentials", # generic code, not DB message
)
# Better still: make the leak impossible rather than remembering not to
# do it. A structlog processor scrubs known-sensitive keys on every
# event, including the ones a future call site adds carelessly.
SENSITIVE = {"password", "api_key", "token", "card", "email"}
def scrub(logger, method_name, event_dict):
for key in event_dict:
if key.lower() in SENSITIVE:
event_dict[key] = "[REDACTED]"
return event_dict
structlog.configure(processors=[scrub, ...])// UNSAFE, never do this
log.error({
email: user.email, // PII leak
password: req.body.password, // catastrophic
api_key: config.openaiKey, // secret leak
}, 'login_failed')
// SAFE, IDs and correlation only
log.error({
user_id: user.id,
correlation_id: req.get('X-Request-ID'),
reason: 'invalid_credentials', // generic code, not DB message
}, 'login_failed')
// Best: configure redaction ONCE, so the unsafe version above is
// neutralised even when somebody writes it by accident.
const log = pino({
redact: {
paths: ['*.password', '*.api_key', '*.token', '*.card', 'req.headers.authorization'],
censor: '[REDACTED]',
},
})
// Note what redaction does NOT cover: `log.error(user)` where the whole
// object is logged, or an error whose MESSAGE contains the token. The
// list is a safety net, not a substitute for choosing the fields.// Redaction is a runtime safety net. TypeScript can add a compile-time
// one on top, which catches the mistake before it ever ships.
type Sensitive = 'password' | 'email' | 'api_key' | 'token' | 'card'
// A log payload that simply cannot carry a sensitive key: the mapped
// type sets each of them to `never`, so supplying one fails to compile.
type SafeLog = Record<string, unknown> & { [K in Sensitive]?: never }
export function logSafely(event: string, fields: SafeLog): void {
log.error(fields, event)
}
// SAFE, IDs and correlation only
logSafely('login_failed', {
user_id: user.id,
correlation_id: req.get('X-Request-ID'),
reason: 'invalid_credentials', // generic code, not DB message
})
// UNSAFE, and this one is now a BUILD failure rather than a breach:
// logSafely('login_failed', { email: user.email, password: req.body.password })// UNSAFE, never do this
log.error("login_failed email={} password={} api_key={}",
user.email(), req.password(), config.openAiKey());
// SAFE, IDs and correlation only
log.atError()
.setMessage("login_failed")
.addKeyValue("user_id", user.id())
.addKeyValue("correlation_id", req.getHeader("X-Request-ID"))
.addKeyValue("reason", "invalid_credentials") // generic code, not DB message
.log();
// Java's structural guard is `toString()`: a record prints ALL of its
// components, so logging a User record logs the password field with it.
// Override it once, at the type, and every log site is safe by default.
public record User(String id, String email, String passwordHash) {
@Override
public String toString() {
return "User[id=" + id + "]"; // no email, no hash, ever
}
}
// Logback can also mask at the appender with a RegexReplacement rule,
// which catches secrets embedded in exception messages, the case no
// field-level approach can reach.15
References & Further Reading
OWASP Error Handling Cheat Sheet OWASP Authentication Cheat Sheet Go errors package Go slog (structured logging) FastAPI Error Handling structlog (Python) MDN, HTTP Status Codes AWS: Backoff with Jitter Grafana Loki
Backend Field Manual / Error Handling & Fault Tolerance / Chapter 16
Backend from First Principles / Chapter 12 / Errors. Code targets Go 1.22+, Python 3.11+, Node.js 20+, TypeScript 5+ and Java 17+ (Spring Boot).