A detailed backend reference
Logging, Monitoring
& Observability
These are not features, they are practices. A production backend without them is a black box. With them, you can determine the internal state of your entire system just by looking at what it emits. That is the definition of an observable system.
01
Why Observability Matters
Modern backend applications run in distributed environments: multiple services, multiple servers, multiple regions, users spread across the globe. Something will go wrong. The question is not if, but when, and more importantly, how fast can you find out and fix it?
Without logging, monitoring, and observability, when an incident happens you are completely blind. You know something is wrong (users are complaining, conversion dropped), but you have no idea what, where, or why. Every minute of debugging in the dark is revenue lost and user trust eroded.
These practices also require a collective effort. As a developer, you instrument your code. DevOps / SRE teams configure the infrastructure that collects, stores, and displays that data. Neither side alone is sufficient.
02
The Three Pillars of Observability
Observability theory defines three foundational pillars. A system is only fully observable when all three are in place. Each pillar answers a different question about your system’s behaviour.
LOGS
What happened?
Timestamped records of discrete events in your application, a request came in, a database query ran, an error occurred. Think of logs as your backend’s diary.
METRICS
What are the patterns?
Numerical measurements aggregated over time, request rate, error rate, latency p99, CPU usage. Metrics tell you trends, not individual events.
TRACES
Where did it go?
A trace follows a single request through every component it touched, handler -> service -> DB -> external API. Shows component interactions and latency per step.
Logs -> Answer
“User 42 tried to log in at 14:32:01 and failed, invalid password.”
Metrics -> Answer
“In the last 5 minutes, 800 requests/sec, 12% error rate, p99 latency 340ms.”
Traces -> Answer
“This request spent 240ms in the DB query layer and 80ms calling the email provider, that’s where the latency is.”
03
Monitoring vs Observability: The Difference
These terms are often used interchangeably but they are distinct concepts at different levels of capability.
| Dimension | Monitoring | Observability |
|---|---|---|
| Core question | “Is something wrong?” | “What exactly is wrong and why?” |
| Approach | Pre-defined dashboards and alerts on known metrics | Ad-hoc investigation using logs + metrics + traces together |
| Scope | Tracks metrics you decided to measure in advance | Can answer questions you didn’t know to ask when you set it up |
| Output | “CPU is at 95%” -> alert fires | “CPU is 95% because function X is running an N+1 DB query for user segment Y” |
| Historical analogy | Dashboard on a car: speed, fuel, temp | Full diagnostic system: tells you the exact faulty component and cause |
| What it requires | Metrics collection, alert rules | Logs + Metrics + Traces all correlated and linked |
Key insight: Monitoring tells you there is a problem. Observability tells you exactly what the problem is, the specific function, the specific service, the specific user, the specific query. This is only possible if you’ve properly implemented all three pillars.
04
How All Three Work Together: The Debugging Workflow
In practice, when an incident occurs, you navigate from coarse-grained signals to fine-grained details. Here’s the canonical flow used in production systems:
-
1
Alert fires (Monitoring)
Grafana / New Relic detects error rate > 80% and fires a Slack webhook. You get a message: “API service error rate spiked at 14:32.” This is monitoring doing its job.
-
2
Go to Metrics dashboard
Open Grafana / New Relic. You see concrete numbers: 840 requests/min, 82% returning 5xx, p99 latency jumped from 120ms to 4.2s at 14:31. You now know the scale of the problem and when it started.
-
3
Correlate to Logs
From the metrics view, click through to the associated error logs for that 2-minute window. You see hundreds of logs:
ERROR: connection pool exhausted after 4000ms. Now you know what is failing. -
4
Follow the Trace
Click on one of those error logs. It links to a trace. The trace shows the full request journey:
AuthMiddleware (2ms) -> UserHandler (5ms) -> UserService (3ms) -> DB.GetUser (3990ms <- HERE). The DB query is timing out. Root cause found. -
5
Fix & Confirm with Metrics
You add a missing index to the query. Within minutes, the Grafana dashboard shows error rate dropping back to 0.2%, latency back to 110ms. Monitoring confirms the fix worked.
05
Logging: Deep Dive
Logging is the practice of recording all important events throughout your application’s execution lifecycle. Think of it as a journal your backend keeps, timestamped, detailed, structured entries for every significant thing that happens.
What Should You Log?
A good rule of thumb: anything a developer would want to know when debugging an issue at 2 AM, without access to the live system, without the ability to reproduce the bug, with only the logs to go on.
Business events
User created a to-do. Order was placed. Payment succeeded. Subscription cancelled. These are the events that matter to the business and to auditing.
Security events
Login attempt (success or failure). Password reset. Token revoked. Admin action. These are critical for security audits and intrusion detection.
System events
Server startup / shutdown. Database connection established / lost. Background job started / finished. Config loaded.
Errors & exceptions
Every unhandled error, every caught exception, every failed external API call. Always include: timestamp, user ID, request ID, error type, stack trace.
Performance signals
Slow queries (> 500ms), large payload responses, DB connection pool warnings. These are logs that become metrics over time.
What Metadata to Include in Every Log
A log without context is almost useless. Every log entry should carry enough metadata to answer: who did what, when, where, and with what result?
// A well-structured log entry (JSON, production format)
{
"timestamp": "2024-01-15T14:32:01.234Z",
"level": "error",
"message": "database query failed",
"service": "todo-api",
"environment": "production",
"request_id": "req_01J2KXYZ", // unique per request
"trace_id": "abc123def456", // links to distributed trace
"span_id": "span_789",
"user_id": "usr_01J2K",
"method": "POST",
"path": "/todos",
"error": "pq: deadlock detected",
"duration_ms": 4001,
"host": "worker-node-3"
}
JSON Log Entry
06
Log Levels
Every log entry is assigned a severity level. This lets you filter what you see, in development you want everything, in production you care about warnings and above. Your logging library will let you set a minimum level per environment.
DEBUG
Verbose diagnostic info for development only. Variable values, function entry/exit, loop iterations. Never enable in production, the volume would overwhelm your log pipeline and cost a fortune in storage. Used when troubleshooting a specific issue locally.
INFO
Normal, successful application operations. Server started. User logged in. To-do created. Background job completed. These record the normal flow of business events. This is the default level for production.
WARN
Something unexpected, but not a failure. A user typed the wrong password (not your fault). An API responded slowly but successfully. A deprecated config option is still in use. Something you should investigate eventually but the system is still working.
ERROR
Something failed and needs attention. Database query failed. External API returned 500. Validation failed in a way that should not have happened. A task handler threw an unhandled exception. This is the level you build alerts around.
FATAL
The application cannot continue and is shutting down. Cannot connect to database on startup. Config file missing. Critical dependency unavailable. After logging FATAL, the process terminates. Infrastructure will restart it. Use sparingly, only for truly unrecoverable states.
Environment-Specific Level Configuration
JSON in production, a readable console locally, and a level per environment
// logger/logger.go, Environment-aware log level
package logger
import (
"os"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
)
func getLogLevel() zapcore.Level {
env := os.Getenv("APP_ENV")
switch env {
case "production":
return zapcore.InfoLevel // production: INFO and above
case "staging":
return zapcore.WarnLevel // staging: WARN and above only
default:
return zapcore.DebugLevel // local dev: everything
}
}
func New() *zap.Logger {
env := os.Getenv("APP_ENV")
level := getLogLevel()
var cfg zap.Config
if env == "production" {
// Production: JSON format, parseable by Loki, ELK, New Relic
cfg = zap.NewProductionConfig()
cfg.EncoderConfig.TimeKey = "timestamp"
cfg.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
} else {
// Development: coloured console format, human-readable
cfg = zap.NewDevelopmentConfig()
cfg.EncoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
}
cfg.Level = zap.NewAtomicLevelAt(level)
logger, _ := cfg.Build()
return logger
}# logging_config.py, Environment-aware log level
import logging
import os
import structlog
def get_log_level() -> int:
env = os.getenv("APP_ENV", "development")
if env == "production":
return logging.INFO # production: INFO and above
if env == "staging":
return logging.WARNING # staging: WARN and above only
return logging.DEBUG # local dev: everything
def configure() -> None:
env = os.getenv("APP_ENV", "development")
if env == "production":
# Production: JSON format, parseable by Loki, ELK, New Relic
renderer = structlog.processors.JSONRenderer()
else:
# Development: coloured console format, human-readable
renderer = structlog.dev.ConsoleRenderer(colors=True)
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.stdlib.add_log_level,
renderer,
],
# A FILTERING bound logger drops suppressed levels before the
# message is ever formatted, so a debug log in a hot path costs
# almost nothing in production rather than being built and binned.
wrapper_class=structlog.make_filtering_bound_logger(get_log_level()),
)
logging.basicConfig(level=get_log_level())// logger.js, Environment-aware log level
import pino from 'pino'
const env = process.env.APP_ENV ?? 'development'
function getLogLevel() {
switch (env) {
case 'production': return 'info' // production: INFO and above
case 'staging': return 'warn' // staging: WARN and above only
default: return 'debug' // local dev: everything
}
}
export const log = pino({
level: getLogLevel(),
timestamp: pino.stdTimeFunctions.isoTime,
// Production: raw JSON straight to stdout, which is exactly what Loki,
// ELK and New Relic want. Development: coloured, human-readable.
// pino-pretty is a DEV dependency on purpose, it costs real throughput.
transport:
env === 'production'
? undefined
: { target: 'pino-pretty', options: { colorize: true } },
})import pino, { type Level, type Logger } from 'pino'
type Env = 'development' | 'staging' | 'production'
const env = (process.env.APP_ENV ?? 'development') as Env
// A Record keyed by Env means adding a fourth environment will not
// compile until this table decides what level it logs at. A switch with
// a default would have silently given it the dev level instead.
const LEVELS: Record<Env, Level> = {
production: 'info', // production: INFO and above
staging: 'warn', // staging: WARN and above only
development: 'debug', // local dev: everything
}
export const log: Logger = pino({
level: LEVELS[env],
timestamp: pino.stdTimeFunctions.isoTime,
transport:
env === 'production'
? undefined
: { target: 'pino-pretty', options: { colorize: true } },
})<!-- Java keeps this in configuration rather than code, and Spring's
profile blocks are how the environment picks a shape. -->
<configuration>
<!-- Production: one JSON object per line, parseable by Loki / ELK / NR -->
<springProfile name="production">
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<timeZone>UTC</timeZone>
</encoder>
</appender>
<root level="INFO"><appender-ref ref="JSON"/></root>
</springProfile>
<springProfile name="staging">
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>
<root level="WARN"><appender-ref ref="JSON"/></root>
</springProfile>
<!-- Local dev: coloured console, human-readable -->
<springProfile name="default">
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%highlight(%-5level) %cyan(%logger{20}) - %msg%n</pattern>
</encoder>
</appender>
<root level="DEBUG"><appender-ref ref="CONSOLE"/></root>
</springProfile>
</configuration>
<!-- Because the level lives per LOGGER NAME (the class), you can also
raise one package to DEBUG in production without touching the rest:
<logger name="com.example.todo.repo" level="DEBUG"/> -->07
Structured vs Unstructured Logging
There are two fundamental ways to write a log entry. The choice of which to use depends on your environment.
| Property | Unstructured (Plain Text) | Structured (JSON) |
|---|---|---|
| Format | 2024-01-15 14:32 ERROR: DB query failed for user 42 | {"level":"error","user_id":"42","error":"pq: timeout",...} |
| Human readability | Easy to read in a terminal | Harder to read without a viewer |
| Machine parseability | Hard, requires regex to extract user_id, error message, etc. | Native JSON, any tool can parse instantly |
| Field extraction | Error-prone regex patterns | Direct field access: log.user_id |
| Querying in Loki/ELK | Complex, fragile grep patterns | {user_id="42"} |= "error", trivial |
| Use in | Local development | Staging & Production |
What Structured Logs Look Like Side by Side
// UNSTRUCTURED, development console (human-friendly)
14:32:01 DEBUG Connected to PostgreSQL on localhost:5432
14:32:01 INFO Started background job worker
14:32:01 INFO HTTP server listening on :8080
14:32:14 WARN Slow query detected: 520ms query=SELECT * FROM todos WHERE user_id=$1
14:32:15 ERROR Failed to send email user_id=usr_01J2K error=connection refused
Console (Dev)
// STRUCTURED, production JSON (machine-friendly)
{"timestamp":"2024-01-15T14:32:14Z","level":"warn","msg":"slow query","duration_ms":520,"query":"SELECT * FROM todos WHERE user_id=$1","service":"todo-api","env":"production"}
{"timestamp":"2024-01-15T14:32:15Z","level":"error","msg":"failed to send email","user_id":"usr_01J2K","error":"connection refused","trace_id":"abc123","span_id":"span_001","service":"todo-api","env":"production"}
JSON (Production)
08
Metrics: Deep Dive
Metrics are numerical measurements of your system’s behaviour, aggregated over time. Unlike logs (one entry per event), a metric is a counter or gauge that summarises many events into a single number, e.g. “320 requests in the last 15 seconds”. This makes them extremely efficient for dashboards and alerting.
The Four Core Metric Types (Prometheus Model)
Counter
A value that only ever goes up (and resets on restart). Used for: total requests served, total errors, total tasks processed. Query: rate(http_requests_total[5m]) = requests/sec over last 5 min.
Gauge
A value that can go up or down. Used for: current queue depth, current active DB connections, current goroutine count, memory usage right now.
Histogram
Records observations in configurable buckets. Used for: request latency, response size. Lets you compute percentiles (p50, p95, p99), crucial for SLOs. “95% of requests complete under 200ms.”
Summary
Similar to Histogram but calculates percentiles client-side. Less flexible for aggregation across multiple instances. Prefer Histograms in most cases.
Key Metrics Every Backend Should Expose
| Metric | Type | Why It Matters |
|---|---|---|
http_requests_total | Counter | Request throughput. Labels: method, path, status_code. |
http_request_duration_seconds | Histogram | Latency distribution. p99 tells you worst-case user experience. |
http_errors_total | Counter | Count of 4xx / 5xx responses. Spike = something broken. |
db_query_duration_seconds | Histogram | Slow DB queries. Spikes indicate missing indexes or N+1 problems. |
db_connections_open | Gauge | Pool exhaustion risk. Near max = you need more connections or fewer queries. |
task_queue_depth | Gauge | Backlog in your background task system. Rising = workers need scaling. |
task_processing_duration_seconds | Histogram | How long background tasks take. Useful for SLA tracking. |
go_goroutines / process_resident_memory_bytes | Gauge | Runtime health. Goroutine leak = this keeps climbing. |
Scrape Interval & Delay
Prometheus scrapes your /metrics endpoint at a configured interval (typically 15 seconds). This means there is a built-in delay of ~10-15 seconds between reality and what you see on a dashboard. This is acceptable for most use cases. For near-real-time alerting, use sub-15-second scrape intervals, but be aware this increases load on both your service and Prometheus.
09
Distributed Tracing
A trace is the complete record of a single request’s journey through your system. Each step of that journey is called a span. Spans are linked by a shared trace_id that is generated when the request first enters your system and propagated through every subsequent call.
Trace vs Span: Definitions
Trace
The entire end-to-end journey of a single request. Identified by a globally unique trace_id. Contains all spans for that request.
Span
A single unit of work within a trace, one function call, one DB query, one HTTP call. Has a start time, duration, parent span ID, and arbitrary attributes (user_id, query, etc.).
Root Span
The first span in a trace. Typically the HTTP handler or the entry point of the request. Has no parent.
Context Propagation
The mechanism by which trace_id and span_id are passed from function to function (via Go’s context.Context), and from service to service (via HTTP headers like traceparent).
traceparent header
W3C standard header: traceparent: 00-abc123-span001-01. When your service calls another service, it includes this header so the receiving service can join its spans to the same trace.
10
Instrumentation & OpenTelemetry
Instrumentation is the act of adding code to your application to measure its behaviour, recording spans, emitting metrics, enriching logs with context. You cannot observe what you haven’t instrumented.
OpenTelemetry (OTel) is an open-source, vendor-neutral standard for instrumentation. It provides APIs, SDKs, and an agent (the Collector) for all major languages. The key value: instrument once, send to any backend (Jaeger, Grafana Tempo, New Relic, Datadog, Honeycomb) just by changing configuration, no code changes.
OTel Core Concepts
Tracer
The OTel object that creates spans. Each component (service, library) gets its own named tracer. otel.Tracer("todo-service")
Span
Created by the tracer. Has a name, start/end time, status, and key-value attributes. OTel spans automatically link to parent spans via context.
Context Propagation
OTel automatically injects/extracts traceparent headers in HTTP clients and servers (with the right libraries). Zero manual work for standard HTTP.
OTel Collector
A standalone agent/sidecar that receives telemetry from your app (via OTLP protocol), optionally processes/filters it, and exports to one or more backends. Decouples instrumentation from backend choice.
Auto-instrumentation
OTel provides libraries that auto-instrument popular frameworks (net/http, gin, gRPC, database/sql, redis, etc.) with zero code changes. You get traces for DB queries and HTTP calls for free.
11
Code: Full LMO Implementation
A complete example showing logging, metrics and tracing wired into one service handler, in five languages. This is the pattern from the lecture’s to-do application. The libraries differ, Zap and structlog and pino and Logback, but the three pillars are assembled the same way everywhere: a logger that emits JSON with fixed fields, a middleware that opens a span per request, a service method that logs and traces the business event, and a middleware that counts and times every request.
Logger Setup
structured output with a fixed set of fields on every line
// logger/logger.go
package logger
import (
"os"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
)
var Log *zap.Logger
func Init() {
env := os.Getenv("APP_ENV")
var cfg zap.Config
if env == "production" {
cfg = zap.NewProductionConfig() // JSON output
cfg.Level = zap.NewAtomicLevelAt(zapcore.InfoLevel)
} else {
cfg = zap.NewDevelopmentConfig() // coloured console
cfg.Level = zap.NewAtomicLevelAt(zapcore.DebugLevel)
cfg.EncoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
}
Log, _ = cfg.Build(
// Always include these fields in every log entry
zap.Fields(
zap.String("service", "todo-api"),
zap.String("env", env),
),
)
}# logging_config.py, structlog setup for Python
import logging
import os
import structlog
def configure_logging():
env = os.getenv("APP_ENV", "development")
if env == "production":
# JSON renderer for production, parseable by Loki / ELK
renderer = structlog.processors.JSONRenderer()
level = logging.INFO
else:
# Colourful console for local dev, human-readable
renderer = structlog.dev.ConsoleRenderer(colors=True)
level = logging.DEBUG
structlog.configure(
processors=[
structlog.contextvars.merge_contextvars, # merge request-scoped context
structlog.processors.TimeStamper(fmt="iso"),
structlog.stdlib.add_log_level,
structlog.stdlib.add_logger_name,
structlog.processors.StackInfoRenderer(),
structlog.processors.ExceptionRenderer(), # auto-renders exceptions
renderer,
],
logger_factory=structlog.stdlib.LoggerFactory(),
wrapper_class=structlog.stdlib.BoundLogger,
cache_logger_on_first_use=True,
)
logging.basicConfig(level=level)
# Always include these fields in every log entry
log = structlog.get_logger().bind(
service="todo-api",
env=os.getenv("APP_ENV", "development"),
)// logger.js
import pino from 'pino'
const env = process.env.APP_ENV ?? 'development'
export const log = pino({
level: env === 'production' ? 'info' : 'debug',
// `base` is the fixed field set: these two land on EVERY log entry
base: { service: 'todo-api', env },
timestamp: pino.stdTimeFunctions.isoTime,
transport:
env === 'production'
? undefined // JSON output
: { target: 'pino-pretty', options: { colorize: true } }, // coloured console
// Redaction belongs in the logger, not in the hundred call sites that
// might each forget. See sec 21: never log credentials, ever.
redact: ['req.headers.authorization', '*.password', '*.token'],
})import pino, { type Logger } from 'pino'
const env = process.env.APP_ENV ?? 'development'
export const log: Logger = pino({
level: env === 'production' ? 'info' : 'debug',
base: { service: 'todo-api', env }, // on every log entry
timestamp: pino.stdTimeFunctions.isoTime,
redact: ['req.headers.authorization', '*.password', '*.token'],
transport:
env === 'production'
? undefined
: { target: 'pino-pretty', options: { colorize: true } },
})
// A CHILD logger binds fields once and stamps them on every line it
// emits. This is the cheap way to guarantee the request id from chapter
// 06 reaches every log entry in a request, with no call site passing it
// by hand, and the returned type is still a plain Logger, so nothing
// downstream has to know whether it was handed the root or a child.
export function requestLogger(requestId: string, userId: string): Logger {
return log.child({ request_id: requestId, user_id: userId })
}// In Java the CONFIGURATION lives in logback-spring.xml (see sec 06) and
// the code side is deliberately tiny: one logger per class, obtained by
// class, so the class name becomes the logger name.
@Service
public class TodoService {
// Always static final: it is created once per class, not per instance.
private static final Logger log = LoggerFactory.getLogger(TodoService.class);
// The fixed fields ("service", "env") are set once in the encoder:
// <customFields>{"service":"todo-api","env":"production"}</customFields>
void example(String userId, String requestId) {
// MDC is the per-request context from chapter 06. Anything put
// here is added to every JSON log line from this thread, which is
// how a request id reaches a log statement ten frames down that
// has never heard of the request.
MDC.put("user_id", userId);
MDC.put("request_id", requestId);
try {
log.info("creating todo"); // the JSON gains both fields for free
} finally {
MDC.clear(); // threads are pooled; never leak into the next request
}
}
}Tracing Middleware (OpenTelemetry)
extract the incoming trace, open a span, attach the business context
// middleware/tracing.go, Creates a trace span per request
package middleware
import (
"github.com/gin-gonic/gin"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.uber.org/zap"
"myapp/logger"
)
func EnhancedTracing() gin.HandlerFunc {
tracer := otel.Tracer("todo-api")
return func(c *gin.Context) {
// Extract incoming trace context (from load balancer / front-end)
ctx := otel.GetTextMapPropagator().Extract(
c.Request.Context(),
propagation.HeaderCarrier(c.Request.Header),
)
// Start root span for this request
ctx, span := tracer.Start(ctx, c.FullPath())
defer span.End()
// Attach metadata to span (visible in trace UI)
span.SetAttributes(
attribute.String("http.method", c.Request.Method),
attribute.String("http.path", c.FullPath()),
attribute.String("http.user_agent", c.Request.UserAgent()),
attribute.String("user.id", getUserID(c)),
)
// Inject context so downstream handlers can create child spans
c.Request = c.Request.WithContext(ctx)
c.Next()
// After handler returns: record status code on span
statusCode := c.Writer.Status()
span.SetAttributes(attribute.Int("http.status_code", statusCode))
if statusCode >= 500 {
span.SetStatus(codes.Error, "server error")
}
// Structured log entry linking to this trace
logger.Log.Info("request completed",
zap.String("method", c.Request.Method),
zap.String("path", c.FullPath()),
zap.Int("status", statusCode),
zap.String("trace_id", span.SpanContext().TraceID().String()),
)
}
}# tracing.py, OTel setup for FastAPI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
def setup_tracing(app):
# Tracer provider, sends spans to OTel Collector via gRPC
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
# Auto-instrument FastAPI, all routes get spans automatically, and
# the incoming traceparent header is extracted for you
FastAPIInstrumentor.instrument_app(app)
# Auto-instrument SQLAlchemy, all DB queries get spans automatically
SQLAlchemyInstrumentor().instrument(engine=engine)
# main.py, wire it all up
from fastapi import FastAPI
from tracing import setup_tracing
from logging_config import configure_logging, log
configure_logging()
app = FastAPI()
setup_tracing(app)// tracing.js, creates a trace span per request
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { trace, context, propagation, SpanStatusCode } from '@opentelemetry/api'
import { log } from './logger.js'
// Auto-instrumentation patches http, express, pg and redis, so most spans
// exist without you writing them. It MUST start before your app imports
// those modules, so this file is loaded with --import, not required later.
new NodeSDK({
traceExporter: new OTLPTraceExporter({ url: 'http://otel-collector:4317' }),
instrumentations: [getNodeAutoInstrumentations()],
}).start()
const tracer = trace.getTracer('todo-api')
export function enhancedTracing(req, res, next) {
// Extract incoming trace context (from load balancer / front-end)
const parent = propagation.extract(context.active(), req.headers)
// Start root span for this request
const span = tracer.startSpan(req.route?.path ?? req.path, {}, parent)
// Attach metadata to span (visible in trace UI)
span.setAttributes({
'http.method': req.method,
'http.path': req.route?.path ?? req.path,
'http.user_agent': req.get('user-agent'),
'user.id': getUserId(req),
})
res.on('finish', () => {
// After the handler returns: record status code on span
span.setAttribute('http.status_code', res.statusCode)
if (res.statusCode >= 500) {
span.setStatus({ code: SpanStatusCode.ERROR, message: 'server error' })
}
// Structured log entry linking to this trace
log.info({
method: req.method,
path: req.route?.path ?? req.path,
status: res.statusCode,
trace_id: span.spanContext().traceId,
}, 'request completed')
span.end()
})
// Run the rest of the chain INSIDE this span's context, so anything
// downstream that opens a span becomes its child automatically.
context.with(trace.setSpan(parent, span), next)
}import { trace, context, propagation, SpanStatusCode, type Span } from '@opentelemetry/api'
import type { NextFunction, Request, Response } from 'express'
const tracer = trace.getTracer('todo-api')
export function enhancedTracing(req: Request, res: Response, next: NextFunction): void {
// Extract incoming trace context (from load balancer / front-end)
const parent = propagation.extract(context.active(), req.headers)
// Use the ROUTE TEMPLATE as the span name ("/todos/:id"), never the
// real path. A span name per todo id makes the trace UI useless,
// exactly as it would make a Prometheus label useless (sec 10).
const routeName = req.route?.path ?? req.path
const span: Span = tracer.startSpan(routeName, {}, parent)
span.setAttributes({
'http.method': req.method,
'http.path': routeName,
'http.user_agent': req.get('user-agent') ?? 'unknown',
'user.id': getUserId(req),
})
res.on('finish', () => {
span.setAttribute('http.status_code', res.statusCode)
if (res.statusCode >= 500) {
span.setStatus({ code: SpanStatusCode.ERROR, message: 'server error' })
}
log.info({
method: req.method,
path: routeName,
status: res.statusCode,
trace_id: span.spanContext().traceId,
}, 'request completed')
span.end() // in the 'finish' handler, so it runs on every path
})
context.with(trace.setSpan(parent, span), next)
}// Spring needs very little tracing middleware of its own. Adding the
// OpenTelemetry starter instruments every controller, RestClient and JDBC
// call, and extracts the incoming traceparent header automatically:
//
// implementation 'io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter'
//
// otel.service.name: todo-api
// otel.exporter.otlp.endpoint: http://otel-collector:4317
// What is still worth writing by hand is the per-request enrichment.
@Component
public class TracingFilter extends OncePerRequestFilter {
private static final Logger log = LoggerFactory.getLogger(TracingFilter.class);
@Override
protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
FilterChain chain)
throws ServletException, IOException {
// The starter already extracted the incoming trace context and
// started the root span; pick it up rather than starting another.
Span span = Span.current();
// Attach metadata to span (visible in trace UI)
span.setAttribute("http.method", req.getMethod());
span.setAttribute("http.path", req.getRequestURI());
span.setAttribute("http.user_agent", String.valueOf(req.getHeader("User-Agent")));
span.setAttribute("user.id", getUserId(req));
// Putting the trace id in MDC is what joins the three pillars:
// every log line this request emits now carries the id that the
// trace UI is keyed on, so a slow span leads straight to its logs.
MDC.put("trace_id", span.getSpanContext().getTraceId());
try {
chain.doFilter(req, res);
} finally {
int status = res.getStatus();
span.setAttribute("http.status_code", status);
if (status >= 500) {
span.setStatus(StatusCode.ERROR, "server error");
}
log.info("request completed method={} path={} status={}",
req.getMethod(), req.getRequestURI(), status);
MDC.clear();
}
}
}Service Layer: Logging + Tracing + Error Handling
one business operation, instrumented end to end
// services/todo_service.go, The full LMO pattern per function
package services
import (
"context"
"fmt"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.uber.org/zap"
"myapp/logger"
"myapp/models"
)
var tracer = otel.Tracer("todo-service")
func CreateTodo(ctx context.Context, req models.CreateTodoRequest, userID string) (*models.Todo, error) {
// 1. Start a child span (parent span is in ctx from the middleware)
ctx, span := tracer.Start(ctx, "TodoService.CreateTodo")
defer span.End()
// 2. Add business context to the span, visible in trace UI
span.SetAttributes(
attribute.String("user.id", userID),
attribute.String("todo.title", req.Title),
attribute.String("todo.priority", req.Priority),
)
// 3. INFO log, record business event start
logger.Log.Info("creating todo",
zap.String("user_id", userID),
zap.String("title", req.Title),
zap.String("trace_id", span.SpanContext().TraceID().String()),
)
// 4. Execute DB operation (inside its own child span, auto via otelgorm)
todo, err := repo.CreateTodo(ctx, req, userID)
if err != nil {
// ERROR log with full context
logger.Log.Error("failed to create todo",
zap.String("user_id", userID),
zap.String("error", err.Error()),
zap.String("trace_id", span.SpanContext().TraceID().String()),
)
// Record error on span, shows as failed in Jaeger/Grafana Tempo
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return nil, fmt.Errorf("repo.CreateTodo: %w", err)
}
// 5. DEBUG log, only visible in dev, suppressed in production
logger.Log.Debug("todo created",
zap.String("todo_id", todo.ID),
)
// 6. INFO, business event completion log (for audit trail)
logger.Log.Info("todo created successfully",
zap.String("todo_id", todo.ID),
zap.String("user_id", userID),
zap.String("title", todo.Title),
zap.String("priority", todo.Priority),
zap.String("category_id", todo.CategoryID),
)
return todo, nil
}# services/todo_service.py
from opentelemetry import trace
import structlog
tracer = trace.get_tracer("todo-service")
log = structlog.get_logger()
async def create_todo(user_id: str, req: CreateTodoRequest) -> Todo:
# 1. Start child span (the parent is the request span from the middleware)
with tracer.start_as_current_span("TodoService.create_todo") as span:
# 2. Add business context to the span, visible in trace UI
span.set_attribute("user.id", user_id)
span.set_attribute("todo.title", req.title)
# 3. INFO log, business event start
log.info("creating_todo", user_id=user_id, title=req.title)
try:
# 4. Execute DB operation (its own child span, auto-instrumented)
todo = await repo.create_todo(user_id, req)
# 5. DEBUG log, only visible in dev, suppressed in production
log.debug("todo_created", todo_id=todo.id)
# 6. INFO, business event completion log (for audit trail)
log.info("todo_created_successfully",
todo_id=todo.id,
user_id=user_id,
title=todo.title,
priority=todo.priority,
)
return todo
except Exception as e:
# ERROR log with full context + span error recording
log.error("todo_creation_failed", user_id=user_id, error=str(e))
span.record_exception(e)
span.set_status(trace.StatusCode.ERROR, str(e))
raise// services/todoService.js, the full LMO pattern per function
import { trace, SpanStatusCode } from '@opentelemetry/api'
import { log } from '../logger.js'
const tracer = trace.getTracer('todo-service')
export async function createTodo(req, userId) {
// 1. Start a child span. startActiveSpan also makes it the CURRENT
// span, so the DB call below nests under it automatically.
return tracer.startActiveSpan('TodoService.createTodo', async (span) => {
// 2. Add business context to the span, visible in trace UI
span.setAttributes({
'user.id': userId,
'todo.title': req.title,
'todo.priority': req.priority,
})
const traceId = span.spanContext().traceId
// 3. INFO log, record business event start
log.info({ user_id: userId, title: req.title, trace_id: traceId }, 'creating todo')
try {
// 4. Execute DB operation (its own child span, auto-instrumented)
const todo = await repo.createTodo(req, userId)
// 5. DEBUG log, only visible in dev, suppressed in production
log.debug({ todo_id: todo.id }, 'todo created')
// 6. INFO, business event completion log (for audit trail)
log.info({
todo_id: todo.id,
user_id: userId,
title: todo.title,
priority: todo.priority,
category_id: todo.categoryId,
}, 'todo created successfully')
return todo
} catch (err) {
// ERROR log with full context
log.error({ user_id: userId, err, trace_id: traceId }, 'failed to create todo')
// Record error on span, shows as failed in Jaeger / Grafana Tempo
span.recordException(err)
span.setStatus({ code: SpanStatusCode.ERROR, message: err.message })
throw err
} finally {
// startActiveSpan does NOT end the span for you. A span that is
// never ended is a span that never reaches the collector.
span.end()
}
})
}import { trace, SpanStatusCode, type Span } from '@opentelemetry/api'
import { log } from '../logger'
const tracer = trace.getTracer('todo-service')
export async function createTodo(req: CreateTodoRequest, userId: string): Promise<Todo> {
// 1. Start a child span, and make it current for everything inside
return tracer.startActiveSpan('TodoService.createTodo', async (span: Span) => {
// 2. Add business context to the span, visible in trace UI.
// Attribute VALUES must be primitives; passing an object silently
// drops the attribute, so shape it here rather than at the call site.
span.setAttributes({
'user.id': userId,
'todo.title': req.title,
'todo.priority': req.priority,
})
const traceId = span.spanContext().traceId
// 3. INFO log, record business event start
log.info({ user_id: userId, title: req.title, trace_id: traceId }, 'creating todo')
try {
const todo = await repo.createTodo(req, userId) // 4. DB, own child span
log.debug({ todo_id: todo.id }, 'todo created') // 5. dev only
// 6. INFO, business event completion log (for audit trail)
log.info({
todo_id: todo.id,
user_id: userId,
title: todo.title,
priority: todo.priority,
category_id: todo.categoryId,
}, 'todo created successfully')
return todo
} catch (err) {
log.error({ user_id: userId, err, trace_id: traceId }, 'failed to create todo')
span.recordException(err as Error)
span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message })
throw err // re-throw: the caller decides the status code, not us
} finally {
span.end()
}
})
}// TodoService.java, the full LMO pattern per method
@Service
public class TodoService {
private static final Logger log = LoggerFactory.getLogger(TodoService.class);
private final Tracer tracer = GlobalOpenTelemetry.getTracer("todo-service");
public Todo createTodo(CreateTodoRequest req, String userId) {
// 1. Start a child span (the parent is the current Context)
Span span = tracer.spanBuilder("TodoService.createTodo").startSpan();
// 2. Add business context to the span, visible in trace UI
span.setAttribute("user.id", userId);
span.setAttribute("todo.title", req.title());
span.setAttribute("todo.priority", req.priority());
// makeCurrent() is what makes the NEXT span a child of this one.
// Forget it and the trace flattens into a list of orphan spans
// that no longer tells you what called what.
try (Scope scope = span.makeCurrent()) {
// 3. INFO log, record business event start
log.info("creating todo user_id={} title={}", userId, req.title());
// 4. Execute DB operation (its own child span, auto-instrumented)
Todo todo = repo.createTodo(req, userId);
// 5. DEBUG log, only visible in dev, suppressed in production
log.debug("todo created todo_id={}", todo.id());
// 6. INFO, business event completion log (for audit trail)
log.info("todo created successfully todo_id={} user_id={} title={} priority={}",
todo.id(), userId, todo.title(), todo.priority());
return todo;
} catch (RuntimeException e) {
// ERROR log with full context. The exception goes LAST and
// without a placeholder: that is the SLF4J signature that
// logs the full stack trace rather than just its toString.
log.error("failed to create todo user_id={}", userId, e);
// Record error on span, shows as failed in Jaeger / Tempo
span.recordException(e);
span.setStatus(StatusCode.ERROR, e.getMessage());
throw e;
} finally {
span.end(); // ALWAYS, or the span leaks and never reaches the collector
}
}
}Prometheus Middleware (Metrics)
count and time every request, labelled by the route TEMPLATE
// middleware/metrics.go, HTTP metrics instrumentation
package middleware
import (
"strconv"
"time"
"github.com/gin-gonic/gin"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
httpRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total HTTP requests",
}, []string{"method", "path", "status"})
httpDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "HTTP request latency",
Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5},
}, []string{"method", "path"})
)
func PrometheusMetrics() gin.HandlerFunc {
return func(c *gin.Context) {
start := time.Now()
c.Next()
duration := time.Since(start)
status := strconv.Itoa(c.Writer.Status())
httpRequestsTotal.WithLabelValues(c.Request.Method, c.FullPath(), status).Inc()
httpDuration.WithLabelValues(c.Request.Method, c.FullPath()).Observe(duration.Seconds())
}
}# middleware/metrics.py, HTTP metrics instrumentation
import time
from fastapi import Request
from prometheus_client import Counter, Histogram, make_asgi_app
http_requests_total = Counter(
"http_requests_total",
"Total HTTP requests",
["method", "path", "status"],
)
http_duration = Histogram(
"http_request_duration_seconds",
"HTTP request latency",
["method", "path"],
buckets=(.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5),
)
@app.middleware("http")
async def prometheus_metrics(request: Request, call_next):
start = time.perf_counter()
response = await call_next(request)
duration = time.perf_counter() - start
# The route TEMPLATE ("/todos/{id}"), never request.url.path. The raw
# path would mint one time series per todo id, and THAT is what takes
# a Prometheus server down, not the request volume.
route = request.scope.get("route")
path = route.path if route else "unmatched"
http_requests_total.labels(request.method, path, response.status_code).inc()
http_duration.labels(request.method, path).observe(duration)
return response
# Expose /metrics for Prometheus to scrape
app.mount("/metrics", make_asgi_app())// middleware/metrics.js, HTTP metrics instrumentation
import { Counter, Histogram, register } from 'prom-client'
const httpRequestsTotal = new Counter({
name: 'http_requests_total',
help: 'Total HTTP requests',
labelNames: ['method', 'path', 'status'],
})
const httpDuration = new Histogram({
name: 'http_request_duration_seconds',
help: 'HTTP request latency',
labelNames: ['method', 'path'],
buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
})
export function prometheusMetrics(req, res, next) {
const end = httpDuration.startTimer()
res.on('finish', () => {
// req.route.path is the TEMPLATE ("/todos/:id"). req.path is the real
// url and would create one time series per todo id.
const path = req.route?.path ?? 'unmatched'
end({ method: req.method, path })
httpRequestsTotal.inc({ method: req.method, path, status: res.statusCode })
})
next()
}
// Expose /metrics for Prometheus to scrape
app.get('/metrics', async (_req, res) => {
res.set('Content-Type', register.contentType)
res.end(await register.metrics())
})import { Counter, Histogram, register } from 'prom-client'
import type { NextFunction, Request, Response } from 'express'
// Naming the label set as a type is worth the three lines. Prometheus
// costs one time series per distinct label COMBINATION, so the danger is
// never the metric, it is somebody adding `user_id` to this object in six
// months. A closed type makes that a compile error.
type HttpLabels = { method: string; path: string; status?: number }
const httpRequestsTotal = new Counter<keyof HttpLabels>({
name: 'http_requests_total',
help: 'Total HTTP requests',
labelNames: ['method', 'path', 'status'],
})
const httpDuration = new Histogram<keyof HttpLabels>({
name: 'http_request_duration_seconds',
help: 'HTTP request latency',
labelNames: ['method', 'path'],
buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
})
export function prometheusMetrics(req: Request, res: Response, next: NextFunction): void {
const end = httpDuration.startTimer()
res.on('finish', () => {
const labels: HttpLabels = { method: req.method, path: req.route?.path ?? 'unmatched' }
end(labels)
httpRequestsTotal.inc({ ...labels, status: res.statusCode })
})
next()
}// Spring Boot gives you the HTTP metrics for free. Add
// spring-boot-starter-actuator + micrometer-registry-prometheus and every
// controller is counted and timed as http_server_requests_seconds,
// tagged by method, status, and the uri TEMPLATE:
//
// management.endpoints.web.exposure.include: prometheus
// management.metrics.tags.service: todo-api
//
// The uri tag uses "/todos/{id}", never the actual path. That is not an
// accident: tagging by raw path mints one time series per id, which is
// the classic way to take a Prometheus server down.
// What no framework can know is your BUSINESS metrics, so write those.
@Component
public class TodoMetrics {
private final Counter created;
private final Timer dbWrite;
TodoMetrics(MeterRegistry registry) {
this.created = Counter.builder("todos_created_total")
.description("Todos successfully created")
.register(registry);
this.dbWrite = Timer.builder("todo_db_write_seconds")
.description("Time spent writing a todo")
.publishPercentileHistogram() // the histogram buckets
.register(registry);
}
public void recordCreated() {
created.increment();
}
public <T> T timeDbWrite(Supplier<T> work) {
return dbWrite.record(work); // times both the happy and throwing paths
}
}13
Tooling: The Open Source Stack
The open-source observability stack is what most large enterprises run. More control, no vendor lock-in, no per-seat pricing, but requires operational expertise to maintain.
| Tool | Pillar | Role |
|---|---|---|
| Prometheus | Metrics | Scrapes /metrics endpoints every 15s, stores time-series data, evaluates alert rules. The de facto standard metrics backend. |
| Grafana | Dashboards | Visualization layer. Queries Prometheus, Loki, Tempo, and others. Build dashboards, set alert thresholds, create on-call runbooks. |
| Loki | Logs | Log aggregation system by Grafana Labs. Unlike ELK, it indexes only metadata (labels), not full-text, much cheaper. Queryable via LogQL. Push-based via Promtail agent. |
| Promtail | Logs (agent) | Runs on each server, tails log files or Docker/Kubernetes log streams, and ships them to Loki. Zero-code change required. |
| Grafana Tempo | Traces | Distributed tracing backend. Receives OTel spans, stores them cheaply (object storage), queryable from Grafana. Replaces Jaeger for new setups. |
| Jaeger | Traces | Older distributed tracing system (CNCF). Still widely used. UI for navigating traces, service dependency graphs. Now recommends migrating to OTel SDK. |
| Alertmanager | Alerting | Routes Prometheus alerts to Slack, PagerDuty, email, etc. Handles deduplication, grouping, and silencing of alerts. |
| OTel Collector | All three | Receives all telemetry from apps, processes/enriches it, and fans out to Prometheus + Loki + Tempo. Central telemetry pipeline. |
Full Stack Architecture
14
Tooling: Proprietary Solutions
If you don’t have the team to manage the open-source stack, proprietary all-in-one solutions are the pragmatic choice. They bundle logging, metrics, tracing, and alerting into a single platform with managed storage and a polished UI.
| Tool | Strengths | Best For | Pricing Model |
|---|---|---|---|
| New Relic | Full-stack observability, Go/Python/Node agents, APM, browser monitoring, mobile. Easy 10-minute setup. | Teams that want one tool with minimal ops overhead | Free tier (100GB/month), then per GB ingested |
| Datadog | Industry-leading dashboards, 500+ integrations, excellent APM, infra monitoring, security. Very powerful. | Enterprises with complex infra, Kubernetes, multi-cloud | Per host + per GB, can get expensive quickly |
| Sentry | Best-in-class error tracking. Groups identical errors, shows stack traces, user context, release tracking. | Error monitoring specifically, pairs well with Grafana for metrics | Free tier (5k errors/month), then per event |
| Honeycomb | High-cardinality event querying. Built for observability-first teams. Native OTel support. | Teams doing advanced distributed tracing, many microservices | Per event ingested |
New Relic Agent: Quick Integration
one agent, auto-instrumented routes, no per-handler code
// main.go, New Relic agent setup in Go
import (
"github.com/newrelic/go-agent/v3/newrelic"
"github.com/newrelic/go-agent/v3/integrations/nrgin"
)
func main() {
// Initialize New Relic application agent
nrApp, _ := newrelic.NewApplication(
newrelic.ConfigAppName("todo-api"),
newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")),
newrelic.ConfigAppLogForwardingEnabled(true), // forward logs to NR
newrelic.ConfigDistributedTracerEnabled(true), // enable distributed tracing
)
r := gin.New()
// nrgin middleware, auto-instruments every route with NR transactions
r.Use(nrgin.Middleware(nrApp))
r.Run(":8080")
}# Python's agent is mostly configuration, not code. Install it, point it
# at a config file, and start the app THROUGH the admin wrapper:
#
# pip install newrelic
# newrelic-admin generate-config $NEW_RELIC_LICENSE_KEY newrelic.ini
# NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program \
# uvicorn main:app --host 0.0.0.0 --port 8000
# newrelic.ini
# [newrelic]
# app_name = todo-api
# application_logging_forwarding_enabled = true ; forward logs to NR
# distributed_tracing.enabled = true ; enable distributed tracing
# Or initialise in-process, which keeps the wrapper out of your Dockerfile:
import newrelic.agent
newrelic.agent.initialize("newrelic.ini") # MUST run before importing the app
from fastapi import FastAPI
app = FastAPI()
# The agent auto-instruments FastAPI, SQLAlchemy, redis and httpx by
# monkey-patching them on import, which is why initialise must come first.
# Only custom business spans need writing by hand:
@newrelic.agent.function_trace()
async def create_todo(user_id: str, req: CreateTodoRequest) -> Todo:
...// The Node agent instruments by patching modules as they load, so it has
// to be the VERY first thing the process imports, before express or pg.
// node --import newrelic/esm-loader.mjs server.js
// or, for CommonJS, literally the first line of the entry file:
// require('newrelic')
// newrelic.cjs, config lives beside the entry point
exports.config = {
app_name: ['todo-api'],
license_key: process.env.NEW_RELIC_LICENSE_KEY,
application_logging: {
forwarding: { enabled: true }, // forward logs to NR
},
distributed_tracing: { enabled: true }, // enable distributed tracing
}
// server.js
import 'newrelic' // first import, always
import express from 'express'
const app = express()
// Every route is now auto-instrumented as an NR transaction; there is no
// middleware to add. Only custom segments are written by hand:
import newrelic from 'newrelic'
app.get('/todos/:id', async (req, res) => {
const todo = await newrelic.startSegment('repo.findTodo', true, () =>
repo.findTodo(req.params.id))
res.json(todo)
})import 'newrelic' // first import, before express and pg are loaded
import newrelic from 'newrelic'
import express from 'express'
// Worth knowing before you reach for a vendor agent in a TypeScript
// codebase: the agent patches modules at load time, which no type
// system can see. A mis-ordered import type-checks perfectly and
// instruments nothing, so keep the bare `import 'newrelic'` on line one
// and configure the entry point with `--import`, never a lazy import.
const app = express()
// Custom segments are the only hand-written part, and they are generic,
// so the return type of the wrapped work survives the wrapper.
async function findTodo(id: string): Promise<Todo> {
return newrelic.startSegment('repo.findTodo', true, () => repo.findTodo(id))
}
app.get('/todos/:id', async (req, res) => {
res.json(await findTodo(req.params.id))
})
// Prefer OTel + a collector if you may ever change vendor. New Relic,
// Datadog and Honeycomb all ingest OTLP, so the instrumentation in the
// sections above ports between them; a vendor agent does not.// The Java agent is a JVM-level attachment: no dependency, no code, and
// nothing to import. You download newrelic.jar and hand it to the JVM:
//
// java -javaagent:/opt/newrelic/newrelic.jar -jar todo-api.jar
//
// newrelic.yml
// common: &default_settings
// app_name: todo-api
// license_key: ${NEW_RELIC_LICENSE_KEY}
// application_logging:
// forwarding:
// enabled: true # forward logs to NR
// distributed_tracing:
// enabled: true # enable distributed tracing
// Bytecode instrumentation means every Spring controller, JDBC query and
// HTTP client call is traced without a single annotation. Only custom
// business spans are written by hand:
@Service
public class TodoService {
@Trace(dispatcher = true) // com.newrelic.api.agent.Trace
public Todo createTodo(CreateTodoRequest req, String userId) {
NewRelic.addCustomParameter("user.id", userId);
return repo.createTodo(req, userId);
}
}
// Same caveat as everywhere else in this section: the OTel instrumentation
// above is vendor-neutral and New Relic ingests OTLP directly. Reach for
// the agent when you want zero-code coverage, not as the default.15
ELK Stack: Elasticsearch, Logstash, Kibana
The ELK Stack (now called the Elastic Stack) is the classic enterprise logging platform. It provides full-text search across logs, powerful aggregations, and rich dashboards. It’s heavier than Loki but offers more querying flexibility.
Elasticsearch
The storage and search engine. Indexes every field of every log document. Supports complex queries: full-text search, aggregations, range filters. Horizontally scalable.
Logstash
The log pipeline. Ingests logs from many sources (files, Kafka, Redis, Beats agents), transforms/filters them (parse JSON, add geo-IP, mask PII), and outputs to Elasticsearch. Powerful but resource-heavy.
Kibana
The visualization layer. Build dashboards, run queries against Elasticsearch (KQL language), create alerts. The UI equivalent of Grafana for Elastic data.
Filebeat / Fluentd
Lightweight log shipping agents that run on your servers and ship log files to Logstash or Elasticsearch directly. Lower resource overhead than Logstash as a local agent.
ELK vs Grafana Loki
| Property | ELK Stack | Grafana Loki |
|---|---|---|
| Indexing | Full-text indexes every field -> powerful queries | Only indexes labels (metadata) -> cheaper storage |
| Storage cost | High, indexing all fields is expensive | Low, like Prometheus for logs |
| Query power | Very high, KQL, complex aggregations | Good for label-based queries (LogQL) |
| Integration | Self-contained Kibana UI | Grafana, one UI for metrics + logs + traces |
| Best for | High query complexity, security/compliance use cases | Cost-sensitive setups already using Grafana stack |
16
Alert Design: What to Alert On
Not every metric needs an alert. Alert fatigue is a real problem, if engineers receive too many notifications, they start ignoring them. Good alert design means alerting on symptoms that impact users, not internal implementation details.
The Four Golden Signals (Google SRE Book)
Latency
Time to serve a request. Distinguish between successful and failed requests, a fast error is not a good metric. Alert on p99 > threshold.
Traffic
Demand on your system, requests/sec. Sudden drops can indicate an outage just as much as sudden spikes. Alert on both extremes.
Errors
Rate of failed requests (5xx). Alert on error rate, not raw count, a spike in traffic will naturally cause more errors even if the error rate is stable.
Saturation
How “full” is your service, CPU, memory, DB connections, queue depth. Alerts before the system tips into failure. Alert at 80%, page at 95%.
Alert Rules (Grafana / Prometheus format)
# alert_rules.yml, Production-grade alert rules
groups:
- name: api_health
rules:
# P1, Error rate critical
- alert: HighErrorRate
expr: |
rate(http_requests_total{status=~"5.."}[5m])
/ rate(http_requests_total[5m]) * 100 > 5
for: 2m
labels:
severity: critical
annotations:
summary: "HTTP error rate above 5% ({{ $value | printf \"%.1f\" }}%)"
runbook: "https://wiki.company.com/runbooks/high-error-rate"
# P2, High latency
- alert: HighP99Latency
expr: |
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket[5m])
) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "P99 latency above 2s ({{ $value | printf \"%.2f\" }}s)"
# P2, DB connection pool near exhaustion
- alert: DBConnectionPoolExhaustion
expr: db_connections_open / db_connections_max > 0.85
for: 3m
labels:
severity: warning
annotations:
summary: "DB connection pool at {{ $value | humanizePercentage }}"
# P1, Service is down
- alert: ServiceDown
expr: up{job="todo-api"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "todo-api service is unreachable"
17
Production Debugging Workflow: Full Example
Here is what the complete workflow looks like end-to-end when an incident occurs in a system with full LMO in place.
Scenario: API Suddenly Slow + High Errors
-
1
Alert fires at 14:31 UTC
Alertmanager sends Slack message: “[CRITICAL] HighErrorRate: HTTP error rate is 82.4% (was 0.3%). Service: todo-api. Runbook: wiki/…”
-
2
Open Grafana -> Metrics dashboard
P99 latency jumped from 120ms to 4.3s at 14:31. Requests/sec unchanged (not a traffic spike). DB connection pool at 97%. Error type: 100% are 500s, not 4xx. Indicates server-side DB problem.
-
3
Correlate to Logs in Loki / New Relic
Query:
{service="todo-api", level="error"} |= "connection". Hundreds of entries:ERROR: pq: remaining connection slots are reserved for non-replication superuser connections. DB connection pool is exhausted. -
4
Jump to Trace from log entry
Click trace link in the log. Trace shows: HTTP Handler -> AuthMiddleware (3ms) -> TodoService (4290ms) -> DB.GetTodos (4285ms <- STUCK). DB.GetTodos span has attribute:
query="SELECT * FROM todos WHERE user_id=$1". -
5
Root cause identified
A deployment 5 minutes ago added a
SELECT *that performs a full table scan (missing index). Under load, queries pile up, connections are held longer, pool exhausts. Checkgit log, confirms deployment at 14:29 UTC. -
6
Fix + verify
Add missing index migration. Deploy. Grafana shows error rate drops to 0.1% within 2 minutes of deployment. DB connection pool back to 12%. Incident resolved. Total time: 11 minutes.
18
Best Practices
1. Always Include a Request ID
Generate a unique request_id (UUID or ULID) at the entry point of every request, either in your load balancer or in your first middleware. Inject it into the context and include it in every log entry and span for that request. This is what lets you pull all logs for a single failing request out of millions of entries.
2. Log at Boundaries, Not Inside Logic
Log at the entry and exit of major components (handler, service, repository), not scattered throughout every loop and condition. This gives you a clean, readable trail without noise. Inside functions, use spans for granular timing, not log.Debug spam.
3. Never Log Sensitive Data
Passwords, credit card numbers, SSNs, auth tokens, and API keys must never appear in logs. Use structured logging libraries that let you mark fields as redacted. In your CI pipeline, consider a log scanning step that fails the build if certain patterns are found in log statements.
4. Use Sampling for High-Volume Traces
At high traffic (1000+ req/sec), recording every trace is expensive. Use head-based sampling (record 10% of all requests) or tail-based sampling (record 100% of error requests + 1% of success requests). The OTel Collector supports both. Never sample errors, always record traces for failures.
5. Log Levels Discipline
- Never use DEBUG in production, it generates enormous volume and costs storage
- INFO should be your production default, meaningful events, not chatty
- Treat every ERROR log as something that needs a ticket or an alert rule
- FATAL should mean “I need someone paged right now”
6. This is a Spectrum, Not a Checkbox
Start with basic structured logging in JSON format + a simple Prometheus counter for error rate. That alone will save you hours when the first production incident happens. Then iteratively add: histogram metrics, basic tracing, Grafana dashboards, alert rules. You don’t need the full ELK + Jaeger + OTel Collector stack on day one. Build it incrementally.
Backend from First Principles / Chapter 15 / Observability. Code targets Go 1.22+, Python 3.11+, Node.js 20+, TypeScript 5+ and Java 17+ (Spring Boot).