Logging, Monitoring
& Observability

Modern backend applications run in distributed environments — multiple services, multiple servers, multiple regions, users spread across the globe. Something will go wrong. The question is not if, but when — and more importantly, how fast can you find out and fix it?

Without logging, monitoring, and observability, when an incident happens you are completely blind. You know something is wrong (users are complaining, conversion dropped), but you have no idea what, where, or why. Every minute of debugging in the dark is revenue lost and user trust eroded.

These practices also require a collective effort. As a developer, you instrument your code. DevOps / SRE teams configure the infrastructure that collects, stores, and displays that data. Neither side alone is sufficient.

Observability theory defines three foundational pillars. A system is only fully observable when all three are in place. Each pillar answers a different question about your system's behaviour.

These terms are often used interchangeably but they are distinct concepts at different levels of capability.

Key insight: Monitoring tells you there is a problem. Observability tells you exactly what the problem is — the specific function, the specific service, the specific user, the specific query. This is only possible if you've properly implemented all three pillars.

In practice, when an incident occurs, you navigate from coarse-grained signals to fine-grained details. Here's the canonical flow used in production systems:

• 1
  Alert fires (Monitoring)

  Grafana / New Relic detects error rate > 80% and fires a Slack webhook. You get a message: "API service error rate spiked at 14:32." This is monitoring doing its job.

• 2
  Go to Metrics dashboard

  Open Grafana / New Relic. You see concrete numbers: 840 requests/min, 82% returning 5xx, p99 latency jumped from 120ms to 4.2s at 14:31. You now know the scale of the problem and when it started.

• 3
  Correlate to Logs

  From the metrics view, click through to the associated error logs for that 2-minute window. You see hundreds of logs: ERROR: connection pool exhausted after 4000ms. Now you know what is failing.

• 4
  Follow the Trace

  Click on one of those error logs. It links to a trace. The trace shows the full request journey: AuthMiddleware (2ms) → UserHandler (5ms) → UserService (3ms) → DB.GetUser (3990ms ← HERE). The DB query is timing out. Root cause found.

• 5
  Fix & Confirm with Metrics

  You add a missing index to the query. Within minutes, the Grafana dashboard shows error rate dropping back to 0.2%, latency back to 110ms. Monitoring confirms the fix worked.

Logging is the practice of recording all important events throughout your application's execution lifecycle. Think of it as a journal your backend keeps — timestamped, detailed, structured entries for every significant thing that happens.

What Should You Log?

A good rule of thumb: anything a developer would want to know when debugging an issue at 2 AM — without access to the live system, without the ability to reproduce the bug, with only the logs to go on.

What Metadata to Include in Every Log

A log without context is almost useless. Every log entry should carry enough metadata to answer: who did what, when, where, and with what result?

// A well-structured log entry (JSON, production format)
{
  "timestamp":   "2024-01-15T14:32:01.234Z",
  "level":        "error",
  "message":      "database query failed",
  "service":      "todo-api",
  "environment":  "production",
  "request_id":   "req_01J2KXYZ",    // unique per request
  "trace_id":     "abc123def456",    // links to distributed trace
  "span_id":      "span_789",
  "user_id":      "usr_01J2K",
  "method":       "POST",
  "path":         "/todos",
  "error":        "pq: deadlock detected",
  "duration_ms":  4001,
  "host":         "worker-node-3"
}
JSON Log Entry

Every log entry is assigned a severity level. This lets you filter what you see — in development you want everything, in production you care about warnings and above. Your logging library will let you set a minimum level per environment.

Environment-Specific Level Configuration (Go)

// logger/logger.go — Environment-aware log level
package logger

import (
    "os"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

func getLogLevel() zapcore.Level {
    env := os.Getenv("APP_ENV")
    switch env {
    case "production":
        return zapcore.InfoLevel   // production: INFO and above
    case "staging":
        return zapcore.WarnLevel   // staging: WARN and above only
    default:
        return zapcore.DebugLevel  // local dev: everything
    }
}

func New() *zap.Logger {
    env := os.Getenv("APP_ENV")
    level := getLogLevel()

    var cfg zap.Config
    if env == "production" {
        // Production: JSON format — parseable by Loki, ELK, New Relic
        cfg = zap.NewProductionConfig()
        cfg.EncoderConfig.TimeKey = "timestamp"
        cfg.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
    } else {
        // Development: coloured console format — human-readable
        cfg = zap.NewDevelopmentConfig()
        cfg.EncoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
    }

    cfg.Level = zap.NewAtomicLevelAt(level)
    logger, _ := cfg.Build()
    return logger
}
Go / Zap

There are two fundamental ways to write a log entry. The choice of which to use depends on your environment.

What Structured Logs Look Like Side by Side

// UNSTRUCTURED — development console (human-friendly)
14:32:01 DEBUG  Connected to PostgreSQL on localhost:5432
14:32:01 INFO   Started background job worker
14:32:01 INFO   HTTP server listening on :8080
14:32:14 WARN   Slow query detected: 520ms  query=SELECT * FROM todos WHERE user_id=$1
14:32:15 ERROR  Failed to send email  user_id=usr_01J2K  error=connection refused
Console (Dev)

// STRUCTURED — production JSON (machine-friendly)
{"timestamp":"2024-01-15T14:32:14Z","level":"warn","msg":"slow query","duration_ms":520,"query":"SELECT * FROM todos WHERE user_id=$1","service":"todo-api","env":"production"}
{"timestamp":"2024-01-15T14:32:15Z","level":"error","msg":"failed to send email","user_id":"usr_01J2K","error":"connection refused","trace_id":"abc123","span_id":"span_001","service":"todo-api","env":"production"}
JSON (Production)

Metrics are numerical measurements of your system's behaviour, aggregated over time. Unlike logs (one entry per event), a metric is a counter or gauge that summarises many events into a single number — e.g. "320 requests in the last 15 seconds". This makes them extremely efficient for dashboards and alerting.

The Four Core Metric Types (Prometheus Model)

Key Metrics Every Backend Should Expose

Scrape Interval & Delay

Prometheus scrapes your /metrics endpoint at a configured interval (typically 15 seconds). This means there is a built-in delay of ~10–15 seconds between reality and what you see on a dashboard. This is acceptable for most use cases. For near-real-time alerting, use sub-15-second scrape intervals — but be aware this increases load on both your service and Prometheus.

A trace is the complete record of a single request's journey through your system. Each step of that journey is called a span. Spans are linked by a shared trace_id that is generated when the request first enters your system and propagated through every subsequent call.

Trace vs Span — Definitions

Instrumentation is the act of adding code to your application to measure its behaviour — recording spans, emitting metrics, enriching logs with context. You cannot observe what you haven't instrumented.

OpenTelemetry (OTel) is an open-source, vendor-neutral standard for instrumentation. It provides APIs, SDKs, and an agent (the Collector) for all major languages. The key value: instrument once, send to any backend (Jaeger, Grafana Tempo, New Relic, Datadog, Honeycomb) just by changing configuration — no code changes.

OTel Core Concepts

A complete example showing logging (Zap), metrics (Prometheus), and tracing (OpenTelemetry) wired into a Go service handler. This is the pattern from the lecture's to-do application.

Logger Setup (Zap)

// logger/logger.go
package logger

import (
    "os"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

var Log *zap.Logger

func Init() {
    env := os.Getenv("APP_ENV")
    var cfg zap.Config

    if env == "production" {
        cfg = zap.NewProductionConfig()          // JSON output
        cfg.Level = zap.NewAtomicLevelAt(zapcore.InfoLevel)
    } else {
        cfg = zap.NewDevelopmentConfig()         // coloured console
        cfg.Level = zap.NewAtomicLevelAt(zapcore.DebugLevel)
        cfg.EncoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
    }

    Log, _ = cfg.Build(
        // Always include these fields in every log entry
        zap.Fields(
            zap.String("service", "todo-api"),
            zap.String("env", env),
        ),
    )
}
Go / Zap

Tracing Middleware (OpenTelemetry)

// middleware/tracing.go — Creates a trace span per request
package middleware

import (
    "github.com/gin-gonic/gin"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.uber.org/zap"
    "myapp/logger"
)

func EnhancedTracing() gin.HandlerFunc {
    tracer := otel.Tracer("todo-api")

    return func(c *gin.Context) {
        // Extract incoming trace context (from load balancer / front-end)
        ctx := otel.GetTextMapPropagator().Extract(
            c.Request.Context(),
            propagation.HeaderCarrier(c.Request.Header),
        )

        // Start root span for this request
        ctx, span := tracer.Start(ctx, c.FullPath())
        defer span.End()

        // Attach metadata to span (visible in trace UI)
        span.SetAttributes(
            attribute.String("http.method", c.Request.Method),
            attribute.String("http.path", c.FullPath()),
            attribute.String("http.user_agent", c.Request.UserAgent()),
            attribute.String("user.id", getUserID(c)),
        )

        // Inject context so downstream handlers can create child spans
        c.Request = c.Request.WithContext(ctx)
        c.Next()

        // After handler returns: record status code on span
        statusCode := c.Writer.Status()
        span.SetAttributes(attribute.Int("http.status_code", statusCode))
        if statusCode >= 500 {
            span.SetStatus(codes.Error, "server error")
        }

        // Structured log entry linking to this trace
        logger.Log.Info("request completed",
            zap.String("method", c.Request.Method),
            zap.String("path", c.FullPath()),
            zap.Int("status", statusCode),
            zap.String("trace_id", span.SpanContext().TraceID().String()),
        )
    }
}
Go / Gin + OTel

Service Layer — Logging + Tracing + Error Handling

// services/todo_service.go — The full LMO pattern per function
package services

import (
    "context"
    "fmt"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.uber.org/zap"
    "myapp/logger"
    "myapp/models"
)

var tracer = otel.Tracer("todo-service")

func CreateTodo(ctx context.Context, req models.CreateTodoRequest, userID string) (*models.Todo, error) {
    // 1. Start a child span (parent span is in ctx from the middleware)
    ctx, span := tracer.Start(ctx, "TodoService.CreateTodo")
    defer span.End()

    // 2. Add business context to the span — visible in trace UI
    span.SetAttributes(
        attribute.String("user.id", userID),
        attribute.String("todo.title", req.Title),
        attribute.String("todo.priority", req.Priority),
    )

    // 3. INFO log — record business event start
    logger.Log.Info("creating todo",
        zap.String("user_id", userID),
        zap.String("title", req.Title),
        zap.String("trace_id", span.SpanContext().TraceID().String()),
    )

    // 4. Execute DB operation (inside its own child span — auto via otelgorm)
    todo, err := repo.CreateTodo(ctx, req, userID)
    if err != nil {
        // ERROR log with full context
        logger.Log.Error("failed to create todo",
            zap.String("user_id", userID),
            zap.String("error", err.Error()),
            zap.String("trace_id", span.SpanContext().TraceID().String()),
        )
        // Record error on span — shows as failed in Jaeger/Grafana Tempo
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return nil, fmt.Errorf("repo.CreateTodo: %w", err)
    }

    // 5. DEBUG log — only visible in dev, suppressed in production
    logger.Log.Debug("todo created",
        zap.String("todo_id", todo.ID),
    )

    // 6. INFO — business event completion log (for audit trail)
    logger.Log.Info("todo created successfully",
        zap.String("todo_id", todo.ID),
        zap.String("user_id", userID),
        zap.String("title", todo.Title),
        zap.String("priority", todo.Priority),
        zap.String("category_id", todo.CategoryID),
    )

    return todo, nil
}
Go / Zap + OTel

Prometheus Middleware (Metrics)

// middleware/metrics.go — HTTP metrics instrumentation
package middleware

import (
    "strconv"
    "time"
    "github.com/gin-gonic/gin"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    httpRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests",
    }, []string{"method", "path", "status"})

    httpDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request latency",
        Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5},
    }, []string{"method", "path"})
)

func PrometheusMetrics() gin.HandlerFunc {
    return func(c *gin.Context) {
        start := time.Now()
        c.Next()
        duration := time.Since(start)

        status := strconv.Itoa(c.Writer.Status())
        httpRequestsTotal.WithLabelValues(c.Request.Method, c.FullPath(), status).Inc()
        httpDuration.WithLabelValues(c.Request.Method, c.FullPath()).Observe(duration.Seconds())
    }
}
Go / Gin + Prometheus

Structured Logging with structlog

# logging_config.py — structlog setup for Python
import logging
import os
import structlog

def configure_logging():
    env = os.getenv("APP_ENV", "development")

    if env == "production":
        # JSON renderer for production — parseable by Loki / ELK
        renderer = structlog.processors.JSONRenderer()
        level = logging.INFO
    else:
        # ColourfulConsole for local dev — human-readable
        renderer = structlog.dev.ConsoleRenderer(colors=True)
        level = logging.DEBUG

    structlog.configure(
        processors=[
            structlog.contextvars.merge_contextvars,      # merge request-scoped context
            structlog.processors.TimeStamper(fmt="iso"),
            structlog.stdlib.add_log_level,
            structlog.stdlib.add_logger_name,
            structlog.processors.StackInfoRenderer(),
            structlog.processors.ExceptionRenderer(),     # auto-renders exceptions
            renderer,
        ],
        logger_factory=structlog.stdlib.LoggerFactory(),
        wrapper_class=structlog.stdlib.BoundLogger,
        cache_logger_on_first_use=True,
    )

    logging.basicConfig(level=level)

log = structlog.get_logger()
Python / structlog

OpenTelemetry Tracing in FastAPI

# tracing.py — OTel setup for FastAPI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

def setup_tracing(app):
    # Tracer provider — sends spans to OTel Collector via gRPC
    provider = TracerProvider()
    exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

    # Auto-instrument FastAPI — all routes get spans automatically
    FastAPIInstrumentor.instrument_app(app)

    # Auto-instrument SQLAlchemy — all DB queries get spans automatically
    SQLAlchemyInstrumentor().instrument(engine=engine)

# main.py — wire it all up
from fastapi import FastAPI
from tracing import setup_tracing
from logging_config import configure_logging, log

configure_logging()
app = FastAPI()
setup_tracing(app)
Python / FastAPI + OTel

Service Layer with Full LMO Pattern

# services/todo_service.py
from opentelemetry import trace
import structlog

tracer = trace.get_tracer("todo-service")
log = structlog.get_logger()

async def create_todo(user_id: str, req: CreateTodoRequest) -> Todo:
    # 1. Start child span
    with tracer.start_as_current_span("TodoService.create_todo") as span:
        span.set_attribute("user.id", user_id)
        span.set_attribute("todo.title", req.title)

        # 2. INFO log — business event start
        log.info("creating_todo", user_id=user_id, title=req.title)

        try:
            todo = await repo.create_todo(user_id, req)

            # 3. Business event log — audit trail
            log.info("todo_created",
                todo_id=todo.id,
                user_id=user_id,
                title=todo.title,
                priority=todo.priority,
            )
            return todo

        except Exception as e:
            # 4. ERROR log + span error recording
            log.error("todo_creation_failed", user_id=user_id, error=str(e))
            span.record_exception(e)
            span.set_status(trace.StatusCode.ERROR, str(e))
            raise
Python / FastAPI + OTel + structlog

The open-source observability stack is what most large enterprises run. More control, no vendor lock-in, no per-seat pricing — but requires operational expertise to maintain.

Full Stack Architecture

If you don't have the team to manage the open-source stack, proprietary all-in-one solutions are the pragmatic choice. They bundle logging, metrics, tracing, and alerting into a single platform with managed storage and a polished UI.

New Relic Go Agent — Quick Integration

// main.go — New Relic agent setup in Go
import (
    "github.com/newrelic/go-agent/v3/newrelic"
    "github.com/newrelic/go-agent/v3/integrations/nrgin"
)

func main() {
    // Initialize New Relic application agent
    nrApp, _ := newrelic.NewApplication(
        newrelic.ConfigAppName("todo-api"),
        newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")),
        newrelic.ConfigAppLogForwardingEnabled(true),  // forward logs to NR
        newrelic.ConfigDistributedTracerEnabled(true), // enable distributed tracing
    )

    r := gin.New()
    // nrgin middleware — auto-instruments every route with NR transactions
    r.Use(nrgin.Middleware(nrApp))

    r.Run(":8080")
}
Go / New Relic

The ELK Stack (now called the Elastic Stack) is the classic enterprise logging platform. It provides full-text search across logs, powerful aggregations, and rich dashboards. It's heavier than Loki but offers more querying flexibility.

ELK vs Grafana Loki

Not every metric needs an alert. Alert fatigue is a real problem — if engineers receive too many notifications, they start ignoring them. Good alert design means alerting on symptoms that impact users, not internal implementation details.

The Four Golden Signals (Google SRE Book)

Alert Rules (Grafana / Prometheus format)

# alert_rules.yml — Production-grade alert rules
groups:
  - name: api_health
    rules:

      # P1 — Error rate critical
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
          / rate(http_requests_total[5m]) * 100 > 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "HTTP error rate above 5% ({{ $value | printf \"%.1f\" }}%)"
          runbook: "https://wiki.company.com/runbooks/high-error-rate"

      # P2 — High latency
      - alert: HighP99Latency
        expr: |
          histogram_quantile(0.99,
            rate(http_request_duration_seconds_bucket[5m])
          ) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency above 2s ({{ $value | printf \"%.2f\" }}s)"

      # P2 — DB connection pool near exhaustion
      - alert: DBConnectionPoolExhaustion
        expr: db_connections_open / db_connections_max > 0.85
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "DB connection pool at {{ $value | humanizePercentage }}"

      # P1 — Service is down
      - alert: ServiceDown
        expr: up{job="todo-api"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "todo-api service is unreachable"
YAML / Prometheus Rules

Here is what the complete workflow looks like end-to-end when an incident occurs in a system with full LMO in place.

Scenario: API Suddenly Slow + High Errors

• 1
  🔔 Alert fires at 14:31 UTC

  Alertmanager sends Slack message: "[CRITICAL] HighErrorRate: HTTP error rate is 82.4% (was 0.3%). Service: todo-api. Runbook: wiki/…"

• 2
  📊 Open Grafana → Metrics dashboard

  P99 latency jumped from 120ms to 4.3s at 14:31. Requests/sec unchanged (not a traffic spike). DB connection pool at 97%. Error type: 100% are 500s, not 4xx. Indicates server-side DB problem.

• 3
  📋 Correlate to Logs in Loki / New Relic

  Query: {service="todo-api", level="error"} |= "connection". Hundreds of entries: ERROR: pq: remaining connection slots are reserved for non-replication superuser connections. DB connection pool is exhausted.

• 4
  🔗 Jump to Trace from log entry

  Click trace link in the log. Trace shows: HTTP Handler → AuthMiddleware (3ms) → TodoService (4290ms) → DB.GetTodos (4285ms ← STUCK). DB.GetTodos span has attribute: query="SELECT * FROM todos WHERE user_id=$1".

• 5
  🔍 Root cause identified

  A deployment 5 minutes ago added a SELECT * that performs a full table scan (missing index). Under load, queries pile up, connections are held longer, pool exhausts. Check git log — confirms deployment at 14:29 UTC.

• 6
  ✅ Fix + verify

  Add missing index migration. Deploy. Grafana shows error rate drops to 0.1% within 2 minutes of deployment. DB connection pool back to 12%. Incident resolved. Total time: 11 minutes.

1 · Always Include a Request ID

Generate a unique request_id (UUID or ULID) at the entry point of every request — either in your load balancer or in your first middleware. Inject it into the context and include it in every log entry and span for that request. This is what lets you pull all logs for a single failing request out of millions of entries.

2 · Log at Boundaries, Not Inside Logic

Log at the entry and exit of major components (handler, service, repository), not scattered throughout every loop and condition. This gives you a clean, readable trail without noise. Inside functions, use spans for granular timing — not log.Debug spam.

3 · Never Log Sensitive Data

Passwords, credit card numbers, SSNs, auth tokens, and API keys must never appear in logs. Use structured logging libraries that let you mark fields as redacted. In your CI pipeline, consider a log scanning step that fails the build if certain patterns are found in log statements.

4 · Use Sampling for High-Volume Traces

At high traffic (1000+ req/sec), recording every trace is expensive. Use head-based sampling (record 10% of all requests) or tail-based sampling (record 100% of error requests + 1% of success requests). The OTel Collector supports both. Never sample errors — always record traces for failures.

5 · Log Levels Discipline

• Never use DEBUG in production — it generates enormous volume and costs storage
• INFO should be your production default — meaningful events, not chatty
• Treat every ERROR log as something that needs a ticket or an alert rule
• FATAL should mean "I need someone paged right now"

6 · This is a Spectrum, Not a Checkbox

Start with basic structured logging in JSON format + a simple Prometheus counter for error rate. That alone will save you hours when the first production incident happens. Then iteratively add: histogram metrics, basic tracing, Grafana dashboards, alert rules. You don't need the full ELK + Jaeger + OTel Collector stack on day one. Build it incrementally.