Backend from First PrinciplesBFP

Chapter 052-3 hours

Validations & Transformations.

A first-principles walkthrough of the rules and guidelines you keep in mind while designing APIs, the gate every piece of client data passes through before any business logic runs. Why it exists, where it lives, how it works underneath, with complete implementations in Go and Python shown side by side.

Part I / The Mental Model

What They Are & Why They Exist

Validations and transformations are not a single feature you install. They are a set of rules and guidelines you keep in mind while designing your APIs. Almost everything about them traces back to two goals, and those two goals are the lens through which every later decision is made:

Goal 01Data Integrity

Whatever data finally reaches your business logic and your database is in the exact shape your application expects, correct fields, correct types, correct ranges, correct meaning.

Goal 02Security

Untrusted input from the outside world is never allowed to drive your system into an unexpected, unhandled state. The server stays defensive about everything that crosses its boundary.

What counts as “incoming client data”

The validation boundary covers every kind of data a client can push at the server, not just the request body. All of the following must be treated as untrusted until proven otherwise:

  • JSON payloads: the body of a POST / PUT / PATCH request.
  • Query parameters: e.g. ?page=2&limit=20.
  • Path parameters: e.g. the 5 in /users/5.
  • Headers: anything the client sets on the request.

Where It Lives: The Three Layers

To pinpoint where validation happens you first need the standard mental model of a backend: three stacked layers, each with one job. A request travels down the stack and the response travels back up.

BottomRepository Layer

Deals only with the database, connections, queries, inserts, updates, deletes. It does not care why a row is being written, only that it is. Works against a relational DB (Postgres), Redis, or any other persistent store.

MiddleService Layer

The business logic. A single service method may call one or more repository methods, send push notifications, email users, store data, fire a webhook, everything an API is expected to actually do.

TopController Layer

The HTTP-facing entry point. It owns everything HTTP: which status code to return, the response format, and, crucially, the validations. It receives client data, calls the matching service method, and returns whatever that method gives back.

The reason controller and service are deliberately separated is to keep HTTP concerns (error codes, success codes, response shape, validation) in one place, and pure business logic in another. The controller calls the service; the service may or may not call the repository, depending on what the request needs.

CLIENTJSON / query / path / headersCONTROLLER LAYERHTTP concerns / status codes / response shape> VALIDATION & TRANSFORMATION live here <the only layer that touches raw client dataSERVICE LAYERbusiness logic / email / notificationswebhooks / orchestrationREPOSITORY LAYERDB connections / queries / insert / update / deletePostgres / Rediscalls service methodmay call repository

The standard stack, validation is pinned to the top of the controller

The Request Lifecycle

A request does not magically appear inside your controller method. It arrives, gets route-matched, and only then is the controller method assigned to that route invoked. That ordering is the key to placing validation correctly.

  1. Data arrives. A client opens an HTTP connection and sends a request, typically a JSON payload, plus query and path parameters and headers.
  2. Route matching runs. The router compares the incoming path against its patterns and selects the controller method registered for the matched route.
  3. Validation & transformation pipeline fires. Before any significant logic runs, the matched controller passes the raw data through the pipeline.
  4. Business logic runs. Only if the data passed does the controller call the service method.
  5. Service -> repository. The service executes logic, optionally calling repository methods that hit the database.
  6. Response travels back up. Whatever the service returns is handed to the controller, which returns it to the caller over the HTTP connection.
CLIENTsends requestROUTEMATCHINGVALIDATION &TRANSFORMATIONthe gate, runs firstCONTROLLERbusiness pathSERVICE-> repositoryon failure -> 400 Bad Request, never reaches service

The gate sits between route matching and any business logic

The Execution Point

Validation and transformation happen at exactly one point: inside the controller layer, immediately after the route is matched and before any service method is called or any business logic runs. Concretely it is implemented as a reusable middleware or utility function that you hand a schema: the description of what the data should look like, and it walks every field of the incoming payload against that schema.

Why It Is Critical: The 500 vs 400 Story

The clearest way to feel the need for validation is to watch what happens without it. Imagine a “create book” API that expects a JSON body with a name field that should be a string. A client instead sends the number 0:

POST /api/books

Request body (malformed)

{ "name": 0 }

In your Postgres schema the column was almost certainly declared like this, a string column that cannot be null:

schema.sqlSQL

CREATE TABLE books (
  id    SERIAL PRIMARY KEY,
  name  TEXT NOT NULL   -- expects text, refuses an integer
);

With no validation, that 0 sails through the controller, through the service, down into the repository, and becomes an INSERT. Postgres checks the column type, sees a number where it demanded TEXT, and rejects the operation. The database call fails, and the client receives:

Response without validation 500 INTERNAL SERVER ERROR

“Something went wrong on the server”

{ "error": "Internal Server Error" }

That is a poor user experience. A 500 says “something unexpected broke inside us” when in reality the client sent bad input, and gives them nothing to fix. With a validation pipeline at the entry point, the server catches the mistake instantly, the database is never touched, and the client gets an actionable 400 Bad Request:

Response with validation 400 BAD REQUEST

Actionable, returned before any DB call

{ "errors": [ "name: expected a string, received a number" ] }
PATH A / NO VALIDATION{name:0}controllerservicerepositoryDB REJECTStype mismatch500 to client”something broke” / unhelpfulPATH B / WITH VALIDATION AT THE GATE{name:0}VALIDATIONexpected string, got number400 to client (instant)exactly what to fixservice / DBnever touched

Same bad input, two outcomes, 500 disaster vs a clean 400 that never wakes the database

Inside the Pipeline: Step by Step

A robust pipeline processes a field in layers, and the order matters: each check only runs if the previous one passed. Take a single field, name, required to be a string of length 5-100:

  1. Existence check. Does name exist in the payload at all? If not -> "name is required", stop.
  2. Type check. It exists, is it a string, or did the client send an array / boolean / number? If wrong type -> "expected a string", stop.
  3. Constraint check. It is a string, does it satisfy the restrictions? Length 5-100. A 2-character string or a whole paragraph both fail with a tailored message.
incoming field “name”1 / EXISTENCE CHECK2 / TYPE CHECK3 / CONSTRAINT CHECKPASS -> controller”name is required”400”expected a string”400”length must be 5-100”400

Each gate only opens if the previous one passed, failure short-circuits to a 400

In code, libraries express this as a declarative schema. Go commonly uses struct tags with go-playground/validator; Python uses Pydantic, where the same model both validates and transforms. Here is a complete, runnable version of the pipeline in each:

package books

import (
    "encoding/json"
    "fmt"
    "net/http"
    "strings"

    "github.com/go-playground/validator/v10"
)

// one shared, reusable validator (the pipeline engine)
var validate = validator.New()

// CreateBook is the SCHEMA. The tags declare every rule:
//   required      -> existence check
//   the Go type   -> type check (string)
//   min=5,max=100 -> constraint check
type CreateBook struct {
    Name string `json:"name" validate:"required,min=5,max=100"`
}

// runPipeline decodes then validates; returns 400-ready messages.
func runPipeline(r *http.Request) (*CreateBook, []string) {
    var body CreateBook

    // 1. TYPE CHECK during decode: a JSON number for `name`
    //    cannot unmarshal into a Go string -> caught here.
    if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
        return nil, []string{"name: expected a string"}
    }
    // 2. EXISTENCE + CONSTRAINT checks in one pass.
    if err := validate.Struct(body); err != nil {
        return nil, formatErrors(err)
    }
    return &body, nil
}

// formatErrors turns validator output into human messages.
func formatErrors(err error) []string {
    var msgs []string
    for _, e := range err.(validator.ValidationErrors) {
        f := strings.ToLower(e.Field())
        switch e.Tag() {
        case "required":
            msgs = append(msgs, fmt.Sprintf("%s: is required", f))
        case "min":
            msgs = append(msgs, fmt.Sprintf("%s: min %s chars", f, e.Param()))
        case "max":
            msgs = append(msgs, fmt.Sprintf("%s: max %s chars", f, e.Param()))
        default:
            msgs = append(msgs, fmt.Sprintf("%s: failed %s", f, e.Tag()))
        }
    }
    return msgs
}

Part II / The Four Types of Validation

These are not a fixed taxonomy you must memorize, there can be more depending on requirements, but in practice four flavors cover almost everything you will meet while designing APIs. The real takeaway is a question you keep asking yourself: how strict and how specific do I want to be with this data?

Type Validation

The most basic flavor: does the field match the data type the API expects? Is it a string, a number, a boolean, an array, or a nested JSON object, and not something else? It can also apply recursively: an array field may require that every element is a string, so element 0 being a number is itself a type error.

POST /api/valid/type

Wrong types sent

{ "stringField":"x", "numberField":"x", "arrayField":"x", "boolField":"x" }

Response 400

numberField: expected number, received string
arrayField:  expected array,  received string
boolField:   expected boolean, received string

Then an array of numbers where strings are required

{ "arrayField": [1, 2] }  ->  arrayField[0]: expected string, received number

Fully corrected -> 200

{ "stringField":"something", "numberField":10, "arrayField":["one","two"], "boolField":false }
package validate

import (
    "encoding/json"
    "errors"
    "net/http"
)

// *bool so we can tell "false" apart from "missing".
type TypePayload struct {
    StringField string   `json:"stringField" validate:"required"`
    NumberField float64  `json:"numberField" validate:"required"`
    // dive = recurse INTO the slice; each element required
    ArrayField  []string `json:"arrayField" validate:"required,dive,required"`
    BoolField   *bool    `json:"boolField" validate:"required"`
}

func parseTypes(r *http.Request) (*TypePayload, error) {
    var p TypePayload
    dec := json.NewDecoder(r.Body)
    dec.DisallowUnknownFields() // reject stray keys

    // json.Decode enforces the BASE types: a string for
    // numberField / arrayField / boolField fails to unmarshal.
    if err := dec.Decode(&p); err != nil {
        return nil, errors.New("a field has the wrong data type")
    }
    // existence + recursive element check
    if err := validate.Struct(p); err != nil {
        return nil, err
    }
    return &p, nil
}

// {"numberField":"x"} -> expected number, received string
// {"arrayField":[1,2]} -> arrayField[0]: expected string

Syntactic Validation

Here the type is already correct (a string) but the string must follow a specific structural pattern. The validation algorithm checks the shape of the value:

  • Email: a local part, then the @ character, then a domain made of a name plus a top-level domain (.com, .co.in, .org, …). There are thousands of valid TLDs; the structure is what gets checked.
  • Phone number: a country code followed by the right number of digits for that country (e.g. a country code then a 10-digit number).
  • Date: must match the expected structure, e.g. YYYY-MM-DD (year, then month, then day).

POST /api/valid/syntactic

Empty body

{ }

Response 400, the errors double as live documentation

email: required   phone: required   date: required

Bad values

{ "email":"randomstring", "phone":1234567, "date":"2025-11-05" }

Response 400

email: invalid email format
phone: expected a string, received a number

Corrected

{ "email":"test@test.com", "phone":"1234567", "date":"2025-11-05" }  ->  200
package validate

import (
    "errors"
    "regexp"
)

// optional + then 7-15 digits (country code + national no.)
var phoneRe = regexp.MustCompile(`^\+?[0-9]{7,15}$`)

type Contact struct {
    Email string `validate:"required,email"`            // local @ domain.tld
    Phone string `validate:"required"`                  // checked below
    Date  string `validate:"required,datetime=2006-01-02"` // YYYY-MM-DD
}

// checkSyntax handles the phone pattern (validator's
// built-ins cover email + date structure already).
func (c Contact) checkSyntax() error {
    if !phoneRe.MatchString(c.Phone) {
        return errors.New("phone: invalid phone number format")
    }
    return nil
}

// "randomstring"            -> email: invalid email format
// phone sent as JSON number -> received a number, want string
// "2025-13-40"              -> date: does not match YYYY-MM-DD

Semantic Validation

The type is right and the syntax is right, but does the value make sense in the real world? Semantic checks encode domain logic:

  • A date of birth cannot be in the future, if today is 2025-01-11, a DOB of 2025-01-13 is nonsense even though it is a perfectly valid date.
  • An age of 365 (or 430) does not make sense, at least not yet. A reasonable human age range is roughly 1 to 120.

POST /api/valid/semantic

Future DOB

{ "dateOfBirth":"2026-06-12", "age":43 }

Response 400

dateOfBirth: date of birth cannot be in the future

Impossible age

{ "dateOfBirth":"1995-06-12", "age":430 }

Response 400

age: number must be less than or equal to 120

Sensible

{ "dateOfBirth":"1995-06-12", "age":43 }  ->  200
package validate

import (
    "errors"
    "time"
)

type Profile struct {
    DateOfBirth string `validate:"required,datetime=2006-01-02"`
    // gte/lte cover the "430 is impossible" semantic bound
    Age int `validate:"required,gte=1,lte=120"`
}

// Type & syntax can't express "not in the future"
// semantics need real logic against the real clock.
func (p Profile) checkSemantics() error {
    dob, err := time.Parse("2006-01-02", p.DateOfBirth)
    if err != nil {
        return errors.New("dateOfBirth: invalid date")
    }
    if dob.After(time.Now()) {
        return errors.New(
            "dateOfBirth: date of birth cannot be in the future")
    }
    return nil
}

// {"dateOfBirth":"2026-06-12"} -> cannot be in the future
// {"age":430}                  -> age: must be 120 or less

Complex / Dependent Validation

The most powerful flavor: a field’s rules depend on other fields. The pipeline can encode arbitrary cross-field logic:

  • Password confirmation: passwordConfirmation must exactly match password; and password must be at least 8 characters. Two different strings yield “passwords don’t match”; a 6-character password yields “string must contain at least 8 characters”.
  • Conditional requirement: partner is optional when married is false, but the moment married is true the partner name becomes required: “partner name is required when married is true”.

POST /api/valid/complex

Mismatched + short password

{ "password":"random", "passwordConfirmation":"another", "married":false }

Response 400

password: string must contain at least 8 characters
passwordConfirmation: passwords don't match

married=true but no partner

{ "password":"random12", "passwordConfirmation":"random12", "married":true }

Response 400

partner: partner name is required when married is true

Complete

{ "password":"random12", "passwordConfirmation":"random12", "married":true, "partner":"Sam" }  ->  200
package validate

type Signup struct {
    Password string `json:"password" validate:"required,min=8"`
    // eqfield: must equal another field on the struct
    PasswordConf string `json:"passwordConfirmation" \
        validate:"required,eqfield=Password"`
    Married *bool `json:"married" validate:"required"`
    // required_if: partner required only if married == true
    Partner string `json:"partner" \
        validate:"required_if=Married true"`
}

// {password:"random", passwordConfirmation:"another",
//  married:false}
//   -> password: must be at least 8 characters
//   -> passwordConfirmation: passwords don't match
//
// {password:"random12", passwordConfirmation:"random12",
//  married:true}
//   -> partner: required when married is true
THE FOUR FLAVORS, INCREASING STRICTNESS & CONTEXT ->TYPEis it the rightdata type?string / numberbool / arraySYNTACTICright structuralpattern?email / phonedate formatSEMANTICdoes the valuemake sense?DOB not futureage <= 120COMPLEXdepends onother fields?password matchmarried->partner

From “right type” to “right relationship”, pick how strict the endpoint needs to be

Part III / Transformation

Transformation as Type Casting

Transformation means executing operations on the incoming data to convert it into a desirable format, either before validation (so it can pass) or after it (to ready it for the service layer). The textbook case is pagination query parameters.

Consider GET /bookmarks?page=2&limit=20 with these requirements:

  • page, a number, greater than 0 and less than 500.
  • limit, a number, greater than 0 and less than 10,000 (at most 10,000 records returned at a time).

The fix is not to error. It is the server’s responsibility to cast the string into a number first. “Casting” is forcing one data type to become another. Cast "2" -> 2, then run the numeric validations. That casting step is the transformation.

?page=“2”arrives as STRINGTRANSFORMcast “2” -> 2(force the type)page = 2now a NUMBERVALIDATE0 < n < 500

Transformation runs before validation here, cast the string, then the numeric rules can even apply

package validate

import (
    "errors"
    "net/url"
    "strconv"
)

type Pagination struct {
    Page  int
    Limit int
}

// Query params are ALWAYS strings, so we must CAST
// (transform) before we can VALIDATE the numbers.
func parsePagination(q url.Values) (*Pagination, error) {
    // TRANSFORM: force the string "2" into the int 2
    page, err := strconv.Atoi(q.Get("page"))
    if err != nil {
        return nil, errors.New("page: must be a number")
    }
    limit, err := strconv.Atoi(q.Get("limit"))
    if err != nil {
        return nil, errors.New("limit: must be a number")
    }

    // VALIDATE the now-numeric values
    if page <= 0 || page >= 500 {
        return nil, errors.New("page: must be 1..499")
    }
    if limit <= 0 || limit >= 10000 {
        return nil, errors.New("limit: must be 1..9999")
    }
    return &Pagination{Page: page, Limit: limit}, nil
}

Transformation as Normalization

Transformation also cleans up data after it validates, reshaping it into what the service layer prefers. The server quietly normalizes the payload before doing anything with it:

  • An email sent with mixed case, Test@TEST.com, comes back stored as all-lowercase test@test.com.
  • A phone string gets a + prefix injected before it.
  • A date gets recomputed into the canonical format the backend expects.

POST /api/valid/transform

What the client sent

{ "email":"Test@TEST.com", "phone":"1234567", "date":"2025-11-05" }

What the server returns 200, normalized

{ "email":"test@test.com", "phone":"+1234567", "date":"2025-11-05" }
package validate

import "strings"

// normalize runs AFTER validation passes and BEFORE
// the data is handed to the service layer.
func (c *Contact) normalize() {
    // lowercase + trim the email
    c.Email = strings.ToLower(strings.TrimSpace(c.Email))

    // inject the leading + if it is missing
    c.Phone = strings.TrimSpace(c.Phone)
    if !strings.HasPrefix(c.Phone, "+") {
        c.Phone = "+" + c.Phone
    }
}

// "Test@TEST.com" -> "test@test.com"
// "1234567"       -> "+1234567"

One Combined Pipeline

In practice validations and transformations are paired into a single pipeline. The reason is locality: the entire “input data layer”, every requirement and every operation performed on incoming data before any business logic runs, lives in one place. You never have to hunt across files to discover what an endpoint expects or what it quietly does to the data. The pipeline can transform -> validate, or validate -> transform, in either order, as the requirements demand.

VALIDATION & TRANSFORMATION PIPELINE, one placetransformvalidatetransformcast query stringstype / syntax / semantic / complexlowercase / prefix / reformat

All input-layer logic, requirements and operations, collected in a single reusable middleware

Part IV / The Golden Rule

Frontend vs Backend Validation

A common and dangerous mistake is treating frontend validation as a replacement for backend validation. They are not interchangeable, they serve different purposes, and you need both for every API.

FrontendFor UX

A web form validates as the user types and on submit, giving immediate, friendly feedback (“that doesn’t look like an email”) before any API call is made. Its entire job is user experience. It is not about security or data integrity.

BackendFor Security

Server-side validation is mandatory and exists for security and data integrity. It does not matter what the client is or whether the client validated at all, the server must be as strict and specific as possible on its own.

Why can’t the backend trust the frontend? Because a server can have many different clients. A polished web app might validate beautifully, but the same API can also be hit directly through an API client like Postman or Insomnia, where there is no frontend acting as a proxy and therefore no frontend validation at all. If the backend ever depended on the frontend for security or integrity, the server would break the moment the client changed.

WEB APPhas FE validation (UX)POSTMAN / INSOMNIANO FE validation at allMOBILE / 3RD PARTYvalidation unknownSERVERBACKEND VALIDATIONstrict / specific / mandatorysecurity & data integrity, for ALL clients

One server, many clients, only the backend gate is guaranteed to run, so it must never be skippable

When the two are wired up correctly they complement each other: in a web form, an invalid email blocks the submit and shows an inline error, no API call is even made. Fix it to a real address and the submit fires a proper request; the payload is sent, the server validates again, and returns 200. The frontend delivered instant feedback; the backend delivered the guarantee.

Full Annotated Controller

Putting the whole picture together: a complete controller that route-matches, runs the combined pipeline (transform -> validate -> normalize) before any business logic, returns a clean 400 on failure, and only then calls the service layer, which calls the repository. Both ends shown in full.

package main

import (
    "context"
    "encoding/json"
    "net/http"
    "strings"

    "github.com/go-playground/validator/v10"
)

var validate = validator.New()

// ---- schema (the gate) ----
type CreateBook struct {
    Name string `json:"name" validate:"required,min=5,max=100"`
}

type Book struct {
    ID   int    `json:"id"`
    Name string `json:"name"`
}

// ---- helper ----
func writeJSON(w http.ResponseWriter, status int, b any) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(status)
    json.NewEncoder(w).Encode(b)
}

// ---- CONTROLLER: owns HTTP + the validation gate ----
func createBookHandler(w http.ResponseWriter, r *http.Request) {
    var body CreateBook

    // === GATE, runs before ANY business logic ===
    if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
        writeJSON(w, 400, map[string]string{
            "error": "name: expected a string"})
        return // 400, the DB is never touched
    }
    body.Name = strings.TrimSpace(body.Name) // transform
    if err := validate.Struct(body); err != nil {
        writeJSON(w, 400, map[string]string{
            "error": err.Error()})
        return // 400 Bad Request, not a confusing 500
    }

    // === only now: business logic (service -> repo) ===
    book, err := createBook(r.Context(), body.Name)
    if err != nil {
        writeJSON(w, 500, map[string]string{
            "error": "could not create book"})
        return
    }
    writeJSON(w, 201, book)
}

// ---- service + repository (sketched) ----
func createBook(ctx context.Context, name string) (*Book, error) {
    // service logic ... repository INSERT ... then:
    return &Book{ID: 1, Name: name}, nil
}

func main() {
    http.HandleFunc("/api/books", createBookHandler)
    http.ListenAndServe(":8080", nil)
}

Backend from First Principles / Chapter 05 / Validation. Code targets Go 1.22+ and Python 3.11+.