Integration Patterns

Software systems don't work alone. Every B2B product you use connects to something — your CRM syncs with your marketing platform, your data warehouse feeds your BI tool, your payment processor talks to your accounting software, your hiring tool pushes records to your HRIS. When those connections work, they're invisible. When they break, they're the only thing anyone is talking about.

Integration is the plumbing of the modern business stack. Most operators think of it as engineering's problem — something you commission, then forget about. The reality is that integration decisions have meaningful business consequences: how fresh your data is, how reliably notifications reach customers, whether a system change requires three teams to coordinate or one, and whether the engineering team needs six weeks to map your integration landscape before they can change anything.

This guide covers the vocabulary and concepts a business operator needs to participate in integration decisions: how APIs work, what webhooks are and why they're different, when batch processing is the right choice versus streaming, how event-driven systems are built, what integration platforms do, how integration debt accumulates over time, and how data contracts keep integrations from breaking silently.

§1 APIs: the shared language of connected systems Foundational

Almost every software-to-software integration runs on an API — an Application Programming Interface. APIs are the agreed-upon contracts that let two systems communicate: here's what requests I accept, here's what I'll return, here's how to authenticate.

The anatomy of an API call. When your CRM sends a record to your email platform, it's making an HTTP request — the same protocol that powers the web. The request has a few components:

The method (verb): GET fetches something. POST creates something. PUT and PATCH update something. DELETE removes something. The verb declares the intent.

The endpoint (URL): the specific resource the request targets. https://api.example.com/contacts/12345 identifies the contact with ID 12345. The URL structure is the API's vocabulary.

Authentication: APIs don't trust any caller. The request includes credentials — an API key (a long secret string passed in a header), or a Bearer token from an OAuth flow that established trust at the session level. OAuth is the pattern behind "Sign in with Google" — a server authenticates once and issues a token the calling system uses for subsequent requests.

The body: for POST and PATCH requests, the data being sent — typically JSON (a structured text format both sides can read).

The response: the server returns a status code and a response body. Status codes communicate the outcome: 200 success, 201 created, 400 bad request, 401 unauthorized, 404 not found, 500 server error. Integration code that doesn't check status codes and handle errors gracefully will silently fail.

REST (Representational State Transfer) is the dominant API style in modern software. RESTful APIs use HTTP verbs to describe actions, URLs to describe resources, and JSON for data exchange. You'll also encounter GraphQL — a query language that lets the caller specify exactly what fields it wants, which reduces over-fetching. Older enterprise systems often use SOAP, which wraps requests in XML envelopes and is more verbose than REST.

Rate limits. Every API constrains how many requests a caller can make in a given window — typically expressed as requests per minute or per hour. When a caller exceeds the limit, the API returns a 429 Too Many Requests response. An integration that doesn't handle rate limits gracefully will fail under load — a common failure when an integration that works fine at low volume breaks after customer growth.

Related glossary: API, REST, API key, OAuth, rate limiting, JSON, endpoint

§2 Webhooks: when the API calls you Foundational

A standard API interaction is pull: your system asks another system for data. A webhook inverts this: the other system pushes data to your system the moment something happens.

The difference isn't just technical — it changes the latency and efficiency profile of the integration entirely.

The polling problem. Before webhooks were common, integrations worked by polling: your system would ask the external service every few minutes, "anything new?" Most of the time the answer was no, and the requests were wasted. A system checking for new orders every five minutes will always be up to five minutes behind, and it generates 288 API calls per day whether or not anything actually happened.

How webhooks work. When you configure a webhook, you register a URL on your server — your webhook endpoint. When a triggering event occurs (an order is placed, a payment processes, a form is submitted), the external system sends an HTTP POST request to your URL with the event data in the body. Your system receives it immediately — no polling required.

What gets sent is an event payload — a JSON object describing what happened. For an order webhook: {"event": "order.created", "order_id": "78234", "amount": 149.00, "timestamp": "2026-06-01T14:22:31Z"}. Your system reads the payload and decides what to do.

Reliability and idempotency. Webhooks introduce failure modes that polling doesn't have. If your server is down when the webhook fires, the delivery fails. Most providers retry with exponential backoff — wait 1 minute, then 5 minutes, then 15 minutes — but retries can deliver the same event more than once. Idempotency is the property that processing the same event twice produces the same result as processing it once. A well-built webhook handler checks "have I already processed this event ID?" before acting. Without it, a double-delivery can create a duplicate order, a duplicate charge, or a duplicate notification.

Signature verification. A webhook endpoint is a public URL — anyone who knows it could send a fake event. Webhook providers sign their payloads with a cryptographic signature (typically an HMAC) generated from the payload and a shared secret. Your handler verifies the signature before processing. Skip this step and your system will process spoofed events.

Related glossary: webhook, event, idempotency, payload, HMAC, polling

§3 Batch vs streaming: choosing the right latency Building

Not every integration needs to move data the moment it exists. The choice between batch processing and streaming is a latency-versus-cost trade-off, and the right answer depends on how stale your data can afford to be.

The latency spectrum. Think of data freshness on a spectrum:

Scheduled batch — data moves on a fixed schedule: daily, hourly, every 30 minutes. The ETL job runs at 2am and loads yesterday's records into the warehouse. Simplest to build, lowest operational complexity, highest latency. Right for analytics that don't require intraday freshness — finance reporting, weekly business reviews, model training runs.

Micro-batch — batches run frequently: every 5, 10, or 15 minutes. Looks like near-real-time from the user's perspective but is fundamentally still batch processing. Latency is minutes; operational complexity is moderate.

Near-real-time streaming — data moves within seconds of being created. Events flow continuously; the processing layer consumes them as they arrive. Right for fraud detection, live dashboards, alerting pipelines.

Real-time / low-latency streaming — sub-second latency. Required for high-frequency trading, live bidding systems, real-time gaming. Very complex to build and operate; most business applications don't need it.

When batch is right. Historical reporting, financial close, marketing attribution, most BI dashboards, data science model training. These use cases can all afford to be hours or a day old. Batch is simpler, cheaper, and more auditable — a batch job either completes or fails; a streaming pipeline can silently fall behind without triggering an error.

When streaming is right. Fraud detection (the decision must happen before the transaction clears), recommendation engines on live user sessions (what the user saw in the last 10 seconds matters), operational alerting (you need to know a server is down now, not tomorrow), and any customer-facing feature where latency is part of the product.

The infrastructure cost difference. Batch pipelines run on scheduled compute — a job fires, loads data, terminates. Streaming pipelines run continuously. Compute costs are constant rather than occasional. Managing streaming infrastructure requires thinking about consumer groups, partition management, offset tracking, backpressure, and recovery from failures — none of which appear in a daily batch job.

Related glossary: batch processing, streaming, ETL, ELT, latency, event-driven

§4 Event-driven architecture: systems that talk without being called Building

In a point-to-point integration, System A calls System B directly. The caller knows who the receiver is; both sides are tightly coupled. If System B is slow, System A waits. If System B is down, System A fails.

Event-driven architecture solves this coupling problem by introducing a message broker in the middle. System A publishes an event — "something happened" — and doesn't know or care who receives it. System B, C, and D all subscribe to that event type and process it independently. Neither side knows the other exists; the broker is the only shared dependency.

The three components. An event-driven system has:

Producers (publishers) — systems that emit events when something happens. An order service emits order.created. A payment service emits payment.processed. The producer's job is to emit the event and move on — not to know what happens next.

The broker / queue / topic — the infrastructure that receives events from producers and delivers them to consumers. Apache Kafka is the dominant open-source event streaming platform; Amazon SQS is the AWS-managed queue; Azure Service Bus and Google Pub/Sub are cloud-native equivalents. The broker provides persistence, delivery guarantees, and fan-out.

Consumers (subscribers) — services that process specific event types. A fulfillment service subscribes to order.created and kicks off the pick-and-pack workflow. A notification service subscribes to the same event and sends a confirmation email. Both run independently; neither knows about the other.

Fan-out. The power of this pattern is that one event can trigger multiple downstream processes simultaneously without the producer knowing about any of them. When a new customer signs up, a single user.created event can simultaneously kick off the onboarding email, the CRM record creation, the analytics tracking event, and the provisioning workflow — four systems, one event, zero coupling between them.

Dead letter queues. Events that fail processing need somewhere to go that isn't "lost." A dead letter queue (DLQ) is the holding area for failed messages. Operations teams inspect the DLQ to understand what failed and why, then replay events after the problem is fixed. Without a DLQ, failed events vanish silently — and the failure only surfaces when someone notices the downstream system is out of sync.

At-least-once delivery. Most message brokers guarantee that events will be delivered at least once — but not exactly once. Which means consumers need to be idempotent: processing the same event twice should produce the same result as processing it once. This is a design discipline, not something the infrastructure enforces.

Related glossary: event-driven architecture, message broker, dead letter queue, Kafka, pub/sub, consumer, producer

§5 Integration platforms: the no-code middle layer Building

Most integrations don't require engineers to write code from scratch. A whole category of software — integration platforms as a service (iPaaS) — exists to wire systems together through configuration and low-code interfaces. Understanding when these platforms are the right tool (and when they aren't) is useful for any operator who's been told "the engineering team will get to that in Q3."

The spectrum. Integration tooling spans a wide range:

At the citizen integrator tier: Zapier, Make (formerly Integromat), and n8n. Designed for non-engineers — connect App A to App B, trigger on this event, do this action. Fast to set up, low cost, limited complexity. Zapier handles simple trigger-action flows well. The limitation: they manage simple linear flows and struggle with complex multi-step conditional logic, custom authentication patterns, or high-volume workloads.

At the business automation tier: Workato, Tray.io. These handle more complex logic: branching workflows, error handling, data transformation, loops, API calls with custom auth schemes. More powerful than citizen-tier tools; still primarily no-code. Used by operations and RevOps teams to build integrations that would otherwise require engineering.

At the enterprise iPaaS tier: MuleSoft (Salesforce), Informatica, IBM App Connect. These govern integration at scale — managing dozens or hundreds of integrations, providing monitoring, versioning, security controls, and SLA tracking. Complex to set up, expensive to license, powerful at scale. An IT team runs them.

The custom-build trade-off. Every integration platform adds a vendor dependency and a cost floor. The alternative — custom-built integration code — gives full control, no per-task pricing, and no intermediary. The cost: engineering time to build, test, maintain, and debug. The right frame (from §3 of FinOps & Unit Economics): is this integration a source of differentiation or a commodity requirement? Custom code is right when the integration requires complex business logic the platform can't express, when performance requirements exceed what the platform can deliver, or when data sensitivity makes routing through a third-party service a compliance problem.

Related glossary: iPaaS, Zapier, integration platform, workflow automation, no-code, low-code

§6 Integration topology: when the plumbing becomes the problem Strategic

One integration is manageable. Ten is still manageable. Fifty integrations built over five years by different teams using different patterns and tools is something else — it's a liability that slows every system change and introduces failure modes that are difficult to trace.

The point-to-point anti-pattern. The natural way to build integrations is one at a time: connect System A to System B, then System A to System C, then System B to System D. This produces a point-to-point mesh — every system has direct connections to multiple others. With five systems, that's up to 10 direct connections. With ten systems, 45. The math is N(N-1)/2: at 15 systems, you have 105 connections. Each connection is a dependency: when System B changes its API, every system connected directly to System B needs updating.

Hub-and-spoke. One solution to point-to-point complexity is a central integration hub — all systems connect to the hub, and the hub routes messages between them. The hub handles format translation in one place. When a system's API changes, only the hub-to-system connection needs updating. iPaaS platforms in §5 are often deployed as hub-and-spoke: the platform is the hub. The limitation: the hub becomes a single point of failure and a performance bottleneck at high volume.

Event mesh. The mature alternative is the event-driven architecture from §4: systems publish and subscribe to events; no central router is required. The event bus is the only shared infrastructure. Systems are coupled only to event schemas — not to each other. Adding a new system means publishing new events and subscribing to existing ones without touching any existing integration code.

Integration debt. Integration debt accumulates like technical debt — invisibly until it becomes the dominant engineering risk. Point-to-point that seems manageable at 10 integrations becomes a liability at 50. Symptoms: routine system changes require "what else connects to this?" archaeology, unexplained data inconsistencies trace back to a deprecated integration nobody knew was still running, new integrations require weeks of discovery before any code is written.

Governance at scale also includes: an API versioning policy (when a new API version ships, how long does the old version stay live?), observability on the integrations themselves (latency, error rate, volume per integration — not just the systems they connect), and deprecation discipline (integration code for systems that no longer exist is a maintenance burden that can be mistaken for live traffic).

Related glossary: integration mesh, hub-and-spoke, event bus, API versioning, integration debt, observability

What to remember from this guide

Integration is the plumbing that makes the modern business stack work — and like plumbing, the time to design it well is before the pipes are in. APIs are the request/response contracts that let systems communicate; understanding their anatomy (verb, endpoint, auth, status codes) helps you ask better questions when engineering describes an integration. Webhooks are the event-driven inverse of polling — they deliver data when it happens, not when you ask, and they require idempotency to handle retries safely. The batch-vs-streaming decision is a latency-versus-cost trade-off: real-time infrastructure is expensive and complex, and the right tier depends on how stale the data can actually afford to be. Event-driven architecture with message brokers decouples producers from consumers and makes fan-out and independent scaling possible. iPaaS platforms exist on a spectrum from citizen-tier no-code tools to enterprise governance platforms — the right tier depends on who's building it, how complex the logic is, and how business-critical the integration is. And integration topology is something that needs deliberate attention before the point-to-point mesh becomes the dominant engineering risk. The catalog is the cheapest insurance you'll ever buy.

Related guides: Data Foundations — §1 (ETL/ELT), §5 (modern data stack) for the pipelines that run on integration patterns; Decision Infrastructure — §4 (alerts) and §5 (ML loops) for downstream uses of event-driven data; FinOps & Unit Economics — §4 (vendor lock-in) for the API/data lock-in risks created by integration choices

§7 Data contracts and schema governance Strategic

A data contract is a formal agreement between the team that produces data (the producer) and the team that consumes it (the consumer) defining exactly what the data will look like: which fields exist, what types they carry, what values are valid, how fresh the data will be, and how breaking changes will be communicated before they happen. Without this agreement, schema drift — the silent mutation of column names, types, or structure as the producing system evolves — becomes a chronic source of downstream failures.

The cost of schema drift compounds quietly. A column rename is a one-line change in the source system. Downstream, it might break three transformation jobs, invalidate a dashboard that business stakeholders review weekly, corrupt a ML feature pipeline, and cause a customer-facing alert to stop firing — none of which the engineer who renamed the column knew about or intended. By the time the breakage surfaces, the connection to the root cause is often unclear, and the debugging time dwarfs the time cost of having a contract in place. This is not a pathological scenario; it is the normal condition in organizations that haven't invested in schema governance.

Schema registries enforce contracts at the infrastructure layer. When a producer publishes a message or writes to a shared table, the schema registry validates the payload against the declared contract. If the change is backward-compatible (adding an optional field), it passes. If it's breaking (removing a field, changing a type, renaming a column), the registry rejects it and notifies the producer. Tools like Confluent Schema Registry (for Kafka-based event streams), Apache Avro and Protobuf (for serialization with embedded schema evolution rules), and dbt contracts (which enforce column-level type and constraint expectations in SQL transformations) implement this pattern at different layers of the stack. The goal is the same: make breaking changes impossible to ship silently.

The data mesh framing connects contracts to organizational design. In a data mesh architecture, domain teams own their data products end-to-end — including the quality, freshness, and schema stability of the data they publish. A contract is what makes a "data product" a product rather than just a table: it's the interface contract that allows consumers to build dependably on top of it. Platform engineering teams often provide the tooling layer (the registry, the contract validation CI, the schema catalog), but the contractual accountability lives with the domain teams who own the data.

Without contract