Observability and Cost

You enter a cloud contract

In production, serverless’s defining trait is scalability: managed services auto-scale from zero to peak with pay-per-use billing. But you enter a cloud contract — AWS provides autoscaling and HA while you must operate within service limits and know your units of scale: Lambda = concurrency, Kinesis = buffer limits, DynamoDB = read/write capacity, API Gateway = response latency. Annotate architecture diagrams with expected usage and limits. As systems shift from custom function logic toward application integration, bugs move from code to operations — and “there is no substitute for production.”

Metrics, logs, and traces

Observability is a sociotechnical practice, not a tool — inferring internal state from external outputs via three signals:

graph TD
S["Serverless system"] --> M["Metrics<br/>(CloudWatch: errors,<br/>throttles, duration)"]
S --> L["Logs<br/>(event-driven, structured)"]
S --> T["Traces<br/>(X-Ray: segments,<br/>annotations)"]
M --> AL["Alarm (metric + threshold)<br/>use percentages"]
AL --> AT["Alert: Obvious,<br/>Actionable, Singular"]
T --> WW["Decouple what is wrong<br/>from why"]

• Metrics → Amazon CloudWatch. An alarm is a metric plus threshold (use percentages for errors to absorb spikes); an alert is an action when an alarm fires — make alerts Obvious, Actionable, Singular. Build a critical health dashboard with the RED method (Rate, Errors, Duration) and use composite alarms for capability-based alerting instead of noisy per-resource alarms.
• Logs. Use event-driven logging with a standard envelope (e.g., CloudEvents). Logs are overused — they add cost and security risk.
• Traces → AWS X-Ray. Prefer traces to logs: logs tell what, traces tell where. Components: segments, subsegments, indexed annotations (≤50/trace), non-indexed metadata, and correlation IDs tying traces across systems. Use sampling (100% at launch, reduced over time).

Define SLIs (binary good/bad), SLOs (targets, e.g., 99.98% within 6s), and error budgets — the flip side of SLOs that grant permission to ship. SLO-based alerts answer what is wrong while you investigate why.

The bill mirrors the architecture

Pay-per-use must be approached “with caution as well as optimism” — costs scale down and up, but inefficiencies are punished with disproportionate bills at scale. Three cost factors: compute, storage, and outbound data transfer. The broader lens is total cost of ownership (TCO): engineering (usually the largest cost — the humans), delivery, operations (the monthly bill), and maintenance.

• Compute — Lambda is “rarely the most expensive line”; priced on requests + duration. “The cheapest Lambda function is one that is never invoked” — block invalid invocations at the API Gateway. Step Functions standard is billed per state transition (each retry = an extra transition); express per request + duration.
• Storage — S3 priced by size × duration × storage class; use lifecycle policies and TTL. DynamoDB billed on requests + storage with on-demand vs. provisioned modes.

Recovery and resilience

Two failure categories: transient faults (auto-retried with exponential backoff and jitter) and permanent faults (retries exhausted → DLQs, restore from backup). Push retry/backoff into Step Functions rather than functions, eliminate single points of failure, and use the core analysis loop: verify the problem on the health dashboard → find outliers → filter wider telemetry → remedy or loop.

graph LR
V["Verify on health dashboard"] --> O["Find outliers"]
O --> F["Filter wider telemetry"]
F --> R{"Remedy found?"}
R -- Yes --> X["Remediate"]
R -- No --> V

See also

• Testing serverless — observe and recover complete the square of balance.
• The Well-Architected lens — the operational excellence, reliability, and cost pillars.
• Event-driven serverless patterns — the integrations you instrument.

When to use it — and when not

Related topics