Event-Driven Architecture with EventBridge - Lessons from 3 Years in Production
Three years of building serverless event-driven systems with AWS EventBridge. What works, what doesn't, and patterns that actually scale.
By Subash Adhikari · · 5 min read
Updated .
After three years of running event-driven architectures in production, I've learned that EventBridge is both powerful and full of sharp edges. This post covers the patterns that survived production load and the mistakes I'd avoid if I were starting today.
Why EventBridge?#
When I started building serverless applications, I reached for SNS for everything. It was simple, familiar, and it worked. But as our systems grew, its limitations got painful. There was no event filtering at the source, so every subscriber got everything. There was no built-in schema validation, very little visibility into how events flowed, and publishers and subscribers ended up tightly coupled.
EventBridge solved those problems and kept the serverless model I liked: pay per use, no servers to manage, and native AWS integration.
The Architecture That Worked#
This is the pattern I settled on after a few iterations. A service publishes to one event bus, and rules filter that stream onto two Lambda handlers and a Fargate worker.
Lesson 1: Event Structure Matters More Than You Think#
I started with simple flat events:
{
"userId": "123",
"action": "order.created",
"data": { ... }
}This was a mistake. Once we had 50+ event types, debugging became a nightmare. Now I use a consistent envelope:
interface DomainEvent<T extends string, P> {
specversion: "1.0";
type: T;
source: string;
id: string;
time: string;
datacontenttype: "application/json";
data: P;
}
// Example usage
type OrderCreatedEvent = DomainEvent<
"orders.orderCreated.v1",
{
orderId: string;
customerId: string;
total: number;
items: Array<{ sku: string; quantity: number }>;
}
>;This follows the CloudEvents spec, which EventBridge supports natively. You get standard tooling, since there are CloudEvents libraries in every language. The EventBridge Schema Registry understands the structure. And for debugging, source, type, and id together give you a complete trace.
Lesson 2: Use EventBridge Schema Registry#
I ignored the Schema Registry for the first year. Big mistake.
// sst.config.ts
const bus = new sst.aws.Bus("Bus", {
rules: {
orderCreated: {
pattern: {
source: ["orders"],
"detail-type": ["orders.orderCreated.v1"],
},
targets: { handler: "src/order.handler" },
},
},
});
// Enable schema discovery
new aws.cloudwatch.EventBus("DiscoveredSchemas", {
name: bus.name,
eventBridgeBus: bus.arn,
});With schema discovery turned on, EventBridge infers schemas from your events and stores them in the registry. That means you can generate TypeScript types from the discovered schemas, catch malformed events before they cause problems, and keep living documentation of your event contracts.
Lesson 3: Dead Letter Queues Are Non-Negotiable#
In my first production EventBridge setup, I didn't configure DLQs. When a Lambda handler failed, events were silently dropped. We lost order notifications for 6 hours before anyone noticed.
Now every target gets a DLQ:
// sst.config.ts
const dlq = new sst.aws.Queue("DLQ");
const bus = new sst.aws.Bus("Bus", {
rules: {
orderCreated: {
pattern: {
source: ["orders"],
"detail-type": ["orders.orderCreated.v1"],
},
targets: {
handler: {
handler: "src/order.handler",
deadLetterQueue: dlq.arn,
},
},
},
},
});And I monitor DLQ depth with a CloudWatch alarm:
new aws.cloudwatch.MetricAlarm("DLQAlarm", {
metricName: "ApproximateNumberOfMessagesVisible",
namespace: "AWS/SQS",
dimensions: { QueueName: dlq.name },
threshold: 0,
evaluationPeriods: 1,
comparisonOperator: "GreaterThanThreshold",
});Lesson 4: Event Replay Saved My Job#
EventBridge archives are underrated. When I accidentally deployed a bug that processed orders incorrectly, I needed to replay 3 hours of events.
# Enable archive
aws events create-archive \
--archive-name order-events \
--event-source-arn $BUS_ARN \
--retention-days 30
# Replay events
aws events start-replay \
--replay-name fix-orders-2024-03-15 \
--event-source-arn $BUS_ARN \
--event-start-time 2024-03-15T10:00:00Z \
--event-end-time 2024-03-15T13:00:00Z \
--destination '{"Arn": "'$BUS_ARN'", "FilterArns": ["'$FIXED_RULE_ARN'"]}'Replay let me fix the bug, redeploy, and reprocess only the affected events. No manual data fixing.
Lesson 5: Cross-Account EventBridge Is Worth the Complexity#
When we split into multiple AWS accounts (dev, staging, prod, and services), I first used SNS and SQS for cross-account communication. It worked, but it felt fragile.
EventBridge cross-account event buses are cleaner:
// In the producer account
new aws.cloudwatch.EventBusPolicy("CrossAccountPolicy", {
eventBusName: bus.name,
policy: JSON.stringify({
Version: "2012-10-17",
Statement: [{
Sid: "AllowConsumerAccount",
Effect: "Allow",
Principal: { AWS: "arn:aws:iam::CONSUMER_ACCOUNT_ID:root" },
Action: "events:PutEvents",
Resource: bus.arn,
}],
}),
});
// In the consumer account
new aws.cloudwatch.EventRule("CrossAccountRule", {
eventBusName: "arn:aws:events:REGION:PRODUCER_ACCOUNT_ID:event-bus/Bus",
eventPattern: JSON.stringify({
source: ["orders"],
}),
});With this pattern, the event bus is centralized in the producer account and consumers live in their own accounts. Ownership is clear too. The producer owns the schema and each consumer owns its handlers.
What I'd Do Differently#
I'd start with schema validation. Use JSON Schema or Protobuf from day one, because migrating 100+ event types later is painful.
I'd version events aggressively, as in orders.orderCreated.v1, v2, and so on. EventBridge can filter on detail-type, which keeps migrations manageable.
I'd use SaaS integrations sparingly. EventBridge has 200+ of them. They're convenient, but they create vendor lock-in. I prefer polling webhooks into my own event bus.
And I'd monitor event latency from the start. EventBridge promises 500ms delivery but doesn't guarantee it, so add CloudWatch metrics for end-to-end latency.
The Code I Use Today#
This is my current SST v3 setup:
// sst.config.ts
export default $config({
app() {
return {
name: "events",
removal: "remove",
home: "aws",
};
},
async run() {
// DLQ for failed events
const dlq = new sst.aws.Queue("EventDLQ");
// Main event bus with schema discovery
const bus = new sst.aws.Bus("Bus", {
rules: {
// Order events
orderCreated: {
pattern: {
source: ["orders"],
"detail-type": ["orders.orderCreated.v1"],
},
targets: {
handler: {
handler: "src/handlers/orderCreated.handler",
deadLetterQueue: dlq.arn,
},
},
},
// Inventory events
inventoryUpdated: {
pattern: {
source: ["inventory"],
"detail-type": ["inventory.inventoryUpdated.v1"],
},
targets: {
handler: {
handler: "src/handlers/inventoryUpdated.handler",
deadLetterQueue: dlq.arn,
},
},
},
},
});
// Archive for replay
new aws.cloudwatch.EventArchive("EventArchive", {
name: $app.name,
eventSourceArn: bus.arn,
retentionDays: 30,
});
// DLQ alarm
new aws.cloudwatch.MetricAlarm("DLQAlarm", {
metricName: "ApproximateNumberOfMessagesVisible",
namespace: "AWS/SQS",
dimensions: { QueueName: dlq.name },
statistic: "Sum",
threshold: 0,
evaluationPeriods: 1,
comparisonOperator: "GreaterThanThreshold",
});
return {
busArn: bus.arn,
dlqArn: dlq.arn,
};
},
});Final Thoughts#
EventBridge isn't perfect. The CloudWatch Logs integration is clunky, debugging event flows means jumping between services, and the 256KB event size limit forces you to put large payloads in S3.
But after three years, it's still my default for serverless event-driven architectures. Schema validation, dead letter queues, event replay, and native AWS integrations together are hard to beat.
If you're starting a new project, skip SNS and go straight to EventBridge. Your future self will thank you.
Related posts
Thanks for reading. You can also follow along on X or with RSS.