Deduplication at System Boundaries: Why Every Event Will Fire Twice
Same signal, same handler, same window. Two trades placed. Two emails sent. Two deploys triggered. Deduplication is not a feature — it is the gate that prevents your system from doubling every action.
Your 5-minute candle closes. The signal fires. The handler executes. A trade is placed. Clean execution.
Then the same candle triggers again. Same signal. Same handler. Same trade. You now have two identical positions — one real, one duplicate. Your exposure is doubled. Your risk model is wrong. Your P&L will show a loss on the duplicate when you unwind it, even if the trade was correct.
This is not a hypothetical. This is the default behavior of any event-processing system that does not deduplicate at the boundary. Duplicate events are not a bug. They are a certainty. The only question is whether your system handles them or doubles your actions.
Any system that processes external events without deduplication will eventually process the same event twice. It is not a question of if. It is a question of when, and how much damage it causes.
Why Events Fire Twice
Events are not surgical instruments. They are broadcast signals with noise, retries, and overlap.
Network retries. A webhook fires. The receiving server takes 3 seconds to respond. The sender's timeout is 2 seconds. The sender retries. The receiver now processes the same event twice — once from the original, once from the retry.
Overlapping processing windows. A cron job runs every 5 minutes. Execution takes 6 minutes. The next cron fires while the previous is still running. Both process events from the overlapping window.
At-least-once delivery. Most message queues — SQS, Kafka, PubSub — guarantee at-least-once delivery, not exactly-once. The trade-off is explicit: the queue will never lose your message, but it may deliver it more than once. Your handler must be prepared for duplicates.
Source-side duplicates. The event source itself may emit duplicates. A price feed might send the same candle data twice due to a correction or re-broadcast. Your system sees two events that are semantically identical.
None of these are failures. They are inherent properties of distributed systems. Any architecture that assumes events arrive exactly once is an architecture that will double-fire.
Anything that can go wrong will go wrong. And anything that can happen twice will happen twice — usually at the worst possible time.
— Murphy's Law (origin: Edwards Air Force Base, 1949) · Aerospace Engineering Principle
The Dedup Gate
Deduplication belongs at one place in your architecture: the boundary where external events enter your system. Before the event reaches any handler, before any business logic executes, before any side effect occurs — the dedup check happens.
The gate is simple in concept. An event arrives. A dedup key is generated. The key is checked against durable storage. If the key exists, the event is a duplicate — log it and return. If the key does not exist, set it and process the event.
The simplicity is deceptive. Every decision in this gate — key construction, flag ordering, storage choice, TTL policy — has consequences.
Dedup Key Construction
The key must uniquely identify the event within its processing window. Too broad, and legitimate events are suppressed. Too narrow, and duplicates slip through.
The format: {entity}_{event_type}_{window_identifier}
Trading signals: ETH_momentum_5m_2026-03-01T14:30:00Z — the asset, the signal type, the candle window timestamp. Two momentum signals for ETH in the same 5-minute window are duplicates. A momentum signal in the next window is new.
Webhook events: stripe_payment_succeeded_evt_1234abcd — the source, event type, and the event's own unique ID. Stripe already provides unique event IDs. Use them.
Cron jobs: blog_autopilot_2026-03-01T09:00:00Z — the job name and scheduled execution time. Two triggers of the same job at the same scheduled time are duplicates. The next scheduled time is new.
The window identifier is the critical component. It defines what "duplicate" means for this specific event type.
For a 5-minute candle signal, the window is the candle timestamp. For a daily cron, the window is the date. For a webhook, the window is the event ID. Match the identifier to the semantics of the event, not to an arbitrary time window.
The Ordering Problem: Idempotency vs. Retry
Where the dedup flag is set — before or after processing — determines how the system handles failures.
Idempotent ordering (check → set → process): If processing fails after the flag is set, the next attempt sees the flag and skips. The event is never retried. Use this when double-fire is more costly than a missed event — financial transactions, trade placement, payment charges.
Retriable ordering (check → process → set): If processing fails before the flag is set, the next attempt sees no flag and retries. The event gets another chance. Use this when a missed event is more costly than a duplicate — notification delivery, data synchronization, metrics collection.
The choice is a product decision, not a technical one. What costs more — processing twice or processing zero times? For trading, the answer is always idempotent. Two trades on the same signal is worse than zero trades. For notifications, the answer is usually retriable. Two emails is better than zero emails.
The critical mistake is placing the dedup flag after an early return. If your handler has multiple return paths — validation failures, feature flags, error conditions — and the dedup flag is set only on the happy path, every failed attempt leaves the gate open for the next attempt to process the same event again.
Map every return path. The dedup flag must be set on the path that matches your ordering choice — before ALL processing for idempotent, or after ALL processing for retriable. Not somewhere in the middle.
Durable Storage: Surviving Restarts
Dedup state in memory dies when the process restarts. A system that restarts at 14:35 will re-process every event from the current window because its dedup memory is empty. The gate was open, the process walked through, and the same trades fired again.
Durable storage options:
- SQLite — simplest for single-process systems. One table:
dedup_key TEXT PRIMARY KEY, created_at TIMESTAMP. Query is a simple existence check. - Filesystem — lock files for cron jobs.
/tmp/blog_autopilot_2026-03-01.lock. Presence of file = already processed. - Redis — for distributed systems where multiple processes need shared dedup state. Built-in TTL support via
SETEX.
The choice depends on your system's architecture. Single process on one machine? SQLite. Cron job that should not overlap? Lock file. Distributed microservices? Redis.
The non-negotiable requirement: the dedup state must survive process restarts. If it does not, every restart resets the gate and every event in the current window re-processes.
TTL-Based Cleanup
Dedup entries accumulate. Without cleanup, your storage grows unbounded. A trading system processing 100 signals per day accumulates 36,500 dedup entries per year. That is manageable for SQLite but wasteful — most entries are meaningless after their window closes.
The rule: TTL = window_length + buffer
A 5-minute candle signal's dedup entry is meaningless after the candle closes. Set TTL to 10 minutes — the window plus a buffer for late-arriving duplicates. A daily cron's dedup entry is meaningless after midnight. Set TTL to 48 hours — one day plus buffer.
Cleanup runs periodically — every hour, every day, depends on volume. A simple query: DELETE FROM dedup WHERE created_at < NOW() - ttl. The storage stays bounded. Old entries do not interfere with current processing.
Real-World Dedup Patterns
You are not inventing this. Industry has solved dedup at scale. Study these implementations:
Stripe idempotency_key — you send a unique key with each API call. Stripe stores it. If you send the same key again within 24 hours, Stripe returns the original response without re-processing. The client controls dedup, not the server.
AWS SQS MessageDeduplicationId — messages with the same dedup ID within a 5-minute window are treated as duplicates. SQS delivers only one. The queue handles dedup at the infrastructure level.
Cron lock files — /tmp/job_name.lock created at job start, deleted at job end. If the lock file exists when the job starts, the previous run has not finished — the new run exits. Simple, filesystem-based, survives nothing (which is correct for overlapping cron prevention).
GitHub Actions concurrency groups — a concurrency key ensures only one workflow with that key runs at a time. Duplicate triggers queue or cancel. The platform handles dedup for CI/CD.
Each pattern solves the same fundamental problem: preventing the same action from executing twice within a window where duplicates are semantically identical.
The dedup gate is the first line of defense for any event-driven system. It does not make your system correct. It prevents your system from being wrong in the specific, expensive way that duplicates cause — doubled trades, doubled emails, doubled deploys, doubled charges.
Build the gate before you build the handler. The handler assumes events are unique. The gate makes that assumption true.
Lesson 37 Drill
Pick one event-driven system you operate — a trading bot, a webhook handler, a cron pipeline, a notification system.
Answer these questions:
- What is the dedup key for each event type? Write it out in
{entity}_{type}_{window}format. - Where is the dedup state stored? If the answer is "in memory" or "nowhere," you have a restart vulnerability.
- Is the dedup flag set before or after processing? Does that ordering match your cost model (idempotent vs. retriable)?
- What is the TTL for dedup entries? If there is no TTL, your storage is growing unbounded.
- What happens if the process restarts mid-window? Will events from the current window re-process?
Implement the dedup gate this week. A SQLite table with three columns — key, created_at, ttl — and a check at the top of your event handler. That is the minimum viable dedup. It takes 30 minutes to build and prevents every double-fire that will otherwise happen.
Bottom Line
Events fire twice. Webhooks retry. Crons overlap. Queues deliver at-least-once. Every system that processes external events lives in a world where duplicates are the default, not the exception.
The dedup gate is not a feature. It is a requirement. Build it at the boundary where events enter. Use durable storage. Set TTLs. Choose your ordering based on cost — idempotent for financial actions, retriable for notifications.
Thirty minutes of dedup implementation prevents every double-trade, double-email, and double-deploy that your system will otherwise produce. The events will come twice. Your system only needs to act once.
Explore the Invictus Labs Ecosystem