How WarmySender handles load

What this page covers

WarmySender is a 4-pillar outreach platform — Cold Emailing, Email Warmup, LinkedIn Outreach, and Multichannel sequences. Across these pillars we send tens of thousands of emails and LinkedIn actions per day on behalf of customers, and we ramp gracefully past traffic spikes without missing sends, over-sending against caps, or putting customer accounts at risk. This page documents the architecture, the safety guarantees, and what you should expect during peak times.

The short version:

• Postgres is the source of truth. Every send is a row in campaign_send_jobs. If anything goes wrong (Redis hiccup, worker crash, deployment), Postgres still has the work to do and we recover automatically.
• Upstash Redis is the hot path. Atomic cap reservations, queue dispatching via BullMQ, and singleton client connections — all per the published Upstash best practices for Redis at scale.
• Recurring heals are evergreen. Two 6-hour ticks sweep the database for any prospects who hit edge-case races (late accepts arriving after a wait_accept timeout, or post-accept follow-ups that didn't enqueue) and self-heal without operator intervention.
• Account safety always wins. Caps fail-CLOSED on Redis outage. Circuit breakers EXPIRE-and-defer instead of throw-and-retry. The platform never bursts past LinkedIn or email-provider rate limits, even when the queue is hot.

Queue architecture

Every send (email or LinkedIn) is materialized as a row in the campaign_send_jobs table with status pending when it's first scheduled. A scheduler tick reads pending jobs whose run_at has arrived and pushes them onto a BullMQ queue backed by Upstash Redis. Worker processes pull jobs from BullMQ, attempt the send via the appropriate provider (SMTP for email; Unipile for LinkedIn), and write the result back to Postgres.

The two-layer design (Postgres for source-of-truth + Redis for fast dispatching) means we get the best of both worlds:

• Durability. Postgres rows survive worker crashes, deploy restarts, and Redis outages. Even if Upstash goes down for an hour, the work is still there in Postgres and resumes when Redis comes back.
• Throughput. Redis lets us dispatch jobs at sub-millisecond latency. BullMQ handles retries, backoff, and concurrency limits. We don't have to poll Postgres for every dispatch.
• Recovery. If a worker dies mid-send, the job's processing_started_at timestamp lets a watchdog recover it (we flip stale processing rows back to pending after a configurable timeout — typically 15 minutes for email, longer for LinkedIn). Per LL#324 (May 6, 2026), Upstash post-addBulk silent drops also auto-heal: a 5-second probe after every batch dispatch verifies every job ID still exists in BullMQ, and if any are missing, the corresponding Postgres rows flip back to pending so the next sync tick re-enqueues them.

For a deeper dive on the Phase 1 (in-process) vs Phase 2 (Redis/BullMQ) auto-switch architecture, see the main documentation.

Cap enforcement

LinkedIn campaigns enforce per-action daily caps via an atomic Redis Lua script. The full design is documented in How invite, message, and InMail caps work together; here's the load-handling summary:

• Atomic INCR with rollback. Each campaign + action_type + UTC date has a Redis key. Before any send, we run a Lua script that atomically increments the counter and rolls back via DECR if the new count exceeds the cap. Two concurrent workers cannot both be allowed past the boundary — Lua runs as one Redis op.
• Cold-start backfill from Postgres. If the Redis key is missing (cache eviction, fresh deploy), we SELECT COUNT(*) from campaign_events for the matching action+day and SET the key with TTL-to-midnight-UTC. Subsequent INCR/DECR runs on the warm key.
• Fail-CLOSED on Redis outage. If Redis is unavailable, the cap helper returns circuit-open and the caller falls back to a Postgres SELECT-COUNT gate. Per CLAUDE.md "account safety always wins" — banned-domain risk is greater than missed-send risk, so we err on the side of NOT sending during a cap-system outage.
• TTL-to-midnight-UTC. Every cap key auto-expires at the day rollover, so we don't accumulate stale keys.

For email campaigns, daily limits are enforced at the mailbox-pacer layer with a similar atomic INCR pattern keyed on mailbox + UTC date. The fail-CLOSED behavior is identical: if the pacer can't talk to Redis, we defer the send rather than burst.

Recurring heals (the evergreen sweepers)

Some race conditions cannot be fully eliminated at write time — Unipile webhook delivery skew, post-accept pipeline transients, timing-window edge cases at the boundary between wait_accept timeout and accept arrival. Rather than hand-waving "it's eventually consistent," we run two recurring heal sweeps every 6 hours that find any prospect stuck in one of these edge cases and self-heal:

• Late-accepts heal. Sweeps for prospects whose accept webhook arrived AFTER the matcher had already excluded them (per late-accept SAFE design). For each, stamps late_accept_observed_at on a separate column that no forward-mover reads, and (if the campaign has a configured post-accept step) enqueues ONE follow-up message via the existing campaign_send_jobs pipeline so caps/ramps/cooldowns are still enforced. The user-visible result: the dashboard shows the late accept and the follow-up fires at the next valid sending slot.
• Stuck post-accepts heal. Sweeps for prospects with linkedin_accepted_at populated whose post-acceptance follow-up never enqueued (transient pipeline failure, expired job that wasn't replaced). Dispatches one follow-up per row via the canonical pipeline insert path.

Both sweeps:

• Make zero Unipile API calls directly — they only stamp database sentinels and enqueue follow-ups via the existing send pipeline.
• Are idempotent — each row has a WHERE-NULL guard on its idempotency column (late_accept_observed_at IS NULL / post_accept_stuck_followup_sent_at IS NULL), so two ticks back-to-back never double-process.
• Are capped at 200 rows per tick as defense-in-depth against runaway cohort growth.
• Write a per-tick audit row to linkedin_events with ll_ref=LL#329 so ops can grep and verify.
• Stagger their first ticks 30 minutes apart so they never both fire on the same wall-clock minute.
• Can be disabled via LINKEDIN_RECURRING_HEALS_ENABLED=false as an off-switch (default ON).

Why 6 hours and not faster? Because the cohorts these sweeps target are small (typically 0–20 rows platform-wide at any given time after a deploy) and the underlying live engine is already self-healing for the common case. Six hours is a sweet spot between "fast enough that customers don't notice the gap" and "infrequent enough that we never thrash the database."

Upstash Redis best practices we follow

We follow the published Upstash Redis best practices across every Redis-touching code path:

• One singleton connection per process. We never create a new Redis client per request, per tick, or per worker — that would burn Upstash connection slots quickly. The shared client is set up at module load with maxRetriesPerRequest: null (BullMQ requires this for blocking commands), keepAlive: 10000 (10s TCP keepalive prevents Upstash 310-second idle eviction), and enableOfflineQueue: false (reject commands when disconnected rather than silently buffer).
• Bounded retry strategy. The retry function never returns null (which would permanently kill the connection); it always retries with capped exponential backoff (2s, 4s, 6s, ... cap 30s).
• TTL on every key. Cap keys expire at midnight UTC. Lock keys have explicit EX TTLs (5–30 minutes depending on the lock). Metric counters expire after 7 days. We never accumulate unbounded keys.
• SETNX with EX is atomic. Distributed locks use the canonical SET key value NX EX seconds single-command idiom — no race where the value is set but the TTL is missed.
• No SCAN in production hot path. SCAN is O(N) and cursor-iterating across the full keyspace would be slow. Every Redis read is by exact key, by INCR/DECR, or by Lua atomic.
• Pipelining for batch operations. Where appropriate (e.g., enqueueing many jobs at once), we pipeline commands so we get one round-trip per batch instead of one per command.
• Circuit breakers fail-CLOSED. When Redis is unavailable, every cap-touching code path returns "circuit-open" and falls back to a Postgres-only gate that still blocks at-or-above the cap. Per CLAUDE.md "account safety always wins."

Load capacity limits

The platform's current sustained throughput envelope (single deployment region):

• Email sending — capped at the customer's mailbox-level daily limits (typically 30–80/day per Gmail/Outlook mailbox under warmup). Aggregate platform throughput across customers comfortably handles 100,000+ sends per hour with sub-second pacing latency.
• LinkedIn sending — capped at LinkedIn-cited per-account limits (typically 80–100 invites/day for paid accounts). Aggregate throughput is bounded by Unipile's own rate limits, which we honor with conservative client-side spacing.
• Webhook ingestion — Unipile delivers webhook events at up to ~10 events/second per workspace. Our handler queue absorbs bursts without dropping events; the inbox-sync queue catches anything that lands during a deploy or transient outage.
• Database writes — Postgres on Neon comfortably handles the platform's current write rate (single-digit thousands of writes per minute at peak). The schema is indexed on every hot-path query (campaign_id, mailbox_id, prospect_id, enrollment_id) so reads stay fast even on multi-million-row tables.

If you ever feel like sends are slower than expected, check the campaign-not-sending diagnosis page first — most "slow" reports trace back to the campaign's configured cap, sending window, or per-account ramp ceiling, not to platform-wide load. We monitor queue depth + processing latency + cap-block rate per workspace, and we'd see the issue before you do if it were systemic.

What "deferred" status means

If you see a campaign or prospect in deferred status, it means the engine tried to send but a safety gate said "not now." Common reasons:

• Daily cap reached. The campaign's per-action cap (invites, messages, or InMails) hit its limit for the UTC day. Sends defer to tomorrow's first sending-window slot. This is the most common deferred reason and is exactly what we want — better deferred than burst.
• Account ramp ceiling. The LinkedIn account is in its first weeks of activity, so the ramp schedule limits daily volume to 10–40/day (gradually increasing). Sends defer to the next day.
• Sending window closed. The campaign's configured sending window (e.g., 9:00–17:00 IST) is closed at the moment the engine tried. Sends queue for the next valid window slot.
• Per-prospect cooldown. The same prospect was sent something within the last N hours; the spacing rule prevents back-to-back sends.
• Mailbox/account paused. The mailbox or LinkedIn account is in paused, cooldown, or disconnected state. Resume the account or wait for cooldown to lift.

Deferred sends are never lost. They sit in campaign_send_jobs with status='pending' and run_at set to the next valid slot. The next scheduler tick picks them up and dispatches when the gate clears. If you want to see what's blocking, the campaign detail page has a "Why deferred?" section that lists the active gates per pending send.

Incident response posture

If something goes wrong (Upstash outage, Unipile downtime, Postgres latency spike), our incident response is built around three principles:

• Account safety first. Every safety gate fails-CLOSED. We'd rather defer 10,000 sends for an hour than burst 10 sends past a cap and lose customer accounts. Customer can always wait for sends; a banned LinkedIn account is unrecoverable.
• Postgres is the source of truth. If Redis is down, we still have every pending job. Recovery is "wait for Redis to come back, then resume." No data loss, no double-sending.
• Idempotent recovery. Every recovery script we run is idempotent. Re-running it after a partial recovery does the right thing — it won't re-process rows that already healed.

Post-incident, we publish what happened, what the customer impact was, and what we changed to prevent recurrence — see the missed-accept, InMail count correction, and other troubleshooting pages for examples of past incidents and how they shaped the platform.

Common questions

Will my campaign keep sending if Upstash Redis goes down?

Email campaigns: yes, with degraded-mode pacing. The mailbox-pacer falls back to a Postgres-only gate that still enforces daily cap, but loses atomic-strength under high concurrency. With single-worker deployments (typical), this is functionally equivalent to the Redis fast path. LinkedIn campaigns: cap enforcement returns circuit-open, and the caller falls back to the Postgres SELECT-COUNT gate which still blocks at-or-above the cap. Sends continue but with slightly higher per-send latency. When Redis recovers (typically within minutes for Upstash incidents), the cap counters cold-start-backfill from Postgres on the next miss — no manual intervention.

How does the platform recover from a worker crash mid-send?

Every job in processing status has a processing_started_at timestamp. A watchdog scans for processing rows older than 15 minutes (email) or longer for LinkedIn, and flips them back to pending for re-dispatch. The send is idempotent at the provider boundary (we use Unipile's idempotency_key for LinkedIn; SMTP duplicates are rare and detected by the message-ID dedup). Per LL#324 (May 6, 2026), Upstash post-addBulk silent drops also auto-heal at the queue layer.

What's the typical end-to-end latency from "scheduled" to "sent"?

For a job scheduled with run_at = NOW(): the scheduler tick runs every 60 seconds, picks up the job, dispatches to BullMQ, the worker pulls it within milliseconds, and the actual send happens within a few seconds (Unipile API latency for LinkedIn; SMTP handshake for email). End-to-end: 60–90 seconds typical, ~3 minutes p99. Recurring heal latency is bounded by the 6-hour tick cadence — see recurring heals above for why we picked that cadence.

How do I know if my campaign is hitting the cap vs. sending normally?

The campaign detail page shows daily counts vs. cap for each action type (invites, messages, InMails). If the cap is hit, the badge color shifts and the "Why deferred?" section lists "daily cap" as the active gate. The platform also logs [Cap-Redis] decision=blocked on every cap-block event; admin users can grep these from the deployment console.

Does the recurring heal scheduler cost extra Unipile API quota?

No. The heal sweeps make zero Unipile API calls — they only stamp database sentinels and enqueue follow-ups via the existing send pipeline. The follow-ups themselves count against the campaign's daily message cap (one slot per follow-up), not invite cap. If the cap is exhausted, the follow-up defers to the next day.

What happens if both ticks (late-accepts + stuck-post-accepts) try to fire at the same time?

They don't, by design — the second tick is staggered 30 minutes after the first. If somehow both did fire, each holds its own Redis SETNX lock with a 30-minute TTL, so a second invocation while one is mid-flight returns "lock held" and skips. Multi-pod deployments use the same Redis lock, so we never double-heal across pods. If Redis is unavailable, each tick falls back to a process-local lock (we may double-heal across pods during a Redis outage, but the underlying helper is idempotent at the row level, so the worst case is a few duplicate audit rows in linkedin_events — no double-sends).

Is the platform multi-region today?

No, currently single-region (us-east-1 on Neon + Upstash + Replit). Multi-region adds significant complexity (cap-counter consensus, webhook routing, DB-replication latency) and we haven't seen the throughput need to justify it. If your campaign is highly latency-sensitive (e.g., regulatory windows in Europe), let us know and we'll prioritize accordingly.

How do I reach the team if I see a load issue?

Email hello@warmysender.com with the campaign name, time-window, and what behavior you're seeing (deferred status, slow sends, missed webhook). We'll dig into the queue depth and per-account ramp state for that workspace. For real-time issues, the Support page has the on-call rotation contact.

Related guides

• How invite, message, and InMail caps work together — Cap split semantics + Unipile-cited limits + worked examples
• LinkedIn rate limits — Per-account daily and weekly limits, ramp schedule
• What is a "late accept" — The SAFE-design tolerance + recurring heal sweeper
• If you suspect a missed LinkedIn accept — Webhook delivery + polling fallback
• Why isn't my LinkedIn campaign sending? — Common diagnoses for "campaign appears stuck"
• LinkedIn campaign documentation — Schedule, sending windows, ramp, acceptance lag, disconnect flow
• Full documentation — All 90+ guides
• Support — How to get in touch