A/B Testing & Auto-Optimization
A/B testing lets you compare different email variants to find what works best.
How It Works:
- When creating a campaign step, add multiple variants — each with its own subject line and body content.
- Each variant gets a weight that determines what percentage of prospects receive it (e.g., 50/50 split for two variants).
- As emails are sent, the system tracks open, click, reply and opt-out rates for each variant independently. Replies, bounces and opt-outs count toward the variant that prospect was actually sent.
- The variant assignment is stored per prospect for consistent tracking.
Auto-Optimization:
WarmySender can automatically pick the winning variant using statistical analysis:
• Turn on 'Auto-Optimize Variants' in the campaign builder's settings (it appears once a step has two or more variants)
• Set a minimum sample size (default: 100 sends) — the system needs enough data before making a decision
• The system uses a chi-squared statistical test comparing reply rates across variants, at 95% confidence
• When one variant is significantly better than every other variant, it gets 100% of future sends
• It never picks a winner before every variant in the test has been sent and at least one reply has come in, so a variant you just added is always tried first. A variant you set to 0% is left out of the test
• Auto-optimize checks run every hour
What to Test:
- Subject lines — Usually the biggest impact on open rates
- Opening lines — First sentence determines if the email gets read
- Call-to-action — Different asks (meeting, reply, link click)
- Email length — Short vs. detailed approaches
- Personalization level — More vs. fewer merge tags
Where to see the results:
Once a step is running with multiple variants, open the campaign's Sequence tab and expand the step to see one row per variant (A, B, C…) with its own Sent, Opened, Clicked, Replied and Opt-outs numbers side by side. In each column the best variant is shown in green and the worst in red (for opt-outs, fewer is better), and a line under the variants tells you which one is leading: it judges on replies first, then clicks, then opens, says 'too early to call' until each variant has at least 20 sends, and says when the gap is large enough to trust (95% confidence). The step row also shows a badge such as 'B leads on replies', so you can see the leader without expanding the step. To see which variant each individual recipient received, open the Prospects tab — the Variant column sits right next to that person's own Opens and Clicks, and the Prospects CSV export includes it too. See the 'See Opens and Clicks by Email Variant (A/B)' guide for the full walkthrough.
Best Practice: Test one variable at a time for clear results. Start with subject line testing as it has the most measurable impact on open rates.