A/B Testing & Auto-Optimization
A/B testing lets you compare different email variants to find what works best.
How It Works:
- When creating a campaign step, add multiple variants — each with its own subject line and body content.
- Each variant gets a weight that determines what percentage of prospects receive it (e.g., 50/50 split for two variants).
- As emails are sent, the system tracks open rates, click rates, and reply rates for each variant independently.
- The variant assignment is stored per prospect for consistent tracking.
Auto-Optimization:
WarmySender can automatically pick the winning variant using statistical analysis:
• Enable 'Auto-Optimize' on the campaign step
• Set a minimum sample size (default: 100 sends) — the system needs enough data before making a decision
• Set a confidence level (default: 95%) — how sure the system needs to be
• The system uses a chi-squared statistical test comparing reply rates across variants
• When statistical significance is reached, the winning variant gets 100% of future sends
• Auto-optimize checks run every hour
What to Test:
- Subject lines — Usually the biggest impact on open rates
- Opening lines — First sentence determines if the email gets read
- Call-to-action — Different asks (meeting, reply, link click)
- Email length — Short vs. detailed approaches
- Personalization level — More vs. fewer merge tags
Best Practice: Test one variable at a time for clear results. Start with subject line testing as it has the most measurable impact on open rates.