Skip to main content
A/B Testing lets you compare different versions of an email campaign to see which one performs better. Send each version to a portion of your audience, wait for results, and automatically deliver the winning version to everyone else. Screenshot of Email Creation in an email campaign A/B test

​
What to test

Changes that often result in the most significant changes in engagement include:

Subject Line

Try different wording, length, or tone in your subject line.

Preview Text

Test what appears in the inbox preview to boost open rates.

Offer or Coupon

Compare different discount amounts or no discount at all.

Design or Layout

Test different layouts or visual styles.

Image

Try product images vs. lifestyle images, or image vs. no image.
We recommend testing one change per version so you can identify what made the difference in your results.

​
Create an A/B Test

To start a new A/B Test:
  • Navigate to Campaigns.
  • Click Create Campaign.
  • Choose A/B Test Email.
  • Name your test and click Continue.

​
Email Creation

In the Email Creation tab, you’ll create 2 to 5 versions of your campaign. Version A is your control. It uses the campaign’s existing email. To add more versions:
  • Click Add Version to add Version B, C, D, or E.
  • Give each version a Version Name for internal tracking.
  • Customize each version’s Email Subject, Email Preview Text (Optional), and design using Edit Design.
  • Use Send Test Email to preview how each version looks in your inbox.
Each version can have its own coupon. The coupon merge tag must be added to that version’s design.
You can duplicate or remove versions at any time. You need at least two versions to continue.
Edit Sender Details sets the From Business Name, From Email, and Reply To Email for all versions. These details are shared across the test.

​
Recipients

Select one or more segments to send your test to. You can also exclude a segment if needed. The audience is frozen when you send the test. Changes to your segments after sending won’t affect who receives the campaign.

​
Test Settings

Screenshot of Test Settings in an email campaign A/B test Configure how your test will run: Winning Metric Choose how Privy will determine the winner:
  • Open Rate — The version with the highest percentage of opens wins.
  • Click Rate — The version with the highest percentage of clicks wins.
Test Duration Set how long Privy will collect data before choosing a winner. You can specify hours or days. The test duration starts after all test-cohort sends are complete. Recipient Allocation Set what percentage of your audience receives each version during the test. You can allocate from 1% to 100% per version. The allocations are approximate, especially on smaller lists. If you allocate less than 100% total, the remaining recipients will receive the winning version after the test completes. If you allocate 100%, there will be no one left to send the winner to.
Smaller allocations make it harder to identify a clear winner. Privy will show guidance when your allocation is too small to produce reliable results.

​
Review and Send

The Review tab shows an overview of your test: number of versions, winning metric, test duration, and total recipients. When you’re ready:
  • Click Send to start the test immediately.
After you send, the test content, audience, allocations, metric, and duration are frozen. You can cancel an in-flight test, but messages already sent cannot be recalled.

​
How the winner is chosen

Privy calculates a rate for each version:
  • Open Rate = opens for that version ÷ emails delivered for that version
  • Click Rate = clicks for that version ÷ emails delivered for that version
The version with the highest rate wins. A larger allocation does not automatically win — only the rate matters. If there’s an exact tie, Version A (the control) wins. If no version has any engagement data, Version A wins. The decision uses data collected through the test deadline. Later activity may change the live report but won’t change which version won. After the winner is chosen, Privy sends that version to the remaining recipients (if any are left outside the test allocation).

​
Reporting

Screenshot of completed A/B test showing Version B as winner with 21.3% click rate To view your A/B test results:
  • Navigate to Campaigns.
  • Locate your test. A/B Test Email campaigns appear with A/B Test or Email A/B Test and show the version count.
  • Click to open the campaign report.
At the top of the report, you’ll see an A/B Test Overview card. This card compares each version on the winning metric you selected and marks the Winner. The status shows whether your test is Live, Completed, or Canceled:
  • Live tests display Time Remaining and Recipients Tested So Far.
  • Completed tests show Ended and Recipients Tested.
Use Select Version to scope the rest of the report to a specific version. When you select a version, the Overview, Contact Activity, Link Clicks, and View Email tabs all show data for that version only.
Later opens and clicks may change the live report numbers, but they do not change the winner. The winner is determined by data collected through the test deadline. See How the winner is chosen for details.

​
Tips for reliable results

  • Test one element at a time so you know what caused the difference.
  • Allocate enough of your audience to each version. Very small percentages rarely produce a clear winner.
  • Give your test enough time. Open and click behavior takes hours to unfold.
  • Make sure each version’s design is complete before sending. You cannot edit versions after the test starts.