What to change in an email to understand the difference
A valid test has one variable. If version B has a new subject line, a different offer and a new call to action, the result explains nothing: any of those changes may have caused the reply. Keep the segment, sender, sending conditions and remaining copy fixed, then change one part of the message.
| What you test | Version A | Version B |
|---|---|---|
| Opening line | I see that you are expanding your dealer network | Do you work with dealers across several regions? |
| Call to action | I can send you the calculation framework | Would it help to review the framework in a 15-minute call? |
| Offer | Reduce manual reconciliation of enquiries | Remove errors when enquiries are passed into 1C |
Test the subject line separately from the email body: it earns attention, but does not prove the offer is valuable. For the first touch, it matters more that the decision-maker can answer briefly: “Yes, send it” or “Not relevant to us.”
How to run a test without choosing a false winner
Split the target list into two equal parts before sending. Do not give one version all companies from one industry, region or size band: an audience difference can easily look like a copy advantage. If several people work at one company, they should receive the same version.
- State a testable assumption, such as: a specific operational problem will generate more relevant replies than a general service description.
- Prepare two versions that differ by one sentence or one block.
- Send them to equal, mixed parts of the list under the same conditions.
- Classify replies by meaning: interest, refusal, referral, or request to remove the contact.
- Keep the winning version for the next wave, then test the next hypothesis.
Why the email with more replies can lose
Consider a provider of business systems writing to manufacturing directors. Version A promised to “optimise business processes”; version B asked who reconciles shift reports and inventory between the shop floor and 1C. A received more polite requests for a presentation, while B received fewer replies but several questions about implementation timing. If the campaign goal is an implementation conversation, B is stronger.
Set the target action before launch. If you are testing whether the market understands the offer, count substantive replies. If the goal is to hand conversations to sales, record qualification and the next step; align those rules in advance with reply handling and lead qualification.
A test will not rescue an unclear offer, a random company list, or a team with nobody ready to answer interest. It is also a poor fit for a very narrow account-based list where every company needs a unique approach. In that case, compare personalisation logic rather than two identical mailings; account-based marketing is the more appropriate model.
Mistakes that invalidate an A/B test
- Changing the subject line, copy and segment in the same wave.
- Comparing replies without separating interest from automated notifications and refusals.
- Stopping one version too early or giving recipients unequal time to reply.
- Keeping both versions live after choosing a winner instead of recording the new control email.
Check the technical side before testing. Strong copy cannot produce a reliable signal if some addresses cannot receive the message. See email list verification and email deliverability for the relevant checks. Related terms: cold email, reply rate and outreach sequence.