Frameworks for testing ads are easy to find. Many come from creative-analytics vendors. Segwise, which sells creative analytics, publishes a five-pillar framework for Meta ads that lets you define a winner as a CPA ceiling, a CTR lift or a ROAS gain1. Frameworks like that help you organise creative. This piece adds the money end: which ad brings new customers at a cost and margin you can live with.
What should a creative test decide?
A test exists to make one decision. Write it down before you launch.
Typical decisions:
- Which concept gets more budget next month?
- Which concept do we retire?
- Does a new angle deserve a second round, or was it noise?
"Which ad has the best click-through rate (CTR)" is not a decision. It is a metric. A cheap click that never buys is a hobby.
Our view: the unit to judge is cost per new customer, read against margin. Our convention: contribution margin means revenue minus the variable costs of the order (product cost, shipping, payment fees, returns), before any marketing cost. Your new-customer CAC (nCAC) is the acquisition spend divided by first-time customers only. Put the two side by side and a winner has to clear both.
CTR shows attention, not profit.
How many creatives can your budget and volume test properly?
Fewer than you think. Every variant needs enough purchases to be read. Split the same purchases across more ads and each ad gets a thinner slice. Then every result looks like noise.
An A/B test splits an audience at random between two versions and compares them on a metric chosen in advance. More arms make that harder, not easier. Statistical significance tells you whether a gap is unlikely to be chance. It does not tell you whether the gap is big enough to change a budget.
Example (illustrative): a brand gets about 100 purchases a week from Meta. Testing ten variants means about 10 purchases each, before any split by audience. Testing three means about 33 each. The three-variant test is the one that can say something. These round numbers show the logic, not a threshold.
Our view: pick the number of variants from the purchases you expect during the test, not from how many ideas the creative team has. Surplus ideas go in a queue.
Meta's retrieval system adds a wrinkle. Meta Andromeda is the stage that narrows a very large pool of candidate ads to a shortlist. Segwise, a creative-analytics vendor, writes that Andromeda groups near-identical creatives under one ID, so they compete as one ad2. That is the vendor's reading, and it cites no Meta documentation for it. Our view: either way, treat ten small tweaks of one concept as one idea, not ten.
Which metrics do you read at each stage, from hook to purchase?
Read metrics in order, and know what each one can and cannot tell you.
| Stage | Metric | What it tells you | What it does not |
|---|---|---|---|
| Attention | Hook rate, thumb-stop (people who watch the opening) | Whether the first seconds hold people | Whether they buy |
| Interest | CTR | Whether the message earns a click | Whether the click is a likely buyer |
| Intent | Add-to-cart, checkout starts | Whether the promise matches the page | Whether the buyer is new |
| Purchase | Cost per new customer | What the ad paid for a first-time buyer | Lifetime value or returns |
| Money | Contribution margin per order | Whether that customer was worth buying | Anything after the first order |
Early metrics are for diagnosis. If an ad has a strong hook and weak purchase, the creative may be fine and the page may be the problem. If the hook is weak, fix the first seconds before you judge the rest.
The final judge is the bottom two rows. Platform ROAS alone can flatter an ad that mostly reaches people who would have bought anyway. Check it against new customers. Our view: if platform ROAS and cost per new customer disagree, trust the second.
How do you structure tests in Meta without breaking delivery?
This is also the answer to how to set up a creative testing system in Meta Ads, and to what a creative testing framework for Meta ads looks like in practice. Keep it plain.
- One variable per test. Change the hook, or the offer, or the format. Not all three.
- Same audience and objective for all variants. Otherwise you are testing the audience.
- Few variants. See the volume check above.
- Do not edit mid-test. Change budgets or swap ads halfway and you are reading two tests mixed into one.
- Check spend per variant before you read results. If one variant got most of the spend, you did not run an equal test. Read each result against what that variant actually received.
- Keep a log. Hypothesis, variants, dates, decision. If it is not logged, you will run the same test twice.
For the detail of campaign structures, see our Meta creative testing setup page. This article stays on the framework.
When is a winner a winner?
Agree the rule before the test starts. A winner needs three things:
- It beat the control on cost per new customer, by a margin you set in advance.
- It held that result across the full test window, not on one good day.
- After costs, the order still leaves contribution margin you can use.
If the result is close, say so. "No clear winner" is a valid output. It saves you from scaling noise.
Watch for the opposite problem too. Winners decay. Jon Loomer, a Meta ads consultant, writes that Ads Manager can show "Creative Fatigue" or "Creative Limited" in the Delivery column when an ad's cost per result is higher than it has been in the past3. Read more on creative fatigue and how to fix creative fatigue.
How do you run a weekly creative review?
Same hour, same agenda, every week. Looking at a dashboard is not a review. A decision is.
- Status of live tests. Which are running, which have enough volume to call.
- Decisions. Scale, retire or extend. Each with a reason tied to cost per new customer and margin.
- Retire on a schedule. Ads that lost the last read get paused. Not "when we feel like it".
- Next tests. Pull from the queue, within what your volume can read.
- Log it.
Our view: retiring on a schedule matters more than finding winners. A losing ad that stays live because nobody wanted to decide is a standing cost.
