AI, AI Everywhere… How to Keep Your Tests Pure
When Meta wants to improve your ad—and accidentally ruins your experiment
Be careful what you wish for.
For years, I wished for someone who could create images for my ads, write the copy, suggest hooks and headlines, and generally make advertising easier.
Well, that “someone” has arrived.
Actually, several someones, collectively labeled AI.
Some are polite, friendly, and extremely helpful. Others work for Meta.
When you open Ads Manager these days, Meta’s AI is everywhere. It offers to generate new images, rewrite your primary text, produce alternative headlines, add related media, create videos, resize your creative, and generally improve things you may not have realized needed improving.
And Meta is very enthusiastic about the potential benefits.
Try this enhancement and your click-through rate might increase.
Add that variation and your conversion rate might improve.
Wouldn’t you like a few additional versions of your image?
How about a video?
Perhaps another headline?
Maybe your author photograph?
At some point, you begin to feel less like an advertiser and more like someone trying to leave a department store while an exceptionally persistent salesperson follows you toward the exit.
For many advertisers, all this automation can be useful.
For authors trying to run a clean test, however, it can become a problem very quickly.
The problem authors have with Meta
If you run an online shop that you control, Meta can often receive conversion data from your website and use that information to optimize toward people likely to buy.
Authors selling books on Amazon are in a different position.
We can advertise the book on Meta, but we can’t simply place our own Meta pixel on an Amazon product page and tell Meta:
Find me more people who behave like the ones who bought this book.
That changes the game.
If we optimize primarily for traffic, Meta becomes very good at finding people who click.
Unfortunately, people who like clicking things and people who like buying books are not necessarily the same population.
A cheap click can be wonderful. It can also be very cheap for a reason.
Which means authors need to do more of the filtering ourselves.
One way to do that is through the ad itself: the image, hook, headline, and copy should attract not just traffic, but the kind of traffic most likely to be interested in the book.
And that means testing.
Carefully.
Systematically.
With as few moving parts as possible.
Which is exactly where Meta’s helpful AI can become decidedly unhelpful.
When Cover A is no longer Cover A
Suppose I want to know which of two book covers attracts more reader attention.
Simple enough.
I create:
Ad A → Cover A
and
Ad B → Cover B
Everything else stays the same.
Then I compare the results.
At least, that’s the experiment I think I’m running.
But if Meta quietly creates variations of the images, attaches additional media, changes text, or otherwise “enhances” the ads, Cover A versus Cover B is no longer the only difference.
Now I’m testing:
Cover A plus whatever Meta decided to do
against
Cover B plus whatever Meta decided to do.
Scientifically speaking, we have left the laboratory and entered the casino.
How I learned this the annoying way
While preparing the launch of my latest thriller, Nemesis, I was testing several images.
Normally, even when results surprise me, I can usually see some logic in them.
This time, the numbers were all over the place.
Nothing seemed to fit.
So I did what every advertiser eventually learns to do when Meta produces results that look peculiar:
I assumed I had made a mistake.
I went back through the ads.
Audience?
Fine.
Copy?
Fine.
Images?
Fine.
The obvious AI enhancements?
Off.
Nothing looked wrong.
And yet something clearly was.
Then I noticed a small line under the Media section:
Related media
Beside it were several image thumbnails.
Aha.
Or perhaps:
AARGH.
Meet “Related media”
Related media allows Meta to pull images or videos from your other ads and serve them alongside the media you intentionally selected.
From Meta’s perspective, this makes perfect sense.
If the system has another image that might improve performance, why not give it a chance?
If your goal is simply to sell more of a product and Meta has enough information to optimize effectively, that flexibility may be useful.
But imagine what happens during a controlled creative test.
I am trying to compare Cover A with Cover B.
Meta helpfully decides that my Cover B ad could use some related media.
Perhaps Cover A.
Now readers may see Cover A associated with the Cover B test.
And I am sitting at my desk carefully comparing results from an experiment that no longer exists.
That was essentially what had happened to me.
Meta had “improved” my test into uselessness.
AI isn’t the villain
This is not an argument against Meta’s AI tools. Some of them are quite good.
I’ve seen alternative primary text and headline suggestions that made me think:
Hmm. I wish I’d written that.
And if Meta generates an idea you like, wonderful. Steal it from the machine. Well—perhaps “adopt” is the more diplomatic word.
Then create a separate ad and test that version intentionally.
The issue isn’t whether AI-generated creative is good or bad.
The issue is whether you know what variable you are testing.
If you want to compare two headlines, you need two headlines—not two headlines plus four AI-generated alternatives being served unpredictably.
If you want to compare two images, you need two images—not two images plus related media, animated versions, automatically generated videos, backgrounds, overlays, and your author headshot making an unexpected guest appearance.
A test can only answer the question you designed it to answer if the test stays intact.
Meta wants variety. You want control.
There is a basic tension here.
Meta increasingly wants to give its system more options.
More text.
More images.
More placements.
More formats.
More variations.
Its logic is understandable: give the algorithm enough ingredients and let it find the combination most likely to produce the desired result.
When you are testing, however, you often want exactly the opposite.
You want fewer ingredients.
You want one controlled difference.
You want to know:
Did this hook beat that hook?
Did this image beat that image?
Did women respond differently from men?
Did the cityskype background outperform the symbolic one?
The more variables Meta quietly changes, the harder it becomes to answer those questions.
So during testing, my philosophy is very simple:
Turn the extras off.
Systematically.
Without mercy.
My test-hygiene checklist
Because Meta changes constantly, I’m deliberately not giving you a rigid map of which button lives under which submenu.
By the time you read this, Meta may have moved the button, renamed it, split it into three buttons, or replaced it with a cheerful AI assistant asking whether you’d like to “unlock more performance.”
Instead, I use a process.
Before launching a test:
- Decide exactly what variable you are testing.
Image? Hook? Headline? Primary text? Audience? Change one meaningful thing at a time whenever possible. - Disable creative enhancements you did not ask for.
Text alternatives, image variations, related media, generated video—whatever happens to be available. - Check the media section carefully.
Don’t assume that because the obvious AI switch is off, nothing else has been added. - Study every ad preview.
Not casually. Look at what Meta is actually preparing to show people. - Compare the preview with your original test design.
If you intended Image A + Text 1, make sure that is actually what is being served. - Check again after publishing.
Meta has taught me not to regard “Publish” as the end of quality control.
And if Meta offers you a brilliant enhancement you genuinely like?
Save the idea.
Test it separately.
Keep the laboratory clean
The irony of modern advertising is that AI makes it easier than ever to produce creative—and easier than ever to lose track of what you are actually testing.
That matters particularly for authors.
We usually don’t have enormous advertising budgets.
We can’t afford to spend hundreds of dollars learning the wrong lesson from a contaminated test.
If Cover A really beats Cover B, I want to know that.
If Hook 3 consistently beats Hook 4, I want to know that too.
But I can only trust the result if Meta didn’t quietly introduce Covers C, D, and E while I wasn’t looking.
So use AI.
Borrow its ideas.
Let it inspire you.
Just don’t let it redesign the experiment behind your back.
Because before clicks can tell you a useful story, you have to make sure you know which story you asked them to tell.
And that brings us to the next step.
In You’ve Got the Clicks – Now Make Them Testify, we’ll leave the AI gremlins behind and look at how to structure tests so the results actually mean something.