From research “Mismeasuring the Drivers of Ad Performance”
A Conversation with SMU Cox Marketing Professor Gijs Overgoor, Late 2025. A new version of the research is described at the end.
Original interview, diving into using LLMs to handle data on ad performance
Jennifer Warren: What was the motivation for your study?
Gijs Overgoor: We’ve been studying ad creatives and their effects for a while, and we were trying to figure out the best way to identify what’s actually in these ads. Looking at the existing literature, we noticed that most researchers hire and train a group of research assistants to code TV ads against a set of questions. At the same time, we saw a broader trend of marketing and business researchers using AI for this kind of content coding — mostly with text so far, a bit with images, and now increasingly with video.
That gave us a dual problem to work with. On one hand, we have a huge amount of TV advertising data we’d like to study, but we’re limited by scale — research assistants can do a good job, but they can’t code 5,000, 10,000, or 100,000 videos. On the other hand, AI is improving quickly, especially with multimodal models that can process video, text, and audio, but we weren’t sure how good it would actually be at this specific task. So we set out to see whether AI could help solve that scale problem.
JW: You used Alibaba’s Video LLaMA 2 as your baseline LLM. What did you find — where did it perform well, and where did it fall short?
GO: We deliberately didn’t jump straight to the newest technology. We wanted to see how the tools that are directly available to researchers — ideally open source and easy to use — actually perform. Video LLaMA 2, which combines more traditional machine learning with Alibaba’s open-source Qwen 2.0 model, was one of the top performers in that category.
We ran it and compared its output to our human-coded responses. On the surface, it did a reasonable job — it picks up on a lot of information when you ask it questions. But when you dig in, especially at scale, it misclassifies a lot of information and tends to default to “yes” on binary questions. Ask it “Is this an emotional ad?” — yes. “Is this a funny ad?” — yes. It looks like a reasonable response on the surface, but once you convert those answers into actual yes/no variables, it mismatches broadly, and it’s not easy to predict where it will get things wrong. That inconsistency can have serious downstream consequences if you rely on it — for example, if you’re making ad-spend decisions based on the assumption that the model’s read on an ad is accurate.
JW: You also fine-tuned a model you call “Ad Notator,” which ended up outperforming the human labelers. Can you talk about that finding?
GO: We took the baseline model and asked whether we could train it into a better annotator for ads specifically. We fine-tuned it on the questions and answers our human coders had produced, and it improved substantially — it picked up on the information needed to answer a wide range of questions accurately. Some subjective categories, like emotion or humor, are still somewhat open to interpretation, but in most cases the model agreed with the humans, especially where we had a majority answer across multiple coders.
That’s actually where it got interesting: for ads that had at least three human coders, our fine-tuned model — combined with the human training data — outperformed the very humans it was trained on. Using that broad base of AI knowledge and inserting ad-specific training data filtered out a lot of the noise and the reflexive “yes” answers from the model, but it also filtered out the noise from the humans — the instances where a coder might miss something or find a question ambiguous. That combination really cut down on inter-annotator noise.
JW: You also tested Gemini Flash, which performed better than the off-the-shelf baseline model but still not as well as your fine-tuned model. What did you find there?
GO: It’s worth acknowledging that there’s probably going to be a model that does all of this a little better at some point — that wasn’t really the point of the exercise. The point was to understand what happens when you use a model as-is, before you’ve figured out how reliable it actually is for this task.
To our surprise, Gemini still defaulted to “yes” on a lot of questions. That’s a broader tendency across many LLMs — a kind of sycophancy, where the model is almost trained to please the person asking. Gemini’s answers ended up close to human levels overall, but it still misperformed on a range of individual questions. Our fine-tuned model — despite not being the most current or best-performing open-source model available — was still more accurate than Gemini at moderating that “yes” rate.
JW: So your model wasn’t as much of a “yes man.”
GO: Exactly — and I think we have enough of those already. It’s worth being critical rather than just agreeable.
JW: At a high level, what were some of your findings about what actually makes an ad successful?
GO: The outcome measure we used came from iSpot, a TV advertising platform that lets advertisers pretest and track ads — including things like how often viewers turn the TV off. iSpot maintains a consumer panel of at least 500 U.S. customers who rate ads on a range of items, which rolls up into something close to a persuasion or ad-quality score. They also ask viewers about purchase intent — whether the ad made them more or less likely to buy.
If you’re using AI to study a phenomenon or to pretest ads, getting this wrong can lead to badly incorrect conclusions — and then spending real money running ads that aren’t actually right for your brand. — Overgoor
We had our human coders rate the same 2,000 ads on questions grouped roughly into action, information, emotion, and sensory categories — sensory meaning things like whether the ad is visually pleasing or sounds good. A few findings stood out. Action-oriented content — explicitly encouraging someone to buy, to go to the store, to go online — was actually negatively associated with perceived ad quality. Informational content — details about the product, the brand, the price — was positively associated. And sensory qualities had a notably strong positive effect: if an ad was visually pleasing, cute, or had sex appeal, it was well received. Pure emotional or “feel-good” content, on its own, didn’t show the same positive association.
JW: This work could serve as a benchmark for using these tools responsibly going forward. Can you talk about that?
GO: The main takeaway is that you can use AI tools to answer questions about ads, and the output looks reasonable on the surface. But when you actually study how those answers influence downstream measures of consumer response, the results can look very different depending on whether you use our baseline (Video LLaMA), a model like Gemini that defaults heavily to “yes,” or human coders — who, it turns out, are also fairly noisy annotators and can miss or misidentify features that relate to ad response. Our fine-tuned model reduces both that noise and that bias, which gets you more accurate downstream measurements.
So yes, you can use an off-the-shelf model for this kind of task, but you should vet it carefully. You can use our model for similar questions, or — and this is something we’re continuing to work on — future researchers should ideally collect their own data and train a model on it to embed that domain-specific ad knowledge. Even with the most current frontier models — OpenAI’s, Claude, Gemini — it’s still worth running additional tests to check you’re not introducing bias into your advertising research. If you’re using AI to study a phenomenon or to pretest ads, getting this wrong can lead to badly incorrect conclusions — and then spending real money running ads that aren’t actually right for your brand.
JW: Any final thoughts on how this helps advertisers decide where and how to spend their creative dollars?
GO: There’s also a question of whether your best-performing ad — the one people most enjoy watching — is actually the one that drives revenue downstream. In some follow-up work, we’re finding that those two things can diverge. It’s important to ask different kinds of questions to figure out what’s actually working about a given ad for a given audience, and I think our tool can help tease that apart.
For example, in a related paper where we applied this tool, we found that viewers are more likely to turn off the TV when an ad is more informational, even though the action-oriented and informational elements are actually more effective at getting people to the store — it just comes at the cost of how much people say they enjoy the ad. We tend to think about Super Bowl ads as the gold standard because they’re clever and entertaining, but the open question is whether those are the ads that actually drive what advertisers want to drive.
Link to SMU Cox write up and full video interview at bottom. Posted Jan 2026.
Update per Overgoor: “The new version of the paper shifts more toward understanding the impact of advertising, whether large-language models (LLMs) can measure it, and — if we can get at what a smaller sample of humans think — how to correct the estimates obtained from the LLM.
A key issue is that LLMs make inferences about ads that can sometimes have the outcome baked in, which is a problem when you want an independent assessment of the advertisement or the specific element of the ad you’re studying. Humans do this far less. From there, if you have a small human-coded sample alongside a full set of LLM codings, you can compare not just the elements and agreement between the two, but also the downstream relationship each produces once entered into a statistical model.
Our paper then offers guidance on how to infer what the “true” relationship likely is, and what to watch for along the way.”