Common A/B Test Pitfalls to Avoid

Common A/B Test Pitfalls to Avoid

Most A/B tests are designed in a way that all but ensures the learnings will not translate into an ongoing lift in sales.

Ultimately, to know with confidence that one creative variation will outperform another, tests must be deployed at the user level, and not the impression level. When tests are run at the impression level, results are far more likely to indicate no difference between creatives. That’s because impressions don’t engage with brands, people do

The large number of inconclusive A/B tests resulting from testing impressions rather than users unfortunately discourages many advertisers from continued testing and optimization. User-focused digital marketers are smarter, however, and demand creative A/B testing that is administered at the user level.

Users Split by Line Item

Tests are all-too-often conducted by comparing the performance of two campaigns or line items being run simultaneously. The problem with this approach is that it does not hold the audience constant, which is a fundamental requirement of A/B testing. Different line items, even with identical settings, will inevitably access slightly different inventory over the course of the test. A proper A/B test in digital advertising must be run on a single line item with the ad server assigning users into groups at each impression so that audience used for the test truly is identical across the test groups.

Testing Plan Siloed from Media Plan

It is critical that creative A/B testing is coordinated with your media agency or in- house programmatic team. Programmatic media buying that is not coordinated with creative optimization can undermine creative tests in the following ways:

Buying low CPM inventory that is less viewable limits the impact of any creative
As discussed in the introduction, creative is growing in importance for multiple reasons, one of which is that digital advertisers are increasingly getting the viewability and attention to their ads that they have been missing. Your creative testing will only be as informative as your ads are viewable. Ad placements that compete with 5-10 other placements on the page may be low in CPM, but the attention garnered makes creative less effective, thus making creative testing less effective. If you’re impressions are 40% unviewable, 40% competing with 5-10 other placements, and only 20% both viewable and prominent, the impact of creative will simply be minimal.

Targeting the same users in the A/B test with other programmatic campaigns
Sometimes when an agency sets up a creative A/B test, they generate a separate placement in the ad server for the test and trafEic it to a line item or package with their trading desk. Such tests are less likely to measure the actual differences in performance between creative variations, because users in the test are being served creatives from other programmatic campaigns at the same time. When providing multiple placements or ad tags to a trading desk, ensure that the trading desk isolates users in a test from other programmatic campaigns.

Creative Test Obscured by Media Attribution

Ad servers, other than Adacus, apply attribution models by default to all conversion reporting. In other words, when a user converts after having seen ads from a test as well as ads from other placements or channels, the ad server will likely not attribute that conversion to the test group to which the user was assigned.
This makes A/B testing all but impossible, for two reasons:

  1. Attribution across channels removes most conversions from the results of an A/B test, thus increasing the amount of testing time required to achieve significance from weeks to months.
  2. Multi-channel attribution introduces noise into the A/B test results. Evaluation of an A/B test does not require multi-touch attribution as users are only presented with the A ad or the B ad throughout the duration of the test.

The solution is to report creative test performance separately from media performance across channels. In the creative test report, include all users that converted after seeing an ad in the test regardless of whether or not they were exposed to media from outside of the test as well.

Learn how to avoid more common pitfalls in our eBook

"The Essential Guide to A/B Testing for Digital Advertisers"

What is Statistical Significance?

What is Statistical Significance?

Excerpt from free eBook, "The Essential Guide to A/B Testing for Digital Advertisers"

Marketers perpetually ask if test results are “statistically significant,” but what does that term even mean?  Probably not what you think.  

Traditional statistics relies on p-values as a measure of the “statistical significance” of test results.  Most marketers are surprised to learn that p-values do not, in fact, measure the probability that one creative or treatment will outperform another.  What p-values measure is far more abstract and removed from the decisions that marketers make based on A/B tests.

In every A/B test one variation will perform at least slightly better than the other.  P-values measure the probability that a test result (say, creative variation B outperforms creative variation A by 10%) would have occurred if in fact there were no difference between the two creatives at all.  That “95% confidence level” threshold you’ve probably heard bantered about simply means that there is a 5% chance that, were the two variations identical, you would have observed as large a difference in performance between them as you did.  The p-value is an important measure in other fields of study to account for what is known as in traditional statistics as Type I error.4In our experience, we have yet to hear a digital marketer ask us for this specific probability.  And why would they.

Learn why Adacus uses Bayesian statistics in our latest eBook

"The Essential Guide to A/B Testing for Digital Advertisers"

Why We're Excited to Exhibit at  AD:TECH NYC 2016

Why We're Excited to Exhibit at AD:TECH NYC 2016

Adacus, like any growth-stage company, has to pick and choose conferences very judiciously.  For better or worse, the digital marketing industry offers more than it's fair share of conferences from which to choose.   

We've gone big at DMEXCO - a 17-ring circus of digital marketing shock-and-awe - and small at Mediapost's OMMA Programmatic - an intimate gathering of understated digital marketing professionals.

Ad:tech, on the other hand, sits comfortably somewhere along the continuum between those two. Adweek ranks it as one of the top 10 digital marketing conferences to attend.  By the conference's own account, "is the original industry authority for marketing and media technology, where marketing, technology and media communities assemble to share new ways of thinking, build strong partnerships, and define new strategies to compete in an ever-changing marketplace."

Beyond the fancy language, we're excited to exhibit among other companies hustling to leverage technology for positive change within some niche of the greater industry.  It is the place to learn what's next and to talk with the companies making it happen.

Come meet Adacus at booth NX20 on the exhibition floor.  There just might be some swag in it for you.

What Comes After DCO? Creative-Side Platforms

What Comes After DCO? Creative-Side Platforms

This article originally appeared in The Drum on September 21, 2016.

Dynamic Creative Optimization (DCO) plucked the low-hanging fruit of creative optimization in digital advertising. By turning a creative into a creative template, DCO changes the value of a text or image element using a large data feed, usually location or last product viewed. But once a marketer has dynamic locations and product images, what comes next? If DCO excels at contextualizing a single creative, a Creative-Side Platform (CSP) goes further by optimizing an entire creative plan. 

CSPs represent as big an opportunity for digital marketers as DSPs. CSPs do for creative plans what DSPs did for media buying. That DSPs preceded CSPs is a matter of economics – arbitrage is a big money-maker, so that’s where investment capital went first. But this also tells us something about the potential of CSPs – they drive lift without a penny more in media spend.

If DCO excels at contextualizing a single creative, a Creative-Side Platform (CSP) goes further by optimizing an entire creative plan.

What exactly is a CSP? A CSP is an ad server with tons of audience data. CSPs access audience data by integrating with DMPs but also by building in third-party audience data like demographics and interests. CSPs leverage all of this audience data with endless creative testing, decisioning and optimization tools that make it easy to develop and deploy a creative segmentation strategy.

You can run an A/B test that shows which creative variation wins with men vs women, with parents vs non-parents, or with new customers vs long-time customers. You can figure out how to drive sales from millennials by running an A/B/C test of 3 creatives only when a millennial is served an impression. And, ultimately, you can deploy a creative decision tree that captures the creative segmentation insights generated from all this testing.

CSPs represent the elevation of programmatic from an ad delivery platform to an insight-generating platform. The messaging segmentation strategy that results is a proprietary asset of the brand, as tightly guarded as the segmentation strategy developed by traditional database marketers. In fact, that’s a telling comparison, as CSPs represent the introduction of martech best practices into digital advertising. DCO, by comparison, is a thoroughly adtech phenomenon.

Hold it, you may be thinking. I can already serve different creatives to men and women using my DSP. Not so fast.

First, media targets are not creative targets. Men respond to a message tailored for men regardless of how much the media they consume costs. The latter, media costs, are incredibly dynamic from day to day, as anyone who has worked the controls of a DSP console can tell you. A creative segmentation strategy, on the other hand, is more stable over time, though it does evolve with changes to the brand and to consumer preferences over time.

Second, ask an ad operations team whether they can serve 5 creatives to 5 different demographic and interest segments. Here’s the ad ops ordeal this creates: buying 5 media targets from a DMP, creating 5 line items on the DSP, cookie syncing the 5 media targets with the 5 line items, generating 5 ad tags from your ad server, and trafficking those ad tags to their respective DSP line items - and then testing all of this to make sure no mistakes were made. Even then, your creative decisioning rules are ignored by all other line items on your DSP, other DSPs you use and direct buy publishers.

Third, testing to see if this creative decision makes sense on the DSP isn’t possible because creative testing requires holding the media constant between creative groups. Creative testing has to be done on an ad server.

The slow feedback loops of direct marketing channels are about to be replaced with far faster and more data-rich feedback loops from programmatic. CSPs are the orchestrators of this feedback loop and will not only drive future lift in digital advertising, they will drive the segmentation insights that are leveraged across all direct marketing channels.

Adacus Goes Beyond DCO at DMEXCO 2016

Adacus Goes Beyond DCO at DMEXCO 2016

Just in case you haven't heard of it, DMEXCO is - as the kids say - "kind of a big deal."

The Digital Marketing Exposition & Conference hosts more than 800 exhibitors and 30,000 visitors annually from around the world, making DMEXCO one of Europe's largest digital marketing conferences.

Yes, Europe.  Not Mexico.  Cologne, Germany to be exact.  This year the two day event will be held on September 14-15 and Adacus will not just have a booth - we will be presenting.

Mark your calendars for September 15 at 1:30pm (local Cologne time) to hear Joel Sadler share how the Adacus Creative-Side Platform can help digital creative teams break free from the restrictions of DCO to deliver personalized messaging.

Come by our table in Hall 6 anytime during the conference to chat with our team and learn more about the Adacus CSP.

In the meantime, check out latest FAQ "Adacus v DCO" to see how the Adacus Creative-Side Platform goes beyond DCO.