Testing Overview

As creative is growing in importance, so too is creative A/B and multivariate testing. Creative testing is the digital feedback loop that powers agile marketing and continuous creative optimization.  Despite its growing importance, creative A/B and multivariate testing is broken, as other ad servers simply don't allow A/B testing and thus attempts to do tests don’t translate into ongoing sales lift. This is an enormous untapped opportunity for digital advertisers who understand proper testing.

Fixing Creative Testing is Critical and Urgent

Fixing A/B testing in digital marketing has never been more urgent. Creative testing mattered less when impressions were competing with numerous placements on low-value inventory or were unviewable, because creative itself mattered less then. But with the higher quality that brands are increasingly demanding, and the leaner ad experiences being rolled out by publishers, advertisers are finally getting the viewability and attention to their ads they have longed for, and must not squander customer attention with untested creative.

As creative grows in importance, traditional copy testing just won’t do. Pre-testing with a panel is slow, costly and not reflective of the constantly changing media contexts in which ads are experienced. The pace and volume of creative production demands continuous delivery of creatives, leveraging mid-Elight feedback to optimize creative and creative plans.

Two Types of Creative Tests

There are two types of tests that a digital marketer uses to optimize creative: A/B or multivariate tests and creative rotation attribution Tests.  

A/B and Multivariate: The first, A/B and multivariate tests, provide insights about creative elements within a creative.  These tests answer questions such as the following: Do price or feature-oriented ads perform best?  Do multi-frame or single-frame ads perform best?  Do feature-oriented ads outperform price-oriented ads with any audience segments?  Do images improve the performance of price-oriented ads or detract from the performance?  And so on.  

Creative Rotation Attribution: The second, creative rotation attribution tests, provide insights about which creatives to include in a rotation.  Advertisers typically produce dozens of creatives over time and end up with large rotations as a result.  In fact, with the average impression frequency of prospecting campaigns at 1-2 impressions per user per month, the number of creatives in most rotations far exceeds the number of creatives that users actually see.  That's why it's important to identify the subset of creatives to include in a rotation that will maximize engagement and conversion.  Creative Rotation Attribution tests are special multivariate tests that serve different rotation combinations to different users in order to assess the partial attribution of conversions to each creative and decide which creative to include in a rotation.

Testing Overview

Pitfalls to Avoid in Creative Testing

  • Faint Signal Problem
  • Not Controlling for Media
  • Test Plan Siloed from Media Plan
  • Rotation-based Testing on DCM, Facebook

A/B and Multivariate Testing

  • Test Planning
  • Test Setup
  • Analyzing Test Results

Creative Rotation Attribution Testing

  • Test Planning
  • Test Setup
  • Analyzing Test Results