Jessica at PPC Hero makes an excellent point in her recent post “Never Odd or Even: PPC Ad Text Testing Made Easy.” It’s a factor in split testing AdWords text ads that I’d overlooked, and perhaps you have also:
To make a valid test, you need to try to keep keywords and landing pages as consistent as possible throughout the ad text test. This is why I said sometimes our desire to optimize gets in the way of our desire to test- we want to add modified broad match keywords! We want to change the headline on the landing page! We wanna use Website Optimizer! But we’re going to screw our ad text test up, and that’s something we’re going to have to keep in mind.
In other words, if you change other variables connected to the ads you’re testing (keywords, landing pages) during the test period, you throw off the validity of the test. In a valid A/B test, you change just one variable (in this case, some difference in text between one add and the other) and keep all other variables the same.
Since if you are properly managing your AdWords accounts you will be changing keywords and landing pages all the time, it might be best to do ad testing on just a few select ad groups at a time, ad groups you’re willing to leave alone for enough time to generate valid test data for the ads you’re testing.