Why you are never done with A / B testing


At the start of an optimization process on the website, you would like to know when the process has been completed. After all, with most products or services, you agree on what a product or service will deliver after a certain period. A frequently asked question is: when are you done with A / B testing? The answer is simple: never!

The influence of context on behavior

Behavior arises because you are motivated to do something. For example: you eat because you are hungry. This is how behavior comes about. But motivation isn’t the only thing that determines behavior. Context has a great influence on behavior. You know it: you are at a party with a drink in your hand at a standing table with snacks. Chances are that you will eat the snacks without being hungry. The context, in this case the table with snacks, therefore causes you to eat (display behavior).

But what do these snacks have to do with optimization? A lot of! But before I tell you more about that, I want to go back to the basics: why do we actually optimize?

We optimize websites to ensure that visitors reach their goal. We make it easier to buy a bed, book a hotel room or take out a mobile subscription. Through A / B testing we find out what makes visitors get closer to their goal. We draw up behavioral models that we test and adjust with A / B tests. Based on this, we are gaining better insight into how behavior on this specific (place on the) website can be influenced. And that is a very important point: how behavior on the website can be influenced.

How do you respond to context?

What we don’t include in behavioral models is context. So everything that takes place outside the website. For example the weather, an economic crisis or political tensions. These things also influence behavior. Think of how much money people want to spend, how much risk they take and to what extent they are able to make a conscious choice. But also think of smaller things such as someone’s personal context. Is someone looking at your website with full attention in the evening, or is they cooking in the meantime or listening to a meeting with half an ear? This all plays a role.

Look, and that’s where the snacks come back. The fact that there is context. In other words: the snacks on the table influence your behavior. The snacks ensure that you eat at the party. And, the fact that there is an economic crisis means that people spend less money. Likewise, if someone is sitting on the couch with children running around, this will make someone pay less attention to the website.

There is always context

You have no influence on what happens outside the website. But you do have this on your website! You try to set up the website in such a way that it is always easy and accessible for the customer.

An economic crisis may cause people to spend less money, but it does not mean that people do not need a new bed. People will still want to make this purchase. By A / B testing you find out what influence you can exert on the behavior of your visitor, despite the context. This knowledge is very valuable and most likely forms the basis of your optimization process. After all, there is always context.

Best before…

Unfortunately, this knowledge has a best before date. This is because the context is changeable. And, so does the behavior of your visitor. The weather changes, a fad disappears, the economic crisis turns into an economic boom and political tensions disappear.

The longer you experiment, the more likely the context will change. The tricky part is that these kinds of changes often take place very gradually. A / B testing makes identifying such a change easier. You notice that behavior changes and your A / B tests turn out differently than expected.

So you see that less money is spent during an economic crisis. The moment we recover from the economic crisis, you will see that your visitors exhibit different behavior. And that your website, optimized for the economic crisis, no longer performs as before.

Fortunately, A / B testing will help you find out more quickly, so you can intervene in time. This allows you to relearn what your visitor does need (to buy that bed, to book the hotel room or to take out a mobile subscription).

Never finished A / B testing

In short, behavior is subject to all kinds of influences on a website and beyond. We can check this on a website, but not outside of it. Because context is changeable, so is behavior. Continuous A / B testing ensures that you are always aware of change. So you are never done with A / B testing!

3 tips to extend the shelf life of your knowledge:

  1. Be aware that the changing context causes behavior to change. Keep track of the baseline in a dashboard. If numbers are out of line, it could mean that the context is changing. Then it’s time to adjust your A / B testing.
  2. Intervene in a changing event! Do you sell airline tickets and do people suddenly stop flying? Do a data analysis to see what is happening on your website. Last year we conducted corona analyzes at our customers to see what effect this major event had on visitor behavior.
  3. Do regular retesting or back-testing to see if the effect still exists. For example, we have experienced that the biggest winner ever became the biggest loser ever five years later. This was a very clear indication to choose a different course.

In short: always keep A / B testing 🙂


Source: Frankwatching by feedproxy.google.com.

*The article has been translated based on the content of Frankwatching by feedproxy.google.com. If there is any problem regarding the content, copyright, please leave a report below the article. We will try to process as quickly as possible to protect the rights of the author. Thank you very much!

*We just want readers to access information more quickly and easily with other multilingual content, instead of information only available in a certain language.

*We always respect the copyright of the content of the author and always include the original link of the source article.If the author disagrees, just leave the report below the article, the article will be edited or deleted at the request of the author. Thanks very much! Best regards!