Challenges with experimentation
Experimentation is one of the most crucial processes in any marketing department. All aspects of campaigns such as creatives, messaging, targeting, new channels or media sources need to be experimented with as it’s not predictable where and how to find the most valuable customers.
Although data science is central to today’s marketing, teams often have to rely on limited internal or external resources to analyze campaigns and carry out experiments.
For instance the standard practice to analyse the results of an experiment is significance testing. It’s a way of comparing two variations (A/B) of a website, app, creative, campaign or any other marketing facet to determine which is the most effective at reaching the desired goal. However, there are some inherent and prominent issues associated with significance testing, such as long test periods, inconclusive results, lifts not being quantified, limitations to the number of variations that can be tested at once and most importantly the dependency on expert resources to conclude tests.
Always-on experimentation to the rescue
Always-on experimentation eliminates these issues by providing a drastically more efficient experimentation process.
Implementing always-on experimentation can have a tremendous impact on how businesses conduct their marketing practices and drive growth.
The process needs to be supported by software that automates statistical modelling and simplifies the setup and analysis of experiments. By implementing always-on experimentation marketing teams are able to carry out and understand the results of experiments on their own, take full control of their experimentation roadmap and are ultimately enabled to make data-driven decisions at pace.
Benefits of always-on experimentation
Independence from expert resources
Manually setting up and analysing tests can enormously slow down all testing and learning activities. Specifically significance tests require elaborate preparation, usually under the supervision of a data scientist. After successfully implementing always-on experimentation marketers have tools in place that will automate these manual and time-consuming steps so they can focus on the analysis of their data.
Ability to make data-driven decisions fast
For most businesses, data is a trustworthy and reliable source of information. Learning from it, and enabling informed decisions based on observations at pace is the ultimate goal of always-on experimentation.
Harnessing a culture of experimentation
Experiments should be an integral part of not only every marketing team but also every business. They give way for people to learn which products, solutions, activities or innovations work, which do not, and how it’s possible to improve them. This article describes how to incorporate an experimentation culture in great detail.
Snowball effect on growth and performance
The combination of the mentioned benefits increases the testing velocity, leads to faster learnings and can yield a competitive advantage. Always-on experimentation has an exponential effect on business operations, results and development.
Why it is required to work agile when implementing always-on experimentation?
Agile marketing and always-on experimentation are deeply intertwined and more than just complementary methodologies. The implementation of both methodologies can significantly enhance the production, efficiency and quality of all marketing activities.
Agile marketing is an organizational effectiveness strategy in which teams identify and focus their collective efforts on high-value projects, complete those projects cooperatively, measure their impact, and then continuously and incrementally improve the results over time.
If done right, agile marketing teams can carry out large amounts of campaigns while developing new ideas on a weekly basis.
Here is a link to a guide on how to implement agile marketing into your business.
Cornerstones of agile marketing
Collaboration is key
A communicative work process filled with feedback and suggestions from people who have the same work-oriented goals is one of the focal points for why agile marketing is so useful.
Reactivity over following a plan
Agile marketing is all about iteration and being adaptive to new information. When working in an agile way this is more important than dedicated adherence to a scheme.
Constant modification, testing and iteration over slam-dunk projects
Huge campaigns are the complete opposite of the goals of agile marketing since completing them takes a lot of time, effort, and resources.
Data-driven decisions over opinions or general wisdom
It is important to base the decisions of the team on actual data that is flowing from your marketing channels – otherwise, the resolution might be unfactual, inaccurate and even costly. Understanding and analysis of your campaigns and metrics is an absolute must before making a decision.
Multiple small experiments are the goal
It's inefficient and resourceful to experiment with a few big jam-packed campaigns that take a lot of time to implement. A lot of small experiments are a much safer bet.
Combining both methodologies
From performance, functionality and practicality perspectives agile marketing and always-on experimentation work the absolute best when implemented together. When a business implements both the efficiency and results start to increase exponentially due to their endogenous relationship. When one process starts to be used it increases the effectiveness of the other and, in turn, this increases the effectiveness of the former process. Moreover, the principles of agile marketing validate for a tool that helps marketers do their job fast, make data-driven decisions and experiment continuously to not miss out on new opportunities.
We’ve seen non-agile teams trying to implement always-on experimentation and the results are usually falling short from the expectations. If your team operates in a more traditional way, creates few big marketing campaigns a year, and does not rely on data to constantly learn and make better decisions, then implementing always-on experimentation is bound to fail. For it to work it needs a constant flow of data, many small experiments, and, most importantly, the right mindset coming from the team.
Software that supports always-on experimentation
While there are many tools available that support agile processes and planning, there are not many that will allow teams to run always-on experimentation.
Our experimentation engine Quantify is the only tool that supports always-on experimentation across any marketing channel and media source out of the box. Quantify takes a new approach to making core marketing performance metrics credible and actionable by adding robust statistical models that qualify performance metrics for all tested variation. Without the need for manual interaction of dedicated experts, Quantify enables marketers to easily track and drill into ongoing campaigns and tests, and retrospectively explore past data for insights.
Here are some of the ways how Quantify eases the process of data analysis and executing marketing experiments:
Test peeking
Unlike with significance testing that requires a concrete number of observations before the results can become conclusive, Quantify allows marketers to look at the results throughout the test. As results are available from day one it can show early trends or issues with the setup of the test itself.
Flexible number of variations
Due to machine learning and advanced statistical modelling, it’s possible to test a nearly unlimited number of variations at once. Furthermore, variations can be added and removed during tests, without the need to start over again.
Quantification of results
Equally important is that traditional significance testing makes no statement on the actual lift measured in the test. It indicates whether the hypothesis is statistically significant, but doesn’t quantify the lift. If your hypothesis is that variation A has a 5% higher ROI than B and it turns out to be significant, you still don’t know how much highter it actually is (it might be 10% higher). Quantify always provides a clear indication of lifts.
~10x less data required
In comparison to significance testing Quantify requires 60-90% less data (less observations) to provide statistically accurate results. That means you will have to spend up to 10x less budget to produce the same learnings while being able to act much faster.
No prior setup
Once your data sources are onboarded (which is usually a one-time effort) tests can be analysed without any further setup. Quantify provides marketers with all the necessary data science tools through a comprehensive and intuitive user interface.
Exploratory experimentation
Additionally to the confirmatory approach to experimentation that requires formulating a hypothesis that can be falsified, Quantify enables a whole new approach to experimentation: Exploratory experimentation enables teams to discover insights that they wouldn’t have thought of coming up with a hypothesis for (more details can be found in this article).
Conclusion
Agile teams that leverage always-on experimentation are proven to not only be faster, but also more efficient. Our experience shows that companies that adhere to more traditional marketing practices have trouble implementing always-on experimentation. This is because the mindsets that are latently attached to these differing methodologies are clashing. Utilising always-on experimentation in an agile environment can exponentially improve the results produced by marketing teams.
Get in touch if you want to talk about implementing always-on experimentation in your team.
