Running growth experiments isn’t rocket science. However, many founders and startups overlook the importance of structuring their experiments and tests and get too excited by the test itself. When you are testing something within your business, learning is paramount. Everything you can learn will further guide your business towards success. When you have qualified a new idea for your business, the following growth experiments template is an excellent way to lay it out so that you and everyone on your team knows what the test is, why you are doing it and whether it worked or not.
The 7 steps of an experimentation framework are:
This is a brief synopsis of what the problem is or the area you are trying to improve.
Example: No one is opening our emails so no one is seeing our amazing deals.
Every test should start with a testable hypothesis. For example:
By [doing] or [not doing] [something – an action] we believe that [a metric] can be impacted.
Example: By changing the time we send our emails, we believe we can increase email open rates.
A control is what we are measuring against. Usually historical data, it serves as your baseline.
Example: Over the past month, our email open rate has been 12% when we send at 8am.
This is the entity that you are changing, adding, or removing which is measured against your control. You may have more than one variant that you would like to run either simultaneously or sequentially.
Example: We are going to test sending emails at 2pm and 6pm.
Determine how you will measure the results and over what time period. You should establish how to measure before starting anything. Make sure you know what’s happening. Also make sure you stay consistent in what you are measuring by using the same set of data from the same platform throughout. Consistency is key in ensuring you have results that you trust.
Example: We will measure the email open rates using our email provider for the next month – we will finalize the results a week after the last email of the month is sent.
What did we learn with each test or variant?
Track detailed notes and metrics on a daily or weekly basis. Testing doesn’t happen in a silo so be sure to make note of any external or internal affects which may have had an affect on the results.
Example: Variant B was over 100% more successful each week than our baseline so our test was a success. Variant A and B both outperformed the baseline across the month with Variant B being the most successful – it appears the later we send the email, the more likely it is to open. Other notes included a successful paid acquisition campaign so we had a lot more qualified emails come in for Week 4 which may explain it being the strongest week.
Will you run another iteration?
Is this a test to kill?
Will you try something else?
Does the test become a roadmap item?
Example: We will update emails sent to later times and kill the time based element of the test. Given the large increase in performance, we will also start testing what days we send email for a new item.
Learn more about running great experiments
This layout is designed to be used from start, middle to end of experiment and give uniformity and visibility into testing which helps get team buy-in when you want to go from successful test to product build.
We dig into this further as well as how to decide what experiments to do and so much more in our Mastering Growth program. You can get access for the first week for only $1 so check it out!
If you love this, check out our Experiment training package!