How to run growth experiments
Running growth experiments isn’t rocket science. However, many founders and startups overlook the importance of structuring their experiments and tests and get too excited by the test itself. When you are testing something within your business, learning is paramount. Everything you can learn will further guide your business towards success. When you have qualified a new idea for your business, the following growth experiments template is an excellent way to lay it out so that you and everyone on your team knows what the test is, why you are doing it, and whether it worked or not.
The 7 steps of an experimentation framework are:
Overview
This is a brief synopsis of what the problem is or the area you are trying to improve.
Example: No one is opening our emails so no one is seeing our amazing deals.
Hypothesis
Every test should start with a testable hypothesis. For example:
By [doing] or [not doing] [something – an action] we believe that [a metric] can be impacted.
Example: By changing the time we send our emails, we believe we can increase email open rates.
Control
A control is what we are measuring against. Usually historical data, it serves as your baseline.
Example: Over the past month, our email open rate has been 12% when we send at 8am.
Test
This is the entity that you are changing, adding, or removing which is measured against your control. You may have more than one variant that you would like to run either simultaneously or sequentially.
Example: We are going to test sending emails at 2pm and 6pm.
Measurement
Determine how you will measure the results and over what time period. You should establish how to measure before starting anything. Make sure you know what’s happening. Also make sure you stay consistent in what you are measuring by using the same set of data from the same platform throughout. Consistency is key in ensuring you have results that you trust.
Example: We will measure the email open rates using our email provider for the next month – we will finalize the results a week after the last email of the month is sent.
Results
What did we learn with each test or variant?
Track detailed notes and metrics on a daily or weekly basis. Testing doesn’t happen in a silo so be sure to make note of any external or internal affects which may have had an affect on the results.
Example: Variant B was over 100% more successful each week than our baseline so our test was a success. Variant A and B both outperformed the baseline across the month with Variant B being the most successful – it appears the later we send the email, the more likely it is to open. Other notes included a successful paid acquisition campaign so we had a lot more qualified emails come in for Week 4 which may explain it being the strongest week.
Next steps
Will you run another iteration?
Is this a test to kill?
Will you try something else?
Does the test become a roadmap item?
Example: We will update emails sent to later times and kill the time based element of the test. Given the large increase in performance, we will also start testing what days we send email for a new item.
Learn more about running great experiments
We have a whole set of growth models in our Velocity-Growth-Mastering-Growth-Tools spreadsheet.
If you want to dive a lot deeper, I’d urge you to watch our Mastering Growth program. We used to sell this for $1,000 per seat, but it’s now free 🙂
Craig Zingerline is a 6 time founder who has helped dozens of companies scale their growth. Prior to Velocity Growth, Craig was the Chief Product Officer @ Sandboxx, Head of Growth at Upside Travel, CEO of Votion, Head of Growth at Red Tricycle, and VP at New Signature. In addition to in-house roles, Craig has advised and consulted with dozens of high growth startups (4 exits). He’s an award winning product strategist who has mentored hundreds of founders on growth, marketing, and product management.
