Since experimentation is the key to learning cheap and fast, we made it a fundamental component of Playing Lean.

But how does one experiment?

One of the five principles of Lean Startup is the Build - Measure - Learn loop. In order to design an effective experiment you have to start backwards! Ask yourself - what is it that you want to learn? How can you measure that? What do you need to build in order to learn that?

Playing Lean Experiment Report is a template that guides you how to design an experiment and how to capture the learning from that same experiment. 

Let's go through each block.

Design an experiment

1. Background

Why are you doing this experiment? What is the context? Is there a decision to be made? How does it relate to what you are trying to achieve? Are there some assumptions that need to be tested?

Those are some of the questions you should be asking when writing down the experiment background. This is not an "admin" field, and will provide much needed context for interpretation once the results are in. 

Each experiment report should be self-standing, meaning that people that haven't run should be able to make sense of it. It's good practice that'll help you onboard new team members.

Background should be concise and to the point. No problem if you begin verbose, but work on cleaning and simplifying it until you have a sharp background statement. Don't be afraid to include numbers and visuals that support it. Use the limitation of the medium as a creative constraint.

2. Falsifiable Hypothesis

Translate your assumption into a falsifiable hypothesis by restating it with numbers.

We have included the simplest format to get you started, which is:

[Specific repeatable action] will [expected measurable outcome].

Example:

Adding photographs to our listings will reduce the time it takes for the listing to be booked by 30%.

Once you feel like you are getting comfortable with phrashing hypotheses, I suggest to start using one of the following two formats:

Hypothesis format from The Real Startup Book:

The change - the metric - the impact - the timeframe.

Example:

If we add a lock icon next to the credit card information, the completion of the checkout process will increase by 15% in 3 months.

XYZ format by Alberto Savoia:

At least X% of Y will Z.

Example:

At least 15% of dog owners will add a six-pack of beer for dogs for $4 when they buy dog food.

Depending on how much you know you might not be able to formulate a falsifiable hypothesis. If that is the case, rename the field to Assumption, and specify what is that you want to learn.

3. Details

How are you going to test the hypothesis? Who are you going to test it with? Don't just write experiment type (e.g. customer interview), be specific! 

Think of cookbooks, and lay it down step by step. It is easy to bungle up experiments by our interference, and that's why it is important to have the steps clearly written. Some experiments are linear (you can execute them by following steps in the exact order), some are not, and that should not prevent you from writing them down.

I usually list experiment type, key questions, and experiment outline. Everything else (e.g. interview script, variant images, etc) I either include as attachment or link to.

Capture the learning from the experiment

Once you have run the experiment you want to capture that sweet, sweet, learning.

4. Results

Start by putting down raw numbers. No interpretations, no fluff, just write them down. 

Here are some examples:

7/11 interviewees ranked problem #2 as the top issue.

7% (432/6177) of website visitors left their email.

21% (23/111) of commuters purchased the £17 milkshake with thin straw.

Easy!

5. Validated learning

This is where you capture your interpretation! 

Transform into Dora the Explorer or The Grand Inquisitor, whichever imagery you prefer, and ask the following:

• What do the numbers mean? 
• If they are below what you expected, why? 
• If they are above what you expected, why? 
• How do the findings relate to the experiment background and the falsifiable hypothesis? 
• What was clarified? 
• What was not clarified?
• What new facts or uncertainties have been surfaced? 
• What are some options for going forward? 
• What are the implications for future development?

Again, it is all right if you begin quite verbose, and then work through it to make it concise.

6. Next action

Specific, actionable next step.

Pivot? Where to, why, and how? 

Persevere? More experiments? If so, which one and what for? If not, what activities, when, by whom, and by what time? 

Perish, perhaps? There is no shame in deciding to stop exploring certain opportunities or leads, if it is backed by your findings.

Once you have written down your next action, take a moment to reflect upon the whole experiment.

Read it in reverse order, from next action to background. Then read it in proper order. Does the action you are proposing make sense? Is everything logically connected? Give it to a colleague, and ask them to do the same. If anything pops up then revisit Validated learning and Next action.

May the fastest learner win!

Playing Lean Experiment Report is one of the templates Playing Lean Facilitators get access to. It has been adapted with permission from Ash Maurya.