Skip to content

Structured Experiments

Experiments are used to enable continuous improvement by producing insights that can lead to incremental change. Any team or individual can conduct an experiment. As we continue to learn from our experiences, everything we do can be considered an experiment.

Structured Experiments, are purposeful and targeted experiments, with specific purposes and defined parameters.

Experimentation can be time consuming and costly, so it is important to ensure they are performed effectively, with minimal waste, and that the insights produced are maintained.

Performing Structured Experiments

Anyone can perform a Structured Experiment. Typical scenarios include:

  1. A team have identified a constraint and have an idea of how to remove it
  2. A hit squad are working on solving an inter-team problem, and have identified a possible solution
  3. An individual wants to try a new approach to personal work, and wants to share their learnings

Structured Experiments can be performed at any time, so long as those impacted by the experiment are aware and have agreed with the proposed approach, and that the experiment does not knowingly negatively impact another experiment or area of work. Be mindful of introducing WIP, and the burden of additional cognitive load.

Structure

A Structured Experiment requires the following parameters, actions, and outputs:

Parameters

  • Objective: what we are looking to learn, what problem are we solving
  • Hypothesis: what we are expecting to happen given specific experiment being performed, including success criteria.
  • Definition: what the experiment be testing, and how the experiment will run, e.g.:
  • Timeframe: the start and end of the experiment
  • Ownership: those who will be responsible for creating and performing this experiment, and who will be accountable for its outcomes?

Actions

  • Planning: Structured Experiments require planning to ensure that they are able to be successful.
  • Monitoring: Structured Experiments require data to be gathered, as well as oversight to ensure that they are running effectively.
  • Analysis: Data and outcomes must be analysed in order to produce insights. Analysis should include environmental influences.
  • Conclusion:
    • Summarises the outcome, determining whether the hypothesis has been proven or disproven, and whether the experiment has been successful or not.
    • Makes recommendations for next steps.
  • Knowledge Sharing: Stakeholders must be appraised of the conclusion, and this information must be made accessible for future review.

Outputs

  • Documentation including:
    • All experiment parameters, which will provide a complete written summary of the experiment that has been/will be performed
    • Data captured and analysis notes
    • Conclusion report

Pollution

Interactions between activities may cause unintended side affects, or pollution. It is important to consider:

  1. How will this experiment interact with other experiments.

    Running multiple experiments concurrently may make it impossible to identify which experiment caused what outcome. Consider overlapping areas of concern between experiments, and how they are timed.

    Additionally, multiple experiments can reduce focus, cause confusion, and reduce buy in.

  2. How will this experiment interact with other work.

    It is important to be aware if there is sensitive, complex, or time critical work in progress or upcoming which may negatively impact, or be impacted by, an experiment.

Outcomes from Structured Experiments

Successful experiments are experiments that produce useful insights in an effective way. This means that the experiment may prove or disprove a hypothesis, and still be considered successful.

There are therefore 3 primary outcomes:

  1. Success, hypothesis proven
  2. Success, hypothesis disproven
  3. Failure, experiment unable to prove or disprove hypothesis

Whichever outcome, it is important that the learnings are captured and made available for future use. As experiments are subject to the environmental conditions, it is critical that as much of the environmental context is also captured.

It is especially important that when an experiment concludes with failure, that the reasons for failure are determined and communicated.

Ownership

Owner Responsibility
Experiment Team Responsible for defining parameters, performing or coordinating actions, and producing outputs. Accountable for the outcome.
Participants Responsible for participating in the experiment, performing the actions and providing feedback or producing data.

Suggested Flow

  1. Scope: Set out the objective and define a hypothesis. This process may require research and iterative discussion, in order to align on the most valuable scope. Remember that a hypothesis requires success criteria. This is a sensible time to determine ownership.
  2. Plan: Define the solution being experimented and set out the practical parameters of the experiment:
    1. who is involved?
    2. what is being done?
    3. when will it start?
    4. when will it finish?
    5. how will it be measured / how is information gathered?
  3. Commence: Start the experiment
  4. Monitor: monitor the experiment to ensure it is running correctly, gather data
  5. Analyse: once enough data is collected, analyse it
  6. Conclude and Share: determine conclusions, and present findings to stakeholders
  7. Iterate: This experiment is now over. Is further learning needed? Are more experiments required?