Test

Test The Results

At the end of the four month iteration, there is a test of the experiment.

The expected results are weighed against the actual results.


Pivot or Persevere Event

The test is assessed at the Pivot or Persevere event.

At this meeting, the results of the experiment are assessed and the decision is made to either ‘persevere’ with the project as it is, or to ‘pivot’, that is change the experiment to test another aspect of the viability of the initial idea.

The process for each experiment is:

  1. Review the primary metric/s
  2. Discuss the progress of the experiment
  3. Choose to
    • Persevere
    • Pivot
  4. Schedule a planning session, as needed


1. Review the Primary Metric/s

When an experiment starts, at least one metric is chosen that will help us to understand if the experiment is successful.

The key question to ask is: “What would we expect to see by the next Pivot or Persevere meeting if the experiment is successful ?”

The metric needs to be measurable and unambiguous. There will be discussion at the meeting around the implications of the metric, but the metric itself must be clear.

Example Metrics

For the CAT support project:

  • At least 1 weekly meeting is organised by the CATs team themselves
  • At least 2 CAT volunteers are added in the iteration
Vanity Metrics

It is essential that the metric used indicates real progress towards the aim of the experiment. It can be easy to focus on ‘vanity metrics’, ie things that you can measure that don’t really matter, for example number of followers on social media.



2. Discuss Progress

In addition to the quantifiable metrics, the meeting is an opportunity to discuss how the experiment is progressing. The metrics provide stimulus for discussion. The intention is not that the metrics alone will simply dictate whether the experiment has passed or failed.

Attendees

As we are a community-based organisation, the meeting is open to the public. The decision to persevere or pivot is not put to a public vote, but the input of the stakeholders is essential to helping to understand real progress.



3a Choose to Persevere

Following the discussion, the decision will be taken as to how to proceed with the experiment. The decision is the responsibility of the Create team. If the decision is to persevere with the experiment as it is, then the details are merely noted and the meeting moves on.

If necessary, the metrics may be changed, but the aim of the experiment will remain the same.



3b Choose to Pivot

If it is decided that the experiment needs to change, then there is a further discussion on how to pivot the experiment.

A pivot is a change of direction, but not a change of strategy. That is, the original idea remains, but a new hypothesis is selected to test, and an appropriate new experiment devised.

Types of Pivot

There are many different ways to pivot, but in essence we can change 5 areas:

  • The Solution
    • We can focus on just one aspect of the solution (the product or service for an identified user need) or incorporate it into a bigger solution, perhaps hosting multiple solutions. We could also use a different technology
  • The User
    • We can optimise the solution for a different type of user than originally intended
  • The Charge
    • We can charge a lot of people a small amount or a few people a large amount. We could also change the monetisation method, eg from direct to indirect sales.
  • The Marketing Channel
    • We can choose different ways to market the same solution, typically through customer referral, customer retention, and/or various paid advertising options.
  • The Need
    • Sometimes the identified user need simply doesn’t really exist and we need to pivot to a new need requiring a different solution
Examples of Pivots
Pivot the SolutionThe CATs project started with a few volunteers running events at Create, but we pivoted to provide training and support for them to run events and activities in many locations
Pivot the User
We started serving food and drink to coworkers working in the Create space, with food bought in from a local cafe. We pivoted to providing our own food, to provide work experience for users who were finding it difficult to gain paid employment.
Pivot the ChargeWe set up a coworking membership scheme that provided benefits for annual membership, but pivoted to charging for individual services
Pivot the Marketing ChannelWe changed to a loyalty card for escape room participants to encourage customer retention
Pivot the NeedWe initially offered the Create space to small business digital nomads, but discovered that there was little need for this, so pivoted to providing more space for the escape room


Schedule Planning Session

The Pivot or Persevere meeting is a decision-making meeting, but it is not intended to be an in-depth planning meeting. It sets out the experiment aims and metrics to monitor, but an additional planning time will often be needed to work out how the experiment will be implemented. It is more important that each experiment is assessed on the day than they are planned in detail.

If necessary, a planning session should be scheduled by those who are involved in the experiment for soon after the day.



Top