Testing and piloting services: the Cabinet Office guidance, explained

The key facts

• The guidance note builds on chapter 4 of the Sourcing Playbook and covers the use of testing approaches, including pilots, in outsourcing and insourcing projects.
• Piloting is only one of several testing mechanisms. The others named are policy trials, proofs of concept, scoping phases and test and learn.
• Agile approaches and Innovation Partnerships are not testing mechanisms in themselves, but testing must be built into any project using them.
• Where a service is being outsourced for the first time, the Sourcing Playbook says a pilot should be run as part of a programme of testing.
• Where a pilot is not recommended for a complex outsourcing project, that decision should be stated clearly in the business case with justification.
• Testing objectives should be SMART and supported by an evaluation methodology that scores results objectively against those objectives.
• Any testing that involves suppliers must comply with procurement and state aid law, and legal advice may be needed to avoid giving a bidder unfair advantage.
• The Model Services Contract is a useful starting point for a single stage procurement that includes a pilot phase, though it needs amendment for pilot mechanics.

What the guidance covers and who it is for

The Testing and Piloting Services guidance note was published by the Cabinet Office in May 2021. It expands on chapter 4 of the Sourcing Playbook and is aimed at practitioners: commercial professionals, project delivery professionals and the teams building business cases for major service change. It is presented as a collection of best practice rather than legal advice, and it is explicit that departments still need suitably qualified and experienced people running robust procurements with proper legal input.

The starting point is that testing and piloting belong inside the wider strategic delivery model assessment, previously known as the make or buy decision, and inside the overall commercial strategy and business case. Testing is not a bolt on activity that happens once a contract is signed. Planning which testing approaches to use, and whether a pilot is needed, should begin at the earliest strategic stages of a project, before any procurement process starts.

The guidance also sets out where to get help. Departments considering how testing approaches fit into complex projects are directed to consult the Cabinet Office, with the Sourcing Programme supporting complex outsourcing projects alongside the Complex Transactions Team. The Cabinet Office Policy Lab team is named as a source of support for policy trials specifically.

The five testing approaches, and what each one answers

The guidance is built around a simple observation: across government, piloting has come to mean many different things, and that vagueness causes projects to run the wrong test at the wrong time. It therefore separates the mechanisms by project phase. Early testing covers policy trials and proofs of concept. Developing and building requirements covers scoping phases, alongside agile approaches and Innovation Partnerships. Testing in a live environment covers test and learn and pilots.

A policy trial tests whether a proposed policy will deliver the intended outcomes or behaviours, assessed against a control scenario where no change is assumed. It can be desktop work such as market research or statistical comparison of options, or it can involve target populations trying new approaches. A proof of concept is a practical or theoretical demonstration of a product, process or service to show whether it can be turned into reality, and its outcome should be a clear service definition. The guidance also flags the option of a proof of value, which tests the economic case for implementation as well as the functional one.

A scoping phase is used where there is a clear problem but the deliverables and delivery mechanism are unclear, and its output is a specification of requirements that can be used in a later procurement. Test and learn tests one or more delivery options live, with the lessons used to finalise requirements that then form the basis of a further competitive procurement. A pilot is the final stage before full rollout: the preferred approach implemented on a smaller scale, regionally, for a target population, for part of the service or with a single supplier, to iron out operational and logistical issues.

The guidance is clear about what a pilot is not for. A pilot should not be used to test the appropriateness of policies, to look at alternative options or to justify a business case. Other approaches are better suited to those purposes, and contract award still follows the usual compliant procurement process.

When to pilot, and when not to

Testing should be proportionate to the size, complexity and level of uncertainty involved in delivering the service. The guidance treats robust, proportionate testing as good practice in all project plans, including second and later generations of outsourcing, insourcing, and any significant change to operational delivery, policy implementation or commercial models.

It sets out clear triggers for considering a pilot: significant transformation of service delivery including insourcing, a large scope of delivery, citizens having to interact differently with the service, potential far reaching unintended consequences, costly implementation, and delivery that would be difficult to reverse. It also names situations where a pilot adds less value: a continuation of an existing service with no change to core requirements or delivery approach, cases where other testing has already de risked full scale implementation, and agile programmes where testing is embedded at each sprint. Even then, agile does not automatically rule out a pilot before implementation.

Once a decision to outsource has been made, the guidance says the testing approach, including any pilot, should be tested with the market. Suppliers can offer valuable feedback on how they are best engaged, on contractual and financial models, and on realistic timescales for each phase. The testing programme should align to key project milestones, sit in the business case, and provide evidence for assurance reviews, fitting where relevant within the Integrated Assurance and Approval Plan agreed with the Infrastructure and Projects Authority.

Designing, running and evaluating a test

Design is where the guidance is most practical. Objectives should be tied to project objectives but should not be confused with them, because some benefits only appear at the scale and volume of full rollout. Objectives should be SMART, define exactly what is being tested, and set clear success criteria. For pilots, that might mean testing the feasibility of operational procedures, the validity of financial or operational assumptions, delivery capability, the practical effects of policy in the real world, the capability of systems and IT, and interdependencies between teams, processes and policy decisions.

Scope and scale, resources and governance, and timescales each get their own set of considerations. Tests should be realistic and representative, with thought given to regional or population differences and how far results can be replicated elsewhere. A governance model with clear escalation routes should be agreed, budget confirmed including external resource, and internal training needs reviewed. On duration, the guidance is candid: a simple test may last weeks, while a complex test and learn or pilot could take several months or more than a year to test an end to end service, and time must be planned in to close the feedback loop before scaling.

Running a test means reviewing progress against objectives regularly, capturing data on internal and external factors affecting delivery, and keeping communication open with all parties so lessons are captured at the right moments rather than only at the end. Evaluation means collecting what worked, what did not and what had to change, adding insight from interviews with stakeholders and end users, and producing a final report against the success criteria that identifies gaps and any service changes needed. If a pilot goes well, the transition plan agreed at procurement is reviewed and refined. If it does not, the guidance treats that as the point of the exercise: learning in a controlled environment before rollout.

Commercial and legal consequences

Section 9 links testing to commercial strategy. Departments need to decide whether to commit to final supplier selection in a single stage procurement or to run two or more procurements covering different phases, for example advisory support to build requirements, then a test and learn, then the pilot and rollout. Where services are already outsourced and simply transitioning, or where requirements are well established in the market, a single stage tender including a pilot may be quicker and more cost effective. The Model Services Contract is offered as a starting point for that route, with amendments for pilot mechanics, rollout and termination if the pilot fails.

The commercial considerations that should be addressed in tender documentation are specific. They include responsibility for delivering the test, transfer of assets and associated liabilities, whether TUPE applies to departmental employees with New Fair Deal pension protection factored in, intellectual property ownership with the starting position that IP rests with the department so far as legally possible, cost variation between pilot pricing and full implementation, how learnings translate into specification changes, termination rights with and without cause, and liability, risk and dependencies.

Two risks run through all of this. First, material changes to the service specification after a pilot may trigger a requirement to retender unless the original tender documents provided for them, and substantial cost changes raise the same question. Second, where suppliers are involved in trials, scoping or testing, the department must ensure that involvement does not give a real or perceived advantage in later bidding. Legal advice should be sought where testing involves suppliers, and tests must comply with procurement and state aid law. On funding, the guidance sets out the trade off between costing the pilot as a distinct priced phase, which is transparent and allows a clean break but transfers cost risk to the department, and folding pilot costs into total contract pricing, which shifts risk to the provider but reduces transparency and can raise total cost of ownership.

How eSourcing Data helps

The guidance asks buyers to plan a testing programme early, evidence it through the business case, and keep it aligned with the commercial strategy and procurement documents. That is a documentation and audit trail problem as much as a delivery one. eSourcing Data gives procurement teams a single place to hold the sourcing strategy, the tender documentation and the evaluation records for each phase, so that the reasoning behind a scoping phase, a test and learn or a pilot is captured at the time rather than reconstructed for an assurance review.

Because the guidance often points to multiple procurement stages, the platform is useful for keeping those stages connected. Notices, market engagement records, clarification questions and supplier correspondence for a scoping or test and learn exercise sit alongside the later competition for full implementation, which supports the fairness point at the heart of section 9: showing that a supplier involved in early work did not gain an unfair advantage, and that any subsequent procurement was run competitively and transparently.

Market engagement and evaluation are the other natural fits. The guidance recommends testing the pilot design with potential suppliers before finalising it, and eSourcing Data supports structured pre market engagement, supplier communications and a documented evaluation of responses against published criteria. During delivery, supplier management and reporting records help teams track performance information from the test period and carry the evidence forward into the decision on rollout, without overstating what any system can do about the underlying operational judgement, which remains the department's.

What to do about it

1. 1Decide, at the strategy stage rather than after award, which testing approaches your project needs and record the purpose, objectives and success metrics for each one.
2. 2If the service is being outsourced for the first time, plan a pilot. If you decide not to pilot a complex outsourcing project, state that decision and the justification in the business case.
3. 3Write SMART objectives and an evaluation methodology before the test starts, including what you will do if the results are only partly successful.
4. 4Test the proposed testing and pilot design with the market before it is finalised, and use supplier feedback on engagement models, contractual models and timescales.
5. 5Decide early whether the programme needs a single stage procurement or several, and make sure the tender documents allow for the specification and cost changes a pilot is likely to produce.
6. 6Settle intellectual property, TUPE, termination rights and cost variation in the tender documentation, taking legal advice where suppliers are involved in any test.
7. 7Build enough time into the project timeline to evaluate results, implement lessons and refine the transition plan before scaling up.