Plumb · Report
The assessment, in full.
Five Service Standard points, one question each, written up as a working document. Three points sit amber, one green, one red. The gaps cluster around recovery paths and the joined‑up read across channels. Yours to keep, share with your team, or take into a Service Standard assessment as background.
Verdicts
Point 01Service Standard
Understand users and their needs.
Overall
Amber.
The user‑research story is strong on the AI‑enabled channel itself. Less so on the people who are in the loop. The team has done careful work understanding the citizen using the service, but the caseworker reviewing model output sits closer to the edge of the research than to the centre.
- Q.01
What does the AI in your service know about its users, and where did that knowledge come from?
Strong on the citizen using the service: two rounds of qualitative research plus a survey covering edge cases, in a repository the team uses. Thinner on the caseworker reviewing model output, who sits at the edge of the research, and on the groups the model is most likely to misclassify.
Point 02Service Standard
Solve a whole problem for users.
Overall
Green.
The AI‑enabled element sits inside a service that solves a whole problem. The model isn't a side feature. It's where the heavy lifting happens, with handoffs designed where they should be.
- Q.02
When the AI can't finish the job, where does the user end up?
The model handles the heavy lifting, and where it can't finish, the handoffs are designed: the user reaches a person who can complete the work, not a stranded loop or a dead end.
Point 03Service Standard
Provide a joined‑up experience across all channels.
Overall
Amber.
The AI and non‑AI channels read as one service most of the time. The seam between them is visible in the right direction (model handing off to human) but invisible in the other (human handing back to model). Where channels share state, they share it well. Where they don't, the gap is real and worth closing.
- Q.03
What does the user notice moving between the AI and the rest of your service?
Voice and terminology stay consistent across channels, and there's one source of truth that both the AI and the caseworkers read from. The seam shows in one direction only: the model-to-human handoff is announced, the human-back-to-model handoff often isn't, and the document-upload flow gives no notice at all.
Point 04Service Standard
Make the service simple to use.
Overall
Red.
The happy path is well‑tested and simple. The recovery path, when the model is wrong, is the gap. Users see a generic error and are returned to the start of the journey rather than to the step where the model intervened. The team knows about it. It isn't yet in the backlog.
- Q.04
When the AI gets something wrong, what does the user see and do?
First‑time use of the happy path tests well in research. Recovery path is not designed: generic error message, return to start of journey, no acknowledgement of what the model got wrong. The single biggest improvement available.
Point 05Service Standard
Make sure everyone can use the service.
Overall
Amber.
WCAG 2.2 AA holds across both channels. Assistive‑technology testing covers the AI‑enabled flow. The gap is at the populations most likely to be misclassified by the model: they're identified in metrics but not yet in the testing programme.
- Q.05
Who is the AI most likely to get wrong, and what do those users experience?
Assistive technology testing is regular. Low‑literacy testing is in place but light. Misclassification‑risk populations are identified by the model metrics team but the design and research teams aren't running them through the user‑research programme yet.