Skip to content
MosaicAGI

Could you show a board why this AI deployment was signed off?

Answer seventeen questions about a planned or live AI deployment, across performance, manipulation, fairness, the surrounding system and sign-off. See where the evidence is thin and what to ask for first. Runs in your browser. No account, no card, no call.

The deployment

Answer for one planned or live deployment, as things stand today. Nothing is sent.

Performance on your own tasks

Has the system been tested on your own tasks and data, not only on vendor benchmarks?

Are accuracy results reported with proper statistics, such as sample size and uncertainty?

Were pass criteria agreed before testing began?

Do you know how the system fails, not only how often it is right?

Manipulation and data

Has it been tested for prompt injection, including through documents and retrieved content?

Has it been tested for jailbreaks against the behaviour your policies forbid?

Has it been tested for data extraction?

Fairness and calibration

Has performance been compared across the populations you serve?

When the system says it is confident, is it right that often?

Is it defined and tested what the system does when it is unsure?

The surrounding system

Have the actions the system can take through tools been tested, not only its answers?

Has retrieval been tested, including when it returns the wrong document or nothing?

Will performance be monitored after go-live, with regression tests on production traffic?

Sign-off

Is a named person accountable for signing off the deployment?

Could a regulator or board read the evidence and follow it?

Was the evidence produced independently of the vendor?

Have you checked which of the vendor's evaluation claims apply to your own use?

Answer the questions to see your score. It updates as you go.

Method: yes scores a question in full, partly scores half and no scores nothing. Each is weighted from one to three by how much the missing evidence would weaken a sign-off, and the score is the weighted share out of a hundred. Unanswered questions are left out.

Could you show a board why this AI deployment was signed off?

Answer seventeen questions about a planned or live AI deployment, across performance, manipulation, fairness, the surrounding system and sign-off. See where the evidence is thin and what to ask for first. Runs in your browser. No account, no card, no call.

Open the free tool

It runs in your browser. MosaicAGI never sees your inputs.