Module 2 of 2 · 45 minutes
Relying on an output, and telling the client
By the end of this module you will be able to
- Write a decision on the reliability of an output
- Say who must take responsibility for that decision
- Use dip sampling where outputs are automated or high in volume
- Tell a client in writing, in advance
- Answer a client's request for an explanation
Work through it
1 interactive for this module, built on the WAJD Teach engine. Nothing moves until you ask it to, and every one has a written version if you would rather read it.
Watch: Emma and George talk it through
3 minutes. Captions are on, and the same conversation is written out in full below. The voices are computer generated.
Emma George, the tool's given me a lease summary and it looks right. Can I just use it?
George Not on looks. The standard says you must apply professional judgement to decide on the reliability of the output, and you must document that decision in writing.
Emma What counts as professional judgement?
George Four things. Knowledge, skills, experience and professional scepticism. Scepticism is in the definition. You're expected to doubt it before you rely on it.
Emma What goes in the written decision?
George Any assumptions you made. Your key concerns about reliability, including the data underneath. The reason for each concern. Whether anything could lessen it. And a conclusion.
Emma What sort of conclusion?
George A statement of whether the output can reasonably be used for its intended purpose.
Emma Who signs it?
George It must be prepared by, or under the supervision of, an appropriately qualified and named surveyor who accepts responsibility. A person. Not a team and not a system.
Emma And if my conclusion is no?
George Then you tell the client in writing, with your reasoning or a summary of it.
Emma We run thousands of automated valuations. I can't write a decision for each.
George You're not asked to. For automated or high volume outputs, the standard says that isn't proportionate. But you remain accountable for each one. So you must undertake randomised dip samples at regular intervals.
Emma Why randomised?
George Because checking only the ones that look odd tells you nothing about the ones that look fine.
Emma Does the client need to know any of this?
George Yes. In writing and in advance. When AI is to be used, and for what purpose.
Emma And in the terms of engagement?
George When AI will be involved and in which parts of the process. The extent of your professional indemnity cover for AI use, if available. How to contest its use. How to seek redress. And how to opt out, if at all.
Emma What if a client asks how the tool works?
George On request, you must be able to give written information. The type of system. How it works and its limits. Your due diligence. How you manage the risks. And your decisions on reliability.
Emma That's a lot to produce.
George It's everything you've already kept. A firm with its register, its due diligence and its decisions can answer in an afternoon.
The written material
A written decision on reliability
Members and firms must apply professional judgement, meaning knowledge, skills, experience and professional scepticism, to decide on the reliability of any output that will have a material impact, and must document that decision in writing.
The written decision must set out any relevant assumptions, the key areas of concern about reliability including the underlying data, the reason for each concern, whether anything could lessen it, and the effect on the overall reliability of the output. It ends with a statement concluding whether the output can reasonably be used for its intended purpose.
- Assumptions made
- Concerns, and the reason for each
- What could lessen each concern
- Conclusion: can it reasonably be used for this purpose?
A named surveyor, and what to do when the answer is no
The decision must be prepared by, or under the supervision of, an appropriately qualified and named surveyor who accepts responsibility for its use. Not a team, not a system: a person.
Where the conclusion is that an output cannot reasonably be used for its intended purpose, that conclusion must be communicated in writing to the client, with the reasoning or a summary of it.
High volumes: dip sampling
Where AI automates an output or produces a high volume of them, it is generally neither necessary nor proportionate to write a decision on each. Firms remain accountable for every output all the same.
So members and firms must undertake randomised dip samples of the outputs at regular intervals, to scrutinise and assure their quality. Randomised matters: checking only the outputs that look odd tells you nothing about the ones that look fine.
Telling the client
Members and firms using AI with a material impact must make clear to clients, in writing and in advance, when and for what purpose AI is to be used.
The terms of engagement and related documents must detail when AI will be involved, the parts of the process in which it will be involved, the extent of professional indemnity cover for the use of AI if available, the internal processes to contest the use of an AI system, the processes to seek redress, and how a client can opt out, if at all.
Explaining it on request
A firm must be able to provide, on request, written information about the type of AI system used, its basic ways of working and limitations, the due diligence carried out before using it, how the relevant risks are identified and managed, and the decisions made about the reliability of the output.
Every item on that list is something the earlier requirements have already produced. A firm that kept its register, its due diligence record and its reliability decisions can answer in an afternoon. A firm that did not cannot answer at all. Firms that build their own systems carry further duties, including a written record of the system's application, risks and alternatives before it is deployed.
Knowledge check
The knowledge check and your certificate need a free account, so that your progress and results can be saved as evidence.
The learning itself stays free and open. You are reading all of it right now without an account.
Was this module useful? Tell us in two minutes, it decides what we improve next.