Clause 9: Performance Evaluation
Monitoring, measurement, analysis, evaluation, internal audit, and management review of the AI Management System.
Chapter Overview
Clause 9 covers how you evaluate whether your AIMS is working effectively. It includes monitoring and measurement, internal audits, and management reviews - the "Check" phase of the PDCA cycle that provides feedback for improvement.
Clause Structure
| Sub-clause | Title | Focus |
|---|---|---|
| 9.1 | Monitoring, measurement, analysis and evaluation | Performance metrics and evaluation |
| 9.2 | Internal audit | Audit program and execution |
| 9.3 | Management review | Top management evaluation of AIMS |
9.1 Monitoring, Measurement, Analysis and Evaluation
Requirement
The organization shall determine:
- What needs to be monitored and measured
- The methods for monitoring, measurement, analysis, and evaluation to ensure valid results
- When the monitoring and measuring shall be performed
- When the results shall be analyzed and evaluated
The organization shall evaluate the AI management system performance and effectiveness.
The organization shall retain appropriate documented information as evidence of the results.
You must define WHAT you measure, HOW you measure it, WHEN you measure, and WHO analyzes the results. Without measurement, you cannot demonstrate AIMS effectiveness or drive improvement.
What to Monitor and Measure
| Category | Metrics Examples |
|---|---|
| AIMS Performance | Objectives achievement, control effectiveness, process compliance |
| AI System Performance | Accuracy, reliability, availability, response time |
| Risk Management | Risk assessment completion, treatment status, residual risk levels |
| Compliance | Policy compliance, regulatory compliance, audit findings |
| Incidents | AI incidents, near-misses, response times, resolution rates |
| Training | Training completion, competence levels, awareness scores |
| Stakeholder | Customer satisfaction, complaints, feedback scores |
AI-Specific Monitoring
| AI Aspect | Monitoring Approach |
|---|---|
| Model Performance | Accuracy metrics, precision/recall, F1 scores over time |
| Model Drift | Data drift detection, concept drift monitoring |
| Bias | Fairness metrics across protected groups |
| Explainability | Explanation quality, user understanding |
| Human Oversight | Override rates, review coverage, intervention frequency |
| Data Quality | Completeness, accuracy, timeliness of data |
Measurement Framework
For each metric, define:
• Metric name and description
• Purpose (why measure this?)
• Calculation method
• Data source
• Measurement frequency
• Target/threshold
• Responsible person
• Reporting format
• Analysis frequency
• Escalation criteria
9.2 Internal Audit
Requirement
The organization shall conduct internal audits at planned intervals to provide information on whether the AIMS:
- Conforms to the organization's own requirements for its AIMS
- Conforms to the requirements of ISO 42001
- Is effectively implemented and maintained
9.2.2 Audit Program Requirements
The organization shall:
- Plan, establish, implement, and maintain an audit program including frequency, methods, responsibilities, planning requirements, and reporting
- Define audit criteria and scope for each audit
- Select auditors and conduct audits ensuring objectivity and impartiality
- Ensure results are reported to relevant management
- Retain documented information as evidence
Frequency factors:
• Importance of processes
• Risk levels of AI systems
• Results of previous audits
• Changes to AIMS or AI systems
• Organizational changes
• External factors (regulatory, technology)
Audit Program Components
| Component | Description |
|---|---|
| Audit Schedule | Annual plan showing when each area will be audited |
| Audit Scope | What will be covered in each audit |
| Audit Criteria | Requirements against which to audit (ISO 42001, policies) |
| Audit Methods | Document review, interviews, observation, testing |
| Auditor Selection | Competent, objective, impartial auditors |
| Audit Reporting | Findings, nonconformities, recommendations |
| Follow-up | Corrective action tracking and verification |
Internal Audit Process
- Planning: Define scope, criteria, schedule, team
- Preparation: Review documentation, prepare checklists
- Opening Meeting: Confirm scope, explain process
- Evidence Gathering: Interviews, document review, observation
- Analysis: Evaluate evidence against criteria
- Findings: Identify conformities and nonconformities
- Closing Meeting: Present findings, agree actions
- Reporting: Document audit results
- Follow-up: Verify corrective actions
Auditor Competence
Internal auditors should have:
- Understanding of ISO 42001 requirements
- Knowledge of AI systems and governance
- Audit skills and techniques
- Independence from areas being audited
- Objectivity and impartiality
Template: Internal Audit Checklist (Sample)
5.1 Leadership and Commitment
☐ Is there evidence of top management commitment to AIMS?
☐ Has the AI policy been approved by top management?
☐ Are adequate resources allocated to AIMS?
☐ Is AI governance integrated into business processes?
☐ Is there evidence of management communication about AI governance?
5.2 AI Policy
☐ Does the AI policy exist and is it documented?
☐ Is the policy appropriate to the organization's purpose?
☐ Does it provide a framework for AI objectives?
☐ Does it include commitment to requirements and improvement?
☐ Is the policy communicated to personnel?
☐ Is the policy available to interested parties?
5.3 Roles and Responsibilities
☐ Are AIMS roles and responsibilities defined?
☐ Are they communicated and understood?
☐ Is someone responsible for AIMS conformance?
☐ Is someone responsible for reporting to top management?
9.3 Management Review
Requirement
Top management shall review the organization's AIMS at planned intervals to ensure its continuing suitability, adequacy, and effectiveness.
9.3.2 Management Review Inputs
The management review shall include consideration of:
- Status of actions from previous management reviews
- Changes in external and internal issues relevant to the AIMS
- Feedback on AI management system performance, including trends in nonconformities and corrective actions, monitoring and measurement results, audit results, and achievement of AI objectives
- Opportunities for continual improvement
9.3.3 Management Review Outputs
The outputs of the management review shall include decisions related to:
- Continual improvement opportunities
- Any need for changes to the AIMS
The organization shall retain documented information as evidence of the results of management reviews.
ISO 42001 requires reviews at "planned intervals" but doesn't specify frequency. Common approaches:
Annual: Comprehensive review of entire AIMS
Quarterly: More frequent for new or high-risk AIMS
Integrated: Combined with other management system reviews
Consider more frequent reviews during initial implementation or after significant changes.
Template: Management Review Agenda
AI MANAGEMENT SYSTEM - MANAGEMENT REVIEW
Date: [Date]
Attendees: [List top management and key personnel]
AGENDA
1. Previous Actions Review
• Status of actions from last review
• Outstanding items
2. External/Internal Changes
• Regulatory changes
• Technology developments
• Organizational changes
• Market/competitive changes
3. AIMS Performance
• AI objectives status
• KPI dashboard review
• Nonconformities and corrective actions
• Monitoring and measurement results
4. Audit Results
• Internal audit findings
• External audit results (if applicable)
• Certification status
5. AI System Performance
• AI incident summary
• Risk assessment updates
• Impact assessment findings
• Control effectiveness
6. Resource Review
• Resource adequacy
• Competence gaps
• Budget status
7. Improvement Opportunities
• Proposed improvements
• Innovation opportunities
• Stakeholder feedback
8. Decisions and Actions
• Changes to AIMS
• Resource decisions
• Improvement initiatives
• Action items with owners and deadlines
Documented Information Requirements
Required:
• Monitoring and measurement results (9.1)
• Internal audit program (9.2)
• Internal audit results (9.2)
• Management review results (9.3)
Recommended:
• Measurement framework/KPI definitions
• Audit checklists
• Audit reports
• Management review minutes
• Action tracking register
Sample Audit Questions
9.1 Monitoring and Measurement:
• What do you monitor and measure for AIMS effectiveness?
• How do you measure AI system performance?
• Show me your KPIs and dashboards
• How often do you analyze results?
• Who is responsible for performance evaluation?
9.2 Internal Audit:
• Show me your audit program
• How do you ensure auditor competence and independence?
• Show me a recent audit report
• How are audit findings addressed?
• How do you determine audit frequency?
9.3 Management Review:
• When was the last management review?
• Show me the management review minutes
• Were all required inputs considered?
• What decisions were made?
• How are actions tracked to completion?
Common Nonconformities
| Type | Nonconformity | How to Avoid |
|---|---|---|
| Major | No internal audits conducted | Establish and implement audit program |
| Major | No management review conducted | Schedule and conduct reviews |
| Major | Management review missing required inputs | Use checklist of required inputs |
| Minor | Auditors not independent from audited areas | Assign independent auditors |
| Minor | No evidence of performance monitoring | Document monitoring activities |
| Minor | Audit findings not followed up | Track corrective actions |
| Minor | Management review actions not tracked | Maintain action register |
1. Define what, how, when, and who for monitoring and measurement
2. Internal audits must verify conformance AND effectiveness
3. Auditors must be competent, objective, and impartial
4. Management review has specific required inputs (9.3.2)
5. All three sub-clauses require documented information
6. Results feed into continual improvement (Clause 10)
• Know the four things to determine for monitoring (what, how, when, who analyzes)
• Remember internal audit must check conformance AND effectiveness
• Know the required inputs for management review (9.3.2)
• Auditors must be objective and impartial (cannot audit own work)
• Management review outputs include improvement opportunities and changes to AIMS
• All sub-clauses require documented information (records)