Performance Monitoring
Purpose
This plan defines what Rygen monitors and measures in its AI Management System (AIMS), how, and how often, to evaluate the performance and effectiveness of the AIMS and its AI systems per ISO/IEC 42001:2023 Clause 9.1.
Scope
This plan covers:
- AIMS objectives performance
- AI system performance
- Risk management effectiveness
- Process conformity and effectiveness
Monitoring and Measurement Framework
AIMS Objectives Monitoring
The objectives and their targets are set in AI-002: AIMS Objectives. The Principal AI Engineer measures each one for every management review and reports the result in the management review report (AI-012).
| Objective | Method | Frequency | Responsible |
|---|---|---|---|
| Trustworthy Delivery | Review of each system’s design specification and decision record for the oversight and explainability controls its assessments require | Every management review | Principal AI Engineer |
| Performance Excellence | Evaluation results for each production system, against the targets in its design specification | Every management review | Principal AI Engineer |
| Responsible Governance | Check that each system deployed in the period has a risk and impact assessment dated before deployment | Every management review | Principal AI Engineer |
| Client Value Through Innovation | Count of new AI systems deployed in the past year | Every management review | Principal AI Engineer |
| Operational Resilience | GCP monitoring of Rygen’s AI services and the AI providers’ published status pages; incident records | Every management review | Principal AI Engineer |
| Compliance | Internal and external audit results and open nonconformities | Every management review | Principal AI Engineer |
AI System Performance Monitoring
Each AI system has a monitoring plan (MON-XXX) in the aims-governance repository. The plan lists only the metrics that are collected today, with the data source and the role that looks at each. Anything not measured is listed as a limitation, with the control that covers the risk instead.
System performance is also checked when it changes and when it fails:
- On change: Evaluation suites run before a change that can alter a system’s AI behavior is deployed, as AI-013 (Changes to a System in Service) sets.
- Model deprecations: A monthly review checks the OpenAI, Anthropic and Google deprecation notices against the models in-scope systems use.
- Incidents: AI incidents are handled and reviewed under AI-023.
- User reports: Problems reported by users are logged and reviewed as each plan’s User Feedback Monitoring section states.
Risk Management Effectiveness
The risk register, treatment status, and residual risk levels are reviewed at each management review (AI-008, AI-012). Risks are reassessed on the triggers AI-008 (Monitoring & Review phase), AI-013 (Changes to a System in Service) and AI-023 (Post-Incident Activities) set.
Process Conformity and Effectiveness
Conformity of AIMS processes is checked by the internal audit (AI-015), at least once a year. Nonconformities found at any time are handled under AI-014.
Data Collection and Analysis Methods
Data Sources
- Evaluation results for each AI system
- The data sources named in each system’s monitoring plan
- The risk register and treatments
- Incident records and nonconformities
- Internal audit reports
- The monthly model deprecation reviews
Analysis Methods
For each management review, the Principal AI Engineer compares each objective’s result with its target and with the previous review, and states the cause of any shortfall.
Reporting and Communication
Monitoring results are reported to top management in the management review report, at every management review (AI-012). Incidents are reported as AI-023 states. There are no separate monthly or annual monitoring reports.
Evaluation Schedule
Continuous Monitoring
- GCP platform monitoring of AI service availability
- User reports and incidents as they occur
Periodic Evaluation
- Monthly: Model deprecation review
- At each management review (AI-012): AIMS objectives and system monitoring results
- Annually: Internal audit
Triggered Evaluation
- Following an AI incident
- Before deploying a change that can alter a system’s AI behavior (AI-013, Changes to a System in Service)
- When a provider deprecates a model in use
- When an audit, regulator, or stakeholder raises a concern
Performance Evaluation Criteria
The management review judges the AIMS on three questions:
- Effective: Are the objectives met, and are nonconformities corrected?
- Adequate: Do the resources and monitoring cover every AI system in scope?
- Suitable: Does the AIMS still fit Rygen’s business and context?
Documentation and Records
- This plan, in the aims-handbook repository
- Each system’s monitoring plan, in the aims-governance repository
- Evaluation results, in an evaluation report in the aims-governance repository, or in the system repository or MLflow linked from the Jira ticket or merge request (AI-013 Phase 4)
- Management review reports and minutes, in the aims-governance repository
- Monthly model deprecation reviews, in Jira
Records are retained for at least 3 years. Management review records are retained permanently.
Roles and Responsibilities
- Principal AI Engineer: Owns this plan, measures the objectives, and reports results at each management review.
- AI Engineer: Runs the evaluations and the monthly model deprecation review.
- CTO: Reviews the results at management review and decides on resources.
- AI Governance Committee: Raises monitoring concerns in the #ai-governance Slack channel and meets as needed.
- Team Leads: Provide the data their systems produce.
Continuous Improvement
This plan is reviewed at least once a year, and when the AIMS scope or objectives change. A monitoring gap found at management review or audit is either fixed or recorded as a limitation in the system’s monitoring plan.
Integration with Management Review
The results of this plan are an input to every management review (AI-012). At each review, the Principal AI Engineer confirms that each metric in a system’s monitoring plan has a result for every check its schedule or trigger called for in the period; a check that was not due or not triggered needs no result. A metric with a missing result is removed from the plan or recorded as a nonconformity.
Revision History
| Version | Date | Author | Summary of Change |
|---|---|---|---|
| 1.0 | 2025-06-05 | Field Bradley | Initial draft. |
| 1.1 | 2025-09-02 | Field Bradley | Migrated to markdown and gitlab |
| 1.2 | 2025-10-08 | Field Bradley | Updated operational resiliency target from 99.9% to 99% |
| 1.3 | 2025-12-04 | Field Bradley | Updated trustworthiness delivery objective to require explainability or human oversight controls |
| 1.4 | 2026-07-14 | Field Bradley | Aligned AIMS objectives/management review and comprehensive risk review to the every-four-months review cadence (AI-1759) |
| 1.5 | 2026-09-15 | Field Bradley | Corrected this plan’s own maintenance location from Confluence to the aims-handbook repository (AI-1976) |
| 2.0 | 2026-11-04 | Field Bradley | Describes the monitoring Rygen does: objectives measured at each management review, per-system plans listing only collected metrics, evaluations on change, the monthly model deprecation review, and incident reviews. Removed the weekly and monthly reviews and reports, which are not done. Objective targets now live only in AI-002 (AI-2166). Evaluation results for a change are linked in its Jira ticket or merge request; evaluations run only for changes that can alter AI behavior (AI-2131). |
| 2.1 | 2026-11-09 | Field Bradley | Cites AI-008, AI-012, AI-013 and AI-023 instead of repeating their rules, requires a plan for every AI system, and accepts evaluation results kept in the repository or MLflow (AI-2131). The monthly model deprecation review names the providers it checks. |