Performance Monitoring

Purpose

This plan defines what Rygen monitors and measures in its AI Management System (AIMS), how, and how often, to evaluate the performance and effectiveness of the AIMS and its AI systems per ISO/IEC 42001:2023 Clause 9.1.

Scope

This plan covers:

  • AIMS objectives performance
  • AI system performance
  • Risk management effectiveness
  • Process conformity and effectiveness

Monitoring and Measurement Framework

AIMS Objectives Monitoring

The objectives and their targets are set in AI-002: AIMS Objectives. The Principal AI Engineer measures each one for every management review and reports the result in the management review report (AI-012).

ObjectiveMethodFrequencyResponsible
Trustworthy DeliveryReview of each system’s design specification and decision record for the oversight and explainability controls its assessments requireEvery management reviewPrincipal AI Engineer
Performance ExcellenceEvaluation results for each production system, against the targets in its design specificationEvery management reviewPrincipal AI Engineer
Responsible GovernanceCheck that each system deployed in the period has a risk and impact assessment dated before deploymentEvery management reviewPrincipal AI Engineer
Client Value Through InnovationCount of new AI systems deployed in the past yearEvery management reviewPrincipal AI Engineer
Operational ResilienceGCP monitoring of Rygen’s AI services and the AI providers’ published status pages; incident recordsEvery management reviewPrincipal AI Engineer
ComplianceInternal and external audit results and open nonconformitiesEvery management reviewPrincipal AI Engineer

AI System Performance Monitoring

Each AI system has a monitoring plan (MON-XXX) in the aims-governance repository. The plan lists only the metrics that are collected today, with the data source and the role that looks at each. Anything not measured is listed as a limitation, with the control that covers the risk instead.

System performance is also checked when it changes and when it fails:

  • On change: Evaluation suites run before a change that can alter a system’s AI behavior is deployed, as AI-013 (Changes to a System in Service) sets.
  • Model deprecations: A monthly review checks the OpenAI, Anthropic and Google deprecation notices against the models in-scope systems use.
  • Incidents: AI incidents are handled and reviewed under AI-023.
  • User reports: Problems reported by users are logged and reviewed as each plan’s User Feedback Monitoring section states.

Risk Management Effectiveness

The risk register, treatment status, and residual risk levels are reviewed at each management review (AI-008, AI-012). Risks are reassessed on the triggers AI-008 (Monitoring & Review phase), AI-013 (Changes to a System in Service) and AI-023 (Post-Incident Activities) set.

Process Conformity and Effectiveness

Conformity of AIMS processes is checked by the internal audit (AI-015), at least once a year. Nonconformities found at any time are handled under AI-014.

Data Collection and Analysis Methods

Data Sources

  • Evaluation results for each AI system
  • The data sources named in each system’s monitoring plan
  • The risk register and treatments
  • Incident records and nonconformities
  • Internal audit reports
  • The monthly model deprecation reviews

Analysis Methods

For each management review, the Principal AI Engineer compares each objective’s result with its target and with the previous review, and states the cause of any shortfall.

Reporting and Communication

Monitoring results are reported to top management in the management review report, at every management review (AI-012). Incidents are reported as AI-023 states. There are no separate monthly or annual monitoring reports.

Evaluation Schedule

Continuous Monitoring

  • GCP platform monitoring of AI service availability
  • User reports and incidents as they occur

Periodic Evaluation

  • Monthly: Model deprecation review
  • At each management review (AI-012): AIMS objectives and system monitoring results
  • Annually: Internal audit

Triggered Evaluation

  • Following an AI incident
  • Before deploying a change that can alter a system’s AI behavior (AI-013, Changes to a System in Service)
  • When a provider deprecates a model in use
  • When an audit, regulator, or stakeholder raises a concern

Performance Evaluation Criteria

The management review judges the AIMS on three questions:

  • Effective: Are the objectives met, and are nonconformities corrected?
  • Adequate: Do the resources and monitoring cover every AI system in scope?
  • Suitable: Does the AIMS still fit Rygen’s business and context?

Documentation and Records

  • This plan, in the aims-handbook repository
  • Each system’s monitoring plan, in the aims-governance repository
  • Evaluation results, in an evaluation report in the aims-governance repository, or in the system repository or MLflow linked from the Jira ticket or merge request (AI-013 Phase 4)
  • Management review reports and minutes, in the aims-governance repository
  • Monthly model deprecation reviews, in Jira

Records are retained for at least 3 years. Management review records are retained permanently.

Roles and Responsibilities

  • Principal AI Engineer: Owns this plan, measures the objectives, and reports results at each management review.
  • AI Engineer: Runs the evaluations and the monthly model deprecation review.
  • CTO: Reviews the results at management review and decides on resources.
  • AI Governance Committee: Raises monitoring concerns in the #ai-governance Slack channel and meets as needed.
  • Team Leads: Provide the data their systems produce.

Continuous Improvement

This plan is reviewed at least once a year, and when the AIMS scope or objectives change. A monitoring gap found at management review or audit is either fixed or recorded as a limitation in the system’s monitoring plan.

Integration with Management Review

The results of this plan are an input to every management review (AI-012). At each review, the Principal AI Engineer confirms that each metric in a system’s monitoring plan has a result for every check its schedule or trigger called for in the period; a check that was not due or not triggered needs no result. A metric with a missing result is removed from the plan or recorded as a nonconformity.

Revision History

VersionDateAuthorSummary of Change
1.02025-06-05Field BradleyInitial draft.
1.12025-09-02Field BradleyMigrated to markdown and gitlab
1.22025-10-08Field BradleyUpdated operational resiliency target from 99.9% to 99%
1.32025-12-04Field BradleyUpdated trustworthiness delivery objective to require explainability or human oversight controls
1.42026-07-14Field BradleyAligned AIMS objectives/management review and comprehensive risk review to the every-four-months review cadence (AI-1759)
1.52026-09-15Field BradleyCorrected this plan’s own maintenance location from Confluence to the aims-handbook repository (AI-1976)
2.02026-11-04Field BradleyDescribes the monitoring Rygen does: objectives measured at each management review, per-system plans listing only collected metrics, evaluations on change, the monthly model deprecation review, and incident reviews. Removed the weekly and monthly reviews and reports, which are not done. Objective targets now live only in AI-002 (AI-2166). Evaluation results for a change are linked in its Jira ticket or merge request; evaluations run only for changes that can alter AI behavior (AI-2131).
2.12026-11-09Field BradleyCites AI-008, AI-012, AI-013 and AI-023 instead of repeating their rules, requires a plan for every AI system, and accepts evaluation results kept in the repository or MLflow (AI-2131). The monthly model deprecation review names the providers it checks.