Incident Response

Purpose

This procedure sets how Rygen identifies, manages and communicates incidents involving AI systems, and uses them to improve the AI Management System (AIMS), in line with ISO/IEC 42001:2023 Section 8.1 and Annex A.8.4.

Scope

Incidents involving AI systems within the AIMS scope: X1 Platform AI features, Corsair TMS AI capabilities, internal AI tools and third-party AI services, and AI development and deployment processes.

AI Incident Types

An AI incident is any unplanned event that affects, or could affect:

  • System availability: outages, failures or unavailability.
  • Performance: degradation below defined thresholds, or accuracy issues.
  • Trustworthiness: bias, fairness or explainability failures, or lack of human oversight.
  • Data: quality or integrity problems, or unauthorized access.
  • Security and privacy: breaches involving AI systems or training data.
  • Compliance: violations of regulations, policies or contracts.
  • Safety: potential harm to people or operations.

Severity Classification

Severity levels match Rygen’s operational incident severity framework.

SeverityAI Impact ExamplesEscalation
SEV1 (Critical)Complete outage for all users; major data breach or loss; critical safety issue; severe bias causing harm; regulatory violation with immediate impactCTO, Principal AI Engineer, relevant team leads
SEV2 (High)Partial outage or degradation for multiple users; performance below thresholds; data quality issues affecting decisions; moderate bias or fairness concerns; compliance concerns needing investigationPrincipal AI Engineer, relevant team leads
SEV3 (Medium)Minor user-visible issues; intermittent problems; non-critical performance degradationTeam lead; Principal AI Engineer notified

Issues below SEV3, such as minor bugs and small performance variations within acceptable ranges, are standard Jira issues and do not start incident response. Documentation and process gaps are not incidents; they are nonconformities handled under AI-014. If unsure of the severity, ask the relevant team lead or the Principal AI Engineer.

Incident Response Process

Detection and Initial Response

Incidents are detected by team members, automated monitoring or stakeholder reports. On detection:

  1. Notify the relevant system lead and the Principal AI Engineer at once.
  2. Assess the initial severity and AI-specific impacts, such as trustworthiness, bias and safety.
  3. Start operational incident response under Rygen’s Engineering Incident Response Procedure.
  4. For SEV1 and SEV2, engage the Information Security Officer if there are security, privacy or data concerns.

Operational Execution

Rygen’s Engineering Incident Response Procedure runs the operational response: role assignments (Incident Commander, Technical Lead, Communications Lead), a Slack incident channel, a Jira incident ticket (DEVOPS project, Incident type), investigation, remediation and stakeholder communication.

For AI systems:

  • Containment: consider disabling affected features, or reverting to a non-AI fallback if one exists.
  • Evidence preservation: capture model versions, configuration, input and output samples, and logs before making changes.
  • Bias and fairness issues: involve diverse stakeholders in the assessment, and consider mitigating at once even if the root cause is unclear.
  • Data issues: assess the scope of affected data and its downstream impact on decisions or operations.
  • Third-party AI services: engage vendor support where it applies, and record the provider’s response and resolution.

Communication Requirements

Communication follows the Engineering Incident Response Procedure, with AIMS recipients and channels per AI-007 (Communication Triggers and Protocols, AI Incidents). The Communications Lead manages it during an active incident.

Post-Incident Activities

SEV1 and SEV2 incidents need a postmortem within 5 business days, using Rygen’s standard postmortem template. It is optional for SEV3.

The postmortem or its follow-up actions cover:

  1. Nonconformity assessment: did we fail to follow an AIMS procedure, miss a required control, or violate a policy or objective? If yes, create an NCR under AI-014, which handles root cause analysis and corrective action.
  2. Risk register update: record new risks or update risk assessments.
  3. Corrective actions: Jira tasks for technical remediation, if no NCR covers it.
  4. Lessons learned: update the relevant AIMS documents.
  5. Management review input: an incident summary in the next management review.

Remediation tasks are tracked in Jira with owners and due dates. The Principal AI Engineer reviews completion and effectiveness. An incident is closed only after its corrective actions are verified effective.

Roles and Responsibilities

Principal AI Engineer

Accountable for AI incident response. Makes sure the AIMS integration steps are done, approves significant corrective actions, reports incidents to the CTO and the AI Governance Committee, updates the risk register and drives process improvements.

Incident Commander (Operational Role)

Leads coordination and resolution, makes sure AI-specific considerations are addressed, and works with the Principal AI Engineer on AIMS integration.

Technical Lead (Operational Role)

Investigates the technical root cause, implements and validates fixes, and gives technical input to the postmortem.

Communications Lead (Operational Role)

Manages stakeholder communication under AI-007 and works with the Principal AI Engineer on AI-specific messages.

Information Security Officer

Assesses security, privacy and compliance impact, decides whether regulatory notification is needed, and makes sure security controls are updated.

Team Leads

Support incident response in their area, implement corrective actions, and make sure their team learns from incidents.

Metrics and Monitoring

Each management review reports the AI incidents in the period, with their severity, the time to recovery, any repeat of an earlier root cause, and the status of their postmortems and corrective actions (AI-012).

Integration with Other AIMS Processes

Incidents inform risk identification and assessment (AI-008), may lead to an NCR and corrective action (AI-014), and feed AIMS performance evaluation (AI-017). AI-007 defines stakeholder communication. Incident trends and lessons learned are reported at each management review (AI-012).

Operational Implementation

Rygen’s Engineering Incident Response Procedure carries out this procedure and holds the detailed operational guidance.

Review and Improvement

Reviewed every year, after any SEV1 incident, when the Engineering Incident Response Procedure changes, and when audit findings or lessons learned call for it.

Revision History

VersionDateAuthorSummary of Change
1.02025-01-14Field BradleyInitial version integrating AIMS requirements with operational incident response
1.12026-07-14Field BradleyRemoved “quarterly” qualifier from management-review reporting to track the every-four-months cadence (AI-1759)
1.22026-11-04Field BradleySEV3 incidents are reported to the CTO via Slack; monthly reports are not done (AI-2173)
1.32026-11-09Field BradleyShortened every section, cites AI-007 for communication, treats documentation gaps as nonconformities, and names the incident measures the management review reports (AI-2173). Removes response times that no procedure sets, and vendor support is engaged where it applies. Communication follows the Engineering Incident Response Procedure; time to recovery reported.