Incident Response
Purpose
This procedure sets how Rygen identifies, manages and communicates incidents involving AI systems, and uses them to improve the AI Management System (AIMS), in line with ISO/IEC 42001:2023 Section 8.1 and Annex A.8.4.
Scope
Incidents involving AI systems within the AIMS scope: X1 Platform AI features, Corsair TMS AI capabilities, internal AI tools and third-party AI services, and AI development and deployment processes.
AI Incident Types
An AI incident is any unplanned event that affects, or could affect:
- System availability: outages, failures or unavailability.
- Performance: degradation below defined thresholds, or accuracy issues.
- Trustworthiness: bias, fairness or explainability failures, or lack of human oversight.
- Data: quality or integrity problems, or unauthorized access.
- Security and privacy: breaches involving AI systems or training data.
- Compliance: violations of regulations, policies or contracts.
- Safety: potential harm to people or operations.
Severity Classification
Severity levels match Rygen’s operational incident severity framework.
| Severity | AI Impact Examples | Escalation |
|---|---|---|
| SEV1 (Critical) | Complete outage for all users; major data breach or loss; critical safety issue; severe bias causing harm; regulatory violation with immediate impact | CTO, Principal AI Engineer, relevant team leads |
| SEV2 (High) | Partial outage or degradation for multiple users; performance below thresholds; data quality issues affecting decisions; moderate bias or fairness concerns; compliance concerns needing investigation | Principal AI Engineer, relevant team leads |
| SEV3 (Medium) | Minor user-visible issues; intermittent problems; non-critical performance degradation | Team lead; Principal AI Engineer notified |
Issues below SEV3, such as minor bugs and small performance variations within acceptable ranges, are standard Jira issues and do not start incident response. Documentation and process gaps are not incidents; they are nonconformities handled under AI-014. If unsure of the severity, ask the relevant team lead or the Principal AI Engineer.
Incident Response Process
Detection and Initial Response
Incidents are detected by team members, automated monitoring or stakeholder reports. On detection:
- Notify the relevant system lead and the Principal AI Engineer at once.
- Assess the initial severity and AI-specific impacts, such as trustworthiness, bias and safety.
- Start operational incident response under Rygen’s Engineering Incident Response Procedure.
- For SEV1 and SEV2, engage the Information Security Officer if there are security, privacy or data concerns.
Operational Execution
Rygen’s Engineering Incident Response Procedure runs the operational response: role assignments (Incident Commander, Technical Lead, Communications Lead), a Slack incident channel, a Jira incident ticket (DEVOPS project, Incident type), investigation, remediation and stakeholder communication.
For AI systems:
- Containment: consider disabling affected features, or reverting to a non-AI fallback if one exists.
- Evidence preservation: capture model versions, configuration, input and output samples, and logs before making changes.
- Bias and fairness issues: involve diverse stakeholders in the assessment, and consider mitigating at once even if the root cause is unclear.
- Data issues: assess the scope of affected data and its downstream impact on decisions or operations.
- Third-party AI services: engage vendor support where it applies, and record the provider’s response and resolution.
Communication Requirements
Communication follows the Engineering Incident Response Procedure, with AIMS recipients and channels per AI-007 (Communication Triggers and Protocols, AI Incidents). The Communications Lead manages it during an active incident.
Post-Incident Activities
SEV1 and SEV2 incidents need a postmortem within 5 business days, using Rygen’s standard postmortem template. It is optional for SEV3.
The postmortem or its follow-up actions cover:
- Nonconformity assessment: did we fail to follow an AIMS procedure, miss a required control, or violate a policy or objective? If yes, create an NCR under AI-014, which handles root cause analysis and corrective action.
- Risk register update: record new risks or update risk assessments.
- Corrective actions: Jira tasks for technical remediation, if no NCR covers it.
- Lessons learned: update the relevant AIMS documents.
- Management review input: an incident summary in the next management review.
Remediation tasks are tracked in Jira with owners and due dates. The Principal AI Engineer reviews completion and effectiveness. An incident is closed only after its corrective actions are verified effective.
Roles and Responsibilities
Principal AI Engineer
Accountable for AI incident response. Makes sure the AIMS integration steps are done, approves significant corrective actions, reports incidents to the CTO and the AI Governance Committee, updates the risk register and drives process improvements.
Incident Commander (Operational Role)
Leads coordination and resolution, makes sure AI-specific considerations are addressed, and works with the Principal AI Engineer on AIMS integration.
Technical Lead (Operational Role)
Investigates the technical root cause, implements and validates fixes, and gives technical input to the postmortem.
Communications Lead (Operational Role)
Manages stakeholder communication under AI-007 and works with the Principal AI Engineer on AI-specific messages.
Information Security Officer
Assesses security, privacy and compliance impact, decides whether regulatory notification is needed, and makes sure security controls are updated.
Team Leads
Support incident response in their area, implement corrective actions, and make sure their team learns from incidents.
Metrics and Monitoring
Each management review reports the AI incidents in the period, with their severity, the time to recovery, any repeat of an earlier root cause, and the status of their postmortems and corrective actions (AI-012).
Integration with Other AIMS Processes
Incidents inform risk identification and assessment (AI-008), may lead to an NCR and corrective action (AI-014), and feed AIMS performance evaluation (AI-017). AI-007 defines stakeholder communication. Incident trends and lessons learned are reported at each management review (AI-012).
Operational Implementation
Rygen’s Engineering Incident Response Procedure carries out this procedure and holds the detailed operational guidance.
Review and Improvement
Reviewed every year, after any SEV1 incident, when the Engineering Incident Response Procedure changes, and when audit findings or lessons learned call for it.
Revision History
| Version | Date | Author | Summary of Change |
|---|---|---|---|
| 1.0 | 2025-01-14 | Field Bradley | Initial version integrating AIMS requirements with operational incident response |
| 1.1 | 2026-07-14 | Field Bradley | Removed “quarterly” qualifier from management-review reporting to track the every-four-months cadence (AI-1759) |
| 1.2 | 2026-11-04 | Field Bradley | SEV3 incidents are reported to the CTO via Slack; monthly reports are not done (AI-2173) |
| 1.3 | 2026-11-09 | Field Bradley | Shortened every section, cites AI-007 for communication, treats documentation gaps as nonconformities, and names the incident measures the management review reports (AI-2173). Removes response times that no procedure sets, and vendor support is engaged where it applies. Communication follows the Engineering Incident Response Procedure; time to recovery reported. |