Research area

Root Cause Analysis & AIOps

How can we turn operational telemetry into reliable diagnosis and actionable incident response?

MSRG develops methods for root cause analysis and AI for IT operations (AIOps) in distributed systems. We combine metrics, logs, and traces with service dependency graphs and LLM agents to investigate failures and produce actionable diagnoses. Our work connects anomaly detection, incident response, automated repair, and realistic evaluation, with a focus on microservices and the reliability of AI agent executions.

We acknowledge the support of Bell Mobility and the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant ALLRP614128-25.

Bell Mobility Natural Sciences and Engineering Research Council of Canada (NSERC)

10 researchers in this area · 50 publications · 0 data sets

Outcomes · Publications · People

From telemetry to root causes

A failure in one service can surface as errors, slowdowns, or unusual resource usage elsewhere. Root cause analysis (RCA) connects these symptoms to the underlying fault. Doing so requires reasoning across heterogeneous telemetry and the dependencies between services.

We study how AI for IT operations (AIOps) can help engineers move from detecting an anomaly to understanding a failure and deciding what to do next. Our research brings together statistical analysis, graph structure, and LLM agents, with an emphasis on diagnoses and repair decisions grounded in operational evidence.

AIOps research vision: observe distributed systems, detect anomalies, diagnose causes, and support incident response, with realistic benchmark evaluation.

Research vision: connect operational evidence to diagnosis and response, and evaluate each step in realistic systems.

GALA+ workflow: telemetry and service dependencies feed initial hypotheses, followed by graph-guided investigation using local evidence, producing diagnoses and incident response recommendations evaluated with SURE-Score.

Technical example: GALA+ uses service dependencies and local telemetry to guide LLM investigation; SURE-Score assesses the quality of its outputs.

Graph-guided diagnosis and incident response

GALA, whose updated framework is called GALA+, uses service dependencies to guide LLM agents through local metrics, logs, and traces. It produces ranked diagnoses, incident summaries, and action recommendations. SURE-Score, developed with industry site reliability engineers, assesses diagnostic quality beyond text similarity.

Message-level anomaly detection

Effective investigation starts with finding useful signals in large volumes of telemetry. FAME studies anomaly detection at the level of individual log messages. It uses an LLM during offline preparation to help organize failure domains, then trains a lightweight router and specialized experts for online detection with limited labeling effort.

Telemetry-grounded repair

Diagnosis can also provide context for changing code and configuration. ORCA compares failure and reference telemetry to build a fault signature, locate candidate repair sites, and guide patch generation. Its verification process examines patch correctness, test integrity, and telemetry replay to connect automated repair with the incident that motivated it.

Reliable agents and realistic evaluation

AI agents are both tools for incident response and systems whose own failures need diagnosis. AgentChaosBench studies fault detection and localization from agent execution telemetry, including faults at tool, model, guardrail, and inter-agent boundaries.

SREGym evaluates SRE agents in live cloud-native environments with injected faults and ambient noise. These complementary settings help us examine whether an agent can identify a fault, locate its origin, and support incident resolution under realistic operating conditions.

GF-Bench extends failure detection to distributed GPU workloads, while SRE-Marathon studies continuous operation with overlapping incidents and code or configuration changes.

Current directions

  • Correlating metrics, logs, and traces across service dependencies
  • Graph-guided and evidence-grounded LLM investigation
  • Connecting root cause diagnosis to remediation and automated repair
  • Evaluating diagnostic quality, agent reliability, and operational usefulness

This research connects with our work in Data Management. We also continue research in distributed AI and machine learning, including resource allocation and scheduling for training and inference, privacy-preserving and federated learning on embedded and edge devices, and scalable graph processing and graph neural networks.

Research outcomes

FAME

Message-level log anomaly detection using offline failure-domain grouping and a lightweight router with specialized experts.

Related publication

SREGym

A live cloud-native benchmark for evaluating how SRE agents diagnose and mitigate injected operational failures.

Related publication

GF-Bench artifacts

Tools for generating multi-GPU failures, collecting NCCL logs, and evaluating anomaly detection, with sample data and scripts to reproduce the experiments.

Related publication

Recent publications

Recent publications in root cause analysis, AIOps, and our broader distributed AI and machine learning research.

Browse all publications in this area

Researchers in this area

Current and visiting MSRG members associated with our AIOps and distributed AI/ML research are shown below. View the full MSRG team.

Professor

Hans-Arno Jacobsen

Hans-Arno Jacobsen

Professor

Toronto, Canada

Post-Docs

Yiran Li

Yiran Li

Post-Doc

Toronto, Canada

Graph AnalyticsGraph LearningQuantum Data ManagementAI Agent Reliability

Graduate Students PhD

Yifang Tian

Yifang Tian

PhD Student

Toronto, Canada

Machine Learning SystemsDistributed Systems
Michalis Bachras

Michalis Bachras

PhD Student

Toronto, Canada

Data ManagementDistributed Systems
Dawn (Mengli) Duan

Dawn (Mengli) Duan

PhD Student

Toronto, Canada

Machine Learning Systems
Ruiwu Liu

Ruiwu Liu

PhD Student

Toronto, Canada

VLMRLAIOps
YG

Yuanchen Gao

PhD Student

HKUST, Hong Kong

Root Cause AnalysisAutomated Program RepairAIOps

Graduate Students Master's

Yifeng (Jason) He

Yifeng (Jason) He

MSc Student

Toronto, Canada

Research Trainees Undergraduate

Ben Cheng

Ben Cheng

Undergraduate Student

Toronto, Canada

Machine Learning SystemsData Management
CZ

Chenkai Zhang

Undergraduate Student

Toronto, Canada

AI Agent ReliabilityFault Diagnosis