Large Language Models for Failure Mode Classification: An Investigation

Michael Stewart, Melinda Hodkiewicz, Sirui Li
9/15/2023
cs.CL

No official code found. Best implementation: nlp-tlp/llm-fmc (12 stars, last commit 2023-09-20).

Abstract

In this paper we present the first investigation into the effectiveness of Large Language Models (LLMs) for Failure Mode Classification (FMC). FMC, the task of automatically labelling an observation with a corresponding failure mode code, is a critical task in the maintenance domain as it reduces the need for reliability engineers to spend their time manually analysing work orders. We detail our approach to prompt engineering to enable an LLM to predict the failure mode of a given observation using a restricted code list. We demonstrate that the performance of a GPT-3.5 model (F1=0.80) fine-tuned on annotated data is a significant improvement over a currently available text classification model (F1=0.60) trained on the same annotated data set. The fine-tuned model also outperforms the out-of-the box GPT-3.5 (F1=0.46). This investigation reinforces the need for high quality fine-tuning data sets for domain-specific tasks using LLMs.

View on arXivView PDF

Code Implementations(1)

nlp-tlp/llm-fmcCommunity100%

Experiments on using ChatGPT for failure mode classification

125PythonAug 16, 20233 years ago

Similar Papers

Learning from Failures: A Failure-Driven Prompt Refinement for LLM-Based Vulnerability Analysis

83% match2026

Understanding Defects in Generated Codes by Language Models

82% match2024

Breaking the Homogeneity Assumption: Specialized Multi-Generator Adversarial Learning for Rare Failure Detection in Predictive Maintenance

82% match2026

Donkii: Can Annotation Error Detection Methods Find Errors in Instruction-Tuning Datasets?

82% match2023

PHM-Bench: A Domain-Specific Benchmarking Framework for Systematic Evaluation of Large Models in Prognostics and Health Management

81% match2025

MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models

81% match1 repo232026

Cite this paper

@article{stewart2023large,
  title  = {Large Language Models for Failure Mode Classification: An Investigation},
  author = {Michael Stewart and Melinda Hodkiewicz and Sirui Li},
  year   = {2023},
  eprint = {2309.08181},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.08181v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.08181.svg)](https://codeofpaper.com/paper/2309.08181)

Discussion