RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation

Ivan Bondarenko, Roman Derunets, Oleg Sedukhin, Mikhail Komarov, Ivan Chernov +1 more
5/6/2026
cs.CLcs.AIcs.LG

Official code: RaguTeam/ragu_mtrag_semeval (8 stars, MIT, last commit 2026-05-04).

Abstract

We present our winning system for Task~B (generation with reference passages) in SemEval-2026 Task~8: MTRAGEval. Our method is a heterogeneous ensemble of seven LLMs with two prompting variants, where a GPT-4o-mini judge selects the best candidate per instance. We ranked 1st out of 26 teams, achieving a conditioned harmonic mean of 0.7827 and outperforming the strongest baseline (gpt-oss-120b, 0.6390). Ablations show that diversity in model families, scales, and prompting strategies is essential, with the ensemble consistently beating any single model. We also introduce Meno-Lite-0.1, a 7B domain-adapted model with a strong cost--performance trade-off, and analyse MTRAGEval, highlighting annotation limitations and directions for improvement. Our code is publicly available: https://github.com/RaguTeam/ragu_mtrag_semeval

View on arXivView PDF

Code Implementations(1)

RaguTeam/ragu_mtrag_semevalOfficial100%
MIT

First place in Task B 2026 solution in SemEval MTRAG track by RAGU team.

82Shell, PythonMay 4, 20265 months agoMIT

Similar Papers

PSK at SemEval-2026 Task 9: Multilingual Polarization Detection Using Ensemble Gemma Models with Synthetic Data Augmentation

83% match2026

IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English

81% match2026

DEI: Diversity in Evolutionary Inference for Quality-Diversity Search

81% match2026

Adam-Smith at SemEval-2023 Task 4: Discovering Human Values in Arguments with Ensembles of Transformer-based Models

80% match1 repo142023

CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data

80% match2026

FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches

80% match1 repo2026

Cite this paper

@article{bondarenko2026raguteam,
  title  = {RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation},
  author = {Ivan Bondarenko and Roman Derunets and Oleg Sedukhin and Mikhail Komarov and Ivan Chernov and Mikhail Kulakov},
  year   = {2026},
  eprint = {2605.04523},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2605.04523v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2605.04523.svg)](https://codeofpaper.com/paper/2605.04523)

Discussion