Skip to content
FreeEval Documentation

MT-Bench

MT-Bench

The mt_bench step implements the Multi-turn Benchmark (MT-Bench), a comprehensive evaluation framework that assesses model performance through structured multi-turn dialogues across diverse categories.

Overview

MT-Bench is a specialized benchmark designed to evaluate large language models (LLMs) on their conversation and instruction-following abilities. Unlike traditional benchmarks that focus on closed-ended questions with short responses, MT-Bench challenges models with open-ended, multi-turn dialogues that better reflect real-world applications. The benchmark was introduced by Zheng et al. (2023) as part of a systematic study on using LLMs as judges for evaluating other language models, addressing the challenge of efficiently measuring human preference alignment.

Technical Implementation

MT-Bench consists of 80 carefully crafted multi-turn questions spanning 8 categories that represent common use cases:

  1. Writing: Creative and professional writing tasks
  2. Roleplay: Simulating specific roles or characters
  3. Extraction: Finding and organizing information
  4. Reasoning: Logical deduction and analysis
  5. Math: Mathematical problem-solving
  6. Coding: Programming and software development
  7. Knowledge I: Questions on STEM subjects
  8. Knowledge II: Questions on humanities and social sciences

Each question consists of two turns, with the second turn typically requiring the model to build upon or modify its first response. This multi-turn structure specifically tests a model’s ability to maintain context and follow complex instructions across a conversation.

The evaluation process employs strong LLMs (such as GPT-4) as judges to rate responses on a 10-point scale. This “LLM-as-a-judge” approach has been validated to achieve over 80% agreement with human evaluators, making it a reliable and scalable alternative to traditional human evaluation.

Key Features

MT-Bench provides several advantages for comprehensive LLM evaluation:

  • Multi-turn Assessment: Tests a model’s ability to maintain context across conversation turns
  • Diverse Categories: Covers a wide range of capabilities required in real-world applications
  • Standardized Scoring: Uses a consistent 10-point scoring system across all questions
  • Automated Evaluation: Employs LLM judges to provide scalable, reproducible assessments
  • Human-Aligned Metrics: Focuses on criteria that correlate with human preferences

The benchmark is particularly effective at detecting differences between models that might perform similarly on traditional capability-based benchmarks but differ in their alignment with human preferences.

Model Compatibility

MT-Bench is designed for evaluating dialogue-capable instruction-tuned language models. It is applicable to:

  • Chat-optimized LLMs (e.g., ChatGPT, Claude, Llama-2-Chat)
  • Instruction-tuned foundation models
  • Multi-turn conversational assistants

Base language models without instruction-following capabilities may struggle with this benchmark as it requires understanding complex, often multi-part instructions and maintaining coherence across turns.

When to Use

MT-Bench is particularly valuable for:

  • Evaluating a model’s conversation and instruction-following abilities
  • Comparing models that perform similarly on traditional knowledge-based benchmarks
  • Assessing how well models handle complex, multi-step instructions
  • Measuring alignment with human preferences in interactive scenarios
  • Benchmarking models across diverse skill categories

This benchmark complements traditional capability-focused benchmarks by providing insights into aspects of model performance that more closely align with real-world user satisfaction.

Implementation Details

The implementation of MT-Bench follows these steps:

  1. Question Generation: Present the model with a carefully crafted first-turn question
  2. Response Collection: Record the model’s response to the first question
  3. Follow-up: Present a related second-turn question that builds on the first exchange
  4. Final Response: Record the model’s response to the second question
  5. Evaluation: Use an LLM judge to evaluate each response on a 10-point scale
  6. Scoring: Calculate the average score across all questions and turns

The evaluation can be performed using either pairwise comparison (comparing responses from two different models) or single-answer grading (scoring each response independently). The latter is more scalable and has been shown to correlate well with human judgments.

To address potential biases in LLM judges, techniques such as position swapping, chain-of-thought prompting, and reference-guided evaluation may be employed, particularly for questions involving complex reasoning or mathematical calculations.

Further Reading

For a comprehensive understanding of MT-Bench and the LLM-as-a-judge approach, refer to:

Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. https://arxiv.org/abs/2306.05685

This paper presents the design of MT-Bench, an analysis of the LLM-as-a-judge approach, and extensive experiments validating its alignment with human preferences.