Configuration Builder

Configuration Builder

This interactive tool helps you create FreeEval evaluation configurations. Choose your evaluation type and model, then customize the settings to generate a configuration file.

1. Choose Evaluation Type

CategoryMultiple Choice / Question AnsweringOpen-ended GenerationSpecial

Evaluation MethodMultiple ChoiceCloze Prompt

Step Name

Give this evaluation step a descriptive name

Basic multiple choice evaluation step.

2. Choose Model to be Evaluated

🤖

OpenAI API

Evaluate OpenAI models via API

🌐

Remote TGI

Evaluate models via Text Generation Inference API

💻

Local Model

Evaluate models on local GPUs

Required Settings

Model Name

TGI Server URLs

+

Optional Settings

Request Timeout (seconds)

Number of Workers

3. Configure Dataset

Dataset TypeARC ChallengeMMLUTruthfulQAHellaSwagCEvalGSM8KHotpotQAReClor

Random Seed

Splittrainvalidationtest

Number of Few-shot Examples

Few-shot Splittrain

Generated Configuration

Copy to Clipboard

{
  "results_output_path": "./results/evaluation_results.json",
  "steps": [
    {
      "step_type": "simple_multiple_choice",
      "step_name": "",
      "output_path": "./output_test/",
      "dataset_config": {
        "type": "arc_challenge",
        "dataset_kwargs": {
          "seed": 2,
          "split": "test",
          "name_or_path": "ai2_arc",
          "fewshot_split": "train",
          "fewshot_num": 5
        }
      },
      "inference_config": {
        "type": "remote_hf",
        "inference_kwargs": {
          "model_name": "",
          "base_url": [
            ""
          ],
          "timeout": 60,
          "generation_config": {
            "max_new_tokens": 2000,
            "stop_sequences": [
              "\n\n"
            ]
          }
        }
      }
    }
  ]
}