Custom Datasets

此内容尚不支持你的语言。

Custom Datasets

FreeEval allows you to create and use custom datasets for evaluating language models. This guide explains how to implement your own dataset classes and integrate them with the framework.

Dataset Builder Tool

Use this interactive tool to generate code for implementing a new multiple choice dataset:

Custom Dataset Builder

Dataset Information

Dataset Structure

FreeEval datasets inherit from either MultipleChoiceDataset or AugmentedMultipleChoiceDataset base classes. The key components of a dataset implementation include:

Data Loading: Loading data from HuggingFace, local files, or other sources
Data Parsing: Converting raw data into standardized MultipleChoiceProblem instances
Prompt Generation: Creating prompts for model evaluation

Manual Implementation

If you prefer to implement your dataset manually, follow these steps:

1. Create a new dataset class

Create a new file in freeeval/datasets/ with your dataset implementation:

from freeeval.datasets.multiple_choice import (
    MultipleChoiceDataset,
    AugmentedMultipleChoiceDataset,
    MultipleChoiceProblem,
)
from datasets import load_dataset

class YourDataset(AugmentedMultipleChoiceDataset):
    def __init__(
        self,
        seed=1,
        split="test",
        name_or_path=None,
        config_name=None,
        fewshot_split="train",
        fewshot_num=0,
        **kwargs
    ):
        super().__init__(seed=seed, **kwargs)
        # Initialize dataset-specific attributes
        # Load dataset

    def parse_data_instance(self, data, extra={}):
        # Parse a single data instance into a MultipleChoiceProblem
        pass

    def parse_hf_dataset(self):
        # Parse all instances in the dataset
        pass

2. Register your dataset

Update freeeval/datasets/__init__.py to register your dataset:

from freeeval.datasets.your_dataset import YourDataset

TYPE_TO_DATASET = {
    # Existing datasets...
    "your_dataset": YourDataset,
}

3. Use your dataset in evaluations

Create a configuration file that uses your dataset:

{
  "steps": [
    {
      "step_type": "simple_multiple_choice",
      "dataset_config": {
        "type": "your_dataset",
        "dataset_kwargs": {
          "seed": 2,
          "split": "test"
        }
      }
    }
  ]
}

Advanced Dataset Features

Few-shot Learning: Configure few-shot examples with fewshot_num and fewshot_split
Dataset Augmentation: Use augment_dataset_kwargs to create variations of problems
Custom Prompts: Override prompt generation for specialized formats