鱼皮AI导航
鱼皮AI导航
  • 主页
  • 社区
  • AI工具
  • AI知识库
  • AI提示词
  • MCP
  • Skills
  • AI学习资源
  • AI资讯
  • GitHub开源
登录/注册

evaluation-lm-evaluation-harness

提供对大语言模型的标准化学术评测能力,覆盖推理、代码、事实性等多维度任务,支持主流模型加载与分布式加速,适用于模型质量对比、训练过程监控及科研结果复现。

数据与分析研究
访问 GitHub
文档Skills.MD
name
evaluating-llms-harness
description
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
version
1.0.0
author
Orchestra Research
license
MIT
tags["Evaluation","LM Evaluation Harness","Benchmarking","MMLU","HumanEval","GSM8K","EleutherAI","Model Quality","Academic Benchmarks","Industry Standard"]
dependencies["lm-eval","transformers","vllm"]

lm-evaluation-harness - LLM Benchmarking

Quick start

lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

Installation:

▼
bash
复制代码
pip install lm-eval

Evaluate any HuggingFace model:

▼
bash
复制代码
lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu,gsm8k,hellaswag \ --device cuda:0 \ --batch_size 8

View available tasks:

▼
bash
复制代码
lm_eval --tasks list

Common workflows

Workflow 1: Standard benchmark evaluation

Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval).

Copy this checklist:

▼
text
复制代码
Benchmark Evaluation: - [ ] Step 1: Choose benchmark suite - [ ] Step 2: Configure model - [ ] Step 3: Run evaluation - [ ] Step 4: Analyze results

Step 1: Choose benchmark suite

Core reasoning benchmarks:

  • MMLU (Massive Multitask Language Understanding) - 57 subjects, multiple choice
  • GSM8K - Grade school math word problems
  • HellaSwag - Common sense reasoning
  • TruthfulQA - Truthfulness and factuality
  • ARC (AI2 Reasoning Challenge) - Science questions

Code benchmarks:

  • HumanEval - Python code generation (164 problems)
  • MBPP (Mostly Basic Python Problems) - Python coding

Standard suite (recommended for model releases):

▼
bash
复制代码
--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge

Step 2: Configure model

HuggingFace model:

▼
bash
复制代码
lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf,dtype=bfloat16 \ --tasks mmlu \ --device cuda:0 \ --batch_size auto # Auto-detect optimal batch size

Quantized model (4-bit/8-bit):

▼
bash
复制代码
lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf,load_in_4bit=True \ --tasks mmlu \ --device cuda:0

Custom checkpoint:

▼
bash
复制代码
lm_eval --model hf \ --model_args pretrained=/path/to/my-model,tokenizer=/path/to/tokenizer \ --tasks mmlu \ --device cuda:0

Step 3: Run evaluation

▼
bash
复制代码
# Full MMLU evaluation (57 subjects) lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu \ --num_fewshot 5 \ # 5-shot evaluation (standard) --batch_size 8 \ --output_path results/ \ --log_samples # Save individual predictions # Multiple benchmarks at once lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \ --num_fewshot 5 \ --batch_size 8 \ --output_path results/llama2-7b-eval.json

Step 4: Analyze results

Results saved to results/llama2-7b-eval.json:

▼
json
复制代码
{ "results": { "mmlu": { "acc": 0.459, "acc_stderr": 0.004 }, "gsm8k": { "exact_match": 0.142, "exact_match_stderr": 0.006 }, "hellaswag": { "acc_norm": 0.765, "acc_norm_stderr": 0.004 } }, "config": { "model": "hf", "model_args": "pretrained=meta-llama/Llama-2-7b-hf", "num_fewshot": 5 } }

Workflow 2: Track training progress

Evaluate checkpoints during training.

▼
text
复制代码
Training Progress Tracking: - [ ] Step 1: Set up periodic evaluation - [ ] Step 2: Choose quick benchmarks - [ ] Step 3: Automate evaluation - [ ] Step 4: Plot learning curves

Step 1: Set up periodic evaluation

Evaluate every N training steps:

▼
bash
复制代码
#!/bin/bash # eval_checkpoint.sh CHECKPOINT_DIR=$1 STEP=$2 lm_eval --model hf \ --model_args pretrained=$CHECKPOINT_DIR/checkpoint-$STEP \ --tasks gsm8k,hellaswag \ --num_fewshot 0 \ # 0-shot for speed --batch_size 16 \ --output_path results/step-$STEP.json

Step 2: Choose quick benchmarks

Fast benchmarks for frequent evaluation:

  • HellaSwag: ~10 minutes on 1 GPU
  • GSM8K: ~5 minutes
  • PIQA: ~2 minutes

Avoid for frequent eval (too slow):

  • MMLU: ~2 hours (57 subjects)
  • HumanEval: Requires code execution

Step 3: Automate evaluation

Integrate with training script:

▼
python
复制代码
# In training loop if step % eval_interval == 0: model.save_pretrained(f"checkpoints/step-{step}") # Run evaluation os.system(f"./eval_checkpoint.sh checkpoints step-{step}")

Or use PyTorch Lightning callbacks:

▼
python
复制代码
from pytorch_lightning import Callback class EvalHarnessCallback(Callback): def on_validation_epoch_end(self, trainer, pl_module): step = trainer.global_step checkpoint_path = f"checkpoints/step-{step}" # Save checkpoint trainer.save_checkpoint(checkpoint_path) # Run lm-eval os.system(f"lm_eval --model hf --model_args pretrained={checkpoint_path} ...")

Step 4: Plot learning curves

▼
python
复制代码
import json import matplotlib.pyplot as plt # Load all results steps = [] mmlu_scores = [] for file in sorted(glob.glob("results/step-*.json")): with open(file) as f: data = json.load(f) step = int(file.split("-")[1].split(".")[0]) steps.append(step) mmlu_scores.append(data["results"]["mmlu"]["acc"]) # Plot plt.plot(steps, mmlu_scores) plt.xlabel("Training Step") plt.ylabel("MMLU Accuracy") plt.title("Training Progress") plt.savefig("training_curve.png")

Workflow 3: Compare multiple models

Benchmark suite for model comparison.

▼
text
复制代码
Model Comparison: - [ ] Step 1: Define model list - [ ] Step 2: Run evaluations - [ ] Step 3: Generate comparison table

Step 1: Define model list

▼
bash
复制代码
# models.txt meta-llama/Llama-2-7b-hf meta-llama/Llama-2-13b-hf mistralai/Mistral-7B-v0.1 microsoft/phi-2

Step 2: Run evaluations

▼
bash
复制代码
#!/bin/bash # eval_all_models.sh TASKS="mmlu,gsm8k,hellaswag,truthfulqa" while read model; do echo "Evaluating $model" # Extract model name for output file model_name=$(echo $model | sed 's/\//-/g') lm_eval --model hf \ --model_args pretrained=$model,dtype=bfloat16 \ --tasks $TASKS \ --num_fewshot 5 \ --batch_size auto \ --output_path results/$model_name.json done < models.txt

Step 3: Generate comparison table

▼
python
复制代码
import json import pandas as pd models = [ "meta-llama-Llama-2-7b-hf", "meta-llama-Llama-2-13b-hf", "mistralai-Mistral-7B-v0.1", "microsoft-phi-2" ] tasks = ["mmlu", "gsm8k", "hellaswag", "truthfulqa"] results = [] for model in models: with open(f"results/{model}.json") as f: data = json.load(f) row = {"Model": model.replace("-", "/")} for task in tasks: # Get primary metric for each task metrics = data["results"][task] if "acc" in metrics: row[task.upper()] = f"{metrics['acc']:.3f}" elif "exact_match" in metrics: row[task.upper()] = f"{metrics['exact_match']:.3f}" results.append(row) df = pd.DataFrame(results) print(df.to_markdown(index=False))

Output:

▼
text
复制代码
| Model | MMLU | GSM8K | HELLASWAG | TRUTHFULQA | |------------------------|-------|-------|-----------|------------| | meta-llama/Llama-2-7b | 0.459 | 0.142 | 0.765 | 0.391 | | meta-llama/Llama-2-13b | 0.549 | 0.287 | 0.801 | 0.430 | | mistralai/Mistral-7B | 0.626 | 0.395 | 0.812 | 0.428 | | microsoft/phi-2 | 0.560 | 0.613 | 0.682 | 0.447 |

Workflow 4: Evaluate with vLLM (faster inference)

Use vLLM backend for 5-10x faster evaluation.

▼
text
复制代码
vLLM Evaluation: - [ ] Step 1: Install vLLM - [ ] Step 2: Configure vLLM backend - [ ] Step 3: Run evaluation

Step 1: Install vLLM

▼
bash
复制代码
pip install vllm

Step 2: Configure vLLM backend

▼
bash
复制代码
lm_eval --model vllm \ --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \ --tasks mmlu \ --batch_size auto

Step 3: Run evaluation

vLLM is 5-10× faster than standard HuggingFace:

▼
bash
复制代码
# Standard HF: ~2 hours for MMLU on 7B model lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu \ --batch_size 8 # vLLM: ~15-20 minutes for MMLU on 7B model lm_eval --model vllm \ --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=2 \ --tasks mmlu \ --batch_size auto

When to use vs alternatives

Use lm-evaluation-harness when:

  • Benchmarking models for academic papers
  • Comparing model quality across standard tasks
  • Tracking training progress
  • Reporting standardized metrics (everyone uses same prompts)
  • Need reproducible evaluation

Use alternatives instead:

  • HELM (Stanford): Broader evaluation (fairness, efficiency, calibration)
  • AlpacaEval: Instruction-following evaluation with LLM judges
  • MT-Bench: Conversational multi-turn evaluation
  • Custom scripts: Domain-specific evaluation

Common issues

Issue: Evaluation too slow

Use vLLM backend:

▼
bash
复制代码
lm_eval --model vllm \ --model_args pretrained=model-name,tensor_parallel_size=2

Or reduce fewshot examples:

▼
bash
复制代码
--num_fewshot 0 # Instead of 5

Or evaluate subset of MMLU:

▼
bash
复制代码
--tasks mmlu_stem # Only STEM subjects

Issue: Out of memory

Reduce batch size:

▼
bash
复制代码
--batch_size 1 # Or --batch_size auto

Use quantization:

▼
bash
复制代码
--model_args pretrained=model-name,load_in_8bit=True

Enable CPU offloading:

▼
bash
复制代码
--model_args pretrained=model-name,device_map=auto,offload_folder=offload

Issue: Different results than reported

Check fewshot count:

▼
bash
复制代码
--num_fewshot 5 # Most papers use 5-shot

Check exact task name:

▼
bash
复制代码
--tasks mmlu # Not mmlu_direct or mmlu_fewshot

Verify model and tokenizer match:

▼
bash
复制代码
--model_args pretrained=model-name,tokenizer=same-model-name

Issue: HumanEval not executing code

Install execution dependencies:

▼
bash
复制代码
pip install human-eval

Enable code execution:

▼
bash
复制代码
lm_eval --model hf \ --model_args pretrained=model-name \ --tasks humaneval \ --allow_code_execution # Required for HumanEval

Advanced topics

Benchmark descriptions: See references/benchmark-guide.md for detailed description of all 60+ tasks, what they measure, and interpretation.

Custom tasks: See references/custom-tasks.md for creating domain-specific evaluation tasks.

API evaluation: See references/api-evaluation.md for evaluating OpenAI, Anthropic, and other API models.

Multi-GPU strategies: See references/distributed-eval.md for data parallel and tensor parallel evaluation.

Hardware requirements

  • GPU: NVIDIA (CUDA 11.8+), works on CPU (very slow)
  • VRAM:
    • 7B model: 16GB (bf16) or 8GB (8-bit)
    • 13B model: 28GB (bf16) or 14GB (8-bit)
    • 70B model: Requires multi-GPU or quantization
  • Time (7B model, single A100):
    • HellaSwag: 10 minutes
    • GSM8K: 5 minutes
    • MMLU (full): 2 hours
    • HumanEval: 20 minutes

Resources

  • GitHub: https://github.com/EleutherAI/lm-evaluation-harness
  • Docs: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs
  • Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc.
  • Leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard (uses this harness)
数据与分析
研究

快捷安装

复制到终端去安装 Skill

npxskillsadddavila7/claude-code-templates--skill"evaluation-lm-evaluation-harness"
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
🔥热门工具
朱雀AI检测朱雀AI检测
朱雀AI检测助手朱雀AI检测助手
LovartLovart
老鱼简历老鱼简历
WritePass官网WritePass官网
面多多面多多
CursorCursor
星月写作星月写作
朱雀AI检测助手朱雀AI检测助手
零感AI零感AI
Skills 推荐
definition-of-done-generator
自动生成项目完成标准的规范定义与配置文件,提供符合企业级工作流要求的最佳实践指导,输出可直接投入生产环境的代码和验证结果,适用于需要标准化流程审批、合规审查及跨系统集成的复杂项目管理场景。
sagemaker-endpoint-deployer
自动化部署机器学习模型至云端服务端点,支持从配置生成到生产验证的全流程,适用于模型上线、服务监控与推理优化等场景,确保部署方案符合工业级标准。
docx
支持专业级 Word 文档的全生命周期处理,包括从零创建结构化文档、精准编辑内容、管理修订痕迹与批注、提取纯文本或原始 XML 数据、转换为 PDF 或图像用于视觉分析,适用于法律、学术及企业场景下的高保真文档协作与自动化处理。
skills-search
提供对 Claude Code 技能生态的全周期管理能力,支持按功能关键词检索技能、查看详细信息、安装或卸载指定技能,并区分用户级与项目级作用域,覆盖发现、评估、部署及维护各环节。
moai-foundation-core
构建高质量、可扩展的AI开发工作流,通过领域驱动设计与模块化架构实现需求规范先行,结合自动化质量门禁、精细化上下文管理与渐进式知识呈现,在保障安全与可维护性的同时提升开发效率与协作清晰度。