鱼皮AI导航
鱼皮AI导航
  • 主页
  • 社区
  • AI工具
  • AI知识库
  • AI提示词
  • MCP
  • Skills
  • AI学习资源
  • AI资讯
  • GitHub开源
登录/注册

inference-serving-vllm

基于 PagedAttention与连续批处理技术实现高吞吐量大模型推理服务,支持在有限显存下部署大型语言模型,提供兼容 OpenAI 接口的 API 服务,适用于生产环境中的低延迟、高并发场景。

数据与分析
访问 GitHub
文档Skills.MD
name
serving-llms-vllm
description
Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.
version
1.0.0
author
Orchestra Research
license
MIT
tags["vLLM","Inference Serving","PagedAttention","Continuous Batching","High Throughput","Production","OpenAI API","Quantization","Tensor Parallelism"]
dependencies["vllm","torch","transformers"]

vLLM - High-Performance LLM Serving

Quick start

vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests).

Installation:

▼
bash
复制代码
pip install vllm

Basic offline inference:

▼
python
复制代码
from vllm import LLM, SamplingParams llm = LLM(model="meta-llama/Llama-3-8B-Instruct") sampling = SamplingParams(temperature=0.7, max_tokens=256) outputs = llm.generate(["Explain quantum computing"], sampling) print(outputs[0].outputs[0].text)

OpenAI-compatible server:

▼
bash
复制代码
vllm serve meta-llama/Llama-3-8B-Instruct # Query with OpenAI SDK python -c " from openai import OpenAI client = OpenAI(base_url='http://localhost:8000/v1', api_key='EMPTY') print(client.chat.completions.create( model='meta-llama/Llama-3-8B-Instruct', messages=[{'role': 'user', 'content': 'Hello!'}] ).choices[0].message.content) "

Common workflows

Workflow 1: Production API deployment

Copy this checklist and track progress:

▼
text
复制代码
Deployment Progress: - [ ] Step 1: Configure server settings - [ ] Step 2: Test with limited traffic - [ ] Step 3: Enable monitoring - [ ] Step 4: Deploy to production - [ ] Step 5: Verify performance metrics

Step 1: Configure server settings

Choose configuration based on your model size:

▼
bash
复制代码
# For 7B-13B models on single GPU vllm serve meta-llama/Llama-3-8B-Instruct \ --gpu-memory-utilization 0.9 \ --max-model-len 8192 \ --port 8000 # For 30B-70B models with tensor parallelism vllm serve meta-llama/Llama-2-70b-hf \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.9 \ --quantization awq \ --port 8000 # For production with caching and metrics vllm serve meta-llama/Llama-3-8B-Instruct \ --gpu-memory-utilization 0.9 \ --enable-prefix-caching \ --enable-metrics \ --metrics-port 9090 \ --port 8000 \ --host 0.0.0.0

Step 2: Test with limited traffic

Run load test before production:

▼
bash
复制代码
# Install load testing tool pip install locust # Create test_load.py with sample requests # Run: locust -f test_load.py --host http://localhost:8000

Verify TTFT (time to first token) < 500ms and throughput > 100 req/sec.

Step 3: Enable monitoring

vLLM exposes Prometheus metrics on port 9090:

▼
bash
复制代码
curl http://localhost:9090/metrics | grep vllm

Key metrics to monitor:

  • vllm:time_to_first_token_seconds - Latency
  • vllm:num_requests_running - Active requests
  • vllm:gpu_cache_usage_perc - KV cache utilization

Step 4: Deploy to production

Use Docker for consistent deployment:

▼
bash
复制代码
# Run vLLM in Docker docker run --gpus all -p 8000:8000 \ vllm/vllm-openai:latest \ --model meta-llama/Llama-3-8B-Instruct \ --gpu-memory-utilization 0.9 \ --enable-prefix-caching

Step 5: Verify performance metrics

Check that deployment meets targets:

  • TTFT < 500ms (for short prompts)
  • Throughput > target req/sec
  • GPU utilization > 80%
  • No OOM errors in logs

Workflow 2: Offline batch inference

For processing large datasets without server overhead.

Copy this checklist:

▼
text
复制代码
Batch Processing: - [ ] Step 1: Prepare input data - [ ] Step 2: Configure LLM engine - [ ] Step 3: Run batch inference - [ ] Step 4: Process results

Step 1: Prepare input data

▼
python
复制代码
# Load prompts from file prompts = [] with open("prompts.txt") as f: prompts = [line.strip() for line in f] print(f"Loaded {len(prompts)} prompts")

Step 2: Configure LLM engine

▼
python
复制代码
from vllm import LLM, SamplingParams llm = LLM( model="meta-llama/Llama-3-8B-Instruct", tensor_parallel_size=2, # Use 2 GPUs gpu_memory_utilization=0.9, max_model_len=4096 ) sampling = SamplingParams( temperature=0.7, top_p=0.95, max_tokens=512, stop=["</s>", "\n\n"] )

Step 3: Run batch inference

vLLM automatically batches requests for efficiency:

▼
python
复制代码
# Process all prompts in one call outputs = llm.generate(prompts, sampling) # vLLM handles batching internally # No need to manually chunk prompts

Step 4: Process results

▼
python
复制代码
# Extract generated text results = [] for output in outputs: prompt = output.prompt generated = output.outputs[0].text results.append({ "prompt": prompt, "generated": generated, "tokens": len(output.outputs[0].token_ids) }) # Save to file import json with open("results.jsonl", "w") as f: for result in results: f.write(json.dumps(result) + "\n") print(f"Processed {len(results)} prompts")

Workflow 3: Quantized model serving

Fit large models in limited GPU memory.

▼
text
复制代码
Quantization Setup: - [ ] Step 1: Choose quantization method - [ ] Step 2: Find or create quantized model - [ ] Step 3: Launch with quantization flag - [ ] Step 4: Verify accuracy

Step 1: Choose quantization method

  • AWQ: Best for 70B models, minimal accuracy loss
  • GPTQ: Wide model support, good compression
  • FP8: Fastest on H100 GPUs

Step 2: Find or create quantized model

Use pre-quantized models from HuggingFace:

▼
bash
复制代码
# Search for AWQ models # Example: TheBloke/Llama-2-70B-AWQ

Step 3: Launch with quantization flag

▼
bash
复制代码
# Using pre-quantized model vllm serve TheBloke/Llama-2-70B-AWQ \ --quantization awq \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.95 # Results: 70B model in ~40GB VRAM

Step 4: Verify accuracy

Test outputs match expected quality:

▼
python
复制代码
# Compare quantized vs non-quantized responses # Verify task-specific performance unchanged

When to use vs alternatives

Use vLLM when:

  • Deploying production LLM APIs (100+ req/sec)
  • Serving OpenAI-compatible endpoints
  • Limited GPU memory but need large models
  • Multi-user applications (chatbots, assistants)
  • Need low latency with high throughput

Use alternatives instead:

  • llama.cpp: CPU/edge inference, single-user
  • HuggingFace transformers: Research, prototyping, one-off generation
  • TensorRT-LLM: NVIDIA-only, need absolute maximum performance
  • Text-Generation-Inference: Already in HuggingFace ecosystem

Common issues

Issue: Out of memory during model loading

Reduce memory usage:

▼
bash
复制代码
vllm serve MODEL \ --gpu-memory-utilization 0.7 \ --max-model-len 4096

Or use quantization:

▼
bash
复制代码
vllm serve MODEL --quantization awq

Issue: Slow first token (TTFT > 1 second)

Enable prefix caching for repeated prompts:

▼
bash
复制代码
vllm serve MODEL --enable-prefix-caching

For long prompts, enable chunked prefill:

▼
bash
复制代码
vllm serve MODEL --enable-chunked-prefill

Issue: Model not found error

Use --trust-remote-code for custom models:

▼
bash
复制代码
vllm serve MODEL --trust-remote-code

Issue: Low throughput (<50 req/sec)

Increase concurrent sequences:

▼
bash
复制代码
vllm serve MODEL --max-num-seqs 512

Check GPU utilization with nvidia-smi - should be >80%.

Issue: Inference slower than expected

Verify tensor parallelism uses power of 2 GPUs:

▼
bash
复制代码
vllm serve MODEL --tensor-parallel-size 4 # Not 3

Enable speculative decoding for faster generation:

▼
bash
复制代码
vllm serve MODEL --speculative-model DRAFT_MODEL

Advanced topics

Server deployment patterns: See references/server-deployment.md for Docker, Kubernetes, and load balancing configurations.

Performance optimization: See references/optimization.md for PagedAttention tuning, continuous batching details, and benchmark results.

Quantization guide: See references/quantization.md for AWQ/GPTQ/FP8 setup, model preparation, and accuracy comparisons.

Troubleshooting: See references/troubleshooting.md for detailed error messages, debugging steps, and performance diagnostics.

Hardware requirements

  • Small models (7B-13B): 1x A10 (24GB) or A100 (40GB)
  • Medium models (30B-40B): 2x A100 (40GB) with tensor parallelism
  • Large models (70B+): 4x A100 (40GB) or 2x A100 (80GB), use AWQ/GPTQ

Supported platforms: NVIDIA (primary), AMD ROCm, Intel GPUs, TPUs

Resources

  • Official docs: https://docs.vllm.ai
  • GitHub: https://github.com/vllm-project/vllm
  • Paper: "Efficient Memory Management for Large Language Model Serving with PagedAttention" (SOSP 2023)
  • Community: https://discuss.vllm.ai
数据与分析

快捷安装

复制到终端去安装 Skill

npxskillsadddavila7/claude-code-templates--skill"inference-serving-vllm"
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
🔥热门工具
朱雀AI检测朱雀AI检测
朱雀AI检测助手朱雀AI检测助手
LovartLovart
老鱼简历老鱼简历
WritePass官网WritePass官网
面多多面多多
CursorCursor
星月写作星月写作
朱雀AI检测助手朱雀AI检测助手
零感AI零感AI
Skills 推荐
definition-of-done-generator
自动生成项目完成标准的规范定义与配置文件,提供符合企业级工作流要求的最佳实践指导,输出可直接投入生产环境的代码和验证结果,适用于需要标准化流程审批、合规审查及跨系统集成的复杂项目管理场景。
sagemaker-endpoint-deployer
自动化部署机器学习模型至云端服务端点,支持从配置生成到生产验证的全流程,适用于模型上线、服务监控与推理优化等场景,确保部署方案符合工业级标准。
docx
支持专业级 Word 文档的全生命周期处理,包括从零创建结构化文档、精准编辑内容、管理修订痕迹与批注、提取纯文本或原始 XML 数据、转换为 PDF 或图像用于视觉分析,适用于法律、学术及企业场景下的高保真文档协作与自动化处理。
skills-search
提供对 Claude Code 技能生态的全周期管理能力,支持按功能关键词检索技能、查看详细信息、安装或卸载指定技能,并区分用户级与项目级作用域,覆盖发现、评估、部署及维护各环节。
moai-foundation-core
构建高质量、可扩展的AI开发工作流,通过领域驱动设计与模块化架构实现需求规范先行,结合自动化质量门禁、精细化上下文管理与渐进式知识呈现,在保障安全与可维护性的同时提升开发效率与协作清晰度。