鱼皮AI导航
鱼皮AI导航
  • 主页
  • 社区
  • AI工具
  • AI知识库
  • AI提示词
  • MCP
  • Skills
  • AI学习资源
  • AI资讯
  • GitHub开源
登录/注册

optimization-flash-attention

大幅提升 Transformer 模型的注意力计算效率,在长序列场景下实现 2 至 4 倍速度提升和 10 至 20 倍显存占用降低,兼容 PyTorch 原生 SDPA、flash-attn 库、H100 FP8 加速及滑动窗口机制,适用于训练与推理阶段的显存瓶颈或长上下文处理需求。

软件开发
访问 GitHub
文档Skills.MD
name
optimizing-attention-flash
description
Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training/running transformers with long sequences (>512 tokens), encountering GPU memory issues with attention, or need faster inference. Supports PyTorch native SDPA, flash-attn library, H100 FP8, and sliding window attention.
version
1.0.0
author
Orchestra Research
license
MIT
tags["Optimization","Flash Attention","Attention Optimization","Memory Efficiency","Speed Optimization","Long Context","PyTorch","SDPA","H100","FP8","Transformers"]
dependencies["flash-attn","torch","transformers"]

Flash Attention - Fast Memory-Efficient Attention

Quick start

Flash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation.

PyTorch native (easiest, PyTorch 2.2+):

▼
python
复制代码
import torch import torch.nn.functional as F q = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16) # [batch, heads, seq, dim] k = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16) v = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16) # Automatically uses Flash Attention if available out = F.scaled_dot_product_attention(q, k, v)

flash-attn library (more features):

▼
bash
复制代码
pip install flash-attn --no-build-isolation
▼
python
复制代码
from flash_attn import flash_attn_func # q, k, v: [batch, seqlen, nheads, headdim] out = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)

Common workflows

Workflow 1: Enable in existing PyTorch model

Copy this checklist:

▼
text
复制代码
Flash Attention Integration: - [ ] Step 1: Check PyTorch version (≥2.2) - [ ] Step 2: Enable Flash Attention backend - [ ] Step 3: Verify speedup with profiling - [ ] Step 4: Test accuracy matches baseline

Step 1: Check PyTorch version

▼
bash
复制代码
python -c "import torch; print(torch.__version__)" # Should be ≥2.2.0

If <2.2, upgrade:

▼
bash
复制代码
pip install --upgrade torch

Step 2: Enable Flash Attention backend

Replace standard attention:

▼
python
复制代码
# Before (standard attention) attn_weights = torch.softmax(q @ k.transpose(-2, -1) / math.sqrt(d_k), dim=-1) out = attn_weights @ v # After (Flash Attention) import torch.nn.functional as F out = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)

Force Flash Attention backend:

▼
python
复制代码
with torch.backends.cuda.sdp_kernel( enable_flash=True, enable_math=False, enable_mem_efficient=False ): out = F.scaled_dot_product_attention(q, k, v)

Step 3: Verify speedup with profiling

▼
python
复制代码
import torch.utils.benchmark as benchmark def test_attention(use_flash): q, k, v = [torch.randn(2, 8, 2048, 64, device='cuda', dtype=torch.float16) for _ in range(3)] if use_flash: with torch.backends.cuda.sdp_kernel(enable_flash=True): return F.scaled_dot_product_attention(q, k, v) else: attn = (q @ k.transpose(-2, -1) / 8.0).softmax(dim=-1) return attn @ v # Benchmark t_flash = benchmark.Timer(stmt='test_attention(True)', globals=globals()) t_standard = benchmark.Timer(stmt='test_attention(False)', globals=globals()) print(f"Flash: {t_flash.timeit(100).mean:.3f}s") print(f"Standard: {t_standard.timeit(100).mean:.3f}s")

Expected: 2-4x speedup for sequences >512 tokens.

Step 4: Test accuracy matches baseline

▼
python
复制代码
# Compare outputs q, k, v = [torch.randn(1, 8, 512, 64, device='cuda', dtype=torch.float16) for _ in range(3)] # Flash Attention out_flash = F.scaled_dot_product_attention(q, k, v) # Standard attention attn_weights = torch.softmax(q @ k.transpose(-2, -1) / 8.0, dim=-1) out_standard = attn_weights @ v # Check difference diff = (out_flash - out_standard).abs().max() print(f"Max difference: {diff:.6f}") # Should be <1e-3 for float16

Workflow 2: Use flash-attn library for advanced features

For multi-query attention, sliding window, or H100 FP8.

Copy this checklist:

▼
text
复制代码
flash-attn Library Setup: - [ ] Step 1: Install flash-attn library - [ ] Step 2: Modify attention code - [ ] Step 3: Enable advanced features - [ ] Step 4: Benchmark performance

Step 1: Install flash-attn library

▼
bash
复制代码
# NVIDIA GPUs (CUDA 12.0+) pip install flash-attn --no-build-isolation # Verify installation python -c "from flash_attn import flash_attn_func; print('Success')"

Step 2: Modify attention code

▼
python
复制代码
from flash_attn import flash_attn_func # Input: [batch_size, seq_len, num_heads, head_dim] # Transpose from [batch, heads, seq, dim] if needed q = q.transpose(1, 2) # [batch, seq, heads, dim] k = k.transpose(1, 2) v = v.transpose(1, 2) out = flash_attn_func( q, k, v, dropout_p=0.1, causal=True, # For autoregressive models window_size=(-1, -1), # No sliding window softmax_scale=None # Auto-scale ) out = out.transpose(1, 2) # Back to [batch, heads, seq, dim]

Step 3: Enable advanced features

Multi-query attention (shared K/V across heads):

▼
python
复制代码
from flash_attn import flash_attn_func # q: [batch, seq, num_q_heads, dim] # k, v: [batch, seq, num_kv_heads, dim] # Fewer KV heads out = flash_attn_func(q, k, v) # Automatically handles MQA

Sliding window attention (local attention):

▼
python
复制代码
# Only attend to window of 256 tokens before/after out = flash_attn_func( q, k, v, window_size=(256, 256), # (left, right) window causal=True )

Step 4: Benchmark performance

▼
python
复制代码
import torch from flash_attn import flash_attn_func import time q, k, v = [torch.randn(4, 4096, 32, 64, device='cuda', dtype=torch.float16) for _ in range(3)] # Warmup for _ in range(10): _ = flash_attn_func(q, k, v) # Benchmark torch.cuda.synchronize() start = time.time() for _ in range(100): out = flash_attn_func(q, k, v) torch.cuda.synchronize() end = time.time() print(f"Time per iteration: {(end-start)/100*1000:.2f}ms") print(f"Memory allocated: {torch.cuda.max_memory_allocated()/1e9:.2f}GB")

Workflow 3: H100 FP8 optimization (FlashAttention-3)

For maximum performance on H100 GPUs.

▼
text
复制代码
FP8 Setup: - [ ] Step 1: Verify H100 GPU available - [ ] Step 2: Install flash-attn with FP8 support - [ ] Step 3: Convert inputs to FP8 - [ ] Step 4: Run with FP8 attention

Step 1: Verify H100 GPU

▼
bash
复制代码
nvidia-smi --query-gpu=name --format=csv # Should show "H100" or "H800"

Step 2: Install flash-attn with FP8 support

▼
bash
复制代码
pip install flash-attn --no-build-isolation # FP8 support included for H100

Step 3: Convert inputs to FP8

▼
python
复制代码
import torch q = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16) k = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16) v = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16) # Convert to float8_e4m3 (FP8) q_fp8 = q.to(torch.float8_e4m3fn) k_fp8 = k.to(torch.float8_e4m3fn) v_fp8 = v.to(torch.float8_e4m3fn)

Step 4: Run with FP8 attention

▼
python
复制代码
from flash_attn import flash_attn_func # FlashAttention-3 automatically uses FP8 kernels on H100 out = flash_attn_func(q_fp8, k_fp8, v_fp8) # Result: ~1.2 PFLOPS, 1.5-2x faster than FP16

When to use vs alternatives

Use Flash Attention when:

  • Training transformers with sequences >512 tokens
  • Running inference with long context (>2K tokens)
  • GPU memory constrained (OOM with standard attention)
  • Need 2-4x speedup without accuracy loss
  • Using PyTorch 2.2+ or can install flash-attn

Use alternatives instead:

  • Standard attention: Sequences <256 tokens (overhead not worth it)
  • xFormers: Need more attention variants (not just speed)
  • Memory-efficient attention: CPU inference (Flash Attention needs GPU)

Common issues

Issue: ImportError: cannot import flash_attn

Install with no-build-isolation flag:

▼
bash
复制代码
pip install flash-attn --no-build-isolation

Or install CUDA toolkit first:

▼
bash
复制代码
conda install cuda -c nvidia pip install flash-attn --no-build-isolation

Issue: Slower than expected (no speedup)

Flash Attention benefits increase with sequence length:

  • <512 tokens: Minimal speedup (10-20%)
  • 512-2K tokens: 2-3x speedup
  • 2K tokens: 3-4x speedup

Check sequence length is sufficient.

Issue: RuntimeError: CUDA error

Verify GPU supports Flash Attention:

▼
python
复制代码
import torch print(torch.cuda.get_device_capability()) # Should be ≥(7, 5) for Turing+

Flash Attention requires:

  • Ampere (A100, A10): ✅ Full support
  • Turing (T4): ✅ Supported
  • Volta (V100): ❌ Not supported

Issue: Accuracy degradation

Check dtype is float16 or bfloat16 (not float32):

▼
python
复制代码
q = q.to(torch.float16) # Or torch.bfloat16

Flash Attention uses float16/bfloat16 for speed. Float32 not supported.

Advanced topics

Integration with HuggingFace Transformers: See references/transformers-integration.md for enabling Flash Attention in BERT, GPT, Llama models.

Performance benchmarks: See references/benchmarks.md for detailed speed and memory comparisons across GPUs and sequence lengths.

Algorithm details: See references/algorithm.md for tiling strategy, recomputation, and IO complexity analysis.

Advanced features: See references/advanced-features.md for rotary embeddings, ALiBi, paged KV cache, and custom attention masks.

Hardware requirements

  • GPU: NVIDIA Ampere+ (A100, A10, A30) or AMD MI200+
  • VRAM: Same as standard attention (Flash Attention doesn't increase memory)
  • CUDA: 12.0+ (11.8 minimum)
  • PyTorch: 2.2+ for native support

Not supported: V100 (Volta), CPU inference

Resources

  • Paper: "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (NeurIPS 2022)
  • Paper: "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning" (ICLR 2024)
  • Blog: https://tridao.me/blog/2024/flash3/
  • GitHub: https://github.com/Dao-AILab/flash-attention
  • PyTorch docs: https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html
软件开发

快捷安装

复制到终端去安装 Skill

npxskillsadddavila7/claude-code-templates--skill"optimization-flash-attention"
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
🔥热门工具
朱雀AI检测朱雀AI检测
朱雀AI检测助手朱雀AI检测助手
LovartLovart
老鱼简历老鱼简历
WritePass官网WritePass官网
面多多面多多
CursorCursor
星月写作星月写作
朱雀AI检测助手朱雀AI检测助手
零感AI零感AI
Skills 推荐
debug
debug 是一个系统化定位软件缺陷根因的技能,适用于用户报告 'fix'、'broken'、'error' 等问题或测试意外失败时。它通过可验证的假设、最小复现、因果链分析和回归测试,确保在修复前准确定位根本原因。
retrieving-developer-knowledge
retrieving-developer-knowledge 用于搜索、获取和整合 Google 官方开发者文档,覆盖 Google Cloud、AI/Gemini、Android、Chrome、Web、Flutter、Go、Firebase 等平台,适用于查询 gcloud CLI 命令、API 语法、IAM 权限或架构对比等场景。
diff-intake
diff-intake 读取 PR 或分支差异,生成结构化 YAML 变更摘要,用于下游 analytics instrumentation 技能。适用于用户提交 PR 链接、分支对比或原始 diff 后需理解变更内容、评估分析覆盖或启动埋点流程的场景。
implement
实现 Linus Dashboard 的功能开发与集成,基于 TypeScript 和 Python 构建前后端组件,遵循严格的代码规范与架构模式,完成视图、卡片、服务及传感器等模块的创建与验证,确保高质量交付并兼容 Home Assistant 生态。
pypi-server
支持从源码构建 Python 软件包,并搭建本地 PyPI 服务,实现私有包的托管与分发,使团队可通过 pip 直接安装内部开发的模块。