鱼皮AI导航
鱼皮AI导航
  • 主页
  • 社区
  • AI工具
  • AI知识库
  • AI提示词
  • MCP
  • Skills
  • AI学习资源
  • AI资讯
  • GitHub开源
登录/注册

debug-cuda-crash

提供 CUDA 程序崩溃前的输入张量快照,支持在非法内存访问、NaN/Inf、显存溢出及数据类型不匹配等错误发生时,捕获形状、设备、dtype 和数值统计等关键信息,辅助快速定位问题根源。

效率工具
访问 GitHub
文档Skills.MD
name
debug-cuda-crash
description
Tutorial for debugging CUDA crashes using API logging

Tutorial: Debugging CUDA Crashes with API Logging

This tutorial shows you how to debug CUDA crashes and errors in FlashInfer using the @flashinfer_api logging decorator.

Goal

When your code crashes with CUDA errors (illegal memory access, out-of-bounds, NaN/Inf), use API logging to:

  • Capture input tensors BEFORE the crash occurs
  • Understand what data caused the problem
  • Track tensor shapes, dtypes, and values through your pipeline
  • Detect numerical issues (NaN, Inf, wrong shapes)

Why Use API Logging?

Problem: CUDA errors often crash the program, leaving no debugging information.

Solution: FlashInfer's @flashinfer_api decorator logs inputs BEFORE execution, so you can see what caused the crash even after the program terminates.

Step 1: Enable API Logging

Basic Logging (Function Names Only)

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=1 # Log function names export FLASHINFER_LOGDEST=stdout # Log to console python my_script.py

Output:

▼
text
复制代码
[2025-12-18 10:30:45] FlashInfer API Call: batch_decode_with_padded_kv_cache

Detailed Logging (Inputs/Outputs with Metadata)

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 # Log inputs/outputs with metadata export FLASHINFER_LOGDEST=debug.log # Save to file python my_script.py

Output in debug.log:

▼
text
复制代码
================================================================================ [2025-12-18 10:30:45] FlashInfer API Logging - System Information ================================================================================ FlashInfer version: 0.6.0 CUDA toolkit version: 12.1 GPU 0: NVIDIA H100 PCIe Compute capability: 9.0 (SM90) PyTorch version: 2.1.0 ================================================================================ ================================================================================ [2025-12-18 10:30:46] FlashInfer API Call: batch_decode_with_padded_kv_cache -------------------------------------------------------------------------------- Positional input arguments: arg[0]: Tensor( shape=(32, 8, 128) dtype=torch.bfloat16 device=cuda:0 requires_grad=False is_contiguous=True ) Keyword input arguments: kv_cache= Tensor( shape=(1024, 2, 8, 128) dtype=torch.bfloat16 device=cuda:0 requires_grad=False is_contiguous=True )

Full Logging (With Tensor Statistics)

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=5 # Log with min/max/mean/nan/inf export FLASHINFER_LOGDEST=debug.log python my_script.py

Additional output:

▼
text
复制代码
Tensor( shape=(32, 8, 128) dtype=torch.bfloat16 device=cuda:0 requires_grad=False is_contiguous=True min=-3.125000 max=4.250000 mean=0.015625 nan_count=0 inf_count=0 )

Step 2: Reproduce the Crash

Example: Shape Mismatch

Your code crashes with:

▼
text
复制代码
RuntimeError: CUDA error: an illegal memory access was encountered

Enable logging and run again:

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 export FLASHINFER_LOGDEST=crash_log.txt python my_script.py

The log shows inputs before the crash:

▼
text
复制代码
[2025-12-18 10:32:15] FlashInfer API Call: batch_decode_with_padded_kv_cache Positional input arguments: arg[0]: Tensor( shape=(32, 8, 128) # Query tensor ... ) Keyword input arguments: kv_cache= Tensor( shape=(1024, 2, 8, 64) # ❌ Wrong! Should be (..., 128) not (..., 64) ... )

Found the bug: head_dim mismatch (64 vs 128)

Step 3: Common CUDA Errors and How to Debug

Error 1: Illegal Memory Access

Error Message:

▼
text
复制代码
RuntimeError: CUDA error: an illegal memory access was encountered

Enable logging:

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 python my_script.py

What to check in logs:

  • ✅ Tensor shapes match expected dimensions
  • ✅ All tensors are on CUDA (not CPU)
  • ✅ Tensor strides are reasonable
  • ✅ is_contiguous=True (if required)

Common causes:

  • Wrong tensor dimensions
  • CPU tensor passed to GPU kernel
  • Incorrect stride patterns

Error 2: NaN or Inf Values

Error Message:

▼
text
复制代码
RuntimeError: Function ... returned nan or inf

Enable statistics logging:

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=5 # Level 5 shows nan_count, inf_count python my_script.py

What to check in logs:

▼
text
复制代码
Tensor( ... min=-1234567.000000 # ❌ Suspiciously large max=9876543.000000 # ❌ Suspiciously large mean=nan # ❌ NaN detected nan_count=128 # ❌ 128 NaN values! inf_count=0 )

Common causes:

  • Division by zero in previous operation
  • Numerical overflow/underflow
  • Uninitialized memory

Error 3: Out of Memory

Error Message:

▼
text
复制代码
RuntimeError: CUDA out of memory

Enable logging:

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 python my_script.py

What to check in logs:

  • ✅ Tensor shapes (are they unexpectedly large?)
  • ✅ Batch size
  • ✅ Sequence length

Example:

▼
text
复制代码
Tensor( shape=(1024, 8192, 128, 128) # ❌ Way too large! Should be (1024, 128, 128)? ... )

Error 4: Wrong Dtype

Error Message:

▼
text
复制代码
RuntimeError: expected scalar type BFloat16 but found Float16

Enable logging:

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 python my_script.py

What to check in logs:

▼
text
复制代码
Tensor( dtype=torch.float16 # ❌ Should be torch.bfloat16 ... )

Step 4: Multi-Process Debugging

When running with multiple GPUs/processes, use %i pattern:

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 export FLASHINFER_LOGDEST=debug_rank_%i.txt # %i = process ID torchrun --nproc_per_node=4 my_script.py

This creates separate logs:

  • debug_rank_12345.txt (process 12345)
  • debug_rank_12346.txt (process 12346)
  • debug_rank_12347.txt (process 12347)
  • debug_rank_12348.txt (process 12348)

Now you can debug each rank independently.

Step 5: Advanced Debugging with compute-sanitizer

For harder bugs, combine API logging with CUDA tools:

Use compute-sanitizer (Memory Checker)

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 export FLASHINFER_LOGDEST=debug.log compute-sanitizer --tool memcheck python my_script.py

Output shows exact memory errors:

▼
text
复制代码
========= COMPUTE-SANITIZER ========= Invalid __global__ write of size 4 bytes ========= at 0x1234 in ScaleKernel<float> ========= by thread (256,0,0) in block (10,0,0) ========= Address 0x7f1234567890 is out of bounds

Check debug.log to see what inputs caused this kernel to fail.

Use cuda-gdb (Debugger)

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 export FLASHINFER_LOGDEST=debug.log cuda-gdb --args python my_script.py

In gdb:

▼
text
复制代码
(cuda-gdb) run (cuda-gdb) where # Show stack trace when it crashes

Check debug.log for the inputs that led to the crash.

Step 6: Kernel-Level Debugging with printf()

You can use printf() inside CUDA kernels for debugging:

Basic Usage

▼
cpp
复制代码
__global__ void MyKernel(const float* input, float* output, int n) { int idx = blockIdx.x * blockDim.x + threadIdx.x; // Print from one thread to avoid spam if (threadIdx.x == 0 && blockIdx.x == 0) { printf("n=%d, input[0]=%f\n", n, input[0]); } if (idx < n) { output[idx] = input[idx] * 2.0f; } }

Important: Flush printf buffer after kernel:

▼
python
复制代码
my_kernel(input, output) torch.cuda.synchronize() # ← Flushes printf output

⚠️ Warp-Specialized Kernels: Choosing the Right Print Thread

Problem: threadIdx.x == 0 doesn't work for all warps (warp starting at thread 32 won't have thread 0).

Solution: Choose one representative thread per specialization group.

▼
cpp
复制代码
__global__ void WarpSpecializedKernel(...) { // Define your group's representative thread // e.g., first thread of each warp: threadIdx.x % 32 == 0 // e.g., first thread of each 4-warp group: threadIdx.x % 128 == 0 if (is_group_representative) { printf("Group %d processing\n", group_id); } }

Common mistake ❌:

▼
cpp
复制代码
// ❌ Only warp 0 will print! if (threadIdx.x == 0) { printf("Warp %d processing\n", threadIdx.x / 32); }

Quick Reference

Kernel TypePrint ConditionNotes
Simple kernelthreadIdx.x == 0One thread per block
Warp-specializedOne thread per groupDepends on kernel design

Other Kernel Debugging Tools

▼
cpp
复制代码
// Assert for invariants assert(value >= 0.0f && "Value must be non-negative"); // Compile-time checks static_assert(BLOCK_SIZE % 32 == 0, "BLOCK_SIZE must be multiple of warp size");

Environment Variables Reference

VariableValuesDescription
FLASHINFER_LOGLEVEL0No logging (default)
1Function names only
3Inputs/outputs with metadata
5+ Tensor statistics (min/max/mean/nan/inf)
FLASHINFER_LOGDESTstdoutLog to console (default)
stderrLog to stderr
<path>Log to file
log_%i.txtMulti-process: %i = process ID

Best Practices

1. Always Start with Level 3

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3

Level 3 provides tensor metadata (shape, dtype, device) without overwhelming output.

2. Use Level 5 for Numerical Issues

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=5

Only use level 5 when debugging NaN/Inf problems (adds statistics).

3. Log to File for Crashes

▼
bash
复制代码
export FLASHINFER_LOGDEST=crash_log.txt

Console output may be lost when program crashes. File logs persist.

4. Compare Before/After

Enable logging and compare:

  • Last successful API call (inputs logged, outputs logged) ✅
  • First failed API call (inputs logged, no outputs) ❌ ← This is where it crashed!

5. Disable Logging in Production

▼
bash
复制代码
unset FLASHINFER_LOGLEVEL # or export FLASHINFER_LOGLEVEL=0

Logging has zero overhead when disabled (decorator returns original function).

Troubleshooting

No Logs Appearing

Problem: Set FLASHINFER_LOGLEVEL=3 but no logs appear

Solutions:

  1. Check if API has the decorator: Not all FlashInfer APIs have @flashinfer_api yet (work in progress)

  2. Verify environment variable:

    ▼
    bash
    复制代码
    echo $FLASHINFER_LOGLEVEL # Should print "3"
  3. Check log destination:

    ▼
    bash
    复制代码
    echo $FLASHINFER_LOGDEST # Should print path or "stdout"

Too Much Output

Problem: Level 5 produces too much output

Solution: Use level 3 instead:

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 # Skip tensor statistics

Statistics Skipped in CUDA Graph

Warning: [statistics skipped: CUDA graph capture in progress]

What it means: Level 5 statistics are automatically skipped during CUDA graph capture (to avoid synchronization)

This is normal: The framework protects you from graph capture issues.

Quick Examples

Debug Shape Mismatch

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 export FLASHINFER_LOGDEST=stdout python my_script.py # Check tensor shapes in output

Debug NaN/Inf

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=5 # Show statistics export FLASHINFER_LOGDEST=debug.log python my_script.py # Check nan_count and inf_count in debug.log

Debug Multi-GPU Training

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 export FLASHINFER_LOGDEST=rank_%i.log # Separate log per rank torchrun --nproc_per_node=8 train.py # Check rank_*.log files

Combine with Memory Checker

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 export FLASHINFER_LOGDEST=inputs.log compute-sanitizer --tool memcheck python my_script.py # inputs.log shows what data caused the memory error

Example: Full Debug Session

Your code crashes:

▼
python
复制代码
import torch from flashinfer import batch_decode_with_padded_kv_cache q = torch.randn(32, 8, 128, dtype=torch.bfloat16, device="cuda") kv = torch.randn(1024, 2, 8, 64, dtype=torch.bfloat16, device="cuda") # Wrong dim! output = batch_decode_with_padded_kv_cache(q, kv) # ❌ Crashes

Enable logging:

▼
bash
复制代码
export FLASHINFER_LOGLEVEL=3 export FLASHINFER_LOGDEST=debug.log python test.py

Check debug.log:

▼
text
复制代码
[2025-12-18 10:45:23] FlashInfer API Call: batch_decode_with_padded_kv_cache Positional input arguments: arg[0]: Tensor( shape=(32, 8, 128) dtype=torch.bfloat16 ... ) arg[1]: Tensor( shape=(1024, 2, 8, 64) # ❌ Found it! Last dim should be 128 dtype=torch.bfloat16 ... )

Fix the bug:

▼
python
复制代码
kv = torch.randn(1024, 2, 8, 128, dtype=torch.bfloat16, device="cuda") # ✅ Fixed

Success!

▼
bash
复制代码
python test.py # No crash, outputs logged successfully

Summary

  1. Enable logging before the crash:

    ▼
    bash
    复制代码
    export FLASHINFER_LOGLEVEL=3 export FLASHINFER_LOGDEST=debug.log
  2. Run your code - inputs are logged BEFORE crash

  3. Check the log - last API call shows what caused the crash

  4. Fix the issue based on logged input metadata

  5. Disable logging when done:

    ▼
    bash
    复制代码
    export FLASHINFER_LOGLEVEL=0

Related Documentation

  • See CLAUDE.md "API Logging with @flashinfer_api" for technical details
  • See flashinfer/api_logging.py for implementation
  • See CUDA documentation for compute-sanitizer and cuda-gdb
效率工具

快捷安装

复制到终端去安装 Skill

npxskillsaddflashinfer-ai/flashinfer--skill"debug-cuda-crash"
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
🔥热门工具
朱雀AI检测朱雀AI检测
朱雀AI检测助手朱雀AI检测助手
LovartLovart
老鱼简历老鱼简历
WritePass官网WritePass官网
面多多面多多
CursorCursor
星月写作星月写作
朱雀AI检测助手朱雀AI检测助手
零感AI零感AI
Skills 推荐
webapp-testing
提供本地 Web 应用的端到端测试能力,支持页面导航、元素交互、表单填写、动态等待、断言验证、截图捕获、浏览器日志收集及网络请求拦截,适用于前端功能验证与 UI 行为调试。
definition-of-done-generator
自动生成项目完成标准的规范定义与配置文件,提供符合企业级工作流要求的最佳实践指导,输出可直接投入生产环境的代码和验证结果,适用于需要标准化流程审批、合规审查及跨系统集成的复杂项目管理场景。
implement
实现 Linus Dashboard 的功能开发与集成,基于 TypeScript 和 Python 构建前后端组件,遵循严格的代码规范与架构模式,完成视图、卡片、服务及传感器等模块的创建与验证,确保高质量交付并兼容 Home Assistant 生态。
cloud-scheduler-job-creator
自动处理 Google Cloud Platform 中的云调度任务创建,提供符合最佳实践的配置方案与生产级代码生成,支持计算、存储、大数据及机器学习服务的集成部署。
planning-with-files
通过创建任务计划、笔记与交付文件三个持久化文档,实现对复杂任务的结构化管理。在多阶段工作中保持目标清晰,动态更新进度并沉淀研究过程,适用于需长期跟踪的规划、研究及创作场景。