[Bug][CI Failure]: EAGLE Spec Decode failing with Triton Attention Backend

[zhewenl]

Name of failing test

test_eagle_correctness[TRITON_ATTN-qwen3_eagle3]

Basic information

• Flaky test
• Can reproduce locally
• Caused by external libraries (e.g. bug in transformers)

🧪 Describe the failing test

Summary

EAGLE speculative decoding is failing on AMD MI325X (gfx942) GPUs with a HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION error. The error occurs immediately when processing prompts starts, after successful CUDA graph capturing.
It can be reproduced with spec decode + EAGLE + Triton Attention Backend(default for AMD), example:

python3 offline_inference/spec_decode.py --test --method eagle --num_spec_tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --temp 0 --top-p 1.0 --top-k -1 --tp 1 --enable-chunked-prefill --max-model-len 2048

Error Details

:0:rocdevice.cpp:3675: Callback: Queue 0x7f4db0300000 aborting with error:
HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION: The agent attempted to access memory beyond the largest legal address. code: 0x29

(using a different backend could work: eg. ROCM_AITER_FA, ROCM_AITER_UNIFIED_ATTN, but TritonAttentionBackend is the default attention backend for AMD: gist:fb8bbb2cbde391905d86908ca4a46c02)

What Works

1. Model initialization
2. Weight loading (target model + EAGLE draft model)
3. CUDA graph capturing (both PIECEWISE and FULL modes)
4. KV cache allocation

What Fails

Prompt processing - Fails immediately when execute_model is called, before processing even the first token

Error Call Stack

multiproc_executor.py:694 worker_busy_loop
  → worker_base.py:353 execute_model
    → gpu_worker.py:491 execute_model
      → gpu_model_runner.py:2512 execute_model
        → gpu_model_runner.py:2404 _model_forward
          → self.model() [MEMORY VIOLATION]

Affecting tests: V1 Test e2e + engine, Example Test

📝 History of failing test

https://buildkite.com/vllm/ci/builds/36286#019a1ead-5029-45e3-b576-d1ac8cd5ac43
https://buildkite.com/vllm/amd-ci/builds/632#019a27b6-9f6a-4b99-842d-55eef24ea7cd

CC List.

@mxz297 @yeqcharlotte @Alexei-V-Ivanov-AMD @luccafong @njhill @LucasWilkinson

[github-actions]

CC @hongxiayang @tjtanaa @vllmellm for ROCm-related issue

[vllmellm]

Hi @zhewenl , Thank you for submitting this issue. I've investigated this issue thoroughly and performed reproduction tests on NVIDIA platforms. I can confirm that this is a cross-platform bug, not just limited to AMD ROCm.

Here are my findings:

1. On AMD (ROCm): As you can see, the test crashes with the error message "Memory access fault by GPU node-2 (Agent handle: 0x562b70c57570) on address 0x82e5d3f88000. Reason: Unknown." The address I reproduced locally (0x82e5d3f88000) is different from the one on CI, but as you said, it points to the same type of problem: an illegal memory access occurred at the GPU hardware level.

2. On NVIDIA (CUDA): Using the same test command (python3 examples/offline_inference/spec_decode.py --method eagle ...), an essentially identical error is reported on NVIDIA GPUs: RuntimeError: Triton Error [CUDA]: an illegal memory access was encountered.

The crashes on both platforms are different manifestations of the same underlying bug.

This bug involves the Triton backend + EAGLE speculative decoding, and its root cause lies in v1/core/kv_cache_manager.py and its related call chain (KVCacheCoordinator, FullAttentionManager, etc.). Given that this is a systemic issue deeply embedded in the core allocation logic of the v1 backend (rather than a simple Python fix), we may need to temporarily shelve it. We will use this issue to track this cross-platform bug for more in-depth underlying debugging later.

[zhewenl]

@vllmellm Thanks for your comment! I think it's reasonable to use this issue to track it since it's a broader problem(and I have updated the title accordingly).

From testing/CI's perspective, do you think if it makes sense to change attention backend to AITER FA in AMD CI to test spec decode? The existing tests are failing because AMD is using Triton backend as default(Nvidia is using FA so it doesn't have the issue in CI).

[vllmellm]

Hi @zhewenl , This title is a very apt description of a real bug. actually you just said it that's a good method. TRITON_ATTN+EAGLE mode does have a problem, so let's try a different approach first.
so when you testing/CI's about EAGLE Spec Decode on AMD cards, you should set export VLLM_ROCM_USE_AITER=1 on terminal, then run your command: python3 examples/offline_inference/spec_decode.py --method eagle ...) and will run successful, because that command point and output : INFO 10-31 06:09:20 [rocm.py:286] Using Aiter Flash Attention backend on V1 engine. the final result like this:

--------------------------------------------------
total_num_output_tokens: 17006
num_drafts: 7495
num_draft_tokens: 22485
num_accepted_tokens: 9524
mean acceptance length: 2.27
--------------------------------------------------
acceptance at token 0: 0.68
acceptance at token 1: 0.39
acceptance at token 2: 0.21
Test passed! Expected AL: 2.296, got 2.2707138092061374