[Bug]: WSL2(also Docker) 1 GPU work but 2 not,(--tensor-parallel-size 2 )

[goodmaney]

Your current environment

PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.3
Libc version: glibc-2.35

Python version: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.5.119
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 4090
GPU 1: NVIDIA GeForce RTX 4090

Nvidia driver version: 552.22
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i9-14900KF
CPU family: 6
Model: 183
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 1
BogoMIPS: 6374.40
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities
Virtualization: VT-x
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 768 KiB (16 instances)
L1i cache: 512 KiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 36 MiB (1 instance)
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] torchaudio==2.3.0+cu121
[pip3] torchvision==0.18.0+cu121
[pip3] triton==2.3.0
[pip3] vllm_nccl_cu12==2.18.1.0.4.0
[conda] magma-cuda124 2.6.1 1 pytorch
[conda] mkl-include 2024.1.0 intel_691 intel
[conda] mkl-static 2024.1.0 intel_691 intel
[conda] numpy 1.26.4 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi
[conda] torch 2.3.0 pypi_0 pypi
[conda] torchaudio 2.3.0+cu121 pypi_0 pypi
[conda] torchvision 0.18.0+cu121 pypi_0 pypi
[conda] triton 2.3.0 pypi_0 pypi
[conda] vllm-nccl-cu12 2.18.1.0.4.0 pypi_0 pypiROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.4.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS N/A
GPU1 SYS X N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

python -m vllm.entrypoints.openai.api_server
--model ~/.cache/modelscope/hub/Yi-34B-Chat-4bits/
--served-model-name Yi-34B-Chat
--trust-remote-code
--max-model-len 4096 -q awq
--enforce-eager
--tensor-parallel-size 2

#########################################################################################
WARNING 06-01 01:06:19 config.py:205] awq quantization is not fully optimized yet. The speed can be slower than non-quantized models.
2024-06-01 01:06:20,504 INFO worker.py:1749 -- Started a local Ray instance.
INFO 06-01 01:06:21 llm_engine.py:100] Initializing an LLM engine (v0.4.2) with config: model='/home/xx/.cache/modelscope/hub/Yi-34B-Chat-4bits/', speculative_config=None, tokenizer='/home/xx/.cache/modelscope/hub/Yi-34B-Chat-4bits/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=awq, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=Yi-34B-Chat)
INFO 06-01 01:06:23 utils.py:660] Found nccl from library /home/xx/.config/vllm/nccl/cu12/libnccl.so.2.18.1
(RayWorkerWrapper pid=7596) INFO 06-01 01:06:23 utils.py:660] Found nccl from library /home/xx/.config/vllm/nccl/cu12/libnccl.so.2.18.1
WARNING 06-01 01:06:23 utils.py:465] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
(RayWorkerWrapper pid=7596) WARNING 06-01 01:06:23 utils.py:465] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
INFO 06-01 01:06:23 selector.py:27] Using FlashAttention-2 backend.
(RayWorkerWrapper pid=7596) INFO 06-01 01:06:23 selector.py:27] Using FlashAttention-2 backend.
INFO 06-01 01:06:23 pynccl_utils.py:43] vLLM is using nccl==2.18.1
(RayWorkerWrapper pid=7596) INFO 06-01 01:06:23 pynccl_utils.py:43] vLLM is using nccl==2.18.1
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] Error executing method init_device. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] Traceback (most recent call last):
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] File "/home/xx/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/worker/worker_base.py", line 137, in execute_method
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] return executor(*args, **kwargs)
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] ^^^^^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] File "/home/xx/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/worker/worker.py", line 111, in init_device
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] init_worker_distributed_environment(self.parallel_config, self.rank,
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] File "/home/xx/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/worker/worker.py", line 305, in init_worker_distributed_environment
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] pynccl_utils.init_process_group(
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] File "/home/xx/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/distributed/device_communicators/pynccl_utils.py", line 44, in init_process_group
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] comm = NCCLCommunicator(group=group)
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] File "/home/xx/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/distributed/device_communicators/pynccl.py", line 264, in init
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] NCCL_CHECK(
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] File "/home/xx/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/distributed/device_communicators/pynccl.py", line 73, in NCCL_CHECK
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] raise RuntimeError(f"NCCL error: {error_str}")
(RayWorkerWrapper pid=7596) ERROR 06-01 01:06:23 worker_base.py:145] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(RayWorkerWrapper pid=7596) *** SIGSEGV received at time=1717175184 on cpu 31 ***
(RayWorkerWrapper pid=7596) PC: @ 0x7f6feb07e905 (unknown) ncclProxyService()
(RayWorkerWrapper pid=7596) @ 0x7f78f39a4520 (unknown) (unknown)
(RayWorkerWrapper pid=7596) [2024-06-01 01:06:24,352 E 7596 7712] logging.cc:365: *** SIGSEGV received at time=1717175184 on cpu 31 ***
(RayWorkerWrapper pid=7596) [2024-06-01 01:06:24,352 E 7596 7712] logging.cc:365: PC: @ 0x7f6feb07e905 (unknown) ncclProxyService()
(RayWorkerWrapper pid=7596) [2024-06-01 01:06:24,352 E 7596 7712] logging.cc:365: @ 0x7f78f39a4520 (unknown) (unknown)
(RayWorkerWrapper pid=7596) Fatal Python error: Segmentation fault
(RayWorkerWrapper pid=7596)
(RayWorkerWrapper pid=7596)
(RayWorkerWrapper pid=7596) Extension modules: psutil._psutil_linux, psutil._psutil_posix, msgpack._cmsgpack, google._upb._message, setproctitle, yaml._yaml, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, uvloop.loop, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, sentencepiece._sentencepiece, PIL._imaging (total: 33)
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff6aba5c566d171c4cbbfe879201000000 Worker ID: a0821e052de9a053e99bf41cf5c73372eacb37375c5d5b113d38a490 Node ID: 3263fc6978985245b12fc99c23d1dca8db72ce8aa8be6c96f1724a35 Worker IP address: 172.21.21.248 Worker port: 37315 Worker PID: 7596 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
####################################################################################
Then it hanging

Docker version get the same problem
docker run --gpus all
-v ~/.cache/modelscope/hub/Yi-34B-Chat-4bits:/models/Yi-34B-Chat-4bits
-p 8000:8000
--ipc=host
vllm/vllm-openai:latest
--model /models/Yi-34B-Chat-4bits
--served-model-name Yi-34B-Chat
--trust-remote-code
--max-model-len 4096 -q awq
--enforce-eager
--tensor-parallel-size 2

2 GPU worked on other inference tool, and Lora SFT.

[wizd]

try add --privileged to docker

[goodmaney]

try add --privileged to docker

not work.
(RayWorkerWrapper pid=2077) Fatal Python error: Segmentation fault

[goodmaney]

The error change after I update vllm to 0.5.0,and the process hanging.

LLM:4726:4726 [1] NCCL INFO Connected all rings
LLM:2660:2660 [0] NCCL INFO Connected all rings
LLM:2660:2660 [0] NCCL INFO Connected all trees
LLM:4726:4726 [1] NCCL INFO Connected all trees
LLM:4726:4726 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
LLM:2660:2660 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
LLM:2660:2660 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
LLM:4726:4726 [1] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
LLM:2660:2660 [0] NCCL INFO comm 0xc6354a0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 1000 commId 0x40c3d6bafe85590b - Init COMPLETE
LLM:4726:4726 [1] NCCL INFO comm 0xc451140 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 4000 commId 0x40c3d6bafe85590b - Init COMPLETE
Traceback (most recent call last):
File "/home/xx/anaconda3/envs/vllm/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/psm_fb5220fe'
INFO 06-17 20:55:24 custom_all_reduce_utils.py:169] generating GPU P2P access cache in /home/xx/.config/vllm/gpu_p2p_access_cache_for_0,1.json
Error in sys.excepthook:
object address : 0x7f713c4cbd00
object refcount : 1
object type : 0x877520
object type name: ValueError
object repr : ValueError('I/O operation on closed file.')
lost sys.stderr

Original exception was:
object address : 0x7f713c4cbb80
object refcount : 3
object type : 0x877520
object type name: ValueError
object repr : ValueError('I/O operation on closed file.')
lost sys.stderr
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

The gpu_p2p_access_cache_for_0,1.json dosent exist

[goodmaney]

after I terminating the process . It sitll use my cuda.

[youkaichao]

to debug hangs, please follow https://docs.vllm.ai/en/latest/getting_started/debugging.html .

[goodmaney]

to debug hangs, please follow https://docs.vllm.ai/en/latest/getting_started/debugging.html .

Thanks for reply. Where is the path of log generation. I find 2 log files, I don't know if it's all log files. I change the environment to the 'Install with pip' in vllm document's by the way.

VLLM_TRACE_FUNCTION_for_process_2488_thread_140479325135936_at_2024-06-19_224822.654027.log

VLLM_TRACE_FUNCTION_for_process_423_thread_140297769940032_at_2024-06-19_224820.393076.log

[youkaichao]

the log seems to indicate it is stuck here:

2024-06-19 22:48:23.193433 Call to barrier in /home/cc/anaconda3/envs/vllm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py:435 from gpu_p2p_access_check in /home/cc/anaconda3/envs/vllm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py:178

although not sure why, I think you can try #5669 , which fixes some issues for custom_all_reduce_utils.py .

[goodmaney]

the log seems to indicate it is stuck here:

2024-06-19 22:48:23.193433 Call to barrier in /home/cc/anaconda3/envs/vllm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py:435 from gpu_p2p_access_check in /home/cc/anaconda3/envs/vllm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py:178

although not sure why, I think you can try #5669 , which fixes some issues for custom_all_reduce_utils.py .

I download the custom_all_reduce_utils.py (288c99a#diff-e4918ab3e2126c1178b0d8fb0f8ab0175e209daa0578501adf3a7017e56f5e4d) and the envs.py(https://github.com/vllm-project/vllm/pull/5669/files/2e2731637f1b25d7ca3086b7951a7e671e023f71#diff-b7538eee1327212c7cfbe3f7874600dfcd41b5fb5662695cd5f396294610ddf9). vllm not reports error but still stuck .

VLLM_TRACE_FUNCTION_for_process_51496_thread_140453947786304_at_2024-06-20_17_26_12.687434.log
VLLM_TRACE_FUNCTION_for_process_53579_thread_140453947786304_at_2024-06-20_17_26_12.687583.log

[ciaoyizhen]

May I ask if you have resolved this issue?
We are in a very, very similar situation
We are ok locally though, but the docker situation is consistent

[goodmaney]

May I ask if you have resolved this issue? We are in a very, very similar situation We are ok locally though, but the docker situation is consistent

no , do you try the last version? I haven't update yet