Hi, team,

I’ve been running lmms_eval on the ChartQA benchmark using the Qwen/Qwen2-VL-2B-Instruct model under two different settings, and I’m seeing significant discrepancies in performance, despite using the same model.

1. Standard Python version

python3 -m lmms_eval \
    --model=qwen2_vl \
    --model_args=pretrained=Qwen/Qwen2-VL-2B-Instruct,device_map=cuda \
    --tasks=chartqa \
    --batch_size=1 \
    --log_samples \
    --log_samples_suffix=qwen2_vl \
    --output_path="./logs1"

Results:

2. vLLM version

python3 -m lmms_eval \
    --model vllm \
    --model_args model_version=Qwen/Qwen2-VL-2B-Instruct,tensor_parallel_size=4 \
    --tasks chartqa \
    --batch_size 300 \
    --log_samples \
    --log_samples_suffix vllm \
    --output_path ./logs2

Results:

Issue:

Even though both configurations use Qwen/Qwen2-VL-2B-Instruct, I got lower performance results when I used the vLLM version.

Is there any known limitation or additional configuration needed when using the vllm backend?