Description

When running the command python -m lmms_eval --tasks list_with_num --verbosity=DEBUG, I encounter a NameError because the get_task_dict function is not properly imported or defined in the scope where it's being called.

Error Details

Command:

python -m lmms_eval --tasks list_with_num --verbosity=DEBUG

Error Output:

  File "/.../lmms-eval/lmms_eval/__main__.py", line 435, in cli_evaluate_single
    task_dict = get_task_dict([task_name], model_name="llava")
                ^^^^^^^^^^^^^
NameError: name 'get_task_dict' is not defined. Did you mean: 'task_dict'?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
...
Task :   fail to load
Exception :
name 'get_task_dict' is not defined

Root Cause Analysis

1. Missing Import: The get_task_dict function is not imported in lmms_eval/__main__.py
2. Function Signature Mismatch: The function is called with model_name="llava" parameter, but the actual function definition in lmms_eval/tasks/__init__.py has a different signature:

Current call in __main__.py:

task_dict = get_task_dict([task_name], model_name="llava")

Actual function definition in lmms_eval/tasks/__init__.py:

def get_task_dict(
    task_name_list: Union[str, List[Union[str, Dict, Task]]],
    task_manager: Optional[TaskManager] = None,
):

Environment

• Installation: pip install -e .
• Commit: c4a3128