Common issues or Feature Requests

Before you raise a question, you can check if your questions can be solved here or read the docs

More issues will be updated.

Q : How to use local dataset?
A : Though might need to edit some code inside the repo, there are multiple ways to achieve this. You can refer to these issues #335 #179 #124 #23 #21 #25

Q : Exact match is not found
A : This is also related to the network issue. You can refer to this issue #75

Q : How to change the default settings of the model?
A : You can change the pass in parameters in the --model_args flag. A simple example would be param1=xxx,param2=xxx.

Q : How to evaluate on big models?
A : For most of the models, you can try setting device_map=auto in the model args. For llava, you can try the srt api model that use sglang for faster speed.

Q : About the link for worldqa
A : The old branch for worldqa has been removed as the dataset has been integrated into the main branch. You can directly evaluate using the main branch.

Q : Task not found error
A : This is likely due to some errors occur when trying to register the tasks. The errors are most likely to occur in the utils.py or other python scripts written for the tasks. You might check if you have the dependencies needed. For video tasks, you are recommended to set HF_HOME explicitly in your envs

New Feature Requests

Q : New Datasets?
A : Sorry but we won't adding new datasets request from the communities unless we actually need to evaluate on then as we don't have time or member to do so. If you actually needs them on the lmms-eval, we welcome the PR and we will review your code as quickly as possible.

Q : New models?
A : This is same as the datasets. You are welcome to raise the PR.