I encountered a similar issue as This one, when running lmms-eval with an offline machine(no Internet). load_dataset method still tries to reach Hugging Face Hub when I set HF_DATASETS_OFFLINE to 1.

I looked into this issue from huggingface Datasets and found it is a bug from datasets library, where load_dataset method still tries to reach Hugging Face Hub after settingHF_DATASETS_OFFLINE to 1.

And the bug is fixed with this PR since Datasets Version 2.19.0. And it has been verified here that updating Datasets to newer version ACTUALLY enables lmms-eval to run without bug in offline environment.

So I suggest to update Datasets Version to >= 2.19.0 so that lmms-eval can be used in fully offline environment. Any future plans for that?

(Although there are currently some workarounds for running lmms-eval in offline environment 179 21, I think them inconvenient when you need to evaluate MANY tasks. And I think supporting lmms-eval in offline environment will help a lot of users.)