I'm thinking it should be possible to use the VLM as the policy and evaluation, just with different prompts.
I'm trying to use Qwen2.5-VL-3B-Instruct as basis to create an agent to play gameboy games.

So far the best closest thing I can think that would be simply accomplishable would be to use the hidden layers of the pretrained models and train heads on top for the actor and critics.

The biggest problem I am having though is that evaluating the PyBoy observation in the step function, I think, should be using the same model being trained, or at the least the base model with the new critic head on it.
Evaluating with the base (instruct) model isn't very difficult as it can be defined inside the environment and be static.
However since the environment doesn't know about the net until training the evaluator seemingly can't be dynamically modified to match the trained model.