Yep that's the current approach I'm using. One important thing I forgot to mention is that it seems like index.css is being taken into account and even when it goes back to the default theme, an image I had added on index.css still remains on the default theme. However, my changes to variables.css are not taken into account.

My console looks like:
Unchecked runtime.lastError: The message port closed before a response was received.
jlab_core.c1feae0690cd2229a88c.js?v=c1feae0690cd2229a88c:1

Language pack 'en_US' not installed!
fetch @ jlab_core.c1feae0690cd2229a88c.js?v=c1feae0690cd2229a88c:1

JupyterLab extension <my_theme> is activated!
821.a2a81361bc4cb2cbac34.js?v=a2a81361bc4cb2cbac34:1

so it says it is activated and the image through the theme is there however when I go to Settings->Theme it says JupyterLab Light and not my custom theme. My custom theme is an option it's just that it's not selected

Ah ok figured it out. For reference, if anyone is planning to run all this through a docker container:
My environment was set up such that I was creating a virtual env and installing jupyter lab through it. If you are facing the same issues I was then I would highly suggest to go into the docker container using exec and typing jupyter lab path to figure out where you need to copy over the overrides.json file. You are most likely facing a path issue.