@nkoep I implemented the numpy backend for all the functions you added to the TODO list.

@nkoep, I changed quite a lot of things, but we are getting close to having the first version of the multi-backend support for manifolds operations!

For now, I focused on having this working for numpy and torch. This means you can still use the 4 backends (numpy, torch, jax and tensorflow) to define your loss function but for jax and tensorflow, operations on the manifold are still performed with numpy. For Pytorch, all operations are performed with Pytorch and there are no more conversions to numpy arrays. The list of available backends for numericals is available at nx.NUMERICS_SUPPORTED_BACKENDS.

To get an overview of this PR, you can begin with examples/dominant_eigenvector.py. It is nearly identical as before expect the point returned by the optimiser is now a torch.tensor when using PyTorch backend.

The manifolds are backend agnostic, i.e. they instantiated without specifying the backend of input data. The only case when the manifold must know the backend is when sampling a point with the random_point method. In this case, the user need to set manifold.backend if he wants something else than a numpy array. Indeed, by default, a given manifold samples points with numpy.

When running optimizer.run(problem), the backend is inferred from the decorator of the loss function @pymanopt.function.XXX(manifold).

All manifolds are still tested with numpy. Sphere is tested with all nx.NUMERICS_SUPPORTED_BACKENDS, i.e numpy and PyTorch thanks to TestBackendManifoldFactory. The only change in Sphere is the random_tangent_vector where an nx.array_as is added to change the backend of the generated vector. random_point is not changed since it is decorated by Manifold, which automatically changes the backend of the generated data.

When a manifold stores data like in _SphereSubspaceIntersectionManifold, then data are now stored in self.parameters. The latter automatically changes the backend of what is stored inside when manifold.backend is changed.

Some modules had to be adapted such as tools/multi.py and tools/testing.py. tools/multi.py is fully tested with all nx.NUMERICS_SUPPORTED_BACKENDS.

Overall, most of the remaining work concerns the tests of all manifolds except Sphere.

@nkoep, tell me what you think about all of this!