PairwiseDistancesReductions have been introduced as a hierarchy of Cython classes to implement back-ends of some scikit-learn algorithms.

Pieces of work include:

• ENH Introduce PairwiseDistancesReduction and PairwiseDistancesArgKmin (feature branch) #22134
• ENH Introduce PairwiseDistancesRadiusNeighborhood #22320
• FEA Add support for float32 on PairwiseDistancesReduction using Tempita #23865
• Support sparse-dense data-sets pairs:
  • FEA CSR support for all DistanceMetric #23604
  • FEA Fused sparse-dense support for PairwiseDistancesReduction  #23585
• MAINT Adapt scheduling of PairwiseDistancesRadiusNeighborhood #22829
• DOC Fix docstrings and comments for PairwiseDistancesReduction #23978
• Introduce a specialised back-end for KNeighbors*.predict* which would remove the costly sequential portion after the current call to kneighbors
  • PERF Implement PairwiseDistancesReduction backend for KNeighbors.predict #24076
• MAINT Adapt PairwiseDistancesReduction heuristic for strategy="auto" #24043
• FEA Introduce PairwiseDistances #23958

Subsequent work include:

• Create a private asv benchmark suite for the private submodule
• Use a stable sort to support more distance metrics (see this TODO)
• Support "precomputed" distances
• Force the use of PairwiseDistancesReductions for F-contiguous arrays
  • use a scikit-learn configuration entry to accept and convert F-contiguous arrays to C-contiguous arrays
• Document and communicate about the changes of behaviour regarding n_jobs
• Release the GIL before calling parallel_on_{X|Y} instead of releasing it in those methods
• Revisit KMeans implementations using PairwiseDistancesReductions
• Integrate mimalloc to have proper memory allocation for multi-threaded implementations:
  • Changing the default implementation of malloc(1) might come with unexpected changes and might break the whole ecosystem. Also maintaining it might be costly.
• Introduce a specialised back-end for RadiusNeighbors*.predict* which would remove the costly sequential portion after the current call to radius_neighbors
• Improve tests:
  • assert_argkmin_results_quasi_equality to report the original distances (before rounding) for the nighbor indices in the 2 rounded dist groups in the AssertionError message to help understand the nature of the failures found
  • parametrise old tests for public API backed by PairwiseDistancesReductions
  • systematically test public API on combinations of sparse and dense datasets

Note that this needs not be personal work, I would be really glad having others help on this subject, proposing changes and implementations! 🙂