Unable to successfully install scikit-learn development version with MacBookPro with chip Apple M1 Pro #22581
Labels
Comments
|
This could be an issue with the latest version of |
|
I tried using I can also confirm that I have the same issue with Here's the -vvv log for the older versions ( |
|
I can reproduce in the following env I create a few days ago. I build scikit-learn from scratch ( In this env, if I run pytest I get the killed python error: When trying to debug this with lldb to get a post-mortem backtrace I got the surprise that running this in lldb got the lldb process itself to get killed: Then I tried again with a new env and this time got an explicit error message instead of a crash: and then I get the error message that tells me to install scikit-learn before trying to import it. If you run: then everything works fine. And it does work fine in both environments! @marenwestermann how did you build and install scikit-learn? Did run |
|
Maybe there was a transient problem in one of the packages that was updated between the 2 envs. |
|
Thank you for your detailed response, @ogrisel! Regarding your questions: I got a new computer recently (the MacBookPro with chip Apple M1 Pro that I'm using now). I backed up the repos that I had on my old computer to an external hard drive and from there copied these over to the new computer. There, in case of scikit-learn, I made sure that origin and upstream are correct, did a I ran The output of the last command is this: My new conda environment looks like this: I haven't tried completely uninstalling scikit-learn and re-installing it. I could give that a try as a next step. (Maybe the copy-pasting of the repo has corrupted some files?) |
|
From your logs, it looks like the compiler being used is not the conda-forge one. Can you try install the compilers first? conda create -n sklearn-env -c conda-forge python=3.9 numpy scipy cython compilers
conda activate sklearn-env
mamba install -c conda-forge joblib threadpoolctl pytest
make clean
pip install --verbose --no-build-isolation --editable .(An alternative is to deactivate and reactivate the environment after installing |
|
Also, you can do a |
|
@thomasjpfan command should work. But using mamba to create the env and install everything should even be faster: |
|
Also, feel free to use |
|
Thank you for your replies! I tried deactivating and reactivating the environment after installing I cloned a new copy in a separate folder (just to be sure), then followed @ogrisel 's instructions and still got the same error trying to do Looking at the build log, I checked if the used compiler is the one from the environment, and it seems it is: And looking at the crash log, it seems the correct Just in case it's relevant, my system info again: |
|
Interestingly, when I create a venv using the python installed on my machine (3.8), I can build and import sklearn and show versions (I have clang installed system-wide): Of course this doesn't help me in the development env since I can't actually install sklearn (due to lack of openmp I think): It probably would be nice to figure this out before the Paris sprint. |
|
From your build logs in #22581 (comment) I see:
On my M1 Max, I get:
In the conda environment, can you try running
|
|
Note that: cannot always be trusted. Better do: or use: |
|
@marenwestermann in output of your build in the Python 3.8 venv you have a warning about openmp but this is just a warning and should not prevent you from building the project (without openmp parallelism). The actual error is: which makes me think that your system Python install is lacking some dependencies. Maybe |
|
Hum I tried again the instructions of #22581 (comment) and now I can reproduce the crash. This is really weird, I am sure it did work (after running the pip install command previously). |
|
I think it might be related to python 3.9 vs 3.10. @marenwestermann can you please try again with: EDIT: actually no, I just tried again and I still get the "killed process". But this is weird because I am doing something that makes it disappear at some point but I am not sure what. |
|
Ok I think I found the cause. It might be the OS that's killing new processes that try to allocate too much memory when the system is already under memory pressure. See https://developer.apple.com/forums/thread/8664 I launched "Activity Monitor" and realized that might system was under pressure. I stopped firefox and then tried to launch the I started firefox again but the memory usage is still fine and my python process now always work. This is very confusing. It might be a bug of macOS. |
|
No this is not it. I tried again, and this time my memory usage is fine I still get the process killed problem on a new env. |
|
Hum, on an env that fails to import sklearn, I tried to use while: |
I've noticed I need to install Here's what's worked so far (on @marenwestermann 's computer):
Then build and install. |
|
re: and building in this env results in the same crash. |
|
I found the culprit, it's the clang-12 compiler toolchain from conda-forge. If you install the clang-11 version, everything is fine. Just add |
|
I will report to conda-forge. |
|
My question is why @thomasjpfan can't reproduce it then. |
|
Here is the bug report: conda-forge/compilers-feedstock#41
Maybe he still had a clang-11 based env activated in his path when he ran the tests. Also sometimes the problem would disappear even in my |
If you build the env with conda-forge as explained in the doc then you don't need to install it explicitly in my experience. |
|
I can confirm that when using |
|
@marenwestermann the upstream bug has been resolved in the conda-forge packages installed via the Let me close this issue. Thanks for the report. |
|
I can also confirm that the issue is fixed. I created a new conda environment with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
marenwestermann commentedFeb 22, 2022
Describe the bug
I installed the development version of scikit-learn on my computer (MacBookPro with chip Apple M1 Pro). When I ran pytest I got the following error:
I investigated what's going on an this is the result:
https://gist.github.com/marenwestermann/9ffddb7a2f0ef6798d350f3595997ed1
I'm using a conda environment. I used the Miniforge3-MacOSX-arm64 download from here: https://github.com/conda-forge/miniforge#miniforge
This is the content of my sklearn-dev conda environment:
https://gist.github.com/marenwestermann/c01fc4d3044522e329ce83d16680e1a5
Steps/Code to Reproduce
Expected Results
The expected result would be that when the command
pytestis run all test are either skipped or are successful.Actual Results
Pytest result:
Further investigation:
https://gist.github.com/marenwestermann/9ffddb7a2f0ef6798d350f3595997ed1
Versions
function>The text was updated successfully, but these errors were encountered: