The Wayback Machine - https://web.archive.org/web/20240106173916/https://github.com/matplotlib/matplotlib/issues/25811
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: pypy3.9 wheel has hidden dependency on numpy>=1.23.0 #25811

Open
jvesely opened this issue May 4, 2023 · 11 comments
Open

[Bug]: pypy3.9 wheel has hidden dependency on numpy>=1.23.0 #25811

jvesely opened this issue May 4, 2023 · 11 comments
Labels
Build

Comments

@jvesely
Copy link

jvesely commented May 4, 2023 •
edited

Bug summary

The pypy3.9 wheel indicates dependency on numpy >=1.20.0, but in fact needs numpy >=1.23.0

Code for reproduction

$ ~/pypy39/pypy/goal/pypy3.9-c -m venv test-venv
$ source ~/test-venv/bin/activate
(test-venv) $ pip install -U pip wheel setuptools
(test-venv) $ pip install numpy==1.22.4
Collecting numpy==1.22.4
  Using cached numpy-1.22.4.zip (11.5 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: numpy
  Building wheel for numpy (pyproject.toml) ... done
  Created wheel for numpy: filename=numpy-1.22.4-pp39-pypy39_pp73-linux_x86_64.whl size=5838730 sha256=39acecdee26139b21ea2273993304465c9d67d88ec8b9900ce12630d224a1174
  Stored in directory: /home/$USER/.cache/pip/wheels/59/3a/25/8abbffc79457432dd0c73e1616ce79e808d5c92677524655e7
Successfully built numpy
Installing collected packages: numpy
Successfully installed numpy-1.22.4
(test-venv) $ pip install matplotlib
Collecting matplotlib
  Using cached matplotlib-3.7.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.5 MB)
Collecting contourpy>=1.0.1 (from matplotlib)
  Downloading contourpy-1.0.7-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (252 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 252.2/252.2 kB 6.2 MB/s eta 0:00:00
Collecting cycler>=0.10 (from matplotlib)
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0 (from matplotlib)
  Using cached fonttools-4.39.3-py3-none-any.whl (1.0 MB)
Collecting kiwisolver>=1.0.1 (from matplotlib)
  Downloading kiwisolver-1.4.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.4/79.4 kB 7.5 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.20 in ./test-venv/lib/pypy3.9/site-packages (from matplotlib) (1.22.4)
Collecting packaging>=20.0 (from matplotlib)
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting pillow>=6.2.0 (from matplotlib)
  Using cached Pillow-9.5.0-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl (3.3 MB)
Collecting pyparsing>=2.3.1 (from matplotlib)
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting python-dateutil>=2.7 (from matplotlib)
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting importlib-resources>=3.2.0 (from matplotlib)
  Using cached importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Collecting zipp>=3.1.0 (from importlib-resources>=3.2.0->matplotlib)
  Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting six>=1.5 (from python-dateutil>=2.7->matplotlib)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: zipp, six, pyparsing, pillow, packaging, kiwisolver, fonttools, cycler, contourpy, python-dateutil, importlib-resources, matplotlib
Successfully installed contourpy-1.0.7 cycler-0.11.0 fonttools-4.39.3 importlib-resources-5.12.0 kiwisolver-1.4.4 matplotlib-3.7.1 packaging-23.1 pillow-9.5.0 pyparsing-3.0.9 python-dateutil-2.8.2 six-1.16.0 zipp-3.15.0
(test-venv) $ python -c 'from matplotlib._path import affine_transform'

Actual outcome

(test-venv) $ python -c 'from matplotlib._path import affine_transform'
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf . Check the section C-API incompatibility at the Troubleshooting ImportError section at https://numpy.org/devdocs/user/troubleshooting-importerror.html#c-api-incompatibility for indications on how to solve this problem .
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/$USER/test-venv/lib/pypy3.9/site-packages/matplotlib/__init__.py", line 131, in <module>
    from . import _api, _version, cbook, _docstring, rcsetup
  File "/home/$USER/test-venv/lib/pypy3.9/site-packages/matplotlib/rcsetup.py", line 27, in <module>
    from matplotlib.colors import Colormap, is_color_like
  File "/home/$USER/test-venv/lib/pypy3.9/site-packages/matplotlib/colors.py", line 56, in <module>
    from matplotlib import _api, _cm, cbook, scale
  File "/home/$USER/test-venv/lib/pypy3.9/site-packages/matplotlib/scale.py", line 22, in <module>
    from matplotlib.ticker import (
  File "/home/$USER/test-venv/lib/pypy3.9/site-packages/matplotlib/ticker.py", line 138, in <module>
    from matplotlib import transforms as mtransforms
  File "/home/$USER/test-venv/lib/pypy3.9/site-packages/matplotlib/transforms.py", line 49, in <module>
    from matplotlib._path import (
ImportError: numpy.core.multiarray failed to import

Expected outcome

$ python -c 'from matplotlib._path import affine_transform'

Additional information

$ pip install matplotlib --no-build-isolation --no-binary matplotlib

works around the problem

Operating system

No response

Matplotlib Version

3.7.1

Matplotlib Backend

No response

Python version

No response

Jupyter version

No response

Installation

pip
@jvesely
Copy link
Author

jvesely commented May 4, 2023

pypy3.8 wheel doesn't have this problem

@tacaswell
Copy link
Member

tacaswell commented May 4, 2023 •
edited

We use cibuildwheel and oldest-supported-numpy to build the wheels

matplotlib/.github/workflows/cibuildwheel.yml

Lines 124 to 133 in b3bd929

- name: Build wheels for PyPy
uses: pypa/[email protected]
env:
CIBW_BUILD: "pp38-* pp39-*"
CIBW_SKIP: "*-musllinux*"
CIBW_BEFORE_BUILD: >-
pip install certifi oldest-supported-numpy &&
git clean -fxd build
CIBW_ARCHS: ${{ matrix.cibw_archs }}
if: matrix.cibw_archs != 'aarch64'

In turn it looks like there are not pinnings for numpy in oldest-supported-numpy (see https://github.com/scipy/oldest-supported-numpy/blob/b0b3376f6838abd91a9988b936f70939d4acb775/setup.cfg) so we ended up with an un-constrained numpy used in the build. There do not appear to be any wheels for numpy and pypy39 which is probably why it is not pinned.

I think this is another example of the problems inherent in wheels (wheels do not have enough metadata to encode ABI constraints only known at build time) and the way that wheels are built (every project builds and publishes on their own). See https://pypackaging-native.github.io for a lot more details (and other problems!).

I'm inclined to say the course of action here is to pull our wheels for pypy39 and stop building them until numpy first publishes wheels.

@tacaswell
Copy link
Member

I put this on the agenda for today's developer call.

@rgommers
Copy link

rgommers commented May 4, 2023

I'm inclined to say the course of action here is to pull our wheels for pypy39 and stop building them until numpy first publishes wheels.

That sounds about right to me. I suspect numpy 1.25.0 will have a wheel for PyPy 3.9 - @mattip is this on your radar?

@tacaswell
Copy link
Member

pypi does not let you yank by file, only whole releases.

We have questionable pypy39 wheels up for 3.6.0, 3.6.1, 3.6.2, 6.6.3, 3.7.0, and 3.7.1 .

@QuLogic
Copy link
Member

QuLogic commented May 4, 2023

FWIW, NumPy has been able to build PyPy3.9 for some time, since we've had to build it in the process of building our wheels. I can't say whether the tests would work though.

@ksunden
Copy link
Member

ksunden commented May 4, 2023

Can we pin and make .post0 releases built with older numpy?

@mattip
Copy link
Contributor

mattip commented May 4, 2023

I suspect numpy 1.25.0 will have a wheel for PyPy 3.9 - @mattip is this on your radar?

Yes. PyPy3.9 is tested in CI, I will submit a PR to create wheels on main.

I see matplotlib has a 32-bit wheel for linux. Does that pin to a version of NumPy that still had 32-bit linux wheels (I think the last one was 1.21)? So the proposal to create .post0 wheels would need to create a 1.21 one? That might be a big ask. We might be able to create .post0 wheels for 1.24, if the release manager agrees.

Going forward, I would suggest you not create 32-bit wheels for PyPy.

@tacaswell
Copy link
Member

tacaswell commented May 4, 2023 •
edited

On main we no longer build 32 bit wheels and will not publish them in the future.

@ksunden
Copy link
Member

ksunden commented May 4, 2023

My thought on .post0 releases is actually independent of what numpy does (numpy could provide pypy39 wheels retroactively if you like, but matplotlib has wheels that claim to be compatible with older np that aren't.) I don't think that numpy would even need to mark them as post0 if they never were uploaded in the first place.

So on our end we could spin up envs for each mpl version where we can have a built-from-sdist numpy of the version we want, create a new (.post0) release for each pypy39 wheel, and add that to pypi so that new installs will prefer the back-compatible wheel. That would avoid yanking the releases entirely while providing preferred wheels to those who want them.

Perhaps the preferred wheel filename change would actually be build tag rather than .post0 in the version, the latter is really only meant for non-code/runtime changes like docs.

https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-format

@mattip
Copy link
Contributor

mattip commented May 5, 2023

I suspect numpy 1.25.0 will have a wheel for PyPy 3.9 - @mattip is this on your radar?

Yes. PyPy3.9 is tested in CI, I will submit a PR to create wheels on main.

PyPy3.9 wheels already are uploaded to https://anaconda.org/scipy-wheels-nightly/numpy/files and will be part of the next release.

@jvesely jvesely changed the title [Bug]: pypy3.9 wheel is hidden dependency on numpy>=1.23.0 [Bug]: pypy3.9 wheel has hidden dependency on numpy>=1.23.0 Jun 19, 2023
@radarhere radarhere mentioned this issue Jul 3, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Build
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@rgommers @tacaswell @QuLogic @mattip @ksunden @melissawm @jvesely