Closed
Description
Describe your issue.
A couple of weeks back we have noticed that some of our workloads have been hanging.
I attached the code that reproduces the bug
Works with any consumer grade Intel CPUs that I tried. It only has problems on servers.
Works with any AMD machine that I tried (consumer or on servers)
Works with Intel Xeon Haswell CPUs E5-2676 v3 (Amazon AWS t2.micro instance or Google GCP N1 instance)
Does not work under any circumstances with Cascade Lake CPUs, for instance Xeon(R) Platinum 8252C CPU (Amazon AWS m5zn.large or Google CGP N2 / C2 with Cascade Lake)
Works (somewhat) with Intel Xeon Ice Lake Cpus, for instance Xeon(R) Platinum 8375C (Amazon AWS
m6i.large or Google GCP N2 with Ice Lake)
On Ice Lake it only works if you specify the numpy dependency as numpy==1.20.3 , if you let scipy download 1.22.2 then it will hang
On Cascade Lake it hangs regardless of the versions and on Haswell it works regardless of the versions.
Here's a sample docker file if you want deploy it as a docker
Reproduces with every version of scipy since 1.4.0
Sample Dockerfile (our original dockerfile does actually compile python from source but I used this as an example to keep it very simple, it reproduces this way too)
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y python3-pip
RUN mkdir /home/app
COPY main.py /home/app/
COPY requirements.txt /home/app/
RUN pip3 install -r /home/app/requirements.txt
CMD python3 /home/app/main.py
To reproduce on Ice Lake/Cascade Lake cpus
requirements.txt (pretty much any scipy version since 1.4.0)
scipy==1.7.0
To reproduce working on Ice Lake but hanging on Cascade Lake
numpy==1.20.3
scipy==1.7.0
I included the Dockerfile for convenience, the issue reproduces if i execute the python process in the VM without docker.
Reproducing Code Example
main.py
import numpy
import scipy.optimize
csx = [[0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0],
[0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0],
[0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0],
[0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0], [0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0, 0], [0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[-1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 1.0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 1.0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 1.0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 1.0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 1.0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0],
[0, 0, 1.0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0], [0, 0, 1.0, 0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0],
[0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0], [0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0, 0],
[0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0], [0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0],
[0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 1.0, 0], [0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 1.0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, -1.0, 0], [0, 0, 0, 1.0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0], [0, 1.0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 1.0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 1.0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0],
[0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0, 0], [0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0],
[0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0], [1.0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[1.0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0], [1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0],
[1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0], [0, 0, 0, -1.0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, -1.0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1.0, -1.0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, -1.0, 0, 0], [0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, -1.0, 0],
[0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, -1.0], [0, 0, 0, -1.0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, -1.0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, -1.0, 0, 1.0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, -1.0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, -1.0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, -1.0], [0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 1.0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1.0, 1.0], [0, 0, 0, 0, 1.0, 0, -1.0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0, 0, -1.0, 0], [0, 0, 0, -1.0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0],
[0, 0, 0, 0, -1.0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, -1.0, 0, 0, 1.0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, -1.0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, -1.0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, -1.0], [0, 0, 0, -1.0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, -1.0, 0, 0, 1.0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, -1.0, 1.0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, -1.0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, -1.0, 0],
[0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0, 0, 0, -1.0], [0, 0, 0, -1.0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0],
[0, 0, 0, 0, -1.0, 0, 0, 0, 0, 0, 1.0, 0, 0, 0], [0, 0, 0, 0, 0, 0, -1.0, 0, 0, 0, 1.0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, -1.0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, -1.0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0, 0, -1.0], [0, 0, -0.0625, 0.18181818, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, -0.0625, 0, 0.18181818, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0.08333333, -0.0625, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, -0.0625, 0, 0, 0.09090909, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, -0.0625, 0, 0, 0, 1.33333333, 0, 0, 0, 0, 0, 0, 0],
[0, 0, -0.0625, 0, 0, 0, 0, 0.09090909, 0, 0, 0, 0, 0, 0],
[0, 0, -0.0625, 0, 0, 0, 0, 0, 0.09090909, 0, 0, 0, 0, 0],
[0, 0, -0.0625, 0, 0, 0, 0, 0, 0, 0.09090909, 0, 0, 0, 0],
[0, 0, -0.0625, 0, 0, 0, 0, 0, 0, 0, 0.08333333, 0, 0, 0],
[0, 0, -0.0625, 0, 0, 0, 0, 0, 0, 0, 0, 0.18181818, 0, 0], [0, 0, -0.0625, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0],
[0, 0, -0.0625, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.18181818],
[0, 0, 0, 0, 0, 0, 1.33333333, 0, 0, 0, 0, 0, -1.0, 0],
[0, 0, 0, 0, 0, 0, 1.33333333, 0, 0, 0, 0, -0.18181818, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -0.18181818, 1.0, 0],
[0, 0, 0, -0.18181818, 0, 0, 1.33333333, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, -0.18181818, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0],
[0, -0.08333333, 0, 0.18181818, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, -0.08333333, 0, 0, 0, 0, 1.33333333, 0, 0, 0, 0, 0, 0, 0],
[0, -0.08333333, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.18181818, 0, 0],
[0, -0.08333333, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0],
[-0.07142857, 0, 0, 0.18181818, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[-0.07142857, 0, 0, 0, 0.18181818, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[-0.07142857, 0, 0, 0, 0, 0.09090909, 0, 0, 0, 0, 0, 0, 0, 0],
[-0.07142857, 0, 0, 0, 0, 0, 1.33333333, 0, 0, 0, 0, 0, 0, 0],
[-0.07142857, 0, 0, 0, 0, 0, 0, 0.09090909, 0, 0, 0, 0, 0, 0],
[-0.07142857, 0, 0, 0, 0, 0, 0, 0, 0.09090909, 0, 0, 0, 0, 0],
[-0.07142857, 0, 0, 0, 0, 0, 0, 0, 0, 0.09090909, 0, 0, 0, 0],
[-0.07142857, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.18181818, 0, 0],
[-0.07142857, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0],
[-0.07142857, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.18181818],
[0, 0, 0, 0.18181818, 0, -0.09090909, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0.18181818, -0.09090909, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, -0.09090909, 1.33333333, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0, 0, 0.18181818, 0, 0],
[0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0, 0, 0, 1.0, 0],
[0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0, 0, 0, 0, 0.18181818],
[0, 0, 0, 0.18181818, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0.18181818, 0, 0, 0, -0.09090909, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 1.33333333, 0, -0.09090909, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, -0.09090909, 0, 0, 0.18181818, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 1.0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0, 0.18181818],
[0, 0, 0, 0, 0, 0, 1.33333333, 0, 0, 0, 0, 0, 0, -0.18181818],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, -0.18181818],
[0, 0, 0, 0, -0.18181818, 0, 1.33333333, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, -0.18181818, 0, 0, 0, 0, 0, 0, 0, 1.0, 0],
[0, 0, 0, 0.18181818, 0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0],
[0, 0, 0, 0, 0.18181818, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 1.33333333, 0, 0, -0.09090909, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, -0.09090909, 0, 0.18181818, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, -0.09090909, 0, 0, 1.0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0.18181818],
[0, 0, 0, 0.18181818, 0, 0, 0, -0.09090909, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0.18181818, 0, 0, -0.09090909, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 1.33333333, -0.09090909, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0.18181818, 0, 0],
[0, 0, 0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0, 1.0, 0],
[0, 0, 0, 0, 0, 0, 0, -0.09090909, 0, 0, 0, 0, 0, 0.18181818],
[0, 0, 0, 0.18181818, 0, 0, 0, 0, 0, 0, -0.08333333, 0, 0, 0],
[0, 0, 0, 0, 0.18181818, 0, 0, 0, 0, 0, -0.08333333, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 1.33333333, 0, 0, 0, -0.08333333, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -0.08333333, 0.18181818, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -0.08333333, 0, 1.0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -0.08333333, 0, 0, 0.18181818]]
csy = [5.72, 0.63, 0.49, 4.39, 4.36, 8.8, 2.86, 8.78, 8.78, 4.39, 4.39, 8.78, 8.78, 8.8, -14.91, -14.91, -14.91, 8.99,
-8.99, 0.99, -0.99, 1.29, -1.29, 6.99, -6.99, 6.94, -6.94, 11.99, -11.99, 4.99, -4.99, 0.05, 0.05, 0.05, 0.05,
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05,
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05,
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
def constraint2(ix, iy):
return lambda x: numpy.dot(ix, x) - iy
def minimize():
def fun(x):
index_ = -(x[6] ** 2)
print(index_)
return index_
constraints = []
for ix, iy in zip(csx, csy):
constraints.append({'type': 'ineq', 'fun': constraint2(ix, iy)})
x0 = [4.99, 11.99, 8.99, 6.94, 6.94, 11.99, 0.99, 11.99, 11.99, 11.99, 11.99, 6.99, 1.29, 6.94]
scipy.optimize.minimize(fun, x0, constraints=constraints, method='COBYLA')
if __name__ == '__main__':
print('starting')
minimize()
print('finished')Error message
There is no error, the process just hangs and the shell that started it becomes completely unusable, there appears to be some sort of access violation and the whole process is corrupt.
Ice Lake example (It hangs and no amount of Ctrl+C will you get you out of this one, that shell session is dead)
Haswell example (It completes successfully)
SciPy/NumPy/Python version information
1.7.3 1.22.2 sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)