CentOS 7 GCC 4.8.5 miscompiles optimized build: <built-in function compile> returned NULL without setting an exception #94825

jdhedden · 2022-07-13T20:55:40Z

Bug report

Attempts to build 3.11.0b4 fail with:
Fatal Python error: init_import_site: Failed to import the site module

Commands used:

./configure --enable-optimizations
make

The full error is:

./_bootstrap_python ./Programs/_freeze_module.py abc ./Lib/abc.py Python/frozen_modules/abc.h
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
File "/home/centos/Python-3.11.0b4/Lib/site.py", line 73, in
import os
File "/home/centos/Python-3.11.0b4/Lib/os.py", line 29, in
from _collections_abc import _check_methods
SystemError: returned NULL without setting an exception
make[1]: *** [Python/frozen_modules/abc.h] Error 1
make[1]: Leaving directory `/home/centos/Python-3.11.0b4'
make: *** [profile-opt] Error 2

Your environment

CPython versions tested on: 3.11.0b4
Operating system and architecture:

CentOS Linux release 7.9.2009 (Core)
Linux 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed May 18 16:02:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)

EOF

jdhedden · 2022-07-14T01:14:22Z

I believe this is a parallel processing issue. If I try the build multiple time, it does succeed for perhaps 1 out of 3 times. I think some rule needs to be added to the makefile to ensure the site module is built prior it its being needed by other aspects of the build.

tiran · 2022-07-14T12:33:38Z

I don't see any job parameters. Do you run make with a -j options or do you have MAKEFLAGS env var set? Are there any env vars with PYTHON prefix in your build environment?

The SystemError should not happen. It might be the cause of your problem.

from _collections_abc import _check_methods
SystemError: returned NULL without setting an exception

jdhedden · 2022-07-14T12:55:25Z

All I did was untar the distribution, cd'ed into it and ran:

./configure --enable-optimizations
make

No environment vars were set. 'make' was run without arguments.

tiran · 2022-07-14T13:15:55Z

So no parallel builds then.

I can reproduce the issue on a CentOS 7.9 machine with GCC 4.8.5.

$ ./_bootstrap_python ./Programs/_freeze_module.py abc ./Lib/abc.py Python/frozen_modules/abc.h
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/root/cpython/Lib/site.py", line 73, in <module>
    import os
  File "/root/cpython/Lib/os.py", line 29, in <module>
    from _collections_abc import _check_methods
SystemError: <built-in function compile> returned NULL without setting an exception

Python's builtin compile() fails because PyList_Sort() fails to sort dict keys in dictbytype.

#0  PyList_Sort (v=<optimized out>, v@entry=0x7ffff70b6080) at Objects/listobject.c:2588
#1  0x000000000055c98b in dictbytype (src=0x7ffff704ef40, scope_type=scope_type@entry=5, flag=flag@entry=0, offset=offset@entry=0)
    at Python/compile.c:654
#2  0x00000000005655e6 in compiler_enter_scope (c=c@entry=0x7fffffffc5f0, name=<optimized out>, scope_type=scope_type@entry=0, 
    key=key@entry=0xc04ab8, lineno=lineno@entry=1) at Python/compile.c:1739
#3  0x00000000005ed90f in compiler_mod (mod=0xc04ab8, c=0x7fffffffc5f0) at Python/compile.c:2177
#4  _PyAST_Compile (mod=0xc04ab8, filename=filename@entry=0x7ffff7fcf450, flags=flags@entry=0x7fffffffc710, optimize=optimize@entry=-1, 
    arena=arena@entry=0x7ffff70486b0) at Python/compile.c:581
#5  0x00000000005fd8aa in Py_CompileStringObject (
    str=0xa155c0 "# Copyright 2007 Google, Inc. All Rights Reserved.\n# Licensed to PSF under a Contributor Agreement.\n\n\"\"\"Abstract Base Classes (ABCs) for collections, according to PEP 3119.\n\nUnit tests are in test_col"..., filename=filename@entry=0x7ffff7fcf450, start=257, 
    flags=flags@entry=0x7fffffffc710, optimize=optimize@entry=-1) at Python/pythonrun.c:1803
#6  0x00000000005e7174 in builtin_compile_impl (module=<optimized out>, feature_version=<optimized out>, optimize=-1, dont_inherit=<optimized out>, 
    flags=0, mode=0x7ffff7f7fb20 "exec", filename=0x7ffff7fcf450, source=<optimized out>) at Python/bltinmodule.c:829
#7  builtin_compile (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at Python/clinic/bltinmodule.c.h:328
#8  0x00000000004f5957 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7ffff7f78ea0, args=<optimized out>, nargsf=<optimized out>, 
    kwnames=<optimized out>) at Objects/methodobject.c:443
#9  0x00000000004b90a5 in _PyVectorcall_Call (kwargs=0x4f5910 <cfunction_vectorcall_FASTCALL_KEYWORDS>, tuple=0x7ffff7034340, 
    callable=0x7ffff7f78ea0, func=0x4f5910 <cfunction_vectorcall_FASTCALL_KEYWORDS>, tstate=0x9a3808 <_PyRuntime+166312>) at Objects/call.c:257
#10 _PyObject_Call (kwargs=0x4f5910 <cfunction_vectorcall_FASTCALL_KEYWORDS>, args=0x7ffff7034340, callable=0x7ffff7f78ea0, 
    tstate=0x9a3808 <_PyRuntime+166312>) at Objects/call.c:328
#11 PyObject_Call (callable=callable@entry=0x7ffff7f78ea0, args=args@entry=0x7ffff7034340, kwargs=kwargs@entry=0x7ffff7035940) at Objects/call.c:355

The dict looks fine to me:

(gdb) p _PyObject_Dump(src)
object address  : 0x7ffff704ef40
object refcount : 1
object type     : 0x95c7c0
object type name: dict
object repr     : {'ABCMeta': 2192, 'abstractmethod': 2176, 'sys': 2176, 'GenericAlias': 2066, 'type': 6160, 'list': 6160, 'int': 6160, 'EllipsisType': 2050, '_f': 2066, 'FunctionType': 2050, '__all__': 2050, '__name__': 2050, 'bytes_iterator': 2066, 'iter': 6160, 'bytearray_iterator': 2066, 'bytearray': 6160, 'dict_keyiterator': 2066, 'dict_valueiterator': 2066, 'dict_itemiterator': 2066, 'list_iterator': 2066, 'list_reverseiterator': 2066, 'reversed': 6160, 'range_iterator': 2066, 'range': 6160, 'longrange_iterator': 2066, 'set_iterator': 2066, 'set': 6160, 'str_iterator': 2066, 'tuple_iterator': 2066, 'zip_iterator': 2066, 'zip': 6160, 'dict_keys': 2066, 'dict_values': 2066, 'dict_items': 2066, 'mappingproxy': 2066, 'generator': 2066, '_coro': 2066, 'coroutine': 2066, '_ag': 2066, 'async_generator': 2066, '_check_methods': 2050, 'Hashable': 2050, 'Awaitable': 2066, 'Coroutine': 2066, 'AsyncIterable': 2066, 'AsyncIterator': 2066, 'AsyncGenerator': 2066, 'Iterable': 2066, 'Iterator': 2066, 'Reversible': 2066, 'Generator': 2066, 'Sized': 2066, 'Container': 2066, 'Collection': 2066, '_CallableGenericAlias': 2050, '_is_param_expr': 2050, '_type_repr': 2050, 'Callable': 2050, 'Set': 2066, 'frozenset': 6160, 'MutableSet': 2066, 'Mapping': 2066, 'MappingView': 2066, 'KeysView': 2066, 'ItemsView': 2066, 'ValuesView': 2066, 'MutableMapping': 2066, 'dict': 6160, 'Sequence': 2066, 'tuple': 6160, 'str': 6160, 'memoryview': 6160, 'ByteString': 2066, 'bytes': 6160, 'MutableSequence': 2066}

This might be a compiler bug caused by an invalid optimization.

tiran · 2022-07-14T13:29:06Z

$ echo "" > empty.py
$ ./_bootstrap_python -S empty.py
$ echo "import os" > importos.py
$ ./_bootstrap_python -S importos.py 
Traceback (most recent call last):
  File "/root/cpython/importos.py", line 1, in <module>
    import os
  File "/root/cpython/Lib/os.py", line 29, in <module>
    from _collections_abc import _check_methods
SystemError: <built-in function compile> returned NULL without setting an exception
$ echo "import _collections_abc" > importcol.py
$ ./_bootstrap_python -S importcol.py 
Traceback (most recent call last):
  File "/root/cpython/importcol.py", line 1, in <module>
    import _collections_abc
SystemError: <built-in function compile> returned NULL without setting an exception

tiran · 2022-07-14T14:30:59Z

I can't reproduce the problem with devtoolset 7 and GCC 7.3.1 on CentOS / RHEL 7. This points to a compiler bug with GCC 4.8.

@encukou @vstinner @pablogsal

vstinner · 2022-08-01T11:52:27Z

You can get a more recent GCC version on RHEL 7 and Centos 7 via https://www.softwarecollections.org/en/scls/rhscl/devtoolset-8/ for example.

IMO Python ./configure script should fail if trying to build Python with an old GCC version known to have compiler bugs.

Python 3.11 now requires a C compiler compatible with C11 (without optional) features):

vstinner · 2022-08-01T11:54:29Z

If someone is very motivated, maybe the faulty GCC optimization can be identified and it can be disabled when building Python 3.11 on old GCC versions (ex: use -O2 instead of -O3). But it sounds like a boring whack-a-mole game :-(

cpach · 2023-01-04T21:45:44Z

I have managed to succesfully compile Python 3.11.1 on CentOS 7. I have also managed to trigger this bug, but AFAICT it seems to work fine when doing make -j 1.

I configured the build like this basically:

./configure \
    --with-openssl="/opt/openssl-3.0.7" \
    --enable-optimizations \
    --with-system-ffi \
    --with-computed-gotos \
    --enable-loadable-sqlite-extensions

CentOS version: 7.9.2009 (Core), kernel: 3.10.0-1160.81.1.el7.x86_64. GCC version: 4.8.5 20150623 (Red Hat 4.8.5-44).

jdhedden added the type-bug An unexpected behavior, bug, or error label Jul 13, 2022

AlexWaygood added the build The build process and cross-build label Jul 14, 2022

tiran changed the title ~~Optimized build failure: Failed to import the site module~~ CentOS 7 GCC 4.8.5 miscompiles in optimized build: <built-in function compile> returned NULL without setting an exception Jul 14, 2022

tiran changed the title ~~CentOS 7 GCC 4.8.5 miscompiles in optimized build: <built-in function compile> returned NULL without setting an exception~~ CentOS 7 GCC 4.8.5 miscompiles optimized build: <built-in function compile> returned NULL without setting an exception Jul 14, 2022

Dec	JAN	Feb
	04
2022	2023	2024

CentOS 7 GCC 4.8.5 miscompiles optimized build: <built-in function compile> returned NULL without setting an exception #94825

CentOS 7 GCC 4.8.5 miscompiles optimized build: <built-in function compile> returned NULL without setting an exception #94825

jdhedden commented Jul 13, 2022

jdhedden commented Jul 14, 2022

tiran commented Jul 14, 2022

jdhedden commented Jul 14, 2022

tiran commented Jul 14, 2022

tiran commented Jul 14, 2022

tiran commented Jul 14, 2022

vstinner commented Aug 1, 2022

vstinner commented Aug 1, 2022

cpach commented Jan 4, 2023

CentOS 7 GCC 4.8.5 miscompiles optimized build: <built-in function compile> returned NULL without setting an exception #94825

CentOS 7 GCC 4.8.5 miscompiles optimized build: <built-in function compile> returned NULL without setting an exception #94825

Comments

jdhedden commented Jul 13, 2022

EOF

jdhedden commented Jul 14, 2022

tiran commented Jul 14, 2022

jdhedden commented Jul 14, 2022

tiran commented Jul 14, 2022

tiran commented Jul 14, 2022

tiran commented Jul 14, 2022

vstinner commented Aug 1, 2022

vstinner commented Aug 1, 2022

cpach commented Jan 4, 2023