I'm not sure this is a problem with python itself or if I'm messing something up, however I've now been stuck on this problem for weeks and couldn't find any sort of solution to it. I'm on MacOs and I'm trying to embed python inside a c shared library. Disclaimer: there are two installations of python involved with this question. One is my "main" python, so to speak, installed at /Library/Frameworks/Python.framework/Versions/3.10/bin/python3, that will be referred as "main python"; and the other one is a subdirectory of my current working directory containing python source code that i will build and embed, and it will be referred as embedded python.
For clarity, I have reproduced a minimalistic example of my problem that gives the same results.

Consider that the current working directory contains:

• python310 (a directory containing python source code)

I configure it using the following command:

./configure --prefix=__path_to_the_directory_containing_python_source__ --exec-prefix=__path_to_the_directory_containing_python_source__ --enable-shared --enable-optimizations --with-lto

and compiled it using make && make altinstall
I have used altinstall because I will use this distribution only inside my application and I didn't want to replace my normal python installation.

• test_interpreter.c:

#include "__path_to_the_directory_containing_python_source__/Include/Python.h"

int test() {
    Py_SetProgramName(L"__path_to_the_directory_containing_python_source__/python.exe");
    Py_Initialize();

    // adds the current directory to sys.path to make test_import.py importable
    PyObject* sysPath = PySys_GetObject("path"); // borrowed reference
    PyObject* cur_dir = PyUnicode_FromString(""); // new reference
    PyList_Append(sysPath, cur_dir);

    PyRun_SimpleString("import test_import");
    Py_DECREF(cur_dir);
    Py_Finalize();
    return 0;
}

• test_import.py:

# will be called by embedded python
print("Inside python")
...

• test_interpreter.py

# will be called by main python
import ctypes

lib = ctypes.CDLL("/__path_to_current_directory__/libtest.so")
tp = ctypes.CFUNCTYPE(ctypes.c_int)
test = tp(("test", lib))
test()

Now the problem is that, when test_import.py imports some builtin modules (note it doesn't happen with every module but just some), i get errors like segfault (e.g. when importing ctypes) or abort trap (e.g. when importing sqlite3). The most interesting part is that the errors do not occur if the interpreter is embedded inside an executable instead than a shared library.

So, if I compile test_interpreter.c into libtest.so using: gcc -L__path_to_the_directory_containing_python_source__ -lpython3.10 -shared -install_name __current_directory_path__/libtest.so -o libtest.so test_interpreter.c,

then modify test_import.py to for example

# will be called by embedded python
print("Inside python")
import decimal

and execute python3 test_interpreter.py (using main python; version is still 3.10) i get:

Inside python
Segmentation fault: 11

Other modules that give me the same error message are:

• tkinter
• ctypes

Also, if it can be usefull, I managed to understand that when importing ctypes inside the embedded interpreter the error occures when the line from _ctypes import Union, Structure, Array (of ctypes' __ init __.py) is executed.

If i modify test_interpreter.py to:

print("Inside python")
import sqlite3

and run the same command i get:

Inside python
Python(86563,0x10a7105c0) malloc: *** error for object 0x101d56360: pointer being freed was not allocated
Python(86563,0x10a7105c0) malloc: *** set a breakpoint in malloc_error_break to debug
Abort trap: 6

Other modules that give me the same error message are:

• dbm
• distutils
• random
• json
• zoneinfo
• base64
• csv
• calendar
• pickle

Note that if I compile test_interpreter.c as an executable (after changing test function name to main) using gcc -L__path_to_the_directory_containing_python_source__ -o test test_interpreter.c, or if I run the python executable (without embedding it) and try to import those module, everything works fine.

Thanks in advance to everyone who will try to understand what's going on.