Description
Description
When OPcache is enabled and either NewRelic or the zend_test extensions are active, we're seeing a segmentation fault on a fairly complicated WordPress app. We've unfortunately not been able to come up with a simple PHP script that would allow us to simplify reproduction steps. The site's code is maintained by a customer of ours and has a large number of plugins so it's difficult to pinpoint the code that's causing the issue.
The issue has also been reported in NewRelic's PHP agent repo, but it looks like it's most likely not caused by their agent, since it's possible to reproduce it with the zend_test extension. The original reporter mentioned that the issue may have been caused by the font-awesome plugin but we've not been able to reproduce it with that so far:
It's not clear what's changed in NewRelic's PHP agent that started triggering this bug (it started happening in version 10.18.0.8), but we believe they've stopped using the old tracing method of replacing the zend_execute_ex function in minit/mshutdown and switched to using the Observer API in all cases.
A similar issue has also been reported in this repo so hopefully this isn't a duplicate, our crash seems to be different enough:
OPcache and JIT
The issue goes away completely if we disabled OPcache:
opcache.enable = 0
We also have JIT disabled by default (buffer size is set to 0):
opcache.jit => tracing => tracing
opcache.jit_buffer_size => 0 => 0
However, the issue still occurs even if we completely disable it while OPcache is still enabled:
opcache.jit = disable
Core dump debugging
We've looked at a few core dumps, and when OPcache is enabled, the func pointer in the execute_data variable ends up pointing at an invalid address which then crashes PHP when it dereferences it in the Observer API's exit handler:
Here's the backtrace and a some more debugging information that we've been able to extract from a core dump:
Core was generated by `php-fpm:'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 nr_php_backtrace_get_call_site () at /usr/local/src/newrelic-php-agent/agent/php_stack.c:220
220 /usr/local/src/newrelic-php-agent/agent/php_stack.c: No such file or directory.
(gdb) bt
#0 nr_php_backtrace_get_call_site () at /usr/local/src/newrelic-php-agent/agent/php_stack.c:220
#1 nr_php_frame_info () at /usr/local/src/newrelic-php-agent/agent/php_stack.c:267
#2 nr_php_backtrace_fd () at /usr/local/src/newrelic-php-agent/agent/php_stack.c:462
#3 0x00007f05b23172dc in nr_php_fatal_signal_handler () at /usr/local/src/newrelic-php-agent/agent/php_minit.c:740
#4 <signal handler called>
#5 call_end_observers (return_value=0x0, execute_data=0x55734da87d48) at /usr/src/php/Zend/zend_observer.c:265
#6 zend_observer_fcall_end_all () at /usr/src/php/Zend/zend_observer.c:293
#7 0x0000557382500b99 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/php/main/main.c:1841
#8 0x0000557382248a39 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/php/sapi/fpm/fpm/fpm_main.c:1970
(gdb) up 5
#5 call_end_observers (return_value=0x0, execute_data=0x55734da87d48) at /usr/src/php/Zend/zend_observer.c:265
265 /usr/src/php/Zend/zend_observer.c: No such file or directory.
(gdb) source /core-dumps/php-gdbinit
(gdb) zbacktrace
[0x55734da87d48] Cannot access memory at address 0x23e00000357
(gdb) print execute_data
$1 = (zend_execute_data *) 0x55734da87d48
(gdb) set $ed = (zend_execute_data *) 0x55734da87d48
(gdb) print $ed->func
$2 = (zend_function *) 0x23e00000347
(gdb) set $edf = (zend_function *) 0x23e00000347
(gdb) print $edf->common.function_name->val
Cannot access memory at address 0x23e0000034f
(gdb) print $edf->type
Cannot access memory at address 0x23e00000347
(gdb) maintenance info sections
Exec file:
`/usr/local/sbin/php-fpm', file type elf64-x86-64.
[0] 0x5573820002a8->0x5573820002c4 at 0x000002a8: .interp ALLOC LOAD READONLY DATA HAS_CONTENTS
[...]
[25] 0x5573836064a0->0x557383627990 at 0x01406498: .bss ALLOC
[26] 0x00000000->0x00000027 at 0x01406498: .comment READONLY HAS_CONTENTS
[27] 0x00000000->0x00007340 at 0x014064c0: .debug_aranges READONLY HAS_CONTENTS
[28] 0x00000000->0x00bf3119 at 0x0140d800: .debug_info READONLY HAS_CONTENTS
[29] 0x00000000->0x00088872 at 0x02000919: .debug_abbrev READONLY HAS_CONTENTS
[30] 0x00000000->0x0066edef at 0x0208918b: .debug_line READONLY HAS_CONTENTS
[31] 0x00000000->0x00220f7c at 0x026f7f7a: .debug_str READONLY HAS_CONTENTS
[32] 0x00000000->0x00e610db at 0x02918ef6: .debug_loc READONLY HAS_CONTENTS
[33] 0x00000000->0x002e3a90 at 0x03779fd1: .debug_ranges READONLY HAS_CONTENTS
[34] 0x00000000->0x001b2373 at 0x03a5da61: .debug_macro READONLY HAS_CONTENTS
Core file:
`/core-dumps/core-php-fpm.68.php-43789.1711460016', file type elf64-x86-64.
[0] 0x00000000->0x0000d358 at 0x0000a108: note0 READONLY HAS_CONTENTS
[1] 0x00000000->0x000000d8 at 0x0000a18c: .reg/68 HAS_CONTENTS
[2] 0x00000000->0x000000d8 at 0x0000a18c: .reg HAS_CONTENTS
[3] 0x00000000->0x00000080 at 0x0000a31c: .note.linuxcore.siginfo/68 HAS_CONTENTS
[4] 0x00000000->0x00000080 at 0x0000a31c: .note.linuxcore.siginfo HAS_CONTENTS
[5] 0x00000000->0x00000150 at 0x0000a3b0: .auxv HAS_CONTENTS
[6] 0x00000000->0x0000c299 at 0x0000a514: .note.linuxcore.file/68 HAS_CONTENTS
[7] 0x00000000->0x0000c299 at 0x0000a514: .note.linuxcore.file HAS_CONTENTS
[8] 0x00000000->0x00000200 at 0x000167c4: .reg2/68 HAS_CONTENTS
[9] 0x00000000->0x00000200 at 0x000167c4: .reg2 HAS_CONTENTS
[10] 0x00000000->0x00000a88 at 0x000169d8: .reg-xstate/68 HAS_CONTENTS
[11] 0x00000000->0x00000a88 at 0x000169d8: .reg-xstate HAS_CONTENTS
[12] 0x557344200000->0x557382000000 at 0x00018000: load1 ALLOC LOAD HAS_CONTENTS
[...]
[859] 0x7fffebdc6000->0x7fffebdca000 at 0x497c5000: load732 ALLOC LOAD READONLY HAS_CONTENTS
[860] 0x7fffebdca000->0x7fffebdcc000 at 0x497c9000: load733 ALLOC LOAD READONLY CODE HAS_CONTENTS
[861] 0xffffffffff600000->0xffffffffff601000 at 0x497cb000: load734 ALLOC LOAD READONLY CODE HAS_CONTENTS
The backtrace with zend_test enabled is almost identical, the only difference is that NewRelic's signal handler isn't invoked after the segmentation fault:
Core was generated by `php-fpm:'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 call_end_observers (return_value=0x0, execute_data=0x56205f087ec8) at /usr/src/php/Zend/zend_observer.c:265
265 /usr/src/php/Zend/zend_observer.c: No such file or directory.
(gdb) bt
#0 call_end_observers (return_value=0x0, execute_data=0x56205f087ec8) at /usr/src/php/Zend/zend_observer.c:265
#1 zend_observer_fcall_end_all () at /usr/src/php/Zend/zend_observer.c:293
#2 0x0000562093b00b99 in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/php/main/main.c:1841
#3 0x0000562093848a39 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/php/sapi/fpm/fpm/fpm_main.c:1970
(gdb) source /core-dumps/php-gdbinit
(gdb) zbacktrace
[0x56205f087ec8] Cannot access memory at address 0x23e00000357
The following config options are set for the zend_test extension:
zend_test.observer.enabled=1
zend_test.observer.observe_all=1
zend_test.observer.show_output=0
More info about the environment
The site's running in a Docker container, using a slightly modified image that's based on the official PHP Docker images - https://github.com/docker-library/php/tree/master/8.3/bullseye/fpm.
Some notable changes:
- PHP is compiled with
--enable-embed - We're installing jemalloc but it's not in use by default and isn't enabled for the site that's crashing
- A few additional extensions are baked into the image
List of PHP modules:
PHP Version
PHP 8.3.4
Operating System
Ubuntu 20.04