Fudong

Fudong

I love kernel.
I love debugging.

A List of Common Reasons for Segmentation Faults

15 minute read

Segment fault are common issues for native developers, it is caused by accessing memory location that is not allowed to, I’m gonna list the possible reasons of segfault, and show how to analyze these kind of issues, a tombstone was generated when segment fault occurs.

Possible Reasons that get segment fault

Google has developed crasher which shows the reasons that lead to the native crash, and also I find some in this answer, in this section those cases will be illustrated, and other cases that I found will be also be listed here.

Abort by calling abort/assert/raise/LOG(FATAL)

The first type of crash was made by calling abort, assert, raise or LOG(FATAL) and the friends on purpose, the exception logs are much similar, the fault addresses are N/A, the following is created with crasher abort:

F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
F DEBUG   : LineageOS Version: '16.0-20200803-UNOFFICIAL-rpi3'
F DEBUG   : Build fingerprint: 'Raspberry/lineage_rpi3/rpi3:9/PQ3A.190801.002/fdbai08031438:userdebug/test-keys'
F DEBUG   : Revision: '0'
F DEBUG   : ABI: 'arm'
F DEBUG   : pid: 2384, tid: 2384, name: crasher  >>> crasher <<<
F DEBUG   : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG   :     r0  00000000  r1  00000950  r2  00000006  r3  00000008
F DEBUG   :     r4  00000950  r5  00000950  r6  fff99e0c  r7  0000010c
F DEBUG   :     r8  00000000  r9  00000000  r10 00000000  r11 00000000
F DEBUG   :     ip  fff9a010  sp  fff99df8  lr  ea9a5ed9  pc  ea99dd2a
F DEBUG   :
F DEBUG   : backtrace:
F DEBUG   :     #00 pc 0001cd2a  /system/lib/libc.so (abort+58)
F DEBUG   :     #01 pc 000010f3  /system/bin/crasher (maybe_abort+26)
F DEBUG   :     #02 pc 000014e3  /system/bin/crasher (do_action+686)
F DEBUG   :     #03 pc 00002341  /system/bin/crasher (main+68)
F DEBUG   :     #04 pc 00088bc1  /system/lib/libc.so (__libc_init+48)
F DEBUG   :     #05 pc 00001097  /system/bin/crasher (_start_main+38)
F DEBUG   :     #06 pc 00000306  <anonymous:eadee000>

The key messages used to identify these kind of issues are listed below, and the tombstone files are also included:

Type Key Messages Log
Abort F DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG : #00 pc 0001cd2a /system/lib/libc.so (abort+58)		 abort
Assert F DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG : Abort message: ‘some_file.c:123: assertion “false” failed’		 assert
assert2
LOG(FATAL) F DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG : #01 pc 000068fb /system/lib/libbase.so (android::base::DefaultAborter(char const*)+6)		 LOG(FATAL)
Raise F DEBUG : signal 8 (SIGFPE), code -6 (SI_TKILL), fault addr -------- raise

Null pointer dereference

If fault addr is 0 or lower address, this indicates a null pointer dereference, this could be caused by use after free, uninitialized pointer, wild pointer or missing argument in printf/sprintf, also malloc is not guaranteed to be successful, missing the return value check can also lead to undefined behavior.

This is an example crash log of use after free case:

F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
F DEBUG   : LineageOS Version: '16.0-20200803-UNOFFICIAL-rpi3'
F DEBUG   : Build fingerprint: 'Raspberry/lineage_rpi3/rpi3:9/PQ3A.190801.002/fdbai08031438:userdebug/test-keys'
F DEBUG   : Revision: '0'
F DEBUG   : ABI: 'arm'
F DEBUG   : pid: 2936, tid: 2936, name: crasher  >>> crasher <<<
F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
F DEBUG   : Cause: null pointer dereference
F DEBUG   :     r0  00000005  r1  00000068  r2  00000006  r3  ffffffff
F DEBUG   :     r4  00000000  r5  aac429bf  r6  00000006  r7  00000000
F DEBUG   :     r8  00000000  r9  00000000  r10 00000000  r11 00000000
F DEBUG   :     ip  aac44fc8  sp  ffebfc38  lr  aac3fe8f  pc  ec9c98ce
F DEBUG   :
F DEBUG   : backtrace:
F DEBUG   :     #00 pc 000398ce  /system/lib/libc.so (__strncpy_chk2+26)
F DEBUG   :     #01 pc 00001e8b  /system/bin/crasher (uaf+106)
F DEBUG   :     #02 pc 0000158f  /system/bin/crasher (do_action+710)
F DEBUG   :     #03 pc 000024b5  /system/bin/crasher (main+68)
F DEBUG   :     #04 pc 00088bc1  /system/lib/libc.so (__libc_init+48)
F DEBUG   :     #05 pc 0000112b  /system/bin/crasher (_start_main+38)
F DEBUG   :     #06 pc 00000306  <anonymous:ecdab000>

Other examples in crasher regarding null pointer dereferences are as follows:

Type Key Messages Log
strlen-NULL F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
F DEBUG : Cause: null pointer dereference		 strlen null
call-null F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
F DEBUG : Cause: null pointer dereference		 call null
myprintf F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x5
F DEBUG : Cause: null pointer dereference		 missing argument

The last one will report a warning if -Wall or -Wformat option is on, or an error if -Werror was also enabled:

system/core/debuggerd/crasher/crasher.cpp:199:17: error: more '%' conversions than data arguments [-Werror,-Wformat]
    printf("%s %d\n", i);

FORTIFY failure

bionic c library keep involving along with android, in Android P, formerly null pointer dereference cases such as fprintf-NULL and readdir-NULL are now fall into FORTIFY failure case, a sanity check on FILE* for nullptr is added, this is done by the following macro:

#define CHECK_FP(fp) \
  if (fp == nullptr) __fortify_fatal("%s: null FILE*", __FUNCTION__)

which is defined in libc/stdio/local.h.

In bionic, buffer size is also checked before the real syscall, this is a FORTIFY failure case try to read 32-byte data into a 10-byte array:

F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
F DEBUG   : LineageOS Version: '16.0-20200803-UNOFFICIAL-rpi3'
F DEBUG   : Build fingerprint: 'Raspberry/lineage_rpi3/rpi3:9/PQ3A.190801.002/fdbai08031438:userdebug/test-keys'
F DEBUG   : Revision: '0'
F DEBUG   : ABI: 'arm'
F DEBUG   : pid: 3118, tid: 3118, name: crasher  >>> crasher <<<
F DEBUG   : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG   : Abort message: 'FORTIFY: read: prevented 32-byte write into 10-byte buffer'
F DEBUG   :     r0  00000000  r1  00000c2e  r2  00000006  r3  00000008
F DEBUG   :     r4  00000c2e  r5  00000c2e  r6  ffcfe9e4  r7  0000010c
F DEBUG   :     r8  00000000  r9  00000000  r10 00000000  r11 00000000
F DEBUG   :     ip  ffcfec00  sp  ffcfe9d0  lr  e9374ed9  pc  e936cd2a
F DEBUG   :
F DEBUG   : backtrace:
F DEBUG   :     #00 pc 0001cd2a  /system/lib/libc.so (abort+58)
F DEBUG   :     #01 pc 00039113  /system/lib/libc.so (__fortify_fatal(char const*, ...)+26)
F DEBUG   :     #02 pc 0003952b  /system/lib/libc.so (__read_chk+62)
F DEBUG   :     #03 pc 00001613  /system/bin/crasher (do_action+842)
F DEBUG   :     #04 pc 000024b5  /system/bin/crasher (main+68)
F DEBUG   :     #05 pc 00088bc1  /system/lib/libc.so (__libc_init+48)
F DEBUG   :     #06 pc 0000112b  /system/bin/crasher (_start_main+38)
F DEBUG   :     #07 pc 00000306  <anonymous:e96c5000>
Type Key Messages Log
fortify F DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG : Abort message: ‘FORTIFY: read: prevented 32-byte write into 10-byte buffer’		 fortify
fprintf-NULL F DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG : Abort message: ‘FORTIFY: fprintf: null FILE*’		 fprintf null
readdir-NULL F DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG : Abort message: ‘FORTIFY: readdir: null DIR*’		 readdir null

Check the Abort message and __fortify_fatal in the backtrace to see if it is FORTIFY related.

Array index past the end of the array

void oob()
{
	int a[10];

	a[10] = 100;
}

The above code creates an out of bound issue, it is undefined behaviour and can be detected by Bound Sanitizer as we shown in the previous post Android native memory debugging, the compiler option -fstack-protector-all is also served for this purpose.

Type Key Messages Log
Out of Bounds F DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG : Abort message: ‘stack corruption detected (-fstack-protector)’		 smash stack

Seccomp SIGSYS from a disallowed system call

Seccomp (SECure COMPuting with filters) is a kernel feature that used for filtering unwanted syscalls by applying BPF, android take this advantage to harden android system from Android O, the filter is applied to zygote, thus all syscall from app will be checked, if there is any syscall that is forbidden it will crash, seccomp filter must set explicitly for native applications with set_system_seccomp_filter(), for example, swapon/wapoff are blocked as they are considered to be unsecure to the system, if you apps make calls to them, the below exception will be reported in logcat:

F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
F DEBUG   : LineageOS Version: '16.0-20200803-UNOFFICIAL-rpi3'
F DEBUG   : Build fingerprint: 'Raspberry/lineage_rpi3/rpi3:9/PQ3A.190801.002/fdbai08031438:userdebug/test-keys'
F DEBUG   : Revision: '0'
F DEBUG   : ABI: 'arm'
F DEBUG   : pid: 2521, tid: 2521, name: crasher  >>> crasher <<<
F DEBUG   : signal 31 (SIGSYS), code 1 (SYS_SECCOMP), fault addr --------
F DEBUG   : Cause: seccomp prevented call to disallowed arm system call 115
F DEBUG   :     r0  b1d93b3d  r1  a1b570dc  r2  00000001  r3  00000000
F DEBUG   :     r4  b1d915b9  r5  ff9ed544  r6  00000002  r7  00000073
F DEBUG   :     r8  00000000  r9  00000000  r10 00000000  r11 00000000
F DEBUG   :     ip  ff9ed550  sp  ff9ed378  lr  b1d908bd  pc  edd1fac8
F DEBUG   :
F DEBUG   : backtrace:
F DEBUG   :     #00 pc 00055ac8  /system/lib/libc.so (swapoff+12)
F DEBUG   :     #01 pc 000018b9  /system/bin/crasher (do_action+1592)
F DEBUG   :     #02 pc 000025fd  /system/bin/crasher (main+68)
F DEBUG   :     #03 pc 00088bc1  /system/lib/libc.so (__libc_init+48)
F DEBUG   :     #04 pc 0000112b  /system/bin/crasher (_start_main+38)
F DEBUG   :     #05 pc 00000306  <anonymous:edf3b000>
Type Key Messages Log
swapoff F DEBUG : signal 31 (SIGSYS), code 1 (SYS_SECCOMP), fault addr --------
F DEBUG : Cause: seccomp prevented call to disallowed arm system call 115		 seccomp

Write access to read-only memory address

Trying to write to read-only memory location will cause app crash with error code SEGV_ACCERR, for example:

void accerr()
{
	char *str;

	str = "hello";
	str[1] = 'w';
}

The code will lead to below exception:

F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
F DEBUG   : LineageOS Version: '16.0-20200803-UNOFFICIAL-rpi3'
F DEBUG   : Build fingerprint: 'Raspberry/lineage_rpi3/rpi3:9/PQ3A.190801.002/fdbai08031438:userdebug/test-keys'
F DEBUG   : Revision: '0'
F DEBUG   : ABI: 'arm'
F DEBUG   : pid: 1392, tid: 1392, name: crasher  >>> crasher <<<
F DEBUG   : signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xb3ad2a8c
F DEBUG   :     r0  f1afbe0c  r1  b3ad2a8b  r2  00000077  r3  00000000
F DEBUG   :     r4  b3ad0535  r5  ff8ab164  r6  00000002  r7  ff8ab170
F DEBUG   :     r8  00000000  r9  00000000  r10 00000000  r11 00000000
F DEBUG   :     ip  00000000  sp  ff8aaf88  lr  b3acf563  pc  b3acff40
F DEBUG   :
F DEBUG   : backtrace:
F DEBUG   :     #00 pc 00001f40  /system/bin/crasher (accerr+24)
F DEBUG   :     #01 pc 0000155f  /system/bin/crasher (do_action+734)
F DEBUG   :     #02 pc 00002579  /system/bin/crasher (main+68)
F DEBUG   :     #03 pc 00088bc1  /system/lib/libc.so (__libc_init+48)
F DEBUG   :     #04 pc 0000112b  /system/bin/crasher (_start_main+38)
F DEBUG   :     #05 pc 00000306  <anonymous:f1cc5000>
Type Key Messages Log
write to RO F DEBUG : signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xb3ad2a8c write to read only

Invalid use of free

I think this is not a common issue, passing address of variable in stack to free will cause app to crash:

F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
F DEBUG   : LineageOS Version: '16.0-20200803-UNOFFICIAL-rpi3'
F DEBUG   : Build fingerprint: 'Raspberry/lineage_rpi3/rpi3:9/PQ3A.190801.002/fdbai08031438:userdebug/test-keys'
F DEBUG   : Revision: '0'
F DEBUG   : ABI: 'arm'
F DEBUG   : pid: 1407, tid: 1407, name: crasher  >>> crasher <<<
F DEBUG   : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG   : Abort message: 'Invalid address 0xfff18d0c passed to free: value not allocated'
F DEBUG   :     r0  00000000  r1  0000057f  r2  00000006  r3  00000008
F DEBUG   :     r4  0000057f  r5  0000057f  r6  fff18ca4  r7  0000010c
F DEBUG   :     r8  f5b0b008  r9  00000000  r10 00000000  r11 00000000
F DEBUG   :     ip  00000000  sp  fff18c90  lr  f5ca8ed9  pc  f5ca0d2a
F DEBUG   :
F DEBUG   : backtrace:
F DEBUG   :     #00 pc 0001cd2a  /system/lib/libc.so (abort+58)
F DEBUG   :     #01 pc 0007c3a5  /system/lib/libc.so (ifree+880)
F DEBUG   :     #02 pc 0007c4c1  /system/lib/libc.so (je_free+68)
F DEBUG   :     #03 pc 00001cb9  /system/bin/crasher (abuse_heap+20)
F DEBUG   :     #04 pc 00001827  /system/bin/crasher (do_action+1446)
F DEBUG   :     #05 pc 00002579  /system/bin/crasher (main+68)
F DEBUG   :     #06 pc 00088bc1  /system/lib/libc.so (__libc_init+48)
F DEBUG   :     #07 pc 0000112b  /system/bin/crasher (_start_main+38)
F DEBUG   :     #08 pc 00000306  <anonymous:f60c7000>
Type Key Messages Log
F DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG : Abort message: ‘Invalid address 0xfff18d0c passed to free: value not allocated’		 heap  

Returning value of local variable

This is similar to OOB, which may or may not cause a crash, but as it is undefined behavior, the results depending on the compiler implementation, there is a compiler option -Wreturn-stack-address to give it a warning if it is enabled:

system/core/debuggerd/crasher/crasher.cpp:204:9: error: address of stack memory associated with local variable 's' returned [-Werror,-Wreturn-stack-address]
        return s;

This is an example code of accessing address of s after local_var returned:

char *local_var() {
	char s[] = "hello";
	return s;
}

void test_local_var() {
	printf("The string is:%s!\n", local_var());
}

List all the reasons which may cause a segment fault is virtually impossible, do our best to avoid them by at least NEVER IGNORE ANY COMPILER WARNING, and keep below options always on:

-Werror
-Wall

Analyzing native crash log

The very first thing I do when I found a native crash is to decode the backtrace with development/scripts/stack, and review the line of code and the context, save the crash log to a file and feed to the script like this:

$ source build/envsetup.sh
$ lunch lineage_rpi3-userdebug
$ stack --arch=arm accerr
build/make/core/combo/TARGET_linux-arm.mk:43: warning: cortex-a53 is armv8-a.
build/make/core/combo/TARGET_linux-arm.mk:45: warning: TARGET_ARCH_VARIANT, armv7-a-neon, ignored! Use armv8-a instead.
Reading native crash info from stdin
Reading symbols from /opt/workdir/lineageos/lineageos-16/out/target/product/rpi3/symbols
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xb2f8ac28 in tid 2686 (crasher), pid 2686 (crasher)
Revision: '0'
pid: 2686, tid: 2686, name: crasher  >>> crasher <<<
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xb2f8ac28
     r0  f1aeee0c  r1  b2f8ac27  r2  00000077  r3  00000000
     r4  b2f886a9  r5  ff9939d4  r6  00000002  r7  ff9939e0
     r8  00000000  r9  00000000
     ip  00000000  sp  ff9937f8  lr  b2f87597  pc  b2f87fb4
Using arm toolchain from: /opt/workdir/lineageos/lineageos-16/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin/

Stack Trace:
  RELADDR   FUNCTION        FILE:LINE
  00001fb4  accerr+24       system/core/debuggerd/crasher/crasher.cpp:193
  00001593  do_action+734   system/core/debuggerd/crasher/crasher.cpp:309
  000026ed  main+68         system/core/debuggerd/crasher/crasher.cpp:406
  00088bc1  __libc_init+48  bionic/libc/bionic/libc_init_dynamic.cpp:129
  0000115f  _start_main+38  bionic/libc/arch-common/bionic/crtbegin.c:45
  00000306  <unknown>       <anonymous:f1d34000>

One thing needs to be mentioned here is the source code must be kept unchanged, or the backtrace and the code maybe unmatched, if you find the line of code is definitely not the suspected one, then check the BuildId in tombstone file and the output of file command to see if they are the same one:

$ file out/target/product/rpi3/system/bin/crasher
out/target/product/rpi3/system/bin/crasher: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /system/bin/linker, BuildID[md5/uuid]=99f763bbd73b1764ad54af6c2bbdb94f, stripped

$ rg crasher tombstone_09 | rg BuildId
--->b2f86000-b2f8bfff r-x         0      6000  /system/bin/crasher (BuildId: 99f763bbd73b1764ad54af6c2bbdb94f)

Another less likely to happen is the executable binary is put to the target file system in different directory as android build output directory, this might be a problem when you debugging native applications, this will cause the backtrace cannot be parsed correctly by the script, unless you fix the path manually before passing it to the stack script.

The key information in tombstone

Summarize the information in crash log and tombstone we might be interested:

  • The signal and code
  • The fault address
  • The registers
  • Frame #00 is 00000000 (call-null case)
  • The backtrace
  • Memory map (permission)
  • Open files info in tombstone (fd leak)
  • BuildID

Online debugging with gdb

If the crash can be reproduced, then the recommended way to do is using gdb, run gdb server in android:

# gdbserver :8989 stack
[ 8214.476024] init: Untracked pid 1624 received signal 9
Process stack created; pid = 1625
Listening on port 8989

Then connect it from host:

$ adb forward tcp:8989 tcp:8989
$ arm-linux-gnueabi-gdb -q -nh										\
	-ex 'set solib-search-path out/target/product/rpi3/symbols'		\
	-ex 'set solib-absolute-prefix out/target/product/rpi3/symbols'	\
	-ex 'target extended-remote ip.add.re.ss:8989'					\
	out/target/product/rpi3/symbols/system/bin/crasher

(gdb) monitor exit /* quit gdbserver */

You can find the prebuilt gdbserver in prebuilts/misc/android-arm/gdbserver/ in case it is not packaged into android system image.

Cordeump

Sometimes the tombstone file along with the crash information is not sufficient, core dump can provide more information than that, by default, it is disabled in Android, and it need kernel support.

# zcat /proc/config.gz | grep -E 'ELF_CORE|COREDUMP'
CONFIG_ELF_CORE=y
CONFIG_COREDUMP=y

Make sure the above options are enabled in your kernel image.

Use ulimit to enable/disable cordeump:

# Enable coredump
ulimit -S -c unlimited

# Disable coredump
ulimit -S -c 0

By default coredump file will be created in the current working directory named core, unless the core_pattern was set, this can be either done in command line:

sysctl kernel.core_pattern="/data/coredump/core.%e.%p"

or put the following in init.rc:

write  /proc/sys/kernel/core_pattern /data/coredump/core.%e.%p

See man core for more information.

Analyze core dump with gdb

The executable binary used in gdb needs to be the one with debug information.

$ adb pull /data/coredump/
$ arm-linux-gnueabi-gdb -q -nh out/target/product/rpi3/symbols/system/bin/crasher
Reading symbols from out/target/product/rpi3/symbols/system/bin/crasher...
(gdb) set solib-search-path out/target/product/rpi3/symbols/
(gdb) set solib-absolute-prefix out/target/product/rpi3/symbols
(gdb) core coredump/core.crasher.3007
[New LWP 3007]
Core was generated by `crasher seccomp'.
Program terminated with signal SIGSYS, Bad system call.
#0  swapoff () at bionic/libc/arch-arm/syscalls/swapoff.S:10
10	    mov     r7, ip
(gdb) where
#0  swapoff () at bionic/libc/arch-arm/syscalls/swapoff.S:10
#1  0xafccc90a in do_action (arg=0xffcbdb3b "seccomp") at system/core/debuggerd/crasher/crasher.cpp:360
#2  0xafccd6f0 in main (argc=2, argv=0xffcbd6e4) at system/core/debuggerd/crasher/crasher.cpp:406
(gdb) frame 1 /* switch to frame 1 */
#1  0xafccc90a in do_action (arg=0xffcbdb3b "seccomp") at system/core/debuggerd/crasher/crasher.cpp:360
360	        swapoff("/dev/null");
(gdb) l
355	        munmap(map, sizeof(int));
356	        map[0] = '8';
357	    } else if (!strcasecmp(arg, "seccomp")) {
358	        set_system_seccomp_filter();
359	        //syscall(99999);
360	        swapoff("/dev/null");
[...]
(gdb) disassemble
(gdb) info files
(gdb) info threads
(gdb) maintenance info sections
(gdb) i[nfo] proc m[appings]
Mapped address spaces:

	Start Addr   End Addr       Size     Offset objfile
	0xab6fc000 0xab702000     0x6000        0x0 /system/bin/crasher
	0xab702000 0xab703000     0x1000     0x5000 /system/bin/crasher
	0xeb66f000 0xeb673000     0x4000        0x0 /system/lib/libnetd_client.so
	[...]
	0xebb04000 0xebb24000    0x20000        0x0 /dev/__properties__/properties_serial
	0xebb25000 0xebb28000     0x3000        0x0 /dev/__properties__/property_info
	0xebb39000 0xebbf7000    0xbe000        0x0 /system/bin/linker
	0xebbf7000 0xebbfd000     0x6000    0xbd000 /system/bin/linker

Any access out of the range shown above is illegal.

Use below command to save some time:

$ arm-linux-gnueabi-gdb -q -nh										\
	-ex 'set solib-search-path out/target/product/rpi3/symbols'		\
	-ex 'set solib-absolute-prefix out/target/product/rpi3/symbols'	\
	-ex 'core coredump/core.crasher.3007'							\
	out/target/product/rpi3/symbols/system/bin/crasher

If the gdb complain about exec file is newer, please be caution, the source code may changed in anyway, this may lead to parsing issue:

warning: exec file is newer than core file.

Core dump file is actually a ELF file, which means you can inspect it with readelf or objdump.

$ file coredump/core.crasher.3007
coredump/core.crasher.3282: ELF 32-bit LSB core file ARM, version 1 (SYSV), SVR4-style, from 'crasher seccomp', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/system/bin/crasher', platform: 'v8l'

You may also enjoy

Run clang 15 executable on legacy device

1 minute read

For a suspected memory leak issue on a legacy device running OpenWrt build with gcc-8, I need a toolchain with leak sanitizer support, so I try to generate o...