Enable pstore for kernel oops/panic dumping
Debugging devices without serial console is big challenge especially for kernel panic issues, pstore (persistent store) writes kernel messages before system crashes, which is very helpful for kernel debugging. In order to use this feature a reserved memory block is required, and several parameters needs to be configured, there are three ways to do this which is documented here, I pick the second one and added these to device tree blob for Raspberry Pi board:
diff --git a/arch/arm/boot/dts/bcm2837.dtsi b/arch/arm/boot/dts/bcm2837.dtsi
index 9e95fee78..15df73e9d 100644
--- a/arch/arm/boot/dts/bcm2837.dtsi
+++ b/arch/arm/boot/dts/bcm2837.dtsi
@@ -18,6 +18,20 @@
};
};
+ reserved-memory {
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges;
+
+ ramoops@32000000 {
+ compatible = "ramoops";
+ reg = <0x32000000 0x00100000>;
+ record-size = <0x00020000>;
+ console-size = <0x00020000>;
+ ftrace-size = <0x00020000>;
+ };
+ };
+
arm-pmu {
compatible = "arm,cortex-a53-pmu";
interrupt-parent = <&local_intc>;
The meaning of these size paramters can be found here:
- record-size: maximum size in bytes of each kmsg dump.
- console-size: size in bytes of log buffer reserved for kernel messages
- ftrace-size: size in bytes of log buffer reserved for function tracing
- pmsg-size: size in bytes of log buffer reserved for userspace messages
Build kernel with these options enabled:
CONFIG_PSTORE=y
CONFIG_PSTORE_CONSOLE=y
CONFIG_PSTORE_FTRACE=y
CONFIG_PSTORE_RAM=y
Put kernel image and arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b-plus.dtbto
boot partition then boot the Pi, during boot up, you can see pstore is enabled,
and ramoops is attached:
rpi3:/ # dmesg |grep -E 'ramoops|pstore'
[ 0.708678] console [pstore-1] enabled
[ 0.709367] pstore: Registered ramoops as persistent store backend
[ 0.709524] ramoops: attached 0x100000@0x32000000, ecc: 0/0
[ 6.257516] pstore: Using compression: deflate
After system is up, fake a panic with sysrq-trigger:
echo 5 > /proc/sys/kernel/panic
echo c > /proc/sysrq-trigger
System will reboot in 5 seconds, and panic logs were saved in `/sys/fs/pstore:
rpi3:/ # ls -l /sys/fs/pstore/
total 0
-r--r----- 1 system log 58326 1969-12-31 13:00 console-ramoops-0
-r--r--r-- 1 root root 61349 2020-08-03 00:13 dmesg-ramoops-0
-r--r--r-- 1 root root 61609 2020-08-03 00:13 dmesg-ramoops-1
cat /sys/fs/pstore/dmesg-ramoops-0