Android performance monitoring with bcc

Build kernel with bpf support

In order to use bpf, kernel should be at least build with these options:

+CONFIG_BPF=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_BPF_EVENTS=y
+CONFIG_HAVE_EBPF_JIT=y

Then build and update new kernel image with:

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- lineageos_rpi3b_defconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j16
mount -t vfat /dev/block/mmcblk0p1 /data/boot/
adb push arch/arm64/boot/Image /data/boot/kernel8.img

Install bcc with adeb

bcc is not supported by android, adeb is designed for Android to use bcc for kernel tracing.

The first thing you need to do in order to use bcc with android is to prepare the host environment, grab adeb from github and make a symlink:

sudo apt-get install qemu-user-static debootstrap
git clone --depth=1 https://github.com/joelagnel/adeb.git
sudo ln -s $(pwd)/adeb /usr/bin/adeb

Then download the latest release (about 288 MB) with script adeb, enable debug mode with --debug option if needed:

# Download adeb release from github
adeb prepare --full

# For debugging
adeb prepare --full --debug

This will involve downloading tarball from web, push it to the target and then unpacking file system in device. After installation, you can do adeb shell to confirm if it woks, it will prompt:

$ adeb shell

##########################################################
# Welcome to androdeb environment running on Android!    #
# Questions to: Joel Fernandes <joel@joelfernandes.org>  #
                                                         #
 Try running vim, gcc, clang, git, make, perf, filetop   #
  ..etc or apt-get install something.                    #
##########################################################

root@localhost:/#

Make sure to replace tar with busybox applet, as tar in the release package was broken (see trouble shooting section):

root@localhost:/usr/bin# rm /usr/bin/tar
root@localhost:/usr/bin# ln -s /vendor/bin/busybox /usr/bin/tar

Build and make archive for later use

The bcc in release is a little bit old, at the time of this writing, klockstat is not included, so if you want updated bcc toolkit, the best way is to build it from scratch:

$ adeb prepare --full --buildtar /opt/software/adeb/arm64/

This will install adeb to target board and generate an archive after installation.

Make use of prebuilt archive

If you have made your customized adeb rootfs or already have adeb release downloaded you can use them:

adeb prepare --archive /opt/software/adeb/arm64/androdeb-fs.tgz

DO NOT USE androdeb-fs.tgz.zip, androdeb script has issue with this format.

In kernel header support

Build bcc need kernel headers in target file system, joelagnel, the author of adeb submitted a patch titled Provide in-kernel headers to make extending kernel easier, and it was merged into mainline kernel at rc3 of v5.1.0, this patch puts the in-kernel header into procfs, now it was moved to sysfs at /sys/kernel/kheaders.tar.xz

Backport those two patches to current kernel then rebuild it with CONFIG_IKHEADERS selected to support in-kernel headers support.

At this point, bcc should work, summarize hard irq as an example:

root@localhost:/# hardirqs 10
Tracing hard irq event time... Hit Ctrl-C to end.

HARDIRQ                    TOTAL_usecs
3f00b880.mailbox                   113
dwc_otg                         150538
dwc_otg_sim-fiq                 921158

If hardirqs report above output, then it’s time to do more exploring about bcc.

BCC tools

BCC (BPF Compiler Collection) is a toolkit for kernel tracing, as the name implies, bcc provides a bunch of tools and examples, from top level application layer to the kernel level such as scheduler, virtual memory, file system etc, this picture shows all the tools that are currently included in bcc:

tools	examples
argdist	argdist
biolatency	biolatency
biotop	biotop
biosnoop	biosnoop
bitesize	bitesize
bpflist	bpflist
capable	capable
cachestat	cachestat
cachetop	cachetop
cpudist	cpudist
cpuunclaimed	cpuunclaimed
criticalstat	criticalstat
dcsnoop	dcsnoop
dcstat	dcstat
deadlock	deadlock
drsnoop	drsnoop
execsnoop	execsnoop
exitsnoop	exitsnoop
ext4dist	ext4dist
ext4slower	ext4slower
filelife	filelife
fileslower	fileslower
filetop	filetop
funccount	funccount
funclatency	funclatency
funcslower	funcslower
hardirqs	hardirqs
killsnoop	killsnoop
memleak	memleak
offcputime	offcputime
offwaketime	offwaketime
oomkill	oomkill
opensnoop	opensnoop
pidpersec	pidpersec
profile	profile
reset-trace	reset-trace
runqlen	runqlen
slabratetop	slabratetop
softirqs	softirqs
stackcount	stackcount
syncsnoop	syncsnoop
tplist	tplist
trace	trace
vfscount	vfscount
vfsstat	vfsstat
wakeuptime	wakeuptime

There are tutorials for both bcc users and developers in official repo.

If you are interested in development of bcc, refer to bcc Reference Guide for more detail.

Trouble shooting

Unable to find kernel headers

root@localhost:/# hardirqs 10
sh: modprobe: command not found
Unable to find kernel headers. Try rebuilding kernel with CONFIG_IKHEADERS=m (module)
chdir(/lib/modules/4.14.135-v8+/build): No such file or directory
Traceback (most recent call last):
  File "/usr/share/bcc/tools/hardirqs", line 145, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 343, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

A: backport these two patches:

• Provide in-kernel headers to make extending kernel easier
• kheaders: Move from proc to sysfs

and make sure it was built into kernel:

root@localhost:/# zcat /proc/config.gz | grep CONFIG_IKHEADERS

tar is broken

root@localhost:/# hardirqs 10
tar (child): xz: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Unable to find kernel headers. Try rebuilding kernel with CONFIG_IKHEADERS=m (module)
chdir(/lib/modules/4.14.135-v8+/build): No such file or directory
Traceback (most recent call last):
  File "/usr/share/bcc/tools/hardirqs", line 145, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 343, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

A: remove original tar in /usr/bin/ and make a symlink to busybox

root@localhost:/usr/bin# rm /usr/bin/tar
root@localhost:/usr/bin# ln -s /vendor/bin/busybox /usr/bin/tar

header file missing

root@localhost:/# hardirqs 10
In file included from <built-in>:2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
include/uapi/linux/types.h:5:10: fatal error: 'asm/types.h' file not found
#include <asm/types.h>
         ^~~~~~~~~~~~~
1 error generated.
Traceback (most recent call last):
  File "/usr/share/bcc/tools/hardirqs", line 145, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 343, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

A: export ARCH=arm64

required tracing events are not available

root@localhost:/# criticalstat
ERROR: required tracing events are not available
Make sure the kernel is built with CONFIG_DEBUG_PREEMPT and CONFIG_PREEMPTIRQ_EVENTS enabled. Also please disable CONFIG_PROVE_LOCKING and CONFIG_LOCKDEP on older kernels.

A: Enable DEBUG_PREEMPT and PREEMPTIRQ_EVENTS

References

• A debian-based shell environment designed for Android and adb
• BPF Compiler Collection (BCC)
• BPF Features by Linux Kernel Version