Find Out Who Deleted a File

Previously I wrote a post about how to use ftrace via trace-cmd, now here comes a real world example showing how to find who deleted a file in Android system.

Install trace-cmd

First clone trace-cmd with:

git clone --depth=1 https://github.com/rostedt/trace-cmd.git

Then make target trace-cmd:

make LDFLAGS=-static CC=arm-linux-gnueabi-gcc trace-cmd

or

make LDFLAGS=-static CC=aarch64-linux-gnu-gcc trace-cmd

Make sure trace-cmd is statically linked:

fdbai@fdbai-desktop:~/perf/trace-cmd$ file tracecmd/trace-cmd
tracecmd/trace-cmd: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=e307879b33992a3d0c0b9552cd9049d8e9c31ac0, not stripped

Tracing ext4 activity

Firstly, find out what traces are supported by ext4 file system:

rpi3:/ # trace-cmd list |grep ext4
ext4:ext4_getfsmap_mapping
ext4:ext4_getfsmap_high_key
ext4:ext4_getfsmap_low_key
ext4:ext4_fsmap_mapping
ext4:ext4_fsmap_high_key
ext4:ext4_fsmap_low_key
ext4:ext4_es_shrink
ext4:ext4_insert_range
ext4:ext4_collapse_range
ext4:ext4_es_shrink_scan_exit
ext4:ext4_es_shrink_scan_enter
ext4:ext4_es_shrink_count
ext4:ext4_es_lookup_extent_exit
ext4:ext4_es_lookup_extent_enter
ext4:ext4_es_find_delayed_extent_range_exit
ext4:ext4_es_find_delayed_extent_range_enter
ext4:ext4_es_remove_extent
ext4:ext4_es_cache_extent
ext4:ext4_es_insert_extent
ext4:ext4_ext_remove_space_done
ext4:ext4_ext_remove_space
ext4:ext4_ext_rm_idx
ext4:ext4_ext_rm_leaf
ext4:ext4_remove_blocks
ext4:ext4_ext_show_extent
ext4:ext4_get_reserved_cluster_alloc
ext4:ext4_find_delalloc_range
ext4:ext4_ext_in_cache
ext4:ext4_ext_put_in_cache
ext4:ext4_get_implied_cluster_alloc_exit
ext4:ext4_ext_handle_unwritten_extents
ext4:ext4_trim_all_free
ext4:ext4_trim_extent
ext4:ext4_journal_start_reserved
ext4:ext4_journal_start
ext4:ext4_load_inode
ext4:ext4_ext_load_extent
ext4:ext4_ind_map_blocks_exit
ext4:ext4_ext_map_blocks_exit
ext4:ext4_ind_map_blocks_enter
ext4:ext4_ext_map_blocks_enter
ext4:ext4_ext_convert_to_initialized_fastpath
ext4:ext4_ext_convert_to_initialized_enter
ext4:ext4_truncate_exit
ext4:ext4_truncate_enter
ext4:ext4_unlink_exit
ext4:ext4_unlink_enter
ext4:ext4_fallocate_exit
ext4:ext4_zero_range
ext4:ext4_punch_hole
ext4:ext4_fallocate_enter
ext4:ext4_direct_IO_exit
ext4:ext4_direct_IO_enter
ext4:ext4_load_inode_bitmap
ext4:ext4_read_block_bitmap_load
ext4:ext4_mb_buddy_bitmap_load
ext4:ext4_mb_bitmap_load
ext4:ext4_da_release_space
ext4:ext4_da_reserve_space
ext4:ext4_da_update_reserve_space
ext4:ext4_forget
ext4:ext4_mballoc_free
ext4:ext4_mballoc_discard
ext4:ext4_mballoc_prealloc
ext4:ext4_mballoc_alloc
ext4:ext4_alloc_da_blocks
ext4:ext4_sync_fs
ext4:ext4_sync_file_exit
ext4:ext4_sync_file_enter
ext4:ext4_free_blocks
ext4:ext4_allocate_blocks
ext4:ext4_request_blocks
ext4:ext4_mb_discard_preallocations
ext4:ext4_discard_preallocations
ext4:ext4_mb_release_group_pa
ext4:ext4_mb_release_inode_pa
ext4:ext4_mb_new_group_pa
ext4:ext4_mb_new_inode_pa
ext4:ext4_discard_blocks
ext4:ext4_journalled_invalidatepage
ext4:ext4_invalidatepage
ext4:ext4_releasepage
ext4:ext4_readpage
ext4:ext4_writepage
ext4:ext4_writepages_result
ext4:ext4_da_write_pages_extent
ext4:ext4_da_write_pages
ext4:ext4_writepages
ext4:ext4_da_write_end
ext4:ext4_journalled_write_end
ext4:ext4_write_end
ext4:ext4_da_write_begin
ext4:ext4_write_begin
ext4:ext4_begin_ordered_truncate
ext4:ext4_mark_inode_dirty
ext4:ext4_drop_inode
ext4:ext4_evict_inode
ext4:ext4_allocate_inode
ext4:ext4_request_inode
ext4:ext4_free_inode
ext4:ext4_other_inode_update_time

Secondly, use ls -i <filename> to confirm inode number, sample output of busybox inode number:

rpi3:/data # ls -i /system/bin/busybox
2044 /system/bin/busybox

Then trace ext4 with below command in background:

rpi3:/ # trace-cmd record -e ext4:ext4_unlink_enter -e ext4:ext4_unlink_exit -o /data/trace.dat&

After seeing Hit Ctrl^C to stop recording, do some test to reproduce issue formerly occurred, after the issue reproduced, stop by hitting Ctrl^C to stop, then use trace-cmd report to generate final report.

Finally, search inode number in the trace report to find which process do the wrong thing.

     rm-23190 [000]  2440.362945: ext4_unlink_enter:    dev 179,4 ino 108 size 0 parent 2
     rm-23190 [000]  2440.363097: ext4_unlink_exit:     dev 179,4 ino 108 ret 0