Monitoring Embedded Devices with Munin

Recently I was debugging a system hang problem on arm board, I reproduced this issue once, and noticed when this occurred, the interrupts on can interface is incredibly high, more than 2 billion interrupts, and the sender only have around 5 million times, and find the following message in kernel log:

[sched_delayed] sched: RT throttling activated

I don’t know why this happens, but the cpu usage might be very high, so monitoring cpu and interrupts may give some hints.

Munin is a resource monitoring tool that can be used to analyze resource trends, it includes Munin server and the devices that to be monitored, the Munin server is called master, can be run on Ubuntu, and the devices is called nodes.

A server can monitor many nodes at the same time.

Munin master

In my case, Munin server will be installed on Armbian, and configured using below commands:

sudo apt install -y munin apache2 libcgi-fast-perl libapache2-mod-fcgid

sudo sed -i 's/#dbdir/dbdir/g' /etc/munin/munin.conf
sudo sed -i 's/#htmldir/htmldir/g' /etc/munin/munin.conf
sudo sed -i 's/#logdir/logdir/g' /etc/munin/munin.conf
sudo sed -i 's/#rundir/rundir/g' /etc/munin/munin.conf
sudo sed -i 's/#tmpldir/tmpldir/g' /etc/munin/munin.conf

sudo systemctl reload apache2
sudo systemctl enable munin
sudo systemctl start munin

Don’t worry if you see the Munin service is active (exited), it is OK:

$ sudo systemctl status munin
● munin.service - LSB: Create munin master directories on boot
     Loaded: loaded (/etc/init.d/munin; generated)
     Active: active (exited) since Sat 2021-05-01 02:18:09 UTC; 1min 8s ago
       Docs: man:systemd-sysv-generator(8)
    Process: 27871 ExecStart=/etc/init.d/munin start (code=exited, status=0/SUCCESS)

In order to access from other machine other than localhost, the ip address should be granted:

# For Armbian
sudo sed -i '/^    Require local*/a \    Require ip 192.168.0.8' /etc/munin/apache24.conf
# For Ubuntu
sudo sed -i '/^        Require local*/a \        Require ip 192.168.0.8' /etc/munin/apache24.conf
sudo systemctl restart apache2

You can view the Munin node at http://192.168.0.20/munin

Munin node

Install Munin Node on ARM Devices

Munin was written in perl, so it cannot be installed to arm board directly, there is a c implementation of munin node, cross compile it and copy munin-inetd-c and munin-node-c to the target board:

$ git clone --depth 1 https://github.com/munin-monitoring/munin-c.git
$ autoreconf -i -I m4
$ ./configure --host=arm-linux
$ make

Create symlinks for the plugins supported by munin-c:

plugindir=/usr/local/etc/munin/plugins
mkdir -p ${plugindir}
for f in acpi cpu cpuspeed df df_inode diskstats entropy forks fw_packets if_err_eth0 if_err_ip6tnl0 if_eth0 if_ip6tnl0 interrupts irqstats load memory netstat nfs4_client nfs_client nfsd nfsd4 open_files open_inodes processes proc_pri swap threads uptime users vmstat
do
echo "${f}"
ln -s /sbin/munin-plugins-c "${plugindir}"/"${f}"
done

In order to collect metrics from munin node, an entry with its IP address need to be added to munin.conf:

cat << EOF | sudo tee -a /etc/munin/munin.conf

[F33]
    address 192.168.0.33
    use_node_name yes
EOF
sudo systemctl start munin

Now run munin-node and wait for several minutes to see the results:

# munin-inetd-c 4949 /sbin/munin-node-c &

We can see the log files are saved in /var/lib/munin in rrd format:

tail -f /var/log/munin/munin-update.log
[...]
2021/05/03 20:22:20 [INFO] creating rrd-file for irqstats->can0: '/var/lib/munin/F33/F33-irqstats-can0-d.rrd'
[...]

You can find the screenshots of monitored munin node at the end of this post.

Armbian Munin Node

It’s pretty easy to setup munin node for Armbian system:

sudo apt install -y munin-node munin-plugins-extra
sudo sed -i '/^allow \^::1$*/a allow \^192\\.168\\.0\\.3$' /etc/munin/munin-node.conf
sudo sed -i 's/#host_name localhost.localdomain/host_name phicomm-N1/g' /etc/munin/munin-node.conf
sudo service munin-node restart

Remove All History

In case you want to remove all the history, just do the following should be suffice:

sudo rm -fr /var/cache/munin/www
sudo rm -fr /var/lib/munin
sudo mkdir -p /var/lib/munin/cgi-tmp/munin-cgi-graph
sudo chown -R munin:munin /var/lib/munin
sudo chown -R munin:www-data /var/lib/munin/cgi-tmp/
sudo chmod g+w /var/lib/munin/cgi-tmp/munin-cgi-graph

sudo mkdir -p /var/cache/munin/www
sudo chown -R munin:munin /var/cache/munin/www

Troubleshooting

Failed to enable unit: Unit file munin.service is masked.

I find this issue when trying to install Munin master on Ubuntu 18.04, it failed with below messages:

$ sudo systemctl enable munin
Synchronizing state of munin.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable munin
Failed to enable unit: Unit file /lib/systemd/system/munin.service is masked.

And find out that the service file is linked to /dev/null:

$ file /lib/systemd/system/munin.service
/lib/systemd/system/munin.service: symbolic link to /dev/null

Remove it and create a new service like this:

$ sudo rm /lib/systemd/system/munin.service
$ cat << EOF | sudo tee -a /lib/systemd/system/munin.service

[Unit]
Description=Munin service
Documentation=man:munin-cron(8)

[Service]
User=munin
Type=oneshot
ExecStart=/usr/bin/munin-cron
EOF
$ sudo systemctl start munin

Munin Screenshots