Hello everyone, how's it going? 🙂

In this post I'd like to explain and teach how to enable debugging of system components, Click apps as well as Snap packages. This is intended for developers choosing to contribute to the Operating System's development, wanting to fix a bug in their Click apps, or release Ubuntu Touch apps on the Snap Store for both Ubuntu Touch and all Snap-supporting Linux distributions.

Prerequisites

The Snap Store offers a static version of gdb which we're going to make use of for this setup:

sudo snap install gdb-static

Make sure to add this snippet to your ~/.bashrc file:

export PATH="/snap/gdb-static/current/usr/bin:$PATH"

After logging into a shell you are now able to run the Snap-provided gdb and gdbserver commands directly, but they don't yet work when using sudo with short-hand commands like sudo gdb.

Running gdb & strace with sudo

Since these versions of debugging tools don't sit in a PATH that is allowed for sudo-invoked commands, we will have to teach sudo to set them in its default PATH without breaking the Operating System's expectations of the read-only rootfs. So to make this work, we will have to create an underlay for /etc/sudoers.d which configures sudo to keep our tools paths intact.

First let's create an underlay containing our sudoers addition:

sudo mkdir -p /userdata/custom/overlays/etc/sudoers.d
cat - | sudo tee /userdata/custom/overlays/etc/sudoers.d/gdb-strace-path <<EOF
Defaults        secure_path="/snap/gdb-static/current/usr/bin:/snap/strace-static/current/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
EOF
sudo chmod 440 /userdata/custom/overlays/etc/sudoers.d/gdb-strace-path
sudo chmod 550 /userdata/custom/overlays/etc/sudoers.d

This effectively configures sudo to set a custom PATH to gdb, gdbserver and strace (described later) when executing i.e sudo gdb or when snap run is launched with the --gdbserver argument.

Next create a systemd mount unit to effectively apply this underlay immediately and enable on boot:

cat - | sudo tee /etc/systemd/system/etc-sudoers.d.mount <<EOF
[Unit]
Wants=userdata.mount
After=userdata.mount
Before=local-fs.target

[Mount]
What=etc-sudoers.d-underlay
Where=/etc/sudoers.d
Type=overlay
Options=ro,lowerdir=/etc/sudoers.d:/userdata/custom/overlays/etc/sudoers.d

[Install]
WantedBy=local-fs.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now etc-sudoers.d.mount

This underlay is not modifying the immutable file system and is secure against Operating System updates changing the underlying files and defaults other than what we adapted.

Debugging your app

You're now able to run any command with the gdb command and attaching to your desired application's process, simply using sudo gdb -p <PID-of-the-process>, whether it being a system process or Click app.

You can also safely debug your Snaps using built-in commands. Let's take the wasted Snap as an example:

snap run --gdbserver wasted

This will prompt you to 1) enter your sudo password and 2) to execute the outlined gdb command in a separate terminal:

phablet@fairphone5:~$ snap run --gdbserver wasted

Welcome to "snap run --gdbserver".
You are right before your application is run.
Please open a different terminal and run:

gdb -ex="target remote :42957" -ex=continue -ex="signal SIGCONT"
(gdb) continue

or use your favorite gdb frontend and connect to :42957

Running the mentioned command and typing cont in the resulting gdb session will launch the debugging process. You are also free to connect to the server's TCP port it mentions from your development PC, allowing you to comfortably and remotely debug the app.

Tip: If your app receives SIGSTOP signals frequently during start-up, filter those signals out of process management:

(gdb) handle SIGSTOP nostop

syscall tracing

The Snap Store also has strace-static in store for debugging a process' use of syscalls at runtime. Since we've already set up the strace-static PATH in the /etc/sudoers.d underlay we can proceed with the installation:

sudo snap install strace-static

If you wish to use this version of strace over the system default one, add this to your ~/.bashrc:

export PATH="/snap/strace-static/current/bin:$PATH"

You can now simply run a command with strace prefixed, or attach to all running process' threads and children using sudo strace -p <PID-of-the-process> -ff. All the regularly expected strace command arguments apply.

On the Snap side, with a simple snap run --strace wasted we're able to capture each syscall the Snap makes to the kernel. snap run --strace wasted 2>strace.log will save the captured trace to strace.log for further inspection.

How does this work?

Unlike an apt-installed version of gdb, this one is an all-it-can-support static build of gdb into one file, same as strace-static for strace. This means those can be executed on any Linux OS, regardless of dependencies provided by your host's OS. But since these aren't dynamically linked binaries they cannot be extended by various dependencies, especially because in this case the required sources currently expect to be compiled & linked dynamically.

As a result of all this gdb-static lacks debuginfod support for debug symbol retrieval because libdebuginfod upstream still requires yet-to-be-seen changes to build statically.

Memory leak detection using Valgrind

Setting up Valgrind is also possible, albeit with a bit of tweaking and manual setup. It requires an additional underlay below /usr/lib which places debug symbols from a user-writable location into the missing /usr/lib/debug path.

First off, install Valgrind:

sudo snap install --classic valgrind

No additional setup for running as root needed, but running Valgrind at this stage will result in an error stemming from a missing debug symbol package, libc6-dbg. It must be fetched and extracted by hand.

We require an up-to-date apt cache for that. To prepare a temporary apt environment which will get reset during next reboot:

sudo mount -t tmpfs apt-cache /var/cache/apt
sudo mount -t tmpfs apt-lib /var/lib/apt
sudo apt update

Then download and extract the contents of the libc6-dbg package on top of our custom path, for Valgrind to launch properly:

apt download libc6-dbg
sudo dpkg -x ./libc6-dbg_*.deb /userdata/custom/overlays/

Now on to the /usr/lib systemd mount unit. Extra precaution has been made to not make user-provided files override system ones, as /usr/lib usually contains crucial OS files.

cat - | sudo tee /etc/systemd/system/usr-lib.mount <<EOF
[Unit]
Wants=userdata.mount
After=userdata.mount
Before=local-fs.target

[Mount]
What=usr-lib-underlay
Where=/usr/lib
Type=overlay
Options=ro,lowerdir=/usr/lib:/userdata/custom/overlays/usr/lib

[Install]
WantedBy=local-fs.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now usr-lib.mount

You're all set now! Executing valgrind with your desired shell command or a binary should now be able to spawn a process with memory leak detection running.

Extending discovered debug symbols

The newly created path /userdata/custom/overlays/usr/lib/debug is where your custom debug symbol files will be placed for the tools to find them, thanks to the usr-lib.mount unit underlaying it's contents underneath the file system. This way you can extend the gdb debugger's stack frame details and stack trace accuracy.

To add further debug symbols you'll have to apt download a dbgsym .ddeb and extract the full package using sudo dpkg -x into /userdata/custom/overlays:

apt download qtubuntu-qt6-dbgsym
sudo dpkg -x ./qtubuntu-qt6-dbgsym_*.ddeb /userdata/custom/overlays

With Ubuntu Touch's apt repository defaults we get all dbgsym packages built by UBports CI presented to us, ready to be fetched and used in our debugging sessions.

Final words

You did it! You have successfully set up your Ubuntu Touch for debugging purposes without sacrificing on the system's immutability, OTA stability guarantee, or ease of use.

We just had to override the system-default PATH for regular sudo commands in a way that would survive any OTA coming in, and without touching our beloved immutable file system. For Valgrind and extending gdb's functionality a new directory to contain debug symbols ensures you'll find your bug with less of a hassle.

Congratulations, you now know how to debug on Ubuntu Touch. 🙂