Upstream describes LLDB as a next generation, high-performance debugger.
It is built on top of LLVM/Clang toolchain, and features great
integration with it. At the moment, it primarily supports debugging C,
C++ and ObjC code, and there is interest in extending it to more
languages.
In February 2019, I have started working on LLDB, as contracted by the NetBSD
Foundation. So far I've been working on reenabling continuous
integration, squashing bugs, improving NetBSD core file support,
extending NetBSD's ptrace interface to cover more register
types and fix compat32 issues, fixing watchpoint and threading support,
porting to i386.
During the last month, I've finally managed to create proper reproducers
(and tests) for the remaining concurrent signal delivery problems.
I have started working on backtracing through signal trampolines,
and prepared a libc++ update.
NetBSD concurrent signal updates
While finishing the last report, I was trying to reproduce some
of the concurrent test failures in LLDB with plain ptrace(). I've
finally managed to do that and therefore discover the factor causing
all my earlier attempts to fail — concurrent signal delivery works fine
unless the signal is actually delivered to the process and handled
by it.
Let me explain this a bit. When a signal is delivered to a debugged
process (or one of its threads), it is stopped and the debugger receives
stopping signal via waitpid(). Now, if the debugger wishes the signal
to be delivered to the process (thread), it needs to pass the signal
number as an argument to PT_CONTINUE. If it neglects to do so (passes
0), the signal is discarded.
My tests so far were doing precisely that — discarding the signal.
However, once I modified them to pass it back, they started failing
similarly to how LLDB tests are failing.
Whenever the debugged program receives concurrent signals to different
threads and the debugger requests their delivery, the process is stopped
with some of the signals multiple times. Curiously enough, during my
testing every signal to a thread was reported at least once which means
no signals were lost. I suspect that in an attempt to deliver pending
concurrent signals the kernel is passing them again to the debugger
rather than to the process itself.
I've used this research to extend testing of concurrent behavior.
More specifically, I have:
-
Made signal concurrency test into a reusable factory.
-
Started testing passing signal back to the process.
-
Extended the test to verify that signal is actually being delivered.
-
Included catching newly-created processes in the test.
-
Added concurrent breakpoints to the test.
-
Added concurrent watchpoints to the test.
-
Finally, started testing combination of simultaneous signals, breakpoints and watchpoints.
Research into backtrace through signal trampoline
The most important of the remaining tasks was to enhance LLDB with
NetBSD signal trampoline support.
Signal trampolines on NetBSD
Signal trampolines are shortly covered by Signal delivery chapter
of NetBSD Internals.
When a signal is delivered to a running program, the system needs to
interrupt its execution and run its defined signal handler. Once
the signal handler finishes, the program execution resumes where it left
off. How this is achieved differs from system to system.
On NetBSD, so-called signal trampoline is used. The kernel (this is
done by sendsig_siginfo() e.g. in amd64/machdep.c
function on newer ABIs) saves the program context and executes the
signal handler. When the signal handler returns, it returns to a
trampoline function defined by the libc that restores the saved context
and therefore resumes the program execution.
From debugger's perspective, the backtrace for a process interrupted
in midst of a signal handler ends on this trampoline function. However,
it is often considered useful to be able to know the status
of the process just before the signal was received — and therefore,
the point where program execution will continue. The goal in this point
was to make LLDB aware of NetBSD's trampoline design and capable
of locating and using the saved context to produce full backtrace.
The two possible solutions
There are two approaches to implementing signal trampoline handling:
-
Explicitly detecting and processing signal trampolines in debugger.
-
Adding CFI code to signal trampoline implementation in order to store
the necessary information in libc itself.
GDB on NetBSD is currently using the first approach. The code (found
in
nbsd-tdep.c
and e.g.
amd64-nbsd-tdep.c)
explicitly establishes whether the current frame corresponds to a signal
trampoline, finds the saved context and processes it.
Long-term, the second approach is preferable. Instead of explicitly
writing platform-specific code, we add CFI annotations to the trampoline
code (e.g. in
__sigtramp2.S).
Those annotations are consumed by the toolchain and used to construct
frame information inside the executable that can be afterwards consumed
by the debugger.
Both approaches are therefore roughly equivalent. The main difference
is that approach 1. stores platform-specific logic in the debugger,
while approach 2. stores it in the executable for all debuggers to
consume.
libc++ update
Another task to undergo during this period was to update libc++
in NetBSD src tree. It was last imported in 2015, to the version
roughly corresponding to LLVM 3.7 release. This version is dated
and has some bugs, particularly it is prone to miscompilation due
to undefined behavior (e.g. segfault in
std::map). I've decided
to upgrade to the commit corresponding to the most recent LLVM/Clang update.
max_align_t visibility
The first problem I've hit after upgrading is that max_align_t
is declared on NetBSD only for C11/C++11. However, on NetBSD libc++
is exposing it
unconditionally.
Kamil Rytarowski proposed to expose max_align_t unconditionally
in our headers as well. Joerg Sonnenberger on the other hand wants
to change libc++ instead.
Missing errno constants
Another issue I've found is that NetBSD is missing the two errno
constants for robust mutexes: EOWNERDEAD and ENOTRECOVERABLE.
While libc++ has a hack to redefine them when missing,
it seemed a better idea to assign them on our end.
I've learned that adding errno constants involves a few changes
besides adding new constants:
-
Adding mapping to Linux compat in
sys/compat/linux/common/linux_errno.c.
-
Adding descriptions to manpage
lib/libc/sys/intro.2.
-
Adding messages to
libc catalogs.
-
Enabling appropriate features in libstdc++.
-
Adding new error codes to libdtrace.
-
Adding errno mapping to NFS support in
sys/nfs/nfs_subs.c.
While at it, I've made sure to make it harder to accidentally miss doing
some of that in the future. Notably:
-
I've added ATF tests to make sure that libc catalogs stay in sync
with errno and signal descriptions in code.
-
I've added a script to autogenerate libdtrace errno lists.
-
I've added a compile-time assertion that NFS errno mapping covers
all values.
The complete list of commits:
-
Sync errno messages between catalog and errno.h
-
Sync signal messages between catalog and sys_siglist
-
Add tests for missing libc catalog entries
-
PR standards/44921: Add errno consts for robust mutexes
-
Enable EOWNERDEAD & ENOTRECOVERABLE in libstdc++
-
Update dtrace errno.d mapping and add a script for it
-
Update NFS errno mapping and add assert for correctness
The update
I have sent libc++ update to 01f3a59fb3e2542fce74c768718f594d0debd0da
to the mailing list for review. The proposed patch set includes:
-
Adjust the cleanup script for the new version.
-
Cleaning up extraneous files from the old import (to make the diff
clearer).
-
Importing the new version and updating Makefiles.
-
Moving headers to standard /usr/include/c++/v1 location for better
interoperability.
-
Moving libc++ to apache2 license group.
Future plans
This is the final month of my contract and therefore I would like
to primarily focus on importing LLDB into src tree. As time permits,
I will continue attempting to improve support for backtracing through
signal trampolines.
The exact list of remaining tasks in my contract follows:
-
Add support to backtrace through signal trampoline and extend the support to
libexecinfo, unwind implementations (LLVM, nongnu). Examine adding CFI
support to interfaces that need it to provide more stable backtraces (both
kernel and userland).
-
Add support for aarch64 target.
-
Stabilize LLDB and address breaking tests from the test suite.
-
Merge LLDB with the base system (under LLVM-style distribution).
This work is sponsored by The NetBSD Foundation
The NetBSD Foundation is a non-profit organization and welcomes any
donations to help us continue funding projects and services
to the open-source community. Please consider visiting the following URL
to chip in what you can:
https://netbsd.org/donations/#how-to-donate
