Upstream describes LLDB as a next generation, high-performance debugger.
It is built on top of LLVM/Clang toolchain, and features great
integration with it. At the moment, it primarily supports debugging C,
C++ and ObjC code, and there is interest in extending it to more
languages.
In February, I have started working on LLDB, as contracted by the NetBSD
Foundation. So far I've been working on reenabling continuous
integration, squashing bugs, improving NetBSD core file support,
extending NetBSD's ptrace interface to cover more register
types and fix compat32 issues and fixing watchpoint support. Then,
I've started working on improving thread support. You can read more
about that in my July 2019
report.
I've been on vacation in August, and in September I've resumed the work
on LLDB. I've started by fixing new regressions in LLVM suite, then
improved my previous patches and continued debugging test failures
and timeouts resulting from my patches.
LLVM 8 and 9 in NetBSD
Updates to LLVM 8 src branch
I have been asked to rebase my llvm8 branch
of NetBSD src tree. I've done that, and updated it to LLVM 8.0.1 while
at it.
LLVM 9 release
The LLVM 9.0.0 final has been tagged in September. I have been doing
the pre-release testing for it, and discovered that the following tests
were hanging:
LLVM :: ExecutionEngine/MCJIT/eh-lg-pic.ll
LLVM :: ExecutionEngine/MCJIT/eh.ll
LLVM :: ExecutionEngine/MCJIT/multi-module-eh-a.ll
LLVM :: ExecutionEngine/OrcMCJIT/eh-lg-pic.ll
LLVM :: ExecutionEngine/OrcMCJIT/eh.ll
LLVM :: ExecutionEngine/OrcMCJIT/multi-module-eh-a.ll
I couldn't reproduce the problem with LLVM trunk, so I've instead
focused on looking for a fix. I've came to the conclusion that
the problem was fixed through adding missing linked library. I've
requested backport in bug 43196
and it has been merged in r371042.
I didn't put more effort into figuring out why the lack of this
linkage caused issues for us. However, as Lang Hames said on the bug,
‘adding the dependency was the right thing to do’.
LLVM 9 for NetBSD src
Afterwards, I have started working on updating my NetBSD src branch
for LLVM 9. However, in middle of that I've been informed that Joerg
has already finished doing that independently, so I've stopped.
Furthermore, I was informed that LLVM 9.0.0 will not make it to src,
since it still lacks some fixes (most notably, adding a pass to lower
is.constant and objectsize intrinsics).
Joerg plans to import some revision of the trunk instead.
Buildbot regressions
Initial regressions
The first problem that needed solving was LLDB build failure caused by
replacing std::once_flag with llvm::once_flag.
I've came to the conclusion that the build fails because the call site
in LLDB combined std::call_once with llvm::once_flag. The solution
was to replace the former with llvm::call_once.
After fixing the build failure, we had a bunch of test failures on buildbot
to address. Kamil helped me and tracked one of them down to a new test
for stack exhaustion handling.
The test author decided that it ‘is only a best-effort mitigation for
the case where things have already gone wrong’, and marked it unsupported
on NetBSD.
On the plus side, two of the tests previously failing on NetBSD have been
fixed upstream. I've un-XFAIL-ed them appropriately.
Five new test failures in LLDB were related to those tests being
unconditionally skipped before — I've marked them XFAIL
pending further investigation in the future.
Another set of issues was caused by enabling -fvisibility=hidden for libc++
which caused problems when building with GCC. After being pinged, the author
decided to enable it only for builds done using clang.
New issues through September
During September, two new issues arose. The first one was my fault,
so I'm going to cover it in appropriate section below. The second one
was new thread_local test failing. Since it was a newly added test
that failed on most of the supported platforms, I've just added NetBSD
to the list of failing platforms.
Current buildbot status
After fixing the immediate issues, the buildbot returned to previous
status. The majority of tests pass, with one flaky test repeatedly
timing out. Normally, I would skip this specific test in order to
have buildbot report only fresh failures. However, since it is
threading-related I'm waiting to finish my threading update and reassert
afterwards.
Furthermore, I have added --shuffle to lit arguments in order to randomize
the order in which the tests are run. According to upstream, this reduces
the chance of load-intensive tests being run simultaneously and therefore
causing timeouts.
The buildbot host seems to have started crashing recently. OpenMP tests were
causing similar issues in the past, and I'm currently trying to figure out
whether they are the culprit again.
__has_feature(leak_sanitizer)
Kamil asked me to implement a feature check for leak sanitizer being
used. The __has_feature(leak_sanitizer) preprocessor macro
is complementary to __SANITIZE_LEAK__ used in NetBSD gcc and is used to avoid
reports when leaks are known but the cost of fixing them exceeds the gain.
Progress in threading support
Fixing LLDB bugs
In the course of previous work, I had a patch for threading support in LLDB
partially ready. However, the improvements have also resulted in some of
the tests starting to hang. The main focus of my late work as investigating
those problems.
The first issue that I've discovered was inconsistency in expressing no signal
sent. In some places, LLDB used LLDB_INVALID_SIGNAL (-1) to express that,
in others it used 0. So far this went unnoticed since the end result
in ptrace calls was the same. However, the reworked NetBSD threading support
used explicit PT_SET_SIGINFO which — combined with wrong signal parameter —
wiped previously queued signal.
I've fixed C packet handler,
then fixed c, vCont and s handlers
to use LLDB_INVALID_SIGNAL correctly. However, I've only tested the fixes
with my updated thread support, causing regression in the old code. Therefore,
I've also had to fix LLDB_INVALID_SIGNAL handling in NetBSD plugin for the time being.
Thread suspend/resume kernel problem
Sadly, further investigation of hanging tests led me to the conclusion
that they are caused by kernel bugs. The first bug I've noticed is that
PT_SUSPEND/PT_RESUME do not cause the thread to be resumed correctly.
I have written the following reproducer for it:
#include <assert.h>
#include <lwp.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
void* thread_func(void* foo) {
int i;
printf("in thread_func, lwp = %d\n", _lwp_self());
for (i = 0; i < 100; ++i) {
printf("t2 %d\n", i);
sleep(2);
}
printf("out thread_func\n");
return NULL;
}
int main() {
int ret;
int pid = fork();
assert(pid != -1);
if (pid == 0) {
int i;
pthread_t t2;
ret = ptrace(PT_TRACE_ME, 0, NULL, 0);
assert(ret != -1);
printf("in main, lwp = %d\n", _lwp_self());
ret = pthread_create(&t2, NULL, thread_func, NULL);
assert(ret == 0);
printf("thread started\n");
for (i = 0; i < 100; ++i) {
printf("t1 %d\n", i);
sleep(2);
}
ret = pthread_join(t2, NULL);
assert(ret == 0);
printf("thread joined\n");
}
sleep(1);
ret = kill(pid, SIGSTOP);
assert(ret == 0);
printf("stopped\n");
pid_t waited = waitpid(pid, &ret, 0);
assert(waited == pid);
printf("wait: %d\n", ret);
printf("t2 suspend\n");
ret = ptrace(PT_SUSPEND, pid, NULL, 2);
assert(ret == 0);
ret = ptrace(PT_CONTINUE, pid, (void*)1, 0);
assert(ret == 0);
sleep(3);
ret = kill(pid, SIGSTOP);
assert(ret == 0);
printf("stopped\n");
waited = waitpid(pid, &ret, 0);
assert(waited == pid);
printf("wait: %d\n", ret);
printf("t2 resume\n");
ret = ptrace(PT_RESUME, pid, NULL, 2);
assert(ret == 0);
ret = ptrace(PT_CONTINUE, pid, (void*)1, 0);
assert(ret == 0);
sleep(5);
ret = kill(pid, SIGTERM);
assert(ret == 0);
waited = waitpid(pid, &ret, 0);
assert(waited == pid);
printf("wait: %d\n", ret);
return 0;
}
The program should run a two-threaded subprocess, with both threads
outputting successive numbers. The second thread should be suspended
shortly, then resumed. However, currently it does not resume.
I believe that this caused by ptrace_startstop()
altering process flags without reimplementing the complete logic as used
by lwp_suspend()
and lwp_continue().
I've been able to move forward by calling the two latter functions
from ptrace_startstop(). However, Kamil has indicated that he'd like
to make those routines use separate bits (to distinguish LWPs stopped
by process from LWPs stopped by debugger), so I haven't pushed my patch
forward.
Multiple thread reporting kernel problem
The second and more important problem is related to how new LWPs are
reported to the debugger. Or rather, that they are not reported
reliably. When many threads are started by the process in a short time
(e.g. in a loop), the debugger receives reports only for some of them.
This can be reproduced using the following program:
#include <assert.h>
#include <lwp.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
void* thread_func(void* foo) {
printf("in thread, lwp = %d\n", _lwp_self());
sleep(10);
return NULL;
}
int main() {
int ret;
int pid = fork();
assert(pid != -1);
if (pid == 0) {
int i;
pthread_t t[10];
ret = ptrace(PT_TRACE_ME, 0, NULL, 0);
assert(ret != -1);
printf("in main, lwp = %d\n", _lwp_self());
raise(SIGSTOP);
printf("main resumed\n");
for (i = 0; i < 10; i++) {
ret = pthread_create(&t[i], NULL, thread_func, NULL);
assert(ret == 0);
printf("thread %d started\n", i);
}
for (i = 0; i < 10; i++) {
ret = pthread_join(t[i], NULL);
assert(ret == 0);
printf("thread %d joined\n", i);
}
return 0;
}
pid_t waited = waitpid(pid, &ret, 0);
assert(waited == pid);
printf("wait: %d\n", ret);
assert(WSTOPSIG(ret) == SIGSTOP);
struct ptrace_event ev;
ev.pe_set_event = PTRACE_LWP_CREATE | PTRACE_LWP_EXIT;
ret = ptrace(PT_SET_EVENT_MASK, pid, &ev, sizeof(ev));
assert(ret == 0);
ret = ptrace(PT_CONTINUE, pid, (void*)1, 0);
assert(ret == 0);
while (1) {
waited = waitpid(pid, &ret, 0);
assert(waited == pid);
printf("wait: %d\n", ret);
if (WIFSTOPPED(ret)) {
assert(WSTOPSIG(ret) == SIGTRAP);
ptrace_siginfo_t info;
ret = ptrace(PT_GET_SIGINFO, pid, &info, sizeof(info));
assert(ret == 0);
struct ptrace_state pst;
ret = ptrace(PT_GET_PROCESS_STATE, pid, &pst, sizeof(pst));
assert(ret == 0);
printf("SIGTRAP: si_code = %d, ev = %d, lwp = %d\n",
info.psi_siginfo.si_code, pst.pe_report_event, pst.pe_lwp);
ret = ptrace(PT_CONTINUE, pid, (void*)1, 0);
assert(ret == 0);
} else
break;
}
return 0;
}
The program starts 10 threads, and the debugger should report 10 SIGTRAP
events for LWPs being started (ev = 8) and the same number for LWPs
exiting (ev = 16). However, initially I've been getting as many
as 4 SIGTRAPs, and the remaining 6 threads went unnoticed.
The issue is that do_lwp_create()
does not raise SIGTRAP directly but defers that to mi_startlwp()
that is called asynchronously as the LWP starts. This means that the former
function can return before SIGTRAP is emitted, and the program can start
another LWP. Since signals are not properly queued, multiple SIGTRAPs
can end up being issued simultaneously and lost.
Kamil has already worked on making simultaneous signal deliver more reliable. However, he reverted his commit as it caused regressions.
Nevertheless, applying it made it possible for the test program to get all
SIGTRAPs at least most of the time.
The ‘repeated’ SIGTRAPs did not include correct LWP information, though.
Kamil has recently fixed that by moving the relevant data from process information
to signal information struct.
Combined with his earlier patch, this makes my test program pass most
of the time (sadly, there seem to be some more race conditions involved).
Summary of threading work
My current work-in-progress patch can be found on Differential as D64647.
However, it is currently unsuitable for merging as some tests start failing
or hanging as a side effect of the changes. I'd like to try to get as many
of them fixed as possible before pushing the changes to trunk, in order
to avoid causing harm to the build bot.
The status with the current set of Kamil's work-in-progress patches applied
to the kernel includes approximately 4 failing tests and 10 hanging tests.
Other LLVM news
Manikishan Ghantasala has been working on NetBSD-specific clang-format
improvements
in this year's Google Summer of Code. He is continuing to work on clang-format,
and has recently been given commit access to the LLVM project!
Besides NetBSD-specific work, I've been trying to improve a few other areas
of LLVM. I've been working on fixing regressions in stand-alone build support
and regressions in support for BUILD_SHARED_LIBS=ON builds. I have to admit
that while a year ago I was the only person fixing those issues, nowadays I see
more contributions submitting patches for breakages specific to those builds.
I have recently worked on fixing bad assumptions in LLDB's Python support.
However, it seems that Haibo Huang has taken
it from me and is doing a great job.
My most recent endeavor was fixing LLVM_DISTRIBUTION_COMPONENTS support
in LLVM projects. This is going to make it possible to precisely fine-tune
which components are installed, both in combined tree and stand-alone builds.
Future plans
My first goal right now is to assert what is causing the test host to crash,
and restore buildbot stability. Afterwards, I'd like to continue investigating
threading problems and provide more reproducers for any kernel issues we may
be having. Once this is done, I'd like to finally push my LLDB patch.
Since threading is not the only goal left in the TODO, I may switch between
working on it and on the remaining TODO items. Those are:
-
Add support to backtrace through signal trampoline and extend the support to
libexecinfo, unwind implementations (LLVM, nongnu). Examine adding CFI
support to interfaces that need it to provide more stable backtraces (both
kernel and userland).
-
Add support for i386 and aarch64 targets.
-
Stabilize LLDB and address breaking tests from the test suite.
-
Merge LLDB with the base system (under LLVM-style distribution).
This work is sponsored by The NetBSD Foundation
The NetBSD Foundation is a non-profit organization and welcomes any
donations to help us continue funding projects and services
to the open-source community. Please consider visiting the following URL
to chip in what you can:
https://netbsd.org/donations/#how-to-donate
