NetBSD Planet

July 20, 2019

Roy Marples open_memstream

open_memstream is one of the more important functions added to POSIX libc of late. It's so important because it makes the generation of strings really easy - you no longer need to care about allocating the right amount of memory as the library will do it for you. Now, there's many functions that already help with this, such as asprintf but that's not standard and if you want to create many strings in one area you still need to care about the size of the area. You want to create an area if you have many strings, because it's more efficient for malloc and if you keep the area around and re-use it then it avoids memory fragmentation.

Now, to be clear, you have been able to do this since forever using fopen, writing to the file and then allocating your area based on the final file size. Still, it requires some memory management still but more importantly it writes to a file. Writing to a file is slow and reduces the life span of the disk you're writing to. It's only been fairly recently that tmpfs was a thing, but even today not all OS's have /tmp mounted as tmpfs. Requiring this isn't exactly ideal for a generic program to do - the setup and install should be easy. Because of all these reasons, most programs worked the string length needed and either allocated an area or string and then finally write the string. However, while saving the disk, it's also a lot more error prone because you need to work out the length of everything and that's not always trivial, especially for things like a DHCP client which is always building strings based on the information given by the DHCP server.

Here's an example of open_memstream in action:

 * Example program which manages an area of environment strings
 * to send to child programs.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

static const char *foo = "foo";
static const char *bar = "bar";

int main(void)
        char *argv[] = { "/usr/bin/env", NULL };
        char *buf, *ep, *p, **env, **envp;
        size_t buflen, nenv, i;
        FILE *fp = open_memstream(&buf, &buflen);

        fprintf(fp, "FOO=%s", foo);
        fputc('\0', fp);
        fprintf(fp, "BAR=%s", bar);
        fputc('\0', fp);

        /* We could keep fp around as our area and just rewind it. */

        /* execve relies on a trailing NULL */
        nenv = 1;
        for (p = buf, ep = p + buflen; p < ep; p++) {
                if (*p == '\0')

        /* reallocarray(3) should be standard really */
        envp = env = malloc(nenv * sizeof(char *));
        *envp++ = buf;
        for (p = buf, ep--; p < ep; p++) {
                if (*p == '\0')
                        *envp++ = p + 1;
        *envp = NULL;

        execve(argv[0], argv, env);

As you can see, we only manage the environment array handed to to execve - open_memstream is managing our string area and fprintf is working out the length each string needs to be for us. This vastly reduces the complexity and increases the security and reliabilty of creating large environment strings, which most DHCP clients do. We could also write a helper function to write the string AND the trailing NULL terminator for the string to be more efficient. You'll get to see this in dhcpcd-8 which should be released later this year.

DragonFly BSD Digest In Other BSDs for 2019/07/20

And overflow continues!  I am secretly pleased.

July 19, 2019

DragonFly BSD Digest BSDNow 307: Twitching with OpenBSD

BSD Now 307 accomplishes another trifecta week, mentioning Free, Net, and Open, and also mentioning that vBSDCon’s call for papers closes tomorrow – I’m mentioning it now cause it’ll be too late to mention for In Other BSDs this weekend.

July 17, 2019

Roy Marples openresolv-3.9.1 released

Minor update with the following changes:

The last upate was in 2016, so this is proving to be very stable :)

July 13, 2019

Jeremy C. Reed 2019-July-13 pfSense Essentials Book Writing

This week I received my printed proof from the printer and enabled it to be printed. It is now for sale at Amazon and Barnes and Noble,

I set an alarm to work on it very early a few days a week and it took me a few years. (I am blessed to only commute a few times a year, so I make sure that I don't waste that gifted time.)

This book was written using Docbook using NetBSD and vi. The print-ready book was generated with Dblatex version 0.3.10 with a custom stylesheet, pdfTeX 3.14159265-2.6-1.40.19 (Web2C 2018), and the TeX document production system installed via Tex Live and Pkgsrc. Several scripts and templates were created to help have a consistent document.

The book work was managed using the Subversion version control software. I carefully outlined my steps in utilizing the useful interfaces and identified every web and console interface. The basic writing process included adding over 350 special comment tags in the docbook source files that identified topics to cover and for every pfSense web interface PHP script (highlighting if they were main webpages from the pfSense menu). As content was written, I updated these special comments with a current status. A periodic script checked the docbook files and the generated book and reported on writing progress and current needs.

During this writing, nearly every interface was tested. In addition, code and configurations were often temporarily customized to simulate various pfSense behaviors and system situations. Most of the pfSense interface and low-level source code was studied, which helped with identifying pfSense configurations and features that didn't display in standard setups and all of its options. The software was upgraded several times and installed and ran in multiple VMs and hardware environments with many wireless and network cards, including with IPv6. In addition, third-party documentation and even source code was researched to help explain pfSense configurations and behaviors.

As part of this effort, I documented 352 bugs (some minor and some significant) and code suggestions that I found from code reading or from actual use of the system. (I need to post that.)

The first subversion commit for this book was in July 2014. It has commits in 39 different months with 656 commits total. The book's docbook source had 3789 non-printed comments and 56,193 non-blank lines of text. The generated book has over 180,000 words. My subversion logs show I have commits on 41 different Saturdays. Just re-reading with cleanup took me approximately 160 hours.

DragonFly BSD Digest In Other BSDs for 2019/07/13

Done early, for once!  I managed to complete this by Thursday night.

July 11, 2019

Stack Overflow configuration of tty on BSD system

For a command like this one on Linux debian-linux 4.19.0-1-amd64 #1 SMP Debian 4.19.12-1 (2018-12-22) x86_64 GNU/Linux with xfce I get :

[email protected]:~$ dbus-send --system --type=method_call --print-reply --dest
=org.freedesktop.DBus /org/freedesktop/DBus org.freedesktop.DBus.ListActivatable  

The same command on OpenBSD LeOpenBSD 6.4 GENERIC.MP#364 amd64 with xfce I get :

ktop/DBus org.freedesktop.DBus.ListActivatableNames   <

On linux, at the end of screen, we go to next line.
On BSD(OpenBSD-NetBSD), the command line continue on the same line and the first words disapear.
It's the same in xfce-terminal-emulator, xterm or in TTY (Alt-Ctrl-F3)

I try to add am in gettytab in the defaut section with no avail.
Termcap man page say :
If the display wraps around to the beginning of the next line when the cursor reaches the right margin, then it should have the am capability.
What can I do ?

July 09, 2019

NetBSD Package System (pkgsrc) on DaemonForums Zabbix Frontend Dependencies
Hi All
I used pkgsrc to install the zabbix frontend. I notice though that it automatically installs some php71 dependencies. I really wanted to use php73 though as php71 has some vulns. Is there a way to do that?
NetBSD Blog GSoC 2019 Report: Incorporating the memory-hard Argon2 hashing scheme into NetBSD
This report was prepared by Jason High as a part of Google Summer of Code 2019

Argon2 is a modern memory-hard hashing scheme designed by Biryukov et al.[1] Compared to currently supported hashing algorithms in NetBSD, memory-hard Argon2 provides improved resistance against Time Memory Trade-off (TMTO) and side-channel attacks. In our project, we are working to incorporate Argon2 into the local password management framework of NetBSD.

Phase 1 goals and work completed

Phase 1 of the project focused on incorporating the Argon2 reference implementation into NetBSD. As such, we focused on building the associated libraries and integrating the functionality into the existing password management framework. Our initial phase 1 goals were as follows

Towards these goals, we have added the Argon2 reference code into the external source tree and created the necessary build scripts. This work allows us to successfully add Argon2 into our system by adding MKARGON2=yes to /usr/share/mk/ After successfully building and installation, we have the following


We then extended the functionality of pwhash(1) and libcrypt(3) to support Argon2 encoding. Currently we support all three Argon2 variants, although not all variants are recommended (see [1][2]). We support the following standard parameters: execution time (t), memory utiltized (m), and degree of parallelism (p). Salt length is currently fixed at the recommended 16 bytes.[1]

With our phase 1 goals successfully completed, we have the following functionality available. The argon2(1) binary allows us to easily validate parameters and encodings

m2# echo -n password|argon2 somesalt -id -p 3 -m 8
Type:           Argon2id
Iterations:     3
Memory:         256 KiB
Parallelism:    3
Hash:           97f773f68715d27272490d3d2e74a2a9b06a5bca759b71eab7c02be8a453bfb9
Encoded:        $argon2id$v=19$m=256,t=3,p=3$c29tZXNhbHQ$l/dz9ocV0nJySQ09LnSiqbBqW8p1m3Hqt8Ar6KRTv7k
0.000 seconds
Verification ok
Argon2 support has been added to pwhash(1) using the -A flag, using the form -A variant[params], where variant is one of the following: argon2i, argon2d, or argon2id. [params] is a comma-delimited list of the following: p=%d, m=%d, or t=%d (see man pwhash(1)). For example, to create an encoding of 'password' using the argon2id variant, we may execute the following
m2# pwhash -A argon2id password
To encode 'password' using the argon2id variant with explicit specification for both parallelism and memory, we execute
m2# pwhash -Aargon2id,p=3,m=8192  password 
We support local password hashing using passwd.conf(5). We accept the same parameters as pwhash(1). For example
m1# grep -A1 testuser /etc/passwd.conf 
        localcipher = argon2i,t=6,m=4096,p=1
With the above configuration in place, we are able to support standard password management. For example
m1# id testuser 
uid=1001(testuser) gid=100(users) groups=100(users)

m1# grep testuser /etc/master.passwd                                                                                          

m1# passwd testuser
Changing password for testuser.
New Password:
Retype New Password:

m1# grep testuser /etc/master.passwd  

Plans for next phase

Phase 2 will focus on code cleanup and incorporation of any improvements suggested during review. We are also extending our ATF test-set and will begin our performance evaluation. Primary deliverables for phase 2 will be a performance evaluation.


We have successfully integrated Argon2 into NetBSD using the native build framework. We have extended existing functionality to support local password management using Argon2 encoding. Moving forward in phase 2, we will work on cleanup, validation, and performance evaluation.


[1] Biryukov, Alex, Daniel Dinu, and Dmitry Khovratovich. "Argon2: new generation of memory-hard functions for password hashing and other applications." 2016 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2016.
[2] Alwen, Joël, and Jeremiah Blocki. "Towards practical attacks on argon2i and balloon hashing." 2017 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2017.

July 08, 2019

NetBSD Blog Implementation of DRM ioctl Support for NetBSD kernel

This report was prepared by Surya P as a part of Google Summer of Code 2019

What is DRM ioctl ?

Ioctls are input/output control system calls and DRM stands for direct rendering manager The DRM layer provides several services to graphics drivers, many of them driven by the application interfaces it provides through libdrm, the library that wraps most of the DRM ioctls. These include vblank event handling, memory management, output management, framebuffer management, command submission & fencing, suspend/resume support, and DMA services.

Native DRM ioctl calls

NetBSD was able to make native DRM ioctl calls with hardware rendering once xorg and proper mesa packages where installed. We used the glxinfo and glxgears applications to test this out.

X desktop glxgears

DRM ioctl calls from emulation

In order to make sure DRM ioctl calls where also made from the linux emulation layer of NetBSD . We used rpm and suse131 packages In specific base,compat,X11,libdrm,libglx,libexpat packages where used. To my surprise the applications kept segfaulting . I used glxgears and glxinfo rpm packages for this test .when I analyzed the segfault and traced the process , I was able to identify the cause of the segfault which was caused due to broken suse131 libdrm package which did not support nouveau based cards. To further make user that the problem was with the suse packages , I downgraded to suse121 and as expected the glxinfo and glxgears rpm packages ran, but it was using software rendering instead of hardware rendering but nevertheless we were still able to see the DRM ioctl calls made by the emulation layer hence we added some print statements in the kernel source to identify the calls made.

Bash shell showing ioctls


Fixing the Suse131 package and enabling hardware rendering from emulation is of highest priority , I have also planned to port steam and its dependencies to NetBSD to incorporate some gaming on NetBSD! And finally conversion between 32bit DRM ioctl calls 64bit DRM ioctl calls will be implemented.

Last but not the least I would like to thank my mentor @christos , @maya , @leot for helping me out and guiding me throughout the process and Google for providing me with such a wonderful opportunity to work with NetBSD community.

Server Fault Webserver farm with NFS share (autofs failure)

I am trying to set up the farm of webservers, consisting of the internal, external and worker servers.

  1. The actual sites content is stored on internal NFS server deep in internal network. All sites contents management is centralized.

  2. BSD-based external servers have Lighttpd doing all the HTTP/HTTPS job, serving static content. Main NFS share is auto-mounted via special path, like /net/server/export/www/site/ (via amd).

  3. Every Lighttpd have fastcgi parameters pointing to several worker servers, which have php-fpm working (for example). Different sites may require different php versions or arrangement, so www01 and www02 may serve site "A" having php-fpm over PHP 5.6 and www05 and www06 will serve site "B" having php-fpm over PHP 7.2.

  4. Every worker get requests for certain sites (one or more) with path /net/server/export/www/site and execute PHP or any other code. They also have amd (for BSD) and autofs (for Linux) working.

  5. For some sites Lighttpd may not forward fastcgi, but do proxying instead, so workers can have Apache or other web-server (even Java-based) working.

External servers are always BSD, internal servers too, but workers can be different upon actual needs.

This all work good when workers are BSD. If we are using Linux on workers - it stops working when share is automatically unmounted. When one tries to access the site he will get error 404. When I connect to server via ssh I will see no mounted share on "df -h". If I do any "ls" on /net/server/export - it is self-mounted as intended and site starts to work. On BSD-systems df show amd shares always mounted despite of 60 seconds dismount period.

I believe there is a difference between amd and autofs approach, php-fpm calls on Linux become some kind of "invisible" to autofs and do not cause auto-mount, because any other access to /net/server/ work at any time and do cause auto-mount. Also, this happens not with php-fpm only, Apache serving static content on auto-mounted NFS share behave same way.

Sorry for long description, but I tried to describe it good. The main question here - is anyone know why calls to /net/server may not cause auto-mount in autofs and how to prevent this behavior.

For lot of reasons I do not consider using static mounting, so this is not an option here. As for Linux versions - mostly it was tested on OEL 7.recent.

Server Fault ssh tunnel refusing connections with "channel 2: open failed"

All of a sudden (read: without changing any parameters) my netbsd virtualmachine started acting oddly. The symptoms concern ssh tunneling.

From my laptop I launch:

$ ssh -L 7000:localhost:7000 [email protected] -N -v

Then, in another shell:

$ irssi -c localhost -p 7000

The ssh debug says:

debug1: Connection to port 7000 forwarding to localhost port 7000 requested.
debug1: channel 2: new [direct-tcpip]
channel 2: open failed: connect failed: Connection refused
debug1: channel 2: free: direct-tcpip: listening port 7000 for localhost port 7000, connect from port 53954, nchannels 3

I tried also with localhost:80 to connect to the (remote) web server, with identical results.

The remote host runs NetBSD:

bash-4.2# uname -a
NetBSD host 5.1_STABLE NetBSD 5.1_STABLE (XEN3PAE_DOMU) #6: Fri Nov  4 16:56:31 MET 2011  [email protected]:/m/obj/m/src/sys/arch/i386/compile/XEN3PAE_DOMU i386

I am a bit lost. I tried running tcpdump on the remote host, and I spotted these 'bad chksum':

09:25:55.823849 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 67, bad cksum 0 (->3cb3)!) > P, cksum 0xfe37 (incorrect (-> 0xa801), 1622402406:1622402421(15) ack 1635127887 win 4096 <nop,nop,timestamp 5002727 5002603>

I tried restarting the ssh daemon to no avail. I haven't rebooted yet - perhaps somebody here can suggest other diagnostics. I think it might either be the virtual network card driver, or somebody rooted our ssh.


July 07, 2019

NetBSD Blog LLDB: watchpoints, XSTATE in ptrace() and core dumps

Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages.

In February, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support and lately extending NetBSD's ptrace interface to cover more register types and fix compat32 issues. You can read more about that in my May 2019 report.

In June, I have finally finished the remaining ptrace() work for xstate and got it merged both on NetBSD and LLDB end (meaning it's going to make it into NetBSD 9). I have also worked on debug register support in LLDB, effectively fixing watchpoint support. Once again I had to fight some upstream regressions.

ptrace() XSTATE interface

In the previous report, I was comparing two approaches to resolving unpredictable XSAVE data offsets. Both solutions had their merits but I eventually went with having a pair of requests with a single predictable, extensible structure. As a result, I have implemented two new ptrace() requests:

The main features of this API are:

  1. It provides single call to obtain all supported XSAVE components. This is especially useful for YMM or ZMM registers whose contents are split between disjoint XSAVE components.

  2. It provides a xs_rfbm bitfield that clearly indicates which XSAVE components were available, and which can be used to issue partial updates via PT_SETXSTATE.

  3. It requires the caller to explicitly specify structure size. As a result, new fields (= component types) can be added to it without breaking compatibility with already built programs.

  4. It provides identical API to i386 and amd64 programs, removing the need for code duplication.

  5. It provides backwards compatibility with FSAVE- and FXSAVE-only systems, with xs_rfbm clearly indicating which fields were filled.

  6. It can replace disjoint PT_GETFPREGS and PT_GETXMMREGS APIs on i386/amd64 with a single convenient method.

From user's perspective, the main gain is ability to read YMM (AVX) registers. The code supports ZMM (AVX-512) registers as well but I have not been able to test it due to lack of hardware. That said, if one of the readers is running NetBSD on AVX-512 capable CPU and is willing to help, please contact me and I'll give you some tests to run.

The two relevant commits are:

The two new calls are covered by tests for reading and writing MM (MMX), XMM (SSE) and YMM (AVX) registers. I have also done some work on ZMM (AVX-512) test but I did not complete it due to aforementioned lack of hardware.

On the LLDB end, the change was preceded with some bugfixes and cleanup suggested by Pavel Labath. The relevant commits are:

XSTATE in core dumps

The ptrace() XSTATE supports provides the ability to introspect registers in running programs. However, in order to improve the support for debugging crashed programs the respective support needs to be also added to core dumps.

NetBSD core dumps are built on ELF file format, with additional process information stored in ELF notes. Notes can be conveniently read via readelf -n. Each note is uniquely identified by a pair of name and numeric type identifier. NetBSD-specific notes are split into two groups:

Two process-specific notes are used at the moment:

  1. ELF_NOTE_NETBSD_CORE_PROCINFO containing process information — including killing signal information, PIDs, UIDs, GIDs…

  2. ELF_NOTE_NETBSD_CORE_AUXV containing auxiliary information provided by the dynamic linker.

The LWP-specific notes currently contain register dumps. They are stored in the same format as returned by ptrace() calls, and use the same numeric identifiers as PT_GET* requests.

Previously, only PT_GETREGS and PT_GETFPREGS dumps were supported. This implies that i386 coredumps do not include MMX register values. Both requests were handled via common code, with a TODO for providing machdep (arch-specific) hooks.

My work on core dumps involved three aspects:

  1. Writing ATF tests for their correctness.

  2. Providing machdep API for injecting additional arch-specific notes.

  3. Injecting PT_GETXSTATE data into x86 core dumps.

To implement the ATF tests, I've used PT_DUMPCORE to dump core into a temporary file with predictable filename. Afterwards, I've used libelf to process the ELF file and locate notes in it. The note format I had to process myself — I have included a reusable function to find and read specific note in the tests.

Firstly, I wrote a test for process information. Then, I refactored register tests to reduce code duplication and make writing additional variants much easier, and created matching core dump tests for all existing PT_GET* register tests. Finally, I implemented the support for dumping PT_GETXSTATE information.

Of this work, only the first test was merged. The relevant commits and patches are:

LLDB debug register / watchpoint support

The next item on my TODO was fixing debug register support in LLDB. There are six debug registers on x86, and they are used to support up to four hardware breakpoints or watchpoints (each can serve as either). Those are:

DR4 and DR5 are obsolete synonyms for DR6 and DR7.

For each breakpoint, the control register provides the following options:

  1. Enabling it as global or local breakpoint. Global breakpoints remain active through hardware task switches, while local breakpoints are disabled on task switches.

  2. Setting it to trigger on code execution (breakpoint), memory write or memory write or read (watchpoints). Read-only hardware watchpoints are not supported on x86, and are normally emulated via read/write watchpoints.

  3. Specifying the size of watched memory to 1, 2, 4 or 8 bytes. 8-byte watchpoints are not supported on i386.

According to my initial examination, watchpoint support was already present in LLDB (most likely copied from relevant Linux code) but it was not working correctly. More specifically, the accesses were reported as opaque tracepoints rather than as watchpoints. While the program was correctly stopped, LLDB was not aware which watchpoint was triggered.

Upon investigating this further, I've noticed that this happens specifically because LLDB is using local watchpoints. After switching it to use global watchpoints, NetBSD started reporting triggered watchpoints correctly.

As a result, new branch of LLDB code started being used… and turned out to segfault. Therefore, my next goal was to locate the invalid memory use and correct it. In this case, the problem lied in the way thread data was stored in a list. Specifically, the program wrongly assumed that the list index will match LWP number exactly. This had two implications.

Firstly, it suffered from off-by-one error. Since LWPs start with 1, and list indexes start with 0, a single-threaded program crashed trying to access past the list. Secondly, the assumption that thread list will be always in order seemed fragile. After all, it relied on LWPs being reported with successive numbers. Therefore, I've decided to rewrite the code to iterate through thread list and locate the correct LWP explicitly.

With those two fixes, some of the watchpoint tests started passing. However, some are still failing because we are not handling threads correctly yet. According to earlier research done by Kamil Rytarowski, we need to copy debug register values into new LWPs as they are created. I am planning to work on this shortly.

Additionally, NetBSD normally disallows unprivileged processes from modifying debug registers. This can be changed via enabling security.models.extensions.user_set_dbregs. Since LLDB tests are normally run via unprivileged users, I had to detect this condition from within LLDB test suite and skip watchpoint tests appropriately.

The LLDB commits relevant to this topic are:

Regressions caught by buildbot

Finally, let's go over the regressions that were caught by our buildbot instance throughout the passing month:

Future plans

Since Kamil has managed to move the kernel part of threading support forward, I'm going to focus on improving threading support in LLDB right now. Most notably, this includes ensuring that LLDB can properly handle multithreaded applications, and that all thread-level actions (stepping, resuming, signalling) are correctly handled. As mentoned above, this also includes handling watchpoints in threads.

Of course, I am also going to finish the work on XSTATE in coredumps, and handle any possible bugs I might have introduced in my earlier work.

Afterwards I will work on the remaining TODO items, that are:

  1. Add support to backtrace through signal trampoline and extend the support to libexecinfo, unwind implementations (LLVM, nongnu). Examine adding CFI support to interfaces that need it to provide more stable backtraces (both kernel and userland).

  2. Add support for i386 and aarch64 targets.

  3. Stabilize LLDB and address breaking tests from the test suite.

  4. Merge LLDB with the base system (under LLVM-style distribution).

This work is sponsored by The NetBSD Foundation

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:

July 06, 2019

DragonFly BSD Digest In Other BSDs for 2019/07/06

I have a backlog of more BSD links to add, beyond this – it was a busy week!

July 05, 2019

DragonFly BSD Digest Japan NetBSD Users Group tomorrow

There’s a meeting of the NetBSD Japan user’s group, and an informal users session, tomorrow, in Tokyo.  Go, if you are near.  (via)

July 04, 2019

Roy Marples Red-Black Tree

Pretty much every piece of software I've seen uses a list of objects. When you need to easily grow and shrink this list easily you then need something like a Linked List. dhcpcd has used very popular and widely available BSD based queue(3), specifically a tailq. The main advantages of this type of list are:

However, it's just a list. And like any list, to find something we need to pick a starting point and then search in a single direction and match each item until we find it. This is described as O(N) and it's performance roughly corrolates to the Number of objects in the list. If you need to insert at an arbitary point (for example to order the list) you need to search it, so this is O(N) also.

But is this slow?

Typically for dhcpcd, the answer is no because on a desktop, tablet, phone or SOHO router you only have a handful of objects in all the lists. However, someone ran dhcpcd on a switch with millions of routes and dhcpcd was taking minutes of CPU time for routing table modifications. So in this corner case ... yes! dhcpcd is slow! A suggestion to improve matters was given in that report - use a Red-Black Tree instead of a Linked List. A RB tree guarantees O(log N) performance for searching. A patch was even submitted based on OpenBSD's tree implementation! This reduced the dhcpcd runtime from minutes to seconds in his situation.

Now you can't just swap in a RB Tree for a Linked List. Each object within an RB Tree has to have a unique key. Also, you need to have comparison function to define an order to the objects within the tree. Only then can you start to implement it. To find an object within the tree, you just need to know it's key and off you go. I settled on using NetBSD's rbtree(3) implementation instead of the initial suggestion to use the generic BSD tree(3). This mainly to reduce the binary size of dhcpcd because it's in libc on NetBSD and on other systems it will creates a smaller binary due to the complexity of the tree(3) macros. Testing also showed it was slightly faster and also easier to actually develop with.

So should you replace all LinkedLists with RB Trees? No! That would be silly :) After all, sometimes you don't actually need to search or order a list - a lot of the time, lists are used for push/pop operations and thus gain nothing from RB Tree.

You'll get to see this in dhcpcd-8 which should be released later this year.

NetBSD Blog Forking code support in ptrace(2)
I've finished all the planned tasks regarding fork(2), vfork(2), clone(2)/__clone(2), and posix_spawn(3) in the context of debuggers. There are no longer any known kernel issues for any of these calls. All of the calls are covered with ATF regression tests.

Debugger related changes

NetBSD might be the only mainstream OS that implements posix_spawn(3) with a dedicated syscall. All the other recognized kernels have an implementation as a libc wrapper around either fork(2), vfork(2) or clone(2). I've introduced a new ptrace(2) event, PTRACE_POSIX_SPAWN, improved the posix_spawn(3) documentation, and introduced new ATF regression tests for posix_spawn(3).

The new ptrace(2) code has been exercised under load with LLDB test-suite, picotrace, and NetBSD truss.

I intend to resume porting of edb-debugger from work I began two years ago. I hope to use it for verifying FPU registers. I've spent some time porting this debugger to NetBSD/amd64 and managed to get a functional process attached. Unfortunately the code is highly specific to a single Operating System. That OS is the only one that is really functional at the moment and the code needs rework to be more agnostic with respect to the different semantics of kernels.

Issues with threaded debugging

I've analyzed the problems with multiple thread programs under debuggers. They could be classified into the following groups:

I've found that focusing on kernel correctness now and fixing thread handling bugs can have paradoxically random impact on GDB/NetBSD. The solution for multiple events reported from multiple threads concurrently has a fix in progress, but the initial changes caused some fallout in GDB support so I have decided to revert it for now. This pushed me to the conclusion that before fixing LWP events, there is a priority to streamline the GDB support: modernize it, upstream it, run regression tests.

Meanwhile there is an ongoing work on covering more kernel code paths with fuzzers. We will catch and address problems out there only because they're able to be found. I'm supervising 3 ongoing projects in this domain: syzkaller support enhancements, TriforceAFL porting and AFL+KCOV integration. This work makes the big picture of what is still needed to be fixed clearer and lowers the cost of improving the quality.


The original implementation of Leak Sanitizer for NetBSD was developed for GCC by Christos Zoulas. The whole magic of a functional LSan software is the Stop-The-World operation. This means that a process suspends all other threads while having the capability to investigate the register frame and the stacks of suspended threads.

Until recently there were two implementations of LSan: for Linux (ab)using ptrace(2) and for Darwin using low-level Mach interfaces. Furthermore the Linux version needs a special version of fork(2) and makes assumptions about the semantics of the signal kernel code.

The original Linux code must separately attach to each thread via a pid. This applies to all other operations, such as detaching or killing a process through ptrace(2). Additionally listing threads of a debugged process on Linux is troublesome as there is need to iterate through directories in /proc.

The implementation in GCC closely reused the semantics of Linux. There was room for enhancement. I've picked this code as an inspiration and wrote a clean implementation reflecting the NetBSD kernel behavior and interfaces. In the end the NetBSD code is simpler than the Linux code without needing any recovery fallbacks or port specific kludges (in Linux every CPU needs discrete treatment.)

Much to my chagrin, this approach abuses the ptrace(2) interface, making sanitizing for leaking programs incompatible with debuggers. The whole StopTheWorld() operation could be cleanly implemented on the kernel side as a new syscall. I have a semicompleted implementation of this syscall, however I really want to take care of all threading issues under ptrace(2) before moving on. The threading issues must be fully reliable in one domain of debugging before implementing other kernel code. Both LLVM 9.0 and NetBSD-9 are branching soon and it will be a good enough solution for the time being. My current goal before the LLVM branching is to address a semantic behavioral difference in atexit(3) that raises a false positive in LSan tests. Request for comments on this specific atexit(3) issue will be available pending feedback from upstream.

Plan for the next milestone

Modernize GDB/NetBSD support, upstream it, and run GDB regression tests for NetBSD/amd64. Switch back to addressing threading kernel issues under a debugger.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:

OS News OpenBSD is now my workstation
Why OpenBSD? Simply because it is the best tool for the job for me for my new-to-me Lenovo Thinkpad T420. Additionally, I do care about security and non-bloat in my personal operating systems (business needs can have different priorities, to be clear). I will try to detail what my reasons are for going with OpenBSD (instead of GNU/Linux, NetBSD, or FreeBSD of which I’m comfortable using without issue), challenges and frustrations I’ve encountered, and what my opinions are along the way. I’ve never managed to really get into the BSDs, as Linux has always served my needs for a UNIX-like operating system quite well. I feel like the BSDs are more pure and less messy than Linux, but is that actually true, or just my perception?

July 03, 2019

Super User Using a Console-only NetBSD VM

I am experimenting with NetBSD and seeing if I can get the Fenrir screenreader to run on it. However, I hit a snag post install; the console that I was using for the installation was working perfectly fine, however it stopped working alltogether once I completed the install. For reference, here is the line I used for virt-install:

virt-install --connect qemu:///system -n netbsd-testing \
             --ram 4096 --vcpus=8 \
             --cpu=host \
             -c /home/sektor/Downloads/boot-com.iso  \
             --os-type=netbsd --os-variant=netbsd8.0 \
             --disk=pool=devel,size=100,format=qcow2 \
             -w network=default --nographics 

When it asked me for the type of terminal I was using (this being the NetBSD install program), I accepted the default which was VT200. As I recall, I told it to use the BIOS for booting, and not any of the comm serial ports. Has anyone had any further experience with using no graphics on a Libvirt virtualized machine, and have any points as to how to get a working console?


July 02, 2019

NetBSD Blog Write your own fuzzer for NetBSD kernel! [Part 1]
This report was written by Maciej Grochowski as a part of developing the AFL+KCOV project.

How Fuzzing works? The dummy Fuzzer.

The easy way to describe fuzzing is to compare it to the process of unit testing a program, but with different input. This input can be random, or it can be generated in some way that makes it unexpected form standard execution perspective.

The simplest 'fuzzer' can be written in few lines of bash, by getting N bytes from /dev/rand, and putting them to the program as a parameter.

Coverage and Fuzzing

What can be done to make fuzzing more effective? If we think about fuzzing as a process, where we place data into the input of the program (which is a black box), and we can only interact via input, not much more can be done.

However, programs usually process different inputs at different speeds, which can give us some insight into the program's behavior. During fuzzing, we are trying to crash the program, thus we need additional probes to observe the program's behaviour.

Additional knowledge about program state can be exploited as a feedback loop for generating new input vectors. Knowledge about the program itself and the structure of input data can also be considered. As an example, if the input data is in the form of HTML, changing characters inside the body will probably cause less problems for the parser than experimenting with headers and HTML tags.

For open source programs, we can read the source code to know what input takes which execution path. Nonetheless, this might be very time consuming, and it would be much more helpful if this can be automated. As it turns out, this process can be improved by tracing coverage of the execution.

AFL (American Fuzzy Lop) is one of the first successful fuzzers. It uses a technique where the program is compiled with injected traces for every execution branch instruction. During the program execution, every branch is counted, and the analyzer builds a graph out of execution paths and then explores different "interesting" paths.

Now, fuzzing has become a mainstream technique, and compilers provide an option to embed fuzzing hooks at compilation time via switches.

The same process can be applied to the kernel world. However, it would be quite hard to run another program on the same machine outside of the kernel to read these counters. Because of that, they usually are made available inside the kernel.

To illustrate how that is done, we can compile a hello world program written in C for tracing the Program Counter (PC).

gcc main.c -fsanitize-coverage=trace-pc

/usr/local/bin/ld: /tmp/ccIKK7Eo.o: in function `handler':
main.c:(.text+0xd): undefined reference to `__sanitizer_cov_trace_pc'
/usr/local/bin/ld: main.c:(.text+0x1b): undefined reference to `__sanitizer_cov_trace_pc'

The compiler added additional references to the __sanitizer_cov_trace_pc, but we didn't implement them, or link with something else to provide the implementation. If we grep the head of the NetBSD kernel sources for the same function, we will find within sys/kern/subr_kcov.c an implementation: kcov(4).

Which Fuzzer should I choose?

In recent years, AFL has grown into an industry standard. Many projects have integrated it into their development process. This has caused many different bugs and issues to be found and fixed in a broad spectrum of projects (see AFL website for examples). As this technique has become mainstream, many people have started developing custom fuzzers. Some of them were just modified clones of AFL, but there were also many different and innovative approaches. Connecting a custom fuzzer or testing some unusual execution path is no longer considered as just a hackathon project, but part of security research.

I personally believe that we are still in the early state of fuzzing. A lot of interesting work and research is already available, but we cannot explain or prove why one way is better than another one, or how the reference fuzzer should work, and what are its technical specifications.

Many approaches have been developed to do efficient fuzzing, and many bugs have been reported, but most of the knowledge comes still from empirical experiments and comparison between different techniques.

Modular kcov inside the kernel

Coverage metrics inside kernel became a standard even before the fuzzing era. A primary use-case of coverage is not fuzzing, but testing, and measuring test coverage. While code coverage is well understood, kernel fuzzing is still kind of a Wild West, where most of the projects have their own techniques. There are some great projects with a large community around them, like Honggfuzz and Syzkaller. Various companies and projects manitain several fuzzers for kernel code. This shows us that as a kernel community, we need to be open and flexible for different approaches, that allow people interested in fuzzing to do their job efficiently. In return, various fuzzers can find different sets of bugs and improve the overall quality of our kernel.

In the past, Oracle made some effort to upstream an interface for AFL inside Linux kernel (see the patch here. However, the patches were rejected via the kernel community for various reasons.

We did our own research on the needs of fuzzers in context of kcov(4) internals, and quickly figured out that per-fuzzer changes in the main code do not scale up, and can leave unused code inside the kernel driver.

In NetBSD, we want to be compatible with AFL, Hongfuzz, Syzkaller and few other fuzzers, so keeping all fuzzer specific data inside one module would be hard to maintain.

One idea that we had was to keep raw coverage data inside the kernel, and process it inside the user space fuzzer module. Unfortunately, we found that current coverage verbosity in the NetBSD kernel is higher than in Linux, and more advanced traces can have thousand of entries. One of the main requirements for fuzzers is performance. If the fuzzer is slow, even if it is smarter than others, it will most likely will find fewer bugs. If it is significantly slower, then it is not useful at all. We found that storing raw kernel traces in kcov(4), copying the data into user-space, and transfoming it into the AFL format, is not an option. The performance suffers, and the fuzzing process becomes very slow, making it not useful in practice.

We decided to keep AFL conversion of the data inside the kernel, and not introduce too much complexity to the coverage part. As a current proof of concept API, we made kcov more modular, allowing different modules to implement functionality outside of the core requirements. The current code can be viewed here or on the GitHub.

KCOV Modules

As we mentioned earlier, coverage data available in the kernel is generated during tracing by one of the hooks enabled by the compiler. Currently, NetBSD supports PC and CMP tracing. The Kcov module can gather this data during the trace, convert it and expose to the user space via mmap. To write our own coverage module for new PoC API, we need to provide such operations as: open, free, enable, disable, mmap and handling traces.

This can be done via using kcov_ops structure:

static struct kcov_ops kcov_mod_ops = {
	.open = kcov_afl_open,
	.free = kcov_afl_free,
	.setbufsize = kcov_afl_setbufsize,
	.enable = kcov_afl_enable,
	.disable = kcov_afl_disable,
	.mmap = kcov_afl_mmap,
	.cov_trace_pc = kcov_afl_cov_trace_pc,
	.cov_trace_cmp = kcov_afl_cov_trace_cmp

During load or unload, the module must to run kcov_ops_set or kcov_ops_unset. After set, default kcov_ops are overwritten via the module. After unset, they are returned to the default.

Porting AFL as a module

The next step would be to develop a sub-module compatible with the AFL fuzzer.

To do that, the module would need to expose a buffer to user space, and from kernelspace would need to keep information about the 64kB SHM region, previous PC, and thread id. The thread id is crucial, as usually fuzzing runs few tasks. This data is gathered inside the AFL context structure:

typedef struct afl_ctx {
	uint8_t *afl_area;
	struct uvm_object *afl_uobj;
	size_t afl_bsize;
	uint64_t afl_prev_loc;
	lwpid_t lid;
} kcov_afl_t;

The most important part of the integration is to translate the execution shadow, a list of previous PCs along the execution path, to the AFL compatible hash-map, which is a pair of (prev PC, PC). That can be done according to the documentation of AFL by this method:

++afl->afl_area[(afl->afl_prev_loc ^ pc) & (bsize-1)];
afl->afl_prev_loc = pc;

In our implementation, we use a trick by Quentin Casasnovas of Oracle to improve the distribution of the counters, by storing the hashed PC pairs instead of raw.

The rest of operations, like open, mmap, and enable, can be reviewed in the GitHub repository together with the testing code that dumps 64kB of SHM data.

Debug your fuzzer

Everyone knows that kernel debugging is more complicated than programs running in the user space. Many tools can be used for doing that, and there is always a discussion about usability vs complexity of the setup. People tend to be divided into two groups: those that prefer to use a complicated setup like kernel debugger (with remote debugging), and those for which tools like printf and other simple debug interfaces are sufficient enough.

Enabling coverage brings even more complexity to kernel debugging. Everyone's favourite printf also becomes traced, so putting it inside the trace function will result in a stack overflow. Also, touching any kcov internal structures becomes very tricky and, should be avoided if possible.

A debugger is still a sufficient tool. However, as we mentioned earlier, trace functions are called for every branch, which can be translated to thousand or even tens of thousand break points before any specific condition will occur.

I am personally more of a printf than gdb guy, and in most cases, the ability to print variables' contents is enough to find the issues. For validating my AFL kcov plugin, I found out that debugcon_printf written by Kamil Rytarowski is a great tool.

Example of debugcon_printf

To illustrate that idea, lets say that we want to print every PC trace that comes to our AFL submodule.

The most intuitive way would be put printf("#:%p\n", pc) at very beginning of the kcov_afl_cov_trace_pc, but as mentioned earlier, such a trick would end up with a kernel crash whenever we enable tracing with our module. However, if we switch printf to the debugcon_printf, and add a simple option to our QEMU:

-debugcon file:/tmp/qemu.debug.log -global isa-debugcon.iobase=0xe9

we can see on our host machine that all traces are written to the file qemu.debug.log.

kcov_afl_cov_trace_pc(void *priv, intptr_t pc) {
	kcov_afl_t *afl = priv;

	debugcon_printf("#:%x\n", pc);

	++afl->afl_area[(afl->afl_prev_loc ^ pc) & (afl->afl_bsize-1)];
	afl->afl_prev_loc = _long_hash64(pc, BITS_PER_LONG);


Future work

The AFL submodule was developed as part of the AFL FileSystems Fuzzing project to simplify the fuzzing of different parts of the NetBSD kernel.

I am using it currently for fuzzing different filesystems. In a future article I plan to show more practical examples.

Another great thing to do will be to refactor KLEAK, which is using PC trace data and is disconnected from kcov. A good idea would be to rewrite it as a kcov module, to have one unified way to access coverage data inside NetBSD kernel.


In this article, we familiarized the reader with the technique of fuzzing, starting from theoretical background up to the level of kernel fuzzing.

Based on these pieces of information, we demonstrated the purpose of the a modular coverage framework inside the kernel and an example implementation of submodule that can be consumed by AFL.

More details can be learned via downloading and trying the sample code shown in the example.

At the end of this article, I want to thank Kamil, for such a great idea for a project, and for allowing me to work on NetBSD development.

June 29, 2019

NetBSD General on DaemonForums View X session of instance in VirtualBox via VNC
Does anyone have a working howto on how to attach X session on NetBSD running within VirtualBox to VNC on the host computer?

June 26, 2019

Roy Marples dhcpcd-ui-0.7.6 released

It's been a very long time since the last release - over 2.5 years! It's seen a lot of changes since then, but mainly minor improvements here and there. Some of the important changes are:

Roy Marples dhcpcd-7.2.3 released

Minor update with the following changes:

June 12, 2019

Super User NetBSD - no pkg

After full installation of latest NetBSD I'm tried to launch pkgin, but received pkgin not found, also I've got same for pkgsrc. Then I've found, that there's no /usr/pkg location.

That's normal or I've did something wrong?

June 01, 2019 New Developers in May 2019

May 31, 2019 NetBSD 8.1 released

May 24, 2019

NetBSD Installation and Upgrading on DaemonForums no bootable device after installtion
After installing NetBSD 8 I have a couple problems.
1. If the USB drive with the installation image is not inserted the system will not boot.
2. Running X -configure causes a reboot.

1. Without the installation USB:

PXE-M0F: Exiting PXE ROM.
No bootable -- insert boot disk and press any key

The first time I thought I made a mistake and did something to the BIOS, but the partitions looks fine, just like it should in The Guide:

a:  0    472983    472984    FFSv2
b:  472984    476939    3985    swap
c:  0    476939    476939    NetBSD partition
d:  0    476939    476940    whole disc
e:  0    0    0    unused

I am at a bit of a loss, since as far as I know it should not be possible to set an installation medium as the boot source of an OS.

2. I do not know if this is unsupported hardware or related to #1.

DRM error in radeon_get_bios:
Unable to locate a BIOS ROM
radeon0: error: Fatal error during GPU init

I am trawlling through documrntation, but with a telephone. So I also cannot post a dmesg, although I can look through other threads where it is posted and copy it. (A little later in the day.)

May 20, 2019 NetBSD 8.1_RC1 binaries available

May 02, 2019 New Security Advisories: NetBSD-SA2019-00[2-3]

May 01, 2019 New Developer in April 2019

April 18, 2019

Unix Stack Exchange Automatic install NetBSD ISO

Is there some form of automated NetBSD installation?

I would like to automate this manual process:

But I found no examples.

April 10, 2019

Unix Stack Exchange NetBSD desktop install

I've had NetBSD 7 installed for several years, running MariaDB 5.5.52 and gcc 4.8.4 and Geany. I would like to install a slightly more advanced desktop than XWindows.

When I cd into /usr/pkgsrc/meta-pkgs/lxdew or xfce4 and do

make install clean 

I get the same error msg for both.

Conflicting PLIST with glib1-1.48.2: bin/glib-genmarshal

How do I resolve this conflict? Can I overwrite the existing glib file without harm to MariaDB or GCC? If so, how?

March 15, 2019

Stack Overflow host netbsd 1.4 or 1.5 i386 cross-compile target macppc powerpc g3 program

For some reason, I want develop program which can work on netbsd 1.4 or 1.5 powerpc ,target cpu is power750(powerpc platform,nearly 20 years old system),but I can't find how to make this kind cross-compile enviroment vmware host:i386 netbsd 1.5 + egcs1.1.1 + binutils 2.9.1 ---> target host:macppc powerpc netbsd 1.5 + egcs 1.1.1 I download and install netbsd 1.5 vmware and download pkgsrc,when I make /usr/src/pkgsrc/cross/powerpc-netbsd,I got gcc work on i386 but not cross-gcc,why? Thank you if any help!

March 07, 2019

Amitai Schlair NYCBUG: Maintaining qmail in 2019

On Wednesday, March 6, I attended New York City BSD User Group and presented Maintaining qmail in 2019. This one pairs nicely with my recent DevOpsDays Ignite talk about why and how to Run Your @wn Email Server! That this particular “how” could be explained in 5 minutes is remarkable, if I may say so myself. In this NYCBUG talk — my first since 2014 — I show my work. It’s a real-world, open-source tale of methodically, incrementally reducing complexity in order to afford added functionality.

My abstract:

qmail 1.03 was notoriously bothersome to deploy. Twenty years later, for common use cases, I’ve finally made it pretty easy. If you want to try it out, I’ll help! (Don’t worry, it’s even easier to uninstall.) Or just listen as I share the sequence of stepwise improvements from then to now — including pkgsrc packaging, new code, and testing on lots of platforms — as well as the reasons I keep finding this project worthwhile.

Here’s the video:

February 23, 2019

Stack Overflow How to perform a 308 open redirect with php and apache?

I want to perform an open redirect so,

would redirect to
Here’s /index.cgi which of course has exec permissions.

header("Location: ".$_GET["endpoint"], true, 307);

and Here’s /flashredirect/.htaccess

Options FollowSymLinks
Options +ExecCGI
AddHandler cgi-script .cgi .pl
RewriteEngine On
RewriteBase /
FallbackResource /index.cgi

Obviously, there’s an error somewhere but where ? Also accessing error logs is payfull on so I can’t know the problem.

February 18, 2019

Stack Overflow NetBSD long double trouble

I have simple code:

 #include <stdio.h>

 int main()
      //char d[10] = {0x13, 0x43, 0x9b, 0x64, 0x28, 0xf8, 0xff, 0x7f, 0x00, 0x00};
      //long double rd = *(long double*)&d;
      long double rd = 3.3621e-4932L;
      printf("%Le\n", rd);
      return 0;

On my Ubuntu x64 it prints as expected 3.362100e-4932. On my NetBSD it prints 1.681050e-4932

Why it happens and how can I fix it? I try clang and gcc with same result.

My system (VM inside VirtualBox 5.0):

 uname -a
 NetBSD netbsd.home 7.0 NetBSD 7.0 (GENERIC.201509250726Z) amd64

 gcc --version
 gcc (nb2 20150115) 4.8.4

 clang --version
 clang version 3.6.2 (tags/RELEASE_362/final)
 Target: x86_64--netbsd
 Thread model: posix


/usr/include/x86/float.h defines as LDBL_MIN as 3.3621031431120935063E-4932L And this value is greater than printf result.

February 06, 2019

Stack Overflow Disabling/Enabling interrupts on x86 architectures

I am using NetBSD 5.1 for x86 systems. While studying some driver related code, I see that we use splraise and spllower to block or allow interrupts. I searched some of the mechanisms on internet to understand how these mechanisms work in reality. Did not get any real info on that.

When I disassembled I got the mechanism but still do not understand how all these assembly instruction yield me the result. I know x86 instruction individually, but not how the whole stuff works in its entirety.

Need your help in understanding its principles for x86 system. I understand that we need to disable Interrupt Enable (IE) bit, but this assembly seems to be doing more than just this work. Need help.

  (gdb) x/50i splraise
   0xc0100d40:  mov    0x4(%esp),%edx
   0xc0100d44:  mov    %fs:0x214,%eax
   0xc0100d4a:  cmp    %edx,%eax
   0xc0100d4c:  ja     0xc0100d55
   0xc0100d4e:  mov    %edx,%fs:0x214
   0xc0100d55:  ret
   0xc0100d56:  lea    0x0(%esi),%esi
   0xc0100d59:  lea    0x0(%edi,%eiz,1),%edi
   (gdb) p spllower
   $38 = {<text variable, no debug info>} 0xc0100d60
   0xc0100d60:  mov    0x4(%esp),%ecx
   0xc0100d64:  mov    %fs:0x214,%edx
   0xc0100d6b:  cmp    %edx,%ecx
   0xc0100d6d:  push   %ebx
   0xc0100d6e:  jae,pn 0xc0100d8f
   0xc0100d71:  mov    %fs:0x210,%eax
   0xc0100d77:  test   %eax,%fs:0x244(,%ecx,4)
   0xc0100d7f:  mov    %eax,%ebx
   0xc0100d81:  jne,pn 0xc0100d91
   0xc0100d84:  cmpxchg8b %fs:0x210
   0xc0100d8c:  jne,pn 0xc0100d71
   0xc0100d8f:  pop    %ebx
   0xc0100d90:  ret
   0xc0100d91:  pop    %ebx
   0xc0100d92:  jmp    0xc0100df0
   0xc0100d97:  mov    %esi,%esi
   0xc0100d99:  lea    0x0(%edi,%eiz,1),%edi
   0xc0100da0:  mov    0x4(%esp),%ecx
   0xc0100da4:  mov    %fs:0x214,%edx
   0xc0100dab:  cmp    %edx,%ecx
   0xc0100dad:  push   %ebx
   0xc0100dae:  jae,pn 0xc0100dcf
   0xc0100db1:  mov    %fs:0x210,%eax
   0xc0100db7:  test   %eax,%fs:0x244(,%ecx,4)
   0xc0100dbf:  mov    %eax,%ebx
   0xc0100dc1:  jne,pn 0xc0100dd1
   0xc0100dc4:  cmpxchg8b %fs:0x210
   0xc0100dcc:  jne,pn 0xc0100db1
   0xc0100dcf:  pop    %ebx
   0xc0100dd0:  ret
   0xc0100dd1:  pop    %ebx
   0xc0100dd2:  jmp    0xc0100df0
   0xc0100dd7:  mov    %esi,%esi
   0xc0100dd9:  lea    0x0(%edi,%eiz,1),%edi
   0xc0100de0:  nop
   0xc0100de1:  jmp    0xc0100df0

The code seems to be using a helper function cx8_spllower starting at address 0xc0100da0.

Unix Stack Exchange Setting wallpaper in NetBSD JWM

I have installed "NetBSD JWM" in the virtual environment of VMware Workstation 14 Pro. I'd like to set wallpaper but I do not know how to do it.

January 25, 2019

Amitai Schlair DevOpsDays NYC: Run Your @wn Email Server!

In late January, I was at DevOpsDays NYC in midtown Manhattan to present Run Your @wn Email Server!

My abstract:

When we’re responsible for production, it can be hard to find room to learn. That’s why I run my own email server. It’s still “production” — if it stays down, that’s pretty bad — but I own all the decisions, take more risks, and have learned lots. And so can you! Come see why and how to get started.

With one command, install famously secure email software. A couple more and it’s running. A few more and it’s encrypted. Twiddle your DNS, watch the mail start coming in, and start feeling responsible for a production service in a way that web hosting can’t match.

January 07, 2019

Amitai Schlair 2018Q4 qmail updates in pkgsrc

Happy 2019! Another three months, another stable branch for pkgsrc, the practical cross-platform Unix package manager. I’ve shipped quite a few improvements for qmail users in our 2018Q4 release. In three sentences:

  1. qmail-run gains TLS, SPF, IPv6, SMTP recipient checks, and many other sensible defaults.
  2. Most qmail-related packages — including the new ones used by qmail-run — are available on most pkgsrc platforms.
  3. rc.d-boot starts rc.conf-enabled pkgsrc services at boot time on many platforms.

In one:

It’s probably easy for you to run qmail now.

On this basis, at my DevOpsDays NYC talk in a few weeks, I’ll be recommending that everyone try it.

Try it

Here’s a demo on CentOS 7, using binary packages:

The main command I ran:

$ sudo env PKG_RCD_SCRIPTS=yes pkgin -y install qmail-run rc.d-boot

Here’s another demo on Debian 9, building from source packages:

The commands I ran:

$ cd ...pkgsrc/mail/qmail-run && make PKG_RCD_SCRIPTS=yes install
$ cd ../../pkgtools/rc.d-boot && make PKG_RCD_SCRIPTS=yes install

These improvements were made possible by acceptutils, my redesigned TLS and SMTP AUTH implementation that obviates the need for several large and conflicting patches. Further improvements are expected.

Here’s the full changelog for qmail as packaged in pkgsrc-2018Q4.




December 16, 2018

Unix Stack Exchange pkgin installation problem (NetBSD)

I just installed NetBSD 7.1.1 (i386) on my old laptop.

During the installation, I could not install pgkin (I don't know why), so I skipped it and now I have a NetBSD 7.1.1 installed on my laptop without pkgin.

My problem is "How to install pkgin on NetBSD (i386) ?"

I found this (Click) tutorial and I followed it:

I tried :

#export PKG_PATH=""
# pkg_add -v pkgin

And I got :

pkg_add: Can't process*: Not Found
pkg_add: no pkg found for 'pkgin',sorry.
pkg_add: 1 package addition failed

I know this is a wrong command because this ftp address is for amd64 while my laptop and NetBSD is i386. (I can't find the correct command for i386 )

I also followed instructions of (Click), and I did

git clone

on another computer and copied the output (which is a folder name pkgin) to my NetBSD (my NetBSD doesn't have 'git' command)

and then I did :

./configure --prefix=/usr/pkg --with-libraries=/usr/pkg/lib --with-includes=/usr/pkg/include

and then :


but I got :

#   compile  pkgin/summary.o
gcc -O2    -std=gnu99    -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-sign-compare  -Wno-traditional  -Wa,--fatal-warnings -Wreturn-type -Wswitch -Wshadow -Werror    -DPKGIN_VERSION=\""0.9.4 for NetBSD-7.1.1 i386"\" -DNETBSD  -g -DLOCALBASE=\"/usr/local\"           -DPKG_SYSCONFDIR=\"/usr/local/etc\"         -DPKG_DBDIR="\"/var/db/pkg\""           -DDEF_LOG_DIR="\"/var/db/pkg\""         -DPKGIN_DB=\"/var/db/pkgin\"            -DPKGTOOLS=\"/usr/local/sbin\" -DHAVE_CONFIG_H -D_LARGEFILE_SOURCE -D_LARGE_FILES -DCHECK_MACHINE_ARCH=\"i386\" -Iexternal -I. -I/usr/local/include  -c    summary.c
*** Error code 1

make: stopped in /root/pkgin

I think this error is because of the dependencies. (which is mentioned in but still, don't know how to install those dependencies.

EDIT: I found "" but it still says

no pkg fond for 'pkgin', sorry


Problem solved by writing 7.1 instead of 7.1.1

Unix Stack Exchange How to use 'pkg_add -uu' to upgrade all packages?

According to NetBSD's wiki I can use pkg_add -uu to upgrade packages. However, when I attempt to use pkg_add -uu it results in an error.

pkg_add -uu
pkg_add: missing package name(s)

pkg_add -uu *
pkg_add: no pkg found for `*`, sorry

pkg_add -uu all
pkg_add: no pkg found for `all`, sorry

I've tried to parse the pkg_add man page but I can't tell what the command it to update everything.

I can't use pkg_chk because its not installed, and I can't get the package system to install it:

pkg_chk -b
pkg_chk: command not found

pkg_add pkg_chk
pkg_add: no pkg found for `pkg_chk`, sorry

What is the secret command to get the OS to update everything?

September 15, 2018

Amitai Schlair Coding Tour Summer 2018: Conclusion

After my fourth and final tour stop, we decamped to Mallorca for a week. With no upcoming workshops to polish and no upcoming plans to finalize, the laptop stayed home. Just each other, a variety of beaches, and the annual Les Festes del Rei En Jaume that Bekki and I last saw two years ago on our honeymoon. The parade was perhaps a bit much for Taavi.

Looking away

The just-released episode 99 of Agile for Humans includes some reflections (starting around 50 minutes in) from partway through my coding tour. As our summer in Germany draws to a close, I’d like to reflect on the tour as a whole.

Annual training

I’ve made a habit of setting aside time, attention, and money each year for focused learning. My most recent trainings, all formative and memorable:

I hoped Schleier, Coding Tour would fit the bill for 2018. It has.

Geek joy

At the outset, I was asked how I’d know whether the tour had gone well. My response: “It’s a success if I get to meet a bunch of people in a bunch of places and we have fun programming together.”

I got to program with a bunch of people in a bunch of places. We had fun doing it. Success!

New technologies

My first tour stop offered such an ecumenical mix of languages, tools, and techniques that I began writing down each new technology I encountered. I’m glad I started at the beginning. Even so, this list of things that were new or mostly new to me is probably incomplete:

In the moment, learning new technologies was a source of geek joy. In the aggregate, it’s professionally useful. I think the weight clients tend to place on consultants needing to be expert in their tech stack is dangerously misplaced, but it doesn’t matter what I think if they won’t bring me in. Any chance for me to broaden my tech background is a chance for a future client to take advantage of all the other reasons I can be valuable to them.


As Schmonz’s Theorem predicts, code-touring is both similar to and different from consulting.

When consulting, I expect most of my learning to be meta: the second loop (at least) of double-loop learning. When touring, I became reacquainted with the simple joys of the first loop, spending all day learning new things to be able to do. It often felt like play.

When consulting, I initially find myself being listened to in a peculiar way, my words being heard and measured carefully for evidence of my real intentions. My first tasks are to demonstrate that I can be trusted and that I can be useful, not necessarily in that (or any) order. Accomplishing this as a programmer on tour felt easier than usual.

When I’m consulting, not everyone I encounter wants me there. Some offer time and attention because they feel obligated. On this tour, even though some folks were surprised to find out their employer wasn’t paying me anything, I sensed people were sharing their time and attention with me out of curiosity and generosity. I believe I succeeded in making myself trusted and useful to each of them, and the conversation videos and written testimonials help me hold the belief.

Professional development

With so much practice designing and facilitating group activities, so much information-rich feedback from participants, and so many chances to try again soon, I’ve leveled up as a facilitator. I was comfortable with my skills, abilities, and material before; I’m even more comfortable now. In my tour’s final public meetup, I facilitated one of my legacy code exercises for three simultaneous mobs. It went pretty well — in large part because of the participants, but also because of my continually developing skill at designing and facilitating learning experiences.

As a consultant, it’s a basic survival skill to quickly orient myself in new problem spaces. As a coach, my superpower might be that I help others quickly orient themselves in their problem spaces. Visiting many teams at many companies, I got lots of practice at both. These areas of strength for me are now stronger, the better with which to serve my next clients.

On several occasions I asked mobs not to bother explaining the current context to me before starting the timer. My hypothesis was, all the context I’d need would reveal itself through doing the work and asking a question or two along the way. (One basis among many for this hypothesis: what happened when I showed up late to one of Lennart Fridén’s sessions at this spring’s Mob Programming Conference and everyone else had already read the manual for our CPU.) I think there was one scenario where this didn’t work extremely well, but my memory’s fuzzy — have I mentioned meeting a whole bunch of people at a whole bunch of workplaces, meetups, and conferences? — so I’ll have to report the details when I rediscover it.

You can do this too, and I can help

When designing my tour, I sought advice from several people who’d gone on one. (Along the way I met several more, including Ivan Sanchez at SPA in London and Daniel Temme at SoCraTes in Soltau.)

If you’re wondering whether a coding tour is something you want to do, or how to make it happen, get in touch. I’m happy to listen and offer my suggestions.

What’s next for me, and you can help

Like what I’m doing? Want more of it in your workplace?

I offer short, targeted engagements in the New York metro area — coaching, consulting, and training — co-designed with you to meet your organization’s needs.

More at


Yes, lots.

It’s been a splendid set of privileges to have the free time to go on tour, to have organizations in several countries interested to have me code with them, and to meet so many people who care about what I care about when humans develop software together.

Five years ago I was discovering the existence of a set of communities of shared values in software development and my need to feel connected to them. Today I’m surer than ever that I’ve needed this connection and that I’ve found it.

Thanks to the people who hosted me for a week at their employer: Patrick Drechsler at MATHEMA/Redheads in Erlangen, Alex Schladebeck at BREDEX in Braunschweig, Barney Dellar at Canon Medical Research in Edinburgh, and Thorsten Brunzendorf at codecentric in Nürnberg and München. And thanks to these companies for being willing to take a chance on bringing in an itinerant programmer for a visit.

Thanks and apologies in equal measure to Richard Groß, who did all the legwork to have me visit MaibornWolff in Frankfurt, only to have me cancel at just about the last minute. At least we got to enjoy each other’s company at Agile Coach Camp Germany and SoCraTes (the only two people to attend both!).

Thanks to David Heath at the UK’s Government Digital Service for inviting me to join them on extremely short notice when I had a free day in London, and to Olaf Lewitz for making the connection.

Thanks to the meetups and conferences where I was invited to present: Mallorca Software Craft, SPA Software in Practice, pkgsrcCon, Hackerkegeln, JUG Ostfalen, Lean Agile Edinburgh, NEBytes, and Munich Software Craft. And thanks to Agile Coach Camp Germany and SoCraTes for the open spaces I did my part to fill.

Thanks to Marc Burgauer, Jens Schauder, and Jutta Eckstein for making time to join me for a meal. Thanks to Zeb Ford-Reitz, Barney Dellar, and their respective spice for inviting me into their respective homes for dinner.

Thanks to J.B. Rainsberger for simple, actionable advice on making it easy for European companies to reimburse my expenses, and more broadly on the logistics of going on European consulting-and-speaking tours when one is from elsewhere. (BTW, his next tour begins soon.)

Thanks all over again to everyone who helped me design and plan the tour, most notably Dr. Sal Freudenberg, Llewellyn Falco, and Nicole Rauch.

Thanks to Woody Zuill, Bryan Beecham, and Tim Bourguignon for that serendipitous conversation in the park in London. Thanks to Tim for having been there in the park with me. (No thanks to Woody for waiting till we’d left London before arriving. At least David Heath and GDS got to see him. Hmph.)

Thanks to Lisi Hocke for making my wish a reality: that her testing tour and my coding tour would intersect. As a developer, I have so much to learn about testing and so few chances to learn from the best. She made it happen. A perfect ending for my tour.

Thanks to Ryan Ripley for having me on Agile for Humans a couple more times as the tour progressed. I can’t say enough about what Ryan and his show have done for me, so this’ll have to be enough.

Thanks to everyone else who helped draw special attention to my tour when I was seeking companies to visit, most notably Kent Beck. It really did help.

Another reason companies cited for inviting me: my micropodcast, Agile in 3 Minutes. Thanks to Johanna Rothman, Andrea Goulet, Lanette Creamer, Alex Harms, and Jessica Kerr for your wonderful guest episodes. You’ve done me and our listeners a kindness. I trust it will come back to you.

Thank you to my family for supporting my attempts at growth, especially when I so clearly need it.

Finally, thanks to all of you for following along and for helping me find the kind of consulting work I’m best at, close to home in New York. You can count on me continuing to learn things and continuing to share them with you.


September 01, 2018

Amitai Schlair Coding Tour Stop #4: codecentric

[Update: Thorsten wrote a few words about our week together. They’re quite nice.]

The final week of my coding tour came right on the heels of SoCraTes. On Sunday evening I went straight from Soltau to Nuremberg. On Monday morning, when Thorsten Brunzendorf met me in the lobby of my hotel, it was the first time I’d seen him since… Sunday afternoon. Like several other already-familiar-by-now faces, he’d been at SoCraTes too.

This tour stop was different from the start: we’d spend our first three days in one city and the last two in another.




Mobbing in Scala with Thorsten and Martin


Introducing ‘Strangle Your Legacy Code'


Navigating during Lisi's session

Here are all #CodingTour tweets from the week.


Even though the odds keep stacking higher and higher, the pattern of each tour stop varying meaningfully from the previous ones continued at codecentric. For instance, we chose not to do Learning Hours. I think one reason I didn’t particularly miss it is that I was so busy learning. I’d seen Scala once before, for an hour; here we worked in it all week, both in katas and on production code. And property-based testing had been on my long list of techniques I know I need to learn about. Now I’ve had it explained to me simply and well by Thorsten and I’ve used it enough to know some ways it can help me go where I want to go.

Lisi and me

I’m a fan of Lisi’s testing tour and of how she writes about it. The chance to learn with her felt like the perfect ending for my tour. Here’s her post about it. Next week I’ll offer my reflections on the tour as a whole.

Thorsten, Martin, and I got together Friday afternoon to share some of our highlights from the week. Have a look.

March 17, 2018

Hubert Feyrer The adventure of rebuilding g4u from source
I was asked by a long-time g4u user on help with rebuilding g4u from sources. After pointing at the instructions on the homepage, we figured out that a few lose odds and ends didin't match. After bouncing some advices back and forth, I ventured into the frabjous joy of starting a rebuild from scratch, and quick enough ran into some problems, too.

Usually I cross-compile g4u from Mac OS X, but for the fun of it I did it on NetBSD (7.0-stable branch, amd64 architecture in VMware Fusion) this time. After waiting forever on the CVS checkout, I found that empty directories were not removed - that's what you get if you have -P in your ~/.cvsrc file.

I already had the hint that the "g4u-build" script needed a change to have "G4U_BUILD_KERNEL=true".

From there, things went almost smooth: building indicated a few files that popped up "variable may be used uninitialized" errors, and which -- thanks to -Werror -- bombed out the build. Fixing was easy, and I have no idea why that built for me on the release. I have sent a patch with the required changes to the g4u-help mailing list. (After fixing that I apparently got unsubscribed from my own support mailing list - thank you very much, Sourceforge ;)).

After those little hassles, the build worked fine, and gave me the floppy disk and ISO images that I expected:

>       ls -l `pwd`/g4u*fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u1.fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u2.fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u3.fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u4.fs
>       ls -l `pwd`/g4u.iso
>       -rw-r--r--  2 feyrer  staff  6567936 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u.iso
>       ls -l `pwd`/g4u-kernel.gz
>       -rw-r?r--  1 feyrer  staff  6035680 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u-kernel.gz 
Next steps are to confirm the above changes as working from my faithful tester, and then look into how to merge this into the build instructions .

January 12, 2018

Super User What is the default File System in NetBSD? What are it's benefits and shortcommings?

I spent some time looking through the documentation, but honestly, I have not found any good answer.

I understand NetBSD supports many FS types as USER SPACE, but I would like to know what is the default FS created by the installer, and the one which I could boot from.

January 04, 2018

Hubert Feyrer NetBSD 7.1.1 released
On December 22nd, NetBSD 7.1.1 was released as premature christmas present, see the release annoucement.

NetBSD 7.1.1 is the first update with security and critical fixes for the NetBSD 7.1 branch. Those include a number of fixes for security advisories, kernel and userland.

Hubert Feyrer New year, new security advisories!
So things have become a bit silent here, which is due to reallife - my apologies. Still, I'd like to wish everyone following this here a Happy New Year 2018! And with this, a few new security advisories have been published:
Hubert Feyrer 34C3 talk: Are all BSDs created equally?
I haven't seen this mentioned on the NetBSD mailing lists, and this may be of interest to some - there was a talk about security bugs in the various BSDs at the 34th Chaos Communication Congress:

In summary, many reasons for bugs are shown in many areas of the kernel (system calls, file systems, network stack, compat layer, ...), and what has happened after they were made known to the projects.

As a hint, NetBSD still has a number of Security Advisories to publish, it seems. Anyone wants to help out the security team? :-)

November 03, 2017

Super User Install Linux on Old AirPort Extreme?

I have a very old AirPort Extreme, the A1408. Is it possible to install Linux on it, using the AirPort functionally as a hard disk, and then boot from that? I have also heard that AirPorts run NetBSD. Can you boot into that and run commands?

August 06, 2017

Super User Trying to install NetBSD 7.1 on QEMU/KVM

I am using OpenSUSE Leap 42.2 on a Dell Inspiron 1545.  An error occurs when I try to load NetBSD into QEMU/KVM: "module 'cd9660' pushed by bootloader already exists." I installed Lubuntu 16.04 LTS in QEMU/KVM previously.

June 22, 2017

Server Fault How to log ssh client connection/command?

I would like to know how i could log SSH command lines a user is using on a server. For exemple, if the user Alex on my server is doing the following set of commands :

$ cd /tmp
$ touch myfile
$ ssh [email protected]
$ ssh [email protected]
$ vim anotherfile
$ ssh [email protected]

I would like to log the ssh commands used on the server in a file which looks like :

[2014-07-25 10:10:10] Alex : ssh [email protected]
[2014-07-25 10:18:20] Alex : ssh [email protected]
[2014-07-25 11:15:10] Alex : ssh [email protected]

I don't care what he did during his ssh session, i just want to know WHEN and TO WHERE he made a connection to another server.

The user is not using bash and i would like to avoid manipulating .bash_history anyway as the user can modify it.

Any clue on this ?

Thank you :)

edit : to be more specific :

a user connects to a server A and then connects from the server A to server B. I want to track down to which server he connects through ssh from server A.

June 11, 2017

NetBSD Installation and Upgrading on DaemonForums Update via CVS
Regarding to:

How can I tell NetBSD to only download x86 regarding files?
Is there a way to upgrade/install updates, which is more comparable to MS Windows updates?


June 08, 2017

Hubert Feyrer g4u 2.6 released
After a five-year period for beta-testing and updating, I have finally released g4u 2.6. With its origins in 1999, I'd like to say: Happy 18th Birthday, g4u!

About g4u: g4u ("ghosting for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

The past: When I started g4u, I had the task to install a number of lab machines with a dual-boot of Windows NT and NetBSD. The hype was about Microsoft's "Zero Administration Kit" (ZAK) then, but that did barely work for the Windows part - file transfers were slow, depended on the clients' hardware a lot (requiring fiddling with MS DOS network driver disks), and on the ZAK server the files for installing happened do disappear for no good reason every now and then. Not working well, and leaving out NetBSD (and everything elase), I created g4u. This gave me the (relative) pain of getting things working once, but with the option to easily add network drivers as they appeared in NetBSD (and oh they did!), plus allowed me to install any operating system.

The present: We've used g4u successfully in our labs then, booting from CDROM. I also got many donations from public and private instituations plus comanies from many sectors, indicating that g4u does make a difference.

In the mean time, the world has changed, and CDROMs aren't used that much any more. Network boot and USB sticks are today's devices of choice, cloning of a full disk without knowing its structure has both advantages but also disadvantages, and g4u's user interface is still command-line based with not much space for automation. For storage, FTP servers are nice and fast, but alternatives like SSH/SFTP, NFS, iSCSI and SMB for remote storage plus local storage (back to fun with filesystems, anyone? avoiding this was why g4u was created in the first place!) should be considered these days. Further aspects include integrity (checksums), confidentiality (encryption). This leaves a number of open points to address either by future releases, or by other products.

The future: At this point, my time budget for g4u is very limited. I welcome people to contribute to g4u - g4u is Open Source for a reason. Feel free to get back to me for any changes that you want to contribute!

The changes: Major changes in g4u 2.6 include:

The software: Please see the g4u homepage's download section on how to get and use g4u.


June 05, 2017

NetBSD General on DaemonForums X11R7 on NetBsd
I installed NetBsd on a older computer, and it is working basicly.
This is my first time with NetBsd,
However, I am having trouble getting the spanish keyboard to work
after I 'startx'.
From the console, when I first boot, the keyboard is fine, and the keys
work as expected, it is after I start the xsever that it become a problem.
I did read some manuals and info,..generated a

# X -config ~/
As instructed here:

But I can not find any X11 in /etc as mentioned:

If the above test was successful, move the file into place (as either /etc/X11/xorg.conf or /etc/X11/XF86Config) and you are ready to go. The following sections may be of interest or use, but are not required reading.
So I am wondering, not only what to put in the .conf file, the example they show is
for a german keyboard, and reveseing some keys,...I do not want to do that.
Just want it to know it is a spanish keyboard, no reversed keys, etc.
So, what to put in the file, and where to place the . conf file.
Thank you

February 23, 2017

Julio Merino Easy pkgsrc on macOS with pkg_comp 2.0

This is a tutorial to guide you through the shiny new pkg_comp 2.0 on macOS using the macOS-specific self-installer.

Goals: to use pkg_comp 2.0 to build a binary repository of all the packages you are interested in; to keep the repository fresh on a daily basis; and to use that repository with pkgin to maintain your macOS system up-to-date and secure.

This tutorial is specifically targeted at macOS and relies on the macOS-specific self-installer package. For a more generic tutorial that uses the pkg_comp-cron package in pkgsrc, see Keeping NetBSD up-to-date with pkg_comp 2.0.

Getting started

Start by downloading and installing OSXFUSE 3 and then download the standalone macOS installer package for pkg_comp. To find the right file, navigate to the releases page on GitHub, pick the most recent release, and download the file with a name of the form pkg_comp-<version>-macos.pkg.

Then double-click on the file you downloaded and follow the installation instructions. You will be asked for your administrator password because the installer has to place files under /usr/local/; note that pkg_comp requires root privileges anyway to run (because it uses chroot(8) internally), so you will have to grant permission at some point or another.

The installer modifies the default PATH (by creating /etc/paths.d/pkg_comp) to include pkg_comp’s own installation directory and pkgsrc’s installation prefix. Restart your shell sessions to make this change effective, or update your own shell startup scripts accordingly if you don’t use the standard ones.

Lastly, make sure to have Xcode installed in the standard /Applications/ location and that all components required to build command-line apps are available. Tip: try running cc from the command line and seeing if it prints its usage message.

Adjusting the configuration

The macOS flavor of pkg_comp is configured with an installation prefix of /usr/local/, which means that the executable is located in /usr/local/sbin/pkg_comp and the configuration files are in /usr/local/etc/pkg_comp/. This is intentional to keep the pkg_comp installation separate from your pkgsrc installation so that it can run no matter what state your pkgsrc installation is in.

The configuration files are as follows:

Note that these configuration files use the /var/pkg_comp/ directory as the dumping ground for: the pkgsrc tree, the downloaded distribution files, and the built binary packages. We will see references to this location later on.

The cron job

The installer configures a cron job that runs as root to invoke pkg_comp daily. The goal of this cron job is to keep your local packages repository up-to-date so that you can do binary upgrades at any time. You can edit the cron job configuration interactively by running sudo crontab -e.

This cron job won’t have an effect until you have populated the list.txt file as described above, so it’s safe to let it enabled until you have configured pkg_comp.

If you want to disable the periodic builds, just remove the pkg_comp entry from the crontab.

On slow machines, or if you are building a lot of packages, you may want to consider decreasing the build frequency from @daily to @weekly.

Sample configuration

Here is what the configuration looks like on my Mac Mini as dumped by the config subcommand. Use this output to get an idea of what to expect. I’ll be using the values shown here in the rest of the tutorial:

$ pkg_comp config
AUTO_PACKAGES = autoconf automake bash colordiff dash emacs24-nox11 emacs25-nox11 fuse-bindfs fuse-sshfs fuse-unionfs gdb git-base git-docs glib2 gmake gnuls libtool-base lua52 mercurial mozilla-rootcerts mysql56-server pdksh pkg_developer pkgconf pkgin ruby-jekyll ruby-jekyll-archives ruby-jekyll-paginate scmcvs smartmontools sqlite3 tmux vim
CVS_ROOT = :ext:[email protected]:/cvsroot
CVS_TAG is undefined
DISTDIR = /var/pkg_comp/distfiles
EXTRA_MKCONF = /usr/local/etc/pkg_comp/
GIT_BRANCH = trunk
LOCALBASE = /opt/pkg
PACKAGES = /var/pkg_comp/packages
PBULK_PACKAGES = /var/pkg_comp/pbulk-packages
PKG_DBDIR = /opt/pkg/libdata/pkgdb
PKGSRCDIR = /var/pkg_comp/pkgsrc
SANDBOX_CONFFILE = /usr/local/etc/pkg_comp/sandbox.conf
SYSCONFDIR = /opt/pkg/etc
VARBASE = /opt/pkg/var

SANDBOX_ROOT = /var/pkg_comp/sandbox
SANDBOX_TYPE = darwin-native

Building your own packages by hand

Now that you are fully installed and configured, you’ll build some stuff by hand to ensure the setup works before the cron job comes in.

The simplest usage form, which involves full automation and assumes you have listed at least one package in list.txt, is something like this:

$ sudo pkg_comp auto

This trivially-looking command will:

  1. clone or update your copy of pkgsrc;
  2. create the sandbox;
  3. bootstrap pkgsrc and pbulk;
  4. use pbulk to build the given packages; and
  5. destroy the sandbox.

After a successful invocation, you’ll be left with a collection of packages in the /var/pkg_comp/packages/ directory.

If you’d like to restrict the set of packages to build during a manually-triggered build, provide those as arguments to auto. This will override the contents of AUTO_PACKAGES (which was derived from your list.txt file).

But what if you wanted to invoke all stages separately, bypassing auto? The command above would be equivalent to:

$ sudo pkg_comp fetch
$ sudo pkg_comp sandbox-create
$ sudo pkg_comp bootstrap
$ sudo pkg_comp build <package names here>
$ sudo pkg_comp sandbox-destroy

Go ahead and play with these. You can also use the sandbox-shell command to interactively enter the sandbox. See pkg_comp(8) for more details.

Lastly note that the root user will receive email messages if the periodic pkg_comp cron job fails, but only if it fails. That said, you can find the full logs for all builds, successful or not, under /var/pkg_comp/log/.

Installing the resulting packages

Now that you have built your first set of packages, you will want to install them. This is easy on macOS because you did not use pkgsrc itself to install pkg_comp.

First, unpack the pkgsrc installation. You only have to do this once:

$ cd /
$ sudo tar xzvpf /var/pkg_comp/packages/bootstrap.tgz

That’s it. You can now install any packages you like:

$ PKG_PATH=file:///var/pkg_comp/packages/All sudo pkg_add pkgin <other package names>

The command above assume you have restarted your shell to pick up the correct path to the pkgsrc installation. If the call to pkg_add fails because of a missing binary, try restarting your shell or explicitly running the binary as /opt/pkg/sbin/pkg_add.

Keeping your system up-to-date

Thanks to the cron job that builds your packages, your local repository under /var/pkg_comp/packages/ will always be up-to-date; you can use that to quickly upgrade your system with minimal downtime.

Assuming you are going to use pkgtools/pkgin as recommended above (and why not?), configure your local repository:

$ sudo /bin/sh -c "echo file:///var/pkg_comp/packages/All >>/opt/pkg/etc/pkgin/repositories.conf"

And, from now on, all it takes to upgrade your system is:

$ sudo pkgin update
$ sudo pkgin upgrade


February 18, 2017

Julio Merino Keeping NetBSD up-to-date with pkg_comp 2.0

This is a tutorial to guide you through the shiny new pkg_comp 2.0 on NetBSD.

Goals: to use pkg_comp 2.0 to build a binary repository of all the packages you are interested in; to keep the repository fresh on a daily basis; and to use that repository with pkgin to maintain your NetBSD system up-to-date and secure.

This tutorial is specifically targeted at NetBSD but should work on other platforms with some small changes. Expect, at the very least, a macOS-specific tutorial as soon as I create a pkg_comp standalone installer for that platform.

Getting started

First install the sysutils/sysbuild-user package and trigger a full build of NetBSD so that you get usable release sets for pkg_comp. See sysbuild(1) and pkg_info sysbuild-user for details on how to do so. Alternatively, download release sets from the FTP site and later tell pkg_comp where they are.

Then install the pkgtools/pkg_comp-cron package. The rest of this tutorial assumes you have done so.

Adjusting the configuration

To use pkg_comp for periodic builds, you’ll need to do some minimal edits to the default configuration files. The files can be found directly under /var/pkg_comp/, which is pkg_comp-cron’s “home”:

Lastly, review root’s crontab to ensure the job specification for pkg_comp is sane. On slow machines, or if you are building many packages, you will probably want to decrease the build frequency from @daily to @weekly.

Sample configuration

Here is what the configuration looks like on my NetBSD development machine as dumped by the config subcommand. Use this output to get an idea of what to expect. I’ll be using the values shown here in the rest of the tutorial:

# pkg_comp -c /var/pkg_comp/pkg_comp.conf config
AUTO_PACKAGES = autoconf automake bash colordiff dash emacs-nox11 git-base git-docs gmake gnuls lua52 mozilla-rootcerts pdksh pkg_comp-cron pkg_developer pkgin sqlite3 sudo sysbuild sysbuild-user sysupgrade tmux vim zsh
CVS_ROOT = :ext:[email protected]:/cvsroot
CVS_TAG is undefined
DISTDIR = /var/pkg_comp/distfiles
EXTRA_MKCONF = /var/pkg_comp/
GIT_BRANCH = trunk
LOCALBASE = /usr/pkg
PACKAGES = /var/pkg_comp/packages
PBULK_PACKAGES = /var/pkg_comp/pbulk-packages
PKG_DBDIR = /usr/pkg/libdata/pkgdb
PKGSRCDIR = /var/pkg_comp/pkgsrc
SANDBOX_CONFFILE = /var/pkg_comp/sandbox.conf
VARBASE = /var

NETBSD_NATIVE_RELEASEDIR = /home/sysbuild/release/amd64
NETBSD_RELEASE_RELEASEDIR = /home/sysbuild/release/amd64
SANDBOX_ROOT = /var/pkg_comp/sandbox
SANDBOX_TYPE = netbsd-release

Building your own packages by hand

Now that you are fully installed and configured, you’ll build some stuff by hand to ensure the setup works before the cron job comes in.

The simplest usage form, which involves full automation, is something like this:

# pkg_comp -c /var/pkg_comp/pkg_comp.conf auto

This trivially-looking command will:

  1. checkout or update your copy of pkgsrc;
  2. create the sandbox;
  3. bootstrap pkgsrc and pbulk;
  4. use pbulk to build the given packages; and
  5. destroy the sandbox.

After a successful invocation, you’ll be left with a collection of packages in the directory you set in PACKAGES, which in the default pkg_comp-cron installation is /var/pkg_comp/packages/.

If you’d like to restrict the set of packages to build during a manually-triggered build, provide those as arguments to auto. This will override the contents of AUTO_PACKAGES (which was derived from your list.txt file).

But what if you wanted to invoke all stages separately, bypassing auto? The command above would be equivalent to:

# pkg_comp -c /var/pkg_comp/pkg_comp.conf fetch
# pkg_comp -c /var/pkg_comp/pkg_comp.conf sandbox-create
# pkg_comp -c /var/pkg_comp/pkg_comp.conf bootstrap
# pkg_comp -c /var/pkg_comp/pkg_comp.conf build <package names here>
# pkg_comp -c /var/pkg_comp/pkg_comp.conf sandbox-destroy

Go ahead and play with these. You can also use the sandbox-shell command to interactively enter the sandbox. See pkg_comp(8) for more details.

Lastly note that the root user will receive email messages if the periodic pkg_comp cron job fails, but only if it fails. That said, you can find the full logs for all builds, successful or not, under /var/pkg_comp/log/.

Installing the resulting packages

Now that you have built your first set of packages, you will want to install them. On NetBSD, the default pkg_comp-cron configuration produces a set of packages for /usr/pkg so you have to wipe your existing packages first to avoid build mismatches.

WARNING: Yes, you really have to wipe your packages. pkg_comp currently does not recognize the package tools that ship with the NetBSD base system (i.e. it bootstraps pkgsrc unconditionally, including bmake), which means that the newly-built packages won’t be compatible with the ones you already have. Avoid any trouble by starting afresh.

To clean your system, do something like this:

# ... ensure your login shell lives in /bin! ...
# pkg_delete -r -R "*"
# mv /usr/pkg/etc /root/etc.old  # Backup any modified files.
# rm -rf /usr/pkg /var/db/pkg*

Now, rebootstrap pkgsrc and reinstall any packages you previously had:

# cd /
# tar xzvpf /var/pkg_comp/packages/bootstrap.tgz
# echo "pkg_admin=/usr/pkg/sbin/pkg_admin" >>/etc/pkgpath.conf
# echo "pkg_info=/usr/pkg/sbin/pkg_info" >>/etc/pkgpath.conf
# export PATH=/usr/pkg/bin:/usr/pkg/sbin:${PATH}
# export PKG_PATH=file:///var/pkg_comp/packages/All
# pkg_add pkgin pkg_comp-cron <other package names>

Finally, reconfigure any packages where you had have previously made custom edits. Use the backup in /root/etc.old to properly update the corresponding files in /etc. I doubt you made a ton of edits so this should be easy.

IMPORTANT: Note that the last command in this example includes pkgin and pkg_comp-cron. You should install these first to ensure you can continue with the next steps in this tutorial.

Keeping your system up-to-date

If you paid attention when you installed the pkg_comp-cron package, you should have noticed that this configured a cron job to run pkg_comp daily. This means that your packages repository under /var/pkg_comp/packages/ will always be up-to-date so you can use that to quickly upgrade your system with minimal downtime.

Assuming you are going to use pkgtools/pkgin (and why not?), configure your local repository:

# echo 'file:///var/pkg_comp/packages/All' >>/etc/pkgin/repositories.conf

And, from now on, all it takes to upgrade your system is:

# pkgin update
# pkgin upgrade


February 17, 2017

Julio Merino Introducing pkg_comp 2.0 (and sandboxctl 1.0)

After many (many) years in the making, pkg_comp 2.0 and its companion sandboxctl 1.0 are finally here!

Read below for more details on this launch. I will publish detailed step-by-step tutorials on setting up periodic package rebuilds in separate posts.

What are these tools?

pkg_comp is an automation tool to build pkgsrc binary packages inside a chroot-based sandbox. The main goal is to fully automate the process and to produce clean and reproducible packages. A secondary goal is to support building binary packages for a different system than the one doing the builds: e.g. building packages for NetBSD/i386 6.0 from a NetBSD/amd64 7.0 host.

The highlights of pkg_comp 2.0, compared to the 1.x series, are: multi-platform support, including NetBSD, FreeBSD, Linux, and macOS; use of pbulk for efficient builds; management of the pkgsrc tree itself via CVS or Git; and a more robust and modern codebase.

sandboxctl is an automation tool to create and manage chroot-based sandboxes on a variety of operating systems. sandboxctl is the backing tool behind pk_comp. sandboxctl hides the details of creating a functional chroot sandbox on all supported operating systems; in some cases, like building a NetBSD sandbox using release sets, things are easy; but in others, like on macOS, they are horrifyingly difficult and brittle.

Storytelling time

pkg_comp’s history is a long one. pkg_comp 1.0 first appeared in pkgsrc on September 6th, 2002 as the pkgtools/pkg_comp package in pkgsrc. As of this writing, the 1.x series are at version 1.38 and have received contributions from a bunch of pkgsrc developers and external users; even more, the tool was featured in the BSD Hacks book back in 2004.

This is a long time for a shell script to survive in its rudimentary original form: pkg_comp 1.x is now a teenager at its 14 years of age and is possibly one of my longest-living pieces of software still in use.

Motivation for the 2.x rewrite

For many of these years, I have been wanting to rewrite pkg_comp to support other operating systems. This all started when I first got a Mac in 2005, at which time pkgsrc already supported Darwin but there was no easy mechanism to manage package updates. What would happen—and still happens to this day!—is that, once in a while, I’d realize that my packages were out of date (read: insecure) so I’d wipe the whole pkgsrc installation and start from scratch. Very inconvenient; I had to automate that properly.

Thus the main motivation behind the rewrite was primarily to support macOS because this was, and still is, my primary development platform. The secondary motivation came after writing sysbuild in 2012, which trivially configured daily builds of the NetBSD base system from cron; I wanted the exact same thing for my packages.

One, two… no, three rewrites

The first rewrite attempt was sometime in 2006, soon after I learned Haskell in school. Why Haskell? Just because that was the new hotness in my mind and it seemed like a robust language to drive a pretty tricky automation process. That rewrite did not go very far, and that’s possibly for the better: relying on Haskell would have decreased the portability of the tool, made it hard to install it, and guaranteed to alienate contributors.

The second rewrite attempt started sometime in 2010, about a year after I joined Google as an SRE. This was after I became quite familiar with Python at work, wanting to use the language to rewrite this tool. That experiment didn’t go very far though, but I can’t remember why… probably because I was busy enough at work and creating Kyua.

The third and final rewrite attempt started in 2013 while I had a summer intern and I had a little existential crisis. The year before I had written sysbuild and shtk, so I figured recreating pkg_comp using the foundations laid out by these tools would be easy. And it was… to some extent.

Getting the barebones of a functional tool took only a few weeks, but that code was far from being stable, portable, and publishable. Life and work happened, so this fell through the cracks… until late last year, when I decided it was time to close this chapter so I could move on to some other project ideas. To create the focus and free time required to complete this project, I had to shift my schedule to start the day at 5am instead of 7am—and, many weeks later, the code is finally here and I’m still keeping up with this schedule.

Granted: this third rewrite is not a fancy one, but it wasn’t meant to be. pkg_comp 2.0 is still written in shell, just as 1.x was, but this is a good thing because bootstrapping on all supported platforms is easy. I have to confess that I also considered Go recently after playing with it last year but I quickly let go of that thought: at some point I had to ship the 2.0 release, and 10 years since the inception of this rewrite was about time.

The launch of 2.0

On February 12th, 2017, the authoritative sources of pkg_comp 1.x were moved from pkgtools/pkg_comp to pkgtools/pkg_comp1 to make room for the import of 2.0. Yes, the 1.x series only existed in pkgsrc and the 2.x series exist as a standalone project on GitHub.

And here we are. Today, February 17th, 2017, pkg_comp 2.0 saw the light!

Why sandboxctl as a separate tool?

sandboxctl is the supporting tool behind pkg_comp, taking care of all the logic involved in creating chroot-based sandboxes on a variety of operating systems. Some are easy, like building a NetBSD sandbox using release sets, and others are horrifyingly difficult like macOS.

In pkg_comp 1.x, this logic used to be bundled right into the pkg_comp code, which made it pretty much impossible to generalize for portability. With pkg_comp 2.x, I decided to split this out into a separate tool to keep responsibilities isolated. Yes, the integration between the two tools is a bit tricky, but allows for better testability and understandability. Lastly, having sandboxctl as a standalone tool, instead of just a separate code module, gives you the option of using it for your own sandboxing needs.

I know, I know; the world has moved onto containerization and virtual machines, leaving chroot-based sandboxes as a very rudimentary thing… but that’s all we’ve got in NetBSD, and pkg_comp targets primarily NetBSD. Note, though, that because pkg_comp is separate from sandboxctl, there is nothing preventing adding different sandboxing backends to pkg_comp.


Installation is still a bit convoluted unless you are on one of the tier 1 NetBSD platforms or you already have pkgsrc up and running. For macOS in particular, I plan on creating and shipping a installer image that includes all of pkg_comp dependencies—but I did not want to block the first launch on this.

For now though, you need to download and install the latest source releases of shtk, sandboxctl, and pkg_comp—in this order; pass the --with-atf=no flag to the configure scripts to cut down the required dependencies. On macOS, you will also need OSXFUSE and the bindfs file system.

If you are already using pkgsrc, you can install the pkgtools/pkg_comp package to get the basic tool and its dependencies in place, or you can install the wrapper pkgtools/pkg_comp-cron package to create a pre-configured environment with a daily cron job to run your builds. See the package’s MESSAGE (with pkg_info pkg_comp-cron) for more details.


Both pkg_comp and sandboxctl are fully documented in manual pages. See pkg_comp(8), sandboxctl(8), pkg_comp.conf(5) and sandbox.conf(5) for plenty of additional details.

As mentioned at the beginning of the post, I plan on publishing one or more tutorials explaining how to bootstrap your pkgsrc installation using pkg_comp on, at least, NetBSD and macOS. Stay tuned.

And, if you need support or find anything wrong, please let me know by filing bugs in the corresponding GitHub projects: jmmv/pkg_comp and jmmv/sandboxctl.

February 09, 2017

BSD Talk bsdtalk266 - The nodes take over
We became tired of waiting. File Info: 7Min, 3MB. Ogg Link:

January 22, 2017

Emile Heitor CPU temperature collectd report on NetBSD

pkgsrc’s collectd does not support the thermal plugin, so in order to publish thermal information I had to use the exec plugin:

LoadPlugin exec
# more plugins

<Plugin exec>
Exec "nobody:nogroup" "/home/imil/bin/"

And write this simple script that reads CPUs temperature from NetBSD’s envstat command:

$ cat bin/ 


while :
envstat|awk '/cpu[0-9]/ {printf "%s %s\n",$1,$3}'|while read c t
echo "PUTVAL ${hostname}/temperature/temperature-zone${c#cpu} interval=${interval} N:${t%%.*}"
sleep ${interval}

I then send those values to an influxdb server:

LoadPlugin network
# ...

<Plugin network>
Server "" "25826"

And display them using grafana:

grafana setup
NetBSD temperature in grafana