NetBSD Planet


September 25, 2020

NetBSD Blog Google Summer of Code 2020: [Final Report] RumpKernel Syscall Fuzzing

This post is the third update to the project RumpKernel Syscall Fuzzing.

Part1 - https://blog.netbsd.org/tnf/entry/gsoc_reports_fuzzing_rumpkernel_syscalls1

Part2 - https://blog.netbsd.org/tnf/entry/gsoc_reports_fuzzing_rumpkernel_syscalls

The first and second coding period was entirely dedicated to fuzzing rumpkernel syscalls using hongfuzz. Initially a dumb fuzzer was developed to start fuzzing but it soon reached its limits.

For the duration of second coding peroid we concentrated on crash reproduction and adding grammar to the fuzzer which yielded in better results as we tested on a bug in ioctl with grammar. Although this works for now crash reproduction needs to be improved to generate a working c reproducer.

For the last coding period I have looked into the internals of syzkaller to understand how it pregenerates input and how it mutates data. I have continued to work on integrating buildrump.sh with build.sh. buildrump eases the task fo building the rumpkernel on any host for any target.

buildrump.sh is like a wrapper around build.sh to build the tools and rumpkernel from the source relevant to rumpkernel. So I worked to get buildrump.sh working with netbsd-src. Building the toolchain was successfull from netbsd-src. So binaries like rumpmake work just fine to continue building the rumpkernel.

But the rumpkernel failed to build due to some warnings and errors similar to the following. It can be due to the fact that buildrump.sh has been dormant recently I faced a lot of build issues.

nbmake[2]: nbmake[2]: don't know how to make /root/buildrump.sh/obj/dest.stage/usr/lib/crti.o. Stop

nbmake[2]: stopped in /root/buildrump.sh/src/lib/librumpuser
>> ERROR:
>> make /root/buildrump.sh/obj/Makefile.first dependall

Few of the similar errors were easily fixed but I couldn't integrate it during the time span of the coding period.

To Do

GSoC with NetBSD has been an amazing journey throughout, in which I had a chance to learn from awesome people and work on amazing projects. I will continue to work on the project to achieve the goal of integrating my fuzzer with OSS Fuzz. I thank my mentors Siddharth Muralee, Maciej Grochowski, Christos Zoulas for their support and Kamil for his continuous guidance.

NetBSD Blog Google Summer of Code 2020: [Final Report] Curses Library Automated Testing
My GSoC project under NetBSD involves the development of the test framework of curses. This is the final blog report in a series of blog reports; you can look at the first report and second report of the series.

The first report gives a brief introduction of the project and some insights into the curses testframe through its architecture and language. To someone who wants to contribute to the test suite, this blog can act as the quick guide of how things work internally. Meanwhile, the second report discusses some of the concepts that were quite challenging for me to understand. I wanted to share them with those who may face such a challenge. Both of these reports also cover the progress made in various phases of the Summer of Code.

This being the final report in the series, I would love to share my experience throughout the project. I would be sharing some of the learning as well as caveats that I faced in the project.

Challenges and Caveats

By the time my application for GSoC was submitted, I had gained some knowledge about the curses library and the testing framework. Combined with compiler design and library testing experience, that knowledge proved useful but not sufficient as I progressed through the project. There were times when, while writing a test case, you have to look into documentation from various sources, be it NetBSD, FreeBSD, Linux, Solaris, etc. One may find questioning his understanding of the framework, documentation, or even curses itself. This leads to the conclusion that for being a tester, one has to become a user first. That made me write minimal programs to understand the behavior. The experience was excellent, and I felt amazed by the capability and complexity of curses.

Learnings

The foremost learning is from the experience of interacting with the open-source community and feeling confident in my abilities to contribute. Understanding the workflows; following the best practices like considering the maintainability, readability, and simplicity of the code were significant learning.

The project-specific learning was not limited to test-framework but a deeper understanding of curses as I have to browse through codes for the functions tested. As this blog says, getting the TTY demystified was a long-time desire, which got fulfilled to some extent.

Some tests from test suite

In this section, I would discuss a couple of tests of the test suite written during the third phase of GSoC. Curses input model provides a variety of ways to obtain input from keyboard. We will consider 2 tests keypad and halfdelay that belong to input processing category but are somewhat unrelated.

Keypad Processing

An application can enable or disable the tarnslation of keypad using keypad() function. When translation is enabled, curses attempts to translate input sequence into a single key code. If disabled, curses passes the input as it is and any interpretation has to be made by application.

include window
call $FALSE is_keypad $win1
input "\eOA"
call 0x1b wgetch $win1
call OK keypad $win1 $TRUE
input "\eOA"
call $KEY_UP wgetch $win1

# disable assembly of KEY_UP
call OK keyok $KEY_UP $FALSE
input "\eOA"
call 0x1b wgetch $win1

As keypad translation is disabled by default, on input of '\eOA', the input sequence is passed as it is and only '\e' (0x1b is hex code) is received on wgetch(). If we enable the translation, then the same input is translated as KEY_UP. In curses, one can disable assembly of specific key symbols using keyok(). See related man page.

Input Mode

Curses lets the application control the effect of input using four input modes; cooked, cbreak, half-delay, raw. They specify the effect of input in terms of echo-ing and delay. We will discuss about the halfdelay mode. The half-delay mode specifies how quickly certain curses function return to application when there is no terminal input waiting since the function is called.

include start
delay 1000
# input delay 1000 equals to 10 tenths of seconds
# getch must fail for halfdelay(5) and pass for halfdelay(15)
input "a"
call OK halfdelay 15
call 0x61 getch
call OK halfdelay 5
input "a"
call -1 getch

We have set the delay for feeding input to terminal with delay of 1s(10 tenths of second). If the application sets the halfdelay to 15, and makes a call to getch() it receives the input. But it fails to get the input with haldelay set to 5. See related man page.

Project Work

The work can be merged into organisation repository https://github.com/NetBSD/src under tests/lib/libcurses.

This project involved:

  1. Improvement in testframework:
    • Automation of the checkfile generation.
    • Enhnacement of support for complex character
    • Addition of small features and code refactoring
  2. Testing and bug reports:
    • Tests for a family of routines like wide character, complex character, line drawing, box drawing, pad, window operations, cursor manipulations, soft label keys, input-output stream, and the ones involving their interactions.
    • Raising a bunch of Problem Report (PR) under lib category some of which have been fixed. The list of PRs raised can be found here

Future Work

Acknowledgements

I want to extend my heartfelt gratitude to my mentor Mr. Brett Lymn, who helped me through all the technical difficulties and challenges I faced. I also thank my mentor Martin Huseman for valuable suggestions and guidance at various junctures of the project. A special thanks to Kamil Rytarowski for making my blogs published on the NetBSD site.


September 23, 2020

UnitedBSD Opening pinentry in xterm from passmenu(dmenu)

I am using pass to manage my passwords, and I also use the [passmenu](https://git.zx2c4.com/password-store/tree/contrib/dmenu/passmenu), a script that uses dmenu to select which password you want to copy to the clipboard.

Now on my Linux machine running Gnome, upon selecting a password from the passmenu a full screen authentication overlay (Pinentry?) appears asking for the master password. Please correct me if this is not pinentry
It looks like this:

On my NetBSD machine however, using only a stand alone WM, I am unable to authenticate pass when launched from passmenu as no pinentry is triggered. When launched from within a terminal however with pass -c github.com for example, you get this:

Essentially passmenu is useless if I cant authenticate it. Therefore my question is, would it be possible to use passmenu to spawn this pinentry program/event in an xterm ?


September 22, 2020

UnitedBSD samba problem

hi guys ,i need some favour,i install and config samba follow "https://wiki.netbsd.org/tutorials/how_to_set_up_a_samba_server/", it show " protocol negotiation failed: NT_STATUS_IO_TIMEOUT" when i run "smbclient -Usamba -L localhost" i dont know what i miss?thanks

and this my step and config file
pkgin install samba
cp /usr/pkg/share/examples/rc.d/smbd /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/nmbd /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/samba /etc/rc.d/
and in /etc/rc.conf
smbd=YES
nmbd=YES
samba=YES
and my /usr/pkg/etc/samba/smb.conf
[global]
workgroup = WORKGROUP
server string = Samba Server
hosts allow = 192.168.31.
log file = /var/log/log.%m
max log size = 50
dns proxy = no
[homes]
comment = Home Directories
browseable = no
writable = yes
[shared]
comment = Shared
path = /home/zero/work
browseable = yes
writable = yes
guest ok = yes
[printers]
comment = All Printers
path = /usr/spool/samba
browseable = no
guest ok = no
writable = yes
printable = yes


September 21, 2020

NetBSD Package System (pkgsrc) on DaemonForums samba problem
hi guys ,i need some favour,i install and config samba follow "https://wiki.netbsd.org/tutorials/how_to_set_up_a_samba_server/",
it show " protocol negotiation failed: NT_STATUS_IO_TIMEOUT" when i run "smbclient -Usamba -L localhost" i dont konw what i miss?thanks

and this my step and config file
pkgin install samba
cp /usr/pkg/share/examples/rc.d/smbd /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/nmbd /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/samba /etc/rc.d/
and in /etc/rc.conf
smbd=YES
nmbd=YES
samba=YES
follow is my /usr/pkg/etc/samba/smb.conf
Code:

# smbclient -L localhost
protocol negotiation failed: NT_STATUS_IO_TIMEOUT
# vim /usr/pkg/etc/samba/smb.conf
# cat  /usr/pkg/etc/samba/smb.conf
# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options (perhaps too
# many!) most of which are not shown in this example
#
# For a step to step guide on installing, configuring and using samba,
# read the Samba-HOWTO-Collection. This may be obtained from:
#  http://www.samba.org/samba/docs/Samba-HOWTO-Collection.pdf
#
# Many working examples of smb.conf files can be found in the
# Samba-Guide which is generated daily and can be downloaded from:
#  http://www.samba.org/samba/docs/Samba-Guide.pdf
#
# Any line which starts with a ; (semi-colon) or a # (hash)
# is a comment and is ignored. In this example we will use a #
# for commentry and a ; for parts of the config file that you
# may wish to enable
#
# NOTE: Whenever you modify this file you should run the command "testparm"
# to check that you have not made any basic syntactic errors.
#
#======================= Global Settings =====================================
[global]

# workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
  workgroup = WORKGROUP

# server string is the equivalent of the NT Description field
  server string = Samba Server

# Server role. Defines in which mode Samba will operate. Possible
# values are "standalone server", "member server", "classic primary
# domain controller", "classic backup domain controller", "active
# directory domain controller".
#
# Most people will want "standalone server" or "member server".
# Running as "active directory domain controller" will require first
# running "samba-tool domain provision" to wipe databases and create a
# new domain.
#  server role = standalone server

# This option is important for security. It allows you to restrict
# connections to machines which are on your local network. The
# following example restricts access to two C class networks and
# the "loopback" interface. For more examples of the syntax see
# the smb.conf man page
  hosts allow = 192.168.31.

# Uncomment this if you want a guest account, you must add this to /etc/passwd
# otherwise the user "nobody" is used
;  guest account = pcguest

# this tells Samba to use a separate log file for each machine
# that connects
  log file = /var/log/log.%m

# Put a capping on the size of the log files (in Kb).
  max log size = 50

# Specifies the Kerberos or Active Directory realm the host is part of
;  realm = MY_REALM

# Backend to store user information in. New installations should
# use either tdbsam or ldapsam. smbpasswd is available for backwards
# compatibility. tdbsam requires no further configuration.
;  passdb backend = tdbsam

# Using the following line enables you to customise your configuration
# on a per machine basis. The %m gets replaced with the netbios name
# of the machine that is connecting.
# Note: Consider carefully the location in the configuration file of
#      this line.  The included file is read at that point.
;  include = /usr/local/samba/lib/smb.conf.%m

# Configure Samba to use multiple interfaces
# If you have multiple network interfaces then you must list them
# here. See the man page for details.
;  interfaces = 192.168.12.2/24 192.168.13.2/24

# Where to store roving profiles (only for Win95 and WinNT)
#        %L substitutes for this servers netbios name, %U is username
#        You must uncomment the [Profiles] share below
;  logon path = \\%L\Profiles\%U

# Windows Internet Name Serving Support Section:
# WINS Support - Tells the NMBD component of Samba to enable it's WINS Server
;  wins support = yes

# WINS Server - Tells the NMBD components of Samba to be a WINS Client
#      Note: Samba can be either a WINS Server, or a WINS Client, but NOT both
;  wins server = w.x.y.z

# WINS Proxy - Tells Samba to answer name resolution queries on
# behalf of a non WINS capable client, for this to work there must be
# at least one  WINS Server on the network. The default is NO.
;  wins proxy = yes

# DNS Proxy - tells Samba whether or not to try to resolve NetBIOS names
# via DNS nslookups. The default is NO.
  dns proxy = no

# These scripts are used on a domain controller or stand-alone
# machine to add or delete corresponding unix accounts
;  add user script = /usr/sbin/useradd %u
;  add group script = /usr/sbin/groupadd %g
;  add machine script = /usr/sbin/adduser -n -g machines -c Machine -d /dev/null -s /bin/false %u
;  delete user script = /usr/sbin/userdel %u
;  delete user from group script = /usr/sbin/deluser %u %g
;  delete group script = /usr/sbin/groupdel %g


#============================ Share Definitions ==============================
[homes]
  comment = Home Directories
  browseable = no
  writable = yes

[shared]
comment = Shared
path = /home/zero/work
browseable = yes
writable = yes
guest ok = yes
# Un-comment the following and create the netlogon directory for Domain Logons
; [netlogon]
;  comment = Network Logon Service
;  path = /usr/local/samba/lib/netlogon
;  guest ok = yes
;  writable = no
;  share modes = no


# Un-comment the following to provide a specific roving profile share
# the default is to use the user's home directory
;[Profiles]
;    path = /usr/local/samba/profiles
;    browseable = no
;    guest ok = yes


# NOTE: If you have a BSD-style print system there is no need to
# specifically define each individual printer
[printers]
  comment = All Printers
  path = /usr/spool/samba
  browseable = no
# Set public = yes to allow user 'guest account' to print
  guest ok = no
  writable = yes
  printable = yes

# This one is useful for people to share files
;[tmp]
;  comment = Temporary file space
;  path = /tmp
;  read only = no
;  public = yes

# A publicly accessible directory, but read only, except for people in
# the "staff" group
;[public]
;  comment = Public Stuff
;  path = /home/samba
;  public = yes
;  writable = no
;  printable = no
;  write list = @staff

# Other examples.
#
# A private printer, usable only by fred. Spool data will be placed in fred's
# home directory. Note that fred must have write access to the spool directory,
# wherever it is.
;[fredsprn]
;  comment = Fred's Printer
;  valid users = fred
;  path = /homes/fred
;  printer = freds_printer
;  public = no
;  writable = no
;  printable = yes

# A private directory, usable only by fred. Note that fred requires write
# access to the directory.
;[fredsdir]
;  comment = Fred's Service
;  path = /usr/somewhere/private
;  valid users = fred
;  public = no
;  writable = yes
;  printable = no

# a service which has a different directory for each machine that connects
# this allows you to tailor configurations to incoming machines. You could
# also use the %U option to tailor it by user name.
# The %m gets replaced with the machine name that is connecting.
;[pchome]
;  comment = PC Directories
;  path = /usr/pc/%m
;  public = no
;  writable = yes

# A publicly accessible directory, read/write to all users. Note that all files
# created in the directory by users will be owned by the default user, so
# any user with access can delete any other user's files. Obviously this
# directory must be writable by the default user. Another user could of course
# be specified, in which case all files would be owned by that user instead.
;[public]
;  path = /usr/somewhere/else/public
;  public = yes
;  only guest = yes
;  writable = yes
;  printable = no

# The following two entries demonstrate how to share a directory so that two
# users can place files there that will be owned by the specific users. In this
# setup, the directory should be writable by both users and should have the
# sticky bit set on it to prevent abuse. Obviously this could be extended to
# as many users as required.
;[myshare]
;  comment = Mary's and Fred's stuff
;  path = /usr/somewhere/shared
;  valid users = mary fred
;  public = no
;  writable = yes
;  printable = no
;  create mask = 0765


September 19, 2020

NetBSD General on DaemonForums New Shared Lib on i386
Hi,

NetBSD i386 9.0 (GENERIC) #0: Fri Feb 14 00:06:28 UTC 2020

I have my own personal shared library, which complies and installs fine. When I compile programs that link against that library, the compile succeeds.

But when I attempt to run one of the programs I get:

Quote:

/usr/local/lib/libj_lib.so: text relocations
/usr/local/lib/libj_lib.so: Cannot write-enable text segment: Permission denied
I even tried a reboot without luck.

I did many searches but all I found was issues compiling mplayer and to use ld.so.conf, but nothing else. I read ld.so.conf is no longer needed and man pages has nothing I can find.

Does anyone know what I can do to fix this issue ?

edit: no luck with "-Wl,-R/usr/local/lib" as noted in the elf FAQ

Thanks
John
DragonFly BSD Digest In Other BSDs for 2020/09/19

No theme, just everything in a bucket.

UnitedBSD wfsb

I feel this is probably one of the bigger "problems" for people new to NetBSD, and although I'm not really familiar enough with it to be sure of that, if I wanted to help NetBSD I would promote wfsb for people with the problems it is a solution to.

Will probably turn into a support thread-- that's cool, I mean I've got a NetBSD machine right next to me that could benefit, though I really did bring it up to champion the idea. The good news is it already exists! I won't say I'm too surprised, I thought as an idea it was pretty obvious.


September 17, 2020

NetBSD General on DaemonForums Howto power off usb devices
Under FreeBSD, I can power off a usb device with usbconfig's "power_off" option. How can I do that under NetBSD?

Code:

# usbstats
Controller /dev/usb0:
      1069 control
        0 isochronous
 154806725 bulk
  8954336 interrupt
Controller /dev/usb1:
      375 control
        0 isochronous
  33390421 bulk
        2 interrupt

# usbdevs
addr 1: EHCI root hub, NetBSD
 addr 2: Rate Matching Hub, Intel
  addr 5: External USB 3.0, Toshiba
  addr 3: USB Optical Mouse, Primax Electronics
  addr 4: Kensington U+P Keyboard, NOVATEK
addr 1: EHCI root hub, NetBSD
 addr 2: Rate Matching Hub, Intel
  addr 3: USB2.0-CRW, Generic

I want to power off (and later on) External USB 3.0, Toshiba, which I believe is at usb0, addr 5.

September 15, 2020

Server Fault Non-standard IP address with dashes

I ran the who command on a shared NetBSD box, and this weird user IP came up:

<redacted> pts/33   May 13 02:13  (XXX.XXX.XXX.XXX)
<redacted> pts/35   May 12 20:59  (202-172-110-147-)
<redacted> pts/36   May  6 20:36  (XXX.XXX.XXX.XXX)

I've never seen an IP like that. Obviously, ping 202-172-110-147- will complain with "Cannot resolve ... (Unknown host)".

There was a similar question posted 7 years ago, which posited that it was a non-standard way of denoting IP ranges, but seeing there's a - at the end of the address, it doesn't seem like a similar thing.


Edit:

I've tried reverse DNS with nslookup 202-172-110-147-, which errors with "** server can't find 202-172-110-147-: NXDOMAIN"

Doing w <user> returns:

9:49AM  up 89 days,  7:46, 1 user, load averages: 0.23, 0.18, 0.17
USER          TTY     FROM              [email protected]  IDLE WHAT
<redacted>    pts/35  202-172-110-147- Tue08PM  4:13

Edit 2: This is on NetBSD, not Linux like I mentioned at the beginning (I thought the box was Linux):

$ uname -rsv
NetBSD 8.1 NetBSD 8.1 (GENERIC) #0: Fri May 31 08:43:59 UTC 2019  [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC

Edit 3: Following @NStorm update, I ran w -n to display network addresses as numbers. I still see the same result

$ w -n <user>
 9:57AM  up 89 days,  7:54, 1 user, load averages: 0.12, 0.12, 0.14
USER          TTY     FROM              [email protected]  IDLE WHAT
<redacted>    pts/35  202-172-110-147- Tue08PM  4:22

September 14, 2020

Ruben Schade Feedback on my Mr Orange post

Good morning! My post about Mr Orange generated the most email feedback I’ve had since my encrypted-ZFS on NetBSD post, all of it negative. I can only assume someone shared it among Orange supporters.

I’d publish each message in full, but they’re all just a little too tragic. One of the more civil gentleman—though we’re coming from a low baseline here—attempted to debunk everything I said, but the substance of each quote was that:

It spoke to the internal machinations and mental gymnastics of someone desperate to absolve their leader of culpability. All the facts cited were wrong, and only served to reinforce my original thought that his supporters have absolutely no idea how the rest of the world perceives them. This kind of transparent projection and self-ownage isn’t unique to their flavour of politics, but it does attract a disproportionate amount of it.

I do feel a responsibility to reach these people. Each Orange supporter has their reasons, and they have the same family, job security and other worries we all do. Perhaps even moreso, which is why they’re so motivated to accept any convenient excuse for why the man they support continues to brazenly abuse them. But when you’re coming from a place of bad faith to start, I have no motivation to engage with you.

And that’s the problem. We could have a reasonable discussion if we were coming from a place of honesty. When the Dear Leader regularly lies, and people like Scott Adams say it’s a political tactic, we have no basis upon which to even start. Facts are sacrosanct, just as so many far-right people ironically state on their Twitter profiles. He didn’t have the biggest turnout at his inauguration. He didn’t get Mexico to pay for his wall. America isn’t doing the best at dealing with COVID. Wind turbines don’t cause cancer. He didn’t get Kim Jong Un to do anything. We can debate the finer points of his other policies, but if people can’t even admit to these, they’re being just as dishonest as him.

(I also see this playing out to a lesser extent with Scott Morrison in Australia, who has made no secret of his admiration for the American knob-in-chief).

Thanks for taking the time to respond everyone, but your tactics to win over another supporter in your “culture war” have backfired. And unlike so many of you who still cling to him because he’s “owning lefties”, I take no pleasure whatsoever in pointing that out. The sooner they realise they’re transparently doing the work of our common enemies better than anyone else, the sooner we can begin to rebuild what we’ve all lost. Because being an outspoken friend of America has been hard these last few years.


This post originally appeared on Rubenerd.

Unix Stack Exchange Modifying firewall rules with pfctl on NetBSD 4

I'm wanting to modify some firewall/NAT rules on a device (Apple Airport) running NetBSD 4.0. I'm not that familiar with BSD and pf, so want to check the right approach. I can change the pf.conf file (using sed) to the required configuration, then load using pfctl -f /etc/pf.conf, but am wondering if this clears out the old rules first?

Alternately, what would be the best approach to doing this directly via the command line (i.e. removing old the rdr pass and nat rules, and adding new ones?)

(for context, the aim here is to be able to change NAT rules on an Apple Airport Extreme without having to restart the device, which brings the whole network down for a minute or so; I've successfully gained ssh access)


September 13, 2020

Ruben Schade Forgetting to set UTF normalisation on a ZFS pool

Clara and I were getting some bizarre behavior while accessing a new FreeBSD pool over Netatalk and Samba. A subset of files with CJK names were showing up in the macOS Finder as expected, but would error out with file not found if you tried to open them.

We store a lot of files in Japanese and Korean, especially music and holiday photo directories with place names, so I’ve always been careful about using UTF-8 globally. I confirmed I had this in my /etc/login.conf:

default:\[...]:charset=UTF-8:\:lang=en_US.UTF-8:

(NOTE: I’ve read this isn’t advisable because it can break ports that weren’t designed for UTF-8. I’ve never had that issue, but it’s something to keep in mind. I’d also be worried if software in 2020 still had that limitation, but that’s a topic for another post).

Then I confirmed the ZFS pool was set up for UTF-8:

# zfs get utf8only pool==> NAME  PROPERTY  VALUE  SOURCE==> zten  utf8only  on     -

So what was going on?

# zfs get normalization pool==> NAME  PROPERTY       VALUE  SOURCE==> zten  normalization  none   -

Whoops!

Normalisation is a field of information science that fills entire textbooks, but in a nutshell ZFS uses it to reconcile filenames. This is especially important in UTF-8 where characters from disparate languages may appear superficially the same, such as Chinese-derived kanji in Japanese. How the filename is represented internally, and presented to the operator, can vary in unexpected ways.

Unfortunately, normalisation can’t be set after the filesystem is created. So this weekend I dropped one of the drives from my mirror, created a new pool with normalisation to transfer data back to, then resilvered the mirror back to full redundancy:

# zpool -O normalization=formD [...]

Now previously-inaccessible files can be opened.


This post originally appeared on Rubenerd.


September 12, 2020

DragonFly BSD Digest In Other BSDs for 2020/09/12

Perhaps a performance tweaks mini-theme this week?

 

 

 

NetBSD Blog GSoC Reports: Benchmarking NetBSD, third evaluation report

This report was written by Apurva Nandan as part of Google Summer of Code 2020.

Introduction

This blog post is in continuation of GSoC Reports: Benchmarking NetBSD, first evaluation report and GSoC Reports: Benchmarking NetBSD, second evaluation report blogs, and describes my progress in the final phase of GSoC 2020 under The NetBSD Foundation.

In the third phase, I upgraded to the latest stable version Phoronix Test Suite (PTS) 9.8.0 in pkgsrc-wip, resolved the TODOs and created patches for more test-profiles to fix their installation and runtime errors on NetBSD-current.

Progress in the third phase of GSoC

wip/phoronix-test-suite TODO and update

As a newer stable version of the Phoronix Test Suite was available in upstream, I upgraded the Phoronix Test Suite from version 9.6.1 to 9.8.0 in pkgsrc-wip and is available as wip/phoronix-test-suite. You can have a look at the PTS Changelog to know about the improvements between these two versions.

To get the package ready for merge in pkgsrc upstream, I also resolved the pkgsrc-wip TODOs.

pkgsrc-wip commits:

If any new problems are encountered, please document them in wip/phoronix-test-suite/TODO file and/or contact me.

Testing of automated benchmarking framework

I had been assigned a remote testing machine having Intel 6138 dual processor, 40 cores, 80 threads, 192GB of RAM. I spent time reproducing my automated framework i.e., Phoromatic-Anita Integration on the machine. I was able to reproduce the integration framework working without networking configuration, but the network bridge needs to be setup on the remote machine and the integration script to be tested with it. I shall continue this task in the post-GSoC period.

Benchmarking Results

I also performed benchmarking of NetBSD-9 amd64 native installation by running 50 test-profiles on a remote machine assigned to me by mentors and uploaded the benchmark results to OpenBenchmarking.org at:

Test-profile debugging

I then continued the task of maintaining/porting test-profiles and fixed the following test-profiles:

Timed FFmpeg Compilation

This test times how long it takes to build FFmpeg. This test is part of Processor Test category.

Original Test-profile:

https://openbenchmarking.org/test/pts/build-ffmpeg

Patched Test-profile:

https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/build-ffmpeg-1.0.1

Commit:

Compile Bench

Compilebench tries to age a filesystem by simulating some of the disk IO common in creating, compiling, patching, stating and reading kernel trees. It indirectly measures how well filesystems can maintain directory locality as the disk fills up and directories age. This test is part of Disk Test category.

Original Test-profile:

https://openbenchmarking.org/test/pts/compilebench

Patched Test-profile:

https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/compilebench-1.0.2

Commit:

Timed MAFFT Alignment

This test performs an alignment of 100 pyruvate decarboxylase sequences. This test is part of Processor Test category.

Original Test-profile:

https://openbenchmarking.org/test/pts/mafft

Patched Test-profile:

https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/mafft-1.5.0

Commits:

Future Plans

This officially summarizes my GSoC project: Benchmark NetBSD, and my end goal of the project that is to integrate Phoronix Test Suite with NetBSD and Anita for automated benchmarking is complete and its deployment on benchmark.NetBSD.org will be continued to be worked on with the coordination of moderators and merging the wip of Phoronix Test Suite 9.8.0 will be done by the pkgsrc maintainers in next days.

I want to thank my mentors and the NetBSD community without whose constant support I wouldn't have achieved the goals.

UnitedBSD How can I boot NetBSD from a different HD?

Hey everyone.
(First off: I'm a newcomer to both BSD and this forum, so I apologize in advance if I posted in the wrong place or if the solution is embarrassingly obvious.)

What's the right way™ to boot NetBSD from a different physical drive than the one it's installed on?

I'm planning to run NetBSD on an HP Microserver (Gen 8), which has quirky booting behavior.
Without going into too much gory detail, the problem is that the server's only SATA port is not recognized by the BIOS unless all other drives are unplugged, or unless hardware RAID is enabled. (And neither is an option for me.)
What does always work though is booting from any of the USB ports or the internal card reader, so until now (with Linux) I simply threw /boot/ and GRUB onto a microSD and called it a day.

I tried to replicate that kind of setup with NetBSD, but for the life of me can't get it to work.
Tried fiddling around with installboot, fdisk, and disklabel for hours on end, but it seems like I'm either missing something in the manpages or generally misunderstand how the boot process works in NetBSD.
First off, none of the tools seem to toggle the MBR boot flag, which causes my buggy BIOS to completely ignore the device at startup. Secondly, when I "manually" toggle that flag the bootloader does launch, but instantly exits, claiming that the /boot file can't be found (despite the file being there on both the microSD and the HDD).
This was the case with both FFS w. bootxx_ffsv2 and FAT32 w. bootxx_msdos.

Any kind of hints/pointers are appreciated 👀
Thanks


September 10, 2020

NetBSD Blog The GNU GDB Debugger and NetBSD (Part 4)
The NetBSD team of developers maintains two copies of GDB:

The base-system version of GDB (GPLv3) still relies on a set of local patches. I set a goal to reduce the local patches to bare minimum, ideally reaching no local modifications at all.

GDB changes

Over the past month I worked on gdbserver for NetBSD/amd64 and finally upstreamed it to the GDB mainline, just in time for GDB 10.

What is gdbserver? Let's quote the official GDB documentation:

gdbserver is a control program for Unix-like systems, which allows you to connect your program with a remote GDB via target remote or target extended-but without linking in the usual debugging stub.

gdbserver is not a complete replacement for the debugging stubs, because it requires essentially the same operating-system facilities that GDB itself does. In fact, a system that can run gdbserver to connect to a remote GDB could also run GDB locally! gdbserver is sometimes useful nevertheless, because it is a much smaller program than GDB itself. It is also easier to port than all of GDB, so you may be able to get started more quickly on a new system by using gdbserver. Finally, if you develop code for real-time systems, you may find that the tradeoffs involved in real-time operation make it more convenient to do as much development work as possible on another system, for example by cross-compiling. You can use gdbserver to make a similar choice for debugging.

GDB and gdbserver communicate via either a serial line or a TCP connection, using the standard GDB remote serial protocol. remote

This illustrated that gdbserver is especially useful for debugging applications on embedded and thin devices, connected to a controlling computer equipped with full distribution sources, toolchain, debugging information etc. Eventually, this approach of gdb and gdbserver can replace the native gdb plugin entirely and spawn all connections debugging sessions using this protocol. This design decision was already introduced into LLDB, where remote process plugin is the only supported program on Linux, NetBSD and highly recommended for other kernels.

I've picked amd64 as the first target as it's the easiest to develop and test.

An example debugging session looks like this:

$ uname -rms
NetBSD 9.99.72 amd64
$ LC_ALL=C date
Thu Sep 10 22:43:10 CEST 2020
$ ./gdbserver/gdbserver --version                
GNU gdbserver (GDB) 10.0.50.20200910-git
Copyright (C) 2020 Free Software Foundation, Inc.
gdbserver is free software, covered by the GNU General Public License.
This gdbserver was configured as "x86_64-unknown-netbsd9.99"
$ ./gdbserver/gdbserver localhost:1234 /usr/bin/nslookup
Process /usr/bin/nslookup created; pid = 26383
Listening on port 1234

Then on the other terminal:

$ ./gdb/gdb 
GNU gdb (GDB) 10.0.50.20200910-git
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-unknown-netbsd9.99".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
--Type  for more, q to quit, c to continue without paging--
    .

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
Reading /usr/bin/nslookup from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /usr/bin/nslookup from remote target...
Reading symbols from target:/usr/bin/nslookup...
Reading /usr/bin/nslookup.debug from remote target...
Reading /usr/bin/.debug/nslookup.debug from remote target...
Reading /usr/libdata/debug//usr/bin/nslookup.debug from remote target...
Reading /usr/libdata/debug//usr/bin/nslookup.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/bin/nslookup.debug...
process 28353 is executing new program: /usr/bin/nslookup
Reading /usr/bin/nslookup from remote target...
Reading /usr/bin/nslookup from remote target...
Reading /usr/bin/nslookup.debug from remote target...
Reading /usr/bin/.debug/nslookup.debug from remote target...
Reading /usr/libdata/debug//usr/bin/nslookup.debug from remote target...
Reading /usr/libdata/debug//usr/bin/nslookup.debug from remote target...
Reading /usr/libexec/ld.elf_so from remote target...
Reading /usr/libexec/ld.elf_so from remote target...
Reading /usr/libexec/ld.elf_so.debug from remote target...
Reading /usr/libexec/.debug/ld.elf_so.debug from remote target...
Reading /usr/libdata/debug//usr/libexec/ld.elf_so.debug from remote target...
Reading /usr/libdata/debug//usr/libexec/ld.elf_so.debug from remote target...
warning: Invalid remote reply: timeout [kamil: repeated multiple times...]
Reading /usr/lib/libbind9.so.15 from remote target...
Reading /usr/lib/libisccfg.so.15 from remote target...
Reading /usr/lib/libdns.so.15 from remote target...
Reading /usr/lib/libns.so.15 from remote target...
Reading /usr/lib/libirs.so.15 from remote target...
Reading /usr/lib/libisccc.so.15 from remote target...
Reading /usr/lib/libisc.so.15 from remote target...
Reading /usr/lib/libkvm.so.6 from remote target...
Reading /usr/lib/libz.so.1 from remote target...
Reading /usr/lib/libblocklist.so.0 from remote target...
Reading /usr/lib/libpthread.so.1 from remote target...
Reading /usr/lib/libpthread.so.1.4.debug from remote target...
Reading /usr/lib/.debug/libpthread.so.1.4.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libpthread.so.1.4.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libpthread.so.1.4.debug from remote target...
Reading /usr/lib/libgssapi.so.11 from remote target...
Reading /usr/lib/libheimntlm.so.5 from remote target...
Reading /usr/lib/libkrb5.so.27 from remote target...
Reading /usr/lib/libcom_err.so.8 from remote target...
Reading /usr/lib/libhx509.so.6 from remote target...
Reading /usr/lib/libcrypto.so.14 from remote target...
Reading /usr/lib/libasn1.so.10 from remote target...
Reading /usr/lib/libwind.so.1 from remote target...
Reading /usr/lib/libheimbase.so.2 from remote target...
Reading /usr/lib/libroken.so.20 from remote target...
Reading /usr/lib/libsqlite3.so.1 from remote target...
Reading /usr/lib/libcrypt.so.1 from remote target...
Reading /usr/lib/libutil.so.7 from remote target...
Reading /usr/lib/libedit.so.3 from remote target...
Reading /usr/lib/libterminfo.so.2 from remote target...
Reading /usr/lib/libc.so.12 from remote target...
Reading /usr/lib/libgcc_s.so.1 from remote target...
Reading symbols from target:/usr/lib/libbind9.so.15...
Reading /usr/lib/libbind9.so.15.0.debug from remote target...
Reading /usr/lib/.debug/libbind9.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libbind9.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libbind9.so.15.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libbind9.so.15.0.debug...
Reading symbols from target:/usr/lib/libisccfg.so.15...
Reading /usr/lib/libisccfg.so.15.0.debug from remote target...
Reading /usr/lib/.debug/libisccfg.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libisccfg.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libisccfg.so.15.0.debug from remote target...
--Type  for more, q to quit, c to continue without paging--
Reading symbols from target:/usr/libdata/debug//usr/lib/libisccfg.so.15.0.debug...
Reading symbols from target:/usr/lib/libdns.so.15...
Reading /usr/lib/libdns.so.15.0.debug from remote target...
Reading /usr/lib/.debug/libdns.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libdns.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libdns.so.15.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libdns.so.15.0.debug...
Reading symbols from target:/usr/lib/libns.so.15...
Reading /usr/lib/libns.so.15.0.debug from remote target...
Reading /usr/lib/.debug/libns.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libns.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libns.so.15.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libns.so.15.0.debug...
Reading symbols from target:/usr/lib/libirs.so.15...
Reading /usr/lib/libirs.so.15.0.debug from remote target...
Reading /usr/lib/.debug/libirs.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libirs.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libirs.so.15.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libirs.so.15.0.debug...
Reading symbols from target:/usr/lib/libisccc.so.15...
Reading /usr/lib/libisccc.so.15.0.debug from remote target...
Reading /usr/lib/.debug/libisccc.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libisccc.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libisccc.so.15.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libisccc.so.15.0.debug...
Reading symbols from target:/usr/lib/libisc.so.15...
Reading /usr/lib/libisc.so.15.0.debug from remote target...
Reading /usr/lib/.debug/libisc.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libisc.so.15.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libisc.so.15.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libisc.so.15.0.debug...
Reading symbols from target:/usr/lib/libkvm.so.6...
Reading /usr/lib/libkvm.so.6.0.debug from remote target...
Reading /usr/lib/.debug/libkvm.so.6.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libkvm.so.6.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libkvm.so.6.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libkvm.so.6.0.debug...
Reading symbols from target:/usr/lib/libz.so.1...
Reading /usr/lib/libz.so.1.0.debug from remote target...
Reading /usr/lib/.debug/libz.so.1.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libz.so.1.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libz.so.1.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libz.so.1.0.debug...
Reading symbols from target:/usr/lib/libblocklist.so.0...
Reading /usr/lib/libblocklist.so.0.0.debug from remote target...
Reading /usr/lib/.debug/libblocklist.so.0.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libblocklist.so.0.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libblocklist.so.0.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libblocklist.so.0.0.debug...
Reading symbols from target:/usr/lib/libgssapi.so.11...
Reading /usr/lib/libgssapi.so.11.0.debug from remote target...
Reading /usr/lib/.debug/libgssapi.so.11.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libgssapi.so.11.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libgssapi.so.11.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libgssapi.so.11.0.debug...
Reading symbols from target:/usr/lib/libheimntlm.so.5...
Reading /usr/lib/libheimntlm.so.5.0.debug from remote target...
Reading /usr/lib/.debug/libheimntlm.so.5.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libheimntlm.so.5.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libheimntlm.so.5.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libheimntlm.so.5.0.debug...
Reading symbols from target:/usr/lib/libkrb5.so.27...
Reading /usr/lib/libkrb5.so.27.0.debug from remote target...
Reading /usr/lib/.debug/libkrb5.so.27.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libkrb5.so.27.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libkrb5.so.27.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libkrb5.so.27.0.debug...
Reading symbols from target:/usr/lib/libcom_err.so.8...
Reading /usr/lib/libcom_err.so.8.0.debug from remote target...
Reading /usr/lib/.debug/libcom_err.so.8.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libcom_err.so.8.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libcom_err.so.8.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libcom_err.so.8.0.debug...
Reading symbols from target:/usr/lib/libhx509.so.6...
Reading /usr/lib/libhx509.so.6.0.debug from remote target...
Reading /usr/lib/.debug/libhx509.so.6.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libhx509.so.6.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libhx509.so.6.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libhx509.so.6.0.debug...
Reading symbols from target:/usr/lib/libcrypto.so.14...
Reading /usr/lib/libcrypto.so.14.0.debug from remote target...
Reading /usr/lib/.debug/libcrypto.so.14.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libcrypto.so.14.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libcrypto.so.14.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libcrypto.so.14.0.debug...
Reading symbols from target:/usr/lib/libasn1.so.10...
Reading /usr/lib/libasn1.so.10.0.debug from remote target...
Reading /usr/lib/.debug/libasn1.so.10.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libasn1.so.10.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libasn1.so.10.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libasn1.so.10.0.debug...
Reading symbols from target:/usr/lib/libwind.so.1...
Reading /usr/lib/libwind.so.1.0.debug from remote target...
Reading /usr/lib/.debug/libwind.so.1.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libwind.so.1.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libwind.so.1.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libwind.so.1.0.debug...
Reading symbols from target:/usr/lib/libheimbase.so.2...
Reading /usr/lib/libheimbase.so.2.0.debug from remote target...
Reading /usr/lib/.debug/libheimbase.so.2.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libheimbase.so.2.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libheimbase.so.2.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libheimbase.so.2.0.debug...
Reading symbols from target:/usr/lib/libroken.so.20...
Reading /usr/lib/libroken.so.20.0.debug from remote target...
Reading /usr/lib/.debug/libroken.so.20.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libroken.so.20.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libroken.so.20.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libroken.so.20.0.debug...
Reading symbols from target:/usr/lib/libsqlite3.so.1...
Reading /usr/lib/libsqlite3.so.1.4.debug from remote target...
Reading /usr/lib/.debug/libsqlite3.so.1.4.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libsqlite3.so.1.4.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libsqlite3.so.1.4.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libsqlite3.so.1.4.debug...
Reading symbols from target:/usr/lib/libcrypt.so.1...
Reading /usr/lib/libcrypt.so.1.0.debug from remote target...
Reading /usr/lib/.debug/libcrypt.so.1.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libcrypt.so.1.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libcrypt.so.1.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libcrypt.so.1.0.debug...
Reading symbols from target:/usr/lib/libutil.so.7...
Reading /usr/lib/libutil.so.7.24.debug from remote target...
Reading /usr/lib/.debug/libutil.so.7.24.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libutil.so.7.24.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libutil.so.7.24.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libutil.so.7.24.debug...
Reading symbols from target:/usr/lib/libedit.so.3...
Reading /usr/lib/libedit.so.3.1.debug from remote target...
Reading /usr/lib/.debug/libedit.so.3.1.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libedit.so.3.1.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libedit.so.3.1.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libedit.so.3.1.debug...
Reading symbols from target:/usr/lib/libterminfo.so.2...
Reading /usr/lib/libterminfo.so.2.0.debug from remote target...
Reading /usr/lib/.debug/libterminfo.so.2.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libterminfo.so.2.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libterminfo.so.2.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libterminfo.so.2.0.debug...
Reading symbols from target:/usr/lib/libc.so.12...
Reading /usr/lib/libc.so.12.217.debug from remote target...
Reading /usr/lib/.debug/libc.so.12.217.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libc.so.12.217.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libc.so.12.217.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libc.so.12.217.debug...
Reading symbols from target:/usr/lib/libgcc_s.so.1...
Reading /usr/lib/libgcc_s.so.1.0.debug from remote target...
Reading /usr/lib/.debug/libgcc_s.so.1.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libgcc_s.so.1.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/libgcc_s.so.1.0.debug from remote target...
Reading symbols from target:/usr/libdata/debug//usr/lib/libgcc_s.so.1.0.debug...
Reading /usr/libexec/ld.elf_so from remote target...
_rtld_debug_state () at /usr/src/libexec/ld.elf_so/rtld.c:1577
1577            __insn_barrier();
(gdb) b main
Breakpoint 1 at 0x211c00: file /usr/src/external/mpl/bind/bin/nslookup/../../dist/bin/dig/nslookup.c, line 990.
(gdb) c
Continuing.

Breakpoint 1, main (argc=1, argv=0x7f7fffffe768)
    at /usr/src/external/mpl/bind/bin/nslookup/../../dist/bin/dig/nslookup.c:990
990     main(int argc, char **argv) {
(gdb) bt
#0  main (argc=1, argv=0x7f7fffffe768)
    at /usr/src/external/mpl/bind/bin/nslookup/../../dist/bin/dig/nslookup.c:990
(gdb) info threads 
  Id   Target Id          Frame 
* 1    Thread 28353.28353 main (argc=1, argv=0x7f7fffffe768)
    at /usr/src/external/mpl/bind/bin/nslookup/../../dist/bin/dig/nslookup.c:990
(gdb) b pthread_setname_np
Breakpoint 2 at 0x7f7ff4e0c9e4: file /usr/src/lib/libpthread/pthread.c, line 792.
(gdb) c
Continuing.
[New Thread 28353.27773]

Thread 1 hit Breakpoint 2, pthread_setname_np (thread=0x7f7ff7e41000, 
    [email protected]=0x7f7fffffe610 "work-0", [email protected]=0x0)
    at /usr/src/lib/libpthread/pthread.c:792
792     {
(gdb) info threads
  Id   Target Id          Frame 
* 1    Thread 28353.28353 pthread_setname_np (thread=0x7f7ff7e41000, 
    [email protected]=0x7f7fffffe610 "work-0", [email protected]=0x0)
    at /usr/src/lib/libpthread/pthread.c:792
  2    Thread 28353.27773 0x00007f7ff0aa623a in ___lwp_park60 () from target:/usr/lib/libc.so.12
(gdb) n
796             pthread__error(EINVAL, "Invalid thread",
(gdb) n
799             if (pthread__find(thread) != 0)
(gdb) 
802             namelen = snprintf(newname, sizeof(newname), name, arg);
(gdb) 
803             if (namelen >= PTHREAD_MAX_NAMELEN_NP)
(gdb) 
806             cp = strdup(newname);
(gdb) 
807             if (cp == NULL)
(gdb) 
810             pthread_mutex_lock(&thread->pt_lock);
(gdb) 
811             oldname = thread->pt_name;
(gdb) 
812             thread->pt_name = cp;
(gdb) 
813             (void)_lwp_setname(thread->pt_lid, cp);
(gdb) 
814             pthread_mutex_unlock(&thread->pt_lock);
(gdb) n
816             if (oldname != NULL)
(gdb) n
isc_taskmgr_create (mctx=, [email protected]=1, default_quantum=, 
    [email protected]=0, [email protected]=0x0, [email protected]=0x418638 )
    at /usr/src/external/mpl/bind/lib/libisc/../../dist/lib/isc/task.c:1431
1431            for (i = 0; i < workers; i++) {
(gdb) info threads
  Id   Target Id                   Frame 
* 1    Thread 28353.28353          isc_taskmgr_create (mctx=, [email protected]=1, 
    default_quantum=, [email protected]=0, [email protected]=0x0, 
    [email protected]=0x418638 )
    at /usr/src/external/mpl/bind/lib/libisc/../../dist/lib/isc/task.c:1431
  2    Thread 28353.27773 "work-0" 0x00007f7ff0aa623a in ___lwp_park60 ()
   from target:/usr/lib/libc.so.12
(gdb) dis 1
(gdb) b exit
Breakpoint 3 at 0x7f7ff0b530e0: exit. (2 locations)
(gdb) c
Continuing.

Thread 1 hit Breakpoint 2, pthread_setname_np (thread=0x7f7ff7e42c00, 
    [email protected]=0x7f7ff5e6324e "isc-timer", [email protected]=0x0)
    at /usr/src/lib/libpthread/pthread.c:792
792     {
(gdb) dis 2
(gdb) c
Continuing.
Reading /usr/lib/i18n/libUTF8.so.5.0 from remote target...
Reading /usr/lib/i18n/libUTF8.so.5.0.debug from remote target...
Reading /usr/lib/i18n/.debug/libUTF8.so.5.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/i18n/libUTF8.so.5.0.debug from remote target...
Reading /usr/libdata/debug//usr/lib/i18n/libUTF8.so.5.0.debug from remote target...

Then, back to the first terminal:

> netbsd.org
Server:         62.179.1.62
Address:        62.179.1.62#53

Non-authoritative answer:
Name:   netbsd.org
Address: 199.233.217.205
Name:   netbsd.org
Address: 2001:470:a085:999::80
> exit

Thread 1 hit Breakpoint 3, exit (status=1) at /usr/src/lib/libc/stdlib/exit.c:55
55      {
(gdb) info threads 
  Id   Target Id          Frame 
* 1    Thread 28353.28353 exit (status=1) at /usr/src/lib/libc/stdlib/exit.c:55
(gdb) bt
#0  exit (status=1) at /usr/src/lib/libc/stdlib/exit.c:55
#1  0x0000000000206122 in ___start ()
#2  0x00007f7ff7c0c840 in ?? () from target:/usr/libexec/ld.elf_so
#3  0x0000000000000001 in ?? ()
#4  0x00007f7fffffed20 in ?? ()
#5  0x0000000000000000 in ?? ()
(gdb) kill
Kill the program being debugged? (y or n) y
[Inferior 1 (process 28353) killed]

It worked!

In order to get this functionality operational I had to implement multiple GDB functions, in particular: create_inferior, post_create_inferior, attach, kill, detach, mourn, join, thread_alive, resume, wait, fetch_registers, store_registers, read_memory, write_memory, request_interrupt, supports_read_auxv, read_auxv, supports_hardware_single_step, sw_breakpoint_from_kind, supports_z_point_type, insert_point, remove_point, stopped_by_sw_breakpoint, supports_qxfer_siginfo, qxfer_siginfo, supports_stopped_by_sw_breakpoint, supports_non_stop, supports_multi_process, supports_fork_events, supports_vfork_events, supports_exec_events, supports_disable_randomization, supports_qxfer_libraries_svr4, qxfer_libraries_svr4, supports_pid_to_exec_file, pid_to_exec_file, thread_name, supports_catch_syscall.

NetBSD is the first BSD and actually the first Open Source UNIX-like OS besides Linux to grow support for gdbserver.

Plan for the next milestone

Introduce AArch64 support for GDB/NetBSD.


September 09, 2020

UnitedBSD NetBSD installer

A good work has been done on the server infrastructure, which is now much more reliable.
The next step towards easy and reproducible installations is improving sysinst.
I don't know why it has remained so - let's say, "perfectible" - for such a long time.
If the only reason is nobody is willing to work on it, I would be interested in helping on this task, hence the following questions:

I'm thinking of picking good ideas from FreeBSD, Void Linux and Debian's installers for user experience, and solving partitioning and boot issues for reproducibility.


September 08, 2020

Unix Stack Exchange Does *BSD have the ability to encrypt a system partition with full disk encryption?

Does FreeBSD, NetBSD, or OpenBSD have an encryption feature like Linux's dm-crypt? And will it work for a system partition?

Benny Siegert pkgsrc Developer Monotony
Somehow, my contributions to NetBSD and pkgsrc have become monotonous. Because I am busy with work, family and real life, the amount of time I can spend on open source is fairly limited, and I have two commitments that I try to fulfill: Member of pkgsrc-releng: I process most of the pull-ups to the stable quarterly branch. Maintainer of Go and its infrastructure. Unfortunately, these things are always kinda the same.

September 07, 2020

Frederic Cambus Playing with Kore JSON API

Kore 4.0.0 has been released a few days ago, and features a brand new JSON API allowing to easily parse and serialize JSON objects.

During the last couple of years, I have been using Kore for various projects, including exposing hardware sensor values over the network via very simple APIs. In this article, I would like to present a generalization of this concept and show how easy it is to expose system information with Kore.

This small API example allows to identify hosts over the network and has been tested on Linux, OpenBSD, NetBSD, and macOS (thanks Joris!).

After creating a new project:

kodev create identify

Populate src/identify.c with the following code snippet:

#include <sys/utsname.h>

#include <kore/kore.h>
#include <kore/http.h>

#if defined(__linux__)
#include <kore/seccomp.h>

KORE_SECCOMP_FILTER("json",
	KORE_SYSCALL_ALLOW(uname)
);
#endif

int		page(struct http_request *);

int
page(struct http_request *req)
{
	char *answer;

	struct utsname u;

	struct kore_buf buf;
	struct kore_json_item *json;

	if (uname(&u) == -1) {
		http_response(req, HTTP_STATUS_INTERNAL_ERROR, NULL, 0);
		return (KORE_RESULT_OK);
	}

	kore_buf_init(&buf, 1024);
	json = kore_json_create_object(NULL, NULL);

	kore_json_create_string(json, "system", u.sysname);
	kore_json_create_string(json, "hostname", u.nodename);
	kore_json_create_string(json, "release", u.release);
	kore_json_create_string(json, "version", u.version);
	kore_json_create_string(json, "machine", u.machine);

	kore_json_item_tobuf(json, &buf);

	answer = kore_buf_stringify(&buf, NULL);
	http_response(req, 200, answer, strlen(answer));

	kore_buf_cleanup(&buf);
	kore_json_item_free(json);

	return (KORE_RESULT_OK);
}

And finally launch the project:

kodev run

The kodev tool will build and run the project, and we can now query the API to identify hosts:

{
  "system": "OpenBSD",
  "hostname": "foo.my.domain",
  "release": "6.8",
  "version": "GENERIC.MP#56",
  "machine": "amd64"
}

September 05, 2020

DragonFly BSD Digest In Other BSDs for 2020/09/05

It’s a good mix this week.

 


September 03, 2020

Ruben Schade The roles of OSs have changed

I had an overdue realisation last night:

All of this is such a shift from before. Macs used to be seen as creative tools, not something you’d use for business. DOS and 1990s Windows were endlessly frustrating, not sources of fun. Linux couldn’t have been broadly used for games.

I can still remember in the early 2000s bringing my iBook G3 into a corporate office for school work experience, and all the ensuing discussion about what it was, and whether it could be supported by their corporate LAN. They were even more confused when I booted it into NetBSD. How things change.

Though I do wonder sometimes, with just a slight tweak to history, how things might have been different. In another dimension somewhere, I’m using the latest BeOS-powered PowerPC laptop, and a shiny new Palm smartphone. Both of these represented the pinnacle of UI design in the 1990s, and still in the 2020s have yet to be surpassed. People call me an Apple fanboy, but I’d drop all of it in a second for that gear.


This post originally appeared on Rubenerd.


September 02, 2020

NetBSD General on DaemonForums restart network
I ran the command
Code:

# ifconfig mue0 down
to take the computer offline, but how do I now bring it back up?
When I try
Code:

# ifconfig mue0 up
I get
Code:

exec_matches: No buffer space available
error

What is the NetBSD equivalent of the FreeBSD command
Code:

# /etc/rc.d/netif restart
or OpenBSD equivalent
Code:

# /etc/netstart
I want to be able to restart all network services

August 31, 2020

Frederic Cambus Modernizing the OpenBSD console

At the beginning were text mode consoles. Traditionally, *BSD and Linux on i386 and amd64 used text mode consoles which by default provided 25 rows of 80 columns, the "80x25 mode". This mode uses a 8x16 font stored in the VGA BIOS (which can be slightly different across vendors).

OpenBSD uses the wscons(4) console framework, inherited from NetBSD.

CRT monitors allowed to set the resolution you wanted, so on bigger monitors a 80x25 console in textmode was fairly large but not blurry.

Framebuffer consoles allowed taking advantage of larger monitor sizes, to fit more columns and row. With the switch to LCD monitors, also in part driven by the decreasing costs of laptops, the fixed size panels became a problem as the text mode resolution needed to be stretched, leading to distortion and blurriness.

One thing some people might not realize, is the huge discrepancy between text mode and framebuffer consoles regarding the amount of data you have to write to cover the whole screen. In text mode, we only need to write 2 bytes per character: 1 byte for the ASCII code, and 1 byte for attributes. So in 80x25 text mode, we only need to write 80 * 25 * 2 bytes of data, which is 4000 bytes, and the VGA card itself takes care of plotting characters to the screen. In framebuffer however, to fill a 4K UHD-1 (3840x2160) screen in 32bpp mode we need to send 3840 * 2160 * 4 bytes of data, which is 33177600 bytes (approximately 33 MB).

On framebuffer consoles, OpenBSD uses the rasops(9) subsystem (raster operations), imported from NetBSD in 2001.

While they had been used for a while on platforms without VGA cards, framebuffer consoles were only enabled on i386 and amd64 in 2013 for inteldrm(4) and radeondrm(4).

In recent years, rasops(9) itself and framebuffer drivers have seen some improvements:

General improvements:

Performance related improvements:

Console fonts improvements:

There is an article about Spleen in the OpenBSD Journal with more information, notably on the font selection mechanism relative to screen resolution.

And work slowly continues to make framebuffer consoles more usable.

It is interesting to note that while NetBSD has been adding a lot of features to rasops(9) over the years, OpenBSD has taken a more conservative approach. There is however one major feature that NetBSD currently has which would be beneficial: the capability for loading fonts of different metrics and subsequently resizing screens.

Looking forward, performance of various operations could likely still be improved, possibly by leveraging the new OpenBSD dynamic tracing mechanism to analyze bottlenecks.

Another open question is UTF-8 support, Miod Vallat started work in this direction back in 2013 but there are still a few things missing. I have plans to implement sparse font files support in the future, at least so one can take advantage of box drawing and possibly block elements characters.

Lastly, a major pain point has been the lack of larger fonts in RAMDISK kernels, making installations and upgrades very difficult and error-prone on large DPI monitors as the text is basically unreadable. There is no technical blocker to make this happen, which ironically makes it the most difficult kind of issue to tackle.


August 30, 2020

NetBSD Blog GSoC 2020: Report-2: Fuzzing the NetBSD Network Stack in a Rumpkernel Environment
This report was written by Nisarg S. Joshi as part of Google Summer of Code 2020.

The objective of this project is to fuzz the various protocols and layers of the network stack of NetBSD using rumpkernel. This project is being carried out as a part of GSoC 2020. This blog post is regarding the project, the concepts and tools involved, the objectives and the current progress and next steps.

You can read the previous post/report here.

Overview of the work done:

The major time of the phase 1 and 2 were spent in analyzing the input and output paths of the particular protocols being fuzzed. During that time, 5 major protocols of the internet stack were taken up:

  1. IPv4 (Phase 1)
  2. UDP (Phase 1)
  3. IPv6 (Phase 2)
  4. ICMP (Phase 2)
  5. Ethernet (Phase 2)

Quite a good amount of time was spent in understanding the input and output processing functions of the particular protocols, the information gathered was to be applied in packet creation code for that protocol. This is important so that we know which parts of the packet can be kept random by the fuzzer based input and which part of the packet need to be set to proper fixed values. Fixing some values in the data packet to be correct is important so that the packet does not get rejected for trivial cases like IP Protocol Version or Internet Checksum. (The procedure to come up with the decisions and the code design and flow is explained in IPv4 Protocol section as an example)

For each protocol, mainly 2 things needed to be implemented: 

  1. The Network Config: the topology for sending and receiving packets example using a TUN device or a TAP device, the socket used and so on. Configuring these devices was the first step in being able to send or receive packets
  2. Packet Creation: Using the information gathered in the code walkthrough of the protocol functions, packet creation is decided where certain parts of the packet are kept fixed and others random from the fuzzer input itself. Doing so we try to gain maximum code coverage. Also one thing to be noted here, we should not randomly change the fuzzer input, rather do it deterministically following the same rules for each packet, otherwise the evolutionary fuzzer cannot easily grow the corpus.

In the next section, a few of the protocols will be explained in detail.

Protocols

In this section we will talk about the various protocols implemented for fuzzing and talk about the approach taken to create a packet. 

IPv4:

IPv4 stands for the Internet protocol version 4. It is one of the most widely used protocols in the internet family of protocols. It is used as a network layer protocol for routing and host to host packet delivery using an addressing scheme called IP Address(A 32 bit address scheme). IP Protocol also handles a lot of other functions like fragmentation and reassembly of packets to accommodate for transmission of packets over varying sizes of physical channel capacities. It also supports the concept of multicasting and broadcasting (Via IP Options).

In order to come up with a strategy for fuzzing, the first step was to carry out a code walkthrough of relevant functions/APIs and data structures involved in the IPv4 protocol. For that the major files and components studied were:

These sections of code represent the working of the input and output processing paths of IPv4 protocol and the struct ip is the main IPv4 header. On top of that other APIs related to mbuf (The NetBSD packet), ip_forward(), IP assembly and fragmentation etc. were also studied in order to determine information about packet structure that could be followed.

In order to be able to reach these various aspects of the protocol and be able to fuzz it, we went forward with packet creation that took care of basic fields of the IP Header so that it would not get rejected in trivial cases as mentioned before. Hence we went ahead and fixed these fields:

Other fields were allowed to be populated randomly by fuzzer input. Here is an illustration of the IPv4 header with the fields marked in red as fixed.

The packet creation code lies in the following section inside [pkt_create.c]. Another important component is the network configuration [located here net_config] where the code related to configuring a TUN/TAP device is present. All the code uses the rumpkernel exposed APIs and syscalls (prepended with rump_sys_) so as to utilize the rumpkernel while executing the application binary. After packet creation and network config is handled the main fuzzing function is written where a series of steps are followed:

  1. We call rump_init() to initialize the rumpkernel linked via libraries
  2. We setup the Client and server IP addresses
  3. We setup the TUN device by calling the network config functions described above
  4. We create the packet using the packet creation function utilizing the random buffer passed by the fuzzer and transforming that into a semi-random buffer.
  5. Pass this forged packet into the network stack of the rumpkernel linked with the application binary by calling rump_sys_write on the TUN device setup.

IPv6:

IPv6 stands for the Internet protocol version 4. It is the successor of the IPv4 protocol. It came into existence in order to overcome the addressing requirements that could not fit in a 32 bit IPv4 address. It is used as a network layer protocol for routing and host to host packet delivery using an addressing scheme called IPv6 Address(A 128 bit address scheme). It also supports almost similar other functions as IPv4 except some things like fragmentation, broadcast(instead uses multicast). 

In order to be able to reach these various aspects of the protocol and be able to fuzz it, we went forward with packet creation that took care of basic fields of the IP Header so that it would not get rejected in trivial cases as mentioned before. Hence we went ahead and fixed these fields:

Other fields were allowed to be populated randomly by fuzzer input. Allowing the payload len value to be randomly populated allowed processing of various “next headers” or ”Extension headers”. Extension headers carry optional Internet Layer information, and are placed between the fixed header and the upper-layer protocol header. The headers form a chain, using the Next Header fields. The Next Header field in the fixed header indicates the type of the first extension header; the Next Header field of the last extension header indicates the type of the upper-layer protocol header in the payload of the packet. A further work can be done to set the value of the next header chain and form packets for multiple scenarios with a combination of various next headers.

UDP:

UDP stands for User Datagram Protocol. It is one of the simplest protocols and is designed to be simple so that it simply carries payload with minimal overhead. It does not have many options except for checksum information and ports in order to demultiplex the packet to the processes. 

Since UDP runs at the transport layer and hence is wrapped up in an IP header. Since we do not want to fuzz the IP code section, we form a well formed IP header so that the packet does not get rejected in the IP processing section. We only randomize the UDP header using the fuzzer input. We used previously built out IP packet creation utilities to form the IP header and then use the fuzzer input for UDP header. 

In UDP, we fix the following fields:

ICMP:

ICMP stands for Internet control message protocol. This protocol is sometimes called a sister protocol of IP protocol and is used as a troubleshooting protocol at the network layer. It is used for major 2 purposes:

  1. Error messages
  2. Request-Reply Queries.

ICMP has a lot of options and is quite generic in the sense that it handles a lot of error messages and queries. Although ICMP is generally considered at the network layer, it is actually wrapped inside an IP header, hence it has its own protocol number(= 1). Again similar to UDP, we wrap the ICMP headers inside IP headers, hence we do not randomize the IP header and only the ICMP headers using fuzzer input.

In order to test various ICMP messages and queries, we could not fix values for the type and code fields in the ICMP header since they decide the ICMP message type. Also if we allowed random input, most of the packets would get rejected since the number of options of type and code fields are limited and most other values would discard the packet while processing. Hence we came up with a solution where we deterministically modified the input bits from the fuzzer corresponding to the code and type fields. For the type field we simply took a modulo of the number of types(ICMP_NTYPES macro used here). For the value of code , we had to fix values in a certain range based on the type value set already. This technique allowed us to cover all different ICMP message types via the fuzzer input. We also ensured that the input buffer was not modified completely randomly, since that is a bad practice for a feedback-driven fuzzer like ours. Apart from this we fixed the ICMP Checksum field as well by calculating the checksum using the internet checksum algorithm.

Ethernet:

Ethernet protocol defined by the IEEE 802.3 standard is a widely used data link layer protocol. The ethernet packet called a frame carries an IP(or the network layer protocol) datagram. The header is simple with Link Layer Addresses called MAC address (used for switching at data link layer which is a part of addressing), for source and destination each of 6 octets(=48 bytes) present, followed by a 4 octet Ethertype and QTag field. This is followed by payload and finally the FCS(frame check sequence) which is a four-octet cyclic redundancy check (CRC) that allows detection of corrupted data within the entire frame as received on the receiver side. 

In case of Ethernet protocol fuzzing, we had to use a TAP device instead of a TUN device, since the TUN device supports passing an IP packet to the network stack, whereas a TAP device accepts an ethernet frame. 

For packet creation, we set the source and destination MAC address and let the payload and ethertype be randomly populated by the fuzzer.



Current Progress and Next steps

The project currently has reached a stage where many major internet family protocols have been covered for fuzzing. As described above a structured approach to fuzzing them have been taken by forming packets based on the internal workings of the protocols. Also as mentioned in the previous post, Rumpkernel environment is being used for fuzzing all these protocols. In order to get better results as compared to raw fuzzing, we have taken these steps. In the next report we shall talk about and compare the coverage of raw fuzzing with our approach.

For the next phase of GSoC, the major focus would be to validate this process of fuzzing by various methods to check the penetration of packets into the network stack as well as the code coverage. Also the code would be made more streamlined and standardized so that it can be extended for adding more protocols even beyond the scope of the GSoC project.


August 29, 2020

DragonFly BSD Digest In Other BSDs for 2020/08/29

No theme, but plenty of variety.

 

Nikita Gillmann GSoC 2020 Final Report - Report part 2

This report was prepared by Nikita Gillmann as a part of Google Summer of Code 2020

This is my second and final report for the Google Summer of Code project I am working on for NetBSD.

My code can be found at github.com/teknokatze/src in the gsoc2020 branch, at the time of writing some of it is still missing. The test facilities and logs can be found in github.com/teknokatze/gsoc2020. A diff can be found at github which will later be split into several patches before it is sent to QA for merging.

The initial and defined goal of this project was to make system(3) and popen(3) use posix_spawn(3) internally, which had been completed in June. For the second part I was given the task to replace fork+exec calls in our standard shell (sh) in one scenario. Similar to the previous goal we determine through implementation if the initial motivation, to get performance improvements, is correct otherwise we collect metrics for why posix_spawn() in this case should be avoided. This second part meant in practice that I had to add and change code in the kernel, add a new public libc function, and understand shell internals.

Summary of part 1

Prior work: In GSoC 2012 Charles Zhang added the posix_spawn syscall which according to its SF repository at the time (maybe even now, I have not looked very much into comparing all other systems and libcs + kernels) is an in-kernel implementation of posix_spawn which provides performance benefits compared to FreeBSD and other systems which had a userspace implementation (in 2012).

After 1 week of reading POSIX and writing code, 2 weeks of coding and another 1.5 weeks of bugfixes I have successfully implemented posix_spawn in usage in system(3) and popen(3) internally.

The biggest challenge for me was to understand POSIX, to read the standard. I am used to reading more formal books, but I can't remember working with the posix standard directly before.

system(3)

system(3) was changed to use posix_spawnattr_ (where we used sigaction before) and posix_spawn (which replaced execve + vfork calls).

popen(3) and popenve(3)

Since the popen and popenve implementation in NetBSD's libc use a couple of shared helper functions, I was able to change both functions while keeping the majority of the changes focused on (some of ) the helper functions (pdes_child).

pdes_child, an internal function in popen.c, now takes one more argument (const char *cmd) for the command to pass to posix_spawn which is called in pdes_child.

On a high level what happens in pdes_child() and popen is that we first lock the pidlist_mutex. Then we create a file file action list for all concurrent popen() / popenve() instances and the side of the pipe not necessary, and the move to stdin/stdout. We unlock the pidlist_mutex. Finally we return the list and destroy.

In the new version of this helper function which now handles the majority of what popen/popenve did, we have to initialize a file_actions object which by default contains no file actions for posix_spawn() to perform. Since we have to have error handling and a common return value for the functions calling pdes_child() and deconstruction, we make use of goto in some parts of this function.

The close() and dup2() actions now get replaced by corresponding file_actions syscalls, they are used to specify a series of actions to be performed by a posix_spawn operation.

After this series of actions, we call _readlockenv(), and call posix_spawn with the file_action object and the other arguments to be executed. If it succeeds, we return the pid of the child to popen, otherwise we return -1, in both cases we destroy the file_action object before we proceed.

In popen and popenve our code has been reduced to just the 'pid == -1' branch, everything else happens in pdes_child() now.

After readlockenv we call pdes_child and pass it the command to execute in the posix_spawn'd child process; if pdes_child returns -1 we run the old error handling code. Likewise for popenve.

The outcome of the first part is, that thanks to how we implement posix_spawn in NetBSD we reduced the syscalls being made for popen and system. A full test with proper timing should indicate this, my reading was based on comparing old and new logs with ktrace and kdump.

sh, posix_spawn actions, libc and kernel - Part 2

Motivation

The main goal of part 2 of this project was to change sh(1) to determine which simple cases of (v)fork + exec I could replace, and to replace them with posix_spawn where it makes sense.

fork needs to create a new address space by cloning the address space, or in the case of vfork update at least some reference counts. posix_spawn can avoid most of this as it creates the new address space from scratch.

Issues

The current posix_spawn as defined in POSIX has no good way to do tcsetpgrp, and we found that fish just avoids posix_spawn for foreground processes.

Implementation

Since, roughly speaking, modern BSDs handle "#!" execution in the kernel (probably since before the 1990s, systems which didn't handle this started to disappear most likely in the mid to late 90s), our main concern so far was in the evalcmd function the default cmd switch case ('NORMALCMD').

After adjusting the function to use posix_spawn, I hit an issue in the execution of the curses application htop where htop would run but input would not be accepted properly (keysequences pressed are visible). In pre-posix_spawn sh, every subprocess that sh (v)forked runs forkchild() to set up the subprocess's environment. With posix_spawn, we need to arrange posix_spawn actions to do the same thing.

The intermediate resolution was to switch FORK_FG processes to fork+exec again. For foreground processes with job control we're in an interactive shell, so the performance benefit is small enough in this case to be negligible. It's really only for shell scripts that it matters.

Next I implemented a posix_spawn file_action, with the prototype

int posix_spawn_file_actions_addtcsetpgrp(posix_spawn_file_actions_t *fa, int fildes)

The kernel part of this was implemented inline in sys/kern/kern_exec.c, in the function handle_posix_spawn_file_actions() for the new case 'FAE_TCSETPGRP'.

The new version of the code is still in testing and debugging phase and at the time of writing not included in my repository (it will be published after Google Summer of Code when I'm done moving).

Future steps

posix_spawnp kernel implementation

According to a conversation with [email protected], the posix_spawnp() implementation we have is just itterating over $PATH calling posix_spawn until it succeeds. For some changes we might want a kernel implementation of posix_spawnp(), as the path search is supposed to happen in the kernel so the file actions are only ever run once:

some of the file actions may be "execute once only",
they can't be repeated (eg: handling "set -C; cat foo >file" - file
can only be created once, that has to happen before the exec (as the fd
needs to be made stdout), and then the exec part of posix_spawn is
attempted - if that fails, when it can't find "cat" in $HOME/bin (or
whatever is first in $PATH) and we move along to the next entry (maybe /bin
doesn't really matter) then the repeated file action fails, as file now
exists, and "set -C" demands that we cannot open an already existing file
(noclobber mode).   It would be nice for this if there were "clean up on
failure" actions, but that is likely to be very difficult to get right,
and each would need to be attached to a file action, so only those which
had been performed would result in cleanup attempts.

Replacing all of fork+exec in sh

Ideally we could replace all of (v)fork + exec with posix_spawn. According to my mentors there is pmap synchronisation as an impact of constructing the vm space from scratch with (v)fork. Less IPIs (inter-processor interrupts) matter for small processes too.

posix_spawn_file_action_ioctl

Future directions could involve a posix_spawn action for an arbitrary ioctl.

Thanks

My thanks go to fellow NetBSD developers for answering questions, most recently [email protected] for sharing invaluable sh knowledge, Riastradh and Jörg as the mentors I've interacted with most of the time and for their often in-depth explanations as well as allowing me to ask questions I sometimes felt were too obvious. My friends, for sticking up with my "weird" working schedule. Lastly would like to thank the Google Summer of Code program for continuing through the ongoing pandemic and giving students the chance to work on projects full-time.


August 26, 2020

Julio Merino pkgdb belongs in libdata, not var
Right after discussing where rc.d should live, it’s time to tackle a different but related pet peeve of mine: the location of the installed packages database. For this, I’m going to focus on the system I know best, pkgsrc, which keeps its database under /var/db/pkg/ by default. I think this location is wrong and the database should move to /usr/pkg/libdata/pkgdb/. From a cursory look, it seems that FreeBSD’s and OpenBSD’s ports databases, as well as dpkg’s and rpm’s, are also affected by this “problem”—but I do not know enough about their internals to say with certainty.

August 24, 2020

Julio Merino rc.d belongs in libexec, not etc
Let’s open with the controversy: the scripts that live under /etc/rc.d/ in FreeBSD, NetBSD, and OpenBSD are in the wrong place. They all should live in /libexec/rc.d/ because they are code, not configuration. This misplacement is something that has bugged me for ages but I never had the energy to open this can of worms back when I was very involved in NetBSD. I suspect it would have been a draining discussion and a very difficult thing to change.

August 22, 2020

DragonFly BSD Digest In Other BSDs for 2020/08/22

Started this with overflow from last week.


August 20, 2020

Ruben Schade You don’t need tmux or screen for ZFS

Back in January I mentioned how to add redundancy to a ZFS pool by adding a mirrored drive. Someone with a private account on Twitter asked me why FreeBSD—and NetBSD!—doesn’t ship with a tmux or screen equivilent in base in order to daemonise the process and let them run in the background.

ZFS already does this for its internal commands. For example, I used zfs add to add a drive to fix a older RAIDZ2 pool, so I can just run:

# zpool status

Under status:

status: One or more devices is currently being resilvered.The pool will continue to function, possibly in a degraded state.

The pool will also continue to be available, though potentially with reduced IO performance while it (re-)replicates.

The only time I use tmux is during a large ZFS send/receive operation between machines over SSH. At that stage we’ve introduced networks into the mix, which even the most robust, trustworthy storage system in the world can’t guarantee will stay up!


This post originally appeared on Rubenerd.


August 14, 2020

NetBSD Package System (pkgsrc) on DaemonForums Samba : sh: /usr/bin/lex: not found
Hi everybody

I'm trying to install samba, like that :
Code:

[email protected] /usr/pkgsrc/net/samba # make install clean
But I've got this message :
Code:

sh: /usr/bin/lex: not found
ERROR: This package has set PKG_FAIL_REASON:
ERROR: samba-3.6.25nb23 requires a working dlopen().
*** Error code 1

Does anyone can help me ?
I found nothing.

Thank You
Guillaume ( NetBsd 9 amd64 Zsh )

August 09, 2020

Emile Heitor Make Postfix Trigger Blacklistd on Failed Authentication
The other day, I realized that from time to time, alpine, my console mail client for about 20+ years now, would close the connection to the IMAP server because of an “error”. Digging in the logs, I realized my server was being bruteforced for months, if not years. NetBSD being the fantastic OS it is, it actually had nearly no effect on my server’s behaviour, only those annoying connections closing from time to time.

August 07, 2020

Ruben Schade Trademarks disputes in tech

Trademark disputes and confusion have exited since they have; a tautological statement for which I couldn’t resist. But in the spirit of realising when I’ve been previously wrong about something, I’m here to explore the idea that there may be something to trademark antsiness in tech after all.

The term podcast was my first experience with this. Everyone I knew and respected in the tech and online media communities couldn’t stand the phrase when it came out in the mid 2000s. Frank Nora already had his New Time Radio moniker to describe downloadable online broadcasts. Jim Kloss famously used Internet Magazine at Whole Wheat Radio because he didn’t think the future only belonged to Apple. A former TechTV titan preferred netcast because it attributed the Internet as the enabling tech, like satellite radio.

(Incidently, this is the etymology behind the NetBSD name as well. I always thought it looked elegant compared to the other BSDs).

I used to think New Time Radio and Audio Magazine were more fun, clever, and relatable to a wider audience than podcast, but I’ll admit, deep down, I thought the association with Apple’s then-ubiquitous iPod was overblown. Despite what Apple’s legal team may have thought at the time, they didn’t invent the word pod. And besides, everyone knew they could download and listen to Ze Frank or Digital Flotsam wherever they wanted.

But just as I did with a new bottle of Sriracha Tabasco in a pasta dish, I grossly under-estimated the term’s impact. Years before podcasting became as cool and ubiquitous as it is now, I still fielded questions for my own silly show asking why they needed an Apple device to listen to it. I ended up linking to the BBC’s podcasting FAQ page where they took great pains to say it worked on your desktop, iRiver, flip phone, or anything else that supported MP3s.

Meanwhile GitHub was taking off as the Internet’s preferred code repository and collaboration platform. I still remember a lecturer at uni assuring us we didn’t need to use GitHub to use Git for our major assignment. I thought he was joking, but even afterwards we had group members who were confused. “GitHub is Git, right?” they all asked, before we decided on Subversion.

Another example is rsync.net. I recently noticed it among a vendor list on a client proposal. I thought they’d just listed the website for the indispensable file sync tool, but it referred to a commercial backup service. I realised my mistake, but how many other stakeholders were caught out too? Large companies can defend their trademarks, but I worry when I see the name of a free or open source tool being used in one. They may have the blessing of the tool’s creator—in some cases they may not have a choice—but it can introduce confusion, and the potential to assume that one endorses the other.

CentOS being Red Hat Enterprise Linux in all but name is one of the worst-kept secrets in the industry, so it’s clear it’s possible to build your own reputation and pedigree without piggy-backing off another title. I suppose that’s harder, though.


This post originally appeared on Rubenerd.


August 06, 2020

Frederic Cambus NetBSD on the NanoPi NEO2

The NanoPi NEO2 from FriendlyARM has been serving me well since 2018, being my test machine for OpenBSD/arm64 related things.

As NetBSD/evbarm finally gained support for AArch64 in NetBSD 9.0, released back in February, I decided to give it a try on this device. The board only has 512MB of RAM, and this is where NetBSD really shines. Things have become a lot easier since [email protected] now provides bootable ARM images for a variety of devices, including the NanoPi NEO2.

On first boot, the system will resize the filesystem to automatically expand to the size of the SD card.

Growing ld0 MBR partition #1 (1052MB -> 60810MB)
Growing ld0 disklabel (1148MB -> 60906MB)
Resizing /
/dev/rld0a: grow cg |************************************                 |  69%

Once the system is up and running, we can add a regular user in the wheel group:

useradd -m -G wheel username

And add a password to the newly created user:

passwd username

From there we do not need the serial console anymore and can connect to the device using SSH.

NetBSD has binary packages available for this architecture, and installing and configuring pkgin can be done as follow:

export PKG_PATH=https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/aarch64/9.0/All/
pkg_add pkgin
echo $PKG_PATH > /usr/pkg/etc/pkgin/repositories.conf
pkgin update

The base system can be kept up to date using sysupgrade, which can be installed via pkgin:

pkgin in sysupgrade

The following variable need to be set in /usr/pkg/etc/sysupgrade.conf:

RELEASEDIR="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/evbarm-aarch64"

Lastly, the device has two user controllable LEDs which can be toggled on and off using sysctl.

To switch both LEDs on:

sysctl -w hw.led.nanopi_green_pwr=1
sysctl -w hw.led.nanopi_blue_status=1

To switch off the power LED automatically at boot time:

echo "hw.led.nanopi_green_pwr=0" >> /etc/sysctl.conf

Here is a dmesg for reference purposes:

[     1.000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
[     1.000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
[     1.000000]     2018, 2019, 2020 The NetBSD Foundation, Inc.  All rights reserved.
[     1.000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
[     1.000000]     The Regents of the University of California.  All rights reserved.

[     1.000000] NetBSD 9.0_STABLE (GENERIC64) #0: Wed Aug  5 15:20:21 UTC 2020
[     1.000000] 	[email protected]:/usr/src/sys/arch/evbarm/compile/GENERIC64
[     1.000000] total memory = 497 MB
[     1.000000] avail memory = 479 MB
[     1.000000] timecounter: Timecounters tick every 10.000 msec
[     1.000000] armfdt0 (root)
[     1.000000] simplebus0 at armfdt0: FriendlyARM NanoPi NEO 2
[     1.000000] simplebus1 at simplebus0
[     1.000000] simplebus2 at simplebus0
[     1.000000] cpus0 at simplebus0
[     1.000000] simplebus3 at simplebus0
[     1.000000] psci0 at simplebus0: PSCI 1.1
[     1.000000] cpu0 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
[     1.000000] cpu0: package 0, core 0, smt 0
[     1.000000] cpu0: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
[     1.000000] cpu0: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
[     1.000000] cpu0: Dcache line 64, Icache line 64
[     1.000000] cpu0: L1 32KB/64B 2-way read-allocate VIPT Instruction cache
[     1.000000] cpu0: L1 32KB/64B 4-way write-back read-allocate write-allocate PIPT Data cache
[     1.000000] cpu0: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT Unified cache
[     1.000000] cpu0: revID=0x180, PMCv3, 4k table, 64k table, 16bit ASID
[     1.000000] cpu0: auxID=0x11120, FP, CRC32, SHA1, SHA256, AES+PMULL, NEON, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add
[     1.000000] cpu1 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
[     1.000000] cpu1: package 0, core 1, smt 0
[     1.000000] cpu2 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
[     1.000000] cpu2: package 0, core 2, smt 0
[     1.000000] cpu3 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
[     1.000000] cpu3: package 0, core 3, smt 0
[     1.000000] gic0 at simplebus1: GIC
[     1.000000] armgic0 at gic0: Generic Interrupt Controller, 224 sources (215 valid)
[     1.000000] armgic0: 16 Priorities, 192 SPIs, 7 PPIs, 16 SGIs
[     1.000000] fclock0 at simplebus2: 24000000 Hz fixed clock (osc24M)
[     1.000000] sunxisramc0 at simplebus1: SRAM Controller
[     1.000000] fclock1 at simplebus2: 32768 Hz fixed clock (ext_osc32k)
[     1.000000] gtmr0 at simplebus0: Generic Timer
[     1.000000] gtmr0: interrupting on GIC irq 27
[     1.000000] armgtmr0 at gtmr0: Generic Timer (24000 kHz, virtual)
[     1.000000] timecounter: Timecounter "armgtmr0" frequency 24000000 Hz quality 500
[     1.000010] sun8ih3ccu0 at simplebus1: H3 CCU
[     1.000010] sun8ih3rccu0 at simplebus1: H3 PRCM CCU
[     1.000010] sunxide2ccu0 at simplebus1: DE2 CCU
[     1.000010] sunxigpio0 at simplebus1: PIO
[     1.000010] gpio0 at sunxigpio0: 94 pins
[     1.000010] sunxigpio0: interrupting on GIC irq 43
[     1.000010] sunxigpio1 at simplebus1: PIO
[     1.000010] gpio1 at sunxigpio1: 12 pins
[     1.000010] sunxigpio1: interrupting on GIC irq 77
[     1.000010] fregulator0 at simplebus0: vcc3v3
[     1.000010] fregulator1 at simplebus0: usb0-vbus
[     1.000010] fregulator2 at simplebus0: gmac-3v3
[     1.000010] sun6idma0 at simplebus1: DMA controller (12 channels)
[     1.000010] sun6idma0: interrupting on GIC irq 82
[     1.000010] com0 at simplebus1: ns16550a, working fifo
[     1.000010] com0: console
[     1.000010] com0: interrupting on GIC irq 32
[     1.000010] sunxiusbphy0 at simplebus1: USB PHY
[     1.000010] sunxihdmiphy0 at simplebus1: HDMI PHY
[     1.000010] sunximixer0 at simplebus1: Display Engine Mixer
[     1.000010] sunxilcdc0 at simplebus1: TCON1
[     1.000010] sunxilcdc0: interrupting on GIC irq 118
[     1.000010] sunxirtc0 at simplebus1: RTC
[     1.000010] emac0 at simplebus1: EMAC
[     1.000010] emac0: Ethernet address 02:01:f7:f9:2f:67
[     1.000010] emac0: interrupting on GIC irq 114
[     1.000010] rgephy0 at emac0 phy 7: RTL8211E 1000BASE-T media interface
[     1.000010] rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
[     1.000010] h3codec0 at simplebus1: H3 Audio Codec (analog part)
[     1.000010] sunximmc0 at simplebus1: SD/MMC controller
[     1.000010] sunximmc0: interrupting on GIC irq 92
[     1.000010] motg0 at simplebus1: 'otg' mode not supported
[     1.000010] ehci0 at simplebus1: EHCI
[     1.000010] ehci0: interrupting on GIC irq 104
[     1.000010] ehci0: EHCI version 1.0
[     1.000010] ehci0: 1 companion controller, 1 port
[     1.000010] usb0 at ehci0: USB revision 2.0
[     1.000010] ohci0 at simplebus1: OHCI
[     1.000010] ohci0: interrupting on GIC irq 105
[     1.000010] ohci0: OHCI version 1.0
[     1.000010] usb1 at ohci0: USB revision 1.0
[     1.000010] ehci1 at simplebus1: EHCI
[     1.000010] ehci1: interrupting on GIC irq 110
[     1.000010] ehci1: EHCI version 1.0
[     1.000010] ehci1: 1 companion controller, 1 port
[     1.000010] usb2 at ehci1: USB revision 2.0
[     1.000010] ohci1 at simplebus1: OHCI
[     1.000010] ohci1: interrupting on GIC irq 111
[     1.000010] ohci1: OHCI version 1.0
[     1.000010] usb3 at ohci1: USB revision 1.0
[     1.000010] sunxiwdt0 at simplebus1: Watchdog
[     1.000010] sunxiwdt0: default watchdog period is 16 seconds
[     1.000010] /soc/[email protected] at simplebus1 not configured
[     1.000010] gpioleds0 at simplebus0: nanopi:green:pwr nanopi:blue:status
[     1.000010] /soc/[email protected] at simplebus1 not configured
[     1.000010] /soc/[email protected] at simplebus1 not configured
[     1.000010] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
[     1.000010] cpu2: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
[     1.000010] cpu2: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
[     1.040229] cpu2: Dcache line 64, Icache line 64
[     1.040229] cpu2: L1 32KB/64B 2-way read-allocate VIPT Instruction cache
[     1.050220] cpu2: L1 32KB/64B 4-way write-back read-allocate write-allocate PIPT Data cache
[     1.060220] cpu2: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT Unified cache
[     1.070220] cpu2: revID=0x180, PMCv3, 4k table, 64k table, 16bit ASID
[     1.070220] cpu2: auxID=0x11120, FP, CRC32, SHA1, SHA256, AES+PMULL, NEON, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add
[     1.090221] cpu1: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
[     1.090221] cpu1: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
[     1.100222] cpu1: Dcache line 64, Icache line 64
[     1.110221] cpu1: L1 32KB/64B 2-way read-allocate VIPT Instruction cache
[     1.110221] cpu1: L1 32KB/64B 4-way write-back read-allocate write-allocate PIPT Data cache
[     1.120222] cpu1: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT Unified cache
[     1.130222] cpu1: revID=0x180, PMCv3, 4k table, 64k table, 16bit ASID
[     1.140223] cpu1: auxID=0x11120, FP, CRC32, SHA1, SHA256, AES+PMULL, NEON, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add
[     1.150222] cpu3: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
[     1.160223] cpu3: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
[     1.160223] cpu3: Dcache line 64, Icache line 64
[     1.170223] cpu3: L1 32KB/64B 2-way read-allocate VIPT Instruction cache
[     1.180223] cpu3: L1 32KB/64B 4-way write-back read-allocate write-allocate PIPT Data cache
[     1.180223] cpu3: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT Unified cache
[     1.190223] cpu3: revID=0x180, PMCv3, 4k table, 64k table, 16bit ASID
[     1.200224] cpu3: auxID=0x11120, FP, CRC32, SHA1, SHA256, AES+PMULL, NEON, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add
[     1.210224] sdmmc0 at sunximmc0
[     1.240225] uhub0 at usb0: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
[     1.240225] uhub0: 1 port with 1 removable, self powered
[     1.240225] uhub1 at usb2: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
[     1.250226] uhub1: 1 port with 1 removable, self powered
[     1.250226] uhub2 at usb1: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
[     1.260226] uhub2: 1 port with 1 removable, self powered
[     1.260226] uhub3 at usb3: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
[     1.275641] uhub3: 1 port with 1 removable, self powered
[     1.275641] IPsec: Initialized Security Association Processing.
[     1.350228] sdmmc0: SD card status: 4-bit, C10, U1, A1
[     1.350228] ld0 at sdmmc0: <0x03:0x5344:SC64G:0x80:0x0cd9141d:0x122>
[     1.360690] ld0: 60906 MB, 7764 cyl, 255 head, 63 sec, 512 bytes/sect x 124735488 sectors
[     1.370228] ld0: 4-bit width, High-Speed/SDR25, 50.000 MHz
[     1.990242] boot device: ld0
[     1.990242] root on ld0a dumps on ld0b
[     2.000243] root file system type: ffs
[     2.010242] kern.module.path=/stand/evbarm/9.0/modules

July 11, 2020

Nikita Gillmann GSoC 2020 - Notes on sh(1)

Misc notes on the 3 relevant files for adding posix_spawn usage to sh, from https://github.com/teknokatze/gsoc2020/blob/default/sh.txt and summary from today's IRC meeting with my mentors. This week and last week I've read through sh's code, with no clue how to start replacing the logic.

We have 3 files (jobs, exec, eval) which seem to contain the main logic i need to investigate. Between fork and exec we sometimes fork before we exec and have too much disconnection between those two, or sometimes we exec in a vforked shell, and I need to understand more about what is going on or just start testing.

According to Riastradh the shell is going to seem a bit more complicated because it often forks subshells without directly reducing to exec. The low-hanging fruit is where it vforks, because you can't do much besides exec after vfork -- so that's the path that is already an opportunity for posix_spawn, because pretty much the only things it will do are set up signals and file descriptors and exec. In evalcommand, the relevant path is limited to the case where cmdentry.cmdtype == CMDNORMAL and (!cmd->ncmd.backgnd or cmd->ncmd.redirect == NULL), in which case it does listmklocal, forkchild, redirect, and shellexec. It's really down to what's between VFORK_BLOCK, and the case CMDNORMAL of switch (cmdentry.cmdtype) down below (which is the 'default' of the switch). This is mostly just redirect and shellexec. The easiest approach would probably be to rip out everything in VFORK_BLOCK/END and replace it by posix_spawn, don't bother falling through to the switch below; just incorporate the redirect and shellexec into posix_spawn there.


June 30, 2020

Nikita Gillmann GSoC 2020 Report 1

This is my first report for the Google Summer of Code project I am working on for NetBSD.

Update 2020-07-14: The more polished version has been published and reviewed at our TNF blog.

Update 2020-07-10: Prior work. In GSoC 2012 Charles Zhang added the posix_spawn syscall which according to its SF repository at the time (maybe even now, I have not looked very much into comparing all other systems and libcs + kernels) is an in-kernel implementation of posix_spawn which provides performance benefits compared to FreeBSD and other systems which had a userspace implementation.

The code snippets in this article are under the same license as their respective files, ie 3-clause BSD licensed, Copyright (c) 1988, 1993 The Regents of the University of California. For full details refer to netbsd.org/about/redistribution and the full source code of the functions described. This is not legal advice and I'm only the author of (some portions of) the new version of the functions.

After 1 week of reading POSIX and writing code, 2 weeks of coding and another 1.5 weeks of bugfixes I have succesfully implemented posix_spawn in usage in system(3) and popen(3) internally.

The biggest challenge for me was to understand POSIX, to read the standard. I am used to reading more formal books, but I can't remember working with the posix standard directly before.

The next part of my Google Summer of Code project will focus on similar rewrites of NetBSD's sh(1).

system(3)

The prototype

int system(const char *command);

remains the same. Below I'm just commenting on the differences, not the whole function, and therefore only include code block where the versions differ the most. The full work can be found at gsoc2020 as well as src and will be submitted for inclusion later in the project.

Previously we'd use vfork, sigaction, and execve in this stdlib function.

The biggest difference to the 2015 version of our system version is the usage of posix_spawnattr_ where we'd use sigaction before, and posix_spawn where execve executes the command in a vfork'd child:

   posix_spawnattr_init(&attr);
   posix_spawnattr_setsigmask(&attr, &omask);
   posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETSIGDEF|POSIX_SPAWN_SETSIGMASK);
   (void)__readlockenv();
   status = posix_spawn(&pid, _PATH_BSHELL, NULL, &attr, __UNCONST(argp), environ);
   (void)__unlockenv();
   posix_spawnattr_destroy(&attr);

The full version can be found here.

The prototype of posix_spawn is:

int posix_spawn(pid_t *restrict pid, const char *restrict path, const posix_spawn_file_actions_t *file_actions, const posix_spawnattr_t *restrict attrp, char *const argv[restrict], char *const envp[restrict]);

We first initialize a spawn attributes object with the default value for all of its attributes set. A spawn attributes object is used to modify the behavior of posix_spawn().

The previous fork-exec switch included calls to sigaction to set the behavior associated with SIGINT and SIGQUIT as defined by POSIX:

The system() function shall ignore the SIGINT and SIGQUIT signals, and shall block the SIGCHLD signal, while waiting for the command to terminate. If this might cause the application to miss a signal that would have killed it, then the application should examine the return value from system() and take whatever action is appropriate to the application if the command terminated due to receipt of a signal. source: https://pubs.opengroup.org/onlinepubs/9699919799/functions/system.html

This has been achieved with a combination of posix_spawnattr_setsigmask() and posix_spawnattr_setflags() in the initialized attributes object referenced by attr.

As before we call __readlockenv() and then call posix_spawn() which returns the process ID of the child process in the variable pointed to by 'pid', and returns zero as the function return value.

The old code:

   (void)__readlockenv();
   switch(pid = vfork()) {
   case -1:                        /* error */
           (void)__unlockenv();
           sigaction(SIGINT, &intsa, NULL);
           sigaction(SIGQUIT, &quitsa, NULL);
           (void)sigprocmask(SIG_SETMASK, &omask, NULL);
           return -1;
   case 0:                         /* child */
           sigaction(SIGINT, &intsa, NULL);
           sigaction(SIGQUIT, &quitsa, NULL);
           (void)sigprocmask(SIG_SETMASK, &omask, NULL);
           execve(_PATH_BSHELL, __UNCONST(argp), environ);
           _exit(127);
   }
   (void)__unlockenv();

popen(3), popenve(3)

As with system, the prototype of both functions remains the same:

FILE * popenve(const char *cmd, char *const *argv, char *const *envp, const char *type); FILE * popen(const char *cmd, const char *type);

Since the popen and popenve implementation in NetBSD's libc use a couple of shared helper functions, I was able to change both functions while keeping the majority of the changes focused on (some of ) the helper functions (pdes_child).

pdes_child, an internal function in popen.c, now takes one more argument (const char *cmd) for the command to pass to posix_spawn which is called in pdes_child.

pdes_child previously looked like this:

static void
pdes_child(int *pdes, const char *type)
{
        struct pid *old;

        /* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
           from previous popen() calls that remain open in the 
           parent process are closed in the new child process. */
        for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
                (void)close(old->fd); /* don't allow a flush */
#else
                (void)close(fileno(old->fp)); /* don't allow a flush */
#endif

        if (type[0] == 'r') {
                (void)close(pdes[0]);
                if (pdes[1] != STDOUT_FILENO) {
                        (void)dup2(pdes[1], STDOUT_FILENO);
                        (void)close(pdes[1]);
                }
                if (type[1] == '+')
                        (void)dup2(STDOUT_FILENO, STDIN_FILENO);
        } else {
                (void)close(pdes[1]);
                if (pdes[0] != STDIN_FILENO) {
                        (void)dup2(pdes[0], STDIN_FILENO);
                        (void)close(pdes[0]);
                }
        }
}

My current blog layout and formating will also mess up the readability of this function, this is the new version (the whole file is here):

static int
pdes_child(int *pdes, const char *type, const char *cmd)
{
        struct pid *old;
        posix_spawn_file_actions_t file_action_obj;
        pid_t pid;
        const char *argp[] = {"sh", "-c", NULL, NULL};
        argp[2] = cmd;
        int error;

        error = posix_spawn_file_actions_init(&file_action_obj);
        if (error) {
                goto fail;
        }
        /* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
           from previous popen() calls that remain open in the
           parent process are closed in the new child process. */
        for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
        error = posix_spawn_file_actions_addclose(&file_action_obj, old->fd); /* don't allow a flush */
        if (error) {
                goto fail;
        }
#else
        error = posix_spawn_file_actions_addclose(&file_action_obj, fileno(old->fp)); /* don't allow a flush */
        if (error) {
                goto fail;
        }
#endif
        if (type[0] == 'r') {
                error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
                if (error) {
                        goto fail;
                }
                if (pdes[1] != STDOUT_FILENO) {
                        error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[1], STDOUT_FILENO);
                        if (error) {
                                goto fail;
                        }
                        error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
                        if (error) {
                                goto fail;
                        }
                }
                if (type[1] == '+') {
                        error = posix_spawn_file_actions_adddup2(&file_action_obj, STDOUT_FILENO, STDIN_FILENO);
                        if (error) {
                                goto fail;
                        }
                }
        } else {
                error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
                if (error) {
                        goto fail;
                }
                if (pdes[0] != STDIN_FILENO) {
                        error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[0], STDIN_FILENO);
                        if (error) {
                                goto fail;
                        }
                        error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
                        if (error) {
                                goto fail;
                        }
                }
        }
        (void)__readlockenv();
        error = posix_spawn(&pid, _PATH_BSHELL, &file_action_obj, 0, __UNCONST(argp), environ);
        if (error) {
                (void)__unlockenv();
                goto fail;
        }
        (void)__unlockenv();
        error = posix_spawn_file_actions_destroy(&file_action_obj);
        /*
         * TODO: if _destroy() fails we have to go on, otherwise we
         * leak the pid.
         */
        if (error) {
                errno = error;
                return -1;
        }
        return pid;

fail:
        errno = error;
        posix_spawn_file_actions_destroy(&file_action_obj);
        return -1;
}

On a high level what happens in pdes_child() and popen is that we first lock the pidlist_mutex. Then we create a file file action list for all concurrent popen() / popenve() instances and the side of the pipe not necessary, and the move to stdin/stdout. We unlock the pidlist_mutex. Finally we return the list and destroy.

In the new version of this helper function which now handles the majority of what popen/popenve did, we have to initialize a file_actions object which by default contains no file actions for posix_spawn() to perform. Since we have to have error handling and a common return value for the functions calling pdes_child() and deconstruction, we make use of goto in some parts of this function.

The close() and dup2() actions now get replaced by corresponding file_actions syscalls, they are used to specify a series of actions to be performed by a posix_spawn operation.

After this series of actions, we call _readlockenv(), and call posix_spawn with the file_action object and the other arguments to be executed. If it succeeds, we return the pid of the child to popen, otherwise we return -1, in both cases we destroy the file_action object before we proceed.

In popen and popenve our code has been reduced to just the 'pid == -1' branch, everything else happens in pdes_child() now.

After readlockenv we call pdes_child and pass it the command to execute in the posix_spawn'd child process; if pdes_child returns -1 we run the old error handling code. Likewise for popenve.

popen, old:

FILE *
popen(const char *cmd, const char *type)
{
        struct pid *cur;
        int pdes[2], serrno;
        pid_t pid;

        _DIAGASSERT(cmd != NULL);
        _DIAGASSERT(type != NULL);

        if ((cur = pdes_get(pdes, &type)) == NULL)
                return NULL;

        MUTEX_LOCK();
        (void)__readlockenv();
        switch (pid = vfork()) {
        case -1:                        /* Error. */
                serrno = errno;
                (void)__unlockenv();
                MUTEX_UNLOCK();
                pdes_error(pdes, cur);
                errno = serrno;
                return NULL;
                /* NOTREACHED */
        case 0:                         /* Child. */
                pdes_child(pdes, type);
                execl(_PATH_BSHELL, "sh", "-c", cmd, NULL);
                _exit(127);
                /* NOTREACHED */
        }
        (void)__unlockenv();

        pdes_parent(pdes, cur, pid, type);

        MUTEX_UNLOCK();

        return cur->fp;
}

popen, new:

FILE *
popen(const char *cmd, const char *type)
{
        struct pid *cur;
        int pdes[2], serrno;
        pid_t pid;

        _DIAGASSERT(cmd != NULL);
        _DIAGASSERT(type != NULL);

        if ((cur = pdes_get(pdes, &type)) == NULL)
                return NULL;

        MUTEX_LOCK();
        (void)__readlockenv();
        pid = pdes_child(pdes, type, cmd);
        if (pid == -1) {
                /* Error. */
                serrno = errno;
                (void)__unlockenv();
                MUTEX_UNLOCK();
                pdes_error(pdes, cur);
                errno = serrno;
                return NULL;
                /* NOTREACHED */
        }
        (void)__unlockenv();

        pdes_parent(pdes, cur, pid, type);

        MUTEX_UNLOCK();

        return cur->fp;
}

popenve, old:

FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
        struct pid *cur;
        int pdes[2], serrno;
        pid_t pid;

        _DIAGASSERT(cmd != NULL);
        _DIAGASSERT(type != NULL);

        if ((cur = pdes_get(pdes, &type)) == NULL)
                return NULL;

        MUTEX_LOCK();
        switch (pid = vfork()) {
        case -1:                        /* Error. */
                serrno = errno;
                MUTEX_UNLOCK();
                pdes_error(pdes, cur);
                errno = serrno;
                return NULL;
                /* NOTREACHED */
        case 0:                         /* Child. */
                pdes_child(pdes, type);
                execve(cmd, argv, envp);
                _exit(127);
                /* NOTREACHED */
        }

        pdes_parent(pdes, cur, pid, type);

        MUTEX_UNLOCK();

        return cur->fp;
}

popenve, new:

FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
        struct pid *cur;
        int pdes[2], serrno;
        pid_t pid;

        _DIAGASSERT(cmd != NULL);
        _DIAGASSERT(type != NULL);

        if ((cur = pdes_get(pdes, &type)) == NULL)
                return NULL;

        MUTEX_LOCK();
        pid = pdes_child(pdes, type, cmd);
        if (pid == -1) {
                /* Error. */
                serrno = errno;
                MUTEX_UNLOCK();
                pdes_error(pdes, cur);
                errno = serrno;
                return NULL;
                /* NOTREACHED */
        }

        pdes_parent(pdes, cur, pid, type);

        MUTEX_UNLOCK();

        return cur->fp;
}
NetBSD.org pkgsrc-2020Q2 released

June 20, 2020

Benny Siegert Getting Started with NetBSD on the Pinebook Pro
If you buy a Pinebook Pro now, it comes with Manjaro Linux on the internal eMMC storage. Let’s install NetBSD instead! The easiest way to get started is to buy a decent micro-SD card (what sort of markings it should have is a science of its own, by the way) and install NetBSD on that. On a warm boot (i.e. when rebooting a running system), the micro-SD card has priority compared to the eMMC, so the system will boot from there.

June 18, 2020

Emile Heitor Fakecracker: NetBSD as a Function Based MicroVM
In November 2018 AWS published an Open Source tool called Firecracker, mostly a virtual machine monitor relying on KVM, a small sized Linux kernel, and a stripped down version of Qemu. What baffled me was the speed at which the virtual machine would fire up and run the service. The whole process is to be compared to a container, but safer, as it does not share the kernel nor any resource, it is a separate and dedicated virtual machine.

June 16, 2020

Stack Overflow screen fails on NetBSD, reporting "poll: Invalid argument"

I have installed and used screen many times on several different operating systems. Recently I installed it on a NetBSD-8.0 virtual machine.

$ sudo pkgin install screen
calculating dependencies...done.

1 package to install:
  screen-4.8.0nb1

0 to refresh, 0 to upgrade, 1 to install
0B to download, 1098K to install

proceed ? [Y/n] Y
installing screen-4.8.0nb1...
screen-4.8.0nb1: setting permissions on /usr/pkg/bin/screen-4.8.0 (o=root, g=wheel, m=4511)
screen-4.8.0nb1: adding /usr/pkg/bin/screen to /etc/shells
screen-4.8.0nb1: registering info file /usr/pkg/info/screen.info
===========================================================================
$NetBSD: MESSAGE,v 1.5 2005/12/28 17:53:24 reed Exp $
[snip]
===========================================================================
pkg_install warnings: 0, errors: 0
reading local summary...
processing local summary...
marking screen-4.8.0nb1 as non auto-removable

However, when I went to use it, I got an immediate failure.

$ uname -mrs
NetBSD 8.0 amd64
$ ls -l /usr/pkg/bin/screen
lrwxr-xr-x  1 root  wheel  12 Apr  6 02:50 /usr/pkg/bin/screen -> screen-4.8.0
$ groups
users wheel
$ screen
poll: Invalid argument

This problem persists even when I first remove, then reinstall the screen package. Any suggestions as to what's wrong?


June 06, 2020

Frederic Cambus OpenBSD framebuffer console and custom color palettes

On framebuffer consoles, OpenBSD uses the rasops(9) subsystem, which was imported from NetBSD in March 2001.

The RGB values for the ANSI color palette in rasops have been chosen to match the ones in Open Firmware, and are different from those in the VGA text mode color palette.

Rasops palette:

Rasops palette

VGA text mode palette:

VGA text mode palette

As one can see, the difference is quite significant, and decades of exposure to MS-DOS and Linux consoles makes it quite difficult to adapt to a different palette.

RGB values for the ANSI color palette are defined in sys/dev/rasops/rasops.c, and here are the proper ones to use to match the VGA text mode palette:

#define	NORMAL_BLACK	0x000000
#define	NORMAL_RED	0xaa0000
#define	NORMAL_GREEN	0x00aa00
#define	NORMAL_BROWN	0xaa5500
#define	NORMAL_BLUE	0x0000aa
#define	NORMAL_MAGENTA	0xaa00aa
#define	NORMAL_CYAN	0x00aaaa
#define	NORMAL_WHITE	0xaaaaaa

#define	HILITE_BLACK	0x555555
#define	HILITE_RED	0xff5555
#define	HILITE_GREEN	0x55ff55
#define	HILITE_BROWN	0xffff55
#define	HILITE_BLUE	0x5555ff
#define	HILITE_MAGENTA	0xff55ff
#define	HILITE_CYAN	0x55ffff
#define	HILITE_WHITE	0xffffff

And here is a diff doing just that, which I sent to [email protected] back in January 2017.

EDIT: The enthusiasm around this article led me to make another try, which didn't fare any better.


May 31, 2020

Benny Siegert Pinebook Pro, First Impressions
Note: This post was written on the Pinebook Pro :) After seeing it in action at FOSDEM (from afar, as the crowd was too large), I decided to buy a Pinebook Pro for personal use. From the beginning, the intention was to use it for pkgsrc development, with NetBSD as the main OS. It was finally delivered on Thursday, one day earlier than promised, so I thought I would write down my first impressions.

May 27, 2020

Frederic Cambus OpenBSD/armv7 on the CubieBoard2

I bought the CubieBoard2 back in 2016 with the idea to run OpenBSD on it, but because of various reliability issues with the onboard NIC, it ended up running NetBSD for a few weeks before ending up in a drawer.

Back in October, Mark Kettenis committed code to allow switching to the framebuffer "glass" console in the bootloader on OpenBSD/armv7, making it possible to install the system without using a serial cable.

>> OpenBSD/armv7 BOOTARM 1.14
boot> set tty fb0
switching console to fb0

This prompted me to plug the board again, and having support for the framebuffer console is a game changer. It also allows running Xenocara, if that's your thing.

Here is the output of running file on executables:

ELF 32-bit LSB shared object, ARM, version 1

And this is the result of the md5 -t benchmark:

MD5 time trial.  Processing 10000 10000-byte blocks...
Digest = 52e5f9c9e6f656f3e1800dfa5579d089
Time   = 1.340000 seconds
Speed  = 74626865.671642 bytes/second

For the record, LibreSSL speed benchmark results are available here.

System message buffer (dmesg output):

OpenBSD 6.7-current (GENERIC) #299: Sun May 24 18:25:45 MDT 2020
    [email protected]:/usr/src/sys/arch/armv7/compile/GENERIC
real mem  = 964190208 (919MB)
avail mem = 935088128 (891MB)
random: good seed from bootblocks
mainbus0 at root: Cubietech Cubieboard2
cpu0 at mainbus0 mpidr 0: ARM Cortex-A7 r0p4
cpu0: 32KB 32b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu0: 256KB 64b/line 8-way L2 cache
cortex0 at mainbus0
psci0 at mainbus0: PSCI 0.0
sxiccmu0 at mainbus0
agtimer0 at mainbus0: tick rate 24000 KHz
simplebus0 at mainbus0: "soc"
sxiccmu1 at simplebus0
sxipio0 at simplebus0: 175 pins
sxirtc0 at simplebus0
sxisid0 at simplebus0
ampintc0 at simplebus0 nirq 160, ncpu 2: "interrupt-controller"
"system-control" at simplebus0 not configured
"interrupt-controller" at simplebus0 not configured
"dma-controller" at simplebus0 not configured
"lcd-controller" at simplebus0 not configured
"lcd-controller" at simplebus0 not configured
"video-codec" at simplebus0 not configured
sximmc0 at simplebus0
sdmmc0 at sximmc0: 4-bit, sd high-speed, mmc high-speed, dma
"usb" at simplebus0 not configured
"phy" at simplebus0 not configured
ehci0 at simplebus0
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Generic EHCI root hub" rev 2.00/1.00 addr 1
ohci0 at simplebus0: version 1.0
"crypto-engine" at simplebus0 not configured
"hdmi" at simplebus0 not configured
sxiahci0 at simplebus0: AHCI 1.1
scsibus0 at sxiahci0: 32 targets
ehci1 at simplebus0
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Generic EHCI root hub" rev 2.00/1.00 addr 1
ohci1 at simplebus0: version 1.0
"timer" at simplebus0 not configured
sxidog0 at simplebus0
"ir" at simplebus0 not configured
"codec" at simplebus0 not configured
sxits0 at simplebus0
com0 at simplebus0: ns16550, no working fifo
sxitwi0 at simplebus0
iic0 at sxitwi0
axppmic0 at iic0 addr 0x34: AXP209
sxitwi1 at simplebus0
iic1 at sxitwi1
"gpu" at simplebus0 not configured
dwge0 at simplebus0: address 02:0a:09:03:27:08
rlphy0 at dwge0 phy 1: RTL8201L 10/100 PHY, rev. 1
"hstimer" at simplebus0 not configured
"display-frontend" at simplebus0 not configured
"display-frontend" at simplebus0 not configured
"display-backend" at simplebus0 not configured
"display-backend" at simplebus0 not configured
gpio0 at sxipio0: 32 pins
gpio1 at sxipio0: 32 pins
gpio2 at sxipio0: 32 pins
gpio3 at sxipio0: 32 pins
gpio4 at sxipio0: 32 pins
gpio5 at sxipio0: 32 pins
gpio6 at sxipio0: 32 pins
gpio7 at sxipio0: 32 pins
gpio8 at sxipio0: 32 pins
usb2 at ohci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00 addr 1
usb3 at ohci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00 addr 1
simplefb0 at mainbus0: 1920x1080, 32bpp
wsdisplay0 at simplefb0 mux 1: console (std, vt100 emulation)
scsibus1 at sdmmc0: 2 targets, initiator 0
sd0 at scsibus1 targ 1 lun 0: <SD/MMC, SC64G, 0080> removable
sd0: 60906MB, 512 bytes/sector, 124735488 sectors
uhidev0 at uhub2 port 1 configuration 1 interface 0 "Lenovo ThinkPad Compact USB Keyboard with TrackPoint" rev 2.00/3.30 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub2 port 1 configuration 1 interface 1 "Lenovo ThinkPad Compact USB Keyboard with TrackPoint" rev 2.00/3.30 addr 2
uhidev1: iclass 3/1, 22 report ids
ums0 at uhidev1 reportid 1: 5 buttons, Z and W dir
wsmouse0 at ums0 mux 0
uhid0 at uhidev1 reportid 16: input=2, output=0, feature=0
uhid1 at uhidev1 reportid 17: input=2, output=0, feature=0
uhid2 at uhidev1 reportid 19: input=8, output=8, feature=8
uhid3 at uhidev1 reportid 21: input=2, output=0, feature=0
uhid4 at uhidev1 reportid 22: input=2, output=0, feature=0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
bootfile: sd0a:/bsd
boot device: sd0
root on sd0a (f7b555b0fa0e8c49.a) swap on sd0b dump on sd0b

Sensors output:

$ sysctl hw.sensors
hw.sensors.sxits0.temp0=39.50 degC
hw.sensors.axppmic0.temp0=30.00 degC
hw.sensors.axppmic0.volt0=4.95 VDC (ACIN)
hw.sensors.axppmic0.volt1=0.03 VDC (VBUS)
hw.sensors.axppmic0.volt2=4.85 VDC (APS)
hw.sensors.axppmic0.current0=0.11 A (ACIN)
hw.sensors.axppmic0.current1=0.00 A (VBUS)
hw.sensors.axppmic0.indicator0=On (ACIN), OK
hw.sensors.axppmic0.indicator1=Off (VBUS)

May 20, 2020

Amitai Schlair notqmail 1.08 released

notqmail logo

I’m pleased to announce notqmail 1.08, the latest update to notqmail. It addresses security vulnerabilities in qmail 1.03 reported yesterday by Qualys. As a fork, we’ve inherited qmail’s relatively few bugs; as an active qmail fork, we’ve addressed the vulnerabilities with a timely release; and as the community-driven open-source successor to qmail, we show our work.

The 1.08 release also includes many other small improvements we’ve made since 1.07, including fewer compiler warnings, less dead code, published intent to remove more presumed-dead code, my QMAILREMOTE patch, and our first few unit tests. For more detail, see the notqmail 1.08 release notes.


May 16, 2020

Nikita Gillmann On plant - update 2

Continued musings about plant on a saturday evening, still not 100% verified and opinionated as long as I don't double check with my existing notes. It's a bit chaotic, just taking notes.

So, what exactly do I want?

First it's mostly an experiment at this point.

I'm contributing to Guix, I'm contributing (rarely) to Nix, and I'm a pkgsrc developer. I've used and packaged for various other package managers in the past. And I really am unhappy with all of them, all have nice features, and all make compromises at some points, all make me annoyed when working with them for too long.

I could happily settle for Guix if it wasn't for a few nitpicks where the most unchangable ones can be reduced to code structure and certain parts being unchangable.

Before I dropped/indefinitely paused infotropique/OS core, I reworked my fork of Guix in a way that some issues were addressed, but the amount of work and time to rebase on a newer guix version made me reconsider using guix for small gains in it. I effectively had something which wanted to go in a new direction, but running with an engine I had no full control over. Notes prior to joining NetBSD shifted to the point where I often came back to basically wanting NetBSD but structured differently together with a package manager but not as in the old "this BSD can be dissected like Linux Distributions" mistake.

plant is my ideal package manager, and so is infotropique OS/core constructed from it in templates.

It'd base on MesCC, and some core tools.

As roughly listed here already:

Since the idea is to get to an modular extensible approach to package managers, we want the core to be relatively small and where necessary you should be able to replace the code.

It seems to point at some involvement of Lisp again or one of the few other languages which allow this type of code to be written.

Addendum:

In https://issues.guix.gnu.org/41286#1 which started my more public effort of porting Guix to NetBSD, Ludovic Courtès writes:

Hi Nikita,

Nikita Gillmann skribis:

as mentioned in IRC I have begun porting Guix to NetBSD (with the path taken not yet decided upon, just plain building guix itself for now).

Glibc provides argp. Arguably we don't have to check for argp because Guix targets glibc. But I am quiete certain that there will be people who will attempt to do what I am doing and run into this.

Guix targets glibc-based systems, so as you write, it’s reasonable to assume argp is present.

Also, using AC_CHECK_HEADERS doesn’t achieve anything: it only defines ‘HAVE_ARGP_H’ to zero or one.

Last, I don’t want to discourage anyone from porting, but I also want to be clear about what it entails. I’m strongly in favor of supporting only glibc because: (1) after all, it’s about GNU as a system, and (2) my experience with Nixpkgs is that supporting multiple C libraries is just too much work to maintain good support.

Thus I’m closing for now.

In a chat today with Janneke of GNU Mes I got enough pointers to start with adding support for NetBSD to mes. It's a start, but it will take a long time with everything else going on in my life.


May 15, 2020

Nikita Gillmann On plant - update 1

A quick reflection on what plant wanted to be:

In 2015 I started exploring the reliable creation of a live CD system for an application, which lead to thinking about the concept of an Operating System based on Guix, which rapidly diverged from Guix.

One of the main ideas was to provide building blocks instead of a monolithic approach to the core (you will find that you can modify Guix to a decent amount for an application, but some parts are designed in a way that (last tested in 2017) you can't add/override/etc). Furthermore I knew I wanted to test out ideas which would not work with the way Guix as a group works. While Guix and Nix allow yourself to be pretty independent from a central group, in practice you rely on their flow of work, their decisions, the way they upgrade and their infrastructure. Which isn't bad. I just want to test more decentralization, but made the (decision / mistake) to rely for that part at that time on a network which still needs some tweaks to be ready for this. Furthermore I disagree with some decisions which justify the existence of certain kernel flavors based on interpretation of blobs, licenses, how hardware works, and what some people tell others who ask to run this system (just buy a new/supported device is not an option for many people). Next I dislike the way updates are handled, but we all have to make compromises in what can be maintained, security updates handling, etc. I have my own ideas, which I guess/hope are not that new but just encode certain expectations and contracts. Modularity was also a big goal. Did I mention this should be a generic package manager and also operating system builder through templating and inheritance of the language used (maybe you don't know that Gentoo, Guix, Nix, etc are package managers). And much more.

I am going to extend on these points as I go through old stacks of papers and digital notes I assembled, but not all of them right now. This is the first copy of nightly notes I've started again after a long break on (publicly) thinking about package managers. This one is from 4 days ago. Don't take this as finished, technically correct, or whatever. It's just to try and start documenting more publicly what I do. No concept diagrams for now included. It's mostly written without much editing, and I hope to construct a more long-form technical preview once I get to a point where I can start working on it.

plant should be a set of concepts, protocols, and idealized package guidelines. My version of it would then mean I get to use an implementation of it, for example in Common Lisp. Maye the core could be in C. Maybe it could be entirely Lisp. The goal is just that recipes and binaries can be exchanged. If someone ships an extension to read ebuild-like syntax to build a subset of Gentoo with this? It's okay. But there needs to be a native package format first. I'm okay with NetBSD, so I don't really see a future in building yet another Unix-like system (which doesn't mean that infotropique OS/core is abandoned). When implementations can produce bit by bit the same output for builds, this should work. build-system etc can all be extensons people get however they want them to get (implicit: obtaining and depending on them is managed and documented for packages). I think this is similar to how guix conceptually works, but making all of it hackable (tricky for reprpducibility, so future compromises might apply), without any strong opinion on what plant should look like or run. Even the way you use the network, which network, will be up to you. i2p? tor? "clearnet"? GNUnet? plant doesn't care. Trade recipes, no binary builds on a central server, ... Groups can and probably will form to collaborate, but t really should just be well documented to give everyone what they want it to be. This doesn't have to contradict reproducible builds, it just has to work differently (and makes it harder).

lose notes, incomplete,...:


May 05, 2020

Unix Stack Exchange NetBSD/FreeBSD and TP-Link TL-WN727N: Second encounter

Moved from SU.
First part at https://superuser.com/questions/975564/netbsd-and-tp-link-tl-wn727n-atheros-ar9271-or-ralink-rt5370

So, I've installed NetBSD 7 and device shown again as ugen(ugein, lol).

ugen0 at uhub4 port 8 
ugen0: Mediatek 802.11 n WLAN, rev 2.01/00, addr 2

Then I'm installed FreeBSD 10.2 and ugen again.

usbconfig gives me ugen4.3: <product 0x7601 vendor 0x148f> at usbus4, cfg=0 md=HOST spd=HIGH(480Mbps) pwr=ON (90ma)

So, what's next? Buying new dongle is a last thing, which I'll make.


May 01, 2020

NetBSD.org New Developer in April 2020

April 21, 2020

Kimmo Suominen Serial console for NetBSD

I’ve been switching my NetBSD machines to using a serial console lately, as it is easier to copy output from it (no more screen shots). I would still need to figure out how to setup rtty1 to log console output, so unexpected output could also be inspected later (hello, panic).

On the NetBSD VM its bootblocks must be reinstalled to select a serial port as the console. I’ve been using the auto option, so I can easily switch between the VGA and serial consoles. I set speed to zero, which will use the speed configured in BIOS. The only available “speed” with qemu is 115200 bps.

installboot -v -o console=auto,speed=0 /dev/rsd0a /usr/mdec/bootxx_ffsv2

In Proxmox a serial port must be added to the VM on the Hardware tab. It is important to leave the Display setting at Default (or VGA) — for some reason input from the serial console will not work in the boot menu otherwise. (It will work in BIOS and at the login prompt.)


  1. Also known as remote-tty in Debian. 


April 16, 2020

Brett Lymn Thomas E. Dickey on NetBSD curses


I was recently pointed at a web page on Thomas E. Dickeys site talking about NetBSD curses.  It seems initially that the page was intended to be a pointer to some differences between ncurses and NetBSD curses and does appear to start off in this vein but it seems that the author has lost the plot as the document evolved and the tail end of it seems to be devolving into some sort of slanging match.  I don't want to go through Mr. Dickey's document point by point, that would be tedious but I would like to pick out some of the things that I believe to be the most egregious.  Please note that even though I am a NetBSD developer, the opinions below are my own and not the NetBSD projects.

The first thing that I would like to mention is that the author seems to be incapable of spelling my name correctly.  It isn't hard, 4 whole letters but the only time that it is spelt correctly is when the text is copied from a NetBSD commit.  If you ever do read this Mr. Dickey, I am not offended, plenty of people struggle with my last name, I am quite used to it.

Before going further, just a bit of background as to how I started working on the NetBSD curses library.  I originally was using a commercial x86 port of System V UNIX on my home PC but started looking elsewhere when I found that the SL/IP implementation was bugged.  At the time Linux had no networking at all but the BSD source code had just been ported to the PC in a project known as 386BSD.  Since this had networking I picked this up and then switched to NetBSD early on.  Being used to the SYSV curses I was disappointed to find that there was no real way of handling cursor and function keys in a terminal independent manner as there is in the SYSV curses.  I developed and submitted a patch to the NetBSD curses library that added the support for the keypad() function and modified getch() to perform the conversion of key sequences into key symbols.  It was on the basis of this patch that I was offered the opportunity to be a NetBSD developer.

When working on NetBSD curses my overall philosophy is to follow The Single UNIX® Specification, Version 2: X/Open Curses, Issue 4 Version 2 
(I will refer to this as SUSv2 from now on) where possible, if a behaviour is undefined then I look to ncurses and match the behaviour there if appropriate on the basis that this normally results in less issues for someone porting an application.  I add ncurses extensions in response to problem reports (PRs).

Let's now have a look at Mr. Dickey's comments (I am tempted to use the word diatribe but that seems a bit strong...).  I won't go over all of them but will pick up the topics that he has mentioned going down his page.

I would like to pick up on the comments about making the window structure opaque.  I don't apologise for this, I believe that the fact that WINDOW * was available for applications to poke at is a violation of software layering and just wrong.  Later in Mr. Dickey's page he takes NetBSD to task over error checking (more on this later) but by making the contents of WINDOW available for the application there is little use in error checking if the application can just write what they want anyway.  It took ncurses until 2007 to opaque WINDOW.  Mr. Dickey makes some ominous sounding statements about adding functions to compensate for the opaquing.  As far as I am concerned the interface to update a window is specified by SUSv2, if there are extensions that need to be added they will be added as they are needed.
Given I made WINDOW opaque nearly 20 years ago, I have yet to see a PR to indicate that more is needed.  The positives of making WINDOW opaque are not only enforcing correct layering of software but it also means that the contents of WINDOW can be modified without requiring a major bump of the library since nothing outside the curses library has access so backward compatibility is maintained.

Making the character string in scanw const just makes sense the function does not need to modify the string so it should be a constant.  The NetBSD project does not need to worry about the compiler objecting because the base compiler works with this statement.

Wide curses was a Google Summer of Code project that was done by Ruibiao Qiu.  I think that this was a great success and added some very useful functionality to NetBSD curses.  It seems that Mr. Dickey is a bit confused by this, in actual fact, he is totally wrong on a number of points:
  1. The non-spacing characters are not stored as a linked list, they are an array
  2. The interface for setting non-spacing characters is anything that uses the complex character type (cchar_t)
He seems a bit mystified by the term non-spacing character, this term comes from SUSv2 described here
The non-spacing characters are dynamically added when required, this was my design as I thought that simply having a fixed array for every character cell in the display was a waste of memory, it would be more efficient to only allocate memory as required for the non-spacing characters.

The function specification for ungetch is ambiguous in SUSv2, NetBSD will accept the output from getch on the basis that the input stream has already been processed and it makes no sense to save the last character in what could be a multi-byte sequence.

As for testing of curses, yes, this is mine.  I am happy that at least some of our curses functions are being tested to ensure that any changes made don't affect the on-screen output and that if there are changes that they are highlighted and can be analysed for correctness.  It is too easy with terminal output to have something that "looks right" but is doing the wrong thing in the background.

Now we get to something I wrote on a mailing list:
 
Re: curses: create panel from stdscr?

I stand by my comments there.  I was referring to my attempt to report what I thought was an issue with the ncurses libmenu implementation.  I no longer have the email nor the reply from Mr. Dickey but my clear recollection of the tone and attitude was that I would not be bothered raising issues about ncurses with Mr. Dickey ever again.  I wrote the libmenu and libform libraries for NetBSD using the book "Programmer's Guide: Character User Interface (FMLI and ETI)", this is UNIX System Laboratories book so I regard it as definitive for the libraries.  I noticed a variance in the ncurses libmenu implementation but as I have stated my efforts of reporting that were not fruitful.  Mr. Dickey did reach out to me in late 2017 asking for the email I was referencing, his comment was that my response was vague, actually, my exact words were:
I will try and dig it up.  It was a very long time ago.  I think it was
related to some behaviours in libmenu.  I will try and dig back through
my mail history but it may take a bit of time.
Alas I never found the original email but looking at my commits around libmenu I would have sent that email in 1999 some time so I don't think that this is very surprising.  At the time of the reply I thought that perhaps there was some interest in actually addressing the issue but the actual fact is that all it was was a fishing expedition to provide more ammunition for an attack on NetBSD curses which is disappointing.

Getting back to the email quoted above, in the book I have create_panel() is documented as taking a WINDOW as parameter.  Though, internally, stdscr is treated as a window the books description of create_panel() does make it clear that WINDOW is expected to be the return from newwin() not just stdscr.

As for missing functions, they will be added as and when the developers have time.  A PR may guide us in what is needed first but given that NetBSD is a volunteer project the time that the developers can devote to adding new features can be variable.  We are aiming to have something that is useful and not necessarily a clone of ncurses.

Mr. Dickey makes a point about portability.  I myself have never been concerned about making NetBSD curses portable, this is not an aim that I have ever had.  I would much rather be adding new features and fixing bugs for NetBSD than struggling with the complexities of making the NetBSD curses library portable.  If others wish to do this and feed back fixes then they will be considered but in the main, portability is not a focus.

He also turns on some random gcc flags and then berates NetBSD curses for producing extra warnings.  It is difficult to determine what flags he has turned on to produce these warnings.  The NetBSD build system treats any warnings as errors and the build terminates, the NetBSD curses library builds with the standard NetBSD gcc options without issue.  Without knowing what options were uses and what defines were in effect the information that Mr. Dickey has presented is useless.  He is welcome to submit a PR using the public interface and it will be looked into.

He then notes the output differences in the wide character implementations.  Unfortunately he has not reached out to me (at least) with this information.  It would be useful to have the tools to test this issue and rectify it.  I will note that he used a rather old version of NetBSD, it would be interesting to see if these issues still exists.

We then have a snide little bit about the number of error returns and error checking.  He says "Error-checking in NetBSD curses is seen as a problem by its developers" which is absolutely untrue and uncalled for.  He seems to be forgetting at this by that, by his own admission, that NetBSD curses does not have all the functions that ncurses has so there should be less returns of status simply from that.  Even if this were not true, if we look at the ratio of ERR/OK returns per number of statements - using the numbers from his page - NetBSD curses has 12987 statements which translates to a ERR/OK return every 25 statements where as ncurses has 24566 statements which translates to a ERR/OK return ever 38 statements.  So NetBSD curses actually assigns status codes more often than ncurses.  Regardless, I view this metric as specious - simply counting returns is rubbish, look at this:

if ((x > maxx) || (y > maxy))
         return ERR;

VS:

if (x > maxx)
        return ERR;

if (y > maxy)
        return ERR;


Both do the same thing but we get double the ERR returns in the second one.  I am not saying this is what is happening but just pointing out the fallacy of counting ERR/OK assignments and trying to draw a conclusion from it without properly analysing the code.

He lambasts NetBSD for commenting out unsupported functions in pycurses, this is just a reflection of the NetBSD curses not having all the functions at the moment.  This is a pkgsrc thing and locally applied, pycurses is unique in that it wants all the curses functions so it can provide bindings, since NetBSD curses doesn't have them then the unsupported ones are commented out.  It would be ideal if they all existed, hopefully one day they will.

The final thing I would like to pick up on is the comment:

"Without documentation, there is no point in making bug reports."

Which is just a lame excuse, even the lack of documentation is something that can be a bug report.  There is a public interface for reporting bugs.  Not using it and then complaining on your own website is not helpful to NetBSD but then again, perhaps that is the intention after all.

April 06, 2020

NetBSD.org pkgsrc-2020Q1 released

April 02, 2020

NetBSD.org NetBSD 8.2 released
NetBSD.org Extended support of the netbsd-7 branch

March 20, 2020

Unix Stack Exchange How to change the font size in NetBSD (TTY no GUI)

NetBSD 7.1 (GENERIC.201703111743Z) amd64

In TTY without gui , the better screen resolution is set at boot time but the font is really to small.
So how to change that

On debian dpkg-reconfigure console-setup
in grub GRUB_GFXMODE=800x600

I find wsconscfg with a example :

wsconscfg -t 80x50 -e vt100 3

but I get

wsconscfg: WSDISPLAYIO_ADDSCREEN: Device not configured

I look at man wsconscfg wsconsctl wscons wsdisplay vga etc..
All of them link to the other but I can't find a way.


March 17, 2020

Super User Most Windows-like text editor that can be used through PuTTY in NetBSD?

Please help me find an editor that behaves more like the modern Windows editors do, that I could use on a NetBSD box through PuTTY.

This includes the absense of the distinction between view/insert/append modes, basic shortcuts such as Ctrl+Left/Right to jump a word at a time, and other niceties such as perhaps Home/End and PageUp/Down moving the cursor around (in my vi they just ding).

I can live without macros because I only ever tweak an occasional setting this way, and nothing else.

pico is pretty close, but I was wondering if there are other editors that might fit this bill?


March 15, 2020

Unix Stack Exchange pkgin installation problem (NetBSD)

I just installed NetBSD 7.1.1 (i386) on my old laptop.

During the installation, I could not install pgkin (I don't know why), so I skipped it and now I have a NetBSD 7.1.1 installed on my laptop without pkgin.

My problem is "How to install pkgin on NetBSD (i386) ?"

I found this Unixmen tutorial and I followed it:

I tried :

#export PKG_PATH="http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/6.0_BETA_current/All/"
# pkg_add -v pkgin

And I got :

pkg_add: Can't process ftp://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/amd64/6.0_BETA_current/All/%0d/pkgin*: Not Found
pkg_add: no pkg found for 'pkgin',sorry.
pkg_add: 1 package addition failed

I know this is a wrong command because this ftp address is for amd64 while my laptop and this NetBSD is i386. (I can't find the correct command for i386 )

I also followed instructions of pkgin.net, and I did

git clone https://github.com/NetBSDfr/pkgin.git

on another computer and copied the output (which is a folder name pkgin) to my NetBSD (my NetBSD doesn't have 'git' command)

and then I did :

./configure --prefix=/usr/pkg --with-libraries=/usr/pkg/lib --with-includes=/usr/pkg/include

and then :

make

but I got :

#   compile  pkgin/summary.o
gcc -O2    -std=gnu99    -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-sign-compare  -Wno-traditional  -Wa,--fatal-warnings -Wreturn-type -Wswitch -Wshadow -Werror    -DPKGIN_VERSION=\""0.9.4 for NetBSD-7.1.1 i386"\" -DNETBSD  -g -DLOCALBASE=\"/usr/local\"           -DPKG_SYSCONFDIR=\"/usr/local/etc\"         -DPKG_DBDIR="\"/var/db/pkg\""           -DDEF_LOG_DIR="\"/var/db/pkg\""         -DPKGIN_DB=\"/var/db/pkgin\"            -DPKGTOOLS=\"/usr/local/sbin\" -DHAVE_CONFIG_H -D_LARGEFILE_SOURCE -D_LARGE_FILES -DCHECK_MACHINE_ARCH=\"i386\" -Iexternal -I. -I/usr/local/include  -c    summary.c
*** Error code 1

Stop.
make: stopped in /root/pkgin

I think this error occurs because of the dependencies. (which is mentioned in pkgin.net) but still, don't know how to install those dependencies.

EDIT: I found "http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/7.1.1/All/" but it still says

no pkg fond for 'pkgin', sorry

SOLVED:

** I solved the problem by writing 7.1 instead of 7.1.1**


March 06, 2020

Stack Overflow How do I set root password for mysql 5.7 on netbsd

I'm trying to get mySQL 5.7 working on NetBSD (unix). When I try "mysql -u root" I get

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)

When I try "mysqld --initialize-insecure" I get

[ERROR] --initialize specified but the data directory has files in it.

What do I do?


March 02, 2020

Emile Heitor Let's Encrypt certificates using LEGO

This post is more like a self-reminder on how I setup automatic SSL/TLS certificate renewal on my servers.

I chose LEGO to handle my certificates renewal with Let’s Encrypt because it’s simple to use, has no dependency, great documentation and is worked on at a constant pace.

I found this and this articles very useful, but they are outdated in their use of the tls and http parameters. So here are my notes.

This procedure is Debian GNU/Linux based but I also used it pretty much as-is on NetBSD and FreeBSD, only nginx related PATHs changed.

nginx

In /etc/nginx/sites-available/default:

1
include /etc/nginx/sites-available/letsencrypt;

Which contains:

1
2
3
4
location /.well-known/acme-challenge {
proxy_pass http://127.0.0.1:8181;
proxy_set_header Host $host;
}

Check nginx.conf syntax then reload it.

1
2
$ sudo nginx -t
$ sudo nginx -s reload

Let’s Encrypt

Go to a writable directory where LEGO will write the challenges

1
$ cd ~/www/letsencrypt

To request and install a new certificate, first prepare nginx.conf:

/etc/nginx/sites-available/https:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
listen443 ssl;
listen[::]:443 ssl;
server_namekorriban.imil.net;

ssl_certificate/usr/local/etc/letsencrypt/certificates/korriban.imil.net.crt;
ssl_certificate_key/usr/local/etc/letsencrypt/certificates/korriban.imil.net.key;

# ...

# for future use of the tls challenge
location /.well-known/acme-challenge {
proxy_pass http://127.0.0.1:4444;
proxy_set_header Host $host;
}

# ...

Then execute LEGO with desired parameters:

1
$ sudo lego --email="[email protected]" --domains="korriban.imil.net" --http.port :8181 --http --path=/usr/local/etc/letsencrypt run

And check / reload nginx.

If you need a multi-domain certificate, simply add multiples --domains.

To renew the certificate using the tls challenge if it expires within 30 days:

1
$ sudo /usr/local/bin/lego --email="[email protected]" --domains="korriban.imil.net" --tls.port :4444 --tls --path=/usr/local/etc/letsencrypt renew --days 30

Automatic renewal

1
$ cat bin/lerenew.sh
1
2
3
4
5
#!/bin/sh

cd /home/imil/www/letsencrypt
/usr/local/bin/lego --email="[email protected]" --domains="korriban.imil.net" --tls.port :4444 --tls --path=/usr/local/etc/letsencrypt renew --days 30
nginx -s reload

cron

1
00 3 * * * /home/imil/bin/lerenew.sh >/home/imil/log/lerenew.log 2>&1

reload mail system

My mail system also uses Let’s Encrypt certificates, it is hosted on virtual machines in the same subnet as the web server, which exports /usr/local/etc/letsencrypt/ via NFS. The mail server runs NetBSD. In order to watch any modifications in the certificates directory, I use direvent, which will call a script to reload both dovecot and postfix:

1
$ cat /usr/pkg/etc/direvent.conf
1
2
3
4
5
6
7
8
9
10
11
syslog {
facility local0;
tag "direvent";
print-priority yes;
}

watcher {
path /usr/local/etc/letsencrypt/certificates;
event write;
command "/home/imil/bin/mailrestart.sh";
}
1
$ cat bin/mailrestart.sh
1
2
3
4
5
6
#!/bin/sh

/usr/sbin/postfix reload
/usr/pkg/bin/doveadm reload

su imil -c 'echo "done"|mail -s "postfix and dovecot reloaded" imil'

direvent is started from /etc/rc.local as there’s no init script given with the package.

1
/usr/pkg/bin/direvent /usr/pkg/etc/direvent.conf