NetBSD Planet


October 24, 2020

Ruben Schade Firefox 82.0 resolves macOS stuttering scrolling

My new MacBook Pro coincided with the release of Firefox 81.x, which lead me to think there was something wrong with the discrete GPU on this refurbished machine. Each time I loaded a site and scrolled, regardless of how heavy the page was, it would occasionally stop then lurch in an attempt to catch up. I joked with colleagues that it was a *nix VESA desktop emulation mode.

Safari and Vivaldi didn’t have the same issue, which thankfully ruled out a hardware.

Firefox icon

I’m pleased to report now that the issue is gone as of Firefox 82. Either that, or an extension I use also updated in the interim. Either way, I’m unreasonably happy.

I used to use Phoenix/Firebird/Firefox back in the day to push against IE. Now the few of us still using it are at it again, only we use it to push against Chrome hegemony. Please use it; it’s a great browser and especially quick since the Quantum update. We need its user agent in server logs to show the world there’s still value in cross-browser testing and development. We’re already starting to see Chrome-only sites again, presumably written by people who either weren’t alive or don’t remember the lessons of the first browser wars.

Special thanks to these fine contributors for maintaining the Homebrew Cask for Firefox, the FreeBSD Gecko team, and ryoon for pkgsrc. A lot of work goes into people like me being able to install Firefox on our various platforms with a single command.


This post originally appeared on Rubenerd.

DragonFly BSD Digest In Other BSDs for 2020/10/24

It’s apparently release week?


October 23, 2020

UnitedBSD A sad day for BSD and Python

Since I realise that I'm in the minority on this issue, let me point out that the (at the time) vice president of the FSF has written about this same problem, and I also knew a former Nokia developer who felt similarly-- about the Python part, not OpenBSD.

OpenBSD is my favourite operating system. It's new to me, I've spent time with NetBSD and FreeBSD, and 6.8 may mean I'm increasing the time I spend with NetBSD.

The main thing I need an operating system for is running PyPy. Python2 will do in a pinch, but I realise the Python Foundation (while getting paid by Microsoft) has abandoned Python2. No problem, right? PyPy2 fulfils my needs.

Only in FreeBSD and in OpenBSD, and probably in NetBSD if this keeps up, PyPy is deprecated because Python 2 is-- and Python 2 is used to build PyPy.

Essentially, PyPy2 is a well-maintained Python2. All the projects that need Python2 may find it easier to port to PyPy than Python3. I've already got a Python3 port of my favourite software, but it is certainly inferior-- I tested it for about half a year (I'm also the author.)

I am not interested in the Python Foundation's hegemony.

Python 2 is needed to build PyPy, but FreeBSD and NetBSD have Linux compatibility layers, and there is a pre-built (portable) PyPy2 for GNU/Linux.

Today, I installed BSD over my last GNU/Linux installation. Good riddance. But I wish I'd installed 6.7 instead of 6.8, because 6.8 is the first version of OpenBSD that lacks Python2. That wouldn't bother me if it had PyPy2 instead.

Again, PyPy2 is actively maintained. So not building it because it needs Python 2 to build is silly, but that's how things are working in BSD. It's a big catch 22.

I've talked to FreeBSD maintainers about this-- that was the first BSD I isntalled. I've talked to the person who did the port for Python2 or PyPy in OpenBSD, and I've talked to the PyPY maintainers.

This seems unresolved. I really won't enjoy computing until I can use PyPy again. I may eventually ditch my own favourite language and create a new one, based on something else-- but right now, this is very depressing.

I've been waiting for years to ditch Linux now. It's a pretty hollow victory with this taking place. But it was my choice to use OpenBSD 6.8 as the reason to stop using my last GNU/Linux install.

If I'd just installed 6.7, I would be a lot happier right now. I was excited about 6.8, but I knew this was a possibility.

NetBSD is a possible fix, until they do this too. I almost wish there was binary compatibility and I could just copy Python from NetBSD to OpenBSD-- but there are pretty good reasons that binaries are not compatible. I might try copying Python from 6.7 to 6.8, but I know there are reasons that might not work (and it's certainly no good for production if it works, but that's a minor concern.)

I won't miss GNU/Linux. I DO miss Python. And I have no interest in Python 3. I'd rather use a different language-- but I watch those closely, and I don't love most of them.

For example, I started working on a somewhat different version of my Python software in JavaScript. It's a bit apples and oranges, but it shows the amount of trouble I'm willing to go to sometimes.

Anyway, first impressions of OpenBSD 6.8:

It's my favourite operating system in the world. I hate it.

I'm not saying this won't improve. I bet it takes years to get fixed though. The Python Foundation is incredibly arrogant; it's one thing to ditch software people want and need. That's how it goes, it's not their job to do work that doesn't suit them.

The way they've campaigned against anybody working on compatibility is the incredibly selfish part. Projects like Trinity (KDE 3.5 fork) and MATE (GNOME 2 fork) and all projects that work hard to maintain things that were abandoned by devs but not fans-- they deserve more respect.

PyPy deserves better, so do its users. I hope the PyPy devs will make a portable PyPy for OpenBSD-- they did one for FreeBSD, but not in a while.

I've known about Python 3 for years-- there are computer science majors who aren't convinced it's better, I'm thoroughly convinced it's not. Python 3 is pushed on people who don't want it. "You'll like it if you just" "It's really not that hard to" NO. It's not even true. It's only true for some, and by waving away the many people it's most certainly not true for, it becomes a circular argument.

Long live PyPy. And hooray for the Linux compatibility layer (we really shouldn't need it, but sometimes it's nice.) It's not in OpenBSD anymore, NetBSD still has it.


October 21, 2020

NetBSD Blog NetBSD 9.1 released

After a small delay*, the NetBSD Project is pleased to announce NetBSD 9.1, the first feature and stability maintenance release of the netbsd-9 stable branch.

The new release features (among various other changes) many bug fixes, a few performance enhancements, stability improvements for ZFS and LFS and support for USB security keys in a mode easily usable in Firefox and other applications.

For more details and instructions see the 9.1 announcement.

Get NetBSD 9.1 from our CDN (provided by fastly) or one of the ftp mirrors.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, and other services may be found at https://www.NetBSD.org/mirrors/.

* for the delay: let us say there was a minor hickup and we took the opportunity to provide up to date timezone files for NetBSD users in Fiji.


October 20, 2020

NetBSD.org NetBSD 9.1 released

October 19, 2020

NetBSD Blog Google Summer of Code 2020: [Final Report] Enhancing Syzkaller support for NetBSD
This report was written by Ayushu Sharma as part of Google Summer of Code 2020.

This post is a follow up of the first report and second report. Post summarizes the work done during the third and final coding period for the Google Summer of Code (GSoc’20) project - Enhance Syzkaller support for NetBSD

Sys2syz

Sys2syz would give an extra edge to Syzkaller for NetBSD. It has a potential of efficiently automating the conversion of syscall definitions to syzkaller’s grammar. This can aid in increasing the number of syscalls covered by Syzkaller significantly with the minimum possibility of manual errors. Let’s delve into its internals.

A peek into Syz2syz Internals

This tool parses the source code of device drivers present in C to a format which is compatible with grammar customized for syzkaller. Here, we try to cull the details of the target device by compiling, and then collocate the details with our python code. For further details about proposed design for the tool, refer to previous post.

Python code follows 4 major steps:

Extraction:

This step involves fetching the possible ioctl commands for the target device driver and getting the files which have to be included in our dev_target.txt file. We have already seen all the commands for device drivers are defined in a specific way. These commands defined in the header files need to be grepped along with the major details, regex comes in as a rescue for this


	io = re.compile("#define\s+(.*)\s+_IO\((.*)\).*")
	iow = re.compile("#define\s+(.*)\s+_IOW\((.*),\s+(.*),\s+(.*)\).*")
	ior = re.compile("#define\s+(.*)\s+_IOR\((.*),\s+(.*),\s+(.*)\).*")
	iowr = re.compile("#define\s+(.*)\s+_IOWR\((.*),\s+(.*),\s+(.*)\).*")

Code scans through all the header files present in the target device folder and extracts all the commands along with their details using compiled regex expressions. Details include the direction of buffer(null, in, out, inout) based on the types of Ioctl calls(_IO, _IOR, _IOW, _IOWR) and the argument of the call. These are stored in a file named ioctl_commands.txt at location out/&lttarget_name&gt. Example output:


out, I2C_IOCTL_EXEC, i2c_ioctl_exec_t

Preprocessing:

Preprocessing is required for getting XML files, about which we would look in the next step. Bear plays a major role when it comes to preprocessing C files. It records the commands executed for building the target device driver. This step is performed when setup.sh script is executed.

Extracted commands are modified with the help of parse_commands() function to include ‘-E’ and ‘-fdirectives’ flags and give it a new output location. Commands extracted by this function are then used by the compile_target function which filters out the unnecessary flags and generates preprocessed files in our output directory.

Generating XML files

Run C2xml on the preprocessed files to fetch XML files which stores source code in a tree-like structure, making it easier to collect all the information related to each and every element of structures, unions etc. For eg:


	&ltsymbol type="struct" id="_5970" file="am2315.i" start-line="13240" start-col="16" end-line="13244" end-col="11" bit-size="96" alignment="4" offset="0">
		&ltsymbol type="node" id="_5971" ident="ipending" file="am2315.i" start-line="13241" start-col="33" end-line="13241" end-col="41" bit-size="32" alignment="4" offset="0" base-type-builtin="unsigned int"/<
		&ltsymbol type="node" id="_5972" ident="ilevel" file="am2315.i" start-line="13242" start-col="33" end-line="13242" end-col="39" bit-size="32" alignment="4" offset="4" base-type-builtin="int"/>
		&ltsymbol type="node" id="_5973" ident="imasked" file="am2315.i" start-line="13243" start-col="33" end-line="13243" end-col="40" bit-size="32" alignment="4" offset="8" base-type-builtin="unsigned int"/>
	</symbol>
	&ltsymbol type="pointer" id="_5976" file="am2315.i" start-line="13249" start-col="14" end-line="13249" end-col="25" bit-size="64" alignment="8" offset="0" base-type-builtin="void"/>
	&ltsymbol type="array" id="_5978" file="am2315.i" start-line="13250" start-col="33" end-line="13250" end-col="39" bit-size="288" alignment="4" offset="0" base-type-builtin="unsigned int" array-size="9"/>

We would further see how attributes like - idents, id, type, base-type-builtin etc conveniently helps us to analyze code and generate descriptions in a trouble-free manner .

Descriptions.py

Final part, which offers a txt file storing all the required descriptions as its output. Here, information from the xml files and ioctl_commands.txt are combined together to generate descriptions of ioctl commands and their arguments.

Xml files for the given target device are parsed to form trees,


for file in (os.listdir(self.target)):
	tree = ET.parse(self.target+file)
	self.trees.append(tree)

We then traverse through these trees to search for the arguments of a particular ioctl command (particularly _IOR, _IOW, _IOWR commands) by the name of the argument. Once an element with the same value for ident attribute is found, attributes of the element are further examined to get its type. Possible types for these arguments are - struct, union, enum, function, array, pointer, macro and node. Using the type information we determine the way to define the element in accordance with syzkaller’s grammar syntax.

Building structs and unions involves defining their elements too, XML makes it easier. Program analyses each and every element which is a child of the root (struct/union) and generates its definitions. A dictionary helps in tracking the structs/unions which have been already built. Later, the dictionary is used to pretty print all the structs and union in the output file. Here is a code snippet which depicts the approach


            name = child.get("ident")
            if name not in self.structs_and_unions.keys():
                elements = {}
                for element in child:
                    elem_type = self.get_type(element)
                    elem_ident = element.get("ident")
                    if elem_type == None:
                        elem_type = element.get("type") 
                    elements[element.get("ident")] = elem_type

                element_str = ""
                for element in elements: 
                    element_str += element + "\t" + elements[element] + "\n"
                self.structs_and_unions[name] = " {\n" + element_str + "}\n"
            return str(name)

Task of creating descriptions for arrays is made simpler due to the attribute - `array-size`. When it comes to dealing with pointers, syzkaller needs the user to fill in the direction of the pointer. This has already been taken care of while analyzing the ioctl commands in Extractor.py. The second argument with in/out/inout as its possible value depends on ‘fun’ macros - _IOR, _IOW, _IOWR respectively.

There is another category named as nodes which can be distinguished using the base-type-builtin and base-type attributes.

Result

Once the setup script for sys2syz is executed, sys2syz can be used for a certain target_device file by executing the python wrapper script (sys2syz.py) with :

#bin/sh
python sys2syz.py -t &ltabsolute_path_to_device_driver_source> -c compile_commands.json -v

This would generate a dev_&ltdevice_driver&gt.txt file in the out directory. An example description file autogenerated by sys2syz for i2c device driver.


#Autogenerated by sys2syz
include 

resource fd_i2c[fd]

syz_open_dev$I2C(dev ptr[in, string["/dev/i2c"]], id intptr, flags flags[open_flags]) fd_i2c

ioctl$I2C_IOCTL_EXEC(fd fd_i2c, cmd const[I2C_IOCTL_EXEC], arg ptr[out, i2c_ioctl_exec])

i2c_ioctl_exec {
iie_op	flags[i2c_op_t_flags]
iie_addr	int16
iie_buflen	len[iie_buf, intptr]
iie_buf	buffer[out]
iie_cmdlen	len[iie_cmd, intptr]
iie_cmd	buffer[out]
}

Future Work

Though we have a basic working structure of this tool, yet a lot has to be worked upon for leveling it up to make the best of it. Perfect goals would be met when there would be least of manual labor needed. Sys2syz still looks forward to automating the detection of macros used by the flag types in syzkaller. List of to-dos also includes extending syzkaller’s support for generation of description of syscalls.

Some other yet-to-be-done tasks include-

Summary

We have surely reached closer to our goals but the project needs active involvement and incremental updates to scale it up to its full potential. Looking forward to much more learning and making more contribution to NetBSD community.

Atlast, a word of thanks to my mentors William Coldwell, Siddharth Muralee, Santhosh Raju and Kamil Rytarowski as well as the NetBSD organization for being extremely supportive. Also, I owe a big thanks to Google for giving me such a glaring opportunity to work on this project.


October 17, 2020

DragonFly BSD Digest In Other BSDs for 2020/10/17

This list of links runs in the same order of the BSD RSS feeds in my reader.  What a coincidence!

Unix Stack Exchange Does *BSD have the ability to encrypt a system partition with full disk encryption?

Does FreeBSD, NetBSD, or OpenBSD have an encryption feature like Linux's dm-crypt? And will it work for a system partition?

UnitedBSD Various issues getting netbsd up and running on amd64

I recently tried setting up a netbsd box for fun on an amd64 laptop. There have been/still are several issues I encountered, some of which I've implemented hacky workarounds for.

For the installer:

Next, pkgsrc cannot install anything by default. I have gcc 7.3.0 from the comp set, but it's looking for libstdc++.so.6, which doesn't exist. Comp gives libstdc++.so.9 instead. For pkgsrc, changing mk.conf to require a more recent gcc, and installing said gcc, works. For other tasks (such as build.sh), this doesn't work. I copied libstdc++.so.9 into libstdc++.so.6 as a workaround. This may be addressed by doing a manual install instead of going through sysinst as well, I am assuming. This will probably be my favored workflow from now on.

Battery is currently detected but battery state (powered/discharging) does not seem to be detected (at least I see no messages in /var/log/messages and mate-desktop doesn't see the change). Would enabling acpi power monitor in the kernel help?

Touchpad options are a bit limited but otherwise work well. I couldn't find how to disable pointer acceleration, though, and I have to reset the scale/maxspeed at every boot for now. Probably missing what config file to use here? Or just .xinitrc or such?

Trying to get emul-linux working, I found that the suse13 packages are quite old and the handful of packages I tried did not seem to like it (my "goal test case" is to get portable palemoon working: it's a relatively complex program but doesn't necessarily require the likes of gtk3, pulseaudio, and such, so hopefully it could have worked with the suse13-based core). Palemoon just segfaults (it's a non-PIE linux x86_64 executable and COMPAT_LINUX and EXEC_ELF64 are active in the running kernel), and trying to run linux-ldd on it also causes linux-ldd to segfault (code 139).
I am currently trying to substitute it with a gentoo stage3 tarball, I pull the extra packages I need from the cloveros BINHOST. I setup the emul filesystems in fstab as instructed, but I'm missing something if I want to chroot into the emul root to have an easier time installing dependencies. I haven't investigated further here, although trying to execute binaries from this gentoo-like system fails due to a missing interpreter (e.g. when trying to execute ldd). Probably path-related? I'm half-aware that what I'm doing is crazy,. If someone has a saner solution, I'd like to hear about it. In any case, this should still more or less be in line with https://man.netbsd.org/compat_linux.8 although I'm going with the whole stage3 because of versioned dependency incompatibilities avalanching through the whole stack.

UPDATE: As expected, this did not help. Both the cloveros binary for palemoon and the upstream binary were tried, both segfaulted on launch. gdb is of no help here because it's not able to work with linux ABI binaries.

The cpu on the device is a kabylake, whose gpu seems to work out of the box. However, it does complain about not being able to find the i915 firmware (kbl_guc_ver9_16.bin and kbl_dec_ver1.bin are both mentioned). Said firmware doesn't actually appear to be available. Can I steal this from linux and put it somewhere? (If so: where? messages tells me this is an internal firmware so it has to be put in the initramfs or in-kernel, but I find no instructions to do either in the manual).

The device also features a touchscreen. It seems this is actually detected by netbsd on boot, however:

uts0: autoconfiguration error: touchscreen has no range support

And it fails to work.

Other than that, I'm surprised about how the rest of the system seems to be working for now. I'm not getting audio jack detection on linux, yet on netbsd it works out of the box. Wifi also worked straight away, and so on.


October 16, 2020

UnitedBSD Welcome to NetBSD 9.1!

It has been just tagged.

http://mail-index.netbsd.org/source-changes/2020/10/16/msg122981.html

Now, we are waiting for the official install media and the formal announcement.

UnitedBSD Use of disklabel, MBR and GPT

Hello!
After partitioning a new disk with MBR, creating two primary partitions, I was trying to define a disklabel in each of them. Quite soon I realized it was not possible. After an IRC conversation, which was very instructive yesterday, here are some considerations.

An important premise is that disklabel, MBR and GPT are data structures intended to describe a whole disk, not a single partition: internally, they then divide the disk into partitions. These single partitions can not host themselves a disklabel, an MBR or a GPT to further subdivide them, because they are not an entire disk.

Maybe this is trivial, but during the installation of NetBSD a disklabel is actually nested inside a primary MBR partition. So, I would like to share some further clarification.

Assume that you have this example MBR:

1. NTFS, Windows system partition
2. NTFS, Windows data
3. (empty)
4. (empty)

(for simplicity, I will not consider the case of extended partitions, but it would be similar).

Assume that, when installing NetBSD with sysinst(8), partition 3. is chosen for NetBSD. sysinst(8) does not change MBR in this case (thus preserving primary partitions 1. and 2.).

There are several other options:

As you may note, the use of MBR is never abandoned. This is done for two reasons:

Note that the above behaviour is about amd64; for other ports it may change.

Whatever choices the user makes, in the end there will be an MBR primary partition dedicated exclusively to NetBSD. Inside it, sysinst(8) installs disklabel. disklabel(8) allows to further divide this NetBSD-dedicated MBR partition, originating disklabel partitions a (which is generally used for the NetBSD root directory /), b (generally used for swap), e (it may be /home, for example), f (for example, /usr), g (for example, /var), etc.; in amd64, partitions c and d are reserved, to represent respectively the NetBSD MBR primary partition and the whole disk, as shown here.

It may seem that this disklabel, nested into an MBR primary partition, is used to further divide internally this partition: so, for another MBR primary partition on the same disk, there may be another disklabel which can be used the same way. This is wrong.

Disklabel is an alternative, and not an integration, to MBR.

In the case of an amd64 NetBSD installation inside a PC/DOS disk, the disklabel is placed inside the MBR primary partition dedicated to NetBSD, in the second block (and the first block is used for the NetBSD bootloader). This is done to make sure that the disklabel data structure is preserved and not altered or deleted by the action of other OSs: in fact, the MBR primary partition dedicated to NetBSD is not used by DOS or similar OSs, but it is recognized by them. Without this issue, disklabel (whose location is flexible) could be placed in the second sector of the disk (while MBR is in the first), not the second sector of a specific partition.

When the machine is booted with NetBSD, disklabel is the only way NetBSD looks at the disk, the entire disk; MBR in this case is not considered at all. (Obviously, during the installation, a reference to the NetBSD bootloader is placed in the boot sector of MBR, so that NetBSD can be booted). Disklabel provides exactly all the information MBR provides, but in a way NetBSD can understand. It provides the full size of the disk (disklabel partition d for amd64), the size and location of the usable portion of the disk (disklabel partition c, for example, which represents in amd64 the NetBSD primary partition), and the structure of this portion of the disk (disklabel custom partitions a, b, e, f, g, etc.). Thinking about MBR, it actually provides the same information, but in a way the other OSs can understand: the full size of the disk, the size and location of the usable portion(s) of the disk and their structure (all the primary partitions whose type is DOS-compatible).

Disklabel and MBR are then just two alternative ways to represent the same reality, the whole disk. The former is used by NetBSD and the latter is used by DOS-compatible OSs like Windows, for the same disk. The location of disklabel (nested inside an MBR primary partition) is just a consequence of possible compatibility issues with other OSs. But both MBR and disklabel are meant to be unique inside a disk (and in systems where PC/IBM compatibility issues are absent, they should not even coexist: disklabel only is used; on the other hand, in a Windows PC, only MBR is used). A disk can not host more than one MBR, or more than one disklabel. My initial intention of creating multiple disklabels, one for each MBR primary partition, was wrong.

Here, Figure 2.1 well describes the way disklabel (and so, NetBSD) looks at the whole disk. Refer to the blue labels, on the right of the image. The same Figure also well describes the way MBR (and so, a DOS-compatible OS) looks at the whole disk: refer to the red labels, in the center of the image.

The fact that NetBSD only uses disklabel for his awareness of the disk is also proved by the /dev directory: for each disk, for example wd0, only the disklabel partitions are available: wd0a, wd0b, etc. (along with the corresponding raw partitions rwd0a, rwd0b, etc.). There is no way to reference the other MBR primary partitions (so, installing a disklabel on them would be impossible, other than uncorrect), because from the NetBSD (and disklabel) perspective, they simply do not exist: it only exists a portion of the disk which is not used by any disklabel partition (and where actually are placed the eventual other MBR primary partitions).

All the above considerations are assuming that a single NetBSD OS is installed in a disk. They may change if two or multiple NetBSD systems are installed in the same disk, but this is a possible conflicting and confusing scenario, which I would discourage.

This is different on FreeBSD, where the MBR primary partitions are accessible for each disk: for example, /dev/ada0s1 refers to the first MBR primary partition of disk ada0 (s means slice, another way MBR primary partitions are called). In FreeBSD, /dev also has ada0s1a, ada0s1b, etc., if ada0s1 has a UFS filesystem). I don't know if FreeBSD also manages in a different way the coexistence of MBR and disklabel.

GPT is used for the same purpose as MBR and disklabel: it describes the structure of an entire disk. There's however a difference in the way NetBSD integrates with it: while MBR is an alien system for NetBSD, GPT is supported. So, for example, if a new disk is added to an existing NetBSD system, and this disk is partitioned with GPT, there's no need of a disklabel. For each new partition, GPT in NetBSD automatically creates a wedge (dk0, dk1, etc.). Wedges are the equivalent of a partition, not a disk, so a disklabel can not be placed on them (also, /dev has no dk0a, dk0b, etc., but only dk0, dk1, etc.). A new filesystem (FFSv2, for example) can be directly created on a wedge. I don't know if a disklabel is installed when GPT is used to partition a NetBSD system disk (the one hosting the root / directory), but I don't think so.

GPT is available to partition a system disk when performing a UEFI installation, but IIRC a traditional BIOS installation does not offer GPT.

As well, if a new disk is added to an existing NetBSD system (so, it is not a boot disk), it may be partitioned using only disklabel, not MBR or GPT. Avoiding in amd64 the reserved partitions c and d, the other letters may be arbitrary related to any data contents (with any mountpoint).

These observations would not have been possible without the help of the users in the #netbsd channel on Freenode IRC. I would like to thank them so much. If any of them wants to integrate these notes and/or correct something, is welcome.

Thank you again!

Rocky

The original message is from the @netbsd-users mailing list: I thought it could also be useful here.


October 14, 2020

UnitedBSD Requests for packages in pkgsrc/pkgin

Please list packages you miss in pkgsrc, so the developers and external contributors can add them on your demand.


October 13, 2020

NetBSD.org New Security Advisory: NetBSD-SA2020-003

October 10, 2020

DragonFly BSD Digest In Other BSDs for 2020/10/10

I should have set this to post at 10:10AM GMT.


October 09, 2020

NetBSD.org pkgsrc-2020Q3 released

October 07, 2020

NetBSD Blog The GNU GDB Debugger and NetBSD (Part 5)
The NetBSD developers maintain two copies of GDB:

The base-system version of GDB (GPLv3) still relies on local patching to work. I have set a goal to reduce the number of custom patches to bare minimum, ideally achieving the state of GDB working without any local modifications at all.

GDB changes

Last month, the NetBSD/amd64 support was merged into gdbserver. This month, the gdbserver target support was extended to NetBSD/i386 and NetBSD/aarch64. The gdbserver and gdb code was cleaned up, refactored and made capable of introducing even more NetBSD targets.

Meanwhile, the NetBSD/i386 build of GDB was fixed. The missing include of x86-bsd-nat.h as a common header was added to i386-bsd-nat.h. The i386 GDB code for BSD contained a runtime assert that verified whether the locally hardcoded struct sigcontext is compatible with the system headers. In reality, the system headers are no longer using this structure since 2003, after the switch to ucontext_t, and the validating code was no longer effective. After the switch to newer GCC, this was reported as a unused local variable by the compiler. I have decided to remove the check on NetBSD entirely. This was followed up by a small build fix.

The NetBSD team has noticed that the GDB's agent.cc code contains a portability bug and prepared a local fix. The traditional behavior of the BSD kernel is that passing random values of sun_len (part of sockaddr_un) can cause failures. In order to prevent the problems, the sockaddr_un structure is now zeroed before use. I've reimplemented the fix and successfully upstreamed it.

In order to easily resolve the issue with environment hardening enforced by PaX MPROTECT, I've introduced a runtime warning whenever byte transfers betweeen the debugee and debugger occur with the EACCES errno code.

binutils changes

I've added support for NetBSD/aarch64 upstream, in GNU BFD and GNU GAS. NetBSD still carries local patches for the GNU binutils components, and GNU ld does not build out of the box on NetBSD/aarch64.

Summary

The NetBSD support in GNU binutils and GDB is improving promptly, and the most popular platforms of amd64, i386 and aarch64 are getting proper support out of the box, without downstream patches. The remaining patches for these CPUs include: streamlining kgdb support, adding native GDB support for aarch64, upstreaming local modifications from the GNU binutils components (especially BFD and ld) and introducing portability enhancements in the dependent projects like libiberty and gnulib. Then, the remaining work is to streamline support for the remaining CPUs (Alpha, VAX, MIPS, HPPA, IA64, SH3, PPC, etc.), to develop the missing generic features (such as listing open file descriptors for the specified process) and to fix failures in the regression test-suite.


October 03, 2020

DragonFly BSD Digest In Other BSDs for 2020/10/03

No theme, but lots to read.


October 01, 2020

NetBSD.org New Developer in September 2020

September 30, 2020

OS News Wayland on NetBSD – trials and tribulations

Related to yesterday’s post about NetBSD switching to ctwm:

After I posted about the new default window manager in NetBSD I got a few questions, including “when is NetBSD switching from X11 to Wayland?”, Wayland being X11’s “new” rival. In this blog post, hopefully I can explain why we aren’t yet!

The short answer? Wayland is too Linux-specific to be easily ported or adapted to NetBSD, so don’t expect it any time soon.


September 29, 2020

Stack Overflow Trouble compiling ncurses-st-menu for BSD

I found a package on github (https://github.com/okbob/ncurses-st-menu) and am having trouble compile it for BSD platforms like NetBSD or OpenBSD. The instructions say to do ./autogen.sh, ./configure, and then make. So I install the autoconf, autotools, libtool, gettext, and any other necessary packages and run ./autogen.sh. It works without spitting out any errors. But ./configure says it doesn't support "OS x86_64-unknown-netbsd9.0" if for example on NetBSD. Can someone else try to compile this program? Because if this was done by autotools, it certainly should support any of the four major BSD operating systems.

Unix Stack Exchange Is NetBSD 'primes' utility or equivalent available in any package on MacOS?

Is the NetBSD primes utility (or equivalent) available on MacOS in any package, other than via manual download-and-compile (e.g. curl)? I searched quite a lot and couldn't find any package (other than the NetBSD CVS source).

(NetBSD primes is not a prime-sieve to find large/as-yet-unknown primes, just a simple command-line utility which tells you which integers are prime (or composite) in a given (64b) range).

(Unlike Gnu factor which is available via package coreutils "Finding Prime Numbers - “factor” command not found on MacOS", "Is there a practical use for the GNU factor command?")

Note: this question does not belong on AskDifferent since there is no brew/macports package.

Unix Stack Exchange How to use 'pkg_add -uu' to upgrade all packages?

According to NetBSD's wiki I can use pkg_add -uu to upgrade packages. However, when I attempt to use pkg_add -uu it results in an error.

pkg_add -uu
pkg_add: missing package name(s)
...

pkg_add -uu *
pkg_add: no pkg found for `*`, sorry
...

pkg_add -uu all
pkg_add: no pkg found for `all`, sorry
...

I've tried to parse the pkg_add man page but I can't tell what the command it to update everything.

I can't use pkg_chk because its not installed, and I can't get the package system to install it:

pkg_chk -b
pkg_chk: command not found

pkg_add pkg_chk
pkg_add: no pkg found for `pkg_chk`, sorry

What is the secret command to get the OS to update everything?

Unix Stack Exchange How to use resize_ffs in netbsd

I'm trying to use resize_ffs with NetBSD to increase the size of my partition. I have NetBSD running in a virtual machine, and have expanded the size of the disk, and now wish to grow the partition to match.

Here is the man page for the tool.

I am trying to grow a 300 MB partition to 1 GB.

The tool's man page says that specifying a size is not mandatory, and that if it is not specified it will grow to use available space (ideal behaviour). However this results in an error saying newsize not known.

I have used various online tools to try and calculate the disk geometry, but no matter what I try when I pass a number to -s, I get the error "bad magic number".

I have been unable to find example of using this tool online.

What is the correct way to use resize_ffs to grow a partition to use available disk space?


September 28, 2020

OS News Default window manager switched to CTWM in NetBSD-current

For more than 20 years, NetBSD has shipped X11 with the “classic” default window manager of twm. However, it’s been showing its age for a long time now.

In 2015, ctwm was imported, but after that no progress was made. ctwm is a fork of twm with some extra features – the primary advantages are that it’s still incredibly lightweight, but highly configurable, and has support for virtual desktops, as well as a NetBSD-compatible license and ongoing development. Thanks to its configuration options, we can provide a default experience that’s much more usable to people experienced with other operating systems.

The ctwm website has more information for those interested.

Ruben Schade OpenSSH 8.4 released

OpenSSH 8.4 was released yesterday. It includes several signifigant changes for FIDO/U2F authentication, some of which are listed as potentially-incompatible, but are still great to see. Other things that caught my eye:

scp(1), sftp(1): allow the -A flag to explicitly enable agentforwarding in scp and sftp. The default remains to not forward anagent, even when ssh_config enables it.

sshd(8): allow sshd_config longer than 256k

And I’m always pleased to see NetBSD portability notes:

sshd(8): support NetBSD’s utmpx.ut_ss address field. bz#960

This exquisitely-maintained software powers so much of the Internet. It got me thinking that for all my talk about donations, I should put my money where my mouth is and donate to the OpenBSD Foundation. Even if you’ve never heard of OpenSSH, you’ve also benefited from it.


This post originally appeared on Rubenerd.

NetBSD Blog Wayland on NetBSD - trials and tribulations

After I posted about the new default window manager in NetBSD I got a few questions, including "when is NetBSD switching from X11 to Wayland?", Wayland being X11's "new" rival. In this blog post, hopefully I can explain why we aren't yet!

Last year (and early this year) I was responsible for porting the first working Wayland compositor to NetBSD - swc. I chose it because it looked small and hackable. You can try it out by installing the velox window manager from pkgsrc.

A Wayland compositor running on my NetBSD laptop, with a few applications like Luakit and Dungeon Crawl Stone Soup open.

Difficulties

In a Wayland system, the "compositor" (display server) is responsible for managing displays, input, and window management. Generally, this means a lot of OS-specific code is contained there.

Wayland does not define protocols for features X11 users expect, like screenshots, screen locking, or window management. Either you implement these inside the compositor (lots of work that has to be redone), or you define your own protocol extension.

The Wayland "reference implementation" is a small set of libraries that can be used to build a compositor or a client application. These libraries currently have hard dependencies on Linux kernel APIs like epoll. In pkgsrc we've patched the libraries to add kqueue(2) support, but the patches haven't been accepted upstream. Wayland is written with the assumption of Linux to the extent that every client application tends to #include <linux/input.h> because Wayland's designers didn't see the need to define a OS-neutral way to get mouse button IDs.

So far, all Wayland compositors but swc have a hard dependency on libinput, which only supports Linux's input API (also cloned in FreeBSD). In NetBSD we have an entirely different input API - wscons(4). wscons is actually fairly simple to write code for, someone just needs to go out there and do it. You can use my code in swc as a reference. :)

In general, Wayland is moving away from the modularity, portability, and standardization of the X server.

Is it ready for production?

No, but you can play with it.

Task list

I've decided to take a break from this, since it's a fairly huge undertaking and uphill battle. Right now, X11 combined with a compositor like picom or xcompmgr is the more mature option.

NetBSD Blog Default window manager switched to CTWM in NetBSD-current

For more than 20 years, NetBSD has shipped X11 with the "classic" default window manager of twm. However, it's been showing its age for a long time now.

In 2015, ctwm was imported, but after that no progress was made. ctwm is a fork of twm with some extra features - the primary advantages are that it's still incredibly lightweight, but highly configurable, and has support for virtual desktops, as well as a NetBSD-compatible license and ongoing development. Thanks to its configuration options, we can provide a default experience that's much more usable to people experienced with other operating systems.

Recently, I've been installing NetBSD with some people in real life and was inspired by their reactions to the default twm to improve the situation, so I played with ctwm, wrote a config, and used it myself for a week. It's now the default in NetBSD-current.

We gain some nice features like an auto-generated application menu (that will fill up as packages are installed to /usr/pkg), and a range of useful keyboard shortcuts including volume controls - the default config should be fully usable without a mouse. It should also work at a range of screen resolutions. We can add HiDPI support after some larger bitmap fonts are imported - another advantage of ctwm is that we can support very slow and very fast hardware with one config.

If you're curious about ctwm, check out the ctwm website. It's also included in previous NetBSD releases, though not as the default window manager and not with this config.


September 21, 2020

NetBSD Package System (pkgsrc) on DaemonForums samba problem
hi guys ,i need some favour,i install and config samba follow "https://wiki.netbsd.org/tutorials/how_to_set_up_a_samba_server/",
it show " protocol negotiation failed: NT_STATUS_IO_TIMEOUT" when i run "smbclient -Usamba -L localhost" i dont konw what i miss?thanks

and this my step and config file
pkgin install samba
cp /usr/pkg/share/examples/rc.d/smbd /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/nmbd /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/samba /etc/rc.d/
and in /etc/rc.conf
smbd=YES
nmbd=YES
samba=YES
follow is my /usr/pkg/etc/samba/smb.conf
Code:

# smbclient -L localhost
protocol negotiation failed: NT_STATUS_IO_TIMEOUT
# vim /usr/pkg/etc/samba/smb.conf
# cat  /usr/pkg/etc/samba/smb.conf
# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options (perhaps too
# many!) most of which are not shown in this example
#
# For a step to step guide on installing, configuring and using samba,
# read the Samba-HOWTO-Collection. This may be obtained from:
#  http://www.samba.org/samba/docs/Samba-HOWTO-Collection.pdf
#
# Many working examples of smb.conf files can be found in the
# Samba-Guide which is generated daily and can be downloaded from:
#  http://www.samba.org/samba/docs/Samba-Guide.pdf
#
# Any line which starts with a ; (semi-colon) or a # (hash)
# is a comment and is ignored. In this example we will use a #
# for commentry and a ; for parts of the config file that you
# may wish to enable
#
# NOTE: Whenever you modify this file you should run the command "testparm"
# to check that you have not made any basic syntactic errors.
#
#======================= Global Settings =====================================
[global]

# workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
  workgroup = WORKGROUP

# server string is the equivalent of the NT Description field
  server string = Samba Server

# Server role. Defines in which mode Samba will operate. Possible
# values are "standalone server", "member server", "classic primary
# domain controller", "classic backup domain controller", "active
# directory domain controller".
#
# Most people will want "standalone server" or "member server".
# Running as "active directory domain controller" will require first
# running "samba-tool domain provision" to wipe databases and create a
# new domain.
#  server role = standalone server

# This option is important for security. It allows you to restrict
# connections to machines which are on your local network. The
# following example restricts access to two C class networks and
# the "loopback" interface. For more examples of the syntax see
# the smb.conf man page
  hosts allow = 192.168.31.

# Uncomment this if you want a guest account, you must add this to /etc/passwd
# otherwise the user "nobody" is used
;  guest account = pcguest

# this tells Samba to use a separate log file for each machine
# that connects
  log file = /var/log/log.%m

# Put a capping on the size of the log files (in Kb).
  max log size = 50

# Specifies the Kerberos or Active Directory realm the host is part of
;  realm = MY_REALM

# Backend to store user information in. New installations should
# use either tdbsam or ldapsam. smbpasswd is available for backwards
# compatibility. tdbsam requires no further configuration.
;  passdb backend = tdbsam

# Using the following line enables you to customise your configuration
# on a per machine basis. The %m gets replaced with the netbios name
# of the machine that is connecting.
# Note: Consider carefully the location in the configuration file of
#      this line.  The included file is read at that point.
;  include = /usr/local/samba/lib/smb.conf.%m

# Configure Samba to use multiple interfaces
# If you have multiple network interfaces then you must list them
# here. See the man page for details.
;  interfaces = 192.168.12.2/24 192.168.13.2/24

# Where to store roving profiles (only for Win95 and WinNT)
#        %L substitutes for this servers netbios name, %U is username
#        You must uncomment the [Profiles] share below
;  logon path = \\%L\Profiles\%U

# Windows Internet Name Serving Support Section:
# WINS Support - Tells the NMBD component of Samba to enable it's WINS Server
;  wins support = yes

# WINS Server - Tells the NMBD components of Samba to be a WINS Client
#      Note: Samba can be either a WINS Server, or a WINS Client, but NOT both
;  wins server = w.x.y.z

# WINS Proxy - Tells Samba to answer name resolution queries on
# behalf of a non WINS capable client, for this to work there must be
# at least one  WINS Server on the network. The default is NO.
;  wins proxy = yes

# DNS Proxy - tells Samba whether or not to try to resolve NetBIOS names
# via DNS nslookups. The default is NO.
  dns proxy = no

# These scripts are used on a domain controller or stand-alone
# machine to add or delete corresponding unix accounts
;  add user script = /usr/sbin/useradd %u
;  add group script = /usr/sbin/groupadd %g
;  add machine script = /usr/sbin/adduser -n -g machines -c Machine -d /dev/null -s /bin/false %u
;  delete user script = /usr/sbin/userdel %u
;  delete user from group script = /usr/sbin/deluser %u %g
;  delete group script = /usr/sbin/groupdel %g


#============================ Share Definitions ==============================
[homes]
  comment = Home Directories
  browseable = no
  writable = yes

[shared]
comment = Shared
path = /home/zero/work
browseable = yes
writable = yes
guest ok = yes
# Un-comment the following and create the netlogon directory for Domain Logons
; [netlogon]
;  comment = Network Logon Service
;  path = /usr/local/samba/lib/netlogon
;  guest ok = yes
;  writable = no
;  share modes = no


# Un-comment the following to provide a specific roving profile share
# the default is to use the user's home directory
;[Profiles]
;    path = /usr/local/samba/profiles
;    browseable = no
;    guest ok = yes


# NOTE: If you have a BSD-style print system there is no need to
# specifically define each individual printer
[printers]
  comment = All Printers
  path = /usr/spool/samba
  browseable = no
# Set public = yes to allow user 'guest account' to print
  guest ok = no
  writable = yes
  printable = yes

# This one is useful for people to share files
;[tmp]
;  comment = Temporary file space
;  path = /tmp
;  read only = no
;  public = yes

# A publicly accessible directory, but read only, except for people in
# the "staff" group
;[public]
;  comment = Public Stuff
;  path = /home/samba
;  public = yes
;  writable = no
;  printable = no
;  write list = @staff

# Other examples.
#
# A private printer, usable only by fred. Spool data will be placed in fred's
# home directory. Note that fred must have write access to the spool directory,
# wherever it is.
;[fredsprn]
;  comment = Fred's Printer
;  valid users = fred
;  path = /homes/fred
;  printer = freds_printer
;  public = no
;  writable = no
;  printable = yes

# A private directory, usable only by fred. Note that fred requires write
# access to the directory.
;[fredsdir]
;  comment = Fred's Service
;  path = /usr/somewhere/private
;  valid users = fred
;  public = no
;  writable = yes
;  printable = no

# a service which has a different directory for each machine that connects
# this allows you to tailor configurations to incoming machines. You could
# also use the %U option to tailor it by user name.
# The %m gets replaced with the machine name that is connecting.
;[pchome]
;  comment = PC Directories
;  path = /usr/pc/%m
;  public = no
;  writable = yes

# A publicly accessible directory, read/write to all users. Note that all files
# created in the directory by users will be owned by the default user, so
# any user with access can delete any other user's files. Obviously this
# directory must be writable by the default user. Another user could of course
# be specified, in which case all files would be owned by that user instead.
;[public]
;  path = /usr/somewhere/else/public
;  public = yes
;  only guest = yes
;  writable = yes
;  printable = no

# The following two entries demonstrate how to share a directory so that two
# users can place files there that will be owned by the specific users. In this
# setup, the directory should be writable by both users and should have the
# sticky bit set on it to prevent abuse. Obviously this could be extended to
# as many users as required.
;[myshare]
;  comment = Mary's and Fred's stuff
;  path = /usr/somewhere/shared
;  valid users = mary fred
;  public = no
;  writable = yes
;  printable = no
;  create mask = 0765


September 19, 2020

NetBSD General on DaemonForums New Shared Lib on i386
Hi,

NetBSD i386 9.0 (GENERIC) #0: Fri Feb 14 00:06:28 UTC 2020

I have my own personal shared library, which complies and installs fine. When I compile programs that link against that library, the compile succeeds.

But when I attempt to run one of the programs I get:

Quote:

/usr/local/lib/libj_lib.so: text relocations
/usr/local/lib/libj_lib.so: Cannot write-enable text segment: Permission denied
I even tried a reboot without luck.

I did many searches but all I found was issues compiling mplayer and to use ld.so.conf, but nothing else. I read ld.so.conf is no longer needed and man pages has nothing I can find.

Does anyone know what I can do to fix this issue ?

edit: no luck with "-Wl,-R/usr/local/lib" as noted in the elf FAQ

Thanks
John
DragonFly BSD Digest In Other BSDs for 2020/09/19

No theme, just everything in a bucket.


September 17, 2020

NetBSD General on DaemonForums Howto power off usb devices
Under FreeBSD, I can power off a usb device with usbconfig's "power_off" option. How can I do that under NetBSD?

Code:

# usbstats
Controller /dev/usb0:
      1069 control
        0 isochronous
 154806725 bulk
  8954336 interrupt
Controller /dev/usb1:
      375 control
        0 isochronous
  33390421 bulk
        2 interrupt

# usbdevs
addr 1: EHCI root hub, NetBSD
 addr 2: Rate Matching Hub, Intel
  addr 5: External USB 3.0, Toshiba
  addr 3: USB Optical Mouse, Primax Electronics
  addr 4: Kensington U+P Keyboard, NOVATEK
addr 1: EHCI root hub, NetBSD
 addr 2: Rate Matching Hub, Intel
  addr 3: USB2.0-CRW, Generic

I want to power off (and later on) External USB 3.0, Toshiba, which I believe is at usb0, addr 5.

September 15, 2020

Server Fault Non-standard IP address with dashes

I ran the who command on a shared NetBSD box, and this weird user IP came up:

<redacted> pts/33   May 13 02:13  (XXX.XXX.XXX.XXX)
<redacted> pts/35   May 12 20:59  (202-172-110-147-)
<redacted> pts/36   May  6 20:36  (XXX.XXX.XXX.XXX)

I've never seen an IP like that. Obviously, ping 202-172-110-147- will complain with "Cannot resolve ... (Unknown host)".

There was a similar question posted 7 years ago, which posited that it was a non-standard way of denoting IP ranges, but seeing there's a - at the end of the address, it doesn't seem like a similar thing.


Edit:

I've tried reverse DNS with nslookup 202-172-110-147-, which errors with "** server can't find 202-172-110-147-: NXDOMAIN"

Doing w <user> returns:

9:49AM  up 89 days,  7:46, 1 user, load averages: 0.23, 0.18, 0.17
USER          TTY     FROM              [email protected]  IDLE WHAT
<redacted>    pts/35  202-172-110-147- Tue08PM  4:13

Edit 2: This is on NetBSD, not Linux like I mentioned at the beginning (I thought the box was Linux):

$ uname -rsv
NetBSD 8.1 NetBSD 8.1 (GENERIC) #0: Fri May 31 08:43:59 UTC 2019  [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC

Edit 3: Following @NStorm update, I ran w -n to display network addresses as numbers. I still see the same result

$ w -n <user>
 9:57AM  up 89 days,  7:54, 1 user, load averages: 0.12, 0.12, 0.14
USER          TTY     FROM              [email protected]  IDLE WHAT
<redacted>    pts/35  202-172-110-147- Tue08PM  4:22

September 14, 2020

Ruben Schade Feedback on my Mr Orange post

Good morning! My post about Mr Orange generated the most email feedback I’ve had since my encrypted-ZFS on NetBSD post, all of it negative. I can only assume someone shared it among Orange supporters.

I’d publish each message in full, but they’re all just a little too tragic. One of the more civil gentleman—though we’re coming from a low baseline here—attempted to debunk everything I said, but the substance of each quote was that:

It spoke to the internal machinations and mental gymnastics of someone desperate to absolve their leader of culpability. All the facts cited were wrong, and only served to reinforce my original thought that his supporters have absolutely no idea how the rest of the world perceives them. This kind of transparent projection and self-ownage isn’t unique to their flavour of politics, but it does attract a disproportionate amount of it.

I do feel a responsibility to reach these people. Each Orange supporter has their reasons, and they have the same family, job security and other worries we all do. Perhaps even moreso, which is why they’re so motivated to accept any convenient excuse for why the man they support continues to brazenly abuse them. But when you’re coming from a place of bad faith to start, I have no motivation to engage with you.

And that’s the problem. We could have a reasonable discussion if we were coming from a place of honesty. When the Dear Leader regularly lies, and people like Scott Adams say it’s a political tactic, we have no basis upon which to even start. Facts are sacrosanct, just as so many far-right people ironically state on their Twitter profiles. He didn’t have the biggest turnout at his inauguration. He didn’t get Mexico to pay for his wall. America isn’t doing the best at dealing with COVID. Wind turbines don’t cause cancer. He didn’t get Kim Jong Un to do anything. We can debate the finer points of his other policies, but if people can’t even admit to these, they’re being just as dishonest as him.

(I also see this playing out to a lesser extent with Scott Morrison in Australia, who has made no secret of his admiration for the American knob-in-chief).

Thanks for taking the time to respond everyone, but your tactics to win over another supporter in your “culture war” have backfired. And unlike so many of you who still cling to him because he’s “owning lefties”, I take no pleasure whatsoever in pointing that out. The sooner they realise they’re transparently doing the work of our common enemies better than anyone else, the sooner we can begin to rebuild what we’ve all lost. Because being an outspoken friend of America has been hard these last few years.


This post originally appeared on Rubenerd.

Unix Stack Exchange Modifying firewall rules with pfctl on NetBSD 4

I'm wanting to modify some firewall/NAT rules on a device (Apple Airport) running NetBSD 4.0. I'm not that familiar with BSD and pf, so want to check the right approach. I can change the pf.conf file (using sed) to the required configuration, then load using pfctl -f /etc/pf.conf, but am wondering if this clears out the old rules first?

Alternately, what would be the best approach to doing this directly via the command line (i.e. removing old the rdr pass and nat rules, and adding new ones?)

(for context, the aim here is to be able to change NAT rules on an Apple Airport Extreme without having to restart the device, which brings the whole network down for a minute or so; I've successfully gained ssh access)


September 13, 2020

Ruben Schade Forgetting to set UTF normalisation on a ZFS pool

Clara and I were getting some bizarre behavior while accessing a new FreeBSD pool over Netatalk and Samba. A subset of files with CJK names were showing up in the macOS Finder as expected, but would error out with file not found if you tried to open them.

We store a lot of files in Japanese and Korean, especially music and holiday photo directories with place names, so I’ve always been careful about using UTF-8 globally. I confirmed I had this in my /etc/login.conf:

default:\[...]:charset=UTF-8:\:lang=en_US.UTF-8:

(NOTE: I’ve read this isn’t advisable because it can break ports that weren’t designed for UTF-8. I’ve never had that issue, but it’s something to keep in mind. I’d also be worried if software in 2020 still had that limitation, but that’s a topic for another post).

Then I confirmed the ZFS pool was set up for UTF-8:

# zfs get utf8only pool==> NAME  PROPERTY  VALUE  SOURCE==> zten  utf8only  on     -

So what was going on?

# zfs get normalization pool==> NAME  PROPERTY       VALUE  SOURCE==> zten  normalization  none   -

Whoops!

Normalisation is a field of information science that fills entire textbooks, but in a nutshell ZFS uses it to reconcile filenames. This is especially important in UTF-8 where characters from disparate languages may appear superficially the same, such as Chinese-derived kanji in Japanese. How the filename is represented internally, and presented to the operator, can vary in unexpected ways.

Unfortunately, normalisation can’t be set after the filesystem is created. So this weekend I dropped one of the drives from my mirror, created a new pool with normalisation to transfer data back to, then resilvered the mirror back to full redundancy:

# zpool -O normalization=formD [...]

Now previously-inaccessible files can be opened.


This post originally appeared on Rubenerd.


September 08, 2020

Benny Siegert pkgsrc Developer Monotony
Somehow, my contributions to NetBSD and pkgsrc have become monotonous. Because I am busy with work, family and real life, the amount of time I can spend on open source is fairly limited, and I have two commitments that I try to fulfill: Member of pkgsrc-releng: I process most of the pull-ups to the stable quarterly branch. Maintainer of Go and its infrastructure. Unfortunately, these things are always kinda the same.

September 07, 2020

Frederic Cambus Playing with Kore JSON API

Kore 4.0.0 has been released a few days ago, and features a brand new JSON API allowing to easily parse and serialize JSON objects.

During the last couple of years, I have been using Kore for various projects, including exposing hardware sensor values over the network via very simple APIs. In this article, I would like to present a generalization of this concept and show how easy it is to expose system information with Kore.

This small API example allows to identify hosts over the network and has been tested on Linux, OpenBSD, NetBSD, and macOS (thanks Joris!).

After creating a new project:

kodev create identify

Populate src/identify.c with the following code snippet:

#include <sys/utsname.h>

#include <kore/kore.h>
#include <kore/http.h>

#if defined(__linux__)
#include <kore/seccomp.h>

KORE_SECCOMP_FILTER("json",
	KORE_SYSCALL_ALLOW(uname)
);
#endif

int		page(struct http_request *);

int
page(struct http_request *req)
{
	char *answer;

	struct utsname u;

	struct kore_buf buf;
	struct kore_json_item *json;

	if (uname(&u) == -1) {
		http_response(req, HTTP_STATUS_INTERNAL_ERROR, NULL, 0);
		return (KORE_RESULT_OK);
	}

	kore_buf_init(&buf, 1024);
	json = kore_json_create_object(NULL, NULL);

	kore_json_create_string(json, "system", u.sysname);
	kore_json_create_string(json, "hostname", u.nodename);
	kore_json_create_string(json, "release", u.release);
	kore_json_create_string(json, "version", u.version);
	kore_json_create_string(json, "machine", u.machine);

	kore_json_item_tobuf(json, &buf);

	answer = kore_buf_stringify(&buf, NULL);
	http_response(req, 200, answer, strlen(answer));

	kore_buf_cleanup(&buf);
	kore_json_item_free(json);

	return (KORE_RESULT_OK);
}

And finally launch the project:

kodev run

The kodev tool will build and run the project, and we can now query the API to identify hosts:

{
  "system": "OpenBSD",
  "hostname": "foo.my.domain",
  "release": "6.8",
  "version": "GENERIC.MP#56",
  "machine": "amd64"
}

September 03, 2020

Ruben Schade The roles of OSs have changed

I had an overdue realisation last night:

All of this is such a shift from before. Macs used to be seen as creative tools, not something you’d use for business. DOS and 1990s Windows were endlessly frustrating, not sources of fun. Linux couldn’t have been broadly used for games.

I can still remember in the early 2000s bringing my iBook G3 into a corporate office for school work experience, and all the ensuing discussion about what it was, and whether it could be supported by their corporate LAN. They were even more confused when I booted it into NetBSD. How things change.

Though I do wonder sometimes, with just a slight tweak to history, how things might have been different. In another dimension somewhere, I’m using the latest BeOS-powered PowerPC laptop, and a shiny new Palm smartphone. Both of these represented the pinnacle of UI design in the 1990s, and still in the 2020s have yet to be surpassed. People call me an Apple fanboy, but I’d drop all of it in a second for that gear.


This post originally appeared on Rubenerd.


September 02, 2020

NetBSD General on DaemonForums restart network
I ran the command
Code:

# ifconfig mue0 down
to take the computer offline, but how do I now bring it back up?
When I try
Code:

# ifconfig mue0 up
I get
Code:

exec_matches: No buffer space available
error

What is the NetBSD equivalent of the FreeBSD command
Code:

# /etc/rc.d/netif restart
or OpenBSD equivalent
Code:

# /etc/netstart
I want to be able to restart all network services

August 31, 2020

Frederic Cambus Modernizing the OpenBSD console

At the beginning were text mode consoles. Traditionally, *BSD and Linux on i386 and amd64 used text mode consoles which by default provided 25 rows of 80 columns, the "80x25 mode". This mode uses a 8x16 font stored in the VGA BIOS (which can be slightly different across vendors).

OpenBSD uses the wscons(4) console framework, inherited from NetBSD.

CRT monitors allowed to set the resolution you wanted, so on bigger monitors a 80x25 console in textmode was fairly large but not blurry.

Framebuffer consoles allowed taking advantage of larger monitor sizes, to fit more columns and row. With the switch to LCD monitors, also in part driven by the decreasing costs of laptops, the fixed size panels became a problem as the text mode resolution needed to be stretched, leading to distortion and blurriness.

One thing some people might not realize, is the huge discrepancy between text mode and framebuffer consoles regarding the amount of data you have to write to cover the whole screen. In text mode, we only need to write 2 bytes per character: 1 byte for the ASCII code, and 1 byte for attributes. So in 80x25 text mode, we only need to write 80 * 25 * 2 bytes of data, which is 4000 bytes, and the VGA card itself takes care of plotting characters to the screen. In framebuffer however, to fill a 4K UHD-1 (3840x2160) screen in 32bpp mode we need to send 3840 * 2160 * 4 bytes of data, which is 33177600 bytes (approximately 33 MB).

On framebuffer consoles, OpenBSD uses the rasops(9) subsystem (raster operations), imported from NetBSD in 2001.

While they had been used for a while on platforms without VGA cards, framebuffer consoles were only enabled on i386 and amd64 in 2013 for inteldrm(4) and radeondrm(4).

In recent years, rasops(9) itself and framebuffer drivers have seen some improvements:

General improvements:

Performance related improvements:

Console fonts improvements:

There is an article about Spleen in the OpenBSD Journal with more information, notably on the font selection mechanism relative to screen resolution.

And work slowly continues to make framebuffer consoles more usable.

It is interesting to note that while NetBSD has been adding a lot of features to rasops(9) over the years, OpenBSD has taken a more conservative approach. There is however one major feature that NetBSD currently has which would be beneficial: the capability for loading fonts of different metrics and subsequently resizing screens.

Looking forward, performance of various operations could likely still be improved, possibly by leveraging the new OpenBSD dynamic tracing mechanism to analyze bottlenecks.

Another open question is UTF-8 support, Miod Vallat started work in this direction back in 2013 but there are still a few things missing. I have plans to implement sparse font files support in the future, at least so one can take advantage of box drawing and possibly block elements characters.

Lastly, a major pain point has been the lack of larger fonts in RAMDISK kernels, making installations and upgrades very difficult and error-prone on large DPI monitors as the text is basically unreadable. There is no technical blocker to make this happen, which ironically makes it the most difficult kind of issue to tackle.


August 29, 2020

Nikita Gillmann GSoC 2020 Final Report - Report part 2

This report was prepared by Nikita Gillmann as a part of Google Summer of Code 2020

This is my second and final report for the Google Summer of Code project I am working on for NetBSD.

My code can be found at github.com/teknokatze/src in the gsoc2020 branch, at the time of writing some of it is still missing. The test facilities and logs can be found in github.com/teknokatze/gsoc2020. A diff can be found at github which will later be split into several patches before it is sent to QA for merging.

The initial and defined goal of this project was to make system(3) and popen(3) use posix_spawn(3) internally, which had been completed in June. For the second part I was given the task to replace fork+exec calls in our standard shell (sh) in one scenario. Similar to the previous goal we determine through implementation if the initial motivation, to get performance improvements, is correct otherwise we collect metrics for why posix_spawn() in this case should be avoided. This second part meant in practice that I had to add and change code in the kernel, add a new public libc function, and understand shell internals.

Summary of part 1

Prior work: In GSoC 2012 Charles Zhang added the posix_spawn syscall which according to its SF repository at the time (maybe even now, I have not looked very much into comparing all other systems and libcs + kernels) is an in-kernel implementation of posix_spawn which provides performance benefits compared to FreeBSD and other systems which had a userspace implementation (in 2012).

After 1 week of reading POSIX and writing code, 2 weeks of coding and another 1.5 weeks of bugfixes I have successfully implemented posix_spawn in usage in system(3) and popen(3) internally.

The biggest challenge for me was to understand POSIX, to read the standard. I am used to reading more formal books, but I can't remember working with the posix standard directly before.

system(3)

system(3) was changed to use posix_spawnattr_ (where we used sigaction before) and posix_spawn (which replaced execve + vfork calls).

popen(3) and popenve(3)

Since the popen and popenve implementation in NetBSD's libc use a couple of shared helper functions, I was able to change both functions while keeping the majority of the changes focused on (some of ) the helper functions (pdes_child).

pdes_child, an internal function in popen.c, now takes one more argument (const char *cmd) for the command to pass to posix_spawn which is called in pdes_child.

On a high level what happens in pdes_child() and popen is that we first lock the pidlist_mutex. Then we create a file file action list for all concurrent popen() / popenve() instances and the side of the pipe not necessary, and the move to stdin/stdout. We unlock the pidlist_mutex. Finally we return the list and destroy.

In the new version of this helper function which now handles the majority of what popen/popenve did, we have to initialize a file_actions object which by default contains no file actions for posix_spawn() to perform. Since we have to have error handling and a common return value for the functions calling pdes_child() and deconstruction, we make use of goto in some parts of this function.

The close() and dup2() actions now get replaced by corresponding file_actions syscalls, they are used to specify a series of actions to be performed by a posix_spawn operation.

After this series of actions, we call _readlockenv(), and call posix_spawn with the file_action object and the other arguments to be executed. If it succeeds, we return the pid of the child to popen, otherwise we return -1, in both cases we destroy the file_action object before we proceed.

In popen and popenve our code has been reduced to just the 'pid == -1' branch, everything else happens in pdes_child() now.

After readlockenv we call pdes_child and pass it the command to execute in the posix_spawn'd child process; if pdes_child returns -1 we run the old error handling code. Likewise for popenve.

The outcome of the first part is, that thanks to how we implement posix_spawn in NetBSD we reduced the syscalls being made for popen and system. A full test with proper timing should indicate this, my reading was based on comparing old and new logs with ktrace and kdump.

sh, posix_spawn actions, libc and kernel - Part 2

Motivation

The main goal of part 2 of this project was to change sh(1) to determine which simple cases of (v)fork + exec I could replace, and to replace them with posix_spawn where it makes sense.

fork needs to create a new address space by cloning the address space, or in the case of vfork update at least some reference counts. posix_spawn can avoid most of this as it creates the new address space from scratch.

Issues

The current posix_spawn as defined in POSIX has no good way to do tcsetpgrp, and we found that fish just avoids posix_spawn for foreground processes.

Implementation

Since, roughly speaking, modern BSDs handle "#!" execution in the kernel (probably since before the 1990s, systems which didn't handle this started to disappear most likely in the mid to late 90s), our main concern so far was in the evalcmd function the default cmd switch case ('NORMALCMD').

After adjusting the function to use posix_spawn, I hit an issue in the execution of the curses application htop where htop would run but input would not be accepted properly (keysequences pressed are visible). In pre-posix_spawn sh, every subprocess that sh (v)forked runs forkchild() to set up the subprocess's environment. With posix_spawn, we need to arrange posix_spawn actions to do the same thing.

The intermediate resolution was to switch FORK_FG processes to fork+exec again. For foreground processes with job control we're in an interactive shell, so the performance benefit is small enough in this case to be negligible. It's really only for shell scripts that it matters.

Next I implemented a posix_spawn file_action, with the prototype

int posix_spawn_file_actions_addtcsetpgrp(posix_spawn_file_actions_t *fa, int fildes)

The kernel part of this was implemented inline in sys/kern/kern_exec.c, in the function handle_posix_spawn_file_actions() for the new case 'FAE_TCSETPGRP'.

The new version of the code is still in testing and debugging phase and at the time of writing not included in my repository (it will be published after Google Summer of Code when I'm done moving).

Future steps

posix_spawnp kernel implementation

According to a conversation with [email protected], the posix_spawnp() implementation we have is just itterating over $PATH calling posix_spawn until it succeeds. For some changes we might want a kernel implementation of posix_spawnp(), as the path search is supposed to happen in the kernel so the file actions are only ever run once:

some of the file actions may be "execute once only",
they can't be repeated (eg: handling "set -C; cat foo >file" - file
can only be created once, that has to happen before the exec (as the fd
needs to be made stdout), and then the exec part of posix_spawn is
attempted - if that fails, when it can't find "cat" in $HOME/bin (or
whatever is first in $PATH) and we move along to the next entry (maybe /bin
doesn't really matter) then the repeated file action fails, as file now
exists, and "set -C" demands that we cannot open an already existing file
(noclobber mode).   It would be nice for this if there were "clean up on
failure" actions, but that is likely to be very difficult to get right,
and each would need to be attached to a file action, so only those which
had been performed would result in cleanup attempts.

Replacing all of fork+exec in sh

Ideally we could replace all of (v)fork + exec with posix_spawn. According to my mentors there is pmap synchronisation as an impact of constructing the vm space from scratch with (v)fork. Less IPIs (inter-processor interrupts) matter for small processes too.

posix_spawn_file_action_ioctl

Future directions could involve a posix_spawn action for an arbitrary ioctl.

Thanks

My thanks go to fellow NetBSD developers for answering questions, most recently [email protected] for sharing invaluable sh knowledge, Riastradh and Jörg as the mentors I've interacted with most of the time and for their often in-depth explanations as well as allowing me to ask questions I sometimes felt were too obvious. My friends, for sticking up with my "weird" working schedule. Lastly would like to thank the Google Summer of Code program for continuing through the ongoing pandemic and giving students the chance to work on projects full-time.


August 26, 2020

Julio Merino pkgdb belongs in libdata, not var
Right after discussing where rc.d should live, it’s time to tackle a different but related pet peeve of mine: the location of the installed packages database. For this, I’m going to focus on the system I know best, pkgsrc, which keeps its database under /var/db/pkg/ by default. I think this location is wrong and the database should move to /usr/pkg/libdata/pkgdb/. From a cursory look, it seems that FreeBSD’s and OpenBSD’s ports databases, as well as dpkg’s and rpm’s, are also affected by this “problem”—but I do not know enough about their internals to say with certainty.

August 24, 2020

Julio Merino rc.d belongs in libexec, not etc
Let’s open with the controversy: the scripts that live under /etc/rc.d/ in FreeBSD, NetBSD, and OpenBSD are in the wrong place. They all should live in /libexec/rc.d/ because they are code, not configuration. This misplacement is something that has bugged me for ages but I never had the energy to open this can of worms back when I was very involved in NetBSD. I suspect it would have been a draining discussion and a very difficult thing to change.

August 14, 2020

NetBSD Package System (pkgsrc) on DaemonForums Samba : sh: /usr/bin/lex: not found
Hi everybody

I'm trying to install samba, like that :
Code:

[email protected] /usr/pkgsrc/net/samba # make install clean
But I've got this message :
Code:

sh: /usr/bin/lex: not found
ERROR: This package has set PKG_FAIL_REASON:
ERROR: samba-3.6.25nb23 requires a working dlopen().
*** Error code 1

Does anyone can help me ?
I found nothing.

Thank You
Guillaume ( NetBsd 9 amd64 Zsh )

August 09, 2020

Emile Heitor Make Postfix Trigger Blacklistd on Failed Authentication
The other day, I realized that from time to time, alpine, my console mail client for about 20+ years now, would close the connection to the IMAP server because of an “error”. Digging in the logs, I realized my server was being bruteforced for months, if not years. NetBSD being the fantastic OS it is, it actually had nearly no effect on my server’s behaviour, only those annoying connections closing from time to time.

August 06, 2020

Frederic Cambus NetBSD on the NanoPi NEO2

The NanoPi NEO2 from FriendlyARM has been serving me well since 2018, being my test machine for OpenBSD/arm64 related things.

As NetBSD/evbarm finally gained support for AArch64 in NetBSD 9.0, released back in February, I decided to give it a try on this device. The board only has 512MB of RAM, and this is where NetBSD really shines. Things have become a lot easier since [email protected] now provides bootable ARM images for a variety of devices, including the NanoPi NEO2.

On first boot, the system will resize the filesystem to automatically expand to the size of the SD card.

Growing ld0 MBR partition #1 (1052MB -> 60810MB)
Growing ld0 disklabel (1148MB -> 60906MB)
Resizing /
/dev/rld0a: grow cg |************************************                 |  69%

Once the system is up and running, we can add a regular user in the wheel group:

useradd -m -G wheel username

And add a password to the newly created user:

passwd username

From there we do not need the serial console anymore and can connect to the device using SSH.

NetBSD has binary packages available for this architecture, and installing and configuring pkgin can be done as follow:

export PKG_PATH=https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/aarch64/9.0/All/
pkg_add pkgin
echo $PKG_PATH > /usr/pkg/etc/pkgin/repositories.conf
pkgin update

The base system can be kept up to date using sysupgrade, which can be installed via pkgin:

pkgin in sysupgrade

The following variable need to be set in /usr/pkg/etc/sysupgrade.conf:

RELEASEDIR="https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/evbarm-aarch64"

Lastly, the device has two user controllable LEDs which can be toggled on and off using sysctl.

To switch both LEDs on:

sysctl -w hw.led.nanopi_green_pwr=1
sysctl -w hw.led.nanopi_blue_status=1

To switch off the power LED automatically at boot time:

echo "hw.led.nanopi_green_pwr=0" >> /etc/sysctl.conf

Here is a dmesg for reference purposes:

[     1.000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
[     1.000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
[     1.000000]     2018, 2019, 2020 The NetBSD Foundation, Inc.  All rights reserved.
[     1.000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
[     1.000000]     The Regents of the University of California.  All rights reserved.

[     1.000000] NetBSD 9.0_STABLE (GENERIC64) #0: Wed Aug  5 15:20:21 UTC 2020
[     1.000000] 	[email protected]:/usr/src/sys/arch/evbarm/compile/GENERIC64
[     1.000000] total memory = 497 MB
[     1.000000] avail memory = 479 MB
[     1.000000] timecounter: Timecounters tick every 10.000 msec
[     1.000000] armfdt0 (root)
[     1.000000] simplebus0 at armfdt0: FriendlyARM NanoPi NEO 2
[     1.000000] simplebus1 at simplebus0
[     1.000000] simplebus2 at simplebus0
[     1.000000] cpus0 at simplebus0
[     1.000000] simplebus3 at simplebus0
[     1.000000] psci0 at simplebus0: PSCI 1.1
[     1.000000] cpu0 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
[     1.000000] cpu0: package 0, core 0, smt 0
[     1.000000] cpu0: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
[     1.000000] cpu0: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
[     1.000000] cpu0: Dcache line 64, Icache line 64
[     1.000000] cpu0: L1 32KB/64B 2-way read-allocate VIPT Instruction cache
[     1.000000] cpu0: L1 32KB/64B 4-way write-back read-allocate write-allocate PIPT Data cache
[     1.000000] cpu0: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT Unified cache
[     1.000000] cpu0: revID=0x180, PMCv3, 4k table, 64k table, 16bit ASID
[     1.000000] cpu0: auxID=0x11120, FP, CRC32, SHA1, SHA256, AES+PMULL, NEON, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add
[     1.000000] cpu1 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
[     1.000000] cpu1: package 0, core 1, smt 0
[     1.000000] cpu2 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
[     1.000000] cpu2: package 0, core 2, smt 0
[     1.000000] cpu3 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
[     1.000000] cpu3: package 0, core 3, smt 0
[     1.000000] gic0 at simplebus1: GIC
[     1.000000] armgic0 at gic0: Generic Interrupt Controller, 224 sources (215 valid)
[     1.000000] armgic0: 16 Priorities, 192 SPIs, 7 PPIs, 16 SGIs
[     1.000000] fclock0 at simplebus2: 24000000 Hz fixed clock (osc24M)
[     1.000000] sunxisramc0 at simplebus1: SRAM Controller
[     1.000000] fclock1 at simplebus2: 32768 Hz fixed clock (ext_osc32k)
[     1.000000] gtmr0 at simplebus0: Generic Timer
[     1.000000] gtmr0: interrupting on GIC irq 27
[     1.000000] armgtmr0 at gtmr0: Generic Timer (24000 kHz, virtual)
[     1.000000] timecounter: Timecounter "armgtmr0" frequency 24000000 Hz quality 500
[     1.000010] sun8ih3ccu0 at simplebus1: H3 CCU
[     1.000010] sun8ih3rccu0 at simplebus1: H3 PRCM CCU
[     1.000010] sunxide2ccu0 at simplebus1: DE2 CCU
[     1.000010] sunxigpio0 at simplebus1: PIO
[     1.000010] gpio0 at sunxigpio0: 94 pins
[     1.000010] sunxigpio0: interrupting on GIC irq 43
[     1.000010] sunxigpio1 at simplebus1: PIO
[     1.000010] gpio1 at sunxigpio1: 12 pins
[     1.000010] sunxigpio1: interrupting on GIC irq 77
[     1.000010] fregulator0 at simplebus0: vcc3v3
[     1.000010] fregulator1 at simplebus0: usb0-vbus
[     1.000010] fregulator2 at simplebus0: gmac-3v3
[     1.000010] sun6idma0 at simplebus1: DMA controller (12 channels)
[     1.000010] sun6idma0: interrupting on GIC irq 82
[     1.000010] com0 at simplebus1: ns16550a, working fifo
[     1.000010] com0: console
[     1.000010] com0: interrupting on GIC irq 32
[     1.000010] sunxiusbphy0 at simplebus1: USB PHY
[     1.000010] sunxihdmiphy0 at simplebus1: HDMI PHY
[     1.000010] sunximixer0 at simplebus1: Display Engine Mixer
[     1.000010] sunxilcdc0 at simplebus1: TCON1
[     1.000010] sunxilcdc0: interrupting on GIC irq 118
[     1.000010] sunxirtc0 at simplebus1: RTC
[     1.000010] emac0 at simplebus1: EMAC
[     1.000010] emac0: Ethernet address 02:01:f7:f9:2f:67
[     1.000010] emac0: interrupting on GIC irq 114
[     1.000010] rgephy0 at emac0 phy 7: RTL8211E 1000BASE-T media interface
[     1.000010] rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
[     1.000010] h3codec0 at simplebus1: H3 Audio Codec (analog part)
[     1.000010] sunximmc0 at simplebus1: SD/MMC controller
[     1.000010] sunximmc0: interrupting on GIC irq 92
[     1.000010] motg0 at simplebus1: 'otg' mode not supported
[     1.000010] ehci0 at simplebus1: EHCI
[     1.000010] ehci0: interrupting on GIC irq 104
[     1.000010] ehci0: EHCI version 1.0
[     1.000010] ehci0: 1 companion controller, 1 port
[     1.000010] usb0 at ehci0: USB revision 2.0
[     1.000010] ohci0 at simplebus1: OHCI
[     1.000010] ohci0: interrupting on GIC irq 105
[     1.000010] ohci0: OHCI version 1.0
[     1.000010] usb1 at ohci0: USB revision 1.0
[     1.000010] ehci1 at simplebus1: EHCI
[     1.000010] ehci1: interrupting on GIC irq 110
[     1.000010] ehci1: EHCI version 1.0
[     1.000010] ehci1: 1 companion controller, 1 port
[     1.000010] usb2 at ehci1: USB revision 2.0
[     1.000010] ohci1 at simplebus1: OHCI
[     1.000010] ohci1: interrupting on GIC irq 111
[     1.000010] ohci1: OHCI version 1.0
[     1.000010] usb3 at ohci1: USB revision 1.0
[     1.000010] sunxiwdt0 at simplebus1: Watchdog
[     1.000010] sunxiwdt0: default watchdog period is 16 seconds
[     1.000010] /soc/[email protected] at simplebus1 not configured
[     1.000010] gpioleds0 at simplebus0: nanopi:green:pwr nanopi:blue:status
[     1.000010] /soc/[email protected] at simplebus1 not configured
[     1.000010] /soc/[email protected] at simplebus1 not configured
[     1.000010] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
[     1.000010] cpu2: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
[     1.000010] cpu2: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
[     1.040229] cpu2: Dcache line 64, Icache line 64
[     1.040229] cpu2: L1 32KB/64B 2-way read-allocate VIPT Instruction cache
[     1.050220] cpu2: L1 32KB/64B 4-way write-back read-allocate write-allocate PIPT Data cache
[     1.060220] cpu2: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT Unified cache
[     1.070220] cpu2: revID=0x180, PMCv3, 4k table, 64k table, 16bit ASID
[     1.070220] cpu2: auxID=0x11120, FP, CRC32, SHA1, SHA256, AES+PMULL, NEON, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add
[     1.090221] cpu1: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
[     1.090221] cpu1: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
[     1.100222] cpu1: Dcache line 64, Icache line 64
[     1.110221] cpu1: L1 32KB/64B 2-way read-allocate VIPT Instruction cache
[     1.110221] cpu1: L1 32KB/64B 4-way write-back read-allocate write-allocate PIPT Data cache
[     1.120222] cpu1: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT Unified cache
[     1.130222] cpu1: revID=0x180, PMCv3, 4k table, 64k table, 16bit ASID
[     1.140223] cpu1: auxID=0x11120, FP, CRC32, SHA1, SHA256, AES+PMULL, NEON, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add
[     1.150222] cpu3: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
[     1.160223] cpu3: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
[     1.160223] cpu3: Dcache line 64, Icache line 64
[     1.170223] cpu3: L1 32KB/64B 2-way read-allocate VIPT Instruction cache
[     1.180223] cpu3: L1 32KB/64B 4-way write-back read-allocate write-allocate PIPT Data cache
[     1.180223] cpu3: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT Unified cache
[     1.190223] cpu3: revID=0x180, PMCv3, 4k table, 64k table, 16bit ASID
[     1.200224] cpu3: auxID=0x11120, FP, CRC32, SHA1, SHA256, AES+PMULL, NEON, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add
[     1.210224] sdmmc0 at sunximmc0
[     1.240225] uhub0 at usb0: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
[     1.240225] uhub0: 1 port with 1 removable, self powered
[     1.240225] uhub1 at usb2: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
[     1.250226] uhub1: 1 port with 1 removable, self powered
[     1.250226] uhub2 at usb1: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
[     1.260226] uhub2: 1 port with 1 removable, self powered
[     1.260226] uhub3 at usb3: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
[     1.275641] uhub3: 1 port with 1 removable, self powered
[     1.275641] IPsec: Initialized Security Association Processing.
[     1.350228] sdmmc0: SD card status: 4-bit, C10, U1, A1
[     1.350228] ld0 at sdmmc0: <0x03:0x5344:SC64G:0x80:0x0cd9141d:0x122>
[     1.360690] ld0: 60906 MB, 7764 cyl, 255 head, 63 sec, 512 bytes/sect x 124735488 sectors
[     1.370228] ld0: 4-bit width, High-Speed/SDR25, 50.000 MHz
[     1.990242] boot device: ld0
[     1.990242] root on ld0a dumps on ld0b
[     2.000243] root file system type: ffs
[     2.010242] kern.module.path=/stand/evbarm/9.0/modules

July 11, 2020

Nikita Gillmann GSoC 2020 - Notes on sh(1)

Misc notes on the 3 relevant files for adding posix_spawn usage to sh, from https://github.com/teknokatze/gsoc2020/blob/default/sh.txt and summary from today's IRC meeting with my mentors. This week and last week I've read through sh's code, with no clue how to start replacing the logic.

We have 3 files (jobs, exec, eval) which seem to contain the main logic i need to investigate. Between fork and exec we sometimes fork before we exec and have too much disconnection between those two, or sometimes we exec in a vforked shell, and I need to understand more about what is going on or just start testing.

According to Riastradh the shell is going to seem a bit more complicated because it often forks subshells without directly reducing to exec. The low-hanging fruit is where it vforks, because you can't do much besides exec after vfork -- so that's the path that is already an opportunity for posix_spawn, because pretty much the only things it will do are set up signals and file descriptors and exec. In evalcommand, the relevant path is limited to the case where cmdentry.cmdtype == CMDNORMAL and (!cmd->ncmd.backgnd or cmd->ncmd.redirect == NULL), in which case it does listmklocal, forkchild, redirect, and shellexec. It's really down to what's between VFORK_BLOCK, and the case CMDNORMAL of switch (cmdentry.cmdtype) down below (which is the 'default' of the switch). This is mostly just redirect and shellexec. The easiest approach would probably be to rip out everything in VFORK_BLOCK/END and replace it by posix_spawn, don't bother falling through to the switch below; just incorporate the redirect and shellexec into posix_spawn there.


June 30, 2020

Nikita Gillmann GSoC 2020 Report 1

This is my first report for the Google Summer of Code project I am working on for NetBSD.

Update 2020-07-14: The more polished version has been published and reviewed at our TNF blog.

Update 2020-07-10: Prior work. In GSoC 2012 Charles Zhang added the posix_spawn syscall which according to its SF repository at the time (maybe even now, I have not looked very much into comparing all other systems and libcs + kernels) is an in-kernel implementation of posix_spawn which provides performance benefits compared to FreeBSD and other systems which had a userspace implementation.

The code snippets in this article are under the same license as their respective files, ie 3-clause BSD licensed, Copyright (c) 1988, 1993 The Regents of the University of California. For full details refer to netbsd.org/about/redistribution and the full source code of the functions described. This is not legal advice and I'm only the author of (some portions of) the new version of the functions.

After 1 week of reading POSIX and writing code, 2 weeks of coding and another 1.5 weeks of bugfixes I have succesfully implemented posix_spawn in usage in system(3) and popen(3) internally.

The biggest challenge for me was to understand POSIX, to read the standard. I am used to reading more formal books, but I can't remember working with the posix standard directly before.

The next part of my Google Summer of Code project will focus on similar rewrites of NetBSD's sh(1).

system(3)

The prototype

int system(const char *command);

remains the same. Below I'm just commenting on the differences, not the whole function, and therefore only include code block where the versions differ the most. The full work can be found at gsoc2020 as well as src and will be submitted for inclusion later in the project.

Previously we'd use vfork, sigaction, and execve in this stdlib function.

The biggest difference to the 2015 version of our system version is the usage of posix_spawnattr_ where we'd use sigaction before, and posix_spawn where execve executes the command in a vfork'd child:

   posix_spawnattr_init(&attr);
   posix_spawnattr_setsigmask(&attr, &omask);
   posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETSIGDEF|POSIX_SPAWN_SETSIGMASK);
   (void)__readlockenv();
   status = posix_spawn(&pid, _PATH_BSHELL, NULL, &attr, __UNCONST(argp), environ);
   (void)__unlockenv();
   posix_spawnattr_destroy(&attr);

The full version can be found here.

The prototype of posix_spawn is:

int posix_spawn(pid_t *restrict pid, const char *restrict path, const posix_spawn_file_actions_t *file_actions, const posix_spawnattr_t *restrict attrp, char *const argv[restrict], char *const envp[restrict]);

We first initialize a spawn attributes object with the default value for all of its attributes set. A spawn attributes object is used to modify the behavior of posix_spawn().

The previous fork-exec switch included calls to sigaction to set the behavior associated with SIGINT and SIGQUIT as defined by POSIX:

The system() function shall ignore the SIGINT and SIGQUIT signals, and shall block the SIGCHLD signal, while waiting for the command to terminate. If this might cause the application to miss a signal that would have killed it, then the application should examine the return value from system() and take whatever action is appropriate to the application if the command terminated due to receipt of a signal. source: https://pubs.opengroup.org/onlinepubs/9699919799/functions/system.html

This has been achieved with a combination of posix_spawnattr_setsigmask() and posix_spawnattr_setflags() in the initialized attributes object referenced by attr.

As before we call __readlockenv() and then call posix_spawn() which returns the process ID of the child process in the variable pointed to by 'pid', and returns zero as the function return value.

The old code:

   (void)__readlockenv();
   switch(pid = vfork()) {
   case -1:                        /* error */
           (void)__unlockenv();
           sigaction(SIGINT, &intsa, NULL);
           sigaction(SIGQUIT, &quitsa, NULL);
           (void)sigprocmask(SIG_SETMASK, &omask, NULL);
           return -1;
   case 0:                         /* child */
           sigaction(SIGINT, &intsa, NULL);
           sigaction(SIGQUIT, &quitsa, NULL);
           (void)sigprocmask(SIG_SETMASK, &omask, NULL);
           execve(_PATH_BSHELL, __UNCONST(argp), environ);
           _exit(127);
   }
   (void)__unlockenv();

popen(3), popenve(3)

As with system, the prototype of both functions remains the same:

FILE * popenve(const char *cmd, char *const *argv, char *const *envp, const char *type); FILE * popen(const char *cmd, const char *type);

Since the popen and popenve implementation in NetBSD's libc use a couple of shared helper functions, I was able to change both functions while keeping the majority of the changes focused on (some of ) the helper functions (pdes_child).

pdes_child, an internal function in popen.c, now takes one more argument (const char *cmd) for the command to pass to posix_spawn which is called in pdes_child.

pdes_child previously looked like this:

static void
pdes_child(int *pdes, const char *type)
{
        struct pid *old;

        /* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
           from previous popen() calls that remain open in the 
           parent process are closed in the new child process. */
        for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
                (void)close(old->fd); /* don't allow a flush */
#else
                (void)close(fileno(old->fp)); /* don't allow a flush */
#endif

        if (type[0] == 'r') {
                (void)close(pdes[0]);
                if (pdes[1] != STDOUT_FILENO) {
                        (void)dup2(pdes[1], STDOUT_FILENO);
                        (void)close(pdes[1]);
                }
                if (type[1] == '+')
                        (void)dup2(STDOUT_FILENO, STDIN_FILENO);
        } else {
                (void)close(pdes[1]);
                if (pdes[0] != STDIN_FILENO) {
                        (void)dup2(pdes[0], STDIN_FILENO);
                        (void)close(pdes[0]);
                }
        }
}

My current blog layout and formating will also mess up the readability of this function, this is the new version (the whole file is here):

static int
pdes_child(int *pdes, const char *type, const char *cmd)
{
        struct pid *old;
        posix_spawn_file_actions_t file_action_obj;
        pid_t pid;
        const char *argp[] = {"sh", "-c", NULL, NULL};
        argp[2] = cmd;
        int error;

        error = posix_spawn_file_actions_init(&file_action_obj);
        if (error) {
                goto fail;
        }
        /* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
           from previous popen() calls that remain open in the
           parent process are closed in the new child process. */
        for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
        error = posix_spawn_file_actions_addclose(&file_action_obj, old->fd); /* don't allow a flush */
        if (error) {
                goto fail;
        }
#else
        error = posix_spawn_file_actions_addclose(&file_action_obj, fileno(old->fp)); /* don't allow a flush */
        if (error) {
                goto fail;
        }
#endif
        if (type[0] == 'r') {
                error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
                if (error) {
                        goto fail;
                }
                if (pdes[1] != STDOUT_FILENO) {
                        error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[1], STDOUT_FILENO);
                        if (error) {
                                goto fail;
                        }
                        error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
                        if (error) {
                                goto fail;
                        }
                }
                if (type[1] == '+') {
                        error = posix_spawn_file_actions_adddup2(&file_action_obj, STDOUT_FILENO, STDIN_FILENO);
                        if (error) {
                                goto fail;
                        }
                }
        } else {
                error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
                if (error) {
                        goto fail;
                }
                if (pdes[0] != STDIN_FILENO) {
                        error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[0], STDIN_FILENO);
                        if (error) {
                                goto fail;
                        }
                        error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
                        if (error) {
                                goto fail;
                        }
                }
        }
        (void)__readlockenv();
        error = posix_spawn(&pid, _PATH_BSHELL, &file_action_obj, 0, __UNCONST(argp), environ);
        if (error) {
                (void)__unlockenv();
                goto fail;
        }
        (void)__unlockenv();
        error = posix_spawn_file_actions_destroy(&file_action_obj);
        /*
         * TODO: if _destroy() fails we have to go on, otherwise we
         * leak the pid.
         */
        if (error) {
                errno = error;
                return -1;
        }
        return pid;

fail:
        errno = error;
        posix_spawn_file_actions_destroy(&file_action_obj);
        return -1;
}

On a high level what happens in pdes_child() and popen is that we first lock the pidlist_mutex. Then we create a file file action list for all concurrent popen() / popenve() instances and the side of the pipe not necessary, and the move to stdin/stdout. We unlock the pidlist_mutex. Finally we return the list and destroy.

In the new version of this helper function which now handles the majority of what popen/popenve did, we have to initialize a file_actions object which by default contains no file actions for posix_spawn() to perform. Since we have to have error handling and a common return value for the functions calling pdes_child() and deconstruction, we make use of goto in some parts of this function.

The close() and dup2() actions now get replaced by corresponding file_actions syscalls, they are used to specify a series of actions to be performed by a posix_spawn operation.

After this series of actions, we call _readlockenv(), and call posix_spawn with the file_action object and the other arguments to be executed. If it succeeds, we return the pid of the child to popen, otherwise we return -1, in both cases we destroy the file_action object before we proceed.

In popen and popenve our code has been reduced to just the 'pid == -1' branch, everything else happens in pdes_child() now.

After readlockenv we call pdes_child and pass it the command to execute in the posix_spawn'd child process; if pdes_child returns -1 we run the old error handling code. Likewise for popenve.

popen, old:

FILE *
popen(const char *cmd, const char *type)
{
        struct pid *cur;
        int pdes[2], serrno;
        pid_t pid;

        _DIAGASSERT(cmd != NULL);
        _DIAGASSERT(type != NULL);

        if ((cur = pdes_get(pdes, &type)) == NULL)
                return NULL;

        MUTEX_LOCK();
        (void)__readlockenv();
        switch (pid = vfork()) {
        case -1:                        /* Error. */
                serrno = errno;
                (void)__unlockenv();
                MUTEX_UNLOCK();
                pdes_error(pdes, cur);
                errno = serrno;
                return NULL;
                /* NOTREACHED */
        case 0:                         /* Child. */
                pdes_child(pdes, type);
                execl(_PATH_BSHELL, "sh", "-c", cmd, NULL);
                _exit(127);
                /* NOTREACHED */
        }
        (void)__unlockenv();

        pdes_parent(pdes, cur, pid, type);

        MUTEX_UNLOCK();

        return cur->fp;
}

popen, new:

FILE *
popen(const char *cmd, const char *type)
{
        struct pid *cur;
        int pdes[2], serrno;
        pid_t pid;

        _DIAGASSERT(cmd != NULL);
        _DIAGASSERT(type != NULL);

        if ((cur = pdes_get(pdes, &type)) == NULL)
                return NULL;

        MUTEX_LOCK();
        (void)__readlockenv();
        pid = pdes_child(pdes, type, cmd);
        if (pid == -1) {
                /* Error. */
                serrno = errno;
                (void)__unlockenv();
                MUTEX_UNLOCK();
                pdes_error(pdes, cur);
                errno = serrno;
                return NULL;
                /* NOTREACHED */
        }
        (void)__unlockenv();

        pdes_parent(pdes, cur, pid, type);

        MUTEX_UNLOCK();

        return cur->fp;
}

popenve, old:

FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
        struct pid *cur;
        int pdes[2], serrno;
        pid_t pid;

        _DIAGASSERT(cmd != NULL);
        _DIAGASSERT(type != NULL);

        if ((cur = pdes_get(pdes, &type)) == NULL)
                return NULL;

        MUTEX_LOCK();
        switch (pid = vfork()) {
        case -1:                        /* Error. */
                serrno = errno;
                MUTEX_UNLOCK();
                pdes_error(pdes, cur);
                errno = serrno;
                return NULL;
                /* NOTREACHED */
        case 0:                         /* Child. */
                pdes_child(pdes, type);
                execve(cmd, argv, envp);
                _exit(127);
                /* NOTREACHED */
        }

        pdes_parent(pdes, cur, pid, type);

        MUTEX_UNLOCK();

        return cur->fp;
}

popenve, new:

FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
        struct pid *cur;
        int pdes[2], serrno;
        pid_t pid;

        _DIAGASSERT(cmd != NULL);
        _DIAGASSERT(type != NULL);

        if ((cur = pdes_get(pdes, &type)) == NULL)
                return NULL;

        MUTEX_LOCK();
        pid = pdes_child(pdes, type, cmd);
        if (pid == -1) {
                /* Error. */
                serrno = errno;
                MUTEX_UNLOCK();
                pdes_error(pdes, cur);
                errno = serrno;
                return NULL;
                /* NOTREACHED */
        }

        pdes_parent(pdes, cur, pid, type);

        MUTEX_UNLOCK();

        return cur->fp;
}
NetBSD.org pkgsrc-2020Q2 released

June 20, 2020

Benny Siegert Getting Started with NetBSD on the Pinebook Pro
If you buy a Pinebook Pro now, it comes with Manjaro Linux on the internal eMMC storage. Let’s install NetBSD instead! The easiest way to get started is to buy a decent micro-SD card (what sort of markings it should have is a science of its own, by the way) and install NetBSD on that. On a warm boot (i.e. when rebooting a running system), the micro-SD card has priority compared to the eMMC, so the system will boot from there.

June 18, 2020

Emile Heitor Fakecracker: NetBSD as a Function Based MicroVM
In November 2018 AWS published an Open Source tool called Firecracker, mostly a virtual machine monitor relying on KVM, a small sized Linux kernel, and a stripped down version of Qemu. What baffled me was the speed at which the virtual machine would fire up and run the service. The whole process is to be compared to a container, but safer, as it does not share the kernel nor any resource, it is a separate and dedicated virtual machine.

June 16, 2020

Stack Overflow screen fails on NetBSD, reporting "poll: Invalid argument"

I have installed and used screen many times on several different operating systems. Recently I installed it on a NetBSD-8.0 virtual machine.

$ sudo pkgin install screen
calculating dependencies...done.

1 package to install:
  screen-4.8.0nb1

0 to refresh, 0 to upgrade, 1 to install
0B to download, 1098K to install

proceed ? [Y/n] Y
installing screen-4.8.0nb1...
screen-4.8.0nb1: setting permissions on /usr/pkg/bin/screen-4.8.0 (o=root, g=wheel, m=4511)
screen-4.8.0nb1: adding /usr/pkg/bin/screen to /etc/shells
screen-4.8.0nb1: registering info file /usr/pkg/info/screen.info
===========================================================================
$NetBSD: MESSAGE,v 1.5 2005/12/28 17:53:24 reed Exp $
[snip]
===========================================================================
pkg_install warnings: 0, errors: 0
reading local summary...
processing local summary...
marking screen-4.8.0nb1 as non auto-removable

However, when I went to use it, I got an immediate failure.

$ uname -mrs
NetBSD 8.0 amd64
$ ls -l /usr/pkg/bin/screen
lrwxr-xr-x  1 root  wheel  12 Apr  6 02:50 /usr/pkg/bin/screen -> screen-4.8.0
$ groups
users wheel
$ screen
poll: Invalid argument

This problem persists even when I first remove, then reinstall the screen package. Any suggestions as to what's wrong?


June 06, 2020

Frederic Cambus OpenBSD framebuffer console and custom color palettes

On framebuffer consoles, OpenBSD uses the rasops(9) subsystem, which was imported from NetBSD in March 2001.

The RGB values for the ANSI color palette in rasops have been chosen to match the ones in Open Firmware, and are different from those in the VGA text mode color palette.

Rasops palette:

Rasops palette

VGA text mode palette:

VGA text mode palette

As one can see, the difference is quite significant, and decades of exposure to MS-DOS and Linux consoles makes it quite difficult to adapt to a different palette.

RGB values for the ANSI color palette are defined in sys/dev/rasops/rasops.c, and here are the proper ones to use to match the VGA text mode palette:

#define	NORMAL_BLACK	0x000000
#define	NORMAL_RED	0xaa0000
#define	NORMAL_GREEN	0x00aa00
#define	NORMAL_BROWN	0xaa5500
#define	NORMAL_BLUE	0x0000aa
#define	NORMAL_MAGENTA	0xaa00aa
#define	NORMAL_CYAN	0x00aaaa
#define	NORMAL_WHITE	0xaaaaaa

#define	HILITE_BLACK	0x555555
#define	HILITE_RED	0xff5555
#define	HILITE_GREEN	0x55ff55
#define	HILITE_BROWN	0xffff55
#define	HILITE_BLUE	0x5555ff
#define	HILITE_MAGENTA	0xff55ff
#define	HILITE_CYAN	0x55ffff
#define	HILITE_WHITE	0xffffff

And here is a diff doing just that, which I sent to [email protected] back in January 2017.

EDIT: The enthusiasm around this article led me to make another try, which didn't fare any better.


May 31, 2020

Benny Siegert Pinebook Pro, First Impressions
Note: This post was written on the Pinebook Pro :) After seeing it in action at FOSDEM (from afar, as the crowd was too large), I decided to buy a Pinebook Pro for personal use. From the beginning, the intention was to use it for pkgsrc development, with NetBSD as the main OS. It was finally delivered on Thursday, one day earlier than promised, so I thought I would write down my first impressions.

May 27, 2020

Frederic Cambus OpenBSD/armv7 on the CubieBoard2

I bought the CubieBoard2 back in 2016 with the idea to run OpenBSD on it, but because of various reliability issues with the onboard NIC, it ended up running NetBSD for a few weeks before ending up in a drawer.

Back in October, Mark Kettenis committed code to allow switching to the framebuffer "glass" console in the bootloader on OpenBSD/armv7, making it possible to install the system without using a serial cable.

>> OpenBSD/armv7 BOOTARM 1.14
boot> set tty fb0
switching console to fb0

This prompted me to plug the board again, and having support for the framebuffer console is a game changer. It also allows running Xenocara, if that's your thing.

Here is the output of running file on executables:

ELF 32-bit LSB shared object, ARM, version 1

And this is the result of the md5 -t benchmark:

MD5 time trial.  Processing 10000 10000-byte blocks...
Digest = 52e5f9c9e6f656f3e1800dfa5579d089
Time   = 1.340000 seconds
Speed  = 74626865.671642 bytes/second

For the record, LibreSSL speed benchmark results are available here.

System message buffer (dmesg output):

OpenBSD 6.7-current (GENERIC) #299: Sun May 24 18:25:45 MDT 2020
    [email protected]:/usr/src/sys/arch/armv7/compile/GENERIC
real mem  = 964190208 (919MB)
avail mem = 935088128 (891MB)
random: good seed from bootblocks
mainbus0 at root: Cubietech Cubieboard2
cpu0 at mainbus0 mpidr 0: ARM Cortex-A7 r0p4
cpu0: 32KB 32b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu0: 256KB 64b/line 8-way L2 cache
cortex0 at mainbus0
psci0 at mainbus0: PSCI 0.0
sxiccmu0 at mainbus0
agtimer0 at mainbus0: tick rate 24000 KHz
simplebus0 at mainbus0: "soc"
sxiccmu1 at simplebus0
sxipio0 at simplebus0: 175 pins
sxirtc0 at simplebus0
sxisid0 at simplebus0
ampintc0 at simplebus0 nirq 160, ncpu 2: "interrupt-controller"
"system-control" at simplebus0 not configured
"interrupt-controller" at simplebus0 not configured
"dma-controller" at simplebus0 not configured
"lcd-controller" at simplebus0 not configured
"lcd-controller" at simplebus0 not configured
"video-codec" at simplebus0 not configured
sximmc0 at simplebus0
sdmmc0 at sximmc0: 4-bit, sd high-speed, mmc high-speed, dma
"usb" at simplebus0 not configured
"phy" at simplebus0 not configured
ehci0 at simplebus0
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Generic EHCI root hub" rev 2.00/1.00 addr 1
ohci0 at simplebus0: version 1.0
"crypto-engine" at simplebus0 not configured
"hdmi" at simplebus0 not configured
sxiahci0 at simplebus0: AHCI 1.1
scsibus0 at sxiahci0: 32 targets
ehci1 at simplebus0
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Generic EHCI root hub" rev 2.00/1.00 addr 1
ohci1 at simplebus0: version 1.0
"timer" at simplebus0 not configured
sxidog0 at simplebus0
"ir" at simplebus0 not configured
"codec" at simplebus0 not configured
sxits0 at simplebus0
com0 at simplebus0: ns16550, no working fifo
sxitwi0 at simplebus0
iic0 at sxitwi0
axppmic0 at iic0 addr 0x34: AXP209
sxitwi1 at simplebus0
iic1 at sxitwi1
"gpu" at simplebus0 not configured
dwge0 at simplebus0: address 02:0a:09:03:27:08
rlphy0 at dwge0 phy 1: RTL8201L 10/100 PHY, rev. 1
"hstimer" at simplebus0 not configured
"display-frontend" at simplebus0 not configured
"display-frontend" at simplebus0 not configured
"display-backend" at simplebus0 not configured
"display-backend" at simplebus0 not configured
gpio0 at sxipio0: 32 pins
gpio1 at sxipio0: 32 pins
gpio2 at sxipio0: 32 pins
gpio3 at sxipio0: 32 pins
gpio4 at sxipio0: 32 pins
gpio5 at sxipio0: 32 pins
gpio6 at sxipio0: 32 pins
gpio7 at sxipio0: 32 pins
gpio8 at sxipio0: 32 pins
usb2 at ohci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00 addr 1
usb3 at ohci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00 addr 1
simplefb0 at mainbus0: 1920x1080, 32bpp
wsdisplay0 at simplefb0 mux 1: console (std, vt100 emulation)
scsibus1 at sdmmc0: 2 targets, initiator 0
sd0 at scsibus1 targ 1 lun 0: <SD/MMC, SC64G, 0080> removable
sd0: 60906MB, 512 bytes/sector, 124735488 sectors
uhidev0 at uhub2 port 1 configuration 1 interface 0 "Lenovo ThinkPad Compact USB Keyboard with TrackPoint" rev 2.00/3.30 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub2 port 1 configuration 1 interface 1 "Lenovo ThinkPad Compact USB Keyboard with TrackPoint" rev 2.00/3.30 addr 2
uhidev1: iclass 3/1, 22 report ids
ums0 at uhidev1 reportid 1: 5 buttons, Z and W dir
wsmouse0 at ums0 mux 0
uhid0 at uhidev1 reportid 16: input=2, output=0, feature=0
uhid1 at uhidev1 reportid 17: input=2, output=0, feature=0
uhid2 at uhidev1 reportid 19: input=8, output=8, feature=8
uhid3 at uhidev1 reportid 21: input=2, output=0, feature=0
uhid4 at uhidev1 reportid 22: input=2, output=0, feature=0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
bootfile: sd0a:/bsd
boot device: sd0
root on sd0a (f7b555b0fa0e8c49.a) swap on sd0b dump on sd0b

Sensors output:

$ sysctl hw.sensors
hw.sensors.sxits0.temp0=39.50 degC
hw.sensors.axppmic0.temp0=30.00 degC
hw.sensors.axppmic0.volt0=4.95 VDC (ACIN)
hw.sensors.axppmic0.volt1=0.03 VDC (VBUS)
hw.sensors.axppmic0.volt2=4.85 VDC (APS)
hw.sensors.axppmic0.current0=0.11 A (ACIN)
hw.sensors.axppmic0.current1=0.00 A (VBUS)
hw.sensors.axppmic0.indicator0=On (ACIN), OK
hw.sensors.axppmic0.indicator1=Off (VBUS)

May 20, 2020

Amitai Schlair notqmail 1.08 released

notqmail logo

I’m pleased to announce notqmail 1.08, the latest update to notqmail. It addresses security vulnerabilities in qmail 1.03 reported yesterday by Qualys. As a fork, we’ve inherited qmail’s relatively few bugs; as an active qmail fork, we’ve addressed the vulnerabilities with a timely release; and as the community-driven open-source successor to qmail, we show our work.

The 1.08 release also includes many other small improvements we’ve made since 1.07, including fewer compiler warnings, less dead code, published intent to remove more presumed-dead code, my QMAILREMOTE patch, and our first few unit tests. For more detail, see the notqmail 1.08 release notes.


May 16, 2020

Nikita Gillmann On plant - update 2

Continued musings about plant on a saturday evening, still not 100% verified and opinionated as long as I don't double check with my existing notes. It's a bit chaotic, just taking notes.

So, what exactly do I want?

First it's mostly an experiment at this point.

I'm contributing to Guix, I'm contributing (rarely) to Nix, and I'm a pkgsrc developer. I've used and packaged for various other package managers in the past. And I really am unhappy with all of them, all have nice features, and all make compromises at some points, all make me annoyed when working with them for too long.

I could happily settle for Guix if it wasn't for a few nitpicks where the most unchangable ones can be reduced to code structure and certain parts being unchangable.

Before I dropped/indefinitely paused infotropique/OS core, I reworked my fork of Guix in a way that some issues were addressed, but the amount of work and time to rebase on a newer guix version made me reconsider using guix for small gains in it. I effectively had something which wanted to go in a new direction, but running with an engine I had no full control over. Notes prior to joining NetBSD shifted to the point where I often came back to basically wanting NetBSD but structured differently together with a package manager but not as in the old "this BSD can be dissected like Linux Distributions" mistake.

plant is my ideal package manager, and so is infotropique OS/core constructed from it in templates.

It'd base on MesCC, and some core tools.

As roughly listed here already:

Since the idea is to get to an modular extensible approach to package managers, we want the core to be relatively small and where necessary you should be able to replace the code.

It seems to point at some involvement of Lisp again or one of the few other languages which allow this type of code to be written.

Addendum:

In https://issues.guix.gnu.org/41286#1 which started my more public effort of porting Guix to NetBSD, Ludovic Courtès writes:

Hi Nikita,

Nikita Gillmann skribis:

as mentioned in IRC I have begun porting Guix to NetBSD (with the path taken not yet decided upon, just plain building guix itself for now).

Glibc provides argp. Arguably we don't have to check for argp because Guix targets glibc. But I am quiete certain that there will be people who will attempt to do what I am doing and run into this.

Guix targets glibc-based systems, so as you write, it’s reasonable to assume argp is present.

Also, using AC_CHECK_HEADERS doesn’t achieve anything: it only defines ‘HAVE_ARGP_H’ to zero or one.

Last, I don’t want to discourage anyone from porting, but I also want to be clear about what it entails. I’m strongly in favor of supporting only glibc because: (1) after all, it’s about GNU as a system, and (2) my experience with Nixpkgs is that supporting multiple C libraries is just too much work to maintain good support.

Thus I’m closing for now.

In a chat today with Janneke of GNU Mes I got enough pointers to start with adding support for NetBSD to mes. It's a start, but it will take a long time with everything else going on in my life.


May 15, 2020

Nikita Gillmann On plant - update 1

A quick reflection on what plant wanted to be:

In 2015 I started exploring the reliable creation of a live CD system for an application, which lead to thinking about the concept of an Operating System based on Guix, which rapidly diverged from Guix.

One of the main ideas was to provide building blocks instead of a monolithic approach to the core (you will find that you can modify Guix to a decent amount for an application, but some parts are designed in a way that (last tested in 2017) you can't add/override/etc). Furthermore I knew I wanted to test out ideas which would not work with the way Guix as a group works. While Guix and Nix allow yourself to be pretty independent from a central group, in practice you rely on their flow of work, their decisions, the way they upgrade and their infrastructure. Which isn't bad. I just want to test more decentralization, but made the (decision / mistake) to rely for that part at that time on a network which still needs some tweaks to be ready for this. Furthermore I disagree with some decisions which justify the existence of certain kernel flavors based on interpretation of blobs, licenses, how hardware works, and what some people tell others who ask to run this system (just buy a new/supported device is not an option for many people). Next I dislike the way updates are handled, but we all have to make compromises in what can be maintained, security updates handling, etc. I have my own ideas, which I guess/hope are not that new but just encode certain expectations and contracts. Modularity was also a big goal. Did I mention this should be a generic package manager and also operating system builder through templating and inheritance of the language used (maybe you don't know that Gentoo, Guix, Nix, etc are package managers). And much more.

I am going to extend on these points as I go through old stacks of papers and digital notes I assembled, but not all of them right now. This is the first copy of nightly notes I've started again after a long break on (publicly) thinking about package managers. This one is from 4 days ago. Don't take this as finished, technically correct, or whatever. It's just to try and start documenting more publicly what I do. No concept diagrams for now included. It's mostly written without much editing, and I hope to construct a more long-form technical preview once I get to a point where I can start working on it.

plant should be a set of concepts, protocols, and idealized package guidelines. My version of it would then mean I get to use an implementation of it, for example in Common Lisp. Maye the core could be in C. Maybe it could be entirely Lisp. The goal is just that recipes and binaries can be exchanged. If someone ships an extension to read ebuild-like syntax to build a subset of Gentoo with this? It's okay. But there needs to be a native package format first. I'm okay with NetBSD, so I don't really see a future in building yet another Unix-like system (which doesn't mean that infotropique OS/core is abandoned). When implementations can produce bit by bit the same output for builds, this should work. build-system etc can all be extensons people get however they want them to get (implicit: obtaining and depending on them is managed and documented for packages). I think this is similar to how guix conceptually works, but making all of it hackable (tricky for reprpducibility, so future compromises might apply), without any strong opinion on what plant should look like or run. Even the way you use the network, which network, will be up to you. i2p? tor? "clearnet"? GNUnet? plant doesn't care. Trade recipes, no binary builds on a central server, ... Groups can and probably will form to collaborate, but t really should just be well documented to give everyone what they want it to be. This doesn't have to contradict reproducible builds, it just has to work differently (and makes it harder).

lose notes, incomplete,...:


April 21, 2020

Kimmo Suominen Serial console for NetBSD

I’ve been switching my NetBSD machines to using a serial console lately, as it is easier to copy output from it (no more screen shots). I would still need to figure out how to setup rtty1 to log console output, so unexpected output could also be inspected later (hello, panic).

On the NetBSD VM its bootblocks must be reinstalled to select a serial port as the console. I’ve been using the auto option, so I can easily switch between the VGA and serial consoles. I set speed to zero, which will use the speed configured in BIOS. The only available “speed” with qemu is 115200 bps.

installboot -v -o console=auto,speed=0 /dev/rsd0a /usr/mdec/bootxx_ffsv2

In Proxmox a serial port must be added to the VM on the Hardware tab. It is important to leave the Display setting at Default (or VGA) — for some reason input from the serial console will not work in the boot menu otherwise. (It will work in BIOS and at the login prompt.)


  1. Also known as remote-tty in Debian. 


April 16, 2020

Brett Lymn Thomas E. Dickey on NetBSD curses


I was recently pointed at a web page on Thomas E. Dickeys site talking about NetBSD curses.  It seems initially that the page was intended to be a pointer to some differences between ncurses and NetBSD curses and does appear to start off in this vein but it seems that the author has lost the plot as the document evolved and the tail end of it seems to be devolving into some sort of slanging match.  I don't want to go through Mr. Dickey's document point by point, that would be tedious but I would like to pick out some of the things that I believe to be the most egregious.  Please note that even though I am a NetBSD developer, the opinions below are my own and not the NetBSD projects.

The first thing that I would like to mention is that the author seems to be incapable of spelling my name correctly.  It isn't hard, 4 whole letters but the only time that it is spelt correctly is when the text is copied from a NetBSD commit.  If you ever do read this Mr. Dickey, I am not offended, plenty of people struggle with my last name, I am quite used to it.

Before going further, just a bit of background as to how I started working on the NetBSD curses library.  I originally was using a commercial x86 port of System V UNIX on my home PC but started looking elsewhere when I found that the SL/IP implementation was bugged.  At the time Linux had no networking at all but the BSD source code had just been ported to the PC in a project known as 386BSD.  Since this had networking I picked this up and then switched to NetBSD early on.  Being used to the SYSV curses I was disappointed to find that there was no real way of handling cursor and function keys in a terminal independent manner as there is in the SYSV curses.  I developed and submitted a patch to the NetBSD curses library that added the support for the keypad() function and modified getch() to perform the conversion of key sequences into key symbols.  It was on the basis of this patch that I was offered the opportunity to be a NetBSD developer.

When working on NetBSD curses my overall philosophy is to follow The Single UNIX® Specification, Version 2: X/Open Curses, Issue 4 Version 2 
(I will refer to this as SUSv2 from now on) where possible, if a behaviour is undefined then I look to ncurses and match the behaviour there if appropriate on the basis that this normally results in less issues for someone porting an application.  I add ncurses extensions in response to problem reports (PRs).

Let's now have a look at Mr. Dickey's comments (I am tempted to use the word diatribe but that seems a bit strong...).  I won't go over all of them but will pick up the topics that he has mentioned going down his page.

I would like to pick up on the comments about making the window structure opaque.  I don't apologise for this, I believe that the fact that WINDOW * was available for applications to poke at is a violation of software layering and just wrong.  Later in Mr. Dickey's page he takes NetBSD to task over error checking (more on this later) but by making the contents of WINDOW available for the application there is little use in error checking if the application can just write what they want anyway.  It took ncurses until 2007 to opaque WINDOW.  Mr. Dickey makes some ominous sounding statements about adding functions to compensate for the opaquing.  As far as I am concerned the interface to update a window is specified by SUSv2, if there are extensions that need to be added they will be added as they are needed.
Given I made WINDOW opaque nearly 20 years ago, I have yet to see a PR to indicate that more is needed.  The positives of making WINDOW opaque are not only enforcing correct layering of software but it also means that the contents of WINDOW can be modified without requiring a major bump of the library since nothing outside the curses library has access so backward compatibility is maintained.

Making the character string in scanw const just makes sense the function does not need to modify the string so it should be a constant.  The NetBSD project does not need to worry about the compiler objecting because the base compiler works with this statement.

Wide curses was a Google Summer of Code project that was done by Ruibiao Qiu.  I think that this was a great success and added some very useful functionality to NetBSD curses.  It seems that Mr. Dickey is a bit confused by this, in actual fact, he is totally wrong on a number of points:
  1. The non-spacing characters are not stored as a linked list, they are an array
  2. The interface for setting non-spacing characters is anything that uses the complex character type (cchar_t)
He seems a bit mystified by the term non-spacing character, this term comes from SUSv2 described here
The non-spacing characters are dynamically added when required, this was my design as I thought that simply having a fixed array for every character cell in the display was a waste of memory, it would be more efficient to only allocate memory as required for the non-spacing characters.

The function specification for ungetch is ambiguous in SUSv2, NetBSD will accept the output from getch on the basis that the input stream has already been processed and it makes no sense to save the last character in what could be a multi-byte sequence.

As for testing of curses, yes, this is mine.  I am happy that at least some of our curses functions are being tested to ensure that any changes made don't affect the on-screen output and that if there are changes that they are highlighted and can be analysed for correctness.  It is too easy with terminal output to have something that "looks right" but is doing the wrong thing in the background.

Now we get to something I wrote on a mailing list:
 
Re: curses: create panel from stdscr?

I stand by my comments there.  I was referring to my attempt to report what I thought was an issue with the ncurses libmenu implementation.  I no longer have the email nor the reply from Mr. Dickey but my clear recollection of the tone and attitude was that I would not be bothered raising issues about ncurses with Mr. Dickey ever again.  I wrote the libmenu and libform libraries for NetBSD using the book "Programmer's Guide: Character User Interface (FMLI and ETI)", this is UNIX System Laboratories book so I regard it as definitive for the libraries.  I noticed a variance in the ncurses libmenu implementation but as I have stated my efforts of reporting that were not fruitful.  Mr. Dickey did reach out to me in late 2017 asking for the email I was referencing, his comment was that my response was vague, actually, my exact words were:
I will try and dig it up.  It was a very long time ago.  I think it was
related to some behaviours in libmenu.  I will try and dig back through
my mail history but it may take a bit of time.
Alas I never found the original email but looking at my commits around libmenu I would have sent that email in 1999 some time so I don't think that this is very surprising.  At the time of the reply I thought that perhaps there was some interest in actually addressing the issue but the actual fact is that all it was was a fishing expedition to provide more ammunition for an attack on NetBSD curses which is disappointing.

Getting back to the email quoted above, in the book I have create_panel() is documented as taking a WINDOW as parameter.  Though, internally, stdscr is treated as a window the books description of create_panel() does make it clear that WINDOW is expected to be the return from newwin() not just stdscr.

As for missing functions, they will be added as and when the developers have time.  A PR may guide us in what is needed first but given that NetBSD is a volunteer project the time that the developers can devote to adding new features can be variable.  We are aiming to have something that is useful and not necessarily a clone of ncurses.

Mr. Dickey makes a point about portability.  I myself have never been concerned about making NetBSD curses portable, this is not an aim that I have ever had.  I would much rather be adding new features and fixing bugs for NetBSD than struggling with the complexities of making the NetBSD curses library portable.  If others wish to do this and feed back fixes then they will be considered but in the main, portability is not a focus.

He also turns on some random gcc flags and then berates NetBSD curses for producing extra warnings.  It is difficult to determine what flags he has turned on to produce these warnings.  The NetBSD build system treats any warnings as errors and the build terminates, the NetBSD curses library builds with the standard NetBSD gcc options without issue.  Without knowing what options were uses and what defines were in effect the information that Mr. Dickey has presented is useless.  He is welcome to submit a PR using the public interface and it will be looked into.

He then notes the output differences in the wide character implementations.  Unfortunately he has not reached out to me (at least) with this information.  It would be useful to have the tools to test this issue and rectify it.  I will note that he used a rather old version of NetBSD, it would be interesting to see if these issues still exists.

We then have a snide little bit about the number of error returns and error checking.  He says "Error-checking in NetBSD curses is seen as a problem by its developers" which is absolutely untrue and uncalled for.  He seems to be forgetting at this by that, by his own admission, that NetBSD curses does not have all the functions that ncurses has so there should be less returns of status simply from that.  Even if this were not true, if we look at the ratio of ERR/OK returns per number of statements - using the numbers from his page - NetBSD curses has 12987 statements which translates to a ERR/OK return every 25 statements where as ncurses has 24566 statements which translates to a ERR/OK return ever 38 statements.  So NetBSD curses actually assigns status codes more often than ncurses.  Regardless, I view this metric as specious - simply counting returns is rubbish, look at this:

if ((x > maxx) || (y > maxy))
         return ERR;

VS:

if (x > maxx)
        return ERR;

if (y > maxy)
        return ERR;


Both do the same thing but we get double the ERR returns in the second one.  I am not saying this is what is happening but just pointing out the fallacy of counting ERR/OK assignments and trying to draw a conclusion from it without properly analysing the code.

He lambasts NetBSD for commenting out unsupported functions in pycurses, this is just a reflection of the NetBSD curses not having all the functions at the moment.  This is a pkgsrc thing and locally applied, pycurses is unique in that it wants all the curses functions so it can provide bindings, since NetBSD curses doesn't have them then the unsupported ones are commented out.  It would be ideal if they all existed, hopefully one day they will.

The final thing I would like to pick up on is the comment:

"Without documentation, there is no point in making bug reports."

Which is just a lame excuse, even the lack of documentation is something that can be a bug report.  There is a public interface for reporting bugs.  Not using it and then complaining on your own website is not helpful to NetBSD but then again, perhaps that is the intention after all.