NetBSD Planet


May 10, 2021

Pullup 9 [pullup-9 #1271] Please pullup sysinst MSDOS FS fixes
Ruben Schade Umbrage at trying FreeBSD and Linux in VMs

Picture this: you’re browsing through a forum, mailing list, or social network, and someone posts that they’re trying a BSD, illumos, Linux, or another operating system they’re not familiar with. The community rallies around this convert, and offers words of encouragement and technical assistance.

All goes well, until an eagle-eyed reader chances upon something that causes them, to use a phrase popular in Singapore at the moment, umbrage. Buried in dmesg output, or a screenshot, is evidence that the OS the person installed is running in… hold your hats… a virtual machine. And boy do they let the poster know their disdain.

Gatekeeping of this nature is as rife as it is baffling. Not only is it counterproductive to mock people for testing a new OS in a VM, but the entire premise of this elitism is technically flawed. If you’re going to be pedantic about something, at least take the time to be correct. Think of the other pendants!

I always read pedant as peanut. Maybe that’s why so many people have allergies to them.

Open source software communities, especially those not associated with large corporate sponsors, don’t have the luxury of big advertising budgets, training programmes, and social media campaigns. Someone coming to you, of their own volition, to learn about your OS is fantastic. Nurturing someone today could mean a contributor or advocate tomorrow. Driving them away because they had the audacity to share their experience in a VM is neither useful or nice.

Virtual machines are perfect for trying new OSs, precisely for their composability, ability to rollback learner mistakes without damage to the host system, and being able to leverage existing hardware. There’s a reason VMs are popular in academia and education: they let you start with a clean slate and focus on picking up the fundamentals. I don’t begrudge anyone for wanting to avoid the potential pitfalls of dual booting, especially if it’s your primary or only machine. VMs are fun, convenient, and inexpensive.

But even leaving all that aside, the worst kept secret in IT today is that the bulk of Unix-like OSs run in virtual machines! They’re in containers, cloud instances, VPSs, or even in on-premise hypervisors. A bhyve VM on FreeBSD, or a VMware VM on Workstation, or any number of other VMs, can even be migrated and booted elsewhere. Knowing how an OS will perform within a virtualised environment, from networking to memory use, is also valuable skill. How does your target database handle not having direct IO, for example? What other considerations, from performance to security, must you factor in?

Professionals use VMs for everything from building, testing, and production. People like me run the hypervisors to make it work!

Which leads us to the real reason why these delightful people try and take down newbies starting with a VM. They’re seen as easy takedowns in the zero sum game world they see. You can’t be a serious computator if you’re running it in a VM on your laptop, right? Those people likely won’t ever be swayed by facts and logic, so instead I’d propose that serious computators don’t discriminate over such an arbitrary metric. They’re bigger than that.

By Ruben Schade in Sydney, 2021-05-10.


May 09, 2021

Pullup 9 [pullup-9 #1270] Fwd: CVS commit: src/sys/dev/pci

May 08, 2021

DragonFly BSD Digest In Other BSDs for 2021/05/08

Lots to read, yet I still have unposted tabs!

 

Pullup 9 [pullup-9 #1269] Turn on constty instead of ttyE0 for mac68k
Ruben Schade Two types of comparison sites

It’s human nature to compare systems you’re researching to ones with which you’re already familiar. Varnish and nginx for web proxying, or Firefox and Safari for web browsing, for example. Do a search in 2021, and chances are you’ll come across one of these:

I mentioned something similar at my Linux.conf.au talk about using FreeBSD in the field. Sites tell us that we should go with FreeBSD for broad support, NetBSD for portability, OpenBSD for security, and DragonFly BSD because something something Amiga. That’s all great, but what does any of this mean?

You could be forgiven for thinking that with sufficient data mining you could extract information and provide quantitatively-informed advice. Sports commentary is now largely written by bots, so why couldn’t the same apply to software?

The thing is, algorithms can never replace expertise. Expertise comes from experience and knowledge, and is opinionated.

In conclusion, Varnish and nginx are both web proxy software with different use cases, strengths, and weaknesses. Choose the one that’s right for you!

We don’t need a thousand articles telling us that Varnish and nginx are both caching servers, we want to know your battle stories, experience using in the field, how you think their configuration compares, the implications elsewhere in the stack over choosing one, how responsive and cordial their community is, whether the documentation is useful, how frequently it’s updated, and so on. You can’t get that information by parsing Wikipedia infoboxes.

I’ve been talking about the rise and fall of blogging again for the last few years in the context of independent publishing and RSS. Maybe I need to broaden my scope to wish for more human writers!

By Ruben Schade in Sydney, 2021-05-08.


May 07, 2021

Ruben Schade Answering @klarainc: Where I started with ZFS

Klara Inc have done a great job of late advocating for FreeBSD, including its licencing and unique features. I can speak to the experience of pitching a solution with a FreeBSD component only to have people—whom you’d otherwise assume are well informed—not know what it is. Their most recent tweet asked what we started with ZFS on, and what we did with it.

I namedrop ZFS and OpenZFS here a lot, but as a primer for those unaware, it’s a file system and volume manager designed for the express purpose of making storage easier to administer. The two key features for me:

I started using ZFS back in the OpenSolaris days, mostly out of curiosity. Sun Microsystems had open-sourced their Solaris OS and tooling, which gave us this kickarse new storage system. I used SunOS/Solaris at university in the late 2000s, but OpenSolaris gave me the first experience with how simple ZFS was to use. I installed it on my old ThinkPad X40 with it back in the day, after Fedora had decided it didn’t like my integrated graphics anymore. I even briefly ran it on my MacBook Pro. You could say the seed was well-planted by then, and thanks to ZFS, it wasn’t about to be accidentally deleted.

Which leads me to a patented Rubenerd.com digression, if you’ll indulge me! I’ve mentioned it here a few times, but ZFS isn’t only useful for large data stores on a remote server. ZFS’s integrity features make it ideal for laptops, because your data will remain consistent even during power loss or a borked suspend/hibernation. On workstations, ZFS snapshots let you upgrade and try new software without risk.

FreeBSD’s ZFS integration is the best out there, and is now where I use it for most of my systems. My homelab consists of a few FreeBSD servers running ZFS for data storage, as well as to simplify deployments of jails and Xen domu VMs. At work we use it to present storage over InfiniBand to our secondary VMware cluster, for clients who have workloads that can’t run on our Linux Xen infrastructure.

And as for NetBSD, I started using ZFS with it starting with their pivotal version 9 release! Benedict and Allan sometimes read my silly posts here on their BSD Now! show which makes me unreasonably happy, and I think my post about trying encrypted ZFS on NetBSD was the first one they discussed.

I still haven’t tried running ZFS on Mac or Windows, mostly because I offload whatever I can from those systems. Or DOS! Yes, wouldn’t it be great for me to try a new CONFIG.SYS memory setting on my 486, and be able to roll back after the inevitable fireworks set in? Don’t answer that.

Give FreeBSD with ZFS a try now, and ping Klara Inc if you need enterprise-grade support for it. The aha! moment I had when I first tried ZFS a decade ago now was something rare and special; if you’re at all interested in this stuff I think you’ll like it.

By Ruben Schade in Sydney, 2021-05-08.

Pullup pkgsrc [pullup-pkgsrc #6455] mail/exim multiple vulnerabilities fixed

May 06, 2021

Ruben Schade My state of macOS virtualisation, with FreeBSD and NetBSD digressions

It only occured to me halfway through writing this that what I’ve said here applies to Intel Macs. Clara has an M1, but I’m still on the fence if I’m going that way.

Antranig Vartanian asked me on The Bird Site:

How’s your Virtualization setup on macOS? I have to use macOS for work for a while, since I use a mix of bhyve and VirtualBox on my FreeBSD machines, I don’t know what to do on macOS [..] last time I installed VirtualBox it messed up my USB ports and I was not able to transfer data, only charge.

I had a similar experience with VirtualBox, only with NICs. It added a network bridge that for some reason kept messing up my Ethernet to USB-C dongle when cutting over from Wi-Fi. Or at least, I think it did. I removed it and it magically all worked again. It was frustrating, because VirtualBox is otherwise pretty capable and has good support for tools like Vagrant.

I used to write a lot about virtualisation here, I should get back into it. It’s a fascinating field of IT, and recent developments on mu tipple fronts are intriguing and worth exploring, especially on NetBSD of late too. I got my current job, in no small part because my boss read VM-related stuff I posted here.

But I digress! As with everything on the Mac, I offload whatever I can to my homelab server running FreeBSD. VMs and jails on my Holo server running a mix of FreeBSD, NetBSD, and Debian run Minecraft, PleX, Netatalk file shares, build environments, dev testing, automated backups, and a music server. Some of these are backed onto an OpenVPN tunnel so I can access wherever. I need to get back into bhyve more; I run a Xen dom0 because it’s what we use at work and its what I’m most familiar with.

Screenshot of Parallels Desktop running Telemetry OS and FreeBSD 13, and QEMU running Windows 2000

With that set up, this leaves me three use cases for Mac virtualisation:

I can say from experience that VMware Fusion is also excellent, and was slightly more useful than Parallels when I was also doing more VMware ESXi stuff back in the day. I standardised back on Parallels to make my life simpler, just as I did with the first betas in 2006!

All this works great, which is why I’m hesitant to move to an M1 Mac, at least for now. A small part of me wonders if this might be impetus I need to jump completely to a FreeBSD laptop, assuming I can run the software I need for work and a few games in a Windows guest. bhyve with PCI passthrough for a discrete GPU would be boss.

By Ruben Schade in Sydney, 2021-05-07.

Pullup 8 [pullup-8 #1678] please pullup fixes for building netbsd-8 on GCC 10 hosts

May 04, 2021

Pullup 9 [pullup-9 #1268] audio: do not prefer surround formats

May 03, 2021

Pullup 9 [pullup-9 #1267] Fix PR 56130 fdescfs create nodes with wrong major number

May 02, 2021

Pullup 8 [pullup-8 #1677] fix posix_spawn

May 01, 2021

DragonFly BSD Digest In Other BSDs for 2021/05/01

There’s some opinions mixed in this week!

 


April 30, 2021

Pullup 8 [pullup-8 #1676] Please sync external/public-domain/tz/dist in -8 with HEAD
Pullup 8 [pullup-8 #1675] fix for PR port-xen/50282 (make XEN kernels link without options DDB)

April 29, 2021

Pullup pkgsrc [pullup-pkgsrc #6454] pullup-request: pkgsrc/net/bind916
Pullup pkgsrc [pullup-pkgsrc #6453] pullup-request: pkgsrc/net/bind911
Pullup pkgsrc [pullup-pkgsrc #6452] [[email protected]: CVS commit: pkgsrc/chat/irssi-xmpp]

April 28, 2021

Pullup 8 [pullup-8 #1674] dhcpd.conf fix in install notes

April 26, 2021

Pullup pkgsrc [pullup-pkgsrc #6451] GeoIP segfault fix for netbsd

April 25, 2021

Benny Siegert NetBSD VM on bhyve (on TrueNAS)
My new NAS at home is running TrueNAS Core. So far, it has been excellent, however I struggled a bit setting up a NetBSD VM on it. Part of the problem is that a lot of the docs and how-tos I found are stale, and the information in it no longer applies. TrueNAS Core allows running VMs using bhyve, which is FreeBSD’s hypervisor. NetBSD is not an officially supported OS, at least according to the guest OS chooser in the TrueNAS web UI :) But since the release of NetBSD 9 a while ago, things have become far simpler than they used to be – with one caveat (see below).

April 15, 2021

Server Fault ssh tunnel refusing connections with "channel 2: open failed"

All of a sudden (read: without changing any parameters) my netbsd virtualmachine started acting oddly. The symptoms concern ssh tunneling.

From my laptop I launch:

$ ssh -L 7000:localhost:7000 [email protected] -N -v

Then, in another shell:

$ irssi -c localhost -p 7000

The ssh debug says:

debug1: Connection to port 7000 forwarding to localhost port 7000 requested.
debug1: channel 2: new [direct-tcpip]
channel 2: open failed: connect failed: Connection refused
debug1: channel 2: free: direct-tcpip: listening port 7000 for localhost port 7000, connect from 127.0.0.1 port 53954, nchannels 3

I tried also with localhost:80 to connect to the (remote) web server, with identical results.

The remote host runs NetBSD:

bash-4.2# uname -a
NetBSD host 5.1_STABLE NetBSD 5.1_STABLE (XEN3PAE_DOMU) #6: Fri Nov  4 16:56:31 MET 2011  [email protected]:/m/obj/m/src/sys/arch/i386/compile/XEN3PAE_DOMU i386

I am a bit lost. I tried running tcpdump on the remote host, and I spotted these 'bad chksum':

09:25:55.823849 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 67, bad cksum 0 (->3cb3)!) 127.0.0.1.54381 > 127.0.0.1.7000: P, cksum 0xfe37 (incorrect (-> 0xa801), 1622402406:1622402421(15) ack 1635127887 win 4096 <nop,nop,timestamp 5002727 5002603>

I tried restarting the ssh daemon to no avail. I haven't rebooted yet - perhaps somebody here can suggest other diagnostics. I think it might either be the virtual network card driver, or somebody rooted our ssh.

Ideas..?


April 11, 2021

Ruben Schade Feedback about a slow CF card

Yesterday I wrote about a new CF card with better specifications performing worse as a boot disk on my Pentium 1 tower. I didn’t have hard figures, but the machine was noticeably sluggish when performing writes compared to the slower 16 GiB card it replaced. I got more than a dozen comments from people like you who read this, thank you!

Turns out that among the electrical engineers and developers among you, my theory that there’s a protocol mismatch is almost certainly false. To recap, I thought maybe its use of UDMA7 somehow confused the VIA firmware on this 1998 motherboard, but that’s not how this hardware works. Hales and Jonathan Belford even sent me photos showing the discrete controller next to the flash memory, which makes sense. CF cards are designed to be backwards compatible, and to implement all prior specifications.

I’m still leaving open the remote possibility that’s an incompatibility, if only based on the fact that this motherboard has surprised me before. My favourite was the MIDI not working on my Sound Blaster AWE32 card when I had a PCI Adaptec SCSI card installed, but not a PCI Jaz Jet SCSI accelerator card. It also stubbornly refuses to recognise more than 64 MiB of memory, regardless of the density, layout or voltage of the DIMMs I buy for it, and the board’s supposed support for up to 288 MiB [sic].

But by far the most likely issue in this case is either a faulty card, or a ripoff. We’ve all seen the photos of SSDs being sold that are merely a SD card and some glue in a case. I bought this from a (supposedly!) reputable seller in Australia, so I’d hope it wasn’t the latter. But I’d be interested to see!

Now I’m motivated to do some proper tests. I’ll boot it into NT4 and NetBSD next weekend and run it through its paces. I also remembered I have a PCI ATA controller card that I originally bought for an ISA machine; I might try that too to isolate if the controller on the motherboard is flaky.

By Ruben Schade in Sydney, 2021-04-12.


April 10, 2021

DragonFly BSD Digest In Other BSDs for 2021/04/10

I’m hitting all the unixes.

Super User Wait for process to start on linux/(net)bsd

I'm attempting to make a script which tracks how many times you execute a specific process. I want to detect when the process starts and then log it.

The psuedo-code would be something like this:

while (true) if (process started) then log(process)

Is there an easy way to do this (preferably in shell but C is also fine) on either Linux or NetBSD?


April 09, 2021

Frederic Cambus The state of toolchains in NetBSD

While FreeBSD and OpenBSD both switched to using LLVM/Clang as their base system compiler, NetBSD picked a different path and remained with GCC and binutils regardless of the license change to GPLv3. However, it doesn't mean that the NetBSD project endorses this license, and the NetBSD Foundation's has issued a statement about its position on the subject.

Realistically, NetBSD is more or less tied to GCC, as it supports more architectures than the other BSDs, some of which will likely never be supported in LLVM.

As of NetBSD 9.1, the latest released version, all supported platforms have recent versions of GCC (7.5.0) and binutils (2.31.1) in the base system. Newer (and older!) versions of GCC can be installed via Pkgsrc, and the following packages are available, going all the way back to GCC 3.3.6:

+---------+------------+-------------------+
| Package | Version    |      Release date |
+---------+------------+-------------------+
| gcc10   | GCC 10.2.0 |     July 23, 2020 |
| gcc9    | GCC  9.3.0 |    March 12, 2020 |
| gcc8    | GCC  8.4.0 |     March 4, 2020 |
| gcc7    | GCC  7.5.0 | November 14, 2019 |
| gcc6    | GCC  6.5.0 |  October 26, 2018 |
| gcc5    | GCC  5.5.0 |  October 10, 2017 |
| gcc49   | GCC  4.9.4 |    August 3, 2016 |
| gcc48   | GCC  4.8.5 |     June 23, 2015 |
| gcc3    | GCC  3.3.6 |       May 3, 2005 |
+---------+------------+-------------------+

The focus on GCC doesn't mean that the GNU and LLVM toolchains cannot coexist within NetBSD, and work has in fact been done during the last decade to make it happen.

Despite currently not being built by default in official NetBSD releases, LLVM has been imported in the NetBSD source tree in 2013. Daily images are built from NetBSD-current for selected platforms (at least amd64, i386 and evbarm) with the MKLLVM and HAVE_LLVM build options enabled, and contain LLVM and Clang.

Moreover, NetBSD has invested a lot of work on LLVM during the past few years, including funding some developer contracts for Kamil Rytarowski ([email protected]) and Michał Górny ([email protected]), which allowed them to work on various parts of the LLVM toolchain to add and enhance support for sanitizers, and to improve LLDB support.

They both published several dozen articles on the NetBSD blog along the way, retracing their journey. Kamil's final report about upstreaming support to LLVM sanitizers summarizes the work accomplished. Thanks to this work, sanitizer support on NetBSD is mature and mostly on par with Linux. As a result, because LLVM is upstream for GCC sanitizers, they are also available in GCC on NetBSD. Similarly, Michał's final report on his LLDB work details the achievements on the debuggers front.

As always, work continues towards keeping the toolchains up to date, and upstreaming local changes whenever possible.


April 03, 2021

DragonFly BSD Digest In Other BSDs for 2021/04/03

Well, I made up for last week’s short list.

Semi-BSD-related: One of my 3 workplaces needs a Software Team Lead.  (scroll down)  The main product is FreeBSD-based, though this team position does not directly work with it.


April 01, 2021

The NetBSD Foundation New Developer in March 2021

March 30, 2021

NetBSD Blog GSoC Reports: Make system(3), popen(3) and popenve(3) use posix_spawn(3) internally (Final report)
This report was prepared by Nikita Gillmann as a part of Google Summer of Code 2020

This is my second and final report for the Google Summer of Code project I am working on for NetBSD.

My code can be found at github.com/teknokatze/src in the gsoc2020 branch, at the time of writing some of it is still missing. The test facilities and logs can be found in github.com/teknokatze/gsoc2020. A diff can be found at github which will later be split into several patches before it is sent to QA for merging.

The initial and defined goal of this project was to make system(3) and popen(3) use posix_spawn(3) internally, which had been completed in June. For the second part I was given the task to replace fork+exec calls in our standard shell (sh) in one scenario. Similar to the previous goal we determined through implementation if the initial motivation, to get performance improvements, is correct otherwise we collect metrics for why posix_spawn() in this case should be avoided. This second part meant in practice that I had to add and change code in the kernel, add a new public libc function, and understand shell internals.

Summary of part 1

Prior work: In GSoC 2012 Charles Zhang added the posix_spawn syscall which according to its SF repository at the time (maybe even now, I have not looked very much into comparing all other systems and libcs + kernels) is an in-kernel implementation of posix_spawn which provides performance benefits compared to FreeBSD and other systems which had a userspace implementation (in 2012).

After 1 week of reading POSIX and writing code, 2 weeks of coding and another 1.5 weeks of bugfixes I have successfully implemented posix_spawn in usage in system(3) and popen(3) internally.

The biggest challenge for me was to understand POSIX, to read the standard. I am used to reading more formal books, but I can't remember working with POSIX Standard directly before.

system(3)

system(3) was changed to use posix_spawnattr_ (where we used sigaction before) and posix_spawn (which replaced execve + vfork calls).

popen(3) and popenve(3)

Since the popen and popenve implementation in NetBSD's libc use a couple of shared helper functions, I was able to change both functions while keeping the majority of the changes focused on (some of ) the helper functions (pdes_child).

pdes_child, an internal function in popen.c, now takes one more argument (const char *cmd) for the command to pass to posix_spawn which is called in pdes_child.

On a high level what happens in pdes_child() and popen is that we first lock the pidlist_mutex. Then we create a file file action list for all concurrent popen() / popenve() instances and the side of the pipe not necessary, and the move to stdin/stdout. We unlock the pidlist_mutex. Finally we return the list and destroy.

In the new version of this helper function which now handles the majority of what popen/popenve did, we have to initialize a file_actions object which by default contains no file actions for posix_spawn() to perform. Since we have to have error handling and a common return value for the functions calling pdes_child() and deconstruction, we make use of goto in some parts of this function.

The close() and dup2() actions now get replaced by corresponding file_actions syscalls, they are used to specify a series of actions to be performed by a posix_spawn operation.

After this series of actions, we call _readlockenv(), and call posix_spawn with the file_action object and the other arguments to be executed. If it succeeds, we return the pid of the child to popen, otherwise we return -1, in both cases we destroy the file_action object before we proceed.

In popen and popenve our code has been reduced to the pid == -1 branch, everything else happens in pdes_child() now.

After readlockenv we call pdes_child and pass it the command to execute in the posix_spawn'd child process; if pdes_child returns -1 we run the old error handling code. Likewise for popenve.

The outcome of the first part is, that thanks to how we implement posix_spawn in NetBSD we reduced the syscalls being made for popen and system. A full test with proper timing should indicate this, my reading was based on comparing old and new logs with ktrace and kdump.

sh, posix_spawn actions, libc and kernel - Part 2

Motivation

The main goal of part 2 of this project was to change sh(1) to determine which simple cases of (v)fork + exec I could replace, and to replace them with posix_spawn where it makes sense.

fork needs to create a new address space by cloning the address space, or in the case of vfork update at least some reference counts. posix_spawn can avoid most of this as it creates the new address space from scratch.

Issues

The current posix_spawn as defined in POSIX has no good way to do tcsetpgrp, and we found that fish just avoids posix_spawn for foreground processes.

Implementation

Since, roughly speaking, modern BSDs handle "#!" execution in the kernel (probably since before the 1990s, systems which didn't handle this started to disappear most likely in the mid to late 90s), our main concern so far was in the evalcmd function the default cmd switch case ('NORMALCMD').

After adjusting the function to use posix_spawn, I hit an issue in the execution of the curses application htop where htop would run but input would not be accepted properly (keysequences pressed are visible). In pre-posix_spawn sh, every subprocess that sh (v)forked runs forkchild() to set up the subprocess's environment. With posix_spawn, we need to arrange posix_spawn actions to do the same thing.

The intermediate resolution was to switch FORK_FG processes to fork+exec again. For foreground processes with job control we're in an interactive shell, so the performance benefit is small enough in this case to be negligible. It's really only for shell scripts that it matters.

Next I implemented a posix_spawn file_action, with the prototype

int posix_spawn_file_actions_addtcsetpgrp(posix_spawn_file_actions_t *fa, int fildes)

The kernel part of this was implemented inline in sys/kern/kern_exec.c, in the function handle_posix_spawn_file_actions() for the new case 'FAE_TCSETPGRP'.

The new version of the code is still in testing and debugging phase and at the time of writing not included in my repository (it will be published after Google Summer of Code when I'm done moving).

Future steps

posix_spawnp kernel implementation

According to a conversation with [email protected], the posix_spawnp() implementation we have is just itterating over $PATH calling posix_spawn until it succeeds. For some changes we might want a kernel implementation of posix_spawnp(), as the path search is supposed to happen in the kernel so the file actions are only ever run once:


some of the file actions may be "execute once only",
they can't be repeated (eg: handling "set -C; cat foo >file" - file
can only be created once, that has to happen before the exec (as the fd
needs to be made stdout), and then the exec part of posix_spawn is
attempted - if that fails, when it can't find "cat" in $HOME/bin (or
whatever is first in $PATH) and we move along to the next entry (maybe /bin
doesn't really matter) then the repeated file action fails, as file now
exists, and "set -C" demands that we cannot open an already existing file
(noclobber mode).   It would be nice for this if there were "clean up on
failure" actions, but that is likely to be very difficult to get right,
and each would need to be attached to a file action, so only those which
had been performed would result in cleanup attempts.

Replacing all of fork+exec in sh

Ideally we could replace all of (v)fork + exec with posix_spawn. According to my mentors there is pmap synchronisation as an impact of constructing the vm space from scratch with (v)fork. Less IPIs (inter-processor interrupts) matter for small processes too.

posix_spawn_file_action_ioctl

Future directions could involve a posix_spawn action for an arbitrary ioctl.

Thanks

My thanks go to fellow NetBSD developers for answering questions, most recently [email protected] for sharing invaluable sh knowledge, Riastradh and Jörg as the mentors I've interacted with most of the time and for their often in-depth explanations as well as allowing me to ask questions I sometimes felt were too obvious. My friends, for sticking up with my "weird" working schedule. Lastly would like to thank the Google Summer of Code program for continuing through the ongoing pandemic and giving students the chance to work on projects full-time.

Originally published in August 2020 on https://n0.is/gsoc-2020-final-report---report-part-2.html.


March 29, 2021

NetBSD Blog Hitting donation milestone, financial report for 2020

We nearly hit our 2020 donation milestone set after the release of 9.0 of $50,000. These donations have enabled us to fund significant work on NetBSD in 2020 such as:

If you are interested in seeing more work like this, please consider donating via PayPal or GitHub sponsors.

The financial report for 2020 is also available.

Note: We realize that this data is inconsistent with the website indicator of donations. This is due to the fact the website is updated manually in an error-prone process as the donations are processed. The financial report (just completed) is prepared separately using ledger.

Stack Overflow Turn a BSD .zip file into an installable version

I'm working on a distro of NetBSD, and the code builds without any problems. However, now I have the .zip file and I don't know what to do with it. How can I turn it into an installable image. I've tried using a chroot.

Azure Pipelines GitHub


March 22, 2021

Server Fault Archive & analyse network logs from NetBSD on another machine

I want to use the the new NetBSD's NPF firewall on a gateway that also has the legal obligation to log the traffic for 1 year. To offload the gateway, I though to put the logging stuff (and IDS, quota management, ..) on another machine (probably on Debian).

I though of 2 solutions to do so :

  1. recopying the traffic to the other machine

Unlike FreeBSD and OpenBSD, I'm not sure if NetBSD is able to setup this king of port, and if it is, I don't know how. http://man.netbsd.org/bridge.4 indicates

The bridge driver currently does not support snooping via bpf(4).

but in an older version it was

The bridge driver currently does not support snooping via bpf(4) or transparent filtering.

So maybe there's a way, but I don't know where to start, any help ?

  1. run tcpdump on the NetBSD machine and continuously sync the log file with the analyse machine

But how ? It has to be reliable and adapted to network log file (ie continuously written).


March 20, 2021

DragonFly BSD Digest In Other BSDs for 2021/03/20

Solene is publishing faster than I can link!


March 13, 2021

Kimmo Suominen Patches for UTF-8 bugs in screen

Version 4.8.0nb4 of misc/screen includes a couple of patches from Debian’s salsa repository to address CVE-2021-26937 and another UTF-8 combining character bug as discussed on the screen-devel mailing list.


March 11, 2021

The NetBSD Foundation New Security Advisory: NetBSD-SA2021-001

February 26, 2021

NetBSD General on DaemonForums mkdir & symlink inside kernel code
Could anyone in the know point me at sample code for implementing (presumably via VOP_MKDIR etc.) mkdir/symlink inside the kernel?

I would like to do something like this in dkwedge_add

sys/dev/dkwedge/dk.c

/dev/disk/by-partlabel/dkw->dkw_wname link to ../../dkw->dkw_devname
/dev/disk/by-id/dkw->parent+dkw->dkw_offset link to ../../dkw->dkw_devname
/dev/disk/by-owner/dkw->dkw_wname link to ../../dkw->dkw_parent

The relevant sections on netbsd.org are not filled out (VOP_MKDIR)

February 25, 2021

Server Fault chroot not able to start service - not found. What is missing?

I would like to run a service inside a chroot in a NetBSD 9.1 amd64 system. The service runs if invoked from OS. The service in question is dendrite-monolith-server. I just copied the file for ease of use to start sitting inside the chroot in /bin/.

# ldd bin/start 
bin/start:
        -lpthread.1 => /usr/lib/libpthread.so.1
        -lc.12 => /usr/lib/libc.so.12

They are hard linked:

# ls -l usr/lib
total 8560
-r--r--r--  2 root  pe  2079984 Feb 22 23:40 lc.12
-r--r--r--  2 root  pe  2079984 Feb 22 23:40 libc.so.12
-r--r--r--  2 root  pe    93656 Feb 22 23:40 libpthread.so.1
-r--r--r--  2 root  pe    93656 Feb 22 23:40 lpthread.1

In the chroot /dev, did MAKEDEV all to create the devices.

Copied ld.elf_so to the chroot /libexec directory

# ls -l /libexec/
total 324
-r-xr-xr-x  1 0  1000  164344 Feb 22 23:47 ld.elf_so

ksh93 is statically linked:

# chroot ./ /bin/ksh93
#
# /bin/start 
/bin/ksh93: /bin/start: not found

What's wrong or missing?


February 24, 2021

NetBSD General on DaemonForums Computer Crashes
Hi all,

My desktop (i3-2120 from 2012) runs NetBSD 9 for the last 11 months. It runs 24/7 and crashes once a month. This is very bad as I rely on it being remotely accessible.

Before, it was not my computer and I don't know of any known hardware problems.

After the crash:
  • the power LED is still on and the fan continues to spin. This makes me believe it's not a hardware problem.
  • the monitor does not detect a signal from the graphics card. Here I'm not sure what that means. If the kernel crashes, what is supposed to be on the screen? Who is driving the graphics card/monitor?
  • pings are not answered.
  • there was no power outage - all other computers in the household are still running.
  • BIOS is set up to "keep the power-state" after a power-failure, e.g. when it was previously on, it will be turned on. As NetBSD did not boot again, the power was never really lost.
  • it was not a graceful shutdown. When NetBSD came up after a manual power-cycle, filesystem checks took place (well, the journal is replayed)

I follow NetBSD cvs netbsd-9 branch. I believe that is the stable branch. The NetBSD config is the default.

NetBSD XXX 9.1_STABLE NetBSD 9.1_STABLE (GENERIC) #2: Sun Jan 3 11:19:52 PST 2021 [email protected]:/usr/obj/sys/arch/amd64/compile/GENERIC amd64

The last crash must have happened early morning of Feb 24 (I power-cyceled the computer at 9:00am. Last cron job activity at 3:00am.) Log files are "blank" in the early morning. /var/log/messages:
Code:

Feb 21 13:16:04 XXX /netbsd: [ 3620751.3847045] kern error: [drm:(/usr/src/sys/external/bsd/drm2/dist/drm/i915/intel_fifo_underrun.c:230)cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder A
Feb 21 13:16:04 XXX /netbsd: [ 3620751.3847045] kern error: [drm:(/usr/src/sys/external/bsd/drm2/dist/drm/i915/intel_fifo_underrun.c:381)intel_pch_fifo_underrun_irq_handler] *ERROR* PCH transcoder A FIFO underrun
Feb 21 14:33:25 XXX /netbsd: [ 3625392.1028982] kern error: [drm:(/usr/src/sys/external/bsd/drm2/dist/drm/i915/intel_fifo_underrun.c:230)cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder A
Feb 21 14:33:25 XXX /netbsd: [ 3625392.1028982] kern error: [drm:(/usr/src/sys/external/bsd/drm2/dist/drm/i915/intel_fifo_underrun.c:381)intel_pch_fifo_underrun_irq_handler] *ERROR* PCH transcoder A FIFO underrun
Feb 24 09:03:22 XXX syslogd[184]: restart
Feb 24 09:03:22 XXX /netbsd: [  1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,

No other log files have timestamps around that time (well, besides cron.log).

What do you suspect, hardware problem or kernel crash?

Is there a knob to keep the kernel in a debugger upon a crash?

What can I do to root-cause the crash?

February 22, 2021

Super User BASH: Run function every n seconds, but allow it to be interrupted by another keystroke

I'm looking to create a function for a script where another function will be called every few seconds, however the user should be able to issue other commands to the program while this is happening. Because of this, I doubt I'll be able to use sleep as that holds up the entire script.

I've spent about a day searching for ways to accomplish this, but so far haven't found anything that would work for me.


February 15, 2021

Unix Stack Exchange Why does NetBSD grep slow down nearly 15x on subsequent calls

I'm working through examples in APUE. On a NetBSD 9.0 system, under no major load, I time a call to grep and get an unremarkable result:

apue$ cd /usr/include
apue$ time -p grep __POSIX_SOURCE */*.h > /dev/null
real         0.73
user         0.01
sys          0.63

However, if I repeat the experiment several times, the system time spikes drastically (up to 15x):

apue$ time -p grep _POSIX_SOURCE */*.h > /dev/null
real         0.57
user         0.02
sys          0.54
apue$ time -p grep _POSIX_SOURCE */*.h > /dev/null
real        10.06
user         0.01
sys         10.04
apue$ time -p grep _POSIX_SOURCE */*.h > /dev/null
real         3.57
user         0.01
sys          3.56
apue$ time -p grep _POSIX_SOURCE */*.h > /dev/null
real         4.58
user         0.00
sys          4.58
apue$ time -p grep _POSIX_SOURCE */*.h > /dev/null
real         5.56
user         0.02
sys          5.53
apue$ time -p grep _POSIX_SOURCE */*.h > /dev/null
real         6.57
user         0.00
sys          6.56
apue$ time -p grep _POSIX_SOURCE */*.h > /dev/null
real         2.56
user         0.01
sys          2.54

Is this expected behavior? What could be causing such wide variance?

Update Based on the answer given by @Tim, I took a look at my Buffercache, and saw that it was fully allocated at 100% when grep was struggling with my search. After restarting the VM, the buffer usage had dropped down to around 95%.

$ sysstat bufcache
                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   |

      603 metadata buffers using                5565 kBytes of memory ( 0%).
    15512 pages for cached file data using     62048 kBytes of memory ( 3%).
     3034 pages for executables using          12136 kBytes of memory ( 1%).
     6460 pages for anon (non-file) data       25840 kBytes of memory ( 1%).
   468172 free pages                         1872688 kBytes of memory (93%).

File System          Bufs used   %   kB in use   %  Bufsize kB   %  Util %
/                          577  95        5378  97        5418  97      99

Total:                     577  95        5378  97        5418  97      99

January 29, 2021

Frederic Cambus NetBSD on the EdgeRouter Lite

NetBSD-current now has pre-built octeon bootable images (which will appear in NetBSD 10.0) for the evbmips port, so I decided to finally give it a try. I've been happily running OpenBSD/octeon on my EdgeRouter Lite for a few years now, and have previously published some notes including more detail about the CPU.

Contrary to the OpenBSD/octeon port which is very stable and runs SMP kernels, things are a little less polished on the NetBSD side for this platform. The system runs an uniprocessor kernel and there are still some stability issues.

EdgeRouter Lite

Here is the U-Boot configuration to boot the image:

Octeon ubnt_e100# set bootcmd 'fatload usb 0 $loadaddr netbsd;bootoctlinux $loadaddr coremask=0x3 root=wedge:octeon-root'
Octeon ubnt_e100# saveenv
Saving Environment to Flash...
Un-Protected 1 sectors
Erasing Flash...
. done
Erased 1 sectors
Writing to Flash... 4....3....2....1....done
Protected 1 sectors
Octeon ubnt_e100#

On first boot, the system automatically expands the filesystem:

Resizing / (NAME=octeon-root)
/dev/rdk1: grow cg |*************************************                 |  69%

Here is the login session, for posterity:

Thu Jan 28 23:40:37 UTC 2021

NetBSD/evbmips (octeon) (constty)

login:

Here is the output of running file on executables:

ELF 32-bit MSB pie executable, MIPS, N32 MIPS-III version 1 (SYSV), dynamically
linked, interpreter /libexec/ld.elf_so, for NetBSD 9.99.79, not stripped

For the record, OpenSSL speed benchmark results are available here.

System message buffer (dmesg output):

[     1.000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
[     1.000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
[     1.000000]     2018, 2019, 2020, 2021 The NetBSD Foundation, Inc.  All rights reserved.
[     1.000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
[     1.000000]     The Regents of the University of California.  All rights reserved.

[     1.000000] NetBSD 9.99.79 (OCTEON) #0: Thu Jan 28 18:52:43 UTC 2021
[     1.000000] 	[email protected]:/usr/src/sys/arch/evbmips/compile/OCTEON
[     1.000000] Cavium Octeon CN5020-500
[     1.000000] total memory = 512 MB
[     1.000000] avail memory = 496 MB
[     1.000000] timecounter: Timecounters tick every 10.000 msec
[     1.000000] mainbus0 (root)
[     1.000000] cpunode0 at mainbus0: 2 cores, crypto+kasumi, 64bit-mul, unaligned-access ok
[     1.000000] cpu0 at cpunode0 core 0: 500.00MHz
[     1.000000] cpu0: Cavium CN5020-500 (0xd0601) Rev. 1 with software emulated floating point
[     1.000000] cpu0: 64 TLB entries, 512TB (49-bit) VAs, 512TB (49-bit) PAs, 256MB max page size
[     1.000000] cpu0: 32KB/128B 4-way set-associative L1 instruction cache
[     1.000000] cpu0: 16KB/128B 64-way set-associative write-through coherent L1 data cache
[     1.000000] cpu0: 128KB/128B 8-way set-associative write-back L2 unified cache
[     1.000000] cpu1 at cpunode0 core 1: disabled (uniprocessor kernel)
[     1.000000] wdog0 at cpunode0: default period is 4 seconds
[     1.000000] iobus0 at mainbus0
[     1.000000] iobus0: initializing POW
[     1.000000] iobus0: initializing FPA
[     1.000000] com0 at iobus0 address 0x0001180000000800: ns16650, no ERS, 16-byte FIFO
[     1.000000] com0: console
[     1.000000] com at iobus0 address 0x0001180000000c00 not configured
[     1.000000] octrnm0 at iobus0 address 0x0001180040000000
[     1.000000] entropy: ready
[     1.000000] octtwsi at iobus0 address 0x0001180000001000 not configured
[     1.000000] octmpi at iobus0 address 0x0001070000001000 not configured
[     1.000000] octsmi0 at iobus0 address 0x0001180000001800
[     1.000000] octpip0 at iobus0 address 0x00011800a0000000
[     1.000000] octgmx0 at octpip0
[     1.000000] cnmac0 at octgmx0: address=0x1180008000000: RGMII
[     1.000000] cnmac0: Ethernet address 44:d9:e7:9e:f5:9e
[     1.000000] atphy0 at cnmac0 phy 7: Atheros AR8035 10/100/1000 PHY, rev. 2
[     1.000000] atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
[     1.000000] cnmac1 at octgmx0: address=0x1180008000000: RGMII
[     1.000000] cnmac1: Ethernet address 44:d9:e7:9e:f5:9f
[     1.000000] atphy1 at cnmac1 phy 6: Atheros AR8035 10/100/1000 PHY, rev. 2
[     1.000000] atphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
[     1.000000] cnmac2 at octgmx0: address=0x1180008000000: RGMII
[     1.000000] cnmac2: Ethernet address 44:d9:e7:9e:f5:a0
[     1.000000] atphy2 at cnmac2 phy 5: Atheros AR8035 10/100/1000 PHY, rev. 2
[     1.000000] atphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
[     1.000000] dwctwo0 at iobus0 address 0x0001180068000000
[     1.000000] dwctwo0: Core Release: 2.65a (snpsid=4f54265a)
[     1.000000] usb0 at dwctwo0: USB revision 2.0
[     1.000000] bootbus0 at mainbus0
[     1.000000] timecounter: Timecounter "mips3_cp0_counter" frequency 500000000 Hz quality 100
[     1.000003] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
[     1.059978] uhub0 at usb0: NetBSD (0x0000) DWC2 root hub (0x0000), class 9/0, rev 2.00/1.00, addr 1
[     1.059978] uhub0: 1 port with 1 removable, self powered
[     1.069975] aes: BearSSL aes_ct
[     1.069975] aes_ccm: self-test passed
[     1.069975] chacha: Portable C ChaCha
[     1.079979] blake2s: self-test passed
[     3.609971] umass0 at uhub0 port 1 configuration 1 interface 0
[     3.620226] umass0: vendor 13fe (0x13fe) USB DISK 2.0 (0x4200), rev 2.00/1.00, addr 2
[     3.620226] umass0: using SCSI over Bulk-Only
[     3.620226] scsibus0 at umass0: 2 targets, 1 lun per target
[     3.632383] uhub0: autoconfiguration error: illegal enable change, port 1
[     3.639974] sd0 at scsibus0 target 0 lun 0: <, USB DISK 2.0, PMAP> disk removable
[     3.639974] sd0: 3824 MB, 959 cyl, 255 head, 32 sec, 512 bytes/sect x 7831552 sectors
[     3.659974] sd0: GPT GUID: 6e7b1b6a-2e9f-4915-946a-567dad0caaa4
[     3.669969] dk0 at sd0: "octeon-boot", 163840 blocks at 32768, type: ntfs
[     3.669969] dk1 at sd0: "octeon-root", 7626752 blocks at 196608, type: ffs
[     3.683879] WARNING: 1 error while detecting hardware; check system log.
[     3.691430] boot device: sd0
[     3.691430] root on dk1
[     3.709975] root file system type: ffs
[     3.719976] kern.module.path=/stand/evbmips/9.99.79/modules
[     3.719976] WARNING: no TOD clock present
[     3.729990] WARNING: using filesystem time
[     3.734057] WARNING: CHECK AND RESET THE DATE!

January 16, 2021

Unix Stack Exchange Why Does the NetBSD System Call Manual Mention the Standard C Library?

I'm reviewing man(2) pages on NetBSD 9, and have seen that all of the documents (write(2), open(2), pipe(2)) mention the Standard C Library at the top.

My understanding was that system calls were independent of library functions (such as those in libc). I don't see a similar mention in the Linux System call Manual. Does this mean that invoking these methods is calling some wrapper function included in libc, instead of directly calling a kernel function? Is this generally true, or just a feature of NetBSD?


January 07, 2021

The NetBSD Foundation pkgsrc-2020Q4 released

January 04, 2021

Benny Siegert SSD Rochade
My desktop PC has two NVMe drives, one for Windows and one for NetBSD. With Steam game footprints being what they are, the Windows one (256 GB) has been perpetually overfull, so it was time for something bigger. At the same time, I had bought an NVMe daughter board for my Pinebook Pro. My impression with the PBP is that the I/O performance of the eMMC module is holding it back, so I am excited to give it fast storage :)

December 25, 2020

Stack Overflow Which processor would execute hardware interrupt in a muticore system

In general hardware interrupts need to be processed immediately, at least so as to acknowledge it and do some first level of processing. As I understand this is not scheduled activity. Please correct me.

So the question is how to choose a processor that would actually execute this hardware interrupt handler?

One can answer this for Linux and/or BSD systems


December 21, 2020

NetBSD Blog Allen K. Briggs Memorial Scholarship

Allen Briggs was one of the earliest members of the NetBSD community, pursuing his interest in macBSD, and moving to become a NetBSD developer when the two projects merged. Allen was known for his quiet and relaxed manner, and always brought a keen wisdom with him; allied with his acute technical expertise, he was one of the most valued members of the NetBSD community.

He was a revered member of the NetBSD core team, and keenly involved in many aspects of its application; from working on ARM chips to helping architect many projects, Allen was renowned for his expertise. He was a distinguished engineer at Apple, and used his NetBSD expertise there to bring products to market.

Allen lived in Blacksburg Virginia with his wife and twin boys and was active with various community volunteer groups. His family touched the families of many other NetBSD developers and those friendships have endured beyond his passing.

We have received the following from Allen's family and decided to share it with the NetBSD community. If you can, we would ask you to consider contributing to his Memorial Scholarship.

https://www.ncssm.edu/donate/distance-education/allen-k-briggs-88-memorial-scholarship

The Allen K. Briggs Memorial Scholarship is an endowment to provide scholarships in perpetuity for summer programs at the North Carolina School of Science & Math, which Allen considered to be a place that fundamentally shaped him as a person. We would love to invite Allen's friends and colleagues from the BSD community to donate to this cause so that we can provide more scholarships to students with financial need each year. We are approximately halfway to our goal of $50K with aspirations to exceed that target and fund additional scholarships.

Two quick notes on donating: Important! When donating, you must select "Allen K. Briggs Memorial Scholarship" under designation for the donation to be routed to the scholarship If you have the option to use employer matching (i.e., donating to NCSSM through an employer portal to secure a match from your employer), please email the NCSSM Foundation's Director of Development, April Horton ([email protected]), after donating to let her know you want your gift and employer match to go to the Allen K. Briggs Memorial Scholarship Thanks in advance for your help. I'd be happy to answer any questions you or any others have about this.


December 05, 2020

UnitedBSD login.conf

Netbsd users, what's in your login.conf? Asking so I can get an idea of what I can do with it .

edited: and sysctl.conf as well. I'm trying to see what's possible with net and get the most out of my hardware

UnitedBSD Help Tuning netbsd

I just installed netbsd on an old desktop with an amd A6 and 8 gb of ram. Obviously it's an old cpu, but the performance on a recent debian install was much better. I've taken a cursory look at the docs to fine tune performance by editing kernel configs etc....

Anyone here have good advice on how to get the most of netbsd?


December 04, 2020

UnitedBSD exec dwm "libX11.so.6" not found

I try compile dwm (not in pkgsrc) in NetBSD and run it. I have my config.mk file look like this

PREFIX = /usr/local
MANPREFIX = ${PREFIX}/share/man
X11INC = /usr/pkg/include
X11LIB = /usr/pkg/lib
XINERAMALIBS = -lXinerama
XINERAMAFLAGS = -DXINERAMA
FREETYPELIBS = -lfontconfig -lXft
FREETYPEINC = /usr/pkg/include/freetype2
INCS = -I${X11INC} -I${FREETYPEINC}
LIBS = -L${X11LIB} -lX11 ${XINERAMALIBS} ${FREETYPELIBS}
CPPFLAGS = -D_DEFAULT_SOURCE -D_BSD_SOURCE -D_POSIX_C_SOURCE=2 -DVERSION=\"${VERSION}\"${XINERAMAFLAGS}
CFLAGS = -std=c99 -pedantic -Wall -Wno-deprecated-declarations -Os ${INCS} ${CPPFLAGS}
LDFLAGS = ${LIBS}
CC = cc

and my .xinitrc

exec dwm

But the output when I startx is:

dwm: Shared object "libX11.so.6 not found"
/usr/pkg/bin/xinit: connection to X server lost

What did I do wrong?


December 03, 2020

UnitedBSD Thinkpad install finalization

I'm working on a Thinkpad x230 which I've had for a long time. I love it, and NetBSD is working well. However, I can't seem to find good documentation about which pieces of software, or Kernel modules make things work.

I know these laptops see a lot of use, and I'm sure someone can point me in the right direction.

What I am looking for:

I want an icon to self populate onto the desktop which when clicked, opens a file manager window to an already automounted USB or SD card drive.

I'm still working to get all the pieces of FVWM working, but I like it so far. I've not ruled out going back to xfce, but I wanted to try something different.

Hopefully someone can point me in the right directions with some of this.

cheers!


December 02, 2020

UnitedBSD Can the aarch64 image file provided by netbsd run on rockpis?

Processor SoC RK3308
Quad Cortex-A35 ARM 64bits processor
frequency up to 1.3GHz
Memory 256MB or 512MB DDR3
Storage MicroSD(TF), optional on board 1/2/4/8Gb NAND flash
Wireless 802.11 b/g/n wifi
Bluetooth 4.0(rtl8723DS)
external antenna
USB USB2.0 Type-A HOST x1
USB3.0 Type-C OTG x1
Key maskrom x1
reset x1
Ethernet 100MB ethernet, RTL8201F,optional PoE(additional HAT requried)
IO 26-pin expansion header
I2C x4
PWM x3
SPI x2
UART x3
I2S0 x1
5V DC power in x2
3.3V DC power in x2
Others ---
Power USB Type-C DC 5V
Size 1.7inch square


November 25, 2020

OS News Before the BSD kernel starts

In this article, I will walk through the early kernel initialization process, defining the meaning of this term. System initialization is a broad topic that ranges from the platform’s hardware design all the way up to typical functions of an operating system such as handling I/O operations. It is not possible to cover the entire topic adequately within the scope of an article. In this first part I will describe the well-known AMD64: 64-bit platform. I am going to highlight a very interesting part of the initialization process the early initialization of the kernel. Later, I will compare it with ARM64. In both cases I will discuss the topic in the context of NetBSD, the operating system known for its portability.

Some light reading.


November 22, 2020

Unix Stack Exchange NetBSD sh -c "echo OK" doesn't give any output? [closed]

I'm testing the portability of some stuff I'm writing to BSD. It's working on Linux, FreeBSD, OpenBSD. It isn't working on NetBSD.

The following is on a fresh VM installation I've made just for the purpose of testing this. I've traced the issue to

NetBSD$ uname -a
NetBSD NetBSD.local 9.1 NetBSD 9.1 (GENERIC) #0: Sun Oct 18 19:24:30 UTC 2020 [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC amd64
NetBSD$ cat /etc/shells                                                                                 
#       $NetBSD: shells,v 1.3 1996/12/29 03:23:07 mrg Exp $
#
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells.

/bin/sh
/bin/csh
/bin/ksh
/usr/pkg/bin/zsh
/usr/pkg/bin/bash
NetBSD$ for s in /bin/sh /bin/csh /bin/ksh /usr/pkg/bin/zsh /usr/pkg/bin/bash ; do echo $s; $s -c "echo OK" ; done
/bin/sh
/bin/csh
OK
/bin/ksh
/usr/pkg/bin/zsh
OK
/usr/pkg/bin/bash
OK
NetBSD$ su -
Password:
NetBSD# for s in /bin/sh /bin/csh /bin/ksh /usr/pkg/bin/zsh /usr/pkg/bin/bash ; do echo $s; $s -c "echo OK" ; done
/bin/sh
OK
/bin/csh
OK
/bin/ksh
OK
/usr/pkg/bin/zsh
OK
/usr/pkg/bin/bash
OK

Why doesn't sh -c "echo OK" and ksh -c "echo OK" work when I'm a non-root user, and why do they work when I'm root?

Other shells (csh, zsh, bash) work correctly, as shown above.


November 08, 2020

Unix Stack Exchange How to install a torrent client on a NetBSD server with only SSH access [closed]

I have a remote server running NetBSD 6, to which I have access only via SSH. I have very limited Unix/Linux experience but I guess there are some command line torrent cliens available for Unixes like the ones in the BSD family. Could someone help install a torrent client only using SSH access onto a NetBSD 6 server?

Thank you!


October 21, 2020

NetBSD Blog NetBSD 9.1 released

After a small delay*, the NetBSD Project is pleased to announce NetBSD 9.1, the first feature and stability maintenance release of the netbsd-9 stable branch.

The new release features (among various other changes) many bug fixes, a few performance enhancements, stability improvements for ZFS and LFS and support for USB security keys in a mode easily usable in Firefox and other applications.

For more details and instructions see the 9.1 announcement.

Get NetBSD 9.1 from our CDN (provided by fastly) or one of the ftp mirrors.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, and other services may be found at https://www.NetBSD.org/mirrors/.

* for the delay: let us say there was a minor hickup and we took the opportunity to provide up to date timezone files for NetBSD users in Fiji.


October 20, 2020

The NetBSD Foundation NetBSD 9.1 released

October 19, 2020

NetBSD Blog Google Summer of Code 2020: [Final Report] Enhancing Syzkaller support for NetBSD
This report was written by Ayushu Sharma as part of Google Summer of Code 2020.

This post is a follow up of the first report and second report. Post summarizes the work done during the third and final coding period for the Google Summer of Code (GSoc’20) project - Enhance Syzkaller support for NetBSD

Sys2syz

Sys2syz would give an extra edge to Syzkaller for NetBSD. It has a potential of efficiently automating the conversion of syscall definitions to syzkaller’s grammar. This can aid in increasing the number of syscalls covered by Syzkaller significantly with the minimum possibility of manual errors. Let’s delve into its internals.

A peek into Syz2syz Internals

This tool parses the source code of device drivers present in C to a format which is compatible with grammar customized for syzkaller. Here, we try to cull the details of the target device by compiling, and then collocate the details with our python code. For further details about proposed design for the tool, refer to previous post.

Python code follows 4 major steps:

Extraction:

This step involves fetching the possible ioctl commands for the target device driver and getting the files which have to be included in our dev_target.txt file. We have already seen all the commands for device drivers are defined in a specific way. These commands defined in the header files need to be grepped along with the major details, regex comes in as a rescue for this


	io = re.compile("#define\s+(.*)\s+_IO\((.*)\).*")
	iow = re.compile("#define\s+(.*)\s+_IOW\((.*),\s+(.*),\s+(.*)\).*")
	ior = re.compile("#define\s+(.*)\s+_IOR\((.*),\s+(.*),\s+(.*)\).*")
	iowr = re.compile("#define\s+(.*)\s+_IOWR\((.*),\s+(.*),\s+(.*)\).*")

Code scans through all the header files present in the target device folder and extracts all the commands along with their details using compiled regex expressions. Details include the direction of buffer(null, in, out, inout) based on the types of Ioctl calls(_IO, _IOR, _IOW, _IOWR) and the argument of the call. These are stored in a file named ioctl_commands.txt at location out/&lttarget_name&gt. Example output:


out, I2C_IOCTL_EXEC, i2c_ioctl_exec_t

Preprocessing:

Preprocessing is required for getting XML files, about which we would look in the next step. Bear plays a major role when it comes to preprocessing C files. It records the commands executed for building the target device driver. This step is performed when setup.sh script is executed.

Extracted commands are modified with the help of parse_commands() function to include ‘-E’ and ‘-fdirectives’ flags and give it a new output location. Commands extracted by this function are then used by the compile_target function which filters out the unnecessary flags and generates preprocessed files in our output directory.

Generating XML files

Run C2xml on the preprocessed files to fetch XML files which stores source code in a tree-like structure, making it easier to collect all the information related to each and every element of structures, unions etc. For eg:


	&ltsymbol type="struct" id="_5970" file="am2315.i" start-line="13240" start-col="16" end-line="13244" end-col="11" bit-size="96" alignment="4" offset="0">
		&ltsymbol type="node" id="_5971" ident="ipending" file="am2315.i" start-line="13241" start-col="33" end-line="13241" end-col="41" bit-size="32" alignment="4" offset="0" base-type-builtin="unsigned int"/<
		&ltsymbol type="node" id="_5972" ident="ilevel" file="am2315.i" start-line="13242" start-col="33" end-line="13242" end-col="39" bit-size="32" alignment="4" offset="4" base-type-builtin="int"/>
		&ltsymbol type="node" id="_5973" ident="imasked" file="am2315.i" start-line="13243" start-col="33" end-line="13243" end-col="40" bit-size="32" alignment="4" offset="8" base-type-builtin="unsigned int"/>
	</symbol>
	&ltsymbol type="pointer" id="_5976" file="am2315.i" start-line="13249" start-col="14" end-line="13249" end-col="25" bit-size="64" alignment="8" offset="0" base-type-builtin="void"/>
	&ltsymbol type="array" id="_5978" file="am2315.i" start-line="13250" start-col="33" end-line="13250" end-col="39" bit-size="288" alignment="4" offset="0" base-type-builtin="unsigned int" array-size="9"/>

We would further see how attributes like - idents, id, type, base-type-builtin etc conveniently helps us to analyze code and generate descriptions in a trouble-free manner .

Descriptions.py

Final part, which offers a txt file storing all the required descriptions as its output. Here, information from the xml files and ioctl_commands.txt are combined together to generate descriptions of ioctl commands and their arguments.

Xml files for the given target device are parsed to form trees,


for file in (os.listdir(self.target)):
	tree = ET.parse(self.target+file)
	self.trees.append(tree)

We then traverse through these trees to search for the arguments of a particular ioctl command (particularly _IOR, _IOW, _IOWR commands) by the name of the argument. Once an element with the same value for ident attribute is found, attributes of the element are further examined to get its type. Possible types for these arguments are - struct, union, enum, function, array, pointer, macro and node. Using the type information we determine the way to define the element in accordance with syzkaller’s grammar syntax.

Building structs and unions involves defining their elements too, XML makes it easier. Program analyses each and every element which is a child of the root (struct/union) and generates its definitions. A dictionary helps in tracking the structs/unions which have been already built. Later, the dictionary is used to pretty print all the structs and union in the output file. Here is a code snippet which depicts the approach


            name = child.get("ident")
            if name not in self.structs_and_unions.keys():
                elements = {}
                for element in child:
                    elem_type = self.get_type(element)
                    elem_ident = element.get("ident")
                    if elem_type == None:
                        elem_type = element.get("type") 
                    elements[element.get("ident")] = elem_type

                element_str = ""
                for element in elements: 
                    element_str += element + "\t" + elements[element] + "\n"
                self.structs_and_unions[name] = " {\n" + element_str + "}\n"
            return str(name)

Task of creating descriptions for arrays is made simpler due to the attribute - `array-size`. When it comes to dealing with pointers, syzkaller needs the user to fill in the direction of the pointer. This has already been taken care of while analyzing the ioctl commands in Extractor.py. The second argument with in/out/inout as its possible value depends on ‘fun’ macros - _IOR, _IOW, _IOWR respectively.

There is another category named as nodes which can be distinguished using the base-type-builtin and base-type attributes.

Result

Once the setup script for sys2syz is executed, sys2syz can be used for a certain target_device file by executing the python wrapper script (sys2syz.py) with :

#bin/sh
python sys2syz.py -t &ltabsolute_path_to_device_driver_source> -c compile_commands.json -v

This would generate a dev_&ltdevice_driver&gt.txt file in the out directory. An example description file autogenerated by sys2syz for i2c device driver.


#Autogenerated by sys2syz
include 

resource fd_i2c[fd]

syz_open_dev$I2C(dev ptr[in, string["/dev/i2c"]], id intptr, flags flags[open_flags]) fd_i2c

ioctl$I2C_IOCTL_EXEC(fd fd_i2c, cmd const[I2C_IOCTL_EXEC], arg ptr[out, i2c_ioctl_exec])

i2c_ioctl_exec {
iie_op	flags[i2c_op_t_flags]
iie_addr	int16
iie_buflen	len[iie_buf, intptr]
iie_buf	buffer[out]
iie_cmdlen	len[iie_cmd, intptr]
iie_cmd	buffer[out]
}

Future Work

Though we have a basic working structure of this tool, yet a lot has to be worked upon for leveling it up to make the best of it. Perfect goals would be met when there would be least of manual labor needed. Sys2syz still looks forward to automating the detection of macros used by the flag types in syzkaller. List of to-dos also includes extending syzkaller’s support for generation of description of syscalls.

Some other yet-to-be-done tasks include-

Summary

We have surely reached closer to our goals but the project needs active involvement and incremental updates to scale it up to its full potential. Looking forward to much more learning and making more contribution to NetBSD community.

Atlast, a word of thanks to my mentors William Coldwell, Siddharth Muralee, Santhosh Raju and Kamil Rytarowski as well as the NetBSD organization for being extremely supportive. Also, I owe a big thanks to Google for giving me such a glaring opportunity to work on this project.


October 17, 2020

Unix Stack Exchange Does *BSD have the ability to encrypt a system partition with full disk encryption?

Does FreeBSD, NetBSD, or OpenBSD have an encryption feature like Linux's dm-crypt? And will it work for a system partition?


October 13, 2020

The NetBSD Foundation New Security Advisory: NetBSD-SA2020-003