NetBSD Planet

Michael Larabel reported in Phoronix:

GCC 15 had planned to remove Itanium IA-64 support to close the book on that Intel architecture. That GCC move followed the Linux kernel removing Itanium support last year and more distributions ending Itanium support although many did so years ago. But open-source developer René Rebe stepped up and wants to maintain Itanium’s IA-64 support in the GNU Compiler Collection.

I don’t use GCC, but this sort of thing makes me happy in the way all of NetBSD’s ports do. Good on you, René! It’s one thing to run contemporaneous software and period-correct hardware on legacy machines, but it’s another to introduce them into the future. (It’s also a bit of a commentary about how wasteful modern tech is with resources, but that’s a separate discussion).

Weirdly enough, I did actually mess with Itanium briefly when I was in high school. I remember being hamgstrung by actually being able to boot things, and being frustrated at the limited software support. Little did I know how accurate those (albeit unremarkable) observations were to become.

Part of me does wonder how different the industry would look if everyone followed Itanium instead of amd64. Maybe we would have at least had some more architectural diversity.

By Ruben Schade in Sydney, 2024-10-12.

Speaking of homelabs, I’m looking forward to building a small cluster in Clara’s and my upcoming 25 RU home server rack. I’ve got used to Xen and Bhyve for one-off or mirrored hypervisors, but I’ve only ever used large clusters at work. The plan would be to use this to test distributed storage, RDMA, hypervisor failovers, and tooling like Proxmox, XCP-NG, and K8s that clients at work are migrating from, as well as more Xen, Bhyve, and NVMM testing.

I learn from tinkering, so this would be first and foremost an educational tool I could spin up, tinker with, and tear down without worrying about breaking something. It would be discrete and separate from our FreeBSD and NetBSD homelab “prod” workloads, such as Jellyfin, Minecraft, PostgreSQL, OpenZFS, and our Wireguard VPN that I can’t afford to break.

There are a few options, with pros and cons for each.

Option 1: 2U servers

As mentioned previously, rack-mountable servers aren’t ideal for a homelab owing to their depth and noise; attributes that often miss those people who offer blanket recommendations of “just get an R620!” You can often hack them with PWM fans and separate controllers, but their length also extends farther than the space Clara and I have designated for the depth-adjustable rack.

But I have a few spare boards lying around I could use inside a svelte 2U SilverStone cases and some quiet low-profile Noctua fans. While they’d probably use the most power of all the options here, I could always slowly upgrade the power supplies to more efficient units, and be more aggresive with suspend/resume.

Option B: “Tiny” PCs

Various manufacturers make PCs in a “tiny” form factor (like Clara, cough). While they can’t hold a candle to a modern Apple Silicon Mac Mini in performance, they can be picked up for less than AU$50. Some examples include the Lenovo ThinkCentre, Dell OptiPlex, and HP ProDesk. I’ve also seen Fujitsu and Panasonic units in Japan.

You can mount them in 3D-printed cases, or Jeff Geerling has recommended MyElectronics in the Netherlands (whether they come with stroopwafels is unknown). Upgrading might be a pain, but they’d be small, quiet, and relatively energy efficient.

Option 三: Pi cluster or equivalent

The Raspberry Pi has spawned a world of single-board computers in the same form factor, often with the same standoffs. For example, the Pine Rock64 for an alternative ARM option, the UP 7000 for Intel, and the Milk-V Mars for RISC-V. I’m extremely keen to try all these out.

Like the Clara-sized PCs (cough), these SBCs can be mounted in a rack with a 3D-printed chassis, or you can fit more stacked vertically in a 2U unit. Pair these with a PoE switch in the RU above or below it, and you’d have an efficient, small, dense cluster for testing. Though it’s interesting that the tiny PCs have a lower upfront cost.

Again, Jeff Geerling recommended a fine rack from MyElectronics, though he’s also since recommended some fancier units with a full enclosure and active cooling.

Option δ: Dense backplane of compute modules

This is the option I’m most intrigued by, because I’ve never used them before. The Turing Pi CM3 is probably one of the more well-known units, which is tiny when paired with a 9V barrel jack and/or a picoPSU. I’d just need to find a chassis to mount it into a server rack.

As for the modules, there’s the popular Raspberry Pi Compute Module, though friends of mine speak highly of the SOPINE among others. My concern would be getting something similar for x64 too, if such a thing exists.

Conclusion

I’d probably be leaning towards the tiny PCs, just because I have a local source going for extremely cheap right now. The idea of a SBC RISC-V cluster does excite me in a way that’s hard to describe though.

Who’d have thought The VM Guy would be sharding out stuff to multiple physical machines again? It’s HA!

By Ruben Schade in Sydney, 2024-10-08.

Dear NetBSD Community,

I am considering gradually migrating all of my systems from Linux to NetBSD, drawn by the simplicity and philosophy of NetBSD. However, I have one concern regarding the default filesystem, FFS2. How resilient is this filesystem in the event of power outages?

Some of my servers will be located in a remote area in the central mountains of Spain, where power outages are occasionally caused by storms. Any insights or recommendations on handling these scenarios would be greatly appreciated.

Thank you in advance for your advice.

I’ve redacted the source now, but it’s still pretty funny:

dumbfunk low level netbsd admin guy with a silly fedora and a shitty blog thinks he is [sic] hotshot

Recognition was a long-time coming, but I’m flattered! 🧡

You can view related posts with the NetBSD and BSD tags. Give it a try if you’ve only ever used Linux or other BSDs; there’s a lot to like.

Update: A few of you have expressed concern. Don’t worry, it isn’t half as bad as 90% of the comments I get when I’m featured on Hacker News!

By Ruben Schade in Sydney, 2024-10-06.

Hey, remember Crowdstrike? That update to Windows security software that affected airports, hospitals, supermarkets, schools… and clients who called me desperate to know why their VMs had vanished? It was fun! By which I mean it was an unmitigated disaster, and the ultimate example of a technical externality bourne by everyone.

I haven’t seen much of a postmortem discussion on this; certainly not to the same extent as the wall-to-wall coverage those BSODs garnered. I suspect the press got their story, social media got their memes, and we all went on with our lives. Well, until the next one hits.

Of the people who have been talking about it, the discussion has centred around the companies involved, the processes that lead to the faulty patch being released, the economic and social fallout that such a massive, global issue can cause, and some ironic comparisons being made between malicious actors and security vendors. Why take your actor down with an evil ploy, when they’ll do it themselves by accident?

I’m surprised though that the biggest issue was largely unacknowledged (save for some commentary about how appropriate it is to run Windows in an airport). The scale of the problem could only occur because of how homogeneous our global IT infrastructure has become. The ubiquity of one vendor’s tech all but guarantees such issues are global in scope and impact. That fact alone should be ringing alarm bells.

Take this example of the TOP500 supercomputers by processor family, as visualised by Moxfyre on Wikimedia Commons. Granted we can’t directly extrapolate this to general web servers or end-user devices like workstations and phones, let alone software. Here comes the proverbial posterior prognostication, but… we can broadly see similar trends everywhere.

I talked again last week about browser monocultures, but the same issue is happening across the industry, from consumer operating systems and software to industrial control equipment and servers. Granted we have ARM (and to a far lesser extent RISC-V) offering an alternative, but even then we’re typically running similar software and platforms across these architectures.

I miss the days when there was more technical diversity in the industry purely from the silly perspectives of fun and novelty. But it’s also a security issue. We intuitively understand how important genetic diversity is in nature for the viability and robustness of ecosystems, and I’d argue it’s just as important here. It’s ridiculous that a single software patch can bring down that many systems.

Standards are great, but we need more implementations! Icanhaz a modern SPARC64 server running NetBSD?

By Ruben Schade in Sydney, 2024-10-05.

It’s fascinating to me how scale affects perceptions.

At work we regularly buy dozens of 2U servers, loaded to the gills with DDR5 memory, dual CPUs with more cores than all the computers in my home combined, and terabytes of SSD storage. When a client asks for a hybrid cloud, or a private bare metal fleet for their own hypervisor, I spec out similar machines and provide quotes.

Occasionally I have moments of introspection, normally after we’ve received a shipment and I’m standing in the data centre surrounded by cardboard boxes, pallets of servers, and kilometres of power, Ethernet, and InfiniBand cables splayed out across all the tables. At that point I think huh, that’s a lot. But then we get right back to work.

By comparison, is a phrase with two words. For Clara’s and my homelab upgrade, we’ll pour over specification sheets, individual drives, and look to second-hand markets for deals for months. We always buy previous-generation kit so we’re not beta testers, but also to save money.

And boy does everything change at that scale! For example:

• I can’t justify the price of hot spares like we have at work, so I’ll spend more time digging into exactly what will get us the best balance of performance, cost, and data integrity (ala OpenZFS).

• I’ll check if it’s easily upgradeable and expandable, and how much I can get away with deferring.

• Certain considerations like noise aren’t a factor in data centre kit, but definitely at home. Sure this 4U chassis doesn’t hold anything more than this 2U Supermicro box, but the Noctua fans in the former are far preferable sitting next to me!

• We know exactly what each server in a fleet will be running for its operational life at work, but at home I hedge my bets by sourcing components with the widest compatibility possible. Could it boot FreeBSD? Would this 10G card work with NetBSD? What about if Debian or Alpine ends up on it?

• I’ll forgo niceties like backplanes and drive sleds if it means I can put more money into the drives themselves (then cross my fingers I don’t have to open up the machine to remove a failed drive that often).

It’s obvious why; this stuff gets expensive fast. But it’s still funny how my brain compartmentalises these sorts of things.

By Ruben Schade in Sydney, 2024-10-05.

I have not been at EuroBSDCon for a while, unfortunately! My last EuroBSDCon was EuroBSDcon 2017 in Paris, France (and I have also blogged about it)!

I was very excited to come back to EuroBSDCon. Meet again in person with people. Talk in the "hall track"... and, why not!, have some fun and do some shenanigans in the nights! :)

And... definitely it was very nice, instructive and fun!

I have not fully unpacked the bag but it's time to share some notes!

Friday (20/09): arriving in Dublin

I arrived in Dublin on Friday afternoon. After some sightseeing on foot I got lost in the paintings of the National Gallery of Ireland.

I then spent the rest of the evening and night in Porterhouse Temple Bar. I had a tasty soup and garlic bread and several delicious craft beers!

Saturday (21/09): 1st day of conference talks and social event

My hotel was a 40 minutes walk from University College Dublin (UCD). I arrived a bit early for the registration. I then met some other NetBSD folks that I had missed in person since 2018 and met new ones.

View from O'Reilly Hall, University College Dublin.

After the Opening Session that welcomed us, the conference started with the opening keynote Evidence based Policy formation in the EU what Evidence are we Presenting to the EU? by Tom Smyth. Tom Smyth shared his experience on evidence based policy formation in the European Union from a point of a relatively small ISP. EU is open to feedback and as a BSD community we can shape and influence policies.

Flipping Bits: Memory Errors in the Machine, Taylor R Campbell

Taylor talked about bit flips, the memory errors in the machine.

Memory errors caught in the act: corruption of a filename in Riastradh's local machine.

He started sharing a catch of bit flip in a filename corruption on his local machine in NetBSD src repository. A bit flipped and that resulted from external/gpl3/gdb/dist/gdb/testsuite/gdb.linespec/cpls.cc to e\370ternal/gpl3/gdb/dist/gdb/testsuite/gdb.linespec/cpls.cc (In ASCII lower case x is \170 that is 01111000 in binary, while \370 is 11111000, the most significant bit got flipped!).

He also opened several PRs - due to several experienced kernel panics mostly in ZFS - before he realized that it was bad RAM.

As part of the talk a lot of fundamentals concepts and theory behind Error Detection And Correction (EDAC), causes of memory errors, where memory errors can happen, error severity and error persistence were shared.

Taylor then talked and digged in ACPI Platform Error Interface (APEI) that is the standard interface in ACPI that abstract EDAC device registers.

In NetBSD APEI is supported by the apei(4) driver.

The apei(4) driver also exposes a sysctl interface to APEI EINJ (Error INJection) that permit to also inject errors. Using such interface Riastradh live demoed that and trigger a memory error that was corrected and reported by apei(4)!

Riastradh live demoing a memory error using APEI EINJ via apei(4).

The talk was great and super-interesting. Memory errors are also pretty common. Taylor also shared a lot of anecdotes and that make his talk even more fun and interesting!

An introduction to GPIO in RPi3B+ and NetBSD, building a wind-speed logger as an application, Dr. Nicola Mingotti

Dr. Nicola Mingotti talk was a great introduction (and more) to Generalized Pin Input Output (GPIO)!

He started really from the start by populating a uSD card and installing and configuring NetBSD on a Raspberry Pi 3 Model B+.

He then introduced GPIO, how the RPi3B+ pin maps to the GPIO number and then we were ready to get our hands on GPIO!

As first exercises he showed how to set a PIN state (on/off) and read a PIN state via gpioctl(8). This can be used respectively to turn a LED on/off and to read the state of a switch.

The second series of exercises looked on how fast gpioctl can be. This is limited for several applications and so Nicola introduced how to write and read pin states in C via ioctl(2). This is much faster and with that we can go from switches to square waves!

To avoid bit-banging and polling respectively gpiopwm(4) and gpioirq(4) can be used. Nicola shared several applications of them, like blinking LED and loopback. (Another possible application, left as an exercise to the reader is the "daemon toggler". The "daemon toggler" starts/stops a daemon (e.g. ntpd(8)) based on the state of a physical switch!)

He then shared a much bigger application a Wind-Speed Logger (AKA WSL). This was used by Nicola in order to evaluate if wind turbines could be installed or not. He also shared how he adjusted an RPi case and built housing for it (the RPi will be outside, needs to cool off so needs some ventilation but at the same time the housing should block rain!)

Nicola showing the sensor used to build the Wind-Speed Logger (WSL).

He concluded the talk on why he used NetBSD.

The talk was really educational. Nicola did a great job in summarizing and providing a lot of references. If you are more interested I suggest to catch up with the video recordings, slides and try to do the exercises in it!

"Hall track" and preparing for the social event

After Nicola's talk I have spent some time in the "hall track" talking with other people and missed a couple of talks (recording should be available so I will hopefully catch up!).

I have then attended Stefano Marinelli's talk Why (and how) we're migrating many of our servers from Linux to the BSDs.

Stefano shared his more than 2 decades old experience with BSD systems and how he made his passion his profession.

He shared his philosophy, experience with clients and why it is important to focus on solving problems.

During the talk he shared also several interesting stories with clients. In one of them to avoid possible bias on BSD systems he migrated client hosts without informing them. A client called alarmed because he noticed a massive performance boost!

His talk was inspiring and you can find more in his I Solve Problems blog post.

After Stefano's talk we gathered to join the social event and took a DART train (Dublin Area Rapid Transit).

Social event: BrewDog Dublin Outpost

The social event was in BrewDog Dublin Outpost.

We were in an area dedicated to EuroBSDCon participants so that we can eat, drink and talk. There was a buffet and we received tickets to grab beers.

Several folks gifted me an handful and I have definitely had a pretty ample beer tasting experience too! :)

I also had a Vegan Spicy Meaty pizza: a pizza with seitan, mushrooms, chilli flakes, fresh red chilli, tomatoes and vegan mozzarella. My italian-pizza-side is usally pretty orthodox and I usually go for a pizza marinara! :) But overall that was actually pretty nice and I really appreciated the topping!

I have staid with a couple of folks until the closure. With Christoph Badura (<bad@>) we walked in the desperate search of grabbing some more food. However, at the end we ended up in The Temple Bar Pub for "only another beer"! We met with some friendly Swedish and Swiss tourists and we started talking about BSD systems at 2:00 AM! The weather was pretty nice (it was always pretty cloudy but there was no rain for the entire conference) and we decided to continue walking back to our hotels. At the end we have walked for a bit less than 9 kilometers from Temple Bar to nearly Booterstown! That was a great walk though and definitely we had no traces of hangovers in the morning! :)

Sunday (22/09): 2nd day of conference talks

I wake up a bit late on Sunday and arrived in UCD at around 12:00 and staid until lunch in the "hall track".

For lunch the vegetarian dish was a vegetarian curry, pretty tasty!

On Sunday we had a longer lunch break also to take a family photo.

EuroBSDCon 2024 family picture. You can find more EuroBSDCon photographs taken by Ollivier Robert at EuroBSDCon 2024 - Dublin, Ireland album.

After lunch I have attended FreeBSD at 30 Years: Its Secrets to Success by Kirk McKusick. In this talk Kirk looked back at 30 years of FreeBSD history (and also more for BSD years!) and what made its success. He talked about a lot of different topics, including leadership, development, importance of adopting ideas and codes from NetBSD and OpenBSD, communication, documentation and project culture. He also shared several interesting statistics and demographic about FreeBSD.

I have then attended Confidential Computing with OpenBSD by Hans-Jörg Höxer. Hans-Jörg introduced concepts about confidential computing, the threat model that it cover and then digged in AMD Secure Encrypted Virtualization (SEV) and how he is using that in OpenBSD vmm(4).

Then I have attended Building an open native FreeBSD CI system from scratch with lua, C, jails & zfs by Dave Cottlehuber. In this talk Dave shared the design and implementation of a Continuous Integration (CI) system focused on FreeBSD technologies but that can be ported also to other BSDs.

The final talk I have attended was SIMD-enhanced libc string functions: how it's done by Robert Clausecker and Getz Mikalsen. In this talk Robert shared how several libc string functions were reimplemented in other to use SIMD techniques on amd64 and arm64. Getz worked on porting such work on arm64 as part of Google Summer of Code 2024 and he shared his work and challenges in porting that. The talk was interesting and micro-benchmarking showed performance increase by factor of 5 on average!

Then I have joined the Closing Session.

There was a wrap up of the conference and some stats about it.

And *drumrolls* the next EuroBSDCon location was announced! EuroBSDCon 2025 will be in Zagreb, Croatia!

After the Closing Session with other NetBSD folks we met again for one last dinner. We met with Andy Doran (<ad@>) and we had some junk food and several beers.

Conclusion

I had not traveled a lot in the last years and I have missed several EuroBSDCon-s and I really regret that! EuroBSDCon 2024 was great: very interesting talks, friendly folks and it was some time that I did not had so much fun!

Dublin was also really nice. All the locals were also very friendly. I hope to come back to both Dublin and Ireland to do some much more sightseeing in a more relaxed pace. Enjoy food, beers, drinks and more. Talk with locals.

I would like to thanks a lot to all the EuroBSDCon organizers for the amazing conference!

I also would like to thanks The NetBSD Foundation that funded my EuroBSDCon registration.

If you have never been to EuroBSDCon and you are curious about BSDs... I strongly suggest to attend either as participant or speaker! Folks are super-friendly, there are a lot of interesting tutorials and talks and I'm pretty sure you will have fun too!

And... if you are still reading until here... thank you too! :)

pkgsrc for 2024Q3 has been released

Fixed Link: https://mail-index.netbsd.org/pkgsrc-users/2024/09/30/msg040317.html
Thanks Jay

I just finished updating to pkgsrc 2024Q3, no issues occurred on 10.0 amd64. Note, I use binary packages.

Thanks to the pkgsrc people for a great release!

This report was written by Emmanuel Nyarko as part of Google Summer of Code 2024.

Alternate Queuing has been of great need in the high Performance Computing space since the continuous records of unfair disruption in network quality due to the buffer bloat problem. The buffer bloat problem still persists and not completely gone but modern active queue managements have been introduced to improve the performance of networks.

ALTQ was refactored to basically improve maintainability. Duplicates were handled, some compile time errors were fixed and also performance has been improved too.

This improves the quality of developer experience on maintaining the ALTQ codebase.

The Controlled Delay (CoDel) active queue management has also been integrated into the netbsd codebase. This introduces improvements made in the area of quality of service in the netbsd operating system. CoDel was a research led collaborative work by Van Jacobness and Kathleen Nichols which was developed to manage queues under control of the minimum delay experienced by packets in the running buffer window.

As it stands now, ALTQ in NetBSD is integrated in PF packet filter. I am currently working to integrate it in the NPF packet filter. The code in NetBSD is on the constant pursuit to produce clean and maintainable code.

I'll also be working to improve quality of service in NetBSD through quality and collaborative research driven by randomness in results. As a research computer scientist, I will be working to propose new active queue managements for the NetBSD operating system to completely defeat the long lasting buffer bloat problem.

More details of the work can be found in my Google Summer of Code 2024 work submission.

I am trying to install, for the first time, NetBSD-10.0-amd64 on HP EliteBook 8570w. After choosing "Install BSD" option installer goes to the booting mode and gets interrupted with an error:

502c0
Stopped in pid 294.294 (init) at            netbsd:breakpoint+0x5: leave
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x183
panic() at netbsd:panic+0x3c
cnopen() at netbsd:cnopen+0x104
cdev_open() at netbsd:cdev_open+0x12a
spec_open() at netbsd:spec_open+0x1e0
VOP_OPEN() at netbsd:VOP_OPEN+0x3e
vn_open() at netbsd:vn_open+0x2ec
do_open() at netbsd:do_open+0xc3
do_sys_openat() at netbsd:do_sys_openat+0x74
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1fc
--- syscall (number 5) ---
netbsd:syscall+0x1fc
ds            8
es            2
fs            180
gs            4a80
rdi           0
rsi           ffffffff81d88000
rsi           ffffbe8345a54ad0
rbx           0
rdx           1
rcx           ffffffffffffff
rax           800000000000000
r8            0
r9            0
r10           ffffffff818450e0      x86_mem
r11           fffffffe
r12           ffffffff8139af6f      ostype+0x13aa
r13           ffffbe8345a54b18
r14           104
r15           ffff8046d2cbdbc0
rip           ffffffff80235385      breakpoint+0x5
cs            8
rflags        202
rsp           ffffbe8345a54ad0
ss            10
netbsd:breakpoint +0x5: leave

If I continue the system precedes to reboot.

I'm not familiar with bsd tools so please tell me if I left important information.

Notice

I installed FreeBSD and Arch Linux on the same machine and it worked fine.

After several sessions with intense Google searching and trying several angles with ChatGPT, I seem to be at a dead-end, my problem arises when I try to build OpenSSL from source, it seems that the build process wants to link with libcrypto.so located in /lib, but the system supplied version of OpenSSL is ancient, so this fails miserably, since OpenSSL now includes functionality not present in my version of libcrypto.so, specifically QUIC, its failing on safe_muldiv_uint64_t. It seems like a catch-22, and I have absolutely no idea how to break out of this.

Some of the suggestions I have found involved building OpenSSL in a chroot jail, but I think it seems a little excessive?

So I guess my question is: How do I build OpenSSL without linking with /lib/libcrypto.so, but linking with the version of libcrypto from the source package?

Output:

${LDCMD:-cc} -pthread -Wa,--noexecstack -O2 -O3 -pipe -I/usr/include -I/usr/pkg/include  -L/usr/local/lib -L/usr/pkg/gcc7/lib/gcc/x86_64--netbsd/7.5.0 -Wl,-R/usr/pkg/gcc7/lib/gcc/x86_64--netbsd/7.5.0 -Wl,-zrelro -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib \
        -o fuzz/quic-srtm-test \
        fuzz/quic-srtm-test-bin-fuzz_rand.o \
        fuzz/quic-srtm-test-bin-quic-srtm.o \
        fuzz/quic-srtm-test-bin-test-corpus.o \
        libssl.a libcrypto.a -pthread
...
apps/libapps.a -lssl -lcrypto -pthread
./libssl.so: undefined reference to safe_muldiv_uint64_t
./libssl.so: undefined reference to safe_mul_uint64_t

-L/usr/lib is specified in the above command executed by the Makefile, and that folder contains libcrypto.o and libssl.o from the system supplied version of OpenSSL,

Hello everyone. Does anybody know is there a guide on the Internet about OpenBSD installation via shell and without the installer program? The only ones I could find on the web were NetBSD chroot installation guides. One by JuvenalUrbino (on this forum) and a couple of others. I tried somehow to follow these guides while installing OpenBSD and I failed miserably. The reason why I am asking this is because of my eagerness to learn OpenBSD.

Hello,

Is there support to use USB tethering iPhone in NetBSD?
Does this also work in NetBSD?

https://forums.freebsd.org/threads/howto-iphone-internet-connection-sharing-via-usb.19995/

Hello,

I am encountering issues while executing the [CPython](
https://github.com/python/cpython/issues/124108) test_locale test suite on NetBSD 10.0, particularly with the tests related to the strcoll and strxfrm functions.

localhost$ ./python -m test test_locale -m test_strcoll_with_diacritic -m test_strxfrm_with_diacritic -v
== CPython 3.14.0a0 (heads/main:401fff7423c, Sep 15 2024, 21:20:00) [GCC 10.5.0]
== NetBSD-10.0-amd64-x86_64-64bit-ELF little-endian
== Python build: debug
== cwd: /home/blue/cpython/build/test_python_worker_28241æ
== CPU count: 6
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 1742921980
0:00:00 load avg: 0.18 Run 1 test sequentially in a single process
0:00:00 load avg: 0.18 [1/1] test_locale
test_strcoll_with_diacritic (test.test_locale.TestEnUSCollation.test_strcoll_with_diacritic) ... testing with 'en_US.UTF-8'... FAIL
test_strxfrm_with_diacritic (test.test_locale.TestEnUSCollation.test_strxfrm_with_diacritic) ... testing with 'en_US.UTF-8'... FAIL

======================================================================
FAIL: test_strcoll_with_diacritic (test.test_locale.TestEnUSCollation.test_strcoll_with_diacritic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_locale.py", line 359, in test_strcoll_with_diacritic
    self.assertLess(locale.strcoll('à', 'b'), 0)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 126 not less than 0

======================================================================
FAIL: test_strxfrm_with_diacritic (test.test_locale.TestEnUSCollation.test_strxfrm_with_diacritic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_locale.py", line 368, in test_strxfrm_with_diacritic
    self.assertLess(locale.strxfrm('à'), locale.strxfrm('b'))
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'à' not less than 'b'

----------------------------------------------------------------------
Ran 2 tests in 0.013s

FAILED (failures=2)
test test_locale failed
test_locale failed (2 failures)

== Tests result: FAILURE ==

1 test failed:
    test_locale

Total duration: 266 ms
Total tests: run=2 (filtered) failures=2
Total test files: run=1/1 (filtered) failed=1
Result: FAILURE
localhost$

Environment Variables:
I have configured the following locale-related environment variables in my .profile file:

export LANG="C.UTF-8"
export LC_CTYPE="C.UTF-8"
export LC_COLLATE="C.UTF-8"
export LC_TIME="C.UTF-8"
export LC_NUMERIC="C.UTF-8"
export LC_MONETARY="C.UTF-8"
export LC_MESSAGES="C.UTF-8"

Despite these settings, the tests continue to fail.

I have been unable to find specific documentation in the NetBSD resources regarding the configuration of locale-related environment variables, particularly LC_COLLATE. Could you provide any suggestions or guidance on how to properly configure locale settings for accurate testing on NetBSD?

Thank you very much for your assistance.

For certain use cases, it’s advisable to set up a read-only root file system, which ensures better reliability in case of system issues. Think of scenarios like a router (critical for network access) or a caching reverse-proxy, such as the one described in my series “Make your own CDN“.

While FreeBSD natively supports this configuration and some Linux distributions offer custom solutions (e.g., Alpine Linux), NetBSD stands out as an excellent choice for such devices. It supports nearly all embedded devices, is lightweight, and its stability minimizes the need for frequent updates.

↫ Stefano Marinelli

Exactly what it says on the tin. Friend of the website (a new term I just made up and will use from here on out for some people) Stefano Marinelli, fresh from his series about making your own CDN using the various BSDs, explains how to set up a NetBSD system with a read-only root filesystem for the special use cases where this makes sense.

After covering setting up your own CDN with both FreeBSD and OpenBSD, it’s now time to learn how to set up your own CDN wit NetBSD.

This article is a spin-off from a previous post on how to create a self-hosted CDN, but this time we’ll focus on using NetBSD. NetBSD is a lightweight, stable, and secure operating system that supports a wide range of hardware, making it an excellent choice for a caching reverse proxy. Devices that other operating systems may soon abandon, such as early Raspberry Pi models or i386 architecture, are still fully supported by NetBSD and will continue to be so. Additionally, NetBSD is an outstanding platform for virtualization (using Xen or qemu/nvmm) and deserves more attention than it currently receives.

↫ Stefano Marinelli

All the same from my previous post still applies, and it’s a great thing that Marinelli covers all three of the major BSDs (so far). If you want to run your own CDN on BSD, you can now make a pretty informed decision on which BSD best suits your needs.

Thanks to open source, no technology ever has to become obsolete, so long as a community remains to support it. You can sync Newtons and Palm Pilots with modern desktops, download web browsers for long-discontinued operating systems, or connect vintage computers like the Apple IIe to the modern internet via WiFi. Every year, new cartridges are released for old-school video game consoles like the Nintendo Entertainment System and Game Boy.

People keep old software and online platforms alive as well. The Dreamwidth team forked an old version of the early social network LiveJournal’s source code and built a community around it. The dial-up bulletin board system software WWIV is still maintained and there are plenty of BBSes still around. Teams are working to restore aspects of early online services like AOL and Prodigy. And you can still use Gopher, the hypertext protocol that was — for a brief period in the early 1990s — bigger than the web.

↫ Klint Finley

Retrocomputing is about a lot of things, and I feel like it differs per person. For me, it’s a little bit of nostalgia, but primarily it’s about learning, and experiencing hardware and software I was unable to experience when they were new, either due to high cost or just general unavailability. There’s a lot to learn from platforms that are no longer among us, and often it helps you improve your skills with the modern platforms you do still use.

The linked article is right: open source is playing such a massive role in the retrocomputing community. The number of open source projects allowing you to somehow use decades-old platforms in conjunction with modern technologies is massive, and it goes far beyond just software – projects like BlueSCSI or very niche things like usb3sun highlights there’s also hardware-based solutions for just about anything retro you want to accomplish.

And we really can’t forget NetBSD, which seems to be the go-to modern operating system for bringing new life to old and retro hardware, as it often runs on just about anything. When I got my PA-RISC workstation, the HP Visualize c3750, I couldn’t find working copies of HP-UX, so I, too, opted for NetBSD to at least be able to see if the computer was fully functional. NetBSD is now a tool in my toolbox when I’m dealing with older, unique hardware.

Retrocomputing is in a great place right now, with the exception of the ballooning prices we’re all suffering from, with even successful mainstay YouTubers like LGR lamenting the state of the market. Still, if you do get your hands on something retro – odds are there’s a whole bunch of tools ready for you to make the most of it, even today.

This is a new installation for a wordpress site that hasn't launched yet.

The memcached process is always at near 100% of cpu usage:

load averages:  1.93,  1.78,  1.83;               up 3+22:29:29                                                                                      21:49:18
31 processes: 28 sleeping, 3 on CPU
CPU states: 59.1% user,  0.0% nice,  4.0% system,  0.0% interrupt, 36.8% idle
Memory: 2069M Act, 1014M Inact, 44K Wired, 175M Exec, 2447M File, 54M Free
Swap: 512M Total, 512M Free / Pools: 279M Used / Network: 23K In, 1K Out

  PID USERNAME PRI NICE   SIZE   RES STATE       TIME   WCPU    CPU COMMAND
 5770 memcache  25    0    70M 3112K CPU/0      19:11 94.58% 94.58% memcached

And memcached-tool /var/run/memcached/memcached_0.sock stats reports

#/var/run/memcached/memcached_0.sock Field         Value
              accepting_conns             1
                    auth_cmds             0
                  auth_errors             0
                        bytes             0
                   bytes_read            21
                bytes_written          4435
                   cas_badval             0
                     cas_hits             0
                   cas_misses             0
                    cmd_flush             0
                      cmd_get             0
                     cmd_meta             0
                      cmd_set             0
                    cmd_touch             0
                  conn_yields             0
        connection_structures             2
        crawler_items_checked             0
            crawler_reclaimed             0
             curr_connections             1
                   curr_items             0
                    decr_hits             0
                  decr_misses             0
                  delete_hits             0
                delete_misses             0
              direct_reclaims             0
               evicted_active             0
            evicted_unfetched             0
                    evictions             0
            expired_unfetched             0
                  get_expired             0
                  get_flushed             0
                     get_hits             0
                   get_misses             0
                   hash_bytes        524288
            hash_is_expanding             0
             hash_power_level            16
                    incr_hits             0
                  incr_misses             0
                     libevent 2.1.12-stable
               limit_maxbytes      67108864
          listen_disabled_num             0
             log_watcher_sent             0
          log_watcher_skipped             0
                 log_watchers             0
           log_worker_dropped             0
           log_worker_written             0
            lru_bumps_dropped             0
          lru_crawler_running             0
           lru_crawler_starts             6
       lru_maintainer_juggles     103444004
            lrutail_reflocked             0
                 malloc_fails             0
              max_connections          1024
                moves_to_cold             0
                moves_to_warm             0
             moves_within_lru             0
                          pid          5770
                 pointer_size            64
               read_buf_bytes         98304
          read_buf_bytes_free         32768
               read_buf_count             6
                 read_buf_oom             0
                    reclaimed             0
         rejected_connections             0
                 reserved_fds            20
           response_obj_bytes         49152
           response_obj_count             1
             response_obj_oom             0
         round_robin_fallback             0
                rusage_system      1.053032
                  rusage_user   1136.463840
        slab_global_page_pool             0
   slab_reassign_busy_deletes             0
     slab_reassign_busy_items             0
  slab_reassign_chunk_rescues             0
slab_reassign_evictions_nomem             0
 slab_reassign_inline_reclaim             0
        slab_reassign_rescues             0
        slab_reassign_running             0
                  slabs_moved             0
              store_no_memory             0
              store_too_large             0
                      threads             4
                         time    1724186947
   time_in_listen_disabled_us             0
            total_connections             4
                  total_items             0
                   touch_hits             0
                 touch_misses             0
          unexpected_napi_ids             0
                       uptime          1177
                      version        1.6.23

Memcached is configure to start with:

# grep memcach /etc/rc.conf
memcached=YES
memcached_jobs="job0"
memcached_job0_args="-a 660 -s /var/run/memcached/memcached_0.sock -m 64 -c 1024"

What might be wrong?

After years of inactivity, the Kyua project has graduated as an open source citizen and has a new home under the FreeBSD umbrella!

But uh… wait, what is Kyua and why is this exciting? To resolve confusion and celebrate this milestone, I’d like to revisit what Kyua is, how it came to be, why I stopped working on it for a while, why that was a problem for FreeBSD—and, indirectly, NetBSD—and how Kyua being free software has helped keep it alive.

As usual, partway through a couple weeks in Mallorca, we’re just getting the hang of it. After a few days of only the pool, it’s been pool mornings and beach afternoons. Every day, each kid gets more bold in both. I’ve managed to avoid getting sunburnt so far, though it’s getting harder to stay ahead of the situation. For mealtimes it’s kind of fun to be stuck in a tiny kitchen trying to cook my way out; bedtimes, down a sleepable room due to a broken air conditioner, were less so. My carcass got taken over by mosquitoes, who then rented most of it back to me. In a few days I might be ready to buy them out.

Usually here I’d consider taking a nap when the littles do. Definitely tired enough. But a little solo computer time feels more like what I’m needing: Refactoring some code, backing up some photos, updating pkgsrc stuff, writing posts on my website, that sort of thing.

For this summer vacation we’re hopping around more, which happens to simplify our transatlantic travel days. (Traditionally we’d have connecting flights before arriving anywhere.) One flight into Munich, where we stayed in the area for a few days visiting in-laws. From there a quick hop to Mallorca, the meat in our vacation sandwich. From here a quick-ish hop to Hannover for a week back in the little north-central German village where we lived out the first two years of COVID. Then we’ll drive to Frankfurt to see friends before our return flight to New York. Hopping around like this means we get to see more people and places, in exchange for which we get to find out what happens when kids try to sleep in a wider variety of environments and configurations. So far, sokay.

I wanted to clear all the newsletters items I’ve saved, so you are getting links until my inbox no longer paginates.

• Just Use Postgres for Everything.  (via)
• When web search failed me for Postgres.
• The Future of MySQL is Postgres. (target from previous link)
• From Cloud Chaos to FreeBSD Efficiency.  Or any BSD.   (via)
• Layers and Gateways, a historical view.  Railways vs networks.
• The workstation you wanted in 1990, in your pocket.
• Paul Ceruzzi has a Substack.
• pkgsrc-2024Q2 is out.
• X Window System at 40.  (via)
• Some TENEX history.
• Why shitty robots are the antidote of perfectionism, a video.  Her store has neat stuff.
• Scifiinterfaces.com.  I linked to this in 2013 but it’s still going.  (reminded via)
• Largest old computer I’ve seen a person keep yet.  (via)
• Tiny TV 2.  (via)
• UNLICENSED: Bootlegging As Creative Practice, a book.  (via)
• One Page Dungeon Reviews.  (via)

All that to say, I find that NetBSDs philosophy aligns with mine. The OS is small and cozy, and compared to many minimal Linux distributions, I found it faster to setup. Supported hardware is automatically picked up, for my Thinkpad T480s almost everything (except the trackpad issue I solved above) worked out of the box, and it comes with a minimal window manager and display manager to get you started. It is simple and minimal but with sane defaults. It is a hackable system that teaches you a ton. What more could you want?

↫ Marc Coquand

I spent quite some time using OpenBSD earlier this year, and I absolutely, positively loved it. I can’t quite put into words just how nice OpenBSD felt, how graspable the configuration files and commands were, how good and detailed the documentation, and how welcoming and warm the community was over on Mastodon, with even well-known OpenBSD developers taking time out of their day to help me out with dumb newbie questions.

The only reason I eventually went back to Fedora on my workstation was performance. OpenBSD as a desktop operating system has some performance issues, from a slow file system to user interface stutter to problematic Firefox performance, that really started to grind my gears while trying to get work done. Some of these issues stem from OpenBSD not being primarily focused on desktop use, and some of them simply stem from lack of manpower or popularity. Regardless, nobody in the OpenBSD community was at all surprised or offended by me going back to Fedora.

NetBSD seems to share a lot of the same qualities as OpenBSD, but, as the linked article notes, with a focus on different things. Like I said yesterday, I’m looking to building and testing a system entirely focused on tiled terminal emulators and TUI applications, and I’ve been pondering if OpenBSD or NetBSD would be a perfect starting point for that experiment.

I was not able to get this done early like the last few posts, but there’s still a good range here.

• Can You Make It Bigger? – A Journey In Building Arcade Cabinets.  (via)
• Via previous story, the Picade.
• The 84 Series.  On point for current topics and period stylings.  (via)
• Bat Hauntings on Apple VR.  This, then, is the future.  (via)
• 3D Printing My Teeth.  (via)
• valve.computer.  Hardcore analog.  (via I lost it, sorry)
• Schotter – Georg Nees – Part 1.  (via)
• The Old Computer Challenge v4 (Olympics edition)
• Try out a NetBSD system in 20 seconds (amd64 & amr64).
• Things to Make and Do.

NetBSD 10 was released recently, so a lot of people are experimenting with it and writing down their thoughts. I’ve got two of those for you today, to help you in case you, too, want to install NetBSD 10 and play around with, or just use, it.

First, what if you want to install NetBSD 10 on a UEFI system, but with full disk encryption in case your device gets stolen? It turns out there are countless guides for installing with full-disk encryption on MBR-based systems, but once you use UEFI – as you should be – things get a lot more complicated. The NetBSD installer is apparently rather basic, and a better solution is to drop to a shell and install NetBSD that way instead, and even then, full disk encryption with UEFI is actually not possible, as it seems the root file system – where the operating system itself resides – cannot be encrypted.

The restriction is in the root file-system. It needs to be in plain-text and in a regular partition. It seems to me that rootfs in CGD or LVM is not well supported.

↫ vsis.online

This seems like something the NetBSD team may need to take a look at, since full disk encryption should be an easy option to choose, even, or especially in 2024, on UEFI systems. Such encryption is easily achieved on Linux or Windows systems, and it seems odd to me that NetBSD is lagging behind a bit here. In the meantime, the linked guide will be a good jumping-off point for those of you interested in going a similar route.

The second article I want to highlight concerns NetBSD 10 on the Pinebook Pro, the inexpensive ARM laptop that normally ships with Linux. It turns out there’s a NetBSD 10 image for this device, so installation is quite a bit more straightforward than the more exotic setup I mentioned earlier. It seems most of the hardware works quite well out of the box, with the inly exception being the on-board Wi-Fi, which the author addressed with a USB W-Fi dongle.

Other than that, NetBSD is running well on the Pinebook Pro for the author, which is great to read since that makes this cheap device a great starting point for people interested in running NetBSD.

On my home network, some important jobs are performed by little ARM computers.

AirPlay to sound system

The house came with a decent sound system wired in. The receiver can take 1/8” stereo input — from AirPlay, with help from a decade-old Raspberry Pi 1 Model B Rev 2.

1. Prepare disk

With a 4GB SD card, from macOS:

$ diskutil list    # inspect output
$ SDCARD=disk6
$ diskutil unmountDisk ${SDCARD}
$ links https://raspi.debian.net/tested-images/
$ DISKIMAGE=20231109_raspi_1_bookworm.img.xz
$ fetch https://raspi.debian.net/tested/${DISKIMAGE}
$ xzcat ${DISKIMAGE} \
  | sudo dd of=/dev/r${SDCARD} bs=64k oflag=sync status=progress
$ diskutil eject ${SDCARD}

2. First boot

Place the RPi somewhere convenient. Connect SD card, keyboard, HDMI, Ethernet, and power. Log in as root, no password:

# apt update
# apt -y install etckeeper
# cd /etc
# git branch -M main
# apt -y install sudo
# visudo    # for the sudo group, insert NOPASSWD: before the final ALL
# useradd -m -G sudo -s /bin/bash schmonz
# passwd schmonz
# exit

Log in as schmonz:

$ sudo passwd root
$ sudo sh -c 'echo 127.0.1.1 schleierplay >> /etc/hosts'
$ sudo hostnamectl hostname schleierplay
$ sudo ln -sf /usr/share/zoneinfo/US/Eastern /etc/localtime
$ sudo etckeeper commit -m 'Set root password, hostname, and timezone.'
$ sudo apt -y install shairport-sync
$ sudo vi /etc/shairport-sync.conf
$ sudo etckeeper commit -m 'Set AirPlay name.'
$ sudo shutdown -h now

3. Deployment

Place the RPi where it’ll live. Connect audio cable, Ethernet, and power.

$ ssh-copy-id schleierplay.local

4. Usage

Make sure receiver is set to AUX input. Use AirPlay.

5. Maintenance

As with any Debian:

$ ssh schleierplay.local -t 'sudo apt update && sudo apt -y upgrade && sudo apt -y autoremove'

To back up /etc, git push it someplace trustworthy and private.

6. Wishlist

I’d rather run NetBSD, but on 10.0 with shairport-sync, I saw a lot of AirPlay Speaker Not Available: 'House' is being used by someone else (even when it wasn’t).

AirPrint to old printer

My ancient USB-only HP LaserJet P1006 remains reliable for our basic needs and we’ve still got a pile of toner cartridges. A friend recently sent me a comparatively beefy Pine A64 board.

1. Prepare disk

With a 4GB SD card, from macOS:

$ diskutil list    # inspect output
$ SDCARD=disk6
$ diskutil unmountDisk ${SDCARD}
$ links https://www.armbian.com/pine64/
$ DISKIMAGE=Armbian_24.5.1_Pine64_bookworm_current_6.6.31_minimal.img.xz
$ fetch https://dl.armbian.com/pine64/archive/${DISKIMAGE}
$ xzcat ${DISKIMAGE} \
  | sudo dd of=/dev/r${SDCARD} bs=64k oflag=sync status=progress
$ diskutil eject ${SDCARD}

2. First boot

Place the A64 somewhere convenient. Connect SD card, keyboard, HDMI, Ethernet, and power. Follow the prompts to set the root password, create a user account, and select a locale. Then continue:

# apt update
# apt -y install etckeeper
# cd /etc
# git branch -M main
# visudo    # for the sudo group, insert NOPASSWD: before the final ALL
# exit

Log in as schmonz:

$ sudo sh -c 'echo 127.0.1.1 schleierprint >> /etc/hosts'
$ sudo hostnamectl hostname schleierprint
$ sudo ln -sf /usr/share/zoneinfo/US/Eastern /etc/localtime
$ sudo etckeeper commit -m 'Set root password, hostname, and timezone.'
$ sudo apt -y install hplip avahi-daemon
$ sudo usermod -a -G lpadmin schmonz
$ sudo etckeeper commit -m 'Make myself a printer admin.'
$ sudo shutdown -h now

3. Deployment

Place the A64 where it’ll live. Connect printer, Ethernet, and power.

$ ssh-copy-id schleierprint.local
$ ssh schleierprint.local
$ sudo hp-setup -i    # follow prompts, mostly defaults; name the queue 'hpljp1006'
$ sudo etckeeper commit -m 'Add initial hplip config for P1006.'
$ sudo sed -i \
  -e '/^\*ColorDevice: True$/s|True|False|' \
  -e '/^\*OpenUI \*Duplex\/Double-Sided Printing: PickOne$/,/^\*CloseUI: \*Duplex$/s|^|*% |' \
  -e '/^\*OpenUI \*ColorModel\/Output Mode: PickOne$/,/^\*CloseUI: \*ColorModel$/s|^|*% |' \
  /etc/cups/ppd/hpljp1006.ppd
$ sudo etckeeper commit -m 'Correct advertised printer capabilities.'
$ sudo sed -i \
  -e 's|^Info $|Info HP LaserJet P1006|' \
  /etc/cups/printers.conf
$ sudo lpadmin -d hpljp1006
$ sudo etckeeper commit -m 'Name printer and set it as default.'
$ sudo cupsctl --remote-any
$ sudo etckeeper commit -m 'Let local network talk to CUPS.'
$ sudo sed -i \
  -e '/^WebInterface /a PreserveJobFiles No' \
  /etc/cups/cupsd.conf
$ sudo etckeeper commit -m 'Maybe avoid some disk writes.'
$ sudo systemctl restart cups

4. Usage

On macOS, add the printer. When it autoselects “Generic PostScript Printer”, leave it (details in sidebar). Print.

On iOS, print.

On Windows, add the printer. Print.

5. Maintenance

As with any Debian:

$ ssh schleierprint.local -t 'sudo apt update && sudo apt -y upgrade && sudo apt -y autoremove'

To back up /etc, git push it someplace trustworthy and private.

6. Wishlist

I’d rather run NetBSD, but neither 10.0 nor -current brought up HDMI. I could try writing NetBSD to an SD card, mounting it from another NetBSD system, setting hostname in rc.conf, adding a non-root user, and then booting the A64 from it in order to do the rest over ssh. (Other systems that also didn’t bring up HDMI, wherefore I landed by trial and error on Armbian: FreeBSD 14, OpenBSD 7.5, Debian 12.)

AirPlay to old Sonos

Since one of my old Sonos speakers can’t be upgraded to AirPlay-compatible firmware, I’m not eager to upgrade the other. Instead, I’ve added AirConnect on the Pine A64 as an AirPlay relay.

Contents of /etc/systemd/system/airupnp.service:

[Unit]
Description=AirUPnP bridge
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/home/schmonz/bin/airupnp-linux-aarch64-static -l 1000:2000 -N '%%s' -x /home/schmonz/etc/airupnp.xml -Z
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

Contents of /home/schmonz/etc/airupnp.xml (to omit my UPnP router from the AirPlay list):

<?xml version="1.0"?>
<airupnp>
    <device>
        <udn>uuid:1e38fc78-51f5-5f5d-9268-50c6b1dc59f8</udn>
        <name>Verizon FiOS-G1100 ManageableDevice+</name>
        <mac>bb:bb:bb:bb:bb:bb</mac>
        <enabled>0</enabled>
    </device>
</airupnp>

I’d rather install AirConnect from a system-provided package, but there isn’t one for Debian. Maybe I can puzzle out the AirConnect build system and add it to pkgsrc.

The NetBSD Project is pleased to announce NetBSD 8.3, the third and final release from the NetBSD 8 stable branch.

It represents a selected subset of fixes deemed important for security or stability reasons since the release of NetBSD 8.2 in March 2020, as well as some enhancements backported from the development branch. It is fully compatible with NetBSD 8.0.

This also represents the end-of-life for the netbsd-8 release branch. No further security updates will happen. Users running 8.2 or an earlier release are strongly recommended to upgrade to a newer branch, preferably the recent NetBSD 10.0 release.

Pkgsrc has already desupported the netbsd-8 branch.

See the full release announcement (including download links).

My early imaginings of a collaborative Open Source successor to qmail, let me assure you, did not include going nearly four years between releases. Well, at least it hasn’t been more than four. notqmail 1.09 is here:

• qmail list announcement
• Fediverse and Twitter
• Lobste.rs, Hacker News, and Reddit
• Release notes

For decades, due to each administrator needing to patch in their particular missing bits of functionality, the qmail source code itself has effectively been a public API. Some future release of notqmail will include everything most everyone needs. On that day, we’ll freely make desirable code changes without worrying about breaking people’s patches. On that day, notqmail will have become a relatively normal software project operating under relatively normal constraints.

This is not that day. notqmail remains a uniquely challenging legacy-code rehabilitation project, and 1.09 is merely a solid, long-overdue release that includes the work of a couple dozen new contributors.

Since this release took too long, our next development cycle will be

1. Time-bounded
2. Focused on process improvements

In legacy code, every time we can turn a vicious cycle virtuous, it’s a big win. By making the code easier and safer to change, we’ll have more fun; by having more fun, we’ll make more progress; by making more progress, we’ll get more feedback; by getting more feedback, we’ll have more fun; and so on.

Have fun with notqmail 1.09! Let us know how the upgrade goes for you. (I’ll be updating the pkgsrc package soon.) And if getting involved is your kind of thing, please feel welcome to join us.

I am trying to use OpenVPN as a client under NetBSD using this command:

openvpn --client --config /etc/openvpn/config.ovpn

I am getting the following output and errors:

localhost# openvpn --client --config /etc/openvpn/openvpn.ovpn 
2024-04-26 10:29:35 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
2024-04-26 10:29:35 DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305). OpenVPN ignores --cipher for cipher negotiations. 
2024-04-26 10:29:35 OpenVPN 2.6.10 x86_64--netbsd [SSL (OpenSSL)] [LZO] [LZ4] [MH/PKTINFO] [AEAD]
2024-04-26 10:29:35 library versions: OpenSSL 1.1.1k  25 Mar 2021, LZO 2.10
Enter Auth Username:********
Enter Auth Password:********
2024-04-26 10:32:48 TCP/UDP: Preserving recently used remote address: [AF_INET]**.191.33.**:1701
2024-04-26 10:32:48 Socket Buffers: R=[32768->32768] S=[32768->32768]
2024-04-26 10:32:48 Attempting to establish TCP connection with [AF_INET]**.191.33.**:1701
2024-04-26 10:32:48 TCP connection established with [AF_INET]**.191.33.**:1701
2024-04-26 10:32:48 TCPv4_CLIENT link local: (not bound)
2024-04-26 10:32:48 TCPv4_CLIENT link remote: [AF_INET]**.191.33.**:1701
2024-04-26 10:32:48 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2024-04-26 10:32:48 TLS: Initial packet from [AF_INET]**.191.33.**:1701, sid=0006909e 9b0d208f
2024-04-26 10:32:48 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2024-04-26 10:32:48 VERIFY OK: depth=1, C=US, ST=New York, L=New York, O=Ubiquiti Inc., OU=UniFi_OpenVPN_CA, CN=UniFi_OpenVPN_CA
2024-04-26 10:32:48 VERIFY KU OK
2024-04-26 10:32:48 Validating certificate extended key usage
2024-04-26 10:32:48 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
2024-04-26 10:32:48 VERIFY EKU OK
2024-04-26 10:32:48 VERIFY OK: depth=0, C=US, ST=New York, L=New York, O=Ubiquiti Inc., OU=UniFi_OpenVPN_Server, CN=UniFi_OpenVPN_Server
2024-04-26 10:33:53 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bits RSA, signature: RSA-SHA256, peer temporary key: 253 bits X25519
2024-04-26 10:33:53 [UniFi_OpenVPN_Server] Peer Connection Initiated with [AF_INET]**.191.33.**:1701
2024-04-26 10:33:53 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2024-04-26 10:33:53 TLS: tls_multi_process: initial untrusted session promoted to trusted
2024-04-26 10:33:53 PUSH: Received control message: 'PUSH_REPLY,dhcp-option DNS 192.168.7.1,route 192.168.4.0 255.255.255.0,route 192.168.2.0 255.255.255.0,route 192.168.1.0 255.255.255.0,route 192.168.3.0 255.255.255.0,route-gateway 192.168.7.1,topology subnet,ping 10,ping-restart 60,ifconfig 192.168.7.2 255.255.255.0,peer-id 0,cipher AES-256-GCM'
2024-04-26 10:33:53 OPTIONS IMPORT: --ifconfig/up options modified
2024-04-26 10:33:53 OPTIONS IMPORT: route options modified
2024-04-26 10:33:53 OPTIONS IMPORT: route-related options modified
2024-04-26 10:33:53 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
2024-04-26 10:33:53 TUN/TAP device /dev/tun0 opened
2024-04-26 10:33:53 /sbin/ifconfig tun0 192.168.7.2 192.168.7.1 mtu 1500 netmask 255.255.255.0 up
2024-04-26 10:33:53 /sbin/route add -net 192.168.7.0 192.168.7.1 -netmask 255.255.255.0
add net 192.168.7.0: gateway 192.168.7.1
2024-04-26 10:33:53 /sbin/route add -net **.191.33.** 192.168.1.254 -netmask 255.255.255.255
route: writing to routing socket: File exists
add net **.191.33.**: gateway 192.168.1.254: File exists
2024-04-26 10:33:53 ERROR: OpenBSD/NetBSD route add command failed: external program exited with error status: 1
2024-04-26 10:33:53 /sbin/route add -net 0.0.0.0 192.168.7.1 -netmask 128.0.0.0
add net 0.0.0.0: gateway 192.168.7.1
2024-04-26 10:33:53 /sbin/route add -net 128.0.0.0 192.168.7.1 -netmask 128.0.0.0
add net 128.0.0.0: gateway 192.168.7.1
2024-04-26 10:33:53 /sbin/route add -net 192.168.4.0 192.168.7.1 -netmask 255.255.255.0
add net 192.168.4.0: gateway 192.168.7.1
2024-04-26 10:33:53 /sbin/route add -net 192.168.2.0 192.168.7.1 -netmask 255.255.255.0
add net 192.168.2.0: gateway 192.168.7.1
2024-04-26 10:33:53 /sbin/route add -net 192.168.1.0 192.168.7.1 -netmask 255.255.255.0
route: writing to routing socket: File exists
add net 192.168.1.0: gateway 192.168.7.1: File exists
2024-04-26 10:33:53 ERROR: OpenBSD/NetBSD route add command failed: external program exited with error status: 1
2024-04-26 10:33:53 /sbin/route add -net 192.168.3.0 192.168.7.1 -netmask 255.255.255.0
add net 192.168.3.0: gateway 192.168.7.1
2024-04-26 10:33:53 GID set to nogroup
2024-04-26 10:33:53 UID set to nobody
2024-04-26 10:33:53 Initialization Sequence Completed
2024-04-26 10:33:53 Data Channel: cipher 'AES-256-GCM', peer-id: 0, compression: 'lzo'
2024-04-26 10:33:53 Timers: ping 10, ping-restart 60

I have a working internet connection when running OpenVPN as a client, but I can't access any of the machines on the network **.191.33.**, I know I should be able to SSH into 192.168.1.114, but I can't reach that machine through OpenVPN, there are firewall rules in the Ubuiquity box allowing traffic from 192.168.7.* to 192.168.1.* I know this is working, its testet from Mac and PC using the OpenVPN Client, I just can't get it to work on NetBSD

This is my routing table before running OpenVPN:

Internet:
Destination        Gateway            Flags    Refs      Use    Mtu Interface
default            192.168.1.254      UGS         -        -      -  iwn0
127/8              127.0.0.1          UGRS        -        -  33624  lo0
127.0.0.1          lo0                UHl         -        -  33624  lo0
192.168.1/24       link#2             UC          -        -      -  iwn0
192.168.1.68       link#2             UHl         -        -      -  lo0
192.168.1.254      00:1e:80:a2:2e:ff  UHL         -        -      -  iwn0

This is my routing table when running OpenVPN:

Internet:
Destination        Gateway            Flags    Refs      Use    Mtu Interface
0/1                192.168.7.1        UGS         -        -      -  tun0
default            192.168.1.254      UGS         -        -      -  iwn0
**.191.33.**/32    192.168.1.254      UGS         -        -      -  iwn0
127/8              127.0.0.1          UGRS        -        -  33624  lo0
127.0.0.1          lo0                UHl         -        -  33624  lo0
128/1              192.168.7.1        UGS         -        -      -  tun0
192.168.1/24       link#2             UC          -        -      -  iwn0
192.168.1.68       link#2             UHl         -        -      -  lo0
192.168.2/24       192.168.7.1        UGS         -        -      -  tun0
192.168.3/24       192.168.7.1        UGS         -        -      -  tun0
192.168.4/24       192.168.7.1        UGS         -        -      -  tun0
192.168.7/24       192.168.7.1        UGS         -        -      -  tun0
192.168.7.1        192.168.7.2        UH          -        -      -  tun0
192.168.7.2        tun0               UHl         -        -      -  lo0
192.168.1.254      00:1e:80:a2:2e:ff  UHL         -        -      -  iwn0

This is my routing table after stopping OpenVPN:

Internet:
Destination        Gateway            Flags    Refs      Use    Mtu Interface
0/1                192.168.7.1        UGS         -        -      -  tun0
default            192.168.1.254      UGS         -        -      -  iwn0
**.191.33.**/32    192.168.1.254      UGS         -        -      -  iwn0
127/8              127.0.0.1          UGRS        -        -  33624  lo0
127.0.0.1          lo0                UHl         -        -  33624  lo0
128/1              192.168.7.1        UGS         -        -      -  tun0
192.168.1/24       link#2             UC          -        -      -  iwn0
192.168.1.68       link#2             UHl         -        -      -  lo0
192.168.2/24       192.168.7.1        UGS         -        -      -  tun0
192.168.3/24       192.168.7.1        UGS         -        -      -  tun0
192.168.4/24       192.168.7.1        UGS         -        -      -  tun0
192.168.7/24       192.168.7.1        UGS         -        -      -  tun0
192.168.7.2        tun0               UHl         -        -      -  lo0
192.168.1.254      00:1e:80:a2:2e:ff  UHL         -        -      -  iwn0

This is my routing table when i have destroyed tun0:

ifconfig tun0 destroy

Internet:
Destination        Gateway            Flags    Refs      Use    Mtu Interface
default            192.168.1.254      UGS         -        -      -  iwn0
**.191.33.**/32    192.168.1.254      UGS         -        -      -  iwn0
127/8              127.0.0.1          UGRS        -        -  33624  lo0
127.0.0.1          lo0                UHl         -        -  33624  lo0
192.168.1/24       link#2             UC          -        -      -  iwn0
192.168.1.68       link#2             UHl         -        -      -  lo0
192.168.1.254      00:1e:80:a2:2e:ff  UHL         -        -      -  iwn0

The route to **.191.33.** is still there when stopping OpenVPN and destroying the tunnel tun0, I don't know if this is expected behaviour.

Update I have checked several computers now, and none of them have the 192.168.1/24 route, its only on the PC running NetBSD, I have tried to delete it, with no success. I have also read a lot of man pages and various other documentation, but I have not come up with anything usefull yet.

OpenVPN Config

client
dev tun
proto tcp
remote **.191.33.** 1701
resolv-retry infinite
nobind

# Downgrade privileges after initialization (non-Windows only)
user nobody
group nogroup

persist-key
persist-tun

auth-user-pass
remote-cert-tls server
cipher AES-256-CBC
comp-lzo
verb 3

auth SHA1
key-direction 1

reneg-sec 0

redirect-gateway def1

<ca>
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
</ca>
<tls-auth>
-----BEGIN OpenVPN Static key V1-----
...
-----END OpenVPN Static key V1-----
</tls-auth>
<cert>
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
</key>

Intent

I am trying to connect to a VPN at a remote location, from home. The remote network is protected by a firewall facing the internet, all computers on the network behind the router is accessible, the 192.168.7.* network is standard Ubuiquity and used for VPN clients, I have added a firewall rule to allow traffic from 192.168.7.* to the 192.168.1.* network, this works fine from all computers I have tried it with, Mac, PC, Windows, Linux, MacOS. etc. except a PC running NetBSD.

The network configuration on the PC running NetBSD was performed during installation, and I used the auto-configuration feature, so I have not specified any networks, routes or rules at all. I am able to access the internet when using OpenVPN client, I just cannot access any of the machines on the remote network. So I guess the part I am missing is the routing from 192.168.7.* to 192.168.1.* so I will be able to access computers attached to that network

A few years ago, I wrote a "state of things" blog post about Wayland on NetBSD. It's only natural that I should do one about X11, which is used by far more people to get a graphical environment on NetBSD.

There are a lot of differences from how NetBSD and the typical distributor ship X.Org. For one, we ship it as an optional monolithic package rather than separate individual packages. This means every driver is included on every system, rather than as an optional module. Sometimes, this means we need to fine-tune driver selection to ensure the correct drivers are loaded on the correct hardware, since multiple conflicting drivers can claim a video output. We also want sensible fallbacks, since if you're using a GPU from the future with an old OS version, you probably want X to seamlessly fall back to a regular framebuffer.

Secondly, the way our "xsrc" repository is set up, it's effectively functioning as a fork of X.Org that regularly pulls from upstream freedesktop.org (but does not push back). This allows X development to happen as part of NetBSD.

Thirdly, we use our own build system based purely on BSD makefiles, not X.Org's based on GNU autotools. This fits well with our build.sh cross-compilation system.

We have a number of drivers which have not made their way upstream. Perhaps the most ubiquitous of these is xf86-input-ws, a driver which came from OpenBSD, targets an API from NetBSD, and continues to be developed in both. This is a generic input driver that can support any pointing device that the kernel supports. Unlike xf86-input-mouse, it doesn't assume the device is a mouse, and can support advanced touchpad and touchscreen features. Other NetBSD exclusives include xf86-video-pnozz, xf86-video-mgx, and xf86-video-crime. While these all share the "xf86" name inherited from the historical XFree86 distribution, none of them are exclusively for x86.

There are a number of drivers that are accelerated when used in NetBSD, but the acceleration support is missing upstream. This is mostly due to the work of macallan@, who has diligently worked on drivers for accelerators found on SPARC and PowerPC hardware.

X.Org has historically supported two 2D acceleration modes, XAA and EXA. XAA seems complicated - according to the X.Org Foundation it supports accelerating "patterned fills and Bresenham lines" (eh?). XAA was removed from the X.Org server in 2012, and many old drivers were not updated to support the newer and simpler EXA model, except in NetBSD, over a time period of several years.

Did you know that Nvidia used to have an open source graphics driver? It supported 2D acceleration for a range of cards. In NetBSD, it's retained for platforms that included embedded Nvidia chips and aren't capable of (or predate, or don't want) the modern novueau driver. Six years ago, it was updated in NetBSD to support EXA acceleration.

There are a few ways our X integration could be improved. While lots of attention has been paid to the server, less has been paid to clients (programs). Did you know that X includes a text editor, and that text editor supports syntax highlighting and spell checking?

NetBSD includes its own command-line spell checker and associated dictionaries, spell(1), inherited from the BSD UNIX of yore. It's pretty basic, and only supports variations of English. To get spell checking to work in xedit, you need to install ispell (another command-line spell checker) from pkgsrc, install a dictionary, then set some Xresources (or create symlinks) to make sure xedit finds ispell. This could surely be streamlined by teaching xedit about spell.

We also ship every program that has been included with historical X.Org distributions. This includes well-known things like xterm, slightly less well-known things like xbiff(1), and obscurities like bitmap(1) (apparently a 1-bit-per-pixel alternative to MS Paint). A while ago, we removed some libraries which are no longer used by the modern X server, and maybe we should evaluate whether we need all of these programs too. xmh(1) is a frontend for a mail system that isn't included in base. Together, bitmap and xmh are around 300 kilobytes.

We include fonts, bitmaps and scalable, for a wide range of computing devices. In the latest versions of NetBSD, the font size will automatically scale with the screen size to support HiDPI displays as well as small mobile devices. However, we don't ship a scalable cursor theme at the moment. We're also missing high-resolution fonts for Japanese, a shame considering the popularity of NetBSD in Japan. Koruri looks interesting and is suitably small, maybe we should import it.

While we have many useful simple programs by default (a clock, a calculator, an editor, a window manager, a compositor, a terminal emulator...), we're notably missing a screen locking program for X in the default install, although we have lock(1) for the tty.

The big question - does all this have a future? The good news is that all new hardware has generic support in X. Someone writes either a modesetting kernel driver or a classical wsdisplay kernel driver and they will be automatically supported by the associated drivers in X. The bad news is that to have applications running we require access to a larger open source ecosystem, and that ecosystem has a lot of churn and is easily distracted by shiny new squirrels. The process of upstreaming stuff to X.Org is an ongoing process, but it's likely we'll run into things that will never be suitable for upstream.

Of course, on NetBSD, you also have the option of trying vanilla modular X.Org from pkgsrc, or using something else entirely.

The NetBSD Project is pleased to announce NetBSD 9.4, the fourth release from the NetBSD 9 stable branch.

It represents a selected subset of fixes deemed important for security or stability reasons since the release of NetBSD 9.3 in August 2022, as well as some enhancements backported from the development branch. It is fully compatible with NetBSD 9.0. Users running 9.3 or an earlier release are strongly recommended to upgrade.

The general NetBSD community is very excited about NetBSD 10.0, the latest NetBSD release, but if for some reason you can not (or do not want to) update to 10.0, it is strongly recommended to update to 9.4. This is especially true for users still using a NetBSD 8.x release as that old release branch will be desupported by the end of April 2024.

Full release notes, including download links

If you sorta squint and tilt your head, it’s a games theme this week.

• Roguelike vs. Roguelite: What’s the Difference?  (via)
• Awesome-Selfhosted.  (also via)
• rigg 1.0 released – a new way to run indie games on OpenBSD.
• SEthernet: Modern, low-cost 10/100 Ethernet for the Macintosh SE and SE/30.
• Mastodon for Apple II (][+, //c, IIe, and IIgs)
• Hex marks the spot.  (via)
• pkgsrc-2024Q1 branch released, and NetBSD 10.0.
• Further Explorations, the companion volume to 50 Years of Text Games, is available for standalone purchase.  Related: merchandise.
• The Greenwich prime meridian isn’t where it used to be.
• FreeBSD Speedrun.
• Terminal status bar with only stock tools.

Your unrelated music for the week: New Strategies for Modern Crime Vol 1.  (via)

I am trying to write a driver for NetBSD that will reserve 10 pages of virtual memory, then provide the first 5 pages with physical addresses. At the end, I output the page number, its physical address, and flags such as Valid, User and Modified. However, it seems to me that the flags are not working correctly, since for all pages, all flags have the same value, which is 1 (true). Please help me figure out what I'm doing wrong.

#include <sys/cdefs.h>
#include <sys/module.h>
#include <sys/param.h>
#include <sys/sysctl.h>
#include <uvm/uvm.h>

MODULE(MODULE_CLASS_MISC, driver, NULL);
#define PAGESIZE 0x1000
extern paddr_t avail_end;
vaddr_t va;
struct pglist plist;
static int lab4_modcmd(modcmd_t cmd, void* arg) {
   va = uvm_km_alloc(kernel_map, PAGESIZE*10, 0, UVM_KMF_VAONLY);
   if (va == 0) {
      return 0;
   }
   int error = uvm_pglistalloc(PAGESIZE*5, 0, avail_end, 0, 0, &plist, 5, 0);
   if (!error) printf ("LAB4 loaded\n");
   struct vm_page *page = TAILQ_FIRST(&plist);
   for(int i = 0; page; i++) {
      pd_entry_t *ppte;
      ppte = L2_BASE+pl2_i(va+PAGESIZE*i);
      paddr_t pa = VM_PAGE_TO_PHYS(page);
      printf("Page - %d\n", i+1);
      printf("Valid - %d\n", ((*ppte & PG_V) ? 1 : 0));
      printf("Used - %d\n", ((*ppte & PG_U) ? 1 : 0));
      printf("Modified - %d\n", ((*ppte & PG_M) ? 1 : 0));
      printf("Physical address - 0x%lx\n", pa);
      printf("\n");
      page = TAILQ_NEXT(page, pageq.queue);
   }
   uvm_pglistfree(&plist);
   uvm_km_free(kernel_map, va, PAGESIZE*10, UVM_KMF_VAONLY);
   return 0;
}

I tried to look for other examples where these flags are used.

I have the following code that produces a window with a QTableView visible. I add a QTextEdit to the right but hide it. When I run the code it looks as though space has been reserved on the right for the TextEdit even though the TableView has a setSizePolicy of Expanding. What do I need to set so that the TableView initially takes up all the space in the window, then shrinks to accommodate the TextEdit and then expands fully when the TextEdit is hidden? Thank you in advance.

import sys
from PyQt5.QtWidgets import QApplication, QGridLayout, QMainWindow, QPushButton
from PyQt5.QtWidgets import  QSizePolicy, QTableView, QTextEdit, QWidget

class MainWindow(QMainWindow):
# https://www.pythonguis.com/faq/file-image-browser-app-with-thumbnails/
    def runBtn(self):
#https://www.youtube.com/watch?v=LCJEyuCZlAY
        if self.hdn == True:
            self.text.show()
            self.hdn = False
        else:
            self.text.hide()
            self.hdn=True

    def __init__(self):            
        super().__init__()
        widget = QWidget()
        self.setCentralWidget(widget)
        self.layout = QGridLayout(widget)
        self.hdn = True

        self.Btn = QPushButton('Hide/Show',self)
        self.Btn.clicked.connect(self.runBtn)
        self.layout.addWidget(self.Btn, 0, 0, 1, 1)

        self.view = QTableView()
        self.layout.addWidget(self.view, 1, 0, 48, 50)
        self.view.setSizePolicy(QSizePolicy.Expanding, QSizePolicy.Expanding) # /73522221/

        self.text = QTextEdit()
        self.text.hide()
        self.layout.addWidget(self.text, 1, 50, 48, 2)

app = QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec_()

A few weeks ago, Apple released new versions of Xcode and Command Line Tools. Not thinking too hard about how my pkgsrc developer environment often gets broken by OS or tool updates, I updated promptly. For one thing, I’m kinda used to it. For another, it doesn’t usually break. For a third thing, managing dependencies — anything not my code that can break my code — means responsibility for dealing with the inevitable trouble, and therefore the sooner I find it the better. (More on my approach to life with dependencies.)

A vendor-provided toolchain is a significant dependency. So I accepted the Command Line Tools update, and it commandeered my spare time for two weeks as I hurried carefully to repair one of the world’s biggest continuous-integration cauldrons on one of its most popular platforms. When I ran my usual pkg_rolling-replace -suv to rebuild anything outdated, it did not go well at all.

Did I mention that a few weeks ago we were aiming to stabilize for yesterday’s quarterly release? Suddenly, if we didn’t scramble to straighten things out for macOS users, we’d have to manage a complicated situation for a while. But if we created a mess on other platforms by moving rashly, that’d be even worse.

The usual feedback mechanism for proposed infrastructure changes is to compare full bulk builds before and after. There was no time for that.

Happily, the conclusion of the story is boring: as always, the pkgsrc 2024Q1 stable branch supports macOS and its developer tools, including the latest releases of each. (So does -current pkgsrc, of course, if that’s your thing.)

Curious what we had to do to keep it boring? Read on.

Stricter clang defaults

Upstream Clang 16 and GCC 14 have promoted several warnings to errors by default, and Apple’s Clang 15 has followed suit. (Gentoo has very helpfully documented this for packagers.) These changes are intentional and well-intentioned, pushing maintainers to ship more reliable code. But pkgsrc’s job is to build nearly 30,000 codebases we don’t maintain. And stricter compiler defaults break a lot of builds.

As you might hope, we can make the breakage go away in one place.

In pkgsrc, packages declare which programming languages are required for their build. The compiler framework then selects package-and-platform-appropriate compilers, places them preferentially in the package’s build environment, and — crucially — intercepts compiler invocations and rewrites them for a variety of purposes.

When we look into pkgsrc’s clang logic, we find prior art for this specific class of problem. In September 2020, Xcode 12 (and its associated Command Line Tools) arrived even later in our quarterly schedule and promoted -Wimplicit-function-declaration to an error. The surgical fix: on macOS only, if invoking clang reveals the new stricter default, we pass -Wno-error=implicit-function-declaration to demote the error back to a warning.

Apple Clang 15’s new strictures aren’t observable in the same way, so we adjust our workaround: if clang doesn’t complain when we try demoting the new errors back to warnings, we pass those arguments too, via the same compiler-framework mechanism.

Missing m4 and yacc

This messy regression found only in the Command Line Tools 15.3.0.0.1.1708646388 update — not in the corresponding full Xcode 15.3 (build 15E204a) update — must have been unintended.

On macOS, some of the familiar Unix tools in /usr/bin are in fact stubs. When invoked, they either execute into the corresponding installed program (living somewhere under /Library/Developer) or prompt the user to install the Command Line Tools.

This Command Line Tools update uninstalls m4 and yacc from /Library/Developer. But since the OS-provided /usr/bin/m4 and /usr/bin/yacc stubs still exist, running m4 or yacc still does something: it pops up a window prompting you to reinstall the CLT. Unfortunately, as the latest available version doesn’t provide those tools, reinstalling is a waste of time.

As you might once again hope, we can hide the problem without personally visiting 29,000+ packages.

In pkgsrc, we also have a framework to control which non-compiler tools are invoked during builds. Packages declare which tools are required for their build. The tools framework then selects package-and-platform-appropriate incarnations of the declared tools and places them preferentially in the package’s build environment.

We just got handed a few new twists to handle in the framework, is all.

First, because this clever new CLT failure mode outfoxes our usual tool-detection mechanism, we special-case m4 and yacc detection on macOS, performing an existence check for the stubs’ targets. Then the selection mechanism’s usual fallback logic can provide them some other way. This prevents the primary source of needless CLT install popups. For non-macOS platforms, no change.

Second, because some packages might not yet be declaring all their tool dependencies, we special-case m4 and yacc handling on macOS: when they’re not declared, we place them in the build environment anyway, as no-ops. If the package build happens to invoke them, nothing happens. This prevents the secondary source of needless CLT install popups, at the risk of breaking macOS builds for packages that are missing these tool declarations and have heretofore gotten lucky; in such cases, the breakage will be obvious and the fix easy. For non-macOS platforms, no change. (At leisure, we might like to broaden this approach to help find and fix all undeclared tools on all platforms.)

Third, because the flex tool expects to invoke a GNU-compatible m4, we adjust the tools framework to infer gm4 from a flex declaration so that the framework controls which m4 gets found. This more correctly expresses our intent on all platforms, and in the macOS package build environment it restores /usr/bin/flex to a working state.

Broader xcrun search

We were already relying on xcrun for a couple of things, so when our new tool-detection special cases were sometimes getting surprising results from it, that was concerning. Turns out xcrun no longer looks solely in Apple-controlled locations, but also consults the environment’s $PATH. By invoking xcrun with an empty PATH and --no-cache, we obtain controlled, predictable tool detection.

Conclusion

Under the constraints, we changed as little as possible, as safely as possible, as similarly as possible to previous proven changes, avoiding novel constructs or any whiff of unforeseen consequences. We could not have done nearly as safe or thorough a job without good abstractions already in place. Total lines of pkgsrc infrastructure code changed: less than 100. Now that 2024Q1 is out, we have room to refactor.

These 15.3 updates also include a brand new linker. So far it hasn’t given us any trouble. If that changes, wanna guess whether we have one place to take care of it?

This is the ninth post in my toolchains adventures series. Please check the previous posts in the toolchains category for more context about this journey. There was no Q4 2023 report as there wasn't really anything worthwhile to write about.

I've been taking a break from Pkgsrc to only focus on OpenBSD at this point, for which I updated binutils to version 2.42 in the ports tree.

During this OpenBSD release cycle, the remaining parts required to get pinsyscalls(2) working have been committed, and I added support upstream for the PT_OPENBSD_SYSCALLS segment type to readelf in GNU Binutils, as well as in LLVM versions of objdump and readobj.

Lastly, I also wrote a blog post about Speedbuilding LLVM/Clang in 3 minutes on Power10.

As usual, I’ve also been busy reading different material, and adding new resources to toolchains.net.

binutils commits:

LLVM commits