This report was written by Naveen Narayanan as part of Google Summer of Code 2019.

This report encompasses the progress of the project during the third coding period. You can make sense of the overall progress of the project by going through the first evaluation report and second evaluation report.

WINE on amd64

Wine-4.4 (released on Mar 2019) is working fine on amd64 and i386. I have been able to use a script as a workaround for the problem of setting LD_LIBRARY_PATH. My patch for setting guard size to 0 and hence, precluding Wine from segfaulting, that got upstreamed, can be found here. I have updated the package to the latest development version of Wine which is Wine-4.13 (released on Aug 2019). I have added support to Wine pkgsrc packages to run tests using make test, and at the time of writing, they are failing. I have also noticed them fail on Linux non-pkgsrc environment and hence, will require further investigation. Initially, they were disabled owing to pkgsrc setting FORTIFY_SOURCE which is a macro that provides support for detecting buffer overflows. In pkgsrc, the wip/wine* packages honor PKGSRC_USE_FORTIFY variable passing _FORTIFY_SOURCE macro accordingly. Programs compiled with FORTIFY_SOURCE substitute wrappers for commonly used libc functions that don't do bounds checking regularly, but could in some cases. Wine unconditionally disables that via their configure script because for some platforms that triggered false positives in the past. However, in my experience, no false positive were found.

Running tests on Wine-4.13 throws errors as you can see below:

mode64# make test
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M adsldp.dll -p adsldp_test.exe.so sysinfo && touch sysinfo.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002a:err:winediag:SECUR32_initNTLMSP ntlm_auth was not found or is outdated. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f150,0x00000001,0x22f120) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EEA0 000000000002B028 000000000022EE9C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022F0C8: stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022FB48: stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so cred && touch cred.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002e:fixme:cred:CredReadW unhandled type -1
002e:fixme:cred:CredReadW unhandled flags 0xdeadbeef
002e:err:cred:CredWriteW bad username L"winetest"
002e:err:cred:CredWriteW bad username (null)
002e:err:cred:CredWriteW bad username (null)
002e:fixme:cred:CredDeleteW unhandled type -1
002e:fixme:cred:CredDeleteW unhandled flags 0xdeadbeef
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b3c0, 0x0, 0x22fa50, 0x22f908) stub
cred.c:820: Tests skipped: CRED_TYPE_DOMAIN_VISIBLE_PASSWORD credentials are not supported or are disabled. Skipping
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt && touch crypt.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_lmhash && touch crypt_lmhash.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md4 && touch crypt_md4.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md5 && touch crypt_md5.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_sha && touch crypt_sha.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so eventlog && touch eventlog.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",L"deadbeef") stub
0046:fixme:advapi:OpenEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"deadbeef") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW (L"",L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 1, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 8, 0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:BackupEventLogW (0x0,(null)) stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:546: Tests skipped: We don't have a backup eventlog to work with
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000000,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000001,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000002,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000d,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000e,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000007,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa84) stub
eventlog.c:479: Tests skipped: No records in the 'Application' log
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0020,0x0000,0x00000000,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000005,0x00000000,0x1b1e0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ClearEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"Wine"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"Wine"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000002,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc1"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc1"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc20") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0001,0x00000004,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc300"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc300"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0002,0x00000007,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc1") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc20"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc20"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0002,0x00000009,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc300") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:err:eventlog:ReportEventW L"First string"
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:889: Tests skipped: No events were written to the eventlog
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "this name is too long", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x0) stub
0046:fixme:advapi:StartTraceA (0x0, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:ControlTraceA (cafe4242, "wine", 0x1b1e0, 1) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so lsa && touch lsa.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004a:fixme:advapi:LsaOpenPolicy ((null),0x22f960,0x000f0fff,0x22f930) stub
004a:fixme:security:GetWindowsAccountDomainSid (000000000022F690 0000000000019078 000000000022F68C): semi-stub
004a:fixme:advapi:LsaEnumerateAccountRights (0xcafe,0x22fa20,0x22f990,0x22f92c) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb10,0x000f0fff,0x22fae8) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb00) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb08) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so registry && touch registry.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
registry.c:2850: Tests skipped: HKCR key merging not supported
registry.c:3146: Tests skipped: HKCR key merging not supported
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"Global", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"invalid counter name", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
registry.c:4032: Test failed: [ 9] expected 1168, got 2
registry.c:4032: Test failed: [10] expected 1814, got 2
*** Error code 2

Stop.
make[1]: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64/dlls/advapi32/tests
*** Error code 1

Stop.
make: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64

Running programs on Wine

You can find obligatory screenshots of Wine-4.4/4.13 on amd64/i386 below:

010 Editor (Professional Text/Hex Editor) on Wine-4.13 (amd64)

Notepad++ on Wine-4.4 (i386)

Pinball on Wine-4.13 (amd64)

How to run Wine on NetBSD/amd64

• Compile kernel with USER_LDT enabled.
• Install kernel.
• Clone wip repo.
• cd /usr/pkgsrc/wip/wine64; make install (for 4.4)
or
• cd /usr/pkgsrc/wip/wine64-dev; make install (for 4.13)
• wine notepad

Future Plans

Wine requires the kernel option USER_LDT to be able to run 32-bit applications on amd64 - facilitated by WoW64. Presently, this feature isn't enabled by default on NetBSD and hence, the kernel has to be compiled with USER_LDT enabled. I will work on getting the tests to pass and finding a better approach to deal with LD_LIBRARY_PATH issue. In addition to that, I shall work on incorporating a NetBSD VM to Wine Test Bot infrastructure so we can preclude Wine from getting out of shape on NetBSD in the future (work in progress).

Summary

Presently, Wine-4.4 and Wine-4.13 are working fine on amd64/i386. I have been able to meet all the goals as per my original plan. I would like to attribute the success to the support provided by my mentors @leot, @maya and @christos. I would also like to thank my mentor @maxv for solving the conflict between SVS and USER_LDT kernel options. I intend to maintain the packages wine64/wine64-dev/wine32 for the foreseeable future. As stated above, I shall try to set up a NetBSD VM as Wine Test Bot for purpose of CI. Once again, thanks to Google for enabling me to do what I love.

Source Code

With the following changes:

• DHCP: Work with IP headers with options
• script: Assert that env string are correctly terminated
• script: Terminate env strings with no value
• script: Don't attempt to use an invalid env string
• route: Fix NULL deference error when using static routes
• ARP: Respect IFF_NOARP
• DHCP: Add support for ARPHRD_NONE interfaces
• DHCP: Allow full DHCP support for PtP interfaces, but not by default
• DragonFlyBSD: 500704 announces IPv6 address flag changes
• control: sends correct buffer to listeners

DragonFlyBSD-500704 kernel has the functionality dhcpcd needs to compile without any warnings. There are still improvements to be made to the whole network stack, but none of them are dhcpcd specific.

dhcpcd-ui now correctly reports SSD association and all the addresses obtained (regression from dhcpcd-7)

dhcpcd now supports QMI interfaces in RawIP mode - this is basically PtP interface without any L2 frame header. Because PtP interfaces normally configure their address via a 3rd party tool (dhcpcd waits for this address to appear), DHCP is not enabled by default. You can now enable it like so

interface wwan0
    dhcp

Or just add --dhcp on the command line.

ftp://roy.marples.name/pub/dhcpcd/dhcpcd-8.0.3.tar.xz
https://roy.marples.name/downloads/dhcpcd/dhcpcd-8.0.3.tar.xz

Here’s something I haven’t see before: at the time of me typing this, there are commits in DragonFly, FreeBSD, and I assume NetBSD (haven’t found the commit), but the 2019-5612 CVE entry is still shown as reserved and not public.  This may change by the time you read this article, of course.

Update: the original source, found by an intrepid reader.

First, I can use Java, but for what I want to achieve (building a database where othe only application supporting the format is in Java), I need 100Gb of RAM during 20 hours.

I have access to a server with the required RAM, but not as root and no JRE is available. The same is true for the Xorg libraries.

Here’s the uname :

8.0_STABLE NetBSD 8.0_STABLE (GENERIC) #0: Sun Feb 24 10:50:49 UTC 2019  [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC amd64

The Linux layer is installed, but nothing else is installed : not even Glibc, so the only applications which can be run are the ones which are statically compiled.

So not only Java isn’t Installed, but some of the require shared libraries are missing…
However, I have full write access to my $HOME directory, and I can run my own executables from there.

Is a way to convert a Jar file into a NetBsd Executable or Linux statically linked executable ? I have also the source code of the Jar file if compiling is an acceptable solution.

I only found about ɢᴄᴊ, but I’m unsure if Java 7 is supported…

Running my own email server has its challenges. Chief among them: my favorite mail-server software hasn’t been updated since I started using it in 1998.

The qmail logo

Okay, that’s not entirely true. While qmail hasn’t been updated by its original author, a group of respected users created netqmail, a series of tiny updates that were informed, conservative, and careful. By their design, it was safe for everyone running qmail to follow netqmail, so everyone did. But larger changes in the world of email — authentication, encryption, and ever-shifting anti-spam techniques — remained as puzzles for each qmail administrator to solve in their own way. And netqmail hasn’t been updated since 2007.

One fork per person

In the interim, devotees have continued maintaining their own individual qmail forks. Some have shared theirs publicly. I’ve preferred the design constraints of making minimal, purpose-specific, and conflict-avoidant add-ons and patches. Then again, these choices are motivated by the needs of my qmail packaging, which I suppose is itself a de facto fork.

I’ve found this solo work quite satisfying. I’ve learned more C, reduced build-time complexity, added run-time configurability, and published unusually polished and featureful qmail packages for over 20 platforms. Based on these experiences, I’ve given dozens of workshops and talks. In seeking to simplify system administration for myself and others, I’ve become a better programmer and consultant.

Still, wouldn’t it be more satisfying if we could somehow pool our efforts? If, long after the end of DJB’s brilliant one-man show, a handful of us could shift how we relate to this codebase — and to each other — in order to bring a collaborative open-source effort to life? If, with netqmail as inspiration, we could produce safe updates while also evolving qmail to meet more present-day needs?

One fork per community

My subtle artwork

Say hello to notqmail.

Our first release is informed, conservative, and careful — but bold. It reflects our brand-new team’s rapid convergence on where we’re going and how we’ll get there. In the span of a few weeks, we’ve:

• Started this project
• Grown to four active developers with diverse concerns, opinions, and skills
• Defined our big-picture goals (and non-goals)
• Identified milestones for future releases
• Agreed on standards for pull requests
• Merged only the changes that absolutely had to be in the first release
• Shipped our first release

I say “bold” because, for all the ways we intend to hew to qmail tradition, one of our explicit goals is a significant departure. Back in the day, qmail’s lack of license, redistribution restrictions, technical barriers, and social norms made it hard for OS integrators to create packages, and hard for package users to get help. netqmail 1.06 expressed a desire to change this. In notqmail 1.07, we’ve made packaging much easier. (I’ve already updated pkgsrc from netqmail to notqmail, and some of my colleagues have prepared notqmail RPM and .deb packages.) Further improvements for packagers are part of what’s slated for 1.08.

What’s next

Looking much further ahead, another of our explicit goals is “Meeting all common needs with OS-provided packages”. We have a long way to go. But we couldn’t be off to a better start.

By our design, we believe we’ve made it safe for everyone running qmail to follow notqmail. We hope you’ll vet our changes carefully, then update your installations to notqmail 1.07. We hope you’ll start observing us as we continue the work. We hope you’ll discuss freely on the qmail mailing list. We hope you’ll be a part of the qmail revival in ways that are comfortable for you. And we hope that, in the course of time, notqmail will prove to be the community-driven open-source successor to qmail.

This is a somewhat pre-made post coming off a week on the road, so I packed it chock-full.

• The UNIX Philosophy in 2019.  (via)
• Converse to previous: Brian Will – Replacing the Unix tradition.
• Summer Sun and microsystems.
• How does a “real” Unix system feel compared to *BSDs and Linux?
• GSoC 2019 Report Update: Incorporating the memory-hard Argon2 hashing scheme into NetBSD.
• Valuable News – 2019/08/05.
• Removal of route-collector mode in bgpd(8).
• tpmr(4) driver added to -current.
• OPNsense 19.7.2 released.
• Linux kernel’s inability to gracefully handle low memory pressure.  Linked cause you know what performs well under memory restriction?  Any BSD.  (via)
• Devine Lu Linvega’s Uses This entry.  Interesting to see his workstation details because his workstation is 2 machines, with Plan 9 variations, and segregated on work vs. distraction.  (and sometimes FreeBSD.)
• NetBSD light weight theme.
• Wanting to install Trident to boot from my USB stick…
• LibreSSL 3.0.0 Released.  (via)
• Play Windows Games On FreeBSD.  Encapsulating setup.  (via)

Once upon a time a developer wrote a temperature sensor driver for one system and decided to port it to another, similar, system. For days and then weeks, the developer waited and waited for the other, similar, system to arrive.

One day, joy of joys, the other, similar, system arrived. A National Day of Celebration commenced, and the porting effort began. Over hours, perhaps even as many as five hours of time, the sensors were finally able to tell you whether they were hot or not.

This other, similar, system suddenly was purposeless and was pondering what else life could provide when the Remote Server task popped up and said "Hey, first media file is free", and sadly, this other, similar, system tried its first media file, and then purchased many, many more media files, and suddenly this other, similar, system was suddenly hooked.

Unfortunately, this other, similar, system had a problem talking on the network without falling asleep on the job, so the developer says "let's try a USB network instead!", and initially this seemed like a good idea. Many bits were transferred over USB, but soon whispers of a lurking monster, known to developers, experience or otherwise, as KASSERT, were heard and soon found common.

The developer attempted other USB network as the other, similar, system was destined to be flown many thousands of miles away soon, but the only other option was similarly plagued by the KASSERT monster. The developer reached into his Bag Of Holding and pulled out a magical weapon known capable of slaying the KASSERT monster. The mighty blade of MPSAFE was free again!

After much planning and many failed attacks, the developer was able to slay the KASSERT monster and all the bits wanting to be transferred were able to be.

For a day and for a night there were celebrations. Much food and ale was consumed until finally everyone was asleep, and the music and lights were finally turned off. In the morning a great clean up was needed and while the developer was cleaning off the shiny, happy and working USB network the original USB network was accidentally reconnected. Oh no, the KASSERT monster has returned! Lock your doors and hide your children.

The developer quickly pulled out MPSAFE again, and once again the KASSERT monster was slain, though the face of this monster was different to the previous monster. The developer then searched and searched for surely they were more KASSERT monsters to be found. Indeed, many many others were found, though they retreated to safety after two more of their number were slain by the mighty MPSAFE.

The developer called upon his friends Shared and Code and with them forged a new weapon against the KASSERT monster, using the mighty MPSAFE in ways unheard of before. After much research and planning, and with the help of some friends, the USBNET was born. All the angels and all the birds of the world were said to sing all at once at this moment, for the USBNET would bring happiness to both the Code Deletionist and the Code Sharers, bound to war against each other from time immemorial.

With this new USBNET the developer was able to blaze a trail across the landscape, searching out each KASSERT monster lurking in every USB network corner. All told, fourteen faces of KASSERT monster were found and the developer and his friends have slain seven of these faces, with the remaining seven under attack, life looks grim for them.

The other, similar, system is safe now. Turns out that MPSAFE also has cleared up the sleeping problem using the cousins, NET and FDT in a tight, dual-blade combination.

Let the world rejoice, for soon the KASSERT monster will be no more!

--mrg @ 2019-08-11

tl;dr:

i fixed many bugs across several USB ethernet adapters and got sick of fixing the same bug across multiple drivers so made common code for them to use instead. the original 4 drivers fixed were axen(4), axe(4), cdce(4), and ure(4). the current list of fixed drivers, at time of writing, includes smsc(4), udav(4) and urndis(4). all drivers except umb(4) are ported but either not tested or not yet working with the usbnet(9) framework.

As NetBSD-9 has finally branched and I keep receiving requests to finish the integration of LLVM sanitizers, I have pushed this task forward too. I have also completed a few leftover tasks from my previous months that still needed fixes.

GNU GDB

Originally the GDB debugger was meant for local debugging. However with the request of tracing remote setups, it has been extended to remote debugging capabilities and this move standardized the protocol for many debuggers.

In fact the native / local / non-remote debugging is redundant with the remote capabilities, however this blog entry is not the right place to discuss the technical merits in comparision between them trying to convince someone. What matters: the developers of LLDB (LLVM debugger) already removed their local debugging variation (for Linux) and implements only the remote process plugin. The NetBSD support for LLDB started with this mode of copying the Linux approach. Bypassing no longer exists for Linux local-debugging-only support plugin.

Inside the GNU world, the transition from local process debugging to remote process debugging is progressing slowly. In the current state of affairs there is a need to support two separate (however whenever possible, with code sharing) plugins for each OS. Also, there is intention from some GNU debugger developers to completely drop the native-only process plugin to debugging code. The state in NetBSD one month ago handled only local-debugging and no code for remote process debugging. The state is the same with other BSD derived systems (FreeBSD, DragonFlyBSD...) and this state was in general kept alive with a reduced set of features compared to Linux.

I have decided to refresh the GDB support for NetBSD as this is still the primary debugger on NetBSD (LLDB support is still work-in-progress) for two primary reasons:

• Slight ptrace(2) changes in the kernel code can alter support in GDB/NetBSD. With proper NetBSD handling in GDB and ability to run its regression tests the fallout will be minimized.
• Certain ptrace(2) reported bugs are in fact specific GDB bugs. In order to close them as resolved I need to revamp and validate the GDB support.

I have managed to get a basic GDB client/server session to work for tracing a simple application. The following features have been implemented targetting as of now NetBSD/amd64:

• creating inferior (debuggee in GDB terminology)
• attaching
• killing
• detaching
• resuming (with optional: single step, signal, resuming single thread)
• waiting
• checking whether a thread is alive
• fetching registers
• storing registers
• reading memory
• writing memory
• requesting interruption
• auxv reading
• software breakpoint support
• hardware single step support
• svr4 ELF loader tracing
• retrieving exec file name from pid
• reading thread name

• i386 support
• multilib support amd64/i386
• reading TLS BASE (x86) register (there is still missing a ptrace(2) interface)
• EXEC events handling improvement in the generic code (that maps Linux restricted semantics only)
• hardware watchpoint/breakpoint support
• FPU support
• FORK, VFORK, POSIX_SPAWN events
• reading loadmap
• research for other features: multiprocess, mulifs, async, btrace, tracepoints, new gdb connections..

The main difficulty was with overly elaborated version of Linux proccess tracing in GDB that contains various workaround for various kernel bugs, handles differences between behavior of the Linux kernel (like different set of ptrace(2) calls or.. swapped arguments) between CPUs.. Finally, I have decided to base my work on unfortunately dated (and probably broken today) gdbserver for lynxos/i386 and then keep adding missing features that are implemented for Linux. I have passed most of the past month on debugging spurious crashes and anomalies in my GDB remote port to NetBSD. After much work, I have finally managed to get gdbserver to work and perform successful operations of spawning a process, inserting a breakpoint, single-stepping, detaching and running a process until its natural completion.

I plan to validate the native tracing support for NetBSD, checking whether it needs upgrading and planning to run the GDB regression tests as soon as possible.

LLVM changes

I have updated the list of ioctl(2) operations to the state as of 9.99.3 (+36 ioctls). The support for NVMM ioctl(2)s has been enabled and restricted as of now to amd64 only.

I have also finally managed to resolve the crash of dynamic (as DSO library) Address Sanitizer that blocked the integration with base in Januray 2019. My original suspect on the root cause was finally verified to be false (mismatch in the source code files composing .so libraries). The real reason to ship with broken dynamic ASan was inability to reuse the same implementation of TSD (Thread-Specific Data) in asan-dynamic. The static (default in LLVM) version of ASan can use in early initialization TLS (Thread-Local Storage), while TSD (Thread-Specific Data) is broken. It happened that the state with dynamic ASan is inversed and TLS breaks and TSD works.

I have also landed build rules for LLVM sanitizers into src/, for:

• asan
• ubsan
• tsan
• msan
• xray
• safestack
• libfuzzer

MAXRSS changes

During the work on libfuzzer last year for GSoC-2018 one of the changes that landed the kernel was the restoration of the MAXRSS option. MAXRSS presents the maxiumum resident set size of the process during its lifetime. In order to finalize this work, there was still need to enable printing it in ps(1). I have pushed a kernel fix to update the RSS related values for sysctl(3) query of a process statistics and with help of Krzysztof Lasocki reenabled printing MAXRSS, IDRSS (integral unshared data), ISRSS (integral unshared stack) and IXRSS (integral shared memory size) in ps(1). These values were commented out in the ps(1) program since the inception of NetBSD, the first registered commit.

$ ps -O maxrss,idrss,isrss,ixrss|head -3
 PID MAXRSS IDRSS ISRSS IXRSS TTY    STAT    TIME COMMAND
 910   5844    20    12   216 pts/0  Is   0:00.01 -ksh
1141   5844    20    12   152 pts/0  I+   0:00.01 mg event.h

These change have been backported to the NetBSD-9 branch and will land 9.0. Welcome back!

Plan for the next milestone

Start executing GDB regression tests. Keep enhancing GDB support. Keep detecting ptrace(2) bugs and addressing them.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:

http://netbsd.org/donations/#how-to-donate

Recently I started working on Fuzzing Filesystems on NetBSD using AFL.
You can take a look at the previous post to learn more details about background of this project.
This post summarizes the work that has been done in this area, and is divided into 3 sections:

1. Porting AFL kernel mode to work with NetBSD
2. Running kernel fuzzing benchmark
3. Example howto fuzzing particular Filesystem

AFL Port for NetBSD

AFL is a well known fuzzer for user space programs and libraries, but with some changes it can be used for fuzzing the kernel binary itself.

For the first step to fuzz the NetBSD kernel via AFL, I needed to modify it to use coverage data provided by the kernel instead of compiled instrumentations. My initial plan was to replace the coverage data gathered via afl-as with that provided by kcov(4). In this scenario, AFL would just run a wrapper and see the real coverage from the kernel.

I also saw previous work done by Oracle in this area, where instead of running the wrapper as a binary, the wrapper code was included in a custom library (.so object).

Both approaches have some pros and cons. One thing that convinced me to use a solution based on the shared library with initialization code was the potentially easier integration with remote fork server. AFL has some constraints in the way of managing fuzzed binary, and keeping it on a remote VM is less portable than fuzzing using a shared library and avoiding introducing changes to the original binary fuzzing.
Porting AFL kernel fuzzing mode to be compatible with NetBSD kernel mainly relied on how the operating system manages the coverage data. The port can be found currently on github.

Writing a kernel fuzzing benchmark

Performance is one of the key factors of fuzzing. If performance of the fuzzing process is not good enough, it's likely that the entire solution won't be useful in practice. In this section we will evaluate our fuzzer with a practice benchmark.

One exercise that I want to perform to check the AFL kernel fuzzing in practice is similar to a password cracking benchmark. The high level idea is that a fuzzer based on coverage should be much smarter than bruteforce or random generation.

To do this, we can write a simple program that will take a text input and compare it with a hardcoded value. If the values match, then the fuzzer cracked the password. Otherwise, it will perform another iteration with a modified input.

Instead of "password cracker", I called my kernel program "lottery dev". It's a character device that takes an input and compares it with a string.

The chance to find one 6 byte combination (or the "lucky byte" combination, because of the name) are similar to winning big in the lottery: every byte contains 8 bits, thus we have 2**(8*6) => 281,474,976,710,656 combinations. The coverage based fuzzer should be able to do this much quicker and in fewer iterations, as feedback from code instrumentations will show, compared to blindly guessing.

I performed a similar test using a simple C program: the program read stdio and compared it with the hardcoded pattern. If the pattern matches, the program panics, and otherwise returns zero. Such a test took an AFL about a few hours on my local laptop to break the challenge (some important details can make it faster). The curious reader who wants to learn some basics of AFL should try to do run a similar test on their machine.

I ran the fuzzer on my lottery dev for several days, and after almost a week it was still not able to find the combination. So something was fundamentally not right. The kernel module with the wrapper code can be found here.

Measuring Coverage for a particular function

In the previous article, I mentioned that the NetBSD kernel seems to be 'more verbose' in terms of coverage reporting. I ran my lottery dev wrapper code (the code that writes a given input to the char device) to check the coverage data using standard kcov(4) without the AFL module. My idea was to check the ratio between entries of my code that I wanted to track and other kernel functions that can be considered noise from other subsystems. These operations are caused by the context services, such as Memory Management, File Systems or Power Management, that are executed in the same process.

To my surprise, there was a lot of data, but I could not find any of functions from lottery dev. I quickly noticed that the amount of addresses is equal to the size of kcov(4) buffer, so maybe my data didn't fit in the buffer inside kernel space?

I changed the size of the coverage buffer to make it significantly larger and recompiled the kernel. With this change I reran the test. Now, with the buffer being large enough, I collected the data and printed top 20 entries with a number of occurrences. There were 30578 entries in total.

1544 /usr/netbsd/src/sys/uvm/uvm_page.c:847
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:869
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:890
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:880
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:858
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:70
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:71
 478 /usr/netbsd/src/sys/kern/kern_mutex.c:840
 456 /usr/netbsd/src/sys/arch/x86/x86/pmap.c:3046
 438 /usr/netbsd/src/sys/kern/kern_mutex.c:837
 438 /usr/netbsd/src/sys/kern/kern_mutex.c:835
 398 /usr/netbsd/src/sys/kern/kern_mutex.c:838
 383 /usr/netbsd/src/sys/uvm/uvm_page.c:186
 308 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:129
 307 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:130
 307 /usr/netbsd/src/sys/uvm/uvm_page.c:178
 307 /usr/netbsd/src/sys/uvm/uvm_page.c:1568
 231 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:135
 230 /usr/netbsd/src/sys/uvm/uvm_page.c:1567
 228 /usr/netbsd/src/sys/kern/kern_synch.c:416

It should not be a surprise that the coverage data does not much help our fuzzing with AFL. Most of the information that the fuzzer sees is related to UVM page management and machine-dependent code.

I decided to remove instrumentation from these most common functions to check the difference. Using the attribute no_instrument_function tells the compiler to not put instrumentation for coverage tracing inside these functions.

Unfortunately, after recompiling the kernel, the most common functions did not disappear from the list. As I figured out, the support in GCC 7 may not be fully in place.

GCC 8 for help

To solve this issue, I decided to work on reusing GCC 8 for building the NetBSD kernel. After fixing basic build warnings, I got my basic kernel working. This still needs more work to get kcov(4) fully functional. Hopefully, in the next report, I will be able to share these results.

Fuzzing Filesystem

Given what we already know, we can run Filesystem fuzzing. As a target I chose FFS as it is a default FS that is delivered with NetBSD.

The reader may ask the question: why would you run coverage based fuzzer if the data is not 100% accurate? So here is the trick: For coverage based fuzzers, it is usually recommended to leave the input format as is, as genetic algorithms can do a pretty good job here. There is great post on Michal Zalewski's Blog about this process applied for the JPEG format: "Pulling JPEGs out of thin air". But what will AFL do if we provide an input format that is already correct? We already know what a valid FS image should look like (or we can simply just generate one). As it turns out, AFL will start performing operations on the input in a similar way as mutation fuzzers do - another great source that explains this process can be found here: "Binary fuzzing strategies: what works, what doesn't"

Writing mount wrapper

As we discussed in the previous paragraph, to fuzz the kernel itself, we need some code to run operations inside the kernel. We will call it a wrapper, as it wraps the operations of every cycle of fuzzing. The first step to write a wrapper for AFL is to describe it as a sequence of operations. Bash style scripting is usually good enough to do that.
We need to have an input that will be modified by the fuzzer, and be able to mount it. NetBSD comes with vnd(4) that allows exposing regular files as block devices. The simplest sequence can be described as:

# Expose file from tmpfs as block device
vndconfig vnd0 /tmp/rand.tmp

# Create a new FS image on the blk dev that we created
newfs /dev/vnd0

# Mount our fresh FS
mount /dev/vnd0 /mnt

# Check if FS works fine
echo "FFS mounted!" > /mnt/test

# Undo mount
umount /mnt

# Last undo step
vndconfig -u vnd0

From bash to C and system calls

At this point, the reader has probably figured out that a shell script won't be the best approach for fuzzer usage. We need to change it to C code and use proper syscall/libc interfaces.

vndconfig uses the opendisk(3) combined with vnd_ioctl. mount(2) is a simple system call which can operate directly after file is added to vnd(4)

Below is an example conceptual code for mounting an FS:

	// Structure required by mount()
	struct ufs_args ufs_args;

	// VNConfigure step
	rv = run_config(VND_CONFIG, dev, fpath);
	if (rv) 
		printf("VND_CONFIG failed: rv: %d\n", rv);

	// Mount FS
	if (mount(FS_TYPE, fs_name, mntflags, &ufs_args, 8) == -1) {
		printf("Mount failed: %s", strerror(errno));
	} else {
		// Here FS is mounted
		// We can perform any other operations on it
	
		// Umount FS
		if (unmount(fs_name, 0) == -1) printf("#: Umount failed!\n");
	}

	// VNC-unconfigure
	rv = run_config(VND_UNCONFIG, dev, fpath);
	if (rv) {
		printf("VND_UNCONFIG failed: rv: %d\n", rv);
	}

The complete code can be found here

Ready to fuzz FFS! aka Running FS Fuzzing with a predifined corpus

The first thing that we need to do is to have a wrapper that provides mount/umount functionality. In the previous section, we have already shown how that can be done. For now, we will be fuzzing the same kernel that we are running on. Isn't that dangerous? Taking a saw to the branch we are sitting on? Of course it is! In this exercise I want to illustrate an idea from a technical perspective so the curious reader is able to understand it better and do any modifications by their own. The take away from this exercise is that the fuzzing target is the kernel itself, the same binary that is running the fuzzing process.

Let's come back to the wrapper code. We've already discussed how it works. Now we need to compile it as a shared library. This is not obvious, but should be easy to understand after we already brought this sawing-off metaphor.

To compile the so object:

gcc -fPIC -lutil  -g -shared ./wrapper_mount.c -o wrapper_mount.so 

Now we need to create the input corpus, for the first attempt we will use a large enough empty file.

dd if=/dev/zero of=./in/test1 bs=10k count=8

And finally run. The @@ tells AFL to put here the name of input file that will be used for fuzzing.

./afl-fuzz -k -i ./in -o ./out -- /mypath/wrapper_mount.so @@

Now, as we described earlier, we need a proper FS image to allow AFL perform mutations on it. The only difference is the additional newfs(8) command.

# We need a file, big enough to fit FS image but not too big
dd if=/dev/zero of=./in/test1 bs=10k count=8

# A block is already inside fuzzer ./in
vndconfig vnd0 ./in/test1

# Create new FFS filesystem
newfs /dev/vnd0

vndconfig -u vnd0

Now we are ready for another run!

./afl-fuzz -k -i ./in -o ./out -- /mypath/wrapper_mount.so @@



                  american fuzzy lop 2.35b (wrapper_mount.so)

┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐
│        run time : 0 days, 0 hrs, 0 min, 17 sec       │  cycles done : 0      │
│   last new path : none seen yet                      │  total paths : 1      │
│ last uniq crash : none seen yet                      │ uniq crashes : 0      │
│  last uniq hang : none seen yet                      │   uniq hangs : 0      │
├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤
│  now processing : 0 (0.00%)         │    map density : 17.28% / 17.31%       │
│ paths timed out : 0 (0.00%)         │ count coverage : 3.53 bits/tuple       │
├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤
│  now trying : trim 512/512          │ favored paths : 1 (100.00%)            │
│ stage execs : 15/160 (9.38%)        │  new edges on : 1 (100.00%)            │
│ total execs : 202                   │ total crashes : 0 (0 unique)           │
│  exec speed : 47.74/sec (slow!)     │   total hangs : 0 (0 unique)           │
├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤
│   bit flips : 0/0, 0/0, 0/0                         │    levels : 1          │
│  byte flips : 0/0, 0/0, 0/0                         │   pending : 1          │
│ arithmetics : 0/0, 0/0, 0/0                         │  pend fav : 1          │
│  known ints : 0/0, 0/0, 0/0                         │ own finds : 0          │
│  dictionary : 0/0, 0/0, 0/0                         │  imported : n/a        │
│       havoc : 0/0, 0/0                              │ stability : 23.66%     │
│        trim : n/a, n/a                              ├────────────────────────┘
┴─────────────────────────────────────────────────────┘             [cpu:  0%]

Future work

Support for the NetBSD kernel fuzzing was developed as a part of the AFL FileSystems Fuzzing project, which aims to improve quality of filesystems and catch various issues.
The very next thing that I have on my todo list is to provide support for kernel tracing on GCC 8 to turn off coverage data from other functions that generate a lot of noise.
During the FFS fuzzing, I found a few issues that I need to analyze in detail.
Last but not least, for the next report I plan to show a remote setup of AFL running on a VM, reporting crashes, and being remotely rebooted by the master controller.

I'm trying to use resize_ffs with netbsd to increase the size of my partition. I have NetBSD running in a virtual machine, and have expanded the size of the disk, and now wish to grow the partition to match.

The man_page for the tool is here

https://netbsd.gw.com/cgi-bin/man-cgi?resize_ffs+8+NetBSD-7.0

I am trying to grow a 300mb partition to 1gb.

The tool manpage says that specifiying a size is not mandatory, and that if it is not specified it will grow to use available space (ideal behaviour), however this results in an error saying newsize not known.

I have used various online tools to try and calculate the disk geomtery, but no matter what I try when I pass a number to -s, I get the error 'bad magic number'.

I have been unable to find example of using this tool online.

What is the correct way to use resize_ffs to grow a partition to use available disk space?

A reminder: if you have a BSD user group and I’m not posting about your meetings, please make sure I know about it.

• The OpenBSD Ada Library.  (via)
• BSD Link Roundup 7.29.
• Project Trident 19.07-U1 now available.  No, wait, 12-U1 now available.
• Valuable News – 2019/07/29.
• Nvidia optimus drivers for FreeBSD.  (via)
• OpenBGPD 6.5p1 released.
• Something blogged (on pkgsrcCon 2019).
• Installing Owntracks recorder on FreeBSD.
• Mod-able voxel engine written in common lisp.  (via)
• Thief-like FOSS stealth game “The Dark Mod” running on OpenBSD.  (via)
• Obviously related: Let’s Try on OpenBSD: The Dark Mod (Thief-inspired stealth game).
• Upcoming changes to LISA and SREcon.

This report was prepared by Manikishan Ghantasala as a part of Google Summer of Code 2019

This report encloses the progress of the project Add KNF (NetBSD style) clang-format configuration during the second coding period of GSoC 2019.

Clang-format

Clang-format is a powerful code formatter which is a part of clang. Clang-format formats the code either by a configuration file .clang-format or can be chosen from some predefined coding styles namely LLVM, Google, Chromium, Mozilla, WebKit.

The final goal of the project is to add a new style NetBSD along with them by patching the libFormat to support the missing styles and add the configuration according to NetBSD KNF.

Style options introduced in the first coding period:

1. BitFieldDeclarationsOnePerLine
2. SortNetBSDIncludes

You can also take a look at the first report to learn more about these style options.

Work done in the second coding period

I have worked on the following styles during this second coding period.

1. Withdrawn SortNetBSDIncludes and modified existing sortIncludes.
2. Modified spacesBeforeTralingComments to support block comments.
3. New style option alignConsecutiveListElements introduced.

Sortincludes:

The native SortIncludes sorts the includes/headers in alphabetical order. And we also have IncludeCategories which allows setting custom priorities to group the includes matching via a regex after sorting.

Example:

Configuration:

    
    IncludeCategories:
        -Regex:	        ^<(a|b|c)/
         Priority:	1

        -Regex:	        ^<(foo)/
         Priority:	2

        -Regex:	        .*
         Priority:	3
    

Input

    
        #include exe.h
        #include gaz.h
        #include <a/dee.h>
        #include <foo/b.h>
        #include <a/bee.h>
        #include <exc.h>
        #include <b/dee.h>
        #include <c/abc.h>
        #include <foo/a.h>

    

Output

    
        #include <a/bee.h>
        #include <a/dee.h>
        #include <b/dee.h>
        #include <c/abc.h>

        #include <foo/a.h>
        #include <foo/b.h>

        #include <exc.h>
        #include gaz.h>


    

Modified SortIncludes

The new sortIncludes supports to give custom priority for sorting in addition to grouping priority. Now the IncludeCategories have a new field named SortPriority along with Priority to set the priority while sorting the includes, and the default priority will be alphabetical. The usage and working of the new IncludeCategories style shown in the following example.

Example

    
    IncludeCategories:
        -Regex:	        <^c/
         Priority:	1
         SortPriority:	0

        -Regex:	        ^<(a|b)/
         Priority:	1
         SortPriority:	1

        -Regex:	        ^<(foo)/
         Priority:	2

        -Regex:	        .*
         Priority:	3   
    

Input

    
        #include exe.h
        #include <a/dee.h>
        #include <foo/b.h>
        #include <a/bee.h>
        #include <exc.h>
        #include <b/dee.h>
        #include <c/abc.h>
        #include <foo/a.h>
    

Output

    
        #include <c/abc.h>
        #include <a/bee.h>
        #include <a/dee.h>
        #include <b/dee.h>
        
        #include <foo/a.h>
        #include <foo/b.h>
        
        #include <exc.h>
        #include gaz.h
        
    

As we observe in the above example, the includes having the same Priority are grouped, and SortPriority defines the sort priority.

The patch was accepted and ready to merge, and you can find the patch here -> SortIncludesPatch. This patch also introduces the NetBSD style to clang-format with configurations to supported styles.

Spaces Before Trailing Comments

This option is also a native style option in clang-format which enables a user to decide the number of spaces to be given before trailing comments (// - comments) but not block comments (/* - comments). The reason for this is that block comments have different usage patterns and different exceptional cases. The following is an example of the native style option.

    
    SpacesBeforeTrailingComments: 3

    void f() {
    	if (true) {  //foo
            f();     //bar
    	}            //foo1
    }

    

Modifications to spaceBeforeTrailingComments:

I am working on modifying this style option to support block comments by covering cases where block comments are used. There are some cases yet to be covered. Once the patch is ready, this is the deliverable

The initial revision can be found here -> patch

    
    SpacesBeforeTrailingComments: 3


    void f() {
        if (true) {  /*foo */
            f();     /*bar */
    	}            /*foo1*/
    }
    

AlignConsecutiveListElements

AlignConsecutiveListElements is a new style option that I am going to introduce to clang-format. This style option aligns elements in consecutive definitions, declarations of lists. The style is not yet ready to patch, and I have to modify a lot of functionalities in alignTokens() which aligns tokens to support this style.

Example:

Input

    
    keys[] = {
        	{"all", f_all, 0 },
        	{ "cbreak", f_cbreak, F_OFFOK },
        	{"cols", f_columns, F_NEEDARG },
        	{ "columns", f_columns, F_NEEDARG },
        };
    

Output

    
    keys[] = {
        	{ "all",        f_all,      0 },
        	{ "cbreak",     _cbreak,    F_OFFOK },
        	{ "cols",       f_columns,  F_NEEDARG },
        	{ "columns",    f_columns,  F_NEEDARG },
        };
    

The blocker to this style is the nested structure for list declarations, alignTokens() should be equipped to parse the nested list declarations to support this style option. I will make sure this style option is available by the next report.

Further plans

For the next phase, I will make all the style options that were modified or introduced till now during the first two phases mergeable to upstream along with required unit tests. With these style options ready, I will test the new clang-format with NetBSD style patched across NetBSD source and check for bugs. After the testing will try to fix the bugs and get the NetBSDStyle ready for the final evaluation.

Summary

In the final coding period, the main focus will be on testing the new/modified style options and fix them. Add any missing styles for NetBSD and get the NetBSDStyle by the final evaluation.

I want to thank my mentors Michal, Christos and developers of both LLVM and NetBSD for supporting to complete this project.

A bumper crop this week!

• Video on OpenBSD, a presentation at NYCBUG by Andre Buskvekster, happening on the 7th.  I’ll post a reminder on the 6th.
• OpenBSD ttyplot examples.
• Something blogged (on pkgsrcCon 2019).
• NetBSD 9.0 release process has started.  Aarch64 support sounds good.
• TrueNAS 11.2 out.
• PHP and OpenBSD talk by Mario Campos on August 12th at ChiPHPUG lightning talks.
• OpenBGPD: The OpenBSD BGP internet routing daemon.  The link source comments are the useful part.
• FreeBSD-SA-19:12.telnet: Were these claims even properly investigated, or did Juniper bully an advisory through?  This is a spectator sport item, really, if you can’t tell from the title.
• Exploiting a No-Name FreeBSD Kernel Vulnerability.  (via)
• Porting wine to amd64 on NetBSD, second evaluation report.
• Adapting TriforceAFL for NetBSD, Part 2.
• Enchancing Syzkaller Support for NetBSD, Part 2.  Enhancing, I think it should be.
• Work-in-progress threading support in LLDB.
• Touchpad, Interrupted.

I just installed NetBSD 7.1.1 (i386) on my old laptop.

During the installation, I could not install pgkin (I don't know why), so I skipped it and now I have a NetBSD 7.1.1 installed on my laptop without pkgin.

My problem is "How to install pkgin on NetBSD (i386) ?"

I found this (Click) tutorial and I followed it:

I tried :

#export PKG_PATH="http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/6.0_BETA_current/All/"
# pkg_add -v pkgin

And I got :

pkg_add: Can't process ftp://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/amd64/6.0_BETA_current/All/%0d/pkgin*: Not Found
pkg_add: no pkg found for 'pkgin',sorry.
pkg_add: 1 package addition failed

I know this is a wrong command because this ftp address is for amd64 while my laptop and this NetBSD is i386. (I can't find the correct command for i386 )

I also followed instructions of pkgin.net (Click), and I did

git clone https://github.com/NetBSDfr/pkgin.git

on another computer and copied the output (which is a folder name pkgin) to my NetBSD (my NetBSD doesn't have 'git' command)

and then I did :

./configure --prefix=/usr/pkg --with-libraries=/usr/pkg/lib --with-includes=/usr/pkg/include

and then :

make

but I got :

#   compile  pkgin/summary.o
gcc -O2    -std=gnu99    -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-sign-compare  -Wno-traditional  -Wa,--fatal-warnings -Wreturn-type -Wswitch -Wshadow -Werror    -DPKGIN_VERSION=\""0.9.4 for NetBSD-7.1.1 i386"\" -DNETBSD  -g -DLOCALBASE=\"/usr/local\"           -DPKG_SYSCONFDIR=\"/usr/local/etc\"         -DPKG_DBDIR="\"/var/db/pkg\""           -DDEF_LOG_DIR="\"/var/db/pkg\""         -DPKGIN_DB=\"/var/db/pkgin\"            -DPKGTOOLS=\"/usr/local/sbin\" -DHAVE_CONFIG_H -D_LARGEFILE_SOURCE -D_LARGE_FILES -DCHECK_MACHINE_ARCH=\"i386\" -Iexternal -I. -I/usr/local/include  -c    summary.c
*** Error code 1

Stop.
make: stopped in /root/pkgin

I think this error occurs because of the dependencies. (which is mentioned in pkgin.net) but still, don't know how to install those dependencies.

EDIT: I found "http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/7.1.1/All/" but it still says

no pkg fond for 'pkgin', sorry

SOLVED:

** I solved the problem by writing 7.1 instead of 7.1.1**

With the following changes:

• NetBSD: Can be build without ARP support but listen to kernel DaD
• ND6: Removed NA support from SMALL builds
• ND6: Remove and warn about NA issues on OS's other than NetBSD and Linux
• script: /tmp files are now cleaned up for systems without open_memstream(3)
• configure: open_memstream(3) detected on recent glibc
• DHCP: Avoid duplicate read of UDP socket when BPF is also open
• IP: Avoid adding address if already exists on OS other than Linux
• IP6: Avoid adding address is already exists on Solaris
• route: Fixed a NULL de-reference error on statically configured routes
• DHCP6: Move to REQUEST when any IA has error no-binding in RENEW/REBIND
• DragonFlyBSD: Now compiles and works for
• IP: Accept packets with IP header options

I don't anticipate any more releases for a while as this is looking really good now!

ftp://roy.marples.name/pub/dhcpcd/dhcpcd-8.0.2.tar.xz
https://roy.marples.name/downloads/dhcpcd/dhcpcd-8.0.2.tar.xz

Is there any way to install software from the *.tgz file that is its package, in NetBSD? Or indeed in operating systems with similar package managers such as OpenBSD or FreeBSD?

For example, I can install the nano editor on NetBSD using this command:

pkgin nano

(I could do the same with a similar pkg install nano command on FreeBSD.)

What if I download the package file directly from the operating system's package repository, which would be a URL like http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/7.1/All/nano-2.8.7.tgz?

Having obtained the package file from the repository by hand like this, is there any way to now install nano directly from it? How do I do that?

All over the map this week.

• Broken ports?  (OpenBSD games)
• Realtime bandwidth terminal graph visualization.
• What’s the point of the Single UNIX Specification?
• Resuming ZFS send.
• A Chapter from the FBI’s History with OpenBSD and an OpenSSH Vuln.  (via).
• OpenBSD and NetBSD machines at Open Source Conference 2019 Nagoya.  (via)
• Valuable News – 2019/07/22.
• NetBSD audio – an application perspective.  (via)
• Project Trident STABLE-12 Now Available.  (via)
• A week of pkgsrc #13.
• OPNsense 19.7.1 released.
• New FreeNAS Mini Entry-Level & High-End Models Unveiled by iXsystems.
• Bill Joy interview, 1984.  I once heard him described as a flat-food person.  (PDF, via)

Just three changes of note

• Compile fixes for various build options on various platforms
• IPv4LL is fixed on NetBSD-8
• script buffers are freed on forking

ftp://roy.marples.name/pub/dhcpcd/dhcpcd-8.0.1.tar.xz
https://roy.marples.name/downloads/dhcpcd/dhcpcd-8.0.1.tar.xz

I'm trying to compile fIcy (https://gitlab.com/wavexx/fIcy) for NetBSD/FreeBSD.

When I'm executing the make command nothing happens. Even no error message.

The same source package compiles without problems with Debian 10.

Is the Makefile even compatible with BSD?

https://gitlab.com/wavexx/fIcy/blob/master/Makefile

The commands I used so far on FreeBSD 12:

pkg install gcc
wget https://gitlab.com/wavexx/fIcy/-/archive/master/fIcy-master.tar.gz
tar xfvz fIcy-master.tar.gz
cd fIcy-master
make

type make
make is /usr/bin/make

Huge update! The big points are:

• ARP now supports many requests
• Routing tables now use Red-Black Trees
• Script variables are no longer allocated manually

dhcpcd-7 branch has now entered maintainance only mode, which means it only gets security updates. Minor changes from dhcpcd-7.2.3 include:

• DHCP addresses are added with vltime of the lease time and pltime of the rebind time (Linux only)

ftp://roy.marples.name/pub/dhcpcd/dhcpcd-8.0.0.tar.xz
https://roy.marples.name/downloads/dhcpcd/dhcpcd-8.0.0.tar.xz

open_memstream is one of the more important functions added to POSIX libc of late. It's so important because it makes the generation of strings really easy - you no longer need to care about allocating the right amount of memory as the library will do it for you. Now, there's many functions that already help with this, such as asprintf but that's not standard and if you want to create many strings in one area you still need to care about the size of the area. You want to create an area if you have many strings, because it's more efficient for malloc and if you keep the area around and re-use it then it avoids memory fragmentation.

Now, to be clear, you have been able to do this since forever using fopen, writing to the file and then allocating your area based on the final file size. Still, it requires some memory management still but more importantly it writes to a file. Writing to a file is slow and reduces the life span of the disk you're writing to. It's only been fairly recently that tmpfs was a thing, but even today not all OS's have /tmp mounted as tmpfs. Requiring this isn't exactly ideal for a generic program to do - the setup and install should be easy. Because of all these reasons, most programs worked the string length needed and either allocated an area or string and then finally write the string. However, while saving the disk, it's also a lot more error prone because you need to work out the length of everything and that's not always trivial, especially for things like a DHCP client which is always building strings based on the information given by the DHCP server.

Here's an example of open_memstream in action:

/*
 * Example program which manages an area of environment strings
 * to send to child programs.
 */

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

static const char *foo = "foo";
static const char *bar = "bar";

int main(void)
{
        char *argv[] = { "/usr/bin/env", NULL };
        char *buf, *ep, *p, **env, **envp;
        size_t buflen, nenv, i;
        FILE *fp = open_memstream(&buf, &buflen);

        fprintf(fp, "FOO=%s", foo);
        fputc('\0', fp);
        fprintf(fp, "BAR=%s", bar);
        fputc('\0', fp);

        /* We could keep fp around as our area and just rewind it. */
        fclose(fp);

        /* execve relies on a trailing NULL */
        nenv = 1;
        for (p = buf, ep = p + buflen; p < ep; p++) {
                if (*p == '\0')
                        nenv++;
        }

        /* reallocarray(3) should be standard really */
        envp = env = malloc(nenv * sizeof(char *));
        *envp++ = buf;
        for (p = buf, ep--; p < ep; p++) {
                if (*p == '\0')
                        *envp++ = p + 1;
        }
        *envp = NULL;

        execve(argv[0], argv, env);
}

As you can see, we only manage the environment array handed to to execve - open_memstream is managing our string area and fprintf is working out the length each string needs to be for us. This vastly reduces the complexity and increases the security and reliabilty of creating large environment strings, which most DHCP clients do. We could also write a helper function to write the string AND the trailing NULL terminator for the string to be more efficient. You'll get to see this in dhcpcd-8 which should be released later this year.

This week I received my printed proof from the printer and enabled it to be printed. It is now for sale at Amazon and Barnes and Noble,

I set an alarm to work on it very early a few days a week and it took me a few years. (I am blessed to only commute a few times a year, so I make sure that I don't waste that gifted time.)

This book was written using Docbook using NetBSD and vi. The print-ready book was generated with Dblatex version 0.3.10 with a custom stylesheet, pdfTeX 3.14159265-2.6-1.40.19 (Web2C 2018), and the TeX document production system installed via Tex Live and Pkgsrc. Several scripts and templates were created to help have a consistent document.

The book work was managed using the Subversion version control software. I carefully outlined my steps in utilizing the useful interfaces and identified every web and console interface. The basic writing process included adding over 350 special comment tags in the docbook source files that identified topics to cover and for every pfSense web interface PHP script (highlighting if they were main webpages from the pfSense menu). As content was written, I updated these special comments with a current status. A periodic script checked the docbook files and the generated book and reported on writing progress and current needs.

During this writing, nearly every interface was tested. In addition, code and configurations were often temporarily customized to simulate various pfSense behaviors and system situations. Most of the pfSense interface and low-level source code was studied, which helped with identifying pfSense configurations and features that didn't display in standard setups and all of its options. The software was upgraded several times and installed and ran in multiple VMs and hardware environments with many wireless and network cards, including with IPv6. In addition, third-party documentation and even source code was researched to help explain pfSense configurations and behaviors.

As part of this effort, I documented 352 bugs (some minor and some significant) and code suggestions that I found from code reading or from actual use of the system. (I need to post that.)

The first subversion commit for this book was in July 2014. It has commits in 39 different months with 656 commits total. The book's docbook source had 3789 non-printed comments and 56,193 non-blank lines of text. The generated book has over 180,000 words. My subversion logs show I have commits on 41 different Saturdays. Just re-reading with cleanup took me approximately 160 hours.

For a command like this one on Linux debian-linux 4.19.0-1-amd64 #1 SMP Debian 4.19.12-1 (2018-12-22) x86_64 GNU/Linux with xfce I get :

[email protected]:~$ dbus-send --system --type=method_call --print-reply --dest
=org.freedesktop.DBus /org/freedesktop/DBus org.freedesktop.DBus.ListActivatable  
Names

The same command on OpenBSD LeOpenBSD 6.4 GENERIC.MP#364 amd64 with xfce I get :

ktop/DBus org.freedesktop.DBus.ListActivatableNames   <

On linux, at the end of screen, we go to next line.
On BSD(OpenBSD-NetBSD), the command line continue on the same line and the first words disapear.
It's the same in xfce-terminal-emulator, xterm or in TTY (Alt-Ctrl-F3)

I try to add am in gettytab in the defaut section with no avail.
Termcap man page say :
If the display wraps around to the beginning of the next line when the cursor reaches the right margin, then it should have the am capability.
What can I do ?

I am trying to set up the farm of webservers, consisting of the internal, external and worker servers.

1. The actual sites content is stored on internal NFS server deep in internal network. All sites contents management is centralized.

2. BSD-based external servers have Lighttpd doing all the HTTP/HTTPS job, serving static content. Main NFS share is auto-mounted via special path, like /net/server/export/www/site/ (via amd).

3. Every Lighttpd have fastcgi parameters pointing to several worker servers, which have php-fpm working (for example). Different sites may require different php versions or arrangement, so www01 and www02 may serve site "A" having php-fpm over PHP 5.6 and www05 and www06 will serve site "B" having php-fpm over PHP 7.2.

4. Every worker get requests for certain sites (one or more) with path /net/server/export/www/site and execute PHP or any other code. They also have amd (for BSD) and autofs (for Linux) working.

5. For some sites Lighttpd may not forward fastcgi, but do proxying instead, so workers can have Apache or other web-server (even Java-based) working.

External servers are always BSD, internal servers too, but workers can be different upon actual needs.

This all work good when workers are BSD. If we are using Linux on workers - it stops working when share is automatically unmounted. When one tries to access the site he will get error 404. When I connect to server via ssh I will see no mounted share on "df -h". If I do any "ls" on /net/server/export - it is self-mounted as intended and site starts to work. On BSD-systems df show amd shares always mounted despite of 60 seconds dismount period.

I believe there is a difference between amd and autofs approach, php-fpm calls on Linux become some kind of "invisible" to autofs and do not cause auto-mount, because any other access to /net/server/ work at any time and do cause auto-mount. Also, this happens not with php-fpm only, Apache serving static content on auto-mounted NFS share behave same way.

Sorry for long description, but I tried to describe it good. The main question here - is anyone know why calls to /net/server may not cause auto-mount in autofs and how to prevent this behavior.

For lot of reasons I do not consider using static mounting, so this is not an option here. As for Linux versions - mostly it was tested on OEL 7.recent.

All of a sudden (read: without changing any parameters) my netbsd virtualmachine started acting oddly. The symptoms concern ssh tunneling.

From my laptop I launch:

$ ssh -L 7000:localhost:7000 [email protected] -N -v

Then, in another shell:

$ irssi -c localhost -p 7000

The ssh debug says:

debug1: Connection to port 7000 forwarding to localhost port 7000 requested.
debug1: channel 2: new [direct-tcpip]
channel 2: open failed: connect failed: Connection refused
debug1: channel 2: free: direct-tcpip: listening port 7000 for localhost port 7000, connect from 127.0.0.1 port 53954, nchannels 3

I tried also with localhost:80 to connect to the (remote) web server, with identical results.

The remote host runs NetBSD:

bash-4.2# uname -a
NetBSD host 5.1_STABLE NetBSD 5.1_STABLE (XEN3PAE_DOMU) #6: Fri Nov  4 16:56:31 MET 2011  [email protected]:/m/obj/m/src/sys/arch/i386/compile/XEN3PAE_DOMU i386

I am a bit lost. I tried running tcpdump on the remote host, and I spotted these 'bad chksum':

09:25:55.823849 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 67, bad cksum 0 (->3cb3)!) 127.0.0.1.54381 > 127.0.0.1.7000: P, cksum 0xfe37 (incorrect (-> 0xa801), 1622402406:1622402421(15) ack 1635127887 win 4096 <nop,nop,timestamp 5002727 5002603>

I tried restarting the ssh daemon to no avail. I haven't rebooted yet - perhaps somebody here can suggest other diagnostics. I think it might either be the virtual network card driver, or somebody rooted our ssh.

Ideas..?

I am experimenting with NetBSD and seeing if I can get the Fenrir screenreader to run on it. However, I hit a snag post install; the console that I was using for the installation was working perfectly fine, however it stopped working alltogether once I completed the install. For reference, here is the line I used for virt-install:

virt-install --connect qemu:///system -n netbsd-testing \
             --ram 4096 --vcpus=8 \
             --cpu=host \
             -c /home/sektor/Downloads/boot-com.iso  \
             --os-type=netbsd --os-variant=netbsd8.0 \
             --disk=pool=devel,size=100,format=qcow2 \
             -w network=default --nographics 

When it asked me for the type of terminal I was using (this being the NetBSD install program), I accepted the default which was VT200. As I recall, I told it to use the BIOS for booting, and not any of the comm serial ports. Has anyone had any further experience with using no graphics on a Libvirt virtualized machine, and have any points as to how to get a working console?

Thanks.

After full installation of latest NetBSD I'm tried to launch pkgin, but received pkgin not found, also I've got same for pkgsrc. Then I've found, that there's no /usr/pkg location.

That's normal or I've did something wrong?

Is there some form of automated NetBSD installation?

I would like to automate this manual process: https://www.netbsd.org/docs/guide/en/chap-exinst.html

But I found no examples.

For some reason, I want develop program which can work on netbsd 1.4 or 1.5 powerpc ,target cpu is power750(powerpc platform,nearly 20 years old system),but I can't find how to make this kind cross-compile enviroment vmware host:i386 netbsd 1.5 + egcs1.1.1 + binutils 2.9.1 ---> target host:macppc powerpc netbsd 1.5 + egcs 1.1.1 I download and install netbsd 1.5 vmware and download pkgsrc,when I make /usr/src/pkgsrc/cross/powerpc-netbsd,I got gcc work on i386 but not cross-gcc,why? Thank you if any help!

On Wednesday, March 6, I attended New York City BSD User Group and presented Maintaining qmail in 2019. This one pairs nicely with my recent DevOpsDays Ignite talk about why and how to Run Your @wn Email Server! That this particular “how” could be explained in 5 minutes is remarkable, if I may say so myself. In this NYCBUG talk — my first since 2014 — I show my work. It’s a real-world, open-source tale of methodically, incrementally reducing complexity in order to afford added functionality.

My abstract:

qmail 1.03 was notoriously bothersome to deploy. Twenty years later, for common use cases, I’ve finally made it pretty easy. If you want to try it out, I’ll help! (Don’t worry, it’s even easier to uninstall.) Or just listen as I share the sequence of stepwise improvements from then to now — including pkgsrc packaging, new code, and testing on lots of platforms — as well as the reasons I keep finding this project worthwhile.

• Slides

Here’s the video:

I want to perform an open redirect so,

https://ytrezq.sdfeu.org/flashredirect/?endpoint=https:://www.google.fr/

would redirect to https://www.google.fr.
Here’s /index.cgi which of course has exec permissions.

#!/usr/pkg/bin/php
<?php
header("Location: ".$_GET["endpoint"], true, 307);
?>

and Here’s /flashredirect/.htaccess

Options FollowSymLinks
Options +ExecCGI
AddHandler cgi-script .cgi .pl
RewriteEngine On
RewriteBase /
FallbackResource /index.cgi

Obviously, there’s an error somewhere but where ? Also accessing error logs is payfull on sdf-eu.org so I can’t know the problem.

I have simple code:

 #include <stdio.h>

 int main()
 {
      //char d[10] = {0x13, 0x43, 0x9b, 0x64, 0x28, 0xf8, 0xff, 0x7f, 0x00, 0x00};
      //long double rd = *(long double*)&d;
      long double rd = 3.3621e-4932L;
      printf("%Le\n", rd);
      return 0;
 }

On my Ubuntu x64 it prints as expected 3.362100e-4932. On my NetBSD it prints 1.681050e-4932

Why it happens and how can I fix it? I try clang and gcc with same result.

My system (VM inside VirtualBox 5.0):

 uname -a
 NetBSD netbsd.home 7.0 NetBSD 7.0 (GENERIC.201509250726Z) amd64

 gcc --version
 gcc (nb2 20150115) 4.8.4

 clang --version
 clang version 3.6.2 (tags/RELEASE_362/final)
 Target: x86_64--netbsd
 Thread model: posix

FYI

/usr/include/x86/float.h defines as LDBL_MIN as 3.3621031431120935063E-4932L And this value is greater than printf result.

I am using NetBSD 5.1 for x86 systems. While studying some driver related code, I see that we use splraise and spllower to block or allow interrupts. I searched some of the mechanisms on internet to understand how these mechanisms work in reality. Did not get any real info on that.

When I disassembled I got the mechanism but still do not understand how all these assembly instruction yield me the result. I know x86 instruction individually, but not how the whole stuff works in its entirety.

Need your help in understanding its principles for x86 system. I understand that we need to disable Interrupt Enable (IE) bit, but this assembly seems to be doing more than just this work. Need help.

  (gdb) x/50i splraise
   0xc0100d40:  mov    0x4(%esp),%edx
   0xc0100d44:  mov    %fs:0x214,%eax
   0xc0100d4a:  cmp    %edx,%eax
   0xc0100d4c:  ja     0xc0100d55
   0xc0100d4e:  mov    %edx,%fs:0x214
   0xc0100d55:  ret
   0xc0100d56:  lea    0x0(%esi),%esi
   0xc0100d59:  lea    0x0(%edi,%eiz,1),%edi
   (gdb) p spllower
   $38 = {<text variable, no debug info>} 0xc0100d60
   0xc0100d60:  mov    0x4(%esp),%ecx
   0xc0100d64:  mov    %fs:0x214,%edx
   0xc0100d6b:  cmp    %edx,%ecx
   0xc0100d6d:  push   %ebx
   0xc0100d6e:  jae,pn 0xc0100d8f
   0xc0100d71:  mov    %fs:0x210,%eax
   0xc0100d77:  test   %eax,%fs:0x244(,%ecx,4)
   0xc0100d7f:  mov    %eax,%ebx
   0xc0100d81:  jne,pn 0xc0100d91
   0xc0100d84:  cmpxchg8b %fs:0x210
   0xc0100d8c:  jne,pn 0xc0100d71
   0xc0100d8f:  pop    %ebx
   0xc0100d90:  ret
   0xc0100d91:  pop    %ebx
   0xc0100d92:  jmp    0xc0100df0
   0xc0100d97:  mov    %esi,%esi
   0xc0100d99:  lea    0x0(%edi,%eiz,1),%edi
   0xc0100da0:  mov    0x4(%esp),%ecx
   0xc0100da4:  mov    %fs:0x214,%edx
   0xc0100dab:  cmp    %edx,%ecx
   0xc0100dad:  push   %ebx
   0xc0100dae:  jae,pn 0xc0100dcf
   0xc0100db1:  mov    %fs:0x210,%eax
   0xc0100db7:  test   %eax,%fs:0x244(,%ecx,4)
   0xc0100dbf:  mov    %eax,%ebx
   0xc0100dc1:  jne,pn 0xc0100dd1
   0xc0100dc4:  cmpxchg8b %fs:0x210
   0xc0100dcc:  jne,pn 0xc0100db1
   0xc0100dcf:  pop    %ebx
   0xc0100dd0:  ret
   0xc0100dd1:  pop    %ebx
   0xc0100dd2:  jmp    0xc0100df0
   0xc0100dd7:  mov    %esi,%esi
   0xc0100dd9:  lea    0x0(%edi,%eiz,1),%edi
   0xc0100de0:  nop
   0xc0100de1:  jmp    0xc0100df0

The code seems to be using a helper function cx8_spllower starting at address 0xc0100da0.

In late January, I was at DevOpsDays NYC in midtown Manhattan to present Run Your @wn Email Server!

My abstract:

When we’re responsible for production, it can be hard to find room to learn. That’s why I run my own email server. It’s still “production” — if it stays down, that’s pretty bad — but I own all the decisions, take more risks, and have learned lots. And so can you! Come see why and how to get started.

With one command, install famously secure email software. A couple more and it’s running. A few more and it’s encrypted. Twiddle your DNS, watch the mail start coming in, and start feeling responsible for a production service in a way that web hosting can’t match.

• Slides

Happy 2019! Another three months, another stable branch for pkgsrc, the practical cross-platform Unix package manager. I’ve shipped quite a few improvements for qmail users in our 2018Q4 release. In three sentences:

1. qmail-run gains TLS, SPF, IPv6, SMTP recipient checks, and many other sensible defaults.
2. Most qmail-related packages — including the new ones used by qmail-run — are available on most pkgsrc platforms.
3. rc.d-boot starts rc.conf-enabled pkgsrc services at boot time on many platforms.

In one:

It’s probably easy for you to run qmail now.

On this basis, at my DevOpsDays NYC talk in a few weeks, I’ll be recommending that everyone try it.

Try it

Here’s a demo on CentOS 7, using binary packages:

The main command I ran:

$ sudo env PKG_RCD_SCRIPTS=yes pkgin -y install qmail-run rc.d-boot

Here’s another demo on Debian 9, building from source packages:

The commands I ran:

$ cd ...pkgsrc/mail/qmail-run && make PKG_RCD_SCRIPTS=yes install
$ cd ../../pkgtools/rc.d-boot && make PKG_RCD_SCRIPTS=yes install

These improvements were made possible by acceptutils, my redesigned TLS and SMTP AUTH implementation that obviates the need for several large and conflicting patches. Further improvements are expected.

Here’s the full changelog for qmail as packaged in pkgsrc-2018Q4.

Removed

• From mail/mess822:
  • The SMTP AUTH patch.
• From mail/qmail:
  • The SMTP AUTH patch.
  • The qmail-smtpd half of the TLS patch.
  • The RCPTCHECK patch.
• From mail/qmail-run:
  • Dependency on spamdyke.
  • Dependency on stunnel.
• From sysutils/checkpassword and sysutils/checkpassword-pam:
  • The setuid bit.

Updated

• mail/qmail:
  • To the latest qmail-remote half of the TLS patch.
  • To warn if the queue filesystem is case-insensitive.
• mail/qmail-rejectutils:
  • To let qmail-rcptcheck run under qmail-spp, so that other RCPTCHECK programs can continue to run unmodified.
  • To deprecate qmail-qfilter-ofmipd-queue and qmail-qfilter-smtpd-queue in favor of qmail-qfilter-queue.
• mail/qmail-run’s defaults:
  • To sslserver (from tcpserver).
  • To listen on IPv6 when available.
  • To auto-enable TLS for message submission, incoming SMTP, and POP3 (as well as remote delivery) when certs are in place.
  • To tag log entries with nbqmail/send, nbqmail/smtpd, etc. (inspired by Postfix).
  • To find tcprules in control/tcprules/* (and auto-migrate from /etc/qmail/tcp.*).
  • To rebuild outdated tcprules CDBs on startup.
  • To delay the SMTP greeting by 2 seconds (a simple anti-spam measure).
  • To check the zen.spamhaus.org RBL.
  • To check recipients using qmail’s delivery logic before accepting mail.
  • To record a Received-SPF: header.
  • To skip greylisting (if any) when SPF returns “pass”.
  • To record a Received: header with TLS protocol and ciphers.
  • To let users configure their own ofmipd address-rewriting rules.

Added

• mail/greylisting-spp:
  • For greylisting.
• mail/qmail-spp-spf:
  • For SPF checks.
• pkgtools/rc.d-boot:
  • For starting pkgsrc-provided services at boot on a variety of systems.
• To devel/syncdir:
  • A dlsym()-based implementation, for systems without syscall().
• To mail/qmail:
  • The qmail-spp patch, for flexibly modifying SMTP behavior at runtime.
• To mail/qmail-rejectutils:
  • Manual pages.
• To mail/qmail-run:
  • greylisting-spp-wrapper, for whitelisting recipient addresses or whole domains, and optionally omitting IP address from greylisting’s tuples.
• To mail/qmail and mail/qmail-run:
  • Cleaner uninstall, so people can feel comfortable trying qmail.

After my fourth and final tour stop, we decamped to Mallorca for a week. With no upcoming workshops to polish and no upcoming plans to finalize, the laptop stayed home. Just each other, a variety of beaches, and the annual Les Festes del Rei En Jaume that Bekki and I last saw two years ago on our honeymoon. The parade was perhaps a bit much for Taavi.

The just-released episode 99 of Agile for Humans includes some reflections (starting around 50 minutes in) from partway through my coding tour. As our summer in Germany draws to a close, I’d like to reflect on the tour as a whole.

Annual training

I’ve made a habit of setting aside time, attention, and money each year for focused learning. My most recent trainings, all formative and memorable:

• 2014: Cockburn, Coaching in an Agile Context
• 2015: Weinberg and Derby, Problem Solving Leadership
• 2016: Grenning, Test-Driven Development for Embedded C
• 2017: Larsen, Leading Agile Adoptions with the Agile Fluency Game

I hoped Schleier, Coding Tour would fit the bill for 2018. It has.

Geek joy

At the outset, I was asked how I’d know whether the tour had gone well. My response: “It’s a success if I get to meet a bunch of people in a bunch of places and we have fun programming together.”

I got to program with a bunch of people in a bunch of places. We had fun doing it. Success!

New technologies

My first tour stop offered such an ecumenical mix of languages, tools, and techniques that I began writing down each new technology I encountered. I’m glad I started at the beginning. Even so, this list of things that were new or mostly new to me is probably incomplete:

• Pact, ClickShare, Entity Framework, JMeter, MSTest, Visual Studio Unit Testing Framework, Windows on Raspberry Pi, Eclipse Photon, protothreads, ASCOM, Hibernate, Jubula, JavaFX, REST-assured, mobodoro, mob rotation every 2 minutes 30 seconds, Boost Test, Artifactory, Mercurial, _ for ignoring a parameter, Scala, ScalaTest, WordSpec, property-based testing, ScalaCheck, Google Cloud SQL, Akka

In the moment, learning new technologies was a source of geek joy. In the aggregate, it’s professionally useful. I think the weight clients tend to place on consultants needing to be expert in their tech stack is dangerously misplaced, but it doesn’t matter what I think if they won’t bring me in. Any chance for me to broaden my tech background is a chance for a future client to take advantage of all the other reasons I can be valuable to them.

Insights

As Schmonz’s Theorem predicts, code-touring is both similar to and different from consulting.

When consulting, I expect most of my learning to be meta: the second loop (at least) of double-loop learning. When touring, I became reacquainted with the simple joys of the first loop, spending all day learning new things to be able to do. It often felt like play.

When consulting, I initially find myself being listened to in a peculiar way, my words being heard and measured carefully for evidence of my real intentions. My first tasks are to demonstrate that I can be trusted and that I can be useful, not necessarily in that (or any) order. Accomplishing this as a programmer on tour felt easier than usual.

When I’m consulting, not everyone I encounter wants me there. Some offer time and attention because they feel obligated. On this tour, even though some folks were surprised to find out their employer wasn’t paying me anything, I sensed people were sharing their time and attention with me out of curiosity and generosity. I believe I succeeded in making myself trusted and useful to each of them, and the conversation videos and written testimonials help me hold the belief.

Professional development

With so much practice designing and facilitating group activities, so much information-rich feedback from participants, and so many chances to try again soon, I’ve leveled up as a facilitator. I was comfortable with my skills, abilities, and material before; I’m even more comfortable now. In my tour’s final public meetup, I facilitated one of my legacy code exercises for three simultaneous mobs. It went pretty well — in large part because of the participants, but also because of my continually developing skill at designing and facilitating learning experiences.

As a consultant, it’s a basic survival skill to quickly orient myself in new problem spaces. As a coach, my superpower might be that I help others quickly orient themselves in their problem spaces. Visiting many teams at many companies, I got lots of practice at both. These areas of strength for me are now stronger, the better with which to serve my next clients.

On several occasions I asked mobs not to bother explaining the current context to me before starting the timer. My hypothesis was, all the context I’d need would reveal itself through doing the work and asking a question or two along the way. (One basis among many for this hypothesis: what happened when I showed up late to one of Lennart Fridén’s sessions at this spring’s Mob Programming Conference and everyone else had already read the manual for our CPU.) I think there was one scenario where this didn’t work extremely well, but my memory’s fuzzy — have I mentioned meeting a whole bunch of people at a whole bunch of workplaces, meetups, and conferences? — so I’ll have to report the details when I rediscover it.

You can do this too, and I can help

When designing my tour, I sought advice from several people who’d gone on one. (Along the way I met several more, including Ivan Sanchez at SPA in London and Daniel Temme at SoCraTes in Soltau.)

If you’re wondering whether a coding tour is something you want to do, or how to make it happen, get in touch. I’m happy to listen and offer my suggestions.

What’s next for me, and you can help

Like what I’m doing? Want more of it in your workplace?

I offer short, targeted engagements in the New York metro area — coaching, consulting, and training — co-designed with you to meet your organization’s needs.

More at latentagility.com.

Gratitude

Yes, lots.

It’s been a splendid set of privileges to have the free time to go on tour, to have organizations in several countries interested to have me code with them, and to meet so many people who care about what I care about when humans develop software together.

Five years ago I was discovering the existence of a set of communities of shared values in software development and my need to feel connected to them. Today I’m surer than ever that I’ve needed this connection and that I’ve found it.

Thanks to the people who hosted me for a week at their employer: Patrick Drechsler at MATHEMA/Redheads in Erlangen, Alex Schladebeck at BREDEX in Braunschweig, Barney Dellar at Canon Medical Research in Edinburgh, and Thorsten Brunzendorf at codecentric in Nürnberg and München. And thanks to these companies for being willing to take a chance on bringing in an itinerant programmer for a visit.

Thanks and apologies in equal measure to Richard Groß, who did all the legwork to have me visit MaibornWolff in Frankfurt, only to have me cancel at just about the last minute. At least we got to enjoy each other’s company at Agile Coach Camp Germany and SoCraTes (the only two people to attend both!).

Thanks to David Heath at the UK’s Government Digital Service for inviting me to join them on extremely short notice when I had a free day in London, and to Olaf Lewitz for making the connection.

Thanks to the meetups and conferences where I was invited to present: Mallorca Software Craft, SPA Software in Practice, pkgsrcCon, Hackerkegeln, JUG Ostfalen, Lean Agile Edinburgh, NEBytes, and Munich Software Craft. And thanks to Agile Coach Camp Germany and SoCraTes for the open spaces I did my part to fill.

Thanks to Marc Burgauer, Jens Schauder, and Jutta Eckstein for making time to join me for a meal. Thanks to Zeb Ford-Reitz, Barney Dellar, and their respective spice for inviting me into their respective homes for dinner.

Thanks to J.B. Rainsberger for simple, actionable advice on making it easy for European companies to reimburse my expenses, and more broadly on the logistics of going on European consulting-and-speaking tours when one is from elsewhere. (BTW, his next tour begins soon.)

Thanks all over again to everyone who helped me design and plan the tour, most notably Dr. Sal Freudenberg, Llewellyn Falco, and Nicole Rauch.

Thanks to Woody Zuill, Bryan Beecham, and Tim Bourguignon for that serendipitous conversation in the park in London. Thanks to Tim for having been there in the park with me. (No thanks to Woody for waiting till we’d left London before arriving. At least David Heath and GDS got to see him. Hmph.)

Thanks to Lisi Hocke for making my wish a reality: that her testing tour and my coding tour would intersect. As a developer, I have so much to learn about testing and so few chances to learn from the best. She made it happen. A perfect ending for my tour.

Thanks to Ryan Ripley for having me on Agile for Humans a couple more times as the tour progressed. I can’t say enough about what Ryan and his show have done for me, so this’ll have to be enough.

Thanks to everyone else who helped draw special attention to my tour when I was seeking companies to visit, most notably Kent Beck. It really did help.

Another reason companies cited for inviting me: my micropodcast, Agile in 3 Minutes. Thanks to Johanna Rothman, Andrea Goulet, Lanette Creamer, Alex Harms, and Jessica Kerr for your wonderful guest episodes. You’ve done me and our listeners a kindness. I trust it will come back to you.

Thank you to my family for supporting my attempts at growth, especially when I so clearly need it.

Finally, thanks to all of you for following along and for helping me find the kind of consulting work I’m best at, close to home in New York. You can count on me continuing to learn things and continuing to share them with you.

FIN

• All coding tour posts
• All coding tour tweets (twitter.com)
• All end-of-week conversations (youtube.com)
• My consultancy (latentagility.com)

Usually I cross-compile g4u from Mac OS X, but for the fun of it I did it on NetBSD (7.0-stable branch, amd64 architecture in VMware Fusion) this time. After waiting forever on the CVS checkout, I found that empty directories were not removed - that's what you get if you have -P in your ~/.cvsrc file.

I already had the hint that the "g4u-build" script needed a change to have "G4U_BUILD_KERNEL=true".

From there, things went almost smooth: building indicated a few files that popped up "variable may be used uninitialized" errors, and which -- thanks to -Werror -- bombed out the build. Fixing was easy, and I have no idea why that built for me on the release. I have sent a patch with the required changes to the g4u-help mailing list. (After fixing that I apparently got unsubscribed from my own support mailing list - thank you very much, Sourceforge ;)).

After those little hassles, the build worked fine, and gave me the floppy disk and ISO images that I expected:

>       ls -l `pwd`/g4u*fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u1.fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u2.fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u3.fs
>       -rw-r--r--  2 feyrer  staff  1474560 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u4.fs
>       ls -l `pwd`/g4u.iso
>       -rw-r--r--  2 feyrer  staff  6567936 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u.iso
>       ls -l `pwd`/g4u-kernel.gz
>       -rw-r?r--  1 feyrer  staff  6035680 Mar 17 19:27 /home/feyrer/work/NetBSD/cvs/src-g4u.v3-deOliviera/src/distrib/i386/g4u/g4u-kernel.gz 

I spent some time looking through the documentation, but honestly, I have not found any good answer.

I understand NetBSD supports many FS types as USER SPACE, but I would like to know what is the default FS created by the installer, and the one which I could boot from.

NetBSD 7.1.1 is the first update with security and critical fixes for the NetBSD 7.1 branch. Those include a number of fixes for security advisories, kernel and userland.

• NetBSD Security Advisory 2018-001: Several vulnerabilities in context handling - this affects amd64, i386 and sparc64
• NetBSD Security Advisory 2018-002: Local DoS in virecover

• Video: https://www.youtube.com/watch?v=rRg2vuwF1hY
• Talk: https://events.ccc.de/congress/2017/Fahrplan/events/8968.html

As a hint, NetBSD still has a number of Security Advisories to publish, it seems. Anyone wants to help out the security team? :-)

I have a very old AirPort Extreme, the A1408. Is it possible to install Linux on it, using the AirPort functionally as a hard disk, and then boot from that? I have also heard that AirPorts run NetBSD. Can you boot into that and run commands?

I would like to know how i could log SSH command lines a user is using on a server. For exemple, if the user Alex on my server is doing the following set of commands :

$ cd /tmp
$ touch myfile
$ ssh [email protected]
$ ssh [email protected]
$ vim anotherfile
$ ssh [email protected]

I would like to log the ssh commands used on the server in a file which looks like :

[2014-07-25 10:10:10] Alex : ssh [email protected]
[2014-07-25 10:18:20] Alex : ssh [email protected]
[2014-07-25 11:15:10] Alex : ssh [email protected]

I don't care what he did during his ssh session, i just want to know WHEN and TO WHERE he made a connection to another server.

The user is not using bash and i would like to avoid manipulating .bash_history anyway as the user can modify it.

Any clue on this ?

Thank you :)

edit : to be more specific :

a user connects to a server A and then connects from the server A to server B. I want to track down to which server he connects through ssh from server A.

About g4u: g4u ("ghosting for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

The past: When I started g4u, I had the task to install a number of lab machines with a dual-boot of Windows NT and NetBSD. The hype was about Microsoft's "Zero Administration Kit" (ZAK) then, but that did barely work for the Windows part - file transfers were slow, depended on the clients' hardware a lot (requiring fiddling with MS DOS network driver disks), and on the ZAK server the files for installing happened do disappear for no good reason every now and then. Not working well, and leaving out NetBSD (and everything elase), I created g4u. This gave me the (relative) pain of getting things working once, but with the option to easily add network drivers as they appeared in NetBSD (and oh they did!), plus allowed me to install any operating system.

The present: We've used g4u successfully in our labs then, booting from CDROM. I also got many donations from public and private instituations plus comanies from many sectors, indicating that g4u does make a difference.

In the mean time, the world has changed, and CDROMs aren't used that much any more. Network boot and USB sticks are today's devices of choice, cloning of a full disk without knowing its structure has both advantages but also disadvantages, and g4u's user interface is still command-line based with not much space for automation. For storage, FTP servers are nice and fast, but alternatives like SSH/SFTP, NFS, iSCSI and SMB for remote storage plus local storage (back to fun with filesystems, anyone? avoiding this was why g4u was created in the first place!) should be considered these days. Further aspects include integrity (checksums), confidentiality (encryption). This leaves a number of open points to address either by future releases, or by other products.

The future: At this point, my time budget for g4u is very limited. I welcome people to contribute to g4u - g4u is Open Source for a reason. Feel free to get back to me for any changes that you want to contribute!

The changes: Major changes in g4u 2.6 include:

• Make this build with NetBSD-current sources as of 2017-04-17 (shortly before netbsd-8 release branch), binaries were cross-compiled from Mac OS X 10.10
• Many new drivers, bugfixes and improvements from NetBSD-current (see beta1 and beta2 announcements)
• Go back to keeping the disk image inside the kernel as ramdisk, do not load it as separate module. Less error prone, and allows to boot the g4u (NetBSD) kernel from a single file e.g. via PXE (Testing and documentation updates welcome!)
• Actually DO provide the g4u (NetBSD) kernel with the embedded g4u disk image from now on, as separate file, g4u-kernel.gz
• In addition to MD5, add SHA512 checksums

Enjoy!

This is a tutorial to guide you through the shiny new pkg_comp 2.0 on macOS using the macOS-specific self-installer.

Goals: to use pkg_comp 2.0 to build a binary repository of all the packages you are interested in; to keep the repository fresh on a daily basis; and to use that repository with pkgin to maintain your macOS system up-to-date and secure.

This tutorial is specifically targeted at macOS and relies on the macOS-specific self-installer package. For a more generic tutorial that uses the pkg_comp-cron package in pkgsrc, see Keeping NetBSD up-to-date with pkg_comp 2.0.

Getting started

Start by downloading and installing OSXFUSE 3 and then download the standalone macOS installer package for pkg_comp. To find the right file, navigate to the releases page on GitHub, pick the most recent release, and download the file with a name of the form pkg_comp-<version>-macos.pkg.

Then double-click on the file you downloaded and follow the installation instructions. You will be asked for your administrator password because the installer has to place files under /usr/local/; note that pkg_comp requires root privileges anyway to run (because it uses chroot(8) internally), so you will have to grant permission at some point or another.

The installer modifies the default PATH (by creating /etc/paths.d/pkg_comp) to include pkg_comp’s own installation directory and pkgsrc’s installation prefix. Restart your shell sessions to make this change effective, or update your own shell startup scripts accordingly if you don’t use the standard ones.

Lastly, make sure to have Xcode installed in the standard /Applications/Xcode.app location and that all components required to build command-line apps are available. Tip: try running cc from the command line and seeing if it prints its usage message.

Adjusting the configuration

The macOS flavor of pkg_comp is configured with an installation prefix of /usr/local/, which means that the executable is located in /usr/local/sbin/pkg_comp and the configuration files are in /usr/local/etc/pkg_comp/. This is intentional to keep the pkg_comp installation separate from your pkgsrc installation so that it can run no matter what state your pkgsrc installation is in.

The configuration files are as follows:

• /usr/local/etc/pkg_comp/default.conf: This is pkg_comp’s own configuration file and the defaults configured by the installer should be good to go for macOS.

  In particular, packages are configured to go into /opt/pkg/ instead of the traditional /usr/pkg/. This is a necessity because the latter is not writable starting with OS X El Capitan thanks to System Integrity Protection (SIP).

• /usr/local/etc/pkg_comp/sandbox.conf: This is the configuration file for sandboxctl, which is the support tool that pkg_comp uses to manage the compilation sandbox. The default settings configured by the installer should be good.

• /usr/local/etc/pkg_comp/extra.mk.conf: This is pkgsrc’s own configuration file. In here, you should configure things like the licenses that are acceptable to you and the package-specific options you’d like to set. You should not configure the layout of the installed files (e.g. LOCALBASE) because that’s handled internally by pkg_comp as specified in default.conf.

• /usr/local/etc/pkg_comp/list.txt: This determines the set of packages you want to build automatically (either via the auto command or your periodic cron job). The automated builds will fail unless you list at least one package.

  Make sure to list pkgin here to install a better binary package management tool. You’ll find this very handy to keep your installation up-to-date.

Note that these configuration files use the /var/pkg_comp/ directory as the dumping ground for: the pkgsrc tree, the downloaded distribution files, and the built binary packages. We will see references to this location later on.

The cron job

The installer configures a cron job that runs as root to invoke pkg_comp daily. The goal of this cron job is to keep your local packages repository up-to-date so that you can do binary upgrades at any time. You can edit the cron job configuration interactively by running sudo crontab -e.

This cron job won’t have an effect until you have populated the list.txt file as described above, so it’s safe to let it enabled until you have configured pkg_comp.

If you want to disable the periodic builds, just remove the pkg_comp entry from the crontab.

On slow machines, or if you are building a lot of packages, you may want to consider decreasing the build frequency from @daily to @weekly.

Sample configuration

Here is what the configuration looks like on my Mac Mini as dumped by the config subcommand. Use this output to get an idea of what to expect. I’ll be using the values shown here in the rest of the tutorial:

$ pkg_comp config
AUTO_PACKAGES = autoconf automake bash colordiff dash emacs24-nox11 emacs25-nox11 fuse-bindfs fuse-sshfs fuse-unionfs gdb git-base git-docs glib2 gmake gnuls libtool-base lua52 mercurial mozilla-rootcerts mysql56-server pdksh pkg_developer pkgconf pkgin ruby-jekyll ruby-jekyll-archives ruby-jekyll-paginate scmcvs smartmontools sqlite3 tmux vim
CVS_ROOT = :ext:[email protected]:/cvsroot
CVS_TAG is undefined
DISTDIR = /var/pkg_comp/distfiles
EXTRA_MKCONF = /usr/local/etc/pkg_comp/extra.mk.conf
FETCH_VCS = git
GIT_BRANCH = trunk
GIT_URL = https://github.com/jsonn/pkgsrc.git
LOCALBASE = /opt/pkg
NJOBS = 4
PACKAGES = /var/pkg_comp/packages
PBULK_PACKAGES = /var/pkg_comp/pbulk-packages
PKG_DBDIR = /opt/pkg/libdata/pkgdb
PKGSRCDIR = /var/pkg_comp/pkgsrc
SANDBOX_CONFFILE = /usr/local/etc/pkg_comp/sandbox.conf
SYSCONFDIR = /opt/pkg/etc
UPDATE_SOURCES = true
VARBASE = /opt/pkg/var

DARWIN_NATIVE_WITH_XCODE = true
SANDBOX_ROOT = /var/pkg_comp/sandbox
SANDBOX_TYPE = darwin-native

Building your own packages by hand

Now that you are fully installed and configured, you’ll build some stuff by hand to ensure the setup works before the cron job comes in.

The simplest usage form, which involves full automation and assumes you have listed at least one package in list.txt, is something like this:

$ sudo pkg_comp auto

This trivially-looking command will:

1. clone or update your copy of pkgsrc;
2. create the sandbox;
3. bootstrap pkgsrc and pbulk;
4. use pbulk to build the given packages; and
5. destroy the sandbox.

After a successful invocation, you’ll be left with a collection of packages in the /var/pkg_comp/packages/ directory.

If you’d like to restrict the set of packages to build during a manually-triggered build, provide those as arguments to auto. This will override the contents of AUTO_PACKAGES (which was derived from your list.txt file).

But what if you wanted to invoke all stages separately, bypassing auto? The command above would be equivalent to:

$ sudo pkg_comp fetch
$ sudo pkg_comp sandbox-create
$ sudo pkg_comp bootstrap
$ sudo pkg_comp build <package names here>
$ sudo pkg_comp sandbox-destroy

Go ahead and play with these. You can also use the sandbox-shell command to interactively enter the sandbox. See pkg_comp(8) for more details.

Lastly note that the root user will receive email messages if the periodic pkg_comp cron job fails, but only if it fails. That said, you can find the full logs for all builds, successful or not, under /var/pkg_comp/log/.

Installing the resulting packages

Now that you have built your first set of packages, you will want to install them. This is easy on macOS because you did not use pkgsrc itself to install pkg_comp.

First, unpack the pkgsrc installation. You only have to do this once:

$ cd /
$ sudo tar xzvpf /var/pkg_comp/packages/bootstrap.tgz

That’s it. You can now install any packages you like:

$ PKG_PATH=file:///var/pkg_comp/packages/All sudo pkg_add pkgin <other package names>

The command above assume you have restarted your shell to pick up the correct path to the pkgsrc installation. If the call to pkg_add fails because of a missing binary, try restarting your shell or explicitly running the binary as /opt/pkg/sbin/pkg_add.

Keeping your system up-to-date

Thanks to the cron job that builds your packages, your local repository under /var/pkg_comp/packages/ will always be up-to-date; you can use that to quickly upgrade your system with minimal downtime.

Assuming you are going to use pkgtools/pkgin as recommended above (and why not?), configure your local repository:

$ sudo /bin/sh -c "echo file:///var/pkg_comp/packages/All >>/opt/pkg/etc/pkgin/repositories.conf"

And, from now on, all it takes to upgrade your system is:

$ sudo pkgin update
$ sudo pkgin upgrade

Enjoy!

This is a tutorial to guide you through the shiny new pkg_comp 2.0 on NetBSD.

Goals: to use pkg_comp 2.0 to build a binary repository of all the packages you are interested in; to keep the repository fresh on a daily basis; and to use that repository with pkgin to maintain your NetBSD system up-to-date and secure.

This tutorial is specifically targeted at NetBSD but should work on other platforms with some small changes. Expect, at the very least, a macOS-specific tutorial as soon as I create a pkg_comp standalone installer for that platform.

Getting started

First install the sysutils/sysbuild-user package and trigger a full build of NetBSD so that you get usable release sets for pkg_comp. See sysbuild(1) and pkg_info sysbuild-user for details on how to do so. Alternatively, download release sets from the FTP site and later tell pkg_comp where they are.

Then install the pkgtools/pkg_comp-cron package. The rest of this tutorial assumes you have done so.

Adjusting the configuration

To use pkg_comp for periodic builds, you’ll need to do some minimal edits to the default configuration files. The files can be found directly under /var/pkg_comp/, which is pkg_comp-cron’s “home”:

• /var/pkg_comp/pkg_comp.conf: This is pkg_comp’s own configuration file and the defaults installed by pkg_comp-cron should be good to go.

  The contents here are divided in three major sections: declaration on how to download pkgsrc, definition of the file system layout on the host machine, and definition of the file system layout for the built packages.

  You may want to customize the target system paths, such as LOCALBASE or SYSCONFDIR, but you should not have to customize the host system paths.

• /var/pkg_comp/sandbox.conf: This is the configuration file for sandboxctl. The default settings installed by pkg_comp-cron should suffice if you used the sysutils/sysbuild-user package as recommended; otherwise tweak the NETBSD_NATIVE_RELEASEDIR and NETBSD_SETS_RELEASEDIR variables to point to where the downloaded release sets are.

• /var/pkg_comp/extra.mk.conf: This is pkgsrc’s own configuration file. In here, you should configure things like the licenses that are acceptable to you and the package-specific options you’d like to set. You should not configure the layout of the installed files (e.g. LOCALBASE) because that’s handled internally by pkg_comp as specified in pkg_comp.conf.

• /var/pkg_comp/list.txt: This determines the set of packages you want to build in your periodic cron job. The builds will fail unless you list at least one package.

  WARNING: Make sure to include pkg_comp-cron and pkgin in this list so that your binary kit includes these essential package management tools. Otherwise you’ll have to deal with some minor annoyances after rebootstrapping your system.

Lastly, review root’s crontab to ensure the job specification for pkg_comp is sane. On slow machines, or if you are building many packages, you will probably want to decrease the build frequency from @daily to @weekly.

Sample configuration

Here is what the configuration looks like on my NetBSD development machine as dumped by the config subcommand. Use this output to get an idea of what to expect. I’ll be using the values shown here in the rest of the tutorial:

# pkg_comp -c /var/pkg_comp/pkg_comp.conf config
AUTO_PACKAGES = autoconf automake bash colordiff dash emacs-nox11 git-base git-docs gmake gnuls lua52 mozilla-rootcerts pdksh pkg_comp-cron pkg_developer pkgin sqlite3 sudo sysbuild sysbuild-user sysupgrade tmux vim zsh
CVS_ROOT = :ext:[email protected]:/cvsroot
CVS_TAG is undefined
DISTDIR = /var/pkg_comp/distfiles
EXTRA_MKCONF = /var/pkg_comp/extra.mk.conf
FETCH_VCS = cvs
GIT_BRANCH = trunk
GIT_URL = https://github.com/jsonn/pkgsrc.git
LOCALBASE = /usr/pkg
NJOBS = 2
PACKAGES = /var/pkg_comp/packages
PBULK_PACKAGES = /var/pkg_comp/pbulk-packages
PKG_DBDIR = /usr/pkg/libdata/pkgdb
PKGSRCDIR = /var/pkg_comp/pkgsrc
SANDBOX_CONFFILE = /var/pkg_comp/sandbox.conf
SYSCONFDIR = /etc
UPDATE_SOURCES = true
VARBASE = /var

NETBSD_NATIVE_RELEASEDIR = /home/sysbuild/release/amd64
NETBSD_RELEASE_RELEASEDIR = /home/sysbuild/release/amd64
NETBSD_RELEASE_SETS is undefined
SANDBOX_ROOT = /var/pkg_comp/sandbox
SANDBOX_TYPE = netbsd-release

Building your own packages by hand

Now that you are fully installed and configured, you’ll build some stuff by hand to ensure the setup works before the cron job comes in.

The simplest usage form, which involves full automation, is something like this:

# pkg_comp -c /var/pkg_comp/pkg_comp.conf auto

This trivially-looking command will:

1. checkout or update your copy of pkgsrc;
2. create the sandbox;
3. bootstrap pkgsrc and pbulk;
4. use pbulk to build the given packages; and
5. destroy the sandbox.

After a successful invocation, you’ll be left with a collection of packages in the directory you set in PACKAGES, which in the default pkg_comp-cron installation is /var/pkg_comp/packages/.

If you’d like to restrict the set of packages to build during a manually-triggered build, provide those as arguments to auto. This will override the contents of AUTO_PACKAGES (which was derived from your list.txt file).

But what if you wanted to invoke all stages separately, bypassing auto? The command above would be equivalent to:

# pkg_comp -c /var/pkg_comp/pkg_comp.conf fetch
# pkg_comp -c /var/pkg_comp/pkg_comp.conf sandbox-create
# pkg_comp -c /var/pkg_comp/pkg_comp.conf bootstrap
# pkg_comp -c /var/pkg_comp/pkg_comp.conf build <package names here>
# pkg_comp -c /var/pkg_comp/pkg_comp.conf sandbox-destroy

Go ahead and play with these. You can also use the sandbox-shell command to interactively enter the sandbox. See pkg_comp(8) for more details.

Lastly note that the root user will receive email messages if the periodic pkg_comp cron job fails, but only if it fails. That said, you can find the full logs for all builds, successful or not, under /var/pkg_comp/log/.

Installing the resulting packages

Now that you have built your first set of packages, you will want to install them. On NetBSD, the default pkg_comp-cron configuration produces a set of packages for /usr/pkg so you have to wipe your existing packages first to avoid build mismatches.

WARNING: Yes, you really have to wipe your packages. pkg_comp currently does not recognize the package tools that ship with the NetBSD base system (i.e. it bootstraps pkgsrc unconditionally, including bmake), which means that the newly-built packages won’t be compatible with the ones you already have. Avoid any trouble by starting afresh.

To clean your system, do something like this:

# ... ensure your login shell lives in /bin! ...
# pkg_delete -r -R "*"
# mv /usr/pkg/etc /root/etc.old  # Backup any modified files.
# rm -rf /usr/pkg /var/db/pkg*

Now, rebootstrap pkgsrc and reinstall any packages you previously had:

# cd /
# tar xzvpf /var/pkg_comp/packages/bootstrap.tgz
# echo "pkg_admin=/usr/pkg/sbin/pkg_admin" >>/etc/pkgpath.conf
# echo "pkg_info=/usr/pkg/sbin/pkg_info" >>/etc/pkgpath.conf
# export PATH=/usr/pkg/bin:/usr/pkg/sbin:${PATH}
# export PKG_PATH=file:///var/pkg_comp/packages/All
# pkg_add pkgin pkg_comp-cron <other package names>

Finally, reconfigure any packages where you had have previously made custom edits. Use the backup in /root/etc.old to properly update the corresponding files in /etc. I doubt you made a ton of edits so this should be easy.

IMPORTANT: Note that the last command in this example includes pkgin and pkg_comp-cron. You should install these first to ensure you can continue with the next steps in this tutorial.

Keeping your system up-to-date

If you paid attention when you installed the pkg_comp-cron package, you should have noticed that this configured a cron job to run pkg_comp daily. This means that your packages repository under /var/pkg_comp/packages/ will always be up-to-date so you can use that to quickly upgrade your system with minimal downtime.

Assuming you are going to use pkgtools/pkgin (and why not?), configure your local repository:

# echo 'file:///var/pkg_comp/packages/All' >>/etc/pkgin/repositories.conf

And, from now on, all it takes to upgrade your system is:

# pkgin update
# pkgin upgrade

Enjoy!

After many (many) years in the making, pkg_comp 2.0 and its companion sandboxctl 1.0 are finally here!

Read below for more details on this launch. I will publish detailed step-by-step tutorials on setting up periodic package rebuilds in separate posts.

What are these tools?

pkg_comp is an automation tool to build pkgsrc binary packages inside a chroot-based sandbox. The main goal is to fully automate the process and to produce clean and reproducible packages. A secondary goal is to support building binary packages for a different system than the one doing the builds: e.g. building packages for NetBSD/i386 6.0 from a NetBSD/amd64 7.0 host.

The highlights of pkg_comp 2.0, compared to the 1.x series, are: multi-platform support, including NetBSD, FreeBSD, Linux, and macOS; use of pbulk for efficient builds; management of the pkgsrc tree itself via CVS or Git; and a more robust and modern codebase.

sandboxctl is an automation tool to create and manage chroot-based sandboxes on a variety of operating systems. sandboxctl is the backing tool behind pk_comp. sandboxctl hides the details of creating a functional chroot sandbox on all supported operating systems; in some cases, like building a NetBSD sandbox using release sets, things are easy; but in others, like on macOS, they are horrifyingly difficult and brittle.

Storytelling time

pkg_comp’s history is a long one. pkg_comp 1.0 first appeared in pkgsrc on September 6th, 2002 as the pkgtools/pkg_comp package in pkgsrc. As of this writing, the 1.x series are at version 1.38 and have received contributions from a bunch of pkgsrc developers and external users; even more, the tool was featured in the BSD Hacks book back in 2004.

This is a long time for a shell script to survive in its rudimentary original form: pkg_comp 1.x is now a teenager at its 14 years of age and is possibly one of my longest-living pieces of software still in use.

Motivation for the 2.x rewrite

For many of these years, I have been wanting to rewrite pkg_comp to support other operating systems. This all started when I first got a Mac in 2005, at which time pkgsrc already supported Darwin but there was no easy mechanism to manage package updates. What would happen—and still happens to this day!—is that, once in a while, I’d realize that my packages were out of date (read: insecure) so I’d wipe the whole pkgsrc installation and start from scratch. Very inconvenient; I had to automate that properly.

Thus the main motivation behind the rewrite was primarily to support macOS because this was, and still is, my primary development platform. The secondary motivation came after writing sysbuild in 2012, which trivially configured daily builds of the NetBSD base system from cron; I wanted the exact same thing for my packages.

One, two… no, three rewrites

The first rewrite attempt was sometime in 2006, soon after I learned Haskell in school. Why Haskell? Just because that was the new hotness in my mind and it seemed like a robust language to drive a pretty tricky automation process. That rewrite did not go very far, and that’s possibly for the better: relying on Haskell would have decreased the portability of the tool, made it hard to install it, and guaranteed to alienate contributors.

The second rewrite attempt started sometime in 2010, about a year after I joined Google as an SRE. This was after I became quite familiar with Python at work, wanting to use the language to rewrite this tool. That experiment didn’t go very far though, but I can’t remember why… probably because I was busy enough at work and creating Kyua.

The third and final rewrite attempt started in 2013 while I had a summer intern and I had a little existential crisis. The year before I had written sysbuild and shtk, so I figured recreating pkg_comp using the foundations laid out by these tools would be easy. And it was… to some extent.

Getting the barebones of a functional tool took only a few weeks, but that code was far from being stable, portable, and publishable. Life and work happened, so this fell through the cracks… until late last year, when I decided it was time to close this chapter so I could move on to some other project ideas. To create the focus and free time required to complete this project, I had to shift my schedule to start the day at 5am instead of 7am—and, many weeks later, the code is finally here and I’m still keeping up with this schedule.

Granted: this third rewrite is not a fancy one, but it wasn’t meant to be. pkg_comp 2.0 is still written in shell, just as 1.x was, but this is a good thing because bootstrapping on all supported platforms is easy. I have to confess that I also considered Go recently after playing with it last year but I quickly let go of that thought: at some point I had to ship the 2.0 release, and 10 years since the inception of this rewrite was about time.

The launch of 2.0

On February 12th, 2017, the authoritative sources of pkg_comp 1.x were moved from pkgtools/pkg_comp to pkgtools/pkg_comp1 to make room for the import of 2.0. Yes, the 1.x series only existed in pkgsrc and the 2.x series exist as a standalone project on GitHub.

And here we are. Today, February 17th, 2017, pkg_comp 2.0 saw the light!

Why sandboxctl as a separate tool?

sandboxctl is the supporting tool behind pkg_comp, taking care of all the logic involved in creating chroot-based sandboxes on a variety of operating systems. Some are easy, like building a NetBSD sandbox using release sets, and others are horrifyingly difficult like macOS.

In pkg_comp 1.x, this logic used to be bundled right into the pkg_comp code, which made it pretty much impossible to generalize for portability. With pkg_comp 2.x, I decided to split this out into a separate tool to keep responsibilities isolated. Yes, the integration between the two tools is a bit tricky, but allows for better testability and understandability. Lastly, having sandboxctl as a standalone tool, instead of just a separate code module, gives you the option of using it for your own sandboxing needs.

I know, I know; the world has moved onto containerization and virtual machines, leaving chroot-based sandboxes as a very rudimentary thing… but that’s all we’ve got in NetBSD, and pkg_comp targets primarily NetBSD. Note, though, that because pkg_comp is separate from sandboxctl, there is nothing preventing adding different sandboxing backends to pkg_comp.

Installation

Installation is still a bit convoluted unless you are on one of the tier 1 NetBSD platforms or you already have pkgsrc up and running. For macOS in particular, I plan on creating and shipping a installer image that includes all of pkg_comp dependencies—but I did not want to block the first launch on this.

For now though, you need to download and install the latest source releases of shtk, sandboxctl, and pkg_comp—in this order; pass the --with-atf=no flag to the configure scripts to cut down the required dependencies. On macOS, you will also need OSXFUSE and the bindfs file system.

If you are already using pkgsrc, you can install the pkgtools/pkg_comp package to get the basic tool and its dependencies in place, or you can install the wrapper pkgtools/pkg_comp-cron package to create a pre-configured environment with a daily cron job to run your builds. See the package’s MESSAGE (with pkg_info pkg_comp-cron) for more details.

Documentation

Both pkg_comp and sandboxctl are fully documented in manual pages. See pkg_comp(8), sandboxctl(8), pkg_comp.conf(5) and sandbox.conf(5) for plenty of additional details.

As mentioned at the beginning of the post, I plan on publishing one or more tutorials explaining how to bootstrap your pkgsrc installation using pkg_comp on, at least, NetBSD and macOS. Stay tuned.

And, if you need support or find anything wrong, please let me know by filing bugs in the corresponding GitHub projects: jmmv/pkg_comp and jmmv/sandboxctl.

pkgsrc’s collectd does not support the thermal plugin, so in order to publish thermal information I had to use the exec plugin:

1
2
3
4
5
6

LoadPlugin exec
# more plugins

<Plugin exec>
        Exec "nobody:nogroup" "/home/imil/bin/temp.sh"
</Plugin>

And write this simple script that reads CPUs temperature from NetBSD’s envstat command:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ cat bin/temp.sh 
#!/bin/sh

hostname=$(hostname)
interval=10

while :
do
        envstat|awk '/cpu[0-9]/ {printf "%s %s\n",$1,$3}'|while read c t
        do
                echo "PUTVAL ${hostname}/temperature/temperature-zone${c#cpu} interval=${interval} N:${t%%.*}"
        done
        sleep ${interval}
done

I then send those values to an influxdb server:

1
2
3
4
5
6

LoadPlugin network
# ...

<Plugin network>
    Server "192.168.1.5" "25826"
</Plugin>

And display them using grafana:

HTH!