| Announcing NetBSD 6.1 The NetBSD Project is pleased to announce NetBSD 6.1, the first feature update of the NetBSD 6 release branch. It represents a selected subset of fixes deemed important for security or stability reasons, as well as new features and enhancements. Please note that all fixes in the prior ... |
| BSD magazine :: Issues BSD magazine is a quarterly totally devoted to different distros of BSD. The magazine comes with a DVD with the latest BSD distribution. |
| NetBSD: Recent Changes and News On this page you will find a summary of significant changes to NetBSD. For those interested in every change to the source tree, there is no substitute for subscribing to the source-changes mailing list. Significant news related to the NetBSD Project is also included here. |
The NetBSD Project is pleased to announce NetBSD 6.1, the first feature update of the NetBSD 6 release branch. It represents a selected subset of fixes deemed important for security or stability reasons, as well as new features and enhancements.
Simultaneously, the NetBSD Project is pleased to announce NetBSD 6.0.2, the second security/bugfix update of the NetBSD 6.0 release branch. It represents a selected subset of fixes deemed important for security or stability reasons, without new features.
For more details, please see the 6.1 release notes and the 6.0.2 release notes
Complete source and binaries for NetBSD 6.1 and 6.0.2 are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at http://www.NetBSD.org/mirrors/.The fourth release candidate of NetBSD 6.1 is now available for download at: http://ftp.NetBSD.org/pub/NetBSD/NetBSD-6.1_RC4/. It is expected that this will be the final release candidate, with the official release following very soon.
(Please note that while the third release candidate (RC3) was tagged and built, it was never officially released)
NetBSD 6.1 will be the first feature update for the NetBSD 6 branch. There are many new drivers, some new features, and many bug fixes! Fixes since RC2 include:
A complete list of changes can be found at:
http://ftp.NetBSD.org/pub/NetBSD/NetBSD-6.1_RC4/CHANGES-6.1
Please help us test this and any upcoming release candidates as much as possible. Remember, any feedback is good feedback. We'd love to hear from you, whether you've got a complaint or a compliment.
Release Notes: Integration of the gfio GTK frontend. Better Android integration. The initial file fill honors data compression settings. An option for controlling randomness of I/O (percentage_random) instead of being fully random or fully sequential. Fixes a bug in uneven pattern sizes for verification. Various other important fixes.
Tags: Benchmark, Filesystems
Licenses: GPLv2
Release Notes: cdrtools is now at version 3.01a14, and now supports root-less operation on Linux using fcaps. #include file reordering was undertaken to avoid warnings on older platforms. Job processing code was moved from smake/make.c to smake/job.c. A typo in include/schily/stat.h that caused setting of nanoseonds in timestamps to be impossible for NetBSD and OpenBSD was fixed.
Tags: Software Development, Build Tools, Utilities
Licenses: CDDL
I’ve been obsessed with SaltStack for over a week. This infrastructure management suite is exactly what I needed for both my personal and professional servers: simple but modular, written in python, not depending on a thousand unnecessary complex messaging stacks as it bundles zeromq, capable of both orchestration and configuration management, all this through comprehensive, well documented API and commands.
Only drawback was it had poor NetBSD support. Was :)
It’s been a long time since I’ve dug into python, so it took me a little bit of effort, but Salt now has full support of pkgin in its generic packaging functions, knows how to handle NetBSD services and is capable of dealing with NetBSD‘s sysctl(8) and sysctl.conf.
Those pieces of code have been merged upstream, I hope they’ll be available in version 0.16!
Some examples:
$ cat packages/init.sls
mypkgs:
pkg.installed:
- pkgs:
- vim
- tmux
- bash
- bash-completion
- sudo
$ sudo salt '*' state.sls packages
watto:
----------
State: - pkg
Name: mypkgs
Function: installed
Result: True
Comment: All specified packages are already installed.
Changes:
korriban:
----------
State: - pkg
Name: mypkgs
Function: installed
Result: True
Comment: All specified packages are already installed.
Changes:
tatooine:
----------
State: - pkg
Name: mypkgs
Function: installed
Result: True
Comment: All specified packages are already installed.
Changes:
coruscant:
----------
State: - pkg
Name: mypkgs
Function: installed
Result: True
Comment: All specified packages are already installed.
Changes:
ragnos:
----------
State: - pkg
Name: mypkgs
Function: installed
Result: True
Comment: All specified packages are already installed.
Changes:
exar:
----------
State: - pkg
Name: mypkgs
Function: installed
Result: True
Comment: All specified packages are already installed.
Changes:
$ sudo salt '*' cmd.run 'uname -a'
tatooine:
Linux tatooine 3.2.0-4-686-pae #1 SMP Debian 3.2.41-2 i686 GNU/Linux
watto:
NetBSD watto.home.imil.net 6.1_RC4 NetBSD 6.1_RC4 (GENERIC) i386
exar:
NetBSD exar 6.0_STABLE NetBSD 6.0_STABLE (EXAR) #0: Sun Nov 25 12:39:12 CET 2012 root@exar:/usr/src/sys/arch/i386/compile/EXAR i386
coruscant:
NetBSD coruscant 6.0 NetBSD 6.0 (XEN3_DOM0) amd64
korriban:
NetBSD korriban.imil.net 6.0_STABLE NetBSD 6.0_STABLE (KORRIBAN) #0: Tue Jan 1 23:20:36 CET 2013 [email protected]:/usr/src/sys/arch/amd64/compile/KORRIBAN amd64
ragnos:
NetBSD ragnos 6.0 NetBSD 6.0 (RAGNOS) #2: Wed Oct 17 11:33:31 CEST 2012 root@ragnos:/usr/src/sys/arch/i386/compile/RAGNOS i386
$ sudo salt '*' pkg.version vim
watto:
7.3.762
exar:
7.3.762
korriban:
7.3.712
coruscant:
7.3.762
ragnos:
7.3.762
tatooine:
2:7.3.547-7
$ sudo salt '*' service.status sshd
tatooine:
False
watto:
True
coruscant:
True
exar:
True
korriban:
True
ragnos:
True
If you whish to use these modules without tainting your Salt package installation, simply copy them to a _modules directory within the file_roots.
Happy Salting!
The post NetBSD configuration management appeared first on Emile "iMil" Heitor 's home.
| BSD News 05/06/13 | DiscoverBSD The OpenBSD project today formally released OpenBSD 5.3, the project's 34th release on the steady six monthly release cycle. Notable news in the present release include the a production ready release ... |
I know about estd for a while, that daemon “dynamically sets the CPU-frequency on Enhanced SpeedStep, PowerNow, and APCI P-States-enabled CPUs depending on current cpu-utilization” (manpage excerpt). Thing is, I’ve never seen any CPU changing from its current speed while monitoring the machdep.powernow.frequency.current sysctl.
In order to understand what was happening, I started estd with the -o flag, which outputs the CPU-frequencies as they are set. I then realized that the “ligh watermark percentage” and “low watermark percentage” default values were way too high (respectively 40 and 80) and were never reached, so the CPU speed was never changed.
With lower values, I was able to see the CPU speed increasing and lowering as expected. So I added the following line to the /etc/rc.conf file:
estd_flags="-l 5 -h 15 -a -m 800 -d"
meaning that the low watermark is set at 5 and the high watermark at 15, which were the values I’ve considered being the right ones while watching estd -o -a console output.
Since then, whenever a CPU intensive operation occurs, I can see the CPU speed rising with the following conky parameter:
CPU Frequency: ${alignr}${exec /sbin/sysctl -n machdep.powernow.frequency.current}
The post CPU dynamic scaling on NetBSD appeared first on Emile "iMil" Heitor 's home.
Ever since I realized that the anykernel was the best way to construct a modern general purpose operating system kernel, I have been performing experiments by running unmodified NetBSD kernel drivers in rump kernels in various environments (nb. here driver does not mean a hardware device driver, but any driver like a file system driver or TCP driver). These experiments have included userspaces of various platforms, binary kernel modules on Linux and others, and compiling kernel drivers to javascript and running them natively in a web browser. I have also claimed that the anykernel allows harnessing drivers from a general purpose OS onto more specialized embedded computing devices which are becoming the new norm. This is an attractive possibility because while writing drivers is easy, making them handle all the abnormal conditions of the real world is a time-consuming process. Since the above-mentioned experiments were done on POSIX platforms (yes, even the javascript one), the experiments did not fully support the claim. The most interesting, decidedly non-POSIX platform I could think of for experimentation was the Linux kernel. Even though it had been several years since I last worked in the Linux kernel, my hypothesis was that it would be easy and fast to get unmodified NetBSD kernel drivers running in the Linux kernel as rump kernels.
A rump kernel runs on top of the rump kernel hypervisor. The hypervisor provides high level interfaces to host features, such as memory allocation and thread creation. In this case, the Linux kernel is the host. In principle, there are three steps in getting a rump kernel to run in a given environment. In reality, I prefer a more iterative approach, but the development can be divided into three steps all the same.
Getting basic functionality up and running was a relatively straightforward process. The only issue that required some thinking was an application binary interface (ABI) mismatch. I was testing on x86 where Linux kernel ABI uses -mregparm=3, which means that function arguments are passed in registers where possible. NetBSD always passes arguments on the stack. When two ABIs collide, the code may run, but since function arguments passed between the two ABIs result in garbage, eventually an error will be hit perhaps in the form of accessing invalid memory. The C code was easy enough to "fix" by applying the appropriate compiler flags. In addition to C code, a rump kernel uses a handful of assembly routines from NetBSD, mostly pertaining to optimizations (e.g. ffs()), but also to access the atomic memory operations of the platform. After assembly routines had been handled, it was possible to load a Linux kernel module which bootstraps a rump kernel in the Linux kernel and does some file system operations on the fictional kernfs file system. A screenshot of the resulting dmesg output is shown below.
It is one thing to execute a computation and an entirely different thing to perform I/O. To test I/O capabilities, I ran a rump kernel providing a TCP/IP driver inside the Linux kernel. For a networking stack to be able to do anything sensible, the interface layer needs to be able to shuffle packets. The quickest way to implement the hypercalls for packet shuffling was to use the same method as a userspace virtual TCP/IP stack might use: read/write packets using the tap device. Some might say that doing this from inside the kernel is cheating, but given that the alternative was to copypaste the tuntap driver and edit it slightly, I call my approach constructive laziness.
The demo itself opens a TCP socket to port 80 on vger.kernel.org (IP address 0x43b484d1 if you want to be really precise), does a HTTP get for "/" and displays the last 500 bytes of the result. TCP/IP is handled by the rump kernel, not by the Linux kernel. Think of it as the Linux kernel having two alternative TCP/IP stacks. Again, a screenshot of the resulting dmesg is shown below. Note that unlike in the first screenshot, there is no printout for the root file system because the configuration used here does not include any file system support. Yes, you can ping 10.0.2.17.
As hypothesized, a rump kernel hypervisor for the Linux kernel was easy and straightforward to implement. Furthermore, it could be done without making any changes to the existing hypercall interface thereby reinforcing the belief that unmodified NetBSD kernel drivers can run on top of most any embedded firmwares just by implementing a light hypervisor layer.
There were no challenges in the experiment, only annoyances. As Linux does not support rump kernels, I had to revert back to the archaic full OS approach to kernel development. The drawbacks of the full OS approach include for example suffering multi-second reboot cycles during iterative development. The other tangential issue that I spent a disproportionately large amount of time with was thinking about how releasing this code would affect existing NetBSD code due to GPL involvement. My conclusion was that this does not matter since all code used by the current demo is open source anyway, and if someone wants to use my code in a product, it is their problem, not mine.
For people interested in examining the implementation, I put the source code for the hypervisor along with the test code in a git repo here. The repository also contains the demos linked from this article. The NetBSD kernel drivers I used are available from ftp.netbsd.org or by getting buildrump.sh and running ./buildrump.sh checkout.
Release Notes: A typo in include/schily/stat.h related to nanosecond handling on NetBSD and OpenBSD was fixed. suid-root-less installation is now supported on Linux using fcaps (generally available since Linux 2.6.24). -new-dir-mode in mkisofs now just supersedes the effect of -dir-mode on directories that have been "invented" by mkisofs.
Tags: Software Distribution, Archiving, multimedia, Sound/Audio, CD Audio, CD Writing, CD Ripping, Boot
Licenses: CDDL
I recently switched pkgin‘s repository from SourceForge‘s CVS to GitHub. Long story short, I heard here and there that SF was considering to drop CVS support and I found GitHub service to be more responsive and elegant. Also, I was looking for an excuse to learn git :)
Anyway, GitHub interface may be sexy, they used to have some kind of “upload” section which has been dropped. That may sound like a simple story, but the fact is when it comes to packaging a GitHub-hosted application, things are not that simple when the author has not explicitly tagged a specific release. Another use case, in which I actually am, is when you have an ongoing development, like pkgin in pkgsrc WIP and do not want to tag every test-release.
The way I found to handle that case with pkgsrc is to use GitHub’s commit archives. In short, I will use that kind of URL:
https://github.com/NetBSDfr/pkgin/archive/34b823c158e62e4d347de74499a075a2259382c5.tar.gz
which is redirected like this by GitHub:
HTTP/1.1 302 Found Server: GitHub.com Date: Sun, 21 Apr 2013 21:05:17 GMT Content-Type: text/html; charset=utf-8 Connection: keep-alive Status: 302 Found Cache-Control: max-age=0, private Strict-Transport-Security: max-age=2592000 X-Frame-Options: deny Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Thu, 21-Apr-2033 21:05:17 GMT; HttpOnly Location: https://nodeload.github.com/NetBSDfr/pkgin/tar.gz/34b823c158e62e4d347de74499a075a2259382c5 X-Runtime: 13 Content-Length: 156 Vary: Accept-Encoding HTTP/1.1 200 OK Server: GitHub.com Date: Sun, 21 Apr 2013 21:05:18 GMT Content-Type: application/x-gzip Connection: keep-alive Content-Length: 187510 Content-Disposition: attachment; filename=pkgin-34b823c158e62e4d347de74499a075a2259382c5.tar.gz Vary: Accept-Encoding
and permits to point to a particular commit, no matter if it has been tagged or not.
A typical pkgsrc Makefile will look like this:
VERSION= 34b823c158e62e4d347de74499a075a2259382c5
DISTNAME= ${VERSION}
PKGNAME= pkgin-20130412
CATEGORIES= pkgtools
MASTER_SITES= https://github.com/NetBSDfr/pkgin/archive/
FETCH_USING= curl
# [...]
WRKSRC= ${WRKDIR}/pkgin-${VERSION}
# [...]
Note that FETCH_USING= curl is mandatory here in order to follow redirect codes along with https.
There you go, happy GitHub packaging!
Update
Here’s another approach pointed out by Amitai Schlair (schmonz@):
GIT_COMMIT= dd51ac5
DISTNAME= ${GIT_COMMIT}
PKGNAME= p5-App-Prove-Plugin-ProgressBar-0.01
CATEGORIES= devel perl5
MASTER_SITES= -http://nodeload.github.com/Ovid/App-Prove-Plugin-ProgressBar/tar.gz/${GIT_COMMIT}
# [...]
WRKSRC= ${WRKDIR}/App-Prove-Plugin-ProgressBar-${GIT_COMMIT}
Here, Amitai doesn’t use HTTPS so specifying curl as the fetch method is not mandatory. The dash before the URL in the MASTER_SITES line means that DISTNAME will not be appended when fetching, which is very handy when it comes to GitHub archives.
The post pkgsrc and github archives appeared first on Emile "iMil" Heitor 's home.
Release Notes: This is a stable release, addressing some small issues (no security impact) with 0.99.22. The primary change is a fix to an ospfd issue that has a relatively strong impact, as in 0.99.22 all AS-External routes with a forwarding address are broken when used on point-to-point links. Users of ospfd are therefore strongly urged to update.
Release Tags: stable minor
Tags: Networking, Routing
Licenses: GPLv2
Click on the above link for more information on GSoC in general, there is also a list of proposed projects for this year in NetBSD.
Next steps are:
During the project, students are encouraged to publish news about their process to the world in blogs and other ways found appropriate by their mentoring organizations.
Past NetBSD projects can be found on SourceForce.
Interested? Act now!
Release Notes: This release adds an --eta-newline option to insert ETA new line at fixed intervals. Fixes for the CPU clock, notably PPC. Fixes for workloads with read/write file locking. A fix for skewed latencies with rated IOPS. Fixes a bug with block sizes varying more than 64x. Improves the LFSR implementation. Fixes missing JSON bandwidth output. Various fixes for the Android platform. Uses ginstall on Solaris. Fixes man page installation. Other bugfixes and improvements.
Tags: Benchmark, Filesystems
Licenses: GPLv2
Schmonzcast #14: Last week I returned from Berlin, where I gave a talk at pkgsrcCon 2013. As a pkgsrc developer since 2002, and a member of its steering committee since 2005, I’ve attended every pkgsrcCon but one (Paris 2006, as the spring semester of my freshman year at Columbia was ending). pkgsrcCon is a self-organized, purpose-specific conference for people interested in pkgsrc; it’s short, sweet, and low-key; and thus far it’s always been held in a European city well worth visiting. At this year’s conference I spoke about applying ‘Test-Eventually Development’ to pkglint (a tool for package maintainers), and how all of us volunteer developers ought to team up to slowly but surely apply the same techniques to pkgsrc’s unique and valuable internals. Even if you don’t know much about pkgsrc, a passing familiarity with TDD suffices to follow my line of reasoning. And if you know and love pkgsrc, we’d love to have your help.
Il est là, il est chaud, et il contient deux articles de mon cru:
The post GLMF 159 appeared first on Emile "iMil" Heitor 's home.
These blog articles were very detailed on purpose, to have full logfiles available just in case needed. I have used these logs to prepare my pkgsrcCon 2013 talk about Ansible and Amazon's EC2, so things can be looked at without actually running anything. As it turns out this was good, because the 32bit NetBSD instances that I've used during my pkgsrcCon demonstration actually decided to do a kernel panic, and the presentation was a bit more on the theoretical side than I originally planned.
Now after pkgsrcCon is over, I would like to publish the presentation slides with all the details, and especially the playbooks and all other files to look at - enjoy!
We get a lot of comments asking for tips on using the raspberry pi so I thought I would point out some docs:
evbarm/rpi wiki docs
An example of the rpi.img can be found here:
http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201303221130Z/evbarm/binary/gzimg/ notice the HEAD (NetBSD -current), datestamp, arch path for future reference
There are also some concerns about building a kernel/img on your own.
building NetBSD
build.sh is one of the best features of NetBSD. You can cross compile from almost any other unix-like system with very little difficulty.
The single steps are:
% date Thu Mar 21 00:45:37 CET 2013 % ssh-add -l 2048 d5:25:19:3d:59:40:35:32:03:f7:c5:83:de:19:b6:d0 ../../euca2ools/key-eucaHF.pem (RSA)
% euca-describe-groups ... GROUP sg-ae54b3c5 749335780469 ec2-dbservers Database servers PERMISSION 749335780469 ec2-dbservers ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-dbservers ALLOWS tcp 3306 3306 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-dbservers ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0 GROUP sg-a854b3c3 749335780469 ec2-webservers Web servers PERMISSION 749335780469 ec2-webservers ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS tcp 80 80 FROM CIDR 0.0.0.0/0 PERMISSION 749335780469 ec2-webservers ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0
% ansible-playbook -i hosts-HF config-ec2-prepare-db+web-vm.yml
PLAY [localhost] *********************
TASK: [ec2-webservers | Launch new EC2 instance] *********************
changed: [127.0.0.1]
TASK: [ec2-webservers | Give the system 30 seconds to boot up] *********************
changed: [127.0.0.1]
TASK: [ec2-webservers | Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] *********************
changed: [127.0.0.1]
TASK: [ec2-webservers | Fix /usr/bootstrap.sh to run pkgin with -y] *********************
changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})
TASK: [ec2-webservers | Install pkgin via /usr/bootstrap.sh] *********************
changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})
TASK: [ec2-webservers | Copy over Ansible binary package] *********************
changed: [127.0.0.1]
TASK: [ec2-webservers | Install Ansible dependencies] *********************
changed: [127.0.0.1]
TASK: [ec2-webservers | Install Ansible package (manually)] *********************
changed: [127.0.0.1]
TASK: [ec2-webservers | Setup lame /usr/bin/python symlink] *********************
changed: [127.0.0.1]
TASK: [ec2-dbservers | Launch new EC2 instance] *********************
changed: [127.0.0.1]
TASK: [ec2-dbservers | Give the system 30 seconds to boot up] *********************
changed: [127.0.0.1]
TASK: [ec2-dbservers | Get rid of SSH "Are you sure you want to continue connecting (yes/no)?" query] *********************
changed: [127.0.0.1]
TASK: [ec2-dbservers | Fix /usr/bootstrap.sh to run pkgin with -y] *********************
changed: [127.0.0.1] => (item={'cmd': 'install /usr/bootstrap.sh /usr/bootstrap.sh.orig'})
changed: [127.0.0.1] => (item={'cmd': 'chmod +w /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'sed "s,bin/pkgin update,bin/pkgin -y update," /usr/bootstrap.sh'})
changed: [127.0.0.1] => (item={'cmd': 'chmod -w /usr/bootstrap.sh'})
TASK: [ec2-dbservers | Install pkgin via /usr/bootstrap.sh] *********************
changed: [127.0.0.1] => (item={'cmd': u'env PATH=/usr/sbin:${PATH} /usr/bootstrap.sh binpkg'})
TASK: [ec2-dbservers | Copy over Ansible binary package] *********************
changed: [127.0.0.1]
TASK: [ec2-dbservers | Install Ansible dependencies] *********************
changed: [127.0.0.1]
TASK: [ec2-dbservers | Install Ansible package (manually)] *********************
changed: [127.0.0.1]
TASK: [ec2-dbservers | Setup lame /usr/bin/python symlink] *********************
changed: [127.0.0.1]
PLAY RECAP *********************
127.0.0.1 : ok=18 changed=18 unreachable=0 failed=0
% euca-describe-instances RESERVATION r-a419f9d9 749335780469 ec2-webservers INSTANCE i-21b7c441 ami-5d0f8034 ... RESERVATION r-641efe19 749335780469 ec2-dbservers INSTANCE i-54a2ab3e ami-5d0f8034 ...
% env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-basic.yml PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ********************* TASK: [ping] ********************* ok: [ec2-54-235-44-118.compute-1.amazonaws.com] ok: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Install tcsh] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Add user feyrer] ********************* changed: [ec2-54-234-139-151.compute-1.amazonaws.com] changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Create ~feyrer/.ssh directory] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Enable ssh login with ssh-key] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Install sudo] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] changed: [ec2-54-234-139-151.compute-1.amazonaws.com] TASK: [Enable PW-less sudo-access for everyone in group 'wheel'] ********************* changed: [ec2-54-234-139-151.compute-1.amazonaws.com] changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Disable ssh logins as root] ********************* ok: [ec2-54-235-44-118.compute-1.amazonaws.com] ok: [ec2-54-234-139-151.compute-1.amazonaws.com] PLAY RECAP ********************* ec2-54-234-139-151.compute-1.amazonaws.com : ok=8 changed=6 unreachable=0 failed=0 ec2-54-235-44-118.compute-1.amazonaws.com : ok=8 changed=6 unreachable=0 failed=0
% ssh ec2-54-234-139-151.compute-1.amazonaws.com id uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel) % % ssh ec2-54-235-44-118.compute-1.amazonaws.com id uid=1000(feyrer) gid=100(users) groups=100(users),0(wheel)
% env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-dbserver.yml PLAY [security_group_ec2-dbservers] ********************* TASK: [Install mysql] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Install MySQL rc.d script] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Start MySQL service] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Install python-mysqldb (for mysql_user module)] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Setup DB] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Add db-user] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Copy over DB template] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Import DB data] ********************* changed: [ec2-54-235-44-118.compute-1.amazonaws.com] PLAY RECAP ********************* ec2-54-235-44-118.compute-1.amazonaws.com : ok=8 changed=8 unreachable=0 failed=0- Check and see if the database works as expected:
% ssh -t ec2-54-235-44-118.compute-1.amazonaws.com mysql -u webapp -p webapp Enter password: **** ... mysql> show tables; +------------------+ | Tables_in_webapp | +------------------+ | names | +------------------+ 1 row in set (0.01 sec) mysql> select * from names; +----+--------+------+ | id | first | last | +----+--------+------+ | 1 | Donald | Duck | | 2 | Daisy | Duck | +----+--------+------+ 2 rows in set (0.00 sec) mysql> bye
% env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-webserver.yml
PLAY [security_group_ec2-webservers] *********************
TASK: [Installing ap24-php53 package and dependencies] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Install Apache rc.d script] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Enable and start Apache service] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Enable PHP in Apache config file] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'LoadModule.*mod_php5.so', 'l': 'LoadModule php5_module lib/httpd/mod_php5.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': 'AddHandler.*x-httpd-php', 'l': 'AddHandler application/x-httpd-php .php'})
TASK: [Make Apache read index.php] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Add simple PHP test - see http://10.0.0.181/phptest.php] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Install phpmyadmin] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Enable phpmyadmin in Apache config] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Fix Apache access control for phpmyadmin] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Enable PHP modules in PHP config file] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*zlib.so', 'l': 'extension=zlib.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*zip.so', 'l': 'extension=zip.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mysqli.so', 'l': 'extension=mysqli.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mysql.so', 'l': 'extension=mysql.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mcrypt.so', 'l': 'extension=mcrypt.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*mbstring.so', 'l': 'extension=mbstring.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*json.so', 'l': 'extension=json.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*gd.so', 'l': 'extension=gd.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*gettext.so', 'l': 'extension=gettext.so'})
changed: [ec2-54-234-139-151.compute-1.amazonaws.com] => (item={'re': '^extension.*bz2.so', 'l': 'extension=bz2.so'})
TASK: [Create directory for webapp] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Deploy example webapp] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
TASK: [Create webapp symlink for easy access] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
NOTIFIED: [restart apache] *********************
changed: [ec2-54-234-139-151.compute-1.amazonaws.com]
PLAY RECAP *********************
ec2-54-234-139-151.compute-1.amazonaws.com : ok=14 changed=14 unreachable=0 failed=0
% links -dump ec2-54-234-139-151.compute-1.amazonaws.com/
It works!
%
% links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/phptest.php | head
PHP Logo
PHP Version 5.3.17
System NetBSD ip-10-80-61-33.ec2.internal 6.0.1 NetBSD 6.0.1
(XEN3PAE_DOMU) i386
Build Date Dec 14 2012 10:31:13
'./configure' '--with-config-file-path=/usr/pkg/etc'
'--with-config-file-scan-dir=/usr/pkg/etc/php.d'
'--sysconfdir=/usr/pkg/etc' '--localstatedir=/var'
%
% links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/
Showing table hf.names:
Cannot connect to database: Can't connect to local MySQL server through
socket '/tmp/mysql.sock' (2)(2002)
This step is done by preparing a shell script on both systems, which will then be ran to - depending on the system's security group - perform the proper steps:
% env ANSIBLE_HOSTS=./ec2.py ansible-playbook config-ec2-connections.yml PLAY [security_group_ec2-webservers;security_group_ec2-dbservers] ********************* TASK: [Collect EC2 host information] ********************* ok: [ec2-54-234-139-151.compute-1.amazonaws.com] ok: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Prepare connection-script in /tmp/do-connect-vms.sh] ********************* changed: [ec2-54-234-139-151.compute-1.amazonaws.com] changed: [ec2-54-235-44-118.compute-1.amazonaws.com] TASK: [Run connection-script] ********************* changed: [ec2-54-234-139-151.compute-1.amazonaws.com] changed: [ec2-54-235-44-118.compute-1.amazonaws.com] PLAY RECAP ********************* ec2-54-234-139-151.compute-1.amazonaws.com : ok=3 changed=2 unreachable=0 failed=0 ec2-54-235-44-118.compute-1.amazonaws.com : ok=3 changed=2 unreachable=0 failed=0
% links -dump http://ec2-54-234-139-151.compute-1.amazonaws.com/webapp/
Showing table hf.names:
+--------------------+
| id | first | last |
|----+--------+------|
| 1 | Donald | Duck |
|----+--------+------|
| 2 | Daisy | Duck |
+--------------------+
----------------------------------------------------------------------
Enter new values:
first: _____________________
last: _____________________
[ Submit ]
20 years back from today, NetBSD was initially checked into CVS. Revision 1.1 of src/Makefile was committed on March 21st 1993 on 09:45:37 by Chris Demetriou (cgd@):
% cvs log -Nr1.1 Makefile ... revision 1.1 date: 1993/03/21 09:45:37; author: cgd; state: Exp; branches: 1.1.1; Initial revisionNetBSD was started as successor to the Berkeley System Distribution (BSD) Unix with a focus on multiplatform support.
Personally, I've followed NetBSD since the day in 1993 when the Amiga port popped up, which was the first platform that the newly forked operating system was ported to after its separation from BSD.
Many things have happened in the past 20 years, and a lot could be shown and told for the history books at this point. But I guess that can be done later - I'd be happy to help out with such a project if someone wants to start it, though :)
For today I'm very happy that NetBSD is available on a wide range on platforms, runs the software that I want and gives me the assurrance it will be around tomorrow and hopefully for the next 20 years, too.
Cheers, NetBSD!
Update: Jeremy Reed pointed me at his BSDnewletter posting, which gives a number of details of NetBSD's history. Recommended reading!
I live in the commuting zone of Munich, Germany.
I have just returned from a trip to Tokyo to speak at AsiaBSDcon.
This text is supposed to serve both as a reminder for me for next year (if I can find a topic I'll definitely submit a talk again) and for other first-time visitors.
The trip out was also my first long-distance flight, and in the case of Munich-Tokyo, non-stop means a 12 hour flight.
Visiting a city where you are effectively illiterate and don't speak a word of the language, plus manners and body language are different is an intimidating idea. Fortunately for me, Tokyo is pretty accommodating to visitors who (only) read Latin script and speak English, and while I would not expect to be able to make best possible time between two random points of the Tokyo subway map, I am now confident that I would be able to get from point A to B whatever they are (especially: to get to the hotel or venue) with a bit of extra time and wandering around to find something labelled in English, with zero preparation whatsoever.
Crowds in Tokyo seem to be amazingly pulsed. For me respectively in my experience, body language is off enough to make the automatic evasion algorithms fail, i.e. when I walked through an oncoming crowd I evaded to the same side as the person coming towards me frequently, whereas "at home" this barely ever happens without me even taking notice. Fortunately, when I am in a touristy setting I can just step aside and let the wave pass, and not be a mobile source of turbulence in the flow. When in a hurry, picking a person going the same way and trailing them like a duckling works pretty well - same, obviously, when following a guide. :)
Japan drives on the left side of the street. On escalators, stand on the left.
Mind the 110V line voltage. Get an adapter for a two-pronged socket, there may not be any three-pronged sockets in the hotel room.
Pedestrian traffic lights for street crossings of one or two lanes in total seem to be considered purely advisory by the locals; I picked up their cues (check and cross if there's no cars) without ill effect.
Car drivers seem to be very aware and considerate of pedestrians. A pedestrian giving up right of way for a car that's starving at a crossing seems to be unexpected, just slowing down noticeably and nodding at the driver doesn't seem enough cue.
Street population density, in general: walking around in Akihabara on a Saturday afternoon was about on par with walking between Stachus and Marienplatz off the touristy seasons on a Saturday afternoon, full but not reaching annoying levels, same with Ikebukuro on a Friday afternoon.
Another strong (and pretty immediate) impression I got is that Tokyo is what you get if you take an ordinary large city, and then compress it by a factor of four, while keeping the width of the thoroughfares. Then set another city next to it, repeat a few times.
Some of that compression is achieved by making buildings higher. Also, there are shops on other levels than the ground floor in places that are not something like malls.
Attention, weird people crossing? less goths or punks, instead people (well, women and girls) in neon-colored tulle or furs. About as many women in kimono (I did not notice any men in traditional garb) as you'd expect women in Dirndl in Munich outside of Oktoberfest season; I was told that it was a graduation day for students and that was celebration finery.
Less women in sensible shoes, instead pumps or sneakers (and the latter obviously not to business clothes). Walking in very small steps seems to be considered feminine and some execute it even if not forced by a ridiculously tight skirt.
The canonic length of skirt seems to be "a bit shorter than the hollow of the knee", which makes sense if you see how women sit (when not on chairs). Generally, sitting .. I was taught as a girl to not sit on the floor, it's bad for you (more likelihood of bladder infections for females), it's just not being done. Obviously, in Japan one does sit on the floor (inside, on a cushion, but still). Also, my jeans (any of them) turned out rather not fit to fold up my legs at a significantly acute angle, with the result that when I tried to sit at table, my feet went to sleep in turns due to lack of circulation. If going out to dinner with Japanese hosts, it seems a good idea to either warn them that you have issues sitting down on a cushion (the variant with the hollow under the table obviously just requires being able to get down on the floor, only a potential problem for the elderly or infirm, or seriously tight jeans), or get distinctly wider trousers or a skirt. Not going to wear such a short one though. :)
I've loved most of the Japanese food I got around to trying, yet. I've cheated and been eating at Sushi restaurants most work Fridays for a while, so I can handle chopsticks adequately well. I can imagine learning Japanese to the skill level I have for French: say good morning, good night, please and thank you, and everything that is likely to appear on a menu. Due to the remarkable (and very practical) drinks ordering system, being able to read the words for water, apple juice and grapefruit juice as well as cold tea would be very useful. And I didn't even get around to eating a bowl of soup with udon (which is one of my favorite dishes at home).
Akihabara (if you get out on the right end of the station and actually find the right area) is lots of little shops with electronics, many of them featuring used goods. About Schillerstrasse times ten. From my by necessity cursory inspection, the selection of actual electronics by and large is about the same as available in Munich; the selection of accessories is huge, variable, and including the utterly ridiculous. :) Definitely a sight to see. I got myself a mouse pad with a glittering-big-cat-spots pattern as a souvenir.
Also, prices for a lot of stuff are surprisingly cheap. Others, quite the opposite.
My excursion to Ikebukuro was to buy a toy or two to bring for my cats. I get the distinct impression that keeping a cat (or two) in Tokyo is a luxury not afforded by many (in Germany, a fifth of the households has cats). I picked toys and snacks not available at home (that being the point), and the shiny-bug-on-a-glass-fiber-spring toy is very successful and making even the old cat jump around like a kitten. :) I'd expect a similar item to be half the price in Munich, though, and I went to a department store with a pet section; there seem to be more boutiques selling art with cats on it (if such a shop exists in Munich I've not noticed it) than shops selling supplies for cats. That seems to be a symptom of rather severe cat deprivation.
On speaking English with Japanese (by necessity: I don't speak a word of Japanese and would be shy to try phrases lest I talk about eels in hovercrafts by mispronunciation): Japanese sounds distinctly percussive to me, and the rate of syllables a Japanese speaker can produce is impressive to frightening. Alas, parsing audio input at that rate is a learned skill that I didn't have time to develop, so while talking to (or rather understanding) relaxed (and slow-speaking) Japanese worked rather well, in other instances I had serious trouble following speech even though, in retrospect, it probably consisted of the appropriate words with the appropriate pronunciation, just in rapid fire.
The English skills encountered were as expected, rather well developed with the conference attendants I talked to, understanding-a-lot-better-than-speaking with the random ticket clerks I interacted with. Given the larger difficulty of learning English for Japanese speakers than German speakers, this is actually pretty impressive.
Regarding manners, either the people I interacted with are too used to non-Japanese to bat an eye at mine, were too polite to twitch enough for me to notice or I didn't blunder regarding the major differences between expected behavior that badly. My experience was pleasant, I hope my hosts' experience was too.
The hotel room was well appointed, small but (compared to European hotel rooms I've staid at) not unusually so, and entirely adequate for sleep, freshen up, look at mail, go out schedules like a town visit or a conference. Instead of hotel wifi, a wired ethernet port and two power sockets flush mounted in the top of the desk; an ethernet cable was supplied. There also was a large TV that I didn't turn on once. The blow-dryer was a useful model, not the useless heat-but-no-wind model most European hotels sport due to regulations limiting air throughput. That was fortunate, since the blow-dryer I typically travel with only does 220V (so I had left it at home). The soap dispenser produced bergamot-scented foam, very nice. The fixtures were "two valves, hot and cold, one outlet", no handling problems for me there.
The toilet experience is funny. :) Unless the toilet (room) was unheated (which was not true at any of the places I used), the heated seat seems a bit pointless, though. The "shower" setting might come in handy on occasion, the "bidet" setting for me does not end up anywhere useful.
I'm used to a mattress with at least 15cm give. The bed in the hotel room, true to local custom, has about 5cm. The first morning I woke up with lower back pain from a warped spine (men may not run into that problem due to usually not having wide hips), the second night I seem to have learned to pick a better sleeping position and that problem didn't reoccur. But enough on the TMI section.
I fought with stage nerves and spent most of Friday fiddling with my slides, adding illustrations and examples which were partially culled again. Thankfully, my talk was early, in the second slot on Saturday, so I was able to enjoy the rest of the conference. Feedback to my talk was that it was ok, it would be useful to also receive criticism, especially how useful the distribution of talking and slide points was to attendants with less familiarity with spoken English.
As usual, breakfast, lunch, dinner and breaks were the most valuable parts of the conference. I picked up some ideas from the talks I attended, as well. They are all in the "time-consuming" quarter of Things To Do, so we'll see if and how and when I get around to them.
I quite liked the 10min talks format. Since it allowed to talk about uncooked eggs as well, it was rather information-dense -and- news.
AsiaBSDcon gender distribution is quite the familiar, aka slightly depressing.
The number of Germans, OTOH, was amazing.
EuroBSDcons manage to produce input overflow for me, this AsiaBSDcon was worse in that respect.
I think I met half the NetBSD developers at the conference, talked to a quarter, and will be able to put face and name together for about 2/3 of who I consciously met. Time definitely was too short :) Also, if I manage next year, I'll try to get t-shirt colors and sizes via mail beforehand; this year the flu I had three weeks before the conference seriously impacted plans, and getting the talk and slides done had to take precedence. Jörgs tip that the .jp NetBSD developers tended to "long and thin" and M and L were most useful sizes mostly played out, so those t-shirts I managed to pack found good new homes.
It was fun. I want to do it again (just not this week, or next, and with a bit more preparation time next time. Mercy!). It was exhausting. The latter would get better with repetition, since the sheer amount of new people, new places, new ways to interact would lessen. comments
The second release candidate of NetBSD 6.1 is now available for download at: http://ftp.NetBSD.org/pub/NetBSD/NetBSD-6.1_RC2/
NetBSD 6.1 will be the first feature update for the NetBSD 6 branch. There are many new drivers, some new features, and many bug fixes! Fixes since RC1 include:
A complete list of changes can be found at:
http://ftp.NetBSD.org/pub/NetBSD/NetBSD-6.1_RC2/CHANGES-6.1
Please help us test this and any upcoming release candidates as much as possible. Remember, any feedback is good feedback. We'd love to hear from you, whether you've got a complaint or a compliment.
I’m giving a talk called “Rehabilitating pkglint” later this month at pkgsrcCon, which means I need to do some work worth talking about. pkglint is a tool for pkgsrc developers to find common errors in packages. It’s not part of pkgsrc infrastructure proper, but it’s a key step in the package development cycle. And it has problems. One proposed solution is to rewrite it.
A rewrite may or may not be a good idea (I bet not), but in either case, the beginning of the solution to pkglint’s problems is to first understand and document its current behavior. I’ve retrofitted (very cautiously) a place to put tests, along with a handful of test cases. A long road of adding coverage and safely refactoring lies ahead. To get as far as I possibly can by the 23rd, I need to be able to:
If pkgsrc were managed in a DVCS, I’d be good to go. Since it’s in CVS, I did this:
$ echo .git >> ~/.cvsignore
$ echo CVS >> ~/.gitignore_global
$ cd .../pkgsrc/pkgtools/pkglint
$ git init
$ git add .
$ git commit -am 'Add pkglint to git for offline hacking.'
$ git checkout -b rehab
Now I’m tracking pkglint in its own little git repository. The master branch tracks upstream, and the rehab branch is where I hack:
$ cd .../pkgsrc/pkgtools/pkglint
$ git checkout master
$ cvs update
$ git commit -am 'Track upstream changes from pkgsrc CVS.'
$ git checkout rehab
$ git rebase master
$ vi files/pkglint.t files/pkglint.pl && make test && make clean
$ git commit -am 'More cool stuff on my branch.'
This is a slightly clever trick, but not too much so. I’m happy and productive with it so far.
Damn I love pkgsrc. Let me tell you this story as an example…
A while ago, a couple of pkgin users told me it was a shame that /usr/pkg/etc/pkgin/repositories.conf was still pointing to a 5.0 URL when pkgin is freshly installed. Thing is, pkgin does support the $osrelease variable, but on NetBSD, the result of kern.osrelease can be 6.0_SOMETHING, which would lead to:
ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/6.0_SOMETHING/All
and this does dot exists on the repository.
So in the REPOSITORIES file, which is used to generate the right repositories.conf entry, I added that line:
ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/@OSREL@/All
And wrote the following trick in pkgtools/pkgin‘s Makefile:
.if ${OPSYS} == "NetBSD"
SUBST_CLASSES+= osrel
SUBST_STAGE.osrel= pre-configure
SUBST_MESSAGE.osrel= Adjusting repository OS release
SUBST_FILES.osrel= REPOSITORIES
SUBST_SED.osrel= -e "s|@OSREL@|${OS_VERSION:C/_.*//}|"
.endif
The SUBST framework, part of pkgsrc, will then use (a portable) sed to replace @OSREL@ with the variable ${OS_VERSION}, which is defined in mk/bsd.prefs.mk (included in the Makefile), but will erase anything like the regexp “_.*”. This magic is done by the :C modifier, which behaves like the :S modifier but is capable of replacing regexps instead of simple strings.
Learn to know the power of pkgsrc…
The post pkgtools/pkgin, quick fix appeared first on Emile "iMil" Heitor 's home.
I’m a latecomer to the Destroy All Software party. Lots and lots of catching up to do. Sure would be nice if I could watch at home on my iPad and on the subway with my iPhone and let Downcast keep them in sync about what I’ve seen so far. But DAS doesn’t have a podcast feed and, since it’s coming to an end in the next month or two, probably never will. So I made my own podcast for it. Glossing over most of the details:
I was already running ikiwiki and bozohttpd on my laptop for my personal wiki, so I had already jumped through many of these hoops.
ikiwiki configuration, taking care to add_plugin the inline plugin.index.mdwn at the root of the wiki containing pretty much just this: [[!inline atom=yes pages="media/*" sort=title reverse=yes archive=yes]] (tweak to taste).media subdirectory and put the videos there./path/to/ikiwiki --setup /path/to/ikiwiki.setup, then browse the wiki. All the videos should be linked there.media folder that does this:on adding folder items to this_folder after receiving added_items
do shell script "/path/to/ikiwiki --setup /path/to/ikiwiki.setup"
end adding folder items to
When Gary announces a new screencast:
media.When the iPad and iPhone next refresh their subscriptions, they’ll get the new screencast.
Schmonzcast #13: I’ve attended (and spoken at) more than a few instances of pkgsrcCon, but I’d never attended a full-on costs-money big-crowd software development conference until two weeks ago. Perhaps counterintuitively, the first thing I did there was an all-day Speaker Workshop. While the air at the hotel was thick with code, what was fresh on my mind was the pain of meticulously planning and presenting half-hour talks at work every week for the past couple months. I wanted to know how it might get easier. The instructor, Leon Gersing, combined a background in theater with perceptive, intuitive teaching. I came away much more aware of what to work on. (For example: my personality is slightly goofy in a way that, once I’m comfortable enough in front of a crowd for it to show through, probably works in my favor.) And when Corey Haines organized a session of lightning talks two days later, I decided to get right to work, then and there (or watch on YouTube). Given the format, I like how it went; for next year’s CodeMash, I plan to submit a full-length talk proposal.
[Originally posted to Remember Aaron Swartz]
I maintain pkgsrc packages of some of Aaron’s software, and over the last decade or so would occasionally send him bits and pieces of code. Along the way we somehow discovered that we’d grown up a couple hundred meters apart, separated by a driveway and some trees. (If only we’d known, and been around at the same time, our common interests would doubtless have made us fast friends.)
When sending patches I’d always ask what he’d been up to lately, because it was always something new and ambitious, even audacious. We were just nerds on the Internet whose paths occasionally crossed, but I felt like I knew him better than that. Above all, he had managed to hold fast to a way of thinking about our world — as evidenced by his writing and his actions — that I had forced myself to hold rather more loosely. I always admired him for that.
Had we had the chance to know each other better, I’m quite sure I’d have admired Aaron for much more.
Well my web server rebooted. I don't know why. I will post another story about the significance of this to me. Anyways, my /home (where all my website docs were located) was not mounted.
The disklabel output was strange:
16 partitions: # size offset fstype [fsize bsize cpg/sgs] a: 8192016 63 4.2BSD 2048 16384 27560 # (Cyl. 0*- 8127*) b: 8192016 8192079 swap # (Cyl. 8127*- 16254*) c: 156299312 63 4.2BSD 2048 16384 28752 # (Cyl. 0*- 155058*) d: 156299375 0 unused 0 0 # (Cyl. 0 - 155058*) disklabel: partitions a and c overlap disklabel: partitions b and c overlap
The disk is:
Partition table:
0: NetBSD (sysid 169)
start 63, size 156299312 (76318 MB, Cyls 0-10337/62/30), Active
So I did some math with bc: Size of c is 156299375-63 = 156299312. Size of d is already known (entire disk). Size of e is 156299375-16384095 = 139915280. Start of e is 8192079+8192016 = 16384095 (end of swap plus size of swap).
And did with disklabel -e:
16 partitions: # size offset fstype [fsize bsize cpg/sgs] a: 8192016 63 4.2BSD 2048 16384 27560 # (Cyl. 0*- 8127*) b: 8192016 8192079 swap # (Cyl. 8127*- 16254*) c: 156299312 63 unused 0 0 d: 156299375 0 unused 0 0 e: 139915280 16384095 4.2BSD 2048 16384 28752 # (Cyl. 0*- 155058*)
I guess I could have used disklabel -i for interactive to have it figure out the offsets and sizes for me. Then I fsck'd it. For some reason the fstab had: /dev/wd0c for the home so I changed to wd0e. Then "mount -a" and another reboot to verify it again. Odd that this weird disklabel was in working and in use for long time.
For a few days I worked on an article for BSD Mag that introduced (briefly) the many new tools and daemons included in NetBSD 6.0. Many of this was new development done on NetBSD, while some is third-party software useful enough to be included with the base NetBSD. The article should be published by the magazine sometime in December. Another article needs to be written to introduce the many rump tools.
I noticed several man pages were missing. So I wrote a draft of mkubootimage(1) even though I haven't used it. (It is now at http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/mkubootimage/mkubootimage.1.) Other missing manuals include: sqlite3 (I may handle that), ztest, kcc (heimtools), digest-service, and gsstool.
Also the Google Code-in is in process. This is a Google sponsored event with high school students doing short (couple hour) tasks to help open source projects. NetBSD has many tasks available and I helped review and give feedback to some students.
As a strong C++ dev and googler (hopefully with some #golang exposure), what's your opinion on @rob_pike post? (goo.gl/xlMi4)The answer deserves much more than my original reply included, so here it goes.
--- optiloader.ino 2012-07-18 20:53:16.000000000 +0930
+++ ../optiLoader.pde 2012-07-18 20:51:49.000000000 +0930
@@ -50,15 +50,6 @@
// 9 to power more complex Arduino boards that draw more than 40mA, such
// as the Arduino Uno Ethernet !
//
-// For a leostick the pins are:
-// 0: slave reset (arbitrary allocation)
-// 14: MISO
-// 15: SCK
-// 16: MOSI
-//
-// The pins are only available on the ICSP header, this header
-// also supplies the power to the other board too.
-//
// If the aim is to reprogram the bootloader in one Arduino using another
// Arudino as the programmer, you can just use jumpers between the connectors
// on the Arduino board. In this case, connect:
@@ -98,11 +89,11 @@
/*
* Pins to target
*/
-#define SCK 15
-#define MISO 14
-#define MOSI 16
-#define RESET 0
-#define POWER 1
+#define SCK 13
+#define MISO 12
+#define MOSI 11
+#define RESET 10
+#define POWER 9
// STK Definitions; we can still use these as return codes
#define STK_OK 0x10
This is due to the leostick not having the programming pins in the "normal" arduino place. I made up a cable and managed to program the Etherten bootloader but still no joy. While I was mucking around with this I also found an usb-asp on ebay which is an in circuit programmer for atmel chips which happens to be what the arduinos use. The programmer and a 10 to 6 pin adaptor came to less than $7 including postage so I bought those.The good news is that the usb-asp works fine under NetBSD with the native avrdude, I can upload sketches to the Etherten no problems. Avrdude is complaining that the usbasp needs a firmware update which is another adventure. I have loaded up the AVRisp sketch onto the leostick but I need to make a cross-over adaptor to connect my cable to the usbasp cable, then I can update the firmware. Soon, it will be the Summer (of Code) again, and this year, too, NetBSD will be participating.α At around the same time, last year, I was busy writing technical proposals for what would become xmltools, one of the accepted NetBSD SoC projects of 2009. Well, most of you have probably forgotten about that small project; some may even think it was a useless attempt and a waste of time and money, which could have been better spent elsewhere… Err, in fact, I’m writing this to say that my xmltools are not dead! And I still intend to contribute the code to the NetBSD tree sometime in the (near) future.
α : And if you’re a student who’s into systems and stuff, looking for a cool organization to join, with a broad selection of projects, why not give NetBSD a try? :)
That being said, if you take a look at the Git repo, you’ll surely notice that nothing has moved for months… If you fool around a bit more, though, you may notice another project on the same Git host: regxml. Yep, this is a rewrite of my xmltools!
Why a rewrite, you ask? Well, basically because the old implementation was flawed by design, in addition to having become a mess.
And here’s the long answer, for those who care. Feel free to skip this section if it bores you.
During the second part of last year’s SoC, as I was trying to work on xmlsed, I realized my design was flawed. The main problem was that I had based all my efforts around a concept that was, for our purpose, completely inappropriate: node sets.
So what’s a node set? A node set is simply that: a set of nodes. Now, that’s great if you are extracting data from an XML document, as did xmlgrep, but it sucks if you’re going to operate further on that. In particular, removing arbitrary nodes from an XML tree doesn’t yield a tree, and replacing a node set made no sense at all. (How would you replace two far away nodes with one big chunk of XML? There are many ways to go about it… but they’re all wrong IMO.)
So what emerged at the end of the summer was that… I needed a new design, and as my original algorithm had become crippled with dubious ad hoc attempts to support operations that it never could, I needed a new algorithm too. (Besides, there was a deficiency in my algorithm that made it perform in super-polynomial time with some patterns and data sets, which was bad.)
It took me something like one month (since I wasn’t full-time on it) to jot down the new concepts and a (hopefully) good algorithm on paper. Then, for months, I’d try to implement it, but each time, something would feel really wrong, and I’d trash my new implementation. Well, that was till some days ago, when somehow, I got it right, and so the new xmltools were born!
So all this boils down to the following fundamental change: XML node sets have been replaced by XML intervals, as the core objects the xmltools operate on. An XML interval is really just that: a bunch of adjacent siblings and all their children. You should read the included xmltools(7) manual page for more information, from a user’s point of view.
From a more theoretical viewpoint, intervals have two important advantages over sets:
Inserting or removing an XML interval in an XML tree keeps it a tree. In other words, XML trees are stable by insertion and deletion of arbitrary XML intervals. (It may not have any meaning anymore, but remember xmltools are about syntax, not semantics.)
XML intervals map (injectively) to byte intervals in the XML file, hence all editing operations on XML intervals (sequences of insertions and deletions) can be expressed as operations on sequences of bytes in the file itself.
The first property is what I am convinced will make xmlsed possible. Previously, with node sets, it was a nightmare to give proper semantics to the various operations. But now, we have simple objects that still retain a great deal of expressivity but can be manipulated easily, with simple rules.
Although I do not use the second property yet, I’m definitely thinking about introducing an API for that sometime in the (not quite near) future.
The idea is that you run the XML matcher on a file mapped in memory (mapping new chunks as needed), and get a program script consisting of byte operations, that you can then run very efficiently on the memory-mapped file! Instead of printing an interval, you’d just copy over the entire memory region. Instead of deleting an interval, you’d just ignore that chunk. You get the idea. It could be used for efficient processing of massive data with complex transformation rules, but we’re not quite there, yet. :)
As I’ve discussed with David (<dyoung>), I intend to import my core library, as well as xmlgrep into base sometime in the near future (once it has undergone more thorough testing). It depends on Expat for XML parsing, so I’ll need to import that too.
After much thinking, I think I’ll settle with a subdirectory in
src/external/bsd, though I use the NetBSD build system, as I want to
keep my development model using Git, and the ability to easily make
packages of all my tools for non-NetBSD platforms.
The new code base uses (Net)BSD idioms more extensively than the old
one (because I’ve got the time to read more NetBSD code since
I started in May of last year). (E.g. I’ve followed David’s suggestion
that I should make use of sys/queue.h.) Quite surprisingly, it
compiles almost as is on GNU/Linux, with three additional defines and
two trivial functions. I do provide a simple GNU makefile in addition
to the default NetBSD build system makefiles, for GNU/Linux builds. Of
course, you’re welcome to try to build it on other systems I don’t
have access to!
Well, what’ll come next depends on a lot of factors, some of which are completely outside of my control. But there is one thing you can do: please download, compile, read the man pages, and test xmlgrep and give me some feedback, so I can fix bugs!
$ git clone git://git.huoc.org/regxml.gitOr through Git-Web: http://git.huoc.org/?p=regxml.git;a=summary
As far as the xmltools project is concerned, I’m now going to concentrate on xmlsed. I already have an idea of the implementation, and I think it’s not going to be too hard… hopefully, I’m not wrong. :)
P.S.: Oh, I forgot to talk about the new algorithm. Well, to get the idea, it’s basically an automaton backed by a backtracking memoizing work-queue-based interpreter, which roams the XML tree and try to match intervals. You should read the xmltools(7) man page for a high-level overview of the technique. Maybe I’ll write something more consistent on the subject later…