mtjm’s blog

Personal server exit review

2015-12-11T21:22:47+00:00

I used a Cubieboard as a personal server until October, running Kanboard, tt-rss and some other Web apps there. It allowed me to keep more of my data at home and practice some more system administration.

Why Cubieboard

Initially I wanted a computer that would work during the night in my bedroom. This required a fanless design and therefore a low power usage. (Now I have a separate room for the computers, so they don’t need to be so quiet, while I pay their power bills.)

I could have either used my BeagleBone Black, a Cubieboard or buy a different board. I haven’t chosen the BBB for this since its 512 MiB RAM is not enough for a Web app that maintain in my spare time (Cubieboard’s 1 GiB is enough), and I wanted a more stable storage than microSD: Cubieboard has a SATA port which I connected to an SSD (also no moving parts). Meanwhile, I can use the BBB for flashing and debugging coreboot which requires some downtime.

(Now I think I wouldn’t need the personal server to work at night, but it’s easier: I can read some Web comics when I resume my laptop from suspend and get daily mails from cron and Kanboard. Also I won’t forget to turn it on before accessing these while traveling.)

How it was configured

ARM makes booting slightly more interesting than on x86. It booted u-boot from a microSD card which loaded a kernel from /boot on the same microSD card (I wasn’t able to make it load a kernel from the SSD, while it was documented to support SATA) which mounted a btrfs root filesystem from the SSD.

The server was running Debian Jessie, manually installed using its usual installer (with good support for such boards). I configured nearly all services running on top of it using Ansible. Much configuration was shared with my other computers, e.g. using OpenVPN, Postfix for relaying locally-generated mails to my VPS, etc.

Like all my computers (or, in case of Thinkpads, disks moved between computers), it had a unique hostname. I named it after Sam, the trusted friend of Alice, Bob and Frodo.

How I broke it

After receiving a mail from apticron about Debian package updates being available, I ran aptitude full-upgrade. There was a kernel upgrade, so I rebooted it. This worked many times, but it didn’t once in October.

After getting the serial TTL cable (which required removing the top of its case), I found errors from initramfs. Root couldn’t have been mounted due to filesystem errors. Checking in another computer (a big advantage of SATA over soldered storage chips), I have seen many btrfs errors, while all interesting files could be read.

So I copied the filesystem image to my desktop, ran mkfs.btrfs, copied all files to the new filesystem, in many reboot loops I fixed /etc/fstab and some initramfs configuration. Then it was not booting, probably not being completely configured to use the new filesystem.

Now

Not being able to fix it ‘now’, I migrated services to my desktop computer (really easy with Ansible). I used data restored from the filesystem image and a daily PostgreSQL image. (I couldn’t get the possibly corrupted newer PostgreSQL data: it won’t load files written on a different architecture, requiring using pgdump on armhf.)

Two months passed and I haven’t noticed a need for that server, so I still haven’t fixed it and use the desktop as a personal server. There is a difference in the power bill, while I don’t know how much of it can be attributed to the desktop running more often now.

Future

When I set up a new personal server, I will think about filesystem errors before it stops booting. Maybe periodically running btrfs scrub or choosing an older filesystem would help. Certainly, I should backup before installing any OS update. I should also get a recovery method for when the OS won’t boot (very easy on x86).

My task management: input queues

2015-12-10T22:47:37+00:00

I have a complex task management system, which was partially influenced by Getting Things Done and by blog posts of Leo Babauta, while it seems to have randomly evolved into a different thing since. I’m writing how it currently works (or doesn’t), to better understand what mistakes I made and how I can fix them.

To know what tasks I should do, I use my ‘real’ task management system (really many such systems, including a separate database of bookmarks from several browsers) which is a topic for a future article. I have to write all my ideas and input them to that system: it is too complex to be used in real-time. To efficiently organize this, and never feel guilty of forgetting an idea too complex to fit on a book’s margin, and to blame the ‘real’ system for forgetting important deadlines, I have multiple slightly formalized methods of keeping these ideas until I add them to a real task management system.

I call them input queues. That term is misleading: they are not input, since if I can do a task ‘now’ instead of inputting it to the ‘real’ system, I do it; they are not queues since they are random streams of consciousness which I split, merge and reorder into tasks.

Handwritten notes

The most basic input queue was a square piece of paper with some notes handwritten (using a pencil, nearly never a pen). Each day I checked all such queues, and marked with vertical lines on left the notes of inputted tasks (striking out lines corresponding to already done tasks). I never needed them after inputting all tasks and having no space left. (Two situations motivated me to use these cards as an input queue only and keep a separate digital ‘real system’: previously I had to carry many of these when traveling, not only enough to fill while away from home, and my family had thrown away all my important notes stored in a large cylindrical container.)

Phone notes

Now I usually carry my phone and it’s easier to access than the pencil and paper. I use the Simply Do Android app to maintain a single task list named ‘queue’: I add all ideas as items there and remove them when inputting them to the main system (or when putting the listed products in a shopping basket: a second order input queue). The app is too complex for my needs, but it works well enough. While this doesn’t have such a technical motivation as with paper notes, I type all tasks from this queue to the main system manually: it requires more thinking than using software to automatically copy the list and might lead into better notes.

Email

I have adapted my email inbox into another input queue. I have a rule of having no unread mails, unless they are for tasks to input (or do) during the nearest several days. I keep the inbox empty by periodically moving mails to other formats (this is a task scheduled in the main system; I used to do it once a week before backup, while I’m now moving to more automated daily backups).

Really there are two ‘inbox’ folders: the real inbox for mails not sorted by my Sieve filters to other folders, and a separate one for notification mails, e.g. from mailing list moderation, from apticron and Parabola issue trackers. (Mailing lists get their own folders, inbox mostly gets spam and mails for which I haven’t updated the Sieve script yet.)

I keep the usual input queue rules for unread mails: I mark the uninteresting ones as read when browsing new mails (on phone or from Gnus), when doing the needed task (replying, approving moderated list mails, upgrading systems, etc) or when adding a task in the real system to do more. Rarely I check my sent mails for ideas for blog posts.

IRC

Some entries in the real task system result from discussions on IRC; I need to remember to never refer to IRC logs: they are unreadable and unsearchable. I copy the relevant log fragments if needed. (Same when reporting Parabola issues.)

Browser tabs

I use browser tabs for pages that I’m currently reading (or planning to read in near future), for Kanboard dashboard (one of my ‘real systems’ which I input the tasks to), and a blank tab. The blank tab would be useful if I really cleaned the queue and closed all pages to read: the browser loses my tab groups if it doesn’t have at least two tabs open. (I also used to read a microblogging site which counted the number of unread posts unless their page was in the active tab: so I needed at least one other open tab to know how many posts to read. This was solved by technical changes making that site nearly completely useless for me.)

Tabs don’t work well for the long-term: Firefox-based browsers sometimes forget the page position, e.g. when the page temporarily fails to load. They are worst for pages that I read up to a point and return to after a year of not visiting their tab. (Most smaller browsers that I know don’t keep the page position or tabs at all when restarting them, so they are completely unusable for me. Firefox used to have a reading mode which also had this misfeature; it’s ironic.)

Ebooks

I don’t have a separate ‘Downloads’ directory on my personal computers: all files downloaded from the browser go to the desktop directory. In case of PDF ebooks, I either read them and move to my git-annex repo with such files (in an appropriate directory for read PDFs), or move them to the ‘to read’ directory.

Browsing them in Okular, I reopen them on the page where I stopped reading. I use other systems to manually synchronize page numbers between different computers where I might read the same book.

For EPUB ebooks, I use Cool Reader on my phone. After reading the book, I remove the file from the phone, possibly moving it to the git-annex repo or a random directory on one of my computers. Pages with ebooks are bookmarked; for serialized novels (like most fanfics) I use a custom bookmark record field for tracking reading progress. (Several years ago I had an eink ebook reader device; initially I used it with very poor and buggy vendor firmware, later with OpenInkpot; it is why I now run a newer version of the same application as was included in OpenInkpot on my phone.)

Feed reader

I have a personal instance of Tiny Tiny RSS (previously on my remote VPS, then on various computers that I own and access via my VPN). Most of the feeds are Web comics; but there are also some interesting technical blogs and aggregators. It naturally works as the main source of links in my bookmarks system, both in the ‘read’ and ‘to read’ categories.

Initially I kept all unread articles in tt-rss, but later I lost them by accidentally marking all articles as read (instead of just articles from a single feed). Now I have thousands of the ‘to read’ bookmarks exported from a tt-rss database backup. I plan to periodically export newer ones there too.

Conclusions and future ideas

Writing this helped me notice several significant problems in my current workflow. (‘Current’ is a very relative word here: I changed many parts of the system, but mostly not the input queues, during a several months break between starting and finishing this article.)

Each input queue (other than the handwritten notes) involves a separate set of programs. There is no integration between them and with the ‘real’ systems. This is by design, but I have surely noticed this before calling it a ‘design’. Instead of just several complex task management systems (‘real’ ones), I also have many more for tasks that I move to the more complex ones.

I had many issues with synchronization of data between computers: there should be no such problems here due to the input queue idea. Each queue is separate and all its tasks are quickly moved to the ‘real system’ or done. This doesn’t work well with big tasks like reading a large book (or any sufficiently complex fanfic). (My experience with advocates of task management systems suggests that I’m using my own system incorrectly, so I don’t get its claimed benefits.)

Maybe I need to completely separate input queues from the real systems (and put reading the book in the real system), but open tasks in these systems need reevaluating if they are needed if I’m not doing them for a longer time… becoming another input queue.

To ease synchronization between computers, I preferred storing most data (other than input queues) in DVCS repos. But there are no obvious winners among distributed task management systems; I have chosen using Kanboard as one of the ‘real systems’ instead of writing my own. Now I run Kanboard on one computer and access these various systems from one laptop only, making distributed designs much less important.

I also often forget about synchronizing most input queues (and that I have several others, like files on the phone), so my real systems have tasks reminding me.

Inclusion of licenses longer than licensed works

2015-04-19T13:36:42+00:00

There are licenses known for excessive attribution requirements: in a single project the old four-clause BSD license required including 75 different texts in all advertising materials. The license text itself can be long (GNU FDL 1.3 takes more than 3 500 words, the Web browser that I use would spend nine A4 pages to print it): imagine an award pin with an FDL-licensed image or several pages long document derived from a GNU manual. Both need to include the GNU FDL text. It makes the license, despite being free (possibly in specific cases for FDL; in all cases for GNU GPL), unusable for some kinds of free works.

If you don’t consider award pins sufficiently complex and original, imagine a postcard from a traveling family member. It should have a beautiful photo on one side, like the ones that Wikimedia Commons has, and the whole other side filled by a letter describing their holidays, and your postal address. There is no place to fit nine pages of license text there, and the postcard is distributed by itself, so no separate booklet with required legal texts can be included.

It’s one of the reason for GNU FDL being used for ‘professional’ photos: it’s free, so it is accepted in free culture projects like Wikimedia Commons, but it's unusable so proprietary relicensing businesses work. Wikimedia Commons now discourages using GNU FDL for photos without dual-licensing under a more usable license.

I do believe that this is a significant bug in the license: copyleft licenses should be designed to not support proprietary relicensing or proprietary extensions businesses (i.e. proprietary software businesses) and should not have features that are useful nearly only for such businesses (while FDL has several, possibly since the license was designed to be used by traditional publishers). (There are several different problems in other, more important, copyleft licenses like GNU AGPL or GNU GPL3, e.g. the optional attribution requirement. Some of them are solved in copyleft-next; e.g. the Nullification of Copyleft/Proprietary Dual Licensing clause protects against proprietary relicensing by removing the copyleft for all in some cases.)

How can we solve this problem? By not distributing FDL-licensed works and by not recommending the use of such licenses for cultural works. This requires recommending specific better licenses.

GNU recommends their all-permissive license for short documents like README files. Unless the work is a part of a GNU package, a free Creative Commons license is probably a better solution: copyleft (without source provision requirement) CC-BY-SA, permissive CC-BY or ‘public domain but legal everywhere’ CC0. In its clause 3(a)(1)(C), CC-BY-SA 4.0 requires to

indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License,

so it’s sufficient to fit an URI like https://creativecommons.org/licenses/by-sa/4.0/. (I have seen a much longer text than this URI written on a single pea seed in a local museum, so this surely works for bigger works like award pins or postcards.)

A more general term is used in copyleft-next: ‘inform recipients how they can obtain a copy of this License’ which is obviously satisfied by an URI. (The whole officially recommended license notice is: ‘Licensed under copyleft-next version 0.3.0. See https://gitorious.org/copyleft-next/copyleft-next/raw/master:Releases/copyleft-next-0.3.0 for more information’. Compare the three paragraphs recommended for the GNU GPL.)

This couldn’t have been done several decades ago. There was no Web in 1991 when GNU GPL2 was released (this is why usual GPL legal notices had an FSF address, changed several times after the license was released, until the GPL3 with both an URL and distributed license copy). It was reasonable to assume that the user couldn’t have obtained the license text from the Web, but now it’s probable that every computer user can access the Web, although not necessarily from their home. (How many GPL software recipients can access postal mail to use the source offers and not the Web?)

(This is not the only problem with long licenses or requiring to include their text in the work. It is a bigger problem that some licenses are too complex or too badly written to be understood by users, but that problem cannot be as easily quantified as their texts not fitting in the work: understanding of licenses is ‘cached’ in memories of their readers who have already met e.g. the GNU GPL3 for many other works. It would be also possible, and evil, to write a very short and incomprehensible license.)

PlaneShift and free software

2015-04-11T10:28:40+00:00

On the download page of PlaneShift I see big letters ‘Fully Free Cross-Platform MMORPG’ and ‘Open Source Development!’. They provide the source code of their client, while writing how this helps user’s freedom and security. (I prefer using clearer terms like free software and copyleft for the exact things that they praise. While I played PlaneShift many years ago, I do not have any opinion on it beyond what I write in this essay, since I’m not interested in multiplayer games.)

However, they write that they need ‘some additional bounds [in the license] to keep safe the work of [their] artists and to ensure project success’. This both supports false assumptions (there are safe and successful projects releasing fully free cultural works) and significantly reduces the benefits of their licensing for user’s freedom and security.

They have clearly explained their licensing and its rationale. Source files are licensed under the GPL, while artwork, text and rules in the game use a custom nonfree license (called the PlaneShift Content License).

Despite using a free and copyleft license, the client has significant restrictions on user’s freedom:

‘You cannot distribute the client, sell it or gain any profit from it’
‘You can use our client only to connect to Official PlaneShift Servers’

So of the free software freedoms only a small part of ‘the freedom to study how the program works’ applies. It does not belong ‘to the community of OS developers’, it belongs to Atomic Blue, the organization running PlaneShift. While their licensing is rationalized by making forking as hard as possible, all benefits of free software that they write about require forking.

The ‘content’ license is short and simple. It forbids any distribution or modification of the work, allows using it only (personally) with their official servers and ‘a Planeshift Client, distributed by Atomic Blue’, and disclaims all warranty.

I’m not able to understand what their encouragement for users to ‘experiment with mods and changes to either [their] source code and to [their] art assets’ might mean. Are they recommending infringing their copyright or promoting fair use in a very unclear way?

The requirement to use the artwork ‘only in conjunction with a Planeshift Client, distributed by Atomic Blue’ might forbid using the client software if built from source. So that software, as normally used, is as free as if it was written on stone tablets, impossible to copy nor modify. All security benefits of its source code being free disappear, when a Mallory just needs to backdoor Atomic Blue’s compiler.

Even if a client built from source could be used, GNU/Linux distros wouldn’t be able to include that game, since they wouldn’t be allowed to distribute the needed artwork. The source might be free, but it’s not useful without the nonfree artwork. (Or is it? Write if you know a free derivative of it working without Atomic Blue’s artwork and servers.)

Intel GM45, AMD K10 and Richland compilation benchmark

2015-04-01T19:44:13+00:00

The fastest computers working with a fully free boot firmware distribution are Intel GM45 laptops from 2008 like Lenovo X200 and R400. No newer Intel system can work without signed and nonfree firmware which probably cannot be replaced without a significant breakthrough in number theory. Some AMD systems are known to work without nonfree firmware with coreboot, up to K10 (with some hope of liberating newer systems, up to Richland and Kabini APUs), while no one provides a binary distribution of free boot firmware for them. Is a K10 desktop board from 2010 faster than GM45, making it a useful target for improving performance of systems running completely free software?

Let’s compare compilation time of bash, version 4.3. It should depend mostly on the CPU (using all of their cores) and memory.

Machines

In this article I name computers after their chipsets, since these are most important for support in boot firmware. I have these computers:

GM45 + ICH9: Lenovo R400 with Intel Core2 Duo P8400 CPU (two cores), 8 GiB DDR3 RAM
K10 + RS780: ASUS M4A78LT-M-LE board with AMD Athlon II X2 255 (two cores), 4 GiB DDR3 RAM
Richland (Family 15h model 10h) + Hudson: ASUS F2A85-M board with AMD A8-6600K APU (four cores), 16 GiB DDR3 RAM

The K10 system runs a proprietary BIOS (similar boards are supported in coreboot), while the two other boards use coreboot with custom hacks to not identify R400 as X200 and to use a Richland APU instead of Trinity on F2A85-M.

The GM45 and Richland systems run Debian Sid, while K10 runs Jessie. I have installed the needed dependency packages using aptitude build-dep bash. GCC identifies itself as gcc (Debian 4.9.2-10) 4.9.2 on all three systems.

Benchmark script

#!/bin/sh

set -e -x

build=$(mktemp -d)
cp bash-4.3.tar.gz "$build"
cd "$build"
tar xf bash-4.3.tar.gz
cd bash-4.3
./configure -q
time make -s -j$(nproc)
make -s clean
time make -s -j$(nproc)

cd -
rm -rf "$build"

The package is built twice so some work is hopefully done before the CPU changes to the highest frequency.

I use a tmpfs for /tmp on all my computers, so the sources and built binaries are kept only in memory. This should avoid benchmarking the varied disks in my computers.

Results

GM45:

42.32user 2.76system 0:25.16elapsed 179%CPU (0avgtext+0avgdata 73808maxresident)k
496inputs+0outputs (5major+956964minor)pagefaults 0swaps
42.36user 2.72system 0:25.78elapsed 174%CPU (0avgtext+0avgdata 73868maxresident)k
0inputs+0outputs (0major+956294minor)pagefaults 0swaps

K10:

33.38user 1.84system 0:19.93elapsed 176%CPU (0avgtext+0avgdata 73408maxresident)k
0inputs+0outputs (0major+950564minor)pagefaults 0swaps
33.40user 1.86system 0:19.94elapsed 176%CPU (0avgtext+0avgdata 73376maxresident)k
0inputs+0outputs (0major+950164minor)pagefaults 0swaps

Richland:

36.36user 2.69system 0:11.14elapsed 350%CPU (0avgtext+0avgdata 73768maxresident)k
0inputs+0outputs (0major+954845minor)pagefaults 0swaps
36.63user 2.84system 0:12.38elapsed 318%CPU (0avgtext+0avgdata 73800maxresident)k
0inputs+0outputs (0major+954075minor)pagefaults 0swaps

Compilation times of packages with well-written build systems strongly benefit from having more CPU cores, as shown by the Richland result: my quad core APU doesn’t use less user/system time than the dual core Athlon II. A K10 system with four or six cores would most probably get better results: the exact board that I have supports them, unlike the laptop limited to two cores.

How temporary data loss helped me improve my backup policies

2015-03-29T19:21:15+00:00

Two recent situations have motivated me to rethink and improve my backup policies. Once I have accidentally clicked the ‘Mark as read’ button in my feed reader which marked all 5 000+ unread entries as read, not only the ones from a single feed where I intended to do this. During another day, I have decided to reboot my phone. Then I spent thirty minutes trying various disk encryption passphrases that might be able to unlock it: I finally succeeded, knowing that I guessed correctly that I will forget it when I rebooted it previously a year and a half ago.

I have solved the unread feed issue in three ways: by removing the ‘Mark as read’ button from my installation of Tiny Tiny RSS, by exporting relevant data from a week-old database in my usual server backup to my file with bookmarks from browsers, and by manually marking as unread newer articles. (Why I’m not just marking them as read to shorten my backlog: there are very useful articles among them, often in areas that I don’t need to learn about today, but that I might need in future.)

The database backup wasn’t necessarily consistent and working, but I was able to use it for this purpose. Since that day, I have configured daily backup of all my PostgreSQL databases to a git repo using pg_dumpall. Unlike possibly inconsistent files being written to while the filesystem backup is made, SQL dumps made by pg_dumpall are consistent while other processes write to the databases at the same time. (This isn’t a proper use of git: with a 230+ MiB file changed daily, the repo grows quickly and git gc by default won’t work since it will run out of memory. Two git repack options fixed this: pack.threads=1 and pack.windowMemory=100m, resulting in 76 MiB repository for backup from 18 days.)

Disk encryption protects only systems powered off. It’s obviously worthless on my phone, since it wasn’t powered off for a year and half (longer than these are expected to work outside of the EU). At the same time, it vastly increases my risk of data loss: I remember passwords by typing them, forget by not typing. The solution is to not encrypt phone’s storage, turn all systems with disk encryption off for each night, so I type their passphrases daily, and keep the passphrases written.

This won’t help if I lose the phone itself or if its eMMC chip corrupts itself and destroys the files. For this I need regular backups of its whole /data partition. I should also schedule regularly moving various files from it to my other systems (like photos that I make using the phone and later store on my other computers, or reading progress for various ebooks).

I have much other data that I haven’t lost yet, even for a short time like 30 minutes of typing passphrases. I should get trivial data that I’m much more likely to independently lose in the same ways as my important files, so I can practice restoring it and make my backup policies reliable. A more certain way is to not delete information, storing all changes of important state in a DVCS-like way (with replication and separate backup), and actively use restored backups.

Free software with nonfree dependencies and the emulation argument

2015-03-15T15:12:08+00:00

There is a fallacy that if program X requires nonfree software, then program X is nonfree. (I discuss software that is free as in user’s freedom, not software with external costs. X is a placeholder variable, not the name of a concrete program, while there are relevant issues in X graphics drivers that interpret nonfree code from VGA ROMs and communicate with nonfree firmware loaded by their kernel modules.)

As Richard Stallman explained in his Free but Shackled essay, software being free and software being usable in fully free systems are different issues. His argument is that it can be determined if a program in isolation respects its user’s freedom (an assumption that I don’t believe to apply in unusual cases), while knowing if a program is shackled requires checking its changing set of dependencies and their dependencies (some of which are optional, some aren’t). He calls these programs free, so we have proven that they are free by an argument from authority.

Let’s choose a more amusing way of proving that a program requiring nonfree software can be free. A program is just a sequence of bits (while it’s not obvious if every finite sequence of bits is a program for our purposes). There are no intrinsic dependencies between bit sequences, so we need to reword our proposition in other terms, like uses of software. This leads to the following statement: if use Y of program X requires nonfree software, then program X is nonfree.

This new statement is logically unclear, since it doesn’t specify what quantifier we use for Y: do all possible uses of program X need to require nonfree software, or is one use sufficient for X to be nonfree?

What would for all uses Y of program X, program X requires nonfree software imply here? We introduce the emulation argument: for any program X, X can be used without running it with other software or hardware that it interacts with. (Maybe you don’t consider e.g. symbolic interpretation of a program useful, but there might be a user who does; free software is about uses that a user might consider useful, not only what the author or someone else considers useful.) This makes the left side of our implication false: every program can be used without separate nonfree software.

So our statement should be formalized as if there is a use Y of program X such that Y requires nonfree software, then program X is nonfree. But this leads to a much worse conclusion: since for every program there is a use that requires nonfree software, it would mean that every program is nonfree. The symbolic interpretation from the emulation argument can be done via a nonfree software, making it a use that requires a nonfree program (while it’s a different use than symbolic interpretation via a different program that is free). This contradicts the obvious fact that there are free programs. (The program fix f = f (fix f) is obviously free, while it’s too simple and unoriginal to be copyrighted. It serves an important theoretical purpose and has a very elegant implementation in lazy functional languages.)

In conclusion, we need to discuss freedom of a program without involving the freedom of its dependencies. There are other terms and definitions for important and insufficient criteria for whole system freedom (like for operating system distributions or for devices running free software).

Free software Flash replacements and the JavaScript trap

2014-12-13T14:55:18+00:00

One of the nonfree programs that make it hard for many people to use completely free software operating systems is Adobe Flash. There are several free software projects aiming to replace the Flash interpreter, one of them is an FSF high priority project. I don’t believe that developing such programs will significantly help people stop using nonfree software. (While hardware compatibility issues resulting from free drivers requiring nonfree firmware are well-known and probably more noticeable, they can be easily avoided by buying appropriate hardware, it’s not hard. There are social issues that make people use the same websites as their friends, but not the same computer hardware.)

While Flash has many uses, both as a Web browser plugin and for desktop applications, I will focus on its common use for video players on websites like YouTube.

Replacing Flash is hard

No free software implementation of SWF, the file format used by Flash, can currently support most such files used on the Web. gNewSense contributors mentioned both patents and incomplete specifications making this hard to do. Another issue is the Digital Restriction Management implemented in Flash. A sufficiently complete free implementation would probably violate anti-circumvention laws making DRM an effective restriction of our freedom.

The JavaScript trap

Even if we had a complete and free SWF implementation, it would interpret nonfree programs that websites publish. It is exactly the same problem as the JavaScript trap: using free software interpreters to run untrusted nonfree software from the Web. (I haven’t noticed this issue before reading the RMS’s essay on JavaScript and gNewSense’s page on SWF.)

Some sites like YouTube are moving to providing videos via the HTML5 video tag. It doesn’t solve this problem, since now nonfree JavaScript programs serve the same purpose as previously SWF. I think it might make writing free software replacements easier, due to free development and debugging tools available for JavaScript.

Why we need video downloader programs

Issues with specific video publishing sites are completely solved for their viewers by not running the code that the site provides (either SWF or JavaScript) and using a free software program to obtain the video. This can be done by youtube-dl, a command-line program; UnPlug, a browser extension; and many other programs. There are also extensions that display the video inline on the page without using its builtin player.

These tools support only specific sites, while very many are supported by youtube-dl despite its name. On other sites you can usually find the video URL by reading the source of the HTML page or the included JavaScript code. (It might be a nice fetish to have.) I don’t know what work is needed to use an unsupported site with a free SWF interpreter like Gnash.

Being able to download the video and save it on persistent storage (instead of downloading it just to display it in the player) is needed for at least several useful reasons: we cannot remix without downloading the video, we cannot protect against centralization and copyright censorship while accessing the works from a single centralized site and we cannot share it with our friends (or be a good friend to them) without having a copy. Even the very limited freedoms weakly protected by copyright law as fair use cannot be used without storing a copy of the work.

(While I highly disagree with completely rejecting JavaScript due to its usefulness in free Web applications, the arguments used against it clearly apply to SWF. Video downloader programs and browser extensions are software that we can write to replace nonfree software provided by websites.)

Flash animations

Before Web videos became popular, SWF was often used for vector animations. This might include them in the difficult to reason about area between software and non-functional cultural works, while there is a simple reason to consider it software: it has antifeatures. We need the freedom of free software for such works to make them respect their users.

JavaScript and HTML5 canvas are replacing this use of Flash too, so now nonfree programs using better tools control the animation.

Publishing your own works

If you write an interactive website, use JavaScript. Release your code as free software. If you make videos, release them on your site using a free software-friendly video format like WebM, or use Web applications like GNU MediaGoblin (possibly an instance run by your friend).

To prevent DRMed sites from using your videos to restrict their users, use a free culture license that disallows using ‘effective’ technical restrictions of the freedoms that it protects, like CC-BY-SA 4.0. (YouTube requires giving them a different license, don’t upload your work there.)

My email spam filtering and end-to-end encryption

2014-09-06T19:35:48+00:00

Big email providers use very complex spam filtering methods. Solutions used by Google require distributed real-time processing, access to to plain text of all messages. Their work is closely followed by spammers in an arms race, while it’s not usable for small servers and both sides benefit from reducing user’s privacy. In this article I describe how spam filtering works on my personal server: a solution optimized for low administration effort and not using message content. It involves using only existing known free software packages without much extra configuration beyond what’s needed to have a working mail server.

Email spam that I receive comes from three main sources: zombie computers in botnets, hijacked accounts and Polish businesses. Zombies are easy to block, since they do not comply with mail standards in easily detectable ways. Hijacked accounts are now rare (partially due to the hard work of Google explained in the linked mail; it would be easier if the two Yahoo users who don’t spam moved to other providers).

Spam from Polish companies is my main issue, since they use properly configured servers and their own IP addresses. There is a law that allows sending uninformative spam to everyone, while informative spam can be sent only to companies. They do not check if the recipient has a company.

I use the following methods to filter these kinds of spam on my server using Postfix MTA:

postscreen: It filters much zombie spam by adding a several second delay and checking if the client waits before sending data and doing several other protocol correctness checks.
Sender Policy Framework: Since zombie spammers do not use their own domains (these would be blacklisted by Google), they use fake sender domains which are often real. SPF records specify which servers are authorized to send mails for that domain, so zombie spam using it is blocked. Not enough domains use it. SPF would block some good mails if I used email forwarders without SRS, I don’t, since I have no use for forwarders. (The SPF validator implementation that I use is pypolicyd-spf.)
postgrey: It greylists all mail not from known trusted servers that haven’t successfully delivered a mail recently; i.e. it returns a temporary error code and allows the mail to be sent again after several minutes (proper servers do this; email servers work well without 100% uptime). This leads to delays when getting mails from new servers, annoying for registration emails from shops. It blocks nearly all remaining zombie spam.
static IP address blacklist: For professional Polish spam businesses. For one provider, I have to blacklist entire IP ranges. This solution wouldn’t work for a server with more users.

I don’t use these common methods:

checking reverse DNS records: it fails on real servers and would block much self-hosted servers
using external RBLs: they are bad and block self-hosted mail
DKIM: I don’t find enough value in it to find how to configure it; I think it might be useful for more complex filtering that uses multiple factors to decide if a message is spam and if the provider can motivate administrators of other servers to configure extra things (Google can)
checking message content: it’s complex, has false positives, causes an arms race, needs access to message’s plain text content (preventing end-to-end security or delegating spam filtering to the client); if manual filtering of probable spam messages is needed, the method is at most as good as not doing any filtering.

I tried using ‘unsubscribe’ links in professional spam. They don’t work: they often fail (with e.g. page not found errors), are missing, or are mail addresses (I don’t mail spammers). If they work, they affect only some mails from the provider (only the mail that link was from?): they still send other mails. The IP address blacklist is more effective. I haven’t tried contacting server providers of spam businesses using VPSes or dedicated servers with terms of service prohibiting spam. I don’t know if they have a saner definition of spam than the law.

I would like it if all spammers moved to sending only OpenPGP-encrypted mails (they can easily get my public key from a public keyserver or from my Web site): it wouldn’t affect my spam filtering and it would increase their resource usage.

In this week, I received 11 spam messages (not counting ones from mailing lists), 5 are in English, probably from zombies, 6 are from real Polish businesses with IP addresses that I haven’t blacklisted yet. I don’t count how many were blocked. I consider this good enough to not research better spam filtering methods now.

I don’t offer a solution to the problem of spam: it’s difficult, has economic, legal, technical and educational aspects; what I use is sufficient for my needs and has no problems with securing message texts. I do not know how spam filtering would work if all users moved to their own servers, maybe some post-email protocols with proof-of-work schemes would solve these issues while not supporting sending emails from phones to Google servers.

LibrePlanet 2014 stream recordings

2014-03-23T22:41:34+00:00

The LibrePlanet 2014 conference talks end today. Since there are no official recordings yet, several users published copies of fragments of the live streams that they have downloaded. This page lists the fragments that I know.

See the official program for a complete list of talks. Live stream fragments of these talks are available:

Opening Keynote: video parts 1, 2, 3; with slides
Fighting surveillance with free software: video parts 1, 2, 3, 4; with slides
Opus, Daala, and free codec updates: video parts 1, 2; with slides
Respects Your Freedom hardware certification: video parts 1, 2
An overview of OpenPGP: video; with slides
Get started contributing to MediaWiki: video
Considering the future of copyleft: how will the next generation perceive GPL?: video parts 1, 2, 3, 4; with slides
Your Web apps should talk not just in English, but in español, Kiswahili, 廣州話, and অসমীয়া too: video; with slides
Celebrating one decade of Trisquel GNU/Linux: video parts 1, 2, 3; with slides
Geek knowing: from FAQ to feminism 101: video parts 1, 2, 3; with slides
The creeping techno-surveillance state: how can we fight back?: video; with slides
No more mouse: saving elementary education: video parts 1, 2, 3, 4, 5; with slides
Beyond the women in tech talk: video parts 1, 2, 3, 4, 5, 6; with slides
Mapping for social justice: video parts 1, 2, 3; with slides
Nurturing non-coders: video parts 1, 2; with slides
Current issues in freedom: patents, surveillance, etc., Free Software Awards: video parts 1, 2
Free software for freedom, surveillance and you: video
Free your JavaScript: video; with slides
What does this program do? Reproducible builds, transparency, and freedom: video parts 1, 2, 3; with slides
Building an open digital archive in India: knowledge, access and other issues: video parts 1, 2
1984+30: GNU speech to defeat e-newspeak: video; with slides
State of the goblin: video parts 1, 2, 3; with slides
Distributed free-cultural production and the future of creative economy; video; with slides
Free software activism: a European perspective and experience: video; with slides
Updating Mailman’s UI: video; with slides
Adventures in hackademia: leveraging free software in the classroom: video
Tracking changes: activists using free software across movements: video parts 1, 2, 3, 4; with slides
Choosing between freedom and security, Diversity outreach: video; with slides
Promoting free software adoption (and creation) in the public sector: video parts 1, 2
Lessons in tech activism: video; with slides
IT cooperation: accessible, free, and open: video; with slides
Free software and open science: video
We can’t all be cyborg lawyers: how messaging may be our most important obstacle: video; with slides

Since these are only fragments of live stream downloaded using a faulty script, many are incomplete and some are impossible to watch. I haven’t found slides files for other talks; if the recording shows slides, I’ve listed it above.

I don’t know what copyright licenses apply to the recordings: the conference site lists Creative Commons Attribution-ShareAlike 3.0 US for videos, while I don’t believe some speakers would agree to use a free license for their recordings, so I don’t trust this information.

Lessons learned from using `wget` to download the streams

I believed I knew how to download files via HTTP using the wget program. I was mistaken:

wget -c corrupts the file instead of resuming it
retry on timeout corrupts the file

I used wget in a loop with some delay between retries to download the stream to another file after a part ended. Since the streams are live and different at each time, the commonly used logic for resuming overwrote earlier parts of the streams.

Why there is no standard and supported way to make HTTP servers inform the client that each download gets a different document that shouldn’t be downloaded to the same file?

Watching the talks

I used mpv to watch the downloaded files. The media player program that I usually use, VLC, did not handle them properly.

I hope the FSF will release real and complete recordings on their GNU MediaGoblin instance, so these stream copies won’t be needed.

DRM in free software

2014-03-12T19:31:47+00:00

Free software has less antifeatures than proprietary software and users can remove them. While a well-known distro vendor includes spyware, such bugs usually get fixed. Despite these, some well-known free programs include antifeatures restricting uses or modification of data that these programs should access or edit.

These antifeatures are called DRM which is ‘digital restrictions management’. It is unrelated to the Direct Rendering Manager which despite using the same acronym has no freedom issues other requiring nonfree microcode for Radeon graphics chips. Traditional bugs that make programs mishandle data or crash when using specific files are also different, developers fix them and don’t consider them intentional.

PDF restrictions: Okular, pdftk

The PDF document format includes metadata flags which readers use to determine if the user is allowed to e.g. print the file or copy its text. Okular obeys this restriction by default, while it has an option to respect what the users does.

The main argument for keeping that optional DRM is that the PDF specification requires it and users could use that ‘feature’.

The PDF manipulation program, pdftk, obeys such restrictions with no option to remove them without changing its source. Fortunately Debian fixed this bug in their packages, so it can be used on recent Debian-based systems to modify or fix restricted PDFs.

What if you get a restricted PDF and need to extract its text? Use pdftk input.pdf output output.pdf on a Debian-based system to drop this restriction, or just use the existing file in Okular with disabled DRM or another free PDF reader.

Debian-patched pdftk prints the following warning:

WARNING: The creator of the input PDF:
   drmed.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.

I think it’s an acceptable way to handle such restrictions. Many uses of the restricted features don’t involve violating copyright.

I made that restricted file earlier using pdftk text.pdf output drmed.pdf owner_pw hunter2 allow. It did not warn me that that DRM is bad or that it can be very easily ignored or removed.

The PDF format supports also document encryption with user passwords. It’s not DRM, since it prevents reading the document instead of restricting it in software: can it be used to protect user’s privacy? (I don’t know how secure that encryption is, I would use OpenPGP instead if I had to send an encrypted document to a friend.)

LibreOffice spreadsheet ‘protection’

OpenDocument supports sheet and cell ‘protection’. It allows the user to read the spreadsheet (except for hidden sheets), but not view formulas, copy or edit their data.

This is implemented by adding metadata that tells programs to not allow editing the cells. The document contains an element with hashed password ‘needed’ to unprotect the sheet. It’s easy to change that password or remove protection using a text editor and a ZIP program to access XML files stored in the document.

LibreOffice Calc did not warn me that the added ‘protection’ is useless against users who can use a text editor. It did not warn me that this kind of restrictions is unfriendly and harmful regardless of them being effective.

The reason why I learned about this antifeature is that I once received a spreadsheet document and wanted to learn how its formulas worked. I converted it to ODF using LibreOffice and used jar and sed to change the ‘protection’ password. I learned more than I expected to from that document.

While all complete OpenDocument implementations have this problem, I name LibreOffice specifically here since I use it and recommend for other reasons. This antifeature probably comes from OpenOffice.org or StarOffice which cloned it and other bugs from other proprietary office software.

Like PDF, OpenDocument supports encryption which is unrelated to the discussed restriction.

FontForge

TrueType fonts have metadata flags specifying if a font editor should allow users to modify or embed the font. FontForge supports modifying that metadata and warns the user when opening a font containing it.

The setting responsible for this is ‘Element’ → ‘Font Info’ → ‘OS/2’ → ‘Embeddable’, opening a TrueType font with that value set to ‘Never Embed/No Editing’ shows a dialog box with the following message:

This font is marked with an FSType of 2 (Restricted License). That means it is not editable without the permission of the legal owner.

Do you have such permission?

Accepting it, the program allows me to modify my font and change that setting. I haven’t felt mislead into considering it an effective restriction, unlike when using LibreOffice or pdftk.

DRM and software freedom

All generic DRM issues apply here; I think there are more specific problems when it is used in works edited using free software:

these restrictions make studying or modifying the work harder; LibreOffice and unpatched pdftk don’t suggest a way of solving this
programs offering options to restrict works made using them usually mislead users into believing that that snake oil is secure
it legitimizes preventing users from studying or modifying the digital works that they receive

While all cultural works should be free, these issues apply to functional works like fonts, spreadsheets (non-hacker’s programs), research articles or documentation, for which the freedoms of free software can be most clearly applied.

Solution

Free software that we develop should have no antifeatures. If we find a free program with DRM, we should fix it, like Debian fixed pdftk. Software distributions should have explicit policies against DRM; the No Malware section of the Free System Distribution Guidelines would be appropriate if it was implemented and more widely promoted.

Free software is better: skilled users can fix it and share the changes that allow users to control their own computers.

Buying a freedom-respecting USB wifi card for a Lenovo laptop

2014-02-18T21:14:04+00:00

I received a second-hand Lenovo ThinkPad R400. It’s compatible with Trisquel GNU/Linux 6.0.1 except for its Intel wifi card that needs nonfree firmware (while its fingerprint reader needs an updated userspace driver). I decided to buy a compatible card, since it’s more practical than getting a longer Ethernet cable.

Replacing it with another internal miniPCIe card would be difficult: Lenovo laptops have an antifeature preventing the use of internal wireless cards from other vendors (my other article discussed this and other freedom issues in laptops). I don’t know if an available card would work in this ThinkPad model (all cards that it is shipped with need nonfree firmware) nor how much effort I would need to work around this restriction. These cards are also much harder to buy here than external USB ones. So I decided to buy one in a local shop with an online catalog.

Choosing chipset and device

There are two important aspects of a wifi card: chipset and protocol support. (Appearance and size too is important for external cards, while it is a clearly marketed feature that all shops and vendors show.) I looked for 802.11bgn devices working at 2.4 GHz channels, since 802.11n brings better coverage and performance. Dualband 2.4 and 5.0 GHz devices are harder to find here and much more expensive, while they could be more useful at home with an appropriate AP.

There are practically only two modern chipsets that meets my criteria: Atheros AR9271 and AR7010 using the ath9k_htc driver with free firmware. There are other chipsets used in USB devices that work without nonfree firmware using rtl8187 or carl9170 drivers, while the devices using them are older and not widely available now, and they have technical issues (rtl8187 is for 802.11bg only, carl9170 is for an older version of 802.11n). (I use the word ‘chipset’ for both a single chip and a group of chips working together, this not completely correct usage is common here.)

Which available device has this chipset? An easy solution is to buy one from ThinkPenguin or Tehnoetic, since they offer only devices compatible with free software and they supported freeing AR9271 firmware for their FSF-endorsed cards. (Their single-band cards have higher prices than ones from other vendors, while it might be a worthy cost for supporting development of free software and certainty of the device being supported.) Partially due to lack of local ThinkPenguin shops (leading to much longer and more expensive delivery), I decided to look for a different device in a local shop catalog.

There was no information on what chipsets these devices have, except for some with non-Atheros chipsets that won’t work without nonfree firmware. There was also no information on which versions these devices were: vendors often put completely different chipsets in different versions of the device without changing its name nor making it visible in marketing data.

WikiDevi has information on many devices with versions and chipsets. It allowed me to quickly filter out many useless devices. I’m not sure if this information is complete: vendors release new versions with different chipsets. All devices that I checked were listed and the one that I bought had no other version.

There is another way to learn this: vendors offer nonfree drivers for nonfree operating systems, separately for each device version. These drivers often are ZIP archives that can be browsed without running the included software and contain a .inf file which names the chipset used. If there is any device version that has a driver for an unsupported chipset, don’t buy such a device unless you know it is of a working version. If all versions have listed the AR9271 USB vendor and product identifiers (the modinfo ath9k_htc command lists aliases with identifiers of devices that have a supported chipset), buy it.

Using these methods I found a supported device: TP-Link TL-WN722N that worked out of the box with Trisquel 6.0.1 using the 3.11.0-15-generic kernel. (It needed downloading external free firmware and updating kernel on older 6.0 installations.)

What’s needed to use an internal card with free software?

While the external card works well, having a supported internal one would be better. Why it doesn’t work out of the box and how can this issue be solved?

Intel doesn’t publish sources of their wifi firmware nor documentation needed to understand or reimplement it. They falsely claim that it would be illegal due to FCC software-defined radio regulations, while this doesn’t prevent Atheros from having free firmware (Software Freedom Law Center published an opinion supporting the use of free software with such devices). (It’s not the only area where Intel releases blobs where other companies provide free source code and documentation: they provide sourceless blobs for memory initialization of chipsets used in coreboot and claim that no company provides documentation or source allowing making a free implementation, while companies like AMD and Intel release similar free code for some of their chipsets.)

Lenovo uses a similar argument for the whitelist antifeature, while laptops from some other vendors work with any such cards with probably no legal issues. There is a different argument that might be true: they want users to buy cards with their branding. (If only these were available here…)

What real and complete solutions are possible? There are non-technical solutions: we could convince Intel to free their firmware or release documentation allowing a free reimplementation and make removing the whitelist more profitable for Lenovo than their monopoly on replacement cards. How can this be done?

Reverse engineering by free software hackers gave us free firmware for NVIDIA GPUs and free drivers for many devices, including 802.11bg Atheros chipsets (802.11n devices have a free driver developed by Atheros, 802.11ac have nonfree firmware). Can Intel wifi firmware be reverse engineered and replaced? This is probably very hard, since there are multiple big blobs (more than 300 KiB of binary sent to the device) in an unknown instruction set. There is no project doing it now. Maybe there are other wifi chipsets in devices shipped with Lenovo laptops that have easier to replace firmware, while there are no projects for them yet.

Since it’s hard to make the default card work, can we fix the boot firmware to allow using a different card? The best way would be to port coreboot, the free boot firmware, to the laptop. It is hard, practically impossible if the chipset isn’t supported, easier if it’s similar to a supported device. Even if there is coreboot support for the laptop, it is hard and risky for users to flash and much work needs to be redone for each different laptop model. It’s a good solution if vendors sell many laptops of the same model with preinstalled coreboot and a free wifi card, like Gluglug does for their FSF-endorsed X60.

A simpler and more widely discussed approach is to modify the nonfree boot firmware to change or remove the whitelist. This way has similar flashing risks as coreboot installation and requires using or adapting nonfree software, making results less useful for recommending to others.

It might be also possible to change how a freedom-compatible card identifies itself, to make it accepted by unmodified boot firmware. This requires using a different laptop without the whitelist and might need changing the driver to handle the renamed card. While there are free tools for reading and writing EEPROM of Atheros cards that I have, none of them is able to write it and there is no documentation of how these identifiers are stored there.

Summary

Only USB wifi cards work everywhere. We should recommend buying freedom-friendly ones: there are many ways to do it with varied difficulty and reliability. While we should promote laptop vendors selling hardware without such restrictions, it is important to support freeing systems that users already have.

Thanks to icarious for suggesting several workarounds for the Lenovo whitelist. Thanks to FreeNode users AimHere, Calinou, jxself and oitofelix for advice and encouragement related to recommending compatible USB cards.

Missing source code for non-software works in free GNU/Linux distributions

2013-08-04T18:33:03+00:00

Most software cannot be edited without a source, making source availability necessary for software freedom. Free GNU/Linux distributions have an explicit requirement to provide sources of included software. Despite this, they include works without source. I do believe this is practically acceptable, while it restricts potential uses of the software and limits our ability to reason about software freedom.

The source

Section 1 of the GNU General Public License, version 3, defines the source code of a work as ‘the preferred form of the work for making modifications to it’. This definition is also used outside of the GPL.

However, only the author of the program can know if the given text is the source. C ‘source’ code is usually the source of the program compiled from it, while it isn’t if it was generated from a Bison parser. (Free software projects sometimes do accidentally omit the source for such files.)

Let’s simplify the issue: a source is a form of the work that a skilled user can reasonably modify. Some works, usually not C programs, are distributed in modifiable forms that might be compiled from forms that the author prefers more for editing. (Some generated parsers do get modified, making GPL compliance for them slightly harder.)

(For GPL compliance there is a more important issue of the corresponding source for a non-source work which is certainly harder than deciding if a work is just the source. It is beyond the scope of this essay.)

I believe these issues are trivial in case of C programs like printer drivers that inspired the free software philosophy and rules. For other works, deciding if a form is the source is probably impossible if the software was distributed.

Fonts

Fonts are ‘information for practical use’. They describe the shapes and metrics of letters and symbols, editing them is useful to support minority languages or special symbols needed in computer science. Now most fonts are vector or outline fonts in formats like TrueType. Bitmap fonts have different practical and legal issues.

Fonts legally are considered programs, while their description of glyph shapes just lists points and curves connecting them with no features expected from every programming language. Editors like FontForge can edit TrueType fonts, while it has a different native format with lossy conversion to TrueType which is preferred for editing.

Hinting in TrueType contains ‘real’ programs adapting these shapes to low resolution grids and making them legible on screen. These programs are distributed in a Turing-complete assembly-like language interpreted by a stack-based virtual machine. There are tools like Xgridfit which can compile a higher-level language into these programs. The other popular font formats, PostScript Type 1 and its derivatives, use high-level ‘hints’ like positions of stems and standard heights that the rasterized uses in unspecified ways to grid-fit the glyph.

While there is some benefit of editing the source instead of TrueType files, this is much different for meta-fonts. The Computer Modern project developed by Donald E. Knuth for use with TeX consists of programs using 62 parameters to generate 96 fonts. Modern technologies require drawing every font separately, while the same program describes e.g. a Roman letter for all fonts that contain it and doesn’t need many changes for new fonts. Making a separate set of fonts in a much different style for a single book is possible with meta-fonts, or gradually changing between two different fonts in a single article. (I have made a narrow sans-serif monospace style for a Computer Modern derivative in several hours. It is not published due to a licensing issue.)

However, there are nearly no other uses of meta-fonts as effective as this one. MetaFont, the program that interprets Computer Modern, generates device-specific bitmaps with no Unicode support. All programs that compile meta-fonts to outline font formats do it either by tracing bitmaps produced by MetaFont (resulting in big and unoptimized fonts) or generating outlines directly without support for important features used in Computer Modern. Recent meta-font projects rebuilding their sources from generated outline fonts or not publishing sources do not support this being a successful style today.

Hyphenation patterns

While some languages have reliable rules for hy-phen-a-tion, in English this was done using dictionaries of hyphenated words. This approach has significant problems that were solved by Franklin Liang’s hyphenation algorithm used in TeX, generating rule-like hyphenation patterns from a dictionary. 4447 patterns generated from a non-public dictionary allow TeX to recognize 89.3% of hyphens in the dictionary words.

The patterns are subwords with multiple levels of hyphens to be added or removed. The word hyphenation is hyphenated using hy3ph, he2n, hena4 and six other patterns, resulting in hy-phen-ation. (Not all hyphens are found, this will be fixed by future dictionaries using TeX to derive their hyphens.)

The same algorithm is used for multiple other languages with different patterns. They are usually generated from dictionaries restricted by copyright and not available to the users. Some languages have patterns distributed with the source dictionary. (I believe patterns could be easily written by hand for a language having reliable hyphenation rules depending only on the characters in words, although I haven’t seen any example of this.)

The patterns can be and are edited, while the source dictionaries can be more useful for development of other hyphenation algorithms. This makes them ‘a source’, but not ‘the source’.

(Technically, TeX doesn’t use the patterns directly. INITeX loads macro definitions, hyphenation patterns and font metrics, and saves its memory into a format: a very build-specific file for fast loading by VIRTeX which is normally used to build documents, representing patterns in a difficult to edit packed trie. VIRTeX does not support loading patterns since their compilation needs extra memory and code, now the same program is used for both purposes. Many other macro processors and Lisp implementations have a similar feature under a different name.)

Game data

Video games provide a bigger source of binary data. Many contain bitmaps or animations made using 3D rendering software from unpublished sources. Some games like Flight of the Amazon Queen are published as a single binary with no source and no tools for editing it. (A Trisquel users forum thread about this game originally motivated me to write this essay.)

This game has another interesting issue: a license that forbids selling it alone and allows selling it in larger software distributions. Well-known free licenses for fonts like the SIL Open Font License have the same restriction. It’s ‘useless’ since distributing the work with a Hello World program is allowed and this makes it a free software license.

Lack of source nor tools to edit it is more interesting. The Debian package includes an explanation of its compatibility with the DFSG. The binary is the ‘the preferred form for modification’ and the tools for its editing being lost made modifications equally hard for both Debian users and authors of the game. This is consistent with the source requirement being made to prevent authors from having a monopoly over their works (this explanation looks equivalent to the user’s freedom argument).

In GNU/Linux distributions endorsed by the FSF this is not an issue. Game data is considered non-functional and the only permission required is to distribute unmodified copies. (Debian excludes from the main repository games that are included in these distributions, while they exclude games that other distributions include. The first common issue is lack of data source or modification permission, the second is a restriction of commercial distribution.)

Documentation

Documentation of free software should be free, so it can be shared with the software and updated for modified versions. Most documentation is distributed as HTML or PDF files which are usually generated from various other markup languages.

Not all such documentation has a published source and sometimes software source is distributed with the binary only. (Sourceless PDFs often use nonfree fonts too.)

HTML can be edited and often is the source, while in other cases it is compiled from sources which preserve more semantic information about the document and have better printing support. For this reason we should not consider it the source if the author has a source from which it is compiled. Can we know this?

While the most popular free software licenses require providing the source with binaries, this isn’t true for most documentation licenses. No Creative Commons license protects the practical freedom to modify due to its focus on non-textual works. GNU FDL does and unlike software licenses it also requires the source to be in a free format.

The program-data dualism

Most of the above cases suggest that source code access is needed only for programs, not for data. This isn’t true and is not strict enough to be an useful criterion.

TrueType fonts are both programs and data. The PostScript page description language and typesetting systems based on TeX use Turing-complete programming languages for formatting documents which sometimes do contain nontrivial programs. Scripts describing events (and dialogue) in games are programs.

There is another difference between these works and compiled C programs: they work on multiple architectures. This is not a sufficient criterion for requiring sources, since we do not consider Java programs distributed as class files without source free, while they run on all architectures supported by Java virtual machines. Binaries being architecture-specific make distribution package builds for unpopular architectures like MIPS a more useful way of finding missing sources.

Version control

Most recent free software projects distribute the source in two ways: in distributed version control system repositories and as archives of a specific versions: tarballs which often include generated files that require ‘special’ tools to build that not all Unix systems had.

For development, source is obtained from the version control system, since it has the whole project history explaining why the changes were made. For fulfillment of the source distribution requirements, the tarball is used. Does this mean that the tarball isn’t the ‘the preferred form of the work for making modifications to it’?

Conclusions

We should provide the sources of the works that we make, since only in this case we know that it is the source. The source should be in a public and distributed version control system and include tools to build all non-source files of the work.

Verifying if software written by others has a source is harder. If you can edit it, then maybe it’s free and it’s a source. Don’t distribute software that you don’t use, since you don’t know if it respects the freedom of its users.

How can we improve hardware support in free GNU/Linux distributions?

2013-06-09T23:27:51+00:00

Answering questions on free distro IRC channels I find two main reasons for people to not use these distros: hardware depending on nonfree software and specific nonfree applications that they ‘need’. (There are other reasons, like other free distros being better for the purposes of the user or having newer software. These should become more common.) This article discusses some potential ways to solve the first problem. It is inspired by the LibrePlanet 2013 talk of Richard Stallman discussing also other important current issues like mobile operating systems, DRM and privacy on mobile phones.

Most computers need some nonfree software for some functionality. My previous article discussed three common kinds of such software: boot firmware, graphics drivers and wifi drivers. Here we aim to run an operating system that doesn’t ship nonfree device firmware or drivers while nonfree boot firmware might be used. This leaves most issues with graphics other than VGA ROMs (which are shipped on the card and on Intel systems already did their work, while drivers for other cards use them) and all problems with wifi drivers and firmware.

One of the ways to improve the situation mentioned in that article is to support vendors that do contribute to free drivers and public specifications for their hardware. While this does help, there are also technical solutions with known good results.

Reverse engineering

Since vendors usually don’t support free drivers nor specifications for their devices, we need to make ours via reverse engineering. There are several projects making free drivers with great progress using specifications obtained from reverse engineering, Nouveau might be one of the most well-known (it’s the only project not supported by the vendor that replaced some of nonfree firmware removed in Linux-libre). More recently Lima and other projects for most popular mobile GPUs were started.

I know no general documentation of reverse engineering as used in these projects. This might be a useful area to improve: there are many people interested in helping such projects or starting new ones for other devices.

The FSF reverse engineering project list and the repositories of nonfree firmware included with kernels show which devices might need such projects.

Porting software to completely free non-x86 systems

Since all x86 processors use nonfree microcode usually provided by boot firmware and most have vendors not supporting free drivers for parts of their systems, supporting other CPU architectures might help support free boot firmware in longer term. Of such devices only Lemote Yeeloong laptops with MIPS-compatible CPUs run completely free boot firmware and operating systems.

Not all important free software works on MIPS and many optimizations are missing: assembly using SIMD instructions for video decoding and JIT compilers might be the most important ones. More software is missing on faster 64-bit ABIs.

Removing software dependency on hardware-accelerated graphics

Graphics cards usually work without their nonfree drivers with much worse performance (since all rendering is done on the CPU) and sometimes worse resolution support which is usually solved in future driver releases. This makes the nonfree drivers unnecessary unless using compositing desktop environments or games that need 3D acceleration. (3D modeling or design are less common uses for such hardware.)

While replacing the drivers is difficult, replacing the software depending on them might be easier. Trisquel’s default desktop environment does it well: GNOME Fallback is used instead of GNOME Shell to support systems with AMD or NVIDIA graphics.

We can also recommend games using 2D graphics, like Battle for Wesnoth or text roguelikes.

There might be more opportunities for such improvements on mobile systems: while GNU/Linux traditionally worked without OpenGL hardware, Android normally depends on it, with nearly no phone supporting free drivers for it. Replicant works without accelerated graphics on all supported phones while some useful free applications for it need accelerated graphics.

Documenting compatibility of used hardware

Many systems will have parts not working with free software or will ‘accidentally’ work. Documenting this will make buying working hardware easier for other users. H-node contains reports of hardware compatibility with free distros that are used for this purpose. Improving h-node itself would also help, by e.g. making device submission easier.

This could also support free driver projects by showing which devices are popular and which ones need possibly easy improvements in distros.

Laptops and free software

2013-06-02T23:25:12+00:00

Users of free GNU/Linux distributions often choose computers that can be optimally used without installing nonfree software. This article is based on many discussions of such hardware and problems making this a nontrivial issue.

There are no free laptops: all use CPUs that the user cannot study, manufacture nor modify and many complex parts with this issue. (There are free CPUs like OpenSPARC, none of them are used in notebook computers.) For this reason we use less strict criteria, like for the FSF hardware endorsement which currently only a small number of modified laptops fulfil: Gluglug X60 and other Intel i945 and gm45 laptops running libreboot.

In this article the word free means libre or freedom-respecting software. (I don’t discuss gratis laptops, most probably they don’t respect user’s freedom.) Firmware refers to software running on other CPUs or boot firmware.

I don’t know ARM well, so this article focuses on machines of the x86 and MIPS architectures (no free distro supports other architectures); ARM devices nearly always need nonfree graphics drivers and usually have nonfree boot firmware which is often cryptographically restricted from being modified. The FSF has a list of some computers with these issues.

Boot firmware

Most modern computers and nearly all laptops run nonfree software before starting the operating system. On x86 it’s usually called BIOS, while now it often has a new interface called UEFI. Although UEFI is often criticized for difficulties with installing free operating systems, its implementations have the same important problem as older BIOSes: it is nonfree software and users usually cannot replace it with free solutions.

There are three projects making free boot firmware:

coreboot, often equated with free software boot firmware while for some systems it includes nonfree software, CPU microcode and needs other nonfree software extracted from the original boot firmware: VGA BIOS for integrated cards.
PMON2000 for Loongson devices (but all supported systems except for the YeeLoong using Loongson 2F have nonfree VGA BIOS). For me the most interesting feature is it working as bootloader completely replacing GRUB for my uses: it loads kernels from ext2 or FAT filesystems and supports network booting (via TFTP).
GRUB: while it’s primarily a powerful and portable bootloader, it can replace PMON2000 on YeeLoong and FuLoong based on Loongson 2F (freeing the FuLoong from its default nonfree VGA BIOS).

There are two main issues with mostly free boot firmware: CPU microcode and VGA ROM which I explain in the following sections. Recent systems need much more blobs.

Boot firmware is more difficult to install than operating systems: chips are often soldered, flashing custom code might brick the machine; hardware modifications are used for some boot firmware development. (It’s easier on desktop machines with more motherboards having socketed boot firmware chips.)

Several laptops support completely free boot firmware: Thinkpad X60/X60s/X60t, several Thinkpad T60 models and Lemote YeeLoong (not the Loongson 3A laptop called YeeLoong 8133). There is a fully free binary distribution of coreboot for the Lenovo laptops: libreboot, maintained and used by Gluglug.

Chromebooks are often marketed as devices running coreboot on recent Intel systems. This is true, while they need nonfree software included in the boot firmware, some of which is cryptographically restricted from being replaced by users. Samsung Exynos Chromebooks don’t have Intel ME, while they use a signed and encrypted first stage bootloader and have integrated wi-fi requiring nonfree firmware. (The Acer Chromebook 13 with NVIDIA Tegra K1 is most hopeful, while its soldered wi-fi requires nonfree firmware and limited non-replaceable storage and connectivity options would make it an unusable laptop for me. I don’t know what other nonfree software it needs.)

CPU microcode

CPUs are complex and have bugs, most x86 ones have microcode updates fixing some of these. The hardware includes microcode, boot firmware has updates for it and operating systems might include them. All such microcode is nonfree.

To have free boot firmware, we need to use the original microcode from the CPU chip, while we don’t know which Intel or AMD processors work without updates provided with boot firmware.

MIPS and ARM CPUs usually have no microcode, making support for completely free boot firmware easier. This has a potential disadvantage in having more hardware bugs (userspace programs can hang Loongson 2F CPUs; there are assembler fixes making it unprobable, while no CPU fix is possible without replacing hardware).

Some Intel CPUs have surveillance antifeatures implemented in hardware or boot firmware.

These issues make only a non-x86 system a long-term solution for the boot firmware freedom issue.

Graphics

Only Intel graphics supports 3D acceleration without nonfree software provided by the operating system (i.e. working on free distros).

Both AMD and NVIDIA graphics chips need nonfree VGA BIOS when using free drivers, so no completely free boot firmware supports these systems. A free replacement is used in coreboot on some systems with Intel graphics; this is made easier by the GNU/Linux graphics driver using the BIOS only for data, not code (it previously worked on systems with no VGA BIOS at all), and Intel releasing more documentation of its graphics chipsets.

I personally believe 3D acceleration is not important. Trisquel works well without it and I think this is a reason why there are incorrect reports on h-node suggesting it working on AMD systems. Vendors disagree, Rhombus Tech plans separate CPU cards for the FSF endorsement due to this issue.

The choice between free 3D acceleration and free boot firmware was made harder by AMD’s partial support for coreboot with their chipsets (while their graphics cannot be used without nonfree software) and Intel’s anti-competitive behaviour. AMD helped solve this dilemma by requiring nonfree software for their newer CPUs.

The AMD free GPU drivers need nonfree software, they are literally 95% free except for a necessary part without which they don’t work resulting in unaccelerated graphics or black screen and kernel panics.

Nouveau, the reverse engineered free driver for NVIDIA GPUs, supports 3D acceleration for many older devices, while too new or too old ones are not well-supported. It doesn’t solve the problem of requiring the nonfree VGA BIOS. Despite this it’s an important and effective project. All other modern GPUs that I know about have no vendor-supported free drivers, many have reverse engineering projects.

Wi-fi

Since most currently available systems have nonfree boot firmware that normally isn’t replaced by users, it’s widely accepted. Another common issue is lack of free wi-fi drivers or firmware used by them.

Nearly only Atheros cards work without nonfree software. On GNU/Linux they use the ath5k, ath9k, ath9k_htc and carl9170 drivers, while there are other drivers for less popular cards needing nonfree firmware. Their devices supporting the newer 802.11ac standard are different: their atk10k driver requires nonfree firmware, like all 802.11ac chipsets. (Many Atheros cards integrated in laptops support also Bluetooth that requires nonfree firmware.)

Atheros is the only company that used to free wi-fi firmware: the carl9170 and ath9k_htc drivers use firmware with free source code available.

USB 802.11g cards using the Realtek 8187B chipset work without nonfree firmware while most other Realtek cards don’t.

For Broadcom cards there is reverse-engineered firmware for older cards while Broadcom now makes free drivers they still require nonfree firmware.

There are two problems with replacing internal wi-fi cards in laptops: boot firmware whitelists in devices of IBM, Lenovo, HP, Dell, Toshiba and possibly other vendors (allowing using the machine only with wi-fi cards of the same vendor, usually only ones requiring nonfree firmware) and multiple card format standards. (Some laptops have irreplaceable internal wi-fi.)

USB cards are an often recommended solution for these problems, since many use the Atheros chipset with recently freed firmware (it needs a recent Linux-libre kernel and the separately available firmware); ThinkPenguin offers the card for which the firmware was freed, Tehnoetic sells them in Europe. Buying such devices from other vendors is not reliable: they usually don’t show what chipset is used and change it without changing the model name. This is partially solved by users reporting compatibility of specific versions, while not all vendors show the version number before the device is bought. I’ve written a separate article discussing some tricks for finding compatible cards and the whitelist antifeature.

Not releasing free drivers nor firmware is bad for users and vendors. I know several arguments used by vendors for nonfree firmware: FCC rules that the Software Freedom Law Center considers to accept free firmware (the release of Atheros firmware source clearly shows that at least some hardware designs can have free firmware); complex algorithms that can be used in devices of other vendors; and licensed third party code that they cannot release.

Embedded controller

Even if a laptop has free boot firmware and runs a completely free operating system, it still probably has nonfree software running on other processors. The EC firmware is an important case of such software.

An embedded controller is a microcontroller running software for power management and other laptop-specific functionality. Its firmware is nearly always nonfree, while there is a project for its replacement on OLPC devices. Some Chromebooks have free EC firmware, while their other blobs might have bigger issues.

Nonfree EC firmware has three practical problems: it has bugs (like all software), it’s often undocumented (making support in free boot firmware and drivers harder: this affects power management and hardware monitoring) and makes replacing boot firmware harder.

Since it normally isn’t changed by users and vendors don’t recommend updating it as often as boot firmware, it’s sometimes considered as hardware as the chips that it uses.

There is a similar issue with hard disk and SSD firmware: it’s a big and complex nonfree program using patented algorithms with significant performance, reliability and privacy implications. Since users rarely change it, it’s treated as a part of hardware, not software.

How can this be improved

We should support vendors of mostly free solutions who offer compatible devices and support development of completely free drivers, by e.g. buying 802.11n Atheros wi-fi cards or ThinkPenguin products.

We can also buy used machines, so vendors won’t get money for new non-user-friendly ones. (It’s also better for the environment, like not buying computers that are not needed.) There are social problems with this approach: it motivates people to buy the same device or sell them, so other users pay for new machines. It’s also difficult to buy used devices recommended by others or replace them when they break, since vendors sell them for much less time than they can be used for. Gluglug provides an important partial solution to these issues: their X60 laptops have fully free boot firmware and a compatible wi-fi card preinstalled, while no new devices are manufactured now (so Lenovo won’t be rewarded for their antifeatures and blobs that Gluglug removes) and it is still possible to get replacement parts for them.

Since in important areas most drivers are not supported by vendors, reverse engineering projects are the best way to improve them.

Nonfree firmware in Linux and OpenBSD; why it should be free

2013-05-12T21:22:21+00:00

The FSF doesn't endorse most GNU/Linux distributions for mainly two reasons: they have no rules against including nonfree programs and include nonfree firmware blobs in their kernels. This article shows some specific examples of such firmware, its usually nonfree licenses and problems that could be solved by having free firmware.

I know three practically different uses of the term firmware: it refers to boot firmware (often known as BIOS), programs running on CPUs other than the one running the main operating system (and the boot firmware) and (mostly in case of embedded or formerly embedded systems like phones) the operating system. This article discusses it only in the second meaning.

Some systems (like Fedora or OpenBSD) claim to not contain nonfree software. This is probably true in case of software running on the main CPU, while they contain nonfree firmware that is also software.

Linux

The kernel used in most GNU/Linux distributions contains many firmware files with a clear list of its origin and licenses. Newer firmware is included in a separate repository called linux-firmware which has a similar list.

Some of these files have ‘interesting’ licenses: there are files with unknown licensing (korg1212); ones with a free license made nonfree by lack of published source code (old radeon microcode); with use restrictions (keyspan), typically allowing use only with their vendor’s hardware; ones free with source available (keyspan-pda), unfortunately there aren’t many of them; some are under GPL while not having source available (ambassador), so they are probably non-distributable; ones with clearly nonfree licenses prohibiting reverse engineering (newer radeon microcode).

Several GNU/Linux distributions avoid these legal and ethical problems by not including this firmware. At least these projects free the Linux kernel from such blobs: Linux-libre, Debian and Replicant, the free Android distribution. Although both Linux-libre and Debian kernels (‘Linux distributions’) are free software, they have different approaches towards recommending the blobs, so Debian-based distributions recommended by the FSF use Linux-libre or its scripts to free their kernels.

A useful feature of Linux-libre is its log listing files containing or referring to nonfree software.

OpenBSD

OpenBSD documents itself as not containing blobs: they use this term referring only to software running on the main CPU. (Technically, the OpenBSD kernel has no blobs: they are hexadecimal, not binary, large objects.)

There are firmware files in the OpenBSD kernel sources with issues similar to ones from Linux: some are freely licenses sourceless software, some have also plainly nonfree licenses, some have no license, some have both use restrictions and no reverse engineering clauses.

No derived BSD distribution removing all nonfree software from the upstream distribution has been released yet. The LibertyBSD project is aiming to change this and NuBSD documented some of the work needed in such projects.

Firmware should be free

The main reason why firmware should be free is that all software should be free (that article also shows important arguments for free software in devices like printers or cell phone modems which often aren’t considered general purpose computers). There are several reasons why free firmware can be better even if there are no antifeatures in the original firmware (which cannot be verified with most nonfree software).

Free firmware can implement features that vendors consider unimportant. Wi-fi firmware often doesn’t implement master or monitor modes, while it can be changed to provide this features (although in this case the nonfree firmware is modified).

Although firmware is specific to a device, it might contain algorithms useful in other systems. The firmware running on the GPU used in Raspberry Pi is an atypical example of this: it implements OpenGL ES, like the device-independent parts of most GPU drivers running on the main CPU. Freeing some firmware might make supporting other devices with similar parts easier, since a single design is often used in multiple devices (a similar situation occurred with drivers in kernels for specific router products which are now available in the mainline Linux kernel).

These two arguments miss one important point: free software can be used for purposes of its users, not all of which are known now. New uses will be invented that nonfree software doesn’t implement today and not supporting nonfree firmware can make future systems better for their users.

Manual dynamic memory management might make debugging easier

2013-01-08T21:14:32+00:00

Dynamic memory allocation has an important use in real world programs: data like input lines has no fixed size, so robust programs shouldn’t allocate fixed buffers for it. Writing a homework program I found another reason for it: it leads to more errors that tools like Valgrind can detect.

Using static allocation for such programs has its advantages: all size limits are specified, so it won’t cause errors like truncating input; it’s easy to declare static arrays of structures in C; it’s faster and doesn’t require many additional lines of code for deallocation of the structure nor thinking where to free the objects.

However, this approach makes debugging harder: static memory is valid for the whole run of the program. Tools like Valgrind’s Memcheck won’t complain about uninitialized values being read or memory being accessed after being freed. (There are problems that occur only with dynamic memory, like double frees or frees of unallocated memory, I don’t consider them as common or as hard as the issues that don’t depend on memory allocation style.)

(In real world programs another reason to use dynamic memory is that some of it might be returned to the operating system before the program finishes. This would be useful in long-lived processes doing big allocations for quick computations, although it won’t occur in all cases due to the way how malloc-style routines work.)

The program that motivated me to write this article implemented monotone polygon triangulation using the DCEL structure to represent the polygon. (This structure is not needed for this algorithm. Using it leads to having more bugs, therefore it is educationally useful.) The code managed half-edges to run an algorithm designed for vertices. Storing triangulated parts of the polygon was not needed, so they were deallocated immediately after printing their representation. This resulted in a use-after-free error detected by Valgrind, fixing it corrected an incorrect result on a different polygon. The code added diagonals between half-edges, the resulting graph wasn’t a correct triangulation if the diagonals were added between half-edges of different polygons: some of which were completely triangulated in some cases and thus deallocated before adding the faulty diagonal.

For nearly all mine other programs I use languages with automatic memory management (and I don’t use data structures as complex as DCEL), so there would be no error with the deallocation delayed after the final use of the object. Previously I thought that all such errors would be introduced by incorrect placement of free or delete calls. This program helped me realize that the delete operator can be useful in detecting otherwise incorrect code.

Lemote YeeLoong 8101B with Loongson 2F CPU review

2012-09-03T15:20:24+00:00

The Lemote YeeLoong is a small and free software-friendly laptop and one of the few available non-x86 (and non-ARM) laptops. (It’s sometimes called a ‘netbook’ or a ‘mini notebook’.)

As a user and contributor to a GNU/Linux distribution supporting this device, I’m often asked about it. The information published by the manufacturer and distro maintainers doesn’t reflect what could be seen by a user. This review is based on my experience using it and questions of free software supporters interested in this device.

The YeeLoong I have is 8101B with a 10.1 inch display. The 8089B model probably differs only in display size (8.9 inch), having the same internals. There are newer YeeLoongs with 2G or 3A CPUs being marketed, these are significantly different on most points discussed here. (This review probably won’t be helpful for review-writing classes, there are better resources for them available.)

Hardware

Case

One of the most common marketing claims is that the machine was built by Quanta and the case is of high quality. This seems more reliable than most qualitative opinions stated on the Lemote page.

Shiny lid, the user-visible part matte, the parts I use aren’t visibly shinier. No intrusive logos (the small model name near the screen helps typing it correctly). No scratches after more than two years of using. The display hinge works ok.

(My other laptop, an Asus F3U has many scratches on top and had once many parts changed after the display hinge broke. Despite being newer by about a year, the parts I touched are now much shinier. Completely different experience.)

CPU

Loongson 2F is a single core MIPS3-compatible 64-bit CPU with some custom ISA extensions (not all used in software).

There is a custom SIMD extension, similar to MMX although not well-supported by GCC and with different intrinsics. A Gentoo hacker used them to optimize an important graphics library and posted a great explanation of these issues.

There are easily worked around bugs which would hang the machine (still untrusted code can do it), they were a bigger problem before English documentation was made available.

There is no uploadable microcode, this is one of the reason why x86 systems probably won’t be as free as this one (even the free boot firmware implementation coreboot usually requires nonfree CPU microcode).

The manufacturer claims of buffer overflow protection, this probably refers to an NX bit. I don’t know if it’s used in software.

Video card

The video card is a SiliconMotion SMI712 which does not have any hardware 3D acceleration. The reason why I consider the machine not completely supported by free software is limited 2D or video acceleration in free GNU/Linux distros.

gNewSense metad uses the fbdev driver without support for resolution change or 2D acceleration. Parabola uses the siliconmotion driver with unoptimal support for these features (fbdev is available). Newer X server releases make XAA slower (this is very noticeable when using KDE), while EXA hangs the machine (not a new issue), so fbdev might be faster now. There are legendary drivers with xrandr support, I never used them.

Gentoo has patches making full-screen low quality YouTube videos playable (used WatchVideo for this), this probably could be ported to other distros. There are ongoing discussions on a new SiliconMotion video driver on the X.Org development list, maybe this driver will improve this situation (it has xrandr support).

The VGA output has low colour quality, although I haven’t used such outputs on other machines for a longer time, when all my other machines can use DVI-D. The documentation of the chip claims dual head support at 16 bpp, I never used it successfully.

SMI712 has only 4 MiB of video RAM, using reasonable resolutions might need special X settings to fit in it.

Summing up, I believe the only good thing about this chip is no dependency on a nonfree VBIOS, system-provided microcode nor driver. Unfortunately, no other graphics chip used in laptops or desktop computers known to me has this feature.

Display

Despite all the driver problems, this machine is fast enough to read typical books in a PDF reader. The screen is matte, unlike my other laptop, so it’s useful even during summer days.

However, decreasing the backlight brightness results in a headache-causing flickering (a problem caused by the LED backlight design, probably occurring on other devices). Usually I can use it at full brightness, it’s a less noticeable issue when not using X.

Wi-Fi

There are reports of the Wi-Fi card not working, but I haven’t observed any problems with it. AP mode is not supported by the driver, I never wanted to use it since my other machine with a more powerful Atheros 802.11n card supports it.

The card supports 802.11b and 802.11g, not 802.11a despite what some reviews state.

Webcam

The webcam works with only some programs, depending on kernel version. I probably haven’t tried enough to configure it.

SD card reader

It works with both SD and SD HC cards. Somehow on Parabola reading from the SD card was needed before the partitions were found, so it didn’t work well with the GUIs for mounting storage.

Booting from SD cards is not supported.

Touchpad

There is no middle button and the layout of left and right buttons make simultaneous clicks impossible.

The device is of Sentelic, it doesn’t support absolute positioning in the driver (possibly due to patent issues). I had better experience with an ALPS touchpad supported by the xf86-input-synaptics driver.

There are various non-mainline drivers for Sentelic touchpads (e.g. for MSI Wind), maybe some of them would work with the one in the YeeLoong. I haven’t tried using them.

Fan

It’s loud. It’s too often running, although this might be partially fixed using thinkfan.

RAM

Only 1 GiB is supported by the CPU and boot firmware. The SO-DIMM can be changed, I haven’t found any need to do it.

Disk

The (probably too optimistic in general case) hdparm benchmark shows 20 MB/s transfer speed, even when asked the driver to consider it being connected via an 80 wire cable (a Parabola hacker had similar results with an SSD). The chipset and disk documentation suggests a much higher speed being supported. (Maybe this is related to using a SATA disk with an IDE controller?)

Suspend

Fan works while suspended to RAM, so I use only suspend to disk.

Battery

The machine can work up to two hours on battery. The manufacturer claims lower power use of 12 Watts for the SSD version, data available to the system suggests it being similar for some uses of the HDD version.

Most netbooks work much longer on battery, this results from both bigger batteries and lower CPU power usage. Users who need this use external batteries (I never used them).

Connectors

The device has external connectors for VGA, power, 3.5 mm microphone and speaker, 100 Mbps Ethernet and three USB ports. Its layout prevents using both an Ethernet wire and an USB mouse (the wire would be on the place where I would keep the mouse), this doesn’t change mice being uncomfortable for me in all cases.

Boot firmware

The YeeLoong is often called the only laptop not requiring nonfree software. EC and hard disk firmware are exceptions.

All Lemote machines use a derivative of PMON2000 as boot firmware. It is free (under a four clause BSD license), although on all other devices than YeeLoongs with 2F CPUs it requires a sourceless VGA BIOS blob.

PMON initializes the hardware, shows a menu of kernels to run (using a GRUB 0.97-like configuration file) and boots one of them, supports network booting and flashing itself. It’s not compatible with x86 BIOSes and is more powerful (e.g. it can boot a kernel from an ext2 filesystem, although it doesn’t support newer filesystems).

Booting is fast unless using an initrd (gNewSense and Parabola kernels don’t need it unless using an encrypted root filesystem) or GRUB 2 booted from PMON (I see no benefit of using it).

It is also possible to use GRUB 2 as a PMON replacement, installed directly to a PLCC chip (coreboot doesn’t support the machine). It is difficult to solve potential problems with it due to the PLCC chip being soldered in most devices (or difficult to access).

Software availability

A Debian-based system with very old packages was installed on the machine. I haven’t used it long before installing gNewSense metad. (The installer was broken at that time, so I haven’t used it initially for two weeks before it was fixed; this problem motivated me to use IRC, this led me into contributing to several free software projects and using fully free GNU/Linux distributions.) The review at OSNews has more details on this system (and many other features that I haven’t noticed).

gNewSense has most Debian packages available. There is GHC without the interactive interpreter and a slow Java implementation (without JIT). Mono and Valgrind are not available (although the newest release of Valgrind supports MIPS and is included in Debian Jessie). Gnash is available, although it is too slow to be useful for me and there are better specific tools for most tasks that I could need it for.

While Debian and gNewSense use packages built for any little-endian MIPS system, Parabola has them optimized for Loongson 2F and uses a different ABI called N32 that uses 64-bit registers (and all floating point registers, unlike O32 used in Debian) while 32-bit pointers are used (so a single process can use only 2 GiB of virtual memory: the highest address bit is used for kernel and physical memory). As an advantage, some articles suggest it being 30% faster on some operations. A disadvantage is lack of support for many architecture-specific packages like Java, Valgrind or GHC (and much more portability problems in other packages like WebKit which doesn’t need to use architecture-specific code). Now more packages start to require a JIT, so modern Mozilla software and Qt 5 aren’t available on N32.

One of the reasons for RMS to use such a machine is that it is not supported by popular nonfree operating systems, so it won’t be used to promote them, unlike OLPC.

Performance

The CPU speed is not a problem unless compiling distro packages or using Java or other programs that are optimized for JIT not available or working on the MIPS ABIs used (or when playing videos without large assembly patches using its SIMD extension).

Building GCC, Mozilla browsers or WebKit is too slow to maintain these packages correctly in Parabola. Typical tasks like Web browsing are interactive enough, unless building a package at the same time or viewing a big JPEG image (although this is also slow on my AMD64 machine).

Having played free games like Wesnoth, FreeDink and DCSS (without tiles which require hardware-accelerated OpenGL), I’m not convinced that good games need 3d acceleration.

Availability

There are YeeLoongs available in Europe from Tekmote Electronics (where I bought mine from) and KD85.com. Freedom Included was selling them in the USA with gNewSense preinstalled, now only a limited number of units is available there for developers.

The manufacturer site claims of ‘very competitive price’, this certainly isn’t true in Europe in comparison with non-freedom-respecting x86 netbooks.

Summary

I know three main reasons to use a YeeLoong: it respects user’s freedom, it can be used for MIPS programming and it is a small and portable laptop. I don’t know any good alternatives for the first two of these uses. Except for the graphics performance, I believe the YeeLoong might still be an appropriate device as a general purpose small laptop (although this is not a significant problem for most of my text-oriented needs).

Free software licenses are not a sufficient condition for software freedom

2012-08-14T17:07:40+00:00

A common misconception about free software is that having a free license is both a sufficient and necessary condition for the software to be free. There might be cases when software is too simple to be restricted by copyright, so it is not a necessary condition. There are much more important arguments why it is not sufficient.

Copyright is used to restrict sharing of a work. This is often used to prevent users from cooperating with other users, while this can be used to prohibit some restrictions of the freedom of derivative work users in a practice called copyleft. Although copyleft can limit freedom of users modifying software (works under different copyleft licenses usually cannot be merged in one derived work), this is practically a useful compromise which won’t be discussed in this essay.

The most obvious way to restrict user’s freedom is to not share with them a source form of the program. There are many sourceless blobs of microcode in the kernel called Linux, while they have free licenses ‘giving’ users freedom that they cannot use. Without copyright (or with a very short one), there would be no effect on restrictions of nonfree software, only copyleft would be limited. Digital restrictions management clearly shows that copyright isn’t important for software owners: they make and enforce their own rules restricting what the user can do (actions allowed by copyright laws are also restricted by DRM).

DRM wouldn’t be effective if the user could understand the program (difficult, although possible even without source code as some cases in cryptography show) and install a different one. The second part is prevented by nonmodifiable bootloaders that can load only software signed by the device manufacturer. This is a common practice for phone and tablet operating systems using much GPLv2-licensed software, named tivoization after an early case.

Although free software licenses can protect users from the above problems in derived works, there are different legal issues that won’t be avoided in this way. Software patents are one of them. Governments censoring useful cryptography or pornography are similarly a restriction of software user’s freedom (and non-user’s).

These examples show that free software licenses aren’t sufficient for software to be free, while they don’t suggest any obvious solutions to this problem. They clearly require user’s awareness of their (not necessarily software-related) freedom. The focus on licensing leads to them not considering these issues restrictions of their freedom. It could also limit the visibility of copyright problems making other alternatives (with appropriate replacements for copyleft) more beneficial in a longer term.

Advertising considered harmful

2012-08-12T20:10:50+00:00

It’s said often that if we don’t pay for a service, we are the product. It isn’t generally true, although it seems true for many businesses supported by advertising. In this essay I explain why I believe such advertising to be generally useless for the user and harmful for the society.

There is nearly no benefit from advertising for the user. If they are looking for a better product to solve their problem, they can get good recommendations from their friends or other users of such products. There are multiple reasons why great advertising isn’t equivalent to great products, this makes ‘neutral’ search engines better at finding product information than paid-for ads.

Therefore advertising doesn’t produce anything useful for the society, while it is a huge business. Imagine all people who currently work on making ads and releasing them everywhere and all this creativity that could be used to support well-being of all (or just for leisure, it doesn’t bring the problems of advertising).

No ad can be useful for everyone that views it, it must be useless for someone. There is no difference between spam and other advertising. The one that is copied to the user wastes resources that have better uses. Imagine all paper used for loan ads, all network packets and CPU cycles for drug emails or ads on Web sites. How much could these resources benefit us?

Since it would be otherwise obviously useless for the business, advertisements must encourage people to buy things they don’t need to. They offer to solve all of our problems. The result? We have more problems, more things, less time and resources for what we really want.

On the Web there is additional problem of ads: their publishers can and do track much more data about the user than would be useful to provide their ‘service’.

These are the problems of the medium and motivation, there are also other problems of the message and views promoted in it. Many ads I see when I rarely watch TV promote the view that people should just ‘consume’ the product, do what the masters say, not think for themselves. They assign strict roles for people of various jobs or genders. They promote the harmful to the people family concepts. Most people shown in ads are slaves of the producers and their own genes. We must change this to surpass men, we won’t while supporting advertising by the wealthy ones.

These were the reasons why I won’t choose to support my work by advertising. I use Web browsers with addons like Adblock Plus and NoScript so my computers won’t waste their scarce bandwidth to fetch useless ads and show them instead of the article I’m looking for. Although I disable them for some Web comic sites, since I found many interesting comics via such ads (this might be similar in some other niche areas). The TV ads are an easier problem to avoid, I find creative activities like writing free software that I spend my time on instead of watching TV. Different funding methods for cultural works must be found, ones not connected to the act of using these works.

There is no tree of evolution

2012-08-05T14:17:33+00:00

We often see diagrams called ‘trees’ showing how different beings or things evolve from others. These are used to describe families, species, languages, programs and other entities. Most of them share two problems: they aren’t trees as in graph theory (while we reason about them as trees) and they don’t have discrete, unchanging nodes.

Directed acyclic graphs of evolution

Let’s assume for a while that there are immutable objects with derivative objects being created in atomic ways. These objects are nodes (or vertices), there are edges connecting them from the base one to the derived one.

The ‘tree’ will have nodes with multiple outward edges. However, in many real cases it will also have multiple inward edges. It isn’t a tree, the graph theoretic name for it is a directed acyclic graph, or a DAG. (We won’t get cycles by just adding immutable nodes having immutable lists of predecessor nodes, which seems all what we can do with unidirectional time and really atomic objects.)

Probably the most famous DAGs called ‘trees’ are ‘family trees’. The larger and more complex ones aren’t trees, there is a newer example in The Art of Computer Programming by Donald E. Knuth, Volume 1 Third Edition, Figure 18(a) on page 310 (the family DAGs of Eldar or Edain in the books of J.R.R. Tolkien are more interesting, although improbable without elven lifespans, cases). Some have ancestors aligned on layers depending on the distance from a descendant, these might have multiple nodes on different layers for the same person. It’s an obvious consequence of every person having approximately two parents and there not being exponentially more people several generations ago. As explained in the TAOCP (page 311), these graphs are trees if a node represents ‘a person in the role of mother or father of so-and-so’ (I haven’t seen this definition anywhere else).

There are more software-related examples of graphs not being trees (leaving other natural examples for the next section, since they don’t have obviously atomic nodes). Dependencies graphs of classes, modules and software packages (being specific sets of code, so they are atomic as considered in this article) clearly aren’t trees in general (and common) case. (Dependencies inside a package are often cyclic and this isn’t usually a problem. It is more difficult in case of dependencies between packages, compilers are a well-known case of this.) I believe the assumption that software packages dependency graphs are nearly always trees, optimizing the design of software using such graphs for this case and adding workarounds for a package occurring in multiple parts of the ‘tree’ leads to a much more complex solution than designing for a DAG.

A significantly different tree-vs-DAG understanding issue is that software doesn’t evolve in trees. A common free software ideology point of view is that the user gets a software package, modifies it using only their own ideas and publishes it. This isn’t true, a common and important case is deriving a new package (or technically a version of the package, both are equally nodes here) from multiple other packages. Nearly any use of shared libraries from multiple projects is a case of this. This leads to different software freedoms than the tree view (the Free Software Definition mentions this way of modifying programs, although accepts licenses restricting what licenses the original programs can be under; I see no solution for this better than using only GPL-compatible licenses for all cultural works).

(There is also an interesting, although unrelated to trees, issue of maps being nonplanar graphs. I don’t remember seeing an example of a map having its chromatic number greater than four, although there are many cases requiring four colours. Many countries in these cases are non-contiguous, so a graph of a map containing them doesn’t have to be planar.)

Are there atomic nodes?

In the above examples it was clear what objects were nodes. Each edge referred to complete atomic nodes (e.g. there are parents of a human, not of their specific organs, I don’t know any other cases than aren’t only medically significant unlike the probable uses of family ‘trees’). It isn’t in all common uses of ‘trees’. If the node refers to a mutable union of immutable objects (or a mutable object), the graph can easily have cycles, multiple edges between the same nodes (it even is’t usually called a graph), or just be too unclear to be useful.

A problem of such nodes is that we often don’t know clearly when we have the same node.

Probably the most well-known example is a phylogenetic tree showing a common origin of species (this isn’t the only evolution the title of this article refers to). As described in that article, the ‘ideal’ approximated by such trees isn’t a tree, due to interesting issues like horizontal gene transfer which clearly makes it not a tree. Another issue is that there is no clear separation between species.

The same problem occurs in the evolution of languages. There is no clear separation between languages (except for the political ones). If, in parallel to the definition of a species as a set of animals being able to have common descendants, we define a language as a set of mutually-intelligible sentences, there is no point of separating languages (for each pair of communicators there is at most one language, there is no observable way of finding which of these languages are ‘the same’; this might be a reason for other definitions being widely used).

There is a similar case for software evolution. There is a great graph of GNU/Linux distribution derivation. Since the distributions change, it has many vertical lines when a distribution e.g. changes its base distribution. Somehow this case avoids having the nodes as unclear as species or languages.

Let’s not call the next graph a ‘tree’, unless the objects being modeled are clearly separate, atomic and form trees, not more complex graphs.

Some advantages of shared libraries

2012-07-31T20:49:38+00:00

As a contributor to a GNU/Linux distro, I’ve seen three main ways of using libraries from other projects in a program:

using a shared library from another package
using a static library from another package
copying the source of another project and building it in the same package.

It’s obvious why the third way is bad. A significant problem for the Parabola GNU/Linux-libre mips64el port was WebKit not supporting MIPS N32. This could be easily fixed by disabling JIT support, disabling assembly support and enabling alignment of allocated memory (already done for O32 and other architectures). However, different subsets of these changes needed to be done in at least three packages, one of which builds whole WebKit GTK twice (taking about 30 hours on my machine). It’s still possible that the code was copied into other packages, leading to more errors and long rebuilds.

More typical cases of such problems are fixes for security problems (e.g. in libpng or zlib, these were commonly bundled with other packages despite being installed on practically any GNU/Linux system) or removals of nonfree code in FSDG-compatible distros.

Using static libraries wouldn’t solve the above problems – they would require relinking the programs (i.e. rebuilding in distros, for simplicity and reproducibility) and knowing which programs are affected.

This leads to the following advantages of shared libraries:

a bug can be fixed by changing just one package – reinstalling the library will make the programs use it on next run
programs specify which libraries they use (tools like readelf or scanelf can show this easily), so it’s know what is affected by a library change

However, this assumes that the new library version is compatible with programs built for the previous one. This assumption is incorrect for many libraries (e.g. in case of libpng or poppler updates requiring rebuilding many programs in distros not supporting multiple library versions at once). Despite this problem, such shared libraries are still useful to avoid having multiple copies of them on disk or memory. This is one of the reasons for GHC to support building shared libraries of Haskell packages (any rebuild of a dependency changes the ABI there). Not needing an "evil Perl script" to not link unused functions and better plugin support are other reasons for this.

There is a problem with having both shared and static libraries for a single package: they are usually compiled differently. Static libraries and programs don’t use position independent code which could make them slower, while shared libraries need it on many architectures (x86_64 is the most popular of them). Therefore having only shared libraries for a package might make its build twice faster. In distros like Parabola where development-specific files are not split into separate packages, it would also make the packages much smaller (very useful for LiveCDs).

I know two reasons for using static libraries for typical packages:

it might be faster (no PIC)
this doesn’t require having the needed shared library versions installed

I don’t know any data related to the first argument. The second one is completely unimportant for users of distro packages.

Therefore I think that no package in a new GNU/Linux distro should include a static library unless it’s gcc or glibc (these have other reasons to do this).

Other views

Ulrich Drepper wrote an article arguing why static libraries should never be used on systems with glibc. His article mentions some of the above arguments, address layout randomization and interesting features of glibc using dynamic linking.

The stali project presents a completely opposite view, with multiple reasons why properly designed static libraries are better. I think it’s a sufficiently different case than typical GNU/Linux distros using glibc and having big existing programs to not apply in the cases I described.

Socket activation for FastCGI with systemd

2012-07-26T17:50:03+00:00

The server hosting this blog runs Parabola GNU/Linux-libre with systemd. The HTTP daemon I use is nginx, it serves several domains with Python software connected to it via FastCGI (using MoinMoin as example, it handled a wiki this article was migrated from). Since nginx doesn’t start FastCGI processes on its own and I don’t want to use another daemon for this, I decided to use systemd socket activation to control the FastCGI process. (I’m not discussing the choice of software described here, most reasons I have to use it are unrelated to this issue and there are probably better choices for your use.)

I’ve configured nginx to use the /srv/www/wiki.mtjm.eu/fastcgi socket.

Mostly as documented in the manual, this unit describes the socket:

[Unit]
Description=wiki.mtjm.eu

[Socket]
ListenStream=/srv/www/wiki.mtjm.eu/fastcgi
Accept=false

I put it in /etc/systemd/system/sockets.target.wants/mtjmwiki.socket.

The Accept=false line is the reason for FastCGI: the service (mtjmwiki.service) is started on first request and used for the following requests, unlike with Accept=true that would start it separately for each request. Only the second option is supported with CGI and some older servers (it was typically done via inetd).

The service unit is similarly simple:

[Unit]
Description=wiki.mtjm.eu

[Service]
Type=simple
ExecStart=/srv/www/wiki.mtjm.eu/moin/moin.fcgi
User=mtjmwiki
Group=mtjmwiki
StandardInput=socket

I wrote it to /etc/systemd/system/mtjmwiki.service. The /srv/www/wiki.mtjm.eu/moin/moin.fcgi script is /usr/share/moin/server/moin.cgi adapted to use my paths.

The StandardInput=socket line makes the MoinMoin process get the socket used as specified by the FastCGI protocol. (Without this line it would assume that it’s using CGI.)

Starting the socket units and accessing the wiki has shown that this worked.

A guide to DNS configuration on servers

2012-04-06T13:27:27+00:00

I use DNS daemons on my server for two different tasks:

to find what names maintained by others point to
to publish this mapping for the mtjm.eu. zone

This article aims to explain how these tasks could be done on a small personal server.

DNS servers work in different ways to perform these tasks. Recursive (or caching) servers ask other servers for names and cache the results (the real algorithm is more complex). Authoritative servers get all their answers locally, usually from a file or from data synchronized from other authoritative servers (sometimes called masters, the difference is completely internal).

There are important security, reliability and modularity arguments for using separate recursive and authoritative daemons. The only argument against this known to me is that they need having separate IP addresses, this is not a problem on any machine having both local and public interfaces (and there is no reason to host an authoritative server otherwise). Caches don’t need to and shouldn’t be public.

Unbound recursive server

I use Unbound as a caching server. The default configuration of the package in Parabola GNU/Linux-libre is probably correct, by default it listens on IPv4 and IPv6 localhost addresses.

For other programs to query this daemon, the /etc/resolv.conf file should contain the nameserver 127.0.0.1 line (no other lines are needed). Your network configuration scripts might replace it, a simple and portable way to solve this is to run ‘chattr +i /etc/resolv.conf’ after editing, so no program will change it (without chattr -i).

NSD authoritative server

I use NSD to publish my zone.

The Parabola package includes a foreign rc.d script which makes rc.d not know its status, this is easily solved by using systemd and its nsd.service unit. I haven’t found other problems using it.

This my /etc/nsd/nsd.conf with default comments stripped:

server:
        # These two are the public addresses of the machine running it.
        # The default is to listen on all addresses, which would conflict
        # with Unbound listening on local addresses.
        ip-address: 178.79.140.113
        ip-address: 2a01:7e00::f03c:91ff:fe96:3f65

zone:
        name: "mtjm.eu"
        zonefile: "mtjm.eu.zone"
        # Used to allow zone transfers to secondary server for specific
        # addresses.
        provide-xfr: 216.218.130.2 NOKEY

Writing zone file

The format of zone files is complex and difficult to understand or parse. Somehow most DNS servers use it (djbdns has a much simpler, extensible and somewhat user friendly format).

These are fragments of my zone file, based on various examples found more than a year ago:

$ORIGIN mtjm.eu.     ; designates the start of this zone file in the name space
$TTL 12h                  ; default expiration time of all resource records without their own TTL value
mtjm.eu.  IN  SOA  ns1.he.net. hostmaster.mtjm.eu. (
              2012032801 ; serial number of this zone file
              1d         ; slave refresh (1 day)
              2h         ; slave retry time in case of a problem (2 hours)
              4w         ; slave expiration time (4 weeks)
              1h         ; minimum caching time in case of failed lookups (1 hour)
              )
@      IN NS    ns1.he.net.
       IN NS    ns2.he.net.
       IN NS    ns3.he.net.
       IN NS    ns4.he.net.
       IN NS    ns5.he.net.
; mail
       IN MX    0 elderthing
; WWW
       IN A     178.79.140.113
       IN AAAA  2a01:7e00::f03c:91ff:fe96:3f65
       IN TXT   "v=spf1 ip4:178.79.140.113 ip6:2a01:7e00::f03c:91ff:fe96:3f65 -all"
       IN SPF   "v=spf1 ip4:178.79.140.113 ip6:2a01:7e00::f03c:91ff:fe96:3f65 -all"
; machines
elderthing IN A 178.79.140.113
       IN AAAA  2a01:7e00::f03c:91ff:fe96:3f65
       IN TXT   "v=spf1 ip4:178.79.140.113 ip6:2a01:7e00::f03c:91ff:fe96:3f65 -all"
       IN SPF   "v=spf1 ip4:178.79.140.113 ip6:2a01:7e00::f03c:91ff:fe96:3f65 -all"
; sites on elderthing
b      IN A     178.79.140.113
       IN AAAA  2a01:7e00::f03c:91ff:fe96:3f65
       IN TXT   v=spf1 -all
       IN SPF   v=spf1 -all
; some other sites ommitted
; XMPP
_jabber._tcp IN SRV 0 0 5269 elderthing.mtjm.eu.
       IN TXT   v=spf1 -all
       IN SPF   v=spf1 -all
_xmpp-client._tcp IN SRV 0 0 5222 elderthing.mtjm.eu.
       IN TXT   v=spf1 -all
       IN SPF   v=spf1 -all
_xmpp-server._tcp IN SRV 0 0 5269 elderthing.mtjm.eu.
       IN TXT   v=spf1 -all
       IN SPF   v=spf1 -all

The SOA record has various numbers to make synchronizing zones with secondary servers harder. The secondary servers are provided by the gratis Hurricane Electric DNS service, they have all features that I need (e.g. IPv6 support) and I haven’t observed any problems with them.

The A and AAAA records specify IPv4 and IPv6 addresses of various machines. DNS supports CNAME aliases which shouldn’t be used (that page lists many other problems with DNS design or implementation).

The TXT and SPF records show that most of my records aren’t used in email envelope sender addresses and the other ones are used only for mail sent by elderthing. If enough domains and mail servers used these records, spammers would need to buy their own domains and not forge their sender addresses.

The SRV records here just specify that elderthing provides all XMPP services. They are much more powerful than that.

Run nsdc rebuild && nsdc reload after changing the zone file.

Useful changes not described here

DNS has big security problems and introducing more secure replacements is difficult. Some of these problems can be optionally solved by DNSSEC or DNSCurve (each of them solves different problems). I have no experience with these specifications nor how they could be used.

An unnamed DNS replacement idea

2012-02-27T16:09:24+00:00

DNS solves two problems:

translating between human-readable domain names and machine-usable IP addresses
storing a reliable, hierarchical, distributed database describing which servers provide which services and the above mapping.

It’s known that the second problem is solved inefficiently, insecurely, unreliably and centrally. Thus a different system should be designed to solve it without these problems.

The first problem is solved using globally unique names with unique meanings. This is an unreal assumption enabling useless or harmful activities like domain parking, domain squatting, trademarks being used for censorship, or just making the names difficult to type.

This probably contributes to the fact that users often use machines to store the domain names. Other issues like advertising contribute to using unreadable names and sharing them via e.g. QR codes instead of memorization by humans.

Therefore I believe a good replacement for DNS solving the second problem would not use globally unique human readable names.

Let’s assume that a single being manages the database fragment describing some machines (like a DNS zone). There is no problem with having names in the fragment. The fragment should be signed using a key pair used only for this zone with private part known only to the managing being. Probably any useful and scalable DNS alternative would do this.

The ‘name’ of the zone would be the public key used to sign the zone. It would be random-like and there practically wouldn’t be multiple zones with the same ‘name’, so this would avoid the problems of nonrandom unique names. There are algorithms using elliptic curve cryptography having good enough public keys small enough to use them in a DNS domain name, they could be used here (although having e.g. a multi-kilobyte zone ‘name’ wouldn’t be a problem for machines transferring them).

We could let anyone serve the zone data, since knowing the public key allows knowing if the untrusted server provided us the correct data, assuming the data doesn’t change. In real life such data changes, but having an outdated copy could be detected by e.g. specifying the time when the data is valid in the zone (DNS uses a similar solution, although it wouldn’t need having globally synchronized clocks). There are existing solutions for sharing such data without having a central server.

This leaves the problem of having human-typeable names for the zones in the rare cases when they are useful. This would be solved by having a local daemon having such user-specified mapping and maybe asking other such daemons for other names (e.g. if the user has many machines or uses names shared with other users in a single organization).

mtjm’s blog

Personal server exit review

Why Cubieboard

How it was configured

How I broke it

Now

Future

My task management: input queues

Handwritten notes

Phone notes

Email

IRC

Browser tabs

Ebooks

Feed reader

Conclusions and future ideas

Inclusion of licenses longer than licensed works

PlaneShift and free software

Intel GM45, AMD K10 and Richland compilation benchmark

Machines

Benchmark script

Results

How temporary data loss helped me improve my backup policies

Free software with nonfree dependencies and the emulation argument

Free software Flash replacements and the JavaScript trap

Replacing Flash is hard

The JavaScript trap

Why we need video downloader programs

Flash animations

Publishing your own works

My email spam filtering and end-to-end encryption

LibrePlanet 2014 stream recordings

Lessons learned from using wget to download the streams

Watching the talks

DRM in free software

PDF restrictions: Okular, pdftk

LibreOffice spreadsheet ‘protection’

FontForge

DRM and software freedom

Solution

Buying a freedom-respecting USB wifi card for a Lenovo laptop

Choosing chipset and device

What’s needed to use an internal card with free software?

Summary

Missing source code for non-software works in free GNU/Linux distributions

The source

Fonts

Hyphenation patterns

Game data

Documentation

The program-data dualism

Version control

Conclusions

How can we improve hardware support in free GNU/Linux distributions?

Reverse engineering

Porting software to completely free non-x86 systems

Removing software dependency on hardware-accelerated graphics

Documenting compatibility of used hardware

Laptops and free software

Boot firmware

CPU microcode

Graphics

Wi-fi

Embedded controller

How can this be improved

Nonfree firmware in Linux and OpenBSD; why it should be free

Linux

OpenBSD

Firmware should be free

Manual dynamic memory management might make debugging easier

Lemote YeeLoong 8101B with Loongson 2F CPU review

Hardware

Case

CPU

Video card

Display

Wi-Fi

Webcam

SD card reader

Touchpad

Lessons learned from using `wget` to download the streams