I'm starting to get impatient. I really want to start running a next-generation file system. I'd like to have easy to manage sub volumes, block level checksums, and RAID-like redundancy at a higher level.

Writable snapshots are what I'm most excited about, though. Snapshots of my home directory will help protect me from accidental file deletions and modifications between backups.

I'm currently saving disk space with my QEMU disk images by juggling cloned qcow images. It should be much easier to manage some writable Btrfs snapshots instead. Deduplication would likely save me quite a bit of space as the virtual machines age. ZFS already has deduplilcation, but it sounds like deduplication won't be all that far away for Btrfs.

What are the Best Practices for a Btrfs Root File System?

I have absolutely no idea, but I think I am on the right track. I did not want to use the root of the Btrfs file system as my / file system. I created a sub volume called rootfs instead. I mounted that sub volume as / and I mounted the entire Btrfs volume under /mnt/btrfs.

Doing it this way felt pretty well organized. From this point, it would be pretty easy to create a homefs and a tmpfs. I didn't want to break out /var/ or /usr. That would make it too difficult to snapshot the operating system between upgrades.

Overview of How to Get it All Up and Running

There is a tool to convert and ext3 or ext4 file system to Btrfs. I didn't use them and I don't know if they would allow me to lay out the file system the way I was hoping. I ended up doing it the long way.

This isn't going to be a step-by-step guide. The quick overview of the steps was:

• Install Ubuntu 9.10, small ext2 or ext3 boot partition
• Compile and install kernel 2.6.32 and the latest btrfs-utils (you can skip this if you don't need snapshot removal)
• Add btrfs.ko, zlib_deflate.ko, and crc32c.ko to the initramfs
• Back up everything on the / file system
• Create a Btrfs on the old / block device, create a rootfs sub volume
• Restore the old / file system contents to the rootfs subvolume

GRUB and fstab

You'll have to change your root kernel parameter and add a rootflags parameter:

root=/dev/sda2 rootflags=subvol=rootfs

I left fstab just pointing to /dev/sda2. That seemed to be work fine and allowed me to boot from different snapshots. I also added a line to mount /dev/sda2 under /mnt/btrfs.

Taking a snapshot and booting from a snapshot

I created a snapshot of rootfs called rootfs-1 and then I performed an apt-get dist-upgrade and installed a few extra packages. It wasn't a huge change but it was enough to test booting from a snapshot.

Booting from a snapshot turned out to be easy. At the start of boot, I chose to edit the GRUB entry before booting. I just changed subvol=rootfs to subvol=rootfs-1. It would probably be pretty easy to automatically generate a GRUB menu with all the snapshots as options.

One Little GRUB Hiccup

update-grub ran fine the first time, while my root filesystem was still ext4. update-grub got angry and wouldn't update properly with a Btrfs root. I don't have a good fix for this yet—I just kept fiddling with the GRUB entries during boot.

GRUB, Btrfs, and Kernels

Being able to snapshot root will be very handy. I'd be tempted to set up some automation to snapshot root every day and at every apt-get upgrade.

Unfortunately, GRUB doesn't understand Btrfs or its snapshots. Kernels and initrd images have to be stored on a boot partition that won't support snapshots. That makes the kernel the weak link in the upgrade chain.

It would be nice if the boot loader could pull a kernel straight from the snapshot. That would let you keep your snapshots more self-contained. I don't feel this is very important, though. Snapshots of root alone will be a huge step forward.

Sorry for the Lack of Screen Shots

My X25 died this week and I didn't get around to backing up my Btrfs virtual machine. I didn't think I would need to back up that image because I was planning on deleting it after I wrote this article.

I am seriously considering giving Btrfs a try on my laptop when the replacement X25 gets here. I will post some screen shots if I do.

I bought my first Moka Express about a year ago. It is a wonderful little machine. It is very easy to use, simple to keep clean, and durable. It doesn't take up any counter space and it is easy to take with me when I travel. It also makes a good strong espresso-like coffee.

I should probably mention here that I am not a coffee aficionado. I can't tell the difference between coffee I ground today or last month. A lot of people would probably say I don't even really like coffee.

I do like French Vanilla creamer and caramel syrup, though. I don't make proper coffee beverages. I usually mix about 1/3 Moka Express coffee with 2/3 french vanilla creamer (50/50 if it's iced) with a bit of caramel syrup.

We have a Keurig K-Cup machine. I can only get a strong enough brew out of it to make a coffee the way I like if I use the iced-coffee (3.25 oz) setting and use it to make a hot coffee. Even on that setting it is much, much weaker than what comes out of the Moka Express. It may not be as strong, but it sure is fast and convenient.

I've learned a lot about brewing coffee with the Moka Express over the last 12 months. Some of what I learned is exactly the opposite of what "The Internet" told me.

Don't Grind the Coffee Too Fine, Watch Out For Dust

The first batches of coffee I made with my Moka Express were awesome. After a while, things got bitter. Tiny grounds of coffee were making it through the filter into my beverage.

It turned out that the pre-ground coffee we had lying around was probably processed with a blade-style grinder. No matter how coarse the grind, a blade grinder will generate tiny grains of coffee dust. These pass right through the metal filter and make the brew extremely bitter.

My first attempts at grinding my own beans failed in the same way, since I was using a cheap blade-type grinder. I picked up an inexpensive Nesco burr grinder. Now my brews are consistent and taste better than ever.

My good friend, "The Internet," told me to grind the beans finer than for a drip machine. I'm actually getting much better results on a slightly coarser setting (the 6.5-7.0 setting on my Nesco grinder).

When I ground too fine the water couldn't get through the grounds. The valve would whistle, it'd take forever to brew, and it would taste terrible.

I've made plenty of tasty non-bitter coffee with pre-ground mass-produced coffee. If you're buying pre-ground, you will just have to find one that works. You can also probably grind the coffee in the store.

It is OK to Pack the Coffee

You sure don't want to tamp it down like you would in an espresso machine, but I do lightly pack the coffee into the pot. I probably fit an extra teaspoon or more in there this way.

You know you packed it too tight if the pressure release valve starts whistling.

How Much Heat?

I've read all sorts of ideas related to applying heat…

I crank the stove up all the way. Right before the coffee starts flowing, you'll recognize the sound; I turn it down to about half heat. It brews quite a bit slower than at full heat and it seems to taste better. I haven't really done any sort of blind taste test here, though.

Light or Dark Roast?

This is more about personal taste than anything else. I prefer to use a lighter roast. I've been buying huge, cheap bags of Sam's Club brand breakfast blend coffee. We're happy with the taste and the price.

We managed to buy a huge bag of Sam's Club French Roast coffee. We really didn't know our coffee lingo at the time. We did not know that French Roast was the darkest over-burnt coffee. I feel it is way too dark. It has been handy to have around, though. I sometimes throw a few spoonfuls in with the breakfast blend if we want a darker-tasting coffee.

Is it Hard to Keep Clean?

My old friend, "The Internet," tells me that the coffee tastes better when the walls of the pot are coated with coffee oils. I don't actually know if this is true or not.

I rarely use any soap to clean it, though. As long as I use the pot every other day or so, I just take it apart and rinse it before each use. Just make sure you empty and rinse the top of the pot out after you use it. If you don't you will have scrub out the dried out, caked-on day old-coffee.

How Does it Compare to Proper Espresso?

I have absolutely no idea. I do know that the Moka Express produces a much stronger, more caffeinated brew than a drip machine. They are also much cheaper than a proper Espresso machine and seem easier to maintain.

I've been fairly unhappy with the battery life of my G1. With the stock 1150mAh battery, I am lucky to get 12 hours out of it most of the time. I was once able to use 49% of my battery in an hour and a half by driving to a doctor's office and surfing the web in the waiting room… Bluetooth was at the top of the list of battery consumers, so be sure to turn that off if you don't need it.

I found a cheap 2300mAh battery at dealextreme.com. I've ordered lots of cheap gadgets and cables from them in the past, so I figured it'd be worth spending $11.22 at this battery to try it out. They ship their items, slowly, from Hong Kong. I think it took about 3 weeks to get the battery.

How long does it last?

I haven't actually run it down to empty yet. The first day I had it, I ran it down to 25% in a little over 19 hours. I had everything enabled except Bluetooth. I watched a little over an hour of video, listened to some SHOUTcast radio with Tunewiki, played a bunch of games, read email, and surfed the web. I tried to use the phone as often as I could. That last 25% should easily have put it over 24 hours, but I was tired and I needed a full battery for the next day.

Just how big is that thing, anyway?

I looked at a lot of batteries. Nobody seems to give dimensions. Unfortunately, I don't have anything terribly accurate like a caliper to measure this thing with. I did come up with an easy way to convey the difference in thickness between the stock battery cover and this new battery cover.

The top part of the battery cover that sits over the camera is probably identical in thickness to the stock battery cover. Just below the camera it gets thicker and it maintains that thickness all the way to the bottom of the phone. I was able to stack 3 pennies on the shallow side of the cover and it came out pretty flush with the deep side. My good friend Wikipedia tells me that a United States Penny is 1.55mm thick.

My math tells me that the thickness of my phone has increased from 17.1mm to about 21.75mm. That puts it just a smidge under the thickness of my old Treo 650. I think I can live with that.

Is it heavy? Does the cover fit well?

I may not have a caliper handy, but I do have access to a shipping scale! The stock battery came out at 25g; the beefy new battery came out at 45g.

The first time I put the cover on the phone it didn't feel like it was going to fit. I may have had to press it on a little harder, but it fits very snugly with no gaps anywhere.

Update 2010-01-06:

My phone has mostly been sitting idle the last few days and I forgot to put it on the charger last night. The battery use applet is showing that I've been unplugged for a few minutes shy of 39 hours, with 29% capacity remaining. The three biggest battery wasters seem to be Cell Standby, Bluetooth, and Phone Idle, all clocking in at 29%. Wi-Fi clocked in at 9%, Voice Calls at 6%, and Display at 2%. I was on the phone for about 45 minutes.

All the power-wasting goodies were turned on—GPS, Bluetooth, Wi-Fi, and Sync. It probably helps that I live about a quarter of a mile from a cell tower.

I recently upgraded to Mozilla Thunderbird 3.0. That got me thinking that now might be a good time to clean up my local mail folders. All of my mail from the past few years is stored on my IMAP server. I still have a few gigabytes of old mail from my old POP3 days stored in Thunderbird's Local Folders.

I decided that now might be a good time to do some spring cleaning and not carry around my old POP3 mail anymore. I figured it is also a good time to store this current copy of my old mail with my long-term backups.

My Old Friend rzip

I have been using rzip for quite a few years. Its job is to find and encode large chunks of duplicated data over very large distances, up to 900 MB. Once that is complete, it runs the resulting data through bzip2. For large datasets, I've found it to be much faster than `bzip2 and it usually results in an archive that is about 30% smaller than just using bzip2.

Unfortunately, rzip can’t operate on pipes. All of my automated backup scripts run along pipelines, usually from tar to bzip2 to gpg. They never touch the disk unencrypted, which probably isn't always helpful.

My New Friend lrzip

I recently discovered Con Kolivas' lrzip. lrzip takes rzip a couple steps further. It lets you choose a compressor other than `bzip2 for the second stage of compression. It can also be used in a pipe.

Unfortunately, when used in a pipe it generates a large temp file. This can be a problem if you are trying to generate a large archive and don't have a lot of free disk space, or if you don't want unencrypted data being written to the disk.

Some Benchmarks

I compressed the tarball of my .thunderbird directory every way I could think of that made sense. The default settings for lrzip kept erroring out on me at about 30%. I had to use the -w switch to reduce the window size from 20. I chose 12, which should be about 30% higher than rzip's window.

              Size    Minutes    Ratio
              (MB)
uncompressed  5761         na    1.0:1
lrzip zpaq    1207        265    4.7:1
lrzip lzma    1262         60    4.5:1
lrzip bzip2   1401         27    4.1:1
rzip          1362         20    4.2:1
lzma          1441         97    4.0:1
bzip2         1748         38    3.3:1

Both rzip and lrzip achieved a smaller file size in less time than bzip2. lrzip with zpaq is over 13 times slower than rzip for a savings of 155 MB, or about 12%.

Why Would Anyone Wait for zpaq to Finish?

Most of the time it isn't worth the wait. I'm a huge fan of smaller backups. Backups become significantly more expensive every time a single backup has to span a second (or third, or fourth…) piece of media. It's another floppy, CD, DVD, Blu-ray, tape, hard drive, or flash drive to have to manually swap around and keep safe.

I like flash drives for my personal backups. I have too many CDs and DVDs that are unreadable. I've accidentally run an old compact flash drive through the laundry and it still worked. I'm sure all flash drives won't survive that, but they do tend to be very durable.

Unfortunately, lrzip with zpaq did not get the file size down enough for the archive to fit on the backup flash drive that I keep around the house. Another 100 MB or so would have done the trick and would save me quite a bit of effort.

Which One Should You Use?

For most archives I would probably just choose bzip2. It does a very good job, and a decompressor is always very readily available.

For almost every very large archive, I will definitely be sticking with rzip. It is faster and more space-efficient than bzip2. It is also easier to find than lrzip; my Ubuntu machine has an rzip package available in apt.

I will be sure to keep lrzip with zpaq in mind, though. Sometimes an extra couple hundred MB will save the time, effort, and cost of a second piece of media. The other downside to zpaq is that decompression is also very slow as well.

Update: I very much doubt that anyone should be using this anymore. The discard mount option has been working properly for years. In fact, you probably should use hdparm to manually TRIM your drive. You should be using fstrim. It is available in the software repositories of most recent Linux distributions.

The wiper.sh script that ships with hdparm 9.27 does not work well with the Intel X25 drives. The call to hdparm will fail if it is passed in more than 512 ranges of sectors. I made a quick and dirty modification to the wiper.sh script so that it makes multiple calls to hdparm with 500 ranges each. I've only been running it for a few days, but it seems to be working just fine so far.

I also added a --yes command line switch so that I could more easily call it from cron.

wonko@zaphod:~$ sudo ./wiper-dangerous.sh --commit --yes /

wiper-dangerous.sh: Linux SATA SSD TRIM utility, version 2.5-dangerous, by Mark Lord.
Preparing for online TRIM of free space on /dev/sda2 (ext4 mounted read-write at /).
Creating temporary file (3091782 KB).. 
Syncing disks.. 
Beginning TRIM operations..

/dev/sda:
trimming 6183568 sectors from 159 ranges
succeeded
Removing temporary file..
Syncing disks.. 
Done.

The script also now has a dependency on Perl. Feel free to download a copy of my wiper-dangerous.sh, but I make no promise that it won't eat your data!

I've been carrying a small roll of duct tape in both my wallet and laptop bag for years. I've been taking advantage of the fact that duct tape is the same width as a business card. You just roll some fresh duct tape around an old business card ten or twenty times and you'll get a small roll of duct tape that fits right in your wallet.

A year or two ago I found a roll of cloth tape in an ancient first aid kit that we had lying around the house. The first thing I noticed was how similar it is to duct tape. It is fabric reinforced like duct tape, but it is more flexible and less sticky.

It does a great job holding cables together, and it doesn't seem to leave behind the nasty mess duct tape leaves behind when you remove it a few months later. Cloth tape also works great for labeling things; it takes Sharpie ink very well.

The width of cloth tape does not line up as well with a business card as duct tape does. I've come up with a better idea, though. I now wrap three kinds of tape around the length of a business card: duct tape, cloth tape, and electrical tape. The three added together are slightly too big for a business card, so there is a tiny bit of overhang.

For years I have been keeping a small font printout of my important SSH and GPG private keys hidden and locked away for safe-keeping. I have lost enough floppies, CDs, and DVDs to bit rot, so I do not have much trust in them for the long-term storage of something this important.

The hard copies are nice, but I sure don't want to have to manually type in an error-free copy of my private keys. That is why I now print two copies of each of my keys. One copy is text, the other copy is a QR Code.

QR Codes are popping up all over the place lately. Some of the websites hosting Android software that I use have QR Code images on them, so you can just snap a picture of the code with your smart phone and be taken straight to the website. This one will bring you to this blog:

An alphanumeric QR Code can contain up to 4,296 characters. This was plenty for my SSH and GPG private keys.

Where Should I Store My Backup Keys?

That would depend on your level of paranoia. You could store them in your safe deposit box at your bank, in a safe at home, under your mattress, or you could bury it in your back yard.

Why Bother Storing a Hard Copy of your Keys?

I am mostly worried about losing my GPG private key. I have gigabytes upon gigabytes of backups and important data encrypted with my key. Completely losing that key would turn all that data into useless bits.

Losing my SSH key(s) wouldn't have nearly as much impact. There would just be a few hosts that I would have trouble logging into. The extra effort to print the SSH key along with the GPG is minimal, though.

I updated the firmware on my X25-M today. It was a completely pain-free procedure. Just burn a CD, boot from the CD, say yes a few times, and reboot.

I added a new set of Bonnie++ benchmarks to my first set of Intel X25-M benchmarks.

Unfortunately, the new benchmarks aren't a good comparison. I have been using the drive for a month now and I probably have 100 GB worth of rewrites on the drive.

There were two obvious improvements. Sequential input is up 30%, and CPU usage during random seeks is down 12%.

I can't wait to try running another benchmark after I get some TRIM support in my kernel.

The single most disappointing application that came installed on my T-Mobile G1 was the default email application. The very first thing I noticed was the horrendous IMAP support. I have a very well organized mail account. I might have a have dozen or so folders at the top of my folder hierarchy. The Android email application just flattens out all of my 100+ sub folders out into a huge, difficult to navigate list.

K-9 Mail

My search for a better mail app lead me to K-9 Mail. K-9 is a fork of the stock Android email application with quite a few improvements.

K-9 still flattens out my folder list but lets me choose which folders I would like to display and sync. Fortunately, I really only want quick access to five or six folders while I'm on the go.

Push Email

The beta release of K-9 supports the IMAP IDLE feature. This allows it to stay connected to the IMAP server so that it can instantly be notified when a new message arrives. This is much better for me than waiting up to 5 minutes to see a new message arrive.

Improved Message List

K-9's message list view is a huge improvement over the stock mail client. The view is much more condensed and looks like it fits about twice as many messages on the screen at the same time.

So far there is only one ChatterEmail feature that I miss. It had a summary view that displayed emails from multiple mailboxes on one screen in chronological order. Each message was color coded so that you could very easily tell which account each message belonged to. I saw a few mentions of ChatterEmail in the K-9 issue tracker, so I know I'm not the only one who misses this feature.

I finally broke down and replaced my long in the tooth Palm Treo 650 with a T-Mobile G1 (a.k.a. HTC Dream). The only major complaint I had with the Treo was stability. It used to like to reboot itself about once each day…

So far I'm mostly working on replicating all the functionality that my Treo had. Almost everything on the G1 meets or exceeds the capabilities of the software I had on my Treo. Some software has both improvements and regressions compared to the Treo, but overall I'm pretty happy so far.

Cyanogen 4.1.999

I only ran the stock firmware long enough to install the Cyanogen firmware, so I can't talk about all the specific details that are different from the stock firmware.

There are some interesting improvements behind the scenes, though. The Cyanogen firmware uses compcache and Con Kolivas’s BFS process scheduler. I'm already a heavy user of compcache on my laptop, and I've been testing the BFS scheduler as well.

I've really only had one issue so far with the 4.1.999 firmware. Every once in a while all the icons disappear from the launcher and the phone has to be rebooted to get them back. It seems to be a known issue with the experimental firmware. I doubled the size of the compcache swap space and I didn't see the problem for a few days. I can only assume that the race condition is less likely when the phone is more responsive.

Neat Things I'd Like To Do With Android

I would like to get rsync and cron running on it. I think it would be very nice to have automated wireless backups that are always likely to be in my pocket. I'm hoping to be able to do it without having to rely on having Debian on my SD card.

OpenVPN might be interesting, but I don't currently have a real need for it. A few years ago this would have been a top priority for me, though.