Machine Check Exception

I’m a strong believer in the potential for Firewire 800 to make it feasible for people to have large disk arrays on a budget. There’s enough bandwidth there (plus the ability to chain drives) to run a 1 TB RAID5 system at high end-consumer prices. All the pieces are there: cheap disks, multi-bay firewire enclosures, the hotplug support in firewire, decent software RAID support in Linux and the Logical Volume Manager. Once you figure out where all the pieces go, it’s actally not too hard…

Having tried doing this for 6 months and dealing with the problems, I’m amazed at how much can go wrong. So far LVM2 and RAID have been flawless. Haven’t lost a single byte of data. But I find the communication between the PC and the firewire to be unbelievably fragile. If you try doing too much I/O, the firewire devices jam up, causing the scsi subsytem to slow to a crawl and throw a bunch of errors, and sometimes bomb out. I had these problems earlier, but thankfully by telling the sbp2 driver (which implements the firewire protocol to access block devices) to be very conservative and talk to only one device at a time, I was able to get things running more or less correctly. Any stalls I experienced under high load were not permanent, and only needed some time to reset and clear out.

However, the last two days, I’ve been converting two RAID1 pairs into a combined RAID5 set. I moved all the data off the RAID1 pairs, then wiped their settings and had mdadm create a new RAID5 set out of them. Much like when you make a new RAID1 set, a new RAID5 set will need to do many hours of I/O to get all the disks into a consistent state. Unlike before, though, something about the increased load (now talking to 4 drives at a time instead of 2) caused the firewire system to lose contact with drives within a few minutes. I tried a bunch of different combinations, eventually ending up with sbp serialize_io=1 (like I had before), elevator=cfq (specifying an alternative I/O scheduler), and finally using the -f option with mdadm to make it stop being clever.

I’m still trying to understand the “clever” part. By default, when mdadm makes a new RAID5 array out of N disks, it creates a new array of N-1 good disks, 1 (non-existant) faulty disk, and one “hot spare” (really your last disk). It then proceeds to resync the array like the last disk had just been swapped in. The manpage claims this is faster than the normal resync process. My best guess is that the fast method only reads from the first N-1 drives and only writes to the last drive. The “normal” method (which I assume just reads normal blocks and writes parity blocks) would require reading and writing to all drives since the parity blocks are spread everywhere in RAID5. It seems that the fast method was somehow beating on the last drive too hard and making the firewire communication fail. Switching to the normal method avoided this problem, and appears to be going just as fast as the “fast” method did in the 60 seconds before locking up the drives.

(Update, Oct 9: I spoke too soon. I have been unable to find any arrangement of cables, sbp2 module options or disk configuration that will allow me to reliably write to a RAID5 disk set for any extended period of time without a drive or two getting stuck. I even reflashed the firmware on my Initio 2430L to a newer version just in case there had been stability fixes. I have accepted my fate and reverted back to a RAID 1 configuration.)

Still, the question remains, why is this possible? Why is it that I have to be so careful about the read/write usage patterns of my disks, lest I lose communication with them? I have no good answers. It either has to be hardware or software, and I’m starting to wonder if the problem is that there are a lot of really crappy Firewire-to-IDE bridge chipsets out there. I know that with USB enclosures, I’ve had nothing but problems, partly due to really poor error handling in the usb-storage drivers, but partly due to buggy USB hardware. I had assumed that Firewire would be better, due to a slightly more professional audience. (I’ve seen Firewire drives used in a lot of digital video labs and such.)

Perhaps that faith was misplaced. It’s not hard to imagine that low-end PC hardware has abysmal quality control, but I’ve never been able to tell how much more upscale you have go to get stuff that works. Is it only when a manufacturer knows that you can call them at 3am and they have fix your problem (like if you purchase high-end hardware with service agreements) that they will bother doing comprehensive testing of hardware?

Recently I was asked if it would be possible to setup a network with a NFS server, a dozen compute nodes, and several workstations. The NFS mounts need to be available to all the compute nodes and the workstations. A simple way to do this is to put all clients on a private segment of the network and expose the NFS mounts only to this private segment. The server can have a second interface attached to the Internet, providing the workstations with access to the public network via NAT.

This scenario was impossible in our case because the workstations are spread between two buildings, with no simple way to get them all on the same private network. Exposing the NFS server to the world would be a foolish way to solve the problem. Instead, we need to build a virtual private network that would emulate the previous scenario by tunneling network traffic between the clients and servers through encrypted connections. (You could probably tunnel just NFS using some other means, but the VPN approach gives you everything in one shot.)

I had never setup a VPN before (at least not since Windows NT 4, and I didn’t know what I was doing then), so as a warm up I started with a simpler scenario:

I have two machines, a server and a client. The server runs Gentoo Linux, and the client runs Mac OS X 10.4. I want a VPN that will allow me to safely NFS mount a server directory on the client. The tool that does most of the work is OpenVPN, a really nice package that is available for Linux, Solaris, Windows, Mac OS X, and *BSD.

Step 1: Server Config

On the linux server, I first installed OpenVPN with the usual “emerge openvpn”. Next, I decided to go the simple route, where there is a shared key between the OpenVPN server and client (rather than the full certificate system). (I’m loosely following the Static Key Mini-HOWTO here.) To make the key, run:

openvpn --genkey --secret static.key


On Gentoo, you will need to make a directory under /etc/openvpn with some name (like “myvpn”) that will hold the secret key and the configuration file. (This is how the Gentoo initscripts allow you to have multiple VPNs) After you copy the secret key to /etc/openvpn/myvpn/, you need to create the configuration in a file called /etc/openvpn/myvpn/local.conf. The contents are similar to the HOWTO, but I switched to TCP to allow the client to be behind a NAT router when connecting:

dev tun
ifconfig 10.8.0.1 10.8.0.2
secret static.key
proto tcp-server
comp-lzo

This also adds compression to the link.

You will also need to edit the /etc/exports files and add a line like:

/home 10.8.0.0/24(root_squash,async,insecure,insecure_locks,rw)


Now you should restart the NFS server and start the openvpn service.

Step 2: Firewall

If you have a NAT firewall, you should configure it to forward port 1194 to your NFS server.

Step 3: Client

Someone has nicely compiled the OpenVPN software, the kernel drivers, and a little GUI that sits in the menu bar all for Mac OS X. The package is called Tunnelblick. Download and install it. Then, before you start the app, go create a file called ~/Library/openvpn/openvpn.conf. The contents should look like:

remote your.server.ip.address
dev tun
ifconfig 10.8.0.2 10.8.0.1
secret static.key
proto tcp-client
comp-lzo

Also, copy your static.key from your server to the ~/Library/openvpn/ directory.

Now start the Tunnelblick application. You will probably also want to start the Console app (in Utilities) so you can see status/error messages from Tunnelblick.

The app will put a little icon that looks like a tunnel in your menu bar. Click on it, and select “Connect”. If all goes well, you should see the tunnel icon turn into a “light at the end of the tunnel”.

Step 4: Testing

Now the client has a new network interface called “tun0″ with a local address of 10.8.0.2. The other end of the tunnel (attached to the server) has address 10.8.0.1. You should be able to ping these addresses from both computer.

Now comes the fun part. Switch to Finder, and go to Go/Connect To Server. Now you can type nfs://10.8.0.1/home, and (hopefully) you will now see your NFS mount appear.

Note that all of your normal internet traffic will not go through the VPN. Only traffic destined for the VPN IP addresses will be sent through the encrypted tunnel, as this is just a point-to-point link. Later, I will describe how to go one step further and bridge the VPN onto a private LAN at the server, making it look like your laptop is plugged into the switch next to the server.

Today I finally finished sorting through all the mail that had accumulated while I was gone. The credit card tally was 30 credit card applications received in 60 days (50 if you only count mail service days). That’s an amazing rate and one which makes me wonder about the economics of this. We’re talking about a colossal amount of junk mail with a non-trivial expense. Clearly, credit card companies must make a great deal of money on a customer if such paper spam is actually profitable.

As I understand it, credit companies make money in four ways:

• Yearly fees: Something like $15 - $30 per year, if present. However, it’s often zero and usually can be waived if you ask.
• Merchant processing fees: These are somewhere between 1.5 and 3% of every transaction right off the top.
• Interest: Somewhere between 8% and 20% of the remaining balance, based on my sample.
• Cross-promotional tie ins: About a quarter of my applications were related to other services like eBay, Disney or airlines. Plus, my credit card bills always include ads for really stupid junk, like collectable plates and Bose clock radios. I honestly don’t know which way the money flows in this sort of deal. Maybe credit card companies pay other businesses for these cross-promotions as a way of distinguishing their cards in a ultra-competitive field.

Now, I haven’t paid a yearly fee or interest on a credit card in nearly 4 years, and my card has nothing to do with airline points, DisneyWorld or Paypal. That leaves just the merchant processing fees. So I took a look at a sample of 6 months of my credit card bills to estimate how much they make from these fees. Much to my surprise, they extract ~$250/year from merchants as a result of my purchases.

Not bad at all. Of course, if I’m getting 180 credit card applications per year, all of which I throw away, they must be pretty cheap to send or they would be eating into their profit margins. That still leaves interest fees as a possible cash cow to fund this orgy of mail. I have no idea what the average per-person revenue from interest fees is, and that might actually turn out to be more than the merchant fees. These days it’s probably more lucrative to make money on the purchases people can’t afford rather than the purchases they can afford.

Long ago, I had a functioning, early model Neuros. However, the innards were too old to take more recent firmwares and the internal battery has long since crapped out, so I decided to get another player.

Wanting something a little smaller, I decided to check out the world of flash players. My desired features were:

• Ogg playback — Most all of my music is in Ogg Vorbis format at quality 0, which makes the audiophiles cringe, but is perfect for a flash player where space is at a premium.
• USB Mass Storage support — I have Mac and Linux machines, and the best way to be accessible to both is to have a player which just acts like a USB disk. This also makes it easy to use the player for moving files around too. Most importantly, it means you are not chained to some manufacturer’s file transfer software.

Based on this, I decided to purchase an iRiver iFP-795:

It’s a 512 MB flash player with a monochrome, backlit display. It does MP3 and Ogg playback, FM tuner, and recording to MP3 from both a built in microphone and a line-in jack. The interface is a little complicated, but that doesn’t bother me as long as it is functional. Once I got the hang of it, I found it very quick to move around between options.

However, once the initial marvel of the small size and nice audio quality wore off, I was stunned to discover several shortcomings. Clearly I was very sloppy in my research, and mixed up the features of different flash players. It turns out the iFP-795 really fails both of my criteria for a music player.

First, the current firmware (1.25) can only play Ogg Vorbis files between 96.1 and 225 kbps. You pretty much have to encode at Vorbis quality 3 to get 96 kbps, which really kills one of the best reasons for Vorbis on a flash player. Quality 0 sounds remarkably good despite having an average bitrate between 55 and 75 kbps. That allows 50%-100% more music to fit on the player than you could get with equivalent-sounding MP3s. Not to mention all my music is already in q0, so I would have to upcode it and waste bits, or rerip.

Further experimentation reveals the player is very strict about this bitrate requirement. I encoded a batch of 12 files at quality 2.5 and about half of them were rejected, so it isn’t looking at the nominal bitrate stored in the header (which is the same for all 12 files). The firmware must calculate the average bitrate using the playlength information in the header and filesize, and then refuse to play the file if it isn’t > 96kbps. This is even more lame than I first thought, because Vorbis is highly variable, and so by default there is no absolutely guaranteed minimum bitrate at any quality setting. To get around this, I have to use the -m option with oggenc to force a minimum bitrate, further limiting the compression efficiency of Vorbis.

Second, the iFP-795 no longer appears to have a firmware which supports USB mass storage (iRiver calls this “UMS”) mode. The 1xx, 3xx, and 5xx models have a UMS firmware that enables the mass storage mode, but I was unable to locate a UMS firmware for the 7xx series. Having now seen several reviews talking about the UMS firmware on the iFP-795, I can only conclude that iRiver withdrew the firmware for some reason not explained anywhere on their site.

Even for those players with UMS support, there are numerous reports that the firmware downgrades the recording capability to 96 kbps MP3 (from a normal max of 320 kbps), which is fine for voice, but basically renders the line-in recording option pointless. I found it peculiar that this was not mentioned on the iRiver UMS firmware page.

iRiver does have a Mac version of their file transfer program, which I installed on my iBook. Looks cute but simple, and just acts as a window to drag and drop files and directories into. Of course, they forgot to allow you to drag multiple files already on the device into a new directory. You have to do it one at a time, all the while cursing iRiver for not just implementing USB mass storage support and letting me use the Finder, which has implemented this extremely complicated feature.

Having had enough of dragging the files one at a time to sort them on the device, I thought maybe I would try dragging them out of the iRiver Music Manager window to my desktop, sort them in Finder, and then move them back. Bzzt! No go. I am told that these files are “copy protected” and I cannot copy them back to my hard drive. What?! Some further searching on the iRiver site revealed this tidbit from their FAQ:

Q. Why can’t I upload my MP3 / WMA files from my iFP player?

A. Due to copyright protection laws that apply towards our technology, media files (MP3 / WMA files) cannot be uploaded from an iFP player to a PC. All other non-media files (documents, images, etc.) can be uploaded to a PC from the iFP player.

Frankly, that’s highly misleading if not completely wrong. Let me rephrase this for clarity:

Q. Why can’t I upload my MP3 / WMA files from my iFP player?

A. Given the exceptionally aggressive and litigous nature of the RIAA, we are utterly petrified of getting our pants sued off. Little Cindy could try to give her friend the latest Ashlee Simpson track stored on her iFP player, thus leading to the destruction of the music industry and (more importantly) costly lawsuits against us. We sincerely hope that by adding pointless restrictions like this, we will earn brownie points from the music industry and avoid their wrath as they thrash around, attacking everyone in sight. However, if you ask us why we have decided to penalize you, our customers, we will tell you that “it is the law” and hope you are stupid or confused enough to believe us.

Don’t get me wrong, this player has lots of nice features, but the two that I cared about have awful implementations. If returning the player was an option (it was a clearance item, so not possible), I would strongly consider it. However, I don’t know if the alternatives are much better…