Machine Check Exception

Along with playing with gcj, I’ve been playing around with gcc 4.0. I don’t expect to use it on any production code in the near future, but I want to know what’s coming. In general, reports have been mixed, at least on x86 and AMD64 architectures. I’m still waiting for my university to get the site-licensed copies of Tiger, so I can’t test how GCC 4.0 performs on the iBook G4. I anxiously await benchmarks on the PPC architecture.

In the meantime, I wanted to try out what I consider to be the most interesting addition to GCC: the loop auto-vectorizer. The possibility of using SIMD instructions on CPUs to accelerate loops sounds very promising. This optimization is NOT enabled by default, and while trying to figure out how to turn it on, I ended up reviewing the SIMD options on x86/AMD64:

• MMX - Eight 64-bit wide registers. Integer ops only.
• 3DNow! - Eight 64-bit wide registers. Integer and single-precision float ops.
• SSE - Eight 128-bit wide registers. Single-precision float ops only.
• SSE2 - Eight 128-bit wide registers. Integer, single and double-precision float ops.
• SSE2/AMD64 - Sixteen (!) 128-bit wide registers.

To actually get the vectorization optimizations to be used, you need to use -OX -ftree-vectorize, where X = 1, 2, or whatever your favorite optimization level is. If you leave out -O, the vectorizer will be skipped. On AMD64, this is enough to get the vectorizer going. On x86, you might also need -msse or -msse2. For added fun, you can throw in -fdump-tree-vect -ftree-vectorizer-verbose=8 and check out the .vect file for a detailed explanation of what the compiler is doing when it analyzes loops.

I’ll spare you crappy benchmarks for now. My test code is basically the simplest array loop possible, and totally meaningless. I’m hoping to turn this compiler loose on our FORTRAN code and see what it does with our array loops.

(Update) One thing I will say about performance: With SSE2 on the Opteron, the benefit of vectorizing simple loops appears to be linear in the number of variables you can pack into one SIMD register. So, for 16-bit shorts, the speedup is 8x, for 32-bit floats it is 4x, and for 64-bit doubles it is 2x. This makes sense, but I was surprised to see it work out almost exactly. A little hunting in the gcc manual showed this is because on AMD64, gcc actually uses the SSE registers by default, even for scalar floating point math, and just wastes most of the register.

Having finally found a combination of hardware that would finally POST correctly, it was time to move on to the operating system.

Thankfully, there are quite a lot of nice Linux distributions for AMD64 these days. You can basically use any of the major ones, like Debian, Fedora, Mandrake or Gentoo, and many lesser known ones.

I run Gentoo on my other computers, so I opted to use that distribution. I went to Gentoo AMD64 project page and got familiar with the installation. It pretty much works like any other installation, as you might expect.

….once you get the system to boot, of course. I burned a copy of the Gentoo AMD64 Live CD and tried to boot it. The kernel would start to load, then panic and die with a “Machine Check Exception” (hence the inspiration for my blog name). The MCE crash was pretty confusing, and this early in the installation, quite demoralizing. I went round and round on it for a while, trying different editions of the Gentoo Live CD with no luck.

Fortunately, Opterons can excute older 32 bit code just fine. I dropped an x86 Live CD in, which booted flawlessly. Of course, I don’t want to handicap my Opteron by pretending it is an x86 chip, so this really wasn’t a permanent solution. A couple more hours of Google pinball and I discovered the nomce kernel option. Just pass that to the booting kernel, and it will skip the machine check. (Apparently MCEs are used to detect certain kinds of hardware faults. I was never able to decipher this error, and the system works fine anyway, so I blame a motherboard quirk.)

Adding nomce to the kernel options at boot time fixed my first problem, and I was able to partition, format, and install the base Gentoo system pretty quickly. Then it came time to restart and boot the hard drive directly using the GRUB installation I had performed. No go. All I saw was a blank screen with a blinking cursor. No matter what permutation of BIOS options and boot loaders (GRUB, LILO, EXTLINUX), the result was the same.

Eventually I found the problem purely by accident. When I had first installed Gentoo, I booted the Live CD off of a DVD/CD-ROM drive connected to the motherboard with one of those old 40 conductor IDE cables. I only had one 80 conductor IDE cable which I used for the hard drive. I figured worst case would be the CD-ROM drive would operate slower than if I had used a newer cable. On my way home from some errands, I stopped by a computer shop and picked up a proper 80 conductor IDE cable. When I swapped out the old cable and put in the new one, the hard drive magically was able to boot. This still makes no sense to me. The cable in question was only connected to the CD-ROM drive and not the hard disk, which was the device actually behaving strangely.

Overall, this was a pretty painful computer commissioning experience. I’ve never had this many different things go wrong for seemingly inscrutable reasons. But now it all runs, and I didn’t have to fix anything with duct tape.

Getting the hardware all together and working was the worst part of building the Opteron server. I started with all eBay parts, one of which was bad. However, I had no idea which part was broken, and over the course of a week, I basically purchased a duplicate of each part until I got a working set, then returned the rest. Here’s what I learned:

Opteron heatsinks are very, very scary to attach. A metal plate and mounting bracket sandwich the motherboard and are bolted together. Then you clip the heat sink to the bracket and flip this lever to apply quite a lot of pressure between the heatsink and the chip, and clip the lever to the bracket to keep it from springing back. Thankfully the Opteron has a metal casing on top which contacts the heatsink, so applying all this force seems a little less crazy than the Athlon XP CPUs that actually have the silicon exposed. Still, there was some sweating. Little did I know that I would remove and reattach that heatsink a dozen times while moving CPUs during troubleshooting.

Don’t necessarily believe your motherboard manual. At one point I had both the RioWorks HDAMA and the Tyan K8W motherboards in front of me. The HDAMA uses a Phoenix BIOS which produces a “four digit” beep code out of quick beeps followed by long pauses. I received a beep code of 1-1-1-1 when I put my (unknown to me at the time) defective RAM in. According to every document I could find, that beep code does not exist. Moreover, I got the same beep code without any RAM in the system at all. This incorrectly led me to believe the entire motherboard was hosed. No so–that board is now happily humming away in the corner of my apartment.

The K8W motherboard was acquired to figure out if the HDAMA was toast. I put the defective DIMM into it, and there were no beeps at all. None. I take the memory out, still no beeps. Now I’m thinking, “Oh crap. The CPU must be broken.” Many permutations later, I discover empirically that the K8W will not even beep unless there are two DIMMs present. The manual claims that the motherboard supports both single and double channel memory configurations, but I was unable to even get a beep, much less a full POST, with a known good CPU and one good DIMM. I could get the “your memory is broken” beep code if I put the working DIMM in with the defective one, however. Very annoying.

Eventually I figured out it was the RAM that was bad (after trying just the working DIMM by itself in the HDAMA mobo) and everything fell into place. My confusion about the beep codes on these motherboards was leading me to believe that I had two simultaneous failures, or I was cursed, or something. I was starting to lose faith in rational problem solving. The moral of the story is: Don’t expect more accurate documentation just because your motherboard costs more than the cheap stuff you usually buy.

Oh, and somewhere along the way, I picked up an Antec True550 EPS12V PSU. I highly recommend it. It is much quieter than the first EPS12V power supply I got (going up for sale on eBay today) and has a motherboard connector so you can monitor its speed. It will also control the speed of your other case fans, if you prefer, to keep the noise down.

Once I figured out what parts I wanted, the next major problem was figuring out where to get them. There isn’t a whole lot to say on the subject, but my tips are:

Don’t buy your initial components used. Or rather, don’t buy them used unless you know someone with Opteron hardware who can test them in case they don’t work. We all know used parts on eBay are more likely to be broken, but usually you have spares laying around swap in or can get some cheaply for testing. I underestimated the importance of this and found myself sitting in front of a non-booting Opteron server with no idea which of the CPU/RAM/Motherboard/Power supply components I had purchased were broken. (More on that tomorrow.) Not only will new components be more likely to work, they will be easier to return if they don’t. Once you have a working core system, then you can be really cheap and buy add-ons used from eBay later. Then you will only be debugging one part at a time, and not your entire system.

You’ll probably have to buy all of your parts online. Normally, I buy all my computer parts online, but it is easier to return stuff to local stores, and you can pick up items immediately, which can be invaluable while troubleshooting. This proved to be a huge issue for me when everything went wrong.

CompUSA and Best Buy don’t carry Opteron server parts (even the EPS12V power supplies). Fry’s Electronics did carry Opterons, though much more expensive than from Newegg. They claimed to carry registered DDR memory, but keep their stock so low, I was unable to get any when I needed some. (Fry’s, however, does accept returns without restocking fees, which is very nice for a project like this.) So, I ordered most of my parts from a combination of Newegg (Good Idea!) and eBay (Bad Idea!). Just be prepared for Newegg’s 15% restocking fee if you decide to return something.

If you do go the eBay route, make sure to keep Newegg open in an adjacent window to compare prices. It seems lots of sellers like to post things for more than it costs to get them from Newegg. I was less than amused to discovered that the power supply I bought from eBay was drop-shipped by the seller direct from Newegg, and I had paid him a $10 premium for this “service.” Newegg had forgotten to include the EPS12V keyword in the product description, so my inital searching pass had missed it. Such is the world of ecommerce, I guess.

Even if you are an experienced “computer enthusiast,” there’s a lot of new terminology to learn if you are going to figure out what kind of Opteron you want. I would strongly suggest you hit up Google to learn the basics. Short version: You will probably need to buy all new parts:

CPU

Though I was building a single CPU system now, I wanted the ability to add a second CPU in the future. That only leaves the Opteron 2xx or 8xx model CPUs. These are the socket 940 packages (not to be confused with the socket 939 or 754 Athlon64s), which means your motherboard options are going to be limited and expensive. Keep an eye on the CPU stepping numbers. AMD redesigned some of the slower Opterons (like the 1.8 GHz model 244) to accept DDR-400 memory, whereas the older revisions used to only take DDR-333. Check out this thread on Opteron memory speeds over at the AMD forums for more details on the CPU stepping numbers.

Memory

Your standard DDR memory won’t work. You will need to buy Registered DDR (which is also usually ECC as well). Beware: From other people’s experiences as well as my own, this is the part most likely to cause you problems. It’s probably a good idea to spend more on this item to avoid headache later. Opterons are dual channel, so if you decide to get two DIMMs, you will double your memory bandwidth. That’s my eventual goal, but I decided to start with just one DIMM for now.

Motherboard

There are lots of issues to worry about here. In my case, I wanted PCI-X and NUMA support. Since the memory controller is built right onto the Opteron, in principle you can have your memory bandwidth scale up with the number of CPUs. Each Opteron will have its own banks of memory which it can access at full speed, but they can also access each other’s memory through the HyperTransport bus. This is a huge win for scalability, and Linux is NUMA-aware these days, so it will try to keep processes and their allocated memory on the same CPU if possible. My options were the RioWorks HDAMA, the Tyan Thunder K8W and the Iwill DK8X.

Another issue I found that surprised me were the bus connections of the on-board devices. I found some motherboards would put devices on the slow 32-bit/33 MHz PCI bus, even though there was a faster PCI-X bus available. I was mostly concerned about the GigE and SATA controllers (if present), since I eventually plan to setup a RAID to feed data to my other nodes over GigE. Here are the bus connections for my candidate motherboards (as far as I can tell):

Strangely enough, in the end, I went with the worst motherboard on that chart, the HDAMA, because it was available used for cheap. I already have a PCI-X capable GigE card and planned to get a separate SATA controller anyway, so this didn’t matter so much to me.

Power Supply

You will need an EPS12V power supply. This looks pretty much like an ATX power supply, but the motherboard connector has 24 pins (instead of 20) and it has an extra 8 pin 12V connector. These are not to be confused with the ATX12V supplies which are different, though they may have similar connectors.

Case

Yes, even your case probably needs to be replaced. If you are going with a dual CPU motherboard, chances are it will follow the Extended ATX (EATX) form factor. These are 12″ by 13″ boards that usually need a special, large case. I was lucky and my massive Antec full tower case (purchased before my mini-ATX fetish) has mounting holes for an EATX motherboard. Even then, it was a tight fit. I had to remove the 3.5″ hard drive mounting bracket to make room for the CPU heatsink/fan.

Scared yet?