Machine Check Exception

Getting the hardware all together and working was the worst part of building the Opteron server. I started with all eBay parts, one of which was bad. However, I had no idea which part was broken, and over the course of a week, I basically purchased a duplicate of each part until I got a working set, then returned the rest. Here’s what I learned:

Opteron heatsinks are very, very scary to attach. A metal plate and mounting bracket sandwich the motherboard and are bolted together. Then you clip the heat sink to the bracket and flip this lever to apply quite a lot of pressure between the heatsink and the chip, and clip the lever to the bracket to keep it from springing back. Thankfully the Opteron has a metal casing on top which contacts the heatsink, so applying all this force seems a little less crazy than the Athlon XP CPUs that actually have the silicon exposed. Still, there was some sweating. Little did I know that I would remove and reattach that heatsink a dozen times while moving CPUs during troubleshooting.

Don’t necessarily believe your motherboard manual. At one point I had both the RioWorks HDAMA and the Tyan K8W motherboards in front of me. The HDAMA uses a Phoenix BIOS which produces a “four digit” beep code out of quick beeps followed by long pauses. I received a beep code of 1-1-1-1 when I put my (unknown to me at the time) defective RAM in. According to every document I could find, that beep code does not exist. Moreover, I got the same beep code without any RAM in the system at all. This incorrectly led me to believe the entire motherboard was hosed. No so–that board is now happily humming away in the corner of my apartment.

The K8W motherboard was acquired to figure out if the HDAMA was toast. I put the defective DIMM into it, and there were no beeps at all. None. I take the memory out, still no beeps. Now I’m thinking, “Oh crap. The CPU must be broken.” Many permutations later, I discover empirically that the K8W will not even beep unless there are two DIMMs present. The manual claims that the motherboard supports both single and double channel memory configurations, but I was unable to even get a beep, much less a full POST, with a known good CPU and one good DIMM. I could get the “your memory is broken” beep code if I put the working DIMM in with the defective one, however. Very annoying.

Eventually I figured out it was the RAM that was bad (after trying just the working DIMM by itself in the HDAMA mobo) and everything fell into place. My confusion about the beep codes on these motherboards was leading me to believe that I had two simultaneous failures, or I was cursed, or something. I was starting to lose faith in rational problem solving. The moral of the story is: Don’t expect more accurate documentation just because your motherboard costs more than the cheap stuff you usually buy.

Oh, and somewhere along the way, I picked up an Antec True550 EPS12V PSU. I highly recommend it. It is much quieter than the first EPS12V power supply I got (going up for sale on eBay today) and has a motherboard connector so you can monitor its speed. It will also control the speed of your other case fans, if you prefer, to keep the noise down.

Once I figured out what parts I wanted, the next major problem was figuring out where to get them. There isn’t a whole lot to say on the subject, but my tips are:

Don’t buy your initial components used. Or rather, don’t buy them used unless you know someone with Opteron hardware who can test them in case they don’t work. We all know used parts on eBay are more likely to be broken, but usually you have spares laying around swap in or can get some cheaply for testing. I underestimated the importance of this and found myself sitting in front of a non-booting Opteron server with no idea which of the CPU/RAM/Motherboard/Power supply components I had purchased were broken. (More on that tomorrow.) Not only will new components be more likely to work, they will be easier to return if they don’t. Once you have a working core system, then you can be really cheap and buy add-ons used from eBay later. Then you will only be debugging one part at a time, and not your entire system.

You’ll probably have to buy all of your parts online. Normally, I buy all my computer parts online, but it is easier to return stuff to local stores, and you can pick up items immediately, which can be invaluable while troubleshooting. This proved to be a huge issue for me when everything went wrong.

CompUSA and Best Buy don’t carry Opteron server parts (even the EPS12V power supplies). Fry’s Electronics did carry Opterons, though much more expensive than from Newegg. They claimed to carry registered DDR memory, but keep their stock so low, I was unable to get any when I needed some. (Fry’s, however, does accept returns without restocking fees, which is very nice for a project like this.) So, I ordered most of my parts from a combination of Newegg (Good Idea!) and eBay (Bad Idea!). Just be prepared for Newegg’s 15% restocking fee if you decide to return something.

If you do go the eBay route, make sure to keep Newegg open in an adjacent window to compare prices. It seems lots of sellers like to post things for more than it costs to get them from Newegg. I was less than amused to discovered that the power supply I bought from eBay was drop-shipped by the seller direct from Newegg, and I had paid him a $10 premium for this “service.” Newegg had forgotten to include the EPS12V keyword in the product description, so my inital searching pass had missed it. Such is the world of ecommerce, I guess.

Even if you are an experienced “computer enthusiast,” there’s a lot of new terminology to learn if you are going to figure out what kind of Opteron you want. I would strongly suggest you hit up Google to learn the basics. Short version: You will probably need to buy all new parts:

CPU

Though I was building a single CPU system now, I wanted the ability to add a second CPU in the future. That only leaves the Opteron 2xx or 8xx model CPUs. These are the socket 940 packages (not to be confused with the socket 939 or 754 Athlon64s), which means your motherboard options are going to be limited and expensive. Keep an eye on the CPU stepping numbers. AMD redesigned some of the slower Opterons (like the 1.8 GHz model 244) to accept DDR-400 memory, whereas the older revisions used to only take DDR-333. Check out this thread on Opteron memory speeds over at the AMD forums for more details on the CPU stepping numbers.

Memory

Your standard DDR memory won’t work. You will need to buy Registered DDR (which is also usually ECC as well). Beware: From other people’s experiences as well as my own, this is the part most likely to cause you problems. It’s probably a good idea to spend more on this item to avoid headache later. Opterons are dual channel, so if you decide to get two DIMMs, you will double your memory bandwidth. That’s my eventual goal, but I decided to start with just one DIMM for now.

Motherboard

There are lots of issues to worry about here. In my case, I wanted PCI-X and NUMA support. Since the memory controller is built right onto the Opteron, in principle you can have your memory bandwidth scale up with the number of CPUs. Each Opteron will have its own banks of memory which it can access at full speed, but they can also access each other’s memory through the HyperTransport bus. This is a huge win for scalability, and Linux is NUMA-aware these days, so it will try to keep processes and their allocated memory on the same CPU if possible. My options were the RioWorks HDAMA, the Tyan Thunder K8W and the Iwill DK8X.

Another issue I found that surprised me were the bus connections of the on-board devices. I found some motherboards would put devices on the slow 32-bit/33 MHz PCI bus, even though there was a faster PCI-X bus available. I was mostly concerned about the GigE and SATA controllers (if present), since I eventually plan to setup a RAID to feed data to my other nodes over GigE. Here are the bus connections for my candidate motherboards (as far as I can tell):

Strangely enough, in the end, I went with the worst motherboard on that chart, the HDAMA, because it was available used for cheap. I already have a PCI-X capable GigE card and planned to get a separate SATA controller anyway, so this didn’t matter so much to me.

Power Supply

You will need an EPS12V power supply. This looks pretty much like an ATX power supply, but the motherboard connector has 24 pins (instead of 20) and it has an extra 8 pin 12V connector. These are not to be confused with the ATX12V supplies which are different, though they may have similar connectors.

Case

Yes, even your case probably needs to be replaced. If you are going with a dual CPU motherboard, chances are it will follow the Extended ATX (EATX) form factor. These are 12″ by 13″ boards that usually need a special, large case. I was lucky and my massive Antec full tower case (purchased before my mini-ATX fetish) has mounting holes for an EATX motherboard. Even then, it was a tight fit. I had to remove the 3.5″ hard drive mounting bracket to make room for the CPU heatsink/fan.

Scared yet?

Like any good AMD fanboy, I was very excited by the announcement of the AMD64 architecture several years ago (or x86-64 as they were calling it back then). Single instruction operations on 64-bit ints are nice, but addressing more than 4GB of memory and doubling the number of registers is even better. I knew that some day I would get my hands on one of these systems.

Fast forward to fall 2004, and I decided to combine my 64-bit dreams with an interest in exploring more “upper class” hardware. My cluster of cheap AthlonXP boxes had served me well, but I was starting to see the limits of the PCI bus and single channel DDR memory. When I upgraded the network to gigabit ethernet, the issue was even more obvious. The ethernet controller and the IDE controller were fighting for bandwidth on the PCI bus, and the result was pretty minor improvements in NFS performance over 100BaseT. I wanted a system with way more I/O bandwidth, and room to grow. (And why not make it SMP-capable while I’m at it?)

The upshot was a plan to build an Opteron server on a SMP-motherboard with PCI-X out of parts scavenged from Newegg, Fry’s Electronics and eBay. The result was much expense and pain, but eventually a working system.

The next few entries will go into the problems I encountered and their (non-) solutions.