Building a FreeNAS

I have lots of files, and I like them. They sit on an old and venerable HP Proliant Microserver N40L running openmediavault. That's served [1] us pretty well for the last couple of years, running emby, nfs, samba, and anything else I can throw at it like a champ.

The backup situation left a lot to be desired, though. My media "backup" consisted of the original DVDs/Blu-rays/CDs, because storing all that data twice was prohibitively expensive. Documents and photos were slightly better handled, but the spectre of bitrot hid behind every corner.

And then this happened:

When hard drives go bad

That hard drive is gonna die. :(

This is a great chance to try something that I've wanted to check out for yeeeears: ZFS. That HP microserver was never going to cope with the RAM requirements of ZFS. The official FreeNAS guidelines hint at 1GB of memory per terabyte of disk space, but shy away from declaring it outright. However, the official advice is:

The best way to get the most out of a FreeNAS® system is to install as much RAM as possible.

Lets go shopping.

The bits

I really wanted to build this new server in an proper rack-mountable enclosure, but my lack of server components, and the eye-popping expense made me reconsider.

So I found this lovely wee case: the Node 804 by Fractal Design. It has lots of big fan mounts for quietly cooling things, room for 10 big hard drives and tons of flexibility on how you arrange the guts.

Once I'd decided on my enclosure, I took a trip to pcpartpicker and trawled the existing builds for something close to matching what I was after. Luckily, someone had already built exactly what I had in mind: 35TB Plex Server Intel Xeon E3-1230 V5. I made very few changes to that build and ended up with the following goodies:

Here's what I got:

Type Item Price
CPU Intel Xeon E3-1230 V6 3.5 GHz Quad-Core Processor £273.74
CPU Cooler Noctua NH-U12S 55 CFM CPU Cooler £65.59
Motherboard Asus P10S-M WS Micro ATX LGA1151 Motherboard £209.99
Memory Samsung 16 GB (1 x 16 GB) DDR4-2666 Memory £96.74
Memory Samsung 16 GB (1 x 16 GB) DDR4-2666 Memory £125.68
Storage Samsung 970 Evo 500 GB M.2-2280 NVME Solid State Drive £102.86
Storage Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive £105.14
Storage Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive £105.14
Storage Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive £105.14
Storage Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive £105.14
Case Fractal Design Node 804 MicroATX Mid Tower Case £113.93
Power Supply EVGA SuperNOVA G3 750 W 80+ Gold Certified Fully Modular ATX Power Supply £107.98

PCPartPicker Part List

I had another two new hard drives to fill up the gaps.

All the bits on the carpet

The build

The build was quite fun for the most part; the cable routing options were a bit more limited that previous cases I've owned previous cases I've owned, we managed to get everything pretty nicely tucked away.

I'm really good at focusing on background images with my phone camera.

Empty insides of the Node 804

The case came with a bunch of different screw types, including the one second from the right that isn't mentioned in any of the docs and I haven't found a use for yet..

All the screws

I was pretty disappointed that the two sticks of RAM arrived in the same bag, being bounced off of one another all through the postal system from Poland. That might come back to bite me in the future...

Safely packaged RAM DIMMs

The motherboard, CPU, RAM and cooler went in without a hitch, and the Noctua cooler fits no bother! I was only a little worried that it wouldn't...

The motherboard in place

Plugging in the hard drives above the power supply was really tight, and it's the bit that I'm least happy about. As long as the cables don't slowly drift into the spinning fan, it should be OK, but just a little more clearance would have been lovely.

The hard drive connections just squeaked in

I recorded the last 5 characters of each HDD serial number, and ther position in the case. F means front caddy, B means not the front caddy. Drives were numbered from inside to outside of the case. The idea of this is that when one of them starts to die, I can get the serial number and know which one to pull.

Recording Hard drive placement

The boot

Now was the scary bit. Time to turn it on and see if I'd wired everything up properly, and not mashed the CPU when I installed the heat sink (why is that bit always so difficult?).

The fans spun up, and the motherboard beeped a couple of times at me, but that was it. No posting. No picture. Hmm...

It turns out that onboard graphics on a motherboard doesn't actually mean onboard graphics. It means

This will provide on-board graphics if you use a CPU with a graphics chip.

I did not know this.

More shopping

So, I had to buy a graphics card. I grabbed a Asus GeForce GT 710 2GB after checking that the Nouveau driver would support it. Low powered, passive cooling. I just wanted something that would draw a screen long enough to install ssh.

A few days later, I had the card installed and tried to power on again.

The boot, part 2

Still no life. Hmm... Time to pull out components and try them one by one.

Remember how I said that I was worried the RAM might come back to bite me? One of the DIMMs was completely scuppered; once I'd removed it and RMA'ed it's arse we were booting, complete with fans, POST beeps and a picture.

The burn-in

This machine is going to be a NAS, and you'd better make sure that the components are good. I'd already been bitten by one DIMM; it was time to make sure the other one wasn't slightly less gubbed.

I used an Archlinux live USB to boot into Memtest and ran that for a few days.

At least one stick worked out the bag.

With that under my belt, it was time to burn in the hard drives. There is a bunch of different recommendations on the FreeNAS forums and in the wilds of Reddit, but I was humming and hah-ing over what OS to install in the end, so I grabbed my Archlinux USB stick and booted into that instead.

This is going to take a while, so I installed tmux so I could run the burn-ins in parallel

pacman -Sy tmux

Then I downloaded a script by Spearfoot that sandwiches badblocks between SMART tests.

curl https://raw.githubusercontent.com/Spearfoot/disk-burnin-and-testing/master/disk-burnin.sh -O disk-burnin.sh

There's a "make sure you absolutely positively want to do this" variable in the script called Dry_Run that you need you need to set to 0 for the script to do anything. Which is fair enough; this'll wipe every damn bit of your hard drive and there won't be any way back.

I made the script executable and launched it for each drive in separate tmux windows, then made a cup of tea while I waited.

Then went to bed. Slept. Made more tea. Went to work. Tea, tea, sleep, work, dog walks, tea...

It actually only took 77 hours, which was a lot shorter than I thought it would take. Every harddrive got the error-free stamp of approval :D

Output of disk-burnin.sh

Moar RAM

So that's it. A working canvas upon which to store stuff. I'm still waiting on replacement RAM, but I've installed FreeNAS on it in the mean time so I can have a poke around. This machine is definitely going to use ZFS, so FreeNAS is the best choice for a pure fileserver, but I'm worried that the platform won't have plugins for all the things I might want, and hey; this machine should be good for virtualizing lots of stuff. I'm dithering about putting Ubuntu on there instead.

Do I want to stay in my comfy warm Linux because I'm scared to learn new things, or because I know it'll run docker no problem? And maybe I don't want the appliance of a FreeNAS installation, but rather a playground that happens to have my films.

It'll need to wait until the next post.

[1]If you'll pardon the pun