I have lots of files, and I like them. They sit on an old and venerable HP
Proliant Microserver N40L running openmediavault. That's served us
pretty well for the last couple of years, running emby, nfs, samba, and anything
else I can throw at it like a champ.
The backup situation left a lot to be desired, though. My media "backup"
consisted of the original DVDs/Blu-rays/CDs, because storing all that data twice
was prohibitively expensive. Documents and photos were slightly better handled,
but the spectre of bitrot hid behind every corner.
And then this happened:
That hard drive is gonna die. :(
This is a great chance to try something that I've wanted to check out for
yeeeears: ZFS. That HP microserver was never going to cope with the RAM
requirements of ZFS. The official FreeNAS guidelines hint at 1GB of memory
per terabyte of disk space, but shy away from declaring it outright. However,
the official advice is:
The best way to get the most out of a FreeNAS® system is to install as much RAM as possible.
Lets go shopping.
The bits
I really wanted to build this new server in an proper rack-mountable enclosure,
but my lack of server components, and the eye-popping expense made me
reconsider.
So I found this lovely wee case: the Node 804 by Fractal Design. It has lots
of big fan mounts for quietly cooling things, room for 10 big hard drives and
tons of flexibility on how you arrange the guts.
Once I'd decided on my enclosure, I took a trip to pcpartpicker and trawled the
existing builds for something close to matching what I was after. Luckily,
someone had already built exactly what I had in mind:
35TB Plex Server Intel Xeon E3-1230 V5. I made very few changes to that build
and ended up with the following goodies:
Here's what I got:
| Type |
Item |
Price |
|---|
| CPU |
Intel Xeon E3-1230 V6 3.5 GHz Quad-Core Processor |
£273.74 |
| CPU Cooler |
Noctua NH-U12S 55 CFM CPU Cooler |
£65.59 |
| Motherboard |
Asus P10S-M WS Micro ATX LGA1151 Motherboard |
£209.99 |
| Memory |
Samsung 16 GB (1 x 16 GB) DDR4-2666 Memory |
£96.74 |
| Memory |
Samsung 16 GB (1 x 16 GB) DDR4-2666 Memory |
£125.68 |
| Storage |
Samsung 970 Evo 500 GB M.2-2280 NVME Solid State Drive |
£102.86 |
| Storage |
Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive |
£105.14 |
| Storage |
Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive |
£105.14 |
| Storage |
Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive |
£105.14 |
| Storage |
Western Digital Red 4 TB 3.5" 5400RPM Internal Hard Drive |
£105.14 |
| Case |
Fractal Design Node 804 MicroATX Mid Tower Case |
£113.93 |
| Power Supply |
EVGA SuperNOVA G3 750 W 80+ Gold Certified Fully Modular ATX Power Supply |
£107.98 |
PCPartPicker Part List
I had another two new hard drives to fill up the gaps.
The build
The build was quite fun for the most part; the cable routing options were a bit
more limited that previous cases I've owned previous cases I've owned, we
managed to get everything pretty nicely tucked away.
I'm really good at focusing on background images with my phone camera.
The case came with a bunch of different screw types, including the one second
from the right that isn't mentioned in any of the docs and I haven't found a use
for yet..
I was pretty disappointed that the two sticks of RAM arrived in the same bag,
being bounced off of one another all through the postal system from Poland. That
might come back to bite me in the future...
The motherboard, CPU, RAM and cooler went in without a hitch, and the Noctua
cooler fits no bother! I was only a little worried that it wouldn't...
Plugging in the hard drives above the power supply was really tight, and it's
the bit that I'm least happy about. As long as the cables don't slowly drift
into the spinning fan, it should be OK, but just a little more clearance would
have been lovely.
I recorded the last 5 characters of each HDD serial number, and ther position in
the case. F means front caddy, B means not the front caddy. Drives were
numbered from inside to outside of the case. The idea of this is that when
one of them starts to die, I can get the serial number and know which one to
pull.
The boot
Now was the scary bit. Time to turn it on and see if I'd wired everything up
properly, and not mashed the CPU when I installed the heat sink (why is that bit
always so difficult?).
The fans spun up, and the motherboard beeped a couple of times at me, but that
was it. No posting. No picture. Hmm...
It turns out that onboard graphics on a motherboard doesn't actually mean
onboard graphics. It means
This will provide on-board graphics if you use a CPU with a graphics chip.
I did not know this.
The boot, part 2
Still no life. Hmm... Time to pull out components and try them one by one.
Remember how I said that I was worried the RAM might come back to bite me? One
of the DIMMs was completely scuppered; once I'd removed it and RMA'ed it's arse
we were booting, complete with fans, POST beeps and a picture.
The burn-in
This machine is going to be a NAS, and you'd better make sure that the
components are good. I'd already been bitten by one DIMM; it was time to make
sure the other one wasn't slightly less gubbed.
I used an Archlinux live USB to boot into Memtest and ran that for a few days.
With that under my belt, it was time to burn in the hard drives. There is a
bunch of different recommendations on the FreeNAS forums and in the wilds of
Reddit, but I was humming and hah-ing over what OS to install in the end, so I
grabbed my Archlinux USB stick and booted into that instead.
This is going to take a while, so I installed tmux so I could run the burn-ins
in parallel
Then I downloaded a script by Spearfoot that sandwiches badblocks between
SMART tests.
curl https://raw.githubusercontent.com/Spearfoot/disk-burnin-and-testing/master/disk-burnin.sh -O disk-burnin.sh
There's a "make sure you absolutely positively want to do this" variable in the
script called Dry_Run that you need you need to set to 0 for the script to
do anything. Which is fair enough; this'll wipe every damn bit of your hard
drive and there won't be any way back.
I made the script executable and launched it for each drive in separate tmux
windows, then made a cup of tea while I waited.
Then went to bed. Slept. Made more tea. Went to work. Tea, tea, sleep, work, dog
walks, tea...
It actually only took 77 hours, which was a lot shorter than I thought it would
take. Every harddrive got the error-free stamp of approval :D
Moar RAM
So that's it. A working canvas upon which to store stuff. I'm still waiting on
replacement RAM, but I've installed FreeNAS on it in the mean time so I can have
a poke around. This machine is definitely going to use ZFS, so FreeNAS is the
best choice for a pure fileserver, but I'm worried that the platform won't have
plugins for all the things I might want, and hey; this machine should be good
for virtualizing lots of stuff. I'm dithering about putting Ubuntu on there
instead.
Do I want to stay in my comfy warm Linux because I'm scared to learn new things,
or because I know it'll run docker no problem? And maybe I don't want the
appliance of a FreeNAS installation, but rather a playground that happens to
have my films.
It'll need to wait until the next post.