Migrating Server Data

Posted by david.cowan on July 7, 2019
Posted in: Uncategorized. Leave a Comment

So my FreeNAS server is full.

I wanted to build it again in a rack mount case for inclusion in my new A/V rack in the new cinema room, and bring all my ‘home compute & network’ together in a single place – but I couldn’t afford to take it offline.

Since it’s been a while since I built it, I had a look on eBay for all the same parts (since it’s just a FreeNAS file server, it doesn’t need a whole bunch of power) and I was really quite happy with how solid the original build has been. I therefore purchased all the same parts again (except for a different case)

SuperMicro X9-SCL-O motherboard
Xeon E3-1260v3 Processor
32Gb ECC RAM
8 x 6Tb Seagate NAS drives (increasing from 6 x 3Tb drives)
2 x 120Gb SSD
1 x 800w PSU
1 x Rosewill 4U rack mount case

Apart from the drives, I managed to get the whole lot for about $350 which was a steal compared to my original build.

Throw on the latest FreeNAS 11 and it’s ticking along nicely.

However, I now have about 12Tb of data to move from the old server to the new one, without bringing the old one offline.

I’ve always been a fan of rsync… I know there are faster ways of transferring data, but rsync is solid, handles incremental updates, and is very very mature. But it ain’t fast.

After having a dig around, I finally found a magic incantation which upped my transfer rate from about 150Mbps to about 850Mbps!!!

rsync -aHAXv -e "ssh -T -o Compression=no -x" user@<source>:<source_dir> <dest_dir>

There’s a fantastic discussion here :

https://gist.github.com/KartikTalwar/4393116

[UPDATE] — so, using this to migrate servers is missing some special sauce. Using the command line as-is was resulting in some odd copy failures, mainly related to the mismatch of user/group permissions between the source and target machines.

We need to tell rsync NOT to transfer permissions (and I added a parameter which cleans up the target folder too) – this seems to work well, and saturates my network connection while running 🙂

rsync -aHAXv -A --no-perms --stats --delete -e "ssh -T -o Compression=no -x" remote:/Music/Compressed/ /Media/Music/MP3

So, I built this for Sam…

Posted by david.cowan on May 3, 2018
Posted in: Uncategorized. Leave a Comment

So, I decided to build a full sized arcade machine for Sam’s 5th birthday!  (Actually, I had planned on building it for his Xmas, but things got away from me – so I figured May would be a better target).

I think it turned out pretty damned sweet – and I learned a lot (not just about wood working, cabinetry and power tools), but about Raspberry Pi, RetroPie and MAME configuration!

I really should write up how I built it…

Easing the transition

Posted by david.cowan on March 10, 2014
Posted in: Uncategorized. Leave a Comment

Having been a PC user for almost 30 years now, I’ve changed over to an iMac as my primary machine.  (I have Parallels installed, of course, for when I need to do something PC’ish or use some of my more esoteric development software)…  One of the major things that has been driving me MENTAL is the change in keyboard layout… I can handle SPLAT-C and SPLAT-V for cut/paste instead of CTRL-C/V — but why the hell would anyone mess with HOME or END?  Or event some of the more text-editing centric ones like word jumping left and right, and delete word…

Anyway, one of the things I discovered is that with OS X 10.8 onwards, you can just change all of the default mappings easily by adding a new file with the new keyboard macros!

Create a folder in your user Library called KeyBindings and create a file called DefaultKeyBinding.dict and slap this in there…

~/Library/KeyBindings/DefaultKeyBindings.dict
{
"\UF729" = "moveToBeginningOfLine:";
"^\UF729" = "moveToBeginningOfDocument:";
"$\UF729" = "moveToBeginningOfLineAndModifySelection:";
"^$\UF729" = "moveToBeginningOfDocumentAndModifySelection:";
"\UF72B" = "moveToEndOfLine:";
"^\UF72B" = "moveToEndOfDocument:";
"$\UF72B" = "moveToEndOfLineAndModifySelection:";
"^$\UF72B" = "moveToEndOfDocumentAndModifySelection:";
"^\UF702" = "moveWordLeft:";
"^\UF703" = "moveWordRight:";
"$\UF700" = "moveUpAndModifySelection:";
"$\UF701" = "moveDownAndModifySelection:";
"$\UF702" = "moveLeftAndModifySelection:";
"$\UF703" = "moveRightAndModifySelection:";
"^$\UF702" = "moveWordLeftAndModifySelection:";
"^$\UF703" = "moveWordRightAndModifySelection:";
"^\U007F" = "deleteWordBackward:";
}

Looking for the raw key values?  I picked up some really useful ones here…

http://www.hcs.harvard.edu/~jrus/site/KeyBindings/Windows%20Bindings.dict

Beefy mushroom broth

Posted by david.cowan on January 28, 2014
Posted in: Food. Leave a Comment

OK, so on occasion, I cook a bit… but I’m more ‘self taught’ than classically trained – so I tend to just make something with whatever is lying around.
Most of the time, it works out; on the odd occasion, I knock it out of the park.

The problem with the way I cook, is that I never remember what the hell I did to get there – so my meals suffer from ‘never the same taste twice’.

Last night, I made a fantastic hearty beef and mushroom broth that I wouldn’t mind making again at some point – so this is my attempt at documenting the recipe…

2 x Large sweet onions (sliced)
1lb x mushrooms (roughly chopped into 1/4 or 1/6)
Butter (enough to fry up the onions)
A tsp of thyme
3 or 4 bay leaves
A handful of chopped cilantro
A couple of garlic cloves, chopped
Salt & Pepper
General purpose flour (about 1/4 cup per liter of bouillon)
A fistful of pearl barley
Beef bouillon ("Better than Bouillon" preferably)

 

  • Fry up the onions, garlic, thyme and bay leaves in the butter until they start to caramelize and begin to turn brown (about 25 minutes)
  • Remove the bay leaves
  • Sift and slowly stir in the flour into the onions
  • Continue cooking the onions (and stirring occasionally) for another couple of minutes
  • Bung everything else into the beef broth and put on a low rolling boil for about an hour

And thats it.  I didn’t say it was hard!

FreeNAS, why hast thu forsaken me?

Posted by david.cowan on January 23, 2014
Posted in: Computers, Servers. Leave a Comment

I’ve been running happily with FreeNAS 8.1 for quite a while now, on my HP Microserver N40L.  It has been rock solid, and has handled all the data I’ve thrown at it, along with gracefully dealing with various HDD failures without compromising any data.

So, I was planning on running the latest FreeNAS (9.2) on the new server… The Dell PowerEdge 2950 is pretty vanilla hardware these days, and there’s nothing really that special about my system.  Imagine my surprise then, when I boot up the latest FreeNAS 9.2, to find that it spontaneously reboots when trying to write to the disk pool.  Further questions on the support forums confirm that this is not an isolated problem, and that other 2950 owners are experiencing the same thing.
So far, help has been non-existent — some posts going as far as to suggest “just buy yourself a new server, the 2950 was never on the FreeBSD supported hardware list” (which is ironic, since the 2950 runs the latest FreeBSD 9.2 just fine — only FreeNAS is failing)

Anyway – given that I absolutely MUST have a ZFS file system (there’s nothing out there to touch it yet; btrfs simply isn’t far enough along in development, and Microsoft ReFS parity performance is woeful, and hardware RAID simply doesn’t cut it) – I decided to look into alternatives.

I decided to give a couple of these alternatives a go

  • FreeNAS 9.1 (the latest (and non-working) version is 9.2)
  • openIndiana (based on Open Solaris from SUN)
  • FreeBSD (on which FreeNAS is based)
  • Ubuntu Linux

Out of all of these, Ubuntu was a surprise to me… I’ve used all of the others before.  Last time I built a server, Linux was never an option because the ZFS implementation was unstable, and available only through the FUSE filesystem hooks – so performance was pretty atrocious.   Linux now supports ZFS at a kernel level, and has matured a lot since then.

In each case, I configured the system with the following

  • Dell Poweredge 2950 Gen III
  • 32Gb ECC RAM
  • 6 x 3Tb Western Digital Green Drives
  • 2 x Intel 80Gb SSD
  • 1 x Hitachi 2.5″ 100Gb HDD
  • IBM M1015 (IT Mode) driving the 6x3Tb in the server drive cage
  • Rosewill PCIe SATA controller (Sil3132 chipset) driving the 2 x SSD’s
  • Motherboard SATA driving the Hitachi 100Gb boot drive
  • Intel Pro 100/1000 DUAL NIC (internal Broadcom NIC’s are disabled in BIOS)

The drive pool in each case was configured as

  • ZFS RAID-Z2
  • 6 x 3Tb as storage
  • 2 x 80Gb SSD (8Gb partition as ZIL, mirrored… remaining 72Gb as L2ARC striped)
  • No compression, no de-duplication, sync as standard

The read/write speeds were tested from the console using the dd command

dd if=/dev/zero of=/ZPOOL/test.dat bs=2048k count=50k

(writing out about 150Gb of test file to fully flush any cache)

The network write speeds were testing from an 27″ iMac (3Ghz i7) with 32Gb RAM and a 2Tb Fusion drive, running OS X Mavericks 10.8 – using the ‘nc’ command.

server:   nc -v -l -n 1111 > test2.dat
client:   time yes | nc -v -l -n 192.168.10.109 1111 < FreeNAS-9.1.img

(I chose the FreeNAS image file to transfer, because it was sufficiently large (about 2Gb) and was lying around my drive anyway)

The results actually surprised me a bit…

FreeBSD 9.2 (ashift=9)
——————
W: 744.2 seconds (144279522 bytes/sec)
R: 314.5 seconds (341377873 bytes/sec)


FreeBSD 9.2 (ashift=12 gnop -S 4096)
——————
W: 437.9 seconds (245168343 bytes/sec)
R: 352.1 seconds (304951263 bytes/sec)

nc: 2000000000 transferred in 27.331 seconds (73176978.52 bytes/sec) (69.78Mb/s)




FreeNAS 9.1.1
—————————
W: 437.4 seconds (245460370 bytes/sec)
R: 354.5 seconds (302887486 bytes/sec)

nc: 2000000000 transferred in 28.612 seconds (69900740.94 bytes/sec) (66.66Mb/s)


Open Indiana 151a8
—————————
W: 288.79 seconds (372MB/s)
R: 495.60 seconds (217MB/s)



Linux (Ubuntu 12.04LTS)
—————————
W: 281.277 seconds (382MB/s)
R: 359.095 seconds (299MB/s)

nc: 2000000000 transferred in 30.56 seconds (65427898.456 bytes/sec) (62.40Mb/s)


Linux (Ubuntu 13.10)
—————————
W: 296.71 seconds (362MB/s)
R: 278.72 seconds (385MB/s)


nc: 2000000000 transferred in 19.94 seconds (100300902.71 bytes/sec) (95.65Mb/s)

Silence is golden!

Posted by david.cowan on January 10, 2014
Posted in: Computers, Servers. Leave a Comment

Damn!  Those modifications worked perfectly…  installing the 47ohm resistors across the 4 CPU/Drive chasis fans, and on the 2 fans in both the active and redundant PSU’s (6 fans in all) has dropped the noise level to that of a normal desktop PC!  I can actually be in the same room as the machine when it’s on 🙂
I had some issues where the fan speeds would ‘bounce’ — the resistors brought the spin speeds down so low that the firmware would read them as being below the threshold, and spin them back up — but I followed the instructions on the other blog (using an Ubuntu Live boot CD) and modified the latest DELL firmware with much lower speed thresholds, and flashed it.  Now, it just purrs along like a little kitten 🙂

I took the opportunity to strip out the DELL Perc SATA controller (I wasn’t using it) and added the new IBM M1015 SATA controller.  I used a CD-bootable DOS 6.22 image to boot into DOS, and flashed the M1015 controller with the latest LSI firmware into IT mode.   I plan on installing some variant of ZFS on this machine, but ZFS is a completely software driven RAID solution — hardware RAID just gets in the way; IT mode just removes all of the RAID functionality from the M1015 controller, so that the system just see’s a JBOD disk array.

With the DELL Perc controller removed, there was enough space to install a couple of old 80Gb SSD’s sitting atop the drive cage, and hook them into the unused integrated motherboard SATA ports A/B.  I’ll use these as a boot drive probably.

So, that’s the hardware side of things finished — time to move on to the software side 🙂

Modified PowerEdge 2950

Modified PowerEdge 2950

Harrier Jumpjets!

Posted by david.cowan on January 8, 2014
Posted in: Computers, Servers. Leave a Comment

Holy crap, that was unexpected.  I powered on the PowerEdge 2950, and I’m pretty sure it awoke the neighbors!  That thing is LOUD!
I think, if I could perch this thing vertically, so that the rear of the machine is sitting on the desk – I’m sure that it would hover above the desk with all the fans running at full tilt!!
Considering this beast has to live in the spare bedroom closet (server room), this is unacceptable…

I found a couple of blogs elsewhere addressing the noise issue – I guess I’m not the only person with one or more of these machines in a home environment.

This one explains how to install a set of resistors in series across the fans – http://blindcaveman.wordpress.com/2013/08/23/problem-dell-poweredge-2950-iii-jet-engine-fan-noise/

This one explains how to patch and burn a new BIOS with lower fan speed thresholds, so that they don’t run so hard all the time — http://projects.nuschkys.net/2011/11/15/how-to-adjust-the-fan-thresholds-of-a-dell-poweredge/

Time to nip down to RadioShack for some heat shrink tubing and a couple of 47ohm resistors!

First!!

Posted by david.cowan on January 8, 2014
Posted in: Computers, Servers. Leave a Comment

All the parts of my new home server have arrived!

I ordered a reconditioned Dell PowerEdge 2950, Dual quad core Zeon 5450’s, 32Gb EDD RAM, an IBM M1015 SATA controller and 6 x SATA drive trays for $600 from http://www.stalliontek.com/
That’s a beast of a machine for the price!

I also picked up 6 x Western Digital 3TB Green drives for $85 each on Black Friday from an Amazon marketplace seller.

My current home server setup is made up of

HP N40L ProLiant Microserver, 16Gb EDD RAM, Intel Pro/1000 NIC, IBM M1015 SATA controller and 6 x 2TB HDD’s (a mix of green, non-green and SATA I and II)
HP N40L ProLiant Microserver, 16Gb EDD RAM, Intel Pro/1000 NIC and the stock 250Gb HDD it shipped with

One of the Microservers is running FreeNAS 8.2 and the 6 x 2Tb drive array is set up as ZFS-RAID-Z2 for maximum data resiliancy and redundancy.  The dataset is not partitioned in any way, and is just exported as an iSCSI target
The other Microserver is running Windows Home Server 2011 on the stock 250Gb HDD, and the data drive is the iSCSI target from the FreeNAS Microserver.  The two Intel NIC’s in both machines are hooked up via a crossover ethernet cable, and the iSCSI interface is bound directly to those NIC’s — dedicating that 1000Mb/s link to sharing the ZFS protected space with the Windows Home Server machine.

Originally, I chose this layout because I wanted the protection of ZFS-RAID-Z2, but I also wanted the features of WHS (PC backups, easy SMB shares, etc)

Under normal circumstances, this would suffice for a few years, but I’ve found that I’m in need of more space and some additional services that the Microservers just don’t have the CPU grunt for.