I’ve been running happily with FreeNAS 8.1 for quite a while now, on my HP Microserver N40L. It has been rock solid, and has handled all the data I’ve thrown at it, along with gracefully dealing with various HDD failures without compromising any data.
So, I was planning on running the latest FreeNAS (9.2) on the new server… The Dell PowerEdge 2950 is pretty vanilla hardware these days, and there’s nothing really that special about my system. Imagine my surprise then, when I boot up the latest FreeNAS 9.2, to find that it spontaneously reboots when trying to write to the disk pool. Further questions on the support forums confirm that this is not an isolated problem, and that other 2950 owners are experiencing the same thing.
So far, help has been non-existent — some posts going as far as to suggest “just buy yourself a new server, the 2950 was never on the FreeBSD supported hardware list” (which is ironic, since the 2950 runs the latest FreeBSD 9.2 just fine — only FreeNAS is failing)
Anyway – given that I absolutely MUST have a ZFS file system (there’s nothing out there to touch it yet; btrfs simply isn’t far enough along in development, and Microsoft ReFS parity performance is woeful, and hardware RAID simply doesn’t cut it) – I decided to look into alternatives.
I decided to give a couple of these alternatives a go
- FreeNAS 9.1 (the latest (and non-working) version is 9.2)
- openIndiana (based on Open Solaris from SUN)
- FreeBSD (on which FreeNAS is based)
- Ubuntu Linux
Out of all of these, Ubuntu was a surprise to me… I’ve used all of the others before. Last time I built a server, Linux was never an option because the ZFS implementation was unstable, and available only through the FUSE filesystem hooks – so performance was pretty atrocious. Linux now supports ZFS at a kernel level, and has matured a lot since then.
In each case, I configured the system with the following
- Dell Poweredge 2950 Gen III
- 32Gb ECC RAM
- 6 x 3Tb Western Digital Green Drives
- 2 x Intel 80Gb SSD
- 1 x Hitachi 2.5″ 100Gb HDD
- IBM M1015 (IT Mode) driving the 6x3Tb in the server drive cage
- Rosewill PCIe SATA controller (Sil3132 chipset) driving the 2 x SSD’s
- Motherboard SATA driving the Hitachi 100Gb boot drive
- Intel Pro 100/1000 DUAL NIC (internal Broadcom NIC’s are disabled in BIOS)
The drive pool in each case was configured as
- ZFS RAID-Z2
- 6 x 3Tb as storage
- 2 x 80Gb SSD (8Gb partition as ZIL, mirrored… remaining 72Gb as L2ARC striped)
- No compression, no de-duplication, sync as standard
The read/write speeds were tested from the console using the dd command
dd if=/dev/zero of=/ZPOOL/test.dat bs=2048k count=50k
(writing out about 150Gb of test file to fully flush any cache)
The network write speeds were testing from an 27″ iMac (3Ghz i7) with 32Gb RAM and a 2Tb Fusion drive, running OS X Mavericks 10.8 – using the ‘nc’ command.
server: nc -v -l -n 1111 > test2.dat client: time yes | nc -v -l -n 192.168.10.109 1111 < FreeNAS-9.1.img
(I chose the FreeNAS image file to transfer, because it was sufficiently large (about 2Gb) and was lying around my drive anyway)
The results actually surprised me a bit…
FreeBSD 9.2 (ashift=9) —————— W: 744.2 seconds (144279522 bytes/sec) R: 314.5 seconds (341377873 bytes/sec) FreeBSD 9.2 (ashift=12 gnop -S 4096) —————— W: 437.9 seconds (245168343 bytes/sec) R: 352.1 seconds (304951263 bytes/sec) nc: 2000000000 transferred in 27.331 seconds (73176978.52 bytes/sec) (69.78Mb/s) FreeNAS 9.1.1 ————————— W: 437.4 seconds (245460370 bytes/sec) R: 354.5 seconds (302887486 bytes/sec) nc: 2000000000 transferred in 28.612 seconds (69900740.94 bytes/sec) (66.66Mb/s) Open Indiana 151a8 ————————— W: 288.79 seconds (372MB/s) R: 495.60 seconds (217MB/s) Linux (Ubuntu 12.04LTS) ————————— W: 281.277 seconds (382MB/s) R: 359.095 seconds (299MB/s) nc: 2000000000 transferred in 30.56 seconds (65427898.456 bytes/sec) (62.40Mb/s) Linux (Ubuntu 13.10) ————————— W: 296.71 seconds (362MB/s) R: 278.72 seconds (385MB/s) nc: 2000000000 transferred in 19.94 seconds (100300902.71 bytes/sec) (95.65Mb/s)