Recovering data from my broken Terastation Pro II
A few days ago my Buffalo Terastation Pro II showed a strange error message in the webinterface while I wanted to empty the trash (the Terastation has a trashbox for the samba shares). The error message was:
[BufCommonFileShareInfo.save]File Close Error - /var/lock/cgi_check_gate
The error was only affecting the webinterface and I could not issue any configuration changes any longer. But as it was only an error caused by a lock file I thought a reboot of the Terastation might fix the error condition.
But I was wrong. I rebootet the Terastation and everything I got was this message on the LCD:
System booting ...
I thought it it might be doing a filesystem check or something like that, but after about 6 hours (which should be more than enough to check 700 GB) I unplugged the power cord to cold-reset the Terastation. Now thing got a bit worse and together with a red light and loud beep the LCD showed:
System Error E04 Can't load Krnl!
The advise from Buffalo for this error is to reflash the firmware of the Terastion with a newer version. I could have done this, but I don't fully trust the vendor and the device to not loose my data on a firmware update.
It's my fault that I don't had a backup of the data, which was stored on the Terastation but now I had to go another route of recovering the data without flashing the firmware of the Terastation.
My plan was to put the drives from the Terastation into another system, boot any Linux distribution on the system and reassemble the RAID on that system to copy the data somewhere else.
I know that the Terastation uses the XFS filesystem and that the RAID is a Linux software raid, but Linux software raid is not endian agnostic, so one important point was to find out if I need a PowerPC system or if an x86 system will be enough.
Older Terastations use a PowerPC cpu and are therefor big-endian systems. You need a PPC system to reassemble the RAID. After a long search I found a post in the Buffalo support forum, where someone wrote, that the newer ARM-based Terastations are little-endian systems and therefor an x86 system should be used to reassemble the RAID.
I have a few x86 system lying around, but either they don't have enough Sata ports or they don't have a CD-Rom/DVD-Rom to boot Linux from CD. So I needed to prepare an USB-Stick to boot Linux on a system where I could attach all four disks at once.
Preparing a bootable Linux USB-Stick on OS X
As I currently don't have access to a Linux Workstation one tricky part was to create a USB-Stick, which will boot Linux, on my OS X Workstation.
Many Linux distributions only distribute ISO images and document how to build a bootable USB-Stick from them. All manuals I read involve syslinux at a point, which I couldn't get to compile on OS X.
After searching for a bit I found that Archlinux has ready-made images for USB-Sticks for their releases. After downloading the image just has to be copied to the Stick block-wise. As a note to myself, here are the steps necessary to dd an image to a USB-Stick on OS X:
Insert the USB-Stick (it will be automounted) and check with
diskutilwhich device the stick was assigned to:
ticonderoga:~ arne$ sudo diskutil list Password: /dev/disk0 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *111.8 Gi disk0 1: EFI 200.0 Mi disk0s1 2: Apple_HFS Macintosh HD 111.5 Gi disk0s2 /dev/disk2 #: TYPE NAME SIZE IDENTIFIER 0: FDisk_partition_scheme *3.7 Gi disk2 1: Windows_FAT_32 USB_DISK_01 3.7 Gi disk2s1
The USB-Stick in this case is attached as
/dev/disk2(YMMV). Now umount the Stick but don't eject it:
ticonderoga:~ arne$ sudo diskutil umountDisk /dev/disk2 Unmount of all volumes on disk2 was successful
ddto write the downloaded Archlinux image to the USB-Stick (pay attention to write to the stick itself, not the first partition):
ticonderoga:~ arne$ dd bs=8192 if=./archlinux-2009.08-core-i686.img of=/dev/disk2 46560+1 records in 46560+1 records out 381426176 bytes transferred in 236.369148 secs (1613689 bytes/sec)
That's it, now I have a bootable USB-Stick, which will boot an Archlinux 2009.08, ready to recover the RAID from the Terastation.
Recovering the Data
Having a bootable USB-Stick with Linux and an x86 System with at least 4 Sata ports I could start the recovery of the Terastation RAID.
I attached the disks to the onboard Sata controller and inserted the USB-Stick into one of the onboard USB ports and booted the system. Luckily the BIOS of the onboard Sata controller told me that it detected four disks, so none of the disks was completely bad.
After booting the Archlinux to a shell, I looked at the
dmesg output and
the kernel had detected the four disks with their partitions and everything
fdisk I double checked that all disks had the right paritions and
/dev/sdX6 (with X = a,b,c,d) was the data partition, as it
was the biggest one and I only had one big Raid5 Array configured on the Terastation.
mdadm I inspected all 4 data partitions with
mdadm --examine /dev/sdX6
and it reported that they all belong to a clean Raid5 Array. That meant
that the Array does not got damaged and I could reassemble it with mdadm:
mdadm --assemble /dev/md0 /dev/sda6 /dev/sdb6 /dev/sdc6 /dev/sdd6
And after that mounting the Array, which is formatted with an XFS filesystem, was also not a problem:
mkdir /mnt/raid mount -t xfs /dev/md0 /mnt/raid
At this point it was only a matter of copying the data to another disk or system.
Most resources on the Web about how to recover broken Terastations only cover the older PPC-based systems, and it was a bit hard to find out which endianess my Terastation uses. After I found that missing piece of information it was pretty straight forward to recovery the data. Neither the RAID nor the XFS filesystem was damaged, only the Firmware was broken. In the end it took longer to find a suitable Linux image which was USB-Stick ready than to reassemble the Array.
Now that I have recovered the data the step next will be to reflash the firmware, hoping that it this will fix the problem so that I could sell the Terastation.
The Terastation has no grafical output and no serial-console (the RS-232 port only exists to attach a UPS) and the lesson I learned while troubleshooting the broken Terastation was that it is very hard to fix a problem, if you cannot see what the problem actually is, so most propably I will sell the Terastation as I do not want to go through this trouble again.