RAID6 reconstruction attempt
This is attempt log to recover data from RAID6 NAS disk storage.
Initial state
NAS: LACIE 5Big network 2 (5 disks, 3T each in RAID6 array) online of almost 1000 days. Data value/importance high.
Disk #5 started blinking red. Pff… I thought, not the first time. Just replace a disk and it will be fine. It is RAID6 and only one disk is failing. Anyway quick format and quick attempt to wipe data (stopped due to long estimated time) and plugged again into NAS. It started syncing as usual. After a while noticed speed drop copying files. Not quite sure what I did next, but ended rebooting NAS and it stuck in boot sequence. No online connection was present. Left it overnight thinking it has to sync or whatever. Next day, there were still no connection only main LED was flashing blue while all disk leds were solid blue. Stuck in boot sequence I thought. Rebooted again (this time had to pug power off). Booted pretty soon, but RAID array was gone. Also nas stated that disk #3 was “NEW” (I have replaced it some time ago, and seems like it was not initialized by itself? Though there were partitions present).
What’s next
Let’s not fuck up things worse than it is! Also write numbers on all NAS slots and hard disks.
NAS was in use and pretty important data was stored on it, so naturally I was biased, it did not helped to think clearly. Next I faced many secondary problems like computer refusing to detect keyboard and mouse while booting (dah… switched USB legacy mode for USB3.0). Spent half an hour or more figuring this out. Left it heal by itself, tried to run system on old atom computer. Also failed due to other unknown reasons. Another wasted half an hour or more. Seems like everything is resisting to work. Got back to big computer and made it work (Legacy mode/USB3.0 stuff). Ok, have 3 disks connected:
- W1F3AFDP
- W1F3AJ9H
- empty
- W1F59CLA
- empty
Trivial task connecting disks to computer took longer than expected.
Inspecting
Next thing was to examine disks and reassemble RAID. So tried variety of software:
- ReclaiMe RAID Recovery – Should collect information about RAID to be used later, unfortunately crashed after I left it overnight. More time wasted.
- R-Studio – does not support XFS filesystem. Though is user friendly and nice to use. Some files without actual names and directory structure can be reconstructed.
- UFS explorer recovery – Looks like it should work. Directory structure is visible and some data available. But not sure of RAID parameters to be used.
- NAS Data Recovery – RAID6 not supported
- Home NAS recovery – RAID6 not supported
- Raid Reconstructor – Not eve tried
Linux
Made xubuntu live CD and with unetbootin and started on big computer. Quick installation of some tools:
apt-get install mdadm
apt-get install gparted
apt-get install smartmontools
Quick investigation with mdadm indicated on faulty disk #1 /dev/sdc.
> mdadm --examine /dev/sdc2
mdadm: No md superblock detected on /dev/sdc2.
Meanwhile disks #2 and #4 replied with:
> mdadm --examine /dev/sda2
/dev/sda2:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 16f94400:3b3c023d:1385ea8c:87cf6e8c
Name : NAS:4
Creation Time : Tue Oct 8 10:38:48 2013
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 5856481280 (2792.59 GiB 2998.52 GB)
Array Size : 8784721920 (8377.76 GiB 8995.56 GB)
Super Offset : 5856481536 sectors
Unused Space : before=0 sectors, after=256 sectors
State : clean
Device UUID : 5c5ce57a:e44ca61a:ead63312:1d03a54d
Update Time : Thu Jun 16 21:42:04 2016
Checksum : f13ae31e - correct
Events : 12767478
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : .A.A. ('A' == active, '.' == missing, 'R' == replacing)
Stuck here too. There were an success stories when it was fixed by using older mdadm version (does not use superblock) and by creating raid on top of old one (counter-intuitive, but mdadm should be smart enough and fix problem). Have not tried any of these though. Have no where to backup all data. So experiments with mdadm ended.
Back to windows tools
Connected one more disk. The same NAS indicated as new. Hope it will increase chances of reconstruction.
- W1F3AFDP
- W1F3AJ9H
- WD-WMC4N2178658
- W1F59CLA
- empty
While R-Studio seemed better software, it does not support necessary XFS file system. So only raw files can be extracted. It estimated few days of computer time to collect files, but after being stopped most of the files were corrupt.
UFS explorer recovery. Not so user friendly, but after adding one more disk (and probably setting right RAID parameters) it discovered valid XFS partition. Unfortunately larger files are also corrupt.
S.M.A.R.T
Normally it should be done first thing, but better later than never, right?
- W1F3AFDP – 326 uncorrectable sector count, 2 reallocated
- W1F3AJ9H – OK
- WD-WMC4N2178658 – 1 pending sector count
- W1F59CLA – OK
- YXR83F1W – Failed to start
Disk #5 is the same I formatted earlier and NAS was indicating it was faulty. Let’s not torture it any more for now. Disk #1 is also very problematic and near it’s death.
SeaTool
Windows version of SeaTool for better functionality kindly offered to run DOS version, so I did. Dough… was not working on x64 computer (mysterious error while boot) so switched “brains off” mode and tried on old atom computer which is x32. Worked just fine.
Fast mode detected some errors and suggested running slow more. Estimated time ~4hours for 3T disk. After completing failed miserably by crashing. Only some residual information was left on monitor stating that log’s are saved on disk, but there were no response from keyboard.
When i was assembling a NAS for myself ( basically more a lab for virtualization ) i got tired of making things work with usual pc parts. And then noticed that used corporate rack servers cost almost nothing. HP Gen6 shipped from uk without hard drives – 150e . Supports SATA and SAS. + p410 raid card to support >2TB drives ( 20e ) . I am still happy with my decision as Gen6 is not as loud while power consumption stays low as well. Just my 5c :)
Thank you, Vytautas for your 5c :) I have no problems assembling raid under Linux with recent versions of mdadm (it got really easy tool to use). Problem is I have 3 disks out of 5 damaged. Stopped toying around and asked professionals to step in :)