Server and NAS reconstruct update
Tuesday, August 26, 2008I don't think I've written about this yet, though many may know that something is going on. So I thought I would write a little about it to keep you all informed as well as help me to never have to go through this again! Please be assured, I won't feel bad if you don't read this through! But if you were wondering how things were going, here's a stripped down version of my reconstruction.
About a month ago, I came into work to find several computers shut off. That was interesting, but come to find out, our power went out, and it was down long enough to drain all our UPSs, and henceforth, shutting down our computers. I had some trouble w/ our NAS (Network Attached Storage) box, about 2tb of space. I was able to get that back up just fine (or so I thought) and was all set.About a week later, the NAS went down, similar to what it was like after the power went out. So I worked on it again, but this time couldn't get it back on line. I called a friend and he came over, and after 10 hours of working on it, couldn't get it up either (which made me feel a little better and worse - better that if he couldn't get it going, then I shouldn't feel bad about not being able to, and worse, because now what do you do?) The consensus was that the UPS drained down, and shut down. We think the power came back on briefly, which has the NAS start up, then quit. Since the UPS was already drained, we are thinking it quit suddenly and that's where my problems started. The drives seem fine, but the data integrity was compromised.
I didn't have a good backup. 2tb of data is a lot of space, but it still was not smart of me (us) to be in this position. I was figuring that the NAS (which is a RAID 5) was pretty decent protection. But this is not the case here. I did have a lot archived, but still lost quite a bit of work.
So now what do you do? We sent the drives to a company in NY that specializes in getting data off bad hard drives, and also RAID 5 systems. (For those who don't know, RAID 5 means that your files are stored across a number of hard drives--in this case 7 drives. So you need at least 6 of the drives to get any data back.) We sent the drives to this company. They said they could recover the data, about 98%, for $18,000. We said, "Ok, thanks, um, send the drives back and we'll try it ourselves."
Now I have 7 drives, and some software that is supposed to work at recovering the data. And this was a process in and of itself. What I needed to do, is to 'image' each drive (copy each drive to another location.) Ok, that's 7 drives times 400g, so just doing that will be about 3tb of space. So I ended up buying 6 1tb drives, and put those in my NAS box. (Really, I bought 2 1tb drives, then later realized I needed more, and bought 4 more.) (These 6 1tb drives will serve as part of my backup when all is said and done, so no money wasted.)
To image a drive, you attach the drive to a computer, then run a program that copies the data over to another location. So I figured the server would be the best bet, but it didn't have any extra SATA ports. No problem, I have a SATA card I can stick in. But in doing so, the server wouldn't even boot. So I tried another computer, that didn't work. Finally plugged it in my mac pro (which was really nice because there's no wire connections - just mount to the slide, and shove it in.) This seemed to work the best. (I'm running Win XP on my mac pro.)
For some reason, I couldn't get my NAS (or was it the server) running, and it was Saturday, so I went down to Costco and bought another 1tb external drive. This at least allowed me to work over the weekend. See, when you image a drive around 400gb, it takes around 12 hours. So I was able get get 3 drives done while I got the NAS problem figured out. For $200, extra space is a welcome sight.
While these were imaging, I was able to assemble most of our 2,000 Lamb PowerPoint boxes. Making good use of my time, I hope.
I don't have much hope in our server (an old "Doug 300") and so discussion was to replace that with a new machine. But in the meantime, I managed to get it working again, and continued on the process of imaging drives. So now, when all was said and done, I had one image on my mac pro, 3 on my external tb drive, and 4 on the NAS box (one was duplicated.) I have since ordered another server, which should come in about a week.
The next step, was to run Raid Reconstructor. This analyzes each drive, and comes up with the proper configuration to reconstruct the original image. See, with the RAID 5, data is duplicated, so in the event of a drive failure, the other 6 drives have all the data still. So this program 'weeds out' the extra data. But this program just gave me an xml (text) file, with the proper configuration, to which another program will use to actually get the data back. This program is called GetDataBack for NTFS.
I ran GetDataBack. Again, we are working over 2tb of data, and I figured it would take a day or so. 3 days later I'm still at 47%. Ok, I can wait, at least there are no Read / Write Delayed errors and the server is still running. That was Saturday night. Sunday on the way to church I stopped to look at it, and my mac pro ran out of memory (when GetDataBack was at 50%.)
Well, that's all I can do. Monday came and I ordered another 4g of memory for the mac pro, and will wait for that to come in before starting again. Or so I thought. I decided to create an actual image of the drives into one big image, which is currently running (I started it yesterday morning, and this morning it's at 17%. I'm glad I didn't have my memory shipped overnight!)
Any questions?
For the future, I plan on having a Drobo (w/ 3 1tb drives in it to start) as an external drive to duplicate the NAS to. This will be done daily. And weekly, I plan on swapping that Drobo out for another that I'll keep at my house. Further down, I may have another server, possibly at my house, that will back up my office data nightly. Seems easy enough, but will still probably have Dave come down and help set it up. Or Chuck, if he wants to come to Idaho. I could do tape, and might eventually. I have had bad luck w/ tape drives in the past, as well as them being very expensive. I'm still kicking around ideas for this, so it's not a done deal just yet.
