Hi all.
Wow what a long haul..
The server went down at 11:45 GMT on Tuesday and although the user files and common drive were up and running again quite quickly I ended up with a double whammy, when reconfiguring the Power Vault so I could remove the spare drives on their own array it took the configuration of the data Array down and so I ended up with no Exchange and no Data files!!
It took me 12 hours to get to the stage where I could actually put the data tape in to start the recovery. One being I couldnt boot from the cdrom, eventually i replaced it with one from another server! Rebuild server, install windows 2000, put the service packs on and install symantec backup exec and check the data tapes. With no warning i discovered that the most recent backup was from the 31st October, the backup tapes had been ejecting each day with no warning of failure last week. Windows and Data was the first stage of the reinstall which was done under normal operating conditions. The second was to install the system state this had to be done by loading up in directory services restore safe mode. once the system state was done another reboot and ntdsutil authoritative database restore, reboot again and I could now see Active Directory! This process was completed by about 2:00am GMT.
Before performing the Exchange Disaster recovery I did some server tests to ensure the other 2 DCs were happy to replicate with the recovered server and to my horror they were not! I spent the next 6 hours trying to get the replication working and had no joy! I spoke to the MD at 8:00am GMT and we decided to get a MS guru in to take a look. They arrived at 10:00am GMT and we had a long discussion about all the steps I had performed and we eventually decided to run the System state and NTDSUTIL Authoritative database restore once again. as other actions I had tried had started causing DNS issues etc. System state and NTDSUTIL were run and ready again by 12:00 GMT. The next 4 hours was spent getting the server accepted by the other DC's and the main problem from before was not giving the recovered DC time to replicate fully!! The MS guru said it would be better to not have the exchange server as a DC so we demoted it (dcpromo). 16:00 GMT time to install Exchange.. All the users now had access to data files so we were half way there! The MS Guru said that an Exchange Disaster recovery was not an option because we didnt have the first storage group! so a standard install was performed and it actually integrated quite well it also kept some of the settings which could have only come from the server recovery!! 17:00 GMT we did a restore of one mailbox! Mine! I went home at 17:45 GMT I had been at work for well over 33 hours!!! the mailbox restored ok. We left running a full restore of all mailboxes and public folders
I went home got into bed and slept all the way till 8:00am GMT Thursday!
Knowing that the mailbox was from October 31st we decided to archive the existing email stored on the laptop to a PST file. We then created a new profile and new connection to the server uncached and that worked! The MS Guru popped in for an hour to make sure it was all ok and then left. I moved my emails from 01st November to 11th November out of the pst back up to the server. Then tried to reconnect the mailbox in cached mode but it didn't work. I did a bit of digging and found a cool trick, start outlook with the switch /resetfolders and voila Outlook started caching and all is now well! I then spent the rest of today going round doing the same for all the users. I am abut half way through and will continue tomorrow!!
The Verdict! You can make a successful recovery from a backup of just the mailboxes, you don't need the The First Storage Group or original edb or STM files.
This was a bit a of special case, this post was all about the issues with not being able to back up the first storage group or even doing a file level backup on the STM file.
Thanks for all of your inputs! If you want to know anything more technical of the steps taken give me a shout! i'll try my best to remember!!
Remember to always check your backups!! We were lucky to only lose a weeks worth of data!
I have been looking at VMWARE recently and i think this going to be something I am going to get serious about!
KLewisBPM
