How would you respond? Would you be ready?
by Jim Heynderickx It’s been a fun two days. Early yesterday morning, I found that I could access the Outlook Web Access page, but not log in. At school, I discovered that no one could log in to anything. Reason: our Windows 2000 Server Primary Domain Controller had gone south. Corrupted files. Six services down. The auto-fallback to the second domain controller had not occurred. Yep, it was Monday. It didn’t take long to take down the failed DC, and have the secondary take over. People could log in, get email, etc. We worked on the primary DC, and made the mistake of booting it back onto the network where it promptly knocked out all authentication again. Back to the secondary. Then we noticed strange problems, such as the proxy server not relaying internal server names. No logging into SharePoints. Solution: the primary DNS number for the shut down DC was still dominant and not handing off to the secondary for DNS resolution. Changing the secondary to primary on the network settings solved that. Late Afternoon: things seemed stable on our secondary Domain Controller. Action plan: build a new DC tomorrow on a new machine, and bring it into the domain and make it primary. Problem: 200 OS X machines with fixed IPs pointing at old primary DC. Early this morning: no authentication. Backup DNS on secondary DC had shut down. We brought it back up. Consultant arrives to help with new DC. Good news: he can “swing” the primary roles from the disconnected DC to the secondary to make the secondary the new primary DC. Meanwhile, we have a machine ready to go as a new DC, to which the primary roles will be swung to soon after. Even better news: he can name the new DC the same as the old DC, and give it it same IP so we don’t have to recode network settings in mucho machines. In the end, we’ll likely do a format and rebuild on the old, failed DC, and bring it back as a third DC and a secondary DNS, so we have another layer of protection for next time.