Recent problems with CUDN central nameservers

2007-07-16 - News - Chris Thompson

In the normal state, one machine hosts 131.111.8.37 [authdns0.csx] and 131.111.8.42 [recdns0.csx] while another hosts 131.111.12.37 [authdns1.csx] and 131.111.12.20 [recdns1.csx]. (On each machine the different nameservers run in separate Solaris 10 zones.) On the evening of Friday 13 July, work being done on the second machine (in preparation for keeping the machines running during the electrical testing work on Sunday) caused it to lose power unexpectedly, and recovery took us some time, so that the authdns1 and recdns1 services were unavailable from about 17:24 to 19:20.

Unfortunately, our recovery procedure was flawed, and introduced creeping corruption into the filing system. The relevant machine became unusable at about 14:45 today (Monday 16 July). In order to get the most important services functional again,

  • the recursive nameserver at 131.111.12.20 [recdns1.csx] was moved to a new Solaris 10 zone on the machine already hosting authdns0 & recdns0: this was functional from about 15:45 (although there were some short interruptions later);

  • the non-recursive authoritative nameserver at 131.111.12.37 [authdns1.csx] had its address added to those being serviced by the authdns0 nameserver at about 20:10 this evening.

Of course, we hope to get the failed machine operational again as soon as possible, and authdns1 & recdns1 will then be moved back to it.

Please send any queries about these incidents or their consequences to hostmaster@ucs.cam.ac.uk.