Stealth secondaries and Cisco Jabber

2020-04-30 - News - Tony Finch

The news part of this item is that I've updated the stealth secondary documentation with a warning about configuring servers (or not configuring them) with secondary zones that aren't mentioned in the sample configuration files.

One exception to that is the special Cisco Jabber zones supported by the phone service. There is now a link from our stealth secondary DNS documentation to the Cisco Jabber documentation, but there are tricky requirements and caveats, so you need to take care.

The rest of this item is the story of how we discovered the need for these warnings.

The context

Cisco Jabber is designed around a classic enterprise-style internal/external network architecture with firewalls and DNS views, which doesn't fit the University very well. The special Jabber DNS SRV records (_cisco-uds etc.) have been set up on the phone system's own DNS servers, which are able to support the special split views more easily than the central DNS servers.

If the network requirements are satisfied then you can see the Jabber internal view records, but in practice most clients should see the DNS SRV records for the external view.

The problem

With many people working from home, our colleagues in the telecoms office found that Jabber was not working as expected. After some investigation it became apparent that the internal view _cisco-uds DNS SRV records were leaking: often Virgin Media's DNS servers would return the wrong answers, and sometimes the various public DNS resovers would as well.

This was very mysterious.

We could not find any configuration problems with the phone system's DNS servers, nor with the central DNS servers, nor with the contents of the DNS zones.

The answer

After much head-scratching and many red herrings and blind alleys, I worked out that one of the public DNS servers for the zone was configured as a secondary for Jabber's special internal _cisco-uds view. There was a 1-in-6 chance that people outside the University would get the wrong records, depending on which of our 6 public DNS servers their resolver happened to talk to.

The fix

So we've corrected the configuration mistake, and improved our documentation to reduce the risk of it happening again. But there's a bit more we can do.

One of the things that made this hard to debug was that the usual consistency checking tools such as Zonemaster did not spot the mistake. DNSviz encountered the problem, which gave me a bit of a clue, but DNSviz isn't designed to systematically examine all of a zone's nameservers in the way that Zonemaster does.

The reason Zonemaster didn't find the problem is that it examines a zone's own nameservers for consistency, but it doesn't check that all the zone's parent's nameservers have consistent delegations. In our Jabber case it was one of the parent zone ( servers that was doing the wrong thing with the child _cisco-uds zone.

We have a Zonemaster script for checking all our zones, but it currently uses a rather out-of-date version. I'm hoping that after some operating system upgrades it will be more convenient to use a recent version of Zonemaster, and it will make sense to add some extra checks so that Zonemaster can spot and complain about mistakes like our Cisco Jabber leakage.