What do you mean "we can't get in"?!

Blogger: Ramon Krikken

Hypothetical situation: one day you come in to the office, and find out your senior network administrator was hit by the proverbial bus. How sure are you that you have all the information needed to continue managing the network? If instead he went to work for the competition, how sure are you about the integrity of your network?

If the answer is anything but "We're as sure as we can be. This is accounted for in our security program, business continuity plan, and disaster recovery plan. And yes, we do security audits and BC/DR plan tests on a regular basis", it's probably time to check and make sure.

Real situation: the city of San Francisco puts its its senior network administrator on administrative leave, and then finds out no one else could access their city-wide "FiberWAN" network routers. The official details were vague, but an unofficial story purportedly showed the administrator's side of the story . It appears that the situation isn't the usual cut and dried password recovery or reload and restore from backup.

Does it all sound far-fetched? Well, possibly, but in challenging economic times, and even during better times when mergers and acquisitions take place, it is certainly not unthinkable that someone will leave on bad terms at some point. It may be rather unlikely, but it could still happen ... apparently with potentially significant consequences.

I say potential, because the city can't know. After all, without access to the router, they don't know its configuration. The solution could have been as simple as reloading and recovering the password, but that could lead to losing the configuration if it isn't written in NVRAM and isn't backed up.

Or, it could be that password recovery was disabled with a single configuration statement, available in modern Cisco IOS releases, which results in having to clear the configuration and load it from backup if you lose the password. And no, there doesn't appear to be bypass mechanism, even in hardware, for this.

So there you are with an availability-critical system, designed and configured with priority for confidentiality and integrity of its root password without compensating availability controls (backup, configuration audit, segregation of duties, etc.) ... and you have no idea whether you can easily fix it or not.

So it's certainly possible to assume that a product like this is designed with some fail-open recovery mechanism, and that you should be able to trust high-level administrators to behave a certain way, but it's definitely a mistake to do so. Indicative of a defective risk assessment process, or weakness in the governance process, it's something that needs to be fixed.

Of course getting some basic network management security right is important: proper authentication mechanisms, backups, command restrictions, logging and auditing for configuration changes, and so on. But this all presumes that you've actually thought about it, and that you have more than one person in charge of the network.

In the end, only by having strong governance and proper controls throughout the information systems life cycle, will you be able to have assurance that security objectives - including availability and accountability - can be properly met. This includes assessing risks posed by the implementation itself; something that is easily forgotten.

It's not necessarily a lot of work, it's just creating the methodical approach to it all. A bit of governance goes a long way.