Saving time: How a few committed people helped hold up the Internet...again

Presented at O’Reilly Security.

Susan Sons tells the story of the ongoing intervention to save the troubled but ubiquitous Network Time Protocol’s reference implementation, explaining how social, technical, and resourcing challenges came together to threaten a core piece of Internet infrastructure and how these challenges were overcome.

In February, 2015, NTP—the reference implementation of the Network Time Protocol, which tells nearly every device in the world what time it is—was deeply troubled. Vulnerabilities were going unpatched for months or years, and everyone from script kiddies to APTs were having a field day with this essential service. The code base was not yet C99 compliant—that is, by 2015, it had not caught up to the coding standard of 1999—and documentation was years out of date. The build system was brittle, and the code, while open source, was locked up in a proprietary repository that drive-by contributors could not access. The sole maintainer was a solitary, aging coder operating on a rotting infrastructure.

The security implications of this mess couldn’t be more dire: accurate time is crucial to the stock exchanges and banking and finance generally, as well as cryptography, GPS navigation, scientific experiments around the world, and countless other applications. However, this crucial service is best known in some circles as the ideal jumping-off point for amplification of DDoS attacks on other systems and a great entry point for taking down a system running the service itself.

Plenty of people talk about how to secure shiny, new software at a company with a clear goal and leadership structure. Unfortunately, the money you pay your ISP goes mostly to hardware and their own operations or profits. The software that supports the core protocols of the Internet is, for the most part, maintained ad hoc by open source volunteers. When something goes horribly wrong, it generally isn’t clear who is in charge of fixing it or where the resources to do so should come from. Susan discusses how her team took over this aging, yet critical, behemoth and:

• Organized stopgap resources for a software project that no one owned, no one makes money from, but everyone depends on;
• Built a functional team for securing and improving a delicate piece of software that includes esoteric algorithms almost no one had deep knowledge of in 2015;
• Navigated social and political barriers to getting the job done;
• Overcame more than four decades of technical debt;
• Staged a refactor that would make many developers hide under their beds;
• Built the build, test, and support infrastructure necessary to ensure stability for a user base we have limited information on but which includes critical-but-esoteric applications such as real-time operating systems used in scientific research facilities for high-precision timing of physics, astronomy, geology, and other experiments; and
• Managed to still like one another a year later.