As more and more of our lives are lived online, there is more and more incentive for criminal elements to exploit vulnerabilities in the system.
Major companies whose livelihood depends on the Internet recognize that maintaining consumer trust is paramount, and that high-profile exploits erode that trust. Consequently, many major companies encourage their engineers to seek out and find security vulnerabilities part-time. In addition, Internet companies such as Microsoft, Google, and Facebook offer bug “bounty” programs, to encourage security researchers in the community who discover and report bugs responsibly by rewarding them with cash payments which can reach into the tens of thousands of dollars.
Google has recently upped the ante with an initiative they are calling Project Zero. Project Zero is going to be a well-funded team of dedicated security researchers who will spend their time identifying bugs in widely used Internet software (not necessarily Google software) and working with software developers to get them corrected before criminals are able to find and exploit them.
As Google says:
“You should be able to use the web without fear that a criminal or state-sponsored actor is exploiting software bugs to infect your computer, steal secrets or monitor your communications.”
When bugs go public, whether a patch is available or not, the software developer often issues interim guidance with steps to take in order to reduce the impact or risk of the bug being exploited, until a patch can be developed or deployed to all vulnerable servers.
The challenge administrators then face is how to ensure all the servers under their control are rapidly configured in the recommended way, and then to rapidly deploy the patch once it’s been released and tested. The testing process is an important step which often delays the deployment of security patches.
The notorious Heartbleed bug, discovered earlier this year and which affected critical software broadly used throughout the Internet, shines a spotlight on this problem. Estimates indicate that up to half a million web servers were initially impacted by Heartbleed. Despite significant coverage in the technology world and even mainstream media coverage, six weeks later over ten thousand servers were still estimated to be affected by Heartbleed.
Administrators who struggle to update their servers in a timely manner are often trying to do it manually, and discovering that it’s a very time consuming task. Manual steps are fine if you’re managing one or two servers, but it simply doesn’t scale as the number of servers increases.
Fortunately, there are many tools which help automate this time consuming task. Many of these tools focus exclusively on managing the software patching process, but I’d like to talk a little bit about configuration management tools, a more flexible tool to have in your arsenal.
Configuration management tools like Puppet make it easy to define, track, and manage the configuration of software and infrastructure components in an automated and agile way. As one might expect, these capabilities are enormously useful when needing to make widespread changes to dozens, hundreds, or thousands of servers simultaneously, and particularly valuable following the disclosure of a major software vulnerability.
In the event of a large-scale bug, tools like Puppet can be used to rapidly audit every server in your environment to determine which ones are affected. If a patch is available, an updated software component can be pushed out.
If a patch isn’t available, other actions can be automatically performed to the vulnerable servers: vulnerable servers can be taken offline, or moved to a quarantined network. They can have a non-critical component disabled, or a firewall change can be automatically made to block access to a specific port, removing a vector for exploiting the vulnerability.
Configuration management tools give administrators much more flexibility, power, and control in determining how best to respond to a serious software vulnerability.
Just as important as the initial response to the vulnerability, is to ensure that the configuration change which mitigates the vulnerability remains in place. In a dynamic environment, it can be difficult to ensure that all servers are configured 100% correctly, 100% of the time, as configurations tend to “drift” over time.
In addition to helping rapidly identify and remediate vulnerable servers, tools like Puppet can be used to enforce the correct configuration and ensure that the remediated server stays remediated. These are capabilities which simply aren’t available in most dedicated patch management tools, and are only scratching the surface of what configuration management tools can do. We’ll explore configuration management tools further in future blogs.
The Internet is only becoming more central to our lives, and every year more and more business is conducted online. Security vulnerabilities will continue to be discovered, and administrators can leverage configuration management tools to demonstrate extreme agility in responding to these threats.
What tools are you using to manage your servers and respond to threats like Heartbleed? Are configuration management tools part of your tool box? Why, or why not?