Although the exposure was of a single file of an outdated copy of ESX Server, more source code release has been promised by Hardcore Charlie, the hacker claiming responsibility. Enough code may end up in the public arena to reveal the hypervisor's inner workings and give serious hackers clues and guidance on how exploits might be made to work.
It moves the discussion of the security of ESX Server security back to center stage. The lack of activity in the wild--the absence of successful hacks--had almost removed ESX Server as a subject of continued concern. Hardcore Charlie has put it back on the table.
If even more code gets published, that act at least partially takes away an advantage that VMware has enjoyed over the open source code hypervisor competition. Both Xen and KVM source code were published for everyone to see. ESX Server was proprietary and would stay that way, an extra precaution to IT managers worried about so much operational control moving to a single piece of software.
Enterprises have overwhelmingly voted for VMware, in my opinion, in part because of a high degree of trust in the company. The academic roots and deep x86 instruction set expertise of founder Mendel Rosenblum made it the early brainiac of virtualization. Likewise, IT leaders liked VMware’s ability to set goals and meet them with solid code, with one or two exceptions. Keeping its source code restricted to a few trusted hands was something IT could agree with at an early stage of virtualization.
[Want to learn why an ultra-secure virtualization environment in the form of a public cloud has not been any guarantee of success? See Why Specialized Clouds Will Be A Tough Sell.]
But should customers assume these attributes mean ESX Server can never be breached? Of course not, and even those with full trust in the company never quite got to that point of false confidence. On the contrary, with ESX Server source code in so many hands of partners and co-researchers, as this incident illustrates, it may be time to start assuming the opposite.
Does that mean it's vulnerable to exploits? Probably not much more so than before. It’s not going to be subject to any increase in the kinds of harassment launched by script kiddies and other lightweight creators of malware.
But the source code disclosure forces us to view ESX Server in a different light. Somewhere in the world, there are substantial resources behind efforts to compromise ESX Server, given its prominence in business and government. Think of eager foreign security agencies--Iran has invested substantially in cyber warfare, for example--and what they might try to do inside Defense agencies if they could invent an agent that sat on the hypervisor viewing all the traffic that flows through the hypervisor. That would give them a sniffing point and possibly a control point for everything going on with a set of virtual machines running on a given host. It might give them an avenue out to other hosts and other virtual machines.
This is the long sleeping fear of virtualization users: the compromise of the hypervisor leading to the compromise of data and operations on multiple virtual machines. It hasn't happened yet, to public knowledge. But no software is invulnerable to compromise.
As a warning for what might be possible, consider the Oct. 23, 2009, online version of the MIT Technology Review. Eran Tromer, a post doctoral researcher in MIT’s Computer Science and Artificial Intelligence Labroatory, and three colleagues at the University of California at San Diego, claimed that a snooper could target a multi-tenant server in the Amazon cloud by launching VMs at the same time that the snooper's target user did so. Simply by timing the launch, he said, spying virtual machines could land on the same host as target VMs.
Tromer cited other research that claimed one virtual machine on a shared server can detect activity in another VM by observing hypervisor activity and, under some circumstances, surmise what might be happening. For example, when an idle virtual machine becomes active, the spying VM might assume it was processing a log in name and password. By listening to the timing of keystrokes feeding letters into the system, a snooper might guess what the words were, since the layout of the Qwerty keyboard lends itself to predictable timing of letter sequences in some cases, Tromer said, citing the outside research.
Amazon spokesman Kay Kinton said Amazon had taken steps to make it less predictable for one customer to guess where another customer's virtual machines were running in its server infrastructure. In its early days, EC2 assigned customers IP addresses sequentially, and if, somehow, one customer knew when another was launching VMs, that would open the possibility of the snooper launching VMs at the same time to pick up an IP address on the same server.
Kinton told the MIT publication that Amazon has taken measures to prevent such mapping of VM location but didn't specify what they were. For one thing, Amazon is believed to no longer use strictly sequential IP address numbers. In addition, Amazon officials said EC2 operational characteristics were nothing like the lab circumstances where the keystroke detection and analysis allegedly worked.
In addition, other researchers have been unable to duplicate the feat. It remains unknown whether the MIT and San Diego researchers were describing an observable phenomenon or launching an urban legend. It had just enough plausibility to make virtualization users nervous.
The best antidote to such fears is to implement a higher degree of protection on the hypervisor. One of the experts at operating virtualized environments is Harris Corp., a contractor doing business on many fronts with the Department of Defense. Harris built an ultra secure, Cyber Integration Center in Harrisonburg, Va., that relied on VMware’s ESXi Server. It also imposed security measures, such as calling up a clean copy of the hypervisor from a secure source with the start of a new virtual machine. As the fresh copy launched, Harris matched the bit count of each component with its known number and performed other inspections as a check on whether anything had changed en route to deployment.
It did the same thing with applications and operating systems, capturing a digital fingerprint of the system as whole that would allow it be rechecked each time it was activated. Then it layered an intrusion detection system over the whole, watching for any errant behavior . The system did not need to identify malware or locate compromised code to trigger protective measures. It only needed to see an aberration, some disallowed behavior, to expunge the virtual machine and hypervisor and start over. Even if an intruder got into such a setting, his chance of setting off an alarm and getting eliminated was probably at least as great as his chance of doing persistent harm. Unfortunately, Harris' Cyber Integration Center, an excellent test bed for secure hypervisor operation, closed opened in May 2010 and was announced as headed for closure in February of this year, due to lack of business. Harris had implemented several patented security procedures inside it which may have to wait for a future opportunity to be engaged in a real world, cloud data center test.
At the same time, this failed attempt at an extra secure facility carried a burden of extra charges. Extra security costs more, and there have no proven cases of compromised hypervisors. Harris' closed data center would seem to be a marketplace verdict that existing hypervisor security is good enough.
Nevertheless, that conclusion may come under renewed scrutiny, if Hardcore Charlie publishes 300 MB of VMware source code, instead of a single file, on May 5, as he's said he will do. To be sure it is outdated code, but the core of the how the hypervisor operates may end up exposed. It may be best before May 5 to assume the much of the code for ESX Server is open, or soon will be one way or another, and take adequate protective measures, depending on the sensitivity of the systems. For customers, it's hard to know exactly what those measures should be. Third party security vendors will come up with offerings, but ultimately, it will fall on VMware's shoulders to build protections into standard versions of its products.
Such a move adds expense, but that may be the only way to make sure that the hypervisor is a control point for healthy VM operations, not monitoring and manipulating virtual machines in ways their owners never intended.
InformationWeek is conducting a survey to determine where enterprises stand on their IPv6 deployments, with a focus on security, training, budget, and readiness. Upon completion of our survey, you will be eligible to enter a drawing to receive a 16-GB Apple iPad. Take our D-Day for IPv6 Survey now.
Posts
