Of Tylenol and Telecom

I was struck today by Chris Williams' article in the Register, "BT network 'vulnerable to Chinese attack'" regarding a new dust-up around access nodes and optical equipment purchased by British Telecom from Chinese manufacturer Huawei. In comments from Alex Allan, of the Joint Intelligence Committee, the acquistion, in combination with People's Liberation Army former research head, Ren Zhengfei, at the head of Huawei, has given rise to a series of concerns about the security, propriety, and safety of this equipment.

Following up on a mention of other Huawei/Government conflict within the article, I found this piece from the NY Times on the dissolution of an offer by Huaweii to purchase 3Com, in cooperation with Bain Capital Partners, based on a concern over the likely US Federal Government intervention given the popularity of the 3Com TippingPoint iron in trusted government installations. In this report, Ileana Ros-Lehtinen, the ranking member of the House Foreign Affairs Committee, said that the deal was one that “threatens the national security of the United States.” As though somehow ownership by a company within a foreign nation would unavoidably result in the tainting of the equipment in some unique and malicious way.

Trust me. As you have read here before, I am not one to overlook the possibility of malicious code in equipment, packaged software, and even through the open source community. I would, however, like to draw a little bit of even-handedness into the discussion, because I believe that ordering the words like this, "Chinese-developed" "Vulnerable/Malicious" "Software" places far too much emphasis on pandering to a generalized fear of that which is foreign, as opposed to a more balanced view: Fear of that which is software. In this case, at least according to the reporting, it is the fact that the maker of the equipment is headquartered in China which makes this equipment acquisition a headline and an issue. The real issues should be quotes like this one, "ministers were told...the unspecified measures would not mitigate built-in vulnerabilities", which would naturally apply to almost any equipment, of any vintage, from any country, because so little of it has ever been looked at to identify those vulnerabilities.

So what does Tylenol^TM have to do with any of this? To my mind, the approach taken by Johnson and Johnson, the makers of Tylenol, during the Tylenol Scare of 1982, makes for a simple and elegant model for BT and any other firm to follow. For those of you who have forgotten, in 1982, there were 7 tragic deaths resulting from victims taking tainted doses of Tylenol capsules. Someone had opened the capsules and filled them with cyanide. Facing this tragedy, an unknown assailant, and a general public potentially exposed to more tainted product, Johnson and Johnson recalled it all, about 31,000,000 bottles, and instituted new methods of tamper-resistance: in manufacturing, in packaging, and in user-visible seals. Tylenol's brand recovered, and no more people were injured (with the exception of a woman who was reportedly a victim of a similar attack in 1986, at which point Johnson and Johnson permanently discontinued capsules as a product). This example can provide guidance to firms concerned with their own component and service safety today.

#1: Don't go looking for a subject to blame.
If the unthinkable happens, and you have predicted it, blaming someone else does not lessen your responsibility. If there is malicious code in this equipment, then it is the responsibility of those who buy and deploy it to find it and mitigate it, or otherwise they must remove it from the shelves or from service.

#2: Come clean immediately.
This does not mean a warning that potentially tainted equipment was deployed, it means admitting that the process of procurement ignored this potential issue, and that in the future this mistake will not happen. This admission is vital to provide confidence that the threat has been understood and either mitigated or eliminated.

#3: Improve the process.
Tamper-resistance was pretty much brand new in 1982, and yet Johnson and Johnson retooled their manufacturing to implement it, and now we find it on everything from pharmaceuticals to bottled water. Creating acquisition and validation policies that will similarly analyze and certify software seems no more daunting that a complete sea-change in physical manufacturing, so there is little excuse for not getting started.

#4: Don't make the problem any worse.
If there is really concern over this, and not just idle hand/flag-waving, then do something about it. Stop using the devices until you can verify that they are clean. I am not suggesting that one switches to a domestic product, rather that whatever product is chosen be chosen with this level of inspection in mind.

#5: Stop creating bogeymen to blame.
Whether it is China, or Cyber Jihadists, or Anarchists, or bored teenagers, the reality is that software development, support, and maintenance has become such a diverse melange of new, old, open, and outsourced code, that it is almost equally likely that there will exist maliciousness in any of an organization's applications, regardless of their source. How much development is sent off to be outsourced, how many consultants or contractors touch and application, and how many packages are truly vetted to ensure that they have not been corrupted, prior to application deployment? Far too few.

The recent disclosure by the Joint Intelligence Committee and this ensuing discusion should be put to the most positive use: It is an example of our urgent need to examine our applications for "poisonous" code with a rigor similar to that which we use in the areas of food and pharmaceutical safety. If we do, we can become more certain of our security, regardless of our sources.