No country has even come close to the U.S. in harnessing the power of computer networks to create and share knowledge, produce economic goods, intermesh private and government computing infrastructure including telecommunications and wireless networks, using all manner of technologies to carry data and multimedia communications, and control all manner of systems for its power energy distribution, transportation, manufacturing, etc.
All of this has left the U.S. as the most vulnerable technology ecosystem to those who can steal, corrupt, harm, and destroy public and private users, at a pace often found unfathomable.
I. INTRODUCTION
A. Cyberspace
B. A brief introduction to SolarWinds and her evil sisters
C. My cybersecurity beginnings, and what this series is about
D. A summary of the material and chapters to come
E. So what in hell do we do now, cyber security mavens?
9 February 2021 (Chania, Crete) – The following series is my attempt to bring together a lot of disparate strands in a way that reveals the connections between the development of computers, the internet, spying, and cyber war to understand why today we are in such a state of “SolarWinds computer (in)security”.
And it takes a village. For my source material (foundational texts and interviews) click here. This section will expand as the series continues.
I. INTRODUCTION
An enigmatic telegram arrived at the port of Dover, England just past midnight on the 5th of August 1914. It was in code so its meaning would have been lost on anyone save for its intended recipient, an officer named Superintendent John Bourdeaux: “WHERE COULD SECOND LETTER FIND YOU”
Bourdeaux was in command of the H.M.T.S. Alert – a paddle steamer cable laying ship – as it set sail.
The ship was originally designated the CS Alert, or “Cable Ship Alert. It was owned by the Submarine Telegraph Company and subsequently acquired by the UK government and designated the H.M.T.S. Alert.
The bulk of the crew were ignorant as to their mission. The weather was good. War had been declared with Germany just minutes before the telegram had been sent. The Alert was about to undertake the first offensive act of the conflict. The telegram was the signal to proceed.
The Alert arrived at its first destination at 3.15 a.m., lowered its hook to the seabed and began to dredge. After twenty minutes it had hauled up the end of a cable. Bourdeaux had been instructed to bring it to the surface to check it was the right one. Then he was to cut it and proceed as instructed. By the time a fourth cable was reached, it was six in the morning. Heavy rain and wind were battering the ship, making the grappling much harder.
At this moment, a foreign ship was spotted for the first time. With relief (his ship had no protective escort vessels), Bourdeaux realized it was from Britain’s ally, France. The curious French captain pulled his ship alongside and asked “Alert, what are you doing?”
“Cutting German cables!” Bourdeaux replied with disarming honesty for a man on a secret mission. Later that night in his diary, reciting the events of the day, Bourdeaux wrote:
“The French captain and his crew gave us a splendid round of cheers which we heartily replied”.
By ten in the morning, the Alert was heading back to Dover. The Alert’s cutting of cables marked the beginnings of information warfare and the establishment of an industrial-scale global intelligence operation.
For some background on the World War I “cable wars” click here.
The First World War may seem a slightly odd place to start a history of computers and spies and the internet and cyber warfare and the SolarWinds attack (actually not one attack but by all appearances a multi-headed serpent) since it predates the electronic world. But starting here serves two purposes:
Firstly, it is the time when “signals intelligence” – studying your adversary’s signals (mainly, although not exclusively, communications) – came into being. As Gordon Corera discusses, in much greater detail in his book Intercept, the Alert’s cutting of cables was necessary. These cables gathered material and information in bulk from a commercial cable infrastructure that Britain dominated. Computers were a long way off. But the concepts that would underlie their use in modern espionage – from mass sifting of messages to traffic analysis of data and, of course, code-breaking – would be learned in these years (there were also concerns over privacy).
Secondly, the reference point of the pre-electronic age allows us to see much more clearly what the computer has and has not changed about spying. As Corera points out, Britain’s First World War system parallels the modern intelligence system that the UK and US would built a century later based on access to the cables that convey worldwide data traffic. But those data pipes carry something far richer than the telegraph clicks of the past, and so modern computer-based “digital intelligence”, with its updated form of packet inspection, offers something qualitatively as well as quantitatively different from what was undertaken under fluorescent lights in the Great War.
And of course, for those of you that know your technology history, when the Alert severed those telegraph cables, it did not cut Germany off completely. There was the new medium of radio. Germany (like Britain) had been investing in high-powered kilometre-long radio antenna. For the German and British navies radio was changing warfare, allowing ship-to-shore communication for the first time and centralized command and control.
And German radio would be the crucial stimulus to the conception of Britain’s communications intelligence machine. A decade before the war, a British telegraph company had set up a radio receiver on a small headland above Porthcurno beach in Cornwall, still known as Wireless Point. It was there to spy on the commercial transmissions of Guglielmo Marconi, the flamboyant businessman and inventor who had just sent a radio message across the Atlantic. Britain’s Director of Naval Intelligence saw the immediate value of this new upstart technology.
Britain was learning – as were many countries – that reading someone’s messages to gain intelligence might be more useful than stopping those communications entirely (lessons certainly learned since the days of Francis Walsingham and Queen Elizabeth I). Just as modern commerce today depends on the internet, international trade then depended on the telegraph. Clever detective work could help expose German attempts to evade the economic blockade. Was a company in Amsterdam seeking to buy metal from America really a front for the German firm Krupps? If so, it would be placed on the “Secret Blacklist”.
The next step was using this intelligence as a weapon. When the Irving National Bank of New York was found to be conducting just a little too much business with Germany, its cables were delayed until it changed its behavior. The same trick was used on all commercial traffic to the Netherlands because the Dutch were allowing sand, gravel and cement for the German war effort to transit. This, a British official remarked, inflicted an “infinity of harm” until they agreed to stop.
And the advance of technology? In February 1918, in Frankfurt, Germany, Arthur Scherbius patented what he would eventually call “the Enigma machine”. He built primitive prototypes but the first, fully-operational machine only saw the light of day in 1923. In his book “Turing’s Cathedral”, George Dyson notes:
“… in early 1918 a cryptographic machine had been invented by the German electrical engineer Arthur Scherbius, who proposed it to the German navy, an offer that was declined. Scherbius then founded a company to manufacture the machine, perfected the machine, branding it ‘Enigma’, for enciphering commercial communications, such as transfers between banks. Upon seeing it, the German navy quickly changed its mind and adopted a modified version of the Enigma machine in 1926, followed by the German army in 1928, and the German air force in 1935”.
More on Enigma later in this series.
A. Cyberspace
Perhaps because I am an “armchair physicist” (physics was my second, underdegree at university) I have felt a kinship with Phil Quade (Chief Information Security Officer at Fortinet) who looks at our failures in cyber and technology (and disasters in general) as our failure to acknowledge the fundamentals of physics and chemistry. He is fond of saying that as we solve problems and improve technology, we must work with, not against, the foundation of the laws of mass, force, energy, and chemical reactions. And, like the physical world, he says:
“cybersecurity has its own set of fundamentals: speed and connectivity. When organizations ignore these fundamentals, distracted by sophisticated marketing or new products, we suffer the consequences. We end up with solutions that solve only part of the problem or that simply stop working (or stop us from working when put to the test of real-world conditions).
That’s partly because, to date, cybersecurity has been treated as a cost of doing business, as opposed to a foundational set of primitives and rules that are leveraged to achieve greater things. To build a cybersecurity foundation that will work now and continue to work in a world exponentially faster and more connected, we must start treating cybersecurity more like a science”.
Echoing these thoughts is Chris Inglis, a former Deputy Director of the National Security Agency who now teaches Cyber Security Studies at the United States Naval Academy. He is also on the Advisory Board of the cybersecurity firm Securonix. To Inglis, cyberspace is a domain, with its own rules. You’ll hear more from him later in this series but for this introduction it is important to know his core belief: that we need to lay out a foundation for understanding the essential elements of cyber. As he has said:
Cyberspace is a literal place. Of note, the term cyberspace includes, but is not limited to, the sum of hardware, software, and interconnections that are collectively referred to as the Internet. One of the most important things that the curiosity-minded pioneers of the Scientific Revolution did was to intellectually (and sometimes literally) peel apart a common thing—a leaf, a parasite, a hillside—to better understand what it was made of and how its parts were connected, trying to understand how each layer worked and helped govern the whole.
Various writers have argued that cyberspace is not a domain, since it is man-made and therefore lacking in the enduring and unchanging properties inherent in domains resulting from immutable laws of nature, time, and space. The case for cyberspace as a domain is found in the simple fact that, on the whole, it has unique properties that can be understood, or purposely altered, only by studying cyber as a thing in its own right. It is a center point that is the result of integrating diverse technologies and human actions, while it also serves as a resource enabling widespread collaboration and integration.
Mention the term cyberspace in any otherwise polite conversation and the mind’s eye of the listener immediately conjures up a jumbled mess of technology, wires, people, and communications racing across time and space or stored in vast arrays of storage devices. The resulting rat’s nest of technology, people, and procedures then offers such a complicated and undistinguished landscape that, within the context of the conversation, further use of the word cyber could mean anything, and often does. It is important, then, to tease out the constituent parts of cyberspace to describe their characteristics, their contribution to the overall effect, and their relationship to each other. This, in turn, will yield a taxonomy or roadmap that allows focused discussions about discrete aspects of cyberspace that can be considered in the context of the whole.
He then teases out the layers of cyberspace: physical geography; communications pathways; controlling logic and storage; devices; and people. We’ll discuss this later in the series in the chapter “A Brief History of Cybersecurity”.
But it makes for a nice introduction to my first video clips because Inglis often notes that cyber security is “a sisyphean task”. Because people still believe it is a problem to be solved rather than controlled. Just like rampant misinformation, this is not like a plumbing problem you “fix”. It is the same with cyber security.
So cue up this short video clip from a long interview I had last year with Michael Daniels (the former cyber security czar for the Obama administration, and now President of the Cyber Threat Alliance) who puts the required mindset in perspective:
Plus a chat with Alexander Klimburg (author of “The Darkening Web” and a cyber expert) who, in this clip, discussed why a big country like the U.S., makes an easy target:
I’ll have the longer versions of these videos later in this series.
The computer was born to spy. The first computer was created in secret to aid intelligence work, but all computers (and especially networked computers) are also uniquely useful for – and vulnerable to – espionage. The speed and ingenuity of technological innovation has often blinded us to understanding this historical truth. We have a generation growing up with the latest technology who can make use of the latest tools, which emerge at dizzying speed, driven by the desire and ingenuity of developers and the public’s appetite for the new.
Let’s move on and try to unpack some of this.
B. A brief introduction to SolarWinds and her evil sisters
[ The tactics, techniques and procedures employed in the SolarWinds attack and contemporaneous/subsequent attacks will be examined in detail in a later chapter. Herein an overview. ]
As widely reported, FireEye announced on 16 December 2020 it had suffered an enormous cyber attack, with, among other things, its Red Team tools stolen. FireEye, whose clients include many agencies of the United States government, said it was a very sophisticated attack, one of which they’ve never seen before. They asked for Microsoft’s help to investigate and unleashed 250 steps for companies to protect against potential attacks being deployed using these tools.
While probing for their own hack, they discovered that SolarWinds had been hacked. SolarWinds is a publicly traded company that provides software to tens of thousands of government and corporate customers.
The attackers gained access to SolarWinds software before updates of that software were made available to its customers. Unsuspecting customers then downloaded a corrupted version of the software, which included a hidden back door that gave hackers access to the victim’s network.
It is what is called a supply-chain attack, meaning the pathway into the target networks relies on access to a supplier. Supply-chain attacks require significant resources and sometimes years to execute. They are almost always the product of a nation-state. Evidence in the SolarWinds attack pointed to the Russian intelligence agency known as the S.V.R. (the Russian Foreign Intelligence Service) whose tradecraft is among the most advanced in the world. It’s long been considered Russia’s most advanced intelligence agency in cyber operations.
As the drama and details unfolded I was mindful of three things Allen Woods told me:
1. The security breach involved multiple tactics and there is evidence of additional initial access vectors, other than the SolarWinds Orion platform.
2. The other thing to consider, very, very carefully, is that to date, nothing of this nature has been reported as affecting the Googleverse, Apple, IBM or AWS. That can only mean that this is initiated as an OS level attack specifically and exploiting key elements of the Microsoft architecture.
3. It also means that whoever did this was highly competent technically and highly observant in spotting the nature of the SolarWinds software and its role and customer base, and then targeting the company. This was a quite brilliant attack.
Plus something I learned years ago at FireEye and Crowdstrike cyber intelligence workshops: the U.S. military and U.S. spy agencies, plus U.S. law enforcement, and the U.S. diplomatic corps all have roles in “cyberwar” but they also have limiting boundaries. This necessitates handoffs and generates turf battles between the organizations and within them. The Russians are in an opposite position: they excel in information warfare because they seamlessly integrate cyber operations, influence, intelligence, and diplomacy cohesively; and they don’t obsess over bureaucracy; they employ competing and overlapping efforts. [More on Russia’s cyber operations shortly]
Blasting out of the shoot with a quick series of analysis on Linkedin and on his company website was Steve King. Within a few days he profiled a report by ReversingLabs which revealed that the Russian hackers (they did not say “Russia” but pretty much everybody else did) compromised the SolarWinds software build and code signing infrastructure of the SolarWinds Orion platform as early as October 2019. The plan was to compromise the build system, quietly inject their own code in the source, wait for the company to compile, sign packages and verify updates.
As Steve noted “it worked”. It also spotlighted the next generation of compromises that thrive on access, sophistication and patience. Said Steve:
The genius of this breach, which hid in plain site for months and now possibly years, behind a well-known and trusted global software brand, is that it affords access that a phishing campaign could only dream about. SolarWinds’ support advisory, warning its products may not work properly unless their file directories are exempted from antivirus scans and group policy object (GPO) restrictions, opened and held wide the door through which the bad guys waltzed undetected. My family, who wants to know how something this serious could have happened when we spend tens of billions every year on cyber security, thinks we walked out and left the keys on the porch. A fitting analogy, except that the key ring seemingly held keys to everyone else’s house in the world as well.
He continued in a series of quick-fire, detailed missives across Linkedin for the last two months as more information was revealed, making the point that the SolarWinds hackers put in “painstaking planning” to avoid being detected on the networks of hand-picked targets. The second and third-order effects would be difficult to comprehend, he emphasised.
Most cybersecurity analysts began to form a general consensus. Clearly these cyber adversaries were using advanced tools and resources and were able to infiltrate even the most sophisticated IT environments, tactics we have seen with less sophisticated tools through previous phishing attacks, software vulnerabilities, and other supply chain compromise. Once they established a foothold, they followed these well-established steps in the attack chain:
– Attempt to steal and abuse the identities and credentials of employees or authorized third parties.
– Use these legitimate credentials to move laterally and vertically through the network, looking for high-value targets or to establish persistence. Because attackers appear to be “authorized” users, organizations have a hard time detecting their presence.
– Target privileged account credentials that provide special access to systems or abilities that reach beyond those of a typical user – and work to escalate these privileges until they reach the confidential information they intend to steal or services they wish to disrupt.
– So once these threat actors gained access to internal networks, the hackers escalated access to gain administrator rights and then moved to forge authentication tokens that allowed them to access other cloud-hosted resources inside a company’s network, without needing to provide valid credentials or solve multi-factor authentication challenges.
The SolarWinds breach and the resulting attacks exhibited all of those tried-and-true tactics. Plus, as David Walker and Garett Moreau and others have pointed out some of the operations security (OpSec) methods used by the attackers included methodically avoiding shared indicators of compromise for each compromised host, and exercising an extreme level of variance to avoid setting off alarms.
Worse, as outlined by many, with dramatic cloud migrations underway, and the adoption of transformative digital technologies, the enterprise attack surface is expanding with greater privileged access present across these decentralized environments. Attackers know this, which is why securing privileged access matters more today than ever before.
Oh, and a short note on the semantic battle: was the breach an attack or was it espionage? I’ll let Sue Halpern (who has written extensively about cybersecurity) have the last word on this:
An attack demands a response. Espionage can be dismissed as business as usual—it’s what nation-states do. An attack in the physical world is unmistakable: a bomb explodes, guns are fired, the targets are people and property. In the digital world, where ordnance is constructed from zeros and ones, the distinction is less clear: computers are compromised, networks are infiltrated, and software is weaponized in secret, behind a quiescent scrim that may remain intact for months or years. What initially appears to be a spying operation ultimately may turn out to be an attack—either digital or physical—with a long lead time. Although the consensus seems to be that the SolarWinds breach was straight-up reconnaissance, the truth is that we don’t yet know. CISA [the U.S. Cybersecurity and Infrastructure Security Agency] continues to update its assessment, providing new information about the mechanics of the operation as they are identified.
Consequently, the U.S. has to regard itself as remaining under attack until whatever malicious code has been delivered into its systems has been eliminated, root and branch.
And yes, it’s a mess – and not entirely unlike the COVID response. The impact of things that “shouldn’t happen here” escalates because the country places huge efforts on preventing threats, and less effort on managing successful threats. Once a virus, either physical or electronic, gets into the country there seem to be few internal firewalls. While this reflects the country’s trademark freedoms – no internal borders – it also reflects a perhaps antiquated view that threats come over the horizon in knowable forms. What happens when the threats are invisible and propagated internally?
Thus the catch-22. A U.S. reliant on both (1) being on a global network, and (2) keeping bad actors out appears to be a cyber-nation with weak fundamental view of security. More than likely, any interconnected system should presume it’s going to be compromised. That’s one of the design principles of the blockchain that sticks. Presume your network is insecure and design your protocols to deal with it.
Alexander Culafi of TechTarget started assembling a SolarWinds timeline but he would shortly be overwhelmed:
12/16/2020 — After the FireEye announcement, the FBI, CISA and ODNI released a joint statement saying the SolarWinds attacks are “ongoing” and confirms that several networks of federal agencies have been breached by threat actors.
12/17/20 — Second backdoor discovered in SolarWinds
12/18/20 — Several major technologies companies, including Cisco, VMware and Intel, confirm they were infected by the malicious SolarWinds updates
12/24/20 — SolarWinds addresses ‘Supernova’ backdoor
12/29/20 — SolarWinds statement mentions that there may be other victims
12/30/20 — CISA updates directive for federal agencies
12/31/20 — Microsoft announces breach
1/5/21 — U.S. government acknowledges Russia’s likely involvement
1/6/21 — Department of Justice confirms breach
1/7/21 — Chris Krebs, Alex Stamos hired by SolarWinds
1/11/21 — SolarWinds updates attack timeline
1/12/21 — SolarWinds reveals in an SEC filing that it has found the source of the coding believed to have been used in the recent corporate and government cyberattacks.
1/19/21 — Malwarebytes breached by SolarWinds attackers
1/20/21 — Microsoft publishes new Solorigate/Sunburst deep dive
1/20/21 — FireEye releases open source security tool
And then the hydra was revealed. Investigators were finding concrete evidence that the alleged Russian espionage operation went far beyond the compromise of SolarWinds. It turned out that a third of the victims did not run the SolarWinds software at all. Jake Perez of Linkedin News began compiling news reports that the SolarWinds breach had exploited vulnerabilities in other software by other attackers, software used daily by millions of companies producing millions of victims:
Exploiting authentication vulnerabilities and moving gracefully between different cloud accounts is the threat du jour, and while the victims have yet to be named, the battle-ground corpses will be revealed in the next few weeks.
And then security researchers found a second group of hackers was abusing SolarWinds’ software at the same time as the alleged Russian hack. The software flaw was exploited by a suspected Chinese group and was separate from the one the United States has accused Russian government operatives of using. It marked a new twist in a sprawling cybersecurity breach that every cybersecurity expert said was a national security emergency.
And so the title off this piece changed from “Beyond SolarWinds” to “Beyond SolarWinds and its evil sisters”.
For Garett Moreau, the attacks were unprecedented in their sophistication, scope and scale. However, what these attacks did have in common with other attacks is that the compromise of identities and privileged access played a critical role. But “the pace of compromise is exceeding the pace of securing the infrastructure. Cyber security is very broken and there’s no way to fix it anymore. For hackers, it’s not hard. Once you understand a single one of those bugs, you could then just change a few lines and continue to have working zero-days.”
NOTE: later in the series we’ll talk about Project Zero which operates inside Google as a unique and sometimes controversial team that is dedicated entirely to hunting the enigmatic zero-day flaws (Apple and Microsoft have similar units). These bugs are coveted by hackers of all stripes, and more highly prized than ever before – not necessarily because they are getting harder to develop, but because, in our hyperconnected world, they’re more powerful.
For Bob Carver, it confirmed what he had written about before: today’s cyber adversaries have the advanced tools and resources to infiltrate even the most sophisticated IT environments, whether through phishing attacks, software vulnerabilities, supply chain compromise or other means. Once they establish a foothold, they often follow well-established steps in the attack chain. Once threat actors have gained access to internal networks, the hackers can escalate access to gain administrator rights and then move to forge authentication tokens that allow them to access other cloud-hosted resources inside a company’s network, without needing to provide valid credentials or solve multi-factor authentication challenges. Carver also felt that the Russia hack required a new cybersecurity paradigm but feared there has been little effort to bring various stakeholders together to determine what long-term and strategic technical and policy approaches are needed.
Both Moreau and Carver bring up complex cyber attack issues, and the need for long-term and strategic technical and policy approaches. Both sets of issues will be addressed later in this series.
It also reinforces what both Moreau and Carver have written before, as well as Cliff Kittle and Joe Weiss: we’re in this state because the Internet was never meant/designed for critical systems. It was meant to be a global commons. When will this be realized? Bruce Schneier just might have called it right: “The next president will probably be forced to deal with a large-scale Internet disaster that kills multiple people”.
And there is, of course, the 🤦🏼♂️ from all of these cybersecurity mavens who know the “human element” in cyber security so well. Moreau, after reading SolarWinds’ CEO saying they could not “identify a specific vulnerability in Microsoft Microsoft Office 365 that would have allowed the threat actor to enter our environment. The incident involved the compromise of an email account through the theft of credentials” opined:
What does that tell you? Well, it tells me they were NOT using multi-factor authentication 🤦🏼♂️. How does a tech company (especially at this level) stay afloat so long with so many gaping holes in the boat / awful security practices? The attack was at the discretion of the malicious actor: Dwell time ….. Good grief; I wouldn’t trust these guys to hold water.
Steve King had one better: Kaspersky had released a fascinating study of the disconnect between security vendors and customers and the risk that results. Steve summarised:
-240 CISOs and 2000 UK workers revealed that insecure staff behaviors have actually increased since WFH.
-Over 1/3 of employees said they don’t understand their employers’ security measures and another third believe that their organizations’ security protocols are not important.
-30% have downloaded unauthorized software and connected to a mobile hotspot to bypass security controls.
-The surveyed CISOs point to a poor relationship with their cybersecurity vendors as a major reason.
-They find it difficult to action the guidance or don’t feel the information they receive from vendors is relevant to their organization or are too complicated to even attempt to share with employees.
-But alarmingly, more than half said they don’t believe vendors understand the threats their business faces.
This also goes to the value extraction issue that many vendors face due to short-sighting the post implementation requirements of most security products. Our brand perception work discovers this gap with every engagement. Closing the deal is only half the battle. If the other half is left to the client, your brand will surely suffer.
Andy Jenkinson nailed it. Last week it was revealed the court system in Washington state in the U.S. was found to be insecure, enabling infiltration, domain hijacking, takeover and breaches across a wide reaching network. It sounded like Solarwinds, it could have been. Said Jenkinson:
The blame being aportioned on China or Russia for sophisticated attacks is little more than a magicians slight of hand, concealing oversight, negligence and incompetence. Thousands of breaches are avoidable if basic security hygiene is prioritised in a disciplined manner. Ignore this critical area and you are just waiting for a breach.
Or my favorite, announced this week. An unidentified hacker remotely gained access to a panel that controls the water treatment system in Oldsmar, Florida, and changed a setting that would have drastically increased the amount of sodium hydroxide in the water supply. A legitimate operator saw the change and quickly reversed it. The hacking attempt was a serious threat to the city’s water supply. Sodium hydroxide is also known as lye and can be deadly if ingested in large amounts.
The system was deliberately set up with a piece of remote access software so that authorized users could troubleshoot system problems from other locations. The software, called TeamViewer, is a simple off-the-shelf software that 1,000s of organizations use to remote control computers. The German software company behind TeamViewer said it is one of the most popular softwares in the world and is used primarily to allow users to access and share their desktops remotely. As revealed in a series of articles in the German newspaper Der Spiegel, TeamViewer has always been a target of interest for attackers (for example the Chinese in 2016 and 2019). The astonishing thing in the Florida hack was that, as revealed in a local TV report about the issue, nobody at the water treatment system had considered a hacking as just the “access” you should expect. And in this sprawling system only one person was in charge of cybersecurity.
But the folks I have quoted … Carver, King, Kittle, Moreau, Walker et al. … are all level-headed chaps. As Bob Carver has noted (and I think he speaks for everybody) the simple truth is that cyber defense is hard. In a country like the United States, where so much of the critical infrastructure is privately owned, it’s even harder. Every router, every software program, every industrial controller may inadvertently offer a way for malicious actors to enter and compromise a network. This is compounded by the fact that, even where software patches exist, they are often not applied, and many businesses and municipalities are too cash poor to afford adequate Internet security. We’ll come back to this issue later in the series.
C. My cybersecurity beginnings, and what this series is about
It was almost ten years ago (the summer of 2011) that I was at the Johns Hopkins University School of Advanced International Studies in Washington, D.C. listening to a keynote address by Joseph Nye at a conference about “Power”. It was part of my master’s degree program (a program I would never finish).
Nye’s theories on “soft” and “smart” power need no introduction in the field of international relations. No one has done a better job addressing the evolving notion of power in a new world of non-state actors, weak state actors, and emerging economies and technologies. His latest book at the time, The Future of Power, had just been released.
He was prescient. “The Information Revolution” — the rapid technological advances in computers, communications, and software that have led to dramatic decreases in the cost of creating, processing and transmitting, and searching for information — would “flatten bureaucratic hierarchies and replace them with network organizations. More governmental functions will be handled by private markets as well as by nonprofit entities”. A much larger part of the population both within and among countries will have access to the power that comes from that information.
And then his “power shot”, so to speak. There was an explicit danger. He believed that the earlier “simple dream”– the rapidity and scope of communication would break down barriers between societies and individuals and provide transparency of such magnitude — was NOT the way this would evolve. Just the opposite: cyber networked transparency and the absence of privacy “will propel itself into a world without limits or order, forcing us to careen through crises without comprehending them. And it’s prime target: the U.S.”
His comments struck a chord and they launched me into a full study of cyber war and cyber security via a MOOC (Massive Open Online Course) program at Johns Hopkins plus tutorials and “red team” events sponsored through multiple cyber security vendors. I began attending 6-8 cyber events/conferences a year that included: Black Hat and DEF CON in Las Vegas, Nevada ; Europe’s mega event, the International Cybersecurity Forum in Lille, France; several FireEye and Palo Alto Networks workshops; the Munich Security Conference in Munich, Germany; and the Digital Investigations Conference in Zurich, Switzerland – the latter being a wonderful mix of cyber security, e-discovery and digital forensics. And, in addition to all the people I met at these events, I was aided by an enormous network of cyber security and intelligence community contacts on Linkedin.
NOTE: before I retired, I had the good fortune to work in what once were three discreet technology areas: (1) cyber security, (2) digital and mobile media, and (3) legal technology. I spent a total of almost 40 years across those worlds, although I came late to legaltech – spending only about 12 of those 40 years on the legaltech floorboards. But those 12 years of work did afford some cybersecurity opportunities. I owned a pentest company and we quickly learned how easy it was (and still is) to hack an e-discovery review center. Actually easier now given almost all reviews are now remote/cloud based.
In years gone by those were discreet areas. They are no longer discreet but continue to overlap, though most of the practitioners in each discipline seem stuck in their respective silos, rarely venturing out to learn/understand the other two. Many do make it out of their silo but only make superficial contact with their “neighbors”.
But as I have noted in many (many) previous posts, the modern human (and especially the modern technologist) moves through all three of those myriad, overlapping spheres. They are forever entangled. It is because we live in a world of exponential companies, fundamentally different in characteristics to industrial age ones.
Yes, the relentless barrage of cybersecurity attacks and warnings have given all of us “cyber security fatigue” — much to the worry of cyber experts — but we need to soldier on and attend these events to learn about cyber technology, cyber warfare and cyber security; to have the opportunity to meet the major players in cyber security; and take stock of the tendencies and trends regarding cyber attacks, and especially of the solutions now available.
This series grew out of all my research and reading related to the recent SolarWinds hack (better termed hydra attacks), coupled with my reporting on issues related to technology, intelligence and cyber security over the last 15 years. It is based on interviews and chats with a wide range of individuals who have worked in these fields for years. Many of them I met via those conferences and events listed above, as well as those I have met virtually via Linkedin. I am grateful to all of them for their assistance.
NOTE: footnotes and endnotes and quotes will indicate where people have spoken on the record or published material although there are many others who have provided advice and thoughts who would prefer I not name them. But they, too, have my gratitude.
And to step waaaaay back – or perhaps up is the better word – to the “life-from-10,000-feet level” my guess is that for some time now I’ll bet you feel like I do: history is itself the pandemic, not just the virus. In the U.S. and elsewhere, history has been in overdrive, teeming with evils, flush with collapses, abounding in fear and rage, a wounding contest between the sense of an ending and the sense of a beginning, between inertia and momentum, all with what Leon Wieseltier calls “the terribilities that occur in ages of transition”. We have become connoisseurs of convulsion. It’s not just that we are at sea. Our roiling sea is itself at sea!
I see it every day, I read about it every day – the unbalanced limited thinking much of our modern culture has accumulated and the short-term cul-de-sacs we have rushed into these last 40+ years. Even though retired, I still need to read and think and write a lot or I’ll dry up. I still keep to my life-long regimen of “read-3-books-a-month-and-internalize-them”. I still read my five monthly/quarterly magazines (I’m a tactile guy) that cover culture, politics and technology. Cruise the Net? Less and less. My “netnanny” app controls my iPhone and iPad time. I use my Mac only when I am writing.
But the problem, of course, with such vertiginous study hours is they always come with both clarities and confusions – sometimes (many times) there is a failure in the promise of illumination. Chatting with many of my contacts in the intelligence, cyber and technology communities specifically about this SolarWinds series I was invariably dragged into conversations about coups, Parler, digital platforms, the collapse of society, culture wars, and all kinds of other stuff. But, one needs these brutal attempts to get some perspective, even if partial, on our contemporary cyber and technology tumult.
Writing about cyber security and intelligence is challenging. It was a little like the situation when I began to read and research and write about COVID-19. The challenge was overwhelming. Although I have a bit of a science background (that second degree in physics, plus a life-long passion to read almost everything about all elements of science and medicine), I was venturing onto ground where I had no guarantee of safety or academic legitimacy. So it was not my intention to pass myself off as a scholar, nor as someone of dazzling erudition. It has been enough for me to act as a messenger for those who do have knowledge, and offer my own reflections on that knowledge.
I have more comfort, though, writing about cyber security and intelligence and SolarWinds where I have a more solid base. But the challenge is just as great. Because what I wish to do in this series is bring together a lot of disparate strands in a way that reveals the connections between the development of computers and the internet and spying and cyber war to understand why today we are in such a state of computer (in)security.
We are in a world today in which everything is connected to the internet, in which the physical and virtual have increasingly merged. We have created a rich digital seam that can, for good or for ill, be mined, not just by intelligence agencies but by many others as well. The history told in this series and the issues it raises are not just for technologists or cyber security mavens or intelligence agencies – it is for everyone.
And, yes, some of it is very difficult to understand and even more difficult to write about – to put it in an accessible manner. Yes, it would be fun to simply do some puff pieces, to just do some “digital helicoptering” (a phrase created by good friend Peter Stannack) – hovering above it all, tweeting and posting. Mostly with no consequences.
But that turns us all into mere observers, not full flâneurs. Too many of us have become mere commerce monkeys, commerce machines – writing the mundane, the pedantic, writing in terms of the path to market. Stuck in our occupation silos.
But not the people you will meet in this series. They have all stepped outside all of the usual lanes to cross disciplines, social silos, political tribes and cultural boundaries. It is my goal to take all the myriad spheres and show how literally everything overlaps.
I opened this piece with something I wrote five years ago and have repeated (ad nauseam) : no country has even come close to the U.S. in harnessing the power of computer networks to create and share knowledge, produce economic goods, intermesh private and government computing infrastructure including telecommunications and wireless networks, using all manner of technologies to carry data and multimedia communications, and control all manner of systems for its power energy distribution, transportation, manufacturing, etc. and all this has left the U.S. as the most vulnerable technology ecosystem to those who can steal, corrupt, harm, and destroy public and private users, at a pace often found unfathomable.
Hopefully, this series will explain why.
D. A summary of the material and chapters to come
A subject as complex as this has myriad issues and not all of them can be covered. Many of the people I have consulted in this series have suggested topics that, while important, are not mainstream to my primary intent. But they are persuasive folks so I just might include them. At present, the planned chapters will unfold in the following order:
A short history of the internet, computers and security
The early Internet, constructed decades ago to serve a small, tight-knit and primarily academic community, was built upon principles of game-changing speed and a deep understanding of the importance of connectivity. Security and privacy were not needed for that first small group of trusted users and thus were not part of the original design requirements. Although security and privacy have demonstrated their importance in today’s blisteringly fast, global network, they have not kept up as the Internet has matured. While we are exponentially more connected than at any other time in history, with nearly instantaneously accessible information at our fingertips, the cyberadversaries – not the defenders – are the ones who have mastered speed and connectivity to their advantage. Speed and connectivity serve us well as communication building blocks, but too often have failed us in cybersecurity, because we have failed to establish the foundation of cybersecurity upon those fundamental elements.
The history of how this happened and why is fascinating, especially the evolution of Cloud and the era of “Cloud-Only” technology. Going through several of the foundational texts I’ve listed in my sources/bibliography I’ve done a mash-up and weaved a short story. At every step there were those modals of lost opportunities: “Could have, would have, should have”.
A short history of cybersecurity
The landscape of cybersecurity has now reached a point where we can start thinking about how all of the pieces work together. But cybersecurity didn’t start out with such a broad set of “solutions” or even approaches. Jack Mabry at Crowdstrike told me when he started out 30 years the “first generation of security” was simple. It was focused on connectivity. Firewalls, the initial control points, were placed at the network edge and controlled who and what could connect to the network. A few years later, those firewalls were combined with VPNs to encrypt traffic, and tools like IDS and IPS soon followed to better monitor and secure traffic because threats had become more complex.
But then we quickly moved to a “second generation” of network security which began when threats started to target applications. Content became much richer and more powerful. It was not just text or passive documents being passed between connections, but content that could be programmed to become vectors of agent-based attacks. We had second-generation antivirus firewall that could identify, inspect, and secure application-layer transactions and content — something that first-generation firewalls were not able to do.
And it continued. Especially the complexity. Network, storage, and computing infrastructure became programmable and orchestrated, and we quickly moved to powerful, movable devices and work … right into an expanding attack vector. This chapter will attempt to put all of that in perspective.
The Russian Bear
Russia has a distinct advantage in the cyber realm because on a regular basis it engages the services of non-governmental cyber crime entities, which masks its role in cyber attacks. This is what the U.S. and others do not do – engage proxy cyber warriors. This is not to say we never use them. But as explained to me by Linda Nowak of Crowdstrike:
“What the Russians are saying is that we will make these criminal organizations our partners – recruiting them to do cyber work for the Russian state. The Kremlin promises its criminal partners it will turn a blind eye to their attacking banks, disrupting commerce in the West, stealing money, etc. so long as they make themselves available to do the odd job for Russia’s intelligence services and military”.
You need to start by understanding the Russian dynamic. At the European Electronic Warfare Symposium two years ago, one of the best presentations was by Dr. David Stupples, director of the Centre for Cyber Security Sciences at City University London. He made several points but here are the key ones from the presentation:
• Russia’s intelligence services decided years ago to make cyber warfare a national defense priority. They have become increasingly proficient in cyber operations as a result.
• From around 2007, Russia decided that information warfare was key to winning any world conflict, and that it was this area of capability and technology they decided would benefit from vastly increased military investment. What made this decision easier was that Russia was also home to the largest number of the world’s best hackers.
Information warfare has always been the dominant Russian interest in the cyber domain. Some perspective from Alexander Klimburg in his book The Darkening Web :
Soviet computer systems nearly played a decisive role in world history. Already in the 1940’s, Soviet engineers had begun to make significant strides in computer science, and by the early 1970s some part of Soviet computing were equal to or even better than their US counterparts. Russian programmers have for decades played vital roles in the development of computing in general. Topcoder.com, a community Web site, has consistently ranked Russia the first nation worldwide in terms of providing sophisticated coders (with China second, Ukraine fifth, and the US only sixth). Indeed, brilliant Russian coders have a reputation of having helped build Silicon Valley.
The Soviet in-depth consideration of “cyber” was markedly different from the western approach to network systems. On its surface, the Russian interest in kibernetika, or “cybernetics”, was heavily informed by the writings of American mathematician who first coined the term. Norbert Wiener’s work was largely ignored in the US for decades many having perceived cybernetics as a minor branch of general system theory. Not so in the Soviet Union, where Wiener was celebrated as a philosopher of renown, somewhere between Gramsci and Hegel. As an MIT colleague once put it, “Wiener is the only man I know who conquered Russia, and single-handed at that”. The influence that Wiener’s cybernetics revolution had on post-Stalinist Russia is simply astonishing.
This will be my in-depth “From Russia, With Love” chapter which will include my in-the-field-trips in Ukraine.
SolarWinds and her evil sisters: a deep dive into the attack tactics, techniques and procedures
The hack of SolarWinds was brazen, extensive and damaging. I’ve outlined the attack (and those of its “sisters”) in very broad strokes above. In this chapter we’ll get granular and do a deep dive into the attacks.
And we’ll look at failures. The specter of advanced persistent threats (APTs) has helped sell a lot of cybersecurity technology over the last decade. APTs are all over the marketing campaigns of every major cybersecurity vendor. And yet, apparently, the actors behind the SolarWinds and related hacks easily evaded them all. For endpoint detection and response (EDR), as Sentinel Labs noted in a long report, the threat actor seems to have tested its malware against all the major players. It knew which ones could detect it, which ones it could turn off, and which ones it could not evade.
While we might never know how many companies and government agencies were actively compromised by the campaign, the list keeps growing. It’s therefore safe to assume that there were many opportunities for detection and other categories of tools, including extended detection and response, automated threat hunting platforms, and internal network monitoring tools, were all evaded for over seven months.
And we’ll talk about the gorilla in the room. Not every software firm operates like SolarWinds. Most seek to make money, but few do so with such a combination of malevolence, greed, and idiocy. What makes SolarWinds different? The answer is the specific financial model that has invaded the software industry over the last fifteen years, the recklessness. It is the same story in retail, manufacturing, and so many others.
All aided by its lovely hand maiden, the government. The massive months-long hack of agencies across the U.S. government succeeded, in part, because no one was looking in the right place. The federal government conducts only cursory security inspections of the software it buys from private companies for a wide range of activities, from managing databases to operating internal chat applications. That created the blind spot that the Russian hackers exploited to breach the Treasury Department, the Department of Homeland Security, the National Institutes of Health and other agencies. After embedding code in all that widely used network management software all they had to do was wait for the agencies to download routine software updates from the trusted supplier.
Control systems
Control Systems cybersecurity expert, Joseph Weiss, is an international authority on cybersecurity, control systems and system security. His concern was that the SolarWinds hack can directly affect control systems. A few points from his brief:
• The Russian’s got a “two-fer” in the SolarWinds hack – compromise of the IT infrastructure and direct control of building control system devices. The Russians also got indirect control of “industrial” control systems via the IT network backdoors.
• The SolarWinds webinars and Zoom chats have all focused on the IT networks, network visibility, and data exfiltration/compromise. However, SolarWinds is used to manage all types of Simple Network Management Protocol (SNMP) devices and these include not just IT equipment like servers and switches but, also Industrial Control Systems (ICS) like power and cooling systems. Specifically, SNMP management systems are used to monitor (using the SNMP “get” command”) and control (using the SNMP “set” command) any SNMP device.
NOTE: for a good SNMP “explainer” watch this video.
• Most power and cooling systems in data centers, laboratories, telecom systems, and network closets include SNMP communication cards or chips for the expressed purpose of allowing them to be monitored and remotely controlled. Even the most recent SNMP version is now over 20 years old and has long been shown to be vulnerable. Devices that are using SNMP are insecure and can easily be compromised. The Russians used SNMP communication cards as attack vectors in their 2015 attack on the Ukrainian power grid that left hundreds of thousands without power.
• It has long been speculated that the Russians have been using the Ukraine as the “test laboratory” for more extensive cyberattacks they intend to use against other countries. Unfortunately, we can now say that they have succeeded in this SolarWinds Orion attack.
Weiss goes into more detail and he’ll be the star in this chapter.
E. So what in hell do we do now, cybersecurity mavens?
The SolarWinds hack reflects a level of sophistication that may be impossible to completely block, but cybersecurity technical professionals and policymakers say new “from the ground up” approaches to software development and procurement need to be built to give defenders a fighting chance.
This will be the ending chapter of the series and has been the most difficult to write. But I’ll start by noting Cliff Kittle who has structured a plan, incorporating a positive tone, a holistic approach. As he says “a mature cybersecurity culture is greater than the sum of its parts. Collaboration and teamwork motivated by the investment in preparation for improving the knowledge, skill and awareness of every employee will demonstrate leadership’s perspective of the importance of strengthening the corporation’s security.”
Just a few points while I have your attention.
We still don’t yet understand the scope of these operation or the extent of the damage wrought by the perpetrators. But I rather like the opinion of Alex Stamos who says it will take years to find every piece of spyware embedded into American IT systems. He compared the clean-up operation to the “iron harvest”, the springtime farm work in France and Belgium that even now turns up unexploded bombs and shrapnel from the first and second world wars.
But in the words of General Stanley McChrystal “it takes a network to defeat a network.” He has noted in a series of posts on Cipher Brief and other venues that these recent attacks are highly-sophisticated, highly-disciplined acts of espionage. They have brought home the need for a “totally fundamental redesign of the U.S. cyber defensive posture”.
No, not easily achieved. Cyberspace offers the U.S. adversaries the ultimate asymmetric capability, providing over-the-horizon reach without having to set boots on the ground, the ability to move quickly throughout networks, and all-important cover and concealment to conduct their operations. These adversaries – known as Advanced Persistent Threats or APTs as I noted above – are teams of intelligence gatherers and operators. As Lauren Zabierek and Paul Kolbe of the Harvard Kennedy School’s Belfer Center have noted:
“They are not one-off attacks – they are continuous assaults by networked cells armed with knowledge of their targets, sophisticated tools and techniques, and time–carrying out the interests of their nation. They target the U.S.’s critical infrastructure, its schools and companies, steal our intellectual property, and conduct information operations perpetrated against our electorate–such threats compromise America’s safety and security on a daily basis.
Surely there were clues – between the classified data points and the unclassified observation of activities on domestic servers and networks – but classification restrictions and inadequate infrastructure for data aggregation and sharing likely prevented piecing those clues together before it was too late. Unfortunately, our current defensive cyber analysis and operations across the domestic landscape are stove-piped and uncoordinated, leaving us over-extended and vulnerable. The underpinnings of our modern economy – networks, servers, satellites, the Cloud – are all largely built, managed, and protected by private industry.
Moving toward any kind of whole-of-nation paradigm requires reimagining the concept of national security.”
When one considers what “cyberspace” really is … electrons and photons that modify atoms and molecules … you get this feeling of a “virtual world”, in a sense not being real. While perhaps we like to consider software as language, fundamentally it is moving energy around. The software that controls the dams is on the internet (for efficiency, maybe indirectly) and security is dependent on keeping bad actors out of specific logic addresses and memory spaces in directly interconnected electrical systems.
I sort of agree with Bruce Schneier who said it seems unlikely that there will be true cybersecurity on connected networks, as we’ve seen thirty years of that failing: It’s failing more and faster. And the next wave of technologies are built for insecure systems. You can’t hack something where everything is public.
One of my Linkedin sources said air-gapped networks seem unlikely for that doubles (at least) the costs while posing significant maintenance issues. Can we imagine the NSA laying all new fibre? And then what? It’s cut into and tapped the same way as the old fibre?
I read the Pew Research poll that said more and more cuber security mavens think we’ll move toward some kind of verified internet. Largely, potentially completely, unrecognizable from its current cousin. Kimmie Preston, a colleague of mine who works at FireEye and a fellow film nut, reminded me of the Herzog film Lo and Behold, where one of the UCLA researchers discusses how cybersecurity was never designed into TCP/IP itself because “security” was physical connections between computers. One was either online or not. And if one was online, there was little need for access management. It is a marvellous documentary about how the internet has completely changed civilization. Watch it.
And how do we classify backdoors left in place? Evidently some compromised systems belonged to utilities. Did they have any operational control over any power, water, gas, etc.? To what extent do our current practices and infrastructure constitute a single point of failure? That is they are based on a limited set of low level protocols and a widely dispersed supply chain. If damage to utilities is considered is our move to more and more integrated communication (think 5G) the right direction? For several decades, the holy grail of improved computing has been faster processing. Recently there has been development of systems with substantial hardware support for security. Is that a direction that needs more attention?
Many IT people I spoke with say this current attack is the right place to start an investigation. It is causing a re-look at the context of past penetration of systems by various state and non-state actors. But as told to me by Jak Humpries, a chap who runs a penetration testing company, and deals with this stuff every day:
There is still the protocol that runs the world today. Which is a real triumph of design on one hand. On the other hand, it’s a protocol that is inherently always open and therefore always insecure. Any security layers are attached to the open protocol. You’re always trying to bail a leaky boat with that one, because there’s no direct connection between sender and receiver in the virtual world and a sender and receiver in the real world. The protocol simply doesn’t support it. That is in part what gave us the explosive growth and experimentation of the last three decades. How lightweight and unencumbered TCP/IP was. Now you want to secure it? Doubtful. The U.S. really wants the world to be two oceans and radar, but it’s not. Therefore the entire paradigm of the network has to change.
Everything is ambiguous is cyberspace. Alexander Klimburg, in The Darkening Web, says we can’t even agree on how to spell the very word itself, so it shouldn’t be a surprise that we have difficulties in defining it:
Spelling, as always, is a reflection of one’s preferences: those who spell “cyber space” as two words are implying that the domain is not an entirely separate or unique entity, just as writing “cyber security” as two words implies that it is just another form of security like “maritime security”, and not special at all. Those, like myself, who believe that cyberspace has unique identifiers that make it like a physical domain (like airspace or the seas) spell it as one word, just as the contentious term “cybersecurity” is given in a depth treatment on account of its overall rejection among some parts of the civilian technical community.
Over the last few months as my cyber study intensified, two of my tenents have certainly been reinforced:
• the activities of nation states and quasi/faux nation states are making cyberspace a domain of conflict, and therefore increasingly threatening the overall stability and security not only of the internet but also of our very societies.
• Each aspect of cybersecurity (say, cyber crime, intelligence, military issues, Internet governance, or national crisis management) operates in its own silo, belonging, for instance, to a specific government department or ministry. Each of these silos has its own technical realities, policy solutions, and even basic philosophies. Even if you become proficient in one area, it is likely that you will not have the time to acquire more than a rudimentary knowledge of the others. Your part of the “elephant” will dominate, and inevitably distort, how you see this beast.
I have also come to realize that the media mantra of power grids crashing and armies being immobilized by cyberattacks are puny compared to the comprehensive psychological operations at work. An article in Time magazine (of all places) from 1995 … written in the Stone Age of the Internet, and now lost to history except in a hard copy format, which I found in a bookshop in (wait for it) Saint-Malo, France … made two points that still hold true today. The writer had an instinctive grasp on the two different shades of the debate:
• On the one hand, the kinetic-effect “cyberwar” discourse, which involved hacking and bringing down critical infrastructure and the like
• On the other, the psychological effect “information war” discourse, which the writer (even back then) suggested could covertly influence marketing and political campaigns at a new and much larger scale, using propaganda and other means to subtly influence entire populations.
The writer grasped many of the existential issues this development presented, unconsciously paraphrasing the philosopher Marshall McLuhan when he said that “infowar may only refine the way modern warfare has shifted toward civilian targets.” And, most presciently, the author asserted that “an infowar arms race could be one the U.S. would lose because it is already so vulnerable to such attacks. Indeed, the cyber enhancements that the military is banking on… may be chinks in America’s armor.”
Indeed, as I have written many times before, the worst possible cyber event may not be that the lights go out … but that they will never go out. That it pays for our adversaries to force us to slip into what Klimburg calls “an environment of Orwellian proportions”.
More to come.
