Nearly every day in our digital lives, we see a multitude of apps on our phones or laptops get updated – innocuous maintenance updates that invariably include words to the effect of “This release addresses maintenance updates, stability improvements, and bug fixes.” It was during one of these routine releases late last year that triggered one of the more immensely damaging and wide ranging hacks the US has ever seen, commonly known as the Solar Winds hack. It is named after the company of the same name, and was triggered by an update of their widely used Orion software that helps companies and major technology operations in the US Government monitor network activity. Like a tiger laying in wait, the seemingly boring update triggered what many believe is one of the largest and most damaging hacks the US has ever seen. For many who analyze this sort of thing, the true impact of this will not be able to be calculated for months if not years.
In the months since it happened, NPR has been doing its research to learn more about how exactly it went down and where the checks and balances failed. As was reported when it happened, the brilliance and sophistication of the hack can not be understated.
Network monitoring software is a key part of the backroom operations we never see. Programs like Orion allow information technology departments to look on one screen and check their whole network: servers or firewalls, or that printer on the fifth floor that keeps going offline. By its very nature, it touches everything — which is why hacking it was genius…
The SolarWinds attackers ran a master class in novel hacking techniques. They modified sealed software code, created a system that used domain names to select targets and mimicked the Orion software communication protocols so they could hide in plain sight. And then, they did what any good operative would do: They cleaned the crime scene so thoroughly investigators can’t prove definitively who was behind it. The White House has said unequivocally that Russian intelligence was behind the hack. Russia, for its part, has denied any involvement.
“The tradecraft was phenomenal,” said Adam Meyers, who led the cyber forensics team that pawed through that tainted update on behalf of SolarWinds, providing details for the first time about what they found. The code was elegant and innovative, he said, and then added, “This was the craziest f***ing thing I’d ever seen.”Dina Temple-Raston, NPR
The first foundation blocks of the hack started in late 2019 when the hackers inserted a seemingly simple line of code into the software that would indicate to them if the server used a 32-bit or 64-bit processor. Once the hackers were able to see a response to that simple query, they knew they could wreak some havoc. And five months later, they set down further foundational blocks by inserting code that would inform them whenever there was an impending software update.
Under normal circumstances, developers take the code out of the repository, make changes and then check it back in. Once they finish tinkering, they initiate something called the build process, which essentially translates the code a human can read to the code a computer does. At that point, the code is clean and tested. What the hackers did after that was the trick.
They would create a temporary update file with the malicious code inside while the SolarWinds code was compiling. The hackers’ malicious code told the machine to swap in their temporary file instead of the SolarWinds version. “I think a lot of people probably assume that it is the source code that’s been modified,” Meyers said, but instead the hackers used a kind of bait-and-switch.
But this, Meyers said, was interesting, too. The hackers understood that companies such as SolarWinds typically audit code before they start building an update, just to make sure everything is as it should be. So they made sure that the switch to the temporary file happened at the last possible second, when the updates went from source code (readable by people) to executable code (which the computer reads) to the software that goes out to customers.Dina Temple-Raston, NPR
Similar to the situation that happened in the days and weeks leading up to the 9|11 attacks, there were small signals and flags that were picked up by random people in random places, however none of those pieces were put together to demonstrate that something nefarious was going on. It was only after a network administrator at FireEye discovered that there was a listing for two phones for a single employee that they realized that there was a hacker within their network.
Another reason why this hack was such a ‘work of art’ was that all the normal trackers look for ‘normal techniques’ which usually account for 90 to 95% of all attacks. This one was so unique and so stealth that it completely bypassed all normal checks.
Like other catastrophic failures of all shapes and sizes, this hack had its warning signs. There was just no one there looking at the big picture that could have put these pieces together to see what may be happening just beneath the surface. The NPR article is a really great read, and probably worth a second read, just to really grasp the level of sophistication some of these nefarious hacking organizations have.