“You come to the office one day, open your laptop, log in, and then you see the dreaded black screen with red letters.” This is how Peter Hudak, security testing lead at ABN AMRO Clearing, describes one of the worst nightmares of a technology company: a blackout in production caused by ransom demands.
Production blackouts can devastate businesses in minutes, whether caused by malicious actors, insider threats, or simple human error. Instead of if they happen, the question is why and how to prepare.
Peter Hudak, from Levi9’s partner ABN AMRO Clearing, together with our own Codrin Băleanu and Iulian Constantinescu share their insights into the reality of production environments going dark: from common causes to effective preparation strategies to minimize damage when blackouts inevitably occur.
Production Blackouts are Inevitable
“You will be hacked. No matter what system you build, you will be hacked. Given enough time and effort, someone will find a way to penetrate,” warns Codrin Băleanu, Engineering Lead at Levi9.
Production environments face constant threats, both external and internal, intentional or accidental. As Peter Hudak explains, “Just expose yourself to a public IP and see what’s happening.” The moment any system touches the internet, it becomes a target for various threat actors with different motivations and capabilities.
The Cybercriminal Mind
Examining what drives the various attack types and the typical methods of operation can help organizations better understand and predict security threats.
Professional Ransomwarers
One of the most visible threats causing production blackouts comes from cybercriminals deploying ransomware. These cybercriminal groups have become increasingly professionalized, with organized structures that include people responsible for negotiation, technical operations, and even financial analysis.
“They’re quite well organized,” describes Peter. “They have people who are responsible for negotiation; people who are responsible for analyzing the financials of the target company; and an infrastructure team.” They have HR departments, holidays, and employment benefits.”
What makes these actors particularly dangerous is their opportunistic nature. “Their goal is just to get the money,” says Peter. “Once they get in, they don’t wait. They just move along. They don’t try to be stealthy.”
Advanced Persistent Threats (APTs)
Unlike cybercriminals who seek immediate financial gain, state-sponsored actors known as Advanced Persistent Threats (APTs) have different goals and timelines. They are interested in gaining a geopolitical advantage through intelligence gathering, disruption and sabotage. This means that once they infiltrate a system, they remain hidden, waiting for the right moment to strike.
“Ransomware gangs want to act faster, but APTs have time. Once they get in, they just need to get down and stay hidden. And they wait. And they wait for the opportunity to strike in,” Peter elaborates.
Private Tools for Sale
There is a market for cyber weapons, and there are companies that meet the demand. These companies create software that can compromise devices without any user interaction, called “zero-click, zero-day”. “This market is leveraging on some quasi-legal frameworks and structures, and they hire extremely skilled cyber security specialists,” elaborates Peter. While some are sold to governments for legitimate purposes like tracking criminals, these powerful tools can cause disastrous production blackouts when misused or when they fall into unauthorized hands.
Hacktivists
While all previous threat actors are motivated by money or influence, hacktivists focus on ideological and political causes. Levi9 experienced such an incident when a customer in the United States went public with an anti-LGBT message. Protesters first started to boycott the stores in the US, but soon enough they targeted the website, which was maintained by Levi9. “We had a lot of DDoSing coming our way”, remembers Codrin Băleanu.
The Insider Threat
One of the most efficient sources of production blackouts comes from within organizations themselves. As Iulian Constantinescu, a DevOps architect with 16+ years of experience, points out: “It’s the easiest; you’re already in the house.”
Insiders sometimes act for revenge, as was the case in a crypto-mining disaster that Iulian recalls clearly. “Sometimes, when colleagues leave the company, they leave behind a little present.” He describes a situation where a former colleague had implemented cryptocurrency mining software on production servers, configured to run only when systems were idle. Thanks to resource monitoring, Iulian’s team was able to identify the issue and remove the root cause.
Preparing for the Inevitable Blackout
Since production blackouts are inevitable, organizations should focus on preparation and resilience rather than simply prevention. The Levi9 team suggests several key steps:
1. Implement and Test Backups
When faced with a ransomware situation, the most important factor is whether you have functioning backups. Codrin Băleanu emphasizes that, on top of simply having backups, it’s important to also test them properly. Otherwise, they may give a false sense of security.
2. Have a Ransomware Playbook
Peter Hudak recommends developing specific protocols for responding to blackout scenarios: “You need to have a ransomware playbook plan. This includes proper backups, damage assessment, and what to do if this happens.” Peter, who leads penetration testing and red-team exercises at ABN AMRO Clearing, also advises running tabletop exercises to practice the scenario and make sure the organization has the necessary resilience.
3. Implement Proper Monitoring
Iulian Constantinescu highlights the importance of monitoring: “The example from our colleague mining cryptocurrency was detected because the usage of the systems, especially the production ones, got to an average somewhere around 60 plus.” Proper monitoring enables organizations to detect anomalies before they cause complete blackouts.
For e-commerce environments, Codrin Băleanu relies on a simple metric “we observe euros per minute compared to the previous day.” Some spikes might be normal, but a 10% drop would trigger alarms and the incident will get escalated really fast.
4. Planned Disaster Recovery Exercises
Rather than waiting for blackouts to happen, Codrin suggests proactively testing resilience: “You don’t have to wait for it to happen because it will happen eventually. Especially if you’re in a sensitive area like e-commerce, you can plan for this thing.”
Double Prevention with Resilience
A good cyberdefense strategy includes prevention, and websites like the Mitre Attack Framework can go a long way toward identifying the most likely threat actors and potential ways inside your company. However, the question is not if your systems will experience a blackout, but when.
The key is shifting from a prevention-only mindset to one that embraces preparation and resilience through proper backups, playbooks, monitoring, and regular practice.
“It is a continuous effort”, concludes Peter Hudak. Organizations that acknowledge this reality and prepare accordingly will recover faster and with less damage when darkness inevitably falls on their production environments.
