3 Ways to Mitigate the Growing Risk of Non-Production Environment Breaches

Costly data breaches are nothing new in the enterprise world, but the problem has worsened in recent years. The Identity Theft Resource Center (ITRC) reported 1,862 data breaches in 2021, which remains the all-time record for most breaches counted in a year. Last year, the ITRC counted just 60 less breaches throughout the year— and while the war in Ukraine and cryptocurrency market volatility distracted hackers in the first half of 2022, the number of breaches steadily rose through the second half of the year. 

Even more alarming for enterprises, though, is the emergence of a cybercrime trend in which hackers are targeting enterprises’ non-production environments. LastPass is the latest business to endure this type of breach, having endured two linked breaches of development environments in 2022. The severity of the company’s breach serves as a warning to businesses to secure their non-production environments so that they can avoid similar breaches.

The LastPass Hacks

Both LastPass data breaches transpired in the latter half of 2022, and the hacker leveraged information from the first hack to conduct the second. The first breach transpired in August, when an undisclosed hacker breached a development environment and stole source code and technical information, LastPass reported in a blog post. The hacker was also able to steal unencrypted and encrypted vault data including personally identifiable information (PII) such as names, addresses, and telephone numbers. Over the next two months, the hacker conducted reconnaissance of a cloud storage environment separate from LastPass’s production environment, according to a new statement from LastPass. 

The hacker finally struck LastPass again in November. In this attack, the hacker used information obtained from the first attack as well as a third-party breach to target the home computer of a LastPass senior DevOps engineer who had the high-level security authentication to use decryption keys needed to access the cloud storage service, per LastPass and ZDNet. This engineer was just one of four DevOps engineers who had this level of authentication. The hack allowed the attackers to gain access to the DevOps engineer’s corporate vault, which contained, “encrypted secure notes with access and decryption keys needed to access the AWS S3 LastPass production backups, other cloud-based storage resources, and some related critical database backups,” per LastPass.

The Bigger Picture

The LastPass hack is notable in its level of sophistication, the relation of both attacks to each other, as well as the degree to which the attacker exploited LastPass’s non-production environments. But the LastPass breach is just the latest in an ongoing trend in which hackers are targeting enterprises’ non-production environments, rather than end user-facing production environments. 

One of the first prominent breaches of a non-production environment transpired in 2016, when bad actors exploited Uber’s software development environments to break into the rideshare giant’s cloud storage, where they stole a significant amount of consumer data. In 2021, a hacker leveraged an unprotected router to gain access to T-Mobile’s production, staging, and development servers, which compromised over 48 million social security numbers and other details. 

This trend is worrying for three reasons. First, the security of non-production environments is often an afterthought for most enterprise security teams, who tend to focus more heavily on protecting customer-facing production environments. Second, in our experience, non-production environments constitute up to 80% of an enterprise’s attack surface area in terms of endpoints, privileged user accounts, and data (more on this below). So, not only are non-production environments vulnerable— they’re a large target, too. 

And lastly, IT organizations often create multiple, redundant copies of test data, allowing numerous copies of test data to float around these vulnerable non-production environments. With about 8-10 copies of test data floating around non-production environments for every production environment, these non-production environments have become treasure troves for hackers seeking to steal customer data.

Securing Data in Non-Production Environments 

LastPass has since taken a number of actions as part of its incident response and recovery activities, including upgrading its multi-factor authentication, decommissioning its development environment and constructing a new environment, and adding “additional logging and alerting capabilities to help detect any further unauthorized activity,” per its blog post.

But taking proactive measures to secure data is always better than taking reactive measures following a data compromise. In our experience in working with hundreds of large and regulated enterprises on securing their data in non-production environments, enterprises mitigate the risks by taking the following actions:

1) Maintain an immutable baseline and mitigate vulnerability due to data changes 

Hackers are often just as expert at penetration as they are at covering their tracks.  And, since social engineering, not brute force, is the main way that attackers compromise and gain access to systems, it’s crucially important that hackers be prevented from covering their tracks.

Having an immutable copy of a dataset and associated configuration means that surgical redaction and subtraction can be rapidly identified and corrected, and that hackers are much less able to alter files (especially configuration and log files) to cover their tracks.  Moreover, data immutability provides extra protection against an attempt to destroy data permanently, as the data is out of reach for the hacker.  Non-production environments are often a more favored attack vector since it is usually less protected than the production fortress.

2) Maintain code and data velocity, especially in response to a vulnerability

When responding to a breach, velocity matters. Many breaches are the result of software defects or incorrectly configured software, for which the fix depends on developers and testers (including security testers). Therefore, the speed (velocity) by which your development and quality teams can respond to and correct software is critical. 

But development slows as data must be conditioned to be useful - especially when testing new features. So, when we are made aware of a data vulnerability, the time it takes to retract data from service and the time it takes to get new data out the door are crucial in terms of exposure time and the time to restore service.  

Data figures prominently into code velocity as well.  In response to a zero day attack or a critical code vulnerability, being able to rapidly deliver the right test dataset to test a security patch directly affects lead time for change and the time to restore service.

3) Mask sensitive data used in non-production environments 

The most powerful way to protect data is to make the data useful to the developer and tester but useless to the thief and hacker.  The key phrase here is useful— data protection techniques such as nulling and redaction can’t accomplish this, because developers can’t test with null data, and they need access to all data values in order to run diagnostics as needed.

Development teams need a way to transform the data so that it has valid values that nevertheless don’t correspond to any real situation. Data masking, when done right, provides a no-compromises approach by irreversibly replacing the original, real data values with fictitious but realistic equivalents. Some solutions — like static data masking — can also maintain referential integrity of masked data across datasets that are distributed across different platforms, including both on-prem and public cloud platforms. This carries the added benefit of bringing test data into compliance with virtually all data privacy regulations by simply replacing regulated data with realistic substitutes. 

Because it replaces existing data with fictitious yet realistic data, PII data masking eliminates the risk of personal data exposure in the event of a breach— while also preserving business value. When done right, masking alleviates the privacy fears of users and the business value fears of developers and IT professionals. 

As part of the comprehensive Delphix DevOps Data Platform that automates data delivery across an enterprise, Delphix Continuous Compliance automates the process of detecting and profiling sensitive data in production sources, masking the data wherever it resides and delivering the masked data to downstream environments.

The 2024 State of Data Compliance and Security Report

54% of organizations have experienced data breaches or theft in non-production environments. Find out why — and what you can do about it in The 2024 State of Data Compliance and Security Report. Discover insights from 250 global leaders around sensitive data, compliance, masking, AI, and more.

Get the Data Compliance Report >>