2024 State of DevOps Report Highlights Need for Security and Compliance in Platform Engineering

Two weeks ago, Perforce completed its previously announced acquisition of Delphix. This means that Delphix is now part of a broad portfolio of DevOps-focused solutions that includes Puppet, a pioneer of infrastructure as code and DevOps. In mid-March, Puppet revealed the results of its annual State of DevOps research, based upon responses from 474 participants around the world. This marks Puppet’s 12th such report, representing one of the longest-running research programs on DevOps practices.

Similar to that of last year, this year’s report looked at what successful platform engineering teams do to be successful. While definitions of platform engineering vary, the discipline centers on providing a comprehensive set of tools, practices, and software frameworks designed to streamline and automate the software development lifecycle within an organization. The intended result is an internal developer platform (IDP) that enhances developer productivity, ensures consistency in development processes, and facilitates collaboration among teams. Platform engineering and IDPs often help DevOps teams mature their capabilities, sometimes moving beyond a stage of stagnation with DevOps.

Among its findings, this year’s report highlighted the importance of security and compliance to platform engineering. “Improve security and compliance” ranked second only to “improve speed of product delivery” as a goal of platform engineering, while a large proportion of the respondents reported security and compliance benefits such as improving their security posture and achieving security and compliance KPIs (see charts below). It’s clear that security and compliance have been major components of platform engineering programs and that they’re having a positive impact on reducing risk.

Platform engineering and IDPs have the potential to improve security while enhancing developer experience. When done right, IDPs follow a user-centered security design that makes complying with security policies and practices easy for developers. Instead of having to jump through hoops to build secure applications or adhere to security policies, developers do so seamlessly with IDPs that prioritize both developer experience and security. It’s a rare win-win between better security and greater ease of use.

One of my colleagues showed that DevOps test data management (TDM) is often a missing component of most IDPs. We often see development teams struggling with slow and tedious processes for acquiring test data or dealing with poor-quality test data. The reasons vary, but the main culprit is often security and compliance. Many test data sets begin with sensitive details such as PII or PHI that must be desensitized; this is often accomplished via scripts or other processes that are slow and performed infrequently. It’s not uncommon to see test data refreshed and masked only once per year. For development teams that depend on fresh, production-quality test data, this is unacceptable. Where IDPs have been built without considering test data management, security and compliance often impair ease of use, along with speed and productivity.

Many platform engineers incorporate TDM into their IDPs and do so in a way that makes security and compliance seamless. Sanjeev Sharma, SVP of Engineering and Development Platforms at Dell Technologies, describes his platform model as a “layer cake” that consists of standard and automated infrastructure environments, an application delivery pipeline that includes all the tools developers need (along with observability), and a data pipeline – along with security and compliance.

As you can see from his model, TDM is a core component of the data pipeline layer, and data privacy (masking) is the “bridge” between the pipeline and security and compliance. At Dell, these are inherent features of their platform. Mr. Sharma connects this back to developer experience:

“You need to give your developers permission to act. If you made a platform self-service, but the developers still need to open a ticket to get approval from three people before they can do something, what did you achieve? So the permission to act is very important, but the permission to act needs to come with guardrails so that you can ensure that the developers don't do anything which puts them or your company or your system at risk. By the way, the data masking solution from Delphix is tremendous for that because it's a guardrail. Developers can't expose data if the data is masked.”

In other words, self-service with the proper guardrails gives developers a more seamless experience. In this example, he illustrates this principle by using data masking as part of the data pipeline, but there are many other ways that security can be applied without impeding developers.

This is not just important for today’s development platforms, either. Mr. Sharma describes the importance of this data pipeline for product teams that work with AI/ML:

“[TDM’s] a part of your integral part of your DevOps pipeline. But today, in an AI- and ML-centric world, there is also the data in that data pipeline which our data engineers need, which our data scientists need, which our AI folks need. And they have a very different set of curated use cases from your traditional application developers. And any data pipeline you build today, hopefully with Delphix, needs to have that under consideration so you manage that data properly and provide the right tools for your machine learning and AI data scientists.”

As the 2024 State of DevOps report reminds us, platform engineering should be and often is a conduit to better security and compliance. But if test data management and data pipelines are excluded, security and compliance as well as developer experience will suffer.