The Missing Element from Most Internal Developer Platforms: Test Data Management

Internal developer platforms (IDPs) are tools organizations use to build, deploy, and manage software applications in a more efficient and streamlined way. IDPs provide developers with pre-built components, frameworks, and best practices that can be used to develop and deploy applications in a consistent manner. 

The components of IDPs can vary widely depending on the specific platform and the organization's requirements. However, modern IDPs tend to share some common components:

1. Infrastructure as Code (IaC): An IDP often provides infrastructure resources such as servers, databases, and storage through the concept of Infrastructure as Code (IaC). It allows developers to define and provision the required infrastructure resources programmatically, using tools like Terraform, Puppet, Chef, Ansible, and cloud service providers' APIs (such as AWS CloudFormation and Microsoft Azure Resource Manager).

2. Development Tools: IDPs offer a range of development tools that facilitate coding, testing, and debugging. These tools include integrated development environments (IDEs), code editors, code linters, debuggers, and package managers. They provide a streamlined development experience and support developers in writing high-quality code.

3. Version Control Systems (VCS): IDPs integrate with version control systems (VCSs) like Git to manage source code repositories. VCSs enables developers to track changes, collaborate with teammates, and maintain code version history. It also helps in managing different branches, merging code changes, and facilitating continuous integration.

4. Continuous Integration/Continuous Deployment (CI/CD) Pipelines: IDPs often include CI/CD pipelines that automate the process of building, testing, and deploying software applications. These pipelines define a series of steps and workflows, ensuring that changes in the codebase are automatically validated, tested, and deployed to production or staging environments.

5. Pre-Built Libraries and Frameworks: IDPs may provide a collection of pre-built libraries, frameworks, and modules that can be easily integrated into applications. These components help developers accelerate development by providing ready-to-use functionalities, standardizing code practices, and ensuring consistent application architectures.

6. Containerization and Orchestration: IDPs commonly support containerization technologies like Docker, which package applications and their dependencies into isolated containers. They may also include orchestration tools like Kubernetes, which enable efficient deployment, scaling, and management of containerized applications across a distributed infrastructure.

7. Monitoring and Logging: IDPs often include monitoring and logging capabilities to track the performance, availability, and health of deployed applications. These components help in identifying issues, analyzing system behavior, and providing insights for performance optimization and debugging.

8. Documentation and Knowledge Sharing: IDPs promote knowledge sharing and collaboration through documentation repositories, wikis, and internal forums. These platforms facilitate the dissemination of best practices, guidelines, architectural patterns, and lessons learned, enabling developers to leverage shared knowledge.

9. Security and Access Controls: IDPs prioritize security by offering authentication, authorization, and access control mechanisms. These ensure that only authorized individuals can access and modify sensitive resources. Security components also include vulnerability scanning, secure coding guidelines, and automated security checks.

10. Integration and Extensibility: IDPs can integrate with various third-party tools, APIs, and services to extend their functionality. IDP integration allows developers to connect with external systems, services, and data sources, enriching their applications with additional capabilities.

The actual components and their configurations can vary based on the organization's specific needs, technology stack, and development practices. But most IDPs tend to lack a crucial element needed to efficiently provide test data for developers and testers: DevOps test data management (TDM).

The Missing Link of DevOps Test Data Management

A common goal of an IDP is to automate the steps for building, testing, and deploying software. IaC, CI/CD pipelines, orchestration and other IDP components provide much of this automation. But in our experience, delivering and managing test data often remains a highly manual and time-consuming process, slowing down developers as they wait on data to be subsetted, masked, and copied over into the development environment. A recent poll of 55 DevOps professionals on LinkedIn demonstrated this point, with only 44% stating that they have “mostly automated and optimized” the flow of test data into their development environments.

IaC automates the provisioning of databases for development environments. So you might ask, why should you consider adding another platform component? The reason is that IaC generally creates physical copies of production data for test purposes: this approach is storage-intensive, time-consuming, and puts sensitive data at risk. IaC tools are also incapable of handling complex use cases such as versioning, branching, bookmarking, rewinding, and refreshing test data.

DevOps TDM, when done right, solves these problems by providing the following capabilities:

1. Data Provisioning and Management: DevOps TDM simplifies and accelerates data provisioning for developers. It provides self-service capabilities, allowing developers to quickly and securely access the data they need for development, testing, and debugging purposes. It also virtualizes data, enabling developers to provision lightweight, space-efficient virtual copies of databases or data sets on-demand, without the need for lengthy data provisioning processes or consuming excessive storage resources.

2. Data Masking and Data Privacy: DevOps TDM offers data masking capabilities that allow sensitive data to be anonymized or masked to protect privacy and comply with data protection regulations. This ensures that developers can work with realistic data while safeguarding sensitive information. It helps developers focus on building and testing applications without worrying about exposing sensitive data or violating privacy regulations.

3. Data Versioning and Branching: DevOps TDM enables data versioning and branching, similar to how version control systems work for code. Developers can create virtual copies of data at specific points in time, facilitating parallel development, experimentation, and the ability to quickly switch between different data states. This capability promotes agility and enables developers to work independently on different features or fixes without conflicts.

4. Test Data Refresh: DevOps TDM streamlines test data management by allowing developers to refresh their test environments with up-to-date and relevant data. With the ability to create and refresh data copies in minutes, developers can quickly reset their test environments to a known state, ensuring consistent and reliable testing. This saves time, improves testing efficiency, and reduces dependencies on data operations teams.

5. Data Collaboration and Sharing: DevOps TDM provides collaboration features that enable developers to share data sets with team members, promoting knowledge sharing, collaboration, and consistent development environments. Developers can easily share data subsets or virtual copies with peers, facilitating collaborative development efforts and ensuring everyone has access to the necessary data for their tasks.

6. DevOps Integration: DevOps TDM tools integrate with DevOps tools and processes, supporting continuous integration and continuous deployment (CI/CD) pipelines. DevOps TDM integrates with popular CI/CD tools and platforms, allowing developers to include data operations as part of their automated development workflows. This seamless integration streamlines development processes and reduces friction between development and data operations teams.

DevOps TDM For Dummies® Delphix Special Edition

Find out why DevOps test data management (TDM) is magic in this eBook. You'll explore how DevOps TDM enables speed, quality, and compliance — without manual provisioning or copying of any resource. And you'll get real-world use cases of test data management in financial services, retail, and human resources.

Get DevOps TDM For Dummies >>

Additional Resources

The following videos provide a better perspective of the benefits of incorporating a DevOps TDM solution into a platform engineering approach. They offer varying levels of detail and length: :

1. 3-minute video featuring Arvind Anandam, Director of Platform Engineering for Worldpay describing the benefits that Worldpay’s engineering teams saw from using Delphix’s DevOps TDM solution

2. 20-minute video featuring Sanjeev Sharma, SVP of Development and Engineering Platforms for Dell Technologies, who describes Dell Technologies’ entire developer platform and the role that TDM plays

3. 55-minute webinar recording featuring Cody Taylor, Senior Director of Product Management, Technology Transformation Services for Dell Technologies, who explains how DevOps TDM helped Dell Technologies dramatically increase developer productivity and deliver over 21 million pipeline runs in one year.

By integrating DevOps TDM into IDPs, organizations can overcome the manual and time-consuming processes associated with test data management. It complements existing IDP investments and fills the gaps in data provisioning, data privacy, data versioning, test data refresh, collaboration, and integration. By adopting DevOps TDM as part of their IDP, organizations can further optimize software development, improve collaboration, and ultimately deliver high-quality applications more efficiently.