You Have Delphix. Now What?

You have purchased Delphix and you are ready to take advantage of the agility and flexibility that the sales team promised. Or you are evaluating the product for use within your DevOps or Data Migration project and just don't quite see where the technology fits into the data flow. I had been at that very juncture several years ago when my company looked at data virtualization to solve lengthy data provisioning processes and decided to purchase Delphix. As the lead system architect, I was the person tasked with evaluating, planning, implementing, integrating and operationalizing this new data virtualization software into established data provisioning processes and Infrastructure Technology and Development Operations. In order to succeed, I had to not only understand the Delphix technology and all its great benefits, but I also needed to know where in my company's data infrastructure transaction flow and architecture this virtualization tool would integrate that would gain the most DevOps agility the tool had to offer.

I suspect many new Delphix Clients and Global System Integrators of Delphix find themselves in the same position as I describe. I have since matured in my understanding of Delphix and its many capabilities within infrastructure provisioning, data integration points. However, implementing the technology with a new organization is still a daunting task. I still remember the early days of introducing data virtualization and feeling overwhelmed at the task of bringing my company into the next generation of copy data management. To help the new comers to this technology and to help guide system integrators understand their role in serving virtual data integration plans, I decide to write this blog to highlight Delphix implementations from project planning, to transitioning, to next practices, to operations and to continued sustainability and growth.

In this initial series I will walk though a DevOps Delphix implementation project from start to finish. I will focus on a fictitious organization that has recently purchased 4 Delphix Data Virtualization Engines with Agile Masking and Jetstream Self Service. The organization purchased the initial Delphix engines to support the operations line of business that includes a high availability engine for data protection. After that series I will jump into a project that includes adding Delphix to a hybrid cloud solution and then any other focus areas will come from followers wishing for a specific project or industry vertical rollout.

Delphix in a Nutshell

Delphix virtualizes databases and applications file systems to provide complete fully functional copies that operate in a fraction of the space that a physical copy would consume, with improved mobile agility, simplified manageability, and equal performance. The Delphix Engine is a self-contained operating environment and application that is packaged as OVA to run as a virtual appliance or an AMI in a cloud instance.

Enterprise level storage is attached to the Delphix engine and that storage will be used to house the ingested source copy, to update the source copy as changes occur, and to be the source storage for the downstream non-prod copies of that ingested source. Figure 1 provides a bird's eye view of the Delphix system data flow in a nutshell.

Figure 1 - A Data as a Service platform enables IT organizations to efficiently collect, control, and consume data on demand.

The Delphix Engine links to source physical databases or application file systems via standard Application Programming Interfaces (API) and asks the source databases to send copies of their entire file and log blocks to it. The Delphix Engine uses intelligent filtering and compression to reduce the copy of the source database down to as little as 25% of the original size. The copy of the source database and/or source application stored in the Delphix Engine, along with all incremental updates, is referred to as the dSource in Delphix terminology.

Delphix Processing

When the Delphix Engine is linked to a source, it collects a backup copy of the database or file system, compresses it and stores the backup copy on its storage system. The Delphix Engine then keeps a history (called a TimeFlow) of the state of the source system by periodically taking incremental snapshots of the source and optionally collecting log files from it. From this history, the Delphix Engine can create a virtual copy of the source that is identical to the source at any point in its TimeFlow. The Delphix Engine provisions the virtual copy to a Target Host in as little as 5 to 10 minutes using and NFS or iSCSI protocol. The virtual copy is then made available to downstream users who can stop, start, rewind, refresh bookmark or share their copy without additional support from a Database or Application Administrator. Figure 2 shows the data flow process within the Delphix Engine.

Figure 2 - Delphix delivers multiple masked data copies across heterogeneous systems in minutes vs. days or weeks

Delphix Operating Considerations

The Delphix technology manipulates the ingested sources as single objects at the storage level. Delphix offers data users and developers the opportunity to roll back, rewind, share, branch and bookmark operations on a virtual object level. However, there are caveats one must consider. First and foremost the self-service features of the virtual objects are tied to the objects as a whole, which means internal object components such as schemas, tables, or stored procedures cannot be individually targeted for these operations.

Another thing to consider, revolves around standard resource capacity planning. As with any technology the Delphix system relies on adequate server CPU and memory allocation. The Delphix virtual environments uses these resources and must be monitored closely by system engineers to ensure the operations of rolling forward, rolling, back and provisioning additional virtual environments or dSources remain within acceptable control limits and performance needs of these resources. So I think you have enough background information to follow this blog post without problem. If you need additional information regarding Delphix technology before following my blog, please go to http://www.delphix.com or just search for Delphix online.

Let's get started on our project delivery

Our mission is to design and implement a secure Delphix virtual data solution for databases and respective application binary sets supporting a healthcare business operations system. The overall goal of the project will be to integrate masked virtual data solutions into the DevOps self service automation processes so as to minimize the amount of time it takes to reset or stand-up and to de-identify the non-production system environments.

Project history and required deliverables

The three items mentioned below are critical to the understanding of this system rollout. The documents discussed will be specific enough to be included in the blog but generic enough to allow you to keep, reuse and modify, or throw away as you deem appropriate.

1. The company has already purchased 4 Delphix Data Virtualization Engines with Agile masking and the Jetstream Self Service interface.

2. We have 60 days to complete the transformation

3. We will create a solution design document, a project plan, an integration plan, a test plan, a rollout plan, a requirements traceability matrix, and a final system design document.

The healthcare business operations architecture

The company is software as a service healthcare insurance and claims processing entity. It hosts many clients on its medical processing platforms and the hardware and processing is primarily on-premise. It is important for us to understand the transaction flows for its members, providers, and patient care consulting teams. The project scope is limited to integrating Delphix virtual Database technologies in the Core Claims Processing business flow, the Provider Network processing business flow and the Patient care management transaction flows. To help you better understand these processes I have included sample Healthcare IT component system diagram.

Figure 3 - Depicts the Member, Provider, Claims, Analytics, Operational and Health Exchange data points of a typical Healthcare IT component solution.

Where will Delphix Data Virtualization and Agile Masking have an impact?

In this case Delphix Secure Virtual Data technologies will be used to replace certain non-production physical databases and applications. This means that at some point the systems of those non-prod components will need to be defined, sized and configured to work with the Delphix processes. Many of these systems were most likely identified in the sales process but probably at a very high level in terms of the component functionality itself such as the figure 3.0 provides. Little can be gleaned from the process drawing itself, a detail systems inventory of each of the components identified for scope will need to occur during the project solutioning tasks. A good starting point for this task will be to use the company's business component diagram or Business Architecture Model to act as a guide for discovery by simply annotating the expected replacement points of the physical databases and applications as well as the where the masking of sensitive data will most like occur. As you can see from figure 4, I have identified the integration points where Delphix Virtualization and masking will take place. For this project I will be virtualizing the applications that relate directly to back end databases, marked with the Delphix logo, in order to allow both database and the application virtualization to occur simultaneously. Likewise, I have added a masking symbol to systems that I believe should be reviewed for potential sensitive data that needs to be de-identified.

Figure 4 - The Healthcare IT Non-Production environment components have been identified with Data Virtualization and Data Masking Solution markers for use during the solution design and system inventory tasks.

Next Installment Post Building the Solution Design

This ends the blog for today. Follow my next post where I build out the solution design and deep dive into the customer's database and application inventory. At the end of that blog we will be ready to size the Delphix Virtual Database and Masking Engine's vCPU and Memory components and determine the backend storage that will be assigned.