Enterprise Level Deployment - The Virtual to Physical Transition Process

Hi again!  In today's post I will continue my discussion of my building a Solutions Designs for an enterprise level deployment of Delphix Virtual Data systems in our fictitious healthcare company. If you are new to Delphix technology, please go to https://www.delphix.com or just search for Delphix online.

To bring you up to speed from the last post - The Solution Design, we walked through the first 4 subjects of the Table of Contents. The week I will discuss Next Practices for companies that now have Delphix VCDM. Please refer to my previous blogs for Enterprise Level Deployment of Delphix listed here in order...

You have Delphix Now What | Delphix Goals and Objectives | The Solution Design - Part1

| The solution Design Part2

The Solution Design Document

We have already moved through the first 4 chapters of the solution Design. Lets look at chapter 5 - the Migration Process. This is where the Solution Architect makes the most impact. He or she needs to discuss how the move from the Physical Copy Data Management Solution to the Virtual Copy Data Management solution will transpire. The solution needs to talk directly to the teams that will implement the transition. The architect needs to address nuances of department processes, must solve for any pre-requisites of the solution, and be fully engaged in current internal and external SLAs for the systems impacted.  The detail he or she provides in this chapter will help the engineers design the requirements, motions and events for their specific tasks.

Solution Design Document

Table of Contents

1.0     Project Background.............................................................................................        6

2.0     Technology Descriptions.....................................................................................        8

3.0     Theory of Operations.............................................................................................       19

4.0     Next Practices (What Changes with Delphix)....................................................          25

5.0     Migration Process.................................................................................................       36

5.1    RDBMS ...............................................................................................................      36

5.2    Applications.........................................................................................................       41

5.3    Automation Tasks................................................................................................       48

6.0     Masking Process.................................................................................................        52

6.1    Corporate Security Policies................................................................................           52

6.2    Databases...........................................................................................................       55

6.3    File Systems........................................................................................................       61

6.3    Audit Trails...........................................................................................................       66

7.0     High Level Timeline..............................................................................................        71

7.1    Preplanning and Solutioning ..............................................................................         71

7.2    Initialization and Startup.....................................................................................         72

7.3    Development Configuration and Integration.......................................................          73

7.4    Operations and Maintenance..............................................................................         74

The Migration Process

The migration process in this context means the general approach to replacing the physical copy with a virtual copy. It does not portray outside processes such as any change control or downtime scheduling those considerations will be dealt with during the project plan and project execution. The solution design is to communicate the overall technology solution and relation ships among the different components.  The migration process flow diagram is not meant to dictate to the engineers the steps best practices for tasks within the process those are left to the engineers. The solution architect will have conferred with the respective DBA and application owners and the Delphix admin to ensure all aspects of the migration are taken into consideration.

The Easy Guide to Data Virtualization for DevOps Test Data 

Clarify your understanding of data virtualization with this white paper. You’ll learn how and why data virtualization reduces business risk, simplifies data version control, and boosts productivity. You’ll view diagrams of how it improves physical storage and manual provisioning. And you’ll learn how data virtualization slots into a DevOps test data management solution like Delphix to bring even more benefits to enterprises. 

Get your copy 

The RDBMS Transition

The straight transition from a physical to virtual RDBMS is quite strait forward with Delphix. But before I go into that let me set the ground rules. Delphix is a Virtual Copy Data Management System (VCDMS) for like databases and versions.  The source and Target databases must be the same RDBMS vendor, platform and version see list for DB server and OS platform.  Delphix does not provide cross platform Original Equipment Manufacturer (OEM) virtualization such as Oracle source to SQL Server target. It does however, offer a migration path for Oracle Unix to Linux conversions and I will not be discussing that process here.  Always confer with Delphix documentation for validate latest status.

1. Source and Target Database Servers and OS platform  (use links below for latest status)

   • Oracle

     • Oracle 9.2.0.8, 10.2, 11.1, 11.2, 12.1

     • Solaris (SPARC/X86), RHEL, OEL, SUSE, AIX, HPUX

   • SQL Server

     • SQL Server 2005, 2008, 2008R2, 2012, 2014

     • Windows Server 2003SP2, 2003R2, 2008, 2008R2, 2012, 2012 R2

   • PostgreSQL Server

     • PostgreSQL/Enterprise DB Postgres Plus Advanced Server 9.2

     • RHEL 5, 6

   • SAP ASE Server

     • SAP ASE 12.5, 15.0.3, 15.5, 15.7

     • RHEL 6.2, 6.3, 6.4 | Solaris 10 (SPARC/X86)

   • MYSQL Server

     • MySQL Community 5.5, 5.6, 5.7.7 | Enterprise 5.6 | Maria > 10.0.10

     • RHEL 6.2, 6.3, 6.4 | Solaris 11 (SPARC/X86)

   • DB2 Server

     • DB2 LUW 10.1+

     • RHEL 6.5+, AIX 6.1+

Fortunately, the healthcare company's DB systems for the solution design here, are only Oracle 11.2.0.1 on RHEL and SQL Server 2014 on Windows 2012 R2.  Which means, there are no pre requisite upgrades or platform migrations needed.

The physical to virtual migration process summary is needed to frame the high level overview of the migration efforts required by the Solution Architect, the Delphix administrator, Database Administrators and Data Security Teams. It also provides the management an idea of the level of effort required as they determine resource needs from their respective team.  In the diagrams shown the process boxes are annotated to indicate it is a process that will be executed from within the Delphix technology.

The Applications (vFiles) Transition Process

Here is where my approach to VCDM may differ from my colleagues and that is that I believe the application vFile dSource will start from a non-production staging location and pushed forward through the SDLC environments rather than pulled down from a production deployment.  The vFiles can be an application as complex as and multi file system SAP environment or s simple as custom Web application hosted by Apache. I believe this because in the deployment of an application environment should come from a core set or dry installation of the OEM files. What I mean by dry installation is a location that has the minimum amount of configuration changes required to run. This allows the operations team a documented starting point where a before and after image of configuration edit points for ease of automated deployment methods.  There may be a future need to provision from a dSource ingested from a production environment, but I would expect that would be more in line with break-fix analysis of a production related issue more so the from a standard SDLC promotion process. Since the source vFiles will be coming from a dry installation location there will be no sensitive data lurking about the file systems so the masking and selective data distribution process decision point is unnecessary. So keeping true to those two VCDMS deployment tactics of mine, the swim lane diagram provides the transition process I use for a physical application to virtual application or vFile migration process.

Automation Tasks

At the creation and throughout the life the virtual objects, Delphix provides the option to relate automated tasks before and after the synchronization and provisioning process.  These task automations are called hooks and can be scheduled to run before and after refresh processes, before and after rewind processes, and before and after snapshot processes. They are generally added by the DBA or application administrators to handle routine tasks generally applied during these processes, regardless physical or virtual state of the environment. The tasks may include a shell script to stop an application when a database is stopped during the rewind call. The task may be needed to make appropriate localization edits such as dropping production level DB users and adding developers user accounts after a DB refresh from production. These scripts are critical to provision a DB or application for data as a service. During the Development, Configuration and Integration phase of the project it is expected the appropriate teams will design, develop and assign the scripts necessary to duplicate these actions.

Command Line Interface (CLI)

The Delphix Engine provides a native command line interface (CLI) accessible over SSH. This CLI provides an interactive layer on top of the public web service APIs, and is intended for users that wish to automate interactions with the Delphix Engine, or simply prefer a text based interface. Individual commands passed as arguments to the SSH client will be interpreted as if they had been read from the terminal. More complex scripts can be passed as input to the SSH command. When running SSH in non-interactive mode via these mechanisms, the command line prompt will be suppressed, as will terminal font decorations such as underline and bold. Please review Delphix documentation about CLI functionality.

The command line interface is a direct translation of the web services API to an interactive environment. This allows you to use the CLI to explore functionality with tab completion, integrated help, stronger type checking, and indication of expected types and required fields. When trying to determine how to invoke an operation through the web services or interpret the results, it is recommended that you first learn how to do the same through the CLI, and then use the provided tools to translate that into web services call.

WEB Service API

The Delphix Service provides a set of public stable web service Application Programming Interfaces (API).

The web services form the basis upon with the GUI and CLI are built, and are designed to be public and stable. With allows teams to build automation facilities that easily integrate with third party orchestration tools. The CLI is a thin veneer over the web services. If you are new to the web services, it is recommended you first test out operations with the CLI. Please review Delphix documentation about WebService API functionality for additional Information

The following HTTP methods are supported:

• GET - Retrieve data from the server where complex input is not needed. All GET requests are guaranteed to be read-only, but not all read-only requests are required to use GET. Simple input (strings, number, boolean values) can be passed as query parameters.

• POST - Issue a read/write operation, or make a read-only call that requires complex input. The optional body of the call is expressed as JSON.

• DELETE - Delete an object on the system. For languages that don't provide a native wrapper for DELETE, or for delete operations with optional input, all delete operations can also be invoked as POST to the same URL with /delete appended to it.