Enterprise Level Deployment of Delphix Solution Design - Part 2

Hi Again!  In today's post I will continue my discussion of my building a Solutions Designs for an enterprise level deployment of Delphix Virtual Data systems in our fictitious healthcare company. If you are new to Delphix technology, please go to https://www.delphix.com or just search for Delphix online.

To bring you up to speed from the last post - The solution Design, we walked through the first 4 subjects of the Table of Contents. Next week I discuss the physcial to virtual change over mirgation process solution itself. Please refer to my previous blogs for Enterprise Level Deployment of Delphix listed here in order...

You have Delphix Now What | Delphix Goals and Objectives | The Solution Design - Part1

The Solution Design Document

After the goals and objectives are determined, I generally move on to building out the solution design document. To me this will be the document I refer to most throughout all phases of the implementation. Solution Design Documents vary form architect to architect and the figure below shows the contents that I address for my Delphix Deployments. Feel free to use my template or develop your own.  For brevity the table of contents shown here does not include the 3 level styles.  If you would like the full template, I will be posting the deliverable templates and a PowerPoint on my final installment of this series.  Last week we completed through Theory of Operations now let's discuss Next Practices Chapter 4.

Solution Design Document

Table of Contents

1.0     Project Background.............................................................................................        6

1.1    Solution Goals Objectives...................................................................................        6

2.0     Technology Descriptions.....................................................................................        8

2.1    Delphix Virtual Data Engine ................................................................................         8

2.2   Jet Stream Self-Service.......................................................................................          10

2.3   Agile Masking......................................................................................................         13

2.4    Integration Services.............................................................................................        14

2.5    IT Operations.......................................................................................................       15

2.6    Data Services.......................................................................................................       16

2.7    Hosting ................................................................................................................      16

2.8    Networking...........................................................................................................       17

3.0     Theory of Operations.............................................................................................       19

3.1    VCDMS Provisioning Flow....................................................................................       19

3.2    Masking Process Flow..........................................................................................      19

3.3    Roles and Responsibility......................................................................................       19

3.4    Quality Management............................................................................................       20

4.0     Next Practices (What Changes with Delphix)....................................................          25

4.1    VDB and VApp Request ....................................................................................          25

4.2   Data Self Services....................................................................................................    28

4.3    CMDB Updates................................................................................................           31

4.4    DBA Operations................................................................................................           33

5.0     Migration Process.................................................................................................       36

5.1    RDBMS ...............................................................................................................      36

5.2    Applications.........................................................................................................       41

5.3    Automation Tasks................................................................................................       48

6.0     Masking Process.................................................................................................        52

6.1    Corporate Security Policies................................................................................           52

6.2    Databases...........................................................................................................       55

6.3    File Systems........................................................................................................       61

6.3    Audit Trails...........................................................................................................       66

7.0     High Level Timeline..............................................................................................        71

7.1    Preplanning and Solutioning ..............................................................................         71

7.2    Initialization and Startup.....................................................................................         72

7.3    Development Configuration and Integration.......................................................          73

7.4    Operations and Maintenance..............................................................................         74

Next Practices (What Changes with Delphix)

The section is used to describe what changes for companies now that they have a have the ability to provision large amounts of data in minutes and allow self service to end users of the non-prod environments. Next Practices are about leveraging innovation with your new freedoms associated with the agility that comes with Delphix Virtual Data Copy Data Management Systems (VCDMS).

Virtual Database and/or Virtual Application Request

First, I would like to talk about the provisioning request itself. Last week I provided the process flow diagram, which DBAs and Application Admins would use to provision the system. I apologize now for the marketing look and feel on this topic but this next part is one of the coolest parts of Delphix VCDM. Certainly we need to expect positive change with next generation technology like Delphix and what data consumers can expect from data delivery now is so beneficial and almost limitless.

So what changes can be expected for these end users when Delphix VCDMS enters the world of: Application Developers, DBAs, Automation Engineers, Benefits Administrators, Financial Systems Administrators, and Quality Assurance Teams. The figure below provides a view of what is moved from the IT team to Delphix VCDMS data consumers.

1. Gone are the days of waiting for a new DB or Application environment. In is the ability to stand up a DB and Application Environment in 15 minutes.

2. Gone are the endless numbers of DB refresh requests that require hours and days to pull from backup systems. In is the ability for DBAs to offer these DB refresh requests to end users as consistent functionality of self service activities across all supported platforms,

3. Gone are the days of IT professionals figuring out how to stand up time consistent environments for QA integration testing.  In comes Delphix Software that with the click of 1 button or the single RestAPI call can reset a number of our supported database platforms and applications for the same point-in-time.

4. Gone are the endless hours developers spend creating back-out scripts or processes to ease a reset capability for their development or trial financial close efforts. In is the data bookmark/snapshot capability to allow end users the option to version data and literally run rinse and repeat operations on large data sets in minutes - allowing an increased number of iteration testing without engaging DBAs or other Infra professionals help.

5. Gone are the days of DevOps tools missing the ability to include a full sized current copy of a database within its delivery practices.  In is Delphix's APIs that make it easy to build custom integrations for any DevOps tool, delivering true plug-and-play capabilities.

6. Gone are the days of customers wanting UAT access to their hosted systems only to be challenged with a high-ticket price or clunky VPN access requirements. In is the capability to offer full size UAT environment to customers at the customer facility and provide all he data- on-demand self-service capabilities one can offer to its internal staff.

7. I think you get the idea and you can name any number of efficiencies that Delphix offers as an enterprise level data-provisioning tool. For my follows please sent me your own examples of what is out and what is in with Delphix VCDMS. And if you would like detail explanation on how to make your example happen I will love to help you.

To me, no longer needing to request a data refresh from DBAs, or request the IT team to synchronize an environment for integrated testing or not needing to figure out a data escape route if my test trial close doesn't provide the balance I expected, allows control of my time and have more time to work on task innovations, development efforts, and process improvements.

JetStream Data Self-Service

Here is the fun aspect of the Delphix agile data platform, and is how the tasks mentioned above move from the confines of the IT arm to the data consumer's fingertips. What if you find yourself or development staff in the middle of a release cycle and your boss calls you to research a production data problem? With VCDM, you just need to bookmark where you are in your current development work, and then refresh from production to your current development arena. No need to wait on anyone else in your company. Within 15 minutes you can be actively working the break fix issue. Once complete, send operations the resolution, stop the break-fix environment and return to the exact stopping point of your development efforts.

Here is how it works.... During the implementation phase, is that the Delphix Admin creates a Jet Stream Data Template of any number of integrated DB and application components. The components added, will have been vetted out by the users of the environments:  such as an app development team, benefit configuration team, or a QA team, basically, any team that uses non-production level environments today.  From this template, a Jetstream Data Container that has one or all of the time-flow synchronize components of the template is defined. The data container is assigned a login owner and that owner now has all the self-service functionality and agility of virtual copy data management that I mentioned above and as I point out below. The Delphix descriptions of the software functionality are mentioned here.

JetStream Data Templates: An Overview

A JetStream data template represents a collection of data sources that you can provision to a JetStream user. A data source can be a dSource, a VDB, or vFiles. These sources can be used in multiple data templates. Once you have created a data template, the set of data sources associated with it is fixed; you cannot add data sources to an existing template, nor can you remove data sources from it. In addition to data sources, you can define the set of metadata that is relevant for a given template - for example, notes, descriptions, names for sources that are relevant to an end user, and other configuration details. Once you have created the template, it provides a stencil for provisioning data containers. This, in turn, enables Jet Stream users to have self-service access to a space-efficient copy of the data sources defined in the data template.

Data templates are managed by a Delphix admin. The admin can provision data containers from the data template and assign a data container to an end user. The admin can also create bookmarks on the data template timeline in order to mark meaningful points in time. When creating a data template, it is important to consider the set of users who will own data containers provisioned from it. In Jet Stream, templates effectively define the boundaries of the data that users can share directly with each other. Only owners of data containers created from the same data template are able to share data using bookmarks.

JetStream Data Container Overview

Data containers are provisioned from data templates by administrators and assigned to a JetStream user. A data container represents a socket that is capable of making any data within the data template accessible. The JetStream user controls what data they want to access. JetStream users have effectively been provisioned a set of "physical" resources, A data container is comprised of vDBs or vAPPs provisioned from each source in the data template from which it is created. The data container manages these vDBs, and the data operations performed on a data container will only impact these vDBs. Data containers represent the separation between IT infrastructure and end users. IT determines the set of vDBs or vAPPs to allocate to a data container, and Jet Stream users determine the data that they want accessible in the containers allocated to them.

Data containers can be used to access any data within a single data template, but not across templates. Jet Stream users have the ability to populate the data within their data container from any point in time on the data template, the data container's history, or shared bookmarks from other data containers. Although operations are all accomplished by performing time-flow operations on the underlying vDBs and vAPP, the data containers hide the underlying properties from Jet Stream users. None of the data container operations require additional provisioning; everything is accomplished using the resources assigned when the data container is created.

Data Container Self-Service Toolbar

The Data Container Self-Service Toolbar allows you to perform tasks and activities with data in the current container, by clicking on the following user action icons. The toolbar functionality is available in 3 user interface options: the JetStream Graphical User Interface (GUI), the Command Line Interface (CLI) or via a RestAPI call. The figure below summarizes the Self Service toolbar functions.

Configuration Management Database (CMDB) Updates

This topic, relating to ITIL v3, is not as fun as the one above but just as important and is a critical part of integrating Delphix VCDM to scale. I get asked often "How do you track changes to the seemingly endless databases that can now be available. How would update a CMDB?" This can be challenging for companies that rely on spreadsheets to keep database inventory up to date; but can be very simple for companies that use a commercially available configuration management database systems. But for those reading my Blog you now have a formidable answer and approach to help these companies understand the next practices within the CMDB.

The big change is not that we need to track DB and application copies in a CMDB but at what level do we track now that we have a VCDMS. My recommendation is to track all provisioned databases but stop at the Data Container level. So what does that mean with Delphix, a DBA or App admin will create the vDB or vApp and those are most certainly included as standard entries for tracking as they take server resources and for any kind of server change or outage they can be impacted. The self-service activities such as bookmark, refresh, rewind and branch will be treated as application data changes only  and not recorded in a CMDB.

Some Delphix objects will be new to a CMDB and these new entries with enterprise level deployment of Delphix would be hardware: the Delphix Engines, the Masking engines, the pre-provisioning servers and for DB and App environments: the dSource, the vDBs and vAPPs. A new entity type to CMDB would be the Delphix Template, and Container definitions within the Delphix engine.  Listed here are the objects tracked in Delphix and which objects could be easily added to a CMDB tracking device with RestAPI integration updates to the CMDB tool. The process data flow tpoint hat addresses when to update the CMDB is updated is provided in last weeks blog under Theory of Operations.

The items in red indicate a new CMDB tracking item/entity for companies that are just starting out with Delphix VCDMS.  The new CI Class and CI Family entities may have a new attribute tracking components and will need to be reviewed, updated or created to accurately represent the next practices of Delphix VCDM technology.

DBA Operations

With Delphix now managing the storage and startup/shutdown command for the virtual databases, DBAs need to change the way they interact with the data base level commands of the database. In a non-virtualized environment, the DBA teams are very comfortable starting, stopping, and suspending user operations of their DB from either a command line or an OEM management console. That practice will need to change (i.e. Next Practices) the DBAs now need to consider the Delphix interfaces. If the database is tagged for JetStream, the DBAs should execute the start and stop command from the 3 user interface options open to them: the JetStream Graphical User Interface (GUI), via Command Line Interface (CLI) or via RestAPI calls. If the database is not within the JetStream Self Service, the DBA should execute the start and stop commands from the Data Management GUI and CLI or via RestAPI Calls. There is a command line toolkit called "dxtoolkit" available to Delphix customers that have packaged many of the CLI calls in Python ready for easy use and alterations for scripting.

The old school DB management methods are available to DBAs for specific uses, just remember if one shuts down or starts up a virtual database not using the Delphix interfaces, the un-mount and mount of its supporting file systems will be absent and will keep files mounted in a shutdown operations and may cause errors in startup. There are times when the DBA wants to remove control for the Delphix engine such as when performing a DB upgrade or patch event. Those DBA management processes and tasks are fully defined and explained on the Delphix online documentation relating to the specific DBMS. DBAs should runt though operational scenarios in a lab or non-operational system to practice and document the operational changes that differ from Physical Database operations vs. virtual database operations.

An often-neglected practice in non-production operations, is backing up of the databases. Many companies take the stance that the "restore' method is to use the production backup to restore. With VCDM and the nature of snapshots and bookmarks, companies can expect near up to date restore capabilities of their non-prod databases just because they have invested in the technology. But if there are backups in-place, this is where the practice can be ended and the VCDMS can take over.

Next Installment Post - Continue Building the Solution Design  (The Physical to Virtual migration process)

This ends the blog for today. Follow my next post we will continue to build out the solution design and deep dive into the solution we will use to change from physical to virtual databases and applications.