What is Continuous Integration (CI)?

Continuous Integration (CI) is the first stage of an automated software release pipeline and is referred to as the Development Stage. It enables application development teams to independently build, test, and merge source code changes to applications during the software development lifecycle (SDLC).

Code Repository

The code repository is a core component for building a CI pipeline. Though there are many version control systems in the market, Git and managed offerings such as Github, Gitlab, and AWS Code Commit are common choices.

Git enables development team members to checkout source code from a repository, develop features in an isolated branch, commit source code changes, run tests, and build artifacts from a central code repository.

Supporting DevOps

When moving from waterfall to an agile software development process, software development teams are grouped into smaller software projects with direct ownership of source code and its operation in a production environment.

Implementing continuous integration helps these partitioned teams release code independently while also conforming to organizational controls for the software development practice.

Data for DevOps Teams

Access to high quality data early in the software development lifecycle is critical for Test Driven Development (TDD) initiatives. With fresh realistic test data, a software development team member can write tests with better test coverage.

Test values based on real production data helps to surface data handling exceptions with malformed values. This improves the feedback mechanism for resolving unhandled exceptions in applications earlier in the CI CD pipeline during unit tests–ultimately shifting issues left in the SDLC.

DevOps TDM For Dummies® Delphix Special Edition

Find out why DevOps test data management (TDM) is magic in this eBook. You'll explore how DevOps TDM enables speed, quality, and compliance — without manual provisioning or copying of any resource. And you'll get real-world use cases of test data management in financial services, retail, and human resources.

Get DevOps TDM For Dummies >>

Continuous Integration Server

The Continuous Integration Server (Build Server) is a key component for automation during the software development process. It monitors the Code Repository for a new code pull request initiated by software developers and generates new artifacts following the Continuous Integration automated build process.

The CI process not only compiles a project's source code into an application binary, but also collates machine readable metadata about the process in the shared repository. Once the the continuous integration process is complete, the CI pipeline stage will typically trigger a notification through email, hook operations to external services, or push notification so that the continuous delivery workflow can initiate.

Automated Builds

Continuous integration tools help organizations manage complexity in the development cycle for releasing code at a high cadence in a consistent way. The CI process creates an automated build artifact when new code is ready. It also stamps a release version, captures code review details, compiles the code, runs pre-defined tests, security scans, and publishes builds to a central code repository–with a complete audit log of the process.

Automated Testing

Automated tests are crucial for scaling up Continuous Integration practices. A Continuous Integration test suite is used to check and document if new code meets organizational quality and security standards in the development process. An automated test suite for a larger composite application may execute multiple component unit tests before running a larger Continuous Integration test suite.

Best Practice

Organizations practicing Continuous Integration in their software development lifecycle can greatly improve developer productivity by eliminating complex and error prone manual processes. Continuous Integration ensures that software development teams are on the same page and are following best practice guidelines for releases set by the organization.

Why Continuous Integration?

Reduce Developer Friction

All modern businesses are both software and data companies. As such, business systems must include a continuous feedback mechanism for existing code to be iterated by a software engineering team to deliver new features, fix code defects discovered in production, or patch the latest exploits found during security testing.

This necessitates a modern software development approach–aligning with agile development methodologies–that leverages DevOps automation to quickly build, run automated tests, and release code with improved speed, quality, compliance, and sustainability.

Fast Independent Releases

Organizations looking to accelerate release cycles (i.e. from annual, quarterly, monthly, or daily to multiple times per day) must enable independent code changes in their software development practice. Having small teams helps to improve communication speed between members, produces tight scopes of work, accelerates development of new features, and reduces overall business risk.

Isolation of Feature Changes in Branches

The source code version control system in a CI pipeline enables DevOps teams to checkout code from the main trunk into isolated feature branches. Here DevOps teams can make small code changes without directly impacting other developers.

Merging Update Branches

When code changes are qualified, a developer uses a pull request to signal intent to merge changes to the main branch of a shared repository. There it can be further tested automatically and integrated with other pieces of a larger system. Feature branches help ensure that changes are well contained and can be quickly integrated without breaking the software system as a whole.

Moving Quickly to Continuous Delivery

Comprehensive System Integration Tests take a long time to complete, so accelerating new code through the initial software release process and into the continuous delivery stage is crucial for accelerating Continuous Integration and Continuous Delivery pipelines.

The Continuous Delivery stage takes a qualified build from the Continuous Integration stage and works to automate deployment in operation environment. This involves putting the build onto a host with an operating system with declarative settings defined in a configuration management system. Automated deployment eliminates manual processes for getting code running in downstream environments such as Development (DEV), Quality Assurance (QA), and User Acceptance Testing (UAT) for further integration testing and performance testing.

Hurdles Implementing Continuous Integration

Working Through Integration Testing Issues

Building a modern CI CD workflow requires changes to people, process, and technology. More Continuous Integration automation does not directly eliminate software bugs. In addition, it may take time to integrate new external services into the workflow.

As issues shift left in the SDLC with CI CD improvements, developers can become overwhelmed by the influx of quickly found bugs preventing a release. A blameless culture is necessary to elevate team morale while the development process is still being optimized.

Improving Test Quality

In the beginning, new continuous integration and continuous delivery processes will likely surface issues more consistently and more often as teams push code to the central repository. As teams build self testing methods–such as code linting processes, improved quality standards, and security scanning of libraries–that eliminate issues earlier in the development process, software defects will eventually stabilize.

Required Integration Time

It's nearly impossible to introduce an all new CI CD process at once. Organizations should start small: rolling changes to individual projects or teams–letting a team integrate on their own.

It's most effective to introduce the Continuous Integration process to greenfield applications at the start of the project to prioritize quick wins. Migrating legacy workflows with many dependencies can be extremely time consuming.

Tracking the Right Metrics

The DevOps Research and Assessment (DORA) framework makes it easy for teams to track the two main areas of DevOps progress: speed and stability.

• Change lead time: How long does it take to run your entire workflow? Automating the CI CD pipeline can transform a process that takes weeks or months down to hours or minutes.

• Deployment frequency: How much work is moving through the development pipeline? Automation helps ensure that even the most complicated updates are deployed quickly and easily through a defined CI CD pipeline.

• Change fail percentage: Simply, how many changes to the primary code base result in failures? 

• Mean time to recovery (MTTR): How long does it take to recover from an incident if one does arise?

Key Outcomes of Continuous Integration

Continuous Integration makes it faster and easier for enterprise software teams to independently design, individually test code, and iterate on components of a large composite application—without wasting time waiting on other teams, working through undefined workflows, and stumbling through complex manual processes.

Development Team Empowerment

Continuous Integration improves agile development workflows by reducing communication and coordination friction to get teams on the same page quickly. Developers can work in their own testing environment without impacting adjacent teams and initiate a pull request for changes to get merged, run through automated testing, approved, and then made available for other developers to test with their new code.

Scaling Teams Quickly

Continuous Integration improves developer autonomy, drives accountability, visibility, and communication among software developers. This helps organizations effectively scale their engineering teams as the size of their codebase grows without compromising speed and quality.

Faster Iterations, Less Risk

Automated testing in the Continuous Integration stage helps development teams detect bugs and defects in a codebase earlier—making them easier to fix while reducing the risk involved significantly. When code modifications are isolated in short timespan branches with fast development cycles, developers are able to work in environments that are more closely related to what's in a production environment.

DevOps Culture Shift

Smaller changes closer to the main branch helps organizations improve the feedback mechanism in their software development practice. Teams can better add new features, fix bugs, and apply security patches rapidly. This feedback loop is incredibly valuable for product managers to guide product direction iteratively and prioritize the development of new features in the software development process. 

Declarative Data for Continuous Integration

Most modern teams have automated much of their Continuous Integration, Continuous Delivery, and Continuous Deployment workflows but are still greatly challenged with development bottlenecks, production code defects, and new feature delivery delays. Though they can quickly provision and teardown environments–reducing idle waste, adding parallel environments, and reducing contention of shared resources–their access to fresh realistic data is still limited.

The Heavy Lift of Data

Data is still a heavy lift that is marred by ticketed processes, complex infrastructure & software configurations, and large data compliance hurdles. Teams need a modern approach to test data management that aligns with the needs of DevOps teams to take new features to market quickly. This includes quickly selecting and provisioning fresh, realistic test data from any point in time independently into DEV/TEST/QA environments.

Rise of Cloud Native Technology

The rise of cloud native technologies has greatly reduced the cost of compute, storage, and networking–enabling the delivery of ephemeral environments using declarative infrastructure as code. However, delivering data to test environments during the Continuous Integration stage rarely matches the pace necessary for releases–data is a heavy lift. Data is typically heavy, complex, and filled with risk.

Declarative Data as Code

Data friction creates huge barriers to software delivery speed, code quality, data compliance, and sustainability. Modern DevOps teams need a better way to declare data as code as they declare infrastructure as code to be effective during the Continuous Integration stage. DevOps TDM is designed to eliminate this data management complexity by providing easy to use declarative data APIs and a curated user interface to teams.

Production Environment Sync

Accessing and refreshing data from multiple disparate production environment sources to a test environment typically requires the involvement of IT operations, database administrators, storage specialists, compliance teams, and network administrators— often delaying requests to the point that teams just work with what they have.

Streamlined Data Workflows

DevOps TDM enables modern teams to connect production enterprise data sources to be continuously synchronized, versioned, run through defined masking workflows, cataloged, and delivered to downstream environments (both asynchronously and continuously). This helps to eliminate issues caused by stale production data, partially subsetted data, and purely synthetic data that leads to testing errors, product defects, and preventable workflow mistakes.

Getting to Continuous Deployment

DevOps TDM helps teams spend their valuable time developing and releasing high quality code instead of being tied up coordinating and orchestrating across multiple teams with their own processes. Access to data enables teams to shift issues left during the software development lifecycle and evolve their Continuous Integration and Continuous Delivery processes to eventually mature to a Continuous Deployment process.