When this Fortune 50 company chose to move all of the content at its top-level domain into a Drupal CMS, it undertook an ambitious project. The project involved multiple groups of stakeholders, representing discrete business units within the organization, and required migrating a massive amount of data into a new platform. Unifying the design language, building out dynamic-looking pages that still met the needs of individual business units, and giving stakeholders the ability to make changes was a project that stretched across the entire organization.
To accomplish all this, this Fortune 50 company hired Lullabot as their strategy, design, and development vendor. The development process, while outwardly familiar, was constrained by teams having access to only a single development and testing server, as well as a lengthy release process that slowed velocity and reduced confidence in the accuracy of changes being deployed to production. Lullabot built Tugboat to solve these very challenges.
A common development process that couldn’t keep pace with business needs
This company’s core development process is a common one, used in many organizations:
- The team gets tickets for new features or bug fixes via Jira
- Developers do the work, then create a pull request in GitHub
- A dedicated QA team validates the code and then approves or rejects the changes
- Rework as needed until code changes are accepted
Once code changes are approved, they go into a lengthy release cycle where teams get one week to test and patch bugs before a go/no go point for the release. At that point, code that isn’t ready for production gets pulled, and a release happens with all of the approved code.
The process is simple, but business needs highlighted a few challenges:
- Constraints in how many tickets could be reviewed simultaneously due to limited testing servers; each development team gets only one deploy environment
- Lost development time because:
- Devs who are in the middle of in-progress tickets must destroy working locals to do code reviews or test changes
- Changing branches, pulling down code, resetting the database and waiting for changes to build locally could take hours for a large site; typically a few hours to reset on both sides of testing a ticket
- If the entire team’s release couldn’t be tested and patched in time for the release window to close, the team’s progress was stalled until the next release cycle
With 10 full-time Lullabot developers, plus the organization’s internal developers, that translates to a lot of code going around, exacerbating the bottleneck of a finite number of testing servers. Additional servers and faster resets were required to keep pace with the velocity of the team.
Deploy environments for every pull request
Tugboat’s paradigm of a deploy environment for every pull request solves this organization’s core development needs in a few different ways:
Test multiple pull requests simultaneously
With an ephemeral development environment for every pull request, the dedicated QA team can review multiple pull requests simultaneously. This enables QA to scale as business needs demand; no idle team members waiting for a testing or staging environment to be available so they can load and validate code functionality. Code can be reviewed simultaneously, independent of the availability of the team’s single shared development environment.
Faster resets between pull requests
Because Tugboat’s preview environments are ephemeral and on-demand, they can be building in the background while other pull requests are being reviewed. There is no hours-long delay on either side of reviewing a code change while development environments pull down code, reset the DB, and wait for a build. When QA is ready to start the review, the environment is there, at a secure URL, waiting for someone to perform the review. After the review is complete, the environment can be deleted.
With the addition of Tugboat’s Base Previews, what would normally take hours is completed in a few minutes. Base Previews enable this Fortune 50 client to leverage service commands to front-load the long build times of the large site into a Base Preview that serves as the starting point for subsequent Preview builds. Even if QA wants to validate a pull request that was just created, or a developer pushes out a fix and QA wants to re-review the pull request, the wait time is minutes versus hours when resetting a single traditional environment.
No need to destroy a working local development environment
As any developer who is new to a project can tell you, setting up a working local development environment for a new project at a large enterprise organization can take days — just to get to zero. Over time, developers working on a large feature request or a complex refactoring or migration project can accrete quite a collection of branch-specific database changes, libraries, tweaks, and dependencies.
Pulling down someone else’s code to do code review or testing on a different feature branch or config destroys that developer’s working local development environment. In some scenarios, it can take hours to reload a database and re-wrangle dependencies just to get the new code running locally, and then again on the other side of the process to get back to what the developer was just working on. In some cases, it can be next to impossible to maintain a local that is stable enough for new development work if you’re frequently destroying it to test and review code.
With those reviews removed from the local and instead occurring in a Tugboat environment, the developer’s local environment can remain intact, and the barrier to code review is removed. This process also results in fewer bugs making it through to the pending release, as developers have a more consistent development environment.
Incremental code testing enables rework and improves confidence ahead of the time-limited testing window
By enabling QA to test code in small chunks before it goes into the time-limited testing window along with the entirety of the release, Tugboat makes it possible to identify bugs and regressions earlier and submit code to the release pipeline with more confidence.
Testing code for each and every pull request, before they’re ever merged into a development or release branch, makes it possible to complete testing faster and debug and correct issues more quickly. Additionally, the ability to do this testing and rework asynchronously makes it possible to test more thoroughly and deploy with more confidence to the release pipeline.
A familiar workflow
This enterprise client’s process is a familiar one to Tugboat, so Tugboat is well-positioned to easily solve the bottlenecks in the process. To add Tugboat to the client’s workflow:
- Developers spent some time figuring out the client’s specific configuration, and then created a .tugboat/config.yml to replicate the client’s infrastructure in a Tugboat environment for testing purposes.
- Developers created some helper scripts to optimize the Preview build, such as shortcuts for pulling in large files, etc.
The caveat for this Fortune 50 client is that due to business requirements, it’s not possible to host the client’s production database on Tugboat’s cloud server infrastructure. Instead, Tugboat imports a database that approximates the client’s production configuration. While this approximation goes a long way toward testing simple changes, project managers, developers, and QA testers would have much higher confidence in accuracy and lack of regressions if they were using actual production data. Tugboat’s On-Premise offering solves these sorts of security concerns. Because this enterprise client has experienced firsthand the value of Tugboat’s capabilities, an On-Premise installation of Tugboat is underway.
A faster, more accurate development process
Without Tugboat, development and QA would be constrained to a meager trickle by the bottleneck of a single development environment. Productivity would plummet without multiple on-demand environments, and due to time constraints, it wouldn’t be possible to test code as thoroughly; quality would have to be sacrificed for speed.
With Tugboat in the workflow, the development team can push updates more frequently, and doesn’t have to sacrifice quality and rigorous testing. Code rework can be completed before the code ever goes into the release pipeline. With Tugboat in the process, quality and speed are no longer at odds.
More environments, same cost
Cost is a consideration when it comes to enterprise engineering infrastructure at scale. Part of the reason this Fortune 50 company gives each development team only one shared server is the cost of the hardware. It was easy to make a case for Tugboat as an infrastructure replacement; for the same price as the typical infrastructure’s single development server, development teams using Tugboat have access to many different environments simultaneously for multiple pull request builds.