Category: Continuous Delivery

In the previous episode, we talked about different service design approaches. This time, we dive deeper into database changes.

Even for teams that can effortlessly deploy their application code, database changes can be more stressful. Changing a schema is a high-stakes operation, and there are many ways to do it badly.

Read on to find out why database deployments are different from application deployments, and what techniques you can use to make them worry-free.

Watch the episode

You can watch the episode below, or read on to find some of the key discussion points.

When you change your application’s code and discover an issue, it’s trivial to revert to the previous version. In rare cases where a change affects your data, there may be cleanup to do.

You can use timestamps and a one-off process to fix those rare data-mess-up instances, but they hint at something fundamentally different about database updates. They have different levels and types of risk associated with them.

If you think backups will save you from a bad database deployment, you haven’t yet tried it. Sure, they prevent total data loss, but between your backup and your fix, the data moved. Often by a lot.

There are techniques to apply transactions from the backup up to the point of failure, but if your change caused an issue you needed to roll back, you probably don’t want to apply those transactions automatically. Welcome to the “data remediation project”.

If you added a new column or table as part of your database change and you need to roll back, you have to decide what to do with any data in those tables. Do you forget it, or do you need to keep hold of it and reapply it later when you make a new attempt to extend the schema?

Modern software teams prefer fix-forward for application issues, which are reasonably easy to roll back. Databases take rollbacks to another level.

Crucial modernization steps

Imagine you met a friend for lunch and they told you they store they application’s source code on a network share, rather than in version control. You’d think it was a joke, and when you realize it’s not, you’d form a strong opinion about the kind of sloppy outfit they must be running.

When we meet for lunch, what are you going to tell me about your database schema and static data? Please tell me it’s all in version control, not on a network share.

You should make all database changes by updating the files in version control and deploying them like you would your application code. You progress the change through environments to ensure it works, and you avoid embarrassment caused by the application failing because someone forgot to add the new column in production.

There are further choices to make, which we’ll cover next, but failing to version-control your database is unforgivable.

State-based vs migration-based approaches

On to the first choice for your database project. Do you make state-based or migration-based updates?

State-based schemas describe the desired state of the database. It will list each table with its columns, indexes, and relationships. You use a model-based tool to deploy the database, which compares your current state with the desired state and applies the changes for you.

Some state-based tools convert the differences into standard database scripts, like `ALTER TABLE…` scripts. Others perform migrations by creating a new table with the changes and moving the data into it. This is important if you’re using a technology like replication, which prevents the model migration mechanism from working.

The alternative to state-based database updates is migration-based updates, where you write your own `ALTER TABLE` scripts. You keep all your scripts in version control and use a tool that applies them in order and tracks when each was applied, preventing the same script from being applied twice to the same database.

The main difference between the two is procedural. You can code-review the migration scripts on demand, but you’ll need to review state-based migration scripts after your tools generate the implementation plan, which can make the review task more of a large batch.

Tooling and automation

Whichever approach you use, tooling helps it work. Many of our customers use Redgate tools as part of their deployment process to manage database schema changes, and there’s value in leaning on a tool written by folks who care deeply about the problem and how to solve it.

Crucially, you shouldn’t make any database changes outside of tool-based automation.

Test data management

Once you’ve automated your database schema and static data, it’s worth considering your test data. When automated or manual acceptance tests fail, it’s usually because someone unwittingly messed up the data. Prior test runs often leave data in an inconsistent state, especially if a test failure halts the run.

You can resolve this issue by automating your test data setup. Not only does this make it easy to reset the data during your build and test cycle, but it also lets you provide a self-service runbook for the test team to reset the data in their test environment whenever they need to.

There’s an up-front investment in this, but I can promise that it takes fewer hours than fixing your test data a few times.

Database refactoring patterns

The final thought to ponder concerns the steps you take in changing your database schema. When you’re in the habit of deploying your database and application in the same release process, you start to depend on this change coordination.

You delete a column from the database and immediately deploy the application, with all references to the deleted column removed. It looks like it works smoothly, but it’s a trap.

Imagine you had a critical bug in the application and had to redeploy the previous version. Now you can’t, because the previous version will try to read from a column that doesn’t exist. You no longer have a quick, easy back out plan, since you have to re-add the column, and you’ll also need data to put in it.

This approach also prevents you from making seamless deployments, as even if you progressively roll out the application version, the old version will error out due to the database change. You may deploy in the opposite order, application first, then database. You’ll discover the same problem with new columns that the latest version expects to find in the database.

You need to decouple database and application deployments, and there’s a whole book on the topic, called Refactoring Databases (Ambler, Sadalage). You can start by following the expand/contract pattern, which splits updates into steps. The principle is that you don’t delete a column until the production application has no reads or writes. You add a column and don’t reference it in your code until you deploy to production.

This means you can run the current version and the new version of the software against the same database, which means you can progressively roll out the new version and redeploy the prior version without touching the database.

Databases are only as hard as you make them

The database is high-risk, which is why updating it can be scary. I hope you’ve found this post full of practical advice for making database deployments robust and stress-free.

Database deployments, like application deployments, should be a happy time. You should be celebrating the new features and enhancements you’ve delivered to your users, not biting your nails and worrying that something’s about to go horribly wrong.

After tackling the question of which branching strategy is best, we turn our focus to architecture in this Continuous Delivery Office Hours episode. In particular, we discuss how big your building blocks should be and whether you should opt for a majestic monolith, granular microservices, or something in between.

Traditional monoliths are often described as a *big ball of mud* or *spaghetti code* due to the complicated mess of dependencies. When all the code is co-located, it’s common to find a tangled web of paths, with interfaces and facades skipped in favor of direct access to low-level functions and classes. This kind of code is hard to maintain.

Majestic monoliths are well-structured and loosely coupled, with carefully designed dependencies that are sometimes enforced by architectural tools. By making things modular, you get the benefits often associated with microservices, but without the difficulty of finding the code or the runtime complexity of distributed systems.

Microservices go a step further by physically separating parts of the system into independently deployable units. In addition to the architectural benefits, you can get faster build and test run times because you only build and test each service when it changes, which involves much less code. The trade-off is that your codebase becomes more complex, and you have to deal with versioning and issue pinpointing across all the deployed services.

So, how do you choose between these approaches?

Watch the episode

You can watch the episode below, or read on to find some of the key discussion points.

Architectural decision factors

Many factors influence the decision between monoliths, microservices, and hybrid approaches. One of the most crucial is how your teams are structured and how much autonomy they need. There are different ways to design [team structures](https://octopus.com/devops/culture/team-structures/), like Team Topologies, that make your teams and architecture part of the same conversation.

This is consistent with the *inverse Conway maneuver*, in which you structure your teams to match your target architecture. This works because Conway’s Law states that organizations design systems that mirror their communication structures. Working at the team level is more effective than creating an architectural diagram, as the structure will naturally yield the design you want.

Having a mismatch between the number of teams and the number of components can add complexity, either because too many teams are working on a single module, or you have created so many modules that there aren’t enough teams to own all the maintenance tasks associated with them. A sensible default is to match the number of teams to the number of components, ensuring sufficient autonomy for each team to work independently without overburdening them with excessive complexity.

The matching of service and team design sits between monolith and microservices, and might be described as “mescoservices” to highlight the balance they strike between the macro level of monoliths and the fine-grained level of microservices.

Another factor that can influence your decision is your deployment model. A managed software-as-a-service offering can use microservices without imposing complexity on its customers. In contrast, a self-hosted software offering may benefit from easier installation, troubleshooting, and upgrades if it uses a monolithic architecture.

Common microservice failure modes

There are several common failure modes to watch out for when moving to a microservice architecture.

Distributed monoliths look like microservices, except they have dependencies that mean you have to deploy services in a specific order. If you can’t deploy independently, you don’t have microservices. Similarly, you often find microservices at the application layer, but all sharing the same database. This prevents independent scaling of high-load services and can cause services to access data directly rather than through the appropriate service.

When you deploy large numbers of microservices, you may need to introduce rules about which services can communicate. Allowing any service to call any other service means managing a complex web of dependencies, and it starts to look a lot like the proverbial *big ball of mud*. This is often the result of splitting a monolith into microservices, as the old design tends to persist.

Microservices should bring a high level of team independence, allowing teams to make decisions about their own services, persistence, and deployment schedules.

Pulling microservices together

To make microservices successful, you need to establish data ownership and decide what level of data duplication is acceptable to support the business goals. For example, there’s a temptation to store a single address for a customer, but it may be a better idea to store the address immutably against events, like orders. Even if the customer moves house, the address for a past order remains the same.

Domain-driven design provides a method and a language for making decisions across service boundaries. A principal engineer or architect would usually set standards for service boundaries, contracts, and versioning to help keep services loosely coupled and independently deployable.

Fitness test

The best test of your service-based architecture is independent deployability. If each team can deploy its services without cross-team coordination, then your architecture is working. For majestic monoliths, the test is similar; teams should be able to work independently on modules they own. While deployments will be coupled through a single build and deployment pipeline, they should be able to make and commit changes independently.

Your branching strategy can support Continuous Delivery, or make it an impossible goal. You should assess the impact of how you branch on your ability to deliver software at all times, and you’ll find some branching techniques that work, while others that make software delivery more like walking in the dark through a field of rakes.

Continuous Integration is the practice of integrating code changes frequently, typically multiple times a day. This means you should check your code multiple times a day and keep your main branch deployable. Continuous Integration is a prerequisite for Continuous Delivery.

The name “Continuous Integration” provides solid hints about the crucial parts of the practice. Everyone should integrate their code into a shared branch (feature branches don’t count) and this should be done continuously, which means you’re doing it all the time; at least once a day, but ideally more often.

You’ll often speak to developers who want to stretch the definition of Continuous Integration, but you can’t escape those foundations. Merging the main branch into a long-lived branch feels like Continuous Integration, but you’ll notice no changes come back for ages until someone finally merges to main. Then you have to perform a large, complex merge, which you should avoid.

The DORA research on Continuous Delivery includes the capabilities of Continuous Integration and trunk-based development. The statistics suggest you can get similar benefits as long as you limit yourself to 3 (or fewer) short-lived branches (less than a day old).

Watch the episode

You can watch the episode below, or read on to find some of the key discussion points.

The worst branching strategy

Since the ability to branch code was invented, developers have applied a great deal of creativity to how to use branches. The most common approach is feature branching, where each feature gets a branch of main that evolves separately until the feature is complete, and it’s merged back into main. Release branching allows development to continue on main by taking a cut of the main branch that will be released. This allows hotfixes to be applied to the release branch without further destabilizing it.

The utility of branching strategies is often eroded by the coordination overhead of maintaining the separate branches and merging different changes back together. The more complex the branching strategy, the more likely it is that you’ll have merge conflicts and lost bug fixes. For example, you might fix a release branch and forget to merge the fix back to main, so the next release reintroduces the bug.

This is why the worst branching strategy is Gitflow. This is a complicated branching strategy that creates dedicated branches for features, releases, and hotfixes alongside permanent main and develop branches. The overhead of Gitflow vastly outweighs its benefits.

This is where trunk-based development shines, as it removes unnecessary complexity.

The best strategy is trunk-based development

Trunk-based development is the process of making all commits directly to the main branch. This is complemented by out-of-band reviews that don’t block merging and by feature toggles that decouple deployments from releases. It should be possible to deploy your software from the main branch at all times, even if features aren’t complete.

The DORA research allows up to 3 short-lived branches, which can be useful for teams working remotely (like open-source project teams), who can use branches and pull requests to coordinate their work. Even so, the goal is to keep branches short-lived and to merge frequently.

Trunk-based development is complemented by automated builds and checks that run when code is committed to the main branch. If a problem is found during these checks, the team should prioritize the fix over other development work.

Elite performance comes from small batches

Teams with the best software delivery performance work in small steps. Trunk-based development and Continuous Integration are crucial practices for controlling batch size. Problems are discovered sooner and are easier to fix when you only have a small amount of change to reason about.

Making frequent commits makes it easy to back out a bad change. If a test fails, you can discard changes since the last commit and try again, instead of trying to debug the problem. This is especially true when using AI coding assistants or other code generation techniques.

Ultimately, for trunk-based development to succeed without friction, the entire team must be aligned on the process. It doesn’t work if only some of the team are on board.

When you can’t deploy on demand, you’ve lost control of your software. Risk accumulates in unreleased code, and the more changes you store in one place, the more chance they have of triggering overheads, rework, and failures. When you have blockers stopping you from going live, you’ll start to accumulate dangerously high levels of risk unless you prioritize deployability.

By choosing to do work that returns software to a deployable state ahead of any other kinds of work, like feature development, you’ll avoid the toxicity of tangled work batches. Instead, you can smoothly flow changes between test and production and get the feedback you need to be confident that your software works.

Crucially, if you discover a high-severity bug or security problem, there are no roadblocks to getting fixes into production. You don’t need a special “expedited lane” for these changes, which means you don’t skip steps that let bad changes into your codebase.

Let’s take a look at problem indicators that will help you identify and fix common deployability issues. The goal of answering the deployability question is a mile marker, not the destination, but if you can’t answer the question, you’re going to waste time looking for landmarks!

Watch the episode

You can watch the episode below, or read on to find some of the key discussion points.

The awkward silence problem

When you ask your team, “Are we ready to deploy?”, the correct answer is either “absolutely” or “no way”. The worst possible answer is silence or confusion. If the team doesn’t know whether they can deploy, they’re missing the deployment pipeline that would generate the answer.

When you commit a code change, build errors should be returned to you within a couple of minutes. At the end of 5 minutes from your commit, a suite of fast-running tests should tell you if you’ve broken functionality or the quality attributes of your application. Dependency checks, code scanning, and other static analysis tools should have told you if you have a problem. If you have long-running tests or tests that depend on the new software version being deployed to a test environment, you should know in another 5-20 minutes if they’ve detected a problem.

After all these checks, you should have a version of your software running in a test environment that you have high confidence in. If someone asked you whether you’re ready to deploy, you’d answer “absolutely”.

Deployment automation makes sure the deployment is a non-event. It guarantees the same process is used to deploy to all environments, and makes it trivial to deploy on demand. A solid deployment pipeline contains all the checks you need to know whether you can deploy.

Manual testing is a ceiling, not a floor

Teams often treat manual testing as a foundation for verifying that a software version works. In reality, it acts more like a ceiling, limiting your ability to flow changes to production. As your software becomes more complex, the ceiling descends as the testing takes longer.

Long test cycles make you subvert your process to the speed at which you can test. You’ll notice when this happens because you’ll keep looping back to fix bugs and restart the test process. From the earliest change you make, through each bug list and all the re-testing, right through to the final software version, you are not deployable.

Automating your tests is the real foundation for your software. This raises the ceiling and moves the constraint away from the test cycle.

Your old code wasn’t written to be tested

If your application is successful, it will have some history. Part of that history is often that it wasn’t written with test automation in mind. That means you need to identify where your risk is, work out how to find the seams that will make it testable, and start adding characterizing tests.

Once some old code is wrapped with tests, it becomes far easier to change the code design, because the tests will fail if you break something.

Automation is living documentation

When developers move on, a portion of your institutional knowledge goes with them. High-quality documentation can help teams distribute this knowledge and reduce its loss, and the best kind of documentation is test automation.

Well-written automation, like tests, deployment automation, and infrastructure as code, performs useful functions while effortlessly documenting them. Because you make all changes through the living documentation, it is always up to date.

The hidden cost of undocumented knowledge becomes painfully clear when you have to deploy without the person who normally handles it. You follow their checklist carefully, confirm every step, and everything looks right; yet the deployment still fails. What you didn’t know is that the checklist stopped being accurate months ago, because the person doing the deployments stopped consulting it. All the new steps they introduced lived in their head, not on paper.

The living documentation built into automation tools is especially valuable when onboarding new developers. Rather than relying on tribal knowledge passed down through conversations and shadowing, a new team member can read the test suite and understand not just what the software does, but what it’s supposed to do and why certain behaviors matter. That’s documentation that keeps pace with the code because it is the code.

The value of long-term sustainability

Like its Agile predecessors, Continuous Delivery values long-term sustainability. That means you invest a little more effort up front to constrain maintenance costs over the long term. Writing tests may mean a feature takes 20-25% more time to implement, but the defect density can be 91% lower than similar features not guided by tests (Microsoft VS).

You could reduce 40 hours of bug fixing to just 3.6 hours by guiding feature development with tests, and you also save on other overheads caused by escaped bugs, like reputational damage, customer churn, support costs, pinpointing and debugging, test cycles, and feature delay.

Conclusion

While it takes some effort to set up a strong deployment pipeline, knowing whether a software version is deployable pays dividends. Technical practices like test-driven development and pair programming are needed to keep software economically viable in the long term, even though they require a little more effort up front.

If you can’t answer the question “Is your software deployable?”, you’re sure to run into trouble.

Originally published on octopus.com.

Continuous Delivery promotes low-risk releases, faster time-to-market, higher quality, lower costs, better products, and happier teams. Software is at the core of everything a business does today, so organizations must be able to respond to customer needs more quickly than ever.

Taking a quarter or a month to deliver new functionality puts companies behind their competition and prevents them from serving their customers. Few practices offer as much return on investment as Continuous Delivery, but many organizations continue to resist it, often making their deployment problems worse in the process.

Understanding why Continuous Delivery matters and how to implement it effectively can transform not only your deployment process but also your entire software development approach.

Watch the episode

You can watch the episode below, or read on to find some of the key discussion points.

At its core, Continuous Delivery means you can deploy your software at any time. A good indication of whether a team practices Continuous Delivery is whether they prioritize work that keeps software deployable. Other development styles usually continue working on features and return to deployability issues later.

That means teams must have fast, automated feedback for every change, highlighting when the software has an issue that would prevent its deployment. Deployments to all environments must be automated, with artifacts and deployment processes pinned to avoid unexpected changes between deployments.

The big three: Time, risk, and money

The longer the intervals between deployments, the more you accumulate risk, and the more you delay the value the changes will realize. If you wait six months between deployments, you’re more likely to get caught in a firefighting loop, spending more time pinpointing bug sources because of the volume of changes.

Crucially, until you place new features in users’ hands, you accumulate market risk that the changes won’t solve the underlying problem in a way users accept.

The deployment paradox

Human psychology works against us when deployments go wrong. Having waited six months to deploy, the pain of the firefighting stage and the increased risk of deploying large batches of changes mean people develop an aversion to deployments.

When a process is stressful and goes wrong, we naturally want to do it less often. You might think: “If we do fewer deployments, we’ll have less pain.” But this is precisely backwards.

Decreasing deployment frequency increases batch size, making the next deployment more likely to go wrong and cause pain. This is like avoiding the dentist after a painful checkup; the longer you leave it, the worse the next visit will be.

Risk-averse organizations have instincts that work against their goal of safety. The solution isn’t to deploy less often; it’s to deploy more frequently with smaller batches of changes.

Keeping software deployable during feature development

Another objection to Continuous Integration and Delivery is that features take time to build, so you can’t deploy while a feature is in flight. With an infinity of overlapping feature development, this would result in never deploying (or, more likely, work taking place in long-lived branches).

The solution is to separate deployments from feature release. Trunk-based development (integrating changes into the main branch every day, often many times each day) and feature toggles make it possible to work from a shared code base without making in-flight features visible to users.

There are many benefits to feature toggles beyond supporting Continuous Delivery. They also let you share features early with specific user segments or roll them out progressively rather than all at once.

Changing what deployment success means

When you separate deployment from release, you also transform how you measure deployment success. You’re no longer testing whether new functionality works during deployment. You’re only verifying that the application is running and healthy. This focus makes deployments faster and less stressful.

Feature toggles reduce the stress and burden of deployments because you’ll no longer miss deployment issues while checking functionality or miss functionality problems while monitoring deployments. Separating these concerns means each gets proper attention.

Solving dependency challenges

Feature toggles also address one of the most complex problems in microservices: deployment dependencies. Despite the promise of independently deployable services, teams often create elaborate deployment choreographies to ensure services are deployed in a specific order. Sometimes they give up entirely and deploy everything simultaneously. They accept unpredictable behavior during deployment or direct users to a holding page until it’s complete.

When deployments form a chain of dependencies, the architecture isn’t truly microservices but a distributed monolith. Real microservices should deploy independently. Feature toggles make this possible. Deploy all services when ready, then switch on functionality once dependencies are in place.

Conclusion

Continuous Delivery isn’t just about deploying more often. It’s about reducing risk through smaller changes, separating deployment from release, maintaining deployable code at all times, and giving teams the confidence to move quickly and safely.

The instinct to slow down after problems is natural, but it’s counterproductive. The path to safer deployments runs through more frequent deployments, not fewer. Organizations that embrace this counterintuitive truth gain a competitive advantage through faster feedback, lower risk, and ultimately, better software.