Docker Compose Modularity with “include” | Docker
The docker command line supports many flags to fine-tune your container, and it’s difficult to remember them all when replicating an environment. Doing so is even harder when your application is not a single container but a combination of many containers with various dependency relationships. Based on this, Docker Compose quickly became a popular tool because it lets users declare all the infrastructure details for a container-based application into a single YAML file using a simple syntax directly inspired by the docker run… command line.
Still, an issue persists for large applications using dozens, maybe hundreds, of containers, with ownership distributed across multiple teams. When using a monorepo, teams often have their own “local” Docker Compose file to run a subset of the application, but then they need to rely on other teams to provide a reference Compose file that defines the expected way to run their own subset.
This issue is not new and was debated in 2014 when Docker Compose was a fresh new project and issue numbers had only three digits. Now we’ve introduced the ability to “compose compose files” to address this need — even before this issue reaches its 10th anniversary!
In this article, we’ll show how the new include attribute, introduced in Docker Compose 2.20, makes Compose files more modular and reusable.
Extend a Compose file
Docker Compose lets you reuse an existing Compose file using the extends mechanism. This special configuration attribute lets you refer to another Compose file and select a service you want to also use in your own application, with the ability to override attributes for your own needs.
services:
database:
extends:
file: ../commons/compose.yaml
service: db
That’s a good solution as long as you only need a single service to be shared, and you know about its internal details so you know how to tweak configuration. But, it is not an acceptable solution when you want to reuse someone else’s configuration as a “black box” and don’t know about its own dependencies.
Merge Compose files
Another option is to merge a set of Compose files together. Docker Compose accepts a set of files and will merge and override the service definition to eventually create a composite Compose application model.
This approach has been utilized for years, but it comes with a specific challenge. Namely, Docker Compose supports relative paths for the many resources to be included in the application model, such as: build context for service images, location of file defining environment variables, path to a local directory used in a bind-mounted volume.
With such a constraint, code organization in a monorepo can become difficult, as a natural choice would be to have dedicated folders per team or component, but then the Compose files relative paths won’t be relevant.
Let’s play the role of the “database” team and define a Compose file for the service we are responsible for. Next, we build our own image from a Dockerfile and have a reference environment set as an env file:
services:
database:
builld: .
env-file:
- ./db.env
Now, let’s switch to another team and build a web application, which requires access to the database:
services:
webapp:
depends_on:
- database
Sounds good, until we try to combine those, running the following from the webapp directory: docker compose -f compose.yaml -f ../database/compose.yaml.
In doing so, the relative paths set by the second Compose file won’t get resolved as designed by the authors but from the local working directory. Thus, the resulting application won’t work as expected.
Reuse content from other teams
The include flag was introduced for this exact need. As the name suggests, this new top-level attribute will get a whole Compose file included in your own application model, just like you did a full copy/paste. The only difference is that it will manage relative path references so that the included Compose file will be parsed the way the author expects, running from its original location. This capability makes it way easier for you to reuse content from another team, without needing to know the exact details.
include:
../database/compose.yaml
services:
webapp:
depends_on:
- database
In this example, an infrastructure team has prepared a Compose file to manage a database service, maybe including some replicas, web UI to inspect data, isolated networks, volumes for data persistence, etc.
An application relying on this service doesn’t need to know about those infrastructure details and will consume the Compose file as a building block it can rely on. Thus, the infrastructure team can refactor its own database component to introduce additional services without the dependent teams being impacted.
This approach also comes with the bonus that the dependency on another Compose file is now explicit, and users don’t need to include additional flags on each Compose command they run. They can instead rely on the beloved docker compose up command without any additional knowledge of the application architecture.
Conclusion
With microservices and monorepo, it becomes common for an application to be split into dozens of services, and complexity is moved from code into infrastructure and configuration file. Docker Compose fits well with simple applications but is harder to use in such a context. At least it was, until now.
With include support, Docker Compose makes it easier to modularize such complex applications into sub-compose files. This capability allows application configuration to be simpler and more explicit. It also helps to reflect the engineering team responsible for the code in the config file organization. With each team able to reflect in configuration the way it depends on other’s work, there’s a natural approach to compose.yaml files organization.
Read more about this new include feature on the dedicated Compose specification page and experiment with it by upgrading Docker Compose to v2.20 or later.
