Skip header to navigation
For a long time, across different projects throughout my career, I’ve seen database migrations happen during application startup. These migrations usually run as part of a post-deployment hook, just before the new deployment receives any production traffic. But… what’s happening in this short (or sometimes longer) timeframe after the migration is done and before the new app boots, while the old app is still active?
I had so many questions, but the root one was: “do I have to worry about this at all?” The answer was yes. Because I saw how issues and incidents happened where those questions suddenly became reality.
And over time, I built up an understanding and realized the answers to those questions—but I also realized how this seemingly dangerous process can actually be pretty chill. So let’s do a quick dive into when (and how) to seamlessly migrate your database.
A popular example is how to rename a database column. Let’s take a look into what will happen if you just rename a column in a single migration with a simple ALTER TABLE RENAME COLUMN SQL statement:
Of course, it doesn’t always have to result in an incident. But there’s a short amount of time where users will see errors, and it’s definitely a more risky situation. So what can we do instead?
To prevent all those issues, we have to divide our migrations into multiple backwards compatible changes. Let’s continue with the example of renaming a column, and how we would divide this into multiple steps:
Because we’re doing a bunch of backwards compatible changes, if we fuck up at any stage, we can try again and no one will notice. And we can also do instant rollbacks to a previous version in an emergency. The longer we take to do the last step, the more room we give ourselves to roll back.
Now that we know how and why we safely migrate our database, when and where should we perform these steps?
You can do all of these steps in an async manner. And that’s what I think is the beauty of it. Those steps don’t have to happen as part of your automated deployment pipeline, and you don’t even have to stress about getting it done quickly during a single workday. You can take your time!
In this setup, you wouldn’t even need to migrate the schema during a deployment. It can totally happen during normal operations. And if you don’t want to block the deployment pipeline with a long-running migration, and potentially block any urgent fixes, it maybe shouldn’t happen as part of a regular deployment.
But it’s also a good practice to not raw-dog into your production database from your local machine and execute any SQL.
Prepare your steps as separate scripts, and let your colleagues review them. This isn’t about code style or maintainability—in the end, they’re just one-off scripts. But your colleagues may see some typos, or issues where you’ll accidentally cause high database load.
And then have a way to execute these scripts inside your production infrastructure. For example, a Kubernetes Job which can connect to your database, or any equivalent service in your infrastructure that can connect to your database but is not your production service which handles real traffic. Then you don’t have to directly connect to your database from your local machine or from a deployment machine like GitHub Actions.
I’m working a lot with dockerized TypeScript apps. The final docker image only contains bundled code, and dependencies that are referenced in the bundled code. But migrations aren’t part of the bundle—because they aren’t part of my running app. And a CLI to run migrations, like , also isn’t part of the final bundle!
Knowing that I want to run migrations async in my production environment, I was wondering for some time how the actual fuck I’m supposed to run them within my production docker image—because it doesn’t contain my migrations and the drizzle CLI. I can’t just start that image in a one-off job and execute my tasks. So, should I install in my production image? Copy all migrations? But that would increase the size of my docker image, and I want to keep it small.
The solution to this is quite simple: just have a separate image. You could, for example, execute a Dockerfile as a one-off script:
That’s what I’m doing, with a small tweak that I’m inlining this Dockerfile inside a file, as a separate service called “migrate”. I can just start that service on production, it builds itself, runs the migrations, and then exits.
Or you could even not care about migration files at all, use the schema in your code as the source of truth, and use something like :
As with everything, there are trade-offs, and you don’t need to do this strict approach all the time. Some projects simply don’t have those requirements. And in some projects maybe the few seconds of downtime are okay.
Do you need to do all of this in your side project? Probably not.
Would it be okay if your production docker image is a few MB larger and you execute the migrations, including data migrations, during app boot? Could be okay.
But will you still learn something from practicing it? Absolutely.
For a long time, across different projects throughout my career, I’ve seen database migrations happen during application startup. These migrations usually run as part of a post-deployment hook, just before the new deployment receives any production traffic. But… what’s happening in this short (or sometimes longer) timeframe after the migration is done and before the new app boots, while the old app is still active?
I had so many questions, but the root one was: “do I have to worry about this at all?” The answer was yes. Because I saw how issues and incidents happened where those questions suddenly became reality.
And over time, I built up an understanding and realized the answers to those questions—but I also realized how this seemingly dangerous process can actually be pretty chill. So let’s do a quick dive into when (and how) to seamlessly migrate your database.
A popular example is how to rename a database column. Let’s take a look into what will happen if you just rename a column in a single migration with a simple ALTER TABLE RENAME COLUMN SQL statement:
Of course, it doesn’t always have to result in an incident. But there’s a short amount of time where users will see errors, and it’s definitely a more risky situation. So what can we do instead?
To prevent all those issues, we have to divide our migrations into multiple backwards compatible changes. Let’s continue with the example of renaming a column, and how we would divide this into multiple steps:
Because we’re doing a bunch of backwards compatible changes, if we fuck up at any stage, we can try again and no one will notice. And we can also do instant rollbacks to a previous version in an emergency. The longer we take to do the last step, the more room we give ourselves to roll back.
Now that we know how and why we safely migrate our database, when and where should we perform these steps?
You can do all of these steps in an async manner. And that’s what I think is the beauty of it. Those steps don’t have to happen as part of your automated deployment pipeline, and you don’t even have to stress about getting it done quickly during a single workday. You can take your time!
In this setup, you wouldn’t even need to migrate the schema during a deployment. It can totally happen during normal operations. And if you don’t want to block the deployment pipeline with a long-running migration, and potentially block any urgent fixes, it maybe shouldn’t happen as part of a regular deployment.
But it’s also a good practice to not raw-dog into your production database from your local machine and execute any SQL.
Prepare your steps as separate scripts, and let your colleagues review them. This isn’t about code style or maintainability—in the end, they’re just one-off scripts. But your colleagues may see some typos, or issues where you’ll accidentally cause high database load.
And then have a way to execute these scripts inside your production infrastructure. For example, a Kubernetes Job which can connect to your database, or any equivalent service in your infrastructure that can connect to your database but is not your production service which handles real traffic. Then you don’t have to directly connect to your database from your local machine or from a deployment machine like GitHub Actions.
I’m working a lot with dockerized TypeScript apps. The final docker image only contains bundled code, and dependencies that are referenced in the bundled code. But migrations aren’t part of the bundle—because they aren’t part of my running app. And a CLI to run migrations, like , also isn’t part of the final bundle!
Knowing that I want to run migrations async in my production environment, I was wondering for some time how the actual fuck I’m supposed to run them within my production docker image—because it doesn’t contain my migrations and the drizzle CLI. I can’t just start that image in a one-off job and execute my tasks. So, should I install in my production image? Copy all migrations? But that would increase the size of my docker image, and I want to keep it small.
The solution to this is quite simple: just have a separate image. You could, for example, execute a Dockerfile as a one-off script:
That’s what I’m doing, with a small tweak that I’m inlining this Dockerfile inside a file, as a separate service called “migrate”. I can just start that service on production, it builds itself, runs the migrations, and then exits.
Or you could even not care about migration files at all, use the schema in your code as the source of truth, and use something like :
As with everything, there are trade-offs, and you don’t need to do this strict approach all the time. Some projects simply don’t have those requirements. And in some projects maybe the few seconds of downtime are okay.
Do you need to do all of this in your side project? Probably not.
Would it be okay if your production docker image is a few MB larger and you execute the migrations, including data migrations, during app boot? Could be okay.
But will you still learn something from practicing it? Absolutely.
drizzle-kitdrizzle-kitdocker-composedrizzle-kit pushFROM node:22-slim
WORKDIR /app
COPY package.json pnpm-lock.yaml
RUN pnpm install --frozen-lockfile
COPY ./migrations ./migrations
CMD ["npx", "drizzle-kit", "migrate"]
drizzle-kitdrizzle-kitdocker-composedrizzle-kit pushFROM node:22-slim
WORKDIR /app
COPY package.json pnpm-lock.yaml
RUN pnpm install --frozen-lockfile
COPY ./migrations ./migrations
CMD ["npx", "drizzle-kit", "migrate"]
FROM node:22-slim
WORKDIR /app
COPY package.json pnpm-lock.yaml
RUN pnpm install --frozen-lockfile
COPY . .
CMD ["npx", "drizzle-kit", "push", "--force"]
FROM node:22-slim
WORKDIR /app
COPY package.json pnpm-lock.yaml
RUN pnpm install --frozen-lockfile
COPY . .
CMD ["npx", "drizzle-kit", "push", "--force"]