There’s a secret sauce for productivity: good technology.

More and more, I see people saying that tech doesn’t matter: everything is about culture, working together (DevOps, etc), and "product first" (in the sense of: don’t care about tech, just deliver)…​
Of course, all of this matters, and I even did several talks and articles about this topic over the years.
But if you have shitty tech (or shitty practices around tech), you’ll lower your productivity. Period.

I liked this talk from Nigel Kersten last year at Flowcon. Tech excellence will help you build better products and will make your customers (and your engineers) happy. Let’s, for once, consider tech as an asset and not just something that doesn’t matter and is interchangeable.
We’re in 2025, we have tons of cool and accessible technologies and practices to help us achieve our goals, let’s use them. And pushing technological boundaries is also a good way to change culture, by showing what’s possible!

I don’t like huge monoliths, and every time I encountered large codebases at work it was a mess and the main company pain point, slowing down everyone. I’m sure good-quality large monoliths application exists but never saw them.
I am also not a fan of microservice architectures, especially if badly designed. I even wrote an article (in French but I guess it can be easily translated) about the microservice topic one year ago.

Let’s be pragmatic. I really think that for most organizations there’s a sweet spot on the number of services per team or per developer, especially in large organizations. I don’t know what’s this number. One or two services per team (5-10 engineers handling one or several business domains) maybe? What I can say is that I don’t think we could deploy 200 times per day in production while having a monolith.

Smaller codebases mean smaller cognitive load when working on them, faster CI/CD pipelines (more about testing in the next section), less blast radius when things fail, and easier rollbacks.
Of code services oriented architectures have other challenges (the article I shared before explains a few of them, having strong tech foundations/standards is also important) but it’s a price I would pay because the benefits are huge. Again, I’m not pushing everyone to build crazy microservices architectures but there is a middle ground between "My 5 million LoC monolith that no one can run or launch tests locally" and "my 900 microservices distributed monolith".

Good test coverage for software is essential. Note that I didn’t write "100 % test coverage", it’s useless. The test should cover your software interfaces. Usually, your codebase has internal interfaces, with components containing business logic split per domain. Software also interacts with the outside world, through API for example: starting your software and then testing it as a blackbox also has value. This article is not about testing but there are a few things you should be careful about.

First, avoid slow tests. Slow tests are a productivity killer. It’s just so highly frustrating to wait for tests to execute for 30 minutes, locally or in your CI, especially if they fail at the end (please, avoid flaky tests). Tests should help developers to have a quick feedback loop and this is not possible if they are slow. Some stacks (like Golang) automatically cache test results and will only run tests that are impacted by code changes, which is nice for local development, but it’s still important to be able to run all tests quickly if needed (which will probably be the case on CI). if you have a huge monolith, you’ll have to modulize it to not run all tests for each commit.
Often, slow tests are a "boiling frog" syndrome: people react when it’s too late, and often the effort seems too big, don’t fall into this trap.

Another important thing is keeping your local (and so testing) environment simple. This point is a major problem that is impacting most companies, especially when they start to have several services on the backend side. I wrote an article about this topic two years ago.

I saw (and still see) so many CRAZY systems to run services and/or tests locally in my career. Tons of terrible bash scripts assembled, being forced to run tools like Kubernetes or complex networking stacks locally to run tests, services that can’t be tested without spinning all dependencies (services with which they interact), causing a giant mess. New joiners that can’t run the awful local dev environment after 3 days of trying. It’s ridiculous but not that uncommon. If you’re in this situation, fix it. Having a quick feedback loop is essential for productivity.

If you have a SOA/microservice architecture and can’t test one of the services in isolation without starting its dependencies, you lose. If you can’t just "git clone" a repository, maybe launch a docker compose file to spin up a database/message queue (for integration tests) and immediately run your tests with regular tools (go test, cargo test, mvn test…​), you lose. If you need to deploy complex infrastructure components to run tests, you lose.
The key to success is being able to work in isolation on a given service, without caring about the outside world (this is why we have API contracts). Bonus: if it’s simple locally, it will also be simple on your CI pipeline.

What about end-to-end tests, like tests based on emulating browsers (using Selenium for example)? By experience, they are always slow, flaky, hard, and very costly to maintain. It’s not something to recommend to have in your CI pipeline. But running a few of them constantly (canary testing) on production, on a few key features (like: login) is interesting.

One last thing about tests: avoid on-demand environments, especially for backend services. On-demand environments seem appealing but there false friends because there’s a convenient way to hide deeper issues that will surface in the long run.
I remember an informal discussion I had with a few SREs from one of the most valuated French scale up a few years ago about this topic. They started splitting the historical monolith into several services and asked me my thoughts (knowing that I already worked on this topic before) about how to provide full-featured on-demand environments for backend engineers, started from pull requests in order to manually validate the code before merging. My answer was: DON’T!
It’s a trap: why split a monolith if, in the end, your engineers can’t work without fully rebuilding it (with the accidental complexity it brings to the CI/infrastructure layer) every time they make a change on a given service?

Avoid like hell having to "wait" for a team to deliver changes on one service before being able to ship your part: again, you completely lose the advantage of services-oriented architectures if you have coupling between them when delivering features. I’ll explain later a few techniques to avoid this.

Again, working in isolation is the key to success.

We’re running all our production systems on Kubernetes, with ArgoCD as our GitOps engine. I started using Kubernetes in 2017, wasn’t totally convinced at the beginning but now I can say that it’s a game changer for infrastructure management and to build a reliable internal PaaS. And it’s also thanks to Kubernetes that we’re able to be so productive today (and that I can’t remember the last time I was paged while being on-call).

In any case, you’ll need something to handle stuff like application packaging, rollout (and connection draining), rollbacks, firewalling, service discovery, to detect issues on them (health checks). You will also need autoscaling, resources management, and scheduling capabilities. I could add a lot of stuff to this list.

Yes, you can do all of this without Kubernetes, I did it myself in the past. But in reality your homemade assembly of shell scripts/puppet modules/ansible roles/…​ will just be a much worse, non-standard, bug-ridden, hard-to-maintain version of Kubernetes. Just use Kubernetes, it’s ultra-reliable, even for critical services (we even automatically rebuild the whole infrastructure every 24 or 48 hours thanks to Karpenter running on our clusters for example :D).

As said before, we want to ship early and ship often. We don’t have releases, we just immediately deploy each commit pushed on the main branch. Each commit is a release. It works very well and I encourage you to do the same thing if possible.

Our current CI pipeline for all of our services (they all have continuous deployment enabled without exception, including critical ones) is, for each commit:

But what is a "successful deployment"? Usually, applications have several replicas. Kubernetes will progressively replace replicas running the old version with replicas running the new version. If a new replica fails to start, the rolling restart is aborted and old versions continue to run. It means that in our case production deployment will start only when all replicas are up and running in staging, "up and running" being in the Kubernetes world "the /healthz HTTP endpoint on services are returning 200".
Thanks to Kubernetes and ArgoCD, we can also easily roll back to a previous revision if something fails.

For years, we worked that way and it was enough.

One year ago we introduced Argo Rollout to perform canary deployment for highly critical services. Indeed, if a service successfully starts but then generates tons of errors 5 minutes later, it will not be detected by Kubernetes which will happily continue to roll out new replicas of the faulty services.

Argo rollout allows fine-grained deployment based on services metrics (we’re using Prometheus/Thanos), for example, "deploy 20 % of replicas with the new version, wait for 5 minutes, and periodically check the application HTTP error rate. If the error rate is below the configured threshold, deploy 70 % of the replicas with the new version for 10 minutes, and if it’s still OK deploy 100 %".
If the error rate increases too much, the application is automatically rollbacked to the previous version. Deployments take more time, but we gain confidence. For critical services, it’s worth it, and thanks to our service-oriented architecture we don’t have "CI contention" on them.

I talked at the beginning of the article about decoupling deploying in production and enabling a feature to end users and shipping unfinished work. It’s a super important topic. A new feature can take weeks to be delivered completely but you don’t want to wait weeks before being able to merge work, it’s better to ship continuous improvements every day (Friday included). And I said before, you DON’T want to have teams waiting for each other when shipping new stuff.
Same thing for some smaller changes like existing feature improvements (for example: a new option available on an API endpoint). You may want for example to enable the feature in your staging environment but not in production, or in production but only for some identified users or only a subset of customers…​

The obvious answer for this is using feature flags. Hide your new features or improvements behind a flag and enable it only on a subset of users or on specific environments. Most feature flag systems can also be used by non-tech people, which is nice. It looks easy on paper but feature flags can also become a huge mess:

I saw a lot of cases in my career where feature flags were misused and where they should have been replaced by business logic. One of the most common errors is confusing feature flags with permissions management. If you have a solid permission system (centralized permission store, with tooling and an admin interface, integrated into your services business logic), it’s way easier to use it to enable a feature to specific customers rather than using a feature flag. No conditions in the code to add or remove, just your permission layer dynamically handling everything for you.
I like this talk from Dorra Bartaguiz (in French) about this topic.

Another way to hide features on specific environments is to hide new API endpoints, for a new feature for example. It’s easily doable if you have an API gateway in front of your services for example. It’s also OK to deploy a new service in production but not expose it at all.

Again, a topic on which I already wrote and talked a lot and that would deserve another dedicated article. See for example those slides (in French again!) about this topic.

Your SRE/Platform engineer/sysadmin/whatever new trending team name/…​ team shouldn’t be your organization bottleneck. This means that other teams (product teams, data teams…​) should be able to manage their services in production by themselves, including (non-exhaustive list):

In short, the work doesn’t stop when the code is merged on the main branch.

We don’t ask developers to become cloud or Kubernetes experts. It’s our job to abstract the infrastructure complexity and to provide a simpler interface to end users. You don’t want developers to have to write 10 Kubernetes YAML files to deploy a single service for example, you want them to configure a few key settings (as said before: replicas, autoscaling, env vars/secrets, ports, rollout configuration…​) in a very simple way. Hide the accidental complexity and only expose the relevant settings.

But it’s still not enough. There are tons of actions your developers need to perform in production on their daily work (or during incidents). Here are a few real-world examples:

All of this should be done safely, with authentication, limited scope (you’re only allowed to perform specific actions), audit, and critical actions approvals. There’s no point in putting your developers on call if they can’t perform any action in production when being paged. And even outside of on-call hours, not having tooling immediately available for common situations and having to reach out to another team that may not answer immediately is an issue.

What worked well for us was building our own tooling. We have an API gateway plugged into our central authentication services (with 2FA etc) and we, SRE, built services to expose high-level features. We then provide an official CLI (installed and upgraded automatically in all computers) to interact with them. Here are some subcommands examples, all of them hitting homemade APIs:

We even have an approval system for those actions, where approval should be given by someone else (not necessarily an SRE, it may be someone from your team). You can see examples on these slides where screenshots are in English.

TL:DR about this topic: you should add self-service capabilities to your infrastructure. Just look at your support requests, take the most frequent ones, and build tools to allow teams to perform the actions without having to ask you. Repeat for a few years.
And you will see that it will be easier and easier to build new features with time because you’ll be able to reuse what was done for previous ones (api gateway, authentication, approval systems, tooling…​).

What about observability, one of my favorite topic? I could write pages about it (and this article is already way too long) but here are a few important points to keep in mind:

We should expect things to fail. Bad things will always happen and we’ll have production incidents no matter what we do. And it’s OK. It’s not that bad to have incidents from time to time if you’re able to limit the impact, quickly roll back, and if it doesn’t happen too often.

That’s also why I think SLOs are a really nice tool (see my article about it): they will help you measure the quality of your features on the long run, and so help you decide if you should invest more on not on reliability topics. Business metrics reflecting the user experience have a lot of value and are good candidates to be used for SLOs.

Design systems that tolerate failures. If after an incident you always need to manually perform actions in production to fix customers' issues, like reconciling data in some services or having to manually relaunch actions, it’s a sign of an issue with how you design systems.
Just make sure that next time a similar issue happens, what you had to do manually will be done automatically by the system. You should always ask yourself "What will happen if this component fails" when building features, and plan for automatic recovery.

The end.