I’ve long been a supporter of agile methodologies such as scrum for software development teams and when you’re working on large monolithic applications they make sense. All scrum teams working in synchronised sprints and producing releases of working and tested software regularly is a big step forward from the “bad old days” of vaguely “waterfall” methods which produced annual releases of dodgy software – the result of integration pain from teams that had worked in isolation for too long and requirements processes that ran for months if not quarters of disagreeable meetings.
Moving to agile/scrum isn’t an easy transition for most businesses. It initially seems like a change in R&D process and then rapidly the realisation occurs that it impacts product management, and operations and other business functions as well. Large areas of the company’s processes and roles are required to be adapted. But it’s worth it. When it operates correctly it brings huge benefits. It reduces the latency between requirement and deliverable, it ensures that deliveries are on time (albeit potentially short of a few features that the sales team wanted) and it keeps the various software teams more tightly synchronised.
It isn’t, however, all roses in the garden of scrum. Many teams find that their ability to complete their test suites is the limiting factor to how short their sprint cycles can be. What could be a two week sprint has to be 4 weeks due to the volume of tests. This leads to increased focus on test automation. And even the 4 week sprints don’t actually result in a delivery to customers because the customer support teams can’t get out to all the customer sites to do upgrades that often, and even if they could the customers don’t want them continuously upgrading their installs! The increased testing that results produces more test fails, which pushes workload back on the scrum teams. And because of the test times a decision has been made to overlap test and development sprints – when the testers are working on sprint ‘x’ the scrum teams are already working on sprint ‘x+1’. Hence, when issue arises in test the engineer responsible is pulled back in time to the previous sprint to resolve the issue.
I’ve always maintained that the target you should be aiming for is for the sprint time to be faster than the sales cycle delivery cycle. This means that your sales team can say “yes” to the customer on a feature and the feature will be in the product when it is delivered. Unfortunately if you can’t meet this target, and the deals really are important to you, you’re going to be breaking the rules of scrum and accepting new requirements in the middle of your sprints!
This isn’t criticism of scrum per-se. It’s just how life is in the real world. But can it be different? What is the next step?
Cloud Based Microservices, Containers and Automation
The deployment and operational environment for many of our solutions has changed rapidly and considerably over the last few years. Actually perhaps that’s not true for everyone, but the deployment and operational environment that is POSSIBLE has certainly changed. Cloud based deployments, microservices, containers, container orchestration tools and continuous integration frameworks can allow us to dramatically change how we move from requirement, through development and into operational deployment.
Breaking applications into micro services, when done right, can allow us maintain clear separations between subsystems, deploy them independently and integrate them through clearly defined interfaces. Loose coupling is key here, and the discipline to achieve this is much more readily maintained in a microservices type environment.
Building executables inside containers allows us maintain consistent build environments and continuous integration frameworks alongside container orchestration tools now allow us to automate entire release, test and deploy processes directly from an engineer’s check in on a source code repository. With a single command the source for a microservice can be compiled, unit tested, uploaded to an artefact repository, deployed into a staging environment for further tests then advanced to a live deployment for canary or red/black testing before being rolled out to the entire user base. There are many moving parts and technology choices in this but it is now truly amazing the level of automation and speed that can be achieved, and is being achieved in some companies.
My question then is this – if we can bring levels of automation and test to this point do we need sprints and release cycles anymore? Surely we can move to the point were small independent teams of one or more engineers can pull a requirement off a Kanban, implement the feature and release it – doing the work as fast as possible and getting the feature to the customers with minimal latency. In this environment many of the ceremonies and processes around scrum appear to be superfluous, and without them development is faster and more lean. It’s better for the business and without doubt more empowering and satisfying for the design teams.
Of course it has to be recognised that not everyone is ready for this yet. Not everyone is deploying into these types of environments. Not everyone is building a cloud native application from the ground up. Many will first have the challenge of migrating from their current deployment models and refactoring applications into microservices. Many have to work out their business models for cloud base SAAS and take their customers on the journey with them. Hopefully though that vision of what can be, and the understanding of the business benefits it can deliver, is motivation for some to start on the journey and pursue it with vigour!