Why Develop-Operate-Transition projects need to get the DevOps treatment

The DevOps movement has focused the IT industry on breaking down silos and increase collaboration across the different IT functions. There are popular commercial constructs that did a great job in the past but which are not appropriate for DevOps aligned delivery models. A while ago I talked about the focus on Average Daily Rate, in this post I want to discuss how to change the popular Develop-Operate-Transition (DOT) construct.

Let’s look at the intent behind the DOT construct. The profile of work usually changes over time for a specific system and idealised looks like this:

  • During Development the team needs to be large and deal with complex requirements, new problems need to be solved as the solution evolves
  • During Operate the changes are smaller, the application is relatively stable and changes are not very complex
  • At some stage the application stabilises and changes are rare and of low complexity
  • And then the lifecycle finishes when applications are being decommissioned (– and yes we are not really good at decommissioning applications, somehow we hang onto old systems for way too long. But for the sake of argument let’s assume we do decommission systems)

As an organisation it is quite common to respond to this with a rather obvious answer:

  • During development we engage a team of highly skilled IT workers who can deal with the complexity of building a new system from scratch and we will pay premium rates for this service
  • During Operate we are looking for a ‘commodity’ partner as the work is less complex now and cost-effective labour can be leveraged to reduce the cost profile
  • As the application further stabilises or usage reduces we prefer to take control of the system to use our in-house capabilities

So far so obvious.

If we look at this construct from a DevOps perspective it becomes clear that this construct is sub-optimal as we have two handover points and in the worst case these are between different organisations with different skills and culture. I have seen examples where applications stopped working once one vendor left the building because some intangible knowledge did not get transitioned to the new vendor. It is also understandable if the Develop vendor focuses on aspects that are required to deliver the initial version and less focused on how to keep it running and how to change it after go-live. While the operate vendor would care a lot about those aspects and rather compromise on scope. Now we could try to write really detailed contracts to prevent this from happening. I doubt that we can cover it completely in a contract or at least the contract would become way too extensive and complicated.

What is the alternative you ask? Let’s look at a slight variation:


Here the company is involved from the beginning and is building up some level of internal IP as the solution is being built out. In a time where IT is critical for business success I think it is important to build some level of internal IP about the systems you use to support your business. In this new type of arrangement in the beginning the partner is providing significant additional capabilities, yet the early involvement of both the company itself and the application outsourcing partner makes sure all things are considered and everyone is across the trade-offs that are being made during delivery of the initial project. Once the implementation is complete a part of the team continues on to operate the system in production and makes any necessary changes as required and any additional smaller enhancements. If and when the company chooses to take the application completely back in-house, it is possible to do so as the existing people can continue and the capability can be augmented in-house as required at this point. While there will still be some handover activities the continuous involvement of people makes the process a lot less risky and provides continuity across the different transitions.

Of course having a partner for both implementation and operating is a much better proposition as this will further reduce the fraction. I have now worked on a couple of deals like that and really like that model as it allows for long-term planning and partnership between the two organisations.

Most people I spoke find this quite an intuitive model, so hopefully we will see more of these engagements in the future.

How to Structure Agile Contracts with System Integrators

As you know I work for a Systems Integrator and spend a lot of my time responding to proposals for projects. I am also spending time as consultant with CIOs and IT leadership to help them define strategies and guide DevOps/Agile transformations. An important part is to define successful partnerships.  When you look around it is quite difficult to find guidance on how to structure the relationships between vendor and company better. In this post I want to provide three things to look out for when engaging a systems integrator or other IT delivery partner. Engagements should consider these elements to come to a mutually beneficial commercial construct.

Focus on Dayrates is dangerous

We all know that more automation is better, why is it then that many companies evaluate the ‘productivity’ of a vendor by their dayrates. Normally organisations are roughly organised in a pyramide shape (but the model will work for other structures as well).

It is quite easy to do the math when it comes to more automation. If we automate activities they are usually either low skilled or at least highly repeatable activities which are usually performed by people with lower costs to the company. If we automate more tasks that means our ‘pyramid’ becomes smaller at the bottom. What does this do to the average dayrate? Well of course it brings it up. The overall cost goes down but the average dayrate goes up.


You should therefore look for contracts that work on overall cost not dayrates. A drive for lower dayrates incentives manual activities rather than automation. Besides dayrates it is also beneficial to incentivise automation even further by sharing the upside of automation (e.g. gain sharing on savings from automation, so that the vendor makes automation investments by themselves)

Deliverables are overvalued

To this date many organisations structure contracts around deliverables. This is not in line with modern delivery. In Agile or iterative projects we are potentially never fully done with a deliverable and certainly shouldn’t encourage payments for things like design documents. We should focus on the functionality that is going live (and is documented) and should structure the release schedule so that frequent releases coincide with regular payments for the vendor. There are many ways to ‘measure’ functionality that goes live like story points, value points, function points etc. Each of them better than deliverable based payments.

Here is an example payment schedule:

  • We have 300 story points to be delivered in 3 iterations and 1 release to production. 1000$ total price
  • 10%/40%/30%/20% Payment schedule (First payment at kick-off, second one as stories are done in iterations, third one is once stories are releases to production, last payment after a short period of warranty)
  • 10% = 100% on Signing contract
  • Iteration 1 (50 pts done): 50/300 *0.4 * 1000 = 66$
  • Iteration 2 (100 pts done): 100/300 * 0.4 * 1000 = 133$
  • Iteration 3 (150 pts done): 150/300 * 0.4 * 1000 = 201$
  • Hardening & Go-live: 30% = 300$
  • Warranty complete: 20% = 200$

BlackBox development is a thing of the past

In the past it was a quality of a vendor to take care of things for you in more or less a “blackbox” model. That means you trusted them to use their methodology, their tools and their premises to deliver a solution for you. Nowadays understanding your systems and your business well is an important factor for remaining relevant in the market. Therefore you should ask your vendor to work closely with people in your company so that you can keep key intellectual property in house and bring the best from both parties together, your business knowledge and the knowledge of your application architecture with the delivery capabilities of the systems integrator. A strong partner will be able to help you deliver beyond your internal capability and should be able to do so in a transparent fashion. It will also reduce your risk of nasty surprises. And last but not least in Agile one of the most important things for the delivery team is to work closely with business. That is just not possible if vendor and company are not working together closely and transparently. A contract should reflect the commitment from both sides to work together as it relates to making people, technology and premises available to each other.

One caveat to this guidance is that for applications that are due for retirement you can opt for a more traditional contractual model, but for systems critical to your business you should be strategic about choosing your delivery partner in line with the above.

I already posted some related posts in the past, feel free to read further on:



Agile Reporting at the enterprise level (Part 2) – Measuring Productivity

Thoughts on State of DevOps report 2016

SOD2016And there it is – the most recent State of DevOps report, the 2016 version. If you read my previous blog posts for these kind of reports, you will expect a full summary. Sorry to disappoint. This time I focus on the things that stood out to me – we all know DevOps is a good thing and this report can give you ammunition if you need to convince someone else. But I don’t see the point of reiterating those. Let’s focus on the things that we didn’t know or that surprise us.

It is great to see that the report continues to highlight the importance of people and processes in addition to automation. That is very much in line with my practical experience at clients. High performance organisations have higher Employee Net Promoter Score (ENPS), which makes sense to me. I think there is an argument to be made that you can use ENPS to identify teams or projects with problems. I would love to test this hypothesis as alternative to self-assessments or other more involved tools that might cost more but might not be more accurate and are harder to deploy.

Another key finding is the impact it has when you build quality into your pipeline and don’t have it as a separate activity (e.g. no testing as a dedicated phase – I wrote about modern testing here) – but then the numbers didn’t really convince me on second look. It’s difficult to get this right and especially as the report has to work with guestimates from people at all levels of the organisation. But I agree with the sentiment behind this and there is anecdotal evidence that this holds true. I would love to have some more reliable data on this from real studies of work in organisations, it could be very powerful.

This year is the first time DevOpsSec is reflected in the report and the results are positive which is great. I have always argued that with the level of automation and logging in DevOps security should be a lot easier. The report has some very useful advice on how to integrate security all through the lifecycle on page 28.

We continue to see a good proportion of people coming from DevOps teams which is not surprising as that is the organisational form that most larger organisations choose for practical reasons (at least as a transition state) and flies in the face of a “DevOps team” is an anti-pattern. Glad to see the reality reflected.

On the results side the report uses some pretty impressive numbers on what high performers can do vs low performers. That’s great info, but I would like to see this compared between companies of similar size and complexity – otherwise we compare the proverbial DevOps unicorns with large enterprises and that is not really a fair comparison as the difference is not just in DevOps then. The more detailed data shows in my view the limitations of the comparison and some “kinks” in the data that are not easy to explain. I am glad they printed this data, as it shows that they researchers don’t massage data to fit their purpose which is good.

I really like how the researchers tried to find evidence for the positive benefits of trunk-based development, but I am not convinced this has been fully achieved yet. The same counts for visualising work – I see the point, but the report does not give me more reason and ammunition than I had before in my view.

Similar the ROI calculation is a good start, but nothing revolutionary. Its worth having a read of it but you will likely not find much new here – reduction in downtime, reduction in outage, increase in value through faster delivery.

Overall a good report, but not much revolutionary new. Great to see the trending over the years and that the data remains consistent. Looking forward to next year’s version. And yes I am writing this against the high expectations from previous year, it’s difficult to have revolutionary news each year…

How to Fail at Test Automation

(This post was first published on DevOps.com)

Let me start by admitting that I am not a test automation expert. I have done some work with test automation and have supervised teams who practiced it, but when it comes to the intricacies of it, I have to call a friend. It is from such friends that I have learned why so many test automation efforts fail. Talking to people about test automation validates my impression that this is the DevOps-related practice that people most often fail at.

Let me share the four reasons why test automation fails, in the hope that it will help you avoid these mistakes in your test automation efforts.

Before I go into the four reasons, allow me one more thought: Test automation is actually a bad choice of word. You are not automating testing, you automate quality assessments. What do I mean by that? It is a mistake to think of test automation as automating what you otherwise would do manually. You are finding ways to assess the quality of your product in automated ways and you will execute it way more often than you would do manual testing. This conceptual difference explains to a large degree the four reasons for test automation failure below.

Reason 1: Underestimating the Impact on Infrastructure and the Ecosystem

There is a physical limit of how much pressure a number of manual testers can put on your systems. Automation will put very different stress on your system. What you otherwise do once a week manually you might now do 100 times a day with automation. And into the mix an integrated environment, which means external systems need to respond that frequently, too. So you really have to consider two different aspects: Can your infrastructure in your environments support 100 times the volume it currently supports, and are your external systems set up to support this volume? Of course, you can always choose to reduce the stress on external systems by limiting the real-time interactions and stub out a certain percentage of transactions or use virtual services.

Reason 2: Underestimate the Data Hunger

Very often test automation is used in the same system where manual testing takes place. Test automation is data-hungry, as it needs data for each run of test execution and, remember, this is much more frequent than manual testing. This means you cannot easily refresh all test data whenever you want to run test automation and have to wait until manual testing reaches a logical refresh point. This obviously is not good enough; instead, you need to be able to run your test automation at any time. There are a few different strategies you can use (and you will likely use a combination):

  • Finish the test in the same state of data that you started with;
  • Create the data as part of the test execution;
  • Identify a partial set of data across all involved applications that you can safely replace each time; or
  • Leverage a large base of data sets to feed into your automation to last until the next logical refresh point.

Reason 3: Not Thinking About the System

Test automation often is an orchestration exercise as the overall business process in test flows across many different applications. If you require manual steps in multiple systems, then your automation will depend on orchestrating all those. By just building automation for one system you might get stuck if your test automation solution is not able to be orchestrated across different solutions. Also, some walled-garden test automation tools might not play well together, so think about your overall system of applications and the business processes first before heavily investing in one specific solution for one application.

Reason 4: Not Integrating it into the Software Development Life Cycle

Test automation is not a separate task; to be successful it needs to be part of your development efforts. From the people I have spoken to there is general agreement that a separate test automation team usually doesn’t work for several reasons:

  • They are “too far away” from the application teams to influence “ability to automate testing,” which you want to build into your architecture to be able to test the services below the user interface;
  • Tests often are not integrated in the continuous delivery pipeline, which means key performance constraints are not considered (tests should be really fast to run with each deployment);
  • Tests often are not executed often enough, which means they become more brittle and less reliable. Tests need to be treated at the same level as code and should be treated with the same severity. This is much easier when the team has to run them to claim success for any new feature and is much harder to do when it is a separate team who does the automation. It also will take much longer to understand where the problem lies.

Of course, absence of failure does not mean success. But at least I was able to share the common mistakes I have seen and, as they say, “Learning from others’ mistakes is cheaper.” Perhaps these thoughts can help you avoid some mistakes in your test automation journey. I do have some positive guidance on test automation, too, but will leave this for another post.

And in the case you found your own ways of failing, please share it in the comments to help others avoid those in the future. Failures are part of life and even more so part of DevOps life (trust me, I have some scars to show). We should learn to share those and not just the rosy “conference-ready” side of our stories.

Test automation is for me the practice that requires more attention and more focus. Between some open-source solutions and very expensive proprietary solutions, I am not convinced we in the IT industry have mastered it.

One bonus thought: If you cannot automate testing, automate the recording of your testing.

If you cannot automate testing, find a way to record the screen with each test by default. Once you identify a defect you can use the recording to provide much richer context and make it a lot easier to find the problem and solve it. Verbal descriptions of error are very often lacking and don’t provide all the context of what was done. I keep being surprised how long triage takes because of the lack of context and detail in the defect description. There is really no excuse for not doing this. Record first, discard if successful, attach it to the defect record if you find a problem.

Guide to the Guide to Continuous Delivery Vol 3

CD GuideI am not really objective when I say that I hope you have read the most recent Guide to Continuous Delivery Vol. 3 as I had the honor to contribute an article to it. My article is about mapping out a roadmap for your DevOps journey and I have an extended and updated blog article in draft on that topic that I will push out sometime soon. There is a lot of really good insight in this guide and for the ones with little time or who just prefer the “CliffsNotes”, I want to provide my personal highlights. I won’t go through every article but will cover many of them. Besides articles the guide provides a lot of information on tooling that can help in your DevOps journey.

Key Research Findings

The first article covers the CD survey that was put together for this guide. While less people said they use CD this might indicate more people understanding better what it takes to really do CD, I take this a positive indication for the community. Unsurprisingly Docker is very hot, but its clear that there is a long way to go to make it really work when you look at the survey results.

Five Steps to Automating Continuous Delivery Pipelines

Very decent guidance on how to create your CD pipeline, the two things that stood out for me are “Measure your pipeline”, which is absolutely critical to enable continuous improvement and potentially crucial for measuring the benefits for your CD business case. It also highlights that you sometimes need to include manual steps, which is where many tools fall down a bit. Gradually moving from manual to full automation by enabling a mix of automated and manual steps is a very good way to move forward.

Adding Architectural quality metrics to your cd pipeline

An interesting article on measuring more than just functional testing in your pipeline. It stresses the point to include performance and stress testing in the pipeline and that even without full scale in early environments you can get good insights from measuring performance in early environments and use the relative change to investigate areas of concern.
There is other information that can provide valuable insights into architectural weaknesses like # of calls to external systems, response time and size for those calls, number of exceptions and CPU time.

How to Define your DevOps roadmap – Well read the whole article😉

Four Keys to Successful Continuous Delivery

Three of the keys are quite common: Culture, Automation and Cloud. What I was happy to see what the point about open and extendable tools. Hopefully over time more vendors realise that this is the right way to go.

A scorecard for measuring ROI of Continuous Delivery Investment

An interesting short model for measuring ROI, it uses lots of research based numbers as inputs into the calculations. Could come in handy for some who want a high-level business case.

Continuous Delivery & Release Automation for Microservices

I really liked this article with quite a few handy tips for managing Microservices that match my ideas to a large degree. For example you should only get into Microservices if you already have decent CI and CD skills and capabilities. Microservices require more governance than traditional architectures as you will likely deal with more technology stacks, have to deal with additional testing complexity and require a different ops model. To deal with this you need to have a real-time view of status and dependencies of your Microservices. The article goes into quite some detail and provides a nice checklist.

Top CD resources

No surprise here to see the State of DevOps report, Phoenix Project and the Continuous Delivery book on this list.

Make sure to check out the DevOps Checklist on devopschecklist.com – there is lots of good questions on this checklist that can make you think about possible next steps for your own adoption.

Continuous Delivery for Containerized Applications

A lot of common ground get revisited in this article like the need for immutable Microservices/containers, Canary launches and A/B testing. What I found great about this article is the description of a useful tagging mechanism to govern your containers through the CD pipeline.

Securing a Continuous Delivery Pipeline

Some practical guidance on leveraging the power of CD pipelines to increase security, a topic that was just discussed at the DevOps Forum in Portland too and which means we should see some guidance coming out later in the year. The article highlights that tools alone will not solve all your problems but can provide real insights. When starting to use tools like SonarQube be aware that the initial information can be confusing and it will take a while to make sense of all the information. Using the tools right will allow you to free up time for more meaningful manual inspections where required.

Executive Insights on Continuous Delivery

Based on interviewing 24 executives this articles gathers their insights. Not surprisingly they mention that it is much easier to start in a green fields environment than in brown fields. Even though everyone agrees that tools have significantly improved, the state of CD tools is still not where people would like it to be and many organisations still have to create a lot of homemade automation. The “elephant in the room” that is raised at the end is that in general people rely on intuition still for the ROI of DevOps, there is no obvious recommendation for how to measure this scientifically.

Agile Governance – A short overview

I recently ran a seminar on Agile governance, so what does this even mean? Agile governance requires you to find the right balance between the complete chaos of no governance at all and the smothering of all Agile benefits in overbearing governance. In this article I want to provide an overview on Agile governance across a few levels: Transformation, Program, Project and team level. There is a lot more to be said than I can put into this post, so expect me to come back to this over time.

Agile Transformation governance

To borrow from Lord of the Rings “One does not simply transform an organisation to Agile”. At the transformation level it is important to provide support for teams as they encounter challenges for adopting Agile. Some problems can proactively be addressed like training, coaching and enable good engineering/DevOps practices. Others will be uncovered as you start down the adoption path. Those blockers can be identified from the retrospectives at the team level as well as by speaking to Iteration managers and coaches. These blockers should be prioritised and provide the transformation backlog to address at organisational level. I find the below transformation governance structure very helpful as a starting point:

agile governance


Agile governance at the program/project level

I have shared my view on how to manage at the program level before here. Just one additional word of warning: Do not try to compare teams against each other for performance measurements, even under the guise of gamification this really can only lead you to bad development patterns. There are teams where this works and where their leaderships does not use the information for the wrong reasons, but the risks are higher than the benefits. Defining the success criteria for each team individually is a much better way to deal with this.

Agile governance at the team level

At the team level the SCRUM methodology provides a set of useful governance elements which you should leverage, adopt and adapt:

  • Sprint Reviews
  • Sprint Planning
  • Retrospectives
  • Team based estimations

Additionally you can use Agile maturity models and Agile metrics for teams to evaluate themselves and their progress in maturing their Agile practices.  Good coaches can help teams in this evaluation without being judgmental.

A comment on measuring individual productivity:
How do you measure the productivity of team members in your Agile teams? The short answer is YOU DON’T. I have talked about productivity in this post and if you don’t believe me perhaps Dilbert can help you understand how poisonous this idea is for your teams:



I am looking forward to hear from the community what you use to govern your Agile adoption across the different levels.

Is there such a thing as Hybrid Agile?

I recently wrote an article about Hybrid Agile for InfoQ  because the term has been misused too often. Feel free to read the whole article. Here is my conclusion from the article:

After many discussions, people convinced me that “Hybrid-Agile” is what is otherwise called Water-Scrum-Fall and after some deliberation I admit that this makes sense to me. Of course the term Water-Scrum-Fall or similar phrases are often used with contempt or smiled upon, but when you look closer, this is reality in many places and for good reasons.