A few weeks ago I was in group of people and someone made a statement that application management (AM) is commodity and most people in the group agreed. I didn’t say anything but I think application management is one of the most exciting places to be today. There are so many technology trends that apply to AM: Agile, DevOps, Artificial intelligence, etc.

When I say application management I mean some level of operations (e.g. level 3 support and code fixes) and some level of application development/maintenance (smaller changes that don’t require projects). The nature of those activities are that they are frequent and smaller, which means we have a lot of opportunity to learn and improve. It’s the ideal testbed to showcase how successful modern engineering practices can be assuming the volume of work is large enough to justify investment.

Let’s look at two dimensions of this: the code changes and the platform services

The product team making the actual code changes should have responsibilities for making changes for both new scope and production defects. This team needs to reserve some capacity for urgent production defects, but the bulk of production defects get prioritised against new scope and are just forming part of normal delivery. This is possible because the release frequency is high (at least monthly) and the team can leverage platform services like CI/CD, feature flags, automated environment provisioning and monitoring services. The team collaborates with the platform services team in many ways and focuses on the application they own. Both Kanban and Scrum are possible methodologies this team can use, making this a great Agile working space.

The platform team provides the services which make life easier for the product teams. They also provide the first levels of support in a centralised fashion to protect the product team from too many distractions. There are two goals for this team: to provide platform services to the product team (like CI/CD, environment provisioning, monitoring) and to keep improving the customer experience for end-users. Fast ticket resolution is not the right measure for this aspect, but rather how to reduce the tickets overall. Advanced monitoring of performance, functionality and infrastructure is being used to find problems before the customer finds it. Each user created ticket is an opportunity to improve the monitoring for next time. Chatbots and self service are used to provide the fastest service for standard requests and AI/ML is learning from tickets to make more and more requests become standard and to route tickets to the best team/person. Analytics on all aspects of the platform allows to find patterns and reconcile them against issues. The platform team collaborates with the product team to make sure the application related services keep being improved. Over time this team becomes a small team of automation experts rather than a large team of operators for manual tickets.

Seriously if you look at the above, how are you not super excited about this space?If you do this right you get to use a lot of super modern engineering practices to improve the delivery capability of your organisation. The amount of money/capacity that frees up once you have implemented the first improvements can then be used to make more and more meaningful changes to the products and services. You can do this with a focus on the creative tasks rather than the fire fighting and repetitive manual work that consumed application management teams in the past.

I am working with a couple of organisations now that are as excited as I am about this space and have a similar vision. I am very much looking forward to see what results we will achieve together.

A few weeks ago I published a blog post on simple DevOps questions you can ask your team to find out how mature your adoption of DevOps practices are (you find it here). There is a case to be made that you should have a similar question for your Agile adoption. The problem here is that there are so many different flavors of Agile and that not one is better than the others (no matter what all those zealots out there are saying). The large amount of variations makes it hard to analyse the maturity of Agile unless you understand the context of the organisation and can evaluate the delivery success of the team. But I found a question that helps me differentiate between the “false” or “hybrid” Agile flavors and “real” Agile. I call it my minimum definition of Agile for it to be something I can engage with and think that it can be successful. Here we go, Mirco’s minimum Agile definition:

“One Team, produces a working and tested version of software within each strict timebox of equal length.”

You might think this is a really low bar for an Agile definition, but it rules out a lot of the work that I see being done under the name “Agile”. It rules out all of the following patterns that are suboptimal if not dangerously risky for delivery success:

• Separate design, build and/or test teams – often this is a remnant of existing vendor relationships or strong “fiefdoms” and the Agile change has not been able to break down the barrier. For me this is an absolute showstopper for Agile success. Pause and see how you can enable a joint team.
• Design sprints, followed by build sprints, followed by test sprints – Well let’s call this what it is, Waterfall delivery 😉
• Teams being changed and swapped – Many of the Agile benefits only manifest once the team has been working together for a while, if you need to keep changing people or ramp-up or down then there is something wrong with the work preparation part
• Agile teams building lots of stuff but are unable to validate it end-to-end – This is perhaps an already more advanced problem but I keep seeing it a lot these days. Agile teams deliver software but cannot test it sufficiently and while the velocity is high, once the product gets end-to-end tested it falls apart. There are three root causes that I commonly see: unavailability of test environments, lack of maturity in DevOps practices and lack of up-front-planning of end-to-end scenarios.
• Sprints that have different lengths or are changed at last-minute – Agile is more successful because it is more rigorous and forces you to address problems early and often. If you weaken the conditions you are stepping on a slippery slope that will likely impact you more and more. Step off the slope and fix the problems rather than changing the sprint timebox

I have been happy with this definition for now as it weeds out the most common bad patterns, perhaps it helps you too. Let me know if you have your own question or definition that you use.

DevOps and Agile transformations continue to be challenging for organisations due to the complexity of the problem. It seems that most people agree on what good looks like but struggle to find their way there. Successful organisations like Netflix or Spotify provide examples but their journey cannot be simply copied and organisations which try encounter a lot of challenges as their context organisationally and technology-wise is different. The transformation is different for every organisation, but some patterns are common, so what can one do.

In my book “DevOps for the Modern Enterprise” I tried to write down my experience from working with many large organisations. While there is no one solution, certain patterns that prove successful can be seen and more importantly a certain way of thinking has been common for organisations who made progress. And of course I have been part of many things that did not work out and can share those experiences too to help others avoid costly mistakes. No need to pay for those lessons yourself.

Three themes run throughout the book:

• The transition from managing IT work similar to manufacturing work towards managing a creative process enabled by knowledge workers
• Dan Pink’s three ways to motivate knowledge workers: Autonomy, Mastery & Purpose
• The power of small batches – why it is important to work in small batches and how to enable them

These three themes run like threads throughout all parts of the book. I was motivated by the fact that as a developer myself, I found it very frustrating how for a period of time over-industrialisation created a view of IT work as just a production process that should be optimised for cost and productivity. Something that simply is not appropriate in this day and age.

The book is structured in three dimensions of change which need to be addressed for a transformation to be successful:

• The ecosystem of the organisations including vendors and software partners
• The People Dimension and the organisational structure
• The Technology Architecture of practices and technologies to enable engineering

And in the center of it all is a rigorous continuous improvement process that leverages the scientific method to plan and evaluate the experiments that make up the transformation over time.

Let me give a little summary of each section:

Creating the right ecosystem

The ecosystem is the environment in which the transformation needs to take hold to make real change. This part of the book looks at how to setup the transformation for success and how to identify the right applications in a legacy environment that should be tackled first. It also explores the roles of technology vendors and systems integrators and helps organisations choose the right ones that can help during the transformation.

The People and Organisational Dimension

Nothing happens without people in technology, so how can we structure our processes and our organisation in a way that maximises Autonomy, Purpose and Mastery for our knowledge workers?

I explore how Agile helps to provide all three things to our people, how some organisational structures have shown to be good stepping stones for success and how the organisation has to move from testing to quality engineering.  I also provide some ideas on how to focus more on the people in the team when managing teams based on my own experience of managing teams.

Technology and Architecture Aspects

The last part of the book is the most obvious one that many expect from a DevOps book. What technologies and technical practices should you use and how can you be successful with them. I revisit Continuous Delivery and more advanced delivery patterns. I take a long hard look at the DevOps tools out there and how you can choose the most suitable ones for your organisation while avoiding any specific vendor recommendations as they would age way too quickly. Of course architecture evolution towards Microservices and leveraging the cloud successfully are covered as well.

Organisations who consider all three dimensions are stacking their chances for success. Each chapter of the book comes with practical exercises that I run with clients and that the reader can run for her/himself in her or his organisation. I wanted to make sure that the book is not just for reading but also for taking action and taking the next step of the journey.

DevOps for the Modern Enterprise is widely available from 4 April 2018:

Amazon: Link
iTunes: Link
Bookdepository: Link

I like it simple. And I have a somewhat shaky relationship with maturity models as might have gathered from some of my other posts. I see the power of using a maturity model or assessment to help people understand where they are, but it’s also so prone to Dunning-Kruger (Link). So what else can we do to understand our maturity in IT delivery as it relates to DevOps.

I am running executive training that helps program leads and delivery leads understand how to run programs that use Agile and DevOps methods. As a lead they are unlikely to run stand-ups or configure Jenkins but need to understand enough to support their teams and help stay the course. Hence we created a training that focuses on principles and how things look like from the executive perspective. We call it Modern Engineering Bootcamp. I cheekily usually say in the introduction that the aim is to make executives dangerous for their teams, because they will be able to call things out that teams don’t do well. They will be able to call BS when their teams believe they are doing well but really are not. We had this moment when a participant during the discussion on configuration management spoke up: “Until just then I believed we had our code in configuration management and only after asking a few more questions inspired by this course have I just found out that our Pega configuration is only stored in Pega not in a proper configuration management system. We will change this ASAP to improve our DevOps setup and to be able to automate further”.

This got me thinking, can I come up with a small set of self assessment questions that helps people make the same kind of discoveries where they need to.

Thinking about it for a while, I went through my past projects and realised that I asked my teams certain questions and asked them to do certain things when I got a little bit worried we are missing something, it’s time to share these questions with you (and some of the stories behind it):

Let’s blow away the environment and rebuild it from our stored configuration

When I was still a young developer I was working on a mainframe project. We had a development environment that we used to check our code before deploying to the mainframe. I noticed that the environment was cluttered up with lots of things that seemed to be leftover from earlier iterations and decided to blow the environment away and rebuild it from what we had stored in configuration. We might call this now “infrastructure as code”. It turned out that when I tried to rebuild it, that not only some files were missing (they were not in SCM as they were never migrated when moving to a newer SCM) but also the mainframe CICS configuration was not correct. We spent 2 weeks fixing this and while my boss was initially not happy that I had caused two weeks of extra work, we were lucky to find this in development and not at some later stage in production. Some of the missing files only existed in live environments and not in any SCM system…phew… By the way nowadays this question usually means lets rebuild a cloud environment from scratch, which should cause even less headaches.

When you ask your team to do this you are really looking for two things:

1. Does your team look worried or push back against this request? Explore what the concerns are, it’s likely they know where the skeletons are in the closet.
2. If they are confident and kick this off, measure how long it takes and whether you are happy with the timing.

Keep repeating this regularly to improve speed and reliability.

Let’s delete the application config and redeploy it

In my second project we ran some pretty fancy Control-M schedules to deploy applications and update configuration as required. Jenkins was still to be invented I think (or I hadn’t heard of it at least). Because we ran two releases in parallel and had lots of queues and databases to connect, we used a massive excel sheet to store our environment configuration. By the time I joined the project the project was already under way for a long time. I was worried that some configuration was missed in the excel sheet and decided to build a new environment and deploy the configuration only from the excel sheet. No surprise I found a handful of configurations that were not stored. Luckily I had learned from my previous project not to blow away an environment in use but rather build a new one in parallel to find these problems.

You are using this question the same way as the first one. See whether your team is comfortable and regularly run the exercise.

Let’s redeploy the application even though nothing has changed

So your team told you the application deployment is automated and reliable? Great, ask them to schedule a deployment of the latest package every morning into your dev or test environment before the team comes to work. Given they are deploying the same thing there is very little risk and it’s automated so no one needs to be around for it. On my third larger project this is what I did. And when the team pushed back initially I asked for it anyway. And guess what… for the first few weeks someone had to come to office early as there was always a small hiccup or another until it was fully smooth. The only thing that proves automation and keeps it clean is running it outside of business hours when no one touches the system and doing it very frequently even (and especially) when no changes to the application are required. It keeps proving to everyone that the deployments are reliable and that if after deploying a new version of the software problems occur, they are likely to be found in the code not in the deployment mechanism. Running it every day also allows you to measure performance and keep improving it. Those of you deploying multiple times a day, well done yours must be reliable already to do it so often, for everyone else: Start doing it frequently to become reliable and take the worry away.

Let’s rerun the test suite and then again

In one of my later projects we were struggling to make test automation work. We had a few goes at getting a test automation framework going. At some stage the test team celebrated that they had a regression suite that works. After a successful run I asked them to quickly rerun it and it turned out that we couldn’t as the data had to be completely refreshed and would have blow away the progress made from manual testing that was running at the same time. We had some more work to do. We had other performance related issues too that made it difficult to grow our regression test suite. But that’s a story for another time.

So with these four questions I think I provide you a simple diagnostic for some of the basic principles underpinning DevOps. See how your team reacts when you ask these. If your team cant perform these tasks then start working on improving the situation until all four of these become second nature and the questions don’t even raise eyebrows anymore.

I guess it is time to get back to work. This week I will shift my focus away from teaching my son how to build block towers, how to not kill himself while climbing on and falling off furniture and other required survival skills. I will go back into the complex and exciting world of Agile and DevOps, but before I do this I wanted to share what my paternity time meant for me.

First of all, it was amazing to be full time dad while my son makes exciting progress in development, while I was at home he:

• Learned how to crawl, walk with the help of a pampers box and walk freely – wow what progress in such little time
• Kicked his first soccer ball (YAY!)
• Played with his first LEGO set (YAY!)
• Learned to eat food with his little hands and in the process spread it all over the dining area
• Became even more adorable than he already was

What did it mean for me? Well I:

• Slept an estimated 3 hours per night
• Changed roughly 136783 diapers or so
• Realised that you have to be super-efficient to fit breakfast, shower and getting dressed into a 25 min morning nap
• Caught up on few pop culture things I didn’t know about like Bananas in Pyjamas

The last 7 months or so have been absolutely fabulous. I am looking forward to get back to work, because I really enjoy what I do for a living and have a great time, but I wouldn’t have wanted to miss these special months in my son’s life. I certainly appreciate stay-at-home mums and dads a lot more and will never ever ask them: “So what do you do all day?”

I want to encourage all soon to be dads to take the opportunity if they have it to spend some time as full time dad: It is great for your bonding with your child, it will make you appreciate your life partner even more, it will provide you a new perspective on life.

Now I am looking forward to reconnect with my team, my clients and my friends in the industry, I see a few work dinners and drinks in my near future to share stories 😉

The Dunning Kruger effect (a.k.a. Illusionary Superiority)

If you attended any of my conference talks or spoke to me about maturity models, you would have heard me mention Dunning Kruger. Yet to this day I have not written a blog post about it. It’s well overdue as it is the single best explanation for the behaviours I see in our industry and the source of many of my frustrations. Let’s take a good hard look at ourselves in the mirror.

What is the Dunning-Kruger effect?

The Dunning Kruger effect says that people who are not that knowledgeable about a certain area actually believe that they are quite good at it. Perhaps we start with a few examples:

• 90% of drivers consider themselves to be above average – Daniel Gilbert Stumbling on happiness
• 94% of college teachers consider themselves to be above average – Daniel Gilbert Stumbling on happiness
• In a survey of faculty at the University of Nebraska, 68% rated themselves in the top 25% for teaching ability. – Wikipedia
• In a similar survey, 87% of MBA students at Stanford University rated their academic performance as above the median. – Wikipedia
• How do you think people would rate you as a leader?” It turns out that 74% of the respondents think they’re either above average or the best leader their people have ever had. – SmartBrief on Leadership
• Or consider Lake Wobegon where all children are above average 😉

Perhaps it is best explained by the anecdote that gave it the name, which inspired the original research from Dunning and Kruger:

A man decided rob a bank and did his research to prepare the big heist. He then entered the bank robbed the bank of it’s money and made his way home happy to see everything going smoothly. When he arrived at home the police was already waiting for him and arrested. The man was shocked. In his research he had found out that lemon juice can be used as invisible ink, and so he covered himself in lemon juice to be invisible to the people and cameras due to the invisible ink…

Is Dunning Kruger really a thing?

You might wonder whether Dunning-Kruger is really a thing and why it is important for you. Let me tell where I came across it most often. I do a lot of assessment work to understand where organisations are in their DevOps journey and there is one aspect that keeps baffling me: Continuous Integration. Here is a pretty common dialog that I experience with someone from the development of an organisation, let’s call him Adam:

Mirco: Do you practice Continuous Integration?

Adam: (with a little hesitation) Yes we do.

Mirco: (thinking I could stop here and tick the box, but am curious) How do you know that you are doing CI?

Adam: We have CI server and are using Jenkins.

Mirco: How often does the CI server build your software?

Adam: We run it once a week fully automatically.

Mirco: How often are your developers checking in code?

Adam: Multiple times a day.

Mirco: Does your CI server run cover static code analysis and automated unit tests?

Adam: No.

(…)

I don’t believe that people have the intention of lying in this scenario, I think Dunning-Kruger is working its magic here. I was baffled by seeing this pattern again and again. Once I learned about Dunning-Kruger things fell into place.

Once you know it, you will see it everywhere in the DevOps and Agile world. People believing they are done when they are just at the beginning. People attending conference talks and rather than understanding the difference picking up only the things they have in common to validate their self-evaluation.

This also means that self-assessments are potentially dangerous and that maturity models can be misleading. It might also make you question the validity of some of the surveys you see out there. Well, it might make you doubt yourself? Am I really doing this right or am I falling into Dunning-Kruger?

I think the most important takeaway is that we need to talk more. We need to try to understand the context and the situation much better before we do an evaluation. We need to look for others to take an external look at our situation and compare it as objectively as possible. We need to provide more education for our people as a recipe against Dunning-Kruger. As an alternative to the usual maturity model I once created a “Civilisation”-inspired technology tree for Continuous Delivery with one of my clients, you can read more about this here.

Rather than looking for maturity models, perhaps it is better to look at outcomes and use maturity models as a conversation starter with a coach to help us identify which steps to take next rather than an evaluation of our teams. It’s only one of the tools in our toolbag and we should not put too much emphasis on it.

I will leave you with this final picture that I came across, unfortunately I don’t know the origin (Update: Martin in the comments told me it’s from Simon Wardley), but it does describe the danger zone very nicely in which many people new to Agile and DevOps find themselves:

In November I had one of the most stressful, most nerve-wracking and most amazing days of my life. Of course it won’t compete with the birth of my son or my wedding day, but it was still very special. I was at the DevOps Enterprise Summit to speak and to hand out preview copies (called galleys) of my upcoming book “DevOps for the Modern Enterprise” (available in April 2018 from Amazon, Bookdepository and many other sources).

When I saw a physical copy of the book for the first time in the morning, it was an unreal feeling. There was certainly pride but also a level of disbelief. But damn does it look good 😉 I went on to an interview with Alan Shimmel (which you can find here) to discuss the book. I can’t wait to hear from Alan what he thinks about the book. Before this day only a small number of people had read the book and they are people I consider friends, hence they would have been kind to me in any case, now the book is out in the wild and I am nervously awaiting the feedback from the community, a community that is passionate, opinionated and full of experts in their own rights. Let’s see.

Later in the afternoon I presented my talk “What got you here, wont get you there – a story of transformations”. You can see the video of the presentation here. I think the talk went reasonably well as several people over the next couple of days commented on it with positive feedback. The talk gives a nice glimpses into the content of the book and is a good teaser. If you like the talk, I think you will like the book as it is very similar in tone and content.

And then came the big moment, a 2 hour book signing event in the evening. I was more nervous about this than anything else to be honest. Would anyone even come by my desk when the alternatives are great authors like Mark Schwartz, Gene Kim, Dominica deGrandis and Nicole Forsgren? I didn’t have to worry, many people stopped by to get a signed copy of the book. Lots of people who had been at my talk and wanted to read more. I think I got the hang of the book signing process after a while and apologise to some of the early people in the queue when I was clearly still a bit overwhelmed by the experience. I enjoyed talking to so many people about the conference, the book and my talk.

After the book signing I thought I might go for a drink, but had to admit that my social batteries were depleted after the experience. I am humbled by all the positive comments from people and by so many people from the DevOps community turning up to grab a copy of my book. You made this a really special experience and I hope you enjoy the book. Do let me know what you think.

Happy New year to you all. I am now half-way through my paternity leave and I will have to admit that the “Father of the year” award might be slightly out of reach for me this year. I will get to that, but first let me tell you that I will write to the government and make it clear that this should not be called paternity leave. It is paternity work. I have so much respect for all those full-time parents, your day is fully organised by eating, putting little one to bed, cleaning up, making food, changing baby,… I am keeping up with the schedule but will admit that my wife sent me some handy SMS reminders during the day in the beginning. And then you need to use the few free moments to get the basics done: Shower, eat, check on the Ashes and soccer results,…

You wonder why I don’t think I will win the Father of the year award…well…okay here some not so pretty truth:

• I always thought I would not buy many toys for my kids…but man are they fun. I am buying those toys because its fun to teach him something new with building blocks, and shape sorters, and Duplo blocks, and have you seen the Ballapalooza…you get the gist. By now we are drowning in toys that Papa thought might be nice to play with. And really for an adult who keeps buying lego sets for himself (have you seen the recent millennium falcon set?), should I have really expected anything else from myself?
• Somewhat related is the “we will keep it tidy” idea that I had. And then there is the realisation that your trip from the bedroom to the bathroom at night is now a dangerous endeavour through a minefield of sharp or loud objects…I wonder is it worse to have a sharp pain under your foot or to trigger a loud noise that makes the little one wake up…I let you judge that…I swear the enemy (see picture below) is getting cleverer every day
  
• And of course there is the “we will not let him watch TV until he is old enough”…”Are you thinking what I am thinking B1?”, “I think I am B2”, “It’s papa let’s me watch TV time” Papa gave up and uses a bit of TV once in a while to keep him under control when feeding. And I let you guess which one is his favourite TV show? Oh well I have fond memories of the shows I watched as a kid.

But apart from that the two of us get along well and time flies. We go to weekly swimming lessons together which is fun. We explore the local parks. And since he started walking around Christmas, I am chasing around the house to protect him from the furniture (or is it the other way around?!?).

And then there are those moments when you wonder what is going through his head. I bought him a set of the construction toys, one of them a CAT excavator. And then one of the real ones turns up next door to demolish the house next door. What goes through the head of the little man when his toy all of sudden has a gigantic brother? Is that scary or incredibly cool? I wish he could tell me.

My leave was sweetened by getting promoted to Managing Director in December which gave me reason to celebrate with family – and again just like later at Christmas, I wonder what the little one is thinking? Perhaps: Why is everyone so happy today and why am I not the centre of attention as usual? Our joint birthday is coming up and will mark the end of my paternity leave and for that celebration the little man will be the centre of all attention. The other birthday boy will gladly stand aside and enjoy the loudest birthday party he has had in a long time with the coolest birthday cake (yes I know it’s for the little one, but still 😉) I will post one more time to reflect back before heading back into the office in February.