How to Fail at Test Automation

(This post was first published on DevOps.com)

Let me start by admitting that I am not a test automation expert. I have done some work with test automation and have supervised teams who practiced it, but when it comes to the intricacies of it, I have to call a friend. It is from such friends that I have learned why so many test automation efforts fail. Talking to people about test automation validates my impression that this is the DevOps-related practice that people most often fail at.

Let me share the four reasons why test automation fails, in the hope that it will help you avoid these mistakes in your test automation efforts.

Before I go into the four reasons, allow me one more thought: Test automation is actually a bad choice of word. You are not automating testing, you automate quality assessments. What do I mean by that? It is a mistake to think of test automation as automating what you otherwise would do manually. You are finding ways to assess the quality of your product in automated ways and you will execute it way more often than you would do manual testing. This conceptual difference explains to a large degree the four reasons for test automation failure below.

Reason 1: Underestimating the Impact on Infrastructure and the Ecosystem

There is a physical limit of how much pressure a number of manual testers can put on your systems. Automation will put very different stress on your system. What you otherwise do once a week manually you might now do 100 times a day with automation. And into the mix an integrated environment, which means external systems need to respond that frequently, too. So you really have to consider two different aspects: Can your infrastructure in your environments support 100 times the volume it currently supports, and are your external systems set up to support this volume? Of course, you can always choose to reduce the stress on external systems by limiting the real-time interactions and stub out a certain percentage of transactions or use virtual services.

Reason 2: Underestimate the Data Hunger

Very often test automation is used in the same system where manual testing takes place. Test automation is data-hungry, as it needs data for each run of test execution and, remember, this is much more frequent than manual testing. This means you cannot easily refresh all test data whenever you want to run test automation and have to wait until manual testing reaches a logical refresh point. This obviously is not good enough; instead, you need to be able to run your test automation at any time. There are a few different strategies you can use (and you will likely use a combination):

  • Finish the test in the same state of data that you started with;
  • Create the data as part of the test execution;
  • Identify a partial set of data across all involved applications that you can safely replace each time; or
  • Leverage a large base of data sets to feed into your automation to last until the next logical refresh point.

Reason 3: Not Thinking About the System

Test automation often is an orchestration exercise as the overall business process in test flows across many different applications. If you require manual steps in multiple systems, then your automation will depend on orchestrating all those. By just building automation for one system you might get stuck if your test automation solution is not able to be orchestrated across different solutions. Also, some walled-garden test automation tools might not play well together, so think about your overall system of applications and the business processes first before heavily investing in one specific solution for one application.

Reason 4: Not Integrating it into the Software Development Life Cycle

Test automation is not a separate task; to be successful it needs to be part of your development efforts. From the people I have spoken to there is general agreement that a separate test automation team usually doesn’t work for several reasons:

  • They are “too far away” from the application teams to influence “ability to automate testing,” which you want to build into your architecture to be able to test the services below the user interface;
  • Tests often are not integrated in the continuous delivery pipeline, which means key performance constraints are not considered (tests should be really fast to run with each deployment);
  • Tests often are not executed often enough, which means they become more brittle and less reliable. Tests need to be treated at the same level as code and should be treated with the same severity. This is much easier when the team has to run them to claim success for any new feature and is much harder to do when it is a separate team who does the automation. It also will take much longer to understand where the problem lies.

Of course, absence of failure does not mean success. But at least I was able to share the common mistakes I have seen and, as they say, “Learning from others’ mistakes is cheaper.” Perhaps these thoughts can help you avoid some mistakes in your test automation journey. I do have some positive guidance on test automation, too, but will leave this for another post.

And in the case you found your own ways of failing, please share it in the comments to help others avoid those in the future. Failures are part of life and even more so part of DevOps life (trust me, I have some scars to show). We should learn to share those and not just the rosy “conference-ready” side of our stories.

Test automation is for me the practice that requires more attention and more focus. Between some open-source solutions and very expensive proprietary solutions, I am not convinced we in the IT industry have mastered it.

One bonus thought: If you cannot automate testing, automate the recording of your testing.

If you cannot automate testing, find a way to record the screen with each test by default. Once you identify a defect you can use the recording to provide much richer context and make it a lot easier to find the problem and solve it. Verbal descriptions of error are very often lacking and don’t provide all the context of what was done. I keep being surprised how long triage takes because of the lack of context and detail in the defect description. There is really no excuse for not doing this. Record first, discard if successful, attach it to the defect record if you find a problem.

Guide to the Guide to Continuous Delivery Vol 3

CD GuideI am not really objective when I say that I hope you have read the most recent Guide to Continuous Delivery Vol. 3 as I had the honor to contribute an article to it. My article is about mapping out a roadmap for your DevOps journey and I have an extended and updated blog article in draft on that topic that I will push out sometime soon. There is a lot of really good insight in this guide and for the ones with little time or who just prefer the “CliffsNotes”, I want to provide my personal highlights. I won’t go through every article but will cover many of them. Besides articles the guide provides a lot of information on tooling that can help in your DevOps journey.

Key Research Findings

The first article covers the CD survey that was put together for this guide. While less people said they use CD this might indicate more people understanding better what it takes to really do CD, I take this a positive indication for the community. Unsurprisingly Docker is very hot, but its clear that there is a long way to go to make it really work when you look at the survey results.

Five Steps to Automating Continuous Delivery Pipelines

Very decent guidance on how to create your CD pipeline, the two things that stood out for me are “Measure your pipeline”, which is absolutely critical to enable continuous improvement and potentially crucial for measuring the benefits for your CD business case. It also highlights that you sometimes need to include manual steps, which is where many tools fall down a bit. Gradually moving from manual to full automation by enabling a mix of automated and manual steps is a very good way to move forward.

Adding Architectural quality metrics to your cd pipeline

An interesting article on measuring more than just functional testing in your pipeline. It stresses the point to include performance and stress testing in the pipeline and that even without full scale in early environments you can get good insights from measuring performance in early environments and use the relative change to investigate areas of concern.
There is other information that can provide valuable insights into architectural weaknesses like # of calls to external systems, response time and size for those calls, number of exceptions and CPU time.

How to Define your DevOps roadmap – Well read the whole article 😉

Four Keys to Successful Continuous Delivery

Three of the keys are quite common: Culture, Automation and Cloud. What I was happy to see what the point about open and extendable tools. Hopefully over time more vendors realise that this is the right way to go.

A scorecard for measuring ROI of Continuous Delivery Investment

An interesting short model for measuring ROI, it uses lots of research based numbers as inputs into the calculations. Could come in handy for some who want a high-level business case.

Continuous Delivery & Release Automation for Microservices

I really liked this article with quite a few handy tips for managing Microservices that match my ideas to a large degree. For example you should only get into Microservices if you already have decent CI and CD skills and capabilities. Microservices require more governance than traditional architectures as you will likely deal with more technology stacks, have to deal with additional testing complexity and require a different ops model. To deal with this you need to have a real-time view of status and dependencies of your Microservices. The article goes into quite some detail and provides a nice checklist.

Top CD resources

No surprise here to see the State of DevOps report, Phoenix Project and the Continuous Delivery book on this list.

Make sure to check out the DevOps Checklist on devopschecklist.com – there is lots of good questions on this checklist that can make you think about possible next steps for your own adoption.

Continuous Delivery for Containerized Applications

A lot of common ground get revisited in this article like the need for immutable Microservices/containers, Canary launches and A/B testing. What I found great about this article is the description of a useful tagging mechanism to govern your containers through the CD pipeline.

Securing a Continuous Delivery Pipeline

Some practical guidance on leveraging the power of CD pipelines to increase security, a topic that was just discussed at the DevOps Forum in Portland too and which means we should see some guidance coming out later in the year. The article highlights that tools alone will not solve all your problems but can provide real insights. When starting to use tools like SonarQube be aware that the initial information can be confusing and it will take a while to make sense of all the information. Using the tools right will allow you to free up time for more meaningful manual inspections where required.

Executive Insights on Continuous Delivery

Based on interviewing 24 executives this articles gathers their insights. Not surprisingly they mention that it is much easier to start in a green fields environment than in brown fields. Even though everyone agrees that tools have significantly improved, the state of CD tools is still not where people would like it to be and many organisations still have to create a lot of homemade automation. The “elephant in the room” that is raised at the end is that in general people rely on intuition still for the ROI of DevOps, there is no obvious recommendation for how to measure this scientifically.

Agile Governance – A short overview

I recently ran a seminar on Agile governance, so what does this even mean? Agile governance requires you to find the right balance between the complete chaos of no governance at all and the smothering of all Agile benefits in overbearing governance. In this article I want to provide an overview on Agile governance across a few levels: Transformation, Program, Project and team level. There is a lot more to be said than I can put into this post, so expect me to come back to this over time.

Agile Transformation governance

To borrow from Lord of the Rings “One does not simply transform an organisation to Agile”. At the transformation level it is important to provide support for teams as they encounter challenges for adopting Agile. Some problems can proactively be addressed like training, coaching and enable good engineering/DevOps practices. Others will be uncovered as you start down the adoption path. Those blockers can be identified from the retrospectives at the team level as well as by speaking to Iteration managers and coaches. These blockers should be prioritised and provide the transformation backlog to address at organisational level. I find the below transformation governance structure very helpful as a starting point:

agile governance

 

Agile governance at the program/project level

I have shared my view on how to manage at the program level before here. Just one additional word of warning: Do not try to compare teams against each other for performance measurements, even under the guise of gamification this really can only lead you to bad development patterns. There are teams where this works and where their leaderships does not use the information for the wrong reasons, but the risks are higher than the benefits. Defining the success criteria for each team individually is a much better way to deal with this.

Agile governance at the team level

At the team level the SCRUM methodology provides a set of useful governance elements which you should leverage, adopt and adapt:

  • Sprint Reviews
  • Sprint Planning
  • Retrospectives
  • Team based estimations

Additionally you can use Agile maturity models and Agile metrics for teams to evaluate themselves and their progress in maturing their Agile practices.  Good coaches can help teams in this evaluation without being judgmental.

A comment on measuring individual productivity:
How do you measure the productivity of team members in your Agile teams? The short answer is YOU DON’T. I have talked about productivity in this post and if you don’t believe me perhaps Dilbert can help you understand how poisonous this idea is for your teams:

dilbert2

 

I am looking forward to hear from the community what you use to govern your Agile adoption across the different levels.

Is there such a thing as Hybrid Agile?

I recently wrote an article about Hybrid Agile for InfoQ  because the term has been misused too often. Feel free to read the whole article. Here is my conclusion from the article:

After many discussions, people convinced me that “Hybrid-Agile” is what is otherwise called Water-Scrum-Fall and after some deliberation I admit that this makes sense to me. Of course the term Water-Scrum-Fall or similar phrases are often used with contempt or smiled upon, but when you look closer, this is reality in many places and for good reasons.

Managing the Plethora of DevOps tools

I have been thinking about DevOps tools a lot and discussions about tools often distract from the real problems. But what are the right DevOps tools. Well I will not go into specific tools, instead I will tell what I am looking for in DevOps tools beyond the functionality they provide. In my experience you can build good DevOps tools chains with just about any tool, but some tools might take more effort to integrate than others. I will also provide some guidance on how to manage tools at the enterprise level.

It seems that new DevOps tools appear on the market every month. This is extenuated by the fact that it is difficult to classify all the tools in the DevOps toolbox. One of the best reference guides is the xebialabs periodic table of DevOps tools (https://xebialabs.com/periodic-table-of-devops-tools/ ) which is well worth checking out.

Before I go into the details of what characteristics a good DevOps tool should have, I want to address one other aspect: Should you have one tool or many in the organization?

In general, in large organization it makes sense to have a minimal set of tools to support for several reasons:

  • Optimise license cost
  • Leveraging skills across the organization
  • Minimising complexity of integration

Yet on the other side some tools are much better for specific contexts than others (e.g. your .NET tooling might be very different to your mainframe tooling). And then there are new tools coming out all the time. So how do you deal with this? Here is my preferred approach:

  • Start with small set of standard tools in your organization
  • Allow a certain percentage of teams to diverge from the standard for a period of time (3-6 months perhaps)
  • At the end of the “trial-period’ gather the evidence and decide what to do with the tool in discussion
    • Replace the current standard tool
    • Get it added as an additional tool for specific contexts
    • Discard the tool and the team transitions back to the standard tool

Obviously DevOps tools should support DevOps practices and promote the right culture. This means the tools should not be a “fenced garden” and only work within their own ecosystem. It is very unlikely anyway that a company uses only tools from one vendor or ecosystem. Hence the most important quality of tools is the ability to integrate it with other tools (and yes possibly be able to replace it which is important in such a fast moving market.)

  1. So then the first check is how well APIs are supported. Can you trigger all functionality that is available through the UI via an API (command line or programming language based)?
  2. We should treat our tools just like any other application in the organization, which means we want to version control it. The second check is hence whether all configuration of the tool can be version controlled in an externalised configuration file (not just in the application itself)?
  3. Related to the second point is the functionality to be able to support multiple environments for the tool ( e.g. Dev vs Prod). How easy is it promote configuration? How can you merge configuration of different environments (code lines)?
  4. We want everyone in the company to be able to use the same tool. This has implications for the license model that is appropriate. Of course open source works for us in this case, but what about commercial tools? They are not necessarily bad. What is important is that they don’t discourage usage. For example tools that require agents should not price for every agent as this means people will be tempted to not use it everywhere. Negotiate an enterprise wide license or ‘buckets of agents’ so that not each usage will require a business case.
  5. Visualization and analytics are important aspects of every DevOps toolchain. To make this work we need easy access to the underlying data and that means we likely want to export data or query data. If your data is stored in an obscure data model or you have no way to access the underlying data and export it for analysis and visualization then you will require additional overhead to get good data. Dashboards and reports within the tool are no replacement as you likely want to aggregate and analyse across tools.

I hope these criteria are all relatively clear. What is surprising is how few tools adhere to these. I hope tool vendors will start to realize that if they want to provide DevOps tools they need to adhere to the cultural values of DevOps to be accepted in the community.

Hopefully the tools you are using are adhering to many of these points. Let me know what you think in the comments section of this blogpost. I am very curious to learn how you perceive DevOps tools.

Thoughts on the 10th State of Agile Survey

adopt

10th Agile reportJust recently I reviewed the 9th Agile survey (read about it here) and already the 10th version is out (you can access the survey here), so I thought I provide an update. Given the results are not that different from the previous year, I will write this post a bit different. I will review some of the key results and then will provide in a second section some critical thoughts on the survey.

Part 1 – Review of the Results

Our delivery world is complex (multi-modal, distributed and outsourced)

Looking at the results about company experience, 53% say that less than half of the teams in the organisation uses Agile, only 9% say that everyone is doing Agile. So clearly we live in multi-modal world. 82% are working with distributed teams, which is larger than I expected but reflects the largely distributed nature of IT projects these days. And nearly 70% use outsourcing for their development projects. This complex environment is normal. I find it interesting how many clients tell me there are “special” because of these characteristics, they clearly are not.

What are typical benefits of Agile?

The report highlights the following three to be the top answers (which have been the same last year):
– Ability to manage changing priorities
– Team productivity
– Project visibility

This sits well with me as the usual suspects that we have to debunk are not in this list, e.g. faster time to market (only on rank 6) and cheaper delivery (not in the list at all).

How do you measure the success of Agile?

The answers to this are also stable:
– On-time delivery
– Product Quality
– Customer Satisfaction
Good answers, I think this reflects well what success looks like. I guess these are somehow measurable and provide some success measure for Agile.

How do you scale Agile?

We see a significant jump for SAFe here from 19% to 27% and I am not surprised. It certainly has a lot of momentum in the industry and provides practical guidance that I leverage in my daily work. Other frameworks are really not present (DAD, LESS,…) and have less than 10%.

Of the top 5 tips for Scaling Agile at least two are in my top lessons learned too: Consistent process & practices and Implementation of a common tool across teams. I agree with the other 3 tips as well: Executive Sponsorship, getting help from someone with experience like an Agile consultant and creating an internal support team.

What tools do you use?

Really not much movement here. The previous favourites remain on top: Jira, VersionOne, and TFS.

What makes Agile fail?

Culture has been called out as main reason this year. And it is hard to argue with this. That however is difficult to address directly. The other information is easier to address: lack of experience is the second main reason Agile fails, this means that we should make sure experienced Scrum Masters and coaches support new projects. Lack of management support and lack of support for the cultural shift are next, something we need to work on as an industry. And then a very much addressable one still gets 38% of the votes: Inconsistent agile practices and processes. This we need to fix, I don’t understand how so many people still allow Agile projects to fail because they think Agile is a freeform exercise. At the end of the day there is a project/outcome to be delivered and too many coaches still shy away from that accountability. If you work with Agile coaches make sure they have skin in the game so that they balance learning with delivery responsibility.

Overall I like the results in the report and it is consistent with the previous year which is good to see.

Part 2 – A more critical look at the survey

I like the results of the survey and got some interesting insights, but how valid is the survey? Do I understand the mechanics behind it well enough to use the data?

Let’s look at a few pieces of information that should make us think about the data before blindly quoting it:

  • First of all – who is the sample audience? Clearly the people responding to this have somehow heard about Agile as it is unlikely that others respond to an Agile survey. The people who respond are self-nominating, so they might be more on the extreme ends and feel they have something to contribute. It is not a random sample of people working in IT. This then influence the data to some degree I would think.
  • 1% said that Agile adoption has failed in their organisation. That number seems very low and might be because those people where it has failed are not responding, or it could be because the definition of failing is actually quite difficult to understand. Does failing mean you are completely going back to Waterfall or because you had mostly bad results from Agile. Not sure what my definition would be, so perhaps the lack of definition causes people to not use this value in the survey.
  • Agile techniques used – Now here I really start to struggle with the values and figuring out how this information should help me. Do only people respond to this who use Agile or everyone (even those using Waterfall)? Daily Stand-ups and prioritised backlog seem pretty fundamental to me and other than in rare cases I would expect Agile projects to use these. The numbers seem low on that basis. But wait it gets much worse, only 54% do iteration reviews/showcases, only 45% have devs and testers in the same team. I would argue that if you don’t have those two you are not really Agile – I defined Agile in another blog post as “ONE team delivery an INCREMENT of a product (at least product tested) within ONE iteration”. The problem here is that one cannot relate the info back to the methodology used and hence it becomes difficult to derive insight from this data.
  • Success measures – Velocity is the largest one. I was hoping we got over the velocity race, but clearly it’s still on. The success measures are very contextual and I would caution anyone to use this without thinking. Take for example “Planned vs. Actual stories” – it could be very positive to deliver more stories than planned or even less stories (as long as the most value is delivered). Context is unfortunately king for metrics, so don’t use the information from this survey blindly.

Thoughts on the 9th State of Agile Survey

State of Agile 9The 9th Annual State of Agile report had some confirming and some surprising results. If you have not read it, it is worthwhile having a look here. And yes I know that number 10 is soon coming out, but hey there is still value in the 9th one to look at. Besides the usual statistics around how much Agile people are doing ( key number for me was only 5% working for fully traditional organisations), it provided some interesting answers to some of the more common questions that I hear and the answers are often not surprising but there was the odd surprise. Let’s jump in:

What are typical benefits of Agile?

The report highlights the following three to be the top answers (which have been stable over the last few years):
– Ability to manage changing priorities
– Team productivity
– Project visibility

This sits well with me as the usual suspects that we have to debunk are not in this list, e.g. faster time to market (only on rank 7) and cheaper delivery (not in the list at all).

How do you measure the success of Agile?

Uhhh, tricky one this one. I have heard the question many times and honestly have struggled to give an answer that satisfies senior stakeholders. So what did the report say:
– On-time delivery
– Product Quality
– Customer Satisfaction

Good answers, I think this reflects well what success looks like. Interesting that some of the items mentioned under top benefits are showing up much lower here: Managing priorities obviously speaks to the product quality and customer satisfaction. But team productivity (29% measure it) and visibility (30% measure it) are much lower on the list. An open question for me is how people would measure productivity in the first place (see my other blog on this).

How do you scale Agile?

As much as I come across SAFe it is only used by 19% of the respondents with all the other ones even lower (DAD, LESS). The largest proportion just uses Scrum of Scrums or some custom created method.

Of the top 5 tips for Scaling Agile at least two are in my top lessons learned too: Consistent process & practices and Implementation of a common tool across teams. I agree with the other 3 tips as well: Executive Sponsorship, having a coach and creating an internal support team.

What tools and practices do you use?

Wow – I was surprised, but perhaps I should not have been that Excel and Project are the most used tools…seriously, are we not better than that??? Oh well on the real Agile tools, Jira and VersionOne takes the cake, with TFS close behind. IBM is much much lower. This represents my position as well, JIRA is certainly the one most used and few people complain about it, especially when integrated with Confluence.

There is also information on the practices uses and I was shocked to see that only 69% use retrospectives and 48% have a dedicated product owner. Overall the adoption rates of the practices feels very low, perhaps there is some fundamental flaw in the data if it considers people who run Waterfall projects but use a select few Agile practices…hmmm….

What makes Agile fail?

Good information here as well, lack of experience is the main reason Agile fails, this means that we should make sure experienced Scrum Masters and coaches support new projects. Lack of management support and a non-aligned company culture are the other two main reasons Agile fails. Those are a bit more difficult to tackle but are important to be aware of as you set off on an Agile project.

Overall I like the results in the report and it certainly helps to see some market data validating points that I keep making with my clients.