How to measure Devops work?

Your company now uses the DevOps methodology. You have already integrated development, operation and quality assurance, and have drastically moved away from rapid action and have moved closer to sustainable application development. How do you know how well such a solution works? How do you know if it works at all?

Many of the new companies that take advantage of this approach are surprised that getting started was easier than they thought, but more difficult to keep alive. This can lead to dangerous habits or simply the fact that the team is too overwhelmed. In a word, you need meters.

You need meters not only as a way to measure success (or lack of success) of DevOps, but also as a way to see how this success can be increased, modified or extended. Without them you work blindly. With them you have a holistic point of view – you know where you are, where you are going, where you can go and how to get there. But I’m not talking now about analytical tools that measure actions within development. I’m talking about measuring development itself.

What factors measure DevOps meters? For the most part, they measure aspects such as development speed, implementation and customer response, as well as the frequency of implementations and failures, repair time, volume of repair requests and the rate of change in the above indicators. These meters usually focus on implementation, operation and support (as opposed to e.g. early-stage design and implementation), as most of the ongoing DevOps effort takes place in these areas.

Most DevOps meters fall into three main categories:

• People. People are an integral part of any DevOps process. People-oriented meters measure, among other things, results, capabilities and response time. Always start with people. They can be the most difficult element, and their influence is sometimes difficult to notice.
• Process. In some ways DevOps focuses mainly on the process – a continuous cycle of implementations, operation and support is a continuous package of intertwining processes. However, some measures are more process-oriented than others, especially those involving continuous task execution, response and repair. The lead time for implementation is, for example, a process-oriented meter, as well as the implementation frequency and response time. Process measures can measure speed (where are the bottlenecks? Is the process itself a bottleneck?), adequacy (are all steps relevant?), effectiveness (does the process accomplish the task?) or efficiency (are the steps in the optimal order? Is there free movement in the process?).
• Technology. Technology meters play an important role in DevOps, measuring, among other things, uptime (how long does the system last? What about the network and supporting applications?) and failure rate (what percentage of failed implementations, changes or units?).

Of course, many DevOps meters cover all three categories to a greater or lesser degree. Perhaps the easiest way to check if the meters work in practice is to look at the key Puppet Labs:

Frequency of implementations (or changes)

DevOps practices enable frequent and continuous deployments; powerful, high-traffic websites and cloud-based services make them a must. With fast feedback and small batch development, updated software can be deployed every few days or even several times a day. In a DevOps environment, the frequency of deployments can be an indirect or direct measure of response time, team consistency, development capabilities, development tools effectiveness and overall DevOps team efficiency.

Time to implement the changes

Delivery time is the time from the start of the implementation cycle (first new code) to implementation. It measures the performance of the development process, the complexity of the code and implementation systems, as well as (such as the frequency of implementations) the capabilities of the team and the developer. If the lead time is too long, it may indicate that the development/implementation process at certain stages is inefficient, or that it involves performance issues.

Change failure rate

One of the main goals of DevOps is to replace fast and frequent implementations with daily activities. Of course, for such deployments to be worthwhile, the failure rate must be low. In fact, it should decrease over time, as the experience and capabilities of DevOps teams increase. An increasing failure rate, or one that is high and does not decrease over time, is an effective sign of problems throughout the DevOps process.

Average repair time (MTTR)

This is the time from the appearance of the failure to its repair. This is a good way to measure the team’s capabilities and, like the failure rate, should show a general decrease in time (allowing for occasional longer repair times when the team faces a problem that they are not technically familiar with). MTTRs can be affected by the complexity of the code (or platform), the number of new features being deployed, and changes in the operating environment (such as migration to a new cloud server).

Based on these meters, an ideal DevOps team would contribute to frequent, fast deployments with low (and decreasing) failure rates and short (and decreasing) response times. In practice, of course, there may be factors that contradict these trends – for example, a smaller need or capability for frequent implementations or frequent changes in operating conditions or requirements. In general, however, these measures include some of the most important performance issues in DevOps.

The most important thing is that if you use DevOps methodology, you need meters and good measurement tools. Without them you won’t be able to find out whether DevOps does what you expect or whether it covers problem areas that require your attention.

In any case, good DevOps meters should cover the four areas specified above (deployment frequency, lead time, failure rate and repair time) along with other key indicators relevant to your business. Good DevOps measurement tools should also provide information in a clear, detailed, accurate and easily configurable way, allowing you to quickly focus on potential problem areas and other key elements in the DevOps process.

Tools that measure DevOps performance usually monitor processes and conditions in real time (such as uptime and downtime or traffic volume) and record events (builds, implementations, breakdowns, repair reports, etc.). More generalized project management tools that track factors such as project schedule or goals vs. performance can also provide valuable DevOps meters.

The following aspects are worth noting: when you start looking closely at the meters, you will notice that many of them are available. Before you go into the meters, it is important to understand exactly which meters are important to you and how they will apply to your DevOps program. If you don’t know what you’re looking for, it’s easy to get lost in a labyrinth of data that’s not entirely relevant. But if you’re focused, you can reach straight for the gold that DevOps meters offer and use them well.