Downstream impact of AI on Engineering Analytics and DORA

Top-performing engineering teams know the impact of hitting the right metrics, especially when it comes to speed, quality, and resilience. According to the 2024 State of DevOps report, the best teams achieve lead times as low as 1 minute 29 seconds, and their incident recovery times average just 33.95 minutes. But how many organizations can reach those benchmarks? And more importantly—what does it take?

DORA (DevOps Research and Assessment) metrics exist to answer these questions. They zero in on core performance metrics such as deployment frequency, lead time for changes, mean time to restore (MTTR), and change failure rate.

The State of DevOps report has a a dedicated section pointing out AI's impact on Developer Productivity. AI tools help teams build features faster, leading to better code reviews, real-time diagnostics, feedback cycles, and critical insights into your workflows, which can positively affect your team's scores around each metric.

This guide will explain DORA metrics and its four key metrics and demonstrate how AI can help you improve your team’s performance across these metrics.

What Are DORA Metrics?

DORA metrics provide a quantitative method for measuring your software development and deployment cycle effectiveness. They try to provide some answers to the question, “What makes the best-performing software engineering teams work?”

To understand DORA metrics, you must understand why it all started. According to Jez Humble, one of the co-founders of DORA, the organization was founded to help companies improve their software delivery performance through scientific measurement, benchmarking, and personalized recommendations. It started as a collaboration and eventually became an independent entity before being acquired by Google Cloud in December 2018.

After analyzing thousands of survey responses from professionals over a long period of time, the team identified four key metrics: deployment frequency, lead time for change, change failure rate, and mean time to restore;

These key metrics have now become what we know today as the DORA metrics.

#1: The Deployment Frequency

Deployment frequency measures the number of times an organization deploys code changes to production over a specific period. This period could be daily, weekly, monthly, or any other interval that makes sense for your team. This metric reflects the agility of your release cycle and indicates how well your team can deliver improvements, new features, or fixes to users.

Having a high deployment frequency indicates a fast, efficient development process, while a low frequency can signal bottlenecks and inefficiencies in your continuous integration and delivery processes or pipeline.

How to Measure Your Deployment Frequency

Calculating deployment frequency is straightforward. The formula involves dividing the number of deployments by the length of the measurement period. For example, if a team deploys 20 times in a week, their deployment frequency is 20 deployments per week.

The formula is:

$$Deployment\ Frequency = Number\ of\ Deployments\ /\ Time\ Period$$

The Time Period is the deployment interval selected by you or your team.

Factors Affecting Your Deployment Frequency

Whether using DevOps practices or not, your deployment has to go through a process or pipeline. With the introduction of Continuous Integration/Continuous Deployment (CI/CD) and Infrastructure as Code (IaC), your team can automate most development and operation processes, such as running tests, deploying applications, and even provisioning infrastructures. Additionally, approaches like the GitOps workflow allow your team to manage these processes via a Git repository.

In the GitOps workflow, after you’ve committed your code to a Git repository, the CI/CD pipeline takes over and executes the rest of the processes.

However, if the pipeline is not properly created or there are issues with any steps created within the pipeline configuration, it can slow down how quickly your team can push out updates. IaC configurations usually need to be reviewed before they reach the production environment to prevent security issues, misconfiguration, and more.

How Can You Increase Your Deployment Frequency?

Modern AI tools can significantly improve your deployment frequency through automated code reviews and infrastructure analysis. These tools integrate with CI/CD platforms to review configurations, detect potential issues, and provide actionable insights for improvement.

For example, when your developer updates test cases and creates a CI pipeline to execute the tests.

# .github/workflows/ci.yml
name: Node.js CI

on:
  push:
    branches:
      - main
      - feat/server
  pull_request:
    branches:
      - main
      - feat/server

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '16' # You can specify your desired Node.js version

      - name: Install dependencies
        run: npm install

      - name: Run tests
        run: npm test

This snippet indicates that the action runs when there is a push or pull request to the main and feat/server branches. It sets up Node.js 16 and runs the test.

Once the PR is created, AI can spot potentially missed edge cases the user missed and provide suggestions on mitigating them.

In another scenario, your complete infrastructure configuration is stored and managed on Git. Let's say your DevOps engineer creates a PR to update or provision a new resource; AI can quickly review these configuration updates and suggest improvements to secure and streamline the provisioning process.

With automated configuration reviews and improvement suggestions, you can speed up the process in less time than with the traditional approach.

#2: Lead Time for Changes

Lead time for changes is another essential DORA metric, measuring the time it takes from committing code to deploying it to production. Unlike the Deployment frequency, it measures the efficiency of the DevOps process. Having a high lead time for changes implies that you have a slow code review/code deployment process.

The lead time for change is essential because it can help the project teams set realistic deadlines based on the data obtained from these metrics.

A lead time of less than a day clearly means that you and your team can deliver new changes for your users within a day.

How to Calculate the Lead Time for Changes

To calculate the lead time for a specific change, all you have to do is this:

$$Lead\ time\ for\ changes = time\ of\ deployment - time\ of\ code\ commit$$

to get more statistically related data, you can then get the median value.

Factors Affecting lead time for changes

One of the major factors that can greatly affect your lead time for changes is the code review process. It's the next stage of the development process after the code is committed. Code review is very important as it helps catch bugs and security breaches, improve code standards, and catch code smells.

The Pull request size is another factor that can affect the lead time for changes. Larger PRs take longer to review, while smaller PRs are easier and quicker to review. The longer the time it takes to review, the longer the lead time.

How can you reduce your lead time for changes using AI?

Knowing that the code review process greatly affects the lead time for change, automating it is one of the best ways to reduce it.

For instance, your engineer submits a pull request (PR) for a new feature. Traditionally, this PR has to undergo peer reviews to catch bugs or code smells before being merged into the production environment.

You can augment the traditional peer-review process by letting the AI Code Review tools detect the PR as soon as it's submitted and begin an AI-driven code review for the changes made.

It also provides real-time diagnostics and actionable suggestions, allowing you to make the necessary changes promptly.

AI Code Review does a first pass over the PR and provides several automation benefits that can, in turn, reduce your lead time to changes.

#3: Mean Time To Restore (MTTR)

Mean Time to Restore (MTTR) measures the average time it takes to recover from a production incident like an application failure or system outage. MTTR metric is important for understanding how quickly your team can resolve issues that disrupt end users.

A lower MTTR indicates faster recovery from incidents, ensuring minimal disruption for users and reducing financial and reputational risks. According to the State of DevOps report, elite teams restore services within an hour, demonstrating their ability to respond quickly to unforeseen issues. Measuring and improving MTTR can help your team identify bottlenecks in their incident management processes and build resilience in their systems.

How Do You Calculate MTTR?

To calculate MTTR, use the following formula:

$$MTTR = Total\ Downtime / Number\ of\ Incidents$$

For example, if your service experienced 200 minutes of downtime across five incidents in a month, your MTTR would be:

$$200 ÷ 5 = 40 minutes$$

This calculation shows how long it takes your team to restore service after an issue. Tracking your MTTR over time can reveal patterns in how your team handles incidents. You can use this information to understand what’s working, identify gaps, and make adjustments to reduce downtime in the future.

What Factors Influence MTTR?

Several factors affect how quickly your team can restore service after an incident. Knowing these factors helps identify ways to reduce recovery time:

• Detection Time: How long does it take to recognize that an incident has occurred?

• Diagnosis Time: Once detected, how quickly can your team locate the root cause of the problem? This often depends on clear logs, debugging tools, and collaborative troubleshooting processes.

• Fix Time: How fast can your team create and deploy a resolution? The time required depends on the complexity of the issue, the team’s readiness, and the deployment process.

How AI can help reduce MTTR

Modern observability tools use AI to spot and fix problems faster.

ML models help detect issues early by spotting unusual system behavior. When something goes wrong, AI can look through logs, metrics, and traces to find the root cause quickly. These tools also make alerts more useful by grouping related issues together and filtering out noise.

By learning from past incidents, AI can predict potential failures and suggest fixes based on what worked before. This means teams spend less time searching through logs and more time fixing actual problems.

#4: Change Failure Rate

The Change Failure Rate (CFR) measures the percentage of code changes that result in production failures, such as rollbacks or hotfixes. This metric is crucial for understanding the quality and reliability of code changes. A lower CFR indicates better testing, code reviews, and overall deployment practices, while a high CFR often points to insufficient reviews or inadequate testing processes.

Calculating Change Failure Rate

To calculate CFR, divide the number of failed changes by the total number of changes deployed:

$$CFR = (Number\ of\ Failed\ Changes / Total\ Changes) × 100$$

For example, if a team deploys 50 changes and 2 of them fail, the CFR is:

$$(2 ÷ 50) × 100 = 4\%$$

Using AI and Issue Tracking to Improve Change Failure Rate

Connecting your code review tools with issue trackers like Jira gives you a clearer view of your development process. You can quickly spot bottlenecks and delays when you track code from commit to production. By tying reviews to specific features or fixes, your team stays focused on what matters for each change.

This connection also helps prevent scope creep – you can easily check if code changes match what was originally requested. When production issues occur, you can look back at similar changes to find and fix problems faster. Over time, you'll spot patterns that help prevent the same issues from happening again.

Think of it as a health check for your changes—you see what's working, what's not, and where to focus next.

Leverage DORA Metrics to Continuously Improve Developer Productivity

Now that you understand DORA metrics and how AI tools can help. Here’s what you can do next:

• Check Your Current Metrics: Start by measuring your team’s deployment frequency, lead time for changes, MTTR, and change failure rate. Identify where you need to improve.

• Use AI Tools in Your Workflow: Set up tools like CodeRabbit to automate code reviews, check your pipelines, and provide feedback on your processes. Begin by connecting it to your repositories and CI/CD systems.

• Work With Your Team: Share these metrics with your team and discuss ways to make changes. Use insights from automated code review tools to guide decisions.

• Track Your Progress: Monitor your metrics to see how your changes work. Use reports and dashboards to identify patterns and adjust your approach.

With these steps, you can improve your development process and get better results.

Understanding DORA metrics and using automated code review tools like CodeRabbit can help you make data-driven improvements that benefit your team and users. Start by measuring your current metrics, applying what you’ve learned, and tracking your progress. Sign up for a free trial of CodeRabbit.