Measuring what dashboards can't: health checks and maturity models
A DevOps maturity model rates your practices against levels; a health check asks how the work feels now. What each measures, the public models, how to use them.
System metrics show what your delivery pipeline does. They can’t show how the work feels to the people doing it, or how developed your practices are. Health checks and maturity models fill those two gaps with structured perception data. A health check asks the team how things are right now. A DevOps maturity model rates your practices against written levels. Both are most useful as the start of a team conversation, and both break when they become a scorecard for ranking teams.
This chapter of the agile metrics guide covers why perception data is real data, what DevOps maturity models assess, the main criticism of them, and the public models you’re most likely to meet.
Why perception data counts as data
DORA, the research program behind the DORA metrics, measures culture with surveys rather than system logs, and its guidance on generative culture says why: “Organizational culture is a perceptual measure, and therefore, best measured using survey methods.” Its culture survey asks people to rate statements such as “On my team, information is actively sought” and “Messengers are not punished when they deliver news of failures or other bad news” on a seven-point agree/disagree scale, then averages them into a single Westrum culture score. Westrum’s model describes cultures as pathological, bureaucratic or generative.
That is not a soft add-on. On the same page, DORA reports that “a high-trust, generative culture predicts software delivery and organizational performance in technology,” and cites its 2019 report finding that psychological safety predicts software delivery performance, organizational performance and productivity. So the human side of the work can be a cause of the delivery numbers, not just a nice-to-have alongside them. Our psychological safety guide covers how to build and measure that part.
Perception data beats system data in three situations:
- When the thing you care about has no log. Trust, morale, whether people raise bad news, whether the mission is clear. No pipeline records these. The SPACE framework makes satisfaction and well-being a dimension of its own for this reason, and developer experience metrics build on how developers perceive their work.
- When you have no instrumentation yet. A new team, or delivery data spread across tools, can still ask good questions this week. DORA’s metrics guide warns against putting measurement ahead of improvement and suggests it “might be better to start with having conversations.”
- When you need the reason behind a number. This is the gap between what and why. A deployment-frequency chart shows the drop, and a health check shows that “easy to release” went from green to red the same quarter. Neither tells you the cause. That comes from the team talking it through.
System data still wins for counts and durations, such as how often you deploy. People are poor stopwatches.
Health check vs maturity model
Both use a team survey and a rating scale, so they get confused. They answer different questions.
| Health check | Maturity model | |
|---|---|---|
| Question | How does it feel to work here right now? | How developed are our practices? |
| What is rated | Dimensions of team life (speed, quality, mission, fun, support) | Specific practices against written level descriptions |
| Scale | Traffic light or agree/disagree | Levels, usually ad hoc to optimized |
| Typical cadence | Usually quarterly | Every six to twelve months |
| Output | A trend and a conversation | A gap list and an improvement plan |
| Main risk | Teams compared on color counts | Levels chased as targets |
If a health check keeps flagging the same delivery dimensions quarter after quarter, that is usually the cue to zoom out to a maturity model and look at the practices underneath. For running health checks themselves, see the team health check guide.
What a DevOps maturity model assesses
A DevOps maturity model rates the practices that take a change from commit to production and keep it running. Most models group them into a handful of areas. The Continuous Delivery Maturity Model, published on InfoQ in 2013 by Andreas Rehn, Tobias Palmborg and Patrik Boström, uses five: Culture & Organization, Design & Architecture, Build & Deploy, Test & Verification, and Information & Reporting.
Within each area, the model describes what a practice looks like at each level. “Deployment” at a low level might read as a manual, documented runbook; at a high level, as an automated, low-risk, repeatable release any team member can trigger. The team reads the descriptions, agrees where it sits, and notes the evidence.
That is the difference from the DORA metrics. DORA measures outcomes of the delivery system (how fast, how stable). A maturity model rates the practices that produce those outcomes. The two pair well: the maturity model suggests what to change, and the outcome metrics show whether the change worked.
Common level schemes
There is no single standard scale. Two shapes are common.
CMMI-style levels. CMMI’s maturity levels, published by the CMMI Institute, run from 0 to 5:
| Level | Name | CMMI’s description |
|---|---|---|
| 0 | Incomplete | Ad hoc and unknown |
| 1 | Initial | Unpredictable and reactive |
| 2 | Managed | Managed on the project level |
| 3 | Defined | Proactive, rather than reactive |
| 4 | Quantitatively Managed | Measured and controlled |
| 5 | Optimizing | Stable and flexible |
CMMI separates these staged maturity levels from capability levels (0 to 3), which rate one practice area at a time. That distinction is worth keeping in a DevOps model: rating each capability on its own tells you more than one overall level.
Practitioner levels. The Continuous Delivery Maturity Model uses Base, Beginner, Intermediate, Advanced and Expert. Its authors state that the levels are “not strict and mandatory stages that needs to be passed in sequence”, and advise keeping maturity roughly even across categories, since the categories affect each other.
TeamRetro’s DevOps Maturity Assessment follows the per-capability pattern: the team rates 15 capabilities in five groups on a five-level scale from Ad Hoc to Optimized.
The criticism: maturity models reward box-ticking
A pointed critique comes from the researchers behind DORA. In Accelerate (Forsgren, Humble and Kim, 2018), a section titled “Focus on capabilities, not maturity”, excerpted on TechTarget, makes four points:
- Maturity models help an organization “arrive” at a mature state and declare the journey done, where improvement should be continuous.
- “Maturity models assume that ‘Level 1’ and ‘Level 2’ look the same across all teams and organizations.” They don’t.
- Maturity models often measure tooling or proficiency without tying it to outcomes. Capability models focus on the capabilities that drive outcomes.
- Maturity models define a static target. Capability models allow for a changing environment.
How to use a DevOps maturity model well
None of this means throwing the model out. Written descriptions of good practice give a team a shared vocabulary. The fixes are in how you run it.
- Let the team rate itself. Ratings by an outside assessor turn the model into an audit. Self-rating with evidence (“we say Level 3 because a release needs one approval and runs in under an hour”) turns it into a conversation.
- Rate capabilities, not the team. Report a profile across capabilities, never one overall level. The gaps are the useful part.
- Pick one or two capabilities to move. Not “get to Level 4”. Choose the capability that is the current constraint, agree what the next level looks like for you, and stop there.
- Check it against outcomes. If trunk-based development “moved up a level” but change lead time didn’t move, the rating was optimistic or the capability wasn’t the constraint. Pair the model with DORA or flow metrics.
- Never compare teams on levels. Different systems, different constraints. Compare a team with its own last assessment.
- Revisit every six to twelve months. Practices change slowly, and a quarterly maturity rating tends to measure mood more than practice.
Well-known public models
The Spotify Squad Health Check (2014)
Henrik Kniberg and Kristian Lindwall published the Squad Health Check model on Spotify’s engineering blog in September 2014. Squads rate ten default indicators (Easy to release, Suitable process, Tech quality, Value, Speed, Mission, Fun, Learning, Support, and Pawns or players) as green, yellow or red, plus an arrow for whether each is improving or getting worse. The authors suggest quarterly as a starting cadence and prefer face-to-face workshops to online surveys, because a survey “cuts out the conversation, and that’s the biggest part of the value.”
They are just as clear about the limits. It is “a support tool, not a judgement tool”; it “only works in a high-trust environment”; and “Beware of comparing teams.” It is a health check, not a maturity model: there are no levels to climb.
DORA’s capabilities
DORA publishes a catalog of capabilities that its research links to better software delivery and organizational performance, among them continuous integration, trunk-based development, test automation, loosely coupled teams, generative organizational culture and well-being. DORA’s research page describes the DORA Core Model as “a collection of capabilities, metrics, and outcomes that represent the most firmly-established findings.”
It is a capability model, not a level scheme: there’s no Level 3 to reach, only the capability that constrains your team now.
The Five Ideals (Gene Kim)
The Five Ideals come from Gene Kim’s novel The Unicorn Project (IT Revolution, 2019), and IT Revolution’s summary lists them as Locality and Simplicity; Focus, Flow, and Joy; Improvement of Daily Work; Psychological Safety; and Customer Focus. They describe the conditions developers need rather than practice levels, which makes them a good fit for an agree/disagree survey with one statement per ideal.
If your question is about agile practices rather than the delivery pipeline, TeamRetro’s assessment page and its companion article on measuring and improving agile maturity cover that side.
Frequently asked questions
What is a DevOps maturity model?
A DevOps maturity model rates a team’s delivery practices, such as integration, testing, deployment, operations and security, against a defined scale of levels, usually from ad hoc to optimized. It answers how developed the practices are, not how fast the pipeline runs or how the team feels. Its value is the conversation about evidence for each rating and the one or two capabilities the team chooses to improve next.
What are the levels of a DevOps maturity model?
There is no single standard. Many models borrow a five-step shape from CMMI, whose maturity levels run Initial, Managed, Defined, Quantitatively Managed and Optimizing (with Level 0, Incomplete, below them). The Continuous Delivery Maturity Model uses Base, Beginner, Intermediate, Advanced and Expert. Treat the levels as descriptions of practice to discuss, not as stages to pass in order.
What is the difference between a health check and a maturity model?
A health check asks the team how things are right now, on dimensions like speed, quality, mission and fun, and it is subjective by design. A maturity model rates specific practices against written level descriptions. The health check is a thermometer you read often; the maturity model is a map you revisit every six to twelve months. Most teams get more from using both than from either alone.
Are maturity models still useful?
Yes, with care. The authors of Accelerate argue for focusing on capabilities rather than maturity, because maturity models treat improvement as a destination and assume every team’s Level 2 looks the same. A maturity model stays useful when the team rates itself, discusses the evidence, picks one or two capabilities to improve, and checks the result against outcome metrics such as the DORA measures.
Related reading
- Agile and engineering metrics: the hub, with a map of every framework in this guide.
- What are DORA metrics?: the outcome measures to pair with a maturity model.
- The SPACE framework: where perception data sits alongside system data.
- From numbers to actions: turning a rating or a metric into one change the team makes.
- Team health checks: how to run one, which questions to ask, and what to do with the trend.