DevOps and Infrastructure with AI

The number to start from

In a peer-reviewed measurement covering 458 real-world scenarios, a strong model solved 19.36% of infrastructure-as-code tasks on the first try. The same model scored 86.6% on an established Python task set.

The model did not suddenly get stupid. Infrastructure is a different job: the question here is not whether it runs, but what it allows.

Two further 2026 measurements point the same way:

• On 153 real tasks, six frontier models produced a successfully deployable configuration on the first try 20.8–30.2% of the time — and by established compliance rules, only 8.4% followed the rules.
• According to a study covering 17 models, HCL syntax is harder for models than YAML or JSON.

Three to four times denser defects than in human code

There is a study that does exactly what needs doing: it compares 1,196 machine-generated infrastructure definitions against 634 human-written templates, using three independent policy-checking engines.

Size-adjusted, in every setup examined, the vulnerability density of generated configuration is 3.21–3.87 times that of human configuration.

And there is a counter-intuitive detail inside it: the gap is 4.9x at a single resource, but only 1.4x at twenty resources or more. In other words, the short snippet you grabbed in a hurry is the dangerous one, not the long, deliberated configuration. Exactly the one people paste fastest.

("Extended thinking" settings are roughly 12% better than standard generation — it helps, but it does not solve it.)

It can run fine and still be wide open

This is the most important practical distinction in the whole post.

A 2026 measurement examined seven models across seventeen scenarios, with two policy checkers wired into the pipeline. Two extremes out of the results:

• One open-weight model reached a 77.8% validity rate — meaning the configuration was syntactically sound and the command ran. Compliance rate: zero.
• For one strong closed model, one checker put the pass rate at 23.1% and the other at 92.5% — meaning the choice of checker is itself decisive.

The study's conclusion, verbatim: "prompting alone is not enough: automated, multi-tool scanning remains a necessary complement."

In plain terms: a clean validation run is not a security proof. A wide-open bucket is just as valid a configuration as a locked-down one.

The four recurring failure patterns

In generated infrastructure definitions the same four things come back again and again:

1. Permissive defaults — wildcard permissions, public access.
2. Missing security blocks — encryption and logging are simply not there.
3. Hard-coded credentials.
4. Outdated patterns — deprecated syntax, unversioned images.

None of these is a "bug" in the sense that something crashes because of it. All four work quietly — right up until somebody finds them.

The measurement that took it from 14% to 93%

This is the practical core of the post, and the result is strong enough to be worth walking through.

A 2026 study built a three-stage check into 186 tasks: validity → planned change → policy evaluation. Then it added one element at a time:

And the key sentence: "79% of the compliance failures after refinement stem from missing information and can be resolved when the policy text is visible."

So four fifths of the compliance failures are not a capability gap: the model simply did not know the rule.

The practical recipe follows directly: do not polish the prompt — put the policy text in, and wire the scanner into the loop. Of the jump from 14 to 93 percent, 21.5 points really did come from switching models — but the remaining 57.5 came from the retrieved documentation, the prompt structure, the iterative loop and the policy text, and that last one still added 8.6 points under the strongest model.

AIOps: very readable, moderately accurate

Incident management is the other area where the promise is large. The best measurement on it comes from a big cloud provider, and it is unusually honest.

The data: roughly 180,000 high-severity historical incidents, with testing on 2,900 real incidents. By machine evaluation, the model with the optimised prompt is 21% better than the manual baseline.

But the point is in the human evaluation, in what 47 on-call engineers thought:

This is the most dangerous combination there is. At three in the morning the on-call engineer gets a summary that is convincingly phrased, cleanly structured and half right. What they see is not uncertainty — it is confidence.

For alert-noise reduction and shorter time to repair, we found no independent measurement published with a methodology. What exists is vendor promises.

What does get measured: AI is itself a source of incidents

A reliability analysis processed 177,960 status-page entries between 2018 and June 2026:

From the same source: third-party, vendor incidents make up more than a quarter of the total, and their median time to resolve is 247 minutes — against 96 minutes for your own failures. What you do not run, you do not fix.

Concrete, documented cases: an agent ran an infrastructure-destroying command and 1.94 million rows were lost; a delete-and-recreate strategy caused a 13-hour outage; in another case "the production database and every backup were deleted in a 9-second API call".

And the other side: the same tool resolved an outage

For balance, a public postmortem from March 2026 is worth reading too.

The causal chain is a classic: a race condition in continuous integration skipped a build step, so the deployment ran with the wrong artifact mapping, and the service stopped accepting data for eight hours.

And here AI was not the cause but the tool for finding it. The company's own monitoring agents flagged the failure, and then — in their own description — by pointing a coding agent at the bug report, with cloud and version-control access, they located the broken deployment, rebuilt the image and re-ran the deployment scripts.

Their conclusion was that they will give the agent more permissions, not fewer.

The same technology causes an outage in one place and resolves an outage here. The difference lies in where the permissions stop and in whether the action can be undone.

The workflow

1. THE POLICY TEXT GOES IN THE PROMPT.
   79% of compliance failures are missing information.
   The model does not lack skill - it lacks your rule.

2. SCANNER IN THE LOOP, NOT AFTER IT.
   Generate -> scan -> get the failure back -> fix.
   Without the loop a scanner reports, it does not fix.

3. MORE THAN ONE SCANNER.
   On the same configuration one tool measured 23% and
   the other 92%. One tool is not enough.

4. VALIDITY IS NOT SECURITY.
   One model produced valid config 77.8% of the time
   with ZERO compliance.

5. THE SHORT SNIPPET IS THE DANGEROUS ONE.
   At one resource the vulnerability density is 4.9x the
   human one; at twenty, 1.4x. The fast paste is the bad one.

6. NO AUTOMATIC APPROVAL FOR AN IRREVERSIBLE
   OPERATION. Delete, destroy, production database:
   a human, every single time.

7. THE MACHINE INCIDENT SUMMARY IS A HYPOTHESIS,
   NOT A DIAGNOSIS. Readability 4.21/5, accuracy 2.91/5.

Common misconceptions

• "Validation passed, so it is fine." One model had 77.8% validity with a compliance rate of zero.
• "One scanner is enough." Two tools measured 23% and 92% on the same code.
• "The long, complicated configuration is the risky one." The other way round: the gap is biggest at a single resource.
• "A better model will fix it." A stronger model does a lot — 21.5 points in this measurement — but it does not finish the job: four fifths of compliance failures are missing information, and even under the best model the text of the rule supplied the last 8.6 points.
• "The AI incident summary tells me what is wrong." Very readable, moderately accurate — and that is exactly what makes it convincing at three in the morning.

What to take away

With infrastructure the question is not whether the configuration runs but what it allows — and on that, generated code is measurably worse than human code. The gain, though, does not come from one step: in the measurement that went from 14% to 93%, the model switch accounts for 21.5 of the 79 points and the rest for the retrieved documentation and the policy and scanner wired into the loop. And for irreversible operations, keep a human in the loop — that is the one thing every documented outage has in common.

Setting up agent permissions is covered in the three modes of AI coding post, mode by mode; the wider security picture is the subject of the security bill of AI code post, with the numbers.