AI in the Pipeline: What Actually Sticks

I gave AI access to our pull requests about eighteen months ago. Some of it became load-bearing infrastructure the team would fight me to remove; some of it got disabled within a month. The dividing line was never "is it smart" — it was whether the tool made a verifiable claim or an unverifiable opinion.

Here's my honest ledger of what stuck, what didn't, and the one security problem nobody talks about enough.

What stuck: reviewing the boring 70%

AI code review on every PR was the change I expected to hate. I was sure it would nitpick formatting and hallucinate bugs. Instead it caught the class of mistakes humans skim past: an unclosed resource in an error path, a query built with string concatenation, a test that asserted nothing, a TODO referencing a ticket that closed last year.

The trick was configuration, not magic. Ours reviews with a strict prompt: only comment on correctness, security, and missing tests; no style opinions (the linter already owns style); a hard cap of five comments per PR; silence if nothing substantive. An AI reviewer that writes twelve comments gets muted by developers within a week — ours stays useful because it stays quiet.

Verdict: permanent. Roughly a third of PRs get a real catch from it, and the comments are specific enough to act on.

What stuck: test generation as a first pass

Generated tests are mediocre-to-good, never great — but "mediocre tests written in nine seconds" beats "great tests written never." Our coverage on new modules went up noticeably because the AI drafts the edge cases (empty input, unicode, overflow, timeout) and a human tightens the assertions that matter.

What definitely does not work: letting generated tests merge unreviewed. We had one land that asserted expect(result).toBeDefined() — technically true, entirely useless. Now generated tests face the same review as everything else.

Verdict: permanent, as a draft.

What stuck: flaky test triage and changelogs

The unglamorous winners. Every Monday an agent clusters the week's flaky-test failures, links suspect commits, and posts a digest. It's the job nobody wanted and it's good at pattern-matching across hundreds of runs.

Same for release notes: it reads the merged PR titles, drafts the changelog, and a human edits for tone. Twenty minutes of tedium, gone.

Verdict: permanent.

What got turned off: full-autonomy deploys

We tried agentic deployment — AI watches the deploy, reads the errors, decides whether to roll back. It worked beautifully for three weeks, then confidently rolled back a successful deploy because it misread a 410 as a server error. The blast radius was small; the trust damage wasn't.

Autonomy without a verifiable ground truth is gambling with extra steps. Alarms, health checks, and SLOs are verifiable; "the graphs look fine to me" is not.

Verdict: off. AI proposes, humans approve. Rollback stays one human click.

What got turned off: uncriticized dependency suggestions

An agent proposed a dependency to solve a tricky problem. The package name was one character off from a real one — I won't describe it further because that's exactly the supply-chain trick. Our scanners wouldn't have caught a brand-new squatter package on day one.

Now the rule is simple: AI may suggest dependencies; a human verifies the publisher, download counts, repo activity, and license before anything reaches a lockfile. Same for any command the AI wants to run in CI — allowlisted verbs only, no arbitrary shell.

Verdict: off, permanently, without review.

The security problem: prompt injection in your CI

This is the one I nag about. Your AI tools read issue titles, PR descriptions, README files, and linked web pages — content anyone can write. That's an injection surface pointed straight at machines with repo and deploy credentials.

What we do about it:

• The reviewer agent has read-only repo access, no secrets, no contents: write.
• Anything that executes (deploy-adjacent agents) runs with scoped, short-lived OIDC credentials — never long-lived PATs.
• Web-fetching tools run in a sandbox with no credential environment variables present. Ever.
• Assume any text from an issue is hostile until proven otherwise. A bug report is user input.

The filter I use for every new AI feature

Three questions: Does it make a verifiable claim? (test results, log patterns — yes; vibes — no.) What's the blast radius when it's wrong? (a comment is cheap; a rollback is not.) Is a human in the loop proportional to that blast radius?

Most "AI in DevOps" tools fail the first question. The ones that pass, plus a sane blast radius, have earned a permanent seat in our pipeline — and the ledger above is roughly what yours will look like after eighteen months, too.