Refactoring Safely with AI

The number everything follows from

There is one measurement that does exactly what ought to be done here: instead of running tests, it compares the behaviour of the original and the transformed code across thousands of random inputs. One to two thousand inputs per program, black box, with six different models.

The last row is the point. The trouble is not that the model makes mistakes — it is that more than a fifth of its mistakes are not caught by the existing test suite.

Let us work out what that means:

non-equivalent refactoring:               22.1%
what the test suite does NOT catch:      21.65%

silently changes behaviour AND is green:
   0.221 × 0.2165 = 0.0478  ->  ~4.8%

on harder tasks:
   0.321 × 0.2165 = 0.0695  ->  ~7.0%

(Our own calculation; both base figures are published. The measurement is a preprint.)

One transformation in twenty quietly does something other than what it did before. This is the kind of fault that does not crash but returns a wrong number — and surfaces weeks later.

The trend is improving, but it is not zero

There is a larger-sample, peer-reviewed study as well: 8,096 refactored versions from more than a thousand Java and Python snippets, with four models.

The result: 928 refactored snippets showed a behavioural difference, 180 of them failing outright. The trend is encouraging, though: with the 2023 models the rate was about 17%, with the 2025 ones 6.06%.

And there is a measured mitigation strategy that brought the difference rate down from 17.17% to 4.71% — an improvement of more than twelve percentage points.

Six percent, then, is today's starting point. It is improving, but it is a long way from zero — and we are talking about exactly the kind of fault the user never notices, only the accounts do.

At what diff size does it break down?

There is no continuous measurement for this question — anyone quoting a threshold number is making it up. There are two well-defined points, though, and the difference between them speaks.

• On small, typically single-file fixes, the best systems reach a resolution rate of about 74%.
• On large diffs — where a third of the tasks touch more than ten files and a third change more than two hundred lines — the same models manage 41.2%. The reference fixes here touch 11.4 files and 261.6 lines on average; the most complex case, 182 files.

The same models, roughly half the result.

A benchmark built specifically for refactoring confirms it: 1,099 behaviour-preserving transformations written by developers from eighteen Java projects, split into atomic and composite cases. The best model reached 41.6%, the open-weight ones stayed under 2%, and an agent managed a mere 39.4% on the composite transformations. The authors' summary: today's models "remain effective on local, atomic edits but perform poorly on cross-file and composite transformations".

The real bottleneck: the finding

Here is a small but important distinction that dissolves a lot of confusion.

In one tool vendor's measurement, on a transformation touching seven files the plain agent modified two files. The same task with symbol-level navigation: all seven.

So the constraint is not the editing, it is the finding. The agent transforms what it sees — and it does not see some of the call sites. A refactoring that fixes five places and misses two is worse than not having touched it at all. Where the boundary of that lies is mapped out by the understanding a legacy codebase post, in detail.

Why this matters now in particular

According to a study analysing 623 million code changes between 2022 and 2026, refactoring as a practice is in free fall:

The company's summary: developers today are roughly four times more likely to copy than to move.

(Data from a commercial code analytics company, not peer-reviewed; "moved code" is a heuristic, not a direct measurement. The sample size and the consistent direction across several indicators are serious, though.)

The lesson is not that AI cannot refactor. It is that writing something new looks cheaper — and so the rearranging never happens. Somebody, however, is going to pay the duplication bill.

The order of checks

This is the practical core of the post. Six steps, each with the data above behind it.

1. CHARACTERIZATION TEST FIRST.
   A test that records not WHAT OUGHT TO BE THERE but
   WHAT IS. Refactoring is behaviour-preserving by
   definition - so the current behaviour is what has to
   be set in concrete before you touch anything.

2. COMPILATION + THE FULL TEST SUITE.
   Necessary, but NOT enough: it lets through
   ~21.65% of the broken refactorings.

3. AST-LEVEL CHECK: did the requested transformation happen?
   There is tooling for this; on Java it identifies which
   refactoring took place at 99% precision and 94% recall.
   This is what catches the model "improving things a bit".

4. MUTATION TESTING.
   For refactoring this is a MEANINGFUL signal - because you
   may assume the starting code is good. (Not so in debugging.)

5. DIFFERENTIAL TESTING / FUZZING.
   One to two thousand inputs, old and new code side by side.
   This is what catches the ~5% the tests do not.

6. SMALL DIFF.
   There is no concrete threshold. What we know: past 10+ files
   and 200+ lines the success rate falls from 74% to 41%.

There is a neat connection at the first point that is worth spelling out. It is a known phenomenon that if you put the existing code in the prompt, the model writes assertions for the current behaviour — in debugging that is a disaster, because it sets the bug in concrete. For a characterization test, though, that is exactly what you want. The same tendency, with the opposite sign. From the debugging side the debugging with AI post goes into it further; from the testing side, the safety net of tests post carries it on.

Common misconceptions

• "The tests are green, so the behaviour did not change." More than a fifth of broken refactorings pass the existing test suite.
• "Today's models can do this already." The rate of behavioural difference fell from 17% to 6% in three years. Six percent is not zero.
• "It breaks down above five hundred lines." There is no data for such a threshold. There are two measured points: ~74% on small single-file fixes, 41% above ten files and two hundred lines.
• "The agent cannot rename everywhere." It can — if it finds the places. The documented limit is the finding, not the editing.
• "Code review will catch it." A silent behaviour change is precisely what a diff does not show. For that you have to run it.

What to take away

In refactoring the risk is not the prompt, it is the order of your checks. First record what the current behaviour is, then let the model at it — and know that the full test suite on its own lets a fifth of the faults through. What is worth adding on top: a structural check that the requested transformation is really what happened, and a differential run that catches the few percent nothing else will.

The steps of the human review of a finished change are what the reviewing AI-written code post walks through end to end.