Diff code by behavior instead of by text

A behavioral code diff runs two implementations against the same generated inputs and reports the inputs on which their results differ, instead of comparing their source text line by line.

Compare two snippets

JavaScript, TypeScript and Python run in your browser. Nothing to install, no account.

Why a text diff answers a different question

The diff tools most people already use (git diff, the side-by-side view in an editor, the online diff checkers) compare characters. That suits reviewing a change. It does not answer does this still do the same thing, and the two questions disagree in both directions:

• Large diff, identical behavior. Rewriting a for loop as a reduce changes nearly every character, so a text diff shows two completely different functions. A correct rewrite still returns the same value for every input.
• One-character diff, broken behavior. Changing lo <= hi to lo < hi in a binary search is a one-character diff that is easy to miss in review. It is also a bug that returns the wrong answer for a whole class of inputs.

The size of a text diff carries almost no information about the size of a behavioral change. A behavioral diff compares the behavior you care about, on the inputs it generates.

Text diff, AST diff, behavioral diff

These three terms are often used interchangeably, but they mean different things.

• Text diff compares characters or lines. Reformatting breaks it.
• AST diff compares parsed syntax trees, so it survives reformatting and renaming. It still reasons about structure, so a loop and a reduce stay different to it, and it cannot compare two different languages at all.
• Behavioral diff, which is what Equivl does, ignores both the text and the structure, and compares what the code did: return values, thrown errors, and optionally the intermediate values each version emitted along the way.

Because it only ever looks at inputs and outputs, a behavioral diff works across languages. Comparing a Python function against its JavaScript rewrite is the same operation as comparing two JavaScript functions.

What a run gives you

Both versions receive the same positional arguments, drawn from a seeded pseudo-random generator, so a run is reproducible and a share link reproduces it exactly. You get both results per case, side by side, and a verdict for the run as a whole. When a run diverges, shrinking minimises the failing input, turning “case 47 of 100 diverged on a 30-element array” into “they disagree on [0, -1]”.

When a text diff is still the right tool

A text diff is the right tool for reviewing a change, reading history and resolving a merge, and a behavioral diff adds little there. Use a behavioral diff when you have two versions that are supposed to agree and you want to know whether they do.