The adaptive interpreter: how CPython rewrites itself as it goes

The interpreter edits itself as it goes

The usual picture is this: the compiler turned source into bytecode, and then the interpreter loop runs that bytecode. It is accurate right up to Python 3.11.

From 3.11 there is a third participant. While the code runs, the interpreter watches what exactly passes through it and swaps instructions for narrower ones. It was BINARY_OP — "add anything to anything, work it out as you go". It became BINARY_OP_ADD_INT — "add two integers", no working out.

You can check this in one line, without reading CPython's C: dis has an adaptive flag that shows instructions in their present form.

def attrs(fn):
    return [
        i.opname
        for i in dis.get_instructions(fn, adaptive=True)
        if i.opname.startswith("LOAD_ATTR")
    ]
 
print(attrs(read))    # ['LOAD_ATTR']
for _ in range(500):
    read(p)
print(attrs(read))    # ['LOAD_ATTR_INSTANCE_VALUE']

The same function, the same code object. It changed itself.

It happens almost immediately

The word "specialisation" drags in an expectation of a long warm-up — that is how the JVM works, compiling hot code after thousands of iterations. Python is not like that.

The measurement is simple: a fresh piece of code is executed one run at a time, and after each run it is asked what its instruction is called now.

The practical conclusion: a function called twice is already running on specialised instructions. In particular, any timeit measures the steady state rather than the warm-up.

A place specialises, not an operation

This matters more than it sounds. The specialisation is recorded not "for addition in general" but for one particular line in one particular function. Which is why the same o.x gives different instructions:

Hence a non-obvious consequence: __slots__ is not only about saving memory. It also changes the instruction the attribute is read with.

The check costs money

A specialised instruction has to make sure its assumption still holds — otherwise it would return a wrong answer. That comparison runs on every pass, and if different types go through one place, it fails regularly.

Five classes declared identically do the same thing. The only difference is how many types pass through one line of o.x:

Half again in overhead for nothing — on this benchmark: five classes through one o.x. The cure is not type annotations but splitting hot code so that one place sees one type. Profile first, though: without that you do not know this is the hot place.

What about the JIT?

Since 3.13 CPython has a second mechanism too — a JIT. It does something different: it replaces a whole sequence of instructions rather than one.

The main thing worth knowing about it: you cannot judge it by a version number. In 3.14 it is built but off; you can ask the interpreter directly:

import sys
print(sys._jit.is_available(), sys._jit.is_enabled())   # True False

It is switched on with the PYTHON_JIT=1 environment variable. On four short loops run six times each per mode on one and the same build, switching it on made all four slower: from +10.7% on the dictionary to +36.7% on the list walk. On this code and this machine, "turn the JIT on and it gets faster" did not hold.

What is going on

The mechanism is described in PEP 659 and inside the project itself — in CPython's internal documentation, and the second wording is the more precise:

The key word is rewriting. This is not a compiler optimisation: the compiler emits a plain BINARY_OP knowing nothing about types. The rewriting happens later, at runtime, and the code does it to itself: Python/specialize.c has a _Py_Specialize_* function for each family of instructions.

You can check this without reading any C. dis has an adaptive flag that shows instructions in their present form:

import dis
 
def read(o):
    return o.x
 
class Point:
    def __init__(self):
        self.x = 1
 
def attrs(fn):
    return [
        i.opname
        for i in dis.get_instructions(fn, adaptive=True)
        if i.opname.startswith("LOAD_ATTR")
    ]
 
p = Point()
print("before warm-up:", attrs(read))
for _ in range(500):
    read(p)
print("after:         ", attrs(read))

before warm-up: ['LOAD_ATTR']
after:          ['LOAD_ATTR_INSTANCE_VALUE']

The same function, the same code object. It changed itself.

Where the counter lives

To decide when to specialise, an instruction needs somewhere to keep a counter, and the specialised version needs somewhere to keep what it believes: the type version, the attribute offset, a pointer to a descriptor. That space sits inside the bytecode array, right after the instruction:

And this too is visible from outside, with the show_caches flag:

LOAD_ATTR_INSTANCE_VALUE 0 (x)
CACHE                    0 (counter: 832)
CACHE                    0 (version: 5833536)
CACHE                    0
CACHE                    0 (keys_version: 5833536)
CACHE                    0
CACHE                    0 (descr: 8595768128)
CACHE                    0
CACHE                    0
CACHE                    0

Nine two-byte cells for one LOAD_ATTR — and by the family's rule the first of them is always the counter. Which, incidentally, answers the question everyone asks on first seeing this output: CACHE is not an instruction. The interpreter does not execute it, it steps over it; it lives in the array because that puts the data next to the code that reads it.

The cache size is the same for every member of a family — a correctness requirement: LOAD_ATTR, LOAD_ATTR_SLOT and LOAD_ATTR_MODULE must occupy the same room in the array, or substituting one would shift all the code after it.

What turns into what

Eleven one-line operations, warmed with five hundred calls. On the left what the compiler emitted, on the right what it became (Python 3.13.7):

Look at two adjacent rows: o.x turns into different instructions depending on whether the attribute lives in the instance dictionary or in a slot. That is what "the case that this program encounters" means — what specialises is not an operation in general but a particular place in a particular function.

On 3.13 there are fifteen such families and seventy-four variants inside them. The largest family is CALL, with twenty.

When

The answer is unexpectedly soon:

The measurement is direct: a fresh code object is executed one run at a time and asked for its instruction name after each. On 3.12 and later the name changes after the second pass.

The practical conclusion matters more than the number. "Warm-up" in Python is not thousands of iterations, as in the JVM. Any function called twice is already running on specialised instructions. A microbenchmark that timeit runs with number=1000 measures the steady state, not the warm-up — good news for measuring, bad news for anyone hoping to catch the interpreter cold.

What if the assumption does not hold

The internal documentation promises a revert:

Let's measure it. Five classes declared identically do the same thing; one line of o.x in a loop; the only difference is how many types pass through it:

There are two observations here, and the second contradicts what I expected.

First: a polymorphic call site costs money. Half again in overhead for nothing, given that the work is identical and the classes indistinguishable. That is the price of the check: the specialised instruction compares the type version, and on a foreign type the comparison fails and the generic path has to run.

Second: the instruction does not revert. I expected to see a plain LOAD_ATTR in the last row — and never did. Across the whole run the instruction was rewritten once (generic to specialised) and did not change again, however many types went through it. So "de-optimizing back to the generic version" describes what happens inside the instruction's execution when the check fails, not the bytecode array getting its generic opcode back. Telling one from the other takes a measurement: a retelling of the PEP makes both pictures equally plausible.

One caveat, without which the conclusion would be wider than the measurement: this holds for the LOAD_ATTR family and this scenario. BINARY_OP behaved differently when the type changed — one specialised instruction was replaced by another specialised one fitted to the new type (BINARY_OP_ADD_INT → BINARY_OP_ADD_FLOAT).

Tier two

Since 3.13 CPython has a second mechanism, and it is constantly confused with the first. The internal documentation names the difference in one sentence:

The same page names what counts as hot: the JUMP_BACKWARD instruction — the end of a loop iteration — looks at the counter in its own inline cache and, once past a threshold, asks the optimiser to build a trace.

Whether it is running for you is not a question about a version number. sys._jit answers it:

import sys
print(sys._jit.is_available(), sys._jit.is_enabled(), sys._jit.is_active())

On the build this section was measured on (3.14.7) it is True False False: built, but off. It is switched on with the PYTHON_JIT=1 environment variable.

And since the build is the same and exactly one variable differs, comparing times here is legitimate — unlike comparing versions with each other. Four short loops, six independent pairs of runs, best of eleven repeats in each; the percentage is the ratio of the two modes' fastest runs:

The columns hold not one number but the range of six runs — and that is not pedantry: the machine drifts, and on a single pair of runs a four-percent difference would mean nothing. The runs were interleaved (no JIT, JIT, no JIT, …) so that the drift landed on both modes equally, and the conclusion reduces to one question: do the ranges overlap?

They do not overlap for any of the loads: the fastest run with the JIT stays slower than the slowest run without it.

Six pairs rather than three, and not out of caution for its own sake. The conclusion here rests not on the size of the difference but on the ranges never overlapping, and three pairs are too few to say that.

What these numbers do NOT mean: a verdict on the JIT. Four short loops on two cores in a container are neither pyperformance nor a real program. What they license is one statement: switching tier two on does not by itself guarantee a win, and it has to be checked on your code rather than by version number.

What follows for your code

Three conclusions, each a direct consequence of what was measured above.

Monomorphic call sites are cheaper than polymorphic ones. Not because "Python likes types" but because a specialised instruction checks exactly one assumption. A function that five different classes pass through in a hot loop pays half again — and the cure is not annotations but splitting it into two call sites. That one-and-a-half is the result of a LOAD_ATTR benchmark over five classes, not the price of polymorphism in general: another instruction family and another workload give their own number. And the order of work is the usual one — profile first, reshape the call site after, not the other way round.

__slots__ gives you a different instruction, not just memory savings. LOAD_ATTR_SLOT against LOAD_ATTR_INSTANCE_VALUE are different specialisations, and the choice between them is made once, on the first warm-up.

Microbenchmarks measure a warmed interpreter. Since substitution happens on the second execution, any timeit with more than two repeats is already running specialised code. Measuring the cold start with timeit is not possible — that needs a fresh code object per measurement.

Version history

What measured this

The numbers in this article come from these scripts. Each one opens from here, together with the record of the run: what it was measured on, what came out, and with what spread.

• bench/adaptive/cost.py
• bench/adaptive/jit.py
• bench/adaptive/specialize.py