Testing: unittest, pytest and mock — what is really being checked

What it was checked on: the runs of bench/testing/basics.py and bench/testing/practice.py — CPython 3.11.15, 3.12.3, 3.13.7 and 3.14.7, linux/amd64; the runs with pytest — CPython 3.13.7 and pytest 8.4.2. The output blocks in the text are the 3.13.7 record, with labels and the scripts' Russian strings translated; differences between versions are in the "Version history". The docstrings were checked on 14.10.2026. By that date Python 3.15 was out (9 October 2026); the lesson was not run on it.

Base: what a test is and how unittest differs from pytest

A fragment: add is the function under test, defined elsewhere. A full runnable example is below, in the section "A full example".

# unittest — the standard library
import unittest
 
class AddTest(unittest.TestCase):
    def test_add(self):
        self.assertEqual(add(2, 2), 4)
 
# pytest — a third-party package, plain functions and assert
def test_add():
    assert add(2, 2) == 4

A junior-level answer to "how do you test": I write tests with pytest or unittest: I call a function and check the result; I replace external dependencies with mocks; tests should be independent and fast. That is correct, and it is followed by questions: how does a failed test differ from a broken one, where do you patch a function, what happens if you misspell the name of a check, and in what order do tests run. The mechanisms below answer them with a run.

Mechanism 1: passed, FAIL and ERROR

Output of bench/testing/basics.py, section 1 — four unittest tests of the function add; labels translated:

1) unittest: FAIL versus ERROR
   test_bare_assert -> FAIL: AssertionError
   test_errors      -> ERROR: TypeError: unsupported operand type(s) for +: 'int' and 'str'
   test_fails       -> FAIL: AssertionError: 4 != 5
   test_passes      -> ok
   total: tests 4, failures 2, errors 1

FAIL — the check ran and did not match: the code returned 4, and 5 was expected. For unittest a FAIL is any AssertionError, including one from a bare assert, as in test_bare_assert. That is the signal "the code does the wrong thing". ERROR — any other exception; here a TypeError was raised before the check. That is the signal "the test or the code is broken". unittest counts FAIL and ERROR separately — failures and errors — and when going through failures it is worth looking at errors first: the check in such a test never even ran.

pytest uses the words differently: an exception in the test body is also FAILED, and ERROR appears when a fixture breaks — in the pytest output in the "Deeper" section it is a setup failure.

Mechanism 2: Mock — an object that agrees to everything

Output of bench/testing/basics.py, section 2; labels and the script's Russian strings translated:

2) Mock: attributes, calls, return_value, side_effect
   api.get_user(1)                -> Mock
   api.get_user.call_args         -> call(1)
   api.get_user.call_count        -> 1
   assert_called_once_with(1)     -> passed
   return_value = {...}; call     -> {'id': 1}
   side_effect=[10, 20, exception] -> [10, 20, 'TimeoutError: no reply']

Mock creates an attribute for any name and accepts any call, and a call returns return_value — by default also a Mock, the same one for all calls. It records every call — call_args, call_count — and can check it: assert_called_once_with. Behaviour is set by two fields: return_value — what to return, side_effect — what to do on each call: a list of values in turn or an exception. That is how code that goes to the network is tested: side_effect=[…, TimeoutError(…)] checks that the code survives a timeout, with no network at all.

Mechanism 3: patch — replace where the name is looked up

Two modules return the current time. The first writes import time and calls time.time(), the second from time import time and calls time(). Output of bench/testing/basics.py, section 4; labels translated:

4) patch: replace where the name is looked up
   patch('time.time'):     app_import.now() -> 42.0 | app_from.now() == 42.0 -> False
   patch('app_from.time'): app_from.now()   -> 42.0

patch('module.name') replaces one name in one module for the duration of the test. app_import looks the function up in the time module on every call — the substitute is seen. app_from copied a reference to the real function into its own name time at import and no longer looks into the time module. Hence the rule: patch the name in the module that uses it — app_from.time, not time.time. This follows directly from from … import copying a reference.

Mechanism 4: a spec — spec limits the names, autospec checks the call

PaymentClient has one method — charge(amount, currency). Output of bench/testing/basics.py, section 5; labels translated:

5) a spec: Mock versus create_autospec
   Mock().charge(100)                -> accepted
   autospec.charge(100)              -> TypeError: missing a required argument: 'currency'
   autospec.charge(100, 'RUB')       -> accepted
   Mock().chrage(100, 'RUB')         -> accepted
   autospec.chrage(100, 'RUB')       -> AttributeError: Mock object has no attribute 'chrage'

A bare Mock accepted both the call without a currency and the method with a typo: the test will pass, and the real code will fail. create_autospec builds a mock from the real class: mocked methods will have their arguments checked to check that they are called with the correct signature, and non-existent attributes are an error. The same protection is patch(..., autospec=True). A Mock without a spec is the main source of tests that pass but check nothing.

But "a mock with a spec" means three different parameters with different guarantees, and they are easy to mix up. spec is a list of allowed names: "Accessing any attribute not in this list will raise an AttributeError". spec_set is "a stricter variant of spec": it also forbids assigning new names. The description of neither says anything about the call signature — only create_autospec checks it. Output of bench/testing/basics.py, section 7, every line on a fresh mock; labels translated:

7) Mock, spec, spec_set, autospec: which checks each one switches on
   (PaymentClient has one method charge(amount, currency); a fresh mock for every line)
   attribute existence: m.chrage
      Mock()                                        -> accepted
      Mock(spec=PaymentClient)                      -> AttributeError: Mock object has no attribute 'chrage'
      Mock(spec_set=PaymentClient)                  -> AttributeError: Mock object has no attribute 'chrage'
      create_autospec(PaymentClient, instance=True) -> AttributeError: Mock object has no attribute 'chrage'
   assigning a new attribute: m.extra = 1
      Mock()                                        -> accepted
      Mock(spec=PaymentClient)                      -> accepted
      Mock(spec_set=PaymentClient)                  -> AttributeError: Mock object has no attribute 'extra'
      create_autospec(PaymentClient, instance=True) -> accepted
   call signature: m.charge(100) without currency
      Mock()                                        -> accepted
      Mock(spec=PaymentClient)                      -> accepted
      Mock(spec_set=PaymentClient)                  -> accepted
      create_autospec(PaymentClient, instance=True) -> TypeError: missing a required argument: 'currency'
   a typo in a check: m.was_called_once()
      Mock()                                        -> accepted
      Mock(spec=PaymentClient)                      -> AttributeError: Mock object has no attribute 'was_called_once'
      Mock(spec_set=PaymentClient)                  -> AttributeError: Mock object has no attribute 'was_called_once'
      create_autospec(PaymentClient, instance=True) -> AttributeError: Mock object has no attribute 'was_called_once'
   create_autospec(..., spec_set=True): m.extra = 1 -> AttributeError: Mock object has no attribute 'extra'

The same table in short — every cell comes from a line of the run above:

Hence the precise wording: spec and spec_set limit the names, while a wrong number of arguments is caught only by autospec — create_autospec or patch(..., autospec=True).

Mechanism 5: a typo in the name of a check

Since Mock creates an attribute for any name, a typo in a check method turns into an ordinary call that checks nothing. Mock catches some typos itself — in the run the mock was never called, so a real check must fail. Output of bench/testing/basics.py, section 3; labels and the script's Russian strings translated:

3) a typo in the name of a check (no call was ever made)
   m.assert_called_once()  -> AssertionError: Expected 'mock' to have been called once. Called 0 times.
   m.asert_called_once()   -> AttributeError: 'asert_called_once' is not a valid assertion. Use a spec for the mock if 'asert_called_once' is meant to be an attribute.
   m.assret_called_once()  -> AttributeError: 'assret_called_once' is not a valid assertion. Use a spec for the mock if 'assret_called_once' is meant to be an attribute.
   m.called_once()         -> AttributeError: 'called_once' is not a valid assertion. Use a spec for the mock if 'called_once' is meant to be an attribute.
   m.was_called_once()     -> passed silently — nothing was checked

Mock does not create names that start with assert, assret, asert, aseert and assrt. Since 3.12 the names of checks without the prefix were added to them: e.g. accessing called_once instead of assert_called_once. On 3.11 the m.called_once() line in the same run says "passed silently". And m.was_called_once() is not caught on any version: it is just a new attribute of the mock. The protection is the same as in Mechanism 4 — a spec: a mock with spec or autospec has no non-existent attributes.

A full example: the result comes first, the mock's call is in addition

Two files — they can be copied into an empty directory and run. The working code, the full file bench/testing/checkout.py (comments and the Russian strings translated; the code is the same):

"""Code under test for test_checkout.py: placing an order with a payment.
 
The lesson's full example "Testing" is two files, checkout.py and
test_checkout.py. Copy both into an empty directory and run:
 
    python -m pytest -v
 
pytest is needed (the run record uses 8.4.2). How a defect in this file
turns the test red — checkout_run.py and runs/checkout_run.txt.
"""
 
 
class PaymentClient:
    """A payment service client; in tests a mock replaces it."""
 
    def charge(self, amount, currency):
        raise RuntimeError("a real payment is not needed in a test")
 
 
def checkout(client, prices, discount=0):
    total = sum(prices) - discount
    client.charge(total, "RUB")
    return total

The tests, the full file bench/testing/test_checkout.py (comments translated):

"""Tests for checkout.py — the full example of the lesson "Testing".
 
Run from the directory with both files:  python -m pytest -v
"""
from unittest.mock import create_autospec
 
import pytest
 
from checkout import PaymentClient, checkout
 
 
@pytest.fixture
def client():
    # A mock built from the real class: a non-existent method is an
    # AttributeError, a charge call with wrong arguments is a TypeError.
    return create_autospec(PaymentClient, instance=True)
 
 
def test_checkout_returns_total(client):
    total = checkout(client, [100, 200])
    assert total == 300  # the main thing: the result of the working code
    client.charge.assert_called_once_with(300, "RUB")  # in addition: the interaction
 
 
@pytest.mark.parametrize(
    "prices, discount, expected",
    [([100, 200], 0, 300), ([100], 30, 70), ([], 0, 0)],
)
def test_checkout_discount(client, prices, discount, expected):
    assert checkout(client, prices, discount) == expected

Run python -m pytest -v in the directory with both files. Output of bench/testing/checkout_run.py; labels translated:

0) as is: a green run
   $ python -m pytest -v --tb=line -p no:cacheprovider --no-header -rN
   test_checkout.py::test_checkout_returns_total PASSED
   test_checkout.py::test_checkout_discount[prices0-0-300] PASSED
   test_checkout.py::test_checkout_discount[prices1-30-70] PASSED
   test_checkout.py::test_checkout_discount[prices2-0-0] PASSED
   ============================== 4 passed in <time> ===============================
   exit code: 0

The order of the claims in test_checkout_returns_total is not accidental. The first — assert total == 300 — checks the result of the working code: that is what the test is written for. The second — assert_called_once_with(300, "RUB") — is an additional claim about the interaction: the code really did turn to the payment client, and with which arguments. A test that checks only the mock's calls repeats the implementation instead of checking the behaviour.

A test is useful only if a defect in the code turns it red. The script breaks a copy of bench/testing/checkout.py and runs the same tests:

A) a defect in checkout: the discount is added
   edit in checkout.py: total = sum(prices) - discount  =>  total = sum(prices) + discount
   $ python -m pytest -v --tb=line -p no:cacheprovider --no-header -rN
   test_checkout.py::test_checkout_returns_total PASSED
   test_checkout.py::test_checkout_discount[prices0-0-300] PASSED
   test_checkout.py::test_checkout_discount[prices1-30-70] FAILED
   test_checkout.py::test_checkout_discount[prices2-0-0] PASSED
   <dir>/test_checkout.py:30: AssertionError: assert 130 == 70
   ========================= 1 failed, 3 passed in <time> ==========================
   exit code: 1

The second defect is client.charge(total) without the currency. With the fixture on create_autospec all four tests turned red: every charge call answered TypeError: missing a required argument: 'currency'. And here is the same defect with the fixture on Mock(spec=PaymentClient):

C) the same defect, the fixture on Mock(spec=PaymentClient)
   edit in checkout.py: client.charge(total, "RUB")  =>  client.charge(total)
   edit in test_checkout.py: return create_autospec(PaymentClient, instance=True)  =>  return Mock(spec=PaymentClient)
   edit in test_checkout.py: from unittest.mock import create_autospec  =>  from unittest.mock import Mock, create_autospec
   $ python -m pytest -v --tb=line -p no:cacheprovider --no-header -rN
   test_checkout.py::test_checkout_returns_total FAILED
   test_checkout.py::test_checkout_discount[prices0-0-300] PASSED
   test_checkout.py::test_checkout_discount[prices1-30-70] PASSED
   test_checkout.py::test_checkout_discount[prices2-0-0] PASSED
   <stdlib>/unittest/mock.py:979: AssertionError: expected call not found.
   ========================= 1 failed, 3 passed in <time> ==========================
   exit code: 1

The three tests that look only at the result stayed green: spec does not check the signature, and the call without a currency went through. Only the additional check of the call caught the defect. That is why dependencies with a signature get create_autospec.

Deeper: test order, setUp and pytest

Output of bench/testing/basics.py, section 6; labels translated:

6) test order and setup
   declared: test_zebra, test_apple, test_mango
   ran: ['test_apple', 'test_mango', 'test_zebra']
   setUpClass: 1 | setUp: 3

unittest sorts method names as strings, so the order is alphabetical, not the order of declaration. More precisely, it is string comparison: test_10 comes before test_2. A test that relies on another test running first is a bug: tests should be independent. setUp prepares the environment before every test, setUpClass once per class; so expensive preparation that tests do not change goes into setUpClass, and mutable state into setUp, so that tests do not affect each other.

pytest does the same more briefly: a plain assert, fixtures instead of setUp, parametrize instead of loops. The full file bench/testing/pytest_examples.py — four tests in it fail on purpose, for the text pytest prints (comments and the Russian strings translated; the code is the same):

"""Examples of pytest tests for the lesson "Testing".
 
The file is deliberately not named test_*.py: pytest will not pick it up by
itself, and four tests here fail on purpose — for the text pytest prints on
failure. pytest_run.py runs it.
"""
import pytest
 
 
def add(a, b):
    return a + b
 
 
# ── assert without methods: pytest breaks the expression down itself ─────
def test_plain_assert():
    result = add(2, 2)
    assert result == 5
 
 
# ── parametrize: one test, several data sets ─────────────────────────────
@pytest.mark.parametrize(
    "a, b, expected",
    [(1, 2, 3), (0, 0, 0), (-1, 1, 0), (2, 2, 5)],
)
def test_add_cases(a, b, expected):
    assert add(a, b) == expected
 
 
# ── a fixture: setup and teardown around a test ──────────────────────────
@pytest.fixture
def resource():
    print("\n   [fixture] setup")
    yield {"connected": True}
    print("   [fixture] teardown")
 
 
def test_uses_fixture(resource):
    assert resource["connected"]
 
 
# ── pytest.raises: an expected exception is a success ────────────────────
def test_raises():
    with pytest.raises(TypeError):
        add(1, "1")
 
 
# ── an exception in the test body and a fixture failure: FAILED vs ERROR ─
def test_type_error():
    add(1, "1")
 
 
@pytest.fixture
def broken():
    raise RuntimeError("the fixture did not come up")
 
 
def test_with_broken_fixture(broken):
    assert broken

The file is not named test_*.py, so pytest finds it only when the name is passed explicitly: python -m pytest bench/testing/pytest_examples.py -v -s (-s so that the fixture's print is visible). That is how bench/testing/pytest_run.py runs it. A fragment of the output (labels translated) — a test with a failed assert, four sets of one parametrize and a test with a fixture; the lines about test_raises and the summary are omitted:

pytest_examples.py::test_plain_assert FAILED
pytest_examples.py::test_add_cases[1-2-3] PASSED
pytest_examples.py::test_add_cases[0-0-0] PASSED
pytest_examples.py::test_add_cases[-1-1-0] PASSED
pytest_examples.py::test_add_cases[2-2-5] FAILED
pytest_examples.py::test_uses_fixture 
   [fixture] setup
PASSED   [fixture] teardown

Each parametrize set is a separate test with its own name and its own outcome. A fixture with yield does the setup before the test and the teardown after — like setUp and tearDown in one function. The order here is the order of declaration in the file: pytest does not sort tests the way unittest does. And on a failed assert pytest breaks the expression down by itself — no assertEqual needed; here is the breakdown of the [2-2-5] set:

>       assert add(a, b) == expected
E       assert 4 == 5
E        +  where 4 = add(2, 2)

And the last two tests of the same run — an exception in the test body and a fixture that itself failed during setup:

pytest_examples.py::test_type_error FAILED
pytest_examples.py::test_with_broken_fixture ERROR

In pytest an exception in the body is FAILED, just like a failed assert. ERROR appears when a test could not start or finish: a fixture broke.

How to answer in an interview

Short answer: a test calls code and checks the result; in unittest FAIL is a check that fired (AssertionError), ERROR is any other exception. External dependencies are replaced with Mock via patch — in the module where the name is used. The main claim of a test is about the result of the code; checking the mock's call is an additional one. Mocks are built from the real object: Mock(spec=…) rejects non-existent names, while a wrong call is caught only by create_autospec or autospec=True. Tests are independent: they do not rely on the order in which they run, state is prepared in setUp or a fixture.

That is enough for a correct answer. What follows is what you add if the interviewer digs.

If the interviewer digs deeper

On "I mocked the function, but the test hits the real service". "Most likely the patch is in the wrong place: the module did from x import f, and x.f was patched. In my run patch('time.time') did not replace the time in the module with from time import time, while patch('app_from.time') did."

On "what is dangerous about Mock". "It agrees to everything. In the run a bare Mock accepted a call without a required argument and a method with a typo. With create_autospec the first is a TypeError, the second an AttributeError. And Mock(spec=…) catches only the second: it accepted the call charge(100) without a currency — spec limits the names, not the signature."

On "why is pytest better than unittest". "Less code: a plain assert that pytest breaks down itself on failure, fixtures with yield for setup and teardown, parametrize — each data set as a separate test."

On "in what order do tests run". "In unittest — alphabetically by method name; in my run test_apple ran before test_zebra, although it was declared later. The order cannot be relied on — tests should be independent."

Next they ask

Common misconceptions

Version history

Practice

Two problems. Answer first, then check against the real output: in both, the correct answer is taken from a run of the script rather than assigned.

from unittest.mock import Mock

m = Mock(return_value=10)
print(m(1), m(2), m.call_count)
print(m.call_args)

10 10 2
call(2)

1 2 2
call(2)

10 10 2
call(1)

10 10 1
call(1, 2)

Check yourself

How it was measured

The observations of this lesson come from these scripts. Each one opens right from here — together with the record of its run: what it ran on and what came out.

• bench/testing/basics.py
• bench/testing/checkout.py
• bench/testing/test_checkout.py
• bench/testing/checkout_run.py
• bench/testing/pytest_examples.py
• bench/testing/pytest_run.py
• bench/testing/practice.py