Continuous Integration: Tests on Every Change

The safety net only works if it runs

Everything in this course assumes the tests actually get run, and humans forget. You make a tiny change at 6pm, skip the suite because you are sure, and ship the regression the tests would have caught. Every team learns this the painful way, which is why they stopped trusting memory and gave the job to a machine.

Continuous integration, or CI, is the practice of having a server automatically run the whole test suite on every proposed change, before that change is allowed to join the shared codebase. The workflow at nearly every software company today:

1. you finish a change and propose it to the team
2. the CI server takes a fresh copy of the codebase plus your change and runs every test on a clean machine
3. green means the change may merge into the shared code, and red means the change is blocked until fixed

Two consequences matter. First, "it works on my machine" stops being an argument, because the clean machine is the referee. Second, the pyramid from lesson 6-1 turns into an economic constraint, since CI runs on every change and a slow suite taxes every engineer on the team many times a day.

A miniature CI gate

ci_gate plays the server's role, running every test and then either blessing the change or blocking the merge.

def add(a, b):
    return a + b

def test_adds():
    assert add(2, 3) == 5

def test_zero():
    assert add(0, 0) == 0

def ci_gate(tests):
    failures = 0
    for test in tests:
        try:
            test()
            print("PASS", test.__name__)
        except AssertionError:
            failures += 1
            print("FAIL", test.__name__)
    if failures == 0:
        print("CI: green, change may merge")
    else:
        print(f"CI: red ({failures} failing), merge blocked")

ci_gate([test_adds, test_zero])

Output

PASS test_adds
PASS test_zero
CI: green, change may merge

The gate is the lesson 1-3 runner plus a verdict, so zero failures means merge and anything else means blocked. A real CI service does exactly this and adds fetching your change and setting up the clean machine.

The failures counter is what makes the verdict possible, and note that it counts rather than short-circuiting. A gate that stopped at the first failure would block the merge just as correctly and would tell the author about only one problem per run, which is a worse experience when three tests broke.

The threshold is failures == 0 and not something more forgiving. That strictness is the point of a gate, since any tolerance for a small number of failures immediately becomes a tolerance for the same failures forever.

Reading a red gate

When your change passes on your laptop but CI reports a failure and blocks the merge, the correct reading is that on a clean machine the suite disagrees with your laptop, and the block just stopped a broken change from reaching the whole team.

Removing "works on my machine" from the conversation is CI's entire purpose, so the clean machine wins the disagreement by design.

The usual causes are mundane:

• a file you forgot to include in the change, so the clean machine never got it
• leftover local state your laptop has and a fresh checkout does not, such as a cached file or a database row
• a dependency installed on your machine but missing from the project's declared list
• test order, since a suite that passes only in the order your editor ran it has hidden coupling between tests

The tempting response is to delete or skip the failing test so the gate goes green. That cuts a hole in the unit-1 safety net at team scale, and it converts a five-minute investigation into a bug that ships. The productive response is to reproduce the clean machine's conditions locally, which is exactly the reproduce-first habit from lesson 4-3.

Turning a red gate green

A teammate's change to format_price hit the gate and got blocked. The spec wants exactly two decimals, and before the fix the function returned f"${cents / 100}".

def format_price(cents):
    return f"${cents / 100:.2f}"

def test_two_decimals():
    assert format_price(500) == "$5.00"

def test_small_amount():
    assert format_price(9) == "$0.09"

def ci_gate(tests):
    failures = 0
    for test in tests:
        try:
            test()
            print("PASS", test.__name__)
        except AssertionError:
            failures += 1
            print("FAIL", test.__name__)
    if failures == 0:
        print("CI: green, change may merge")
    else:
        print(f"CI: red ({failures} failing), merge blocked")

ci_gate([test_two_decimals, test_small_amount])

Output

PASS test_two_decimals
PASS test_small_amount
CI: green, change may merge

Before the fix, test_two_decimals fails because f"${500 / 100}" produces $5.0, which is one decimal short. Python prints a float in its shortest faithful form, and money needs a fixed width instead.

F-strings accept a format spec after a colon, so {value:.2f} always prints exactly two decimals. That gives f"${cents / 100:.2f}", and the second test is what confirms the fix works at the other end of the scale, since 9 cents has to format as $0.09 rather than $.09 or $0.1.

Both tests failed with the original code, which makes this a two-failure red report rather than a one-failure one. Seeing both listed is what tells you the problem is the formatting rule itself and not one unusual input, and that is the information a counting gate gives you and a short-circuiting one does not.

The acronym spelled out

CI stands for continuous integration.

Both words carry weight. Continuous means this happens on every proposed change rather than on a schedule or when somebody remembers, and integration means joining the shared codebase, which is the moment the gate protects.

Note that this sense of integration is different from the integration tests of lesson 6-2. There the word describes components meeting each other, and here it describes your change meeting everyone else's code. Both are about a seam, and confusing the two terms is a common early stumble.

When a later lesson mentions reading CI logs, the output of this robot referee is what it means, which is the same pass and fail lines you have been printing since lesson 1-3, produced on a machine you do not control.