Generators, yield, and lazy pipelines

Writing your own iterator the easy way

You could implement __iter__ and __next__ on a class, but Python has a shortcut: a generator function. Use yield instead of return and the function becomes a value factory that can pause and resume:

def countdown(n):
    while n > 0:
        yield n
        n -= 1

Calling countdown(3) runs no code yet. It returns a generator object, which is an iterator. Each next call runs the body until the next yield, hands out that value, and freezes right there, local variables intact. When the function body ends, StopIteration is raised for you. This is called lazy evaluation: values are produced only when asked for.

Generators earn their keep when the data cannot fit in memory: reading a multi-gigabyte log file line by line, or streaming millions of database rows. Production data pipelines are assembled from exactly the pieces in this lesson.

Generator expressions and pipelines

Write a comprehension with parentheses instead of square brackets and you get a generator expression, the lazy cousin of the list comprehension from lesson 1-1:

squares = (n * n for n in range(1_000_000))

No million-item list is built. Values stream out one at a time, so memory stays tiny. Feed one straight into a consuming function and you may drop the extra parentheses:

total = sum(n * n for n in range(1000))

Chain stages to build pipelines: filter with one generator, transform with another, then let sum, max, or a for loop pull values through the whole chain. Use a list when you need indexing or multiple passes, use a generator when you stream through the data once.