Generators and Iterators

Generators and iterators let Python give you values one at a time (this is called lazy evaluation) instead of building the entire collection in memory all at once.

This becomes really important when:

• The data is too large (like files, streams, or APIs)
• The values never end, or you don’t know how many there will be
• You only need some of the results, not all of them

Iterables vs. Iterators

Iterable

An iterable is simply anything you can loop over.

Examples:

• list
• tuple
• string
• set
• dictionary
• file

nums = [1, 2, 3]
for n in nums:
    print(n)

Note

Behind the scenes, Python turns this into an iterator automatically.

Iterator

An iterator is an object that:

• Remembers where it currently is
• Gives you the next value whenever you ask for it

Iterator Protocol (The Rules an Iterator Follows)

An object counts as an iterator if it has these two things:

• __iter__() -> gives back the iterator object itself
• __next__() -> gives back the next value
• It raises StopIteration once there’s nothing left

graph TD A[Iterable] --> B[iter] B --> C[Iterator] C --> D[next] D --> E[Value] C --> F[StopIteration]

Doing Iteration by Hand

nums = [10, 20, 30]

it = iter(nums)

print(next(it))  # 10
print(next(it))  # 20
print(next(it))  # 30
# next(it) -> StopIteration

Making Your Own Iterator

class CountDown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value


for num in CountDown(3):
    print(num)

Generators

Generators are a much easier way to build an iterator.

Instead of writing a whole class, you just use the yield keyword.

The Main Idea

• return -> ends the function completely
• yield -> pauses the function for a moment and remembers exactly where it left off

Example of a Generator Function

def generate_numbers(n):
    i = 1
    while i <= n:
        yield i
        i += 1


gen = generate_numbers(3)

for num in gen:
    print(num)

Execution Flow

graph TD Call --> GeneratorObject GeneratorObject --> StartExecution StartExecution --> Yield Yield --> Pause Pause --> Resume Resume --> Yield Yield --> StopIteration

Things to Remember

• The function does NOT start running the moment you call it
• It only runs when you call next() on it
• It remembers its state (where it stopped) between each call

Generators vs. Lists

# List (memory heavy)
nums = [x*x for x in range(1000000)]

# Generator (memory efficient)
nums = (x*x for x in range(1000000))

A generator:

• Only works out a value when it’s actually needed
• Uses very little memory, no matter how big the sequence is

Generator Expressions

This is a short and quick way to write a generator, similar to how list comprehensions work.

gen = (x * 2 for x in range(5))

for val in gen:
    print(val)

Infinite Generators

Generators can also create sequences that never end.

Caution

You need to control these yourself, or they’ll keep running forever.

Example: A Counter That Never Stops

def infinite_counter():
    num = 1
    while True:
        yield num
        num += 1


counter = infinite_counter()

for _ in range(5):
    print(next(counter))

Output

1
2
3
4
5

Picture It Like This

graph TD Start --> Yield1 Yield1 --> Resume Resume --> Yield2 Resume --> Yield3 Resume --> InfiniteLoop

Sending Values Into a Generator

You can also send input back into a generator using send().

Example

def accumulator():
    total = 0
    while True:
        value = yield total
        total += value


acc = accumulator()

print(next(acc))     # start generator
print(acc.send(10))  # 10
print(acc.send(5))   # 15

Important Rule to Remember

Note

Always call next() once first before sending anything (this first step is called priming)

Closing a Generator

When you call close() on a generator:

• Python raises a GeneratorExit error inside the generator
• This gives you a chance to clean things up properly

Why Do We Need to Close Generators?

Caution

In a normal loop, whatever resources it used get cleaned up automatically once the loop ends. But generators hold onto their state between each yield, so they might keep things like file handles, network connections, or memory tied up for a long time. If you don’t close them properly, you could end up wasting resources without realizing it.

Example

def resource_manager():
    print("Resource opened")
    count = 0

    try:
        while True:
            yield count
            count += 1
    except GeneratorExit:
        print(f"Cleaning up... used {count} times")


gen = resource_manager()

print(next(gen))
print(next(gen))

gen.close()

Output

Resource opened
0
1
Cleaning up... used 2 times

Flow

graph TD Start --> Yield Yield --> Resume Resume --> Yield Close --> GeneratorExit GeneratorExit --> Cleanup Cleanup --> Stop

An Interactive Generator Example

def order_processor():
    total_orders = 0
    print("System Ready")

    try:
        order = yield

        while True:
            print(f"Processing: {order}")
            total_orders += 1
            order = yield

    except GeneratorExit:
        print(f"System Shutdown. Total orders: {total_orders}")


processor = order_processor()

next(processor)  # start

processor.send("Pizza")
processor.send("Burger")

processor.close()

yield from Delegation

This is used to make working with nested generators much simpler.

Without Using yield from

def chain():
    for x in [1, 2]:
        yield x
    for y in [3, 4]:
        yield y

With yield from

def chain():
    yield from [1, 2]
    yield from [3, 4]