Python Generators and yield

In Python learning, topics like if/else, basic function definitions, and ordinary loops are usually easy to understand. Generators are different: they are powerful, memory-friendly, and often confusing at first. This article explains what a generator is, how yield works, and where generators are useful in practice.

1. What Is a Python Generator?

A Python generator is usually created by a function that uses yield. Each time yield runs, it returns a value and pauses the function state. When you call next() again, execution resumes from where it stopped instead of starting from the beginning.

def my_generator():
	print("first element:")
	yield 1
	print("second element:")
	yield 2
	print("third element:")
	yield 3

a = my_generator()
next(a)
next(a)
next(a)

In the example above, the generator instance a pauses each time it reaches yield. The first next(a) returns 1, the second returns 2, and the third returns 3. A fourth next(a) would raise StopIteration because the generator has no more values to produce.

To avoid handling that exception manually, a for loop is often the cleanest way to consume a generator.

2. Difference Between a Generator and a Normal Function

A normal function always starts running from the first line each time it is called, while a generator resumes from the previous yield position.

• A function usually returns one result per call.
• A generator can produce many values over time.
• A function can be called repeatedly without exhaustion.
• A generator instance is exhausted after its final yield or return.

3. Practical Use of Generators

One classic example is generating the Fibonacci sequence:

def fibonacci():
	a, b = 1, 1
	while True:
		yield a
		a, b = b, a + b

fib = fibonacci()

This generator produces one Fibonacci value at a time. You can call next(fib) as many times as needed.

In real projects, generators are useful when processing large datasets because they can reduce memory usage. For example, when reading a very large FASTQ sequencing file, loading everything into memory at once can be expensive, while a generator can stream records one by one.

4. Common Notes and Pitfalls

1. Use yield from for recursive generators

def get_num(data):
	for item in data:
		if isinstance(item, (tuple, list)):
			get_num(item)
		else:
			yield item

for item in get_num([1, 2, (3, 4), 5]):
	print(item)

The code above only yields 1, 2, and 5. If you want 1, 2, 3, 4, and 5, you should delegate with yield from:

def get_num(data):
	for item in data:
		if isinstance(item, (tuple, list)):
			yield from get_num(item)
		else:
			yield item

for item in get_num([1, 2, (3, 4), 5]):
	print(item)

2. Generator return is not used like a normal value return

def yield_with_return(num):
	if num < 0:
		return
	for i in range(num):
		yield i

mynum = yield_with_return(-1)

In ordinary generator usage, return is mainly used to stop iteration rather than to behave like a normal function that returns a main result.

5. Summary

This article introduced the main ideas behind Python generators and yield. To really get comfortable with them, the best approach is to use them in real code.

1. A generator is used to produce a sequence of values.
2. yield both returns a value and saves execution state.
3. Generators are useful when you want lazy evaluation and lower memory usage.
4. yield from is the right tool when delegating to another generator.
5. next(generator) retrieves the next generated value.