Understanding the Differences Between Python Generator Expressions and List Comprehensions

6 min readApr 10, 2023

Python offers two powerful and convenient ways to generate sequences of values: generator expressions and list comprehensions. However, while they share some similarities, they also have significant differences that can affect their performance, memory usage, and other aspects of their behaviour. In this post, we will explore these differences and provide examples of how to use each construct.

Syntax

The most apparent difference between a generator expression and a list comprehension is the syntax used to define them. A list comprehension is enclosed in square brackets ([]) and generates a list of values, while a generator expression is enclosed in parentheses (()) and generates a generator object that can produce values one at a time. Here is an example of each:

# List comprehension
evens_squared = [x**2 for x in range(10) if x % 2 == 0]
print(evens_squared)

# Generator expression
evens_squared = (x**2 for x in range(10) if x % 2 == 0)
print(evens_squared)

<<output>>
[0, 4, 16, 36, 64] 
<generator object <genexpr> at 0x111c96570>

Both constructs use a similar syntax for defining the sequence of values to be generated, in this case, the squares of even numbers between 0 and 9. However, the type of object they produce is different, as we will see in the following sections.

Lazy Evaluation

One of the most critical differences between generator expressions and list comprehensions is their approach to generating values. A list comprehension generates all the values in the sequence immediately, while a generator expression generates them lazily only when needed. This can have important implications for memory usage, performance, and the program’s behaviour.

When we define a list comprehension like this:

evens_squared = [x**2 for x in range(10) if x % 2 == 0]
print(evens_squared)

<<output>>
[0, 4, 16, 36, 64]

Python creates a list of 5 values (0, 4, 16, 36, 64) and stores them in memory. This means that the entire list is available for use immediately, and we can access any element at any time. However, if the list is huge, this can consume significant memory.

When we define a generator expression like this:

evens_squared = (x**2 for x in range(10) if x % 2 == 0)
print(evens_squared)
print(next(evens_squared))
print(next(evens_squared))
print(next(evens_squared))

<<output>>
<generator object <genexpr> at 0x111c969d0>
0
4
16

Python creates a generator object that can produce the values in the sequence one at a time as needed. This means that only one value is in memory at any given time, which can save memory for large sequences. However, this also means that the generator expression cannot be indexed or accessed directly and can only be iterated once.

Memory Usage

As mentioned in the previous section, memory usage is one of the main differences between generator expressions and list comprehensions. List comprehensions generate a complete list of values in memory, which can be accessed multiple times and consume a lot of memory for large sequences. On the other hand, generator expressions generate values one at a time and can save memory for large sequences.

Here is an example of generating a large sequence of even numbers using list comprehension:

# Generate a list of even numbers between 0 and 999999
even_numbers = [x for x in range(1000000) if x % 2 == 0]

This generates a list of 500000 even numbers, which can consume significant memory.

Now, let’s compare this to generating the same sequence of even numbers using a generator expression:

# Generate a sequence of even numbers between 0 and 999999
even_numbers = (x for x in range(1000000) if x % 2 == 0)

This generates a generator object that can produce even numbers one at a time, as needed. This can save memory compared to list comprehension because only one even number is in memory at any given time. However, the tradeoff is that we cannot access the even numbers directly or modify them in any way.

If we need to use the even numbers multiple times, we can still generate a list from the generator expression using the list() function:

# Generate a list of even numbers from the generator expression
even_numbers_list = list(x for x in range(1000000) if x % 2 == 0)

This converts the generator object into a list of even numbers, which we can now access and modify as a regular list. However, this can consume the same amount of memory as a list comprehension because we simultaneously store all the even numbers in memory.

Iteration

Another vital difference between generator expressions and list comprehensions is their behaviour during iteration. A list comprehension immediately generates all the values in the sequence and stores them in memory for fast access and modification. On the other hand, a generator expression generates values lazily, only when needed, and only stores them in memory after they have been produced.

Here is an example of iterating over a list comprehension:

# Iterate over a list comprehension
evens_squared = [x**2 for x in range(10) if x % 2 == 0]
for even_squared in evens_squared:
    print(even_squared)

<<output>>
0
4
16
36
64

This generates the squares of even numbers between 0 and 9, stores them in a list, and then iterates over the list, printing each value. This can be fast and convenient for small sequences but slow and memory-intensive for large sequences.

Here is an example of iterating over a generator expression:

# Iterate over a generator expression
evens_squared = (x**2 for x in range(10) if x % 2 == 0)
for even_squared in evens_squared:
    print(even_squared)

<<output>>
0
4
16
36
64

This generates the squares of even numbers between 0 and 9 but does not store them in memory as a list. Instead, it generates each value lazily, only when needed, and then discards it. This can be slower than a list comprehension for small sequences but faster and more memory-efficient for large sequences.

Performance

Finally, let’s compare the performance of generator expressions and list comprehensions for different use cases. The exact performance characteristics will depend on the specific situation. Still, generator expressions can generally be faster and more memory-efficient than list comprehensions for large sequences that don’t need to be accessed multiple times.

Here is an example of timing the execution of a list comprehension and a generator expression for generating a sequence of even numbers between 0 and 999999:

import time

# Time a list comprehension
start_time = time.time()
even_numbers = [x for x in range(1000000) if x % 2 == 0]
end_time = time.time()
print("List comprehension time:", end_time - start_time)

# Time a generator expression
start_time = time.time()
even_numbers = (x for x in range(1000000) if x % 2 == 0)
end_time = time.time()
print("Generator expression time:", end_time - start_time)

On my machine, running this code produces the following output:

<<output>>
List comprehension time: 0.057240962982177734
Generator expression time: 0.002710103988647461

This shows that the generator expression is significantly faster than the list comprehension by a factor of about 1500. This is because the generator expression only generates the even numbers one at a time, as needed. In contrast, list comprehension generates all the even numbers and stores them in memory.

However, it’s worth noting that the performance of generator expressions and list comprehensions can depend on the specific use case. In general, generator expressions can be faster and more memory-efficient for large sequences that don’t need to be accessed multiple times. At the same time, list comprehension can be faster for small lines that need to be accessed numerous times or modified.

Conclusion

In summary, generator expressions and list comprehensions are powerful tools for generating sequences in Python. They have similar syntax and functionality but differ in memory usage, iteration behaviour, and performance characteristics.

Generator expressions can be more memory-efficient and faster for large sequences that don’t need to be accessed multiple times. At the same time, list comprehensions can be faster for small sequences that need to be accessed numerous times or modified. Understanding the tradeoffs between these two tools can help you write more efficient and effective Python code.

References

Python documentation on List Comprehensions: https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions
Python documentation on Generator Expressions: https://docs.python.org/3/tutorial/classes.html#generator-expressions
Real Python — List Comprehensions and Generator Expressions: https://realpython.com/list-comprehension-python/