Sign in

Why does Python 3 change the return value of filter() to an iterator object? What are the benefits?

qcgza edited in Sat, 28 May 2022

Why does Python 3 change the return value of filter() to an iterator object? What are the benefits? It's not very troublesome to deal with it. I just don't understand the reason why Python 3 is changed in this way

2 Replies
commented on Sat, 28 May 2022

The simple reason: memory saving. Functions such as map and filter return an iterator that has the properties of a class generator. We know that the generator is lazy loading, it only calculates the generated value in the next call, instead of generating all the values in advance like the list, and then returns the next value in the list every time. This method needs to save all the values in the list in advance. When the list is large, it consumes a lot of memory. Consider this scenario: there is a 2GB text document, in which there are tens of millions of user records. Now we need to use filter to find out all the records whose user names begin with ABC and write them to another text document. If the filter returns not the iterator but the list of all matching user records, the memory consumption of saving these user information will be very amazing and unacceptable. It is precisely because the filter returns an iterator that we can read and process user records one by one from such a text document at a very small memory cost.

commented on Sat, 28 May 2022

Because the iterative object processed by map / filter itself may be a generator

If I return a list, it will bring them a lot of restrictions. In 2. X, I can't return an iterator through map and filter. I can only read all the data into memory at one time. If I read a line of a very large file, I can only avoid using map and filter, because memory overflow occurs when I return a generator. At the same time, this problem is avoided, It's no trouble to turn it into a list, just one more step

lock This question has been locked and the reply function has been disabled.