This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Nested generator terminates prematurely
Type: Stage:
Components: Interpreter Core Versions:
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: arigo, mwh, rhettinger, terry.reedy, ygale
Priority: normal Keywords:

Created on 2004-05-18 10:02 by ygale, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (10)
msg20823 - (view) Author: Yitz Gale (ygale) Date: 2004-05-18 10:02
def g(x, y):
  for i in x:
    for j in y:
      yield i, j

r2 = (0, 1)
[e for e in g(r2, g(r2, r2))]

Expected result:

[(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)),
 (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))]

Actual result:

[(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))]
msg20824 - (view) Author: Yitz Gale (ygale) Date: 2004-05-18 10:04
Logged In: YES 
user_id=1033539

Trying again to get the indentation correct:

def g(x, y):
  for i in x:
    for j in y:
      yield i, j
msg20825 - (view) Author: Michael Hudson (mwh) (Python committer) Date: 2004-05-18 11:24
Logged In: YES 
user_id=6656

Um.  I think the answer to this is "generators are not
reiterable".
msg20826 - (view) Author: Yitz Gale (ygale) Date: 2004-05-19 10:57
Logged In: YES 
user_id=1033539

Too bad. What exactly is the restriction?
I didn't find anything in the docs. And things
like this often do work and are useful.

For example:

def primes():
  yield 2
  for n in count(3):
    for p in primes():
      if p > sqrt(n):
        yield n
        break
      if n % p == 0:
        break
msg20827 - (view) Author: Michael Hudson (mwh) (Python committer) Date: 2004-05-19 10:59
Logged In: YES 
user_id=6656

Well, it's impossible in general.  You'd have to store any
arguments the generator took somewhere too, wouldn't you?

What about things like:

def foo(aList):
      while aList:
           yield aList.pop()

?
msg20828 - (view) Author: Yitz Gale (ygale) Date: 2004-05-19 11:59
Logged In: YES 
user_id=1033539

Python functions can be called recursively
in general. They know how to save their
local namespace in a separate
frame for each call. That includes arguments,
since the arguments live in the local namespace
of the function.

Generator functions also seem to be supported,
as my example shows.

There is a restriction on a generator object
that you may not call its next() method again
while a previous call to next() is still running.

But this is definitely not a case of that
restriction - we have two separate generator
instances, and each ought to have its own frame.

If there is some other restriction, I think it ought
to be documented. And if possible, it should
raise an exception, like the other restriction.

This smells like a bug to me, though.
msg20829 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2004-05-19 12:55
Logged In: YES 
user_id=4771

Your issue is that you only create a total of two generator
instances.  The 'inner' one is immediately exhausted. 
Afterwards, this same instance is used again on other 'for'
loops but this has no effect, as it has already been exhausted.

The difference is the same as between r2 and iter(r2).  If
you do that:

it = iter(r2)
for x in it: print x
for x in it: print x

the second loop will be empty beause the first loop has
exhausted the iterator.  Generators are iterators (like it)
and not sequences (like r2).

Using the same iterator on several 'for' loops is useful,
though (e.g. if the first loop can be interrupted with
'break'), so there is no way your code could raise an
exception, short of saying that it is not allowed to call
next() on already-exhausted iterators -- this would be too
big a change.
msg20830 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2004-05-19 16:36
Logged In: YES 
user_id=80475

Marking this as invalid and closing.

Sorry, non-re-iterability is documented fact of life in the
world of generators and iterators.

The work arounds include making the inner generator into a
list or re-instantiating a new generator on every loop:
 
def g(x, y):
    for i in x
        for j in g(x, y)
             yield i, j
msg20831 - (view) Author: Yitz Gale (ygale) Date: 2004-05-19 21:33
Logged In: YES 
user_id=1033539

OK. I can get the semantics I want using the following:

def g(x, y):
  for i in x:
    for j in y:
      yield i, j
g = restartable(g)

where I have defined:

class restartable:
  def __init__(self, genfn):
    self.genfn = genfn
  def __call__(self, *args):
    return restartable_generator(self.genfn, *args)

class restartable_generator:
  def __init__(self, genfn, *args):
    self.genfn = genfn
    self.args = args
  def __iter__(self):
    return self.genfn(*self.args)
msg20832 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2004-06-01 17:08
Logged In: YES 
user_id=593130

Unless you are generating a very long list, you worked too 
hard.  But not as cute.

>>> def g(x, y):
...   y = list(y)
...   for i in x:
...     for j in y:
...       yield i, j
...
>>> r2 = (0, 1)
>>> [e for e in g(r2, g(r2, r2))]
[(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)),
 (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))]
History
Date User Action Args
2022-04-11 14:56:04adminsetgithub: 40263
2004-05-18 10:02:49ygalecreate