This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: call arg of lambda not updating
Type: Stage:
Components: Interpreter Core Versions: Python 2.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: mwh Nosy List: kquick, mwh, rhettinger
Priority: low Keywords:

Created on 2004-10-17 21:55 by kquick, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
lambda_bug.py kquick, 2004-10-18 04:02 lambda_bug.py
cfl.diff mwh, 2004-10-18 21:27 simple "fix" from mwh
Messages (12)
msg22737 - (view) Author: Kevin Quick (kquick) Date: 2004-10-17 21:55
The attachment contains a script intended to do performance testing
on various types of dictionary keys.  Lambda functions are used to
"generate" a sequence of keys with bound arguments to arrays that are
modified during execution.

Script fails with pop() on empty list error, but examination shows that 
the lambda function from the *previous* call is being used for the 
current call; if proper lambda was used, list would not be empty and
operation should succeed.

Script is large and has been "grown", so it's not very elegant, but either 
I'm missing somewhere where I'm making a stupid mistake or else the
lambda argument to the function call isn't being set properly.  Don't be 
put off by the size of the script; the problem is easily demonstrated and 
fairly clearly understood in just a few minutes of looking at the 
generated stack trace relative to the code.

As it exists, script will fail immediately due to the bug.  If it worked as 
expected, at least 3-4 lines of output should be generated before 
encountering new code that hasn't yet been fully debugged.  New code 
could be removed, but it was left in because some of the complexity of 
core code is due to this latter code, and it doesn't obstruct the primary 
bug, so it can be ignored unless of interest.

python
Python 2.3.3 (#1, May 27 2004, 20:44:16) 
[GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on 
linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Thanks for looking at this!
msg22738 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2004-10-17 23:04
Logged In: YES 
user_id=80475

Please resubmit after boiling this down to the simplest
possible thing that doesn't work according to your
expectations. 

Submitting a whole application and playing the "find the
bug" game is not good use of developer time.  Often, when a
submitter starts taking away the unnecessary pieces and
isolates the issue, they find that there was a bug in their
own code or a bug in their own understanding of how the
language works.

When it comes to lambdas, the most frequent misunderstanding
is knowing that default arguments bind only once as the time
of the lambda declaration and that free variables bind
whenever the resulting function is invoked.  Here's an
example of code that looks similar but behaves differently:
 
def once(x): return x
def twice(x): return 2*x
def thrice(x): return 3*x
funcs = [once, twice, thrice]

flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda
x:funcs[2](x)]
flam = [lambda x:f(x) for f in funcs]

print flim[0](1), flim[1](1), flim[2](1)
print flam[0](1), flam[1](1), flam[2](1)

The difference in output is because nested scopes bind
names, while argument binding binds values.

Hope this helps.  If not, please resubmit with the smallest
case that demonstrates the problem.


msg22739 - (view) Author: Kevin Quick (kquick) Date: 2004-10-18 04:02
Logged In: YES 
user_id=6133

OK, example script minimized.

Lambda arguments are not the problem; thanks for the pointer, but I'm not 
having trouble with those.  

I have determined that proper lambda is being used, but not indicated as shown 
by attached script, which forces an exception to occur in the lambda function 
but backtrace shows wrong lambda function.  Reducing 2nd argument of 2nd 
call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the 
error, but the nature of the implementation requires that it is using the correct 
lambda, so it's the backtrace attribution that is apparently wrong.

Mea culpa: my original script version did request wrong 'num_args' causing 
actual bug, but incorrect backtrace led me down an alternate path... I suspected 
that I had some incorrect code, but I couldn't resolve that against the error 
report.  Still can't.  :)
msg22740 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2004-10-18 04:34
Logged In: YES 
user_id=80475

Closing as invalid -- this is a bug in the OP's script, not
in Python itself.

The error is obious when the code is simplified further by
in-lining the function.  Also, it helps to explicitly set
the initial values of variables between the two sections:

#!/usr/bin/env python
num_elem = 100

key_src = range(num_elem+1)
keynums1 = key_src[:]

assert key_src[-1] == 100
for _ in xrange(num_elem):
    keynums1[key_src.pop()]

keynums2 = [0]
for _ in xrange(102):
    key_src[keynums2.pop()]


I think your coding error was in assuming that keynum2 held
a different value before the second half was run.  If so,
that could have been found by using pdb or by inserting
print statements to reveal what is going on.
msg22741 - (view) Author: Kevin Quick (kquick) Date: 2004-10-18 04:48
Logged In: YES 
user_id=6133

Reopened: Problem not addressed.  Please *read* my previous comment.

Yes, the script has a bug in it.  That's not what's being reported here.

The bug is in the backtrace produced by the Python interpreter.  See below.  
Line number reported for lambda function raising exception is line 14 below.  
Actual line number for lambda generating exception is line 18.


$ python lambda_bug.py
Traceback (most recent call last):
  File "lambda_bug.py", line 23, in ?
    seqtest()
  File "lambda_bug.py", line 19, in seqtest
    seq_func1(nextkey2, num_elem+2)
  File "lambda_bug.py", line 5, in seq_func1
    key_genfunc()
  File "lambda_bug.py", line 14, in <lambda>
    nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()]
IndexError: pop from empty list
msg22742 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2004-10-18 05:41
Logged In: YES 
user_id=80475

Confirmed.  Also, the behavior persists into Py2.4b1.

The error disappears when the lambda is replaced by an
equivalent one line def statement:

  def nextkey2(N=keynums2,K=key_src): return K[N.pop()]

The opcodes are the same for both; however, the
co_firstlineno attribute is incorrect for the lambda version.

The pure python compiler module does not make the same error.

msg22743 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2004-10-18 06:42
Logged In: YES 
user_id=80475

Under the hood, the compiler is recognizing that the
function bodies are identical and it produces just one code
object instead of two.  Hence, the co_firstline number with
always reflect the first time the code is defined.

A simple demo shows the id being the same when the bodies
are identical:

src = """
f1 = lambda x=1: x
f2 = lambda x=2: x
"""

from dis import dis
c = compile(src, 'example', 'exec')
dis(c)

For the OP's code, the effect can be shown by
differentiating either lambda body by adding zero to the pop
index:
   nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()+0]

I conclude that there is less to the bug than meets the eye
and that it may even be considered a feature.  Referring to
Jeremy for final disposition.
msg22744 - (view) Author: Kevin Quick (kquick) Date: 2004-10-18 07:37
Logged In: YES 
user_id=6133

Thanks for confirming this issue.

My position is that while minor, this is still a bug and not a feature.  It's good 
to detect identical lambda bodies and optimize for that case, but since 
argument binding occurs at declaration time and not execution time for 
lambdas, the bound arguments should properly be considered part of the body; 
the two lambda functions in the attached script should be disjoint.

From the practical perspective, the python backtrace was misleading in my 
debug efforts.  Internal optimizations should take second place to accuracy of 
displayed information.  func_code.co_code (and possibly others) could still be 
the same, but func_code.co_firstlineno (& func_code.co_filename) should be 
uniquely set.

Hmmm. In thinking more about this, it's not that argument binding occurs 
immediately, but moreso that defaults are supplied for arguments.  If the 
backtrace/debugger points me to a lambda function with entirely different 
default args than the ones that are applicable to the problem, then that would 
be wrong, IMHO.
msg22745 - (view) Author: Michael Hudson (mwh) (Python committer) Date: 2004-10-18 21:27
Logged In: YES 
user_id=6656

I *think* what's happening here is that co_firstlineno is not being 
considered when comparing code objects.  Can you try the 
attached patch?

There may well be other fields not being considered.
msg22746 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2004-10-18 22:29
Logged In: YES 
user_id=80475

Michael, that is a reasonable solution.  Go ahead and apply
it along with a test case verifying the object id's are
distinct after the change but not before.
msg22747 - (view) Author: Kevin Quick (kquick) Date: 2004-10-18 23:24
Logged In: YES 
user_id=6133

Confirmed that the patch does indeed address the problem.  Also verified
that a lambda with the same code appearing at the same line in a different
file is handled separately and distinctly.

Thanks!
msg22748 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2004-10-24 00:11
Logged In: YES 
user_id=80475

Fixed.
See:  Python/compile.c 2.331
History
Date User Action Args
2022-04-11 14:56:07adminsetgithub: 41039
2004-10-17 21:55:17kquickcreate