This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: urllib fails with multiple ftp transfers
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: Sohaib Ahmad, ajaksu2, orsenthil, python-dev, reowen, sschwarzer
Priority: normal Keywords:

Created on 2004-11-16 23:02 by reowen, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
urllibftpbug.py reowen, 2004-11-16 23:02 Sample script; instructions in header
urllibftpbug-non-interactive.py sschwarzer, 2011-06-26 13:54 Test triggering the reported transfer failure without manual intervention in the interpreter.
urllibftpbug-non-interactive-py3.py sschwarzer, 2011-06-26 14:53 Non-interactive test script adapted for Python 3
Messages (11)
msg23169 - (view) Author: Russell Owen (reowen) Date: 2004-11-16 23:02
I find that urllib fails on ftp if:
- I start more than 1 transfer, each in its own thread
- I let those finish
- I again start more than 1 transfer, each in its own thread
One of the latter xfers will fail with:
  [Errno ftp error] 200 Type set to A

I have attached sample code that demonstrates the problem.

Notes:
- I have seen the problem on more than one ftp server, using 
different sets of sample files.
- An explicit ";type=i" suffix on binary file URLs does not help. I have 
not tried text files with explicit "type=a".
msg81794 - (view) Author: Daniel Diniz (ajaksu2) * (Python triager) Date: 2009-02-12 18:17
OP supplied a detailed test. Needs confirmation.
msg139177 - (view) Author: Stefan Schwarzer (sschwarzer) Date: 2011-06-26 13:54
I was able to get some error output with the code of the OP. However, I only saw the "opposite" message, such as:

Retrieval of 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2' failed with error: [Errno ftp error] 200 Switching to Binary mode.

I had to change the used FTP server and path because I had problems connecting to that server.

Even with the guidelines by the OP, I could trigger the bug only occasionally.

I converted the script to be executable with manual intervention (see attachment). Still, sometimes both files are also successfully downloaded in the second iteration (i. e. the "outer" iteration which downloads both files).

These results are from a test with Python 2.6.6. I plan to continue testing with the tip of the Python 2.7 branch.
msg139178 - (view) Author: Stefan Schwarzer (sschwarzer) Date: 2011-06-26 14:13
> I converted the script to be executable with manual intervention (see attachment).

This should have been "without manual intervention". :)
msg139180 - (view) Author: Stefan Schwarzer (sschwarzer) Date: 2011-06-26 14:29
I can confirm the bug for the Python 2.7 tip (changeset b11e7bc76d07) after using the script urllibftpbug-non-interactive.py.
msg139185 - (view) Author: Stefan Schwarzer (sschwarzer) Date: 2011-06-26 14:53
After running the adapted test script three times for Python 3 tip (changeset c5b0585624ef), I didn't get an error message / exception.
msg139186 - (view) Author: Stefan Schwarzer (sschwarzer) Date: 2011-06-26 15:10
The traceback and its context for the exception raised in Python 2.7 is:

...
################################################################################################### 99 %
################################################################################################### 99 %
#################################################################################################### 100 %
Done retrieving 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2'; read 4539023 bytes
Start retrieving 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2'
Start retrieving 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2'
Retrieval of 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2' failed with error: [Errno ftp error] 200 Switching to Binary mode.
Traceback (most recent call last):
  File "urllibftpbug-non-interactive.py", line 62, in _getTask
    self._fromFile = urllib.urlopen(self.fromURL)
  File "/home/schwa/sd/python/cpython/Lib/urllib.py", line 84, in urlopen
    return opener.open(url)
  File "/home/schwa/sd/python/cpython/Lib/urllib.py", line 205, in open
    return getattr(self, name)(url)
  File "/home/schwa/sd/python/cpython/Lib/urllib.py", line 548, in open_ftp
    (fp, retrlen) = self.ftpcache[key].retrfile(file, type)
  File "/home/schwa/sd/python/cpython/Lib/urllib.py", line 886, in retrfile
    conn = self.ftp.ntransfercmd(cmd)
  File "/home/schwa/sd/python/cpython/Lib/ftplib.py", line 326, in ntransfercmd
    host, port = self.makepasv()
  File "/home/schwa/sd/python/cpython/Lib/ftplib.py", line 304, in makepasv
    host, port = parse227(self.sendcmd('PASV'))
  File "/home/schwa/sd/python/cpython/Lib/ftplib.py", line 790, in parse227
    raise error_reply, resp
IOError: [Errno ftp error] 200 Switching to Binary mode.
Total bytes: 4539023
 0 %
 0 %
 0 %
...

I printed the traceback by adding traceback.print_exc() at the end of the _getTask method:

        except Exception, e:
            self._cleanup(e)
            traceback.print_exc()
msg139217 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-26 20:45
New changeset a4d418e8010d by Senthil Kumaran in branch '2.7':
Fix closes issue1067702  The problem with close multiple ftp transfers were due cases where sockets/file were not closed immediately. Tightned those cases and failure is no longer observed.
http://hg.python.org/cpython/rev/a4d418e8010d
msg139249 - (view) Author: Stefan Schwarzer (sschwarzer) Date: 2011-06-27 08:45
Hi Senthil,

I don't yet understand what was going on before it resulted in the traceback. I also don't understand _why_ the patch fixes _this_ bug. (That's not to say it doesn't, but I think it's not obvious either. :-) )

Were you able to reproduce the exception with my attached script before you did the change? Here in the hotel I have a much faster internet connection than I had yesterday at the sprint (where lots of people shared the uplink), and now I can't reproduce the exception after running the test script three times, even _without_ having your change applied.
msg139700 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2011-07-03 20:05
Hello Stefen, 

Yes, I was able to reproduce the issue without the patch and at the
same time, noticed that the problem was not observed in other branches
(py3k), then saw the differences where the possible socket close
delays can happen and fixed it. As it was multiple threads closely
being closed it was most probably due to delay in closing socket
connection that lead the problem in the first place.

If it is required, then manually this can be verified by setting up an
artificial delay in the network layer and trying multiple ftp
transfers. (No, I did not do that to verify. :))
msg274539 - (view) Author: Sohaib Ahmad (Sohaib Ahmad) * Date: 2016-09-06 08:22
The problem is reproducible on latest python 2.7 package (2.7.12).

I tried the same scenario on 2.7.10 and it worked fine. I am not sure if this issue can be reopened or should I create a new one?

In my case first transfer succeeds but second ftp transfer fails with the error:
[Errno ftp error] 200 Type set to I

I am using urllib.urlretrieve(url, local_path) to retrieve two files (one by one) from FTP server.
History
Date User Action Args
2022-04-11 14:56:08adminsetgithub: 41180
2016-09-06 08:22:09Sohaib Ahmadsetnosy: + Sohaib Ahmad
messages: + msg274539
2011-07-03 20:05:49orsenthilsetmessages: + msg139700
2011-06-27 08:45:43sschwarzersetmessages: + msg139249
2011-06-26 20:45:45python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg139217

resolution: fixed
stage: test needed -> resolved
2011-06-26 15:10:55sschwarzersetmessages: + msg139186
2011-06-26 14:53:53sschwarzersetfiles: + urllibftpbug-non-interactive-py3.py

messages: + msg139185
2011-06-26 14:29:11sschwarzersetmessages: + msg139180
2011-06-26 14:13:36sschwarzersetmessages: + msg139178
2011-06-26 13:54:35sschwarzersetfiles: + urllibftpbug-non-interactive.py
title: urllib fails with multiple ftps -> urllib fails with multiple ftp transfers
nosy: + sschwarzer

messages: + msg139177
2010-08-19 18:29:56orsenthilsetassignee: orsenthil
2010-08-19 18:28:23BreamoreBoysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2009-02-12 18:17:35ajaksu2setnosy: + ajaksu2, orsenthil
stage: test needed
type: behavior
messages: + msg81794
versions: + Python 2.6, - Python 2.3
2004-11-16 23:02:15reowencreate