This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: imaplib fetch is broken
Type: Stage:
Components: Library (Lib) Versions: Python 2.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pierslauder Nosy List: jgoerzen, pierslauder
Priority: normal Keywords:

Created on 2002-06-19 21:22 by jgoerzen, last changed 2022-04-10 16:05 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
imaplib.py jgoerzen, 2002-06-24 19:39 Fixed imaplib.py
Messages (11)
msg11261 - (view) Author: John Goerzen (jgoerzen) Date: 2002-06-19 21:22
This bug is 100% reproducible in Python 2.2.1 and the 
imaplib.py from Python's CVS tree. 
 
They'll send the message body in this form: 
{100} 
Headers here, body here, (100 characters worth) 
 
This is specified in RFC2060 secion 4.3.  imaplib's fetch 
chokes on 
"unexpected data" in the response. 
 
imaplib should receive this prefix-quoted string properly 
and return it in a 
non-quoted fashion. 
 
As it is, it is totally impossible to read messages from 
most imap servers. 
 
msg11262 - (view) Author: Piers Lauder (pierslauder) * (Python triager) Date: 2002-06-20 08:46
Logged In: YES 
user_id=196212

I can't duplicate this.

The test code at the end of imaplib.py includes a FETCH
command which is returned as a literal by my imap server.

Perhaps you could run that test on your server and cut
and paste the response.

Run the test like this:
  python2.2 imaplib.py -d5 <imap_server_host_name>

From which I get the following:
	...
 43:26.15 > KDP19 UID FETCH 19570 (FLAGS INTERNALDATE
RFC822.SIZE RFC822.HEADER RFC822.TEXT)
  43:26.46 < * 83 FETCH (UID 19570 FLAGS (\Seen)
INTERNALDATE "20-Jun-2002 18:36:51 +1000" RFC822.SIZE 55
RFC822.HEADER {46}
  43:26.46      matched r'\* (?P<data>\d+)
(?P<type>[A-Z-]+)( (?P<data2>.*))?' => ('83', 'FETCH', '
(UID 19570 FLAGS (\\Seen) INTERNALDATE "20-Jun-2002 18:36:51
+1000" RFC822.SIZE 55 RFC822.HEADER {46}', '(UID 19570 FLAGS
(\\Seen) INTERNALDATE "20-Jun-2002 18:36:51 +1000"
RFC822.SIZE 55 RFC822.HEADER {46}')
  43:26.46      matched r'.*{(?P<size>\d+)}$' => ('46',)
  43:26.47 read literal size 46
	...
and so on.
msg11263 - (view) Author: John Goerzen (jgoerzen) Date: 2002-06-20 14:01
Logged In: YES 
user_id=491567

It turns out that it is a bug in read() in the imaplib.  It does not 
obey the principle that the read() call will return UP TO the 
amount of bytes specified.  The below diff fixes it for me.  I 
have not yet tried append(), but I suspect that the write calls 
will have the same problem. 
 
--- imaplib.py  18 Jun 2002 15:36:03 -0000      1.1 
+++ imaplib.py  20 Jun 2002 04:02:49 -0000      1.2 
@@ -222,8 +222,10 @@ 
 
     def read(self, size): 
         """Read 'size' bytes from remote.""" 
-        return self.file.read(size) 
- 
+        retval = '' 
+        while len(retval) < size: 
+            retval += self.file.read(size - len(retval)) 
+        return retval 
 
     def readline(self): 
         """Read line from remote.""" 
@@ -1056,7 +1059,10 @@ 
 
     def read(self, size): 
         """Read 'size' bytes from remote.""" 
-        return self.sslobj.read(size) 
+        retval = '' 
+        while len(retval) < size: 
+            retval += self.sslobj.read(size - len(retval)) 
+        return retval 
 
 
     def readline(self): 
 
msg11264 - (view) Author: Piers Lauder (pierslauder) * (Python triager) Date: 2002-06-20 22:37
Logged In: YES 
user_id=196212

I'm very surprised by this. According to the Python library
manual for the read method:

read([size])
 Read at most size bytes from the file (less if the read
hits EOF before obtaining size bytes). If the size argument
is negative or omitted, read all data until EOF is reached.
The bytes are returned as a string object. An empty string
is returned when EOF is encountered immediately. (For
certain files, like ttys, it makes sense to continue reading
after an EOF is hit.) Note that this method may call the
underlying C function fread() more than once in an effort to
acquire as close to size bytes as possible.

So read(size) should return size bytes unless EOF is found.
(And as imaplib has been in use for several years, I take it
the manual is factual :-)

Maybe this is an O/S-specific problem in the python interpreter?
What O/S (and version) is your Python 2.2.1 interpreter
running on?
msg11265 - (view) Author: Piers Lauder (pierslauder) * (Python triager) Date: 2002-06-20 22:45
Logged In: YES 
user_id=196212

PS: could you also state which imaplib class you are using -
there may be a difference between the socket.socket and
socket.ssl behaviours.
msg11266 - (view) Author: John Goerzen (jgoerzen) Date: 2002-06-20 23:45
Logged In: YES 
user_id=491567

Note, though, that AFAIK there is no canonical code for 
"read" in python like there is in C.  File objects have a read(), 
socket objects have a read, StringIO has a read, and in this 
case, SSL sockets have a read. 
 
I suspect that it could be an issue unique to SSL sockets. 
 
I'm using Python 2.2.1 with the imaplib.py pulled from the 
development tree's CVS as of two days ago.  Sorry, I don't 
know the exact version number.  I needed that one for the SSL 
support. 
 
It is possible that this could be considered a bug in the SSL 
socket support.  I find it highly annoying that, among other 
things, imaplib.py has to kludge around socket.ssl's lack of 
readline(). 
msg11267 - (view) Author: Piers Lauder (pierslauder) * (Python triager) Date: 2002-06-22 20:41
Logged In: YES 
user_id=196212

I've checked in a new version of imaplib with the IMAP4_SSL
read and send methods modified to deal with short data
results from the sslobj methods.

Let me know if it work for you and I'll mark this bug fixed.
msg11268 - (view) Author: Piers Lauder (pierslauder) * (Python triager) Date: 2002-06-22 21:09
Logged In: YES 
user_id=196212

My cvs checkin failed, perhaps because sourceforge is too
busy,  so the update is postponed for the time being...
msg11269 - (view) Author: Piers Lauder (pierslauder) * (Python triager) Date: 2002-06-23 10:51
Logged In: YES 
user_id=196212

Changes now checked in - please try them out.
msg11270 - (view) Author: John Goerzen (jgoerzen) Date: 2002-06-24 19:39
Logged In: YES 
user_id=491567

 This code is wrong: 
 
data += self.sslobj.read(len(data)-size) 
 
it should be size - len(data) 
 
I'm attaching a version that is known to work. 
msg11271 - (view) Author: Piers Lauder (pierslauder) * (Python triager) Date: 2002-06-26 10:30
Logged In: YES 
user_id=196212

Thanks for debugging! Fixed.

Tino Lange confirms the new version also works for him on
both linux and windows.

I'm marking this bug fixed.
History
Date User Action Args
2022-04-10 16:05:26adminsetgithub: 36770
2002-06-19 21:22:36jgoerzencreate