This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_resource fails when file size is limited
Type: behavior Stage: needs patch
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ajaksu2, giampaolo.rodola, loewis, mdr0, nnorwitz, r.david.murray, sable, wiggin15
Priority: normal Keywords:

Created on 2003-01-31 18:06 by nnorwitz, last changed 2022-04-10 16:06 by admin.

Messages (18)
msg14342 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2003-01-31 18:06
test_resource does:

   print resource.RLIM_INFINITY == max

I'm not sure of the benefit of this line.  For machines 
which have limited file sizes, it causes the test to fail.  
Otherwise the test seems to work properly.
msg14343 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2003-03-03 22:17
Logged In: YES 
user_id=33168

Martin, welcome back!  Now I get to assign some bugs to you
to see if you have any input. :-)
msg14344 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-03-30 16:15
Logged In: YES 
user_id=21627

The rationale of the line is explained in the comment: The
code tests whether the limit is a really large number, i.e.
whether represenation of large limits works.

Unfortunately, it doesn't  (which is a separate issue): On
Linux, RLIM_INFINITY is reported as -1. This comes from
RLIM_INFINITY being the larges unsigned long long number,
i.e. 0xffffffffffffffffuLL. The resource module represents
this incorrectly.

I think there is no way to verify that Python computes the
real limit correctly (unless we invoke the shell's ulimit
for comparison). So you could weaken the test to verify that
the limit is non-negative (and, as a side effect, verify
that it doesn't raise an exception).

If you do so, you'll also have to fix the implementation, so
that the weakened-strengthened test passes on Linux.
msg14345 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2003-03-30 20:25
Logged In: YES 
user_id=33168

Hmmm, sounds like a lot of work.  How about something like this:

    expected_max = resource.RLIM_INFINITY
    if expected_max != max:
        # read the file resource limit
        fsize_ulimit = os.popen('ulimit -f').read()
        try:
            # convert the file size to bytes (from 512 byte
blocks)
            expected_max = int(fsize_ulimit.strip()) * 512
        except ValueError:
            raise TestSkipped, "unable to determine expected
resource value"
    print expected_max == max

This works, but I'm pretty sure this is not portable.  On
Linux blocks are 1k.  This only works when the block size is
512 bytes.  Shall we close this bug as a test environment
problem and won't fix?
msg14346 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-03-30 20:49
Logged In: YES 
user_id=21627

I'm really not that much concerned about fixing the test,
but more about fixing the code itself.
msg14347 - (view) Author: Mark D. Roth (mdr0) Date: 2004-03-10 17:26
Logged In: YES 
user_id=994239

I'm running into this problem under AIX 4.3.3 and 5.1.  Is
this going to cause a problem if I put python into
production, or is it "safe" to ignore it?
msg81850 - (view) Author: Daniel Diniz (ajaksu2) * (Python triager) Date: 2009-02-13 02:32
Is this still a problem?
msg114216 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-18 12:50
Closed as no response to msg81850.
msg116449 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-09-15 14:18
The test does not fail if FSIZE is not max on linux/py3k, but max is still represented as -1 on linux.  So the reported bug is no longer valid, but Martin's concern has not been addressed.
msg116479 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-09-15 20:03
As a point of information, on my gentoo linux system without largefile support in the kernel, any value 4294967295 or above results in getrlimit reporting -1.  Any smaller value is set and reported as itself.  (If a sufficiently large value is passed in to setrlimit, an OverflowError results.)
msg116520 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-09-16 11:24
> As a point of information, on my gentoo linux system without
> largefile support in the kernel, any value 4294967295 or above
> results in getrlimit reporting -1.  Any smaller value is set and
> reported as itself.  (If a sufficiently large value is passed in to
> setrlimit, an OverflowError results.)

I think we really should create new issues for any remaining problems.

AFAICT, the remaining problems are:
- resource.RLIM_INFINITY is -1 on Linux, when it is meant to be
   a really large value
- (as you reported) getrlimit returns -1 to indicate RLIM_INFINITY.

I think the core of the problem is that the resource module considers
rlim_t to be a signed type (or at least representable in "long long").
Using FromUnsignedLongLong in the appropriate place might solve the
issue.
msg117049 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-21 13:53
For info:
this test fails on AIX 6.1 with py3k with the following error:

test test_resource failed -- Traceback (most recent call last):
  File "/san_cis/home/cis/buildbot/buildbot-aix6/py3k-aix6-xlc/build/Lib/test/test_resource.py", line 28, in test_fsize_ismax
    self.assertEqual(resource.RLIM_INFINITY, max)
AssertionError: 2147483647 != 1073741312

The same test works on AIX 5.3.

Is it related or should I open a new issue?
msg117051 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-21 14:21
I suppose the difference comes from the fact that I have:
on AIX 5.3: 
$ ulimit -f
unlimited

on AIX 6.1:
$ ulimit -f 
1048575

I think the test should be updated to not require "ulimit -f" is unlimited.
msg117122 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-09-22 01:37
That's what the original report is about, as opposed to the linux repr issue that Martin wants to break out into a new ticket (which I will do).  Any ideas how to fix the test?  It didn't fail for me on linux, so I don't have a good test platform for trying to fix it.

I'm un-assigning Neal since the original report is no longer the cause of the test error.  (The test has obviously changed since we don't check print output any more.)
msg117130 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-22 09:58
The ideal would be to check that RLIMIT_FSIZE corresponds to the ulimit as it has been suggested by Neal Norwitz in msg14345, but since the value reported by ulimit has a different unit for each platform, that would be quite a lot of trouble.

All I can suggest is the following, which checks that both values are somewhat reasonable.

Index: Lib/test/test_resource.py
===================================================================
--- Lib/test/test_resource.py	(révision 84964)
+++ Lib/test/test_resource.py	(copie de travail)
@@ -20,12 +20,17 @@
         except AttributeError:
             pass
         else:
-            # RLIMIT_FSIZE should be RLIM_INFINITY, which will be a really big
-            # number on a platform with large file support.  On these platforms,
-            # we need to test that the get/setrlimit functions properly convert
-            # the number to a C long long and that the conversion doesn't raise
-            # an error.
-            self.assertEqual(resource.RLIM_INFINITY, max)
+            # RLIMIT_FSIZE should be RLIM_INFINITY if 'ulimit -f' is
+            # set to unlimited
+            # RLIM_INFINITY will be a really big number on a platform
+            # with large file support.  On these platforms, we need to
+            # test that the get/setrlimit functions properly convert
+            # the number to a C long long and that the conversion
+            # doesn't raise an error.            
+            self.assertGreater(resource.RLIM_INFINITY, 0)
+            self.assertLessEqual(cur, max)
+            self.assertGreater(max, 0)
+            self.assertLessEqual(max, resource.RLIM_INFINITY)
             resource.setrlimit(resource.RLIMIT_FSIZE, (cur, max))
 
     def test_fsize_enforced(self):
msg221696 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-27 17:27
The inline patch in msg117130 has never been committed from what I can see.  Can somebody review it please as I'm assuming that it's still valid.
msg227778 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-09-28 21:36
Can somebody take a look at this please.  See also #9917.
msg251772 - (view) Author: Arnon Yaari (wiggin15) * Date: 2015-09-28 16:37
This issue is still relevant and can be reproduced by running:

    ulimit -Hf 1000
    ./python Lib/test/test_resource.py

(On AIX, the default hard limit is not RLIM_INFINITY so it reproduces on that operating system without the first line)

The patch is still relevant and fixes the issue.
It looks like it's doing the right thing, but it can't be committed until #9917 will be fixed - otherwise the tests will break on Linux, where RLIM_INFINITY is -1 and not greater than 0.
History
Date User Action Args
2022-04-10 16:06:23adminsetgithub: 37883
2021-12-17 00:35:56ajaksu2setversions: + Python 3.9, Python 3.10, Python 3.11, - Python 2.7, Python 3.4, Python 3.5
2019-02-24 22:37:46BreamoreBoysetnosy: - BreamoreBoy
2015-09-28 16:37:21wiggin15setnosy: + wiggin15
messages: + msg251772
2015-08-07 10:51:36giampaolo.rodolasetnosy: + giampaolo.rodola
2014-09-28 21:36:35BreamoreBoysetmessages: + msg227778
2014-06-27 17:27:16BreamoreBoysetnosy: + BreamoreBoy

messages: + msg221696
versions: + Python 2.7, Python 3.4, Python 3.5, - Python 3.2
2010-09-22 09:58:44sablesetmessages: + msg117130
2010-09-22 01:37:23r.david.murraysetassignee: nnorwitz ->
components: - Library (Lib)
versions: + Python 3.2
nosy: - BreamoreBoy

messages: + msg117122
stage: test needed -> needs patch
2010-09-21 14:21:20sablesetmessages: + msg117051
2010-09-21 13:53:42sablesetmessages: + msg117049
2010-09-16 11:24:08loewissetmessages: + msg116520
2010-09-15 20:03:00r.david.murraysetmessages: + msg116479
2010-09-15 14:18:18r.david.murraysetnosy: + r.david.murray
messages: + msg116449
2010-09-14 13:07:35sablesetnosy: + sable
2010-08-18 12:50:05BreamoreBoysetnosy: + BreamoreBoy
messages: + msg114216
2009-02-13 02:32:36ajaksu2setnosy: + ajaksu2
type: behavior
messages: + msg81850
components: + Tests
stage: test needed
2003-01-31 18:06:24nnorwitzcreate