This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: release GIL while doing I/O operations in the mmap module
Type: performance Stage: patch review
Components: Extension Modules Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, ZackerySpytz, adiroiban, brian.curtin, jnwatson, luks, neologix, pitrou
Priority: normal Keywords: patch

Created on 2006-10-08 00:26 by luks, last changed 2022-04-11 14:56 by admin.

Pull Requests
URL Status Linked Edit
PR 14114 open ZackerySpytz, 2019-06-15 15:46
Messages (8)
msg61261 - (view) Author: Lukas Lalinsky (luks) Date: 2006-10-08 00:26
There is a few system I/O calls in the mmap module that
can take some time. Would be be possible to put
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around
these to release the GIL and allow other Python threads
to do their work?
msg61262 - (view) Author: Josiah Carlson (josiahcarlson) * (Python triager) Date: 2006-10-16 22:12
Logged In: YES 
user_id=341410

Can you come up with the listing of code blocks where it
would make sense to allow threads, or even provide a patch?
msg84542 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-30 12:26
I don't know mmap myself but patches are welcome.
msg102453 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2010-04-06 11:20
As soon as you're dealing with files (not anonymous mapping), you can get the same type of latency than when using open/read/write...
While it's probably not worth the trouble to release the GIL for every operation involving mmaped-files, there are a couple places where it should be considered, for example:
Modules/mmapmodule.c:new_mmap_object
#ifdef HAVE_FSTAT
#  ifdef __VMS
	/* on OpenVMS we must ensure that all bytes are written to the file */
	if (fd != -1) {
	        fsync(fd);
	}
#  endif

fsync() can take up to a couple seconds, so we should probably allow threads here (this is done only on VMS).
Another place worth looking at is when using msync(), like in mmap_object_dealloc():
	if (m_obj->data!=NULL) {
		msync(m_obj->data, m_obj->size, MS_SYNC);
		munmap(m_obj->data, m_obj->size);
	}

or in mmap_flush_method():
	if (-1 == msync(self->data + offset, size, MS_SYNC)) {
		PyErr_SetFromErrno(mmap_module_error);
		return NULL;
	}

msycn too can take quite a long time to complete.

I can write a patch if someone's willing to review it.
msg102465 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-06 13:56
This shouldn't really care about VMS here. As for the rest, feel free to propose a patch :)
Please note that most non-trivial system calls, such as ftruncate(), mremap(), even mmap() itself, deserve to be enclosed in Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS pairs.

However, you'll have to be careful that the internal state of the mmap object remains consistent, such that using it from several threads doesn't crash the interpreter.
(you can even add multi-threaded unit tests if you want to be sure of this)
msg102466 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-06 13:58
> This shouldn't really care about VMS here.

Correction: /we/ shouldn't really care about VMS here.
And the reason being, of course, that we have neither developers nor
known testers under VMS.
(we don't even know if the current trunk builds there or not)
msg342561 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2019-05-15 11:10
Any news? If a patch is not ready, i can work on a patch too.
msg343989 - (view) Author: Nic Watson (jnwatson) Date: 2019-05-30 17:57
I'll add one more system I/O call that's not GIL-wrapped in the mmap module that can take some time:  mmap itself.

mmap on Linux with MAP_POPULATE (0x8000) as the flags can take quite a bit of time.  That's the flag that prefaults the memory range.  MAP_POPULATE has been around since Linux 2.5.46.

I know that MAP_POPULATE isn't explicitly supported in the module, but mmap.mmap does take arbitrary flags, so it isn't exactly unsupported either.
History
Date User Action Args
2022-04-11 14:56:20adminsetgithub: 44098
2019-06-15 15:48:30ZackerySpytzsetnosy: + ZackerySpytz

components: + Extension Modules, - Library (Lib)
versions: + Python 3.9, - Python 3.1, Python 2.7
2019-06-15 15:46:40ZackerySpytzsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request13963
2019-05-30 17:57:27jnwatsonsetnosy: + jnwatson
messages: + msg343989
2019-05-15 11:10:35BTaskayasetnosy: + BTaskaya
messages: + msg342561
2019-05-02 04:31:53josiahcarlsonsetnosy: - josiahcarlson
2012-04-12 22:03:09adiroibansetnosy: + adiroiban
2010-04-06 14:09:09brian.curtinsetnosy: + brian.curtin
2010-04-06 13:58:17pitrousetmessages: + msg102466
2010-04-06 13:56:05pitrousetmessages: + msg102465
2010-04-06 11:20:40neologixsetnosy: + neologix
messages: + msg102453
2009-03-30 12:27:00pitrousetpriority: normal

type: enhancement -> performance
versions: + Python 3.1
nosy: + pitrou

messages: + msg84542
stage: test needed -> needs patch
2009-03-30 07:21:47ajaksu2setpriority: normal -> (no value)
stage: test needed
versions: + Python 2.7
2006-10-08 00:26:31lukscreate