admin When it comes to Python, there are some oddities to keep in mind. We know that threads share the same memory space, so special precautions must be taken so that two threads don’t write to the same memory location. The CPython interpreter handles this using a mechanism called python multiprocessing vs threading GIL, or the Global Interpreter Lock. Threading is one of the most well-known approaches to attaining parallelism and concurrency in Python. Threading is a feature usually provided by the operating system. Threads are lighter than processes, and share the same memory space.
- The threading module was developed first and it was a specific intention of the multiprocessing module developers to use the same API, both inspired by Java concurrency.
- An interesting real world example is Pytorch Dataloader, which uses multiple subprocesses to load the data into GPU.
- The performance does increase in places where C extensions like numpy, Network, I/O are being used, where a lot of background work is done and GIL is released.
- CPU-bound tasks are normally heavy weight and will provide excellent results when they work with multiprocessing modules in Python.
- Multiprocessing (the right diagram) multiplies a single processor, replicating the code, data and files, which incurs more overhead.
- Both the threading module and the multiprocessing module are intended for concurrency.
Note that the last two on the list, queues and memoryview, are
technically mutable data types, whereas the rest are not. When any
interpreters share mutable data there is always a risk of data races. Cross-interpreter safety, including thread-safety, is a fundamental
feature of queues.
Use Threading for for IO-Bound Tasks
While the de facto reference Python implementation—CPython–has a GIL, this is not true of all Python implementations. For example, IronPython, a Python implementation using the .NET framework, does not have a GIL, and neither does Jython, the Java-based implementation. In fact, the API originally had many camel-case function names, like those in Java, that were later changed to be more Python compliant names via PEP 8 – Style Guide for Python Code in 2001.
Runtime finalization has only a slight, indirect effect on still-running
Python threads, whether in the main interpreter or in subinterpreters. That’s because right away it waits indefinitely for all non-daemon
Python threads to finish. As noted earlier, there is some runtime state that multiple OS threads
share. Some of it is exposed by the sys module, though much is
used internally and not exposed explicitly or only through the C API. Any OS thread may switch which thread state it is currently using, as
long as it isn’t one that another OS thread is already using (or has
been using).
- By putting each job in a Multiprocessing process, each can run on its own CPU and run at full efficiency.
- The subsequent steps to call t1.start() and t1.join() remain the same.
- The multiprocessing library gives each process its own Python interpreter, and each their own GIL.
- Starting a thread is considered to be faster than starting a process.
- The GIL (Global Interpreter Lock) ensures that there is no more than one thread in a state of execution at any one given moment with the CPython interpreter.
By splitting off the database connection into a separate thread you can make the application more responsive. Also because both threads are in the same process, they can access the same data structures – good performance, plus a flexible software design. For some types (builtin singletons), the actual object is shared. For some, the object’s underlying data is actually shared but each
interpreter has a distinct object wrapping that data. For all other
shareable types, a strict copy or proxy is made such that the
corresponding objects continue to match exactly. In cases where
the underlying data is complex but must be copied (e.g. tuple),
the data is serialized as efficiently as possible.
From the Perspective of a Data Scientist
There is no requirement that an interpreter have any thread states,
except as soon as the interpreter is meant to actually be used. When there’s more than one thread state for an OS thread,
PyThreadState_Swap() is used in that OS thread to switch
between them, with the requested thread state becoming the current one. Whatever was running in the thread using the old thread state is
effectively paused until that thread state is swapped back in. GIL helps in the execution of only one thread at one point in time. Let’s say you have a classification problem, and you want to use a random forest classifier for this. As it’s a standard and well-known machine learning algorithm, let’s not reinvent the wheel and just use sklearn.ensemble.RandomForestClassifier.
Parallel Loops in Python
After that, there are only a few small changes made to the existing code. We first create an instance of an RQ Queue and pass it an instance of a Redis server from the redis-py library. Then, instead of just calling our download_link method, we call q.enqueue(download_link, download_dir, link). The enqueue method takes a function as its first argument, then any other arguments or keyword arguments are passed along to that function when the job is actually executed. The Python multiprocessing module is easier to drop in than the threading module, as we don’t need to add a class like the Python multithreading example. Additionally, the operating system may impose limits on the total number of processes supported by the operating system, or the total number of child processes that can be created by a process.
Process
The properties of the thread can be set and configured, e.g. name and daemon, and one thread may join another, causing the caller thread to block until the target thread has terminated. Built In’s expert contributor network publishes thoughtful, solutions-oriented stories written by innovative tech professionals. It is the tech industry’s definitive destination for sharing compelling, first-person accounts of problem-solving on the road to innovation. Whenever a function wants to modify a variable, it locks that variable. When another function wants to use a variable, it must wait until that variable is unlocked.
Additionally, the GIL is a design decision that affects the reference implementation of Python. If you use a different implementation of the Python interpreter (such as PyPy, IronPython, Jython, and perhaps others), then you may not be subject to the GIL and can use threads for CPU bound tasks directly. We can free-up the CPU from IO-bound operations by performing IO-bound operations on another thread of execution.
Multiprocessing refers to the ability of a system to run multiple processors in parallel, where each processor can run one or more threads. Without multiprocessing, Python programs have trouble maxing out your system’s specs because of the GIL (Global Interpreter Lock). Python wasn’t designed considering that personal computers might have more than one core (which shows you how old the language is). Only objects that are specifically supported for passing
between interpreters may be sent through a Queue. Note that the actual objects aren’t sent, but rather their
underlying data. However, the popped object will still be
strictly equivalent to the original.
Fundamentally, an “interpreter” is the collection of (essentially)
all runtime state which Python threads must share. Attempting the same with concurrent.futures is by no means pretty. You could use things such as callbacks, queues, etc., but it would be significantly harder to manage than basic asyncio code. Asyncio is essentially threading where not the CPU but you, as a programmer (or actually your application), decide where and when does the context switch happen. In Python you use an await keyword to suspend the execution of your coroutine (defined using async keyword). Firstly, the per-thread running time for requests is obviously lower than for selenium, as the latter requires spinning up a new process to run a PhantomJS headless browser.
Don’t Use Multiprocessing for IO-Bound Tasks
Now only one json is unloaded into memory at a time, and everything is fine. So basically stick to threading unless you have IO/CPU problems. In general, we cannot benefit from (1) with threading but we can benefit from (2).