multiprocessing (Fork vs Spawn) vs threading vs concurrent.futures
- Last updated
- Prerequisites:
- CPython Global Interpreter Lock (GIL)
- OS Processes vs Threads
- python
- multiprocessing
- threading
- concurrent-futures
- ipc
- fork-vs-spawn
Read at your depth
The practical view
`threading` is suited for I/O-bound tasks (network, disk) where threads release the GIL during syscalls. `multiprocessing` bypasses the GIL entirely by spawning separate OS processes, enabling true multi-core CPU parallelism. `concurrent.futures.ProcessPoolExecutor` and `ThreadPoolExecutor` provide a unified `Future` interface. When creating processes, Python supports `'spawn'` (safe, default on macOS/Windows and Python 3.14+) and `'fork'` (fast on Linux, but causes deadlocks with threads).
The same idea in other frameworks
`ProcessPoolExecutor` (multi-process) vs `ThreadPoolExecutor` (multi-thread).`worker_threads` (isolated V8 isolates) vs `child_process.fork()`.Goroutines multiplexed over OS threads across multiple CPU cores without GIL.`std::thread` and Rayon data-parallel worker pools.Legacy vs modern
GIL-Constrained Threading vs Multi-Core ProcessPoolExecutor
Threads cannot parallelize pure Python CPU loops under the GIL; `ProcessPoolExecutor` utilizes all CPU cores.
with ThreadPoolExecutor() as pool: pool.map(heavy_cpu, data) # ❌ 0% speedup on CPU workwith ProcessPoolExecutor() as pool: pool.map(heavy_cpu, data) # ✅ NEW: Scales across all coresInterview gotchas
Context
Interviewers test your deep understanding of POSIX process cloning and deadlocks.
The mechanical answer
When a process calls `fork()`, POSIX duplicates only the calling thread into the child process; all other threads vanish. If any background thread in the parent (e.g. an internal logging thread, database pool, or C-extension lock) was holding a mutex lock at the moment of `fork()`, that lock remains permanently **locked** in the child process with no surviving thread to release it. Any child code attempting to acquire that lock will **deadlock forever**. `spawn` boots a clean new interpreter, avoiding lock inheritance.
Trap
Never use `multiprocessing.set_start_method('fork')` in applications using multi-threaded libraries or gRPC.
Context
Checks knowledge of IPC serialization and pickling overhead.
The mechanical answer
`multiprocessing` communicates over pipes using `pickle`. Serializing a 5GB object into bytes (`pickle.dumps`), copying it across an OS pipe socket, and deserializing it (`pickle.loads`) in the child process creates massive CPU memory copying overhead. For large datasets, use **`multiprocessing.shared_memory.SharedMemory`** or memory-mapped files (`mmap`) for zero-copy access.
Trap
Large data passed as arguments to `multiprocessing` workers will spike memory and serialization latency.