Multiprocessing

Unlike threading, which is constrained by the Global Interpreter Lock (GIL) for CPU-bound tasks, the multiprocessing package spins up entirely separate Python processes, each with its own interpreter and memory space.

This allows you to bypass the GIL and utilize all CPU cores on your machine, at the cost of higher startup overhead and the need to serialize (pickle) data passed between them.

Process Lifecycle Visualization

Multiprocessing Basics

multiprocessing sidesteps the GIL by not sharing an interpreter at all. Each worker is a genuine OS process with its own interpreter and its own GIL, so Python bytecode really does run on several cores at once.

Spawning a basic process

import multiprocessing
import os

def worker(name):
    print(f"Worker {name} running in process")

if __name__ == '__main__':
    p = multiprocessing.Process(target=worker, args=('A',))
    p.start()
    p.join()

Worker A running in process

Memory Isolation (The Catch)

Unlike threads, processes do not implicitly share memory. Each process has its own isolated memory space. Modifying a global variable in a worker will NOT modify it in the parent!

Memory isolation between processes

import multiprocessing

shared_list = []

def add_item(item):
    shared_list.append(item)
    print(f"Child: {shared_list}")

if __name__ == '__main__':
    p = multiprocessing.Process(target=add_item, args=(1,))
    p.start()
    p.join()
    print(f"Parent: {shared_list}")

Child: [1]
Parent: []

Using Pools for Data Parallelism

A Pool is the convenient front door: hand it a function and an iterable, and it spreads the work across however many worker processes you asked for, collecting the results in order.

Using Pool to map functions

import multiprocessing

def square(x):
    return x * x

if __name__ == '__main__':
    with multiprocessing.Pool(processes=3) as pool:
        results = pool.map(square, [1, 2, 3])
    print(results)

[1, 4, 9]

The Overhead of Processes

Spawning a process means launching an entire Python interpreter. For small tasks, the overhead of creating processes can exceed the benefit of parallelism.

import time
import multiprocessing

def heavy_computation(n):
    return sum(i * i for i in range(n))

work_size = 1000

print(f"Running task of size {work_size}...")

start = time.time()
heavy_computation(work_size)
print(f"Single Process Time: {time.time() - start:.4f}s")

start = time.time()
p = multiprocessing.Process(target=heavy_computation, args=(work_size,))
p.start()
p.join()
print(f"Multi Process Time: {time.time() - start:.4f}s")

Pickling Constraints

Nothing is shared for free. Arguments and return values are pickled, shipped across a pipe, and rebuilt on the other side — which is why lambdas and open file handles cannot be sent, and why tiny tasks can end up slower than doing the work in one process.

Pickling constraint for arguments

import multiprocessing

def worker(func):
    pass

if __name__ == '__main__':
    try:
        p = multiprocessing.Process(target=worker, args=(lambda: print("hi"),))
        p.start()
    except Exception as e:
        print("PickleError")

PickleError