Three long-running encoder processes got OOM-killed at 20-30 GB despite disciplined del + gc.collect() after every single unit of work. The Python heap was clean — the leak was one layer down, in glibc’s allocator. By default glibc doesn’t use a fixed mmap threshold; it raises the threshold every time a large mmap’d chunk is freed, on the bet that a similarly large allocation is coming again soon. For a workload that repeatedly allocates and frees many differently-sized, short-lived, multi-megabyte buffers, that bet is wrong: allocations get progressively demoted onto a heap arena that never gives pages back to the OS. The fix is one line — mallopt(M_MMAP_THRESHOLD, N) pins the threshold and disables the adaptive behavior for the rest of the process’s life. This applies to any long-running process (Python or otherwise) that cycles through many moderately-large, short-lived buffers — image tiles, video frames, ML batches, whatever the buffers happen to be.

A long-running Python 3.12 service leaked 32 OS threads every processing cycle, and even after that got fixed, RSS kept climbing — while every Python-level profiler (tracemalloc, pympler) reported a clean heap. The first bug was structural: nested asyncio.run() calls each spin up a fresh default ThreadPoolExecutor, and fire-and-forget tasks race against that executor’s shutdown, so threads survive cleanup. The second bug was one layer below Python entirely: glibc assigns each new thread to its own memory arena, and short-lived thread pools leave those arenas — and the committed memory inside them — behind forever. The third bug was a footgun in how you fix the second: os.environ.setdefault('MALLOC_ARENA_MAX', ...) silently does nothing, because glibc reads that variable once, before the Python process even starts. None of these three failure modes are specific to this service — they generalize to any long-running process that spawns threads through nested event loops or executors.