1. Summary

Three long-running encoder processes got OOM-killed at 20-30 GB despite disciplined del + gc.collect() after every single unit of work. The Python heap was clean — the leak was one layer down, in glibc’s allocator. By default glibc doesn’t use a fixed mmap threshold; it raises the threshold every time a large mmap’d chunk is freed, on the bet that a similarly large allocation is coming again soon. For a workload that repeatedly allocates and frees many differently-sized, short-lived, multi-megabyte buffers, that bet is wrong: allocations get progressively demoted onto a heap arena that never gives pages back to the OS. The fix is one line — mallopt(M_MMAP_THRESHOLD, N) pins the threshold and disables the adaptive behavior for the rest of the process’s life. This applies to any long-running process (Python or otherwise) that cycles through many moderately-large, short-lived buffers — image tiles, video frames, ML batches, whatever the buffers happen to be.

This started as what looked like an open-and-shut ZFS bug: a kernel panic with a clean stack trace, matching a known upstream issue that already had its own PR number attached. It wasn’t. The actual root cause sat one layer below the kernel entirely — and getting from “random freezes” to “confirmed fixed” meant cross-compiling ZFS off-machine because the live system couldn’t survive its own build, then reverse-engineering an undocumented Ubuntu encryption mechanism that no upstream package ships. Here’s the full chain, and what’s worth keeping from it.

1. Summary

A long-running Python 3.12 service leaked 32 OS threads every processing cycle, and even after that got fixed, RSS kept climbing — while every Python-level profiler (tracemalloc, pympler) reported a clean heap. The first bug was structural: nested asyncio.run() calls each spin up a fresh default ThreadPoolExecutor, and fire-and-forget tasks race against that executor’s shutdown, so threads survive cleanup. The second bug was one layer below Python entirely: glibc assigns each new thread to its own memory arena, and short-lived thread pools leave those arenas — and the committed memory inside them — behind forever. The third bug was a footgun in how you fix the second: os.environ.setdefault('MALLOC_ARENA_MAX', ...) silently does nothing, because glibc reads that variable once, before the Python process even starts. None of these three failure modes are specific to this service — they generalize to any long-running process that spawns threads through nested event loops or executors.

1. The General Problem

The shape of this problem is generic: M consumer call sites need N data types, and each data type can be produced by more than one independent upstream source, each with its own event or payload shape. Left to grow organically, the naive fix is a fallback branch at every call site — an M × N × (source count) surface area, where each branch also has to encode which source to trust first and how to resolve partial failures.

The Symptoms

WavePy is a Python 3.12-based ocean wave data processing system running in a Conda environment called wave312. During recent development, the process started vanishing without warning. No Python traceback, no error logs, just gone.

The last line of the log always stopped at the memory monitor output for scipy.interpolate.griddata:

2026-03-02 01:23:43,251 - MemoryMonitor - INFO - [MEM][interpolate_unstruct_grid:hsig:20260302_060000] RSS: 1440.44 MB

After that, nothing. No traceback, no exception. It was a classic C extension segfault.

SSH is an application-layer encrypted network protocol. It’s used for remote login, remote command execution, and data transfer.

It consists of an SSH client and an SSH server. There are many implementations, with OpenSSH being the default on Ubuntu. The client is ssh, and the server is sshd.