How a gRPC client leak crashed our BEAM: a literal_alloc post-mortem

The crash

Our ingestion tier holds one long-lived gRPC subscription to the Salesforce Pub/Sub API per connected org. One morning a VM died with a slogan most Elixir developers never see:

literal_alloc: Cannot allocate 1048576 bytes of memory (of type "literal").

Not a heap OOM. The BEAM's literal area, the region that holds compile-time constants, was exhausted. Total memory on the box was fine. Nothing in our code stores anything unusual. And the crash recurred, always after periods of network instability.

Why persistent_term can eat the literal area

The clue is that :persistent_term storage lives in the same literal area. It's a wonderful facility with one sharp property: it's designed for values written once and read forever. Erasing a term is expensive (it can trigger a global GC), and the area itself is bounded, by default to a ~1GB super-carrier. Anything that puts unbounded or repeatedly-replaced values into persistent_term is a slow-motion VM crash.

grpc-elixir 0.11 parks every channel's load-balancing state in persistent_term at connect time, including the connection credential. For a TLS connection that means the cacerts bundle, roughly 600KB per entry. And it never erases the entry.

Steady-state, that's a small fixed cost and you'd never notice. Under reconnect churn (a flaky NAT, a maintenance window, an org with a revoked token retrying), each reconnect parks another ~600KB copy in an area nothing reclaims. We measured the leak at about 2MB per minute during sustained reconnects. The ~1GB ceiling arrives in hours.

The disconnect that can never succeed

The obvious fix, calling the library's disconnect, turned out to be impossible in that version. The disconnect handler pattern-matches the channel shape stored by the load-balanced connect path:

# grpc-elixir 0.11.5, paraphrased:
# build_direct_state/4 stores:      real_channels: %{key => channel}
# handle_call({:disconnect, ..}) matches: %{key => {:ok, channel}}
#
# Direct connections store the bare channel, the disconnect clause expects
# the {:ok, channel} tuple - so every disconnect call crashes the Connection
# GenServer with a FunctionClauseError before it disconnects anything.

So for direct connections disconnect didn't leak. It crashed, which is worse, because of what happens next.

The erase-then-resurrect race

The Connection process runs under a supervisor with a :transient restart strategy: crashed processes restart. The restarted Connection re-puts its persistent_term entry. Which means the naive cleanup below silently loses to the supervisor.

:persistent_term.erase(key)  # ...and 40ms later it's back

We confirmed it with :dbg tracing: erase, restart, re-put, in that order, every time. The fix has to kill the process through its supervisor first, then erase, then verify the term stayed gone:

defp reap_connection(key, pid) do
  # 1. Terminate through the DynamicSupervisor so :transient can't restart it.
  #    (Brutal kill as fallback - the process traps exits, so :shutdown alone
  #    would be ignored.)
  DynamicSupervisor.terminate_child(GRPC.Client.Supervisor, pid)

  # 2. Now erase - and re-check after a settle window, bounded to 5 attempts,
  #    because a concurrent restart can re-put the term right after any erase.
  :persistent_term.erase(key)
  verify_erased(key, attempts: 5, settle_ms: 40)
end

Two adjacent fixes rode along. The stream-close path now does a full channel teardown (the channel process previously leaked on every error close). And backoff now resets only when a pushed response proves the stream healthy: Salesforce accepts a subscribe before rejecting bad auth, so resetting on subscribe success had turned exponential backoff into a ~1-second hammer for revoked tokens. That backoff bug was the churn amplifier that made the leak fatal in the first place.

Verification, with a control group

Claims about fixed leaks deserve measurement. We soaked two VMs overnight under sustained forced reconnects: one patched, one not.

patched:   lb_state persistent_terms oscillate between 0 and 4 (bounded)
unpatched: monotonic growth ~2MB/min → identical literal_alloc abort overnight

The unpatched VM reproducing the exact crash slogan is what turned "plausible theory" into "root cause".

Takeaways

If you run grpc-elixir with reconnecting clients, check your persistent_term usage today: :persistent_term.get() |> length() trended over time tells you in a minute whether you're accumulating. Watch for entries keyed by channel.

If you maintain a library: persistent_term is for write-once values. Anything with connection lifetime doesn't qualify.

If you're designing a supervision tree: cleanup that races a restart strategy isn't cleanup. Terminate through the supervisor, then clean, then verify. A bare erase next to a :transient child is a bug with a delay on it.