OutOfMemoryError Playbook (Symptoms → Causes)

OutOfMemoryError has several subtypes, each corresponding to different causes. This article gives a quick reference: symptom → likely cause → diagnosis and fix, plus startup flags, dump analysis, and common remedies.

Overview

• java.lang.OutOfMemoryError: Java heap space: Heap cannot allocate a new object. Causes: leak, heap too small, single allocation too large.
• java.lang.OutOfMemoryError: Metaspace: Class metadata (method area) exceeded. Causes: too many classes, dynamic class generation, CGLIB proxies, hot reload.
• java.lang.OutOfMemoryError: GC overhead limit exceeded: GC took >98% of time and reclaimed <2% of heap. Usually means heap nearly full and ineffective collection.
• java.lang.OutOfMemoryError: Direct buffer memory: Off-heap direct memory exhausted. Causes: NIO direct buffers, Netty buffers not released, or -XX:MaxDirectMemorySize too low.
• java.lang.OutOfMemoryError: unable to create new native thread: Native thread limit reached. Causes: too many threads, low ulimit, large stack size.

Example

Example 1: Heap OOM — enable dump and logs

Plain textCopy
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/heap.hprof
-Xmx2g -Xms2g

• Dump is created on OOM. Use MAT, JProfiler, etc. to see dominant objects and GC roots holding them.

Example 2: Metaspace OOM

Plain textCopy
-XX:MetaspaceSize=128m
-XX:MaxMetaspaceSize=256m

• Cap Metaspace to avoid unbounded growth. Check: dynamic class generation, Groovy/scripts, CGLIB proxies, hot reload, and class unloading.

Example 3: Direct buffer OOM

• Inspect Netty, NIO ByteBuffer.allocateDirect usage for leaks or missed release. You can increase -XX:MaxDirectMemorySize (default is roughly -Xmx).

Example 4: Symptom → cause → action

Error message	Common causes	Action
Java heap space	Leak, small heap, large object	Dump analysis, increase heap, fix leak, chunk large objects
Metaspace	Too many classes, proxies, hot reload	Set MaxMetaspaceSize, audit class loading, reduce dynamic classes
GC overhead limit exceeded	Heap full, ineffective GC	Check leak, increase heap, large objects, tune GC
Direct buffer memory	Off-heap leak, low cap	Check NIO/Netty release, tune MaxDirectMemorySize
unable to create new native thread	Too many threads, ulimit	Reduce threads, use pools, tune ulimit, reduce -Xss

Core Mechanism / Behavior

• Heap: Objects live on heap. When allocation fails, heap space OOM is thrown. Full GC runs first; if still insufficient, OOM follows.
• Metaspace: Class metadata lives in Metaspace (replaces PermGen). Default is unbounded; it may grow until system limit or OOM.
• Direct memory: ByteBuffer.allocateDirect uses off-heap memory limited by MaxDirectMemorySize. GC reclaims via Cleaner; if references are held, leaks occur.

Key Rules

• In production, enable -XX:+HeapDumpOnOutOfMemoryError and set a path so dumps are created for later analysis.
• Map error type to area: heap → objects; Metaspace → class loading; Direct → off-heap and Netty; thread → thread count and ulimit.
• Fix leaks and misuse first, then consider increasing capacity. Raising -Xmx alone may only postpone OOM.

What's Next

See GC Basics, G1 Overview, GC Tuning for heap and GC. See Java Profiling for memory analysis tools.