The tiny API response that used 12 MB of memory
There’s a particularly annoying way to run out of memory: after you’ve already added the check that was supposed to prevent it.
Say your API client starts struggling when requests pile up. You check the response sizes, expecting to find something huge. But the bodies are small. They’re comfortably under the limit.
So now you’re looking at more memory, smaller batches or fewer concurrent requests. All reasonable things to consider. But you still haven’t explained how such small downloads became so much work.
The bit that’s easy to miss is what happens after the download. A compressed body has to be unpacked. Then the JSON parser builds the lists, dictionaries and strings your application actually uses. Each step can leave you dealing with something much larger than the number that passed the check.
I put together a small local example using 25,000 synthetic fixture records. The compressed body was about 69 KB. It passed a 128 KiB size limit with room to spare.
Parsing its contents produced a 12.1 MB peak in traced Python allocations.
That gives us something useful to test: where should the limit run, and what does it actually stop?
One response has three different sizes
The records were deliberately ordinary. Each had an ID, two example team names and a status. I serialised them as compact JSON and compressed the document with gzip.
The decoded document was 32.45 times the compressed size. Peak traced allocations during parsing were about 175.5 times the compressed size. Those ratios describe this example; different records and runtimes will give different results.
I also tried gzip levels 1, 6 and 9. They produced bodies of 73,405, 70,414 and 68,990 bytes respectively. Every one expanded to the same 2,238,891-byte document. Better compression saved transfer bytes while leaving exactly the same input for the JSON parser.
There is a measurement detail worth keeping clear. Python’s tracemalloc tracks Python allocations. I started it immediately before json.loads() and stopped immediately afterwards. The existing compressed and decoded buffers, fixture generation and decompression were outside that window. Total process memory is a separate measurement.
The accompanying lab uses only Python’s standard library. On 64-bit Windows with Python 3.12.14 and zlib 1.3.2, five parsing runs each produced a peak of 12,108,305 traced bytes. MB here means one million bytes; KiB means 1,024 bytes.
Check the decoded bytes before parsing
A compressed-size check would wave this body through. To stop it reaching the JSON parser, the client needs to check the decoded bytes as it reads them.
For the local gzip fixture, the core of that check looks like this:
import gzip
import io
import json
def read_bounded_json(compressed, limit=128 * 1024):
if limit < 0:
raise ValueError("Limit must be non-negative")
with gzip.GzipFile(
fileobj=io.BytesIO(compressed), mode="rb"
) as source:
decoded = source.read(limit + 1)
if len(decoded) > limit:
raise ValueError("Decoded JSON exceeds the limit")
return json.loads(decoded)The extra byte is there for a reason. Reading exactly the limit would leave you unable to tell whether more data followed. Asking for one byte beyond it lets the client reject an oversized document.
With the 128 KiB decoded limit, the reader returned 131,073 bytes and the guard rejected the fixture. The parser was never called. I also checked the boundary against the full document: a limit equal to its decoded length accepted it; one byte below rejected it before parsing.
The 128 KiB value is an example, rather than a recommended default. Choose a limit that fits the documents your client actually needs.
This code starts with the compressed body already in memory. It caps the decoded bytes returned before parsing, while the gzip reader can still use internal buffers. It does not set a hard ceiling on process memory or decompression CPU.
For a real HTTP client, check where the library decompresses the body. Requests automatically decompresses gzip through iter_content(), while its raw stream behaves differently. Count the bytes at the stage you intend to limit, before buffering the entire decoded document.
A record limit can arrive too late
Say you also reject arrays containing more than 24,999 records. Our fixture contains 25,000, so that check catches it.
But if the check comes after json.loads(), Python has already built the array. I tested that too. The record limit rejected the result, but the parser had still run.
That check remains useful for stopping a large batch from reaching later calculations or writes. It cannot prevent allocations the parser has already made.
If you need to stop on record count during parsing, use an incremental parser that can stop as it reads records. Provider-side pagination can reduce batch size as well. Keep a decoded-byte limit alongside it, because a thousand short records and a thousand records with large text fields are very different inputs.
This is also relevant when consuming untrusted responses. Python’s JSON documentation warns that malicious input can consume considerable CPU and memory, and recommends limiting the size of data being parsed. This local test demonstrates a size mismatch without testing a third-party system or alleging a vulnerability.
Count the results you keep as well
There is another detail in the measurements. When parsing returned, about 9.87 MB of traced allocations remained live while the result was held. Finishing the parse had not released most of those allocations.
Think about a worker fetching several pages and putting the parsed results on a queue. The downloads might finish quickly. The results can stay around while the next stage catches up.
That is why a requests-per-minute limit is only part of the picture. The client also needs a limit on work in flight and on batches waiting to be processed. Otherwise, each response can satisfy its individual size limit while the application holds too many together.
I did not measure a concurrent workload in this lab. The retained-allocation result gives a reason to test one with your own data and queueing behaviour, rather than multiplying the single-parse peak and treating it as a capacity guarantee.
Give each limit a clear job
The checks I would look for in a client are quite specific:
Compressed bytes: While receiving and buffering the encoded body
Decoded bytes: While reading decompressed data and before JSON parsing
Record count: During incremental parsing or before later batch processing
Concurrent work and retained results: Before admitting more work or allowing queues to grow
Log which limit caused the rejection. A compressed-byte rejection tells you something different from a decoded-byte rejection or a full processing queue. That distinction makes the next debugging session much easier.
Take one representative response and measure all three sizes. Then send the client something just under each limit, exactly at it and just over it. For the decoded-byte guard, check that the oversized input never reaches the JSON parser.
Our 69 KB body passed the first check. Moving the next check ahead of parsing was what stopped it.
Rhyan Morice runs odds-api.net and works with API payloads, Python consumers and data-delivery contracts.
