Python asyncio Recipes/Process input in batches

Break large workloads into fixed-size async chunks.

Section: Bounded Concurrency

Process input in batches

python
python
import asyncio

async def process(x):
    await asyncio.sleep(0.05)
    return x * 2

async def run_batch(items, size=10):
    results = []
    for i in range(0, len(items), size):
        batch = items[i:i+size]
        results.extend(await asyncio.gather(*(process(x) for x in batch)))
    return results

print(asyncio.run(run_batch(list(range(25)))))
Explanation

Batches help when you want memory control or stricter request pacing.

Learn the surrounding workflow

Compare similar commands or jump into common fixes when this command is part of a bigger troubleshooting path.

Related commands

Same sheet · prioritizing Bounded Concurrency
Bound a worker pool with Semaphore
Throttle async work to a safe level.
OpenIn sheetpythonsame section
Retry an async operation
Retry with simple exponential backoff.
Inspect running tasks
List currently active tasks in the running loop.
Cancel cleanly on shutdown
Maintain and cancel outstanding tasks at shutdown time.
Enable asyncio debug mode
Turn on extra diagnostics while developing.