snipz

cheat sheet

tqdm

Add auto-updating progress bars to any Python loop or CLI pipeline with tqdm. Covers iterables, manual updates, pandas integration, nested bars, async, Jupyter, and byte-piping.

#python#tqdm#progress#cli#jupyter#asyncupdated 04-27-2026

tqdm — Progress Bars

What it is

tqdm wraps any Python iterable and displays an auto-updating progress bar with ETA, rate, and elapsed time in the terminal or Jupyter notebook. It adds less than 60 ns of overhead per iteration and requires no configuration beyond installation — just wrap an iterable and the bar appears. It is the de-facto standard for loop progress in Python data pipelines.

Install

bash
pip install tqdm

Output: (none — exits 0 on success)

Quick example

python
from tqdm import tqdm
import time

for _ in tqdm(range(100)):
    time.sleep(0.02)

Output:

text
100%|████████████████████████████| 100/100 [00:02<00:00, 49.8it/s]

When / why to use it

  • Long-running loops over files, API pages, dataset rows, or model batches where ETA matters.
  • CLI tools where a bar communicates liveness to the user.
  • Data pipelines with pandas (.progress_apply) or PyTorch data loaders.
  • Notebooks where it renders as an HTML widget.
  • Shell pipelines where it acts as a byte-throughput meter (like pv).

Common pitfalls

Wrapping a generator with unknown length — if total= is omitted and tqdm cannot infer the length from len(), the ETA shows ? and the percentage is hidden. Always pass total=len(items) when the iterable is a generator.

Nested bars without position= — each inner bar overwrites the outer bar on the same line. Use position=0 on the outer bar and position=1, leave=False on the inner bar.

print() inside a tqdm loop — ordinary print interleaves with bar output and creates visual noise. Replace with tqdm.write("msg"), which prints above the bar without disturbing it.

tqdm(iterable, desc="Loading") is the most readable one-liner. The desc= prefix appears left of the bar and doubles as a log label when redirected.

from tqdm.auto import tqdm auto-selects the notebook HTML widget when running in Jupyter and falls back to the terminal bar everywhere else — the cleanest single import for code that runs in both environments.

Richer example — file hashing pipeline

python
from tqdm import tqdm
import pathlib, hashlib

files = list(pathlib.Path(".").glob("**/*.py"))

results = {}
with tqdm(files, desc="Hashing", unit="file", colour="green") as pbar:
    for path in pbar:
        pbar.set_postfix(file=path.name, refresh=False)
        results[str(path)] = hashlib.md5(path.read_bytes()).hexdigest()

print(f"Hashed {len(results)} files")

Output:

text
Hashing: 100%|██████████| 42/42 [00:00<00:00, 312.4file/s, file=utils.py]
Hashed 42 files

trange — range shorthand

trange(n) is identical to tqdm(range(n)) and is the idiomatic way to wrap a counter loop. It accepts all the same keyword arguments as tqdm.

python
from tqdm import trange
import time

total = 0
for i in trange(50, desc="Summing", unit="step"):
    total += i
    time.sleep(0.01)

print(f"Total: {total}")

Output:

text
Summing: 100%|████████████████████| 50/50 [00:00<00:00, 87.3step/s]
Total: 1225

Manual updates with update() and set_postfix()

When the loop body controls progress (streaming downloads, chunked reads, batch training), open tqdm as a context manager with total= and call pbar.update(n) to advance by n units. set_postfix attaches live key-value metadata to the right of the bar without interrupting the display.

python
from tqdm import tqdm
import time

with tqdm(total=1000, desc="Download", unit="KB") as pbar:
    downloaded = 0
    while downloaded < 1000:
        chunk = 64
        time.sleep(0.005)
        downloaded += chunk
        pbar.update(chunk)
        pbar.set_postfix(speed="12800 KB/s", refresh=False)

Output:

text
Download: 100%|█████████| 1000/1000 [00:00<00:00, 12.8KB/s, speed=12800 KB/s]

pandas integration — progress_apply

tqdm patches pandas Series, DataFrame, and GroupBy objects with a progress_apply method. Call tqdm.pandas() once at module level to activate the patch; then replace .apply() with .progress_apply() throughout.

python
import pandas as pd
from tqdm import tqdm

tqdm.pandas(desc="Transforming")

df = pd.DataFrame({"value": range(200)})
df["doubled"] = df["value"].progress_apply(lambda x: x * 2)
print(df.tail(3))

Output:

text
Transforming: 100%|████████| 200/200 [00:00<00:00, 4123.7it/s]
   value  doubled
197   197      394
198   198      396
199   199      398

For groupby, use progress_apply on the grouped object the same way:

python
df = pd.DataFrame({"group": ["a", "b"] * 100, "val": range(200)})
result = df.groupby("group")["val"].progress_apply(sum)
print(result)

Output:

text
Transforming: 100%|████| 2/2 [00:00<00:00, 312.4it/s]
group
a    9900
b    10100
Name: val, dtype: int64

Nested bars

Use position= (0-indexed from the bottom of the terminal block) and leave=False on inner bars so they erase themselves when complete. The outermost bar uses position=0 (default) and leave=True (default).

python
from tqdm import tqdm
import time

epochs = 3
batches = 5

for epoch in tqdm(range(epochs), desc="Epoch", position=0):
    for batch in tqdm(range(batches), desc="  Batch", position=1, leave=False):
        time.sleep(0.04)

Output (mid-run):

text
Epoch:  67%|██████████████       | 2/3 [00:01<00:00,  1.39it/s]
  Batch:  60%|████████████       | 3/5 [00:00<00:00,  9.87it/s]

Async — asyncio and concurrent.futures

For asyncio, use tqdm.asyncio.tqdm which provides gather() and as_completed() drop-ins that track coroutine completion. For concurrent.futures, wrap the iterator returned by pool.map with a standard tqdm.

python
import asyncio
from tqdm.asyncio import tqdm as atqdm

async def fetch(i):
    await asyncio.sleep(0.05)
    return i * i

async def main():
    tasks = [fetch(i) for i in range(20)]
    results = await atqdm.gather(*tasks, desc="Fetching")
    print(results[:5])

asyncio.run(main())

Output:

text
Fetching: 100%|████████████████████| 20/20 [00:05<00:00,  3.94it/s]
[0, 1, 4, 9, 16]
python
from concurrent.futures import ThreadPoolExecutor
from tqdm import tqdm
import time

def work(x):
    time.sleep(0.02)
    return x ** 2

with ThreadPoolExecutor(max_workers=4) as pool:
    results = list(tqdm(pool.map(work, range(20)), total=20, desc="Threads"))
print(results[:5])

Output:

text
Threads: 100%|████████████████████| 20/20 [00:00<00:00, 52.3it/s]
[0, 1, 4, 9, 16]

Jupyter / notebook mode

In Jupyter, from tqdm.notebook import tqdm renders an HTML progress widget with colour gradients and smooth updates. The API is identical to the terminal version. from tqdm.auto import tqdm is the recommended import — it picks notebook mode automatically.

python
from tqdm.auto import tqdm   # widget in Jupyter, terminal bar elsewhere
import time

for _ in tqdm(range(50), desc="Training epoch"):
    time.sleep(0.02)

Custom format string and bar characters

bar_format= controls every token in the rendered string. ascii= replaces the default Unicode block characters with a custom set of ASCII fill characters (lowest to highest density).

python
from tqdm import tqdm
import time

fmt = "{desc}: {percentage:3.0f}%|{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}]"

for _ in tqdm(range(60), bar_format=fmt, ascii="░▒█", desc="Custom"):
    time.sleep(0.02)

Output:

text
Custom:  85%|░░░░░░░░░░░░░░░░░░░░░░░░░▒█   | 51/60 [00:01<00:00, 42.1it/s]

Available format tokens: {l_bar}, {bar}, {r_bar}, {n}, {n_fmt}, {total}, {total_fmt}, {percentage}, {elapsed}, {elapsed_s}, {remaining}, {remaining_s}, {rate}, {rate_fmt}, {rate_noinv}, {rate_noinv_fmt}, {postfix}, {desc}, {unit}.

Dynamic description and colour

python
from tqdm import tqdm
import time

stages = ["Loading", "Preprocessing", "Fitting", "Evaluating"]
with tqdm(total=len(stages), colour="cyan") as pbar:
    for stage in stages:
        pbar.set_description(stage)
        time.sleep(0.3)
        pbar.update(1)

Output:

text
Evaluating: 100%|███████████████████████| 4/4 [00:01<00:00,  3.33it/s]

CLI piping — byte-count filter

Invoked as python -m tqdm, tqdm reads stdin, forwards it to stdout, and displays throughput. It is a portable drop-in for pv on systems where pv is unavailable.

bash
cat large_file.bin | python -m tqdm --bytes > /dev/null

Output:

text
 512MB [00:04, 121MB/s]

Count lines instead of bytes:

bash
cat records.jsonl | python -m tqdm --unit=line --unit-scale > output.jsonl

Output: (none — exits 0 on success)

Disabling bars for non-interactive contexts

python
import sys
from tqdm import tqdm

verbose = True  # set via CLI arg or environment variable

# Explicit disable flag
for item in tqdm(range(100), disable=not verbose):
    pass

# Auto-detect TTY — suppresses bar in CI, cron, piped output
for item in tqdm(range(100), disable=not sys.stdout.isatty()):
    pass

Real-world recipes

A set of self-contained snippets that show how tqdm composes with the most common parallelism, networking, and ML primitives.

1. Batched API scraper with retries

python
import httpx
from tqdm import tqdm

def fetch(url: str) -> dict:
    for attempt in range(3):
        try:
            return httpx.get(url, timeout=5).json()
        except httpx.RequestError:
            if attempt == 2:
                raise

urls = [f"https://api.example.com/items/{i}" for i in range(200)]
results = {}
with tqdm(urls, desc="Scraping", unit="page", smoothing=0.3) as bar:
    for url in bar:
        bar.set_postfix(url=url.split("/")[-1], refresh=False)
        try:
            results[url] = fetch(url)
        except Exception as e:
            tqdm.write(f"FAIL {url}: {e}")
print(f"Got {len(results)} / {len(urls)} pages")

Output:

text
Scraping: 100%|██████████| 200/200 [00:14<00:00, 13.7page/s, url=199]
Got 197 / 200 pages

2. ML training loop with nested bars and live metrics

python
import random, time
from tqdm import tqdm

epochs = 4
batches = 12

with tqdm(total=epochs, desc="Epoch", position=0) as ep_bar:
    for epoch in range(epochs):
        loss, acc = 1.0, 0.0
        with tqdm(total=batches, desc="  Batch", position=1, leave=False) as bt_bar:
            for batch in range(batches):
                time.sleep(0.04)
                loss = loss * (0.95 + random.random() * 0.03)
                acc = min(1.0, acc + 0.04 + random.random() * 0.02)
                bt_bar.set_postfix(loss=f"{loss:.3f}", acc=f"{acc:.2%}", refresh=False)
                bt_bar.update(1)
        ep_bar.set_postfix(loss=f"{loss:.3f}", acc=f"{acc:.2%}")
        ep_bar.update(1)

Output (mid-run):

text
Epoch:  50%|██████      | 2/4 [00:01<00:01, loss=0.612, acc=68.40%]
  Batch:  67%|████      | 8/12 [00:00<00:00, loss=0.617, acc=64.20%]

3. Multiprocessing pool with shared progress

python
from multiprocessing import Pool
from tqdm import tqdm
import time

def work(n: int) -> int:
    time.sleep(0.05)
    return n * n

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        # imap_unordered yields as workers finish — bar advances ASAP
        results = list(tqdm(
            pool.imap_unordered(work, range(80)),
            total=80,
            desc="Pool",
            unit="task",
        ))
    print(sum(results))

Output:

text
Pool: 100%|██████████| 80/80 [00:01<00:00, 65.4task/s]
167960

4. asyncio concurrency-bounded fetch

python
import asyncio
from tqdm.asyncio import tqdm as atqdm

semaphore = asyncio.Semaphore(8)

async def fetch(i: int) -> int:
    async with semaphore:
        await asyncio.sleep(0.1)
        return i * 2

async def main():
    coros = [fetch(i) for i in range(40)]
    out = await atqdm.gather(*coros, desc="Fetching", unit="req")
    print(sum(out))

asyncio.run(main())

Output:

text
Fetching: 100%|██████████| 40/40 [00:00<00:00, 78.1req/s]
1560

5. Streaming download with byte progress

python
import httpx, pathlib
from tqdm import tqdm

url = "https://example.com/big.iso"
dst = pathlib.Path("big.iso")

with httpx.stream("GET", url) as r:
    total = int(r.headers.get("Content-Length", 0))
    with open(dst, "wb") as f, tqdm(
        total=total, unit="B", unit_scale=True, unit_divisor=1024, desc=dst.name
    ) as bar:
        for chunk in r.iter_bytes(chunk_size=1024 * 64):
            f.write(chunk)
            bar.update(len(chunk))

Output:

text
big.iso: 100%|██████████| 512M/512M [00:18<00:00, 28.4MB/s]

6. joblib + tqdm — patched parallel callback

python
from joblib import Parallel, delayed
from tqdm import tqdm
import time

def work(x):
    time.sleep(0.02)
    return x * x

# Wrap the input iterator; tqdm advances as each task is dispatched
inputs = list(range(60))
results = Parallel(n_jobs=4)(delayed(work)(i) for i in tqdm(inputs, desc="joblib"))
print(sum(results))

Output:

text
joblib: 100%|██████████| 60/60 [00:00<00:00, 1842it/s]
70210

7. logging interleave — keep the bar at the bottom

python
import logging
from tqdm import tqdm
from tqdm.contrib.logging import logging_redirect_tqdm

log = logging.getLogger("etl")
logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(message)s")

with logging_redirect_tqdm():
    for i in tqdm(range(50), desc="Loading"):
        if i % 10 == 0:
            log.info(f"checkpoint at {i}")

Output:

text
INFO | checkpoint at 0
INFO | checkpoint at 10
Loading:  40%|████      | 20/50 [00:00<00:00, 50.2it/s]

Performance tuning

tqdm adds ~60 ns per iteration in the fast path, but unnecessary refreshes can dominate a tight loop. The right knobs depend on iteration speed.

python
from tqdm import tqdm
import time

# Slow loop (>1 it/s) — defaults are fine
for _ in tqdm(range(50), desc="Default"):
    time.sleep(0.05)

# Fast loop (>10k it/s) — raise miniters to skip per-tick refreshes
for _ in tqdm(range(10_000_000), desc="Fast", miniters=10_000, mininterval=0.2):
    pass

# Streaming-ish loop — smoothing controls ETA averaging window
for _ in tqdm(range(200), desc="Variable", smoothing=0.05):
    time.sleep(0.01)

# Adaptive terminal width — set if columns change mid-run (e.g. in tmux)
for _ in tqdm(range(50), desc="Resize", dynamic_ncols=True):
    time.sleep(0.05)
ParameterDefaultPurpose
mininterval0.1Don't refresh more than once per N seconds
minitersdynamicMinimum iteration count between refreshes
maxinterval10.0Force refresh at least every N seconds (for ETA accuracy)
smoothing0.3EMA factor for rate calculation (0=instantaneous, 1=overall)
dynamic_ncolsFalseRe-query terminal width every refresh
lock_argsNoneLock acquire args — pass (False,) for non-blocking locks in threads

Rules of thumb:

  • For ML training (1–100 it/s): defaults are fine.
  • For tight CPU loops (>10k it/s): miniters=1000+, mininterval=0.5.
  • For variable-rate downloads: smoothing=0.1 (lower = more responsive ETA).
  • For nested terminal pagers (tmux, screen): dynamic_ncols=True.

Troubleshooting common errors

SymptomCauseFix
Bar shows ?it/s and no ETAtotal= not set on a generatorPass total=len(items) explicitly
Bar overwrites itself in CI logs\r carriage returns interpretedtqdm(..., disable=not sys.stdout.isatty())
Multiple bars on one line, garbledNested without position=Outer position=0, inner position=1, leave=False
Bar is duplicated each iterationUsed inside a print() loopReplace print with tqdm.write
ImportError: cannot import name 'tqdm' from 'tqdm.notebook'Importing notebook variant in plain PythonUse from tqdm.auto import tqdm instead
RuntimeError: cannot reuse already awaited coroutineawait atqdm.gather called twiceMaterialise coros into a list once; pass to gather
Bar lags behind real progressOutput buffered (e.g. python -u missing)Run with python -u or set PYTHONUNBUFFERED=1
In Jupyter, bar shows as HBox(...) textWidgets extension not enabledjupyter nbextension enable --py widgetsnbextension
BrokenPipeError writing to tqdm.writePiped to head or similarWrap loop in try/except BrokenPipeError and sys.exit(0)

Integration patterns

Beyond the recipes above, a few cross-cutting patterns repeatedly come up.

Periodic checkpoint with set_postfix:

python
from tqdm import tqdm

best = float("inf")
with tqdm(range(100), desc="Search") as bar:
    for i in bar:
        score = (i - 73) ** 2 + (i % 7)
        if score < best:
            best = score
        bar.set_postfix(best=best, refresh=(i % 10 == 0))

Output:

text
Search: 100%|██████████| 100/100 [00:00<00:00, 12.4kit/s, best=0]

dask.diagnostics progress bar interop:

python
from dask.diagnostics import ProgressBar
import dask.dataframe as dd

# Dask ships its own ProgressBar; tqdm.dask provides an adapter
from tqdm.dask import TqdmCallback

with TqdmCallback(desc="Dask compute"):
    ddf = dd.read_parquet("data/*.parquet")
    result = ddf.groupby("k")["v"].mean().compute()

Output:

text
Dask compute: 100%|██████████| 12/12 [00:03<00:00,  3.42partition/s]

tqdm.contrib.concurrent — one-liner thread/process map:

python
from tqdm.contrib.concurrent import thread_map, process_map
import time

def slow(x):
    time.sleep(0.05)
    return x * 2

print(thread_map(slow, range(40), max_workers=8, desc="Threads")[:5])
print(process_map(slow, range(40), max_workers=4, desc="Procs")[:5])

Output:

text
Threads: 100%|██████████| 40/40 [00:00<00:00, 158.3it/s]
[0, 2, 4, 6, 8]
Procs:   100%|██████████| 40/40 [00:00<00:00,  75.1it/s]
[0, 2, 4, 6, 8]

Testing patterns

In tests, you usually want bars suppressed — they pollute pytest output and interfere with CI log parsers. Disable globally via env var or a fixture.

python
# conftest.py
import os
import pytest
from tqdm import tqdm

@pytest.fixture(autouse=True)
def silence_tqdm(monkeypatch):
    monkeypatch.setenv("TQDM_DISABLE", "1")

# Or explicitly via the API
@pytest.fixture
def quiet_tqdm(monkeypatch):
    monkeypatch.setattr("tqdm.tqdm.__init__",
                        lambda self, *a, **kw: super(tqdm, self).__init__(*a, **{**kw, "disable": True}))
python
# Assert bar emits expected postfix values
def test_postfix_updates(capsys):
    from tqdm import tqdm
    with tqdm(range(3), file=sys.stdout) as bar:
        for i in bar:
            bar.set_postfix(step=i)
    out = capsys.readouterr().err  # tqdm writes to stderr by default
    assert "step=2" in out

Output: (none — exits 0 on success)

TQDM_DISABLE=1 in CI environment variables is the cleanest way to silence every bar without touching application code.

When NOT to use this

tqdm is almost-always the right choice for interactive loops, but a handful of contexts are better served by alternatives.

  • Long-running batch jobs in CI — bars produce ANSI control sequences and \r carriage returns that flood the log. Use a periodic log.info(f"{i}/{total}") every N seconds instead.
  • Web servers / request handlers — never inside a request handler; the user can't see stderr. Emit progress via WebSocket or SSE.
  • Structured-logging environments (Datadog, Splunk) — the carriage-return output line breaks log ingestion. Disable with TQDM_DISABLE=1.
  • Loops with side effects faster than display refresh — for >100k it/s loops, the bar update overhead can be visible; tune miniters or skip the bar entirely.
  • When you want rich tables, panels, or live metrics, prefer rich.progress — it supports multi-column live displays.

Quick reference

FeatureCode
Basic wraptqdm(iterable)
Counter looptrange(n)
With labeltqdm(it, desc="Step")
Manual progresstqdm(total=n) then pbar.update(k)
Postfix metadatapbar.set_postfix(loss=0.42)
Print above bartqdm.write("message")
pandas applytqdm.pandas() then .progress_apply(fn)
Nested barsouter position=0, inner position=1, leave=False
Asynciofrom tqdm.asyncio import tqdm
Thread pooltqdm(pool.map(fn, items), total=n)
Notebook autofrom tqdm.auto import tqdm
Disabledisable=not verbose
Custom charsascii="░▒█"
CLI pipepython -m tqdm --bytes