boto3

What it is

boto3 is the official AWS SDK for Python, maintained by Amazon Web Services. It wraps the botocore low-level library with a higher-level resource interface and a session abstraction over credentials, regions, and endpoints. It is the canonical way to drive any AWS service from Python — S3, DynamoDB, SQS, Lambda, EC2, IAM, STS, and several hundred others — and is generated from the same Smithy service models as the AWS CLI and other AWS SDKs.

Reach for boto3 whenever you need to call AWS APIs from a Python application, script, or Lambda function. For one-off shell operations, the aws CLI (which uses botocore under the hood) is usually quicker; for application code, boto3 is the right layer.

Install

pip install boto3

Output: (none — exits 0 on success; pulls in botocore, s3transfer, jmespath)

uv add boto3

Output: dependency resolved + added to pyproject.toml

poetry add boto3

Output: updated lockfile + virtualenv install

pip install "boto3[crt]"

Output: installs the optional awscrt extra — enables the AWS Common Runtime for faster S3 transfers (multi-part parallelization in native code).

Versioning & Python support

• Current line is the 1.x series; minor releases (1.34, 1.35, …) ship roughly weekly with service-model updates and remain backwards-compatible within the line.
• boto3 versions are tightly coupled to botocore versions — boto3 1.34.x requires botocore 1.34.x. Don't override botocore independently.
• Supports Python 3.8+ on recent releases; new floor moves up roughly once a year following AWS support policy.
• A 1.x → 2.0 migration is occasionally floated but has not been scheduled — pin to a minor range in production and treat AWS's announcement page as the canonical source.

Package metadata

• Maintainer: Amazon Web Services
• Project home: github.com/boto/boto3
• Docs: boto3.amazonaws.com
• PyPI: pypi.org/project/boto3
• License: Apache-2.0
• Governance: Amazon Web Services Inc.
• First released: 2014 (the modern v1 line; predecessor boto dates to 2006)
• Downloads: hundreds of millions per month; consistently in the PyPI top 10

Optional dependencies & extras

• boto3[crt] — installs awscrt, the AWS Common Runtime. Speeds up large S3 transfers via native multipart parallelism and provides additional checksum algorithms.
• No other extras are shipped today; service-specific helpers are imported automatically when you create a client or resource for that service.

Core dependencies pulled in automatically:

• botocore — low-level request/response, signing, retries, service models
• s3transfer — high-level S3 transfer manager (parallel uploads/downloads)
• jmespath — JMESPath expression engine used by Paginator.search() and client.get_paginator(...).search()
• python-dateutil — date parsing for response timestamps (transitive)
• urllib3 — HTTP transport (transitive via botocore)

Alternatives

Common gotchas

1. botocore version lock. Never pip install -U botocore separately — let boto3 pull the matched version. Mismatched versions surface as AttributeError on service operations.
2. Default region resolution. boto3 looks at AWS_REGION, then AWS_DEFAULT_REGION, then the active profile's region setting. If none is set, calls fail with NoRegionError.
3. Credentials precedence. Env vars → shared credentials file → IAM role (EC2 / ECS / Lambda) → SSO cache. Surprising failures usually trace to an unexpected env var winning.
4. Resource vs client. boto3.resource("s3") is a higher-level OO wrapper; boto3.client("s3") is the raw API. Resources don't cover every service — fall back to client when methods are missing.
5. Session is the right boundary for credentials, region, and endpoint overrides. Don't reach for module-level boto3.client(...) in libraries — accept a Session argument.
6. Pagination is opt-in. client.list_objects_v2() returns one page (max 1000). Use client.get_paginator("list_objects_v2").paginate(...) for full traversal.
7. exceptions are dynamic. client.exceptions.NoSuchKey exists, but botocore.exceptions.ClientError is the only thing you can catch statically. Inspect e.response["Error"]["Code"] to branch.

Real-world recipes

The S3 + IAM + STS recipes below are the everyday building blocks; the SQS / DynamoDB / Lambda recipes round out the most-used service surfaces. Each is intentionally minimal so you can drop it into a script and tune.

Recipe 1 — S3 upload and download with a shared session.

import boto3

session = boto3.Session(profile_name="prod", region_name="us-east-1")
s3 = session.client("s3")

s3.upload_file("./report.csv", "my-bucket", "reports/2026/q2.csv",
               ExtraArgs={"ContentType": "text/csv", "ServerSideEncryption": "AES256"})
s3.download_file("my-bucket", "reports/2026/q2.csv", "/tmp/q2.csv")

Output: uploads with server-side encryption headers; downloads to local path. Multipart parallelism kicks in automatically over 8 MB.

Recipe 2 — SQS publish and consume.

import boto3, json

sqs = boto3.client("sqs")
url = sqs.get_queue_url(QueueName="my-queue")["QueueUrl"]

sqs.send_message(QueueUrl=url, MessageBody=json.dumps({"event": "ping"}))

msgs = sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=10, WaitTimeSeconds=20).get("Messages", [])
for m in msgs:
    print(m["Body"])
    sqs.delete_message(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"])

Output: prints each message body; deletes after processing. WaitTimeSeconds=20 is long-polling — preferred over short-polling.

Recipe 3 — DynamoDB CRUD with the resource interface.

import boto3
from boto3.dynamodb.conditions import Key

table = boto3.resource("dynamodb").Table("users")

table.put_item(Item={"user_id": "u-42", "name": "Alice Dev", "tier": "pro"})
got = table.get_item(Key={"user_id": "u-42"})["Item"]
table.update_item(Key={"user_id": "u-42"}, UpdateExpression="SET tier = :t",
                  ExpressionAttributeValues={":t": "enterprise"})
res = table.query(KeyConditionExpression=Key("user_id").eq("u-42"))

Output: item written, fetched, updated, then queried back. Resource layer handles the verbose AttributeValue wire format for you.

Recipe 4 — Synchronous Lambda invoke.

import boto3, json

lam = boto3.client("lambda")
r = lam.invoke(FunctionName="my-fn",
               InvocationType="RequestResponse",
               Payload=json.dumps({"name": "Alice Dev"}))
print(r["StatusCode"], json.loads(r["Payload"].read()))

Output: 200 {...} — status + decoded JSON response. Use InvocationType="Event" for fire-and-forget.

Recipe 5 — STS assume-role + downstream client.

import boto3

sts = boto3.client("sts")
creds = sts.assume_role(RoleArn="arn:aws:iam::123456789012:role/CrossAccount",
                        RoleSessionName="myhost-2026-05")["Credentials"]

s3 = boto3.client("s3",
                  aws_access_key_id=creds["AccessKeyId"],
                  aws_secret_access_key=creds["SecretAccessKey"],
                  aws_session_token=creds["SessionToken"])
print([b["Name"] for b in s3.list_buckets()["Buckets"]][:5])

Output: bucket names visible to the assumed role. Token expires per DurationSeconds (default 1 hour).

Recipe 6 — Pagination pattern for any list-style API.

import boto3

s3 = boto3.client("s3")
pages = s3.get_paginator("list_objects_v2").paginate(Bucket="my-bucket", Prefix="reports/")
for page in pages:
    for obj in page.get("Contents", []):
        print(obj["Key"], obj["Size"])

Output: every key under reports/ regardless of count. Use .search("Contents[?Size > 1000000].Key") for JMESPath filtering inline.

Production deployment

boto3 is a library, not a service, but its production posture matters because it drives every AWS call in the application.

• Use IAM roles, never long-lived access keys, in Lambda / ECS / EC2 / EKS / App Runner workloads. The SDK picks them up automatically.
• Reuse a single Session per process. Sessions cache credentials and signers; per-call sessions cost extra IMDS calls and STS round-trips.
• Pin to a minor range (boto3>=1.34,<1.36) and bump deliberately — weekly releases mean a year-old pin can lag dozens of service model updates.
• Set explicit retry mode to "adaptive" for high-throughput services: Config(retries={"mode": "adaptive", "max_attempts": 10}).
• Set explicit connect_timeout and read_timeout via botocore.config.Config(connect_timeout=5, read_timeout=30) — defaults can be too generous for tight SLAs.
• Enable the CRT (pip install boto3[crt]) for S3-heavy workloads — 2-3× transfer speed on large objects.
• Log API calls in development via boto3.set_stream_logger("botocore") (verbose; never enable in production).

Performance tuning

• CRT (Common Runtime). pip install boto3[crt] and awscrt-backed s3transfer significantly speeds up multipart S3 transfers. Best for large objects (>100 MB).
• Connection pool size. Config(max_pool_connections=N) — default 10 is too low for fan-out workers; raise to match your concurrency.
• Reuse clients. boto3.client(...) is moderately expensive — it parses the service model. Create once per process, share across threads (clients are thread-safe; resources are not).
• Concurrency via threads, not asyncio. boto3 is blocking; for async use aioboto3 or aiobotocore. concurrent.futures.ThreadPoolExecutor is the sweet spot for IO-bound fan-out.
• Avoid the resource layer in hot paths. boto3.client("dynamodb") is leaner than boto3.resource("dynamodb") per call; the resource layer's convenience costs a small per-call overhead.
• botocore retry mode adaptive beats standard under throttling — backs off based on observed quota errors instead of fixed timing.

Version migration guide

• 1.20 → 1.21 — minimum Python is 3.6.
• 1.24 → 1.25 — minimum Python is 3.7. boto3.session.Session.get_available_partitions() added.
• 1.27 → 1.28 — minimum Python is 3.7.10; cross-region presigned URLs default to SigV4.
• 1.34 → 1.35 — minimum Python is 3.8; new retry mode adaptive becomes recommended.
• Service-specific deprecations (e.g. S3 list_objects superseded by list_objects_v2) often span years; AWS announces in release notes well before removal.

# Pre-1.28 default for some clients
s3 = boto3.client("s3", config=Config(signature_version="s3v4"))

# 1.28+: SigV4 is the default; explicit override only needed for legacy buckets
s3 = boto3.client("s3")

Output: same behavior on modern buckets; no need to specify SigV4 in new code.

Security considerations

• Never hardcode aws_access_key_id / aws_secret_access_key. Use IAM roles, profiles, or short-lived STS credentials. Even in test code — accidental commits get scraped.
• Disable shared-config writes from untrusted code. boto3.Session(profile_name=...) reads ~/.aws/config; writing isn't enabled in boto3 directly, but custom auto-config code is a risk surface.
• Scope IAM policies tightly. Use aws:ResourceTag and aws:SourceArn conditions, not wildcards.
• Use VPC endpoints (endpoint_url="https://bucket.vpce-...amazonaws.com") for S3 and DynamoDB to keep traffic off the public internet.
• Validate S3 ACLs and bucket policies separately from your code — least-privilege IAM does not guarantee bucket-level safety.
• Sign with SigV4 always. SigV2 is deprecated; ensure no client overrides this.
• Watch for credential leaks in logs. boto3.set_stream_logger prints request headers, including the Authorization line. Strip before sharing logs.

Testing & CI

moto is the canonical AWS mocking library — it stands up an in-process mock of dozens of AWS services. Pair with pytest fixtures.

# pip install moto[s3] pytest
import boto3, pytest
from moto import mock_aws

@mock_aws
def test_upload_roundtrip(tmp_path):
    s3 = boto3.client("s3", region_name="us-east-1")
    s3.create_bucket(Bucket="test-bucket")
    (tmp_path / "x.txt").write_text("hi")
    s3.upload_file(str(tmp_path / "x.txt"), "test-bucket", "x.txt")
    body = s3.get_object(Bucket="test-bucket", Key="x.txt")["Body"].read()
    assert body == b"hi"

Output: test passes; no real AWS API touched.

# stub-only tests when moto doesn't cover the service
from botocore.stub import Stubber

client = boto3.client("sts")
with Stubber(client) as st:
    st.add_response("get_caller_identity", {"UserId": "u", "Account": "123", "Arn": "arn:..."})
    assert client.get_caller_identity()["Account"] == "123"

Output: assertion holds; stubber raises on unexpected calls.

Ecosystem integrations

• aws-sdk-pandas — Pandas DataFrame I/O against S3, Athena, Glue, Redshift.
• aioboto3 — async wrapper for asyncio applications.
• moto — in-process AWS service mocks for tests.
• localstack — full local AWS emulator (separate process); boto3 clients can target it via endpoint_url.
• smart-open — transparent S3-as-file streaming.
• s3fs — fsspec-compatible filesystem over S3.
• aws-lambda-powertools — structured logging, metrics, tracing for Lambda functions.
• opentelemetry-instrumentation-botocore — distributed tracing.
• aws-cdk-lib / aws-sam-cli — IaC tools that pair naturally with boto3 application code.

Compatibility matrix

When NOT to use this

• One-off shell tasks — aws CLI v2 is faster to write and equally capable. Use boto3 only when you need application-level logic around the call.
• Async services — boto3 is blocking; in asyncio code use aioboto3 or aiobotocore to avoid pinning the event loop.
• Heavy data-frame workloads against S3/Athena — aws-sdk-pandas is a much shorter path than hand-rolling pagination + JSON parsing.
• Multi-cloud abstraction — boto3 is AWS-only. If you need to write provider-agnostic code, look at apache-libcloud or roll your own thin adapter.
• Lambda cold starts that count microseconds — boto3 import time is ~150 ms; for ultra-low-latency Lambdas consider botocore directly or the AWS Common Runtime SDK.

Troubleshooting common errors

See also

• Python: awscli — companion CLI tool
• Concept: API — REST + signing fundamentals
• Concept: HTTP — transport underpinnings
• Official boto3 docs