How do I integrate a proxy into a Python scraper?

Take the provider’s endpoint, your credentials, the supported protocol and any limits. Keep the credentials in environment variables rather than in the script, then pass the proxy through a dictionary, set a timeout, and decide up front which status codes you will accept.

Then the parts that decide whether it survives contact with a real site.

Keep concurrency modest and retry with backoff rather than immediately trying again.
Log the status, the latency and which session or location you used. Never log the credentials.
Validate what comes back before storing it, because a proxy returning a block page still returns a 200 sometimes.

python

import os
import requests

# From the environment, never hardcoded. A proxy URL contains a password,
# and a hardcoded one ends up in version control within the week.
proxy_url = os.environ["PROXY_URL"]  # http://user:pass@gateway:port

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

# A timeout is not optional. Without one a hung connection blocks the worker
# for as long as the operating system allows, which can be minutes.
response = requests.get(
    "https://example.com",
    proxies=proxies,
    timeout=15,
)

The https key selects the proxy used for https destinations. It does not mean the hop to the proxy is encrypted, which is the single most common misreading of this dictionary.

Worth knowing

If the site offers an official API, use it instead. Every scraper is a maintenance commitment that the site can break without warning.

How do I integrate a proxy into a Python scraper?

Other questions