The built in middleware is already on
HttpProxyMiddleware is part of the default downloader middleware set. You do not need to enable it and you do not need to write one. It reads the proxy key from each request meta and applies it, including any credentials in the URL.
import scrapy
PROXY = "http://USERNAME:[email protected]:80"
class PricesSpider(scrapy.Spider):
name = "prices"
def start_requests(self):
for url in ["https://example.com/a", "https://example.com/b"]:
yield scrapy.Request(url, meta={"proxy": PROXY})
def parse(self, response):
yield {"url": response.url, "status": response.status}Applying it to every request without repeating yourself
A small middleware sets the proxy once for the whole spider, which is tidier than putting meta on every yield. This is the one case where writing your own is worth it, and it is six lines rather than the page of code usually suggested.
# middlewares.py
class ProxyMiddleware:
PROXY = "http://USERNAME:[email protected]:80"
def process_request(self, request, spider):
request.meta["proxy"] = self.PROXY
# settings.py
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.ProxyMiddleware": 350,
}Rotation is the gateway, not your code
A rotating pool gives you a different exit address per request from a single endpoint, so you do not need a proxy list and you do not need to cycle through one. If you have written code that picks a random proxy from an array, a rotating gateway replaces all of it.
# No list, no random.choice, no rotation logic.
# One endpoint, a new address per request.
PROXY = "http://USERNAME:[email protected]:80"Concurrency is where the bill comes from
Scrapy defaults to sixteen concurrent requests and will happily saturate a metered plan faster than you expect. Set a per domain limit and a download delay that match what you are willing to spend, and remember that failed and retried requests consume traffic too.
# settings.py
CONCURRENT_REQUESTS = 8
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0.5
RETRY_TIMES = 2What to buy
Datacenter per GB for most crawling, because the majority of pages you will collect do not care where the request came from. Move the specific domains that block you onto residential rather than moving the whole crawl.
Common questions
Do I need a proxy list for rotation?
No, and maintaining one is usually wasted effort. A rotating gateway hands out a different address per request from a single endpoint, so the rotation happens on the network side. Lists are for static addresses, which is a different product for a different job.
Why do my retries cost so much traffic?
Because a retried request is a new request and is billed like one. A crawl hitting a lot of failures can spend a surprising share of its budget on attempts that returned nothing. Lower RETRY_TIMES while you are testing, and look at why the failures are happening before raising it again.
Can I use SOCKS5 with Scrapy?
Not with the built in middleware, which speaks HTTP proxying only. There are third party downloader handlers that add it, but for ordinary crawling there is no advantage, so the simplest answer is to use the HTTP port.