官方方法:
- from w3lib.http import basic_auth_header
- class CustomProxyMiddleware(object):
- def process_request(self, request, spider):
- request.meta['proxy'] = "https://<PROXY_IP_OR_URL>:<PROXY_PORT>"
- request.headers['Proxy-Authorization'] = basic_auth_header(
- '<PROXY_USERNAME>', '<PROXY_PASSWORD>')
- DOWNLOADER_MIDDLEWARES = {
- '<PROJECT_NAME>.middlewares.CustomProxyMiddleware': 350,
- 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
- }
来源: http://www.bubuko.com/infodetail-3385200.html