浏览器里面能输入的最大 url 是有限制的
Safari 最多 一万多
IE 最少 2083
urllength 中间件源码
谷歌和火狐正常 八千多
- """
- Url Length Spider Middleware
- See documentation in docs/topics/spider-middleware.rst
- """
- import logging
- from scrapy.http import Request
- from scrapy.exceptions import NotConfigured
- logger = logging.getLogger(__name__)
- class UrlLengthMiddleware(object):
- def __init__(self, maxlength):
- self.maxlength = maxlength
- @classmethod
- def from_settings(cls, settings):
- maxlength = settings.getint('URLLENGTH_LIMIT')
- if not maxlength:
- raise NotConfigured
- return cls(maxlength)
- def process_spider_output(self, response, result, spider):
- def _filter(request):
- if isinstance(request, Request) and len(request.url)> self.maxlength:
- logger.debug("Ignoring link (url length> %(maxlength)d): %(url)s",
- {'maxlength': self.maxlength, 'url': request.url},
- extra={'spider': spider})
- return False
- else:
- return True
- return (r for r in result or () if _filter(r))
scrapy 设置了默认的长度
如果要自己设置可以在 setting 里面增加配置
URLLENGTH_LIMIT = 60
如果 url 的长度超过了这个设置
会在运行打印日志 忽略这个 url 请求
- logger.debug("Ignoring link (url length> %(maxlength)d): %(url)s",
- {'maxlength': self.maxlength, 'url': request.url},
- extra={'spider': spider})
来源: http://www.bubuko.com/infodetail-3280854.html