内容字号:默认大号超大号

段落设置:段首缩进取消段首缩进

字体设置:切换到微软雅黑切换到宋体

Scrapy Mac Selenium chrome headless

2018-04-16 16:39 出处:清屏网 人气: 评论(0

1. 安装chrome,chromedriver

chrome浏览器要升级到最新版本。

用brew安装chromedriver

用pip安装 Selenium

2. 写middleware

# -*- coding: utf-8 -*-

# Define here the models for your spider middleware
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/spider-middleware.html

from scrapy import signals
from .useragent import agents
import random
import base64
from tutorial import settings

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
# from selenium.webdriver.common.by import By
# from selenium.webdriver.support.ui import WebDriverWait
# from selenium.webdriver.support import expected_conditions as EC
from scrapy.http import HtmlResponse
from logging import getLogger
from selenium.webdriver.chrome.options import Options



class SeleniumMiddleware():
    def __init__(self, timeout=None, service_args=[]):
        self.logger = getLogger(__name__)
        self.timeout = timeout

        chrome_options = Options()
        chrome_options.add_argument('--headless')
        #chrome_options.add_argument('--disable-gpu')
        self.driver = webdriver.Chrome(chrome_options=chrome_options)

    def __del__(self):
        self.driver.close()
    def process_request(self, request, spider):
        if True:
            #request.headers['Referer'] = 'http://jandan.net/ooxx';
            try:
                self.driver.get(request.url)
                return HtmlResponse(url=request.url, body=self.driver.page_source, request=request, encoding='utf-8',
                                    status=200)
            except TimeoutException:
                return HtmlResponse(url=request.url, status=500, request=request)

3. settings

DOWNLOADER_MIDDLEWARES = {
   'tutorial.middlewares.SeleniumMiddleware': 543,
}

4.其他说明

爬虫里的写法不变,这边主要是写了一个downloader middleware,返回一个用我们chrome driver请求后的page_source,这样parse里就是返回后的内容了。这样就可以爬javascript 渲染后的页面,启动后,感觉性能要比那个splash要差。。。

另外本来是用PhantomJs,但是运行后,提示这个不推荐使用了。。。推荐使用firefox和chrome的headless版本。。

分享给小伙伴们:
本文标签: Scrapy

相关文章

发表评论愿您的每句评论,都能给大家的生活添色彩,带来共鸣,带来思索,带来快乐。

CopyRight © 2015-2016 QingPingShan.com , All Rights Reserved.

清屏网 版权所有 豫ICP备15026204号