Scrapy第十一(②)篇：selenium4模拟器中间件-多线程并发

1.安装依赖

pip install scrapy-ajax-utils -i https://pypi.tuna.tsinghua.edu.cn/simple

源码地址：GitHub - kingronjan/scrapy_ajax_utils: utils for crawl ajax page in scrapy project.

2.修改spider文件

①为spider加上装饰器@selenium_support

②将spider.Request()方法换成SeleniumRequest()【重写start_requests方法】

import scrapy
from scrapy.http import HtmlResponse
from scrapy_ajax_utils import selenium_support, SeleniumRequest


@selenium_support
class TestspiderSpider(scrapy.Spider):
    name = 'testspider'
    allowed_domains = ['www.baidu.com']
    start_urls = ['https://www.baidu.com' for i in range(20)]

    def start_requests(self):
        for url in self.start_urls:
            yield SeleniumRequest(url, callback=self.parse, dont_filter=True)

    def parse(self, response: HtmlResponse):
        pass

 3.配置：settings.py（如果使用的是webdriver-manager，如何查看驱动地址：看下面第4点）

# 只支持两种浏览器驱动 chrome（默认） & firefox.
SELENIUM_DRIVER_NAME = 'chrome'
# 浏览器驱动地址（必填）
SELENIUM_DRIVER_PATH = r'C:\Users\sunziwen\.wdm\drivers\chromedriver\win32\98.0.4758\chromedriver.exe'
# 是否为无头浏览器
SELENIUM_HEADLESS = False
# 页面加载超时时间
SELENIUM_DRIVER_PAGE_LOAD_TIMEOUT = 30
# selenium的最小并发数
SELENIUM_MIN_DRIVERS = 5
# selenium的最大并发数
SELENIUM_MAX_DRIVERS = 10

或者也可以直接赋值：

from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager

# 只支持两种浏览器驱动 chrome（默认） & firefox.
SELENIUM_DRIVER_NAME = 'chrome'
# 浏览器驱动地址（必填）
# SELENIUM_DRIVER_PATH = r'C:\Users\sunziwen\.wdm\drivers\chromedriver\win32\98.0.4758\chromedriver.exe'
SELENIUM_DRIVER_PATH = ChromeService(executable_path=ChromeDriverManager().install()).path
# 是否为无头浏览器
SELENIUM_HEADLESS = False
# 页面加载超时时间
SELENIUM_DRIVER_PAGE_LOAD_TIMEOUT = 30
# selenium的最小并发数
SELENIUM_MIN_DRIVERS = 4
# selenium的最大并发数
SELENIUM_MAX_DRIVERS = 5

4.查看驱动地址：

from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager


if __name__ == '__main__':
    # 初始化驱动
    service = ChromeService(executable_path=ChromeDriverManager().install())
    print("驱动地址：",service.path)

5.运行测试：

6.操作浏览器驱动 

①编写操作中间件：middlewares.py

from selenium.webdriver.chrome.webdriver import WebDriver
from scrapy_ajax_utils import SeleniumRequest

class SeleniumMiddleware(object):
    def process_request(self: WebDriver, request: SeleniumRequest, spider):
        # self就是驱动实例
        # 访问网页
        self.get(request.url)
        # 全屏
        self.maximize_window()
        # 阻塞3秒
        time.sleep(3)
        # network记录，这里可以获得所有ajax请求的结果
        # logs = get_xhr_logs(chrome)
        # 动态加载后的网页
        html = self.page_source
        # 退出浏览器
        self.quit()
        return scrapy.http.HtmlResponse(url=request.url, body=html.encode('utf-8'), encoding='utf-8', request=request)

②启动中间件：settings.py

DOWNLOADER_MIDDLEWARES = {
    'testscrapy.middlewares.SeleniumMiddleware': 543,
}  # 下载中间件

 ③修改spider文件，将SeleniumRequest()的handle参数指定为操作中间件。

7.断点测试