scrapy splash爬取动态网页

Scrapy Splash是一个Scrapy插件，用于处理动态网页。它使用了Splash服务来渲染JavaScript并提供网页的HTML源代码。下面是一个使用Scrapy Splash爬取动态网页的示例：

安装Scrapy和Scrapy Splash：
```
pip install scrapy scrapy-splash
```

配置Scrapy Splash：在Scrapy的设置文件中增加以下配置：

SPLASH_URL = 'http://localhost:8050'
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPLASH_COOKIES_DEBUG = False
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

创建Spider：

import scrapy
from scrapy_splash import SplashRequest

class MySpider(scrapy.Spider):
    name = 'example'
    start_urls = ['http://example.com']

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, args={'wait': 0.5})

    def parse(self, response):
        # 处理响应
        pass

运行爬虫：
```
scrapy crawl example
```

在上述示例中，Scrapy Splash使用了Splash服务来渲染JavaScript并等待0.5秒，然后将渲染后的网页的HTML源代码传递给parse方法进行处理。您可以根据需要自定义Splash请求的参数和处理响应的逻辑