Scrapy-Playwright: 使用Playwright 增强 Scrapy 爬虫

Scrapy-Playwright \u662f\u4e00\u4e2a\u57fa\u4e8e Scrapy \u6846\u67b6\u7684\u63d2\u4ef6\uff0c\u7528\u4e8e\u5728\u722c\u866b\u4e2d\u4f7f\u7528 Playwright \u8fdb\u884c\u9875\u9762\u7684\u81ea\u52a8\u5316\u64cd\u4f5c\u548c\u6570\u636e\u62d6\u5f15\u3002Playwright \u662f\u4e00\u4e2a\u8d8a\u80fd\u6d4f\u89c8\u5668\u81ea\u52a8\u5316\u5de5\u5177\uff0c\u652f\u6301\u591a\u79cd\u6d4f\u89c8\u5668\uff0c\u5e76\u4e14\u63d0\u4f9b\u4e86\u5f3a\u5927\u7684 API \u6765\u6a21\u62df\u7528\u6237\u5728\u6d4f\u89c8\u5668\u4e2d\u7684\u64cd\u4f5c\u3002\n\n\u4f7f\u7528 Scrapy-Playwright \u5728\u722c\u866b\u4e2d\u53ef\u4ee5\u6a21\u62df\u7528\u6237\u7684\u64cd\u4f5c\uff0c\u4f8b\u5982\u70b9\u51fb\u6309\u94ae\u3001\u586b\u5199\u8868\u5355\u3001\u6eda\u8f6e\u9875\u9762\u7b49\uff0c\u5e76\u4e14\u53ef\u4ee5\u62d6\u5f15\u9875\u9762\u4e2d\u7684\u6570\u636e\u3002\u76f8\u6bd4\u4e8e\u4f20\u7edf\u7684\u57fa\u4e8e HTTP \u8bf7\u6c42\u7684\u722c\u866b\uff0c\u4f7f\u7528 Playwright \u53ef\u4ee5\u89e3\u51b3\u4e00\u4e9b\u52a8\u6001\u9875\u9762\u6f14\u793a\u548c JavaScript \u4ea4\u4e92\u7684\u95ee\u9898\uff0c\u66f4\u52a0\u7075\u6d3b\u548c\u5f3a\u5927\u3002\n\n\u4f7f\u7528 Scrapy-Playwright \u7684\u6b65\u9aa4\u5982\u4e0b\uff1a\n\n1. \u5b89\u88c5 Scrapy-Playwright \u63d2\u4ef6\uff1a\u53ef\u4ee5\u901a\u8fc7 pip \u5b89\u88c5\uff0c\u8fd0\u884c\u547d\u4ee4 pip install scrapy-playwright\u3002\n\n2. \u5728 Scrapy\u7684\u722c\u866b\u4ee3\u7801\u4e2d\u5f15\u5165 Scrapy-Playwright \u76f8\u5173\u7684\u7c7b\u548c\u65b9\u6cd5\uff1a\u53ef\u4ee5\u5728\u722c\u866b\u7684 settings.py \u6587\u4ef6\u4e2d\u6dfb\u52a0 PLAYWRIGHT_BROWSER_TYPE \u548c PLAYWRIGHT_LAUNCH_OPTIONS \u914d\u7f6e\uff0c\u4ee5\u53ca\u5728\u722c\u866b\u4ee3\u7801\u4e2d\u5f15\u5165 from scrapy_playwright.page import PageCoroutine\u3002\n\n3. \u5728\u722c\u866b\u7684 start_requests \u65b9\u6cd5\u4e2d\u4f7f\u7528 PageCoroutine \u6765\u521b\u5efa\u4e00\u4e2a Playwright\u7684 Page\u5bf9\u8c61\uff0c\u5e76\u4e14\u5728\u5176\u4e2d\u6267\u884c\u81ea\u52a8\u5316\u64cd\u4f5c\u548c\u6570\u636e\u62d6\u5f15\u7684\u903c\u8f91\u3002\u4f8b\u5982\u53ef\u4ee5\u4f7f\u7528 await page.goto(url) \u65b9\u6cd5\u6765\u52a0\u8f7d\u9875\u9762\uff0c\u7136\u540e\u4f7f\u7528 await page.click(selector) \u65b9\u6cd5\u6765\u70b9\u51fb\u9875\u9762\u4e0a\u7684\u67d0\u4e2a\u5143\u7d20\uff0c\u4f7f\u7528 await page.type(selector, text) \u65b9\u6cd5\u6765\u586b\u5199\u8868\u5355\u7b49\u3002\n\n4. \u5728\u81ea\u52a8\u5316\u64cd\u4f5c\u548c\u6570\u636e\u62d6\u5f15\u7684\u903c\u8f91\u4e2d\uff0c\u53ef\u4ee5\u901a\u8fc7 Playwright\u7684 API \u6765\u83b7\u53d6\u9875\u9762\u7684 HTML \u5185\u5bb9\u3001\u6295\u5f71\u3001\u54cd\u5e94\u7b49\uff0c\u5e76\u4e14\u53ef\u4ee5\u4f7f\u7528 Scrapy\u7684 Item \u6765\u5b58\u50a8\u62d6\u5f15\u5230\u7684\u6570\u636e\u3002\n\n5. \u8fd0\u884c Scrapy \u722c\u866b\uff1a\u53ef\u4ee5\u4f7f\u7528\u547d\u4ee4 scrapy crawl spider_name \u6765\u8fd0\u884c\u722c\u866b\uff0c\u5176\u4e2d spider_name \u662f\u722c\u866b\u7684\u540d\u79f0\u3002\n\n\u603b\u7ed3\u8bf4\u6765\uff0cScrapy-Playwright \u662f\u4e00\u4e2a\u65b9\u4fbf\u4f7f\u7528 Playwright \u8fdb\u884c\u9875\u9762\u81ea\u52a8\u5316\u64cd\u4f5c\u548c\u6570\u636e\u62d6\u5f15\u7684\u63d2\u4ef6\uff0c\u53ef\u4ee5\u5728 Scrapy \u722c\u866b\u4e2d\u66f4\u52a0\u7075\u6d3b\u548c\u5f3a\u5927\u5730\u5904\u7406\u52a8\u6001\u9875\u9762\u548c JavaScript \u4ea4\u4e92\u7684\u60c5\u51b5\u3002