Selenium 和 lxml 库：爬取小说网站章节内容

Selenium 和 lxml 库：爬取小说网站章节内容\n\n本教程演示如何使用 Selenium 和 lxml 库爬取小说网站的章节内容。\n\n步骤：\n\n1. 导入库\npython\nfrom selenium import webdriver\nfrom selenium.webdriver.common.keys import Keys\nimport time\nimport random\nfrom lxml import etree\n\n\n2. 设置搜索关键词\npython\nkeyword = "开挂闯异界"\n\n\n3. 创建浏览器对象\npython\ndriver = webdriver.Chrome()\n\n\n4. 打开小说网站首页\npython\ndriver.get("https://www.bige3.cc/")\n\n\n5. 输入搜索关键词并搜索\npython\nsearch_input = driver.find_element_by_xpath("/html/body/div[4]/div[1]/div[2]/form/input[1]")\nsearch_input.send_keys(keyword)\nsearch_input.send_keys(Keys.ENTER)\ntime.sleep(3)\n\n\n6. 点击小说标题打开小说\npython\nelements = driver.find_elements_by_xpath("/html/body/div[5]/div/div/div/div/div[2]/h4/a")\nfor i in range(len(elements)):\n elements = driver.find_elements_by_xpath("/html/body/div[5]/div/div/div/div/div[2]/h4/a")\n element = elements[i]\n print(element.text) # 打印元素文本\n delay = random.uniform(3, 5)\n time.sleep(delay)\n element.click() # 点击元素\n\n driver.switch_to.window(driver.window_handles[-1])\n time.sleep(3)\n\n # 获取每章的标题及其内容\n chapters = driver.find_elements_by_xpath("//*[@class='listmain']/dl/dd/a")\n for chapter in chapters:\n # 重新获取章节元素，避免 StaleElementReferenceException 错误\n chapters = driver.find_elements_by_xpath("//*[@class='listmain']/dl/dd/a")\n chapter = chapters[i]\n chapter_title = chapter.text.replace('、', '') # 获取章节标题\n chapter_url = chapter.get_attribute("href") # 获取章节链接\n\n driver.get(chapter_url)\n time.sleep(3)\n\n chapter_content = driver.find_element_by_xpath("//*[@id='chaptercontent']")\n html = chapter_content.get_attribute("innerHTML")\n tree = etree.HTML(html)\n content = tree.xpath("string(.)") # 获取章节内容的文本形式\n\n print(chapter_title)\n print(content)\n\n driver.back()\n time.sleep(3)\n\n driver.back()\n time.sleep(3)\n\ndriver.quit()\n\n\n解决 StaleElementReferenceException 错误\n\n在获取章节标题之前，重新查找章节元素，避免页面变化导致元素失效。\n\n完整代码：\n\npython\nfrom selenium import webdriver\nfrom selenium.webdriver.common.keys import Keys\nimport time\nimport random\nfrom lxml import etree\n\nkeyword = "开挂闯异界"\n\ndriver = webdriver.Chrome()\ndriver.get("https://www.bige3.cc/")\n\nsearch_input = driver.find_element_by_xpath("/html/body/div[4]/div[1]/div[2]/form/input[1]")\nsearch_input.send_keys(keyword)\nsearch_input.send_keys(Keys.ENTER)\ntime.sleep(3)\n\nelements = driver.find_elements_by_xpath("/html/body/div[5]/div/div/div/div/div[2]/h4/a")\nfor i in range(len(elements)):\n elements = driver.find_elements_by_xpath("/html/body/div[5]/div/div/div/div/div[2]/h4/a")\n element = elements[i]\n print(element.text)\n delay = random.uniform(3, 5)\n time.sleep(delay)\n element.click()\n\n driver.switch_to.window(driver.window_handles[-1])\n time.sleep(3)\n\n chapters = driver.find_elements_by_xpath("//*[@class='listmain']/dl/dd/a")\n for chapter in chapters:\n chapters = driver.find_elements_by_xpath("//*[@class='listmain']/dl/dd/a")\n chapter = chapters[i]\n chapter_title = chapter.text.replace('、', '')\n chapter_url = chapter.get_attribute("href")\n\n driver.get(chapter_url)\n time.sleep(3)\n\n chapter_content = driver.find_element_by_xpath("//*[@id='chaptercontent']")\n html = chapter_content.get_attribute("innerHTML")\n tree = etree.HTML(html)\n content = tree.xpath("string(.)")\n\n print(chapter_title)\n print(content)\n\n driver.back()\n time.sleep(3)\n\n driver.back()\n time.sleep(3)\n\ndriver.quit()\n\n\n注意：\n\n* 确保你已经安装了 Selenium 和 lxml 库。\n* 使用 Chrome 浏览器时，你需要在系统中配置 ChromeDriver 的路径。\n* 网站结构可能会发生变化，你需要根据实际情况修改代码。\n* 使用爬虫时，请尊重网站的 robots.txt 文件和相关协议，避免过度抓取导致网站服务器压力过大。\n