Python 爬虫代码:提取任意网页指定文本数据
import re import requests from bs4 import BeautifulSoup from concurrent.futures import ThreadPoolExecutor
url = input('请输入网页链接:') keywords = input('请输入要查找的关键词,以逗号分隔:').split(",") pattern = '|' .join(keywords) regex = re.compile(pattern)
session = requests.Session() session.headers.update({'User-Agent': 'Mozilla/5.0'}) response = session.get(url, timeout=10) html = response.text
soup = BeautifulSoup(html, 'html.parser')
result = regex.search(html) if result: print(result.group())
links = soup.find_all('a') with ThreadPoolExecutor(max_workers=10) as pool: responses = pool.map(session.get, (link.get('href') for link in links if link.get('href').startswith('http')), timeout=10) for response in responses: sub_html = response.text sub_results = regex.findall(sub_html) for sub_result in sub_results: print(sub_result)
对代码进行了修改,添加了用户输入链接和关键词的功能。
原文地址: https://www.cveoy.top/t/topic/loMs 著作权归作者所有。请勿转载和采集!