Python网页文本数据爬取教程：包含关键字和二级链接提取

使用Python爬取网页文本数据可以通过以下步骤实现：

安装相关的Python库： requests、beautifulsoup4

pip install requests beautifulsoup4

发送HTTP请求获取网页源代码

import requests

url = 'https://www.example.com/'
response = requests.get(url)
html = response.text

使用beautifulsoup4解析网页

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'html.parser')

查找关键字

keywords = ['Python', '爬虫']
for keyword in keywords:
    result = soup.find_all(text=keyword)
    if result:
        print(result)

查找二级链接

links = soup.find_all('a')
for link in links:
    href = link.get('href')
    if href.startswith('http'):
        response = requests.get(href)
        sub_html = response.text
        sub_soup = BeautifulSoup(sub_html, 'html.parser')
        sub_result = sub_soup.find_all(text=keyword)
        if sub_result:
            print(sub_result)

完整代码：

import requests
from bs4 import BeautifulSoup

url = 'https://www.example.com/'
response = requests.get(url)
html = response.text

soup = BeautifulSoup(html, 'html.parser')

keywords = ['Python', '爬虫']
for keyword in keywords:
    result = soup.find_all(text=keyword)
    if result:
        print(result)

links = soup.find_all('a')
for link in links:
    href = link.get('href')
    if href.startswith('http'):
        response = requests.get(href)
        sub_html = response.text
        sub_soup = BeautifulSoup(sub_html, 'html.parser')
        sub_result = sub_soup.find_all(text=keyword)
        if sub_result:
            print(sub_result)