使用python语言应用requests库读取domaintxt内网址添加随机产生五位字母的二级域名并检测是否包含http若无则添加使用requests库对读取出来的网址应用随机user agent及代理取得网页响应的标题并存入resultstxt内
以下是一个可能的实现:
import random
import requests
# 读取domain.txt内网址
with open('domain.txt', 'r') as f:
domains = f.read().splitlines()
# 定义随机user agent列表和代理列表
user_agents = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0'
]
proxies = [
{'http': 'http://10.10.1.10:3128', 'https': 'http://10.10.1.10:1080'},
{'http': 'http://10.10.2.10:3128', 'https': 'http://10.10.2.10:1080'},
{'http': 'http://10.10.3.10:3128', 'https': 'http://10.10.3.10:1080'},
]
# 遍历每个网址,添加随机产生五位字母的二级域名并检测是否包含http://
for domain in domains:
if 'http://' not in domain:
domain = 'http://' + domain
subdomain = ''.join(random.choice('abcdefghijklmnopqrstuvwxyz') for _ in range(5))
url = f"{subdomain}.{domain}"
print(f"访问 {url}")
# 随机选择一个user agent和代理
headers = {'User-Agent': random.choice(user_agents)}
proxy = random.choice(proxies)
# 使用requests库对网页进行访问并获取标题
try:
response = requests.get(url, headers=headers, proxies=proxy, timeout=10)
title = response.text.split('<title>')[1].split('</title>')[0]
print(f"标题为:{title}")
# 存入results.txt内
with open('results.txt', 'a') as f:
f.write(f"{url}: {title}\n")
except:
print("访问出错")
需要注意的是,由于使用代理访问网页可能会被封禁,因此建议使用多个代理,每次随机选择一个,以减少被封禁的风险。另外,由于访问网页可能会很慢,因此需要设置超时时间,避免程序长时间卡在某个访问失败的网页上
原文地址: https://www.cveoy.top/t/topic/fjuC 著作权归作者所有。请勿转载和采集!