Python 协程多进程并发访问URL:解决DeprecationWarning错误并优化代码
这个错误是因为在使用asyncio模块时没有创建事件循环(event loop)对象。可以通过在程序的开始处添加asyncio.get_event_loop()来解决这个问题。以下是一个实现从文本文件中读取URL并使用协程和多进程并发访问的示例代码:
import asyncio
import aiohttp
import concurrent.futures
async def fetch(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.text()
async def process_batch(urls, max_processes):
results = []
with concurrent.futures.ProcessPoolExecutor(max_workers=max_processes) as executor:
loop = asyncio.get_event_loop()
tasks = [loop.run_in_executor(executor, fetch, url) for url in urls]
for response in await asyncio.gather(*tasks):
results.append(response)
return results
async def main(filename, batch_size, max_processes):
with open(filename, 'r') as file:
urls = [line.strip() for line in file][:batch_size]
return await process_batch(urls, max_processes)
if __name__ == '__main__':
filename = 'urls.txt'
batch_size = 10
max_processes = 5
loop = asyncio.get_event_loop()
results = loop.run_until_complete(main(filename, batch_size, max_processes))
loop.close()
for result in results:
print(result[:100]) # 打印每个URL的前100个字符
请确保在运行代码之前安装了aiohttp模块,可以通过运行pip install aiohttp来安装。另外,将filename变量设置为包含URL的文本文件的路径,batch_size变量设置为需要读取的URL行数,max_processes变量设置为每批最多的进程数。
原文地址: https://www.cveoy.top/t/topic/p3Qv 著作权归作者所有。请勿转载和采集!