Python 爬取当当网图书数据：代码示例及详细解析

实现这个爬虫需要一定的编程基础和相关的库的使用。以下是一个简单的示例代码，可以参考。

1. 导入所需库

import requests
from bs4 import BeautifulSoup
import csv
import os
import pymongo

2. 定义函数获取HTML页面

def get_html(url):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
    try:
        r = requests.get(url, headers=headers)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        return r.text
    except:
        print('获取页面失败')
        return ''

3. 定义函数获取书籍信息并存储

def get_book_info(url):
    html = get_html(url)
    soup = BeautifulSoup(html, 'html.parser')
    books = soup.find_all('li', class_='list_num')
    for book in books:
        try:
            img_url = book.find('img')['data-original']
            book_url = book.find('a')['href']
            book_id = book_url.split('/')[-1].split('.')[0]
            book_html = get_html(book_url)
            book_soup = BeautifulSoup(book_html, 'html.parser')
            book_info = book_soup.find('div', class_='messbox_info')
            book_name = book_info.find('h1').get_text().strip()
            book_author = book_info.find('span', class_='t1').find_all('a')[0].get_text().strip()
            book_publisher = book_info.find('span', class_='t1').find_all('a')[1].get_text().strip()
            book_pub_time = book_info.find_all('span', class_='t1')[2].get_text().strip()
            book_price = book_info.find('p', class_='price').find('span', class_='price_n').get_text().strip()
            book_intro = book_soup.find('div', class_='content').get_text().strip()
            save_to_csv([book_id, book_name, book_author, book_publisher, book_pub_time, book_price, book_intro, img_url])
            save_to_db(book_id, book_name, book_author, book_publisher, book_pub_time, book_price, book_intro, img_url)
            save_img(img_url, book_id)
            print('保存书籍信息成功：', book_name)
        except:
            print('保存书籍信息失败')

4. 定义函数将数据存储到CSV文件中

def save_to_csv(book_info):
    with open('book_info.csv', 'a', newline='', encoding='utf-8-sig') as f:
        writer = csv.writer(f)
        writer.writerow(book_info)

5. 定义函数将图片保存到本地

def save_img(img_url, book_id):
    if not os.path.exists('download'):
        os.mkdir('download')
    img_path = 'download/{}.jpg'.format(book_id)
    with open(img_path, 'wb') as f:
        f.write(requests.get(img_url).content)

6. 定义函数将数据存储到MongoDB中

def save_to_db(book_id, book_name, book_author, book_publisher, book_pub_time, book_price, book_intro, img_url):
    client = pymongo.MongoClient('mongodb://localhost:27017/')
    db = client['dangdang_book']
    col = db['book_info']
    book = {
        'book_id': book_id,
        'book_name': book_name,
        'book_author': book_author,
        'book_publisher': book_publisher,
        'book_pub_time': book_pub_time,
        'book_price': book_price,
        'book_intro': book_intro,
        'img_url': img_url
    }
    col.insert_one(book)

7. 定义主函数实现翻页爬取和调用其他函数

def main():
    keyword = 'python'
    for i in range(1, 11):
        url = 'http://search.dangdang.com/?key={}&act=input&page_index={}'.format(keyword, i)
        get_book_info(url)

8. 在主程序中调用主函数

if __name__ == '__main__':
    main()

这是一个简单的示例代码，可以根据需要进行修改和优化。但需要注意的是，在爬取网站数据时需要遵守相关法律法规，不得违反网站的访问规则和反爬虫机制。