Python爬虫实战：批量获取中国知网(CNKI)文献摘要教程

前言

本教程将详细介绍如何使用Python爬虫批量获取中国知网(CNKI)的文献摘要信息。即使你是编程新手，也能通过本教程快速掌握相关技能。教程包含完整源码和详细解释，助你轻松入门网络爬虫开发。

准备工作

所需工具

Python 3.6+
requests库
BeautifulSoup4库
pandas库（用于数据存储）

安装依赖

pip install requests beautifulsoup4 pandas

爬虫实现步骤

1. 分析CNKI搜索页面结构

首先我们需要了解CNKI的搜索URL构成和页面元素结构。以搜索"人工智能"为例：

base_url = "https://kns.cnki.net/kns8/defaultresult/index"
params = {"kw": "人工智能",  # 搜索关键词"korder": "SU",  # 按主题排序"crossDbcodes": "CFLS,CJFD,CDMD,CIPD,CCND,CISD,SNAD"
}

2. 模拟请求获取搜索结果

import requests
from bs4 import BeautifulSoup
import pandas as pdheaders = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
}def get_search_results(keyword, page=1):params = {"kw": keyword,"korder": "SU","crossDbcodes": "CFLS,CJFD,CDMD,CIPD,CCND,CISD,SNAD","page": page}try:response = requests.get("https://kns.cnki.net/kns8/defaultresult/index",params=params,headers=headers,timeout=10)response.raise_for_status()return response.textexcept Exception as e:print(f"获取搜索结果失败: {e}")return None

3. 解析页面获取文献列表

def parse_search_results(html):soup = BeautifulSoup(html, 'html.parser')papers = []items = soup.select('.result-item')for item in items:try:title = item.select_one('.result-item-title a').text.strip()link = item.select_one('.result-item-title a')['href']authors = item.select_one('.author').text.strip() if item.select_one('.author') else ''source = item.select_one('.source').text.strip() if item.select_one('.source') else ''date = item.select_one('.date').text.strip() if item.select_one('.date') else ''abstract = item.select_one('.abstract').text.strip() if item.select_one('.abstract') else ''papers.append({'title': title,'link': 'https://kns.cnki.net' + link if link.startswith('/') else link,'authors': authors,'source': source,'date': date,'abstract': abstract})except Exception as e:print(f"解析文献条目时出错: {e}")continuereturn papers

4. 分页获取所有结果

def get_all_papers(keyword, max_pages=5):all_papers = []for page in range(1, max_pages + 1):print(f"正在获取第 {page} 页数据...")html = get_search_results(keyword, page)if html:papers = parse_search_results(html)all_papers.extend(papers)return all_papers

5. 保存结果到Excel

def save_to_excel(papers, filename):df = pd.DataFrame(papers)df.to_excel(filename, index=False)print(f"数据已保存到 {filename}")

6. 主函数

def main():keyword = input("请输入要搜索的关键词: ")max_pages = int(input("请输入要爬取的页数(每页20条): "))papers = get_all_papers(keyword, max_pages)if papers:filename = f"CNKI_{keyword}_搜索结果.xlsx"save_to_excel(papers, filename)else:print("未获取到任何文献数据")if __name__ == "__main__":main()