实战：Python爬取东方财富股吧财经新闻与动态评论（附完整代码）
1. 环境准备与工具选择做爬虫项目前选对工具相当于厨师选好了菜刀。我推荐用Python 3.8版本主要依赖这几个库requests比urllib更人性化的网络请求库BeautifulSoupHTML解析神器lxmlXPath解析必备openpyxl处理Excel文件安装这些库特别简单一行命令搞定pip install requests beautifulsoup4 lxml openpyxl新手常犯的错误是直接上手写代码结果被反爬机制按在地上摩擦。建议先准备好这些防封杀装备随机User-Agent池至少准备10个不同浏览器标识代理IP池免费的可试试快代理、站大爷请求间隔随机化1-3秒比较安全2. 静态页面抓取实战东方财富股吧的新闻列表页是典型的分页结构URL规律很明显http://guba.eastmoney.com/list,股票代码,1,f_页码.html比如贵州茅台600519的第2页base_url http://guba.eastmoney.com/list,600519,1,f_{}.html抓取标题和阅读量的核心代码def parse_news_list(html): soup BeautifulSoup(html, html.parser) titles [a[title] for a in soup.select(.l3.a3 a)] reads [span.text for span in soup.select(.l1.a1 span)] return list(zip(titles, reads))这里有个坑要注意东方财富的阅读量数据有时候会藏在>reads [span[data-read] for span in soup.select(.l1.a1 span)]3. 动态评论抓取技巧动态内容才是真正的挑战。通过Chrome开发者工具F12我发现了评论数据的API接口http://guba.eastmoney.com/interface/GetData.aspx关键参数是postid这个ID藏在新闻详情页的URL里。比如这个链接https://guba.eastmoney.com/news,600519,123456789.html其中的123456789就是postid。模拟请求的代码示例def get_comments(postid): url http://guba.eastmoney.com/interface/GetData.aspx headers { Referer: fhttps://guba.eastmoney.com/news,600519,{postid}.html, User-Agent: Mozilla/5.0... } data { param: fpostid{postid}sort1sorttype1p1ps50, path: reply/api/Reply/ArticleNewReplyList, env: 2 } response requests.post(url, headersheaders, datadata) return response.json()[re] # 返回评论列表4. 反爬策略实战心得我连续抓取50页后被封IP的经历告诉大家这些防护措施必须做随机延时别用固定间隔from random import uniform sleep(uniform(1.5, 3)) # 1.5-3秒随机间隔请求头伪装特别是Cookie和Refererheaders { User-Agent: random.choice(user_agents), Referer: random.choice(referers), X-Requested-With: XMLHttpRequest }异常处理遇到429状态码立即暂停try: response requests.get(url, timeout10) if response.status_code 429: print(触发反爬等待5分钟...) sleep(300) except Exception as e: print(f请求失败{str(e)})5. 数据存储优化方案原始代码用Excel存储当数据量超过1万条时会变得很卡。我推荐三种进阶方案CSV分批存储每1000条存一个文件import csv def save_to_csv(data, filename): with open(filename, a, newline, encodingutf-8) as f: writer csv.writer(f) writer.writerow(data.values())MySQL存储适合长期积累数据import pymysql conn pymysql.connect(hostlocalhost, userroot, password123456, databaseguba) cursor conn.cursor() sql INSERT INTO news(title,read_count) VALUES(%s,%s) cursor.execute(sql, (茅台股价创新高, 10000)) conn.commit()MongoDB存储处理非结构化评论更方便from pymongo import MongoClient client MongoClient(mongodb://localhost:27017/) db client[guba] db.comments.insert_one({ postid: 123456, content: 茅台永远的神, like: 15 })6. 完整代码示例把上述技巧整合后的完整流程import requests from bs4 import BeautifulSoup from time import sleep from random import uniform, choice import csv user_agents [ Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit..., Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)... ] def crawl_stock_news(stock_code, max_page10): base_url fhttp://guba.eastmoney.com/list,{stock_code},1,f_ with open(result.csv, w, encodingutf-8) as f: writer csv.writer(f) writer.writerow([标题, 阅读量, 评论数]) for page in range(1, max_page1): sleep(uniform(1, 3)) url base_url str(page) .html try: html requests.get(url, headers{User-Agent: choice(user_agents)}).text soup BeautifulSoup(html, html.parser) # 解析新闻列表 news_items soup.select(.articleh) for item in news_items: title item.select(.l3 a)[0][title] read item.select(.l1)[0].text comment item.select(.l2)[0].text writer.writerow([title, read, comment]) except Exception as e: print(f第{page}页抓取出错{str(e)}) continue7. 常见问题解决方案问题1返回数据乱码解决方法强制指定编码response.encoding utf-8 # 或者 response.apparent_encoding问题2动态加载内容缺失解决方案用Selenium模拟浏览器from selenium import webdriver driver webdriver.Chrome() driver.get(url) html driver.page_source问题3验证码拦截应对策略降低采集频率使用打码平台如超级鹰保存Cookies维持会话最后提醒爬虫要遵守robots.txt规则控制请求频率建议在非交易时段如凌晨运行采集任务。完整项目代码我已经上传到GitHub包含异常处理、日志记录等工业级实现需要的话可以私信我获取
