一个获取知乎用户主页信息的多线程Python爬虫程序

上传者: u014534808 | 上传时间: 2023-03-17 21:10:34 | 文件大小: 19.35MB | 文件类型: ZIP
爬虫python入门 个获取知乎用户主页信息的多线程Python爬虫程序。 简介: * 使用[Requests](http://www.python-requests.org/en/master/)模拟HTTP请求/响应,[Beautiful Soup 4](https://www.crummy.com/software/BeautifulSoup/)提取页面信息。 * 使用Python内置的Thread多线程和IP代理提升爬取速度,并绕过知乎的反爬虫机制。 * 使用Python内置的query作为消息队列。 * 用csv文件存储数据。 ## 环境依赖 * beautifulsoup4 * requests ## 使用方法 在项目路径下输入以安装需要的模块: ```shell $ pip install -r requirments.txt ``` 打开proxy.py文件,在""处填写代理隧道验证信息: ```python # 代理隧道验证信息 proxyUser = "" proxyPass = "" ``` (笔者使用一个IP代理隧道,若需要使用普通IP代理,

文件下载

资源详情

[{"title":"( 49 个子文件 19.35MB ) 一个获取知乎用户主页信息的多线程Python爬虫程序 \n","children":[{"title":"Zhihu-Spider-知乎爬虫","children":[{"title":".DS_Store <span style='color:#111;'> 6.00KB </span>","children":null,"spread":false},{"title":"spider","children":[{"title":"crawl.py <span style='color:#111;'> 3.50KB </span>","children":null,"spread":false},{"title":"run.py <span style='color:#111;'> 6.89KB </span>","children":null,"spread":false},{"title":"datafile.py <span style='color:#111;'> 11.22KB </span>","children":null,"spread":false},{"title":"proxy.py <span style='color:#111;'> 697B </span>","children":null,"spread":false}],"spread":true},{"title":".gitattributes <span style='color:#111;'> 33B </span>","children":null,"spread":false},{"title":"image","children":[{"title":"request.png <span style='color:#111;'> 71.51KB </span>","children":null,"spread":false},{"title":"proxytunnel.png <span style='color:#111;'> 52.40KB </span>","children":null,"spread":false},{"title":"datastate.png <span style='color:#111;'> 88.89KB </span>","children":null,"spread":false},{"title":"proxy.png <span style='color:#111;'> 51.00KB </span>","children":null,"spread":false},{"title":"flow.png <span style='color:#111;'> 335.86KB </span>","children":null,"spread":false},{"title":"file.png <span style='color:#111;'> 82.49KB </span>","children":null,"spread":false},{"title":"run.jpg <span style='color:#111;'> 92.90KB </span>","children":null,"spread":false},{"title":"datafilelist.png <span style='color:#111;'> 62.47KB </span>","children":null,"spread":false},{"title":"datafile.png <span style='color:#111;'> 60.48KB </span>","children":null,"spread":false}],"spread":true},{"title":"analysis","children":[{"title":"cloud.ipynb <span style='color:#111;'> 7.21KB </span>","children":null,"spread":false},{"title":"image","children":[{"title":"3D关注和被关注.png <span style='color:#111;'> 342.22KB </span>","children":null,"spread":false},{"title":"收藏和被收藏.png <span style='color:#111;'> 214.17KB </span>","children":null,"spread":false},{"title":"major.png <span style='color:#111;'> 676.79KB </span>","children":null,"spread":false},{"title":"地理分布.png <span style='color:#111;'> 328.12KB </span>","children":null,"spread":false},{"title":"thankedCount.png <span style='color:#111;'> 899.59KB </span>","children":null,"spread":false},{"title":"mask1.png <span style='color:#111;'> 372.65KB </span>","children":null,"spread":false},{"title":"3D关注和被关注.gif <span style='color:#111;'> 3.15MB </span>","children":null,"spread":false},{"title":"answerCount.png <span style='color:#111;'> 931.31KB </span>","children":null,"spread":false},{"title":"问题话题收藏夹专栏.png <span style='color:#111;'> 202.67KB </span>","children":null,"spread":false},{"title":"回答文章提问.png <span style='color:#111;'> 136.43KB </span>","children":null,"spread":false},{"title":"school.png <span style='color:#111;'> 655.01KB </span>","children":null,"spread":false},{"title":"questionCount.png <span style='color:#111;'> 885.40KB </span>","children":null,"spread":false},{"title":"关注和被关注.png <span style='color:#111;'> 178.99KB </span>","children":null,"spread":false},{"title":"mask2.png <span style='color:#111;'> 185.02KB </span>","children":null,"spread":false},{"title":"3D收藏和被收藏.png <span style='color:#111;'> 277.58KB </span>","children":null,"spread":false},{"title":"followerCount.png <span style='color:#111;'> 847.50KB </span>","children":null,"spread":false},{"title":"3D赞同与感谢.png <span style='color:#111;'> 267.47KB </span>","children":null,"spread":false},{"title":"articlesCount.png <span style='color:#111;'> 921.13KB </span>","children":null,"spread":false},{"title":"job.png <span style='color:#111;'> 670.59KB </span>","children":null,"spread":false},{"title":"赞同和感谢.png <span style='color:#111;'> 180.71KB </span>","children":null,"spread":false},{"title":"business.png <span style='color:#111;'> 541.13KB </span>","children":null,"spread":false},{"title":"voteupCount.png <span style='color:#111;'> 910.26KB </span>","children":null,"spread":false},{"title":"company.png <span style='color:#111;'> 672.43KB </span>","children":null,"spread":false},{"title":"map.png <span style='color:#111;'> 87.84KB </span>","children":null,"spread":false},{"title":"favoritedCount.png <span style='color:#111;'> 873.26KB </span>","children":null,"spread":false},{"title":"3D赞同和感谢.gif <span style='color:#111;'> 1.95MB </span>","children":null,"spread":false}],"spread":false},{"title":"heat.ipynb <span style='color:#111;'> 24.83KB </span>","children":null,"spread":false},{"title":"hist3d.ipynb <span style='color:#111;'> 7.29KB </span>","children":null,"spread":false},{"title":"datawash.py <span style='color:#111;'> 2.86KB </span>","children":null,"spread":false},{"title":"hist.ipynb <span style='color:#111;'> 17.63KB </span>","children":null,"spread":false},{"title":"fonts","children":[{"title":"fangzhengqingkebenyuesongjianti.ttf <span style='color:#111;'> 3.00MB </span>","children":null,"spread":false}],"spread":true}],"spread":true},{"title":"requirments.txt <span style='color:#111;'> 80B </span>","children":null,"spread":false},{"title":"README.md <span style='color:#111;'> 12.98KB </span>","children":null,"spread":false}],"spread":true}],"spread":true}]

评论信息

免责申明

【只为小站】的资源来自网友分享,仅供学习研究,请务必在下载后24小时内给予删除,不得用于其他任何用途,否则后果自负。基于互联网的特殊性,【只为小站】 无法对用户传输的作品、信息、内容的权属或合法性、合规性、真实性、科学性、完整权、有效性等进行实质审查;无论 【只为小站】 经营者是否已进行审查,用户均应自行承担因其传输的作品、信息、内容而可能或已经产生的侵权或权属纠纷等法律责任。
本站所有资源不代表本站的观点或立场,基于网友分享,根据中国法律《信息网络传播权保护条例》第二十二条之规定,若资源存在侵权或相关问题请联系本站客服人员,zhiweidada#qq.com,请把#换成@,本站将给予最大的支持与配合,做到及时反馈和处理。关于更多版权及免责申明参见 版权及免责申明