基于javaweb的爬虫项目

上传者: 2301_76166241 | 上传时间: 2025-09-11 21:24:06 | 文件大小: 110KB | 文件类型: ZIP
一个基于Java的爬虫实战项目,主要功能是抓取知乎用户的基本资料,如果觉得不错,请给个star。 修改zhihu/src/main/resources/application.yamlredis、mongodb相关配置,application.yaml 初始化zhihu/src/main/resources/mongo-init.sqlmongodb脚步,mongo-init.sql 设置日志路径,默认在/var/www/logslogback-spring.xml Run with ZhihuCrawlerApplication.java

文件下载

资源详情

[{"title":"( 117 个子文件 110KB ) 基于javaweb的爬虫项目","children":[{"title":".gitignore <span style='color:#111;'> 94B </span>","children":null,"spread":false},{"title":"SimpleHttpClient.java <span style='color:#111;'> 6.45KB </span>","children":null,"spread":false},{"title":"AbstractPageTask.java <span style='color:#111;'> 5.48KB </span>","children":null,"spread":false},{"title":"ZhihuProxyPageDownloadTask.java <span style='color:#111;'> 5.21KB </span>","children":null,"spread":false},{"title":"ZhihuUserTask.java <span style='color:#111;'> 5.06KB </span>","children":null,"spread":false},{"title":"ZhihuConstants.java <span style='color:#111;'> 4.98KB </span>","children":null,"spread":false},{"title":"Proxy.java <span style='color:#111;'> 4.92KB </span>","children":null,"spread":false},{"title":"TopicActivityPageTask.java <span style='color:#111;'> 4.81KB </span>","children":null,"spread":false},{"title":"AbstractHttpClient.java <span style='color:#111;'> 4.77KB </span>","children":null,"spread":false},{"title":"ZhihuTopicPageTask.java <span style='color:#111;'> 3.75KB </span>","children":null,"spread":false},{"title":"TaskQueueService.java <span style='color:#111;'> 3.63KB </span>","children":null,"spread":false},{"title":"ProxyQueue.java <span style='color:#111;'> 3.63KB </span>","children":null,"spread":false},{"title":"Temp.java <span style='color:#111;'> 3.36KB </span>","children":null,"spread":false},{"title":"ZhihuUserTaskSender.java <span style='color:#111;'> 3.10KB </span>","children":null,"spread":false},{"title":"AbstractPageProxyTestTask.java <span style='color:#111;'> 3.07KB </span>","children":null,"spread":false},{"title":"Constants.java <span style='color:#111;'> 3.02KB </span>","children":null,"spread":false},{"title":"TopicPageTaskSender.java <span style='color:#111;'> 3.02KB </span>","children":null,"spread":false},{"title":"ZhihuQuestionMongodbDao.java <span style='color:#111;'> 2.98KB </span>","children":null,"spread":false},{"title":"ProxyPageProxyPool.java <span style='color:#111;'> 2.73KB </span>","children":null,"spread":false},{"title":"BaseReceiver.java <span style='color:#111;'> 2.33KB </span>","children":null,"spread":false},{"title":"ZhihuTopicMongodbDao.java <span style='color:#111;'> 2.32KB </span>","children":null,"spread":false},{"title":"ProxyPageDownloadTaskSender.java <span style='color:#111;'> 2.29KB </span>","children":null,"spread":false},{"title":"ZhihuPageProxyTestTask.java <span style='color:#111;'> 2.24KB </span>","children":null,"spread":false},{"title":"SimpleThreadPoolExecutor.java <span style='color:#111;'> 2.13KB </span>","children":null,"spread":false},{"title":"RedisLockTest.java <span style='color:#111;'> 2.13KB </span>","children":null,"spread":false},{"title":"ThreadPoolUtil.java <span style='color:#111;'> 2.11KB </span>","children":null,"spread":false},{"title":"ZhihuTopicActivityMongodbDao.java <span style='color:#111;'> 2.08KB </span>","children":null,"spread":false},{"title":"ZhihuComponent.java <span style='color:#111;'> 2.03KB </span>","children":null,"spread":false},{"title":"RedisLockUtil.java <span style='color:#111;'> 2.00KB </span>","children":null,"spread":false},{"title":"ZhihuUserMongodbDao.java <span style='color:#111;'> 1.92KB </span>","children":null,"spread":false},{"title":"ProxyPageProxyTestReceiver.java <span style='color:#111;'> 1.91KB </span>","children":null,"spread":false},{"title":"ZhihuProxyPageProxyTestTask.java <span style='color:#111;'> 1.91KB </span>","children":null,"spread":false},{"title":"ProxyPageDownloadTaskReceiver.java <span style='color:#111;'> 1.82KB </span>","children":null,"spread":false},{"title":"ZhihuPageProxyTestReceiver.java <span style='color:#111;'> 1.79KB </span>","children":null,"spread":false},{"title":"Ip181ProxyListPageParser.java <span style='color:#111;'> 1.62KB </span>","children":null,"spread":false},{"title":"LocalIPService.java <span style='color:#111;'> 1.61KB </span>","children":null,"spread":false},{"title":"ThreadPoolMonitor.java <span style='color:#111;'> 1.52KB </span>","children":null,"spread":false},{"title":"CrawledUrlMongodbDao.java <span style='color:#111;'> 1.51KB </span>","children":null,"spread":false},{"title":"ProxyUtil.java <span style='color:#111;'> 1.46KB </span>","children":null,"spread":false},{"title":"CrawlerUtils.java <span style='color:#111;'> 1.41KB </span>","children":null,"spread":false},{"title":"ZhihuTopicActivityMongoPageParser.java <span style='color:#111;'> 1.37KB </span>","children":null,"spread":false},{"title":"ZhihuArticleMongodbDao.java <span style='color:#111;'> 1.31KB </span>","children":null,"spread":false},{"title":"ZhihuAnswerMongodbDao.java <span style='color:#111;'> 1.28KB </span>","children":null,"spread":false},{"title":"SpringMongoConfig.java <span style='color:#111;'> 1.28KB </span>","children":null,"spread":false},{"title":"MultUrlsAbstractPageTask.java <span style='color:#111;'> 1.27KB </span>","children":null,"spread":false},{"title":"QydailiProxyListPageParser.java <span style='color:#111;'> 1.26KB </span>","children":null,"spread":false},{"title":"ZhihuUser.java <span style='color:#111;'> 1.25KB </span>","children":null,"spread":false},{"title":"JedisConfiguration.java <span style='color:#111;'> 1.23KB </span>","children":null,"spread":false},{"title":"ZhihuTopicMongoPageParser.java <span style='color:#111;'> 1.23KB </span>","children":null,"spread":false},{"title":"Ip66ProxyListPageParser.java <span style='color:#111;'> 1.21KB </span>","children":null,"spread":false},{"title":"ZhihuUserPageParser.java <span style='color:#111;'> 1.20KB </span>","children":null,"spread":false},{"title":"Ip89ProxyListPageParser.java <span style='color:#111;'> 1.20KB </span>","children":null,"spread":false},{"title":"KuaiProxyListPageParserTest.java <span style='color:#111;'> 1.16KB </span>","children":null,"spread":false},{"title":"CrawlerMessage.java <span style='color:#111;'> 1.15KB </span>","children":null,"spread":false},{"title":"XicidailiProxyListPageParser.java <span style='color:#111;'> 1.12KB </span>","children":null,"spread":false},{"title":"CrawledUrl.java <span style='color:#111;'> 1.12KB </span>","children":null,"spread":false},{"title":"MimiipProxyListPageParser.java <span style='color:#111;'> 1.12KB </span>","children":null,"spread":false},{"title":"ProxyHttpClient.java <span style='color:#111;'> 1.12KB </span>","children":null,"spread":false},{"title":"Answer.java <span style='color:#111;'> 1.12KB </span>","children":null,"spread":false},{"title":"KuaiProxyListPageParser.java <span style='color:#111;'> 1.12KB </span>","children":null,"spread":false},{"title":"Md5Util.java <span style='color:#111;'> 1.04KB </span>","children":null,"spread":false},{"title":"Page.java <span style='color:#111;'> 1.03KB </span>","children":null,"spread":false},{"title":"CnProxyListPageParser.java <span style='color:#111;'> 1.01KB </span>","children":null,"spread":false},{"title":"ProxyListPageParserFactory.java <span style='color:#111;'> 1004B </span>","children":null,"spread":false},{"title":"BaseMongodbDao.java <span style='color:#111;'> 960B </span>","children":null,"spread":false},{"title":"ShutdownService.java <span style='color:#111;'> 925B </span>","children":null,"spread":false},{"title":"TopicPageTaskReceiver.java <span style='color:#111;'> 891B </span>","children":null,"spread":false},{"title":"ZhihuUserTaskReceiver.java <span style='color:#111;'> 870B </span>","children":null,"spread":false},{"title":"ZhihuHttpClient.java <span style='color:#111;'> 860B </span>","children":null,"spread":false},{"title":"CrawledUrlUtil.java <span style='color:#111;'> 786B </span>","children":null,"spread":false},{"title":"PatternUtil.java <span style='color:#111;'> 749B </span>","children":null,"spread":false},{"title":"Question.java <span style='color:#111;'> 741B </span>","children":null,"spread":false},{"title":"Article.java <span style='color:#111;'> 730B </span>","children":null,"spread":false},{"title":"IhuanProxyListPageParser.java <span style='color:#111;'> 718B </span>","children":null,"spread":false},{"title":"Target.java <span style='color:#111;'> 678B </span>","children":null,"spread":false},{"title":"Author.java <span style='color:#111;'> 673B </span>","children":null,"spread":false},{"title":"Topic.java <span style='color:#111;'> 660B </span>","children":null,"spread":false},{"title":"ZhihuCrawlerConfiguration.java <span style='color:#111;'> 654B </span>","children":null,"spread":false},{"title":"PatternUtilTest.java <span style='color:#111;'> 575B </span>","children":null,"spread":false},{"title":"ZhihuCrawlerApplication.java <span style='color:#111;'> 504B </span>","children":null,"spread":false},{"title":"TopicActivity.java <span style='color:#111;'> 472B </span>","children":null,"spread":false},{"title":"ExceptionTest.java <span style='color:#111;'> 441B </span>","children":null,"spread":false},{"title":"ServletInitalizer.java <span style='color:#111;'> 441B </span>","children":null,"spread":false},{"title":"CommonProperties.java <span style='color:#111;'> 435B </span>","children":null,"spread":false},{"title":"SimpleSpringJUnit4ClassRunner.java <span style='color:#111;'> 428B </span>","children":null,"spread":false},{"title":"AsyncHttpClientUtilTest.java <span style='color:#111;'> 428B </span>","children":null,"spread":false},{"title":"SystemUtil.java <span style='color:#111;'> 405B </span>","children":null,"spread":false},{"title":"ListPageParser.java <span style='color:#111;'> 381B </span>","children":null,"spread":false},{"title":"ZhihuUserRepository.java <span style='color:#111;'> 360B </span>","children":null,"spread":false},{"title":"RetryHandler.java <span style='color:#111;'> 356B </span>","children":null,"spread":false},{"title":"ProxyListPageParser.java <span style='color:#111;'> 332B </span>","children":null,"spread":false},{"title":"CrawledUrlRepository.java <span style='color:#111;'> 302B </span>","children":null,"spread":false},{"title":"ZhihuQuestionRepository.java <span style='color:#111;'> 300B </span>","children":null,"spread":false},{"title":"Location.java <span style='color:#111;'> 300B </span>","children":null,"spread":false},{"title":"ZhihuArticleRepository.java <span style='color:#111;'> 297B </span>","children":null,"spread":false},{"title":"ZhihuAnswerRepository.java <span style='color:#111;'> 294B </span>","children":null,"spread":false},{"title":"ZhihuTopicRepository.java <span style='color:#111;'> 291B </span>","children":null,"spread":false},{"title":"BaseSender.java <span style='color:#111;'> 283B </span>","children":null,"spread":false},{"title":"ZhihuTopicActivityRepository.java <span style='color:#111;'> 274B </span>","children":null,"spread":false},{"title":"SinglePool.java <span style='color:#111;'> 243B </span>","children":null,"spread":false},{"title":"......","children":null,"spread":false},{"title":"<span style='color:steelblue;'>文件过多,未全部展示</span>","children":null,"spread":false}],"spread":true}]

评论信息

免责申明

【只为小站】的资源来自网友分享,仅供学习研究,请务必在下载后24小时内给予删除,不得用于其他任何用途,否则后果自负。基于互联网的特殊性,【只为小站】 无法对用户传输的作品、信息、内容的权属或合法性、合规性、真实性、科学性、完整权、有效性等进行实质审查;无论 【只为小站】 经营者是否已进行审查,用户均应自行承担因其传输的作品、信息、内容而可能或已经产生的侵权或权属纠纷等法律责任。
本站所有资源不代表本站的观点或立场,基于网友分享,根据中国法律《信息网络传播权保护条例》第二十二条之规定,若资源存在侵权或相关问题请联系本站客服人员,zhiweidada#qq.com,请把#换成@,本站将给予最大的支持与配合,做到及时反馈和处理。关于更多版权及免责申明参见 版权及免责申明