c# 多线程蜘蛛爬虫
1.CSpider文件夹下是我开始编写前参考的开源代码，结构比较清晰。单线程
2.任务概述是我这次写spider的过程的描述，我做这次spider的目的是完成一项课程设计。
3.整个spider的开发过程为增量式，从最开始的单页搜索url到最后的多线程spider，这在版本规划中又很明显的体现。每个版本都有自己的readme，其中详细叙述了各个版本的不同
4.由于是初学者，我在每个版本中都是边写边测试，以查找爬虫的bug，所以代码中有很多被注释的段落可能是测试使用。
5.具体版本信息查看各自的readme
另外，爬虫不止提取了url，同时提取了相应的锚文本。这个提取过程导致了爬虫速度的下降，但也是一个必须的开销
文件下载
资源详情
[{"title":"（ 201 个子文件 2.42MB ） c# 多线程蜘蛛爬虫","children":[{"title":"SpiderTest.application <span style='color:#111;'> 1.44KB </span>","children":null,"spread":false},{"title":"SpiderTest.vshost.application <span style='color:#111;'> 1.44KB </span>","children":null,"spread":false},{"title":"SpiderTest.application <span style='color:#111;'> 1.44KB </span>","children":null,"spread":false},{"title":"SpiderTest.application <span style='color:#111;'> 1.44KB </span>","children":null,"spread":false},{"title":"ResolveAssemblyReference.cache <span style='color:#111;'> 6.03KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 8.51KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 8.48KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 8.48KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 7.53KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 5.91KB </span>","children":null,"spread":false},{"title":"analyze_Html.cs <span style='color:#111;'> 4.86KB </span>","children":null,"spread":false},{"title":"analyze_Html.cs <span style='color:#111;'> 4.65KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 3.80KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 2.96KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 2.62KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.09KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.09KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.09KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.09KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.01KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 1.79KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.29KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.29KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.23KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.15KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.15KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.15KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.01KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.01KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.01KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.01KB </span>","children":null,"spread":false},{"title":"rooturl.cs <span style='color:#111;'> 950B </span>","children":null,"spread":false},{"title":"rooturl.cs <span style='color:#111;'> 950B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 928B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 928B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 928B </span>","children":null,"spread":false},{"title":"analyze.cs <span style='color:#111;'> 778B </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 745B </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 741B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 686B </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 630B </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 630B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 579B </span>","children":null,"spread":false},{"title":"analyze.cs <span style='color:#111;'> 554B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 493B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 462B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 455B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 411B </span>","children":null,"spread":false},{"title":"ISpider.cs <span style='color:#111;'> 409B </span>","children":null,"spread":false},{"title":"ISpider.cs <span style='color:#111;'> 401B </span>","children":null,"spread":false},{"title":"store.cs <span style='color:#111;'> 195B </span>","children":null,"spread":false},{"title":"store.cs <span style='color:#111;'> 195B </span>","children":null,"spread":false},{"title":"extractUrl.cs <span style='color:#111;'> 169B </span>","children":null,"spread":false},{"title":"SpiderTest.csproj <span style='color:#111;'> 3.96KB </span>","children":null,"spread":false},{"title":"SpiderTest.csproj <span style='color:#111;'> 3.26KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.69KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.58KB </span>","children":null,"spread":false},{"title":"CSpider.csproj <span style='color:#111;'> 2.33KB </span>","children":null,"spread":false},{"title":"CSpider.csproj <span style='color:#111;'> 2.17KB </span>","children":null,"spread":false},{"title":"UpgradeReport.css <span style='color:#111;'> 3.27KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 20.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 20.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 20.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 20.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 16.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 9.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 9.00KB </span>","children":null,"spread":false},{"title":"任务概述.doc <span style='color:#111;'> 99.50KB </span>","children":null,"spread":false},{"title":"SpiderTest.exe <span style='color:#111;'> 16.00KB </span>","children":null,"spread":false},{"title":"SpiderTest.exe <span style='color:#111;'> 16.00KB </span>","children":null,"spread":false},{"title":"SpiderTest.exe <span style='color:#111;'> 16.00KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"SpiderTest.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 10.00KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 10.00KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 9.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 9.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 9.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 9.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 8.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 8.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 8.50KB </span>","children":null,"spread":false},{"title":"......","children":null,"spread":false},{"title":"<span style='color:steelblue;'>文件过多，未全部展示</span>","children":null,"spread":false}],"spread":true}]
评论信息
其他资源
- 基于msp430f149的FFT算法，结果在1602液晶上显示
- GY2561 TSL2561 光强传感器资料 程序 手册 介绍
- gcc-4.8.5-36.el7.x86_64.rpm
- ArcGIS流域划分工具箱
- 五子棋(含较高级的人机对战)
- TMS570LS1224片内FLASH读写例程（自建工程）
- 作业调度三种算法FCFS，SJF，HRN
- 于剑-机器学习每章重点
- 校园网络系统方案设计.docx
- Gojs设备拓扑图Demo实例（设备，端口均可点击）
- VGA显示汉字
- as3.0 flash 视频播放器
- MAX44009IIC文件
- vb6.0写变速齿轮源代码（VB+VC全套代码）.rar
- 戴尔T5820工作站nvme固态盘驱动
- h5塔防游戏源码
- 川崎机器人 E系列 故障查找和排除手册
- 数据库实验报告（2-13）
- horsematch 赛马程序游戏
- AForge.net视频录制
- Python 贪吃蛇
免责申明
【只为小站】的资源来自网友分享，仅供学习研究，请务必在下载后24小时内给予删除，不得用于其他任何用途，否则后果自负。基于互联网的特殊性，【只为小站】 无法对用户传输的作品、信息、内容的权属或合法性、合规性、真实性、科学性、完整权、有效性等进行实质审查；无论 【只为小站】 经营者是否已进行审查，用户均应自行承担因其传输的作品、信息、内容而可能或已经产生的侵权或权属纠纷等法律责任。
本站所有资源不代表本站的观点或立场，基于网友分享，根据中国法律《信息网络传播权保护条例》第二十二条之规定，若资源存在侵权或相关问题请联系本站客服人员，zhiweidada#qq.com，请把#换成@，本站将给予最大的支持与配合，做到及时反馈和处理。关于更多版权及免责申明参见 版权及免责申明
本站所有资源不代表本站的观点或立场，基于网友分享，根据中国法律《信息网络传播权保护条例》第二十二条之规定，若资源存在侵权或相关问题请联系本站客服人员，zhiweidada#qq.com，请把#换成@，本站将给予最大的支持与配合，做到及时反馈和处理。关于更多版权及免责申明参见 版权及免责申明