c# 多线程蜘蛛爬虫
1.CSpider文件夹下是我开始编写前参考的开源代码，结构比较清晰。单线程
2.任务概述是我这次写spider的过程的描述，我做这次spider的目的是完成一项课程设计。
3.整个spider的开发过程为增量式，从最开始的单页搜索url到最后的多线程spider，这在版本规划中又很明显的体现。每个版本都有自己的readme，其中详细叙述了各个版本的不同
4.由于是初学者，我在每个版本中都是边写边测试，以查找爬虫的bug，所以代码中有很多被注释的段落可能是测试使用。
5.具体版本信息查看各自的readme
另外，爬虫不止提取了url，同时提取了相应的锚文本。这个提取过程导致了爬虫速度的下降，但也是一个必须的开销
文件下载
资源详情
[{"title":"（ 201 个子文件 2.42MB ） c# 多线程蜘蛛爬虫","children":[{"title":"SpiderTest.application <span style='color:#111;'> 1.44KB </span>","children":null,"spread":false},{"title":"SpiderTest.vshost.application <span style='color:#111;'> 1.44KB </span>","children":null,"spread":false},{"title":"SpiderTest.application <span style='color:#111;'> 1.44KB </span>","children":null,"spread":false},{"title":"SpiderTest.application <span style='color:#111;'> 1.44KB </span>","children":null,"spread":false},{"title":"ResolveAssemblyReference.cache <span style='color:#111;'> 6.03KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 8.51KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 8.48KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 8.48KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 7.53KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 5.91KB </span>","children":null,"spread":false},{"title":"analyze_Html.cs <span style='color:#111;'> 4.86KB </span>","children":null,"spread":false},{"title":"analyze_Html.cs <span style='color:#111;'> 4.65KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 3.80KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 2.96KB </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 2.62KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.09KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.09KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.09KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.09KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 2.01KB </span>","children":null,"spread":false},{"title":"pageUrlResearch.cs <span style='color:#111;'> 1.79KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.43KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.29KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.29KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.23KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.15KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.15KB </span>","children":null,"spread":false},{"title":"AssemblyInfo.cs <span style='color:#111;'> 1.15KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.01KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.01KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.01KB </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 1.01KB </span>","children":null,"spread":false},{"title":"rooturl.cs <span style='color:#111;'> 950B </span>","children":null,"spread":false},{"title":"rooturl.cs <span style='color:#111;'> 950B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 928B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 928B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 928B </span>","children":null,"spread":false},{"title":"analyze.cs <span style='color:#111;'> 778B </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 745B </span>","children":null,"spread":false},{"title":"Url.cs <span style='color:#111;'> 741B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 686B </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 630B </span>","children":null,"spread":false},{"title":"Spider.cs <span style='color:#111;'> 630B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 579B </span>","children":null,"spread":false},{"title":"analyze.cs <span style='color:#111;'> 554B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 493B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 462B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 455B </span>","children":null,"spread":false},{"title":"Program.cs <span style='color:#111;'> 411B </span>","children":null,"spread":false},{"title":"ISpider.cs <span style='color:#111;'> 409B </span>","children":null,"spread":false},{"title":"ISpider.cs <span style='color:#111;'> 401B </span>","children":null,"spread":false},{"title":"store.cs <span style='color:#111;'> 195B </span>","children":null,"spread":false},{"title":"store.cs <span style='color:#111;'> 195B </span>","children":null,"spread":false},{"title":"extractUrl.cs <span style='color:#111;'> 169B </span>","children":null,"spread":false},{"title":"SpiderTest.csproj <span style='color:#111;'> 3.96KB </span>","children":null,"spread":false},{"title":"SpiderTest.csproj <span style='color:#111;'> 3.26KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.70KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.69KB </span>","children":null,"spread":false},{"title":"IR.csproj <span style='color:#111;'> 2.58KB </span>","children":null,"spread":false},{"title":"CSpider.csproj <span style='color:#111;'> 2.33KB </span>","children":null,"spread":false},{"title":"CSpider.csproj <span style='color:#111;'> 2.17KB </span>","children":null,"spread":false},{"title":"UpgradeReport.css <span style='color:#111;'> 3.27KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 20.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 20.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 20.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 20.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 16.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 9.00KB </span>","children":null,"spread":false},{"title":"CSpider.dll <span style='color:#111;'> 9.00KB </span>","children":null,"spread":false},{"title":"任务概述.doc <span style='color:#111;'> 99.50KB </span>","children":null,"spread":false},{"title":"SpiderTest.exe <span style='color:#111;'> 16.00KB </span>","children":null,"spread":false},{"title":"SpiderTest.exe <span style='color:#111;'> 16.00KB </span>","children":null,"spread":false},{"title":"SpiderTest.exe <span style='color:#111;'> 16.00KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"SpiderTest.vshost.exe <span style='color:#111;'> 13.99KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 10.00KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 10.00KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 9.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 9.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 9.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 9.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 8.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 8.50KB </span>","children":null,"spread":false},{"title":"ConsoleApplication13.exe <span style='color:#111;'> 8.50KB </span>","children":null,"spread":false},{"title":"......","children":null,"spread":false},{"title":"<span style='color:steelblue;'>文件过多，未全部展示</span>","children":null,"spread":false}],"spread":true}]
评论信息
其他资源
- 基于LMS算法的自适应滤波器FPGA代码实现
- 烟花算法进行函数优化通用matlab代码
- MNIST数据集的gist特征提取
- 车票管理系统（C语言版）
- 波束赋形——matlab程序
- 以太网PHY芯片直接使用FPGA驱动的参考代码
- PyQt5:简单视频播放器
- origin8.0免安装版本
- Qt 下的udp socket通信示例
- ALIENTEK探索者STM32F4开发板案例
- 随机过程及其应用 (第三版) 刘次华 高等教育出版社
- PowerBuilder2019和谐补丁（解压密码：abc123）.rar
- flash喜洋洋与灰太狼动画源文件
- 超大文件切割工具TXT杀手速度快不占内存
- 2019中国中小学校外辅导市场与课业负担状况专题调查报告.pdf
- Rockchip_RK809_Developer_Guide_CN.pdf
- windows1.0_downyi.com.zip
- google pay
- vc6++安装包
- STM8L152R6 STM8L152R8 SCH封装
- 万能BIOS刷新升级工具895
- 基于单片机的多路压力测量仪设计
- android自定义权限
- 基于乘积量化的快速特征匹配
- SWB-A31数据手册
- 整理出div+css网页标准版式经典布局(50种布局方式)
免责申明
【只为小站】的资源来自网友分享，仅供学习研究，请务必在下载后24小时内给予删除，不得用于其他任何用途，否则后果自负。基于互联网的特殊性，【只为小站】 无法对用户传输的作品、信息、内容的权属或合法性、合规性、真实性、科学性、完整权、有效性等进行实质审查；无论 【只为小站】 经营者是否已进行审查，用户均应自行承担因其传输的作品、信息、内容而可能或已经产生的侵权或权属纠纷等法律责任。
本站所有资源不代表本站的观点或立场，基于网友分享，根据中国法律《信息网络传播权保护条例》第二十二条之规定，若资源存在侵权或相关问题请联系本站客服人员，zhiweidada#qq.com，请把#换成@，本站将给予最大的支持与配合，做到及时反馈和处理。关于更多版权及免责申明参见 版权及免责申明
本站所有资源不代表本站的观点或立场，基于网友分享，根据中国法律《信息网络传播权保护条例》第二十二条之规定，若资源存在侵权或相关问题请联系本站客服人员，zhiweidada#qq.com，请把#换成@，本站将给予最大的支持与配合，做到及时反馈和处理。关于更多版权及免责申明参见 版权及免责申明