dwtc-extractor:用于创建德累斯顿网络表语料库的提取代码-源码

上传者: 42139302 | 上传时间: 2021-06-22 19:20:51 | 文件大小: 288KB | 文件类型: ZIP
DWTC-Extractor:用于常见爬网的 Web 表提取器 这是用于创建的完整提取器。 注意:这部分基于最初作为一部分发布的代码。 原始存储库可在找到。 修改包括: 删除不必要的代码片段并更新依赖项 将集成到原始提取代码中,以便能够处理使用 WARC 文件格式的较新版本的 Common Crawl。 这也意味着该提取器是基于 WARC 的新版 Common Crawl 上的任何提取任务的良好基础。 添加网络表格识别和提取代码(在“webreduce”包中) 另请注意,Web Data Commons 项目现在基于旧版本的 Common Crawl 发布。 此代码是在 WDC 语料库发布之前分叉的。 内容 运行提取器(原始文档) 以下是原作者对此代码的原始评论,必要时进行了编辑。 主脚本的文档大多仍然适用,或者经过编辑以与新版本的 CC 配合使用。 此实现使用 AWS 管道

文件下载

资源详情

[{"title":"( 35 个子文件 288KB ) dwtc-extractor:用于创建德累斯顿网络表语料库的提取代码-源码","children":[{"title":"dwtc-extractor-master","children":[{"title":"bin","children":[{"title":"master <span style='color:#111;'> 127B </span>","children":null,"spread":false}],"spread":true},{"title":"pom.xml <span style='color:#111;'> 5.66KB </span>","children":null,"spread":false},{"title":"LICENSE <span style='color:#111;'> 11.09KB </span>","children":null,"spread":false},{"title":"README.markdown <span style='color:#111;'> 5.73KB </span>","children":null,"spread":false},{"title":"src","children":[{"title":"main","children":[{"title":"resources","children":[{"title":"jets3.properties <span style='color:#111;'> 139B </span>","children":null,"spread":false},{"title":"SimpleCart_P2.mdl <span style='color:#111;'> 356.71KB </span>","children":null,"spread":false},{"title":"log4j.properties <span style='color:#111;'> 294B </span>","children":null,"spread":false},{"title":"webreduce.properties <span style='color:#111;'> 856B </span>","children":null,"spread":false},{"title":"SimpleCart_P1.mdl <span style='color:#111;'> 777.73KB </span>","children":null,"spread":false},{"title":"RandomForest_P2.mdl <span style='color:#111;'> 193.46KB </span>","children":null,"spread":false},{"title":"RandomForest_P1.mdl <span style='color:#111;'> 220.32KB </span>","children":null,"spread":false}],"spread":true},{"title":"java","children":[{"title":"webreduce","children":[{"title":"extraction","children":[{"title":"ExtractionAlgorithm.java <span style='color:#111;'> 364B </span>","children":null,"spread":false},{"title":"TableExtractionModule.java <span style='color:#111;'> 1.67KB </span>","children":null,"spread":false},{"title":"mh","children":[{"title":"features","children":[{"title":"FeaturesP1.java <span style='color:#111;'> 17.37KB </span>","children":null,"spread":false},{"title":"FeaturesP2.java <span style='color:#111;'> 24.66KB </span>","children":null,"spread":false}],"spread":false},{"title":"TableClassification.java <span style='color:#111;'> 4.15KB </span>","children":null,"spread":false},{"title":"warcTools","children":[{"title":"ExtractedTable.java <span style='color:#111;'> 959B </span>","children":null,"spread":false}],"spread":false},{"title":"tools","children":[{"title":"ClassificationResult.java <span style='color:#111;'> 1.30KB </span>","children":null,"spread":false},{"title":"Tools.java <span style='color:#111;'> 3.94KB </span>","children":null,"spread":false},{"title":"TableStats.java <span style='color:#111;'> 382B </span>","children":null,"spread":false},{"title":"TableConvert.java <span style='color:#111;'> 5.44KB </span>","children":null,"spread":false},{"title":"ContentType.java <span style='color:#111;'> 331B </span>","children":null,"spread":false},{"title":"CellTools.java <span style='color:#111;'> 2.88KB </span>","children":null,"spread":false}],"spread":false},{"title":"MHExtractionAlgorithm.java <span style='color:#111;'> 6.00KB </span>","children":null,"spread":false}],"spread":true},{"title":"basic","children":[{"title":"BasicExtractionAlgorithm.java <span style='color:#111;'> 10.52KB </span>","children":null,"spread":false}],"spread":false},{"title":"StatsKeeper.java <span style='color:#111;'> 1.55KB </span>","children":null,"spread":false},{"title":"DocumentMetadata.java <span style='color:#111;'> 791B </span>","children":null,"spread":false}],"spread":true},{"title":"terms","children":[{"title":"LuceneNormalizer.java <span style='color:#111;'> 1.93KB </span>","children":null,"spread":false}],"spread":true}],"spread":true},{"title":"org","children":[{"title":"fuberlin","children":[{"title":"wbsg","children":[{"title":"ccrdf","children":[{"title":"ProcessingNode.java <span style='color:#111;'> 4.84KB </span>","children":null,"spread":false},{"title":"StatHandler.java <span style='color:#111;'> 3.17KB </span>","children":null,"spread":false},{"title":"CSVExport.java <span style='color:#111;'> 2.40KB </span>","children":null,"spread":false},{"title":"Master.java <span style='color:#111;'> 42.97KB </span>","children":null,"spread":false},{"title":"Worker.java <span style='color:#111;'> 13.98KB </span>","children":null,"spread":false}],"spread":true}],"spread":true}],"spread":true}],"spread":true}],"spread":true}],"spread":true}],"spread":true},{"title":"2013segments <span style='color:#111;'> 8.64KB </span>","children":null,"spread":false},{"title":".gitignore <span style='color:#111;'> 53B </span>","children":null,"spread":false}],"spread":true}],"spread":true}]

评论信息

免责申明

【只为小站】的资源来自网友分享,仅供学习研究,请务必在下载后24小时内给予删除,不得用于其他任何用途,否则后果自负。基于互联网的特殊性,【只为小站】 无法对用户传输的作品、信息、内容的权属或合法性、合规性、真实性、科学性、完整权、有效性等进行实质审查;无论 【只为小站】 经营者是否已进行审查,用户均应自行承担因其传输的作品、信息、内容而可能或已经产生的侵权或权属纠纷等法律责任。
本站所有资源不代表本站的观点或立场,基于网友分享,根据中国法律《信息网络传播权保护条例》第二十二条之规定,若资源存在侵权或相关问题请联系本站客服人员,zhiweidada#qq.com,请把#换成@,本站将给予最大的支持与配合,做到及时反馈和处理。关于更多版权及免责申明参见 版权及免责申明