操作步骤：
一．数据采集：视频网站访问日志（编辑python脚本）
1.Python日志产生器开发URL ，ip信息，状态码，时间等
2.python日志产生器测试并将日志写入到文件中
3.通过定时调度工具每一分钟产生一批数据
4.使用flume实时收集日志信息
5.对接实时数据到kafka并输出到控制台
6.spark streaming对接kafka的数据进行消费
数据采集详情：项目其他\数据采集.docx
二．数据清洗：见项目
使用spark streaming完成数据清洗操作
三．数据分析：见项目
功能一：
统计到今天为止视频的访问量
yyyyMMdd courseId
使用数据库来进行存储我们的统计结果
sparkstreaming把统计结果写入到数据库里面
hbase表设计：create 'aqy_video_clickcount','info'
RowKey设计：day_videoid
功能二：
统计今天到现在为止从搜索引擎引流过来的实战课程的访问量
功能1+从搜索引擎引流过来的
HBase表设计：
create 'aqy_video_search_clickcount','info'
rowkey设计：根据业务需求20200529+search
功能具体实现步骤：
1.数据库访问dao层方法定义
2.hbase操作工具类开发
3.将spark streaming的处理结果写到hbase中
4.映射到hive数据仓库中
四．数据可视化：见数据可视化项目
2021-08-31 10:57:46
349KB
1
ApacheSpark设计与实现.pdf+ApacheSpark源码剖析.pdf+Spark原著中文版.pdf
2021-08-30 15:55:28
51.99MB
1
Data processing, implementing related algorithms, tuning, scaling up and finally deploying are some crucial steps in the process of optimising any application.
Spark is capable of handling large-scale batch and streaming data to figure out when to cache data in memory and processing them up to 100 times faster than Hadoop-based MapReduce.This means predictive analytics can be applied to streaming and batch to develop complete machine learning (ML) applications a lot quicker, making Spark an ideal candidate for large data-intensive applications.
This book focuses on design engineering and scalable solutions using ML with Spark. First, you will learn how to install Spark with all new features from the latest Spark 2.0 release. Moving on, you’ll explore important concepts such as advanced feature engineering with RDD and Datasets. After studying developing and deploying applications, you will see how to use external libraries with Spark.
2021-08-27 14:51:14
11.47MB
1