利用python从高德地图中爬取到的深圳地铁站信息，包含经纬度信息，抓取的内容已经和高德地图地铁站的数据对照过了，没有遗漏，大家可放心使用。
2021-07-12 16:43:46
243KB
1
主要介绍了Python实现爬取网页中动态加载的数据，文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值，需要的朋友们下面随着小编来一起学习学习吧
2021-07-12 10:57:27
1.72MB
1
主要介绍了python实现爬取百度图片的方法,涉及Python基于requests、urllib等模块的百度图片抓取相关操作技巧,需要的朋友可以参考下
2021-07-11 09:31:35
35KB
1
基于Python Scrapy实现的网易云音乐music163数据爬取爬虫系统 含全部源代码
基于Scrapy框架的网易云音乐爬虫，大致爬虫流程如下：
- 以歌手页为索引页，抓取到全部歌手；
- 从全部歌手页抓取到全部专辑；
- 通过所有专辑抓取到所有歌曲；
- 最后抓取歌曲的精彩评论。
数据保存到`Mongodb`数据库，保存歌曲的歌手，歌名，专辑，和热评的作者，赞数，以及作者头像url。
抓取评论者的头像url，是因为如果大家喜欢，可以将他做web端。
### 运行：
```
$ scrapy crawl music
```
#!/usr/bin/python
#-*-coding:utf-8-*-
import time
from pprint import pprint
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.http import Request
from woaidu_crawler.items import WoaiduCrawlerItem
from woaidu_crawler.utils.select_result import list_first_item,strip_null,deduplication,clean_url
class WoaiduSpider(BaseSpider):
name = "woaidu"
start_urls = (
'http://www.woaidu.org/sitemap_1.html',
)
def parse(self,response):
response_selector = HtmlXPathSelector(response)
next_link = list_first_item(response_selector.select(u'//div[@class="k2"]/div/a[text()="下一页"]/@href').extract())
if next_link:
next_link = clean_url(response.url,next_link,response.encoding)
yield Request(url=next_link, callback=self.parse)
for detail_link in response_selector.select(u'//div[contains(@class,"sousuolist")]/a/@href').extract():
if detail_link:
detail_link = clean_url(response.url,detail_link,response.encoding)
yield Request(url=detail_link, callback=self.parse_detail)
def parse_detail(self, response):
woaidu_item = WoaiduCrawlerItem()
response_selector = HtmlXPathSelector(response)
woaidu_item['book_name'] = list_first_item(response_selector.select('//div[@class="zizida"][1]/text()').extract())
woaidu_item['author'] = [list_first_item(response_selector.select('//div[@class="xiaoxiao"][1]/text()').extract())[5:].strip(),]
woaidu_item['book_description'] = list_first_item(response_selector.select('//div[@class="lili"][1]/text()').extract()).strip()
woaidu_item['book_covor_image_url'] = list
2021-07-10 21:02:57
20KB
Python采集数据存储到Excel
Python采集数据存储到MySQL
MySQL从0安装到穿件数据库和数据表
使用Python链接MySQL
用Python发邮件、定时发邮件
2021-07-10 21:02:55
4MB
基于Python Scrapy实现的爬取豆瓣读书9分榜单的书籍数据采集爬虫系统 含数据集和全部源代码
# -*- coding: utf-8 -*-
import scrapy
import re
from doubanbook.items import DoubanbookItem
class DbbookSpider(scrapy.Spider):
name = "dbbook"
# allowed_domains = ["https://www.douban.com/doulist/1264675/"]
start_urls = (
'https://www.douban.com/doulist/1264675/',
)
URL = 'https://www.douban.com/doulist/1264675/?start=PAGE&sort=seq&sub_type='
def parse(self, response):
# print response.body
item = DoubanbookItem()
selector = scrapy.Selector(response)
books = selector.xpath('//div[@class="bd doulist-subject"]')
for each in books:
title = each.xpath('div[@class="title"]/a/text()').extract()[0]
rate = each.xpath('div[@class="rating"]/span[@class="rating_nums"]/text()').extract()[0]
author = re.search('(.*?)
2021-07-10 17:02:47
19KB
基于Python Scrapy实现的腾讯招聘职位数据爬取爬虫系统 含结果数据集和全部源代码
# -*- coding: utf-8 -*-
import scrapy
from Tencent.items import TencentItem
class TencentpostionSpider(scrapy.Spider):
name = 'tencentPosition'
allowed_domains = ['tencent.com']
url = "http://hr.tencent.com/position.php?&start="
offset = 0
# 起始url
start_urls = [url + str(offset)]
def parse(self, response):
for each in response.xpath("//tr[@class='even'] | //tr[@class='odd']"):
# 初始化模型对象
item = TencentItem()
# 职位名称
item['positionname'] = each.xpath("./td[1]/a/text()").extract()[0]
# 详情连接
item['positionlink'] = each.xpath("./td[1]/a/@href").extract()[0]
# 职位类别
item['positionType'] = each.xpath("./td[2]/text()").extract()[0]
# 招聘人数
item['peopleNum'] = each.xpath("./td[3]/text()").extract()[0]
# 工作地点
item['workLocation'] = each.xpath("./td[4]/text()").extract()[0]
# 发布时间
item['publishTime'] = each.xpath("./td[5]/text()").extract()[0]
yield item
if self.offset < 1680:
self.offset += 10
# 每次处理完一页的数据之后，重新发送下一页页面请求
# self.offset自增10，同时拼接为新的url，并调用回调函数self.parse处理Response
yield scrapy.Request(self.url + str(self.offset), callback=self.parse)
2021-07-10 17:02:45
15KB