Python 爬虫框架 Scrapy 入门与实践之爬取豆瓣电影 Top250 榜单

douban
|-- __init__.py
|-- __pycache__
|   |-- __init__.cpython-37.pyc
|   |-- items.cpython-37.pyc
|   |-- pipelines.cpython-37.pyc
|   `-- settings.cpython-37.pyc
|-- items.py
|-- middlewares.py
|-- pipelines.py
|-- settings.py
`-- spiders
    |-- __init__.py
    |-- __pycache__
    |   |-- __init__.cpython-37.pyc
    |   `-- douban_spider.cpython-37.pyc
    `-- douban_spider.py

3 directories, 13 files

<li>
            <div class="item">
                <div class="pic">
                    <em class="">1</em>
                    <a href="https://movie.douban.com/subject/1292052/">
                        <img width="100" alt=" 肖申克的救赎 " src="https://img3.doubanio.com/view/photo/s_ratio_poster/public/p480747492.webp" class="">
                    </a>
                </div>
                <div class="info">
                    <div class="hd">
                        <a href="https://movie.douban.com/subject/1292052/" class="">
                            <span class="title"> 肖申克的救赎 < span class="tag"></span>
                                    <span class="title"> / The Shawshank Redemption</span>
                                <span class="other"> /  月黑高飞 (港)  /  刺激 1995 (台)</span>
                        </a>


                            <span class="playable">[可播放]</span>
                    </div>
                    <div class="bd">
                        <p class="">
                            导演：弗兰克・德拉邦特 Frank Darabont    主演：蒂姆・罗宾斯 Tim Robbins /...<br>
                            1994 /  美国 & amp;nbsp;/  犯罪 剧情 
                        </p>

                        
                        <div class="star">
                                <span class="rating5-t"></span>
                                <span class="rating_num" property="v:average">9.6</span>
                                <span property="v:best" content="10.0"></span>
                                <span>1338863 人评价 < span class="tag"></span>
                        </div>

                            <p class="quote">
                                <span class="inq"> 希望让人自由。</span>
                            </p>
                    </div>
                </div>
            </div>
        </li>

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy

class DoubanItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    serial_number = scrapy.Field()
    movie_name = scrapy.Field()
    introduce = scrapy.Field()
    star = scrapy.Field()
    evaluate = scrapy.Field()
    describe = scrapy.Field()

# -*- coding: utf-8 -*-

# Scrapy settings for douban project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'douban'

SPIDER_MODULES = ['douban.spiders']
NEWSPIDER_MODULE = 'douban.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 0.5

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'douban.pipelines.DoubanPipeline': 300,
}

mongo_host = '127.0.0.1'
mongo_port = 27017
mongo_db_name = 'douban'
mongo_db_collection = 'douban_movie'

# -*- coding: utf-8 -*-
import scrapy
from douban.items import DoubanItem

class DoubanSpiderSpider(scrapy.Spider):
    name = 'douban_spider'
    allowed_domains = ['movie.douban.com']
    start_urls = ['https://movie.douban.com/top250']

    def parse(self, response):
        movie_list = response.xpath("//div[@class='article']//ol[@class='grid_view']/li")
        for i_item in movie_list:
            douban_item = DoubanItem()
            douban_item['serial_number'] = i_item.xpath(".//div[@class='item']//em/text()").extract_first()
            douban_item['movie_name'] = i_item.xpath(".//div[@class='info']/div[@class='hd']/a/span[1]/text()").extract_first()
            content = i_item.xpath(".//div[@class='info']//div[@class='bd']/p[1]/text()").extract()
            for i_content in content:
                content_s = "".join(i_content.split())
                douban_item['introduce'] = content_s
            douban_item['star'] =  i_item.xpath(".//span[@class='rating_num']/text()").extract_first()
            douban_item['evaluate'] = i_item.xpath(".//div[@class='star']//span[4]/text()").extract_first()
            douban_item['describe'] = i_item.xpath(".//p[@class='quote']//span/text()").extract_first()
            yield douban_item
        next_link = response.xpath("//span[@class='next']/link/@href").extract()
        if next_link:
            next_link = next_link[0]
            yield scrapy.Request("https://movie.douban.com/top250"+next_link,callback=self.parse)

(venv)  wangao@wangao-MAC  ~/Documents/git/wangao/python3/scrapy/douban  python -V
Python 3.7.1
(venv)  wangao@wangao-MAC  ~/Documents/git/wangao/python3/scrapy/douban  scrapy crawl douban_spider -o douban.csv
2019-03-02 15:37:56 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: douban)
2019-03-02 15:37:56 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.19.0, Twisted 18.9.0, Python 3.7.1 (default, Nov  6 2018, 18:46:03) - [Clang 10.0.0 (clang-1000.11.45.5)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0j  20 Nov 2018), cryptography 2.4.2, Platform Darwin-18.2.0-x86_64-i386-64bit
2019-03-02 15:37:56 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'douban', 'DOWNLOAD_DELAY': 0.5, 'FEED_FORMAT': 'csv', 'FEED_URI': 'douban.csv', 'NEWSPIDER_MODULE': 'douban.spiders', 'SPIDER_MODULES': ['douban.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36'}
2019-03-02 15:37:56 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2019-03-02 15:37:56 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-03-02 15:37:56 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-03-02 15:37:56 [scrapy.middleware] INFO: Enabled item pipelines:
['douban.pipelines.DoubanPipeline']
2019-03-02 15:37:56 [scrapy.core.engine] INFO: Spider opened
2019-03-02 15:37:56 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-03-02 15:37:56 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
^C2019-03-02 15:37:56 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force 
2019-03-02 15:37:56 [scrapy.core.engine] INFO: Closing spider (shutdown)
2019-03-02 15:37:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://movie.douban.com/top250> (referer: None)
2019-03-02 15:37:58 [scrapy.core.scraper] DEBUG: Scraped from <200 https://movie.douban.com/top250>
{'describe': '希望让人自由。',
 'evaluate': '1338863 人评价',
 'introduce': '1994 / 美国 / 犯罪剧情',
 'movie_name': '肖申克的救赎',
 'serial_number': '1',
 'star': '9.6'}
2019-03-02 15:37:58 [scrapy.core.scraper] DEBUG: Scraped from <200 https://movie.douban.com/top250>
{'describe': '风华绝代。',
 'evaluate': '989216 人评价',
 'introduce': '1993 / 中国大陆香港 / 剧情爱情同性',
 'movie_name': '霸王别姬',
 'serial_number': '2',
 'star': '9.6'}
2019-03-02 15:37:58 [scrapy.core.scraper] DEBUG: Scraped from <200 https://movie.douban.com/top250>
{'describe': '怪蜀黍和小萝莉不得不说的故事。',
 'evaluate': '1225142 人评价',
 'introduce': '1994 / 法国 / 剧情动作犯罪',
 'movie_name': '这个杀手不太冷',
 'serial_number': '3',
 'star': '9.4'}
2019-03-02 15:37:58 [scrapy.core.scraper] DEBUG: Scraped from <200 https://movie.douban.com/top250>
{'describe': '一部美国近现代史。',
 'evaluate': '1054616 人评价',
 'introduce': '1994 / 美国 / 剧情爱情',
 'movie_name': '阿甘正传',
 'serial_number': '4',
 'star': '9.4'}

describe,evaluate,introduce,movie_name,serial_number,star
 希望让人自由。,1338863 人评价, 1994 / 美国 / 犯罪剧情, 肖申克的救赎, 1,9.6
 风华绝代。,989216 人评价, 1993 / 中国大陆香港 / 剧情爱情同性, 霸王别姬, 2,9.6
 怪蜀黍和小萝莉不得不说的故事。,1224747 人评价, 1994 / 法国 / 剧情动作犯罪, 这个杀手不太冷, 3,9.4
 一部美国近现代史。,1054616 人评价, 1994 / 美国 / 剧情爱情, 阿甘正传, 4,9.4
 最美的谎言。,617064 人评价, 1997 / 意大利 / 剧情喜剧爱情战争, 美丽人生, 5,9.5
 失去的才是永恒的。 ,993900 人评价, 1997 / 美国 / 剧情爱情灾难, 泰坦尼克号, 6,9.3
 最好的宫崎骏，最好的久石让。 ,981959 人评价, 2001 / 日本 / 剧情动画奇幻, 千与千寻, 7,9.3
 拯救一个人，就是拯救整个世界。,550815 人评价, 1993 / 美国 / 剧情历史战争, 辛德勒的名单, 8,9.5
 诺兰给了我们一场无法盗取的梦。,1067442 人评价, 2010 / 美国英国 / 剧情科幻悬疑冒险, 盗梦空间, 9,9.3