Python-scrapy学习(四)

scrapy学习到此告一段落，下图为一个项目的框架图。

在Python2-Scrapy学习(三)学习如何将数据进行存储，接下来学习如何使用selenium解析JS、邮件通知

selenium解析JS

在爬取seebug直接请求无法访问到数据页面，发现访问seebug流程为：访问seebug-解析js-赋值cookie字段-再次访问-成功获取数据。

查阅资料得知scrapy可以使用Splash进行JavaScript渲染，但是根据官网信息显示得与docker进行配合。后面想起来可以通过selenium进行解析，但是scrapy-selenium需要python>=3.6。但是，我使用的是2.7，因此，我直接使用selenium进行解析。Ps：在CentOS6下，无法使用Chrome进行。

安装

直接使用pip安装selenium即可，phantomjs已经暂停项目，新版的selenium已经不支持phantomjs。报错信息：UserWarning: Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead。因此，根据官方文档需要下载相应的浏览器内核（Chrome、Edge、Firefox、Safari）。我下载了Chrome Driver，并将chromedriver文件移动到/usr/local/bin目录下。

pip install selenium

使用middlewares

安装完成之后，我想的是通过scary项目的方式进行调用（例如scrapy-selenium），所以并不会采取直接在爬虫里面selenium的调用。通过阅读scrapy文档发现，可以在middleware.py中将selenium使用封装成类，通过setting.py或者爬虫custom_settings进行调用。

middlewares.py

from selenium import webdriver
from scrapy.http import HtmlResponse
from scrapy.exceptions import IgnoreRequest
from selenium.webdriver.chrome.options import Options


class SeleniumMiddleware(object):
    def __init__(self):
        self.chrome_options = Options()
        self.chrome_options.add_argument('--headless')
        self.chrome_options.add_argument('--disable-gpu')
        self.driver = webdriver.Chrome(chrome_options=self.chrome_options)
        # self.driver = webdriver.Chrome()

    def process_request(self, request, spider):
        self.driver.get(request.url)
        time.sleep(3)
        try:
            body = self.driver.page_source
            return HtmlResponse(self.driver.current_url, body=body, encoding='utf-8', request=request)
        except Exception as e:
            # Timeout on WebDriverWait
            logging.error(e)
            raise IgnoreRequest

seebug.py

import scrapy
import time

from zjyd.items import ZjydItem
from scrapy.utils.project import get_project_settings

settings = get_project_settings()


class SeebugSpider(scrapy.Spider):
    name = 'seebug'
    allowed_domains = ['seebug.org']
    custom_settings = {
        'DOWNLOADER_MIDDLEWARES': {
            'zjyd.pipelines.SeleniumMiddleware': 723,
        },
    }

    def start_requests(self):
        keywords = list(settings['KEYWORDS'])
        for i in keywords:
            yield scrapy.Request(url=('https://www.seebug.org/search/?keywords=%s&category=&page=1' % str(i)), callback=self.parse)

    def parse(self, response):
        item = ZjydItem()
        if response.xpath("//table[@class='table sebug-table table-vul-list']/tbody/tr"):
            page = int(response.url[response.url.find('page=') + 5]) + 1
            next_page = response.url[:response.url.find('page=') + 5] + str(page)
        else:
            next_page = None
        for i in response.xpath("//table[@class='table sebug-table table-vul-list']/tbody/tr"):
            if i.xpath("td[@class='text-center datetime hidden-sm hidden-xs']/text()").extract_first().strip()[:-6] == time.strftime("%Y-%m-%d").decode('utf-8'):
                item['source'] = 'seebug'
                item['title'] = i.xpath("td[@class='vul-title-wrapper']/a[@class='vul-title']/text()").extract_first()
                item['time'] = i.xpath("td[@class='text-center datetime hidden-sm hidden-xs']/text()").extract_first().strip()
                item['url'] = u'https://www.seebug.org' + i.xpath("td[@class='vul-title-wrapper']/a[@class='vul-title']/href").extract_first()
                item['content'] = i.xpath("td[@class='vul-title-wrapper']/a[@class='vul-title']/text()").extract_first()
                item['author'] = u'null'
            else:
                next_page = None
                continue
        if item:
            yield item
        else:
            self.log('%s is none!' % (response.url))
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)
        self.log("seebug sprider:%s" % (response.url))

Or setting.py

DOWNLOADER_MIDDLEWARES = {
   'zjyd.middlewares.ZjydDownloaderMiddleware': 543,
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
    'scrapy_proxies.RandomProxy': 100,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}

通过上面的配置就可以将seebug的内容解析出来，并进行数据存储。

项目整合

现在项目上有freebuf、seebug等爬虫，通过scrapy list可以查看项目下一共有多少个爬虫。

爬虫整合

项目下每个爬虫为一个文件，并非所有爬虫都在一个文件、按照类进行划分。所以需要一个脚本，运行所有的爬虫。通过官方文档可以看到提供了API接口的方式运行爬虫。这里我采用将所有爬虫通过process.crawl进行运行。

from scrapy.crawler import CrawlerProcess
# from scrapy import spiderloader
from scrapy.utils.project import get_project_settings

settings = CrawlerProcess(get_project_settings())

def main():
    process = CrawlerProcess(settings)
    process.crawl('freebuf', domain='freebuf.com')
    process.crawl('vulbox', domain='vulbox.com')
    process.crawl('anquanke', domain='anquanke.com')
    process.crawl('bugbank', domain='bugbank.com')
    process.crawl('seebug', domain='seebug.com')
    process.crawl('cnvd', domain='cnvd.org.cn')
    # spider_loader = spiderloader.SpiderLoader.from_settings(settings)
    # spiders = spider_loader.list()
    # classes = [spider_loader.load(name) for name in spiders]
    # for i in classes:
    #     process.crawl(i)
    
    process.start()

通过上述方法就可以运行项目下的freebuf、vulbox等爬虫。

邮件通知

现在爬虫所获的的数据都会存储在MongoDB数据库中，通过读取数据库中当前日期的数据，进行邮件通知即可。

# -*- coding: utf-8 -*-

# import os
import time
import pymongo
import scrapy
import smtplib
from email.header import Header
from email.mime.text import MIMEText
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from scrapy import spiderloader

settings = get_project_settings()


def send_mail():
    mongo_client = pymongo.MongoClient(host=settings["MONGODB_HOST"], port=settings["MONGODB_PORT"])
    mongo_db = mongo_client[settings["MONGODB_DBNAME"]]
    mongo_query = {'time': time.strftime("%Y-%m-%d").decode('utf-8')}
    result = "邮件更新提醒:\n"
    
    spider_loader = spiderloader.SpiderLoader.from_settings(settings)
    spiders = spider_loader.list()
    for i in spiders:
        mongo_col = mongo_db[i]
        if mongo_col.find(mongo_query).sort("ts",pymongo.ASCENDING).count() != 0:
            result += '%s 有更新,请注意查收!\n' % (i)
        else:
            result += '%s无更新!\n' % (i)

    sender = settings["MAIL_SENDER"]
    receivers = settings["MAIL_RECEIVERS"]

    message = MIMEText(result,'plain','utf-8')
    message['From'] = Header("hywell", 'utf-8')
    message['To'] = receivers
    subject = "信息收集爬虫"
    message['Subject'] = Header(subject, 'utf-8')

    smtpObj = smtplib.SMTP() 
    smtpObj.connect(settings["MAIL_HOST"], 25)
    smtpObj.login(settings["MAIL_USER"], settings["MAIL_PASSWORD"]) 
    smtpObj.sendmail(sender, receivers, message.as_string())

    mongo_client.close()

def main():
    process = CrawlerProcess(settings)
    spider_loader = spiderloader.SpiderLoader.from_settings(settings)
    spiders = spider_loader.list()
    classes = [spider_loader.load(name) for name in spiders]
    for i in classes:
        process.crawl(i)

    process.start()
    send_mail()

if __name__ == "__main__":
    main()

总结

1. python2.7越来越“老旧了”，新的模块很多都已经不支持python2.7，后续还是得开始使用python3进行编程。毕竟，python最主要的就是可以import module；
2. 在进行邮件发送的时候，要注意邮箱服务器的设置，例如：密码是否需要设置为独立密码、发件人信息是否有与账号匹配校验等；
3. 由于Chrome最新版本不支持CentOS6，我通过种种方式安装上了Chrome以及成功运行ChromeDriver，但是在运行时会出现卡在开始连接[urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): 127.0.0.1:1269，后面改用Firefox。

完整代码

完整代码已经上传到我的GiHub。如果有兴趣，不妨移步到Github上一观！Code。