【Spark】岗位数据分析

配置详解

本文是使用Scrapy来获取数据，再使用Spark来进行分析
各版本如下

软件 / 库	版本
Pycharm社区版	2021.3.3
Python	3.8
Pandas	1.4.1
Numpy	1.22.3
PyMySQL	1.0.2
scrapy	2.4.1
MySQL	5.7
Spark	2.0.0

由于社区原因，Scrapy部分不会讲解，只贴代码。

跳转顶部

---

数据获取

MySQL建表语句

DROP TABLE IF EXISTS `job`;
CREATE TABLE `job` (
  `address` varchar(255) DEFAULT NULL,
  `company` varchar(255) DEFAULT NULL,
  `edu` varchar(255) DEFAULT NULL,
  `jobName` varchar(255) DEFAULT NULL,
  `salary` varchar(255) DEFAULT NULL,
  `size` varchar(255) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

跳转顶部

---

Settings文件

BOT_NAME = 'job'

SPIDER_MODULES = ['job.spiders']
NEWSPIDER_MODULE = 'job.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False
LOG_LEVEL = 'ERROR'

ITEM_PIPELINES = {
   
    'job.pipelines.JobPipeline': 300,
}

跳转顶部

---

主项目代码

import scrapy
import re
import lxml.html
from job.items import JobItem


class TestSpider(scrapy.Spider):
    name = 'Test'
    # allowed_domains = ['baidu.com']
    start_urls = ['https://job001.cn/jobs?keyType=0&keyWord=java']
    basic_url = 'https://job001.cn/jobs?keyType=0&keyWord=java&jobTypeId=&jobType=%E8%81%8C%E4%BD%8D%E7%B1%BB%E5%9E%8B&industry=&industryname=%E8%A1%8C%E4%B8%9A%E7%B1%BB%E5%9E%8B&workId=&workPlace=&salary=&entType=&experience=&education=&entSize=&benefits=&reftime=&workTypeId=&sortField=&pageNo=3&curItem='

    def parse(self, response):
        html = response.text
        selector = lxml.html.fromstring(html)
        maxPage = selector.xpath('//*[@id="pagediv"]/a[11]/text()')[0]
        for num in range(int(maxPage)):
            url = self.basic_url.replace('pageNo=3', f'pageNo={
     num <