day19(ES)

2、Elasticsearch介绍

2.1、什么是搜索？

- 百度
- 京东

2.2、如果用数据库做搜索会怎样

- 天气
- 关键词
- 全表扫描

2.3、什么是全文搜索、倒排索引和lucene?

    1.老男孩教育
    2.老男孩教育linux学院
    3.老男孩教育python学院
    4.老男孩教育DBA
    5.老男孩教育oldzhang

老               1,2,3,4,5
老男孩         1,2,3,4,5
教育          1,2,3,4,5
学院          2,3
linux           2
python          3
DBA             4
oldzhang        5

2.4、Elasticsearch功能

1）分布式的搜索引擎和数据分析引擎

2）针对电商搜索

3）对海量数据进行近实时处理

2.5、特点

1）搜索
2）全文搜索
3）分析数据
4）处理海量数据PB
5）高可用高性能分布式搜索引擎数据库

2.6、应用场景

1）网页搜索
2）新闻搜索
3）商品标签
4）日志收集分析展示

3、安装部署

1）安装java
yum install -y java-1.8.0-openjdk.x86_64 

2） 下载安装软件
mkdir  -p /data/es_soft/
cd /data/es_soft/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.rpm
#yum install elasticsearch-6.6.0.rpm -y
rpm -ivh elasticsearch-6.6.0.rpm

3） 配置启动
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
systemctl status elasticsearch.service
systemctl is-active elasticsearch.service


4）检查是否启动成功
ps -ef|grep elastic
lsof -i:9200
curl localhost:9200


5）查看配置文件位置
 rpm -qc elasticsearch 
/etc/elasticsearch/elasticsearch.yml         ---- 主配置文件
/etc/elasticsearch/jvm.options           -----java虚拟机配置
/etc/init.d/elasticsearch         ----init.d启动脚本
/etc/sysconfig/elasticsearch         ----环境变量相关信息，不需要修改
/usr/lib/sysctl.d/elasticsearch.conf           ----环境变量相关
/usr/lib/systemd/system/elasticsearch.service            ----systemd启动脚本

6） 调整jvm配置
/etc/elasticsearch/jvm.options
-Xms1g
-Xmx1g


7）调整elasticsearch配置
/etc/elasticsearch/elasticsearch.yml
[root@db01 ~]# grep "^[a-z]" /etc/elasticsearch/elasticsearch.yml
node.name: node-1
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.0.0.51 
http.port: 9200

mkdir /data/elasticsearch/ -p 
chown -R elasticsearch:elasticsearch /data/elasticsearch/


systemctl edit elasticsearch
[Service]
LimitMEMLOCK=infinity


systemctl daemon-reload
systemctl start elasticsearch

4、名词概念

index       库
type       表
fields       字段
doc        行数据

5、交互方式

RESTful API
curl命令：
    最繁琐
    最复杂
    最容易出错
    不需要安装任何软件，只需要有curl命令

es-head插件
    查看数据方便
    操作相对容易
    需要node环境

kibana
    查看数据以及报表格式丰富
    操作很简单
    需要java环境和安装配置kibana

6、es-head安装部署

yum install nodejs npm openssl screen -y
node -v
npm  -v
npm install -g cnpm --registry=https://registry.npm.taobao.org
tar zxvf elasticsearch-head.tar.gz -C /opt/
cd /opt/elasticsearch-head
npm run start & 


修改es配置文件
vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true 
http.cors.allow-origin: "*"

重启es
systemctl restart elasticsearch
    
网页打开
10.0.0.51:9100
输入
http://10.0.0.51:9200

image.png

6.1、插入数据

curl -XPUT  'localhost:9200/vipinfo/user/2?pretty' -H 'Content-Type: application/json' -d' {
    "first_name": "Jane",
    "last_name" : "Smith",
    "age" : 32,
    "about" : "I like to collect rock albums", "interests": [ "music" ]
}'

curl -XPUT  'localhost:9200/vipinfo/user/3?pretty' -H 'Content-Type: application/json' -d' {
    "first_name": "Jane",
    "last_name" : "Smith",
    "age" : 32,
    "about" : "I like to collect rock albums", "interests": [ "music" ]
}'


curl -XPOST  'localhost:9200/vipinfo/user?pretty' -H 'Content-Type: application/json' -d' {
    "first_name": "Jane",
    "last_name" : "Smith",
    "age" : 32,
    "about" : "I like to collect rock albums", "interests": [ "music" ]
}'

curl -XPUT  'localhost:9200/vipinfo/user/2?pretty' -H 'Content-Type: application/json' -d' {
    "first_name": "zhang",
    "last_name" : "ya",
    "age" : 18,
    "about" : "I like to collect rock albums", "interests": [ "music" ]
}'



curl -XPUT  'localhost:9200/vipinfo/user/3?pretty' -H 'Content-Type: application/json' -d' {
    "first_name": "ya",
    "last_name" : "zhang",
    "age" : 28,
    "about" : "I like to collect rock albums", "interests": [ "music" ]
}'


curl -XPOST  'localhost:9200/vipinfo/user?pretty' -H 'Content-Type: application/json' -d' {
    "first_name": "老男孩Linux",
    "last_name" : "学院",
    "age" : 8,
    "about" : "I like to collect rock albums", "interests": [ "music" ]
}'

6.2、条件查询数据

1. 查询索引中所有的
curl -XGET 'localhost:9200/vipinfo/user/1?pretty'

2. 查询指定文档数据
curl -XGET 'localhost:9200/vipinfo/user/1?pretty'
curl -XGET 'localhost:9200/vipinfo/user/2?pretty’

3. 按条件查询文档数据
curl -XGET 'localhost:9200/vipinfo/user/_search?q=last_name:ya&pretty'
curl -XGET 'localhost:9200/vipinfo/user/_search?q=age:28&pretty'

 
4. 使用Query-string查询 
curl -XGET 'localhost:9200/vipinfo/user/_search?pretty' -H 'Content-Type: application/json' -d'           
{
  "query" : { 
    "match" : {
        "last_name" : "ya"
     }
  } 
}
'

5.更新数据的两种方式
1）PUT更新，需要填写完整的信息
curl -XPUT 'localhost:9200/vipinfo/user/4?pretty' -H 'Content-Type: application/json' -d'
{
    "first_name" : "zhang",
    "last_name": "ya",
    "age" : 27,
    "about" : "i love my cat", "interests": [ "sports", "music" ]
}


2）POST更新，只需要填写更改的信息
curl -XPOST 'localhost:9200/vipinfo/user/4?pretty' -H 'Content-Type: application/json' -d'
{
    "age" : 29
}
curl -XPOST  'localhost:9200/linux58/linux?pretty' -H 'Content-Type: application/json' -d' {
    "id" : 1 ,
    "first_name": "zhang",
    "last_name" : "ya",
    "age" : 18,
    "group" : 1
}'


curl -XPOST  'localhost:9200/linux58/linux?pretty' -H 'Content-Type: application/json' -d' {
    "id" : 2 ,
    "first_name": "li",
    "last_name" : "banzhang",
    "age" : 18,
    "group" : 10
}'

curl -XPOST  'localhost:9200/linux58/linux?pretty' -H 'Content-Type: application/json' -d' {
    "id" : 3 ,
    "first_name": "what",
    "last_name" : "ao",
    "age" : 17,
    "group" : 13
}'

6.3、删除数据

curl -XDELETE 'localhost:9200/vipinfo/user/1?pretty’
{
  "_index" : "vipinfo",
  "_type" : "user",
  "_id" : "1",
  "_version" : 2,
  "result" : "deleted",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 1,
  "_primary_term" : 2
}

6.4、删除索引

curl -XDELETE 'localhost:9200/vipinfo?pretty'

7、集群（所有节点执行）

1） 下载安装软件
mkdir  -p /data/es_soft/
cd /data/es_soft/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.rpm
#yum install elasticsearch-6.6.0.rpm -y
rpm -ivh elasticsearch-6.6.0.rpm


2）配置配置文件
[root@db01 /server/tools]# cat /etc/elasticsearch/elasticsearch.yml 
cluster.name: linux58
node.name: node-1
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.0.0.51,127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.51", "10.0.0.52"]
discovery.zen.minimum_master_nodes: 1
http.cors.enabled: true 
http.cors.allow-origin: "*"

[root@db02 /etc/elasticsearch]# cat /etc/elasticsearch/elasticsearch.yml 
cluster.name: linux58
node.name: node-2
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.0.0.52,127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.51", "10.0.0.52"]
discovery.zen.minimum_master_nodes: 1
http.cors.enabled: true 
http.cors.allow-origin: "*"

[root@db03 /etc/elasticsearch]# cat /etc/elasticsearch/elasticsearch.yml 
cluster.name: linux58
node.name: node-3
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.0.0.53,127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.51", "10.0.0.52", "10.0.0.53"]
discovery.zen.minimum_master_nodes: 1
http.cors.enabled: true 
http.cors.allow-origin: "*"

3）创建数据目录并授权
mkdir /data/elasticsearch/ -p 
chown -R elasticsearch:elasticsearch /data/elasticsearch/

4） 配置启动
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
systemctl status elasticsearch.service
systemctl is-active elasticsearch.service


5）检查是否启动成功
ps -ef|grep elastic
lsof -i:9200
curl localhost:9200

6）添加
systemctl edit elasticsearch
[Service]
LimitMEMLOCK=infinity

systemctl daemon-reload
systemctl restart elasticsearch
curl 10.0.0.51:9200
curl 10.0.0.52:9200
curl 10.0.0.53:9200

7.1、调整限制

限定条件
1.索引一旦创建以后，分片数就固定死了，不能调整了
2.但是副本数可以动态调整

curl -XPUT 'localhost:9200/index2/_settings?pretty' -H 'Content-Type: application/json' -d'         
{
"settings" : { 
  "number_of_replicas" : 2
 } 
}'

curl -XPUT 'localhost:9200/index2/_settings?pretty' -H 'Content-Type: application/json' -d'         
{
"settings" : { 
  "number_of_replicas" : 0
 } 
}'

7.2、查看节点数

curl -XGET 'http://localhost:9200/_cat/nodes?human&pretty'

增加index和设置分片、副本

curl -XPUT 'localhost:9200/index2?pretty' -H 'Content-Type: application/json' -d'       
{
    "settings" : { 
        "number_of_shards" : 3, 
        "number_of_replicas" : 1
    } 
}'

8、安装kibana

1）下载安装

2）修改配置文件
[root@db01 /data/soft]# grep "^[a-z]" /etc/kibana/kibana.yml 
server.port: 5601
server.host: "10.0.0.51"
server.name: "db01"
elasticsearch.hosts: ["http://localhost:9200"]

3）网站连接elasticsearch

image.png

image.png

8.1、安装分词器(1.版本要对应 2.所有的ES节点都需要安装 3.所有es节点都需要重启)

未使用分词器之前查询中文数据
curl -XPOST http://localhost:9200/index2/fulltext/1 -H 'Content-Type:application/json' -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'
curl -XPOST http://localhost:9200/index2/fulltext/2 -H 'Content-Type:application/json' -d'
{"content":"公安部：各地校车将享最高路权"}
'
curl -XPOST http://localhost:9200/index2/fulltext/3 -H 'Content-Type:application/json' -d'
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
'
curl -XPOST http://localhost:9200/index2/fulltext/4 -H 'Content-Type:application/json' -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'


curl -XPOST http://localhost:9200/index2/fulltext/_search?pretty  -H 'Content-Type:application/json' -d'
{
    "query" : { "match" : { "content" : "中国" }},
    "highlight" : {
        "pre_tags" : ["", ""],
        "post_tags" : ["", ""],
        "fields" : {
            "content" : {}
        }
    }
}

1）安装
cd /usr/share/elasticsearch/bin
./elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.6.0/elasticsearch-analysis-ik-6.6.0.zip

2）重启
systemctl restart elasticsearch

8.1.1、分词器测试

创建索引
curl -XPUT http://localhost:9200/index

创建映射
curl -XPOST http://localhost:9200/index/fulltext/_mapping -H 'Content-Type:application/json' -d'
{
        "properties": {
            "content": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_max_word"
            }
        }
}'


创建一些文档
curl -XPOST http://localhost:9200/index/fulltext/1 -H 'Content-Type:application/json' -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'
curl -XPOST http://localhost:9200/index/fulltext/2 -H 'Content-Type:application/json' -d'
{"content":"公安部：各地校车将享最高路权"}
'
curl -XPOST http://localhost:9200/index/fulltext/3 -H 'Content-Type:application/json' -d'
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
'
curl -XPOST http://localhost:9200/index/fulltext/4 -H 'Content-Type:application/json' -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'

查询
curl -XPOST http://localhost:9200/index/fulltext/_search?pretty  -H 'Content-Type:application/json' -d'
{
    "query" : { "match" : { "content" : "中国" }},
    "highlight" : {
        "pre_tags" : ["", ""],
        "post_tags" : ["", ""],
        "fields" : {
            "content" : {}
        }
    }
}
'