CXH728

hive-3.1.3部署文档

提前准备

一个正常运行的hadoop集群
java环境
hive安装包
下载地址：https://archive.apache.org/dist/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
mysql安装包
1、内嵌模式
由于内嵌模式使用场景太少（基本不用），所以仅练习安装查看基础功能

[root@master ~]# tar xf apache-hive-3.1.3-bin.tar.gz -C /opt/
[root@master ~]# cd /opt/apache-hive-3.1.3-bin/
[root@master apache-hive-3.1.3-bin]# cd bin
[root@master bin]# ./schematool -dbType derby -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-3.1.3-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:derby:;databaseName=metastore_db;create=true
Metastore Connection Driver :    org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User:       APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
[root@master bin]# ./hive
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/jdk1.8.0_112/bin:/usr/local/hadoop-3.3.6/bin:/usr/local/hadoop-3.3.6/sbin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-3.1.3-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 4fc56454-1c46-4f18-9af3-cde0d53fecf2

Logging initialized using configuration in jar:file:/opt/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = 26336a34-9d27-4fbb-9314-c6137c9e7d03
hive> show databases;
OK
default
Time taken: 0.477 seconds, Fetched: 1 row(s)
hive> quit;
[root@master bin]#

2、本地模式

2.1、安装数据库——mysql（和mariadb任选一个即可）

1、安装mysql 5.7.18

2、上传mysql的rpm安装包

mysql-community-client-5.7.18-1.el6.x86_64.rpm

mysql-community-libs-5.7.18-1.el6.x86_64.rpm

mysql-community-common-5.7.18-1.el6.x86_64.rpm

mysql-community-server-5.7.18-1.el6.x86_64.rpm

mysql-connector-java-5.1.32-bin.jar

3、到上传目录下执行

rpm -ivh mysql-community-*

报错如下

需要执行如下命令

yum remove mariadb-libs

yum -y install net-tools

[root@master mysql]# ls
mysql-community-client-5.7.18-1.el6.x86_64.rpm  mysql-community-libs-5.7.18-1.el6.x86_64.rpm    mysql-connector-java-5.1.32-bin.jar
mysql-community-common-5.7.18-1.el6.x86_64.rpm  mysql-community-server-5.7.18-1.el6.x86_64.rpm
[root@master mysql]# pwd
/root/mysql
[root@master mysql]# rpm -ivh mysql-community-*
警告：mysql-community-client-5.7.18-1.el6.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
错误：依赖检测失败：
        mariadb-libs 被 mysql-community-libs-5.7.18-1.el6.x86_64 取代
        net-tools 被 mysql-community-server-5.7.18-1.el6.x86_64 需要
[root@master mysql]# mount /dev/sr0 /media/
mount: /dev/sr0 写保护，将以只读方式挂载
[root@master mysql]# yum remove mariadb-libs
已加载插件：fastestmirror
正在解决依赖关系
--> 正在检查事务
---> 软件包 mariadb-libs.x86_64.1.5.5.68-1.el7 将被 删除
--> 正在处理依赖关系 libmysqlclient.so.18()(64bit)，它被软件包 2:postfix-2.10.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libmysqlclient.so.18(libmysqlclient_18)(64bit)，它被软件包 2:postfix-2.10.1-9.el7.x86_64 需要
--> 正在检查事务
---> 软件包 postfix.x86_64.2.2.10.1-9.el7 将被 删除
--> 解决依赖关系完成

依赖关系解决

==============================================================================================================================================================================================================
 Package                                            架构                                         版本                                                   源                                               大小
==============================================================================================================================================================================================================
正在删除:
 mariadb-libs                                       x86_64                                       1:5.5.68-1.el7                                         @anaconda                                       4.4 M
为依赖而移除:
 postfix                                            x86_64                                       2:2.10.1-9.el7                                         @anaconda                                        12 M

事务概要
==============================================================================================================================================================================================================
移除  1 软件包 (+1 依赖软件包)

安装大小：17 M
是否继续？[y/N]：y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在删除    : 2:postfix-2.10.1-9.el7.x86_64                                                                                                                                                             1/2
  正在删除    : 1:mariadb-libs-5.5.68-1.el7.x86_64                                                                                                                                                        2/2
  验证中      : 1:mariadb-libs-5.5.68-1.el7.x86_64                                                                                                                                                        1/2
  验证中      : 2:postfix-2.10.1-9.el7.x86_64                                                                                                                                                             2/2

删除:
  mariadb-libs.x86_64 1:5.5.68-1.el7

作为依赖被删除:
  postfix.x86_64 2:2.10.1-9.el7

完毕！

[root@master yum.repos.d]# yum -y install  net-tools
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
正在解决依赖关系
--> 正在检查事务
---> 软件包 net-tools.x86_64.0.2.0-0.25.20131004git.el7 将被 安装
--> 解决依赖关系完成

依赖关系解决

==============================================================================================================================================================================================================
 Package                                        架构                                        版本                                                          源                                             大小
==============================================================================================================================================================================================================
正在安装:
 net-tools                                      x86_64                                      2.0-0.25.20131004git.el7                                      c7-media                                      306 k

事务概要
==============================================================================================================================================================================================================
安装  1 软件包

总下载量：306 k
安装大小：917 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在安装    : net-tools-2.0-0.25.20131004git.el7.x86_64                                                                                                                                                 1/1
  验证中      : net-tools-2.0-0.25.20131004git.el7.x86_64                                                                                                                                                 1/1

已安装:
  net-tools.x86_64 0:2.0-0.25.20131004git.el7

完毕！
[root@master mysql]# rpm -ivh mysql-community-*
警告：mysql-community-client-5.7.18-1.el6.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:mysql-community-common-5.7.18-1.e################################# [ 25%]
   2:mysql-community-libs-5.7.18-1.el6################################# [ 50%]
   3:mysql-community-client-5.7.18-1.e################################# [ 75%]
   4:mysql-community-server-5.7.18-1.e^[a################################# [100%]
[root@master mysql]#

4、启动数据库

#启动数据库
[root@master mysql]# systemctl start mysqld
#查看数据库初始化密码
[root@master mysql]# cat /var/log/mysqld.log |grep root
2024-10-28T09:30:24.600045Z 1 [Note] A temporary password is generated for root@localhost: g;jryD7tVd*w
#登录mysql数据库
[root@master mysql]# mysql -uroot -p'g;jryD7tVd*w'
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.18
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
#修改初始化密码（必须符合指定的复杂程度规则）
mysql> alter user 'root'@'localhost' identified by 'Class_123';
Query OK, 0 rows affected (0.00 sec)
#调整密码规则等级为0
mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)
#调整密码长度
mysql> set global validate_password_length=6;
Query OK, 0 rows affected (0.00 sec)
#设置新的密码
mysql> set password for 'root'@'localhost'=password('123456');
Query OK, 0 rows affected, 1 warning (0.01 sec)
#赋权root用户可以远程登陆
mysql> grant all privileges on *.* to 'root'@'%' identified by '123456';
Query OK, 0 rows affected, 1 warning (0.00 sec)
#刷新权限
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye
[root@master mysql]#

2.2、安装数据库——mariadb（和mysql任选一个即可）

配置本地yum源
安装mariadb和连接数据库驱动

yum -y install mariadb-server mysql-connector-java

初始化数据库

##启动mariadb
[root@master ~]# systemctl start mariadb
##初始化数据库
[root@master ~]# mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

增加mysql的jar包

cp /usr/share/java/mysql-connector-java.jar /opt/hive/lib/

2.3、配置hive环境

1. 上传hive安装包、解压到指定位置

[root@master ~]# tar xf apache-hive-3.1.3-bin.tar.gz -C /opt/
[root@master ~]# cd /opt/
[root@master ~]# mv apache-hive-3.1.3-bin hive

2. 配置环境变量

vi /etc/profile

export HIVE_HOME=/opt/hive
export PATH=$PATH:$HIVE_HOME/bin

source /etc/profile

3. 修改hive配置

vi /opt/hive/conf/hive-site.xml


  
    hive.metastore.warehouse.dir
    /user/hive/warehouse
  
  
    javax.jdo.option.ConnectionURL
    jdbc:mysql://192.168.128.130:3306/hive?createDatabaseIfNotExist=true
  
  
    javax.jdo.option.ConnectionDriverName
    com.mysql.jdbc.Driver
  
  
    javax.jdo.option.ConnectionUserName
    root
  
  
    javax.jdo.option.ConnectionPassword
    123456
  
  
    datanucleus.schema.autoCreateAll
    true
  
  
    hive.metastore.schema.verification
    false

4. 增加mysql的jar包

使用mysql的jar包

cp mysql-connector-java-5.1.32-bin.jar /opt/hive/lib/

使用mariadb的jar包

cp /usr/share/java/mysql-connector-java.jar /opt/hive/lib/

5. 初始化元数据库

[root@master conf]# schematool -dbType mysql -initSchema
Metastore connection URL:        jdbc:mysql://192.168.128.130:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:       root
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.mysql.sql
Initialization script completed
schemaTool completed

2.4、登录验证

[root@master conf]# hive
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/jdk1.8.0_112/bin:/usr/local/hadoop-3.3.6/bin:/usr/local/hadoop-3.3.6/sbin:/opt/hive/bin:/root/bin)
Hive Session ID = d4f5afa7-fc5d-4985-9889-a2aa8842d1ae
Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = 440763e5-41d1-4d8e-8c70-aa007f51c429
hive> show databases;
OK
default
Time taken: 1.63 seconds, Fetched: 1 row(s)
hive> create database test1;
OK
Time taken: 0.193 seconds
hive> show databases;
OK
default
test1
Time taken: 0.05 seconds, Fetched: 2 row(s)
hive>

3、远程模式访问

我要在192.168.128.131上远程访问上述的192.168.128.130服务器上的hive服务。

3.1、130节点上需要的操作

[root@master conf]# hive --service metastore
2024-10-28 21:10:55: Starting Hive Metastore Server

##如果想后台运行可以写成如下命令
hive --service metastore &

3.2、131上需要配置