一、hadoop2.6.4 hive-2.1.1 presto-server-0.172.tar.gz, jdk1.8u121
二、配置
- hadoop搭建分布式集群
- hive搭建配置
- 安裝
將文件上傳到安裝目錄下/usr/local/
tar -zxvf tar -zxvf apache-hive-2.1.1-bin.tar.gz
mv tar -zxvf apache-hive-2.1.1 hive-2.1
- 配置環境變量，編輯/etc/profile
export HIVE_HOME=/usr/local/hive-2.1
export PATH=$HIVE_HOME/bin:$PATH
執行 source /etc/profile
- hive配置 .../conf/
- 修改 hive.env.sh 增加
export JAVA_HOME=/usr/local/jdk
export HADOOP_HOME=/usr/local/hadoop2
export HIVE_HOME=/usr/local/hive-1.2
* 修改log4j文件
cp hive-log4j.properties.template hive-log4j.properties
將EventCounter修改成org.apache.hadoop.log.metrics.EventCounter
#log4j.appender.EventCounter=org.apache.hadoop.hive.shims.HiveEventCounter
log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter
- 配置hive-site.xml
touch hive-site.xml
寫入：
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name> <value>/usr/hivedata/warehouse</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>root</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://192.168.172.103:9083</value>
</property>
- 安裝mysql并配置hive數據庫及權限
yum install mysql
service mysqld start
對hive元數據庫進行賦權，開放遠程連接，開放localhost連接
grant all privileges on *.* to root@"%" identified by "root" with grant option;
grant all privileges on *.* to root@"localhost" identified by "root" with grant option;
- 問題有解決方案
* 如果報錯Terminal initialization failed; falling back to unsupported;
解決方法：將.../hive2.1/lib 里面的jline2.12替換了hadoop 中.../hadoop-2.6.4/share/hadoop/yarn/lib/jline-0.09*.jar
* Exception in thread "main" java.lang.RuntimeException: Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql);
解決方法：schematool -dbType mysql -initSchema
* hive再插入數據的時候報錯，列超過限制，需要MySQL設置：alter database hive character set latin1;
#coordinator
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
task.max-memory=1GB
discovery-server.enabled=true
discovery.uri=http://192.168.172.103:8080
#work
#coordinator=false
#http-server.http.port=8080
#task.max-memory=512m
#discovery.uri=http://192.168.172.103:8080
#用一臺機器進行測試，那么這一臺機器將會即作為coordinator，也作為worker。
#coordinator=true
#node-scheduler.include-coordinator=true
#http-server.http.port=8080
#task.max-memory=1GB
#discovery-server.enabled=true
#discovery.uri=http://example.net:8080
2.2. jvm.config配置文件
-server
-Xmx1G
-XX:+UseConcMarkSweepGC
-XX:+ExplicitGCInvokesConcurrent
-XX:+CMSClassUnloadingEnabled
-XX:+AggressiveOpts
-XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError=kill -9 %p
-XX:ReservedCodeCacheSize=150M
2.3. log.properties配置文件
com.facebook.presto=INFO
2.4. node.properties配置文件（node.id不能重復）
node.environment=production
node.id=XXXX
node.data-dir=/usr/presto/data
2.5. catalog文件下是數據源配置，可以有hive，kafaka，關系型數據庫等等。。我這里配置了hive.properties
connector.name=hive-hadoop2
hive.metastore.uri=thrift://192.168.172.103:9083
#修改為 hive-metastore 服務所在的主機名稱，這里我是安裝在 cdh1節點
hive.config.resources=/usr/local/hadoop2/etc/hadoop/core-site.xml,/usr/local/hadoop2/e
tc/hadoop/conf/hdfs-site.xml
2.6創建客戶端 下載presto-cli-0.172-executable.jar 將其重命名為 presto-cli 添加執行權限chmode +x
./presto-cli --server 192.168.172.103:8080 --catalog hive --schema default
- 啟動與停止
bin/launcher start
也可以前臺啟動，觀察輸出日志：
bin/launcher run
另外，你也可以通過下面命令停止：
bin/launcher stop
4.啟動與測試
- 啟動 hdfs
- 啟動hive hive --service metastore 、.../hive
#創建并插入數據
create table t1(name string);
load data local inpath '/usr/name.txt' into table t1;
- 啟動
bin/launcher run - 運行測試
./presto-cli --server 192.168.172.103:8080 --catalog hive --schema default
show tables;
