1 概述
本文檔描述CENTOS6.5操作系統部署CDH企業版的過程。Cloudera企業級數據中心的安裝主要分為4個步驟：
1.集群服務器配置，包括安裝操作系統、關閉防火墻、同步服務器時鐘等；
2.外部數據庫安裝
3.安裝Cloudera管理器；
4.安裝CDH集群；
這篇文檔將著重介紹Cloudera管理器與CDH的安裝，并基于以下假設：
1.操作系統版本：CENTOS6.5
2.MySQL數據庫版本為5.1.73
3.CM版本：CM 5.12.0
4.CDH版本：CDH 5.12.0
5.Kudu版本：1.4
6.采用root對集群進行部署
7.您已經下載CDH和CM的安裝包
2 前期準備
1.集群中各個節點之間能互相通信使用靜態IP地址。IP地址和主機名通過/etc/hosts配置，主機名/etc/HOSTNAME進行配置。
以cm節點為例：
/etc/hosts文件如下:
172.31.8.113 cdh1
172.31.14.96 cdh2
172.31.9.234 cdh3
172.31.2.34 cdh4
2.配置ntp時間同步,將172.31.8.113機器作為本地ntp服務器，其他3臺機器與其保持同步，配置片段：
172.31.8.113:
$> sudo vim
/etc/ntp.conf
server? 127.127.1.0???????????? # local clock (LCL)
fudge? 127.127.1.0 stratum 10 # LCL is? unsynchronized
其他機器：
$> sudo vim
/etc/ntp.conf
# server
127.127.1.0
# local clock (LCL)
# fudge?? 127.127.1.0 stratum 10 # LCL is unsynchronized
重啟所有機器的ntp服務：
service ntpd restart
驗證始終同步：ntpq -p
3.關閉防火墻：
在每臺機器上運行下列命令
service iptables stop
chkconfig iptables off
vim /etc/selinux/config改為SELINUX=disabled
檢查所有機器：以cdh1示例
安裝httpd服務：
[root@cdh1 ~]# yum -y? install httpd
4.安裝mysql。
[root@cdh1
~]# yum -y install mysql
[root@cdh1? ~]# yum -y install mysql-server
啟動mysql并配置mysql
[root@cdh1 ~]#? service mysqld start
[root@cdh1 ~]#? /usr/bin/mysql_secure_installation
NOTE: RUNNING ALL? PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
SERVERS IN PRODUCTION USE!PLEASE READ EACH STEP CAREFULLY!
In order to log into? MySQL to secure it, we'll need the current
password for the root? user.If you've just installed MySQL,? and
you haven't set the? root password yet, the password will be blank,
so you should just? press enter here.
Enter current? password for root (enter for none):
OK, successfully used? password, moving on...
Setting the root? password ensures that nobody can log into the MySQL
root user without the? proper authorisation.
Set root password?? [Y/n] Y
New password:
Re-enter new? password:
Password updated? successfully!
Reloading privilege? tables..
... Success!
By default, a MySQL? installation has an anonymous user, allowing anyone
to log into MySQL? without having to have a user account created for
them.This is intended only for testing, and to? make the installation
go a bit? smoother.You should remove them? before moving into a
production? environment.
Remove anonymous? users? [Y/n] Y
... Success!
Normally, root should? only be allowed to connect from 'localhost'.This
ensures that someone? cannot guess at the root password from the network.
Disallow root login? remotely? [Y/n] n
... skipping.
By default, MySQL? comes with a database named 'test' that anyone can
access.This is also intended only for testing, and? should be removed
before moving into a? production environment.
Remove test database? and access to it? [Y/n] Y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the? privilege tables will ensure that all changes made so far
will take effect? immediately.
Reload privilege? tables now? [Y/n] Y
... Success!
Cleaning up...
All done!If you've completed all of the above steps,? your MySQL
installation should? now be secure.
Thanks for using? MySQL!
建立CM和Hive需要的表
[root@cdh1? ~]# mysql -u root -p
Enter? password:
Welcome? to the MySQL monitor.Commands end? with ; or \g.
Your? MySQL connection id is 8
create? database metastore default character set utf8;
CREATE? USER 'hive'@'%' IDENTIFIED BY 'password';
GRANT ALL? PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH
PRIVILEGES;
create? database cm default character set utf8;
CREATE? USER 'cm'@'%' IDENTIFIED BY 'password';
GRANT ALL? PRIVILEGES ON cm. * TO 'cm'@'%';
FLUSH? PRIVILEGES;
create? database am default character set utf8;
CREATE? USER 'am'@'%' IDENTIFIED BY 'password';
GRANT ALL? PRIVILEGES ON am. * TO 'am'@'%';
FLUSH? PRIVILEGES;
create? database rm default character set utf8;
CREATE? USER 'rm'@'%' IDENTIFIED BY 'password';
GRANT ALL? PRIVILEGES ON rm. * TO 'rm'@'%';
FLUSH
PRIVILEGES;
安裝jdbc驅動：
[root@cdh1 ~]# mv? mysql-connector-java-5.1.34.jar /usr/share/java/
[root@cdh1 ~]# ln -s? mysql-connector-java-5.1.34.jar mysql-connector-java.jar
dctest01:/usr/share/java? # ll
總用量944
-rwxr-xr-x 1 root? root 9603723月16 15:58 mysql-connector-java-5.1.34.jar
lrwxrwxrwx 1 root? root313月16 16:15 mysql-connector-java.jar ->? mysql-connector-java-5.1.34.jar
3 cloudera manager 安裝
配置本地repo源，將Cloudera Manager安裝需要的7個rpm包下載到本地，放在同一目錄，執行createrepo命令生成rpm元數據：
[root@cdh1 cm5.12.0]#? ll
total 958396
-rw-r--r-- 1 root? root9674128 Jul7 04:35? cloudera-manager-agent-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root 726564488 Jul7 04:35? cloudera-manager-daemons-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root8704 Jul7 04:35? cloudera-manager-server-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root10608 Jul7 04:36? cloudera-manager-server-db-2-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root31880440 Jul7 04:36? enterprise-debuginfo-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root71204325 Jul7 04:36 jdk-6u31-linux-amd64.rpm
-rw-r--r-- 1 root? root 142039186 Jul7 04:35? oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
[root@cdh1 cm5.12.0]#? createrepo .
Spawning worker 0? with 7 pkgs
Workers Finished
Gathering worker? results
Saving Primary? metadata
Saving file lists? metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
配置Web服務器，比如apache2、nginx等，將上述目錄移動到Web服務器數據目錄，這里為/var/www/html，使得用戶可以通過HTTP訪問這些rpm包。
驗證安裝JDK
yum -y install oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
通過yum安裝Cloudera Manager Server
yum -y install? cloudera-manager-server-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
初始化數據庫
sudo /usr/share/cmf/schema/scm_prepare_database.sh? mysql cm cm password
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Verifying that we can? write to /etc/cloudera-scm-server
Creating SCM configuration? file in /etc/cloudera-scm-server
Executing:/usr/java/jdk1.7.0_67-cloudera/bin/java -cp? /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/*? com.cloudera.enterprise.dbutil.DbCommandExecutor? /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[main]? DbCommandExecutorINFOSuccessfully connected to? database.
All done, your SCM? database is configured correctly!
啟動Cloudera Manager Server
[root@cdh1 init.d]#? service cloudera-scm-server start
這個服務啟動比較慢，稍微等幾分鐘，等可以看到7180端口，就可以了
[root@cdh1? cm5.12]# netstat -lnpt | grep 7180
tcp00 0.0.0.0:71800.0.0.0:*LISTEN1524/java
通過http://54.169.251.123:7180/cmf/login訪問CM
4 CDH安裝
1.admin/admin登錄到CM。
2.點擊繼續
3.輸入主機ip或者名稱，點擊搜索找到主機后點擊繼續
4.點擊“更多選項”
5.點擊“-”刪除其他所有地址，輸入http://54.169.251.123/cdh5.12/和http://54.169.251.123/kudu1.4/點擊確定
6.輸入cm的http地址
7.安裝jdk
8.輸入root密碼
9.開始安裝cloudera manager相關到各臺節點
10.開始安裝cdh和Kudu到各臺節點
11.主機檢查，點擊繼續
12.選擇需要安裝的服務
13.分配角色
14.測試數據庫連接，點擊繼續
15.設置目錄
16.等待各個服務啟動
17.安裝成功
5 Kudu配置
在CM界面添加服務，選擇Kudu
分配主機角色
配置相應的目錄
點擊繼續啟動Kudu服務
添加Kudu服務成功，安裝完畢
6 快速組件服務驗證
1.MapReduce驗證
#增加用戶組
[root]$?groupadd? cloudera-dev
#增加用戶
[root]$?useradd? -g cloudera-dev cloudera-dev
#查看用戶組cloudera-dev中的所有用戶
[root]$?id -g?cloudera-dev
#查看用戶cloudera-dev所屬的所有組
[root]$?id? cloudera-dev
# Hadoop創建相應的用戶
[root]$?sudo -u? hdfs hadoop fs -mkdir /user/cloudera-dev
[root]$?sudo -u? hdfs hadoop fs -chown cloudera-dev:cloudera-dev /user/cloudera-dev
#運行Hadoop wordcount示例程序
[root]$?sudo su? cloudera-dev
[cloudera-dev]$ echo? "Hello World Bye World" > file0
[cloudera-dev]$ echo? "Hello Hadoop Goodbye Hadoop" > file1
[cloudera-dev]$? hadoop fs -mkdir -p /user/cloudera-dev/wordcount/input
[cloudera-dev]$ hadoop?fs? -put file* /user/cloudera-dev/wordcount/input
[cloudera-dev]$? hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-examples.jar wordcount? wordcount/input wordcount/output
[cloudera-dev]$? hadoop fs -getmerge wordcount/output output.txt
[cloudera-dev]$ cat? output.txt
2.Hive驗證
[root]$?sudo su? cloudera-dev
[cloudera-dev]$ echo? "Alex,Cloudera" > file2
[cloudera-dev]$? hadoop fs -mkdir -p hive/input
[cloudera-dev]$? hadoop fs -put file2 hive/input
[cloudera-dev]$ cat? > test.hql
create external table? test (
name string,
company string
)
row format delimited
fields? terminated by ','
location? '/user/cloudera-dev/hive/input'
[cloudera-dev]$ hive? -f?test.hql
[cloudera-dev]$ hive? -e "select * from test" 2> /dev/null
[cloudera-dev]$ hive -e "select? count(*) from test" 2> /dev/null
[cloudera-dev]$ hive -e "drop? table test"
[cloudera-dev]$ exit
3.HBase驗證
[cloudera-dev]$?? cat > test.hbase
create 'record',? {NAME => 'user'}
put 'record', 'Alex',? 'user:company', 'Cloudera'
get 'record', 'Alex'
exit
[cloudera-dev]$?? hbase shell test.hbase
[cloudera-dev]$? hbase shell
hbase(main):001:0>? disable 'record'
0 row(s) in 2.6130? seconds
hbase(main):002:0>? drop 'record'
0 row(s) in 1.2880? seconds
#運行Hive Over HBase測試(Read & Write)
[cloudera-dev]$?? cat test.hbase
create 'record',? {NAME => 'user'}
put 'record', 'Alex',? 'user:company', 'Cloudera'
get 'record', 'Alex'
exit
[cloudera-dev]$?? hbase shell test.hbase
[cloudera-dev]$? cat? readHiveOverHbase.hql
create external table? test (name string, company string)
stored by? 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
with serdeproperties? ("hbase.columns.mapping" = ":key,user:company")
tblproperties? ("hbase.table.name" = "record");
[cloudera-dev]$?? hive -f readHiveOverHbase.hql
[cloudera-dev]$?? hive -e "select * from test" 2> /dev/null
[cloudera-dev]$? hive -e? "select count(*) from test" 2> /dev/null
[cloudera-dev]$? cat? writeHiveOverHbase.hql
create table test1? (name string, firm string)
stored by? 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
with serdeproperties? ("hbase.columns.mapping" = ":key,user:firm")
tblproperties ("hbase.table.name" = "record1");
[cloudera-dev]$?? hive -f writeHiveOverHbase.hql
[cloudera-dev]$?? hive -e "insert overwrite table test1 select name, company from? test"
[cloudera-dev]$?? hbase shell
hbase(main):001:0>? list
TABLE
record
record1
2 row(s) in 0.2400? seconds
=>? ["record", "record1"]
hbase(main):002:0>? scan 'record1'
ROWCOLUMN+CELL
Alexcolumn=user:firm,? timestamp=1500707374796, value=Cloudera
1 row(s) in 0.1360? seconds
4. Kudu驗證
先在Impala的高級配置中配置Kudu Master地址：
--kudu_master_hosts=cdh1:7051
[root@cdh1 ~]# impala-shell -i cdh4
[cdh4:21000]? > CREATE TABLE my_first_table
> (
> id BIGINT,
> name STRING,
> PRIMARY KEY(id)
> )
> PARTITION BY HASH PARTITIONS 16
> STORED AS KUDU;
[cdh4:21000]? > INSERT INTO my_first_table VALUES (99,"sarah");
[cdh4:21000]? > INSERT INTO my_first_table VALUES? (1,"john"),(2,"jane"),(3,"jim");
[cdh4:21000]? > select * from my_first_table;
[cdh4:21000]? > delete from my_first_table where id =99;
[cdh4:21000]? > select * from my_first_table;
[cdh4:21000]? > INSERT INTO my_first_table VALUES (99,"sarah");
[cdh4:21000]? > update my_first_table set name='lilei' where id=99;
[cdh4:21000]? > select * from my_first_table;
[cdh4:21000]? > upsertinto my_first_table? values(1, "john"), (4, "tom"), (99, "lilei1");
[cdh4:21000]? > select * from my_first_table;
