參照官網的步驟：https://www.cloudera.com/documentation/enterprise/latest/topics/installation.html
準備三臺服務器：
| ip 地址（CDH 只支持 ipv4） | 主機名 |
|---|---|
| 10.0.0.0 | master |
| 10.0.0.1 | slave01 |
| 10.0.0.2 | slave02 |
下載安裝包，放在 /root 目錄下（若在線安裝，則可忽略）
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/
https://archive.cloudera.com/cdh6/6.1.0/parcels/
| 主機名 | 所需安裝包 |
|---|---|
| master | cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm cloudera-manager-server-db-2-6.1.0-769885.el7.x86_64.rpm CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel manifest.json |
| slave | cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm |
安裝總共分為三大部分：
Part1: 安裝前準備
Part2: 安裝 Cloudera Manager 和 CDH
Part3: 安裝之后
PART1 安裝前準備
step1: 修改主機名（所有節點，重啟有效）
# master
hostnamectl set-hostname master
# slave01
hostnamectl set-hostname slave01
# slave02
hostnamectl set-hostname slave02
設置映射主機名，將以下內容添加到 /etc/hosts 文件：
10.0.0.0 master
10.0.0.1 slave01
10.0.0.2 slave02
各節點編輯 /etc/sysconfig/network， 填入各自的主機名。比如 master：
HOSTNAME=master
step2: 關閉防火墻（所有節點）
#查看防火墻
systemctl status firewalld.service
#執行關閉命令
systemctl stop firewalld.service
#執行開機禁用防火墻命令
systemctl disable firewalld.service
step3: 關閉 SELINUX （所有節點）
vim /etc/sysconfig/selinux
將 SELINUX=enforcing 改為 SELINUX=disabled
step4: 啟用ntp （所有節點）
安裝 ntp
yum install ntp
啟動 ntp
systemctl enable ntpd
systemctl start ntpd
step5: 配置從 master 到所有 slave 的 ssh 免密登錄
先在 master 上執行：
ssh-keygen -t rsa #一路回車到完成
ssh-copy-id -i ~/.ssh/id_rsa.pub root@master #將公鑰拷貝到本機的authorized_keys上
再在所有 slave 上分別執行：
ssh-keygen -t rsa #一路回車到完成
ssh-copy-id -i ~/.ssh/id_rsa.pub root@master #注意此處不變，將公鑰拷貝到master的authorized_keys上
在 master 上，將 authorized_keys 分發到其他節點服務器：
scp ~/.ssh/authorized_keys root@slave01:~/.ssh/
scp ~/.ssh/authorized_keys root@slave02:~/.ssh/
step6: 禁用透明大頁面壓縮（所有節點）
注：step6 與 step7 雖然不是官網的標準步驟，但如果不執行，創建集群時會報 warning，仍需修改配置。
打開編輯 /etc/rc.local 文件：
vim /etc/rc.local
添加以下內容：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
step7: 優化交換分區（所有節點）
編輯 /etc/sysctl.conf 并添加一行 vm.swappiness=0
vim /etc/sysctl.conf
PART2 安裝 Cloudera Manager 和 CDH
step1: 為 Cloudera Manager 配置存儲庫（master）
- 下載對應操作系統的 cloudera-manager.repo 文件到 Cloudera Manager Server 主機上的 /etc/yum.repos.d/
sudo wget https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
- 導入存儲庫簽名 GPG key：
sudo rpm --import https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPM-GPG-KEY-cloudera
step2: 安裝 JDK 并寫入環境變量（所有節點）
（需要先卸載掉已安裝的 JDK）
（認真對待 step2， 沒裝好會導致 Cloudera Manager 啟動失敗）
- 完成 step 1：為 Cloudera Manager 配置存儲庫后，可以使用軟件包管理器在 Cloudera Manager Server 主機上（master）安裝 Oracle JDK，如下所示：
sudo yum install oracle-j2sdk1.8
如果已經下載了 JDK（下載地址：https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm），直接用 rpm 安裝即可：
rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
或自行安裝：
Oracle 官網下載 Oracle JDK 并安裝，需要 jdk1.8 及以上。
- 寫入環境變量：
/etc/profile 末尾添加以下內容
#set jdk environment
export JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
step3: 安裝 Cloudera Manager 軟件包
在 master 上安裝 Cloudera Manager 軟件包，并可選擇啟用 auto-TLS（推薦啟用）。
- 先安裝下面的內容，因為 cloudera-manager-agent 依賴這些（所有節點）
yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi portmap httpd mod_ssl openssl-devel python-psycopg2 MySQL-python /lib/lsb/init-functions
- 接下來需要安裝：（注意需要先安裝 daemons，再安裝其他）
master:
rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm
slave:
rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
或者可使用在線下載安裝，下載慢，不推薦
sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
- 啟用 auto-TLS：
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager setup --configure-services
啟用 auto-TLS 后，后續訪問 impala 需要指明使用 ssl（默認為不使用）
例如啟動 impala-shell：impala-shell --ssl
- 修改 agent 的服務器地址（所有節點）
編輯 /etc/cloudera-scm-agent/config.ini：
vim /etc/cloudera-scm-agent/config.ini
將 server_host=localhost 改為 master 節點主機名 master
step4: 安裝數據庫（MySQL）
此數據庫用于存儲 Cloudera Manager 配置信息，只安裝在 master 即可
- 安裝 MySQL Server
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum update
yum install mysql-server
systemctl start mysqld
- 配置并啟動 MySQL Server
- 首先停止 MySQL 服務
systemctl stop mysqld
- 將舊的 InnoDB 日志文件 /var/lib/mysql/ib_logfile0 和 /var/lib/mysql/ib_logfile1 從 /var/lib/mysql/ 中移除
rm -rf /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile1
- 更新 /etc/my.cnf 配置文件，推薦的配置如下：
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
- 啟動服務并設置開機啟動
systemctl enable mysqld
systemctl start mysqld
- 設置 root 密碼和其他安全相關設置：
/usr/bin/mysql_secure_installation
[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!
- 安裝 MySQL JDBC Driver
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
- 注意，此時 mysql-connector-java.jar 的權限若為
rwx------，后期 Cloudera Manager Server 啟動時，用非 root 用戶沒有 mysql-connector-java.jar 的權限，將導致服務啟動失敗。因此開放 mysql-connector-java.jar 的權限：
chmod 755 /usr/share/java/mysql-connector-java.jar
- 為 Cloudera Manager 創建 MySQL 數據庫
- root 用戶登錄
mysql -u root -p
Enter password:
- 為下表中每個服務創建 utf8 字符集的數據庫：
CREATE DATABASE <database> DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';
| Service | database | user |
|---|---|---|
| Cloudera Manager Server | scm | scm |
| Activity Monitor | amon | amon |
| Reports Manager | rman | rman |
| Hue | hue | hue |
| Hive Metastore Server | metastore | hive |
| Sentry Server | sentry | sentry |
| Cloudera Navigator Audit Server | nav | nav |
| Cloudera Navigator Metadata Server | navms | navms |
| Oozie | oozie | oozie |
命令如下：
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
flush privileges;
- 檢查 MySQL 確認已創建所有數據庫
SHOW DATABASES;
step5: 設置 Cloudera Manager 數據庫
master 執行：
rm /etc/cloudera-scm-server/db.mgmt.properties
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
得到：
Enter SCM password:
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
step6: 安裝 CDH 和其他軟件
- 準備 parcel 包（master）
若在線安裝，則跳過此步
- 將已下載的 CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel 和 manifest.json 拷貝至 /opt/cloudera/parcel-repo/ 目錄，并且生成 parcel 包的 sha 值：
sha1sum /opt/cloudera/parcel-repo/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
- 將 sha 值填入到 CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha 文件中
vim /opt/cloudera/parcel-repo/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha
- 啟動 cloudera-scm-agent （所有節點）
systemctl start cloudera-scm-agent
- 啟動 Cloudera Manager Server （master）
systemctl start cloudera-scm-server
查看啟動日志：
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
看到 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. 則說明啟動完成
********** 接著在 https://10.0.0.0:7183 創建集群 **********
