一 前言

這是之前寫的一篇文章，現在整理一下，重新發出來。
由于Ambari安裝在ARM機器上問題比較多。主要問題如下：

• ambari依賴的node.js版本是0.10.44，而aarch64機器只支持v4.x以上版本。
• ambari依賴的phantomjs版本是1.9.8，而aarch64機器只支持v2.1.0的以上版本呢
• ambari依賴的一些第三方開源項目，aarch64機器不支持。

因此選擇開源社區版Hadoop組件部署高可用集群。

二 集群架構設計

2.1 基礎環境

2.2 Hadoop組件

2.3 軟件版本

三 下載Hadoop主要組件

下載oracle jdk arm64
wget --no-cookie --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u162-b12/0da788060d494f5095bf8624735fa2f1/jdk-8u162-linux-arm64-vfp-hflt.tar.gz
下載hadoop
wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz
下載hase
wget https://archive.apache.org/dist/hbase/1.1.2/hbase-1.1.2-bin.tar.gz
下載zk
wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
下載spark
wget https://archive.apache.org/dist/spark/spark-2.1.0/spark-2.1.0-bin-hadoop2.7.tgz
下載kafka
wget https://archive.apache.org/dist/kafka/0.10.1.1/kafka_2.11-0.10.1.1.tgz
下載phoenix
wget https://archive.apache.org/dist/phoenix/phoenix-4.7.0-HBase-1.1/bin/phoenix-4.7.0-HBase-1.1-bin.tar.gz
下載scala
wget https://downloads.lightbend.com/scala/2.11.11/scala-2.11.11.tgz
下載zeppelin
wget http://apache.claz.org/zeppelin/zeppelin-0.7.3/zeppelin-0.7.3-bin-all.tgz
下載hue
wget https://github.com/cloudera/hue/archive/release-4.2.0.tar.gz
上傳本地文件到服務器
scp confluent-3.1.2-2.11.tar.gz -p 50300
所有下載目錄解壓后，都做ln -s軟連接

四 基礎環境變量配置

4.0 配置etc/hosts.安裝ntp，關閉防火墻

4.1 添加bigdata賬號

useradd -g bigdata bigdata

4.2 配置集群各節點SSH無密碼連接

創建 authorized_keys 文件 //該文件的權限必須是644或者600，否則無效

4.3 環境變量配置（編輯bigdata用戶的.bash_profile文件）

export SCALA_HOME=/usr/scala/default
export JAVA_HOME=/usr/java/default
export HADOOP_HOME=/home/bigdata/hadoop
export HBASE_HOME=/home/bigdata/hbase 
export SPARK_HOME=/home/bigdata/spark
export HADOOP_CONF_DIR=/home/bigdata/hadoop/etc/hadoop
export HBASE_CONF_DIR=/home/bigdata/hbase/conf
export HADOOP_LOG_DIR=/home/bigdata/log/hdfs  
export ZOOKEEPER_HOME=/home/bigdata/zookeeper
export ZEPPELIN_HOME=/home/bigdata/zeppelin 
export KAFKA_HOME=/home/bigdata/kafka
export CONFLUENT_HOME=/home/bigdata/confluent
export YARN_LOG_DIR=$HADOOP_LOG_DIR
export HUE_HOME=/home/bigdata/hue 
export PATH=$JAVA_HOME/bin:$SCALA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:
$HBASE_HOME/bin:$CONFLUENT_HOME/bin:$PATH

下面的安裝都是基于bigdata用戶操作

五 安裝Zookeeper

5.0 配置zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/home/bigdata/zkdata
dataLogDir=/home/bigdata/zklogs
# the port at which the clients will connect
clientPort=2181
server.1=bigdata3:2888:3888
server.2=bigdata4:2888:3888
server.3=bigdata5:2888:3888

在配置的dataDir目錄下創建相應的myid文件（這個myid文件必須創建，否則啟動會報錯）分別在ZK集群節點創建myid號，myid一定對應好zoo.cfg中配置的server后面1、2、3這個ZK號

5.1 在bigdata3，bigdata4，bigdata5 機器安裝；

錯誤1—— 2臺正常啟動，另一臺則報錯誤報錯

/home/bigdata/zkdata/version-2/acceptedEpoch.tmp (Permission denied)

解決：查看verison-2文件夾的權限果然是root，改為bigdata

5.2 在bigdata3，bigdata4，bigdata5三臺機器啟動zk

zkServer.sh start

六 安裝Hadoop

6.0 配置hadoop-env.sh和yarn_env.sh文件

JAVA_HOME=/usr/java/default

6.1 配置Hadoop文件core-site.xml

<!-- 指定hdfs的nameservice為bigdatacluster，這里的bigdatacluster對應于后面hdfs-site.xml中的dfs.nameservices標簽的值 -->
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://bigdatacluster</value>
</property>
 <!-- 指定hadoop運行時產生文件的存儲路徑-->
<property>
        <name>hadoop.tmp.dir</name>
        <value>/home/bigdata/tmp/hadoop</value>
        <description>Abase for other temporary directories.</description>
</property>
<!-- 指定zookeeper地址，多個用,分割,2181為客戶端連接ZK服務器端口 -->
<property>
        <name>ha.zookeeper.quorum</name>
        <value>bigdata3:2181,bigdata4:2181,bigdata5:2181</value>
</property>
<!-- 配置 hue housts和groups -->
<property>
  <name>hadoop.proxyuser.hue.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.hue.groups</name>
  <value>*</value>
</property>

6.2 配置hdfs文件hdfs-site.xml

  <!--NN存放元數據和日志位置-->
<property>
    <name>dfs.namenode.name.dir</name>
    <value>/data/hdfs/nn</value>
    <final>true</final>
</property>
<!--HDFS文件系統數據存儲位置，可以分別保存到不同硬盤，突破單硬盤性能瓶頸，多個位置以逗號隔開-->
<property>
    <name>dfs.datanode.data.dir</name>
    <value>/data/hdfs/dn</value>
    <final>true</final>
</property>
<property>
    <name>dfs.replication</name>
    <value>2</value>
</property>

 <!-- dfs.nameservices 命名空間的邏輯名稱，多個用,分割 -->
<property>
    <name>dfs.nameservices</name>
    <value>bigdatacluster</value>
</property>
<!-- 指定bigdatacluster下有兩個namenode，分別是bigdata1,bigdata2 -->
<property>
    <name>dfs.ha.namenodes.bigdatacluster</name>
    <value>bigdata1,bigdata2</value>
</property>
<!-- 指定bigdata1的RPC通信地址 -->
<property>
    <name>dfs.namenode.rpc-address.bigdatacluster.bigdata1</name>
    <value>bigdata1:9000</value>
</property>
  <!-- 指定bigdata1的HTTP通信地址 -->
<property>
    <name>dfs.namenode.http-address.bigdatacluster.bigdata1</name>
    <value>bigdata1:50070</value>
</property>
<!-- 指定bigdata2的RPC通信地址 -->
<property>
    <name>dfs.namenode.rpc-address.bigdatacluster.bigdata2</name>
    <value>bigdata2:9000</value>
</property>
  <!-- 指定bigdata2的HTTP通信地址 -->
<property>
    <name>dfs.namenode.http-address.bigdatacluster.bigdata2</name>
    <value>bigdata2:50070</value>
</property>
<!-- 指定namenode的元數據存放的Journal Node的地址，必須奇數，至少三個 -->
<property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://bigdata3:8485;bigdata4:8485;bigdata5:8485/bigdatacluster</value>
</property>
<!--這是JournalNode進程保持邏輯狀態的路徑。這是在linux服務器文件的絕對路徑-->
<property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/data/hdfs/journal</value>
</property>
 <!-- 開啟namenode失敗后自動切換 -->
<property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
</property>
   <!-- 配置失敗自動切換實現方式 -->
<property>
    <name>dfs.client.failover.proxy.provider.bigdatacluster</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔離機制方法，多個機制用換行分割 -->
<property>
    <name>dfs.ha.fencing.methods</name>
    <value>
        sshfence
        shell(/bin/true)
    </value>
</property>
  <!-- 使用sshfence隔離機制時需要ssh免登陸 -->
<property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
    <value>/home/bigdata/.ssh/id_rsa</value>
</property>
  <!-- 配置sshfence隔離機制超時時間30秒 -->
<property>
    <name>dfs.ha.fencing.ssh.connect-timeout</name>
    <value>30000</value>
</property>
<!-- 配置打開webhdfs屬性，hue集成需要 -->
<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>

6.3 配置MR文件mapred-site.xml

<property>
    <!-- 通知框架MR使用YARN -->
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>

6.4 配置YARN 文件yarn-site.xml

<!--啟用RM高可用-->
   <property>
       <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
   </property>
        <!--RM集群標識符-->
   <property>
       <name>yarn.resourcemanager.cluster-id</name>
        <value>rm-cluster</value>
   </property>
   <property>
        <!--指定兩臺RM主機名標識符-->
       <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
   </property>
        <!--RM故障自動切換-->
   <property>
        <name>yarn.resourcemanager.ha.automatic-failover.recover.enabled</name>
        <value>true</value>
   </property>
   <!--RM故障自動恢復
   <property>
       <name>yarn.resourcemanager.recovery.enabled</name> 
        <value>true</value> 
   </property> -->
        <!--RM主機1，如果希望單獨裝在另外兩個節點上，請寫入對應的主機名，后面配置也需要相應修改-->
   <property>
       <name>yarn.resourcemanager.hostname.rm1</name>
        <value>bigdata1</value>
   </property>
        <!--RM主機2，如果希望單獨裝在另外兩個節點上，請寫入對應的主機名，后面配置也需要相應修改-->
   <property>
       <name>yarn.resourcemanager.hostname.rm2</name>
        <value>bigdata2</value>
   </property>
       <!--RM狀態信息存儲方式，一種基于內存(MemStore)，另一種基于ZK(ZKStore)-->
    <property>
       <name>yarn.resourcemanager.store.class</name>
       <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>
        <!--使用ZK集群保存狀態信息-->
    <property>
       <name>yarn.resourcemanager.zk-address</name>
       <value>bigdata3:2181,bigdata4:2181,bigdata5:2181</value>
    </property>
        <!--向RM調度資源地址-->
    <property>
       <name>yarn.resourcemanager.scheduler.address.rm1</name>
        <value>bigdata1:8030</value>
    </property>
    <property>
       <name>yarn.resourcemanager.scheduler.address.rm2</name>
       <value>bigdata2:8030</value>
    </property>
        <!--NodeManager通過該地址交換信息-->
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
       <value>bigdata1:8031</value>
    </property>
    <property>
       <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
       <value>bigdata2:8031</value>
    </property>
        <!--客戶端通過該地址向RM提交對應用程序操作-->
    <property>
       <name>yarn.resourcemanager.address.rm1</name>
       <value>bigdata1:8032</value>
    </property>
    <property>
       <name>yarn.resourcemanager.address.rm2</name>
       <value>bigdata2:8032</value>
    </property>
        <!--管理員通過該地址向RM發送管理命令-->
    <property>
       <name>yarn.resourcemanager.admin.address.rm1</name>
       <value>bigdata1:8033</value>
    </property>
    <property>
       <name>yarn.resourcemanager.admin.address.rm2</name>
        <value>bigdata2:8033</value>
    </property>
    <!--RM HTTP訪問地址,查看集群信息-->
    <property>
       <name>yarn.resourcemanager.webapp.address.rm1</name>
       <value>bigdata1:8088</value>
    </property>
    <property>
       <name>yarn.resourcemanager.webapp.address.rm2</name>
       <value>bigdata2:8088</value>
    </property>
   <!--不限制yarn容器分配的最大內存-->
    <property>
     <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    </property>

配置httpfs-site.xml（如果集成Hue，并且Hadoop集群是HA情況，需要httpFS訪問hdfs）

<property>
  <name>httpfs.proxyuser.hue.hosts</name>
  <value>*</value>
</property>
<property>
  <name>httpfs.proxyuser.hue.groups</name>
  <value>*</value>
</property>

6.5 啟動hadoop集群

6.5.1 確保三臺slave機器已啟動zk

6.5.2 在三臺slave機器啟動jouralnode

hadoop-daemon.sh start journalnode

6.5.3 在bigdata1上，第一次運行分別格式化hdfs，zk

hdfs namenode –format
hdfs zkfc –formatZK //手動輸入，復制拷貝命令，不識別

6.5.4 在bigdata2 上格式化目錄并同步兩個master節點的元數據

# 方法1，通過bigdata1：9000端口連接不到bigdata1，所以采用方法2
hdfs namenode -bootstrapStandby
# 方法2，直接拷貝bigdata1格式化后的元數據到bigdata2
scp -r nn bigdata2:/data/hdfs/

6.5.5 分別在bigdata1和bigdata2上啟動ZKFC來監控NameNode

#在bigdata1 啟動ZKFC來監控NameNode
hadoop-daemon.sh start zkfc
#在bigdata2 啟動ZKFC來監控NameNode
hadoop-daemon.sh start zkfc

6.5.6 在bigdata1上啟動hdfs

start-dfs.sh

報錯：
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
解決：添加編譯好的hadoop aarch64位so文件

6.5.7 在bigdata1上啟動yarn

start-yarn.sh

6.5.8 在bigdata2上啟動yarn

yarn-daemon.sh start resourcemanager

http://bigdata1:50070
http://bigdata1:8088/cluster

七 Hbase安裝

7.0 對于Hbase 修改 ulimit 限制

echo "bigdata  -      nofile  32768" >> /etc/security/limits.conf
echo "bigdata  -      nproc   32000" >>  /etc/security/limits.conf
echo "session required pam_limits.so" >>  /etc/pam.d/common-session 

7.1 配置hbase-env.sh

# The java implementation to use.  Java 1.7+ required.
export JAVA_HOME=/usr/java/default
//配置該路徑，否則在HA情況下，不能解析HDFS路徑
export HADOOP_HOME=/home/bigdata/hadoop
# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=false

配置hbase-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>bigdata3:2181,bigdata4:2181,bigdata5:2181</value>
        <description>The directory shared by RegionServers.
        </description>
    </property>
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/home/bigdata/zkdata</value>
        <description>Property from ZooKeeper config zoo.cfg.
            The directory
            where the snapshot is stored.
        </description>
    </property>
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://bigdatacluster/hbase</value>
        <description>The directory shared by RegionServers.
        </description>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
        <description>The mode the cluster will be in. Possible values are
            false: standalone and pseudo-distributed setups with managed
            Zookeeper
            true: fully-distributed with unmanaged Zookeeper Quorum (see
            hbase-env.sh)
        </description>
    </property>
</configuration>

Regionservers

bigdata3
bigdata4
bigdata5

7.2 啟動Hbase

在bigdata1啟動

start-hbase.sh

在bigdata2啟動

hbase-daemon.sh start master

http://bigdata1:16010/master-status

8 基于HA高可用環境安裝Spark on Yarn (在bigdata1上安裝)

8.1 配置spark-env.sh

export SCALA_HOME=/usr/scala/default
export JAVA_HOME=/usr/java/default
export HADOOP_HOME=/home/bigdata/hadoop
export HBASE_HOME=/home/bigdata/hbase 
export HADOOP_CONF_DIR=/home/bigdata/hadoop/etc/hadoop
export HBASE_CONF_DIR=/home/bigdata/hbase/conf
export HADOOP_LOG_DIR=/home/bigdata/log/hdfs  

8.2 配置spark-defaults

spark.master    yarn
spark.driver.memory    2g
spark.executor.memory  2g
spark.eventLog.enabled  true
#如果hadoop 是HA環境，注意命名空間的名稱
spark.eventLog.dir      hdfs://bigdatacluster/spark-logs
# 歷史日志服務配置
spark.history.provider            org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory     hdfs://bigdatacluster/spark-logs
spark.history.fs.update.interval  10s
spark.history.ui.port             18080

8.3 在hdfs上創建spark日志目錄

hdfs dfs -mkdir /spark-logs

8.4 啟動spark日志

start-history-server.sh

訪問頁面

http://bigdata1:18080/

九 安裝Zeppelin(在bigdata1上安裝)

9.0 修改zeppelin-site.xml文件

# 修改端口號為28080
<property>
  <name>zeppelin.server.port</name>
  <value>28080</value>
  <description>Server port.</description>
</property>

修改 zeppelin-env.sh文件

export JAVA_HOME=/usr/java/default

export HADOOP_CONF_DIR=/home/bigdata/hadoop/etc/hadoop         
#### HBase interpreter configuration ####
export HBASE_HOME=/home/bigdata/hbase                    #
export HBASE_CONF_DIR=/home/bigdata/hbase/conf                

啟動服務

zeppelin-daemon.sh start

http://bigdata1:28080

十 安裝部署Kafka(在bigdata1上安裝)

10.1 首先確保已經安裝啟動ZK服務器

10.2 配置server.properties文件

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
## 監聽本機所有網絡接口(network interfaces)
listeners=PLAINTEXT://0.0.0.0:9092
## 被發布到Zookeeper上，公布給Client讓Client使用
advertised.listeners=PLAINTEXT://bigdata1:9092

num.network.threads=3
# The number of threads doing disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
//配置kafka的日志目錄
log.dirs=/home/bigdata/kafka-logs
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

#log.flush.interval.ms=1000

log.retention.hours=168
#log.retention.bytes=1073741824

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

############################# Zookeeper #############################
# root directory for all kafka znodes.
//配置已安裝好的zk集群地址
zookeeper.connect=bigdata3:2181,bigdata4:2181,bigdata5:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

10.3 啟動kafka

kafka-server-start.sh -daemon config/server.properties

10.4 測試消息

## 1.創建主題
kafka-topics.sh --create --topic TestTopic003 --partitions 1 --replication-factor 1 --zookeeper bigdata3:2181,bigdata4:2181,bigdata5:2181
## 2.發送消息
kafka-console-producer.sh --topic TestTopic003 --broker-list bigdata1:9092
This is a message
This is another message
## 3.消費消息
kafka-console-consumer.sh --topic TestTopic003 --from-beginning --bootstrap-server bigdata1:9092

十一 安裝Confluent

11.1上傳本地3.1.2 版本至服務器，配置環境變量；

11.2 部署hbase-sink.jar包

nohup  schema-registry-start -daemon $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties >nohup_shcema_registry.log> nohup_shcema_registry.err

nohup  connect-standalone -daemon $CONFLUENT_HOME/etc/schema-registry/connect-avro-standalone.properties $CONFLUENT_HOME/etc/kafka-connect-hbase/hbase-sink.properties > nohup_standalone.log>nohup_standalone.err

kafka-avro-console-producer \
--broker-list bigdata1:9092 --topic test \
--property value.schema='{"type":"record","name":"record","fields":[{"name":"id","type":"int"}, {"name":"name", "type": "string"}]}'
{"id": 1, "name": "sz”}
{"id": 2, "name": "bj”}
{"id": 3, "name": “aa”}
{"id": 4, "name": "bb”}

schema-registry-start  $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties 

connect-standalone  $CONFLUENT_HOME/etc/schema-registry/connect-avro-standalone.properties $CONFLUENT_HOME/etc/kafka-connect-hbase/hbase-sink.properties

{"id": 1, "name": "sz”}
{"id": 2, "name": "bj”}
{"id": 3, "name": “aa”}
{"id": 4, "name": "bb”}

錯誤記錄：
部署遠程kafka客戶端寫入kafka集群數據時

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

解決辦法：遠程客戶端，沒有找到服務器，一種方法kafka的server.properties是配置文件中主機名改為ip地址，另一種方式是在遠程客戶端配置host文件中ip地址和主機名的映射；

十二 安裝HUE

這個組件安裝比較復雜，需要編譯安裝

12.1 安裝依賴庫

sudo yum install ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libxml2-devel libxslt-devel make  mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel

注意安裝ant時，會依賴open jdk，如果已經安裝oracle jdk 版本安裝完ant后，卸載openjdk即可。

12.2 安裝maven 3+版本,配置環境變量

12.3 編譯部署(時間較長，耐心等待)

tar -zxvf hue-release-4.2.0
ln -s hue-release-4.2.0 hue
cd hue-release-4.2.0

如果要編譯中文環境，進入hue-release-4.2.0/desktop/core/src/desktop目錄請修改settings.py文件

#注釋英文，添加簡體中文
#LANGUAGE_CODE = 'en-us'
LANGUAGE_CODE='zh_CN'

# 開始編譯
make apps

12.4 配置Hue

進入hue-release-4.2.0/desktop/conf目錄,

cp pseudo-distributed.ini hue.ini

配置hue.ini文件

[desktop]
  # Set this to a random string, the longer the better.
  # This is used for secure hashing in the session store.
  secret_key=jFE93j;2[290-eiw.KEiwN2s3['d;/.q[eIW^y#e=+Iei*@Mn<qW5o
  # Webserver listens on this address and port
  http_host=bigdata1
  http_port=8888
 # 與系統時區保持一致
  time_zone=America/New_York
# 配置Hue界面不顯示的Hadoop組件，我們的集群沒有使用hive，所以屏蔽了。
app_blacklist=security,filebrowser,jobbrowser,rdbms,jobsub,hbase,sqoop,zookeeper,spark,oozie,indexer

[hadoop]
  # 配置Hue的Hadoop（注意我們的集群是高可用的）
  # ------------------------------------------------------------------------
  [[hdfs_clusters]]
    # HA support by using HttpFs 高可用集群只支持httpFS，不支持webhdfs方式
    [[[default]]]
      # 這個名稱和HA hadoop配置的一致
      fs_defaultfs=hdfs://bigdatacluster
      # NameNode logical name.
      logical_name=bigdatacluster
      # Use WebHdfs/HttpFs as the communication mechanism.
      # Domain should be the NameNode or HttpFs host.
      # Default port is 14000 for HttpFs.
      webhdfs_url=http://bigdata1:14000/webhdfs/v1

      # Change this if your HDFS cluster is Kerberos-secured
      ## security_enabled=false

      # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs
      # have to be verified against certificate authority
      ## ssl_cert_ca_verify=True

      # Directory of the Hadoop configuration
      hadoop_conf_dir=/home/bigdata/hadoop/etc/hadoop

  # Configuration for YARN (MR2)
  # ------------------------------------------------------------------------
  [[yarn_clusters]]
    [[[default]]]
      # Enter the host on which you are running the ResourceManager
      resourcemanager_host=bigdata1
      # The port where the ResourceManager IPC listens on
      resourcemanager_port=8032
      # Whether to submit jobs to this cluster
      submit_to=True
      # Resource Manager logical name (required for HA)
      logical_name=rm-cluster
      # Change this if your YARN cluster is Kerberos-secured
      ## security_enabled=false
      # URL of the ResourceManager API
      resourcemanager_api_url=http://bigdata1:8088
      # URL of the ProxyServer API
      ## proxy_api_url=http://localhost:8088
      # URL of the HistoryServer API
      ## history_server_api_url=http://localhost:19888
      # URL of the Spark History Server
      spark_history_server_url=http://bigdata1:18088
[hbase]
  # Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'.
  # Use full hostname with security.
  # If using Kerberos we assume GSSAPI SASL, not PLAIN.
  hbase_clusters=(Cluster|bigdata1:9090)
  hbase_conf_dir=/home/bigdata/hbase/conf

12.5 啟動httpfs服務（HA集群需要）

需要/hadoop/sbin/httpfs.sh start來啟動Bootstrap進程，以服務HttpFS管理

12.6 啟動Hue

參考文章

1.Hadoop 2.7.3 高可用（HA）集群部署. HanBert
2.Spark On Yarn Install, Configure, and Run Spark on Top of a Hadoop YARN Cluster .Florent Houbart
3.Hue官方安裝文檔 .