建議使用阿里云自身的容器服務(wù)吧，畢竟自建k8s集群太折騰

1、資源架構(gòu)

前期使用 3 master + 3 worker配置，后期再擴展worker節(jié)點

架構(gòu)圖

堆疊（Stacked） etcd 拓撲

堆疊的 etcd 拓撲

外部 etcd 拓撲

外部 etcd 拓撲

2、基礎(chǔ)環(huán)境

2.1、軟件版本

由于kubernetes 1.20 版本開始已棄用docker，推薦使用containerd

2.2、系統(tǒng)配置

沒有特別說明的，所有主機需執(zhí)行

• 修改hostname

hostname k8s-master1

• 關(guān)閉防火墻

# systemctl stop firewalld
# systemctl disable firewalld

• 關(guān)閉selinux

# sed -i 's/enforcing/disabled/' /etc/selinux/config  # 永久
# setenforce 0  # 臨時

• 關(guān)閉 swap

# swapoff -a  # 臨時
# sed -ri 's/.*swap.*/#&/' /etc/fstab    # 永久

• 添加 hosts

# cat >> /etc/hosts << EOF
172.18.20.44 k8s-master1
172.18.20.45 k8s-master2
172.18.20.46 k8s-master3
...
EOF

• 配置節(jié)點主機免密訪問(master)

# ssh-keygen 
# ssh-copy-id root@k8s-*

• 將橋接的IPv4流量傳遞到iptables的鏈

# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# sysctl --system 

• 調(diào)整系統(tǒng)內(nèi)核參數(shù)

  # cat > /etc/sysctl.d/kubernetes.conf <<EOF
  net.bridge.bridge-nf-call-iptables=1
  net.bridge.bridge-nf-call-ip6tables=1
  net.ipv6.conf.all.disable_ipv6=1
  net.ipv4.ip_forward=1
  net.ipv4.tcp_tw_recycle=0
  vm.swappiness=0
  fs.file-max=2000000
  fs.nr_open=2000000
  fs.inotify.max_user_instances=512
  fs.inotify.max_user_watches=1280000
  net.netfilter.nf_conntrack_max=524288
  EOF
   
  # modprobe br_netfilter && sysctl -p /etc/sysctl.d/kubernetes.conf
  

• 錯誤提示

  /proc/sys/net/bridge/bridge-nf-call-iptables: 沒有那個文件或目錄
  

  記得運行 modprobe br_netfilter

• 加載ipvs 模塊

  cat > /etc/sysconfig/modules/ipvs.modules <<EOF
  #!/bin/bash
  modprobe -- ip_vs
  modprobe -- ip_vs_rr
  modprobe -- ip_vs_wrr
  modprobe -- ip_vs_sh
  modprobe -- nf_conntrack_ipv4
  EOF
   
  chmod 755 /etc/sysconfig/modules/ipvs.modules
  sh /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_
  

3、etcd 集群部署

k8s集群使用堆疊 ETCD 時，可不單獨創(chuàng)建，本人是為了延長 etcd 證書才單獨部署集群，實際架構(gòu)也是堆疊式

3.1 使用 etcdadm 工具

下載

# wget https://github.com/kubernetes-sigs/etcdadm/releases/download/v0.1.3/etcdadm-linux-amd64
# mv etcdadm-linux-amd64 /usr/local/bin/etcdadm
# chmod +x /usr/local/bin/etcdadm

選擇一個節(jié)點初始化etcd

etcdadm init --install-dir="/opt/etcd/" --name=etcd-1

參數(shù)解析

• --install-dir 安裝目錄
• --name 節(jié)點名稱
• server-cert-extra-sans 負載均衡地址（單獨部署使用）

拷貝證書到其他節(jié)點

scp /etc/etcd/pki/ca.* root@master2:/etc/etcd/pki/
scp /etc/etcd/pki/ca.* root@master3:/etc/etcd/pki/

加入集群

 etcdadm join https://172.18.20.44:2379 --install-dir="/opt/etcd/" --name=etcd-2

安裝完成后查看集群

修改環(huán)境變量

source /etc/etcd/etcdctl.env

查看

# /opt/bin/etcdctl member list 
18ed9897779ce358, started, k8s-master1, https://172.18.20.50:2380, https://172.18.20.50:2379, false
3e04e89693dd5c26, started, k8s-master2, https://172.18.20.51:2380, https://172.18.20.51:2379, false
68562c67a29d6f06, started, k8s-master3, https://172.18.20.49:2380, https://172.18.20.49:2379, false

建議直接使用 etcdctl.sh

注意：etcdadm 創(chuàng)建的證書期限為一年。需要一年更換一次，這對生產(chǎn)環(huán)境來說很不友好。目前有兩個解決方案

• 1、二進制部署 etcd，使用openssl 生成證書

• 2、修改etcdadm源碼

  下載源碼

  git clone https://github.com/kubernetes-sigs/etcdadm.git
  

修改 certs/pkiutil/pki_helpers.go 文件

certificateValidity =time.Hour * 24 * 365 * 10 # 10年期限

const (
        // PrivateKeyBlockType is a possible value for pem.Block.Type.
        PrivateKeyBlockType = "PRIVATE KEY"
        // PublicKeyBlockType is a possible value for pem.Block.Type.
        PublicKeyBlockType = "PUBLIC KEY"
        // CertificateBlockType is a possible value for pem.Block.Type.
        CertificateBlockType = "CERTIFICATE"
        // RSAPrivateKeyBlockType is a possible value for pem.Block.Type.
        RSAPrivateKeyBlockType = "RSA PRIVATE KEY"
        rsaKeySize             = 2048
        certificateValidity    = time.Hour * 24 * 365 * 10
)

編譯（需要golang 1.15 以上版本）

make

  編譯完成后使用 etcdadm 重新生成集群

3.2、二進制部署

太麻煩了，網(wǎng)上找資料吧。ectdadm非常nice

4、安裝 docker 和 containerd (所有節(jié)點)

1.20 版本模式使用 containerd ，可不用安裝docker

添加 yum 源

# yum install -y yum-utils
# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安裝

yum install -y containerd.io

生成默認配置

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

添加私有鏡像倉庫

containerd 修改 config.toml 配置

[plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
    # 阿里云私有鏡像倉庫
        [plugins."io.containerd.grpc.v1.cri".cri.registry.mirrors."registry-vpc.cn-shenzhen.aliyuncs.com"]
          endpoint = ["http://registry-vpc.cn-shenzhen.aliyuncs.com"]

      [plugins."io.containerd.grpc.v1.cri".registry.auths]
        [plugins."io.containerd.grpc.v1.cri".registry.auths."registry-vpc.cn-shenzhen.aliyuncs.com"]
          username = "username"
          password = "password

啟動

systemctl restart containerd
systemctl enable containerd

5、使用kubeadm部署高可用集群

5.1、安裝kubeadm

yum源

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安裝

yum install -y kubeadm kubelet kubectl

無法使用kubeadm啟動kubelet時，需要修改 kubelet 使用 container

# vim /usr/lib/systemd/system/kubelet.services.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

5.2、初始化集群

kubeadm默認會創(chuàng)建一個堆疊式的etcd集群，并不需要另外部署

kubeadm init --control-plane-endpoint "k8s-master-slb:6443" --upload-certs --node-name "k8s-master1"

參數(shù)解析

• --control-plane-endpoint apiserver集群地址
• --upload-certs 證書

由于國內(nèi)無法訪問google的鏡像地址，這里需要使用國內(nèi)的地址來下載鏡像，下載完成后需要更新 tag 為 k8s.gcr.io

5.3、使用外部 etcd 初始化集群

創(chuàng)建 kubeadm 初始化配置文件

# 生成 kubeadm 默認初始化模板
kubeadm config print init-defaults > kubeadm-config.yaml

kubeadm-config 使用說明 https://blog.51cto.com/foxhound/2517491?source=dra

修改

cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
localAPIEndpoint:
  advertiseAddress: 192.168.200.125
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    runtime-cgroups: /system.slice/containerd.service
    kubelet-cgroups: /systemd/system.slice
    container-runtime: remote  
    #container-runtime-endpoint: unix:///var/run/docker.sock
    container-runtime-endpoint: unix:///run/containerd/containerd.sock
    cgroup-driver: systemd
  #criSocket: /var/run/docker.sock
  criSocket: /run/containerd/containerd.sock
  name: k8s-master1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.4
networking:
  dnsDomain: cluster.local
  podSubnet: 10.10.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}
# 負載均衡地址
controlPlaneEndpoint: "k8s-master-slb:7443"
apiServer:
  timeoutForControlPlane: 10m0s
certificatesDir: /etc/kubernetes/pki
clusterName: alw-cluster
controllerManager: {}
dns:
  type: CoreDNS
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
etcd:
    external:
        endpoints:
        - https://etcd-1:2379
        - https://etcd-2:2379
        - https://etcd-3:2379
        caFile: /etc/etcd/pki/ca.crt
        certFile: /etc/etcd/pki/apiserver-etcd-client.crt
        keyFile: /etc/etcd/pki/apiserver-etcd-client.key
EOF

下載鏡像

kubeadm config images pull --config kubeadm-config.yaml

下載鏡像腳本

#!/bin/bash

images=(
kube-apiserver:v1.20.4
kube-controller-manager:v1.20.4
kube-scheduler:v1.20.4
kube-proxy:v1.20.4
pause:3.2
etcd:3.4.13-0
coredns:1.7.0)

for image in ${images[@]}; do
        # docker
        #docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/${image}
        #docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image}
        #docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/${image}
        # cri
        ctr -n k8s.io i tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image}
done

初始化

kubeadm init --config kubeadm-config.yaml  --upload-certs

初始化成功會出現(xiàn)以下信息

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 \
    --control-plane --certificate-key bacab8cd43592812f0e3a186aaa615463c87e9280c0e2ae951b54b138325537d

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 

然后集群二和集群三 運行命令加入

kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 \
    --control-plane --certificate-key bacab8cd43592812f0e3a186aaa615463c87e9280c0e2ae951b54b138325537d

查看集群

# kubectl get nodes
NAME          STATUS     ROLES                  AGE     VERSION
k8s-master1   NotReady   control-plane,master   11m     v1.20.5
k8s-master2   NotReady   control-plane,master   8m54s   v1.20.5
k8s-master3   NotReady   control-plane,master   8m31s   v1.20.5

由于還未部署網(wǎng)絡(luò)插件（cni）,狀態(tài)還是 NotReady

• 注意：使用阿里云 SLB 時，由于apiserver 還未啟動，所以此時 6443 端口并未監(jiān)聽，使用 SLB地址時會無法安裝成功。修改成當(dāng)前服務(wù)器地址安裝成功后再切換 HOST 地址為 SLB地址

  本機地址 172.18.20.44
  負載均衡地址： 172.18.20.55
  172.18.20.44 k8s-master-slb
  

• 注意：token有效期為24小時，失效后請在主節(jié)點使用以下命令重新生成

kubeadm token create --print-join-command

5.4、部署工作節(jié)點

安裝 containerd ,kubeadm,kubelet,kubectl 等

工作節(jié)點運行 join

kubeadm join k8s-master-slb:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:12b4fe0053bafd6b1b0e05482912b44ddcf88d1d1429e3c611d109ad5bf93ac0

master 查看節(jié)點

# kubectl get nodes
NAME          STATUS     ROLES                  AGE     VERSION
k8s-master1   Ready      control-plane,master   6m28s   v1.20.5
k8s-master2   NotReady   control-plane,master   3m25s   v1.20.5
k8s-master3   NotReady   control-plane,master   3m11s   v1.20.5
k8s-worker1   NotReady   <none>                 2m28s   v1.20.5
k8s-worker2   NotReady   <none>                 4s      v1.20.5

由于沒有部署 CNI 網(wǎng)絡(luò)插件，狀態(tài)還是 NotReady

5.5、部署 CNI 網(wǎng)絡(luò)插件

master 節(jié)點運行

需確保kube-flannel.yml文件里的 "Network": "10.10.0.0/16"IP內(nèi)容與 kube-controller-manager.conf 配置的 --cluster-cidr 一致

即 kubeadm-config 里的 podSubnet

# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# kubectl apply -f kube-flannel.yml
# kubectl get pods -n kube-system

再次查看節(jié)點

# kubectl get nodes
NAME          STATUS   ROLES                  AGE   VERSION
k8s-master1   Ready    control-plane,master   18h   v1.20.5
k8s-master2   Ready    control-plane,master   18h   v1.20.5
k8s-master3   Ready    control-plane,master   18h   v1.20.5
k8s-worker1   Ready    <none>                 18h   v1.20.5
k8s-worker2   Ready    <none>                 18h   v1.20.5

5.6、Kubelet驅(qū)逐策略優(yōu)化

修改工作節(jié)點kubelet啟動參數(shù)，更改Pod驅(qū)逐策略

vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 
Environment="EVICTION_HARD=--eviction-hard=memory.available<2Gi,nodefs.available<5Gi,imagefs.available<100Gi"
Environment="EVICTION_RECLAIM=--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=1Gi,imagefs.available=2Gi"

重啟kubelet容器，并查看kubelet進程啟動參數(shù)

systemctl daemon-reload && systemctl restart kubelet

6、單獨部署coredns

不依賴kubeadm的方式，適用于不是使用kubeadm創(chuàng)建的k8s集群，或者kubeadm初始化集群之后，刪除了dns相關(guān)部署。

# 在calico網(wǎng)絡(luò)中也配置一個coredns # 10.96.0.10 為k8s官方指定的kube-dns地址

mkdir coredns && cd coredns
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
chmod +x deploy.sh
./deploy.sh -i 10.96.0.10 > coredns.yml

kubectl apply -f coredns.yml

# 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system

7、Kubernetes 云管理控制器

自建 k8s 想要使用阿里云的負載均衡、存儲等服務(wù)時，需要部署阿里云提供的組件 cloud-controller-manager

github：https://github.com/kubernetes/cloud-provider-alibaba-cloud

幫助文檔：https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/getting-started.md

7.1、安裝Alibaba CloudProvider 組件

修改 kubelet 啟動服務(wù)

修改kubelet 啟動參數(shù)，添加 --cloud-provider=external，并且在kubelet中添加

--hostname-override=${REGION_ID}.${INSTANCE_ID} --provider-id=${REGION_ID}.${INSTANCE_ID}

獲取 region-id 和 instance-id

echo `curl -s http://100.100.100.200/latest/meta-data/region-id`.`curl -s http://100.100.100.200/latest/meta-data/instance-id`

# vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cloud-provider=external --hostname-override=cn-shenzhen.xxxxxxx --provider-id=cn-shenzhen.xxxxxxx

配置阿里云 AccessKeyID，AccessKeySecret

AccessKey & AccessKeySecret 必須以 base64 方式

# base64 AccessKey & AccessKeySecret
$ echo -n "$AccessKeyID" |base64
$ echo -n "$AcceessKeySecret"|base64

$ cat <<EOF >cloud-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-config
  namespace: kube-system
data:
  cloud-config.conf: |-
    {
        "Global": {
            "accessKeyID": "$your-AccessKeyID-base64",
            "accessKeySecret": "$your-AccessKeySecret-base64"
        }
    }
EOF

$ kubectl create -f cloud-config.yaml

添加 kubeconfig 配置文件(所有master節(jié)點)

vim /etc/kubernetes/cloud-controller-manager.conf

kind: Config
contexts:
- context:
    cluster: alw-cluster
    user: system:cloud-controller-manager
  name: system:cloud-controller-manager@alw-cluster
current-context: system:cloud-controller-manager@alw-cluster
users:
- name: system:cloud-controller-manager
  user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: $CA_DATA
    server: https://172.18.20.44:6443
  name: alw-cluster

$CA_DATA 可以使用命令 cat /etc/kubernetes/pki/ca.crt|base64 -w 0 獲取。

server： 修改為本機IP

編寫 cloud-controller-manager.yaml 文件

也可以使用官方推薦的更詳細配置 cloud-controller-manager.yml

# vim cloud-controller-manager.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloud-controller-manager
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:cloud-controller-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: cloud-controller-manager
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: cloud-controller-manager
    tier: control-plane
  name: cloud-controller-manager
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: cloud-controller-manager
      tier: control-plane
  template:
    metadata:
      labels:
        app: cloud-controller-manager
        tier: control-plane
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: cloud-controller-manager
      tolerations:
        - effect: NoSchedule
          operator: Exists
          key: node-role.kubernetes.io/master
        - effect: NoSchedule
          operator: Exists
          key: node.cloudprovider.kubernetes.io/uninitialized
      nodeSelector:
        node-role.kubernetes.io/master: ""
      containers:
        - command:
          -  /cloud-controller-manager
          - --kubeconfig=/etc/kubernetes/cloud-controller-manager.conf
          - --address=127.0.0.1
          - --allow-untagged-cloud=true
          - --leader-elect=true
          - --cloud-provider=alicloud     # Add your own cloud provider here!
          - --use-service-account-credentials=true
          - --cloud-config=/etc/kubernetes/config/cloud-config.conf
          - --configure-cloud-routes=true
          - --allocate-node-cidrs=true
          - --route-reconciliation-period=3m
          # replace ${cluster-cidr} with your own cluster cidr
          - --cluster-cidr=10.10.0.0/16
          image: registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.339-g9830b58-aliyun
          livenessProbe:
            failureThreshold: 8
            httpGet:
              host: 127.0.0.1
              path: /healthz
              port: 10258
              scheme: HTTP
            initialDelaySeconds: 15
            timeoutSeconds: 15
          name: cloud-controller-manager
          resources:
            requests:
              cpu: 200m
          volumeMounts:
            - mountPath: /etc/kubernetes/
              name: k8s
            - mountPath: /etc/ssl/certs
              name: certs
            - mountPath: /etc/pki
              name: pki
            - mountPath: /etc/kubernetes/config
              name: cloud-config
      hostNetwork: true
      volumes:
        - hostPath:
            path: /etc/kubernetes
          name: k8s
        - hostPath:
            path: /etc/ssl/certs
          name: certs
        - hostPath:
            path: /etc/pki
          name: pki
        - configMap:
            defaultMode: 420
            items:
              - key: cloud-config.conf
                path: cloud-config.conf
            name: cloud-config
          name: cloud-config

配置解析

• --cloud-provider=alicloud 云服務(wù)商
• --cluster-cidr 集群 pod 地址

運行

# kubectl apply -f cloud-controller-manager.yaml
# kubectl get pods -n kube-system 
NAME                                  READY   STATUS    RESTARTS   AGE
cloud-controller-manager-7jbzc        1/1     Running   0          109m
cloud-controller-manager-dfpkv        1/1     Running   0          109m
cloud-controller-manager-lqvtz        1/1     Running   4          109m

完成組件的部署后，接下來就可以使用阿里云的負載均衡了

8、部署 Ingress-nginx

安裝 ingress-nginx 控制器

下載：

wget  https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.45.0/deploy/static/provider/cloud/deploy.yaml

國內(nèi)無法訪問 google 的鏡像倉庫，需自己推一個到阿里云鏡像倉庫并設(shè)置為公有

# docker pull k8s.gcr.io/ingress-nginx/controller:v0.45.0
# docker tag k8s.gcr.io/ingress-nginx/controller:v0.45.0 registry.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
# docker login --username=**** --password=**** registry.cn-shenzhen.aliyuncs.com
# docker push registry.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0

修改配置

...
apiVersion: apps/v1
kind: Deployment
    ……
        # 國內(nèi)無法訪問 google 的鏡像倉庫，需自己推一個到阿里云鏡像倉庫并設(shè)置為公有
        image: registry-vpc.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
    ……

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-3.27.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.45.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
  annotations:
    # 指明SLB實例地址類型為私網(wǎng)類型
    # service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet
    # 修改為您的私網(wǎng)SLB實例ID
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: *********-cn-shenzhen-st3-a01
    # 是否自動創(chuàng)建SLB端口監(jiān)聽（會覆寫已有端口監(jiān)聽），也可手動創(chuàng)建端口監(jiān)聽
    #service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: 'true'
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller

運行

# kubectl apply -f deploy.yaml

9、部署Aliyun存儲插件 csi

9.1、官方ack csi插件

阿里云官方文檔：https://help.aliyun.com/document_detail/134722.html?spm=a2c4g.11186623.6.822.7c525ccfsHWlPe

github：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver

RBAC 安裝：

下載RBAC配置文件到操作機，并部署：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/rbac.yaml

執(zhí)行：

$ kubectl apply -f rbac.yaml

CSI-Plugin 安裝：

下載普通模版

1. 下載模板：

下載最新版本的CSI Plugin部署模板：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/ack/csi-plugin.yaml

將部署模板下載到您的操作機，并保存（csi-plugin.yaml）。

2. 適配模板并部署：

根據(jù)集群所在的Region修改模板中的鏡像地址。例如：如果是cn-shenzhen的集群

則將 registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0 中的：

registry 改為 registry-vpc

cn-hangzhou 改為 cn-shenzhen

即：registry-vpc.cn-shenzhen.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0

模板中的其他鏡像也是如此更新；

執(zhí)行部署：

$ kubectl apply -f csi-plugin.yaml

3. 檢查安裝情況：

$ kubectl get pod -nkube-system | grep csi-plugin

$ kubectl describe ds csi-plugin -nkube-system | grep Image

CSI-Provisioner 安裝：

1. 下載模板：

下載最新版本的CSI Provisioner部署模板：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/ack/csi-provisioner.yaml

將部署模板下載到您的操作機，并保存（csi-provisioner.yaml）。

2. 適配模板并部署：

根據(jù)集群所在的Region修改模板中的鏡像地址。例如：如果是cn-beijing的集群

則將 registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v1.6.0-e360c7e43-aliyun 中的：

registry 改為 registry-vpc

cn-hangzhou 改為 cn-shenzhen

即：registry-vpc.cn-shenzhen.aliyuncs.com/acs/csi-provisioner:v1.6.0-e360c7e43-aliyun

模板中的其他鏡像也是如此更新；

執(zhí)行部署：

$ kubectl apply -f csi-provisioner.yaml

3. 檢查安裝情況：

$ kubectl get pod -nkube-system | grep csi-provisioner

$ kubectl describe deploy csi-provisioner -nkube-system | grep Image

10、部署Dashboard

下載部署文件：

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.0/aio/deploy/recommended.yaml

默認Dashboard只能集群內(nèi)部訪問，修改Service為NodePort類型，暴露到外部：

# vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard

部署：

# kubectl apply -f recommended.yaml
# kubectl get pods,svc -n kubernetes-dashboard
NAME                                             READY   STATUS    RESTARTS   AGE
pod/dashboard-metrics-scraper-6b4884c9d5-mjl66   1/1     Running   0          23h
pod/kubernetes-dashboard-7bfbb48676-frmsf        1/1     Running   0          23h

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.0.0.173   <none>        8000/TCP        23h
service/kubernetes-dashboard        NodePort    10.0.0.145   <none>        443:30001/TCP   23h

訪問地址：https://NodeIP:30001

創(chuàng)建service account并綁定默認cluster-admin管理員集群角色：

kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

查看 token：

kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

使用輸出的 token 登錄 Dashboard

結(jié)語

至此，一個生成環(huán)境可用的，基于阿里云ecs的k8s集群就搭建完成。但集群的搭建只是基礎(chǔ)，后續(xù)的維護使用才是重點，包括prometheus監(jiān)控，istio等