前言
Kong是一款功能強大，使用方便，性能優異的網關組件，可以很方便的與K8S ingress集成實現靈活的路由管理，在此之前如果您對k8s網絡對外映射方面不十分熟悉建議先看下這兩篇文章：
http://www.lxweimin.com/p/189fab1845c5/
http://www.lxweimin.com/p/97dd4d59ac5a
基礎概念
k8s對外暴露服務最簡單的模式是通過NodePort直接暴露主機30000以上的一個端口對應一個服務使用，這種模式缺點是占用端口，不能靈活控制。
k8s提供了ingress方案對服務進行統一入口管理，其包含兩大組件：
- ingress：負載管理路由規則，類似于nginx的conf文件，或者您可以直接理解為系統的hosts文件，其更新添加可以通過yaml文件形式由k8s部署。
- ingress controller：負責對外提供入口，簡單說就是網關的實現。
k8s設計時，默認不提供具體的ingress controller實現，而是留給第三方集成，市面上常用的第三方網關組件會對k8s進行適配，網關組件通過與kubernetes API交互，能夠動態的去感知集群中Ingress規則變化，然后讀取規則并按照它自己的模板生成自己的配置規則加載使用；您可以理解為ingress controller是k8s定義的抽象類，而各網關組件是對他的具體實現。
這部分您可以參考這篇詳細了解下ingress controller的選型https://www.cnblogs.com/upyun/p/12372107.html
而本文我們采用的是kong網關組件實現。
1.安裝PostgreSql
指定一臺服務器，然后下載鏡像，我們選擇9.5版本（kong支持9.4以上版本的pg數據庫）
docker pull docker.mirrors.ustc.edu.cn/library/postgres:9.5 #獲取鏡像
mkdir /data/postgresql #創建數據目錄
chmod 777 /data/postgresql #授權目錄
docker run -p 5432:5432 -v /data/postgresql:/var/lib/postgresql/data -e POSTGRES_PASSWORD=123456 -e TZ=PRC -d --name=postgres postgres:9.5
參數說明:
-p端口映射
-v將數據存到宿主機的映射目錄
-e POSTGRES_PASSWORD 密碼（默認用戶名postgres）
-e TZ=PRC時區，中國
-d后臺運行
--name容器名稱
創建用戶及kong數據庫
進入容器內
docker exec -it postgres /bin/bash
su root
su - postgres #切換帳戶
psql #輸入psql
create user kong with password '123456';
create database kong owner kong ; #創建數據庫指定所屬者
\l; # \L查看數據庫
創建kong namespace供后面各組件統一使用
kong-namespaces.yaml
apiVersion: v1
kind: Namespace
metadata:
name: kong
kubectl apply -f kong-namespaces.yaml 創建命名空間
master節點我們創建一個數據庫連接
postgres-service.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: my-postgres
namespace: kong
subsets:
- addresses:
- ip: 192.168.0.230
ports:
- port: 5432
---
apiVersion: v1
kind: Service
metadata:
name: my-postgres
namespace: kong
spec:
type: NodePort
ports:
- port: 5432
protocol: TCP
targetPort: 5432
nodePort: 30432
kubectl apply -f postgres-service.yaml 創建kong postgresql的連接
創建連接的目的是我們可以使用serviceName連接數據庫，通常我們會建議將db/es/redis/mq/等非k8s必須資源獨立于k8s的集群外部署，降低k8s管理的復雜度；而這種獨立在外部部署的資源建議添加一個k8s的endpoint/service指向來描述其調用地址，便于靈活管理及調用方便。
2.kong安裝
為kong節點打標簽
生產環境我們通常會為kong部署多個節點，這些節點通過vip實現NLB方案，而k8s部署默認會隨機分配到某一個節點部署pod，為了保證讓k8s始終將kong的pod分配到特定的有vip的節點，我們需要為運行kong的虛機節點打上標簽，kong根據標簽部署在這些機器，沒打標簽的不會部署。
kubectl get nodes --show-labels #查看標簽
kubectl label k8s-node1 app=ingress-kong #打上這個標簽供后面使用（key/value是我們自定義的）
-----
kubectl label k8s-node1 node=gateway --overwrite #修改/覆蓋標簽
kubectl label k8s-node1 key- #刪除label
-----
創建kong-gateway.yaml
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongconsumers.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .username
description: Username of a Kong Consumer
name: Username
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
group: configuration.konghq.com
names:
kind: KongConsumer
plural: kongconsumers
shortNames:
- kc
scope: Namespaced
validation:
openAPIV3Schema:
properties:
credentials:
items:
type: string
type: array
custom_id:
type: string
username:
type: string
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongcredentials.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .type
description: Type of credential
name: Credential-type
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
- JSONPath: .consumerRef
description: Owner of the credential
name: Consumer-Ref
type: string
group: configuration.konghq.com
names:
kind: KongCredential
plural: kongcredentials
scope: Namespaced
validation:
openAPIV3Schema:
properties:
consumerRef:
type: string
type:
type: string
required:
- consumerRef
- type
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongingresses.configuration.konghq.com
spec:
group: configuration.konghq.com
names:
kind: KongIngress
plural: kongingresses
shortNames:
- ki
scope: Namespaced
validation:
openAPIV3Schema:
properties:
proxy:
properties:
connect_timeout:
minimum: 0
type: integer
path:
pattern: ^/.*$
type: string
protocol:
enum:
- http
- https
- grpc
- grpcs
type: string
read_timeout:
minimum: 0
type: integer
retries:
minimum: 0
type: integer
write_timeout:
minimum: 0
type: integer
type: object
route:
properties:
headers:
additionalProperties:
items:
type: string
type: array
type: object
https_redirect_status_code:
type: integer
methods:
items:
type: string
type: array
preserve_host:
type: boolean
protocols:
items:
enum:
- http
- https
- grpc
- grpcs
type: string
type: array
regex_priority:
type: integer
strip_path:
type: boolean
upstream:
properties:
algorithm:
enum:
- round-robin
- consistent-hashing
- least-connections
type: string
hash_fallback:
type: string
hash_fallback_header:
type: string
hash_on:
type: string
hash_on_cookie:
type: string
hash_on_cookie_path:
type: string
hash_on_header:
type: string
healthchecks:
properties:
active:
properties:
concurrency:
minimum: 1
type: integer
healthy:
properties:
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
successes:
minimum: 0
type: integer
type: object
http_path:
pattern: ^/.*$
type: string
timeout:
minimum: 0
type: integer
unhealthy:
properties:
http_failures:
minimum: 0
type: integer
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
tcp_failures:
minimum: 0
type: integer
timeout:
minimum: 0
type: integer
type: object
type: object
passive:
properties:
healthy:
properties:
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
successes:
minimum: 0
type: integer
type: object
unhealthy:
properties:
http_failures:
minimum: 0
type: integer
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
tcp_failures:
minimum: 0
type: integer
timeout:
minimum: 0
type: integer
type: object
type: object
type: object
slots:
minimum: 10
type: integer
type: object
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongplugins.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .plugin
description: Name of the plugin
name: Plugin-Type
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
- JSONPath: .disabled
description: Indicates if the plugin is disabled
name: Disabled
priority: 1
type: boolean
- JSONPath: .config
description: Configuration of the plugin
name: Config
priority: 1
type: string
group: configuration.konghq.com
names:
kind: KongPlugin
plural: kongplugins
shortNames:
- kp
scope: Namespaced
validation:
openAPIV3Schema:
properties:
config:
type: object
disabled:
type: boolean
plugin:
type: string
protocols:
items:
enum:
- http
- https
- tcp
- tls
type: string
type: array
run_on:
enum:
- first
- second
- all
type: string
required:
- plugin
version: v1
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kong-serviceaccount
namespace: kong
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: kong-ingress-clusterrole
rules:
- apiGroups:
- ""
resources:
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- configuration.konghq.com
resources:
- kongplugins
- kongcredentials
- kongconsumers
- kongingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resourceNames:
- ingress-controller-leader-kong
resources:
- configmaps
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kong-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kong-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: kong-serviceaccount
namespace: kong
---
apiVersion: v1
data:
servers.conf: |
# Prometheus metrics server
server {
server_name kong_prometheus_exporter;
listen 0.0.0.0:9542; # can be any other port as well
access_log off;
location /metrics {
default_type text/plain;
content_by_lua_block {
local prometheus = require "kong.plugins.prometheus.exporter"
prometheus:collect()
}
}
location /nginx_status {
internal;
stub_status;
}
}
# Health check server
server {
server_name kong_health_check;
listen 0.0.0.0:9001; # can be any other port as well
access_log off;
location /health {
return 200;
}
}
kind: ConfigMap
metadata:
name: kong-server-blocks
namespace: kong
---
apiVersion: v1
kind: Service
metadata:
annotations:
#service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
#service.beta.kubernetes.io/aws-load-balancer-type: nlb
name: kong-proxy
namespace: kong
spec:
#externalTrafficPolicy: Local
ports:
- name: proxy
port: 80
protocol: TCP
targetPort: 8000
- name: proxy-ssl
port: 443
protocol: TCP
targetPort: 8443
selector:
app: ingress-kong
type: NodePort
---
apiVersion: v1
kind: Service
metadata:
name: kong-ingress-controller
namespace: kong
spec:
type: NodePort
ports:
- name: kong-admin
port: 8001
targetPort: 8001
nodePort: 30001
protocol: TCP
selector:
app: ingress-kong
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: ingress-kong
name: ingress-kong
namespace: kong
spec:
replicas: 1
selector:
matchLabels:
app: ingress-kong
template:
metadata:
annotations:
prometheus.io/port: "9542"
prometheus.io/scrape: "true"
traffic.sidecar.istio.io/includeInboundPorts: ""
labels:
app: ingress-kong
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
node: kong
#gateway: web
containers:
- env:
- name: KONG_DATABASE
value: postgres
- name: KONG_PG_HOST
value: my-postgres.kong
- name: KONG_PG_PASSWORD
value: "123456" #注意修改
- name: KONG_NGINX_WORKER_PROCESSES
value: "8"
- name: KONG_NGINX_HTTP_INCLUDE
value: /kong/servers.conf
- name: KONG_ADMIN_ACCESS_LOG
value: /dev/stdout
- name: KONG_ADMIN_ERROR_LOG
value: /dev/stderr
- name: KONG_ADMIN_LISTEN
value: 0.0.0.0:8001, 0.0.0.0:8444 ssl
- name: KONG_PROXY_LISTEN
value: 0.0.0.0:80, 0.0.0.0:443 ssl http2
image: 192.168.0.230:8083/kong/kong:1.3.0 #注意修改
securityContext:
runAsUser: 0
#capabilities:
privileged: true
# add:
# - NET_BIND_SERVICE
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- kong quit
livenessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 9001
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: proxy
ports:
- containerPort: 80
name: proxy
protocol: TCP
- containerPort: 443
name: proxy-ssl
protocol: TCP
- containerPort: 9542
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 9001
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
#securityContext:
# runAsUser: 0
volumeMounts:
- mountPath: /kong
name: kong-server-blocks
- args:
- /kong-ingress-controller
- --kong-url=https://localhost:8444
- --admin-tls-skip-verify
- --publish-service=kong/kong-proxy
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: 192.168.0.230:8083/kong/kong-ingress-controller:0.6.2 #注意修改
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: ingress-controller
ports:
- containerPort: 8080
name: webhook
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
initContainers:
- command:
- /bin/sh
- -c
- while true; do kong migrations list; if [[ 0 -eq $? ]]; then exit 0; fi;
sleep 2; done;
env:
- name: KONG_PG_HOST
value: my-postgres.kong
- name: KONG_PG_PASSWORD
value: kong
image: 192.168.0.230:8083/kong/kong:1.3.0 #注意修改
name: wait-for-migrations
serviceAccountName: kong-serviceaccount
volumes:
- configMap:
name: kong-server-blocks
name: kong-server-blocks
---
apiVersion: batch/v1
kind: Job
metadata:
name: kong-migrations
namespace: kong
spec:
template:
metadata:
name: kong-migrations
spec:
containers:
- command:
- /bin/sh
- -c
- kong migrations bootstrap
env:
- name: KONG_PG_PASSWORD
value: "123456" #注意修改
- name: KONG_PG_HOST
value: my-postgres.kong
- name: KONG_PG_PORT
value: "5432"
image: 192.168.0.230:8083/kong/kong:1.3.0 #注意修改
name: kong-migrations
initContainers:
- command:
- /bin/sh
- -c
- until nc -zv $KONG_PG_HOST $KONG_PG_PORT -w1; do echo 'waiting for db';
sleep 1; done
env:
- name: KONG_PG_HOST
value: my-postgres.kong
- name: KONG_PG_PORT
value: "5432"
image: busybox:latest
name: wait-for-postgres
restartPolicy: OnFailure
安裝kongA
kongA是kong的一個開源UI管理組件，使用kongA可以以WEB形式直觀的查看與管理kong的路由規則，該組件為選裝。
說明：集成了k8s ingress后的kong，不建議使用kongA上進行路由的管理，應該使用k8s ingress進行管理路由然后提供給kong使用。
---
***konga-deploy.yaml***
#deploy
apiVersion: apps/v1
kind: Deployment
metadata:
name: kong-konga
namespace: kong
spec:
selector:
matchLabels:
app: kong-konga
replicas: 1
template:
metadata:
labels:
app: kong-konga
spec:
#inodeSelector:
# node: worker
containers:
- name: kong-konga
image: pantsel/konga:0.14.7
imagePullPolicy: IfNotPresent
env:
- name: DB_ADAPTER
value: postgres
- name: DB_HOST
#服務名.命名空間
value: my-postgres.kong
- name: DB_PORT
value: "5432"
- name: DB_USER
value: postgres
- name: DB_DATABASE
value: konga
- name: DB_PASSWORD
value: "123456" #注意修改
- name: NODE_ENV
#value: production
value: development
- name: TZ
value: Asia/Shanghai
ports:
- containerPort: 1337
---
#service
apiVersion: v1
kind: Service
metadata:
name: kong-konga
namespace: kong
spec:
ports:
- port: 80
protocol: TCP
targetPort: 1337
nodePort: 31337
type: NodePort
selector:
app: kong-konga
---
截止目前一共創建了4個yaml文件，您可以根據自己環境實際情況修改鏡像地址，IP及密碼信息。
我們按順序分別執行4個yaml文件
kubectl apply -f kong-namespaces.yaml #創建kong命名空間
kubectl apply -f postgres-service.yaml #創建kong postgresql的連接
kubectl apply -f kong-gateway.yaml #創建kong網關，最重要的一步
kubectl apply -f konga-deploy.yaml #創建kongA管理
3.kong網關的使用
基本測試與配置
部署完畢后，我們測試下：
http://192.168.0.137 kong網關安裝的節點
瀏覽器返回如下：
{"message":"no Route matched with those values"}
該信息是由kong返回的，說明kong已經安裝好，只是沒有配置路由，kong不知道該如何路由。
接下來我們訪問KongA，配置下關聯，出現如下界面，我們注冊一個管理帳戶，帳戶名稱隨便輸入。
http://192.168.0.137:31337/register
首次訪問注冊管理帳戶
注冊后，登錄會提示綁定kong，注意kong admin URL需要輸入內部地址
kongA綁定kong-admin
如果綁定成功，我們是能夠看到kongA獲取到kong的版本號，然后我們點擊列表中的ACTIVE啟用該連接。
激活后，我們頁面左菜單會出多一些管理菜單，我們點擊ROUTES菜單，查看路由
路由管理
由于我們之前已經創建了一些ingress路由，此時已經被kong ingress自動采集上來了。
新建路由規則驗證
我們的域名已經提前添加了A記錄指向kong的公網服務器：api.xxxx.cn，如果您在本地測試建議用hosts文件模擬。
創建一個demo的部署，鏡像是我們之前已經做好的demo程序（.net core寫的）
重點是ingress部分的配置
#kong-netcore-demo.yaml 測試程序部署
#create namespace
apiVersion: v1
kind: Namespace
metadata:
name: mydemos
spec:
finalizers:
- kubernetes
---
#deploy
apiVersion: apps/v1
kind: Deployment
#kind: StatefulSet
metadata:
name: netcore-02-blue
namespace: mydemos
spec:
selector:
matchLabels:
app: netcore-02-blue
replicas: 1
template:
metadata:
labels:
app: netcore-02-blue
spec:
containers:
- name: netcore-02-blue
image: 192.168.0.230:8083/my/netcore-02:2.0.7
imagePullPolicy: Always
env:
- name: TZ
value: Asia/Shanghai
ports:
- containerPort: 8020
---
#service
apiVersion: v1
kind: Service
metadata:
name: netcore-02-blue
namespace: mydemos
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8020
selector:
app: netcore-02-blue
type: NodePort
sessionAffinity: ClientIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: netcore-02-blue
namespace: mydemos
spec:
rules:
#host 網關域名或IP，這個是路由關鍵
- host: api.xxx.cn
http:
paths:
#這個也很重要，path路徑，如service1，這個不需要和服務名完全一致
- path: /netcore-02/
backend:
serviceName: netcore-02-blue
servicePort: 80
我們執行
[root@k8s-master es]# kubectl apply -f kong-netcore-demo.yaml
namespace/mydemos created
deployment.apps/netcore-02-blue created
service/netcore-02-blue created
ingress.extensions/netcore-02-blue created
[root@k8s-master es]# kubectl get pods -n mydemos
NAME READY STATUS RESTARTS AGE
netcore-02-blue-7ddc75cd5d-tfpc2 1/1 Running 0 13s
輸入網址：http://api.xxx.cn/netcore-02/default/index 注意這個路徑由三部分構成：
1:api.xxx.cn：這個是ingress里的host
2:netcore-02：這個是ingress里的path
3:default/index：這個是你程序里的api路徑，我這里默認的controller/action
api路徑訪問
另外此時刷新kongA的service/routes界面，是可以直接看到新創建的服務及路由指向，kong-ingress會自動從ingress中采集并加載，幾乎是實時的。
同理，我們可以將之前通過nginx轉發的相關域名切換到kong-ingress來映射
以apollo的portal界面轉發為例
**apollo-portal-kong.yaml**
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: apollo-config-portal
namespace: apollo
spec:
rules:
- host: config.xxx.cn #host 域名
http:
paths:
- path: / #路徑
backend:
serviceName: service-apollo-portal-server #apollo的service name
servicePort: 8070
這樣我們就可以通過config.xxx.cn訪問到apollo的portal界面
apollo portal界面
附ingress管理命令
如若ingress數據無法清除，可用以下命令清除
[root@master1 apollo]# kubectl get ingress -n kong #查看ingress
NAME HOSTS ADDRESS PORTS AGE
xxx-k8s-web k8s.xxx.cn 192.168.0.28 80 13m
xxx-konga-web konga.xxx.cn 192.168.0.137 80 18m
[root@master1 apollo]# kubectl delete ingress xxx-k8s-web -n kong #刪除
ingress.extensions "xxx-k8s-web" deleted
[root@master1 apollo]# kubectl get ingress -n kong #查看ingress
NAME HOSTS ADDRESS PORTS AGE
xxx-konga-web k8s.xxx.cn 192.168.0.28 80 13m
4.總結
本文采用kong實現了ingress controller的功能，您也可以使用其他網關實現同樣的功能；
k8s網絡體系知識不容易掌握，需要多看多思考；
ingress使用一定要理解其原理，kong讀取ingress的配置實現其controller功能，但不建議使用kong來控制ingress的配置；
