1. 前言
轉載請說明原文出處, 尊重他人勞動成果!
源碼位置: https://github.com/nicktming/istio
分支: tming-v1.3.6 (基于1.3.6版本)
本文承接上文 [istio源碼分析] istio源碼開發(fā)調(diào)試版簡單安裝 進行流量的分析.
本文需要對envoy有一個基本的認識.
2. ingress-gateway -> productpage
因為是訪問
ingress-gateway的內(nèi)部nodeport端口, 并且nodeport 31380:80, 所以kube-proxy會通過iptables轉到ingress-gateway這個pod的podIP:80上.
[root@master ~]# kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-ingressgateway-768778694-sz9kw 1/1 Running 0 7h17m
istio-sidecar-injector-84d5c488d9-jqnx9 1/1 Running 0 7h29m
2.1 查看listener
[root@master analysis]# istioctl -n istio-system proxy-config listener istio-ingressgateway-768778694-sz9kw
ADDRESS PORT TYPE
0.0.0.0 80 HTTP
0.0.0.0 15090 HTTP
[root@master analysis]# istioctl -n istio-system proxy-config listener istio-ingressgateway-768778694-sz9kw --port 80 -o json
[
{
"name": "0.0.0.0_80",
"address": {
...
},
"filterChains": [
{
"filters": [
{
"name": "envoy.http_connection_manager",
"typedConfig": {
...
"routeConfigName": "http.80"
},
...
}
}
]
}
],
"trafficDirection": "OUTBOUND"
}
]
[root@master analysis]#
可以看到當訪問
ingress-gateway的80端口時要根據(jù)路由(route)http.80進行轉發(fā).
2.2 查看route
[root@master analysis]# istioctl -n istio-system proxy-config route --name=http.80 istio-ingressgateway-768778694-sz9kw
NOTE: This output only contains routes loaded via RDS.
NAME VIRTUAL HOSTS
http.80 1
[root@master analysis]#
[root@master analysis]# istioctl -n istio-system proxy-config route --name=http.80 istio-ingressgateway-768778694-sz9kw -o json
[
{
"name": "http.80",
"virtualHosts": [
{
"name": "*:80",
"domains": [
"*",
"*:80"
],
"routes": [
{
"match": {
"path": "/productpage",
"caseSensitive": true
},
"route": {
"cluster": "outbound|9080||productpage.default.svc.cluster.local",
...
},
...
},
...
]
}
],
"validateClusters": false
}
]
[root@master analysis]#
可以看到訪問
ingress-gateway的80端口并且匹配到/productpage的時候, 請求會交給名字為outbound|9080||productpage.default.svc.cluster.local的cluster處理.
2.3 查看clusters
[root@master analysis]# istioctl -n istio-system proxy-config cluster istio-ingressgateway-768778694-sz9kw --fqdn productpage.default.svc.cluster.local -o json
[
{
"name": "outbound_.9080_._.productpage.default.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "0s"
},
"serviceName": "outbound_.9080_._.productpage.default.svc.cluster.local"
},
"connectTimeout": "10s",
"circuitBreakers": {
"thresholds": [
{
"maxRetries": 1024
}
]
}
},
{
"name": "outbound|9080||productpage.default.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "0s"
},
"serviceName": "outbound|9080||productpage.default.svc.cluster.local"
},
"connectTimeout": "10s",
"circuitBreakers": {
"thresholds": [
{
"maxRetries": 1024
}
]
}
}
]
[root@master analysis]#
可以看到名為
outbound|9080||productpage.default.svc.cluster.local的cluster會轉到serviceName, 所以從該serviceName中轉到最終的endpoints.
2.4 查看endpoint
[root@master istio]# istioctl -n istio-system proxy-config endpoint istio-ingressgateway-768778694-sz9kw | grep productpage
10.0.15.34:9080 HEALTHY OK outbound_.9080_._.productpage.default.svc.cluster.local
10.0.15.34:9080 HEALTHY OK outbound|9080||productpage.default.svc.cluster.local
可以看到名為
outbound|9080||productpage.default.svc.cluster.local的cluster有一個endpoint就是10.0.15.34:9080.
2.5 總結
所以訪問流程如下:
traffic-flow.png
3. proxy_init程序
在每個需要注入
sidecar的pod中, 都會有一個init程序, 該程序其實就是執(zhí)行一些iptables規(guī)則, 用于攔截進出該pod的網(wǎng)絡流量.
[root@master ~]# docker ps | grep productpage
fddd22d5f896 5cb4b1355aa0 "/usr/local/bin/pi..." 24 hours ago Up 24 hours k8s_istio-proxy_productpage-v1-8554d58bff-d7j8d_default_fddfd3b3-1338-4e89-979c-33775b9d91be_0
6742b3852dd3 8e754b2df1fe "/bin/sh -c 'tail ..." 24 hours ago Up 24 hours k8s_productpage_productpage-v1-8554d58bff-d7j8d_default_fddfd3b3-1338-4e89-979c-33775b9d91be_0
461d07931e67 k8s.gcr.io/pause:3.1 "/pause" 24 hours ago Up 24 hours k8s_POD_productpage-v1-8554d58bff-d7j8d_default_fddfd3b3-1338-4e89-979c-33775b9d91be_0
[root@master ~]# docker exec -it -u root --privileged=true fddd22d5f896 sh
# bash
root@productpage-v1-8554d58bff-d7j8d:/# iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 43439 packets, 2606K bytes)
pkts bytes target prot opt in out source destination
43441 2606K ISTIO_INBOUND tcp -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 43441 packets, 2606K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 6683 packets, 438K bytes)
pkts bytes target prot opt in out source destination
5453 327K ISTIO_OUTPUT tcp -- * * 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT 6683 packets, 438K bytes)
pkts bytes target prot opt in out source destination
Chain ISTIO_INBOUND (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
43439 2606K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:15020
2 120 ISTIO_IN_REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0
Chain ISTIO_IN_REDIRECT (2 references)
pkts bytes target prot opt in out source destination
2 120 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 redir ports 15006
Chain ISTIO_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * lo 127.0.0.6 0.0.0.0/0
0 0 ISTIO_IN_REDIRECT all -- * lo 0.0.0.0/0 !127.0.0.1
5453 327K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1337
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 owner GID match 1337
0 0 RETURN all -- * * 0.0.0.0/0 127.0.0.1
0 0 ISTIO_REDIRECT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ISTIO_REDIRECT (1 references)
pkts bytes target prot opt in out source destination
0 0 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 redir ports 15001
root@productpage-v1-8554d58bff-d7j8d:/#
關于此部分可以參考 https://blog.csdn.net/luo15242208310/article/details/99290541 和 https://jimmysong.io/posts/envoy-sidecar-routing-of-istio-service-mesh-deep-dive/.
3.1 podIp -> localhost
3.1.1 查看listener
由于每個
istio pod都會有iptables進行攔截, 從上面可以知道進入的流量會被15006攔截.
[root@master analysis]# istioctl proxy-config listener productpage-v1-8554d58bff-d7j8d --port 15006 -o json
[
{
"name": "virtualInbound",
"address": {
"socketAddress": {
"address": "0.0.0.0",
"portValue": 15006
}
},
"filterChains": [
...
{
...
"filters": [
{
"name": "envoy.http_connection_manager",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
"statPrefix": "10.0.15.34_9080",
"routeConfig": {
"name": "inbound|9080|http|productpage.default.svc.cluster.local",
"virtualHosts": [
{
"name": "inbound|http|9080",
"domains": [
"*"
],
"routes": [
{
"name": "default",
"match": {
"prefix": "/"
},
"route": {
"cluster": "inbound|9080|http|productpage.default.svc.cluster.local",
"timeout": "0s",
"maxGrpcTimeout": "0s"
},
"decorator": {
"operation": "productpage.default.svc.cluster.local:9080/*"
},
...
}
]
}
],
"validateClusters": false
},
...
}
}
],
...
}
],
...
}
]
[root@master analysis]#
3.1.2 查看listener
同樣的方法查看
listener
{
"name": "inbound|9080|http|productpage.default.svc.cluster.local",
"virtualHosts": [
{
"name": "inbound|http|9080",
"domains": [
"*"
],
"routes": [
{
"name": "default",
"match": {
"prefix": "/"
},
"route": {
"cluster": "inbound|9080|http|productpage.default.svc.cluster.local",
"timeout": "0s",
"maxGrpcTimeout": "0s"
},
"decorator": {
"operation": "productpage.default.svc.cluster.local:9080/*"
},
"typedPerFilterConfig": {
"mixer": {
"@type": "type.googleapis.com/istio.mixer.v1.config.client.ServiceConfig",
"disableCheckCalls": true,
"mixerAttributes": {
"attributes": {
"destination.service.host": {
"stringValue": "productpage.default.svc.cluster.local"
},
"destination.service.name": {
"stringValue": "productpage"
},
"destination.service.namespace": {
"stringValue": "default"
},
"destination.service.uid": {
"stringValue": "istio://default/services/productpage"
}
}
}
}
}
}
]
}
],
"validateClusters": false
},
流量將被轉到
inbound|9080|http|productpage.default.svc.cluster.local這個cluster.
3.1.3 查看cluster
[root@master analysis]# istioctl proxy-config cluster productpage-v1-8554d58bff-d7j8d --fqdn productpage.default.svc.cluster.local
SERVICE FQDN PORT SUBSET DIRECTION TYPE
productpage.default.svc.cluster.local 9080 - outbound EDS
productpage.default.svc.cluster.local 9080 http inbound STATIC
[root@master analysis]#
[root@master analysis]# istioctl proxy-config cluster productpage-v1-8554d58bff-d7j8d --fqdn productpage.default.svc.cluster.local -o json
[
{
"name": "outbound|9080||productpage.default.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "0s"
},
"serviceName": "outbound|9080||productpage.default.svc.cluster.local"
},
"connectTimeout": "10s",
"circuitBreakers": {
"thresholds": [
{
"maxRetries": 1024
}
]
}
},
{
"name": "inbound|9080|http|productpage.default.svc.cluster.local",
"type": "STATIC",
"connectTimeout": "10s",
"loadAssignment": {
"clusterName": "inbound|9080|http|productpage.default.svc.cluster.local",
"endpoints": [
{
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "127.0.0.1",
"portValue": 9080
}
}
}
}
]
}
]
},
"circuitBreakers": {
"thresholds": [
{}
]
}
}
]
[root@master analysis]#
可以看到將會該
network namespace的9080端口進程服務.
4. productpage -> reviews
因為
productpage服務需要訪問reviews和details服務, 此時在該network namespace下會發(fā)起對reviews:9080和details:9080的訪問, 由于原理一樣, 就以訪問reviews:9080為例子來進行分析.
4.1 查看listener
因為出口流量會被
15001攔截
[root@master analysis]# istioctl proxy-config listener productpage-v1-8554d58bff-d7j8d --port 15001 -o json
[
{
"name": "virtualOutbound",
"address": {
"socketAddress": {
"address": "0.0.0.0",
"portValue": 15001
}
},
"filterChains": [
{
"filterChainMatch": {
"prefixRanges": [
{
"addressPrefix": "10.0.15.34",
"prefixLen": 32
}
]
},
"filters": [
{
"name": "envoy.tcp_proxy",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"statPrefix": "BlackHoleCluster",
"cluster": "BlackHoleCluster"
}
}
]
},
{
"filters": [
{
"name": "mixer",
"typedConfig": {
"@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
"transport": {
"networkFailPolicy": {
"policy": "FAIL_CLOSE",
"baseRetryWait": "0.080s",
"maxRetryWait": "1s"
},
"checkCluster": "outbound|9091||istio-policy.istio-system.svc.cluster.local",
"reportCluster": "outbound|9091||istio-telemetry.istio-system.svc.cluster.local",
"reportBatchMaxEntries": 100,
"reportBatchMaxTime": "1s"
},
"mixerAttributes": {
"attributes": {
"context.proxy_version": {
"stringValue": "1.3.0"
},
"context.reporter.kind": {
"stringValue": "outbound"
},
"context.reporter.uid": {
"stringValue": "kubernetes://productpage-v1-8554d58bff-d7j8d.default"
},
"destination.service.host": {
"stringValue": "PassthroughCluster"
},
"destination.service.name": {
"stringValue": "PassthroughCluster"
},
"source.namespace": {
"stringValue": "default"
},
"source.uid": {
"stringValue": "kubernetes://productpage-v1-8554d58bff-d7j8d.default"
}
}
},
"disableCheckCalls": true
}
},
{
"name": "envoy.tcp_proxy",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"statPrefix": "PassthroughCluster",
"cluster": "PassthroughCluster"
}
}
]
}
],
"useOriginalDst": true
}
]
[root@master analysis]#
"use_original_dst": true的意思是將請求轉發(fā)給和原始目的IP:Port匹配的listener. 參考 https://zhaohuabing.com/post/2018-09-25-istio-traffic-management-impl-intro/ 所以該請求將被轉到名字為0.0.0.0_9080的listener.
查看端口是
9080的listener.
[root@master analysis]# istioctl proxy-config listener productpage-v1-8554d58bff-d7j8d --port 9080
ADDRESS PORT TYPE
10.0.15.34 9080 HTTP // 很明顯這個是處理inbound, ingress-gateway->pod就是走這個listener
0.0.0.0 9080 TCP //處理出口的
查看詳細信息
{
"name": "0.0.0.0_9080",
"address": {
"socketAddress": {
"address": "0.0.0.0",
"portValue": 9080
}
},
"filterChains": [
{
"filterChainMatch": {
"prefixRanges": [
{
"addressPrefix": "10.0.15.34",
"prefixLen": 32
}
]
},
...
},
{
"filters": [
{
"name": "envoy.http_connection_manager",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
"statPrefix": "0.0.0.0_9080",
"rds": {
"configSource": {
"ads": {},
"initialFetchTimeout": "0s"
},
"routeConfigName": "9080"
},
...
}
]
}
],
...
}
可以看到該
listener將請求按照名為9080的route來處理, 也就是說istio將請求按照端口來進行劃分處理.
4.2 查看route
[root@master analysis]# istioctl proxy-config route productpage-v1-8554d58bff-d7j8d --name 9080 -o json
[
{
"name": "9080",
"virtualHosts": [
...
{
"name": "reviews.default.svc.cluster.local:9080",
"domains": [
"reviews.default.svc.cluster.local",
"reviews.default.svc.cluster.local:9080",
"reviews",
"reviews:9080",
"reviews.default.svc.cluster",
"reviews.default.svc.cluster:9080",
"reviews.default.svc",
"reviews.default.svc:9080",
"reviews.default",
"reviews.default:9080",
"169.169.236.122",
"169.169.236.122:9080"
],
"routes": [
{
"name": "default",
"match": {
"prefix": "/"
},
"route": {
"cluster": "outbound|9080||reviews.default.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,resource-exhausted,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"hostSelectionRetryMaxAttempts": "5",
"retriableStatusCodes": [
503
]
},
"maxGrpcTimeout": "0s"
},
...
}
]
},
...
],
"validateClusters": false
}
]
[root@master analysis]#
因為是訪問
reviews:9080, 可以看到名為reviews.default.svc.cluster.local:9080的cluster中的domains含有reviews, 所以可以match到, 所以請求將由outbound|9080||reviews.default.svc.cluster.local的cluster來找到具體的endpoint.
4.3 查看cluster 和 endpoint
[root@master analysis]# istioctl proxy-config cluster productpage-v1-8554d58bff-d7j8d --fqdn reviews.default.svc.cluster.local -o json
[
{
"name": "outbound|9080||reviews.default.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "0s"
},
"serviceName": "outbound|9080||reviews.default.svc.cluster.local"
},
"connectTimeout": "10s",
"circuitBreakers": {
"thresholds": [
{
"maxRetries": 1024
}
]
}
}
]
root@master analysis]# istioctl proxy-config endpoint productpage-v1-8554d58bff-d7j8d | grep reviews.default.svc.cluster.local
10.0.15.30:9080 HEALTHY OK outbound|9080||reviews.default.svc.cluster.local
10.0.15.31:9080 HEALTHY OK outbound|9080||reviews.default.svc.cluster.local
10.0.15.32:9080 HEALTHY OK outbound|9080||reviews.default.svc.cluster.local
可以看到該
cluster有三個endpoint可以訪問, 這個時候會根據(jù)規(guī)則訪問其中某一個endpoint.
{
"name": "outbound|9080||reviews.default.svc.cluster.local",
"addedViaApi": true,
"hostStatuses": [
{
"address": {
"socketAddress": {
"address": "10.0.15.30",
"portValue": 9080
}
...
},
{
"address": {
"socketAddress": {
"address": "10.0.15.31",
"portValue": 9080
}
},
...
},
{
"address": {
"socketAddress": {
"address": "10.0.15.32",
"portValue": 9080
}
},
...
}
]
}
關于
reviews的pod如何接受該請求, 這個需要分析reviews的envoy配置, 原理和ingress-gateway發(fā)請求訪問productpage是一樣的.
圖片.png
5. 總結
這里大致分析了流量是如何轉發(fā)和管理的, 可以看到都是一些配置信息, 那這些配置信息是如何來的呢, 很明顯是從
k8s的service,pod以及自定義資源gateway,virtualService等等中得到的, 那pilot就是做這些事情的, 把這些資源轉成envoy可以識別的配置信息. 那galley做什么呢, 負責給pilot發(fā)送gateway,virtualService等資源.
