2.1 Nginx ingress介绍
2.2 Nginx ingress安装
- 2.2.1 基础依赖环境准备
- 2.2.2 部署Ingress控制器
3.1 Ingress定义
3.2 Ingress动态配置
3.3 Ingress路径转发
3.4 Ingress虚拟主机
3.5 Ingress TLS加密
4.1 定制化参数
- 4.1.1 ConfigMap自定义参数
- 4.1.2 Annotations自定义参数
4.2 虚拟主机和路由

写在前面 #

本章介绍kubernetes系列教程的ingress概念，在kubernetes中对外暴露服务的方式有两种：service（NodePort或者外部LoadBalancer）和ingress，其中service是提供四层的负载均衡，通过iptables DNAT或lvs nat模式实现后端Pod的代理请求。如需实现http，域名，URI，证书等请求方式，service是无法实现的，需要借助于ingress来来实现，本文将来介绍ingress相关的内容。

1. Ingress简介 #

An API object that manages external access to the services in a cluster, typically HTTP. Ingress can provide load balancing, SSL termination and name-based virtual hosting.

引用官方关于ingress的介绍我们可以得知，ingress是一种通过http协议暴露kubernetes内部服务的api对象，即充当Edge Router边界路由器的角色对外基于七层的负载均衡调度机制，能够提供以下几个功能：

负载均衡，将请求自动负载均衡到后端的Pod上；
SSL加密，客户端到Ingress Controller为https加密，到后端Pod为明文的http；
基于名称的虚拟主机，提供基于域名或URI更灵活的路由方式

Ingress Controller

实现Ingress包含的组件有：

Ingress，客户端，负责定义ingress配置，将请求转发给Ingress Controller；
Ingress Controller，Ingress控制器，实现七层转发的Edge Router，通过调用k8s的api动态感知集群中Pod的变化而动态更新配置文件并重载， Controller需要部署在k8s集群中以实现和集群中的pod通信，通常以DaemonSets或Deployments的形式部署，并对外暴露80和443端口，对于DaemonSets来说，一般是以hostNetwork或者hostPort的形式暴露，Deployments则以NodePort的方式暴露，控制器的多个节点则借助外部负载均衡ExternalLB以实现统一接入；
Ingress配置规则，Controller控制器通过service服务发现机制动态实现后端Pod路由转发规则的实现；
Service，kuberntes中四层的负载均衡调度机制，Ingress借助service的服务发现机制实现集群中Pod资源的动态感知；
Pod，后端实际负责响应请求容器，由控制器如Deployment创建，通过标签Labels和service关联，服务发现。

简而言之，ingress控制器借助service的服务发现机制实现配置的动态更新以实现Pod的负载均衡机制实现，由于涉及到Ingress Controller的动态更新，目前社区Ingress Controller大体包含两种类型的控制器：

传统的七层负载均衡如Nginx，HAproxy，开发了适应微服务应用的插件，具有成熟，高性能等优点；
新型微服务负载均衡如Traefik，Envoy，Istio，专门适用于微服务+容器化应用场景，具有动态更新特点；

类型	常见类型	优点	缺点
传统负载均衡	nginx，haproxy	成熟，稳定，高性能	动态更新需reload配置文件
微服务负载均衡	Traefik，Envoy，Istio	天生为微服务而生，动态更新	性能还有待提升

2. Nginx Ingress #

2.1 Nginx ingress介绍 #

By default, pods of Kubernetes services are not accessible from the external network, but only by other pods within the Kubernetes cluster. Kubernetes has a built‑in configuration for HTTP load balancing, called Ingress, that defines rules for external connectivity to Kubernetes services. Users who need to provide external access to their Kubernetes services create an Ingress resource that defines rules, including the URI path, backing service name, and other information. The Ingress controller can then automatically program a frontend load balancer to enable Ingress configuration. The NGINX Ingress Controller for Kubernetes is what enables Kubernetes to configure NGINX and NGINX Plus for load balancing Kubernetes services.

Nginx ingress

Nginx Ingress Controller是实现ingress的具体实现，包含有两个版本：Ngnix OSS和Nginx Plus版，后者是商业化增强版，支持更多的功能，详情参考官方文档介绍https://www.nginx.com/products/nginx/kubernetes-ingress-controller#compare-versions

Nginx ingress版本features对比

2.2 Nginx ingress安装 #

首先需要安装Nginx Ingress Controller控制器，控制器安装方式包含两种：DaemonSets和Deployments。

DaemonSets通过hostPort的方式暴露80和443端口，可通过Node的调度由专门的节点实现部署
Deployments则通过NodePort的方式实现控制器端口的暴露，借助外部负载均衡实现高可用负载均衡

除此之外，还需要部署Namespace，ServiceAccount，RBAC，Secrets，Custom Resource Definitions等资源，如下开始部署。

2.2.1 基础依赖环境准备 #

1、github中下载源码包,安装部署文件在kubernetes-ingress/deployments/目录下

 1[root@node-1 ~]# git clone https://github.com/nginxinc/kubernetes-ingress.git
 2[root@node-1 ~]# tree kubernetes-ingress/deployments/
 3kubernetes-ingress/deployments/
 4├── common
 5│   ├── custom-resource-definitions.yaml  自定义资源
 6│   ├── default-server-secret.yaml        Secrets
 7│   ├── nginx-config.yaml
 8│   └── ns-and-sa.yaml                    Namspace+ServiceAccount
 9├── daemon-set
10│   ├── nginx-ingress.yaml                DaemonSets控制器
11│   └── nginx-plus-ingress.yaml
12├── deployment
13│   ├── nginx-ingress.yaml                Deployments控制器
14│   └── nginx-plus-ingress.yaml
15├── helm-chart                            Helm安装包
16│   ├── chart-icon.png
17│   ├── Chart.yaml
18│   ├── README.md
19│   ├── templates
20│   │   ├── controller-configmap.yaml
21│   │   ├── controller-custom-resources.yaml
22│   │   ├── controller-daemonset.yaml
23│   │   ├── controller-deployment.yaml
24│   │   ├── controller-leader-election-configmap.yaml
25│   │   ├── controller-secret.yaml
26│   │   ├── controller-serviceaccount.yaml
27│   │   ├── controller-service.yaml
28│   │   ├── controller-wildcard-secret.yaml
29│   │   ├── _helpers.tpl
30│   │   ├── NOTES.txt
31│   │   └── rbac.yaml
32│   ├── values-icp.yaml
33│   ├── values-plus.yaml
34│   └── values.yaml
35├── rbac                                RBAC认证授权
36│   └── rbac.yaml
37├── README.md
38└── service                            Service定义
39    ├── loadbalancer-aws-elb.yaml
40    ├── loadbalancer.yaml              DaemonSets暴露服务方式
41    └── nodeport.yaml                  Deployments暴露服务方式

2、创建Namespace和ServiceAccount, kubectl apply -f common/default-server-secret.yaml

 1apiVersion: v1
 2kind: Namespace
 3metadata:
 4  name: nginx-ingress 
 5---
 6apiVersion: v1
 7kind: ServiceAccount
 8metadata:
 9  name: nginx-ingress 
10  namespace: nginx-ingress

3、创建Secrets自签名证书，kubectl apply -f common/default-server-secret.yaml

1apiVersion: v1
2kind: Secret
3metadata:
4  name: default-server-secret
5  namespace: nginx-ingress
6type: Opaque
7data:
8  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN2akNDQWFZQ0NRREFPRjl0THNhWFhEQU5CZ2txaGtpRzl3MEJBUXNGQURBaE1SOHdIUVlEVlFRRERCWk8KUjBsT1dFbHVaM0psYzNORGIyNTBjbTlzYkdWeU1CNFhEVEU0TURreE1qRTRNRE16TlZvWERUSXpNRGt4TVRFNApNRE16TlZvd0lURWZNQjBHQTFVRUF3d1dUa2RKVGxoSmJtZHlaWE56UTI5dWRISnZiR3hsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUwvN2hIUEtFWGRMdjNyaUM3QlBrMTNpWkt5eTlyQ08KR2xZUXYyK2EzUDF0azIrS3YwVGF5aGRCbDRrcnNUcTZzZm8vWUk1Y2Vhbkw4WGM3U1pyQkVRYm9EN2REbWs1Qgo4eDZLS2xHWU5IWlg0Rm5UZ0VPaStlM2ptTFFxRlBSY1kzVnNPazFFeUZBL0JnWlJVbkNHZUtGeERSN0tQdGhyCmtqSXVuektURXUyaDU4Tlp0S21ScUJHdDEwcTNRYzhZT3ExM2FnbmovUWRjc0ZYYTJnMjB1K1lYZDdoZ3krZksKWk4vVUkxQUQ0YzZyM1lma1ZWUmVHd1lxQVp1WXN2V0RKbW1GNWRwdEMzN011cDBPRUxVTExSakZJOTZXNXIwSAo1TmdPc25NWFJNV1hYVlpiNWRxT3R0SmRtS3FhZ25TZ1JQQVpQN2MwQjFQU2FqYzZjNGZRVXpNQ0F3RUFBVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQWpLb2tRdGRPcEsrTzhibWVPc3lySmdJSXJycVFVY2ZOUitjb0hZVUoKdGhrYnhITFMzR3VBTWI5dm15VExPY2xxeC9aYzJPblEwMEJCLzlTb0swcitFZ1U2UlVrRWtWcitTTFA3NTdUWgozZWI4dmdPdEduMS9ienM3bzNBaS9kclkrcUI5Q2k1S3lPc3FHTG1US2xFaUtOYkcyR1ZyTWxjS0ZYQU80YTY3Cklnc1hzYktNbTQwV1U3cG9mcGltU1ZmaXFSdkV5YmN3N0NYODF6cFErUyt1eHRYK2VBZ3V0NHh3VlI5d2IyVXYKelhuZk9HbWhWNThDd1dIQnNKa0kxNXhaa2VUWXdSN0diaEFMSkZUUkk3dkhvQXprTWIzbjAxQjQyWjNrN3RXNQpJUDFmTlpIOFUvOWxiUHNoT21FRFZkdjF5ZytVRVJxbStGSis2R0oxeFJGcGZnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
9  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdi91RWM4b1JkMHUvZXVJTHNFK1RYZUprckxMMnNJNGFWaEMvYjVyYy9XMlRiNHEvClJOcktGMEdYaVN1eE9ycXgrajlnamx4NXFjdnhkenRKbXNFUkJ1Z1B0ME9hVGtIekhvb3FVWmcwZGxmZ1dkT0EKUTZMNTdlT1l0Q29VOUZ4amRXdzZUVVRJVUQ4R0JsRlNjSVo0b1hFTkhzbysyR3VTTWk2Zk1wTVM3YUhudzFtMApxWkdvRWEzWFNyZEJ6eGc2clhkcUNlUDlCMXl3VmRyYURiUzc1aGQzdUdETDU4cGszOVFqVUFQaHpxdmRoK1JWClZGNGJCaW9CbTVpeTlZTW1hWVhsMm0wTGZzeTZuUTRRdFFzdEdNVWozcGJtdlFmazJBNnljeGRFeFpkZFZsdmwKMm82MjBsMllxcHFDZEtCRThCay90elFIVTlKcU56cHpoOUJUTXdJREFRQUJBb0lCQVFDZklHbXowOHhRVmorNwpLZnZJUXQwQ0YzR2MxNld6eDhVNml4MHg4Mm15d1kxUUNlL3BzWE9LZlRxT1h1SENyUlp5TnUvZ2IvUUQ4bUFOCmxOMjRZTWl0TWRJODg5TEZoTkp3QU5OODJDeTczckM5bzVvUDlkazAvYzRIbjAzSkVYNzZ5QjgzQm9rR1FvYksKMjhMNk0rdHUzUmFqNjd6Vmc2d2szaEhrU0pXSzBwV1YrSjdrUkRWYmhDYUZhNk5nMUZNRWxhTlozVDhhUUtyQgpDUDNDeEFTdjYxWTk5TEI4KzNXWVFIK3NYaTVGM01pYVNBZ1BkQUk3WEh1dXFET1lvMU5PL0JoSGt1aVg2QnRtCnorNTZud2pZMy8yUytSRmNBc3JMTnIwMDJZZi9oY0IraVlDNzVWYmcydVd6WTY3TWdOTGQ5VW9RU3BDRkYrVm4KM0cyUnhybnhBb0dCQU40U3M0ZVlPU2huMVpQQjdhTUZsY0k2RHR2S2ErTGZTTXFyY2pOZjJlSEpZNnhubmxKdgpGenpGL2RiVWVTbWxSekR0WkdlcXZXaHFISy9iTjIyeWJhOU1WMDlRQ0JFTk5jNmtWajJTVHpUWkJVbEx4QzYrCk93Z0wyZHhKendWelU0VC84ajdHalRUN05BZVpFS2FvRHFyRG5BYWkyaW5oZU1JVWZHRXFGKzJyQW9HQkFOMVAKK0tZL0lsS3RWRzRKSklQNzBjUis3RmpyeXJpY05iWCtQVzUvOXFHaWxnY2grZ3l4b25BWlBpd2NpeDN3QVpGdwpaZC96ZFB2aTBkWEppc1BSZjRMazg5b2pCUmpiRmRmc2l5UmJYbyt3TFU4NUhRU2NGMnN5aUFPaTVBRHdVU0FkCm45YWFweUNweEFkREtERHdObit3ZFhtaTZ0OHRpSFRkK3RoVDhkaVpBb0dCQUt6Wis1bG9OOTBtYlF4VVh5YUwKMjFSUm9tMGJjcndsTmVCaWNFSmlzaEhYa2xpSVVxZ3hSZklNM2hhUVRUcklKZENFaHFsV01aV0xPb2I2NTNyZgo3aFlMSXM1ZUtka3o0aFRVdnpldm9TMHVXcm9CV2xOVHlGanIrSWhKZnZUc0hpOGdsU3FkbXgySkJhZUFVWUNXCndNdlQ4NmNLclNyNkQrZG8wS05FZzFsL0FvR0FlMkFVdHVFbFNqLzBmRzgrV3hHc1RFV1JqclRNUzRSUjhRWXQKeXdjdFA4aDZxTGxKUTRCWGxQU05rMXZLTmtOUkxIb2pZT2pCQTViYjhibXNVU1BlV09NNENoaFJ4QnlHbmR2eAphYkJDRkFwY0IvbEg4d1R0alVZYlN5T294ZGt5OEp0ek90ajJhS0FiZHd6NlArWDZDODhjZmxYVFo5MWpYL3RMCjF3TmRKS2tDZ1lCbyt0UzB5TzJ2SWFmK2UwSkN5TGhzVDQ5cTN3Zis2QWVqWGx2WDJ1VnRYejN5QTZnbXo5aCsKcDNlK2JMRUxwb3B0WFhNdUFRR0xhUkcrYlNNcjR5dERYbE5ZSndUeThXczNKY3dlSTdqZVp2b0ZpbmNvVlVIMwphdmxoTUVCRGYxSjltSDB5cDBwWUNaS2ROdHNvZEZtQktzVEtQMjJhTmtsVVhCS3gyZzR6cFE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

4、创建ConfigMap自定义配置文件，kubectl apply -f common/nginx-config.yaml

1kind: ConfigMap
2apiVersion: v1
3metadata:
4  name: nginx-config
5  namespace: nginx-ingress
6data:

5、为虚拟云主机和虚拟云主机路由定义自定义资源，支持自定义虚拟主机和虚拟路由，kubectl apply -f common/custom-resource-definitions.yaml

 1apiVersion: apiextensions.k8s.io/v1beta1
 2kind: CustomResourceDefinition
 3metadata:
 4  name: virtualservers.k8s.nginx.org
 5spec:
 6  group: k8s.nginx.org
 7  versions:
 8  - name: v1
 9    served: true
10    storage: true
11  scope: Namespaced
12  names:
13    plural: virtualservers
14    singular: virtualserver
15    kind: VirtualServer
16    shortNames:
17    - vs
18---
19apiVersion: apiextensions.k8s.io/v1beta1
20kind: CustomResourceDefinition
21metadata:
22  name: virtualserverroutes.k8s.nginx.org
23spec:
24  group: k8s.nginx.org
25  versions:
26  - name: v1
27    served: true
28    storage: true
29  scope: Namespaced
30  names:
31    plural: virtualserverroutes
32    singular: virtualserverroute
33    kind: VirtualServerRoute
34    shortNames:
35    - vsr

6、配置RBAC认证授权，实现ingress控制器访问集群中的其他资源，kubectl apply -f rbac/rbac.yaml

 1kind: ClusterRole
 2apiVersion: rbac.authorization.k8s.io/v1beta1
 3metadata:
 4  name: nginx-ingress
 5rules:
 6- apiGroups:
 7  - ""
 8  resources:
 9  - services
10  - endpoints
11  verbs:
12  - get
13  - list
14  - watch
15- apiGroups:
16  - ""
17  resources:
18  - secrets
19  verbs:
20  - get
21  - list
22  - watch
23- apiGroups:
24  - ""
25  resources:
26  - configmaps
27  verbs:
28  - get
29  - list
30  - watch
31  - update
32  - create
33- apiGroups:
34  - ""
35  resources:
36  - pods
37  verbs:
38  - list
39  - watch
40- apiGroups:
41  - ""
42  resources:
43  - events
44  verbs:
45  - create
46  - patch
47- apiGroups:
48  - extensions
49  resources:
50  - ingresses
51  verbs:
52  - list
53  - watch
54  - get
55- apiGroups:
56  - "extensions"
57  resources:
58  - ingresses/status
59  verbs:
60  - update
61- apiGroups:
62  - k8s.nginx.org
63  resources:
64  - virtualservers
65  - virtualserverroutes
66  verbs:
67  - list
68  - watch
69  - get
70---
71kind: ClusterRoleBinding
72apiVersion: rbac.authorization.k8s.io/v1beta1
73metadata:
74  name: nginx-ingress
75subjects:
76- kind: ServiceAccount
77  name: nginx-ingress
78  namespace: nginx-ingress
79roleRef:
80  kind: ClusterRole
81  name: nginx-ingress
82  apiGroup: rbac.authorization.k8s.io

2.2.2 部署Ingress控制器 #

1、部署控制器，控制器可以DaemonSets和Deployment的形式部署，如下是DaemonSets的配置文件

 1apiVersion: apps/v1
 2kind: DaemonSet
 3metadata:
 4  name: nginx-ingress
 5  namespace: nginx-ingress
 6spec:
 7  selector:
 8    matchLabels:
 9      app: nginx-ingress
10  template:
11    metadata:
12      labels:
13        app: nginx-ingress
14     #annotations:
15       #prometheus.io/scrape: "true"
16       #prometheus.io/port: "9113"
17    spec:
18      serviceAccountName: nginx-ingress
19      containers:
20      - image: nginx/nginx-ingress:edge
21        imagePullPolicy: Always
22        name: nginx-ingress
23        ports:
24        - name: http
25          containerPort: 80
26          hostPort: 80            #通过hostPort的方式暴露端口
27        - name: https
28          containerPort: 443
29          hostPort: 443
30       #- name: prometheus
31         #containerPort: 9113
32        securityContext:
33          allowPrivilegeEscalation: true
34          runAsUser: 101 #nginx
35          capabilities:
36            drop:
37            - ALL
38            add:
39            - NET_BIND_SERVICE
40        env:
41        - name: POD_NAMESPACE
42          valueFrom:
43            fieldRef:
44              fieldPath: metadata.namespace
45        - name: POD_NAME
46          valueFrom:
47            fieldRef:
48              fieldPath: metadata.name
49        args:
50          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
51          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
52         #- -v=3 # Enables extensive logging. Useful for troubleshooting.
53         #- -report-ingress-status
54         #- -external-service=nginx-ingress
55         #- -enable-leader-election
56         #- -enable-prometheus-metrics

Deployments的配置文件

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: nginx-ingress
 5  namespace: nginx-ingress
 6spec:
 7  replicas: 1                  #副本的个数
 8  selector:
 9    matchLabels:
10      app: nginx-ingress
11  template:
12    metadata:
13      labels:
14        app: nginx-ingress
15     #annotations:
16       #prometheus.io/scrape: "true"
17       #prometheus.io/port: "9113"
18    spec:
19      serviceAccountName: nginx-ingress
20      containers:
21      - image: nginx/nginx-ingress:edge
22        imagePullPolicy: Always
23        name: nginx-ingress
24        ports:                #内部暴露的服务端口，需要通过NodePort的方式暴露给外部
25        - name: http
26          containerPort: 80
27        - name: https
28          containerPort: 443
29       #- name: prometheus
30         #containerPort: 9113
31        securityContext:
32          allowPrivilegeEscalation: true
33          runAsUser: 101 #nginx
34          capabilities:
35            drop:
36            - ALL
37            add:
38            - NET_BIND_SERVICE
39        env:
40        - name: POD_NAMESPACE
41          valueFrom:
42            fieldRef:
43              fieldPath: metadata.namespace
44        - name: POD_NAME
45          valueFrom:
46            fieldRef:
47              fieldPath: metadata.name
48        args:
49          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
50          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
51         #- -v=3 # Enables extensive logging. Useful for troubleshooting.
52         #- -report-ingress-status
53         #- -external-service=nginx-ingress
54         #- -enable-leader-election
55         #- -enable-prometheus-metrics

2、我们以DaemonSets的方式部署，DaemonSet部署集群中各个节点都是对等，如果有外部LoadBalancer则通过外部负载均衡路由至Ingress中

 1[root@node-1 deployments]# kubectl apply -f daemon-set/nginx-ingress.yaml 
 2daemonset.apps/nginx-ingress created
 3[root@node-1 deployments]# kubectl get daemonsets -n nginx-ingress
 4NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
 5nginx-ingress   3         3         3       3            3           <none>          15s
 6
 7[root@node-1 ~]# kubectl get pods -n nginx-ingress -o wide 
 8NAME                  READY   STATUS    RESTARTS   AGE     IP             NODE     NOMINATED NODE   READINESS GATES
 9nginx-ingress-7mpfc   1/1     Running   0          2m44s   10.244.0.50    node-1   <none>           <none>
10nginx-ingress-l2rtj   1/1     Running   0          2m44s   10.244.1.144   node-2   <none>           <none>
11nginx-ingress-tgf6r   1/1     Running   0          2m44s   10.244.2.160   node-3   <none>           <none>

3、校验Nginx Ingress安装情况，此时三个节点均是对等，即访问任意一个节点均能实现相同的效果，统一入口则通过外部负载均衡，如果在云环境下执行kubectl apply -f service/loadbalancer.yaml创建外部负载均衡实现入口调度，自建的可以通过lvs或nginx等负载均衡实现接入，本文不再赘述，读者可以自行研究。

nginx ingress安装校验

备注说明：如果以Deployments的方式部署，则需要执行service/nodeport.yaml创建NodePort类型的Service，实现的效果和DaemonSets类似。

3. Ingress资源定义 #

上面的章节已安装了一个Nginx Ingress Controller控制器，有了Ingress控制器后，我们就可以定义Ingress资源来实现七层负载转发了，大体上Ingress支持三种使用方式：1. 基于虚拟主机转发，2. 基于虚拟机主机URI转发，3. 支持TLS加密转发。

3.1 Ingress定义 #

1、环境准备，先创建一个nginx的Deployment应用，包含2个副本

1[root@node-1 ~]# kubectl run ingress-demo --image=nginx:1.7.9 --port=80 --replicas=2
2[root@node-1 ~]# kubectl get deployments
3NAME           READY   UP-TO-DATE   AVAILABLE   AGE
4ingress-demo   2/2     2            2           116s

2、以service方式暴露服务端口

1[root@node-1 ~]# kubectl expose deployment ingress-demo --port=80 --protocol=TCP --target-port=80
2service/ingress-demo exposed
3[root@node-1 ~]# kubectl get services 
4NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
5ingress-demo   ClusterIP   10.109.33.91   <none>        80/TCP    2m15s

3、上述两个步骤已创建了一个service，如下我们定义一个ingress对象将起转发至ingress-demo这个service，通过ingress.class指定控制器的类型为nginx

 1[root@node-1 nginx-ingress]# cat nginx-ingress-demo.yaml 
 2apiVersion: extensions/v1beta1
 3kind: Ingress
 4metadata:
 5  name: nginx-ingress-demo
 6  labels:
 7    ingres-controller: nginx
 8  annotations:
 9    kubernets.io/ingress.class: nginx
10spec:
11  rules:
12  - host: www.happylau.cn
13    http:
14      paths:
15      - path: /
16        backend:
17          serviceName: ingress-demo
18          servicePort: 80

4、创建ingress对象

1[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-demo.yaml 
2ingress.extensions/nginx-ingress-demo created
3
4查看ingress资源列表
5[root@node-1 nginx-ingress]# kubectl get ingresses
6NAME                 HOSTS                ADDRESS   PORTS   AGE
7nginx-ingress-demo   www.happylau.cn             80      4m4s

5、查看ingress详情，可以在Rules规则中看到后端Pod的列表，自动发现和关联相关Pod

 1[root@node-1 ~]# kubectl describe ingresses nginx-ingress-demo 
 2Name:             nginx-ingress-demo
 3Namespace:        default
 4Address:          
 5Default backend:  default-http-backend:80 (<none>)
 6Rules:
 7  Host                Path  Backends
 8  ----                ----  --------
 9  www.happylau.cn  
10                      /   ingress-demo:80 (10.244.1.146:80,10.244.2.162:80)
11Annotations:
12  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-demo","namespace":"default"},"spec":{"rules":[{"host":"www.happylaulab.cn","http":{"paths":[{"backend":{"serviceName":"ingress-demo","servicePort":80},"path":"/"}]}}]}}
13
14  kubernets.io/ingress.class:  nginx
15Events:
16  Type    Reason          Age   From                      Message
17  ----    ------          ----  ----                      -------
18  Normal  AddedOrUpdated  9m7s  nginx-ingress-controller  Configuration for default/nginx-ingress-demo was added or updated
19  Normal  AddedOrUpdated  9m7s  nginx-ingress-controller  Configuration for default/nginx-ingress-demo was added or updated
20  Normal  AddedOrUpdated  9m7s  nginx-ingress-controller  Configuration for default/nginx-ingress-demo was added or updated

6、测试验证，ingress规则的配置信息已注入到Ingress Controller中，环境中Ingress Controller是以DaemonSets的方式部署在集群中，如果有外部的负载均衡，则将www.happylau.cn域名的地址解析为负载均衡VIP。由于测试环境没有搭建负载均衡，将hosts解析执行node-1，node-2或者node-3任意一个IP都能实现相同的功能。

ingress测试

上述测试解析正常，当然也可以解析为node-1和node-2的IP，如下：

 1[root@node-1 ~]# curl -I http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.101
 2HTTP/1.1 200 OK
 3Server: nginx/1.17.6
 4Date: Tue, 24 Dec 2019 10:32:22 GMT
 5Content-Type: text/html
 6Content-Length: 612
 7Connection: keep-alive
 8Last-Modified: Tue, 23 Dec 2014 16:25:09 GMT
 9ETag: "54999765-264"
10Accept-Ranges: bytes
11
12[root@node-1 ~]# curl -I http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.102
13HTTP/1.1 200 OK
14Server: nginx/1.17.6
15Date: Tue, 24 Dec 2019 10:32:24 GMT
16Content-Type: text/html
17Content-Length: 612
18Connection: keep-alive
19Last-Modified: Tue, 23 Dec 2014 16:25:09 GMT
20ETag: "54999765-264"
21Accept-Ranges: bytes

3.2 Ingress动态配置 #

上面的章节介绍了ingress资源对象的申明配置，在这个章节中我们探究一下Nginx Ingress Controller的实现机制和动态配置更新机制，以方便了解Ingress控制器的工作机制。

1、查看Nginx Controller控制器的配置文件，在nginx-ingress pod中存储着ingress的配置文件

 1[root@node-1 ~]# kubectl get pods -n nginx-ingress 
 2NAME                  READY   STATUS    RESTARTS   AGE
 3nginx-ingress-7mpfc   1/1     Running   0          6h15m
 4nginx-ingress-l2rtj   1/1     Running   0          6h15m
 5nginx-ingress-tgf6r   1/1     Running   0          6h15m
 6
 7#查看配置文件，每个ingress生成一个配置文件，文件名为：命名空间-ingres名称.conf
 8[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress -- ls -l /etc/nginx/conf.d
 9total 4
10-rw-r--r-- 1 nginx nginx 1005 Dec 24 10:06 default-nginx-ingress-demo.conf
11
12#查看配置文件
13[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress -- cat /etc/nginx/conf.d/default-nginx-ingress-demo.conf
14# configuration for default/nginx-ingress-demo
15
16#upstream的配置，会用least_conn算法，通过service服务发现机制动态识别到后端的Pod
17upstream default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80 {
18	zone default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80 256k;
19	random two least_conn;
20	server 10.244.1.146:80 max_fails=1 fail_timeout=10s max_conns=0;
21	server 10.244.2.162:80 max_fails=1 fail_timeout=10s max_conns=0;
22}
23
24server {
25	listen 80;
26	server_tokens on;
27	server_name www.happylau.cn;
28	location / {
29		proxy_http_version 1.1;
30		proxy_connect_timeout 60s;
31		proxy_read_timeout 60s;
32		proxy_send_timeout 60s;
33		client_max_body_size 1m;
34		proxy_set_header Host $host;
35		proxy_set_header X-Real-IP $remote_addr;
36		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
37		proxy_set_header X-Forwarded-Host $host;
38		proxy_set_header X-Forwarded-Port $server_port;
39		proxy_set_header X-Forwarded-Proto $scheme;
40		proxy_buffering on;
41		proxy_pass http://default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80;	#调用upstream实现代理
42	}
43}

通过上述查看配置文件可得知，Nginx Ingress Controller实际是根据ingress规则生成对应的nginx配置文件，以实现代理转发的功能，加入Deployments的副本数变更后nginx的配置文件会发生什么改变呢？

2、更新控制器的副本数，由2个Pod副本扩容至3个

1[root@node-1 ~]# kubectl scale --replicas=3 deployment ingress-demo 
2deployment.extensions/ingress-demo scaled
3[root@node-1 ~]# kubectl get deployments
4NAME           READY   UP-TO-DATE   AVAILABLE   AGE
5ingress-demo   3/3     3            3           123m

3、再次查看nginx的配置文件，ingress借助于service的服务发现机制，将加入的Pod自动加入到nginx upstream中

nginx动态更新配置

4、查看nginx pod的日志（kubectl logs nginx-ingress-7mpfc -n nginx-ingress），有reload优雅重启的记录，即通过更新配置文件+reload实现配置动态更新。

nginx控制器重启日志记录

通过上述的配置可知，ingress调用kubernetes api去感知kubernetes集群中的变化情况，Pod的增加或减少这些变化，然后动态更新nginx ingress controller的配置文件，并重新载入配置。当集群规模越大时，会频繁涉及到配置文件的变动和重载，因此nginx这方面会存在先天的劣势，专门为微服务负载均衡应运而生，如Traefik，Envoy，Istio，这些负载均衡工具能够提供大规模，频繁动态更新的场景，但性能相比Nginx，HAproxy还存在一定的劣势。往后的章节中，我们再对其他的Ingress控制器做介绍。

3.3 Ingress路径转发 #

Ingress支持URI格式的转发方式，同时支持URL重写，如下以两个service为例演示，service-1安装nginx，service-2安装httpd，分别用http://demo.happylau.cn/news和http://demo.happylau.cn/sports转发到两个不同的service

1、环境准备，创建两个应用并实现service暴露，创建deployments时指定--explose创建service

 1[root@node-1 ~]# kubectl run service-1 --image=nginx:1.7.9 --port=80 --replicas=1 --expose=true 
 2service/service-1 created
 3deployment.apps/service-1 created
 4
 5[root@node-1 ~]# kubectl run service-2 --image=httpd --port=80 --replicas=1 --expose=true 
 6service/service-2 created
 7deployment.apps/service-2 created
 8
 9查看deployment状态
10[root@node-1 ~]# kubectl get deployments 
11NAME           READY   UP-TO-DATE   AVAILABLE   AGE
12ingress-demo   4/4     4            4           4h36m
13service-1      1/1     1            1           65s
14service-2      1/1     1            1           52s
15
16查看service状态，服务已经正常
17[root@node-1 ~]# kubectl get services 
18NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
19ingress-demo   ClusterIP   10.109.33.91     <none>        80/TCP    4h36m
20kubernetes     ClusterIP   10.96.0.1        <none>        443/TCP   101d
21service-1      ClusterIP   10.106.245.71    <none>        80/TCP    68s
22service-2      ClusterIP   10.104.204.158   <none>        80/TCP    55s

2、创建ingress对象，通过一个域名将请求转发至后端两个service

 1[root@node-1 nginx-ingress]# cat nginx-ingress-uri-demo.yaml 
 2apiVersion: extensions/v1beta1
 3kind: Ingress
 4metadata:
 5  name: nginx-ingress-uri-demo
 6  labels:
 7    ingres-controller: nginx
 8  annotations:
 9    kubernets.io/ingress.class: nginx
10    nginx.ingress.kubernetes.io/rewrite-target: /
11spec:
12  rules:
13  - host: demo.happylau.cn
14    http:
15      paths:
16      - path: /news
17        backend:
18          serviceName: service-1 
19          servicePort: 80
20      - path: /sports
21        backend:
22          serviceName: service-2
23          servicePort: 80

3、创建ingress规则，查看详情

 1[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-uri-demo.yaml 
 2ingress.extensions/nginx-ingress-uri-demo created
 3
 4#查看详情
 5[root@node-1 nginx-ingress]# kubectl get ingresses.
 6NAME                     HOSTS              ADDRESS   PORTS   AGE
 7nginx-ingress-demo       www.happylau.cn              80      4h35m
 8nginx-ingress-uri-demo   demo.happylau.cn             80      4s
 9[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-uri-demo 
10Name:             nginx-ingress-uri-demo
11Namespace:        default
12Address:          
13Default backend:  default-http-backend:80 (<none>)
14Rules:              #对应的转发url规则
15  Host              Path  Backends
16  ----              ----  --------
17  demo.happylau.cn  
18                    /news     service-1:80 (10.244.2.163:80)
19                    /sports   service-2:80 (10.244.1.148:80)
20Annotations:
21  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/rewrite-target":"/"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-uri-demo","namespace":"default"},"spec":{"rules":[{"host":"demo.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/news"},{"backend":{"serviceName":"service-2","servicePort":80},"path":"/sports"}]}}]}}
22
23  kubernets.io/ingress.class:                  nginx
24  nginx.ingress.kubernetes.io/rewrite-target:  /
25Events:
26  Type    Reason          Age   From                      Message
27  ----    ------          ----  ----                      -------
28  Normal  AddedOrUpdated  11s   nginx-ingress-controller  Configuration for default/nginx-ingress-uri-demo was added or updated
29  Normal  AddedOrUpdated  11s   nginx-ingress-controller  Configuration for default/nginx-ingress-uri-demo was added or updated
30  Normal  AddedOrUpdated  11s   nginx-ingress-controller  Configuration for default/nginx-ingress-uri-demo was added or updated

4、准备测试，站点中创建对应的路径

1[root@node-1 ~]# kubectl exec -it service-1-7b66bf758f-xj9jh /bin/bash
2root@service-1-7b66bf758f-xj9jh:/# echo "service-1 website page" >/usr/share/nginx/html/news
3
4[root@node-1 ~]# kubectl exec -it service-2-7c7444684d-w9cv9 /bin/bash
5root@service-2-7c7444684d-w9cv9:/usr/local/apache2# echo "service-2 website page" >/usr/local/apache2/htdocs/sports

5、测试验证

1[root@node-1 ~]# curl http://demo.happylau.cn/news --resolve demo.happylau.cn:80:10.254.100.101
2service-1 website page
3[root@node-1 ~]# curl http://demo.happylau.cn/sports --resolve demo.happylau.cn:80:10.254.100.101
4service-2 website page

6、通过上述的验证测试可以得知，ingress支持URI的路由方式转发，其对应在ingress中的配置文件内容是怎样的呢，我们看下ingress controller生成对应的nginx配置文件内容，实际是通过ingress的location来实现，将不同的localtion转发至不同的upstream以实现service的关联，配置文件如下：

 1[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress /bin/bash
 2nginx@nginx-ingress-7mpfc:/$ cat /etc/nginx/conf.d/default-nginx-ingress-uri-demo.conf |grep -v "^$"
 3# configuration for default/nginx-ingress-uri-demo
 4#定义两个upstream和后端的service关联
 5upstream default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80 {
 6	zone default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80 256k;
 7	random two least_conn;
 8	server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
 9}
10
11upstream default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80 {
12	zone default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80 256k;
13	random two least_conn;
14	server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;	
15}
16
17server {
18	listen 80;
19	server_tokens on;
20	server_name demo.happylau.cn;
21	
22  #定义location实现代理，通过proxy_pass和后端的service关联
23	location /news {
24		proxy_http_version 1.1;
25		proxy_connect_timeout 60s;
26		proxy_read_timeout 60s;
27		proxy_send_timeout 60s;
28		client_max_body_size 1m;
29		proxy_set_header Host $host;
30		proxy_set_header X-Real-IP $remote_addr;
31		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
32		proxy_set_header X-Forwarded-Host $host;
33		proxy_set_header X-Forwarded-Port $server_port;
34		proxy_set_header X-Forwarded-Proto $scheme;
35		proxy_buffering on;
36		proxy_pass http://default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80;
37	}
38
39	location /sports {
40		proxy_http_version 1.1;
41		proxy_connect_timeout 60s;
42		proxy_read_timeout 60s;
43		proxy_send_timeout 60s;
44		client_max_body_size 1m;
45		proxy_set_header Host $host;
46		proxy_set_header X-Real-IP $remote_addr;
47		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
48		proxy_set_header X-Forwarded-Host $host;
49		proxy_set_header X-Forwarded-Port $server_port;
50		proxy_set_header X-Forwarded-Proto $scheme;
51		proxy_buffering on;
52		proxy_pass http://default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80;	
53	}	
54}

3.4 Ingress虚拟主机 #

ingress支持基于名称的虚拟主机，实现单个IP多个域名转发的需求，通过请求头部携带主机名方式区分开，将上个章节的ingress删除，使用service-1和service-2两个service来做演示。

1、创建ingress规则，通过主机名实现转发规则

 1apiVersion: extensions/v1beta1
 2kind: Ingress
 3metadata:
 4  name: nginx-ingress-virtualname-demo
 5  labels:
 6    ingres-controller: nginx
 7  annotations:
 8    kubernets.io/ingress.class: nginx
 9spec:
10  rules:
11  - host: news.happylau.cn
12    http:
13      paths:
14      - path: /
15        backend:
16          serviceName: service-1 
17          servicePort: 80
18  - host: sports.happylau.cn
19    http:
20      paths:
21      - path: /
22        backend:
23          serviceName: service-2 
24          servicePort: 80

2、生成ingress规则并查看详情，一个ingress对应两个HOSTS

 1[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-virtualname.yaml 
 2ingress.extensions/nginx-ingress-virtualname-demo created
 3
 4#查看列表
 5[root@node-1 nginx-ingress]# kubectl get ingresses nginx-ingress-virtualname-demo 
 6NAME                             HOSTS                                 ADDRESS   PORTS   AGE
 7nginx-ingress-virtualname-demo   news.happylau.cn,sports.happylau.cn             80      12s
 8
 9#查看详情
10[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-virtualname-demo
11Name:             nginx-ingress-virtualname-demo
12Namespace:        default
13Address:          
14Default backend:  default-http-backend:80 (<none>)
15Rules:
16  Host                Path  Backends
17  ----                ----  --------
18  news.happylau.cn    
19                      /   service-1:80 (10.244.2.163:80)
20  sports.happylau.cn  
21                      /   service-2:80 (10.244.1.148:80)
22Annotations:
23  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-virtualname-demo","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-2","servicePort":80},"path":"/"}]}}]}}
24
25  kubernets.io/ingress.class:  nginx
26Events:
27  Type    Reason          Age   From                      Message
28  ----    ------          ----  ----                      -------
29  Normal  AddedOrUpdated  28s   nginx-ingress-controller  Configuration for default/nginx-ingress-virtualname-demo was added or updated
30  Normal  AddedOrUpdated  28s   nginx-ingress-controller  Configuration for default/nginx-ingress-virtualname-demo was added or updated
31  Normal  AddedOrUpdated  28s   nginx-ingress-controller  Configuration for default/nginx-ingress-virtualname-demo was added or updated

3、准备测试数据并测试

1[root@node-1 ~]# kubectl exec -it service-1-7b66bf758f-xj9jh /bin/bash
2root@service-1-7b66bf758f-xj9jh:/# echo "news demo" >/usr/share/nginx/html/index.html
3
4[root@node-1 ~]# kubectl exec -it service-2-7c7444684d-w9cv9 /bin/bash  
5root@service-2-7c7444684d-w9cv9:/usr/local/apache2# echo "sports demo"  >/usr/local/apache2/htdocs/index.html

测试：

1[root@node-1 ~]# curl http://news.happylau.cn --resolve news.happylau.cn:80:10.254.100.102
2news demo
3[root@node-1 ~]# curl http://sports.happylau.cn --resolve sports.happylau.cn:80:10.254.100.102
4sports demo

4、查看nginx的配置文件内容，通过在server中定义不同的server_name以区分，代理到不同的upstream以实现service的代理。

 1# configuration for default/nginx-ingress-virtualname-demo
 2upstream default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80 {
 3	zone default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80 256k;
 4	random two least_conn;
 5	server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
 6}
 7
 8upstream default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80 {
 9	zone default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80 256k;
10	random two least_conn;
11	server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;
12}
13
14server {
15	listen 80;
16	server_tokens on;
17	server_name news.happylau.cn;
18	location / {
19		proxy_http_version 1.1;
20		proxy_connect_timeout 60s;
21		proxy_read_timeout 60s;
22		proxy_send_timeout 60s;
23		client_max_body_size 1m;
24		proxy_set_header Host $host;
25		proxy_set_header X-Real-IP $remote_addr;
26		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
27		proxy_set_header X-Forwarded-Host $host;
28		proxy_set_header X-Forwarded-Port $server_port;
29		proxy_set_header X-Forwarded-Proto $scheme;
30		proxy_buffering on;
31		
32		proxy_pass http://default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80;
33	
34  }
35}
36server {
37	listen 80;	
38	server_tokens on;
39	server_name sports.happylau.cn;
40
41	location / {
42		proxy_http_version 1.1;
43		proxy_connect_timeout 60s;
44		proxy_read_timeout 60s;
45		proxy_send_timeout 60s;
46		client_max_body_size 1m;
47		proxy_set_header Host $host;
48		proxy_set_header X-Real-IP $remote_addr;
49		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
50		proxy_set_header X-Forwarded-Host $host;
51		proxy_set_header X-Forwarded-Port $server_port;
52		proxy_set_header X-Forwarded-Proto $scheme;
53		proxy_buffering on;
54		
55		proxy_pass http://default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80;	
56
57	}	
58}

3.5 Ingress TLS加密 #

四层的负载均衡无法支持https请求，当前大部分业务都要求以https方式接入，Ingress能支持https的方式接入，通过Secrets存储证书+私钥，实现https接入，同时还能支持http跳转功能。对于用户的请求流量来说，客户端到ingress controller是https流量，ingress controller到后端service则是http，提高用户访问性能，如下介绍ingress TLS功能实现步骤。

1、生成自签名证书和私钥

 1[root@node-1 ~]# openssl req -x509 -newkey rsa:2048 -nodes -days 365 -keyout tls.key -out tls.crt
 2Generating a 2048 bit RSA private key
 3....................................................+++
 4........................................+++
 5writing new private key to 'tls.key'
 6-----
 7You are about to be asked to enter information that will be incorporated
 8into your certificate request.
 9What you are about to enter is what is called a Distinguished Name or a DN.
10There are quite a few fields but you can leave some blank
11For some fields there will be a default value,
12If you enter '.', the field will be left blank.
13-----
14Country Name (2 letter code) [XX]:CN        #国家
15State or Province Name (full name) []:GD    #省份
16Locality Name (eg, city) [Default City]:ShenZhen  #城市
17Organization Name (eg, company) [Default Company Ltd]:Tencent    #公司 
18Organizational Unit Name (eg, section) []:HappyLau  #组织
19Common Name (eg, your name or your server's hostname) []:www.happylau.cn  #域名
20Email Address []:573302346@qq.com       #邮箱地址
21
22
23#tls.crt为证书，tls.key为私钥
24[root@node-1 ~]# ls tls.* -l
25-rw-r--r-- 1 root root 1428 12月 26 13:21 tls.crt
26-rw-r--r-- 1 root root 1708 12月 26 13:21 tls.key

2、配置Secrets，将证书和私钥配置到Secrets中

 1[root@node-1 ~]# kubectl create secret tls happylau-sslkey --cert=tls.crt --key=tls.key 
 2secret/happylau-sslkey created
 3
 4查看Secrets详情,证书和私要包含在data中，文件名为两个不同的key：tls.crt和tls.key
 5[root@node-1 ~]# kubectl describe secrets happylau-sslkey 
 6Name:         happylau-sslkey
 7Namespace:    default
 8Labels:       <none>
 9Annotations:  <none>
10
11Type:  kubernetes.io/tls
12
13Data
14====
15tls.crt:  1428 bytes
16tls.key:  1708 bytes

3、配置ingress调用Secrets实现SSL证书加密

 1apiVersion: extensions/v1beta1
 2kind: Ingress
 3metadata:
 4  name: nginx-ingress-tls-demo
 5  labels:
 6    ingres-controller: nginx
 7  annotations:
 8    kubernets.io/ingress.class: nginx
 9spec:
10  tls:
11  - hosts:
12    - news.happylau.cn
13    - sports.happylau.cn
14    secretName: happylau-sslkey
15  rules:
16  - host: news.happylau.cn
17    http:
18      paths:
19      - path: /
20        backend:
21          serviceName: service-1 
22          servicePort: 80
23  - host: sports.happylau.cn
24    http:
25      paths:
26      - path: /
27        backend:
28          serviceName: service-2 
29          servicePort: 80

4、创建ingress并查看ingress详情

 1[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-tls-demo 
 2Name:             nginx-ingress-tls-demo
 3Namespace:        default
 4Address:          
 5Default backend:  default-http-backend:80 (<none>)
 6TLS:
 7  happylau-sslkey terminates news.happylau.cn,sports.happylau.cn
 8Rules:
 9  Host                Path  Backends
10  ----                ----  --------
11  news.happylau.cn    
12                      /   service-1:80 (10.244.2.163:80)
13  sports.happylau.cn  
14                      /   service-2:80 (10.244.1.148:80)
15Annotations:
16  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-tls-demo","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-2","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["news.happylau.cn","sports.happylau.cn"],"secretName":"happylau-sslkey"}]}}
17
18  kubernets.io/ingress.class:  nginx
19Events:
20  Type    Reason          Age   From                      Message
21  ----    ------          ----  ----                      -------
22  Normal  AddedOrUpdated  22s   nginx-ingress-controller  Configuration for default/nginx-ingress-tls-demo was added or updated
23  Normal  AddedOrUpdated  22s   nginx-ingress-controller  Configuration for default/nginx-ingress-tls-demo was added or updated
24  Normal  AddedOrUpdated  22s   nginx-ingress-controller  Configuration for default/nginx-ingress-tls-demo was added or updated

5、将news.happylau.cn和sports.happylau.cn写入到hosts文件中，并通过https://news.happylau.cn 的方式访问，浏览器访问内容提示证书如下，信任证书即可访问到站点内容。

tls测试页面

查看证书详情，正是我们制作的自签名证书，生产实际使用时，推荐使用CA机构颁发签名证书。

证书详情

6、接下来查看一下tls配置https的nginx配置文件内容，可以看到在server块启用了https并配置证书，同时配置了http跳转，因此直接访问http也能够实现自动跳转到https功能。

 1# configuration for default/nginx-ingress-tls-demo
 2upstream default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80 {
 3	zone default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80 256k;
 4	random two least_conn;
 5	server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;	
 6}
 7
 8upstream default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80 {
 9	zone default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80 256k;
10	random two least_conn;
11	server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;	
12}
13
14server {
15	listen 80;
16
17	listen 443 ssl;     #https监听端口，证书和key，实现和Secrets关联
18	ssl_certificate /etc/nginx/secrets/default-happylau-sslkey;
19	ssl_certificate_key /etc/nginx/secrets/default-happylau-sslkey;
20
21	server_tokens on;
22	server_name news.happylau.cn;
23	
24  #http跳转功能，即访问http会自动跳转至https
25	if ($scheme = http) {
26		return 301 https://$host:443$request_uri;
27	}
28	
29	location / {
30		proxy_http_version 1.1;
31		proxy_connect_timeout 60s;
32		proxy_read_timeout 60s;
33		proxy_send_timeout 60s;
34		client_max_body_size 1m;
35		proxy_set_header Host $host;
36		proxy_set_header X-Real-IP $remote_addr;
37		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
38		proxy_set_header X-Forwarded-Host $host;
39		proxy_set_header X-Forwarded-Port $server_port;
40		proxy_set_header X-Forwarded-Proto $scheme;
41		proxy_buffering on;
42		
43		proxy_pass http://default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80;	
44  }	
45}
46
47server {
48	listen 80;
49	listen 443 ssl;
50	ssl_certificate /etc/nginx/secrets/default-happylau-sslkey;
51	ssl_certificate_key /etc/nginx/secrets/default-happylau-sslkey;
52
53	server_tokens on;
54	server_name sports.happylau.cn;
55
56	if ($scheme = http) {
57		return 301 https://$host:443$request_uri;
58	}
59
60	location / {
61		proxy_http_version 1.1;
62		proxy_connect_timeout 60s;
63		proxy_read_timeout 60s;
64		proxy_send_timeout 60s;
65		client_max_body_size 1m;
66		proxy_set_header Host $host;
67		proxy_set_header X-Real-IP $remote_addr;
68		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
69		proxy_set_header X-Forwarded-Host $host;
70		proxy_set_header X-Forwarded-Port $server_port;
71		proxy_set_header X-Forwarded-Proto $scheme;
72		proxy_buffering on;
73		
74		proxy_pass http://default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80;	
75	}	
76}

4. Nginx Ingress高级功能 #

4.1 定制化参数 #

ingress controller提供了基础反向代理的功能，如果需要定制化nginx的特性或参数，需要通过ConfigMap和Annotations来实现，两者实现的方式有所不同，ConfigMap用于指定整个ingress集群资源的基本参数，修改后会被所有的ingress对象所继承；Annotations则被某个具体的ingress对象所使用，修改只会影响某个具体的ingress资源，冲突时其优先级高于ConfigMap。

4.1.1 ConfigMap自定义参数 #

安装nginx ingress controller时默认会包含一个空的ConfigMap，可以通过ConfigMap来自定义nginx controller的默认参数，如下以修改一些参数为例：

1、定义ConfigMap参数

 1kind: ConfigMap
 2apiVersion: v1
 3metadata:
 4  name: nginx-config
 5  namespace: nginx-ingress
 6data:
 7  proxy-connect-timeout: "10s"
 8  proxy-read-timeout: "10s"
 9  proxy-send-timeout: "10"
10  client-max-body-size: "3m"

2、应用配置并查看ConfigMap配置

 1[root@node-1 ~]# kubectl get configmaps -n nginx-ingress nginx-config -o yaml
 2apiVersion: v1
 3data:
 4  client-max-body-size: 3m
 5  proxy-connect-timeout: 10s
 6  proxy-read-timeout: 10s
 7  proxy-send-timeout: 10s
 8kind: ConfigMap
 9metadata:
10  annotations:
11    kubectl.kubernetes.io/last-applied-configuration: |
12      {"apiVersion":"v1","data":{"client-max-body-size":"3m","proxy-connect-timeout":"10s","proxy-read-timeout":"10s","proxy-send-timeout":"10"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"nginx-config","namespace":"nginx-ingress"}}
13  creationTimestamp: "2019-12-24T04:39:23Z"
14  name: nginx-config
15  namespace: nginx-ingress
16  resourceVersion: "13845543"
17  selfLink: /api/v1/namespaces/nginx-ingress/configmaps/nginx-config
18  uid: 9313ae47-a0f0-463e-a25a-1658f1ca0d57

3 、此时，ConfigMap定义的配置参数会被集群中所有的Ingress资源继承（除了annotations定义之外）

ConfigMap参数验证

有很多参数可以定义，详情配置可参考方文档说明：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/configmap-and-annotations.md#Summary-of-ConfigMap-and-Annotations

4.1.2 Annotations自定义参数 #

ConfigMap定义的是全局的配置参数，修改后所有的配置都会受影响，如果想针对某个具体的ingress资源自定义参数，则可以通过Annotations来实现，下面开始以实际的例子演示Annotations的使用。

1、修改ingress资源，添加annotations的定义,通过nginx.org组修改了一些参数，如proxy-connect-timeout，调度算法为round_robin（默认为least _conn）

 1apiVersion: extensions/v1beta1
 2kind: Ingress
 3metadata:
 4  name: nginx-ingress-demo
 5  labels:
 6    ingres-controller: nginx
 7  annotations:
 8    kubernets.io/ingress.class: nginx
 9    nginx.org/proxy-connect-timeout: "30s"
10    nginx.org/proxy-send-timeout: "20s"
11    nginx.org/proxy-read-timeout: "20s"
12    nginx.org/client-max-body-size: "2m"
13    nginx.org/fail-timeout: "5s"
14    nginx.org/lb-method: "round_robin" 
15spec:
16  rules:
17  - host: www.happylau.cn
18    http:
19      paths:
20      - path: /
21        backend:
22          serviceName: ingress-demo
23          servicePort: 80

2、重新应用ingress对象并查看参数配置情况

Nginx Annotations演示

由上面的演示可得知，Annotations的优先级高于ConfigMapMap，Annotations修改参数只会影响到某一个具体的ingress资源，其定义的方法和ConfigMap相相近似，但又有差别，部分ConfigMap的参数Annotations无法支持，反过来Annotations定义的参数ConfigMap也不一定支持，下图列举一下常规支持参数情况：

通用参数

日志支持

请求头部

认证和安全

upstream支持

ConfigMap和Annotations详细支持说明：链接说明

4.2 虚拟主机和路由 #

安装nginx ingress时我们安装了一个customresourcedefinitions自定义资源，其能够提供除了默认ingress功能之外的一些高级特性如

虚拟主机VirtualServer
虚拟路由VirtualServerRoute
健康检查Healthcheck
流量切割Split
会话保持SessionCookie
重定向Redirect

这些功能大部分依赖于Nginx Plus高级版本的支持，社区版本仅支持部分，对于企业级开发而言，丰富更多的功能可以购买企业级Nginx Plus版本。如下以通过VirtualServer和VirtualServerRoute定义upstream配置为例演示功能使用。

1、定义VirtualServer资源,其配置和ingress资源对象类似，能支持的功能会更丰富一点

 1apiVersion: k8s.nginx.org/v1
 2kind: VirtualServer
 3metadata:
 4  name: cafe
 5spec:
 6  host: cafe.example.com
 7  tls:
 8    secret: cafe-secret
 9  upstreams:
10  - name: tea
11    service: tea-svc
12    port: 80
13    name: tea
14    service: ingress-demo 
15    subselector:
16    version: canary
17    lb-method: round_robin
18    fail-timeout: 10s
19    max-fails: 1
20    max-conns: 32
21    keepalive: 32
22    connect-timeout: 30s
23    read-timeout: 30s
24    send-timeout: 30s
25    next-upstream: "error timeout non_idempotent"
26    next-upstream-timeout: 5s
27    next-upstream-tries: 10
28    client-max-body-size: 2m
29    tls:
30      enable: true
31  routes:
32  - path: /tea
33    action:
34      pass: tea

2、应用资源并查看VirtualServer资源列表

1[root@node-1 ~]# kubectl apply -f vs.yaml 
2virtualserver.k8s.nginx.org/cafe unchanged
3[root@node-1 ~]# kubectl get virtualserver
4NAME                 AGE
5cafe                 2m52s

3、检查ingress控制器的配置文件情况,生成的配置和upstream定义一致

 1nginx@nginx-ingress-7mpfc:/etc/nginx/conf.d$ cat vs_default_cafe.conf 
 2upstream vs_default_cafe_tea {
 3    zone vs_default_cafe_tea 256k;
 4    server 10.244.0.51:80 max_fails=1 fail_timeout=10s max_conns=32;
 5    server 10.244.1.146:80 max_fails=1 fail_timeout=10s max_conns=32;
 6    server 10.244.1.147:80 max_fails=1 fail_timeout=10s max_conns=32;
 7    server 10.244.2.162:80 max_fails=1 fail_timeout=10s max_conns=32;
 8    keepalive 32;
 9}
10
11server {
12    listen 80;
13    server_name cafe.example.com;
14    listen 443 ssl;
15    ssl_certificate /etc/nginx/secrets/default;
16    ssl_certificate_key /etc/nginx/secrets/default;
17    ssl_ciphers NULL;
18    server_tokens "on";
19
20    location /tea {
21        proxy_connect_timeout 30s;
22        proxy_read_timeout 30s;
23        proxy_send_timeout 30s;
24        client_max_body_size 2m;
25        proxy_max_temp_file_size 1024m;
26        proxy_buffering on;
27        proxy_http_version 1.1;
28        set $default_connection_header "";
29        proxy_set_header Upgrade $http_upgrade;
30        proxy_set_header Connection $vs_connection_header;
31        proxy_set_header Host $host;
32        proxy_set_header X-Real-IP $remote_addr;
33        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
34        proxy_set_header X-Forwarded-Host $host;
35        proxy_set_header X-Forwarded-Port $server_port;
36        proxy_set_header X-Forwarded-Proto $scheme;
37
38        proxy_pass https://vs_default_cafe_tea;
39        proxy_next_upstream error timeout non_idempotent;
40        proxy_next_upstream_timeout 5s;
41        proxy_next_upstream_tries 10;   
42    }   
43}

写在最后 #

本文详细介绍了基于nginx的ingress实现，通过实际的案例演示ingress的安装部署，基于虚拟主机的配置，基于TLS加密实现https，高级章节中介绍了负载均衡参数定制，自定义资源虚拟主机和虚拟路由的实现，通过该章节相信能加深对ingress服务暴露机制的理解。实现ingress controller的方式有多种，下一个章节我们将介绍基于HAproxy和TKE Ingress控制器的实现。

参考文献 #

Ingress配置：https://kubernetes.io/docs/concepts/services-networking/ingress/

Ingress控制器：https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/

Nginx ingress安装文档：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md

Nginx ingress文档说明：https://github.com/nginxinc/kubernetes-ingress/tree/master/docs

虚拟主机和路由：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/virtualserver-and-virtualserverroute.md

『转载』该文章来源于网络，侵删。

14 基于nginx Ingress实现服务暴露

Table of Contents