3.4 K8s服务发现之 Ingress

四、Ingress & Ingress Controller

Ingress、Ingress Controller概述：

ingress简称：ing

Ingress为 Kubernetes 集群中的服务提供了外部入口URL、负载均衡、SSL终止、HTTP路由等，而 Ingress Controller 监测 Ingress 和 Service 资源的变更，根据规则配置负载均衡、路由规则和 DNS 等，并提供访问入口。Ingress本质就是配置Ingress Controller的规则。Ingress规则的生效需要集群中运行 Ingress Controller。Ingress Controller 与其他由kube-controller-manager管理的 controller 成员不同，Ingress Controller不受controller-manager管理，需要用户选择最适合自己集群的 Ingress Controller，或者自己实现一个。Ingress Controller本质是一个Pod，可以受Deployment或DaemonSet管，理，将该Pod暴露在集群外部网络，可以通过这个Pod作为桥梁实现集群外部流量访问集群内部服务。

ingress-controller

Ingress 这个玩意，简单的理解就是你要改的Nginx配置，在Ingress中配置各种域名对应哪个Service（这个Service不是被用来调度，仅仅是用来挑选满足条件的Endpoints），现在把这个动作抽象出来，变成一个 Ingress 对象，你可以用 yaml 创建，每次不要去改Nginx了，直接改 yaml 然后创建/更新就行了；那么问题来了：”有了Nginx配置，Nginx 咋整？”。Ingress Controller 这东西就是解决 “Nginx 咋整” 的；Ingress Controoler 通过与 Kubernetes API 交互，动态的去感知集群中 Ingress 规则变化，然后读取他，按照他自己模板生成一段 Nginx 配置，再写到 Nginx Pod 里，最后 reload 一下，工作流程如下图

Ingress

常见的ingress-controller：

Kubernetes currently supports and maintains GCE and nginx controllers.
F5 Networks provides support and maintenance for the F5 BIG-IP Controller for Kubernetes.
Kong offers community or commercial support and maintenance for the Kong Ingress Controller for Kubernetes
Traefik is a fully featured ingress controller (Let’s Encrypt, secrets, http2, websocket…), and it also comes with commercial support by Containous
NGINX, Inc. offers support and maintenance for the NGINX Ingress Controller for Kubernetes
HAProxy based ingress controller jcmoraisjr/haproxy-ingress which is mentioned on this blog post HAProxy Ingress Controller for Kubernetes
Istio based ingress controller Control Ingress Traffic

Ingress Controller 以 Kubernetes Pod 的方式部署。比如可以使用 Nginx Ingress Controller：

helm install stable/nginx-ingress --name nginx-ingress --set rbac.create=true

其他 Ingress Controller 还有：

traefik ingress 提供了一个 Traefik Ingress Controller 的实践案例
kubernetes/ingress-nginx 提供了一个详细的 Nginx Ingress Controller 示例
kubernetes/ingress-gce 提供了一个用于 GCE 的 Ingress Controller 示例

1535027227017

上图中，<Service> ingress-nginx部分可以省略，不过要将<IngressController>以DaemonSet方式使用hostnetwork模式运行；如果按照上图示例，需要将<Service> ingress-nginx运行为NodePort类型，用来接入外部流量。

<Service> site1等Service的作用只是关联后端的Pod用来维护后端Pod的动态变化，将管理的后端Pod配置在Ingress上，但是请求不会经过此类Service，而是直接从Ingress直接到达后端的Pod。

部署Ingress Controller & Ingress

Ingress Controller以Pod方式部署：官方文档入口

ingress-nginx就是一个Nginx的Pod，只不过它能够动态地更新upstream的后端池。如果要让集群外部流量访问到ingress-nginx这个Pod，就需要做这个Pod的服务暴露。

方案1：ingress-nginx普通的Pod+NodePort类型的Service

方案2：ingress-nginx这个Pod使用使用宿主机网络(hostNetwork: true)，推荐。网络效率高。

方式1：使用NodePort方式部署ingress-nginx示例:

[root@k8smaster tmp]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml

此时，ingress-nginx也是一个普通的Pod，此时它只能接受集群内部的访问，我们需要再运行一个NodePort类型的Service，用来接入外部流量到ingress-nginx。

[root@k8smaster tmp]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml

[root@k8smaster tmp]# kubectl get svc -n ingress-nginx
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default-http-backend   ClusterIP   10.96.67.240    <none>        80/TCP                       4h
ingress-nginx          NodePort    10.110.60.149   <none>        80:31616/TCP,443:31713/TCP   1m

接入外部流量当然也可以将ingress-nginx的网络设置为共享宿主机网络(不用经过报文的多层转发，效率最高)，修改以上mandatory.yaml文件中的如下字段：
kind:DaemonSet
hostNetwork: true
image:符合国情的image
删除replicas字段。

查看是否部署成功：

[root@k8smaster ingress]# kubectl  get pod -n ingress-nginx
NAME                                        READY     STATUS    RESTARTS   AGE
default-http-backend-6586bc58b6-8tb2r       1/1       Running   0          3h
nginx-ingress-controller-7675fd6cdb-6x6kf   1/1       Running   0          3h

部署成功后，我们就可以在外部网络通过浏览器访问NodeIP:NodePort访问了。注意此处的NodePort未指定，k8s会随机产生，我们也可以在service-nodeport.yaml配置文件中通过nodePort: 30080进行指定。例如：

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

spec:
  type: NodePort
  ports:
  - name: http
    nodePort: 30080 #指定nodePort
    port: 80
    targetPort: 80
    protocol: TCP
  - name: https
    nodePort: 30443  #指定nodePort
    port: 443
    targetPort: 443
    protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx

方式2：共享宿主机网络方式部署ingress-nginx示例（推荐）

接入外部流量当然也可以将ingress-nginx的网络设置为共享宿主机网络(不用经过报文的多层转发，效率最高)，修改以上mandatory.yaml文件中的如下字段：

[root@k8smaster tmp]# curl -O https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
sed -i 's/quay.io/quay-mirror.qiniu.com/g' mandatory.yaml
sed -i 's/kind: Deployment/kind: DaemonSet/g' mandatory.yaml
对以上的yaml文件做如下修改：
	kind: DaemonSet
	hostNetwork: true
	image:符合国情的image
	删除replicas字段。
	对node进行选择（可选）
[root@k8smaster tmp]# kubectl apply -f mandatory.yaml

例如

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx

---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - update
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - update
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      - ingress-controller-leader-nginx
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - endpoints
    verbs:
      - create
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      nodeSelector:
        publicNet: "true"
      hostNetwork: true
      dnsPolicy: ClusterFirst
      containers:
        - name: controller
          image: quay-mirror.qiniu.com/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=ingress-nginx/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: webhook
              containerPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
  namespace: ingress-nginx
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    rules:
      - apiGroups:
          - extensions
          - networking.k8s.io
        apiVersions:
          - v1beta1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission
        path: /extensions/v1beta1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
rules:
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      - validatingwebhookconfigurations
    verbs:
      - get
      - update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-create
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
spec:
  template:
    metadata:
      name: ingress-nginx-admission-create
      labels:
        helm.sh/chart: ingress-nginx-2.0.3
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 0.32.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: create
          image: jettech/kube-webhook-certgen:v1.2.0
          imagePullPolicy: IfNotPresent
          args:
            - create
            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.ingress-nginx.svc
            - --namespace=ingress-nginx
            - --secret-name=ingress-nginx-admission
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-patch
  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
spec:
  template:
    metadata:
      name: ingress-nginx-admission-patch
      labels:
        helm.sh/chart: ingress-nginx-2.0.3
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 0.32.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: patch
          image: jettech/kube-webhook-certgen:v1.2.0
          imagePullPolicy:
          args:
            - patch
            - --webhook-name=ingress-nginx-admission
            - --namespace=ingress-nginx
            - --patch-mutating=false
            - --secret-name=ingress-nginx-admission
            - --patch-failure-policy=Fail
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx

启用ValidatingAdmissionWebhook

参考连接

说明：新版本ingress-controller部署已经默认开启了ValidatingAdmissionWebhook。对于helm部署，可以通过.Values.controller.admissionWebhooks.enabled配置进行开启。参考：https://zhuanlan.zhihu.com/p/500693796
对于老版本，可以参考如下部分进行开启。

创建一个有问题的ingress,，会影响所有新创建的ingress规则，这导致一个集群级别的Bug诞生了。那么有没有办法,，提前检验ingress配置，有问题就不去reload。那验证步骤肯定要在请求到达nginx-controller之前来做，是不是想到了k8s-admission-webhook，可以在apiserver持久化对象前拦截请求，去实现自定义的验证规则。好在新版本的nginx-ingress-controller(v0.25.0+)已经实现了相关的功能，只需开启对应配置就行。

ApiServer配置

Apiserver开启webhook相关配置, 必须包含MutatingAdmissionWebhook与ValidatingAdmissionWebhook

--admission-control=MutatingAdmissionWebhook,ValidatingAdmissionWebhook

创建webhook相关配置

启用ValidatingAdmissionWebhook必须使用https, 需要配置对应证书，可以使用如下两方式生成证书：

方式1：手动生成:

openssl req -x509 -newkey rsa:2048 -keyout certificate.pem -out key.pem -days 365 -nodes -subj "/CN=ingress-validation-webhook.ingress-nginx.svc"

方式2：CertificateSigningRequest

通过k8s CertificateSigningRequest来创建(controller-manager需要开启–cluster-signing-cert-file与–cluster-signing-key-file)，可通过如下脚本创建, namespace与service替换成自己的。

SERVICE_NAME=ingress-nginx
NAMESPACE=ingress-nginx

TEMP_DIRECTORY=$(mktemp -d)
echo "creating certs in directory ${TEMP_DIRECTORY}"

cat <<EOF >> ${TEMP_DIRECTORY}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${SERVICE_NAME}
DNS.2 = ${SERVICE_NAME}.${NAMESPACE}
DNS.3 = ${SERVICE_NAME}.${NAMESPACE}.svc
EOF

openssl genrsa -out ${TEMP_DIRECTORY}/server-key.pem 2048
openssl req -new -key ${TEMP_DIRECTORY}/server-key.pem \
    -subj "/CN=${SERVICE_NAME}.${NAMESPACE}.svc" \
    -out ${TEMP_DIRECTORY}/server.csr \
    -config ${TEMP_DIRECTORY}/csr.conf

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${SERVICE_NAME}.${NAMESPACE}.svc
spec:
  request: $(cat ${TEMP_DIRECTORY}/server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

kubectl certificate approve ${SERVICE_NAME}.${NAMESPACE}.svc

for x in $(seq 10); do
    SERVER_CERT=$(kubectl get csr ${SERVICE_NAME}.${NAMESPACE}.svc -o jsonpath='{.status.certificate}')
    if [[ ${SERVER_CERT} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${SERVER_CERT} == '' ]]; then
    echo "ERROR: After approving csr ${SERVICE_NAME}.${NAMESPACE}.svc, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi
echo ${SERVER_CERT} | openssl base64 -d -A -out ${TEMP_DIRECTORY}/server-cert.pem

kubectl create secret generic ingress-nginx.svc \
    --from-file=key.pem=${TEMP_DIRECTORY}/server-key.pem \
    --from-file=cert.pem=${TEMP_DIRECTORY}/server-cert.pem \
    -n ${NAMESPACE}

配置ingress controller

ingress controller需要启用如下参数, 挂载需要的tls证书

flag	description	example usage
–validating-webhook	admission webhook的地址	:8080
–validating-webhook-certificate	webhook证书	/usr/local/certificates/validating-webhook.pem
–validating-webhook-key	webhook私钥	/usr/local/certificates/validating-webhook-key.pem

修改ingress-nginx的默认监听端口

有时候80端口被其他服务占用，我们想修改ingress的默认端口

kubectl edit daemonset nginx-ingress-controller -n ingress-nginx
containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --annotations-prefix=nginx.ingress.kubernetes.io
在后面加上
        - --http-port=8080
        - --https-port=8443

删除pod，k8s将重新使用新配置，生成一个新的pod

定义Ingress资源

接下来就可以定义Ingress资源了（可以理解为定义nginx规则），此处定义一个基于虚拟主机的Ingress：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: myapp.magedu.com
    http:
      paths:
      - path:  #此处缺省value，则为“/”
        backend:
          serviceName: myapp
          servicePort: 80

此时我们就可以通过浏览器访问：http://myapp.magedu.com:31616/

Ingress的yaml框架总结：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"  #如果一个集群中有多种ingress-controller，需要指明ingress.class只让某中ingress-controller来适配这些规则。如果集群中只有一种，可以省略。缺省时，全部的ingress-controller都会解析Ingress资源。
    
spec:
  backend: #默认后端，全局的backend，当未匹配到任何规则时使用此backend？？？
    serviceName: testsvc
    servicePort: 80
  rules: #定义规则
  - host: foo.bar.com  #必须为一个FQDN,不能是IP. 如果未指定host，为"*"，则Ingress根据指定的IngressRuleValue路由所有流量。
    http:
      paths:
      - path: /foo
        backend:
          serviceName: svc1
          servicePort: 80
  - host: bar.foo.com
  tls: #定义https
  - hosts:
    - foo.bar.com
    - bar.foo.com
    secretName: ingress-secret

根据 Ingress Spec 配置的不同，Ingress 可以分为以下几种类型：

1、单服务 Ingress

单服务 Ingress 即该 Ingress 仅指定一个没有任何规则的后端服务。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
spec:
  backend:   #默认后端，全局的backend，当未匹配到任何规则时使用此backend。后端通过Service来关联
    serviceName: testsvc
    servicePort: 80

2、多服务的 Ingress

路由到多服务的 Ingress 即根据请求路径的不同转发到不同的后端服务上，比如

foo.bar.com -> 178.91.123.132 -> / foo    s1:80
                                 / bar    s2:80

可以通过下面的 Ingress 来定义：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /  #如果path不是"/",而是“/something” 时需要rewrite，参考：https://kubernetes.github.io/ingress-nginx/examples/rewrite/
spec:
  rules:   #定义Ingress规则，如果缺省或者所有规则都未匹配到，则将流量发送至默认的backend。
  - host: foo.bar.com
    http:
      paths:
      - path: /foo  #如果此处/foo不给出，默认为“/”
        backend:
          serviceName: svc1
          servicePort: 80
      - path: /bar
        backend:
          serviceName: svc2
          servicePort: 80

3、虚拟主机 Ingress

虚拟主机 Ingress 即根据虚拟主机名字的不同转发到不同的后端服务上，而他们共用同一个的 IP 地址，如下所示

foo.bar.com --|                 |-> foo.bar.com s1:80
              | 178.91.123.132  |
bar.foo.com --|                 |-> bar.foo.com s2:80

下面是一个基于 Host header 路由请求的 Ingress：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
spec:
  rules:
  - host: foo.bar.com  #格式为FQDN,目前Ingress的端口是隐含的,隐含在ingress controller中：http为80，https为443。
    http:
      paths:   
      - backend:
          serviceName: svc1
          servicePort: 80
  - host: bar.foo.com
    http:
      paths:
      - backend:
          serviceName: svc2
          servicePort: 80

注：没有定义规则的后端服务称为默认后端服务，可以用来方便的处理 404 页面。

Ingress TLS

#准备证书
[root@physerver manifests]# mkdir pki
[root@physerver manifests]# openssl genrsa -out pki/tls.key 2048
Generating RSA private key, 2048 bit long modulus
.............................................................................+++
...............................+++
e is 65537 (0x10001)
[root@physerver manifests]# ls pki
tls.key
[root@physerver manifests]# openssl req -new -x509 -key pki/tls.key  -out pki/tls.crt -subj "/C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=test.ljzsdut.com"
[root@physerver manifests]# ls pki
tls.crt  tls.key

## 以上2步可是合二为一
#openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=*.ljzsdut.com"

#将证书生成为Secret
[root@physerver manifests]# kubectl create secret tls ljzsdut-ingress-secret --cert=pki/tls.crt --key=pki/tls.key 
secret/ljzsdut-ingress-secret created
[root@physerver manifests]# kubectl get secret
NAME                     TYPE                                  DATA   AGE
ljzsdut-ingress-secret   kubernetes.io/tls                     2      33s
default-token-pp4s5      kubernetes.io/service-account-token   3      20d
[root@physerver manifests]# kubectl describe secrets ljzsdut-ingress-secret 
Name:         ljzsdut-ingress-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1294 bytes
tls.key:  1679 bytes

定义ingress tls

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-main-app
  namespace: default
  annotations: 
    kubernetes.io/ingress.class: "nginx"
spec:
  backend:
    serviceName: cc-main
    servicePort: 80
  rules:
  - host: test.ljzsdut.com
    http:
      paths:
      - path: /
        backend:
          serviceName: cc-main
          servicePort: 80
  tls:
  - hosts:
    - test.ljzsdut.com
    secretName: ljzsdut-ingress-secret

ingress-nginx配置

自定义ingress-nginx的参数

Ingress-nginx背后是Nginx，所以Nginx能实现的Ingress同样可以实现，但是实现方式不太一样，基本都是通过annotations或者ConfigMap实现的，annotations配置的单个Ingress的参数，ConfigMap配置的是全局的所有的Ingress的参数。此外还可以通过自定义Nginx配置文件模板实现更高层次的自定义配置。

例如修改client_max_body_size值为100m，可以通过如下方法：

方法1：在单个Ingress的资源配置文件中加入annotations，实现单个Ingress参数的定制（server块中的配置）：

metadata:
  name: nginx-ingress-main-app
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/proxy-body-size: 100m  #局部配置

方法2：在ConfigMap配置文件中定义data，实现全局配置：

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  proxy-body-size: "90m"  #全局配置
  http-redirect-code: "301"

方法3：修改ingress-nginx的配置文件模板

ingress-nginx的配置文件模板为：/etc/nginx/template/nginx.tmpl。将自定义模板以Volume的方式挂载到容器的/etc/nginx/template上。

        volumeMounts:
          - mountPath: /etc/nginx/template
            name: nginx-template-volume
            readOnly: true
      volumes:
        - name: nginx-template-volume
          configMap:
            name: nginx-template
            items:
            - key: nginx.tmpl
              path: nginx.tmpl

说明：

annotations和ConfigMap两种方式中使用的key名称可能与配置文件参数的名称不同，例如nignx配置文件中的参数“client_max_body_size”对应在annotations和ConfigMap中的key为“proxy-body-size”。具体对应关系需要查阅官方文档：Annotations、ConfigMap、Custom NGINX template

https://kubernetes.github.io/ingress-nginx/examples/

自定义server块中的配置

nginx.ingress.kubernetes.io/server-snippet 注解用于在server块自定义配置，可以写nginx的配置实现更多需求。每个server块只能用一次。 Using the annotation nginx.ingress.kubernetes.io/server-snippet it is possible to add custom configuration in the server configuration block.该注释是将自定义配置加入nginx的server配置中。

例如：

...
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/server-snippet: |
      if ($uri ~* "/gateway/.*") {
          rewrite ^/gateway/(.*) /$1 break;
      }
    nginx.ingress.kubernetes.io/use-regex: "true"
...
spec:
  rules:
  - host: aa.xxx.net
    http:
      paths:
      - backend:
          serviceName: svc-gateway
          servicePort: 8080
        path: /gateway(/|$)(.*)
      - backend:
          serviceName: svc-gateway
          servicePort: 8080
        path: /(admin|bg|auth|monitor)
      - backend:
          serviceName: svc-ui
          servicePort: 80
        path: /
  tls:
  - hosts:
    - aa.xxx.net
    secretName: xxx.net

例如：

nginx.ingress.kubernetes.io/server-snippet: client_header_buffer_size 2046k;

例如：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /    
    nginx.ingress.kubernetes.io/server-snippet: |
      location /mpweixin {
           rewrite ^/mpweixin/(.*)$ https://mp.weixin.qq.com/$1 break;   #小程序外链其他公众号
      }      
  name: approot
  namespace: default
spec:
  rules:
  - host: approot.bar.com

实现了nginx的如下配置：

	## start server approot.bar.com
	server {
		server_name approot.bar.com ;

		listen 80;

		listen [::]:80;

		set $proxy_upstream_name "-";

		location /mpweixin {
			rewrite ^/mpweixin/(.*)$ https://mp.weixin.qq.com/$1 break;   #小程序外链其他公众号
		}

		location / {

			set $namespace      "";
			set $ingress_name   "";
			set $service_name   "";
			set $service_port   "0";
			set $location_path  "/";

			rewrite_by_lua_block {

				balancer.rewrite()

			}

			log_by_lua_block {

				balancer.log()

				monitor.call()
			}

			access_log off;

			port_in_redirect off;

			set $proxy_upstream_name "upstream-default-backend";

			client_max_body_size                    1m;

			proxy_set_header Host                   $best_http_host;

			# Pass the extracted client certificate to the backend

			# Allow websocket connections
			proxy_set_header                        Upgrade           $http_upgrade;

			proxy_set_header                        Connection        $connection_upgrade;

			proxy_set_header X-Request-ID           $req_id;
			proxy_set_header X-Real-IP              $the_real_ip;

			proxy_set_header X-Forwarded-For        $the_real_ip;

			proxy_set_header X-Forwarded-Host       $best_http_host;
			proxy_set_header X-Forwarded-Port       $pass_port;
			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

			proxy_set_header X-Original-URI         $request_uri;

			proxy_set_header X-Scheme               $pass_access_scheme;

			# Pass the original X-Forwarded-For
			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			proxy_set_header Proxy                  "";

			# Custom headers to proxied server

			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;

			proxy_buffering                         off;
			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;
			proxy_request_buffering                 on;

			proxy_http_version                      1.1;

			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;

			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_tries               3;

			proxy_pass http://upstream_balancer;

			proxy_redirect                          off;

		}

	}
	## end server approot.bar.com

自定义location中的配置

#url重写：api.rencaiyoujia.cn/assessh5  --> assessh5.rencaiyoujia.cn
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
            rewrite ^/assessh5(/|$)(.*)$ https://assessh5-test.rencaiyoujia.cn/$2 redirect;
  name: otherfiles-assessh5-rewriteurl
  namespace: oppc-rcyj-web
spec:
  rules:
  - host: api.rencaiyoujia.cn
    http:
      paths:
      - backend:
          serviceName: otherfiles
          servicePort: 80
        path: /assessh5

上面的配置rewrite ^/assessh5(/|$)(.*)$ https://assessh5-test.rencaiyoujia.cn/$2 redirect;会在location /assessh5 {}中出现。

自定义nginx全局配置

使用ConfigMap可以自定义NGINX配置，例如修改超时时间、修改访问日志格式。

kubectl edit cm nginx-configuration -n ingress-nginx configmap修改后，一段时间后，nginx自动生效。

apiVersion: v1
data:
  proxy-connect-timeout: "10"
  proxy-read-timeout: "120"
  proxy-send-timeout: "120"
  # k8s的nginx默认只支持TLS1.2，不支持TLS1.0和TLS1.1。参考文档：https://kubernetes.github.io/ingress-nginx/user-guide/tls/#legacy-tls
  ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"
  ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
kind: ConfigMap
metadata:
  name: nginx-configuration

ingress-nginx的补充配置参考文档:https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md

对于server级别的配置，可是使用nginx.ingress.kubernetes.io/server-snippet实现。

ingress-nginx配置basic-auth

https://kubernetes.github.io/ingress-nginx/examples/auth/basic/

ingress-nginx进行url重写

Name	Description	Values
nginx.ingress.kubernetes.io/rewrite-target	流量必须重定向的目标URI	string
nginx.ingress.kubernetes.io/ssl-redirect	location仅允许访问ssl（Ingress包含证书时默认为True）	bool
nginx.ingress.kubernetes.io/force-ssl-redirect	强制访问https，即使ingress没有启用tls	bool
nginx.ingress.kubernetes.io/app-root	如果Ingress中的path为“/”,则重定向的app-root	string
nginx.ingress.kubernetes.io/use-regex	在Ingress定义的路径是否使用正则表达式	bool

示例1：url路径重写

缩减URI

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2  #改写请求后端的uri
    #nginx.ingress.kubernetes.io/rewrite-target: https://foo.bar.com:8080/$2  #这种带域名的情况会将重写后的请求返回给浏览器重新发起。
  name: rewrite
  namespace: default
spec:
  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)

上例中，会有如下重写：

rewrite.bar.com/something 会重定向到 rewrite.bar.com/ ；

rewrite.bar.com/something/会重定向到 rewrite.bar.com/ ；

rewrite.bar.com/something/new 会重定向到 rewrite.bar.com/new

示例1-2：与上例实现相同效果

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: approot
  namespace: default
spec:
  rules:
  - host: approot.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /enterprise

上例中，访问http://approot.bar.com/enterprise/api会将请求重写为http://approot.bar.com/api发送给后端。等同于nginx的如下配置：

       location /enterprise {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto https;
            proxy_redirect off;
            proxy_pass http://approot.bar.com/;   #注意此处的"/"
       }

示例2：app-root

扩展URI

代理层请求：/mI31jblt57.txt
后端请求： /inflow/mI31jblt57.txt
即：访问/mI31jblt57.txt ，会请求后端的/inflow/mI31jblt57.txt。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/app-root: /inflow
  name: weixin-code
  namespace: api
spec:
  rules:
  - host: approot.bar.com
    http:
      paths:
      - backend:
          serviceName: inflow-exponent-html
          servicePort: 80
        path: /mI31jblt57.txt
  tls:
  - hosts:
    - approot.bar.com
    secretName: approot-bar-com-tls-sercret

上例中，访问http://approot.bar.com/mI31jblt57.txt会将请求重写为http://inflow-exponent-html/inflow/mI31jblt57.txt发送给后端。等同于nginx的如下配置：

       upstream inflow-exponent-html {...}
       .....
       location /mI31jblt57.txt {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto https;
            proxy_redirect off;
            proxy_pass http://inflow-exponent-html/inflow/mI31jblt57.txt; 
       }

示例3：域名重写

#url重写：api.rencaiyoujia.cn/assessh5  --> assessh5.rencaiyoujia.cn
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
            rewrite ^/assessh5(/|$)(.*)$ https://assessh5-test.rencaiyoujia.cn/$2 redirect;
  name: otherfiles-assessh5-rewriteurl
  namespace: oppc-rcyj-web
spec:
  rules:
  - host: api.rencaiyoujia.cn
    http:
      paths:
      - backend:
          serviceName: otherfiles
          servicePort: 80
        path: /assessh5

http重定向https

参考下配置项的文档：https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/

为ingress配置增加注解（annotations）：[nginx.ingress.kubernetes.io/ssl-redirect: ‘true‘ 就可以实现http强制跳转至https

不过默认情况ingress是通过308重定向跳转到https， ie浏览器不一定支持308状态，可以通过如下方式修改ingress配置，让ingress通过301跳转到https

这个重定向默认值是 “308”，k8s路由默认http跳转到https，用的是308跳转，ie浏览器，或者有些低版本的浏览器不支持“308”跳转的，要改成“301”跳转，不然低版本的浏览器会报错。

启用basic认证

文档参考

# docker run -it --rm httpd bash
# 创建文件（auth），并添加用户（dev）
# htpasswd -bc 文件名 用户名 用户密码

# 第一个用户(-c：创建文件)
htpasswd -bc auth dev devpasswd
# 第二个用户(-b：在命令行中提供用户密码)
htpasswd -b auth dev2 devpasswd2

# 创建secret
$ kubectl create secret generic basic-auth --from-file=auth

# 在ingress中添加注解
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-with-auth
  annotations:
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /
        backend:
          serviceName: http-svc
          servicePort: 80
          
          
# 验证
$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' -u 'foo:bar'

TCP或UDP代理

https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

第一步，更改ingress-nginx的deployment启动参数，添加–tcp-services-configmap和–udp-services-configmap参数，开启tcp与udp的支持

containers:
- args:
  - /nginx-ingress-controller
  - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
  - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
  
# 一般都已经开启，POD_NAMESPACE已经在pod中声明为环境变量

第二步，更改ingress-nginx的service，声明tcp和udp用的端口号

  ports:
  - name: proxied-tcp
    nodePort: 30090
    port: 9000
    protocol: TCP
    targetPort: 9000
  - name: proxied-udp
    nodePort: 30091
    port: 9001
    protocol: UDP
    targetPort: 9001
  - name: nginx
    port: 9005
    protocol: TCP
    targetPort: 9005

第三步，定义configmap，格式为<ingress-controller-svc-port>:"<namespace>/<service-name>:<port>",例如下面配置的data第一行表示将default命名空间下的example-go服务的8080端口映射到ingress-controller service的9000端口，即可通过ingress-controller的service ip加9000端口访问到example-go服务

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  23000: "im/im:23000"
  9000: "default/example-go:8080"
  9005: "default/nginx:80"

配置URL重定向的路由服务

Rewrite配置

配置安全的路由服务

配置HTTPS双向认证

配置HTTPS服务转发到后端容器为HTTPS协议

配置域名支持正则化

配置域名支持泛化

通过注解实现灰度发布

使用cert-manager申请免费的HTTPS证书

会话粘性(Sticky sessions)

官方文档

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-test
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "INGRESSCOOKIE"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"

spec:
  ingressClassName: nginx
  rules:
  - host: stickyingress.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: http-svc
            port: 
              number: 80

测试：

$ kubectl describe ing nginx-test
Name:           nginx-test
Namespace:      default
Address:
Default backend:    default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080)
Rules:
  Host                          Path    Backends
  ----                          ----    --------
  stickyingress.example.com
                                /        nginx-service:80 (<none>)
Annotations:
  affinity: cookie
  session-cookie-name:      INGRESSCOOKIE
  session-cookie-expires: 172800
  session-cookie-max-age: 172800
Events:
  FirstSeen LastSeen    Count   From                SubObjectPath   Type        Reason  Message
  --------- --------    -----   ----                -------------   --------    ------  -------
  7s        7s      1   {ingress-nginx-controller }         Normal      CREATE  default/nginx-test


$ curl -I http://stickyingress.example.com
HTTP/1.1 200 OK
Server: nginx/1.11.9
Date: Fri, 10 Feb 2017 14:11:12 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Set-Cookie: INGRESSCOOKIE=a9907b79b248140b56bb13723f72b67697baac3d; Expires=Sun, 12-Feb-17 14:11:12 GMT; Max-Age=172800; Path=/; HttpOnly
Last-Modified: Tue, 24 Jan 2017 14:02:19 GMT
ETag: "58875e6b-264"
Accept-Ranges: bytes

在上面的示例中，您可以看到响应包含带有我们定义的设置的 Set-Cookie 标头。此 cookie 由 NGINX Ingress Controller创建，它包含一个随机生成的密钥，用于该请求的上游server（使用一致哈希选择），并具有一个 Expires 指令。如果客户端发送的 cookie 与上游不对应，NGINX 会选择上游并创建相应的 cookie。

Custom NGINX upstream hashing

官方文档

类似于session粘性，但是更加灵活，可以通过”nginx变量、文本及二者的组合“作为hash key。

nginx.ingress.kubernetes.io/upstream-hash-by: the nginx variable, text value or any combination thereof to use for consistent hashing.

For example: nginx.ingress.kubernetes.io/upstream-hash-by: "$request_uri" or nginx.ingress.kubernetes.io/upstream-hash-by: "$request_uri$host" or

nginx.ingress.kubernetes.io/upstream-hash-by: "${request_uri}-text-value" to consistently hash upstream requests by the current request URI.

“subset” hashing can be enabled setting nginx.ingress.kubernetes.io/upstream-hash-by-subset: “true”. This maps requests to subset of nodes instead of a single one. upstream-hash-by-subset-size determines the size of each subset (default 3).

kind: Service
apiVersion: v1
metadata:
  name: nginxhello
  labels:
    app: nginxhello
spec:
  selector:
    app: nginxhello
  ports:
    - name: http
      port: 80
      targetPort: 8080

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_predictorid"
    nginx.ingress.kubernetes.io/upstream-hash-by-subset: "true"
    nginx.ingress.kubernetes.io/upstream-hash-by-subset-size: "3"
  name: nginxhello-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: foo.bar.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: nginxhello
              port:
                number: 80

其他参考：

https://mritd.me/2017/03/04/how-to-use-nginx-ingress/

https://mritd.me/2016/12/06/try-traefik-on-kubernetes/

https://blog.csdn.net/jacksonary/article/details/94756633