标签：通过   sts   containe   -o   rda   work   入口   alc   doc   

系统环境

1，k8s的版本为1.8.2
2，docker ce的版本为19.03.8-3
3，五台主机操作系统版本为centos7，kernel版本3.10.0-957
4，使用五台主机部署，3台master节点+2台work节点
5，Ingress测试域名t.myk8s.com

部署nginx和apache应用

使用Deployment部署nginx

nginx服务的manifest：

---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.17
        livenessProbe: # 设置存活探针，如果检测失败则重启pod
          httpGet:
            path: /
            port: 80
            scheme: HTTP
          initialDelaySeconds: 30 # 启动容器手册监控检查的等待时间
          timeoutSeconds: 5 # 健康检查请求的超时时间，如果超时，则重启该pod
        readinessProbe: # 设置存活探针，通过探针检测的Pod才会像其转发请求
          httpGet:
            path: /
            port: 80
            scheme: HTTP
        ports:
        - containerPort: 80 # 设置为pod中应用监听的端口

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  ports:
  - name: http
    port: 80 # service服务访问的端口
    protocol: TCP
    targetPort: 80 # 设置为pod中应用监听的端口
  selector:
    app: nginx

apache服务的manifests：

---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: httpd
spec:
  selector:
    matchLabels:
      app: httpd
  replicas: 1 # tells deployment to run 2 pods
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - name: httpd
        image: httpd:2.4
        livenessProbe: # 设置存活探针，如果检测失败则重启pod
          httpGet:
            path: /
            port: 80
            scheme: HTTP
          initialDelaySeconds: 30 # 启动容器手册监控检查的等待时间
          timeoutSeconds: 5 # 健康检查请求的超时时间，如果超时，则重启该pod
        readinessProbe: # 设置存活探针，通过探针检测的Pod才会像其转发请求
          httpGet:
            path: /
            port: 80
            scheme: HTTP
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: httpd
  namespace: default
  labels:
    app: httpd
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: httpd

通过manifest配置k8s，使用--record选项记录命令和版本信息方便后续的升级或者回滚等操作：

# kubectl apply -f nginx-dp.yaml --record
# kubectl apply -f httpd-dp.yaml --record

查看应用状态：

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          3m15s   10.107.199.77    work2   
nginx-5577575ddd-499wr   1/1     Running   0          46m     10.109.125.196   work3   
nginx-5577575ddd-7zwmv   1/1     Running   0          48m     10.99.1.85       work1   

查看services状态，通过services暴露出来的固定ip可以实现对应应用的访问：

# kubectl get services
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
httpd        ClusterIP   10.104.235.200           80/TCP    45m
kubernetes   ClusterIP   10.96.0.1                443/TCP   6d5h
nginx        ClusterIP   10.111.5.31              80/TCP    57m

Ingress配置http访问应用

Ingress控制器的部署方式使用daemon-set+hostNetwork方式，且使用nginx官方提供的nginx-ingress控制器（https://blog.51cto.com/leejia/2495558）。
生成配置Ingress的manifests：

# vim http-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: http-ingress
spec:
  rules:
  - host: t.myk8s.com
    http:
      paths:
      - path: /index.html
        backend:
          serviceName: nginx
          servicePort: 80
      - path: /
        backend:
          serviceName: httpd
          servicePort: 80
# kubectl apply -f http-ingress.yaml

查看状态：

# kubectl get ingress -o wide
NAME           CLASS    HOSTS         ADDRESS   PORTS   AGE
http-ingress      t.myk8s.com             80      102m

master集群中其中一台master的ip为172.18.2.175，在本地pc测试访问:

~ curl -H ‘Host: t.myk8s.com‘ 172.18.2.175
It works!

?  ~ curl -H ‘Host: t.myk8s.com‘ 172.18.2.175/index.html


Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.

For online documentation and support please refer to
nginx.org.

Commercial support is available at
nginx.com.

Thank you for using nginx.

Ingress配置https访问应用

我们使用自签名证书来实现https的访问：

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ingress.key -out ingress.crt -subj "/CN=t.myk8s.com/O=t.myk8s.com"

通过secret存储证书:

# kubectl create secret tls ingress-secret --key ingress.key --cert ingress.crt

# kubectl get secrets
NAME                  TYPE                                  DATA   AGE
ingress-secret        kubernetes.io/tls                     2      2s

注意如下ingress资源要和上面的secret资源在同一个namespace下面：

# vim nginx-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: http-ingress
spec:
  tls:
  - hosts:
    - t.myk8s.com
    secretName: ingress-secret
  rules:
  - host: t.myk8s.com
    http:
      paths:
      - path: /index.html
        backend:
          serviceName: nginx
          servicePort: 80
      - path: /
        backend:
          serviceName: httpd
          servicePort: 80

使用本地pc测试访问，默认http访问也会强制转https：

?  ~ curl https://t.myk8s.com  -k
It works!
?  ~ curl http://t.myk8s.com  -I -k
HTTP/1.1 301 Moved Permanently
Server: nginx/1.17.10
Date: Thu, 21 May 2020 03:53:06 GMT
Content-Type: text/html
Content-Length: 170
Connection: keep-alive
Location: https://t.myk8s.com:443/

?  ~ curl https://t.myk8s.com/index.html  -k





Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.

For online documentation and support please refer to
nginx.org.

Commercial support is available at
nginx.com.

Thank you for using nginx.

可以通过配置nginx-ingress控制器实现http和https同时支持访问，config map的name以及namespace要和nginx -ingress控制器使用的相同：

# vim nginx-cf.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
data:
  ssl-redirect: "false"

# kubectl apply -f nginx-cf.yaml -n nginx-ingress
# kubectl get configmap -n nginx-ingress
NAME           DATA   AGE
nginx-config   1      6d19h

默认nginx-ingress控制器不提供tcp和udp代理功能，但是nginx是支持的，故也可以通过调整nginx-ingress控制器来提供tcp和udp代理服务

配置nginx-ingress做4层tcp代理

打开nginx官方的nginx-ingress仓库，然后修改mainifests，使nginx加载tcp和udp代理的相关配置（即在nginx-ingress.yaml的args下面添加global-configuration配置）：

# git clone https://github.com/nginxinc/kubernetes-ingress/
# cd kubernetes-ingress/deployments
# git checkout v1.7.0
# vim daemon-set/nginx-ingress.yaml
          - -global-configuration=$(POD_NAMESPACE)/nginx-configuration
# kubectl apply -f daemon-set/nginx-ingress.yaml

生成tcp代理相关的manifests：

# vim global-configuration.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration
metadata:
  name: nginx-configuration
  namespace: nginx-ingress
spec:
  listeners:
  - name: nginx-tcp
    port: 8000
    protocol: TCP

# vim global-configuration.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration
metadata:
  name: nginx-configuration
  namespace: nginx-ingress
spec:
  listeners:
  - name: nginx-tcp
    port: 8000
    protocol: TCP
[root@master1 manifests]# cat transport-server-passthrough.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
  name: nginx-tcp
spec:
  listener:
    name: nginx-tcp
    protocol: TCP
  upstreams:
  - name: nginx-app
    service: nginx
    port: 80
  action:
    pass: nginx-app

# kubectl apply -f global-configuration.yaml
# kubectl apply -f transport-server-passthrough.yaml

在本地pc测试访问

?  ~ curl http://172.18.2.175:8000/index.html





Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.

For online documentation and support please refer to
nginx.org.

Commercial support is available at
nginx.com.

Thank you for using nginx.

手动对nginx-ingress的后端nginx服务扩缩容和升级回滚

由2台水平扩容为3台，扩容期间服务无影响：

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77    work2   
nginx-5577575ddd-pz59n   1/1     Running   0          18s     10.99.1.91       work1   
nginx-5577575ddd-qsrb2   1/1     Running   0          18s     10.109.125.207   work3   

# kubectl scale deployment nginx --replicas 3

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77    work2   
nginx-5577575ddd-j4nq4   1/1     Running   0          13s     10.107.199.84    work2   
nginx-5577575ddd-pz59n   1/1     Running   0          68s     10.99.1.91       work1   
nginx-5577575ddd-qsrb2   1/1     Running   0          68s     10.109.125.207   work3   

由3台水平缩容为1台，缩容期间服务无影响：

# kubectl scale deployment nginx --replicas 1

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP              NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77   work2   
nginx-5577575ddd-pz59n   1/1     Running   0          102s    10.99.1.91      work1   

平滑升级，先启动一个新版本的pod，然后流量切到新pod之后，再关闭老版本的pod

# kubectl set image deployment nginx nginx=nginx:1.18 --record

# kubectl rollout status deployment nginx
Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination...

Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination...
deployment "nginx" successfully rolled out

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77    work2   
nginx-648f9b4559-m2b8m   1/1     Running   0          19s     10.109.125.208   work3   

# kubectl get deployment nginx -o yaml|grep  ‘\- image:‘
      - image: nginx:1.18

查看历史操作deployment的命令

# kubectl rollout history deployment nginx
deployment.apps/nginx
REVISION  CHANGE-CAUSE
1         kubectl apply --filename=nginx-dp.yaml --record=true
2         kubectl set image deployment nginx nginx=nginx:1.18 --record=true

可以通过两种方式回滚：
第一种：回滚最近一次操作

# kubectl rollout undo deployment nginx
# kubectl get deployment nginx -o yaml|grep  ‘\- image:‘
      - image: nginx:1.17

第二种：回滚到指定的命令版本，即执行对应的命令

# kubectl rollout undo deployment nginx --to-revision=1

参考文档

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentspec-v1-apps
https://kubernetes.github.io/ingress-nginx/user-guide/tls/
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/
https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/

k8s ingress实现http/https7层和tcp四层代理

标签：通过   sts   containe   -o   rda   work   入口   alc   doc   

原文地址：https://blog.51cto.com/leejia/2497454