Kubernetes——part4-2 Kubernetes集群 服务暴露 Traefik

Kubernetes集群 服务暴露 Traefik

一、认识traefik

1.1 traefik简介

• 参考链接: https://traefik.cn/

• 是一个为了让部署微服务更加便捷而诞生的现代HTTP反向代理、负载均衡工具。

• 它支持多种后台 (Docker, Swarm, Kubernetes, Marathon, Mesos, Consul, Etcd, Zookeeper, BoltDB, Rest API, file…) 来自动化、动态的应用它的配置文件设置。

1.2 traefix 特性

• 非常快
• 无需安装其他依赖，通过Go语言编写的单一可执行文件
• 支持 Rest API
• 多种后台支持：Docker, Swarm, Kubernetes, Marathon, Mesos, Consul, Etcd, 并且还会更多
• 后台监控, 可以监听后台变化进而自动化应用新的配置文件设置
• 配置文件热更新。无需重启进程
• 正常结束http连接
• 后端断路器
• 轮询，rebalancer 负载均衡
• Rest Metrics
• 支持最小化官方docker 镜像
• 前、后台支持SSL
• 清爽的AngularJS前端页面
• 支持Websocket
• 支持HTTP/2
• 网络错误重试
• 支持Let’s Encrypt (自动更新HTTPS证书)
• 高可用集群模式

1.3 traefik与nginx ingress对比

1.4 traefik核心概念及能力

Traefik是一个边缘路由器，它会拦截外部的请求并根据逻辑规则选择不同的操作方式，这些规则决定着这些请求到底该如何处理。Traefik提供自动发现能力，会实时检测服务，并自动更新路由规则。

从上图可知，请求首先会连接到entrypoints，然后分析这些请求是否与定义的rules匹配，如果匹配，则会通过一系列middlewares，再到对应的services上。

这就涉及到以下几个重要的核心组件。

• Providers
• Entrypoints
• Routers
• Services
• Middlewares

下面分别介绍一下：

• Providers

Providers是基础组件，Traefik的配置发现是通过它来实现的，它可以是协调器，容器引擎，云提供商或者键值存储。

Traefik通过查询Providers的API来查询路由的相关信息，一旦检测到变化，就会动态的更新路由。

• Entrypoints

Entrypoints是Traefik的网络入口，它定义接收请求的接口，以及是否监听TCP或者UDP。

• Routers

Routers主要用于分析请求，并负责将这些请求连接到对应的服务上去，在这个过程中，Routers还可以使用Middlewares来更新请求，比如在把请求发到服务之前添加一些Headers。

• Services

Services负责配置如何到达最终将处理传入请求的实际服务。

• Middlewares

Middlewares用来修改请求或者根据请求来做出一些判断（authentication, rate limiting, headers, …），中间件被附件到路由上，是一种在请求发送到你的服务之前（或者在服务的响应发送到客户端之前）调整请求的一种方法。

二、traefik部署

2.1 获取traefik部署前置资源清单文件

2.1.1 创建CRD资源

本次Traefik 是部署在 kube-system Namespace 下，如果不想部署到配置的 Namespace，需要修改下面部署文件中的 Namespace 参数。

此yaml资源清单文件可在traefik.io网站直接复制使用：https://doc.traefik.io/traefik/v2.5/reference/dynamic-configuration/kubernetes-crd/#definitions

# vim traefik-crd.yaml
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.6.2creationTimestamp: nullname: ingressroutes.traefik.containo.us
spec:group: traefik.containo.usnames:kind: IngressRoutelistKind: IngressRouteListplural: ingressroutessingular: ingressroutescope: Namespacedversions:- name: v1alpha1schema:openAPIV3Schema:description: IngressRoute is an Ingress CRD specification.properties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: IngressRouteSpec is a specification for a IngressRouteSpecresource.properties:entryPoints:items:type: stringtype: arrayroutes:items:description: Route contains the set of routes.properties:kind:enum:- Ruletype: stringmatch:type: stringmiddlewares:items:description: MiddlewareRef is a ref to the Middleware resources.properties:name:type: stringnamespace:type: stringrequired:- nametype: objecttype: arraypriority:type: integerservices:items:description: Service defines an upstream to proxy traffic.properties:kind:enum:- Service- TraefikServicetype: stringname:description: Name is a reference to a Kubernetes Serviceobject (for a load-balancer of servers), or to a TraefikServiceobject (service load-balancer, mirroring, etc). Thedifferentiation between the two is specified in theKind field.type: stringnamespace:type: stringpassHostHeader:type: booleanport:anyOf:- type: integer- type: stringx-kubernetes-int-or-string: trueresponseForwarding:description: ResponseForwarding holds configuration forthe forward of the response.properties:flushInterval:type: stringtype: objectscheme:type: stringserversTransport:type: stringsticky:description: Sticky holds the sticky configuration.properties:cookie:description: Cookie holds the sticky configurationbased on cookie.properties:httpOnly:type: booleanname:type: stringsameSite:type: stringsecure:type: booleantype: objecttype: objectstrategy:type: stringweight:description: Weight should only be specified when Namereferences a TraefikService object (and to be precise,one that embeds a Weighted Round Robin).type: integerrequired:- nametype: objecttype: arrayrequired:- kind- matchtype: objecttype: arraytls:description: "TLS contains the TLS certificates configuration of theroutes. To enable Let's Encrypt, use an empty TLS struct, e.g. inYAML: \n \t tls: {} # inline format \n \t tls: \t   secretName:# block format"properties:certResolver:type: stringdomains:items:description: Domain holds a domain name with SANs.properties:main:type: stringsans:items:type: stringtype: arraytype: objecttype: arrayoptions:description: Options is a reference to a TLSOption, that specifiesthe parameters of the TLS connection.properties:name:type: stringnamespace:type: stringrequired:- nametype: objectsecretName:description: SecretName is the name of the referenced KubernetesSecret to specify the certificate details.type: stringstore:description: Store is a reference to a TLSStore, that specifiesthe parameters of the TLS store.properties:name:type: stringnamespace:type: stringrequired:- nametype: objecttype: objectrequired:- routestype: objectrequired:- metadata- spectype: objectserved: truestorage: true
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.6.2creationTimestamp: nullname: ingressroutetcps.traefik.containo.us
spec:group: traefik.containo.usnames:kind: IngressRouteTCPlistKind: IngressRouteTCPListplural: ingressroutetcpssingular: ingressroutetcpscope: Namespacedversions:- name: v1alpha1schema:openAPIV3Schema:description: IngressRouteTCP is an Ingress CRD specification.properties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: IngressRouteTCPSpec is a specification for a IngressRouteTCPSpecresource.properties:entryPoints:items:type: stringtype: arrayroutes:items:description: RouteTCP contains the set of routes.properties:match:type: stringmiddlewares:description: Middlewares contains references to MiddlewareTCPresources.items:description: ObjectReference is a generic reference to a Traefikresource.properties:name:type: stringnamespace:type: stringrequired:- nametype: objecttype: arrayservices:items:description: ServiceTCP defines an upstream to proxy traffic.properties:name:type: stringnamespace:type: stringport:anyOf:- type: integer- type: stringx-kubernetes-int-or-string: trueproxyProtocol:description: ProxyProtocol holds the ProxyProtocol configuration.properties:version:type: integertype: objectterminationDelay:type: integerweight:type: integerrequired:- name- porttype: objecttype: arrayrequired:- matchtype: objecttype: arraytls:description: "TLSTCP contains the TLS certificates configuration ofthe routes. To enable Let's Encrypt, use an empty TLS struct, e.g.in YAML: \n \t tls: {} # inline format \n \t tls: \t   secretName:# block format"properties:certResolver:type: stringdomains:items:description: Domain holds a domain name with SANs.properties:main:type: stringsans:items:type: stringtype: arraytype: objecttype: arrayoptions:description: Options is a reference to a TLSOption, that specifiesthe parameters of the TLS connection.properties:name:type: stringnamespace:type: stringrequired:- nametype: objectpassthrough:type: booleansecretName:description: SecretName is the name of the referenced KubernetesSecret to specify the certificate details.type: stringstore:description: Store is a reference to a TLSStore, that specifiesthe parameters of the TLS store.properties:name:type: stringnamespace:type: stringrequired:- nametype: objecttype: objectrequired:- routestype: objectrequired:- metadata- spectype: objectserved: truestorage: true
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.6.2creationTimestamp: nullname: ingressrouteudps.traefik.containo.us
spec:group: traefik.containo.usnames:kind: IngressRouteUDPlistKind: IngressRouteUDPListplural: ingressrouteudpssingular: ingressrouteudpscope: Namespacedversions:- name: v1alpha1schema:openAPIV3Schema:description: IngressRouteUDP is an Ingress CRD specification.properties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: IngressRouteUDPSpec is a specification for a IngressRouteUDPSpecresource.properties:entryPoints:items:type: stringtype: arrayroutes:items:description: RouteUDP contains the set of routes.properties:services:items:description: ServiceUDP defines an upstream to proxy traffic.properties:name:type: stringnamespace:type: stringport:anyOf:- type: integer- type: stringx-kubernetes-int-or-string: trueweight:type: integerrequired:- name- porttype: objecttype: arraytype: objecttype: arrayrequired:- routestype: objectrequired:- metadata- spectype: objectserved: truestorage: true
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.6.2creationTimestamp: nullname: middlewares.traefik.containo.us
spec:group: traefik.containo.usnames:kind: MiddlewarelistKind: MiddlewareListplural: middlewaressingular: middlewarescope: Namespacedversions:- name: v1alpha1schema:openAPIV3Schema:description: Middleware is a specification for a Middleware resource.properties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: MiddlewareSpec holds the Middleware configuration.properties:addPrefix:description: AddPrefix holds the AddPrefix configuration.properties:prefix:type: stringtype: objectbasicAuth:description: BasicAuth holds the HTTP basic authentication configuration.properties:headerField:type: stringrealm:type: stringremoveHeader:type: booleansecret:type: stringtype: objectbuffering:description: Buffering holds the request/response buffering configuration.properties:maxRequestBodyBytes:format: int64type: integermaxResponseBodyBytes:format: int64type: integermemRequestBodyBytes:format: int64type: integermemResponseBodyBytes:format: int64type: integerretryExpression:type: stringtype: objectchain:description: Chain holds a chain of middlewares.properties:middlewares:items:description: MiddlewareRef is a ref to the Middleware resources.properties:name:type: stringnamespace:type: stringrequired:- nametype: objecttype: arraytype: objectcircuitBreaker:description: CircuitBreaker holds the circuit breaker configuration.properties:expression:type: stringtype: objectcompress:description: Compress holds the compress configuration.properties:excludedContentTypes:items:type: stringtype: arraytype: objectcontentType:description: ContentType middleware - or rather its unique `autoDetect`option - specifies whether to let the `Content-Type` header, ifit has not been set by the backend, be automatically set to a valuederived from the contents of the response. As a proxy, the defaultbehavior should be to leave the header alone, regardless of whatthe backend did with it. However, the historic default was to alwaysauto-detect and set the header if it was nil, and it is going tobe kept that way in order to support users currently relying onit. This middleware exists to enable the correct behavior untilat least the default one can be changed in a future version.properties:autoDetect:type: booleantype: objectdigestAuth:description: DigestAuth holds the Digest HTTP authentication configuration.properties:headerField:type: stringrealm:type: stringremoveHeader:type: booleansecret:type: stringtype: objecterrors:description: ErrorPage holds the custom error page configuration.properties:query:type: stringservice:description: Service defines an upstream to proxy traffic.properties:kind:enum:- Service- TraefikServicetype: stringname:description: Name is a reference to a Kubernetes Service object(for a load-balancer of servers), or to a TraefikServiceobject (service load-balancer, mirroring, etc). The differentiationbetween the two is specified in the Kind field.type: stringnamespace:type: stringpassHostHeader:type: booleanport:anyOf:- type: integer- type: stringx-kubernetes-int-or-string: trueresponseForwarding:description: ResponseForwarding holds configuration for theforward of the response.properties:flushInterval:type: stringtype: objectscheme:type: stringserversTransport:type: stringsticky:description: Sticky holds the sticky configuration.properties:cookie:description: Cookie holds the sticky configuration basedon cookie.properties:httpOnly:type: booleanname:type: stringsameSite:type: stringsecure:type: booleantype: objecttype: objectstrategy:type: stringweight:description: Weight should only be specified when Name referencesa TraefikService object (and to be precise, one that embedsa Weighted Round Robin).type: integerrequired:- nametype: objectstatus:items:type: stringtype: arraytype: objectforwardAuth:description: ForwardAuth holds the http forward authentication configuration.properties:address:type: stringauthRequestHeaders:items:type: stringtype: arrayauthResponseHeaders:items:type: stringtype: arrayauthResponseHeadersRegex:type: stringtls:description: ClientTLS holds TLS specific configurations as client.properties:caOptional:type: booleancaSecret:type: stringcertSecret:type: stringinsecureSkipVerify:type: booleantype: objecttrustForwardHeader:type: booleantype: objectheaders:description: Headers holds the custom header configuration.properties:accessControlAllowCredentials:description: AccessControlAllowCredentials is only valid if true.false is ignored.type: booleanaccessControlAllowHeaders:description: AccessControlAllowHeaders must be used in responseto a preflight request with Access-Control-Request-Headers set.items:type: stringtype: arrayaccessControlAllowMethods:description: AccessControlAllowMethods must be used in responseto a preflight request with Access-Control-Request-Method set.items:type: stringtype: arrayaccessControlAllowOriginList:description: AccessControlAllowOriginList is a list of allowableorigins. Can also be a wildcard origin "*".items:type: stringtype: arrayaccessControlAllowOriginListRegex:description: AccessControlAllowOriginListRegex is a list of allowableorigins written following the Regular Expression syntax (https://golang.org/pkg/regexp/).items:type: stringtype: arrayaccessControlExposeHeaders:description: AccessControlExposeHeaders sets valid headers forthe response.items:type: stringtype: arrayaccessControlMaxAge:description: AccessControlMaxAge sets the time that a preflightrequest may be cached.format: int64type: integeraddVaryHeader:description: AddVaryHeader controls if the Vary header is automaticallyadded/updated when the AccessControlAllowOriginList is set.type: booleanallowedHosts:items:type: stringtype: arraybrowserXssFilter:type: booleancontentSecurityPolicy:type: stringcontentTypeNosniff:type: booleancustomBrowserXSSValue:type: stringcustomFrameOptionsValue:type: stringcustomRequestHeaders:additionalProperties:type: stringtype: objectcustomResponseHeaders:additionalProperties:type: stringtype: objectfeaturePolicy:description: 'Deprecated: use PermissionsPolicy instead.'type: stringforceSTSHeader:type: booleanframeDeny:type: booleanhostsProxyHeaders:items:type: stringtype: arrayisDevelopment:type: booleanpermissionsPolicy:type: stringpublicKey:type: stringreferrerPolicy:type: stringsslForceHost:description: 'Deprecated: use RedirectRegex instead.'type: booleansslHost:description: 'Deprecated: use RedirectRegex instead.'type: stringsslProxyHeaders:additionalProperties:type: stringtype: objectsslRedirect:description: 'Deprecated: use EntryPoint redirection or RedirectSchemeinstead.'type: booleansslTemporaryRedirect:description: 'Deprecated: use EntryPoint redirection or RedirectSchemeinstead.'type: booleanstsIncludeSubdomains:type: booleanstsPreload:type: booleanstsSeconds:format: int64type: integertype: objectinFlightReq:description: InFlightReq limits the number of requests being processedand served concurrently.properties:amount:format: int64type: integersourceCriterion:description: SourceCriterion defines what criterion is used togroup requests as originating from a common source. If noneare set, the default is to use the request's remote addressfield. All fields are mutually exclusive.properties:ipStrategy:description: IPStrategy holds the ip strategy configuration.properties:depth:type: integerexcludedIPs:items:type: stringtype: arraytype: objectrequestHeaderName:type: stringrequestHost:type: booleantype: objecttype: objectipWhiteList:description: IPWhiteList holds the ip white list configuration.properties:ipStrategy:description: IPStrategy holds the ip strategy configuration.properties:depth:type: integerexcludedIPs:items:type: stringtype: arraytype: objectsourceRange:items:type: stringtype: arraytype: objectpassTLSClientCert:description: PassTLSClientCert holds the TLS client cert headers configuration.properties:info:description: TLSClientCertificateInfo holds the client TLS certificateinfo configuration.properties:issuer:description: TLSClientCertificateDNInfo holds the client TLScertificate distinguished name info configuration. cf https://tools.ietf.org/html/rfc3739properties:commonName:type: booleancountry:type: booleandomainComponent:type: booleanlocality:type: booleanorganization:type: booleanprovince:type: booleanserialNumber:type: booleantype: objectnotAfter:type: booleannotBefore:type: booleansans:type: booleanserialNumber:type: booleansubject:description: TLSClientCertificateDNInfo holds the client TLScertificate distinguished name info configuration. cf https://tools.ietf.org/html/rfc3739properties:commonName:type: booleancountry:type: booleandomainComponent:type: booleanlocality:type: booleanorganization:type: booleanprovince:type: booleanserialNumber:type: booleantype: objecttype: objectpem:type: booleantype: objectplugin:additionalProperties:x-kubernetes-preserve-unknown-fields: truetype: objectrateLimit:description: RateLimit holds the rate limiting configuration for agiven router.properties:average:format: int64type: integerburst:format: int64type: integerperiod:anyOf:- type: integer- type: stringx-kubernetes-int-or-string: truesourceCriterion:description: SourceCriterion defines what criterion is used togroup requests as originating from a common source. If noneare set, the default is to use the request's remote addressfield. All fields are mutually exclusive.properties:ipStrategy:description: IPStrategy holds the ip strategy configuration.properties:depth:type: integerexcludedIPs:items:type: stringtype: arraytype: objectrequestHeaderName:type: stringrequestHost:type: booleantype: objecttype: objectredirectRegex:description: RedirectRegex holds the redirection configuration.properties:permanent:type: booleanregex:type: stringreplacement:type: stringtype: objectredirectScheme:description: RedirectScheme holds the scheme redirection configuration.properties:permanent:type: booleanport:type: stringscheme:type: stringtype: objectreplacePath:description: ReplacePath holds the ReplacePath configuration.properties:path:type: stringtype: objectreplacePathRegex:description: ReplacePathRegex holds the ReplacePathRegex configuration.properties:regex:type: stringreplacement:type: stringtype: objectretry:description: Retry holds the retry configuration.properties:attempts:type: integerinitialInterval:anyOf:- type: integer- type: stringx-kubernetes-int-or-string: truetype: objectstripPrefix:description: StripPrefix holds the StripPrefix configuration.properties:forceSlash:type: booleanprefixes:items:type: stringtype: arraytype: objectstripPrefixRegex:description: StripPrefixRegex holds the StripPrefixRegex configuration.properties:regex:items:type: stringtype: arraytype: objecttype: objectrequired:- metadata- spectype: objectserved: truestorage: true
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.6.2creationTimestamp: nullname: middlewaretcps.traefik.containo.us
spec:group: traefik.containo.usnames:kind: MiddlewareTCPlistKind: MiddlewareTCPListplural: middlewaretcpssingular: middlewaretcpscope: Namespacedversions:- name: v1alpha1schema:openAPIV3Schema:description: MiddlewareTCP is a specification for a MiddlewareTCP resource.properties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: MiddlewareTCPSpec holds the MiddlewareTCP configuration.properties:ipWhiteList:description: TCPIPWhiteList holds the TCP ip white list configuration.properties:sourceRange:items:type: stringtype: arraytype: objecttype: objectrequired:- metadata- spectype: objectserved: truestorage: true
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.6.2creationTimestamp: nullname: serverstransports.traefik.containo.us
spec:group: traefik.containo.usnames:kind: ServersTransportlistKind: ServersTransportListplural: serverstransportssingular: serverstransportscope: Namespacedversions:- name: v1alpha1schema:openAPIV3Schema:description: ServersTransport is a specification for a ServersTransport resource.properties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: ServersTransportSpec options to configure communication betweenTraefik and the servers.properties:certificatesSecrets:description: Certificates for mTLS.items:type: stringtype: arraydisableHTTP2:description: Disable HTTP/2 for connections with backend servers.type: booleanforwardingTimeouts:description: Timeouts for requests forwarded to the backend servers.properties:dialTimeout:anyOf:- type: integer- type: stringdescription: The amount of time to wait until a connection toa backend server can be established. If zero, no timeout exists.x-kubernetes-int-or-string: trueidleConnTimeout:anyOf:- type: integer- type: stringdescription: The maximum period for which an idle HTTP keep-aliveconnection will remain open before closing itself.x-kubernetes-int-or-string: trueresponseHeaderTimeout:anyOf:- type: integer- type: stringdescription: The amount of time to wait for a server's responseheaders after fully writing the request (including its body,if any). If zero, no timeout exists.x-kubernetes-int-or-string: truetype: objectinsecureSkipVerify:description: Disable SSL certificate verification.type: booleanmaxIdleConnsPerHost:description: If non-zero, controls the maximum idle (keep-alive) tokeep per-host. If zero, DefaultMaxIdleConnsPerHost is used.type: integerpeerCertURI:description: URI used to match against SAN URI during the peer certificateverification.type: stringrootCAsSecrets:description: Add cert file for self-signed certificate.items:type: stringtype: arrayserverName:description: ServerName used to contact the server.type: stringtype: objectrequired:- metadata- spectype: objectserved: truestorage: true
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.6.2creationTimestamp: nullname: tlsoptions.traefik.containo.us
spec:group: traefik.containo.usnames:kind: TLSOptionlistKind: TLSOptionListplural: tlsoptionssingular: tlsoptionscope: Namespacedversions:- name: v1alpha1schema:openAPIV3Schema:description: TLSOption is a specification for a TLSOption resource.properties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: TLSOptionSpec configures TLS for an entry point.properties:alpnProtocols:items:type: stringtype: arraycipherSuites:items:type: stringtype: arrayclientAuth:description: ClientAuth defines the parameters of the client authenticationpart of the TLS connection, if any.properties:clientAuthType:description: ClientAuthType defines the client authenticationtype to apply.enum:- NoClientCert- RequestClientCert- RequireAnyClientCert- VerifyClientCertIfGiven- RequireAndVerifyClientCerttype: stringsecretNames:description: SecretName is the name of the referenced KubernetesSecret to specify the certificate details.items:type: stringtype: arraytype: objectcurvePreferences:items:type: stringtype: arraymaxVersion:type: stringminVersion:type: stringpreferServerCipherSuites:type: booleansniStrict:type: booleantype: objectrequired:- metadata- spectype: objectserved: truestorage: true
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.6.2creationTimestamp: nullname: tlsstores.traefik.containo.us
spec:group: traefik.containo.usnames:kind: TLSStorelistKind: TLSStoreListplural: tlsstoressingular: tlsstorescope: Namespacedversions:- name: v1alpha1schema:openAPIV3Schema:description: TLSStore is a specification for a TLSStore resource.properties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: TLSStoreSpec configures a TLSStore resource.properties:defaultCertificate:description: DefaultCertificate holds a secret name for the TLSOptionresource.properties:secretName:description: SecretName is the name of the referenced KubernetesSecret to specify the certificate details.type: stringrequired:- secretNametype: objectrequired:- defaultCertificatetype: objectrequired:- metadata- spectype: objectserved: truestorage: true
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.6.2creationTimestamp: nullname: traefikservices.traefik.containo.us
spec:group: traefik.containo.usnames:kind: TraefikServicelistKind: TraefikServiceListplural: traefikservicessingular: traefikservicescope: Namespacedversions:- name: v1alpha1schema:openAPIV3Schema:description: TraefikService is the specification for a service (that an IngressRouterefers to) that is usually not a terminal service (i.e. not a pod of servers),as opposed to a Kubernetes Service. That is to say, it usually refers toother (children) services, which themselves can be TraefikServices or Services.properties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: ServiceSpec defines whether a TraefikService is a load-balancerof services or a mirroring service.properties:mirroring:description: Mirroring defines a mirroring service, which is composedof a main load-balancer, and a list of mirrors.properties:kind:enum:- Service- TraefikServicetype: stringmaxBodySize:format: int64type: integermirrors:items:description: MirrorService defines one of the mirrors of a Mirroringservice.properties:kind:enum:- Service- TraefikServicetype: stringname:description: Name is a reference to a Kubernetes Serviceobject (for a load-balancer of servers), or to a TraefikServiceobject (service load-balancer, mirroring, etc). The differentiationbetween the two is specified in the Kind field.type: stringnamespace:type: stringpassHostHeader:type: booleanpercent:type: integerport:anyOf:- type: integer- type: stringx-kubernetes-int-or-string: trueresponseForwarding:description: ResponseForwarding holds configuration forthe forward of the response.properties:flushInterval:type: stringtype: objectscheme:type: stringserversTransport:type: stringsticky:description: Sticky holds the sticky configuration.properties:cookie:description: Cookie holds the sticky configuration basedon cookie.properties:httpOnly:type: booleanname:type: stringsameSite:type: stringsecure:type: booleantype: objecttype: objectstrategy:type: stringweight:description: Weight should only be specified when Name referencesa TraefikService object (and to be precise, one that embedsa Weighted Round Robin).type: integerrequired:- nametype: objecttype: arrayname:description: Name is a reference to a Kubernetes Service object(for a load-balancer of servers), or to a TraefikService object(service load-balancer, mirroring, etc). The differentiationbetween the two is specified in the Kind field.type: stringnamespace:type: stringpassHostHeader:type: booleanport:anyOf:- type: integer- type: stringx-kubernetes-int-or-string: trueresponseForwarding:description: ResponseForwarding holds configuration for the forwardof the response.properties:flushInterval:type: stringtype: objectscheme:type: stringserversTransport:type: stringsticky:description: Sticky holds the sticky configuration.properties:cookie:description: Cookie holds the sticky configuration based oncookie.properties:httpOnly:type: booleanname:type: stringsameSite:type: stringsecure:type: booleantype: objecttype: objectstrategy:type: stringweight:description: Weight should only be specified when Name referencesa TraefikService object (and to be precise, one that embedsa Weighted Round Robin).type: integerrequired:- nametype: objectweighted:description: WeightedRoundRobin defines a load-balancer of services.properties:services:items:description: Service defines an upstream to proxy traffic.properties:kind:enum:- Service- TraefikServicetype: stringname:description: Name is a reference to a Kubernetes Serviceobject (for a load-balancer of servers), or to a TraefikServiceobject (service load-balancer, mirroring, etc). The differentiationbetween the two is specified in the Kind field.type: stringnamespace:type: stringpassHostHeader:type: booleanport:anyOf:- type: integer- type: stringx-kubernetes-int-or-string: trueresponseForwarding:description: ResponseForwarding holds configuration forthe forward of the response.properties:flushInterval:type: stringtype: objectscheme:type: stringserversTransport:type: stringsticky:description: Sticky holds the sticky configuration.properties:cookie:description: Cookie holds the sticky configuration basedon cookie.properties:httpOnly:type: booleanname:type: stringsameSite:type: stringsecure:type: booleantype: objecttype: objectstrategy:type: stringweight:description: Weight should only be specified when Name referencesa TraefikService object (and to be precise, one that embedsa Weighted Round Robin).type: integerrequired:- nametype: objecttype: arraysticky:description: Sticky holds the sticky configuration.properties:cookie:description: Cookie holds the sticky configuration based oncookie.properties:httpOnly:type: booleanname:type: stringsameSite:type: stringsecure:type: booleantype: objecttype: objecttype: objecttype: objectrequired:- metadata- spectype: objectserved: truestorage: true
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []

2.1.2 创建RBAC权限

基于角色的访问控制（RBAC）策略，方便对Kubernetes资源和API进行细粒度控制

traefik需要一定的权限，需要提前创建ServiceAccount并分配一定的权限。

# vim traefik-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:namespace: kube-systemname: traefik-ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: traefik-ingress-controller
rules:- apiGroups:- ""resources:- services- endpoints- secretsverbs:- get- list- watch- apiGroups:- extensions- networking.k8s.ioresources:- ingresses- ingressclassesverbs:- get- list- watch- apiGroups:- extensionsresources:- ingresses/statusverbs:- update- apiGroups:- traefik.containo.usresources:- middlewares- middlewaretcps- ingressroutes- traefikservices- ingressroutetcps- ingressrouteudps- tlsoptions- tlsstores- serverstransportsverbs:- get- list- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: traefik-ingress-controller
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: traefik-ingress-controller
subjects:- kind: ServiceAccountname: traefik-ingress-controllernamespace: kube-system

2.1.3 创建traefik配置文件

由traefik配置很多，通过CLI定义不方便，一般都通过配置文件对traefik进行参数配置，例如使用ConfigMap将配置挂载至traefik中

# vim traefik-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:name: traefiknamespace: kube-system
data:traefik.yaml: |-serversTransport:insecureSkipVerify: true ## 略验证代理服务的 TLS 证书api:insecure: true  ## 允许 HTTP 方式访问 APIdashboard: true  ## 启用 Dashboarddebug: true  ## 启用 Debug 调试模式metrics:prometheus: ""  ## 配置 Prometheus 监控指标数据，并使用默认配置entryPoints:web:address: ":80" ## 配置 80 端口，并设置入口名称为 webwebsecure:address: ":443"  ## 配置 443 端口，并设置入口名称为 websecuremetrics:address: ":8082" ## 配置 8082端口，并设置入口名称为 metricstcpep:address: ":8083"  ## 配置 8083端口，并设置入口名称为 tcpep,做为tcp入口udpep:address: ":8084/udp"  ## 配置 8084端口，并设置入口名称为 udpep，做为udp入口providers:kubernetesCRD: ""  ## 启用 Kubernetes CRD 方式来配置路由规则kubernetesingress: ""  ## 启用 Kubernetes Ingress 方式来配置路由规则kubernetesGateway: "" ## 启用 Kubernetes Gateway APIexperimental:kubernetesGateway: true  ## 允许使用 Kubernetes Gateway APIlog:filePath: "" ## 设置调试日志文件存储路径，如果为空则输出到控制台level: error ## 设置调试日志级别format: json  ## 设置调试日志格式accessLog:filePath: ""  ## 设置访问日志文件存储路径，如果为空则输出到控制台format: json  ## 设置访问调试日志格式bufferingSize: 0  ## 设置访问日志缓存行数filters:retryAttempts: true  ## 设置代理访问重试失败时，保留访问日志minDuration: 20   ## 设置保留请求时间超过指定持续时间的访问日志fields:             ## 设置访问日志中的字段是否保留（keep 保留、drop 不保留）defaultMode: keep ## 设置默认保留访问日志字段names:ClientUsername: drop  headers:defaultMode: keep  ##  设置 Header 中字段是否保留,设置默认保留 Header 中字段names:  ## 针对 Header 中特别字段特别配置保留模式User-Agent: redactAuthorization: dropContent-Type: keep

2.1.4 应用上述资源清单文件

# kubectl apply -f traefik-crd.yaml

# kubectl apply -f traefik-rbac.yaml

# kubectl apply -f traefik-config.yaml

2.1.5 设置节点Label

由于使用DaemonSet方式部署Traefik，所以需要为节点设置label，当应用部署时会根据节点Label进行选择。

设置节点标签
# kubectl label nodes --all IngressProxy=true

查看节点标签
# kubectl get nodes --show-labels

如需要取消时，可执行下述命令
# kubectl label nodes --all IngressProxy-

2.2 部署Traefik

2.2.1 deploy资源清单文件准备

本次将用Daemonset方式部署traefik,便于后期扩展

本次部署通过hostport方式把Pod中容器内的80、443映射到物理机，方便集群外访问。

# vim traefik-deploy.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:namespace: kube-systemname: traefiklabels:app: traefik
spec:selector:matchLabels:app: traefiktemplate:metadata:labels:app: traefikspec:serviceAccountName: traefik-ingress-controllercontainers:- name: traefikimage: traefik:v2.5.7args:- --configfile=/config/traefik.yamlvolumeMounts:- mountPath: /configname: configports:- name: webcontainerPort: 80hostPort: 80  ## 将容器端口绑定所在服务器的 80 端口- name: websecurecontainerPort: 443hostPort: 443  ## 将容器端口绑定所在服务器的 443 端口- name: admincontainerPort: 8080  ## Traefik Dashboard 端口- name: tcpepcontainerPort: 8083hostPort: 8083  ## 将容器端口绑定所在服务器的 8083 端口- name: udpepcontainerPort: 8084hostPort: 8084  ## 将容器端口绑定所在服务器的 8084 端口protocol: UDPvolumes:- name: configconfigMap:name: traefiktolerations:              ## 设置容忍所有污点，防止节点被设置污点- operator: "Exists"nodeSelector:             ## 设置node筛选器，在特定label的节点上启动IngressProxy: "true"

验证端口配置是否正确
# kubectl get daemonset traefik -n kube-system -o yaml

2.2.2 应用deploy资源清单文件

# kubectl apply -f traefix-deploy.yaml

2.2.3 service资源清单文件准备

# vim traefix-service.yaml
apiVersion: v1
kind: Service
metadata:name: traefiknamespace: kube-system
spec:ports:- protocol: TCPname: webport: 80- protocol: TCPname: adminport: 8080- protocol: TCPname: websecureport: 443- protocol: TCPname: tcpepport: 8083- protocol: UDPname: udpepport: 8084selector:app: traefik

2.2.4 应用service资源清单文件

# kubectl apply -f traefik-service.yaml

2.3 配置访问traefik dashboard路由规则

Traefik 应用已经部署完成，但是想让外部访问 Kubernetes 内部服务，还需要配置路由规则，上面部署 Traefik 时开启了 Traefik Dashboard，这是 Traefik 提供的视图看板，所以，首先配置基于 HTTP 的 Traefik Dashboard 路由规则，使外部能够访问 Traefik Dashboard。这里使用 IngressRoute方式进行演示。

2.3.1 Traefik创建路由规则方法

• 原生ingress
• CRD IngressRoute
• Gateway API

2.3.2 通过原生ingress方式暴露traefik dashboard

# kubectl get svc -n kube-system
NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                     AGE
traefik          ClusterIP   10.96.174.88   <none>        80/TCP,8080/TCP,443/TCP,8083/TCP,8084/UDP   6h44m

# kubectl get endpoints -n kube-system
NAME             ENDPOINTS                                                           AGE
traefik          10.244.135.204:80,10.244.159.141:80,10.244.194.92:80 + 17 more...   6h44m

# cat traefik-dashboard-native-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: traefik-dashboard-ingressnamespace: kube-systemannotations:kubernetes.io/ingress.class: traefiktraefik.ingress.kubernetes.io/router.entrypoints: web
spec:rules:- host: tfni.kubemsb.comhttp:paths:- pathType: Prefixpath: /backend:service:name: traefikport:number: 8080

# kubectl apply -f traefik-dashboard-native-ingress.yaml

# kubectl get ingress -n kube-system
NAME                        CLASS    HOSTS              ADDRESS   PORTS   AGE
traefik-dashboard-ingress   <none>   tfni.kubemsb.com             80      56m

# kubectl describe ingress traefik-dashboard-ingress -n kube-system
Name:             traefik-dashboard-ingress
Namespace:        kube-system
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:Host              Path  Backends----              ----  --------tfni.kubemsb.com/   traefik:8080 (10.244.135.204:8080,10.244.159.141:8080,10.244.194.92:8080 + 1 more...)
Annotations:        kubernetes.io/ingress.class: traefiktraefik.ingress.kubernetes.io/router.entrypoints: web
Events:             <none>

在集群之外主机访问

# vim /etc/hosts
......
192.168.10.12 tfni.kubemsb.com

2.3.3 创建dashboard ingress route资源清单文件

# vim traefik-dashboard-ingress-route.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:name: traefiknamespace: kube-system
spec:entryPoints:- webroutes:- match: Host(`traefik.kubemsb.com`) && PathPrefix(`/`)kind: Ruleservices:- name: traefikport: 8080

2.3.4 应用资源清单文件

# kubectl apply -f traefik-dashboard-ingress-route.yaml

2.3.5 在集群内或集群外主机配置域名解析

把域名解析为kubernetes集群任意节点IP既可。
# vim /etc/hosts
192.168.10.12 traefik.kubemsb.com

2.3.6 通过域名访问traefik dashboard

# firefox http://traefik.kubemsb.com &

三、traefik基础应用

3.1 配置kubernetes dashboard路由规则

必须使用SSL证书创建secret密钥

3.1.1 查看kubernetes dashboard service状态

# kubectl get svc -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.96.100.4   <none>        8000/TCP        21d
kubernetes-dashboard        NodePort    10.96.19.1    <none>        443:30000/TCP   21d

3.1.2 编写访问kubernetes dashboard 路由规则

# ls
6864844_kubemsb.com_nginx.zip  kubemsb.key  kubemsb.pem

# kubectl create secret tls kubemsb-tls --cert=kubemsb.pem --key=kubemsb.key
secret/kubemsb-tls created

# kubectl get secret
NAME                                 TYPE                                  DATA   AGE
default-token-x4qbc                  kubernetes.io/service-account-token   3      24d
kubemsb-tls                          kubernetes.io/tls                     2      11s

# vim kubernetes-dashboard-ir.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:name: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:entryPoints:- websecureroutes:- match: Host(`dashboard.kubemsb.com`)kind: Ruleservices:- name: kubernetes-dashboardport: 443tls:secretName: kubemsb-tls

3.1.3 应用上述路由规则文件

# kubectl apply -f kubernetes-dashboard-ir.yaml

3.1.4 配置域名解析及访问

# vim /etc/hosts
192.168.10.12 dashboard.kubemsb.com

# kubectl get secret -n kubernetes-dashboard
NAME                               TYPE                                  DATA   AGE
......
kubernetes-dashboard-token-8tm5n   kubernetes.io/service-account-token   3      21d

# kubectl describe secret kubernetes-dashboard-token-8tm5n -n kubernetes-dashboard
Name:         kubernetes-dashboard-token-8tm5n
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: kubernetes-dashboardkubernetes.io/service-account.uid: 47d19e14-e3ed-4733-9799-02396dfb436aType:  kubernetes.io/service-account-tokenData
====
ca.crt:     1359 bytes
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImVGc2xjT05uekl0MlVOZ0VCSlhHSURfOXd6WGFvVnZFZmNwREwtVk1STlEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi04dG01biIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ3ZDE5ZTE0LWUzZWQtNDczMy05Nzk5LTAyMzk2ZGZiNDM2YSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.rY5YFoJD3ZUg1e31uxrPhKl0a_mslAmVG5r1ZDia0z7WKKtGjs8XwSu5cIDqKa-1FUV6ixxgfDwm5Rd-nK8TCMVRrDY_7r2I5-u2ebh4rEAXCBvQw0gtJu16-e1Z_TQqNuc73E9fDS0AqHr2qGOWAQcz_FjWGAGjZKzYlKPcA3mFI2wsAIRgxh29-S2f4SxB2zgWyYQdbW-fiHDHWK6dQ-a3glIlCk4jnPtzrX1HNK3BSRPoKaZg_9Ot0dABex2ro5QkQ0wyuT3NLio-n8v21MbhKG5ehBEFRNrTcTPM2CLt4XaUJezphKHShdc3VbUlizPge5DleAJcp9JrFzyqBQ

3.2 配置HTTP路由规则

3.2.1 创建应用及服务资源清单文件并应用

# vim whoami.yaml
kind: Deployment
apiVersion: apps/v1
metadata:name: whoaminamespace: defaultlabels:app: traefiklabsname: whoamispec:replicas: 2selector:matchLabels:app: traefiklabstask: whoamitemplate:metadata:labels:app: traefiklabstask: whoamispec:containers:- name: whoamiimage: traefik/whoamiports:- containerPort: 80---
apiVersion: v1
kind: Service
metadata:name: whoaminamespace: defaultspec:ports:- name: httpport: 80selector:app: traefiklabstask: whoami---
kind: Deployment
apiVersion: apps/v1
metadata:name: whoamitcpnamespace: defaultlabels:app: traefiklabsname: whoamitcpspec:replicas: 2selector:matchLabels:app: traefiklabstask: whoamitcptemplate:metadata:labels:app: traefiklabstask: whoamitcpspec:containers:- name: whoamitcpimage: traefik/whoamitcpports:- containerPort: 8080---
apiVersion: v1
kind: Service
metadata:name: whoamitcpnamespace: defaultspec:ports:- protocol: TCPport: 8080selector:app: traefiklabstask: whoamitcp---
kind: Deployment
apiVersion: apps/v1
metadata:name: whoamiudpnamespace: defaultlabels:app: traefiklabsname: whoamiudpspec:replicas: 2selector:matchLabels:app: traefiklabstask: whoamiudptemplate:metadata:labels:app: traefiklabstask: whoamiudpspec:containers:- name: whoamiudpimage: traefik/whoamiudp:latestports:- containerPort: 8080---
apiVersion: v1
kind: Service
metadata:name: whoamiudpnamespace: defaultspec:ports:- port: 8080protocol: UDPselector:app: traefiklabstask: whoamiudp

# kubectl apply -f whoami.yaml

# kubectl get all
NAME                            READY   STATUS    RESTARTS   AGE
pod/whoami-7d666f84d8-ffbdv     1/1     Running   0          35m
pod/whoami-7d666f84d8-n75wb     1/1     Running   0          35m
pod/whoamitcp-744cc4b47-g98fv   1/1     Running   0          35m
pod/whoamitcp-744cc4b47-s2m7h   1/1     Running   0          35m
pod/whoamiudp-58f6cf7b8-54552   1/1     Running   0          35m
pod/whoamiudp-58f6cf7b8-dzxpg   1/1     Running   0          35mNAME                    TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
service/kubernetes      ClusterIP      10.96.0.1       <none>          443/TCP          25d
service/nginx-metallb   LoadBalancer   10.96.109.6     192.168.10.90   8090:30259/TCP   14d
service/whoami          ClusterIP      10.96.251.213   <none>          80/TCP           35m
service/whoamitcp       ClusterIP      10.96.20.1      <none>          8080/TCP         35m
service/whoamiudp       ClusterIP      10.96.85.175    <none>          8080/UDP         35mNAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/whoami      2/2     2            2           35m
deployment.apps/whoamitcp   2/2     2            2           35m
deployment.apps/whoamiudp   2/2     2            2           35mNAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/whoami-7d666f84d8     2         2         2       35m
replicaset.apps/whoamitcp-744cc4b47   2         2         2       35m
replicaset.apps/whoamiudp-58f6cf7b8   2         2         2       35m

3.2.2 创建whoami应用ingress route资源清单文件并应用

# vim whoami-ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:name: myingressroutenamespace: default
spec:entryPoints:- webroutes:- match: Host(`whoami.kubemsb.com`) && PathPrefix(`/`)kind: Ruleservices:- name: whoamiport: 80

# kubectl apply -f whoami-ingressroute.yaml

# kubectl get ingressroute
NAME             AGE
myingressroute   29m

3.3 配置HTTPS路由规则

如果我们需要用 HTTPS 来访问我们这个应用的话，就需要监听 websecure 这个入口点，也就是通过 443 端口来访问，同样用 HTTPS 访问应用必然就需要证书

3.3.1 自签名证书

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=whoamissl.kubemsb.com"

3.3.2 创建secret

# kubectl create secret tls who-tls --cert=tls.crt --key=tls.key

3.3.3 创建https应用路由规则

# vim whoamissl-ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:name: ingressroutetls
spec:entryPoints:- websecureroutes:- match: Host(`whoamissl.kubemsb.com`)kind: Ruleservices:- name: whoamiport: 80tls:secretName: who-tls

# kubectl apply -f whoamissl-ingressroute.yaml

# kubectl get ingressroute
NAME              AGE
ingressroutetls   17s

# cat /etc/hosts
192.168.10.12 whoamissl.kubemsb.com

3.4 配置TCP路由规则

SNI为服务名称标识，是 TLS 协议的扩展。因此，只有 TLS 路由才能使用该规则指定域名。但是，非 TLS 路由必须使用带有 * 的规则（每个域）来声明每个非 TLS 请求都将由路由进行处理。

3.4.1 实验案例配置

# vim whoami-ingressroutetcp.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:name: ingressroutetcpwho
spec:entryPoints:- tcpeproutes:- match: HostSNI(`*`)services:- name: whoamitcpport: 8080

# kubectl get ingressroutetcp
NAME                 AGE
ingressroutetcpwho   17s

3.4.2 生产案例配置 MySQL部署及traefik代理

3.4.2.1 修改traefik-configmap.yaml

# cat traefik-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:name: traefiknamespace: kube-system
data:traefik.yaml: |-serversTransport:insecureSkipVerify: trueapi:insecure: truedashboard: truedebug: truemetrics:prometheus: ""entryPoints:web:address: ":80"websecure:address: ":443"metrics:address: ":8082"tcpep:address: ":8083"udpep:address: ":8084/udp"mysql:address: ":3312"providers:kubernetesCRD: ""kubernetesingress: ""log:filePath: ""level: errorformat: jsonaccessLog:filePath: ""format: jsonbufferingSize: 0filters:retryAttempts: trueminDuration: 20fields:defaultMode: keepnames:ClientUsername: dropheaders:defaultMode: keepnames:User-Agent: redactAuthorization: dropContent-Type: keep

3.4.2.2 修改traefik-deploy.yaml

# vim traefik-deploy.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:namespace: kube-systemname: traefiklabels:app: traefik
spec:selector:matchLabels:app: traefiktemplate:metadata:labels:app: traefikspec:serviceAccountName: traefik-ingress-controllercontainers:- name: traefikimage: traefik:v2.5.7args:- --configfile=/config/traefik.yamlvolumeMounts:- mountPath: /configname: configports:- name: webcontainerPort: 80hostPort: 80- name: websecurecontainerPort: 443hostPort: 443- name: admincontainerPort: 8080- name: tcpepcontainerPort: 8083hostPort: 8083- name: udpepcontainerPort: 8084hostPort: 8084protocol: UDP- name: mysqlcontainerPort: 3312hostPort: 3306volumes:- name: configconfigMap:name: traefik

# vim traefik-service.yaml
apiVersion: v1
kind: Service
metadata:name: traefiknamespace: kube-system
spec:ports:- protocol: TCPname: webport: 80- protocol: TCPname: adminport: 8080- protocol: TCPname: websecureport: 443- protocol: TCPname: tcpepport: 8083- protocol: UDPname: udpepport: 8084- protocol: TCPname: mysqlport: 3312selector:app: traefik

删除以前的配置及应用
# kubectl delete -f traefix-configmap.yaml
# kubectl delete -f traefix-deploy.yaml
# kubectl delete -f traefix-service.yaml

重新部署
# kubectl apply -f traefix-configmap.yaml
# kubectl apply -f traefix-deploy.yaml
# kubectl apply -f traefix-service.yaml

3.4.2.3 部署mysql应用

关于端口的说明：
traefik Pod：3312:3306（traefik Pod:k8s Node），3312是traefik配置的mysql入口点的端口，3306是k8s Node的端口，traefik请求入口

# vim mysql-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:name: mysqllabels:app: mysqlnamespace: default
data:my.cnf: |[mysqld]character-set-server = utf8mb4collation-server = utf8mb4_unicode_ciskip-character-set-client-handshake = 1default-storage-engine = INNODBmax_allowed_packet = 500Mexplicit_defaults_for_timestamp = 1long_query_time = 10

# vim mysql-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: mysqlname: mysqlnamespace: default
spec:selector:matchLabels:app: mysqltemplate:metadata:labels:app: mysqlspec:containers:- name: mysqlimage: mysql:5.7imagePullPolicy: IfNotPresentenv:- name: MYSQL_ROOT_PASSWORDvalue: abc123ports:- containerPort: 3306volumeMounts:- mountPath: /var/lib/mysqlname: pv- mountPath: /etc/mysql/conf.d/my.cnfsubPath: my.cnfname: cmvolumes:- name: pvhostPath:path: /opt/mysqldata- name: cmconfigMap:name: mysql

# vim mysql-service.yaml
apiVersion: v1
kind: Service
metadata:name: mysqlnamespace: default
spec:ports:- port: 3306protocol: TCPtargetPort: 3306selector:app: mysql

# kubectl apply -f mysql-configmap.yaml
configmap/mysql created

# kubectl apply -f mysql-deploy.yaml
deployment.apps/mysql created

# kubectl apply -f mysql-service.yaml
service/mysql created

3.4.2.4 为mysql应用创建ingressroute

# vim mysql-ingressroutetcp.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:name: mysqlnamespace: default
spec:entryPoints:- mysqlroutes:- match: HostSNI(`*`)services:- name: mysqlport: 3306

# kubectl apply -f mysql-ingressroutetcp.yaml
ingressroutetcp.traefik.containo.us/mysql created

# kubectl get ingressroutetcp
NAME                 AGE
mysql                8s

3.4.2.5 验证

在集群之外主机上执行如下操作

# cat /etc/hosts
......
192.168.10.12 mysql.kubemsb.com

# mysql -h mysql.kubemsb.com  -uroot -pabc123 -P3306
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.36 MySQL Community Server (GPL)Copyright (c) 2000, 2022, Oracle and/or its affiliates.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql>

3.4.3 生产案例 Redis部署及traefik代理

# vim  redis.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: redisnamespace: default
spec:selector:matchLabels:app: redistemplate:metadata:labels:app: redisspec:containers:- name: redisimage: redis:6.2.6ports:- containerPort: 6379protocol: TCP---
apiVersion: v1
kind: Service
metadata:name: redisnamespace: default
spec:ports:- port: 6379targetPort: 6379selector:app: redis

# kubectl apply -f redis.yaml
deployment.apps/redis created
service/redis created

# vim traefik-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:name: traefiknamespace: kube-system
data:traefik.yaml: |-serversTransport:insecureSkipVerify: trueapi:insecure: truedashboard: truedebug: truemetrics:prometheus: ""entryPoints:web:address: ":80"websecure:address: ":443"metrics:address: ":8082"tcpep:address: ":8083"udpep:address: ":8084/udp"mysql:address: ":3312"redis:address: ":6379"providers:kubernetesCRD: ""kubernetesingress: ""log:filePath: ""level: errorformat: jsonaccessLog:filePath: ""format: jsonbufferingSize: 0filters:retryAttempts: trueminDuration: 20fields:defaultMode: keepnames:ClientUsername: dropheaders:defaultMode: keepnames:User-Agent: redactAuthorization: dropContent-Type: keep

删除原configmap,再重新创建
# kubectl delete -f traefik-configmap.yaml# kubectl apply -f traefik-configmap.yaml

# vim traefix-deploy.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:namespace: kube-systemname: traefiklabels:app: traefik
spec:selector:matchLabels:app: traefiktemplate:metadata:labels:app: traefikspec:serviceAccountName: traefik-ingress-controllercontainers:- name: traefikimage: traefik:v2.5.7args:- --configfile=/config/traefik.yamlvolumeMounts:- mountPath: /configname: configports:- name: webcontainerPort: 80hostPort: 80- name: websecurecontainerPort: 443hostPort: 443- name: admincontainerPort: 8080- name: tcpepcontainerPort: 8083hostPort: 8083- name: udpepcontainerPort: 8084hostPort: 8084protocol: UDP- name: mysqlcontainerPort: 3312hostPort: 3306- name: rediscontainerPort: 6379hostPort: 6379volumes:- name: configconfigMap:name: traefik

删除原deploy,再重新创建
# kubectl delete -f traefik-deploy.yaml# kubectl apply -f traefik-deploy.yaml

验证是否添加成功
# kubectl get daemonset traefik -n kube-system -o yaml
...
- containerPort: 6379hostPort: 6379name: redisprotocol: TCP
...

# vim  redis-ingressroutetcp.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:name: redisnamespace: default
spec:entryPoints:- redisroutes:- match: HostSNI(`*`)services:- name: redisport: 6379

# kubectl get ingressroutetcp
NAME                 AGE
redis                8s

在集群之外主机添加域名解析

# cat /etc/hosts
192.168.10.15 redis.kubemsb.com

 # wget http://download.redis.io/releases/redis-3.2.8.tar.gz# tar xf redis-3.2.8.tar.gz# make

# ./src/redis-cli -h redis.kubemsb.com -p 6379
redis.kubemsb.com:6379> ping
PONG
redis.kubemsb.com:6379> set hello world
OK
redis.kubemsb.com:6379> get hello
"world"

3.5 配置UDP路由规则

# vim whoami-ingressrouteudp.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteUDP
metadata:name: ingressrouteudpwho
spec:entryPoints:                  - udpeproutes:                      - services:                  - name: whoamiudp                 port: 8080

# kubectl get ingressrouteudp
NAME                 AGE
ingressrouteudpwho   11s

验证可有性

# kubectl get svc
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGEwhoamiudp       ClusterIP      10.96.85.175    <none>          8080/TCP         6h35m

echo "WHO" | socat - udp4-datagram:10.96.85.175:8080

echo "othermessage" | socat - udp4-datagram:10.96.85.175:8080

输出

# echo "WHO" | socat - udp4-datagram:192.168.10.12:8084
Hostname: whoamiudp-58f6cf7b8-54552
IP: 127.0.0.1
IP: ::1
IP: 10.244.159.187
IP: fe80::1cb4:e1ff:fe66:d9b

# echo "othermessage" | socat - udp4-datagram:192.168.10.12:8084
Received: othermessage

四、traefik中间件 MiddleWare

4.1 traefik中间件介绍 MiddleWare

中间件是 Traefik2.0 中一个非常有特色的功能，我们可以根据自己的各种需求去选择不同的中间件来满足服务，Traefik 官方已经内置了许多不同功能的中间件，其中包括修改请求头信息；重定向；身份验证等等，而且中间件还可以通过链式组合的方式来适用各种情况。例如：强制跳转https、去除访问前缀、访问白名单等。

4.2 traefik 中间件应用案例 ipWhiteList

在工作 中，有一些URL并不希望对外暴露，比如prometheus、grafana等，我们就可以通过白名单IP来过到要求，可以使用Traefix中的ipWhiteList中间件来完成。

# vim deploy-service.yaml
kind: Deployment
apiVersion: apps/v1
metadata:name: nginx-web-middlewarenamespace: default
spec:replicas: 1selector:matchLabels:app: middletemplate:metadata:labels:app: middlespec:containers:- name: nginx-web-cimage: nginx:latestports:- containerPort: 80---
apiVersion: v1
kind: Service
metadata:name: service-middlenamespace: defaultspec:ports:- name: httpport: 80selector:app: middle

# kubectl apply -f deploy-service.yaml
deployment.apps/nginx-web-middleware created
service/service-middle created

# vim middleware-ipwhitelist.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:name: gs-ipwhitelist
spec:ipWhiteList:sourceRange:- 127.0.0.1- 10.244.0.0/16- 10.96.0.0/12- 192.168.10.0/24

# kubectl apply -f middleware-ipwhitelist.yaml
middleware.traefik.containo.us/gs-ipwhitelist created

# vim deploy-service-middle.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:name: ingressroutemiddlenamespace: defaultspec:entryPoints:- webroutes:- match: Host(`middleware.kubemsb.com`) && PathPrefix(`/`)kind: Ruleservices:- name: service-middleport: 80namespace: defaultmiddlewares:- name: gs-ipwhitelist

# kubectl apply -f deploy-service-middle.yaml
ingressroute.traefik.containo.us/ingressroutemiddle created

在集群之外的主机上访问

# vim  /etc/hosts
192.168.10.15 middleware.kubemsb.com

# curl http://middleware.kubemsb.com

把集群外主机所在的网段从白名单中删除，发现无法访问。

# cat middleware-ipwhitelist.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:name: gs-ipwhitelist
spec:ipWhiteList:sourceRange:- 127.0.0.1- 10.244.0.0/16- 10.96.0.0/12

# kubectl apply -f middleware-ipwhitelist.yaml
middleware.traefik.containo.us/gs-ipwhitelist configured

在集群外主机访问

# curl http://middleware.kubemsb.com
Forbidden

五、traefik高级应用

在实际的生产环境，除了上线业务之外，还有更复杂的使用要求。
在开始traefik的高级用法之前，还需要了解一个TraefikService，通过把TraefikService注册到CRD来实现更复杂的请求设置。

TraefikService 目前能用于以下功能servers  load balancing.（负载均衡）services  Weighted Round Robin load balancing.（权重轮询）services  mirroring.（镜像）

5.1 负载均衡

5.1.1 创建deployment控制器类型应用

# vim loadbalancer-deploy.yaml
kind: Deployment
apiVersion: apps/v1
metadata:name: nginx-web1namespace: default
spec:replicas: 1selector:matchLabels:app: v1template:metadata:labels:app: v1spec:containers:- name: nginx-web-cimage: nginx:latestports:- containerPort: 80
---
kind: Deployment
apiVersion: apps/v1
metadata:name: nginx-web2namespace: default
spec:replicas: 1selector:matchLabels:app: v2template:metadata:labels:app: v2spec:containers:- name: nginx-web-cimage: nginx:latestports:- containerPort: 80

# kubectl apply -f loadbalancer-deploy.yaml

修改容器中网站页面信息

# kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
nginx-web1-856c759646-58snd   1/1     Running   0          2m34s
nginx-web2-5d547f7c5f-vd8n7   1/1     Running   0          2m34s

# kubectl exec -it nginx-web1-856c759646-58snd -- sh
# echo "svc1" > /usr/share/nginx/html/index.html
# exit# kubectl exec -it nginx-web2-5d547f7c5f-vd8n7 -- sh
# echo "svc2" > /usr/share/nginx/html/index.html
# exit

# vim loadbalancer-deploy.yaml
kind: Deployment
apiVersion: apps/v1
metadata:name: nginx-web1namespace: default
spec:replicas: 1selector:matchLabels:app: v1template:metadata:labels:app: v1spec:containers:- name: nginx-web-cimage: nginx:latestlifecycle:postStart:exec:command:  ["/bin/sh", "-c", "echo svc1 > /usr/share/nginx/html/index.html"]ports:- containerPort: 80
---
kind: Deployment
apiVersion: apps/v1
metadata:name: nginx-web2namespace: default
spec:replicas: 1selector:matchLabels:app: v2template:metadata:labels:app: v2spec:containers:- name: nginx-web-cimage: nginx:latestlifecycle:postStart:exec:command:  ["/bin/sh", "-c", "echo svc2 > /usr/share/nginx/html/index.html"]ports:- containerPort: 80

5.1.2 创建service

# vim loadbalancer-deploy-service.yaml
apiVersion: v1
kind: Service
metadata:name: svc1namespace: defaultspec:ports:- name: httpport: 80selector:app: v1
---
apiVersion: v1
kind: Service
metadata:name: svc2namespace: defaultspec:ports:- name: httpport: 80selector:app: v2

# kubectl apply -f loadbalancer-deploy-service.yaml

# kubectl get svc
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
svc1            ClusterIP      10.96.136.245   <none>          80/TCP           3s
svc2            ClusterIP      10.96.2.4       <none>          80/TCP           3s

# curl http://10.96.136.245
svc1# curl http://10.96.2.4
svc2

5.1.3 创建ingressroute

# vim loadbalancer-deploy-service-ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:name: ingressrouteweblbnamespace: defaultspec:entryPoints:- webroutes:- match: Host(`lb.kubemsb.com`) && PathPrefix(`/`)kind: Ruleservices:- name: svc1port: 80namespace: default- name: svc2port: 80namespace: default

# kubectl apply -f loadbalancer-deploy-service-ingressroute.yaml

# kubectl get ingressroute
NAME                AGE
ingressrouteweblb   9s

5.1.4 访问验证

在集群之外的主机上访问

# cat /etc/hosts
192.168.10.15 lb.kubemsb.com

# curl http://lb.kubemsb.com
svc1# curl http://lb.kubemsb.com
svc2

5.2 灰度发布

基于上述负载均衡案例基础之上实施。

灰度发布也称为金丝雀发布，让一部分即将上线的服务发布到线上，观察是否达到上线要求，主要通过权重轮询的方式实现。

5.2.1 创建TraefikService

# vim traefikservice.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:name: wrrnamespace: defaultspec:weighted:services:- name: svc1    port: 80weight: 3          # 定义权重kind: Service      # 可选，默认就是 Service - name: svc2port: 80     weight: 1

# kubectl apply -f traefikservice.yaml
traefikservice.traefik.containo.us/wrr created

5.2.2 创建ingressroute

需要注意的是现在我们配置的 Service 不再是直接的 Kubernetes 对象了，而是上面我们定义的 TraefikService 对象

# vim traefik-wrr.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:name: ingressroutewrrnamespace: defaultspec:entryPoints:- webroutes:- match: Host(`lb.kubemsb.com`) && PathPrefix(`/`)kind: Ruleservices:- name: wrrnamespace: defaultkind: TraefikService

# kubectl apply -f traefik-wrr.yaml
ingressroute.traefik.containo.us/ingressroutewrr created

第一次
# curl http://lb.kubemsb.com
svc1第二次
# curl http://lb.kubemsb.com
svc1第三次
# curl http://lb.kubemsb.com
svc2第四次
# curl http://lb.kubemsb.com
svc1

5.3 流量复制

在负责均衡案例基础之上实施

所谓的流量复制，也称为镜像服务是指将请求的流量按规则复制一份发送给其它服务，并且会忽略这部分请求的响应，这个功能在做一些压测或者问题复现的时候很有用。

5.3.1 指定流量来自己于kubernetes service对象

# vim mirror_from_service.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:name: mirror-from-servicenamespace: defaultspec:mirroring:name: svc1       # 发送 100% 的请求到 K8S 的 Service "v1"port: 80mirrors:- name: svc2   # 然后复制 20% 的请求到 v2port: 80percent: 20

# kubectl apply -f mirror_from_service.yaml
traefikservice.traefik.containo.us/mirror-from-service created

# vim mirror-from-service-ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:name: ingressroute-mirrornamespace: defaultspec:entryPoints:- webroutes:- match: Host(`lb.kubemsb.com`) && PathPrefix(`/`) kind: Ruleservices:- name: mirror-from-service         namespace: defaultkind: TraefikService

# kubectl apply -f mirror-from-service-ingressroute.yaml
ingressroute.traefik.containo.us/ingressroute-mirror created

# while true
do
curl http://lb.kubemsb.com
sleep 1
done

5.3.2 通过traefikservice导入流量

# vim mirror-from-traefikservice.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:name: mirror-from-traefikservicenamespace: defaultspec:mirroring:name: mirror-from-service      #流量入口从TraefikService 来kind: TraefikServicemirrors:- name: svc2port: 80percent: 20

# kubectl apply -f mirror-from-traefikservice.yaml
traefikservice.traefik.containo.us/mirror-from-traefikservice created

# vim mirror-from-traefikservice-ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:name: ingressroute-mirror-traefikservicenamespace: defaultspec:entryPoints:- webroutes:- match: Host(`lb.kubemsb.com`) && PathPrefix(`/`)  kind: Ruleservices:- name: mirror-from-traefikservice         namespace: defaultkind: TraefikService

# kubectl apply -f mirror-from-tradfikservice-ingressroute.yaml
ingressroute.traefik.containo.us/ingressroute-mirror-traefikservice created

# while true; do curl http://lb.kubemsb.com; sleep 1; done

5.3.3 流量复制小结

通过上述的演示我们会发现所有的流量100%发送了svc1,有20%的流量被复制到svc2，且用户收到响应均来自svc1，svc2并没有响应，可通过查看svc1及svc2应用日志获取访问日志。

六、Kubernetes Gateway API

6.1 Gateway API介绍

6.1.1 Gateway API架构

Gateway API（之前叫 Service API）是由 SIG-NETWORK 社区管理的开源项目，项目地址：https://gateway-api.sigs.k8s.io/。主要原因是 Ingress 资源对象不能很好的满足网络需求，很多场景下 Ingress 控制器都需要通过定义 annotations 或者 crd 来进行功能扩展，这对于使用标准和支持是非常不利的，新推出的 Gateway API 旨在通过可扩展的面向角色的接口来增强服务网络。

Gateway API 是 Kubernetes 中的一个 API 资源集合，包括 GatewayClass、Gateway、HTTPRoute、TCPRoute、Service 等，这些资源共同为各种网络用例构建模型。

Gateway API 的改进比当前的 Ingress 资源对象有很多更好的设计：

• 面向角色 - Gateway 由各种 API 资源组成，这些资源根据使用和配置 Kubernetes 服务网络的角色进行建模。
• 通用性 - 和 Ingress 一样是一个具有众多实现的通用规范，Gateway API 是一个被设计成由许多实现支持的规范标准。
• 更具表现力 - Gateway API 资源支持基于 Header 头的匹配、流量权重等核心功能，这些功能在 Ingress 中只能通过自定义注解才能实现。
• 可扩展性 - Gateway API 允许自定义资源链接到 API 的各个层，这就允许在 API 结构的适当位置进行更精细的定制。

还有一些其他值得关注的功能：

• GatewayClasses - GatewayClasses 将负载均衡实现的类型形式化，这些类使用户可以很容易了解到通过 Kubernetes 资源可以获得什么样的能力。
• 共享网关和跨命名空间支持 - 它们允许共享负载均衡器和 VIP，允许独立的路由资源绑定到同一个网关，这使得团队可以安全地共享（包括跨命名空间）基础设施，而不需要直接协调。
• 规范化路由和后端 - Gateway API 支持类型化的路由资源和不同类型的后端，这使得 API 可以灵活地支持各种协议（如 HTTP 和 gRPC）和各种后端服务（如 Kubernetes Service、存储桶或函数）。

6.1.2 面向角色设计

无论是道路、电力、数据中心还是 Kubernetes 集群，基础设施都是为了共享而建的，然而共享基础设施提供了一个共同的挑战，那就是如何为基础设施用户提供灵活性的同时还能被所有者控制。

Gateway API 通过对 Kubernetes 服务网络进行面向角色的设计来实现这一目标，平衡了灵活性和集中控制。它允许共享的网络基础设施（硬件负载均衡器、云网络、集群托管的代理等）被许多不同的团队使用，所有这些都受到集群运维设置的各种策略和约束。下面的例子显示了是如何在实践中运行的。

一个集群运维人员创建了一个基于 GatewayClass 的 Gateway 资源，这个 Gateway 部署或配置了它所代表的基础网络资源，集群运维和特定的团队必须沟通什么可以附加到这个 Gateway 上来暴露他们的应用。集中的策略，如 TLS，可以由集群运维在 Gateway 上强制执行，同时，Store 和 Site 应用在他们自己的命名空间中运行，但将他们的路由附加到相同的共享网关上，允许他们独立控制他们的路由逻辑。

这种关注点分离的设计可以使不同的团队能够管理他们自己的流量，同时将集中的策略和控制留给集群运维。

6.1.3 Gateway API概念

在整个 Gateway API 中涉及到3个角色：基础设施提供商、集群管理员、应用开发人员，在某些场景下可能还会涉及到应用管理员等角色。Gateway API 中定义了3种主要的资源模型：GatewayClass、Gateway、Route。

6.1.3.1 GatewayClass

GatewayClass 定义了一组共享相同配置和动作的网关。每个GatewayClass 由一个控制器处理，是一个集群范围的资源，必须至少有一个 GatewayClass 被定义。

这与 Ingress 的 IngressClass 类似，在 Ingress v1beta1 版本中，与 GatewayClass 类似的是 ingress-class 注解，而在Ingress V1 版本中，最接近的就是 IngressClass 资源对象。

6.1.3.2 Gateway

Gateway 网关描述了如何将流量转化为集群内的服务，也就是说，它定义了一个请求，要求将流量从不了解 Kubernetes 的地方转换到集群内的服务。例如，由云端负载均衡器、集群内代理或外部硬件负载均衡器发送到 Kubernetes 服务的流量。

它定义了对特定负载均衡器配置的请求，该配置实现了 GatewayClass 的配置和行为规范，该资源可以由管理员直接创建，也可以由处理 GatewayClass 的控制器创建。

Gateway 可以附加到一个或多个路由引用上，这些路由引用的作用是将流量的一个子集导向特定的服务。

6.1.3.3 Route 资源

路由资源定义了特定的规则，用于将请求从网关映射到 Kubernetes 服务。

从 v1alpha2 版本开始，API 中包含四种 Route 路由资源类型，对于其他未定义的协议，鼓励采用特定实现的自定义路由类型，当然未来也可能会添加新的路由类型。

6.1.3.3.1 HTTPRoute

HTTPRoute 适用于 HTTP 或 HTTPS 连接，适用于我们想要检查 HTTP 请求并使用 HTTP 请求进行路由或修改的场景，比如使用 HTTP Headers 头进行路由，或在请求过程中对它们进行修改。

6.1.3.3.2 TLSRoute

TLSRoute 用于 TLS 连接，通过 SNI 进行区分，它适用于希望使用 SNI 作为主要路由方法的地方，并且对 HTTP 等更高级别协议的属性不感兴趣，连接的字节流不经任何检查就被代理到后端。

6.1.3.3.3 TCPRoute 和 UDPRoute

TCPRoute（和UDPRoute）旨在用于将一个或多个端口映射到单个后端。在这种情况下，没有可以用来选择同一端口的不同后端的判别器，所以每个 TCPRoute 在监听器上需要一个不同的端口。你可以使用 TLS，在这种情况下，未加密的字节流会被传递到后端，当然也可以不使用 TLS，这样加密的字节流将传递到后端。

6.1.4 Gateway API概念之间关系

GatewayClass、Gateway、xRoute 和 Service 的组合定义了一个可实施的负载均衡器。下图说明了不同资源之间的关系:

使用反向代理实现的网关的典型客户端/网关 API 请求流程如下所示：

1. 客户端向 http://foo.example.com 发出请求
2. DNS 将域名解析为 Gateway 网关地址
3. 反向代理在监听器上接收请求，并使用 Host Header 来匹配HTTPRoute
4. (可选)反向代理可以根据 HTTPRoute 的匹配规则进行路由
5. (可选)反向代理可以根据 HTTPRoute 的过滤规则修改请求，即添加或删除 headers
6. 最后，反向代理根据 HTTPRoute 的 forwardTo 规则，将请求转发给集群中的一个或多个对象，即服务。

6.2 kubernetes gateway CRD安装

要在 Traefik 中使用 Gateway API，首先我们需要先手动安装 Gateway API 的 CRDs，使用如下命令即可安装，这将安装包括 GatewayClass、Gateway、HTTPRoute、TCPRoute 等 CRDs：

#  kubectl apply -k "github.com/kubernetes-sigs/service-apis/config/crd?ref=v0.3.0"输出：
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/referencepolicies.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/tcproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/tlsroutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io created

6.3 为traefik授权（RBAC）

# vim traefik-gatewayapi-rbac.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: gateway-role
rules:- apiGroups:- ""resources:- services- endpoints- secretsverbs:- get- list- watch- apiGroups:- networking.x-k8s.ioresources:- gatewayclasses- gateways- httproutes- tcproutes- tlsroutesverbs:- get- list- watch- apiGroups:- networking.x-k8s.ioresources:- gatewayclasses/status- gateways/status- httproutes/status- tcproutes/status- tlsroutes/statusverbs:- update---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: gateway-controllerroleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: gateway-role
subjects:- kind: ServiceAccountname: traefik-ingress-controllernamespace: kube-system 

6.4 Traefik开启Gateway api支持

# vim traefik-configmap.yaml
......providers:kubernetesCRD: ""kubernetesingress: ""kubernetesGateway: "" 添加此行experimental: 添加此行kubernetesGateway: true 添加此行
......

删除
# kubectl delete  -f traefik-configmap.yaml
# kubectl delete -f traefik-deploy.yaml

重新运行
# kubectl apply  -f traefik-configmap.yaml
# kubectl apply  -f traefik-deploy.yaml

6.5 创建Gateway API的GatewayClass

# vim gatewayclass.yaml
apiVersion: networking.x-k8s.io/v1alpha1
kind: GatewayClass
metadata:name: traefik
spec:controller: traefik.io/gateway-controller

# kubectl apply -f gatewayclass.yaml

# kubectl get gatewayclass
NAME      CONTROLLER                      AGE
traefik   traefik.io/gateway-controller   2m59s

6.6 Gateway API应用案例

6.6.1 通过Gateway API方式暴露traefik dashboard

6.6.1.1 创建gateway

# vim gateway.yaml
apiVersion: networking.x-k8s.io/v1alpha1
kind: Gateway
metadata: name: http-gatewaynamespace: kube-system 
spec: gatewayClassName: traefiklisteners: - protocol: HTTPport: 80 routes: kind: HTTPRoutenamespaces:from: Allselector:matchLabels:app: traefik

# kubectl apply -f gateway.yaml
gateway.networking.x-k8s.io/http-gateway created

# kubectl get gateway
NAME           CLASS     AGE
http-gateway   traefik   6s

6.5.1.2 创建HTTPRoute

# vim httproute.yaml
apiVersion: networking.x-k8s.io/v1alpha1
kind: HTTPRoute
metadata:name: traefix-dashboard-gateway-api-route namespace: kube-systemlabels:app: traefik
spec:hostnames:- "traefikdashboard.kubemsb.com"rules:- matches:- path:type: Prefixvalue: /forwardTo:- serviceName: traefik port: 8080weight: 1

# kubectl apply -f httproute.yaml
httproute.networking.x-k8s.io/traefix-dashboard-gateway-api-route created

# kubectl get httproute
NAME                                  HOSTNAMES                          AGE
traefix-dashboard-gateway-api-route   ["traefikdashboard.kubemsb.com"]   6s

6.5.1.3 在集群之外主机访问

# vim /etc/hosts
192.168.10.12 traefikdashboard.kubemsb.com

6.6.2 通过Gateway API方式暴露WEB应用

# cat gatewayapi-web.yaml
kind: Deployment
apiVersion: apps/v1
metadata:name: nginx-web-gatewayapinamespace: default
spec:replicas: 1selector:matchLabels:app: gatewaywebtemplate:metadata:labels:app: gatewaywebspec:containers:- name: nginx-webimage: nginx:latestlifecycle:postStart:exec:command:  ["/bin/sh", "-c", "echo gatewayweb > /usr/share/nginx/html/index.html"]ports:- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:name: nginx-web-gatewayapi-svcnamespace: defaultspec:ports:- name: httpport: 80selector:app: gatewayweb

# kubectl apply -f gatewayapi-web.yaml
deployment.apps/nginx-web-gatewayapi created
service/nginx-web-gatewayapi-svc created

# vim gatewayapi-web-gateway.yaml
apiVersion: networking.x-k8s.io/v1alpha1
kind: Gateway
metadata:name: nginx-web-gateway
spec:gatewayClassName: traefiklisteners:- protocol: HTTPport: 80routes:kind: HTTPRoutenamespaces:from: Allselector:matchLabels:app: gatewayweb

# kubectl apply -f gatewayapi-web-gateway.yaml
gateway.networking.x-k8s.io/nginx-web-gateway created

# kubectl get gateway
NAME                CLASS     AGE
nginx-web-gateway   traefik   7s

# kubectl describe gateway nginx-web-gateway

# cat gatewayapi-web-httproute.yaml
apiVersion: networking.x-k8s.io/v1alpha1
kind: HTTPRoute
metadata:name: nginx-web-gateway-api-routelabels:app: gatewayweb
spec:hostnames:- "nginx.kubemsb.com"rules:- matches:- path:type: Prefixvalue: /forwardTo:- serviceName: nginx-web-gatewayapi-svcport: 80weight: 1

# kubectl apply -f gatewayapi-web-httproute.yaml
httproute.networking.x-k8s.io/nginx-web-gateway-api-route created

kubectl get httproute
NAME                          HOSTNAMES               AGE
nginx-web-gateway-api-route   ["nginx.kubemsb.com"]   6s

在集群之外主机访问

# vim /etc/hosts
192.168.10.12 nginx.kubemsb.com

# curl http://nginx.kubemsb.com
gatewayweb

6.6.3 金丝雀发布

Gateway APIs 规范可以支持的另一个功能是金丝雀发布，假设你想在一个端点上运行两个不同的服务（或同一服务的两个版本），并将一部分请求路由到每个端点，则可以通过修改你的 HTTPRoute 来实现。

# vim gateway-cn-deploy.yaml
kind: Deployment
apiVersion: apps/v1
metadata:name: nginx-web1namespace: kube-system
spec:replicas: 1selector:matchLabels:app: v1template:metadata:labels:app: v1spec:containers:- name: nginx-web-cimage: nginx:latestlifecycle:postStart:exec:command:  ["/bin/sh", "-c", "echo svc1 > /usr/share/nginx/html/index.html"]ports:- containerPort: 80
---
kind: Deployment
apiVersion: apps/v1
metadata:name: nginx-web2namespace: kube-system
spec:replicas: 1selector:matchLabels:app: v2template:metadata:labels:app: v2spec:containers:- name: nginx-web-cimage: nginx:latestlifecycle:postStart:exec:command:  ["/bin/sh", "-c", "echo svc2 > /usr/share/nginx/html/index.html"]ports:- containerPort: 80

# kubectl apply -f gateway-cn-deploy.yaml

# vim gateway-cn-deploy-svc.yaml
apiVersion: v1
kind: Service
metadata:name: svc1namespace: kube-systemspec:ports:- name: httpport: 80selector:app: v1
---
apiVersion: v1
kind: Service
metadata:name: svc2namespace: kube-systemspec:ports:- name: httpport: 80selector:app: v2

# kubectl apply -fgateway-cn-deploy-svc.yaml

# vim gateway-cn-httproute.yaml
apiVersion: networking.x-k8s.io/v1alpha1
kind: HTTPRoute
metadata:labels:app: traefikname: nginxweb-appnamespace: kube-system
spec:hostnames:- canary.kubemsb.comrules:- forwardTo:- port: 80serviceName: svc1weight: 3  # 3/4 的请求到svc1- port: 80serviceName: svc2weight: 1  # 1/4 的请求到svc2

# kubectl apply -f gateway-cn-httproute.yaml
httproute.networking.x-k8s.io/nginxweb-app created

# kubectl get httproute -n kube-system
NAME                          HOSTNAMES                AGE
nginxweb-app                  ["canary.kubemsb.com"]   7s

在 集群之外主机访问

# vim /etc/hosts
192.168.10.12 canary.kubemsb.com

[root@dockerhost ~]# curl http://canary.kubemsb.com
svc1
[root@dockerhost ~]# curl http://canary.kubemsb.com
svc1
[root@dockerhost ~]# curl http://canary.kubemsb.com
svc1
[root@dockerhost ~]# curl http://canary.kubemsb.com
svc2

以上使用 Traefik 来测试了 Kubernetes Gateway APIs 的使用。目前，Traefik 对 Gateway APIs 的实现是基于 v1alpha1 版本的规范，目前最新的规范是 v1alpha2，所以和最新的规范可能有一些出入的地方。