k8s学习(三十八) 使用OpenTelemetry+jaeger实现链路追踪

---

前言

OpenTelemetry 可以用于从应用程序收集数据。它是一组工具、API 和 SDK 集合，我们可以使用它们来检测、生成、收集和导出遥测数据（指标、日志和追踪），以帮助分析应用的性能和行为。
这里基于K8S 1.20.2部署，其他版本可现在对应的operator和镜像即可。

---

提示：以下是本篇文章正文内容，下面案例可供参考

一、安装jaeger

下载镜像,并推送至私服

docker pull  docker.m.daocloud.io/jaegertracing/all-in-one:1.60.0
docker tag docker.m.daocloud.io/jaegertracing/all-in-one:1.60.0  172.16.10.160:88/library/jaegertracing/all-in-one:1.60.0
docker push 172.16.10.160:88/library/jaegertracing/all-in-one:1.60.0

修改jaeger.yaml，修改其中的镜像为172.16.10.160:88私服中的地址
jaeger.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:name: jaeger
spec:replicas: 1selector:matchLabels:app: jaegertemplate:metadata:labels:app: jaegerspec:containers:- name: jaegerimage: 172.16.10.160:88/library/jaegertracing/all-in-one:1.60.0env:- name: COLLECTOR_OTLP_ENABLEDvalue: "true"ports:- containerPort: 16686- containerPort: 14268---
apiVersion: v1
kind: Service
metadata:name: jaeger
spec:selector:app: jaegertype: ClusterIPports:- name: uiport: 16686targetPort: 16686nodePort: 32186- name: collectorport: 14268targetPort: 14268- name: httpprotocol: TCPport: 4318targetPort: 4318- name: grpcprotocol: TCPport: 4317targetPort: 4317type: NodePort

执行：

apply -f jaeger.yaml

二、安装cert-manager

下载镜像并推送至私服

docker pull m.daocloud.io/quay.io/jetstack/cert-manager-cainjector:v1.15.2
docker tag m.daocloud.io/quay.io/jetstack/cert-manager-cainjector:v1.15.2 172.16.10.160:88/library/jetstack/cert-manager-cainjector:v1.15.2
docker push 172.16.10.160:88/library/jetstack/cert-manager-cainjector:v1.15.2docker pull m.daocloud.io/quay.io/jetstack/cert-manager-controller:v1.15.2
docker tag m.daocloud.io/quay.io/jetstack/cert-manager-controller:v1.15.2 172.16.10.160:88/library/jetstack/cert-manager-controller:v1.15.2
docker push 172.16.10.160:88/library/jetstack/cert-manager-controller:v1.15.2docker pull m.daocloud.io/quay.io/jetstack/cert-manager-acmesolver:v1.15.2
docker tag m.daocloud.io/quay.io/jetstack/cert-manager-acmesolver:v1.15.2 172.16.10.160:88/library/jetstack/cert-manager-acmesolver:v1.15.2
docker push 172.16.10.160:88/library/jetstack/cert-manager-acmesolver:v1.15.2docker pull m.daocloud.io/quay.io/jetstack/cert-manager-webhook:v1.15.2
docker tag m.daocloud.io/quay.io/jetstack/cert-manager-webhook:v1.15.2 172.16.10.160:88/library/jetstack/cert-manager-webhook:v1.15.2
docker push 172.16.10.160:88/library/jetstack/cert-manager-webhook:v1.15.2

修改cert-manager中的镜像地址为172.16.10.160:88私服中对应的镜像地址（cert-manager下载地址：https://github.com/cert-manager/cert-manager/releases/download/v1.15.2/cert-manager.yaml）

cert-manager.yaml内容略，可直接从上面地址下载。
执行：

kubectl apply -f cert-manager.yaml

三、安装OpenTelemetry Operator

下载镜像并推送至私服

docker pull m.daocloud.io/ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.85.0
docker tag m.daocloud.io/ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.85.0 172.16.10.160:88/library/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.85.0
docker push 172.16.10.160:88/library/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.85.0docker pull m.daocloud.io/gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1
docker tag m.daocloud.io/gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1 172.16.10.160:88/library/kubebuilder/kube-rbac-proxy:v0.13.1
docker push 172.16.10.160:88/library/kubebuilder/kube-rbac-proxy:v0.13.1

修改opentelemetry-operator.yaml中的镜像地址为172.16.10.160:88私服中对应的镜像地址（cert-manager下载地址：https://github.com/open-telemetry/opentelemetry-operator/releases/download/v0.85.0/opentelemetry-operator.yaml）

opentelemetry-operator.yaml的内容略

执行：

kubectl apply -f opentelemetry-operator.yaml

四、配置 OpenTelemetry Collector

通过创建 CR OpenTelemetryCollector 来配置 Otel 的采集器
open-telemetry-collector.yaml：

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:name: otel
spec:config: |receivers:otlp:protocols:grpc:http:processors:memory_limiter:check_interval: 1slimit_percentage: 75spike_limit_percentage: 15batch:send_batch_size: 10000timeout: 10sexporters:otlp/jaeger:endpoint: "jaeger.default:4317"tls:insecure: trueservice:pipelines:traces:receivers: [otlp]processors: []exporters: [otlp/jaeger]

执行：

kubectl apply -f open-telemetry-collector.yaml

查看自定义的OpenTelemetryCollector

查看otel的pod

查看otel的service

五、配置 Instrumentation

下载java的对应镜像，（其他语言的也可以下载的对应的镜像例如nodejs、python等）

docker pull m.daocloud.io/ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.30.0
docker tag m.daocloud.io/ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.30.0 172.16.10.160:88/library/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.30.0
docker push 172.16.10.160:88/library/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.30.0

Instrumentation 是 Otel Operator 的另一个 CRD，用于自动安装 Otel 探针和配置

open-telemetry-instrumentation.yaml：

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:name: instrumentation-sample
spec:propagators:- tracecontext- baggage- b3sampler:type: parentbased_traceidratioargument: "1"env:- name: OTEL_EXPORTER_OTLP_ENDPOINTvalue: otel-collector.default:4318java:    image: 172.16.10.160:88/library/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.30.0env:- name: OTEL_EXPORTER_OTLP_ENDPOINTvalue: http://otel-collector.default:4317

执行：

kubectl apply -f open-telemetry-instrumentation.yaml

六、编写java示例程序并测试调用链跟踪

为 Pod 添加注解 instrumentation.opentelemetry.io/inject-java: "true" 通知 Otel Operator 该应用的类型以便注入正确的探针。
简单的示例为servicea -> serviceb

servicea和serviceb就是两个普通的java程序，servicea通过K8S服务名调用serviceb,没有其他额外配置

servera调用

package cn.ac.iscas.servicea.controller;import cn.ac.iscas.servicea.util.CustomHttpClient;
import jakarta.annotation.PostConstruct;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;import java.io.IOException;
import java.util.UUID;/*** @author zhuquanwen* @version 1.0* @date 2024/8/9 9:50*/
@RestController
public class TestController {@Value("${serverb.endpoint}")private String servicebEndpoint;private CustomHttpClient httpClient;@PostConstructpublic void init() {httpClient = new CustomHttpClient(new CustomHttpClient.HttpClientProps());}@GetMapping("t1")public String t1() throws IOException, InterruptedException {return httpClient.doGet("http://" + servicebEndpoint + "/t1");}
}

servicea配置：

spring.application.name=service-a
server.port=4001
serverb.endpoint=localhost:4002

serviceb逻辑：

package cn.ac.iscas.serviceb.controller;import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;import java.util.UUID;/*** @author zhuquanwen* @version 1.0* @date 2024/8/9 9:50*/
@RestController
public class TestController {@GetMapping("t1")public String t1() {return "serverb-res-" + UUID.randomUUID();}
}

使用dockerfile将程序打包，并上传至私服

编写两个程序的k8s编排配置，并启动
deploy-servicea.yaml：

apiVersion: apps/v1
kind: Deployment
metadata:name: servicea
spec:replicas: 1selector:matchLabels:app: serviceatemplate:metadata:labels:app: serviceaannotations:instrumentation.opentelemetry.io/inject-java: "true"spec:containers:- name: serviceaimage: 172.16.10.160:88/library/test-micro/servicea:0.0.1ports:- containerPort: 4001---
apiVersion: v1
kind: Service
metadata:name: servicea
spec:selector:app: serviceaports:- name: my-portprotocol: TCPport: 4001targetPort: 4001nodePort: 32401type: NodePort

deploy-serviceb.yaml：

apiVersion: apps/v1
kind: Deployment
metadata:name: serviceb
spec:replicas: 1selector:matchLabels:app: servicebtemplate:metadata:labels:app: servicebannotations:instrumentation.opentelemetry.io/inject-java: "true"spec:containers:- name: servicebimage: 172.16.10.160:88/library/test-micro/serviceb:0.0.1ports:- containerPort: 4002---
apiVersion: v1
kind: Service
metadata:name: serviceb
spec:selector:app: servicebports:- name: my-portprotocol: TCPport: 4002targetPort: 4002nodePort: 32402type: NodePort

部署：

kubectl apply -f deploy-servicea.yaml
kubectl apply -f deploy-serviceb.yaml

通过describe pod 可以看到注入了一个initContainer和一些环境变量

通过浏览器访问servicea

访问jaeger,查看调用链