Jeager๋กœ ์‚ดํŽด๋ณด๋Š” Istio์˜ ๋ถ„์‚ฐ ์ถ”์  ์‹œ์Šคํ…œ๊ณผ ๊ทธ ์›๋ฆฌ ๐ŸฆŒ ์š”์ฒญ์˜ ์‘๋‹ต์ด ๋‚ด๊ฒŒ ๋Œ์•„์˜ค๊ธฐ๊นŒ์ง€์˜ ์—ฌ์ •์„ ๋ณด์—ฌ์ฃผ๋Š” ๊ธธ์žก์ด.

6 minute read

์–˜๊ฑฐ๋งˆ์ด์Šคํ„ฐ, ํ—ˆ๋ธŒํ–ฅ์ด ๋‚˜๋Š” ๋ฆฌํ๋ฅด. ์˜ˆ๊ฑฐ๋ฐค์œผ๋กœ ํ•ด์„œ ๋จน์œผ๋ฉด ๋ง›์žˆ๋‹ค!

โ€œDistributed Tracing(๋ถ„์‚ฐ ์ถ”์ )โ€œ์ด๋ž€ ๋ถ„์‚ฐ ์‹œ์Šคํ…œ์—์„œ ์š”์ฒญ์ด ์‹œ์Šคํ…œ์˜ ๋‹ค์–‘ํ•œ ์„œ๋น„์Šค๋ฅผ ๊ฑฐ์น˜๋ฉฐ ์ง„ํ–‰๋  ๋•Œ, ์š”์ฒญ์˜ ๊ฒฝ๋กœ๋ฅผ ์ถ”์ ํ•˜๊ณ  ์‹œ๊ฐํ™” ํ•˜๋Š” ๊ธฐ์ˆ ์„ ๋งํ•œ๋‹ค.

Tracing์„ ํ†ตํ•ด MSA ๊ตฌ์กฐ๋ฅผ ์ด๋ฃจ๋Š” ๋ณต์žกํ•œ ๋ถ„์‚ฐ ์‹œ์Šคํ…œ์ด ์„œ๋กœ ์–ด๋–ป๊ฒŒ ์—ฐ๊ฒฐ ๋˜์–ด ์žˆ๋Š”์ง€ ํŒŒ์•…ํ•˜๊ณ , ์š”์ฒญ์„ ๋””๋ฒ„๊ทธ ํ•˜๋Š”๋ฐ์— ๋„์›€์ด ๋œ๋‹ค.

์ด๋ฒˆ ํฌ์ŠคํŠธ์—์„œ๋Š” โ€œJaeger(์˜ˆ๊ฑฐ)โ€๋ผ๋Š” Tracing ๋„๊ตฌ๋ฅผ ์‚ฌ์šฉํ•ด Istio์˜ ๋ถ„์‚ฐ ์ถ”์ ์— ๋Œ€ํ•ด ์‚ดํŽด๋ณด๊ณ ์ž ํ•œ๋‹ค. ์ „์ฒด์ ์ธ ๊ณผ์ •์€ Istio ๋ฌธ์„œ์˜ Jeager ์˜ˆ์ œ๋ฅผ ๋”ฐ๋ผํ•œ ๊ฒƒ์ž„์„ ๋ฐํžŒ๋‹ค.

์‚ฌ์ „ ์ค€๋น„

Tracing์„ ์ž˜ ์‚ดํŽด๋ณด๊ธฐ ์œ„ํ•ด MSA ๊ตฌ์กฐ๋ฅผ ๊ฐ€์ง„ ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜์„ ๋„์šฐ์ž. Istio ์˜ˆ์ œ์ธ โ€œbookinfoโ€๋ฅผ ๋„์šด๋‹ค. (์˜ˆ์ œ ํ•ธ์ฆˆ์˜จ์€ ์˜ˆ์ „์— ์จ๋‘” ํฌ์ŠคํŠธ ์ฐธ๊ณ .)

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/bookinfo/platform/kube/bookinfo.yaml

์š”์ฒญ์€ pod์— ์ ‘์†ํ•ด ์•„๋ž˜ ๋ช…๋ น์–ด๋กœ ๋ณด๋‚ผ ์ˆ˜๋„ ์žˆ๊ณ ,

$ k exec -it ... -- sh
~ $ for i in $(seq 1 100); do curl -s -o /dev/null "http://productpage.default.svc.cluster.local:9080/productpage"; done

Istio Gateway์™€ VirtualService๋ฅผ ๋„์›Œ์„œ External IP๋กœ ์š”์ฒญ์„ ๋ณด๋‚ผ ์ˆ˜๋„ ์žˆ๋‹ค.

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/bookinfo/networking/bookinfo-gateway.yaml
$ sh
sh-3.2$ for i in $(seq 1 100); do curl -s -o /dev/null "http://192.168.64.2/productpage"; done

์š”์ฒญ์„ 100๋ฒˆ์”ฉ ๋ฐ˜๋ณตํ•ด์„œ ๋ณด๋‚ด๋Š” ์ด์œ ๋Š” istio์˜ Trace Samping Rate ๋•Œ๋ฌธ์ด๋‹ค. istio ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๋„์šธ ๋•Œ ๋”ฐ๋กœ ์„ค์ •ํ•ด์ฃผ์ง€ ์•Š์•˜๋‹ค๋ฉด ๊ธฐ๋ณธ๊ฐ’์œผ๋กœ 1%์˜ ํŠธ๋ž˜ํ”ฝ๋งŒ์„ ์ƒ˜ํ”Œ๋ง ํ•˜๊ฒŒ ๋œ๋‹ค. ๊ทธ๋ž˜์„œ ์ตœ์†Œ 100๋ฒˆ์˜ ์š”์ฒญ์€ ๋ณด๋‚ด์•ผ ํ•˜๋Š” ๊ฒƒ!

Jaeger Addon ๋„์šฐ๊ธฐ

Istio์˜ ๋‹ค๋ฅธ addon์ธ โ€œPrometheusโ€, โ€œKialiโ€๋ฅผ ๋„์šธ ๋•Œ ํ–ˆ๋˜ ๊ฒƒ์ฒ˜๋Ÿผ istio์—์„œ ์ œ๊ณตํ•˜๋Š” addon sample ์ฝ”๋“œ๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ์†์‰ฝ๊ฒŒ Jeager๋ฅผ ๋„์šธ ์ˆ˜ ์žˆ๋‹ค.

$ kubectl apply \
    -f https://raw.githubusercontent.com/istio/istio/release-1.21/samples/addons/jaeger.yaml

Jaeger๋ฅผ ๋„์šฐ๊ณ  ๋‚˜๋ฉด, istio์—๊ฒŒ ์•ž์œผ๋กœ ์š” ์ฃผ์†Œ๋กœ tracing ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด๋‚ด๋ผ๊ณ  ๋ช…์‹œํ•ด์ค˜์•ผ ํ•œ๋‹ค.

istioctl \
  --set meshConfig.defaultConfig.tracing.zipkin.address=zipkin.istio-system.svc.cluster.local:9411

(์™œ Jaeger์ธ๋ฐ, zipkin ํ•„๋“œ์— ์„ค์ •ํ•˜๋Š”์ง€ ๋‹นํ™ฉํ•˜๋Š” ๋ถ€๋ถ„์ด๋‹ค. istio ์„ค๋ช…์— ๋”ฐ๋ฅด๋ฉด Jaeger๊ฐ€ zipkin์˜ ํฌ๋งท์„ ๋”ฐ๋ฅด๊ธฐ ๋•Œ๋ฌธ์ด๋ผ๊ณ  ํ•œ๋‹ค ใ…‡ใ……ใ…‡)

์ด์ œ Jaeger ๋Œ€์‹œ๋ณด๋“œ์— ์ ‘์† ํ•ด๋ณด์ž! istioctl๋กœ ์†์‰ฝ๊ฒŒ ๊ฐ€๋Šฅํ•˜๋‹ค.

$ istioctl dashboard jaeger

๊ทธ๋ฆฌ๊ณ  ์œ„์˜ ๋ฐฉ์‹๋Œ€๋กœ 100๋ฒˆ ์š”์ฒญ์„ ๋ณด๋‚ด๋ฉด, ์ด๋ ‡๊ฒŒ ๋‚ด๊ฐ€ ๋ณด๋‚ธ ์š”์ฒญ์€ ์–ด๋–ป๊ฒŒ ์ฒ˜๋ฆฌ๋˜๋Š”์ง€ ๊ทธ ํ๋ฆ„์„ ๊ฐ„ํŠธ ์ฐจํŠธ ๋Š๋‚Œ์œผ๋กœ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค!!

์บก์ณ์˜ ์š”์ฒญ์€ productpage๋ผ๋Š” ์›นํŽ˜์ด์ง€์— ์š”์ฒญ์„ ๋ณด๋‚ธ ๊ฒƒ์œผ๋กœ ์ด ์š”์ฒญ์ด ์ฒ˜๋ฆฌ๋˜๊ธฐ ์œ„ํ•ด์„œ details์™€ reviews ์›Œํฌ๋กœ๋“œ์— ์š”์ฒญ์„ ๋ณด๋ƒˆ๊ณ , reviews ์›Œํฌ๋กœ๋“œ๋Š” ๋˜ ratings ์›Œํฌ๋กœ๋“œ์— ์š”์ฒญ์„ ๋ณด๋‚ธ ๊ฑธ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค!!

์ด๋ ‡๊ฒŒ Kiali์ฒ˜๋Ÿผ ํŠธ๋ž˜ํ”ฝ ๊ทธ๋ž˜ํ”„๋„ ํ™•์ธํ•  ์ˆ˜ ์žˆ๊ณ 

Ingress Gateway๋ฅผ ํ†ตํ•ด ๋ณด๋‚ธ ์š”์ฒญ๋„ Jaeger๋กœ ์ถ”์ ์ด ๊ฐ€๋Šฅํ•˜๋‹ค!

Header์™€ Body ์ •๋ณด๋Š” ํ™•์ธ ๋ถˆ๊ฐ€

Istio Envoy Access Logging ํฌ์ŠคํŠธ์—์„œ๋„ ๊ทธ๋žฌ๋Š”๋ฐ, ์š”์ฒญ์˜ Header์™€ Body ์ •๋ณด๋Š” Tracing์—์„œ๋„ ํ™•์ธ ๋ถˆ๊ฐ€๋Šฅ ํ–ˆ๋‹ค.

Zipkin

Istio ์˜ˆ์ œ์—์„œ Zipkin๋„ ์„ธํŒ…ํ•ด๋ณผ ์ˆ˜ ์žˆ์—ˆ๋‹ค.

https://istio.io/latest/docs/ops/integrations/zipkin/#installation

$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.21/samples/addons/extras/zipkin.yaml

๊ทธ๋Ÿฐ๋ฐ ์จ๋ณด๋‹ˆ ๋ญ”๊ฐ€ Jaeger ๋ณด๋‹ค ์ƒ˜ํ”Œ๋ง์ด ์ž˜ ์•ˆ ๋˜๋Š” ๋Š๋‚Œ์ด์—ˆ๊ณ , UI/UX๋„ Jaeger๊ฐ€ ์ข€๋” ๋งˆ์Œ์— ๋“ค์—ˆ๋‹ค!!

Tracing Config

Trace Sampling

Istio์˜ ์ƒ˜ํ”Œ๋ง ๊ธฐ๋ณธ๊ฐ’์€ 1%(1.0)์ด๋‹ค. ๋งŒ์•ฝ ์ด ๊ฐ’์„ ์กฐ์ •ํ•˜๊ณ  ์‹ถ๋‹ค๋ฉด, istio๋ฅผ ์„ค์น˜ํ•  ๋•Œ IstioOperator์— ์žˆ๋Š” Config๋ฅผ ์กฐ์ •ํ•˜๋ฉด ๋œ๋‹ค.

...
spec:
  meshConfig:
    defaultConfig:
      tracing:
        sampling: 100.0

๋˜๋Š” Pod Annotation์œผ๋กœ Sampling Rate๋ฅผ ์กฐ์ •ํ•  ์ˆ˜๋„ ์žˆ๋‹ค.

...
metadata:
  annotations:
    proxy.istio.io/config: |
      tracing:
        sampling: 10

(Deprecated) Pilot Trace

๊ณผ๊ฑฐ์—๋Š” pilot์˜ traceSampling ๊ฐ’์„ ์กฐ์ •ํ•ด์„œ ์ƒ˜ํ”Œ๋ง ํ–ˆ์œผ๋‚˜, ์ด์ œ๋Š” MeshConfig์˜ ๊ฐ’์„ ํ†ตํ•ด ์กฐ์ •ํ•ด์•ผ ํ•œ๋‹ค๊ณ  ํ•œ๋‹ค.

์•„์ง ๋‘ ๋ฐฉ์‹ ๋ชจ๋‘ ์ง€์›ํ•˜๊ณ  ์žˆ์œผ๋‚˜, ๋งŒ์•ฝ ๋‘˜๋‹ค ๊ฐ’์ด ์ •์˜๋˜์–ด ์žˆ๋‹ค๋ฉด MeshConfig์˜ tracing.sampling์œผ๋กœ ๋ฎ์–ด์”Œ์›Œ ์ง„๋‹ค๊ณ  ํ•œ๋‹ค.

Extension Provider

Istio ์˜ˆ์ œ์—์„œ๋Š” Jaeger๋ฅผ Trace Provider๋กœ ์‚ฌ์šฉํ–ˆ๋Š”๋ฐ, Jaeger ์™ธ์—๋„ zipkin, Datadog, stackdriver, Apache SkyWalking ๋“ฑ์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋‹ค๊ณ  ํ•œ๋‹ค.

๋‚˜์—ด๋œ ๊ฒƒ๋“ค์€ ์ „๋ถ€๋‹ค ์•„๋Š” ๊ฑด ์•„๋‹ˆ์ง€๋งŒ Datadog์€ Tracing ๋ฟ๋งŒ ์•„๋‹ˆ๋ผ ๋ชจ๋‹ˆํ„ฐ๋ง๊ณผ ์„œ๋น„์Šค ๋””์Šค์ปค๋ฒ„๋ฆฌ, Anomaly Detection ๋“ฑ ๋‹ค์–‘ํ•œ ๊ธฐ๋Šฅ์ด ๋งŽ๊ณ (ํ•˜์ง€๋งŒ ๋น„์‹ธ๋‹คโ€ฆ!) Apache SkyWakling์€ ์ด๋ฒˆ์— ์ฒ˜์Œ ๋ดค๋Š”๋ฐ Datadog๊ณผ UI๋„ ๋น„์Šทํ•˜๊ณ  ๊ธฐ๋Šฅ๋„ ๊ฑฐ์˜ ๋น„์Šทํ•œ ๊ฒƒ ๊ฐ™๋‹ค. (์˜ค ์ข‹์€ ๊ฒƒ ๊ฐ™์€๋ฐ??)

Trace Context Propagation

์šฐ๋ฆฌ์˜ ๊ฐ“๊ฐ“ Istio ์„œ๋น„์Šค๋ฉ”์‰ฌ๋ฅผ ์ ์šฉํ•˜๊ธฐ๋งŒ ํ•˜๋ฉด โ€œ๋ถ„์‚ฐ ์ถ”์ โ€์„ ๋ˆ„๋ฆด ์ˆ˜ ์žˆ์„ ๊ฒƒ ๊ฐ™์ง€๋งŒ, ํ˜„์‹ค์€ ๊ทธ๋ ‡์ง€ ์•Š๋‹ค ใ… ใ… 

Tracing์„ ๊ตฌํ˜„ํ•˜๋ ค๋ฉด ์š”์ฒญ์„ ๋ฐ›์€ Application์—์„œ ์š”์ฒญ์„ ์ถ”์ ํ•˜๊ธฐ ์œ„ํ•ด ๋””์ž์ธ๋œ Tracing Header๋ฅผ ๋‹ค์Œ ์š”์ฒญ์„ ํฌ์›Œ๋“œ(forward) ํ•ด์ฃผ๋Š” ์ฝ”๋“œ๊ฐ€ ๊ตฌํ˜„๋˜์–ด ์žˆ์–ด์•ผ ํ•œ๋‹ค!!

์–ด๋–ค Tracing Provider๋ฅผ ์“ฐ๋Š๋ƒ์— ๋”ฐ๋ผ์„œ ํฌ์›Œ๋“œ๋ฅผ ๊ตฌํ˜„ํ•ด์•ผ ํ•˜๋Š” ํ—ค๋”๊ฐ€ ๋‹ค๋ฅด์ง€๋งŒ, ์ผ๋‹จ x-request-id ํ—ค๋”๋Š” ๋ฌด์กฐ๊ฑด ํฌ์›Œ๋”ฉ ํ•ด์•ผ ํ•œ๋‹ค.

๋งŒ์•ฝ, Zipkin, Jaeger ๋“ฑ์„ Tracing ๋„๊ตฌ๋กœ ์ฑ„ํƒ ํ–ˆ๋‹ค๋ฉด ์•„๋ž˜์˜ B3 ํ—ค๋”๋“ค์„ ํฌ์›Œ๋“œ ํ•ด์•ผ ํ•œ๋‹ค. (B3๊ฐ€ ์•ฝ์ž๋ผ๊ฑฐ๋‚˜ ํŠน๋ณ„ํ•œ ์˜๋ฏธ๊ฐ€ ์žˆ๋Š”๊ฑด ์•„๋‹ˆ๋‹ค.)

  • x-b3-traceid
  • x-b3-spanid
  • x-b3-parentspanid
  • x-b3-sampled
  • x-b3-flags

์ฐธ๊ณ ๋กœ Jaeger๋„ b3 ํ—ค๋” ํ˜•์‹์„ ๋”ฐ๋ฅด๊ธฐ ๋•Œ๋ฌธ์—, ์š”์ฒญ์„ ๋ฌด์กฐ๊ฑด Tracing ํ•ด์•ผ ํ•œ๋‹ค๋ฉด --header "x-b3-sampled: 1"๋กœ ์„ธํŒ…ํ•˜๋ฉด Sample Rate๋ฅผ ๋ฌด์‹œํ•˜๊ณ  ์ถ”์ ์ด ๋œ๋‹ค ใ…‹ใ…‹

Istio ์˜ˆ์ œ์— ๊ตฌํ˜„ ์ฝ”๋“œ๋„ ์‚ด์ง ๋‚˜์™€์žˆ๋Š”๋ฐ Java์—์„  ์ด๋Ÿฐ ๋Š๋‚Œ์ด๋‹ค.

@GET
@Path("/reviews/{productId}")
public Response bookReviewsById(
    @PathParam("productId") int productId,
    @Context HttpHeaders requestHeaders // ์š”์ฒญ์˜ ํ—ค๋”๋ฅผ
) {
  // ...
  if (ratings_enabled) {
    JsonObject ratingsResponse = getRatings(
      Integer.toString(productId),
      requestHeaders // ๊ทธ๋Œ€๋กœ ๋„˜๊ฒจ์คฌ๋‹ค!!
    );

์ฐธ๊ณ ์ž๋ฃŒ