DevFW-CICD/ingress-nginx-helm
6
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ingress-nginx-helm/user-guide/third-party-addons/opentelemetry/index.html

174 lines
36 KiB
HTML
Raw Normal View History

Deploy GitHub Pages
2023-03-22 18:59:30 +00:00
<!doctype html><html lang=en class=no-js> <head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><link href=https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/ rel=canonical><link rel="shortcut icon" href=../../../assets/images/favicon.png><meta name=generator content="mkdocs-1.1.2, mkdocs-material-6.2.4"><title>OpenTelemetry - NGINX Ingress Controller</title><link rel=stylesheet href=../../../assets/stylesheets/main.15aa0b43.min.css><link rel=stylesheet href=../../../assets/stylesheets/palette.75751829.min.css><meta name=theme-color content=#009485><link rel=preconnect href=https://fonts.gstatic.com crossorigin><link rel=stylesheet href="https://fonts.googleapis.com/css?family=Roboto:300,400,400i,700%7CRoboto+Mono&display=fallback"><style>body,input{font-family:"Roboto",-apple-system,BlinkMacSystemFont,Helvetica,Arial,sans-serif}code,kbd,pre{font-family:"Roboto Mono",SFMono-Regular,Consolas,Menlo,monospace}</style><link rel=stylesheet href=../../../extra.css><script>window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)},ga.l=+new Date,ga("create","UA-118407822-1","kubernetes.github.io"),ga("set","anonymizeIp",!0),ga("send","pageview"),document.addEventListener("DOMContentLoaded",function(){document.forms.search&&document.forms.search.query.addEventListener("blur",function(){if(this.value){var e=document.location.pathname;ga("send","pageview",e+"?q="+this.value)}})}),document.addEventListener("DOMContentSwitch",function(){ga("send","pageview",document.location.pathname)})</script><script async src=https://www.google-analytics.com/analytics.js></script></head> <body dir=ltr data-md-color-scheme data-md-color-primary=teal data-md-color-accent=green> <input class=md-toggle data-md-toggle=drawer type=checkbox id=__drawer autocomplete=off> <input class=md-toggle data-md-toggle=search type=checkbox id=__search autocomplete=off> <label class=md-overlay for=__drawer></label> <div data-md-component=skip> <a href=#opentelemetry class=md-skip> Skip to content </a> </div> <div data-md-component=announce> </div> <header class=md-header data-md-component=header> <nav class="md-header-nav md-grid" aria-label=Header> <a href=https://kubernetes.github.io/ingress-nginx title="NGINX Ingress Controller" class="md-header-nav__button md-logo" aria-label="NGINX Ingress Controller"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 8a3 3 0 003-3 3 3 0 00-3-3 3 3 0 00-3 3 3 3 0 003 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54z"/></svg> </a> <label class="md-header-nav__button md-icon" for=__drawer> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M3 6h18v2H3V6m0 5h18v2H3v-2m0 5h18v2H3v-2z"/></svg> </label> <div class=md-header-nav__title data-md-component=header-title> <div class=md-header-nav__ellipsis> <div class=md-header-nav__topic> <span class=md-ellipsis> NGINX Ingress Controller </span> </div> <div class=md-header-nav__topic> <span class=md-ellipsis> OpenTelemetry </span> </div> </div> </div> <label class="md-header-nav__button md-icon" for=__search> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0116 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 019.5 16 6.5 6.5 0 013 9.5 6.5 6.5 0 019.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5z"/></svg> </label> <div class=md-search data-md-component=search role=dialog> <label class=md-search__overlay for=__search></label> <div class=md-search__inner role=search> <form class=md-search__form name=search> <input type=text class=md-search__input name=query aria-label=Search placeholder=Search autocapitalize=off autocorrect=off autocomplete=off spellcheck=false data-md-component=search-query data-md-state=active required> <label class="md-search__icon md-icon" for=__search> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0116 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516
<span class=nt>enable-opentelemetry</span><span class=p>:</span> <span class=s>&quot;true&quot;</span>
</code></pre></div></p> <p>To enable or disable instrumentation for a single Ingress, use the <code>enable-opentelemetry</code> annotation: <div class=highlight><pre><span></span><code><span class=nt>kind</span><span class=p>:</span> <span class="l l-Scalar l-Scalar-Plain">Ingress</span>
<span class=nt>metadata</span><span class=p>:</span>
<span class=nt>annotations</span><span class=p>:</span>
<span class=nt>nginx.ingress.kubernetes.io/enable-opentelemetry</span><span class=p>:</span> <span class=s>&quot;true&quot;</span>
</code></pre></div></p> <p>We must also set the host to use when uploading traces:</p> <p><div class=highlight><pre><span></span><code><span class=nt>otlp-collector-host</span><span class=p>:</span> <span class=s>&quot;otel-coll-collector.otel.svc&quot;</span>
</code></pre></div> NOTE: While the option is called <code>otlp-collector-host</code>, you will need to point this to any backend that recieves otlp-grpc.</p> <p>Next you will need to deploy a distributed telemetry system which uses OpenTelemetry. <a href=https://github.com/open-telemetry/opentelemetry-collector>opentelemetry-collector</a>, <a href=https://www.jaegertracing.io/ >Jaeger</a> <a href=https://github.com/grafana/tempo>Tempo</a>, and <a href=https://zipkin.io/ >zipkin</a> have been tested.</p> <p>Other optional configuration options: <div class=highlight><pre><span></span><code><span class=c1># specifies the name to use for the server span</span>
<span class="l l-Scalar l-Scalar-Plain">opentelemetry-operation-name</span>
<span class="l l-Scalar l-Scalar-Plain"># sets whether or not to trust incoming telemetry spans</span>
<span class="l l-Scalar l-Scalar-Plain">opentelemetry-trust-incoming-span</span>
<span class="l l-Scalar l-Scalar-Plain"># specifies the port to use when uploading traces, Default</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">4317</span>
<span class="l l-Scalar l-Scalar-Plain">otlp-collector-port</span>
<span class=c1># specifies the service name to use for any traces created, Default: nginx</span>
<span class="l l-Scalar l-Scalar-Plain">otel-service-name</span>
<span class=c1># The maximum queue size. After the size is reached data are dropped.</span>
<span class="l l-Scalar l-Scalar-Plain">otel-max-queuesize</span>
<span class=c1># The delay interval in milliseconds between two consecutive exports.</span>
<span class="l l-Scalar l-Scalar-Plain">otel-schedule-delay-millis</span>
<span class=c1># How long the export can run before it is cancelled.</span>
<span class="l l-Scalar l-Scalar-Plain">otel-schedule-delay-millis</span>
<span class=c1># The maximum batch size of every export. It must be smaller or equal to maxQueueSize.</span>
<span class="l l-Scalar l-Scalar-Plain">otel-max-export-batch-size</span>
<span class=c1># specifies sample rate for any traces created, Default: 0.01</span>
<span class="l l-Scalar l-Scalar-Plain">otel-sampler-ratio</span>
<span class=c1># specifies the sampler to be used when sampling traces.</span>
<span class=c1># The available samplers are: AlwaysOn, AlwaysOff, TraceIdRatioBased, Default: AlwaysOff</span>
<span class="l l-Scalar l-Scalar-Plain">otel-sampler</span>
<span class=c1># Uses sampler implementation which by default will take a sample if parent Activity is sampled, Default: false</span>
<span class="l l-Scalar l-Scalar-Plain">otel-sampler-parent-based</span>
</code></pre></div></p> <p>Note that you can also set whether to trust incoming spans (global default is true) per-location using annotations like the following: <div class=highlight><pre><span></span><code><span class=nt>kind</span><span class=p>:</span> <span class="l l-Scalar l-Scalar-Plain">Ingress</span>
<span class=nt>metadata</span><span class=p>:</span>
<span class=nt>annotations</span><span class=p>:</span>
<span class=nt>nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span</span><span class=p>:</span> <span class=s>&quot;true&quot;</span>
</code></pre></div></p> <h2 id=examples>Examples<a class=headerlink href=#examples title="Permanent link"></a></h2> <p>The following examples show how to deploy and test different distributed telemetry systems. These example can be performed using Docker Desktop.</p> <p>In the <a href=https://github.com/esigo/nginx-example>esigo/nginx-example</a> GitHub repository is an example of a simple hello service:</p> <pre class=mermaid><code>graph TB
subgraph Browser
start["http://esigo.dev/hello/nginx"]
end
subgraph app
sa[service-a]
sb[service-b]
sa --&gt; |name: nginx| sb
sb --&gt; |hello nginx!| sa
end
subgraph otel
otc["Otel Collector"]
end
subgraph observability
tempo["Tempo"]
grafana["Grafana"]
backend["Jaeger"]
zipkin["Zipkin"]
end
subgraph ingress-nginx
ngx[nginx]
end
subgraph ngx[nginx]
ng[nginx]
om[OpenTelemetry module]
end
subgraph Node
app
otel
observability
ingress-nginx
om --&gt; |otlp-gRPC| otc --&gt; |jaeger| backend
otc --&gt; |zipkin| zipkin
otc --&gt; |otlp-gRPC| tempo --&gt; grafana
sa --&gt; |otlp-gRPC| otc
sb --&gt; |otlp-gRPC| otc
start --&gt; ng --&gt; sa
end</code></pre> <p>To install the example and collectors run:</p> <ol> <li> <p>Enable Ingress addon with:</p> <div class=highlight><pre><span></span><code> <span class=nt>opentelemetry</span><span class=p>:</span>
<span class=nt>enabled</span><span class=p>:</span> <span class="l l-Scalar l-Scalar-Plain">true</span>
<span class=nt>image</span><span class=p>:</span> <span class="l l-Scalar l-Scalar-Plain">registry.k8s.io/ingress-nginx/opentelemetry:v20230107-helm-chart-4.4.2-2-g96b3d2165@sha256:331b9bebd6acfcd2d3048abbdd86555f5be76b7e3d0b5af4300b04235c6056c9</span>
<span class=nt>containerSecurityContext</span><span class=p>:</span>
<span class=nt>allowPrivilegeEscalation</span><span class=p>:</span> <span class="l l-Scalar l-Scalar-Plain">false</span>
</code></pre></div> </li> <li> <p>Enable OpenTelemetry and set the otlp-collector-host:</p> <div class=highlight><pre><span></span><code><span class="l l-Scalar l-Scalar-Plain">$ echo &#39;</span>
<span class="l l-Scalar l-Scalar-Plain">apiVersion</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">v1</span>
<span class="l l-Scalar l-Scalar-Plain">kind</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">ConfigMap</span>
<span class="l l-Scalar l-Scalar-Plain">data</span><span class="p p-Indicator">:</span>
<span class=nt>enable-opentelemetry</span><span class=p>:</span> <span class=s>&quot;true&quot;</span>
<span class=nt>opentelemetry-config</span><span class=p>:</span> <span class=s>&quot;/etc/nginx/opentelemetry.toml&quot;</span>
<span class=nt>opentelemetry-operation-name</span><span class=p>:</span> <span class=s>&quot;HTTP</span><span class=nv> </span><span class=s>$request_method</span><span class=nv> </span><span class=s>$service_name</span><span class=nv> </span><span class=s>$uri&quot;</span>
<span class=nt>opentelemetry-trust-incoming-span</span><span class=p>:</span> <span class=s>&quot;true&quot;</span>
<span class=nt>otlp-collector-host</span><span class=p>:</span> <span class=s>&quot;otel-coll-collector.otel.svc&quot;</span>
<span class=nt>otlp-collector-port</span><span class=p>:</span> <span class=s>&quot;4317&quot;</span>
<span class=nt>otel-max-queuesize</span><span class=p>:</span> <span class=s>&quot;2048&quot;</span>
<span class=nt>otel-schedule-delay-millis</span><span class=p>:</span> <span class=s>&quot;5000&quot;</span>
<span class=nt>otel-max-export-batch-size</span><span class=p>:</span> <span class=s>&quot;512&quot;</span>
<span class=nt>otel-service-name</span><span class=p>:</span> <span class=s>&quot;nginx-proxy&quot;</span> <span class=c1># Opentelemetry resource name</span>
<span class=nt>otel-sampler</span><span class=p>:</span> <span class=s>&quot;AlwaysOn&quot;</span> <span class=c1># Also: AlwaysOff, TraceIdRatioBased</span>
<span class=nt>otel-sampler-ratio</span><span class=p>:</span> <span class=s>&quot;1.0&quot;</span>
<span class=nt>otel-sampler-parent-based</span><span class=p>:</span> <span class=s>&quot;false&quot;</span>
<span class=" -Error"> </span><span class=nt>metadata</span><span class=p>:</span>
<span class=nt>name</span><span class=p>:</span> <span class="l l-Scalar l-Scalar-Plain">ingress-nginx-controller</span>
<span class=nt>namespace</span><span class=p>:</span> <span class="l l-Scalar l-Scalar-Plain">ingress-nginx</span>
<span class=s>&#39;</span><span class=nv> </span><span class=s>|</span><span class=nv> </span><span class=s>kubectl</span><span class=nv> </span><span class=s>replace</span><span class=nv> </span><span class=s>-f</span><span class=nv> </span><span class=s>-</span>
</code></pre></div> </li> <li> <p>Deploy otel-collector, grafana and Jaeger backend:</p> <div class=highlight><pre><span></span><code><span class=c1># add helm charts needed for grafana and OpenTelemetry collector</span>
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
<span class=c1># deply cert-manager needed for OpenTelemetry collector operator</span>
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml
<span class=c1># create observability namespace</span>
kubectl apply -f https://raw.githubusercontent.com/esigo/nginx-example/main/observability/namespace.yaml
<span class=c1># install OpenTelemetry collector operator</span>
helm upgrade --install otel-collector-operator -n otel --create-namespace open-telemetry/opentelemetry-operator
<span class=c1># deploy OpenTelemetry collector</span>
kubectl apply -f https://raw.githubusercontent.com/esigo/nginx-example/main/observability/collector.yaml
<span class=c1># deploy Jaeger all-in-one</span>
kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.37.0/jaeger-operator.yaml -n observability
kubectl apply -f https://raw.githubusercontent.com/esigo/nginx-example/main/observability/jaeger.yaml -n observability
<span class=c1># deploy zipkin</span>
kubectl apply -f https://raw.githubusercontent.com/esigo/nginx-example/main/observability/zipkin.yaml -n observability
<span class=c1># deploy tempo and grafana</span>
helm upgrade --install tempo grafana/tempo --create-namespace -n observability
helm upgrade -f https://raw.githubusercontent.com/esigo/nginx-example/main/observability/grafana/grafana-values.yaml --install grafana grafana/grafana --create-namespace -n observability
</code></pre></div> </li> <li> <p>Build and deploy demo app:</p> <div class=highlight><pre><span></span><code><span class=c1># build images</span>
make images
<span class=c1># deploy demo app:</span>
make deploy-app
</code></pre></div> </li> <li> <p>Make a few requests to the Service:</p> <div class=highlight><pre><span></span><code>kubectl port-forward --namespace<span class=o>=</span>ingress-nginx service/ingress-nginx-controller <span class=m>8090</span>:80
curl http://esigo.dev:8090/hello/nginx
StatusCode : <span class=m>200</span>
StatusDescription : OK
Content : <span class=o>{</span><span class=s2>&quot;v&quot;</span>:<span class=s2>&quot;hello nginx!&quot;</span><span class=o>}</span>
RawContent : HTTP/1.1 <span class=m>200</span> OK
Connection: keep-alive
Content-Length: <span class=m>21</span>
Content-Type: text/plain<span class=p>;</span> <span class=nv>charset</span><span class=o>=</span>utf-8
Date: Mon, <span class=m>10</span> Oct <span class=m>2022</span> <span class=m>17</span>:43:33 GMT
<span class=o>{</span><span class=s2>&quot;v&quot;</span>:<span class=s2>&quot;hello nginx!&quot;</span><span class=o>}</span>
Forms : <span class=o>{}</span>
Headers : <span class=o>{[</span>Connection, keep-alive<span class=o>]</span>, <span class=o>[</span>Content-Length, <span class=m>21</span><span class=o>]</span>, <span class=o>[</span>Content-Type, text/plain<span class=p>;</span> <span class=nv>charset</span><span class=o>=</span>utf-8<span class=o>]</span>, <span class=o>[</span>Date,
Mon, <span class=m>10</span> Oct <span class=m>2022</span> <span class=m>17</span>:43:33 GMT<span class=o>]}</span>
Images : <span class=o>{}</span>
InputFields : <span class=o>{}</span>
Links : <span class=o>{}</span>
ParsedHtml : System.__ComObject
RawContentLength : <span class=m>21</span>
</code></pre></div> </li> <li> <p>View the Grafana UI:</p> <p><div class=highlight><pre><span></span><code>kubectl port-forward --namespace<span class=o>=</span>observability service/grafana <span class=m>3000</span>:80
</code></pre></div> In the Grafana interface we can see the details: <img alt="grafana screenshot" src=../../../images/otel-grafana-demo.png title="grafana screenshot"></p> </li> <li> <p>View the Jaeger UI:</p> <p><div class=highlight><pre><span></span><code>kubectl port-forward --namespace<span class=o>=</span>observability service/jaeger-all-in-one-query <span class=m>16686</span>:16686
</code></pre></div> In the Jaeger interface we can see the details: <img alt="Jaeger screenshot" src=../../../images/otel-jaeger-demo.png title="Jaeger screenshot"></p> </li> <li> <p>View the Zipkin UI:</p> <p><div class=highlight><pre><span></span><code>kubectl port-forward --namespace<span class=o>=</span>observability service/zipkin <span class=m>9411</span>:9411
</code></pre></div> In the Zipkin interface we can see the details: <img alt="zipkin screenshot" src=../../../images/otel-zipkin-demo.png title="zipkin screenshot"></p> </li> </ol> </article> </div> </div> </main> <footer class=md-footer> <div class=md-footer-nav> <nav class="md-footer-nav__inner md-grid" aria-label=Footer> <a href=../opentracing/ class="md-footer-nav__link md-footer-nav__link--prev" rel=prev> <div class="md-footer-nav__button md-icon"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12z"/></svg> </div> <div class=md-footer-nav__title> <div class=md-ellipsis> <span class=md-footer-nav__direction> Previous </span> OpenTracing </div> </div> </a> <a href=../../../examples/ class="md-footer-nav__link md-footer-nav__link--next" rel=next> <div class=md-footer-nav__title> <div class=md-ellipsis> <span class=md-footer-nav__direction> Next </span> Introduction </div> </div> <div class="md-footer-nav__button md-icon"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11H4z"/></svg> </div> </a> </nav> </div> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class=md-footer-copyright> Made with <a href=https://squidfunk.github.io/mkdocs-material/ target=_blank rel=noopener> Material for MkDocs </a> </div> </div> </div> </footer> </div> <script src=../../../assets/javascripts/vendor.93c04032.min.js></script> <script src=../../../assets/javascripts/bundle.83e5331e.min.js></script><script id=__lang type=application/json>{"clipboard.copy": "Copy to clipboard", "clipboard.copied": "Copied to clipboard", "search.config.lang": "en", "search.config.pipeline": "trimmer, stopWordFilter", "search.config.separator": "[\\s\\-]+", "search.placeholder": "Search", "search.result.placeholder": "Type to start searching", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.term.missing": "Missing"}</script> <script>
app = initialize({
base: "../../..",
features: ['navigation.tabs', 'navigation.tabs.sticky', 'navigation.instant', 'navigation.sections'],
search: Object.assign({
worker: "../../../assets/javascripts/worker/search.8c7e0a7e.min.js"
}, typeof search !== "undefined" && search)
})
</script> </body> </html>
Reference in a new issue Copy permalink