</code></pre></div></p></li><li>You can validate that the controller is configured for metrics by looking at the values of the installed release, like this: <divclass=highlight><pre><span></span><code>helm get values ingress-controller --namespace ingress-nginx
</code></pre></div></li><li>You should be able to see the values shown below: <divclass=highlight><pre><span></span><code>..
</code></pre></div></li><li>If you are <strong>not using helm</strong>, you will have to edit your manifests like this:<ul><li>Service manifest: <divclass=highlight><pre><span></span><code>apiVersion: v1
</code></pre></div></li></ul></li></ul><h3id=deploy-and-configure-prometheus-server>Deploy and configure Prometheus Server<aclass=headerlinkhref=#deploy-and-configure-prometheus-servertitle="Permanent link"> ¶</a></h3><p>Note that the kustomize bases used in this tutorial are stored in the <ahref=https://github.com/kubernetes/ingress-nginx/tree/main/deploy>deploy</a> folder of the GitHub repository <ahref=https://github.com/kubernetes/ingress-nginx>kubernetes/ingress-nginx</a>.</p><ul><li><p>The Prometheus server must be configured so that it can discover endpoints of services. If a Prometheus server is already running in the cluster and if it is configured in a way that it can find the ingress controller pods, no extra configuration is needed.</p></li><li><p>If there is no existing Prometheus server running, the rest of this tutorial will guide you through the steps needed to deploy a properly configured Prometheus server.</p></li><li><p>Running the following command deploys prometheus in Kubernetes:</p></li></ul><divclass=highlight><pre><span></span><code>kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/
</code></pre></div><h4id=prometheus-dashboard>Prometheus Dashboard<aclass=headerlinkhref=#prometheus-dashboardtitle="Permanent link"> ¶</a></h4><ul><li>Open Prometheus dashboard in a web browser:</li></ul><divclass=highlight><pre><span></span><code><spanclass=go>kubectl get svc -n ingress-nginx</span>
</code></pre></div><ul><li>Obtain the IP address of the nodes in the running cluster:</li></ul><divclass=highlight><pre><span></span><code><spanclass=go>kubectl get nodes -o wide</span>
</code></pre></div><ul><li>In some cases where the node only have internal IP addresses we need to execute:</li></ul><divclass=highlight><pre><span></span><code>kubectl get nodes --selector=kubernetes.io/role!=master -o jsonpath={.items[*].status.addresses[?\(@.type==\"InternalIP\"\)].address}
</code></pre></div><ul><li><p>Open your browser and visit the following URL: <em>http://{node IP address}:{prometheus-svc-nodeport}</em> to load the Prometheus Dashboard.</p></li><li><p>According to the above example, this URL will be http://10.192.0.3:32630</p></li></ul><p><imgalt="Prometheus Dashboard"src=../../images/prometheus-dashboard.png></p><h4id=grafana>Grafana<aclass=headerlinkhref=#grafanatitle="Permanent link"> ¶</a></h4><ul><li>Install grafana using the below command <divclass=highlight><pre><span></span><code>kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/
</code></pre></div></p></li><li><p>Open your browser and visit the following URL: <em>http://{node IP address}:{grafana-svc-nodeport}</em> to load the Grafana Dashboard. According to the above example, this URL will be http://10.192.0.3:31086</p></li></ul><p>The username and password is <code>admin</code></p><ul><li><p>After the login you can import the Grafana dashboard from <ahref=https://github.com/kubernetes/ingress-nginx/tree/main/deploy/grafana/dashboards>official dashboards</a>, by following steps given below :</p><ul><li>Navigate to lefthand panel of grafana</li><li>Hover on the gearwheel icon for Configuration and click "Data Sources"</li><li>Click "Add data source"</li><li>Select "Prometheus"</li><li>Enter the details (note: I used http://CLUSTER_IP_PROMETHEUS_SVC:9090)</li><li>Left menu (hover over +) -> Dashboard</li><li>Click "Import"</li><li>Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json</li><li>Click Import JSON</li><li>Select the Prometheus data source</li><li>Click "Import"</li></ul></li></ul><p><imgalt="Grafana Dashboard"src=../../images/grafana.png></p><h3id=caveats>Caveats<aclass=headerlinkhref=#caveatstitle="Permanent link"> ¶</a></h3><h4id=wildcard-ingresses>Wildcard ingresses<aclass=headerlinkhref=#wildcard-ingressestitle="Permanent link"> ¶</a></h4><ul><li>By default request metrics are labeled with the hostname. When you have a wildcard domain ingress, then there will be no metrics for that ingress (to prevent the metrics from exploding in cardinality). To get metrics in this case you need to run the ingress controller with <code>--metrics-per-host=false</code> (you will lose labeling by hostname, but still have labeling by ingress).</li></ul><h3id=grafana-dashboard-using-ingress-resource>Grafana dashboard using ingress resource<aclass=headerlinkhref=#grafana-dashboard-using-ingress-resourcetitle="Permanent link"> ¶</a></h3><ul><li>If you want to expose the dashboard for grafana using a ingress resource, then you can : <ul><li>change the service type of the prometheus-server service and the grafana service to "ClusterIP" like this : <divclass=highlight><pre><span></span><code>kubectl -n ingress-nginx edit svc grafana
</code></pre></div></li><li>This will open the currently deployed service grafana in the default editor configured in your shell (vi/nvim/nano/other)</li><li>scroll down to line 34 that looks like "type: NodePort"</li><li>change it to look like "type: ClusterIP". Save and exit.</li><li>create a ingress resource with backend as "grafana" and port as "3000"</li></ul></li><li>Similarly, you can edit the service "prometheus-server" and add a ingress resource.</li></ul><h2id=prometheus-and-grafana-installation-using-service-monitors>PROMETHEUS AND GRAFANA INSTALLATION USING SERVICE MONITORS<aclass=headerlinkhref=#prometheus-and-grafana-installation-using-service-monitorstitle="Permanent link"> ¶</a></h2><p>This document assumes you're using helm and using the kube-prometheus-stack package to install Prometheus and Grafana. </p><h3id=verify-nginx-ingress-controller-is-installed>Verify NGINX Ingress controller is installed<aclass=headerlinkhref=#verify-nginx-ingress-controller-is-installedtitle="Permanent link"> ¶</a></h3><ul><li><p>The NGINX Ingress controller should already be deployed according to the deployment instructions <ahref=../../deploy/>here</a>.</p></li><li><p>To check if Ingress controller is deployed, <divclass=highlight><pre><span></span><code>kubectl get pods -n ingress-nginx
</code></pre></div></p></li><li>The result should look something like: <code>NAME READY STATUS RESTARTS AGE ingress-nginx-controller-7c489dc7b7-ccrf6 1/1 Running 0 19h</code></li></ul><h3id=verify-prometheus-is-installed>Verify Prometheus is installed<aclass=headerlinkhref=#verify-prometheus-is-installedtitle="Permanent link"> ¶</a></h3><ul><li>To check if Prometheus is already deployed, run the following command:</li></ul><p><divclass=highlight><pre><span></span><code>helm ls -A
</code></pre></div><divclass=highlight><pre><span></span><code>NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
</code></pre></div> - Notice that prometheus is installed in a differenet namespace than ingress-nginx</p><ul><li>If prometheus is not installed, then you can install from <ahref=https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack>here</a></li></ul><h3id=re-configure-nginx-ingress-controller>Re-configure NGINX Ingress controller<aclass=headerlinkhref=#re-configure-nginx-ingress-controllertitle="Permanent link"> ¶</a></h3><ul><li>The Ingress NGINX controller needs to be reconfigured for exporting metrics. This requires 3 additional configurations to the controller. These configurations are : <divclass=highlight><pre><span></span><code>controller.metrics.enabled=true
</code></pre></div></li><li>The easiest way of doing this is to helm upgrade <divclass=highlight><pre><span></span><code>helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
</code></pre></div></li><li><p>Here <code>controller.metrics.serviceMonitor.additionalLabels.release="prometheus"</code> should match the name of the helm release of the <code>kube-prometheus-stack</code></p></li><li><p>You can validate that the controller has been successfully reconfigured to export metrics by looking at the values of the installed release, like this: <divclass=highlight><pre><span></span><code>helm get values ingress-nginx --namespace ingress-nginx
</code></pre></div></p></li></ul><h3id=configure-prometheus>Configure Prometheus<aclass=headerlinkhref=#configure-prometheustitle="Permanent link"> ¶</a></h3><ul><li>Since Prometheus is running in a different namespace and not in the ingress-nginx namespace, it would not be able to discover ServiceMonitors in other namespaces when installed. Reconfigure your kube-prometheus-stack Helm installation to set <code>serviceMonitorSelectorNilUsesHelmValues</code> flag to false. By default, Prometheus only discovers PodMonitors within its own namespace. This should be disabled by setting <code>podMonitorSelectorNilUsesHelmValues</code> to false</li><li>The configurations required are: <divclass=highlight><pre><span></span><code>prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false
</code></pre></div></li><li>The easiest way of doing this is to use <code>helm upgrade ...</code><divclass=highlight><pre><span></span><code>helm upgrade prometheus prometheus-community/kube-prometheus-stack \
</code></pre></div></li><li>You can validate that Prometheus has been reconfigured by looking at the values of the installed release, like this: <divclass=highlight><pre><span></span><code>helm get values prometheus --namespace prometheus
</code></pre></div></li><li>You should be able to see the values shown below: <divclass=highlight><pre><span></span><code>prometheus:
prometheusSpec:
podMonitorSelectorNilUsesHelmValues: false
serviceMonitorSelectorNilUsesHelmValues: false
</code></pre></div></li></ul><h3id=connect-and-view-prometheus-dashboard>Connect and view Prometheus dashboard<aclass=headerlinkhref=#connect-and-view-prometheus-dashboardtitle="Permanent link"> ¶</a></h3><ul><li>Port forward to Prometheus service. Find out the name of the prometheus service by using the following command: <divclass=highlight><pre><span></span><code>kubectl get svc -n prometheus
</code></pre></div></li></ul><p>The result of this command would look like: <divclass=highlight><pre><span></span><code>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
</code></pre></div> prometheus-kube-prometheus-prometheus is the service we want to port forward to. We can do so using the following command: <divclass=highlight><pre><span></span><code>kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n prometheus 9090:9090
</code></pre></div> When you run the above command, you should see something like: <divclass=highlight><pre><span></span><code>Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
</code></pre></div> - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:9090</p><p><imgalt="Prometheus Dashboard"src=../../images/prometheus-dashboard1.png></p><h3id=connect-and-view-grafana-dashboard>Connect and view Grafana dashboard<aclass=headerlinkhref=#connect-and-view-grafana-dashboardtitle="Permanent link"> ¶</a></h3><ul><li>Port forward to Grafana service. Find out the name of the Grafana service by using the following command: <divclass=highlight><pre><span></span><code>kubectl get svc -n prometheus
</code></pre></div></li></ul><p>The result of this command would look like: <divclass=highlight><pre><span></span><code>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
</code></pre></div> prometheus-grafana is the service we want to port forward to. We can do so using the following command: <divclass=highlight><pre><span></span><code>kubectl port-forward svc/prometheus-grafana 3000:80 -n prometheus
</code></pre></div> When you run the above command, you should see something like: <divclass=highlight><pre><span></span><code>Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000
</code></pre></div> - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:3000 The default username/ password is admin/prom-operator - After the login you can import the Grafana dashboard from <ahref=https://github.com/kubernetes/ingress-nginx/tree/main/deploy/grafana/dashboards>official dashboards</a>, by following steps given below :</p><ul><li>Navigate to lefthand panel of grafana</li><li>Hover on the gearwheel icon for Configuration and click "Data Sources"</li><li>Click "Add data source"</li><li>Select "Prometheus"</li><li>Enter the details (note: I used http://10.102.72.134:9090 which is the CLUSTER-IP for Prometheus service)</li><li>Left menu (hover over +) -> Dashboard</li><li>Click "Import"</li><li>Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json</li><li>Click Import JSON</li><li>Select the Prometheus data source</li><li>Click "Import"</li></ul><p><imgalt="Grafana Dashboard"src=../../images/grafana-dashboard1.png></p></article></div></div></main><footerclass=md-footer><divclass=md-footer-nav><navclass="md-footer-nav__inner md-grid"aria-label=Footer><ahref=../miscellaneous/class="md-footer-nav__link md-footer-nav__link--prev"rel=prev><divclass="md-footer-nav__button md-icon"><svgxmlns=http://www.w3.org/2000/svgviewbox="0 0 24 24"><pathd="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12z"/></svg></div><divclass=md-footer-nav__title><divclass=md-ellipsis><spanclass=md-footer-nav__direction> Previous </span> Miscellaneous </div></div></a><ahref=../multiple-ingress/class="md-footer-nav__link md-footer-nav__link--next"rel=next><divclass=md-footer-nav__title><divclass=md-ellipsis><spanclass=md-footer-nav__direction> Next </span> Multiple Ingress controllers </div></div><divclass="md-footer-nav__button md-icon"><svgxmlns=http://www.w3.org/2000/svgviewbox="0 0 24 24"><pathd="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11H4z"/></svg></div></a></nav></div><divclass="md-footer-meta md-typeset"><divclass="md-footer-meta__inner md-grid"><divclass=md-footer-copyright> Made with <ahref=https://squidfunk.github.io/mkdocs-material/target=_blankrel=noopener> Material for MkDocs </a></div></div></div></footer></div><scriptsrc=../../assets/javascripts/vendor.93c04032.min.js></script><scriptsrc=../../assets/javascripts/bundle.83e5331e.min.js></script><scriptid=__langtype=application/json>{"clipboard.copy":"Copy to clipboard","clipboard.copied":"Copied to clipboard","search.config.lang":"en","search.config.pipeline":"trimmer, stopWordFilter","search.config.separator":"[\\s\\-]+","search.placeholder":"Search","search.result.placeholder":"Type to start searching","search.result.none":"No matching documents","search.result.one":"1 matching document","search.result.other":"# matching documents","search.result.more.one":"1 more on this page","search.result.more.other":"# more on this page","search.result.term.missing":"Missing"}</script><script>
