--> <h1id=troubleshooting>Troubleshooting<aclass=headerlinkhref=#troubleshootingtitle="Permanent link"> ¶</a></h1><h2id=ingress-controller-logs-and-events>Ingress-Controller Logs and Events<aclass=headerlinkhref=#ingress-controller-logs-and-eventstitle="Permanent link"> ¶</a></h2><p>There are many ways to troubleshoot the ingress-controller. The following are basic troubleshooting methods to obtain more information.</p><p>Check the Ingress Resource Events</p><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> kubectl get ing -n <namespace-of-ingress-resource>
<spanclass=go> Normal CREATE 1m nginx-ingress-controller Ingress default/cafe-ingress</span>
<spanclass=go> Normal UPDATE 58s nginx-ingress-controller Ingress default/cafe-ingress</span>
</code></pre></div><p>Check the Ingress Controller Logs</p><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> kubectl get pods -n <namespace-of-ingress-controller>
<spanclass=go>NAME READY STATUS RESTARTS AGE</span>
</code></pre></div><p>Check the Nginx Configuration</p><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> kubectl get pods -n <namespace-of-ingress-controller>
<spanclass=go>NAME READY STATUS RESTARTS AGE</span>
</code></pre></div><p>Check if used Services Exist</p><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> kubectl get svc --all-namespaces
<spanclass=go>NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE</span>
</code></pre></div><h2id=debug-logging>Debug Logging<aclass=headerlinkhref=#debug-loggingtitle="Permanent link"> ¶</a></h2><p>Using the flag <code>--v=XX</code> it is possible to increase the level of logging. This is performed by editing the deployment.</p><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> kubectl get deploy -n <namespace-of-ingress-controller>
<spanclass=go>NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE</span>
<spanclass=gp>#</span> Add --v<spanclass=o>=</span>X to <spanclass=s2>"- args"</span>, where X is an integer
</code></pre></div><ul><li><code>--v=2</code> shows details using <code>diff</code> about the changes in the configuration in nginx</li><li><code>--v=3</code> shows details about the service, Ingress rule, endpoint changes and it dumps the nginx configuration in JSON format</li><li><code>--v=5</code> configures NGINX in <ahref=http://nginx.org/en/docs/debugging_log.html>debug mode</a></li></ul><h2id=authentication-to-the-kubernetes-api-server>Authentication to the Kubernetes API Server<aclass=headerlinkhref=#authentication-to-the-kubernetes-api-servertitle="Permanent link"> ¶</a></h2><p>A number of components are involved in the authentication process and the first step is to narrow down the source of the problem, namely whether it is a problem with service authentication or with the kubeconfig file.</p><p>Both authentications must work:</p><divclass=highlight><pre><span></span><code>+-------------+ service +------------+
</code></pre></div><p><strong>Service authentication</strong></p><p>The Ingress controller needs information from apiserver. Therefore, authentication is required, which can be achieved in two different ways:</p><ol><li><p><em>Service Account:</em> This is recommended, because nothing has to be configured. The Ingress controller will use information provided by the system to communicate with the API server. See 'Service Account' section for details.</p></li><li><p><em>Kubeconfig file:</em> In some Kubernetes environments service accounts are not available. In this case a manual configuration is required. The Ingress controller binary can be started with the <code>--kubeconfig</code> flag. The value of the flag is a path to a file specifying how to connect to the API server. Using the <code>--kubeconfig</code> does not requires the flag <code>--apiserver-host</code>. The format of the file is identical to <code>~/.kube/config</code> which is used by kubectl to connect to the API server. See 'kubeconfig' section for details.</p></li><li><p><em>Using the flag <code>--apiserver-host</code>:</em> Using this flag <code>--apiserver-host=http://localhost:8080</code> it is possible to specify an unsecured API server or reach a remote kubernetes cluster using <ahref=https://kubernetes.io/docs/user-guide/kubectl/kubectl_proxy/>kubectl proxy</a>. Please do not use this approach in production.</p></li></ol><p>In the diagram below you can see the full authentication flow with all options, starting with the browser on the lower left hand side.</p><divclass=highlight><pre><span></span><code>Kubernetes Workstation
</code></pre></div><h3id=service-account>Service Account<aclass=headerlinkhref=#service-accounttitle="Permanent link"> ¶</a></h3><p>If using a service account to connect to the API server, the ingress-controller expects the file <code>/var/run/secrets/kubernetes.io/serviceaccount/token</code> to be present. It provides a secret token that is required to authenticate with the API server.</p><p>Verify with the following commands:</p><divclass=highlight><pre><span></span><code><spanclass=gp>#</span> start a container that contains curl
<spanclass=gp>#</span> when you <spanclass=nb>type</span><spanclass=sb>`</span><spanclass=nb>exit</span><spanclass=sb>`</span> or <spanclass=sb>`</span>^D<spanclass=sb>`</span> the <spanclass=nb>test</span> pod will be deleted.
</code></pre></div><p>If it is not working, there are two possible reasons:</p><ol><li><p>The contents of the tokens are invalid. Find the secret name with <code>kubectl get secrets | grep service-account</code> and delete it with <code>kubectl delete secret <name></code>. It will automatically be recreated.</p></li><li><p>You have a non-standard Kubernetes installation and the file containing the token may not be present. The API server will mount a volume containing this file, but only if the API server is configured to use the ServiceAccount admission controller. If you experience this error, verify that your API server is using the ServiceAccount admission controller. If you are configuring the API server by hand, you can set this with the <code>--admission-control</code> parameter.</p><blockquote><p>Note that you should use other admission controllers as well. Before configuring this option, you should read about admission controllers.</p></blockquote></li></ol><p>More information:</p><ul><li><ahref=http://kubernetes.io/docs/user-guide/service-accounts/>User Guide: Service Accounts</a></li><li><ahref=http://kubernetes.io/docs/admin/service-accounts-admin/>Cluster Administrator Guide: Managing Service Accounts</a></li></ul><h2id=kube-config>Kube-Config<aclass=headerlinkhref=#kube-configtitle="Permanent link"> ¶</a></h2><p>If you want to use a kubeconfig file for authentication, follow the <ahref=../deploy/>deploy procedure</a> and add the flag <code>--kubeconfig=/etc/kubernetes/kubeconfig.yaml</code> to the args section of the deployment.</p><h2id=using-gdb-with-nginx>Using GDB with Nginx<aclass=headerlinkhref=#using-gdb-with-nginxtitle="Permanent link"> ¶</a></h2><p><ahref=https://www.gnu.org/software/gdb/>Gdb</a> can be used to with nginx to perform a configuration dump. This allows us to see which configuration is being used, as well as older configurations.</p><p>Note: The below is based on the nginx <ahref=https://docs.nginx.com/nginx/admin-guide/monitoring/debugging/#dumping-nginx-configuration-from-a-running-process>documentation</a>.</p><ol><li>SSH into the worker</li></ol><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> ssh user@workerIP
<spanclass=go>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</span>
<spanclass=go>d9e1d243156a quay.io/kubernetes-ingress-controller/nginx-ingress-controller "/usr/bin/dumb-init …" 19 minutes ago Up 19 minutes k8s_nginx-ingress-controller_nginx-ingress-controller-67956bf89d-mqxzt_kube-system_079f31ec-aa37-11e8-ad39-080027a227db_0</span>
</code></pre></div><ol><li>Exec into the container</li></ol><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> docker <spanclass=nb>exec</span> -it --user<spanclass=o>=</span><spanclass=m>0</span> --privileged d9e1d243156a bash
</code></pre></div><ol><li>Make sure nginx is running in <code>--with-debug</code></li></ol><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> nginx -V <spanclass=m>2</span>><spanclass=p>&</span><spanclass=m>1</span><spanclass=p>|</span> grep -- <spanclass=s1>'--with-debug'</span>
</code></pre></div><ol><li>Get list of processes running on container</li></ol><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> ps -ef
<spanclass=go>UID PID PPID C STIME TTY TIME CMD</span>
</code></pre></div><ol><li>Attach gdb to the nginx master process</li></ol><divclass=highlight><pre><span></span><code><spanclass=gp>$</span> gdb -p <spanclass=m>21</span>
<spanclass=go>....</span>
<spanclass=go>Attaching to process 21</span>
<spanclass=go>Reading symbols from /usr/sbin/nginx...done.</span>
</code></pre></div><ol><li>Copy and paste the following:</li></ol><divclass=highlight><pre><span></span><code><spanclass=go>set $cd = ngx_cycle->config_dump</span>
</code></pre></div><ol><li><p>Quit GDB by pressing CTRL+D</p></li><li><p>Open nginx_conf.txt</p></li></ol><divclass=highlight><pre><span></span><code><spanclass=go>cat nginx_conf.txt</span>
</code></pre></div></article></div></div></main><footerclass=md-footer><divclass=md-footer-nav><navclass="md-footer-nav__inner md-grid"aria-label=Footer><ahref=../how-it-works/class="md-footer-nav__link md-footer-nav__link--prev"rel=prev><divclass="md-footer-nav__button md-icon"><svgxmlns=http://www.w3.org/2000/svgviewbox="0 0 24 24"><pathd="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12z"/></svg></div><divclass=md-footer-nav__title><divclass=md-ellipsis><spanclass=md-footer-nav__direction> Previous </span> How it works </div></div></a><ahref=../kubectl-plugin/class="md-footer-nav__link md-footer-nav__link--next"rel=next><divclass=md-footer-nav__title><divclass=md-ellipsis><spanclass=md-footer-nav__direction> Next </span> kubectl plugin </div></div><divclass="md-footer-nav__button md-icon"><svgxmlns=http://www.w3.org/2000/svgviewbox="0 0 24 24"><pathd="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11H4z"/></svg></div></a></nav></div><divclass="md-footer-meta md-typeset"><divclass="md-footer-meta__inner md-grid"><divclass=md-footer-copyright> Made with <ahref=https://squidfunk.github.io/mkdocs-material/target=_blankrel=noopener> Material for MkDocs </a></div></div></div></footer></div><scriptsrc=../assets/javascripts/vendor.93c04032.min.js></script><scriptsrc=../assets/javascripts/bundle.83e5331e.min.js></script><scriptid=__langtype=application/json>{"clipboard.copy":"Copy to clipboard","clipboard.copied":"Copied to clipboard","search.config.lang":"en","search.config.pipeline":"trimmer, stopWordFilter","search.config.separator":"[\\s\\-]+","search.placeholder":"Search","search.result.placeholder":"Type to start searching","search.result.none":"No matching documents","search.result.one":"1 matching document","search.result.other":"# matching documents","search.result.more.one":"1 more on this page","search.result.more.other":"# more on this page","search.result.term.missing":"Missing"}</script><script>
