Separate third party NGINX configuration (#10470)

* Document container separation

* Separate configurations

docs/enhancements/20231001-split-containers.md

@@ -0,0 +1,110 @@
+# Proposal to split containers
+
+* All the NGINX files should live on one container
+  * No file other than NGINX files should exist on this container
+  * This includes not mounting the service account
+* All the controller files should live on a different container
+  * Controller container should have bare minimum to work (just go program)
+  * ServiceAccount should be mounted just on controller
+
+* Inside nginx container, there should be a really small http listener just able 
+to start, stop and reload NGINX
+
+## Roadmap (what needs to be done)
+* Map what needs to be done to mount the SA just on controller container
+* Map all the required files for NGINX to work
+* Map all the required network calls between controller and NGINX
+  * eg.: Dynamic lua reconfiguration
+* Map problematic features that will need attention
+  * SSLPassthrough today happens on controller process and needs to happen on NGINX
+
+### Ports and endpoints on NGINX container
+* Public HTTP/HTTPs port - 80 and 443
+* Lua configuration port - 10246 (HTTP) and 10247 (Stream)
+* 3333 (temp) - Dataplane controller http server
+  * /reload - (POST) Reloads the configuration.
+    * "config" argument is the location of temporary file that should be used / moved to nginx.conf
+  * /test - (POST) Test the configuration of a given file location
+    * "config" argument is the location of temporary file that should be tested
+
+### Mounting empty SA on controller container
+
+```yaml
+kind: Pod
+apiVersion: v1
+metadata:
+  name: test
+spec:
+  containers:
+  - name: nginx
+    image: nginx:latest
+    ports:
+    - containerPort: 80
+  - name: othernginx
+    image: alpine:latest
+    command: ["/bin/sh"]
+    args: ["-c", "while true; do date; sleep 3; done"]
+    volumeMounts:
+    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
+      name: emptysecret
+  volumes:
+  - name: emptysecret
+    emptyDir:
+      sizeLimit: 1Mi
+```
+
+### Mapped folders on NGINX configuration
+**WARNING** We need to be aware of inter mount containers and inode problems. If we 
+mount a file instead of a directory, it may take time to reflect the file value on 
+the target container
+
+*  "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;;"; - Lua scripts
+* "/var/log/nginx" - NGINX logs
+* "/tmp/nginx (nginx.pid)" - NGINX pid directory / file, fcgi socket, etc
+* " /etc/nginx/geoip" - GeoIP database directory - OK - /etc/ingress-controller/geoip
+* /etc/nginx/mime.types - Mime types
+* /etc/ingress-controller/ssl - SSL directory (fake cert, auth cert)
+* /etc/ingress-controller/auth - Authentication files
+* /etc/nginx/modsecurity - Modsecurity configuration
+* /etc/nginx/owasp-modsecurity-crs - Modsecurity rules
+* /etc/nginx/tickets.key - SSL tickets - OK - /etc/ingress-controller/tickets.key
+* /etc/nginx/opentelemetry.toml - OTEL config - OK - /etc/ingress-controller/telemetry
+* /etc/nginx/opentracing.json - Opentracing config - OK - /etc/ingress-controller/telemetry
+* /etc/nginx/modules - NGINX modules
+* /etc/nginx/fastcgi_params (maybe) - fcgi params
+* /etc/nginx/template - Template, may be used by controller only
+
+##### List of modules
+```
+ngx_http_auth_digest_module.so    ngx_http_modsecurity_module.so
+ngx_http_brotli_filter_module.so  ngx_http_opentracing_module.so
+ngx_http_brotli_static_module.so  ngx_stream_geoip2_module.so
+ngx_http_geoip2_module.so
+```
+
+##### List of files that may be removed
+```
+-rw-r--r--    1 www-data www-data      1077 Jun 23 19:44 fastcgi.conf
+-rw-r--r--    1 www-data www-data      1077 Jun 23 19:44 fastcgi.conf.default
+-rw-r--r--    1 www-data www-data      1007 Jun 23 19:44 fastcgi_params
+-rw-r--r--    1 www-data www-data      1007 Jun 23 19:44 fastcgi_params.default
+drwxr-xr-x    2 www-data www-data      4096 Jun 23 19:34 geoip
+-rw-r--r--    1 www-data www-data      2837 Jun 23 19:44 koi-utf
+-rw-r--r--    1 www-data www-data      2223 Jun 23 19:44 koi-win
+drwxr-xr-x    6 www-data www-data      4096 Sep 19 14:13 lua
+-rw-r--r--    1 www-data www-data      5349 Jun 23 19:44 mime.types
+-rw-r--r--    1 www-data www-data      5349 Jun 23 19:44 mime.types.default
+drwxr-xr-x    2 www-data www-data      4096 Jun 23 19:44 modsecurity
+drwxr-xr-x    2 www-data www-data      4096 Jun 23 19:44 modules
+-rw-r--r--    1 www-data www-data     18275 Oct  1 21:28 nginx.conf
+-rw-r--r--    1 www-data www-data      2656 Jun 23 19:44 nginx.conf.default
+-rwx------    1 www-data www-data       420 Oct  1 21:28 opentelemetry.toml
+-rw-r--r--    1 www-data www-data         2 Oct  1 21:28 opentracing.json
+drwxr-xr-x    7 www-data www-data      4096 Jun 23 19:44 owasp-modsecurity-crs
+-rw-r--r--    1 www-data www-data       636 Jun 23 19:44 scgi_params
+-rw-r--r--    1 www-data www-data       636 Jun 23 19:44 scgi_params.default
+drwxr-xr-x    2 www-data www-data      4096 Sep 19 14:13 template
+-rw-r--r--    1 www-data www-data       664 Jun 23 19:44 uwsgi_params
+-rw-r--r--    1 www-data www-data       664 Jun 23 19:44 uwsgi_params.default
+-rw-r--r--    1 www-data www-data      3610 Jun 23 19:44 win-utf
+```

docs/user-guide/nginx-configuration/configmap.md

@@ -164,7 +164,7 @@ The following table shows a configuration option's name, type, and the default v
 |[enable-opentelemetry](#enable-opentelemetry)|bool|"false"||
 |[opentelemetry-trust-incoming-span](#opentelemetry-trust-incoming-span)|bool|"true"||
 |[opentelemetry-operation-name](#opentelemetry-operation-name)|string|""||
-|[opentelemetry-config](#/etc/nginx/opentelemetry.toml)|string|"/etc/nginx/opentelemetry.toml"||
+|[opentelemetry-config](#/etc/ingress-controller/telemetry/opentelemetry.toml)|string|"/etc/ingress-controller/telemetry/opentelemetry.toml"||
 |[otlp-collector-host](#otlp-collector-host)|string|""||
 |[otlp-collector-port](#otlp-collector-port)|int|4317||
 |[otel-max-queuesize](#otel-max-queuesize)|int|||

docs/user-guide/third-party-addons/opentelemetry.md

@@ -165,7 +165,7 @@ To install the example and collectors run:
       kind: ConfigMap
       data:
         enable-opentelemetry: "true"
-        opentelemetry-config: "/etc/nginx/opentelemetry.toml"
+        opentelemetry-config: "/etc/ingress-controller/telemetry/opentelemetry.toml"
         opentelemetry-operation-name: "HTTP $request_method $service_name $uri"
         opentelemetry-trust-incoming-span: "true"
         otlp-collector-host: "otel-coll-collector.otel.svc"