Bug - upstream (loki) behind rproxy

[Ronan-WeScale]

I tried to deploy observatorium on dev cluster and call loki on other cluster, loki is exposed by nginx ingress.

It seems the api don't set the right Host header on the call, observatorium log :

level=debug name=observatorium ts=2024-08-09T09:26:42.711366336Z caller=proxy.go:61 request=observatorium-api-bbf9b9dd6-5dsqd/uam7XGqFNW-000018 msg="request to upstream" url="https://loki.xxx.com/loki/api/v1/query?direction=backward&query=vector%281%29%2Bvector%281%29&time=4000000000"
level=debug name=observatorium ts=2024-08-09T09:26:42.714032285Z caller=instrumentation.go:36 request=observatorium-api-bbf9b9dd6-5dsqd/uam7XGqFNW-000018 proto=HTTP/1.1 method=GET status=404 content= path=/api/logs/v1/xxx/loki/api/v1/query duration=3.310348ms bytes=146

With tcpdump I can catch the header and see the Host isn't my loki address "loki.xxx.com" but my observatorium url (obs.xxx.com), so it's normal the nginx ingress answer 404.

Host: obs.xxx.com
User-Agent: Grafana/10.4.3
Accept-Encoding: gzip
Authorization: Bearer yyyy
Forwarded: for=10.128.4.27;host=obs.xxx.com;proto=https
X-Datasource-Uid: e9b1dd3f-49c9-4bff-bdec-f53bd2ee7ea9
X-Forwarded-For: 10.128.4.27
X-Forwarded-Host: obs.xxx.com
X-Forwarded-Port: 443
X-Forwarded-Prefix: /api/logs/v1/infra
X-Forwarded-Proto: https
X-Grafana-Org-Id: 1
X-Id-Token: yyyy

As workaround I had bypass my ingress with metallb to forward traffic directly on loki-gateway svc.

[Ronan-WeScale]

Hi, yes I do this to.

[Falltrades]

To confirm this bug, I confirm it is working with a reverse proxy which correct the host header.

Reverse proxy configuration:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: observatorium-proxy-config
data:
  nginx.conf: |
    events {}

    http {
      server {
        listen 8080;

        location /loki/ {
          proxy_pass http://loki.example.com/;
          proxy_set_header Host loki.example.com;
          proxy_ssl_server_name on;
        }

        location /mimir/ {
          proxy_pass http://mimir.example.com/;
          proxy_set_header Host mimir.example.com;
          proxy_ssl_server_name on;
        }
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: observatorium-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: observatorium-proxy
  template:
    metadata:
      labels:
        app: observatorium-proxy
    spec:
      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:1.28-alpine
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
            - name: run
              mountPath: /run
      volumes:
        - name: config
          configMap:
            name: observatorium-proxy-config
        - name: run
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: observatorium-proxy
spec:
  selector:
    app: observatorium-proxy
  ports:
    - port: 8080
      targetPort: 8080

Using these for datasource url:

    "logs": "http://observatorium-proxy.<namespace>.svc.cluster.local:8080/loki",
    "metrics": "http://observatorium-proxy.<namespace>.svc.cluster.local:8080/mimir