🛠️👷‍♂️ WIP - this post is being worked on

Why?

Probably at least once you want to check external or internal requests on your server, and it’s cumbersome to login via SSH and go through multiple access logs. Fortunately, we have simple to deploy tools that can gather logs from containers and show you in a nice way.

flowchart TD

container-1 & container-2 & container-n --> Docker+Loki-docker-driver -- port 3100 --> Loki

Browser -- port 80/443 --> Grafana -- port 3100 --> Loki
Mermaid source 
flowchart TD

container-1 & container-2 & container-n --> Docker+Loki-docker-driver -- port 3100 --> Loki

Browser -- port 80/443 --> Grafana -- port 3100 --> Loki

Server / NAS / Unraid setup

docker plugin install grafana/loki-docker-driver:2.9.1 --alias loki --grant-all-permissions

Using docker compose and fragments in yaml, you can apply loki logging to selected containers in DRY way. Define logging options once and reuse them in other services you want. This gives you more control over logging vs touching global config.

version: '3.8'
services:
  echo:
    container_name: "echo"
    image: k8s.gcr.io/echoserver:1.3
    restart: always
    logging: &loki
      driver: loki
      options:
        loki-url: "http://<SERVER-ADDRESS>:3100/loki/api/v1/push"
  pma:
    image: phpmyadmin/phpmyadmin
    container_name: pma
    logging: *loki

Mac setup

brew install goaccess logcli

Grafana dashboard

There is no useful Grafana dashboard for HTTP servers for simple, one server setup and Loki 2.0 compatible.

https://grafana.com/blog/2020/04/08/loki-quick-tip-how-to-create-a-grafana-dashboard-for-searching-logs-using-loki-and-prometheus/

Loki / Queries

Fetching a log list from Loki from last 30 days for container systemz-writes with limit of 5000 last log lines

logcli query '{source="stdout", container_name="systemz-writes"} |= ``' -o raw --limit 5000 --from $(date --iso-8601=seconds --date '30 days ago')

Fetching a log list from Loki from last 30 days for container systemz-writes and www-lvlup-pro with limit of 5000 last log lines

logcli query '{source="stdout", container_name=~"systemz-writes|www-lvlup-pro"} |= ``' -o raw --limit 5000 --from $(date --iso-8601=seconds --date '30 days ago')

More:

Goaccess + Loki

Let’s add goaccess to parse logs directly from Loki. Fetching a log list from Loki from last 30 days for container systemz-writes with limit of 5000 last log lines and parse it with goaccess:

(not working yet)

logcli query '{source="stdout", container_name="systemz-writes"} |= ``' -o raw --limit 5000 --from $(date --iso-8601=seconds --date '30 days ago') | LANG="en_US.UTF-8" goaccess --log-format='%h %v %^[%d:%t %^] "%r" %s %b "%R" "%u"' --date-format='%d/%b/%Y' --time-format=%s

Goaccess with multiple apache/nginx containers (not working yet)

logcli query '{source="stdout", container_name=~"pma|home|example"} |= ``' --tail -o raw --limit 1000 | LANG="en_US.UTF-8" goaccess --log-format='%h %v %^[%d:%t %^] "%r" %s %b "%R" "%u"' --date-format='%d/%b/%Y' --time-format=%s

Goaccess HAProxy (not working yet)

logcli query '{source="stdout", container_name="haproxy"} |= ``' --tail -o raw --limit 1000 | LANG="en_US.UTF-8" goaccess --log-format='%^]%^ %h:%^ [%d:%t.%^] %^/%^/%^/%^/%L/%^ %s %b %^"%r"' --date-format='%d/%b/%Y' --time-format='%H:%M:%S'
LANG="en_US.UTF-8" goaccess example-log3.txt --log-format='%^[ %^[ %^[%x.%^] %^ %^/%^/%^/%^/%L %^ %h:%^"%r" %s %^ %^ {%v|%^|%^|%^|%u} %^ %^ %^=%b' --date-format=%s --time-format=%s

More

Loki with log parsing

Parsing apache logs

{source="stdout", container_name="pma"} | pattern `<ip> - - <_> "<method> <uri> <_>" <status> <size> <_> "<agent>" <_>`

HAProxy