101: logging, which logs exist, how to access and understand

đź“‘ How to
, ,
1

related components (LAMP)

Nextcloud traditionally runs as “LAMP” stack: Linux, Apache, Mysql, PHP. Today other components might replace the known products like Nginx as webserver of Postgresql as database but one thing remains common - each component has it’s own configs and logging tools/ways which might be important (especially for more generic problems not related to the application itself)

brief Nextcloud logging overview

Nextcloud application itself has 2 different logging parts:

  • “application log” generic application logging logging - nextcloud.log
    • by default lives in the root of the data directory
    • lists most actions of the system, errors
  • “admit_audit” access and change log auditlog - audit.log
    • by default lives in the root of the data directory
    • log content-related action e.g. creation, removal, access to specific files

review official logging docs for all details.

other components log according to their own habits like Linux journal and different log files.

default logging locations for some relevant applications
component default log location comments
Apache /var/log/apache2/access.log Access logs (requests received)
Apache /var/log/apache2/error.log Error logs
Nginx /var/log/nginx/access.log Access logs
Nginx /var/log/nginx/error.log Error logs
postgres /var/log/postgresql/postgresql-<version>-main.log Main server log
MariaDB /var/log/mariadb/mariadb.log Main server log
default Linux /var/log/.. generic Linux log location
debian 12 systemd-journald command journalctl command e.g. journalctl --since "1 hour ago" or journalctl --tail
Docker /var/lib/docker/containers/<container_id>/<container_id>-json.log Docker maintains a JSON log file for each container, which includes all output from the container
Docker docker logs command STDOUT of the container
Docker Compose docker compose logs command STDOUT of all containers running in specific compose definition or specific container (with {service name})
apt packet manager /var/log/apt/history.log

depending on the format of the log different tool might be required to review and filter the logs: tail, grep, journalctl.
Know about logrotate functionality which might prevent you from accessing older data.

depending on the installation variant logs might recide in different places, below some additional docs, feel free to add another logs

snap

How to manage Nextcloud snap logs
Nextcloud snap Wiki - managing logs

NCP

List of Logs on NCP

Docker (community image)
  • MariaDB and Postgres and Apache logs are written to container STDOUT by default, use docker logs {container name} to review them (docker logs command knows -n parameter to view only last N lines e.g. “docker logs -n 20 nextcloud-app” or “docker compose logs -n 20 app”)

knowing the logs

The 2 Nextcloud logs nextcloud.log and audit.log mentioned above are somewhat special as they don’t follow common log format but write JSON objects. This has advantages as machines can easily parse such logs but at the same time it’s really hard to read JSON formatted as a human. I would recommend you install and use the jq utility to filter and read the logs.

Here are examples how to extract most important data from nextcloud.log:

extract most important fields from nextcloud.log

tail -n 500 /path/to/your/nextcloud.log|jq '{time: .time, remoteIp: .remoteAddr, userAgent: .userAgent, url: .url, message: .message}'
  • tail reads last 500 lines of the log
  • jq extracts time, ip, userAgent, url and message
    with this command we skip many data especially the exception field which is the most verbose and detailed part. it depends on the issue if you need more details… I prefer to get a high-level view first before I dig deep into the problem and for the first look too many details sometimes hide the real problem and limited, filtered “summary” makes it easier to isolate the root cause.

extract specific app logs from nextcloud.log

tail -n 500 /path/to/your/nextcloud.log|jq -r 'select(."app"=="federation")|{time: .time, remoteIp: .remoteAddr, userAgent: .userAgent, url: .url, message: .message}'
  • you can also use jq to filter the logs on specific conditions like an app you want to troubleshoot
  • same as before but only shows logs from the fedration app

extract specific request from nextcloud.log

jq 'select(.reqId == "gnxegghbjQgdEF3AcUEt")' /path/to/your/nextcloud.log

understanding the log

There is no general advice how get a clue of the logs. Each system has it’s own terminology used in the logs. often the message text is clear enough to understand the problem… in other cases internet search using the “native log line” leads you to forum discussions or Github issues related to similar problems. When I refer to a “native” log message I mean the constant message block without variable data parts like timestamp, hostnames, user and file names - so in case you are looking into a problem with federation and see a message

  "message": "error while sending notification for federated share: Client error: `POST https://cloud.mydomain.tld/ocm/notifications` resulted in a `403 Forbidden` response:\n{\"message\":\"RESOURCE_NOT_FOUND\"}\n"

perform an internet search for the terms (quote them as shown) omiting the hostname. In case there are error ID or constant identifiers like above 403 Forbidden and RESOURCE_NOT_FOUND they are often easier to search rather than some free text message (which are translated sometimes):

"error while sending notification for federated share" "Client error" "POST" "RESOURCE_NOT_FOUND" "403 Forbidden"

I would recommend starting with the most specific and complete message and continue removing redundant parts in case you don’t find enough relevant topics.

conclusion

you never know which log part are most relevant - try some of them… after some time you will learn where the relevant parts are. it needs training and practice but once you start you will learn a lot about you system, find new interesting troubleshooting methods and teach yourself to understand and fix issues DYI :clap:

3 Likes