Traefik: Logrotate Large (Access) Log Files

Malte Hellmeier -

Traefik is an excellent reverse proxy for Linux servers, even supporting Docker. When hosting public websites, tools like CrowdSec help to secure the infrastructure from malicious attackers. However, if care is not taken, logs can often consume a significant portion of the hard disk. We will present a solution to this issue.

The Problem

Traefik has two log files: The traefik.log and the access.log file. When writing everything in those files by choosing a log level like TRACE, DEBUG or INFO, those files can get increasingly large and fill a significant majority of the server's disk space. But how can we prevent those big log files? Just changing the log level to ERROR or FATAL might work, but it isn't a good idea when running tools like CrowdSec that help to secure the infrastructure from malicious attackers. The answer to the problem is easy: Logrotate!

Enable Lograte

Rotation of log files means automatically creating a new log file when a criterion is met (such as time or file size) and storing a defined number of old log files. To enable it, there are two possibilities: Changing Traefik's configuration or using the system's logrotate.

Rotate traefik.log

Rotating the traefik.log file is easy and pretty straightforward. Based on the official documentation, the four flags maxSize, maxBackups, maxAge, and compress can be configured to personal needs. Search for your traefik.yml file and open it in your favorite editor. When using Traefik in a Docker container, make sure the configuration file is mounted to your system. Add or change the log part in your config to the following or something similar (YAML version):

log:
  level: "INFO"
  filePath: "/var/log/traefik/traefik.log"
  maxSize: 100
  maxBackups: 5

In this example, the log file located at /var/log/traefik/traefik.log will be rotated if it exceeds the maximum size of 100 MB, while the system keeps a maximum of five log files. If a sixth file were created, the oldest would be deleted.

Rotate access.log

Unfortunately, the previous configuration options like maxSize or maxBackups are not available for the access log file. Based on the official documentation, an external USR1 signal trigger is needed to rotate the logfile. Thus, the accessLog part of the traefik.yml configuration file looks as follows:

accessLog:
  filePath: "/var/log/traefik/access.log"
  bufferingSize: 100

We use the system package logrotate for it, described in the following for an Ubuntu server system. First, create a file /etc/logrotate.d/traefik and open it in your favorite editor with admin permissions. We will add the following configuration for a Docker setup with CrowdSec:

/var/log/crowdsec/traefik/access.log {
  weekly
  rotate 13
  missingok
  notifempty
  dateext
  dateformat .%Y-%m-%d
  create 0644 root root
  postrotate
    docker kill --signal="USR1" traefik
  endscript
}

Remember to change the first line to your log location. This may differ from the previous configuration because logrotate is executed on your host system, while the configuration uses the file structure of your Docker containers. Based on the provided example configuration, it will rotate the logs every week (weekly) and keep 13 log files (rotate 13), which is approximately one-quarter of a year / 90 days. When a size-based log rotation is preferred, use size 100M instead. There won't be errors if the log file is missing (missingok) and the rotation will skip when the log file is empty (notifempty). We will use a specific date-based format (dateext and dateformat). Every new log file is created with 0644 permissions, owned by the root user and group (create 0644 root root). Since we are running Traefik inside Docker, we will send the suggested USR1 signal to the traefik container. Remember to update this line to match your setup, e.g., if you are not using Docker or if the Traefik container has a different name. For non-Docker users, the line must be changed to kill -USR1 `pgrep traefik`.

After creating and saving the file, test it with sudo logrotate --debug /etc/logrotate.d/traefik. If this works, run it once with sudo logrotate /etc/logrotate.d/traefik and manually check and verify if it really works.

Afterward, everything should work as expected. If you want to be sure, restart Traefik or your system to activate everything.

References