The container ships with vector, a light-weight log collector. Vector is used to collect and normalize logs from all of the services and tasks in the container so they can be managed in a similar way.
Logging Output Methods
Docker Logs
The default behaviour is to send all logs to stdout which can then be consumed like normal docker logs.
The exception is when running with exec
or bash
run modes (which are usually only used for debugging or one-off commands), the default is to write logs to disk inside the container at /var/log/docker-logs
so as to keep your stdout clean. If you want to export logs then you can override this default behaviour by mounting a log directory as described below.
Mounted Logs Directory
If you mount /deskpro/logs
, then logs are written to disk in this directory with files named $app/$app.log
or $app/$app.$chan.log
.
Logs in this directory can get very large. If you mount this directory, you will want to make sure that the files are rotated regularly. Log rotation on exported logs is NOT performed by the container; this is something you would have to configure on your host.
If you need to change the output directory for whatever reason, you can specify LOGS_EXPORT_DIR=/some/other/dir
to change the output path.
File Permissions
The logs that get written to disk are the normalized logs collected by vector, not the literal log files being written by each service. That means you don't need to worry about file permissions or ownership. The logs written by vector will be written by the root
user.
If you want logs to be owned by a particular group, then you can make your host logs directory owned by that group and set the sticky bit:
# e.g. on the host
mkdir container-logs
chown root:mygroup container-logs
# set the sticky bit so any files written to this directory will be owned by mygroup
chmod g+s container-logs
# then run the container with that directory mounted copy
Logging Format
Logs are normalized into the simple logfmt
format with these common properties:
ts
is a UTC timestamp in ISO 8601 formatapp
contains the application or service or source of the log linechan
is a particular "channel" (one app may have many different types of logs; we separate them into so-called channels)lvl
contains the message severity level (e.g.INFO
orERROR
)msg
contains the log messageMany logs define other properties such as context information. Scalar values are printed as-is; more complex values may be encoded as JSON strings.
With logfmt, each log entry is a single line. For example:
ts=2023-06-27T16:04:12Z app=entrypoint lvl=DEBUG msg="Cleaning env vars"
ts=2023-06-27T16:04:12Z app=entrypoint lvl=INFO msg="Command: web, Mode: web, Exec: supervisord, Services: nginx php_fpm"
...
ts=2023-06-27T16:04:15Z app=supervisord lvl=INFO msg="success: php_fpm entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)"
ts=2023-06-27T16:04:16Z app=supervisord lvl=INFO msg="success: nginx entered RUNNING state, process has stayed up for > than 3 seconds (startsecs)"
...
ts=2023-06-27T16:04:25Z app=nginx chan=error lvl=INFO msg="client 127.0.0.1 closed keepalive connection" cid=1 pid=192 tid=192
ts=2023-06-27T16:04:29Z app=nginx chan=access lvl=INFO method=GET path=/ status=200 host=localhost scheme=http proto=HTTP/1.1 body_bytes_sent=0 remote_addr=127.0.0.1 request_id=6ddd26905fdaa89f8cde604ba32c305c request_length=78 request_time=0.000 user_agent=curl/8.1.2
...
ts=2023-06-27T16:06:46Z app=php chan=error lvl=WARNING msg="PHP Warning: Undefined variable $foo in Command line code on line 1" copy
In some cases, log lines can get very very long (e.g. an application error with a complicated backtrace). You can use a utility like lnav to make reading and filtering output easier.
Here are the most common logs:
app=entrypoint
-- Emitted during boot (docker entrypoint)app=supervisord
-- supervisord logs (logs to do with starting and stopping services) and the raw stdout of processesapp=nginx
-- Logs for the internal HTTP serverchan=error
-- error logschan=access
-- access logs
app=php
chan=error
-- PHP error logs
app=php_fpm
chan=error
-- php-fpm error log
app=deskpro
andapp=deskpro-dpv5
-- Deskpro application logs. There are a number of channels but notably:chan=general
-- General app logschan=migrations
-- Migration logschan=errors
-- Deskpro error log (some overlap with the PHP error log)
app=deskpro-tasks
-- Output from running Deskpro's various tasks. These can be very noisy and should be rotated frequently.
JSON Output
If you want to output as JSON instead of logfmt, set LOGS_OUTPUT_FORMAT=json
When outputting to JSON, all the same properties are the same, the only difference is that the line will be JSON encoded instead of logfmt.
Log Levels
You can change logging levels by setting env vars:
DESKPRO_LOG_LEVEL
: The default log level for most application logs. Defaults towarning
DESKPRO_LOG_LEVEL_EMAIL_COLLECTION
: Log level for email collection. Defaults towarning
DESKPRO_LOG_LEVEL_EMAIL_PROCESSING
: Log level for email processing. Defaults towarning
PHP_FPM_LOG_LEVEL
-- The value for php-fpm'slog_level
. Default:warning
NGINX_ERROR_LOG_LEVEL
-- The logging level used by the internal web server. Default:info
Boot Log
The boot log (app=entrypoint
) is slightly special because these messages may only be readable from stderr in the event of a problem (i.e. log collection via vector might not exist if vector can't even be started).
These logs are always written to /var/log/docker-boot.log
inside the container. If you have enabled log exporting to a mounted volume, then a log file by the same name will be available in that mounted directory as well.
There are two variables that determine the logging level used for output to stderr:
BOOT_LOG_LEVEL
(default:INFO
) -- This applies when you are running the container in a service mode (e.g. NOT exec or bash). It determines the min log level for messages before they get sent to stderr.BOOT_LOG_LEVEL_EXEC
(default:WARNING
) Is the same, except it applies to "exec" or "bash" run modes. In these modes, you are often running commands manually so you don't want to see noisy verbose INFO messages that happen during container start.
Advanced: Custom Log Shipping
You can provide your own vector configuration by mounting a custom config file to /deskpro/config/vector.d/99-my-output.toml
There are two useful inputs
you can use:
all
will be all log files with a normalized.logprops
structure that includes standard log properties (e.g.app
,chan
,lvl
,msg
, etc.) as well as any 'extra' properties that may be defined by the log file. You can use this as the basis of any other custom transforms or filters.all_formatted
will define a.message
property that is a pre-formatted string (either logfmt or JSON depending on theLOGS_OUTPUT_FORMAT
env var). E.g. you can use this with sinks that haveencoding.codec=raw_message
support.
We suggest you avoid using any other inputs (such as specific log files) as they may change in future versions. Instead, if you wish to handle specific kinds of logs, use the all
input and then filter it by .app
, .chan
, or .lvl
to get just the logs you want.
Please log in or register to submit a comment.