Skip to content
This repository was archived by the owner on Apr 26, 2024. It is now read-only.

Change the default log config to reduce disk I/O and storage #8040

Merged
merged 11 commits into from
Aug 11, 2020
Merged

Change the default log config to reduce disk I/O and storage #8040

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1af663d
Change default log config to buffer by default.
erikjohnston Aug 6, 2020
29b921e
Default to keeping last 4 days of logs.
erikjohnston Aug 6, 2020
ed8a183
Newsfile
erikjohnston Aug 6, 2020
07465f0
Remove spurious level: DEBUG
erikjohnston Aug 6, 2020
5936b1a
Review comments
erikjohnston Aug 7, 2020
c2fc81e
Move setting of Filter into code.
erikjohnston Aug 7, 2020
5590a73
Flush at WARNING logs as well
erikjohnston Aug 7, 2020
dd0b6f2
Revert "Move setting of Filter into code."
erikjohnston Aug 7, 2020
e7d2a9e
Reduce default buffer capacity and doc the option
erikjohnston Aug 7, 2020
62289fa
Document console handler
erikjohnston Aug 10, 2020
538efd0
Update wording as per review
erikjohnston Aug 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/8040.misc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Change the default log config to reduce disk I/O and storage for new servers.
19 changes: 15 additions & 4 deletions docs/sample_log_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,24 @@ filters:

handlers:
file:
class: logging.handlers.RotatingFileHandler
class: logging.handlers.TimedRotatingFileHandler
formatter: precise
filename: /var/log/matrix-synapse/homeserver.log
maxBytes: 104857600
backupCount: 10
when: midnight
backupCount: 3 # Does not include the current log file.
filters: [context]
encoding: utf8

# Default to buffering writes to log file for efficiency. This means that
# will be a delay for WARNING/INFO/DEBUG logs to get written, but ERROR
# logs will still be flushed immediately.
buffer:
class: logging.handlers.MemoryHandler
formatter: precise
level: DEBUG
target: file
capacity: 100
Copy link
Member Author

@erikjohnston erikjohnston Aug 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kind of wonder if 100 is a bit extreme here for a new server?

Copy link
Member

@richvdh richvdh Aug 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as in, too high? maybe a bit.

probably worth adding a comment here saying what the number means, so people can tweak it if they want.

Copy link
Member Author

@erikjohnston erikjohnston Aug 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, even jki.re logs at less than 30 lines a second so a brand new server would probably not actually flush very often at all, which I can imagine will cause confusion. I'm somewhat tempted to lower it to something like ten, which would still be a huge improvement.


console:
class: logging.StreamHandler
formatter: precise
Expand All @@ -38,6 +49,6 @@ loggers:

root:
level: INFO
handlers: [file, console]
handlers: [buffer, console]

disable_existing_loggers: false
19 changes: 15 additions & 4 deletions synapse/config/logger.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,24 @@

handlers:
file:
class: logging.handlers.RotatingFileHandler
class: logging.handlers.TimedRotatingFileHandler
formatter: precise
filename: ${log_file}
maxBytes: 104857600
backupCount: 10
when: midnight
backupCount: 3 # Does not include the current log file.
filters: [context]
encoding: utf8

# Default to buffering writes to log file for efficiency. This means that
# will be a delay for WARNING/INFO/DEBUG logs to get written, but ERROR
# logs will still be flushed immediately.
buffer:
class: logging.handlers.MemoryHandler
formatter: precise
level: DEBUG
target: file
capacity: 100

console:
class: logging.StreamHandler
formatter: precise
Expand All @@ -82,7 +93,7 @@

root:
level: INFO
handlers: [file, console]
handlers: [buffer, console]

disable_existing_loggers: false
"""
Expand Down