Make VM journal volatile by default

[emanruse]

The problem you're addressing (if any)

Considering the way Qubes OS works, journal persistence in client qubes makes no sense, as the volatile volume is volatile anyway. Tracking potential issues can still be done by observing the current boot journal, or explicitly, temporarily enabling journal persistence (in template or using bind-dirs). This is an advanced administrative task anyway, so it is not supposed to be done by users on a regular basis, i.e. won't hurt usability. Other distros (e.g. openSUSE) have volatile journal by default too.

The solution you'd like

In the template:

sed -ri 's/#Storage=.*/Storage=volatile/g' /etc/systemd/journald.conf

Reboot the template, then remove the old persistent journal from /var/log/journal.

The value to a user, and who that user might be

Moving journal to tmpfs would reduce unnecessary SSD writes.

[adrelanos]

Implementation detail if implementing this:

• don't use sed
• drop appropriate snippet in /usr/lib/systemd folder somewhere

rationale:
https://www.kicksecure.com/wiki/Dev/About_Debian_Packaging#Modifying_Default_Configuration_of_Third_Party_Packages

[emanruse]

The oneliner was just an example of what the result should be and how to do it easily. Otherwise editing the conf file is the right thing to do. I don't understand the relation to /usr/lib/systemd. Making journal volatile does not require a systemd unit.

[adrelanos]

Derivative distributions are not supposed to write into /etc if avoidable. That's for system administrators. They're even less so recommended to write directly into a configuration file owned by a different package as this can cause issues when the file is upgraded by either original distribution or derivative distribution. Also shows up under debsums -ce.

The appropriate journal configuration drop-in folder is:
/usr/lib/systemd/journald.conf.d/*.conf

https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html

systemd has always been good in providing drop-in configuration folders to make customization simple.

I can almost guarantee if Qubes implements this, a drop-in will be used. That's why Qubes It makes no sense to debate that. That's what Qubes always did.

find /usr/lib/systemd | grep qubes

So if you want to send a pull request, I would suggest /usr/lib/systemd/journald.conf.d/30_qubes.conf as this seems having a good chance as it matches the style of the many other existing systemd related drop-in config snippets. Talking about the file name only. I don't know yet if Qubes wants to implement this.

[emanruse]

Derivative distributions are not supposed to write into `/etc` if avoidable.

I don't know if Debian and Fedora templates are considered derivative distributions but Qubes OS already writes custom configuration stuff in /etc. Changing one line in a conffile should not be a problem. It would solve a problem.

[marmarek]

No, /etc is for user changes, packages should avoid modifying config files there of other packages if possible. Anyway, if this change is to be made, it's clear how it should be done (via a drop-in file in /usr). What isn't clear, is whether it's a good idea to change it at all. A better solution might be making journal survive qube restart, instead of breaking it even further.

[emanruse]

No, /etc is for user changes, packages should avoid modifying config files of other packages if possible.

It is possible but the question is whether it is justified in the particular context/case.

Anyway, _if_ this change is to be made, it's clear how it should be done (via a drop-in file in /usr).

From FHS: "/usr/etc Theoretically, that's another directory for configuration files. Virtually unused now." Is that what you mean?

What isn't clear, is whether it's a good idea to change it at all. A better solution might be making journal survive qube restart, instead of breaking it even further.

Considering Qubes is a desktop OS: - What would log persistence improve? - How many users actually look at logs? - How many users need to backup tons of logs (stored in each qube)? Will there be a way to exclude logs from backups? - Considering also that, so far, logs have always been volatile in AppVMs, is there any negative experience from that? How many actual complaints are there? Any surveys at all? - Considering it is easy to bind-dir /run/log/journal, in case this is really needed, is persistence by default actually necessary? For what particular purpose? - Considering running many VMs means lots of logging, which of the 2 possibilities will be better for the default overall performance? Maybe answering these questions will make it easier to decide.

[marmarek]

Is that what you mean?

No, see earlier comment by @adrelanos

[adrelanos]

Perhaps journal should be persistent or non-persistent as per Qubes respecting upstream distribution culture, FAQ What is Qubes’ attitude toward changing guest distros?. If Debian has persistent journal by default, Qubes Debian would also have it. Respectively depending on what Fedora does etc.

---

minor: https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html unfortunately does not seem to support ConditionPathExists which then could be used to set a different config if file /run/qubes/this-is-appvm exists. (That status file is already implemented for years.)

• What would log persistence improve?

Log analysis in case of issues without complex instructions "1) make log persistent in Template, 2) reboot App Qube, 3) reproduce the issue, 4) check the log".

• How many users actually look at logs?

Only advanced users and developers. I am a developer but I am also a user. By putting more obstacles into my way, the less productive I get.

For example, if a VM is slow to shut down or crashing, I need to look at the journal of the previous boot. (sudo journalctl -b -1)

But if extra steps are required then that takes extra time and extra mental load. Hence lowering productivity. Therefore, I am all for to make things comfortable for advanced users and developers to keep the ability to easily debug the system.

Will there be a way to exclude logs from backups?

I guess not. That seems complex to implement to have Qubes backup parse the contents of a VM.

• Considering running many VMs means lots of logging, which of the 2 possibilities will be better for the default overall performance?

Performance will be negligible / non-issue / same as now because now all VMs are logging just that it gets lost after App Qube gets shut down.

[emanruse]

Perhaps journal should be persistent or non-persistent as per Qubes respecting upstream distribution culture [...]

Or maybe it is wiser to consider what Qube OS's specifics are and what defaults fit that. It separates system/user data explicitly, so if distro X changes its defaults tomorrow and starts flooding one's AppVMs with logs, or vice versa, there will be no flashing red lights to warn the user. Journal is system, not user data, so it doesn't belong in private volume (unless explicitly requested).

does not seem to support `ConditionPathExists`

In both Fedora and Debian, the default is Storage=auto (from man page):

"auto" behaves like "persistent" if the /var/log/journal directory exists, and "volatile" otherwise (the existence of the directory controls the storage mode).

Perhaps that could be used as a control flag (e.g. a bind-dir which advanced users have no problem with), so no conffiles need to be touched. Re. the answers questions: They are about evaluating the average Qubes user's workflow, not about what you or I prefer. That's why the skipped question #4 is essential. Without any actual statistical data, I can only remind that 90%+ of desktop users will generally abandon software which doesn't behave as they expect, rather than report bugs or analyze logs (or even know what a log is).