Step-by-step guide:

Navigating and Filtering Logs with journalctl

journalctl is a command-line utility for querying and displaying the contents of the systemd journal. Here are some fundamental commands to get you started:

View All Logs: The most basic command, journalctl, will display all log entries from the beginning of the journal.

 

 

Filter by Time: To narrow down logs to a specific time period, use the --since and --until options. For example, journalctl --since "yesterday" shows logs from the last day.

 

 

The —since flag accepts the following date specifications format: “2012-10-30 18:17:16". If the time part is omitted, "00:00:00" is assumed. If only the seconds component is omitted, ":00" is assumed. If the date component is omitted, the current day is assumed. Alternatively, the strings "yesterday", "today", "tomorrow" are understood, which refer to 00:00:00 of the day before the current day, the current day, or the day after the current day, respectively. "now" refers to the current time.

Filter by Service: You can view logs for a specific systemd unit (service) with the -u flag. For instance, journalctl -u sshd.service will show all log entries related to the SSH daemon, journalctl -u docker filters log entries related to Docker, and so on.

 

 

Real-time Monitoring: To "tail" or follow new log entries as they are written, use the -f flag. For instance,: journalctl -f. 

This is similar to tail -f /var/log/syslog.

Adding the -u flag lets you follow a specific service. For instance: journalctl -f -u docker

 

 

Viewing Logs from Previous Boots

 

When troubleshooting a system crash or reboot, it's often necessary to check logs from the previous boot. The -b flag is essential for this.

journalctl -b -1: Displays logs from the immediately previous boot.

 

 

journalctl -b -2: Shows logs from the boot before the previous one.

 

 

journalctl --list-boots: Provides a list of all available boot sessions, showing their index, ID, and a timestamp.

 

 

While -0 is the last boot, -1 is the boot before last, and so on.

Filtering by Priority

Logs are assigned a priority level from 0 (emerg) to 7 (debug). You can filter logs by priority using the -p flag.

journalctl -p err: Shows all messages with a priority of "error" or higher (critical, alert, emergency).

 

 

journalctl -p warning: Displays warnings and messages with a higher priority.

 

Syslog Levels

0 = emerg
1 = alert
2 = crit
3 = err
4 = warning
5 = notice
6 = info
7 = debug

Controlling Journal Size

The journal can grow quite large over time, consuming significant disk space. You can manage this with the --disk-usage and --vacuum options.

journalctl --disk-usage: Reports the current disk space used by the journal.

 

 

sudo journalctl --vacuum-size: removes the oldest archived journal files until the disk space they use falls below the specified size. Accepts the usual "K", "M", "G", and "T" suffixes (to the base of 1024). Vacuuming only affects archived journals, not the active ones.

 

 

sudo journalctl --vacuum-time: removes archived journal files older than the specified timespan. Accepts the usual "s" (default), "m", "h",  "days", "weeks", "months", and "years" suffixes.

 

 

Combining Filters for Advanced Queries

The real power of journalctl comes from combining different filters. You can use multiple flags to pinpoint exactly what you're looking for.

journalctl -u docker --since "1 hour ago" -p err: This command will show all error messages from the docker service that have occurred in the last hour.

 

 

Understanding systemd Journal Export Format

Think of the systemd Journal Export Format as a universal language for transferring log data. It's a simple, binary-safe stream designed specifically to send log entries reliably over a network or to another program. Unlike a regular text file, which might break if it contains special characters, this format ensures all data—even binary content—arrives intact.

You create this format using the command journalctl -o export. This makes it a powerful tool for tasks like streaming logs from one server to a central log collector or backing up a portion of your logs in a structured way.

 

How It's Structured: The "Key-Value" System

Each log entry is a series of key-value pairs. The format is carefully designed to handle different types of data:

For regular text: The data is a simple KEY=VALUE pair on a single line. This is used for standard fields like MESSAGE or SYSLOG_IDENTIFIER. For example: SYSLOG_IDENTIFIER=sshd followed by a newline.

For binary data: This is where the format's robustness shines. It can safely transport data that might contain special characters or be non-textual. Instead of KEY=VALUE, it uses a three-part structure:

• The KEY on its own line.
• A 64-bit size value indicating exactly how long the binary data is.
• The raw binary data itself.

Each complete log entry is then separated by a double newline (\n\n), signaling the end of one record and the start of the next.

Why It's So Useful

This format is perfect for machine-to-machine communication. Because the format is well-defined and handles all data types, it can be easily parsed by other programs, regardless of the operating system. When you're troubleshooting or analyzing data, it's often more efficient to work with this structured stream than with raw, unformatted text.

Conclusion

Mastering journalctl and understanding the systemd journal is a crucial step for anyone who wants to become proficient in Linux system administration. You now know to move beyond simple text log files and leverage a modern, powerful logging system.

By using journalctl's filtering and formatting options, you can quickly find the exact information you need, whether you are diagnosing a recent boot issue, monitoring a specific service in real-time, or performing a historical audit. Furthermore, understanding the Journal Export Format provides you with the means to easily and reliably transport log data between systems, which is essential for centralized logging and advanced analysis.

With these tools at your disposal, you are well-equipped to efficiently monitor, troubleshoot, and maintain the health of your Linux systems.

References/Related Content 

• Journalctl man: https://man7.org/linux/man-pages/man1/journalctl.1.html 
• Journal Export Format: https://systemd.io/JOURNAL_EXPORT_FORMATS/#journal-export-format 

Disclaimer: This post outlines a potential custom workaround for a specific use case or provides instructions regarding a specific task. The solution may not work in all scenarios or Sisense versions, so we strongly recommend testing it in your environment before deployment. If you need further assistance with this, please let us know.