Linux Logs: How to View, Read & Configure

Introduction

Logs provide detailed records of system events, errors, and user activities, which help diagnose and resolve issues. Knowing how to view, read, and configure Linux log files is crucial for system administration and troubleshooting.

The following text explains Linux log files and how to view, read, and configure them.

Prerequisites

• A Linux system (this tutorial uses Ubuntu 22.04).
• Access to the terminal.
• A user account with sudo or root privileges.

What Are Linux Log Files?

All Linux systems create and store information log files for boot processes, applications, and other events. These files are a helpful resource for troubleshooting system issues.

Most Linux log files are stored in plain text files (ASCII format) in the /var/log directory and subdirectories. Logs are generated by the Linux system daemon log, syslogd, or rsyslogd. Properly managing these logs ensures essential data is readily available for analysis and auditing.

Important Linux Logs

Logs provide information about system operations. A good understanding of each log file type helps distinguish between the logs. Most Linux logs can be grouped into one of four categories:

• System Logs
• Event Logs
• Application Logs
• Service Logs

The following sections elaborate on each log type.

System Logs

System log files in Linux contain information about the core operating system activities, including boot processes, kernel messages, and hardware events. These logs are essential for diagnosing and troubleshooting system-level issues. Proper management and analysis of these logs help maintain system stability and security. The most essential system log files are:

• /var/log/syslog and /var/log/messages capture a wide range of system events, including general system messages, notifications from system services, application errors, and other important events.
• /var/log/kern.log contains messages from the kernel, including hardware-related information and kernel module logs.
• /var/log/dmesg logs messages from the kernel ring buffer, primarily capturing information about the boot process, hardware detection, and driver initialization.
• /var/log/daemon.log contains information about events related to running the Linux operation.
• /var/log/faillog records failed login attempts, providing information on the number of unsuccessful login attempts by user accounts.
• /var/log/lastlog maintains a record of the last successful login for each user.
• /var/log/boot.log stores all information related to booting operations.
• /var/log/auth.log stores all authentication logs, including successful and failed attempts.
• /var/log/cron logs activities related to cron jobs, i.e., scheduled tasks running at specified intervals.
• /var/log/debug stores detailed messages related to debugging and helps troubleshoot specific system operations.
• /var/log/yum.log is the yum command log. Records activities related to the Yum package manager.

Application Logs

Application logs store information relevant to an executed application, including error messages, operational details, and signs of potential system compromise. Log files that fall into this category include:

• CUPS Print System logs capture print job details and printer-related errors.
• Rootkit Hunter log is generated by Rootkit Hunter, which scans for potential rootkits and other security vulnerabilities.
• Apache HTTP server logs are logs for the Apache web server. They include access logs (recording client requests) and error logs (capturing server errors and issues).
• Samba SMB server logs are logs for the Samba server, which provides file and print services for SMB/CIFS clients and captures access details and errors.
• The X11 server log is a log related to the X11 window system. It captures details about the graphical server's operations and errors.

Service Logs

Service logs contain information related to system services and background processes. They help monitor the performance and operation of these services. Some system logs, like systemd service logs (/var/log/syslog, /var/log/messages), cron job logs (/var/log/cron), and Samba server logs (/var/log/samba/log.smbd), are also classified as service logs.

Event Logs

Event logs include specific system events and actions, often related to security, authentication, and user activities. They provide detailed records of login attempts, system changes, and other significant occurrences, helping administrators monitor and audit system activities.

Essential event logs include audit logs (/var/log/audit/audit.log) and system logs such as authentication logs (/var/log/auth.log) and Kernel logs (/var/log/kern.log).

Linux Logs Location

In Linux, log files are primarily located in the /var/log directory. This directory contains various subdirectories and files that store logs for different system components, services, and applications.

Application log files are stored in the subdirectories of the the /var/log directory.

View Linux Logs via the Command Line

To view the contents of the /var/log directory, which lists all log files and directories containing logs, do the following:

1. Enable root privileges using the sudo command or switching to root with su.

2. Run the cd command to navigate to the /var/log directory:

cd /var/log

3. To view the logs, type the ls command:

sudo ls

The command displays all Linux log files, such as kern.log and boot.log. These files contain the information necessary for the operating system to function correctly.

How to Read Linux Logs

Linux logs usually follow a structured pattern to ensure consistency. Here's a breakdown of the standard components of a log entry format:

• Timestamp. Indicates the date and time when the event occurred.
• Hostname. The name of the system (computer) where the log entry was generated.
• Service or Application Name. The name of the service or application that generated the log entry.
• Process ID (PID). The identifier of the process that generated the log entry.
• Log Level. The severity of the log entry (e.g., INFO, WARNING, ERROR).
• Message. The actual log message detailing the event.

There are several commands used for reading Linux logs. However, their outputs vary. Each command serves different purposes and is chosen based on the desired filtering, processing, and output customization level for log analysis. For example, dmesg and journalctl provide system-level logs, with journalctl offering more structured and customizable filtering options. grep, awk, and sed are text-processing tools that filter logs based on specific patterns, keywords, or criteria.

The following sections elaborate on the commands.

Read Linux Logs Using cat

The cat command displays the entire contents of a file. For example, to view /var/log/syslog, run:

sudo cat /var/log/syslog

The output includes a timestamp, hostname, process name, PID, and a message.

View Linux Logs Using less

The less command in Linux allows you to view the contents of log files one screen at a time. It allows you to navigate through large files easily without loading the entire file into memory. The less command also supports forward and backward scrolling, searching, and other navigation commands, making it ideal for examining detailed log files efficiently.

To view /var/log/syslog, run:

sudo less /var/log/syslog

To navigate through the output, use the following keys:

• Down Arrow or j to move down one line.
• Up Arrow or k to move up one line.
• Spacebar or Page Down to move down one page.
• Page Up or b to move up one page.
• G to go to the end of the file.
• g to go to the beginning of the file.

Read Linux Log Files Using more

The more command in Linux is used to view the contents of log files one screen at a time, similar to the less command but with limited navigation capabilities. It allows you to scroll down line by line or page by page, making it useful for quickly viewing and analyzing log files too large to fit on one screen.

sudo <span style="background-color: initial; font-family: inherit; font-size: inherit; color: initial;">more /var/log/syslog</span>

Unlike less, the more command does not support backward navigation and lacks advanced search and navigation features. To navigate the output, use:

• Spacebar to move forward one page at a time.
• Enter to move forward one line at a time.
• b to move backward one screen page at a time.
• q to quit the more command screen.

View Linux Logs Using tail

The tail command in Linux displays the last ten entries of a file, including log files. It is convenient for monitoring log files in real time or quickly checking recent entries. View /var/log/syslog with:

sudo tail /var/log/syslog 

Use head to Read Linux Log Files

The head command prints the top part of a file. It's useful for quickly checking a log file's initial entries. View the beginning of /var/log/syslog with:

sudo head /var/log/syslog

Read Linux Logs with grep

The grep command searches for specific patterns or text within a file. When used with log files, grep identifies entries that contain specific keywords or patterns, making it helpful in isolating relevant information during troubleshooting or analysis. For example, look for error in /var/log/syslog with:

sudo grep "error" /var/log/syslog

Using awk to View Linux Logs

awk is a versatile tool for reading and processing log files. It allows the user to extract specific information, filter based on conditions, and efficiently perform various operations on log data.

For example, to scan the /var/log/syslog file for lines containing the word error and then print the first, second, third, and fifth fields' columns of each matching line, run:

sudo awk '/error/ {print $1, $2, $3, $5}' /var/log/syslog

Reading Linux Logs Using sed

The sed command is a stream editor that processes and modifies text in log files. For example, to search for lines containing the word error and print those lines from the /var/log/syslog, run the following:

sudo sed -n '/error/p' /var/log/syslog

Using dmesg to View Linux Log Files

The dmesg command in Linux allows viewing kernel ring buffer messages, which are system logs related to hardware and kernel events. These logs are valuable for diagnosing hardware issues, driver problems, and system initialization processes. By running dmesg, you can access these logs directly from the kernel, providing a low-level view of system activities and events.

To view log files using dmesg, run the command without any options:

sudo dmesg

This command displays the kernel ring buffer messages on your terminal, allowing you to read through the system logs.

Read Linux Logs via journalctl

Use journalctl to view logs collected by the systemd daemon. It also filters logs by service, date, and other criteria. For example, to view logs for a specific service, in this case apache2, run:

sudo journalctl -u apache2.service

To view logs since a specific date, run:

sudo journalctl --since "2024-06-19"

Read Linux Logs Using GUI Tools

The GUI log viewer tools offer intuitive interfaces, filtering options, and search capabilities to help you efficiently read and analyze Linux logs without relying solely on command-line tools.

System Log Viewer is a GUI tool used to monitor system logs.

The interface provides several log management functions, including a log statistics display. It is a user-friendly log monitoring GUI.

Useful features include:

• A live view of logs.
• Number of lines in the log.
• Log size.
• Most recent log dates.
• Modifications made to logs.
• Filters.
• Keyboard Shortcuts.

How to Configure Linux Logs

Configuring Linux logs involves setting up and adjusting configurations for the system's logging daemons. Linux uses rsyslog as the default logging daemon. Here's a step-by-step guide to configure Linux logs:

1. The main configuration file for rsyslog is /etc/rsyslog.conf. Open the main configuration file in a text editor, such as Nano:

sudo nano /etc/rsyslog.conf

2. Within the rsyslog.conf file set global directives such as the maximum file size, the rotation interval, and other parameters. For example:

#Set the default log file owner
$FileOwner syslog
$FileGroup adm
#Set the default log file permissions
$FileCreateMode 0640
#Set the default directory permissions
$DirCreateMode 0755
#Set the maximum log file size (e.g., 10MB)
$MaxMessageSize 10k

3. Define log rules to specify how logs are to be processed. For example, to log all authentication messages to a specific file, type in the following:

# Log all authentication messages 
auth.* /var/log/auth.log

4. After making changes to the configuration files, restart the rsyslog service to apply the changes:

sudo systemctl restart rsyslog

The command has no output.

Custom Log Messages

To create and manage custom log messages on a Linux system, use logger, a command-line tool that provides an interface to the system log, rsyslog. The logger command allows you to add custom log messages to the system log.

To log a simple message, use the following command:

logger "This is a custom log message."

The command has no output, but adds the message "This is a custom log message." to the system log, which can be found in /var/log/syslog.

You can add a tag to your log message to make it easier to identify:

logger -t myscript "This is a custom log message from my script."

The command, which has no output, tags the log entry with myscript.

Another option is to configure rsyslog to handle custom log messages and direct them to a specific log file. To accomplish that, create a configuration file in the /etc/rsyslog.d/ directory. For example:

1. Run the following command:

sudo nano /etc/rsyslog.d/custom-log.conf

2. Add the configuration below to direct messages from the local0 facility to a custom log file:

local0.* /var/log/custom.log

3. Ensure the custom log file exists and has the appropriate permissions using touch, chown, and chmod commands:

sudo touch /var/log/custom.log 
sudo chown syslog:adm /var/log/custom.log 
sudo chmod 640 /var/log/custom.log

The commands have no output.

4. Restart the rsyslog service to apply the changes:

sudo systemctl restart rsyslog

Log Rotation

Log rotation is the process of managing log files to prevent them from consuming excessive disk space and to make log file management easier. It involves moving or archiving old log files and creating new ones to replace them. This process helps ensure logs do not grow indefinitely, which leads to system performance issues and difficulty managing logs.

Log rotation is managed using tools like logrotate.

To rotate a log file, do the following:

1. Create or edit a file in the /etc/logrotate.d/ directory for your custom log rotation. For example:

sudo nano /etc/logrotate.d/customlog

2. In the editor, add the following configuration:

/var/log/lognamehere.log {
    missingok
    notifempty
    compress
    size 20k
    daily
    create 0600 root root
}

The commands perform the actions as follows:

• missingok. Tells logrotate not to output an error if a log file is missing.
• notifempty. Does not rotate the log file if it is empty.
• size. Ensures that the log file does not exceed the specified dimension and rotates it otherwise.
• daily. Rotates the log files on a daily schedule. This can also be done on a weekly or monthly schedule.
• create. Represents a log file where the owner and group are a root user.

3. Save and exit the file.

4. Run logrotate in debug mode to test the configuration without making any changes:

sudo logrotate -d /etc/logrotate.conf

5. To apply the log rotation immediately, force logrotate to run:

sudo logrotate -f /etc/logrotate.conf

The command has no output.

Linux Log Aggregation

Log aggregation is collecting and centralizing log data from multiple sources into a single location for easier management, analysis, and monitoring. It helps troubleshoot, identify patterns, ensure security, and maintain compliance by providing a comprehensive view of the system's operations.

Several tools are available for log aggregation:

• ELK Stack (Elasticsearch, Logstash, Kibana)
  • Elasticsearch. A search and analytics engine that stores and indexes log data.
  • Logstash. A data processing pipeline that ingests data from multiple sources, transforms it, and sends it to Elasticsearch.
  • Kibana. A visualization tool that allows you to explore and visualize the data stored in Elasticsearch.
• Graylog. An open-source log management tool that provides real-time search, analysis, and alerting capabilities. It aggregates logs from various sources and offers a web-based interface for monitoring and analysis.
• Fluentd. An open-source data collector that helps unify the data collection and consumption process. It supports various plugins for data input and output, making it versatile for different use cases.
• Splunk. A comprehensive platform for searching, monitoring, and analyzing machine-generated data. It provides powerful visualization and alerting capabilities and is widely used in enterprise environments.
• Prometheus. Primarily used for monitoring and alerting but also collects and queries logs. Prometheus integrates well with Grafana for visualization purposes.

Conclusion

This article explained how to view, read, and manage Linux logs. It provided an overview of CLI and GUI tools for log management in Linux.

Next, read about the best syslog servers.