What are server logs and what are they for

A web server is an autonomous system. If it is properly configured, programs can run for months without human intervention. At the same time, many events are continuously happening on the server:

• Visitors visit websites, browse pages, and fill out forms;
• The mail server receives and sends mail;
• The backup system creates backups;
• System and other errors occur.

All these events are logged, and server administrators are able to monitor and analyze them.
 

What are logs?

These are messages that record information about events on the server. Server logs are automatically recorded in chronological order. They contain data about users and their actions on the platform, as well as about the operation of the server.
Every time any browser or user agent, including a search engine, requests any resource from your server - a page, an image, a JavaScript file, etc., this information is recorded in the server log file. The log file stores information including the client's IP address, date and time of the request, requested page, HTTP code, etc. This data can be combined into one file or divided into separate logs. They are not available to ordinary Internet users, they are visible only to the developer or other administrators.
 

Why you need logs

Since logs record everything that happens, analyzing logs allows you to use the data in the area of website promotion and maintenance.

• By viewing the server logs, you can determine the cause of system failures and breakdowns.
• Web server logs help you monitor processes more efficiently, make future predictions, and maintain overall control.
• Even if the system is running smoothly, it is recommended to periodically analyze the server logs. This allows you to identify vulnerabilities or flaws at an early stage, before they become a problem.
• Logs help identify malware and hacks.
• Provide valuable information for web site development. They help to collect traffic statistics using incoming technical data.
• From the logs, you can determine where users are coming from, where they are geolocating, and what devices they are using to visit.

 

What is stored in logs and why they are checked

A log file is a kind of a notebook of events where data about errors, user actions and other events that occur on the server or in the system are recorded. A log file contains information in an abbreviated format, which may seem like an incomprehensible set of characters to the average user. However, each entry has its own meaning, and it is important for specialists to be able to understand them.
 

Log classification

Ideally, logs should be written all the time. However, if you "pile up" everything, useful information will be lost. To simplify searching and reading logs, they are divided into classes.

• Debug - recording important state changes, such as database accesses, service start/pause, successful processing of records, and so on.
• Warning - non-standard situation, potential problem: incorrect request format or invalid call parameter.
• Еrror - typical mistake.
• Emergency - a complete failure of performance when there is no access to the database or network, no space on the hard disk.

The log file can be supplemented with two more levels of records:

• Trace - step by step recording of the process. This is useful when it is difficult to find an error.
• Information - general information about the operation of a service.

 

Types of logs

Logs are divided into types for ease of processing:

• system syslog;
• server logs;
• mail logs;
• database logs;
• authorization and authentication logs.

Each type of log has its own log entry. To check authorization logs you should go to access log. To check the system load - go to the dmesg log, for data about user requests - to the access_log. When some logs are written separately from others, it is easier to diagnose the situation and find the source of the problem.
 

How to view and check server logs

Validation can be accomplished in a variety of ways. Manually is a good method for individual servers, but as the server fleet grows and the number of applications increases, it is recommended to use specialized logging systems such as Graylog, ELK, Loggy or Splunk. These tools allow you to organize large-scale monitoring, set up alerts for problems and set thresholds for threat detection.
 

Where server logs are stored and how to view them

Logs are saved in files and most often with the extension .log. You can view their contents with a text editor.
The location where the Linux server logs are located depends on the software, settings and the path specified by the administrator. Usually, you can view server logs in the /var/log/ directory. However, not all services use this directory.
It is recommended to use cloud storage to store logs of networking, engineering equipment, databases and applications. Even with sufficient hard disk space and strong security, equipment can fail and attackers can delete log files. Cloud storage allows you to recover events and investigate an incident even in the event of a complete system failure.
 

How to read

Starting and disabling event logging is done in the administrative panel. Logs can usually be accessed through the "log" or "logs" section. However, saved files are not stored forever.
In the access log (access.log) you will find:

• the address of the resource that was accessed;
• IP address of the user who made the request;
• the date and time of the visit, specifying the time zone;
• type of request (GET/POST) to receive or send data;
• the page accessed by the user;
• the protocol through which the visitor accessed the resource;
• server response code, indicating whether the request was successful or there were problems;
• the number of bytes transferred;
• information about the visitor (or bot) - device, operating system, etc.

Usually this data is enough to analyze the situation and take the necessary measures. For example, you can block a bot that creates an excessive load on the site.
The error file (error.log) records the moments when something goes wrong. The following information can be found in it:

• date, time and type of error, as well as the user's IP address;
• event type;
• file path and error message string.

Server response codes have the following interpretation:

• 1xx - informational messages, e.g. code 102 - the page is being processed;
• 2xx - successful page loading;
• 3xx - redirection (redirect);
• 4xx - content loading error;
• 5xx - server-side error.

 

Analysis

Once the data is decrypted, you still need to analyze it. For this purpose, there are various software such as Weblog Expert, WebAlyzer, Analog, Webtrends, Awstats, SpyLOG Flexolyzer and other paid and free programs. They help to organize and analyze data from logs to extract useful information and take further action.
 

Briefly, what you need to know about logs

A log is a text file that automatically records significant data about the system operation. It is a kind of a diary containing information about errors, user actions, etc. The record is kept by special software that manages internal processes on the server. Developers and engineers often refer to this data when debugging or checking software functionality.