Disk space analysis: ncdu, du

Disk space monitoring, linux disk space analysis is critical to maintaining your system performance and preventing problems with insufficient storage space such as system crashes, data loss and service interruptions.

We recommend regularly checking disk space in Linux if you manage a virtual private server (VPS), maintain databases, and run large-scale applications.

du

The du command is a standard Linux/Unix command that allows the user to quickly get disk usage information on directories and files. It is best applied to specific directories and allows many variations to customize the output to suit your needs.

The syntax of the du utility is as follows:
du [options] [file]

Keys du

The du command offers a number of options that extend its functionality. Let's take a look at some commonly used options:

• -s or --summarize Provides summary disk usage information for a specified directory or file, without displaying individual subdirectory usage information.
• -h or --human-readable Displays sizes in an easy-to-read format using units such as KB, MB, GB, etc. This option makes it easier to interpret disk usage information.
• -c or --total Shows total disk usage in addition to individual usage for directories and files.
• -a or --all Displays disk usage information for all files and directories, including hidden files.
• --time Displays the last modification time of any file or directory
• --exclude Excludes certain directories or files and folders from disk usage calculations based on patterns or names.

The 'du' command in Linux is a versatile tool that can provide valuable information about your disk usage.

Let's start with a simple example. Suppose you want to check disk usage in a particular directory. You can do this by using the 'du' command followed by the path to the directory. For example, to check the disk usage of the /home/user/documents directory, use the following command:
du /home/user/documents
The output of this command will show the total disk usage of the specified directory, including all its subdirectories. The number displayed is the total size of all files and directories contained in the specified directory, measured in kilobytes.

But what if you want to see the size of each subdirectory individually? You can do this by using the '-h' option with the 'du' command. This option makes the output easier to read by displaying the sizes in a “human readable” format (i.e. kilobytes (K), megabytes (M), gigabytes (G), etc.). Here's how you can use this option:
du -h /home/user/documents
In this output, you will see a list of all subdirectories in the specified directory along with their respective sizes and determine which subdirectories take up the most space.

Using the '-a' and '-h' flags

Suppose you want to see the disk usage of all files in a directory, not just the total size of the directory. You can do this by using the '-a' flag. Combine this with the '-h' flag to make the output clearer. Here's an example:
du -ah /home/user/documents
In this output, you will see a list of all files in the specified directory along with their respective sizes. This will help you find out which specific files are taking up the most space.

Using the «--exclude» flag

Finally, the '--exclude=PATTERN' flag allows you to exclude files that match a specific pattern. This can be useful if you want to exclude certain file types. Here's an example:
du -h --exclude='*.txt' /home/user/documents
In this example, 'du' excludes all .txt files from the disk usage calculation, allowing you to focus on other file types.

These are just a few examples of the many ways to use the 'du' command in Linux. By combining the various flags and options, you can customize the command to your specific needs and get a detailed picture of your disk usage.

Using the -c option:

This option displays the grand total:
du -c -h kt.txt pt.txt

8.0K    kt.txt
4.0K    pt.txt
12.0K  total

Using the --time option:

This option is used to display the time of the last modification on the du output.

ncdu

ncdu (NCurses Disk Usage) is an ncurses-based version of the popular 'du with command that offers the fastest way to analyze and track disk space usage in Linux. It is a simple, fast and easy to use program that runs on any minimal Linux/Unix based system with ncurses installed.

Featuring an excellent interface, the ncdu linux utility displays information showing disk space usage in megabytes and gigabytes, along with a graphical usage bar, file/directory names, file deletions, upgrade options, etc.

First, update the system:
apt update
Then execute the update command:
apt upgrade
Let's see how to install ncdu on Debian (10,11 and 12) and Ubuntu (all versions).

On Debian, Ubuntu and Mint
apt install ncdu 
On RHEL/CentOS/Fedora and Rocky/AlmaLinux
yum install ncdu
On Gentoo Linux
emerge -a sys-apps/ncdu
On Alpine Linux
apk add ncdu                 
On Arch Linux
pacman -S ncdu               
On OpenSUSE
zypper install ncdu         
On FreeBSD   
pkg install ncdu
After installation, run ncdu:
ncdu
This command will scan your working directory and open the ncdu interface.

Output format

After the scan is complete, ncdu will show you the file and directory structure using the disk:

Entries in the browser interface can be displayed with a single character. These characters have the following meaning:

• ! An error occurred while reading this directory.
• . An error occurred while reading a subdirectory, so the specified size may be incorrect.
• < The file or directory is excluded from statistics using exclusion patterns.
• > The directory is in a different file system.
• ^ The directory is excluded from statistics because it is a Linux pseudo file system.
• @ It is neither a file nor a folder (symbolic link, socket, ...).
• X The same file has already been counted (hard link).
• e Empty directory.

Navigation

Use the arrow key to navigate through the files. This will allow you to move up and down. To see information about the selected directory (path, disk usage), press 'i'.
Press right, enter, l - Open the selected directory
Press left left, <, h - Go to previous directory

To hide this window, press 'i' again.

Available interface functions

By default, files and directories are sorted by size. However, the ncdu interface offers various options for sorting them.

n Sort by filename (click again to sort in descending order)

s Sort by file size

C Sort by number of items

a Toggles between displaying disk usage and displaying apparent size.

M Organize by last child mtime or modified time.

d Delete the selected file or directory. An error message will be displayed if the contents of the directory do not match or no longer exist in the file system.

t Switch folders before files when sorting.

g Toggle between displaying percent, graph, both, or neither. Percentage is relative to the size of the current directory, graph is relative to the largest item in the current directory.

c Toggles whether to display the number of children.

m Toggles the display of the last child mtime or modified time.

e Show/hide “hidden” or “excluded” files and directories. Note that even if you don't see hidden files and directories, they still exist and are taken into account when calculating directory sizes. If you suspect that the totals displayed at the bottom of the screen are not true, make sure you have not enabled this option.

i Displays information about the currently selected item.

r Updates/recalculates the current catalog.

b Ncdu will determine the preferred shell by the variables “NCDU_SHELL” or “SHELL” (in that order) or call “/bin/sh” if neither is specified. This also allows you to configure another command to execute when the 'b' key is pressed. For example, to invoke the vifm(1) file manager instead of the shell, run ncdu as follows:
export NCDU_SHELL=vifmncdu
Ncdu will set the environment variable 'NCDU_LEVEL' or increment it before running the shell. This variable allows you to determine when your shell is started from ncdu, which can be useful to avoid nesting multiple instances of ncdu. Ncdu itself (currently) does not (currently) warn you when you try to run nested instances.

q exit.

Keys ncdu

-h Outputs a small help message

-q Quiet mode. When calculating disk space, ncdu by default refreshes the screen 10 times per second, in quiet mode this is reduced to once every 2 seconds. Use this feature to save bandwidth on remote connections.

-v Printable version.

-x Counts only files and directories that are in the same file system as the specified dir.

--exclude PATTERN Exclude files that match the pattern. These files will still be displayed by default, but will not be counted in disk usage statistics. This argument can be added more than once to add more templates.

--exclude-from FILE Exclude files matching any template in FILE. Templates must be separated by a newline.

--include-caches, --exclude-caches Include (default) or exclude directories containing CACHEDIR.TAG. These directories will be displayed, but their contents will not be scanned and counted in disk usage statistics.

--include-kernfs, --exclude-kernfs (Linux only) Enable (default) or exclude Linux pseudo-file systems such as /proc (procfs), /sys (sysfs).

Full list of currently known pseudo file systems: binfmt, bpf, cgroup, cgroup2, debug, devpts, proc, pstore, security, selinux, sys, trace.

Nuances

Mismatch of du, ncdu and software data

Although the commands 'ncdu', 'du' is a powerful tool for estimating disk usage in Linux, there are other methods and tools that can also be effective, especially for more complex scenarios or for users who prefer a different approach. Such an alternative is the 'df' command.

Using the 'df' command

The 'df' command in Linux is used to determine the disk space usage of a file system. While 'du' estimates the usage of files and directories on disk, 'df' displays the amount of disk space used and available on file systems.

The basic syntax of df:
df [options] [filesystems]
Here,

• options: These are optional flags that change the output of the command.
• filesystems: You can specify specific filesystems to check their usage instead of getting information for all mounted disks.

If no filename is specified, it displays the space available in all currently mounted filesystems.

If you want to run df in a convenient readable format, use the --human-readable (abbreviated as h ) option, here is an example of how to use the 'df' command:
df -h
In this example, the '-h' flag is used to display the output in a convenient format. You will see a list of all mounted file systems, along with their total size, space used, space available, and percentage of space used.
If deleting unnecessary files on the server does not clear disk space, you can view a list of all deleted files with open file descriptors using this command:
lsof | grep deleted
And then perform a restart
service  restart
Here's a brief comparison of disk space checking methods:

Method	Good	Bad
Сommand 'du'	Detailed information on disk usage, many options	Permission errors may need to be corrected
Command 'df'	Provides for the use of disk file systems	Less detailed than «du»
Third-party tools (e.g. 'ncdu')	User-friendly interface, graphical output	Requires installation, may not be as flexible as command line tools