Fixing the Too Many Open Files Error on Linux

Key Takeaways

In Linux, if you encounter the error message "Too many files open", it means that your process has reached the maximum limit of files it is allowed to open, which is typically set at 1,024. To temporarily raise this limit to, for instance, 2,048 files, you can use the command "ulimit -n 2048". For a permanent increase, you can edit the systemd configuration files.

What Is the Too Many Open Files Error?

System resources on Linux computers are shared among users, meaning that exceeding your allotted portion can result in reaching a maximum limit. Furthermore, this behavior may negatively impact other users or processes by causing bottlenecks.

The kernel of a Linux computer has numerous tasks, one of which is monitoring the usage of limited system resources like RAM and CPU cycles. In a multi-user system, it is important to ensure that individuals and processes do not exceed their fair share of system resources. For instance, it is unfair for one person to monopolize CPU time and cause the computer to slow down for others. Even if you are the sole user of your Linux computer, there are still restrictions in place for the resources your processes can utilize, as you are considered just another user.

Several system resources such as RAM, CPU cycles, and hard drive space are commonly recognized and easy to identify. However, there are numerous other resources that are continuously monitored and have a predetermined maximum limit for each user or user-owned process. One such resource is the maximum number of files that a process can have open simultaneously.

If you have ever encountered the error message "Too many files open" in a terminal window or noticed it in your system logs, this indicates that the process has reached its upper limit and is no longer allowed to open additional files.

Why Are So Many Files Opening?

Linux has a system-wide restriction on the maximum number of concurrently open files. Although this limit is typically quite generous, it does exist. Each user process is granted a separate allocation within this limit, ensuring a fair distribution among all active processes in the system.

The allocation process primarily involves assigning file handles, which are required for each opened file. Even with ample system-wide allocations, file handles can deplete unexpectedly quickly.

Linux operates by abstracting various elements, making them appear as files. These files can range from ordinary ones to directories, with even actions like opening a directory requiring a file handle. Moreover, Linux utilizes block special files as drivers for hardware devices, while character special files are commonly utilized for devices with throughput capabilities, like pipes and serial ports.

Content handling is done differently for block special files and character special files. Block special files handle data in blocks, while character special files handle data character by character. Access to both of these special files is only possible through the use of file handles. Program libraries, streams, and network connections all utilize file handles for accessing these files.

By abstracting the various requirements and treating them as files, interfacing with them becomes simpler. This abstraction enables functionalities such as piping and working with streams.

How to Check File Handle Limits

Linux operates by continuously opening files and utilizing file handles in the background, solely for its own functioning, disregarding the activities of your user processes. The tally of open files encompasses not only the files you have personally opened, but also encompasses virtually every component within the operating system, all of which are reliant on file handles.

The system-wide maximum number of file handles can be seen with this command.

cat /proc/sys/fs/file-max

The result of this computation is an astonishingly enormous number equivalent to 9.2 quintillion. This value represents the absolute maximum limit according to the theoretical framework of the system. It signifies the highest conceivable quantity that can be stored in a 64-bit signed integer. However, the practical ability of your computer to handle such an excessive number of simultaneously open files is an entirely separate concern.

To determine the maximum number of files that one of your processes can open, you can use the ulimit command with the -n (open files) option.

And to find the maximum number of processes a user can have we'll use ulimit with the -u (user processes) option.

ulimit -u

When multiplying 1024 and 7640, the result is 7,823,360. However, it is important to consider that several of these operations are likely being utilized by your desktop environment and other background processes. Therefore, this number serves as a theoretical maximum and is unlikely to be realistically achieved.

The crucial aspect to focus on is the maximum number of files that a process can open, which is initially set at 1024. It is worth noting that opening the same file 1024 times simultaneously is equivalent to opening 1024 distinct files concurrently. Once all available file handles have been used, further file openings are no longer possible.

The number of files a process can open can be adjusted. When adjusting this number, there are two values to consider: the current value or the desired value, known as the soft limit, and the highest value that the soft limit can be raised to, known as the hard limit.

To clarify, the soft limit represents the current value, while the hard limit represents the upper limit that the current value can reach. A regular user can raise their soft limit to any value up to their hard limit, while the root user has the ability to increase their hard limit.

To see the current soft and hard limits, use ulimit with the -S (soft) and -H (hard) options, and the -n (open files) option.

ulimit -Sn

ulimit -Hn

To observe the enforcement of the soft limit, we developed a program named "open-files" that iteratively opens files until it encounters a failure. It subsequently awaits a keystroke prior to releasing all the file handles it had utilized.

It opens 1021 files and fails as it tries to open file 1022.

When subtracting 1021 from 1024, we get a result of 3. What occurred with the remaining three file handles? These were utilized for the STDIN, STDOUT, and STDERR streams, which are automatically generated for every process. Their file descriptor values are consistently 0, 1, and 2. To view these, the lsof command can be utilized with the -p (process) option and the process ID of the open-files program. Conveniently, the program prints its process ID to the terminal window.

lsof -p 11038

To investigate the process that has consumed all the file handles, follow these piped commands. They will provide you with the top fifteen users who have utilized the most file handles on your computer.

To begin, execute the following sequence of commands:

1. lsof | awk '{ print $1 " " $2; }'

2. sort -rn

3. uniq -c

4. sort -rn

5. head -15

To view a greater or lesser number of entries, modify the -15 parameter in the head command. After identifying the process, determine whether it has become uncontrollable and is excessively opening files, or if it genuinely requires these files. If the latter is the case, it is necessary to enhance its file handle limit.

How to Increase the Soft Limit

By raising the soft limit and rerunning our program, we can expect it to open a greater number of files. To accomplish this, we will utilize the ulimit command and specify the -n (open files) option, setting the value as 2048. Consequently, this figure will serve as the updated soft limit.

ulimit -n 2048

This time we successfully opened 2045 files. As expected, this is three less than 2048, because of the file handles used for STDIN , STDOUT , and STDERR.

How to Permanently Change the File Limit

To globally set a new default value for the maximum number of open files a process can have that is persistent and survives reboots, follow these steps:

1. Increasing the soft limit only affects the current shell. Therefore, open a new terminal window and check the soft limit. You'll notice that it still retains the old default value.

2. Out-dated advice used to suggest editing files like "/etc/sysctl.conf" and "/etc/security/limits.conf" for achieving this goal. However, on systemd-based distributions, these edits do not work consistently, particularly for graphical log-in sessions.

To perform this on systemd-based distributions, follow these steps involving two key files. Firstly, navigate to the "/etc/systemd/system.conf" file. To access this file, use the command with sudo privileges:

sudo gedit /etc/systemd/system.conf

To modify the soft limit for processes, locate the line containing "DefaultLimitNOFILE" and remove the "#" at the beginning of the line. Then, change the first number to your desired new limit, such as 4096. Note that the second number represents the hard limit, which we did not modify.

Save the file and close the editor.

We need to repeat that operation on the "/etc/systemd/user.conf" file.

sudo gedit /etc/systemd/user.conf

Make the same adjustments to the line containing the string "DefaultLimitNOFILE."

Save the file and close the editor. To ensure the new settings are applied, either reboot your computer or utilize the systemctl command with the daemon-reexec option to re-execute systemd and incorporate the changes.

sudo systemctl daemon-reexec

Opening a terminal window and checking the new limit should show the new value you set. In our case that was 4096.

ulimit -n

We can test this is a live, operational value by rerunning our file-greedy program.

/open-Files

The program fails to open file number 4094, meaning 4093 were files opened. That's our expected value, 3 less than 4096.

Remember That Everything is a File

That's why Linux is so dependent on file handles. Now, if you start to run out of them, you know how to increase your quota.