Diagnosing too many open files
When facing a “Too Many Open Files” error, it is normally a result of a process taking up too many file descriptors.
To diagnose this, find out the PID of the process which gave out the error.
You can do this with a:
$ ps auwx | grep <name_of_process>
Verify open file limit per process
📝 Note: Each process that runs have a number of open files limit. By default, most of the time the limit is
1024
$ ulimit -a
root@relaye:/home/davidc# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15428
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15428
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Key value to look for is open files
📝 A shortcut you could use is
ulimit -n
Verify the limit of the process that is running
Sometimes a process could have a different limit as the one that is returned by ulimit -n. This is because there exist syscalls that can change a running process’s open files limit (citation needed)
To check for a running process’ actual limit:
$ cat /proc/<pid>/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15428 15428 processes
Max open files 1024 524288 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15428 15428 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Look for the value at Max open files
📝 What is Soft Limit and Hard Limit?
Soft limit is the value that a non-root user can change.
Hard limit is the maximum it can go to. Only root users can change this value.
Verify the number of file descriptors opened
There are two ways to do this.
$ ls /proc/<pid>/fd | wc -l
1024
$ lsof -p <pid> | wc -l
1028
I would recommend using lsof -p <pid> as it will tell you what exactly is the file being opened, so that you can trace and track.