Upgrade error: too many open files

Hi,

I am running nextcloud 29.0.2 on debian 12. Today I wanted to upgrade to 29.0.3 but it fails with message “Too many open files

I am using the command line upgrade process with

sudo -u www-data php --define apc.enable_cli=1 /var/www/nextcloud/./updater/updater.phar

[✔] Check for expected files
[✔] Check for write permissions
[✘] Create backup failed
RecursiveDirectoryIterator::__construct(/var/www/nextcloud/updater/../data/appdata_oco6vecpv4ud/preview/e/f/0/b/e): Failed to open directory: Too many open files

Update failed. To resume or retry just execute the updater again.

When I exceute the script a second time I get a slightly different message:

[ ] Check for expected files ...PHP Warning:  touch(): Unable to create file /var/www/nextcloud/data/updater-oco6vecpv4ud/.step because Too many open files in phar:///var/www/nextcloud/updater/updater.phar/lib/Updater.php on line 1047
PHP Warning:  fopen(/var/www/nextcloud/data/updater.log): Failed to open stream: Too many open files in phar:///var/www/nextcloud/updater/updater.phar/lib/Updater.php on line 1156
[✘] Check for expected files failed
Could not create .step

Update failed. To resume or retry just execute the updater again.

The only location where I have plenty of files is the preview. A few days ago I did a “preview:generate-all” for all my photos. The preview folder became huge > 200 GB.

Can that be an issue? What else can cause a “too many files” error during upgrade?

Maybe check the limit for open files (soft limit and hard limit) for www-data:

sudo -u www-data ulimit -Sn 
sudo - u www-data ulimit -Hn

Then you can change the soft and hard limit for open files. With root edit:

/etc/security/limits.conf

www-data soft nofile 1024
www-data hard nofile 4096

Change the value. Normal users must then log in again. I don’t know how this works for www-data. Maybe check limit for www-data again.

In addition to what @devnull said, your situation is being exasperated due to Updater recurses into datadirectory even though it's later excluded in `createBackup()` · Issue #507 · nextcloud/updater · GitHub.

I would suggest running the Updater, in your case, for the time being with the option --no-backup to skip the internal backup step.

Keep in mind this backup (that I’m suggesting you disable for now) is for the Updater itself internally (mostly) and doesn’t backup many things (like your data) since it’s more meant for the Updater to roll back app code and some manual recovery options of the config file(s). So you should be doing your own backups already anyhow. I only mention this because I’m telling you to use this option to disable backups and I want to avoid misunderstandings. :slight_smile:

1 Like

That did the trick. Thank you!

All other attempts with setting the max files limit did not work. Not the ones @devnull suggested and not other options like setting

DefaultLimitNOFILE=200000

in /etc/systemd/user.conf and /etc/systemd/system.conf

From my point of view this is a bug in the updater.

It is to be expected with big picture databases that the preview cache is growing. My photo folder has more than 80.000 raw files (CR2, RWL,) and more than 110.000 JPG files, not to mention a few PNG and TIFF. I used “preview:generate-all” to generate all possible previews (took a few days). This alone is 200+ GB of data. The updater should be able to handle such a big preview folder. If not, nextcloud needs to make sure that the preview folder is not growing over the limit.

1 Like

I was too early. It failed again. But htis time at he end of the process during cleanup:

[✘] Delete old files failed
RecursiveDirectoryIterator::__construct(/var/www/nextcloud/updater/../data/appdata_oco6vecpv4ud/preview/b/f/f/5/6/f/6/49309): Failed to open directory: Too many open files

Update failed. To resume or retry just execute the updater again.

What next?

What next?

I was afraid you’d likely encounter the same problem in the deletion step. Unfortunately that step can’t be bypassed, unlike the backup one.

First, you can always do a manual update as described in the Admin Manual.

The updater should be able to handle such a big preview folder. If not, nextcloud needs to make sure that the preview folder is not growing over the limit.

The open files limit you’re hitting is still unusual. I was merely providing a way to potentially work around it.

Those limits are a set at the OS level. I’m actually very curious what your limits are that it’s hitting. Or what else it may be that is leading to lots of files being open. There shouldn’t actually be a bunch of open files in those functions to my knowledge.

80,000 is not a ridiculous number. The worst that happens today (if you’re on an officially supported OS/environment) with the backup and delete steps is the Updater gets really slow at those steps. The quantity of files within them does not otherwise matter.

The error you’re seeing is only arising because either somewhere you have a file limit that is unusually low (or) there is something unusual that is causing the Updater to have far more open files than expected.

I run Debian just like you and haven’t encountered open file constraints ever in the Updater.

That said, yes, the automated Updater needs to be optimized so it doesn’t traverse those folders in the way it does currently. That’s a known issue.

Can you share a bit more about your setup? What’s your underlying filesystem? Is either your installation folder or, more likely, your data folder on a network based filesystem perhaps? I’m trying to come up with a good explanation - other than just a really low set of limits - you might be encountering this.

If you re-run the Updater and catch it while it’s in either of those steps, you may be able to capture some interesting information using lsof -p PID -r 30 where PID is the numeric process id of the running Updater process. (The -r 30 specifies how often to update the list). That command will show you precise what files are open.

I tried it again with ulimit changes in /etc/security/limits.conf as @devnull suggested. I did set:

www-data soft nofile 100000

This time the process was running a lot longer than before. The limits seem now to work. But the upgrade still failed with “too many open files”.

Here is the output of the updater script.

# sudo -u www-data php --define apc.enable_cli=1 /var/www/nextcloud/./updater/updater.phar --no-backup

Nextcloud Updater - version: v29.0.0-1-g9293bcb dirty

Current version is 29.0.2.

Update to Nextcloud 29.0.3 available. (channel: "stable")
Following file will be downloaded automatically: https://download.nextcloud.com/server/releases/nextcloud-29.0.3.zip
Open changelog ↗

Steps that will be executed:
[ ] Check for expected files
[ ] Check for write permissions
[ ] Create backup
[ ] Downloading
[ ] Verify integrity
[ ] Extracting
[ ] Enable maintenance mode
[ ] Replace entry points
[ ] Delete old files
[ ] Move new files in place
[ ] Done

Start update? [y/N] y

Info: Pressing Ctrl-C will finish the currently running step and then stops the updater.

[✔] Check for expected files
[✔] Check for write permissions
[✔] Create backup
[✔] Downloading
[✔] Verify integrity
[✔] Extracting
[✔] Enable maintenance mode
[✔] Replace entry points
[✘] Delete old files failed
RecursiveDirectoryIterator::__construct(/var/www/nextcloud/updater/../data/appdata_oco6vecpv4ud/preview/b/6/9/1/6/9/4): Failed to open directory: Too many open files

Update failed. To resume or retry just execute the updater again.

Interstingly, the lsof command did not show a lot of open files. This is the last output of lsof before the upgrade process failed:. It shows only 174 open files.

lsof -p 1635 -r 10
=======
COMMAND  PID     USER   FD      TYPE DEVICE SIZE/OFF    NODE NAME
php     1635 www-data  cwd       DIR  254,1     4096    7954 /root/bin
php     1635 www-data  rtd       DIR  254,1     4096       2 /
php     1635 www-data  txt       REG  254,1  5654232  981508 /usr/bin/php8.2
php     1635 www-data  DEL       REG    0,1             2090 /dev/zero
php     1635 www-data  mem       REG  254,1   118320  979200 /usr/lib/x86_64-linux-gnu/libzip.so.4.0
php     1635 www-data  mem       REG  254,1    92264    7953 /usr/lib/php/20220829/zip.so
php     1635 www-data  mem       REG  254,1   157768  914307 /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.33.1
php     1635 www-data  mem       REG  254,1  1332480  914312 /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.4.1
php     1635 www-data  mem       REG  254,1    96256  959993 /usr/lib/x86_64-linux-gnu/libexslt.so.0.8.20
php     1635 www-data  mem       REG  254,1   268576  959994 /usr/lib/x86_64-linux-gnu/libxslt.so.1.1.35
php     1635 www-data  mem       REG  254,1    34920    8984 /usr/lib/php/20220829/xsl.so
php     1635 www-data  mem       REG  254,1    51304    8983 /usr/lib/php/20220829/xmlwriter.so
php     1635 www-data  mem       REG  254,1    43112    8982 /usr/lib/php/20220829/xmlreader.so
php     1635 www-data  mem       REG  254,1    34920    7064 /usr/lib/php/20220829/tokenizer.so
php     1635 www-data  mem       REG  254,1   333936    7432 /usr/lib/php/20220829/soap.so
php     1635 www-data  mem       REG  254,1   149952  914323 /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.4
php     1635 www-data  mem       REG  254,1   660736    7974 /usr/lib/php/20220829/redis.so
php     1635 www-data  mem       REG  254,1  3307688  936893 /usr/lib/x86_64-linux-gnu/libicui18n.so.72.1
php     1635 www-data  mem       REG  254,1   104552    7060 /usr/lib/php/20220829/sockets.so
php     1635 www-data  mem       REG  254,1   204088  917265 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.4
php     1635 www-data  mem       REG  254,1   216640  921383 /usr/lib/x86_64-linux-gnu/libedit.so.2.0.70
php     1635 www-data  mem       REG  254,1   284776    7057 /usr/lib/php/20220829/phar.so
php     1635 www-data  mem       REG  254,1  2213808  938264 /usr/lib/x86_64-linux-gnu/libfftw3.so.3.6.10
php     1635 www-data  mem       REG  254,1    18536    7063 /usr/lib/php/20220829/sysvshm.so
php     1635 www-data  mem       REG  254,1    14440    7062 /usr/lib/php/20220829/sysvsem.so
php     1635 www-data  mem       REG  254,1    22632    7061 /usr/lib/php/20220829/sysvmsg.so
php     1635 www-data  mem       REG  254,1    59496    8980 /usr/lib/php/20220829/simplexml.so
php     1635 www-data  mem       REG  254,1    14440    7059 /usr/lib/php/20220829/shmop.so
php     1635 www-data  mem       REG  254,1   614016  973100 /usr/lib/x86_64-linux-gnu/libonig.so.5.3.0
php     1635 www-data  mem       REG  254,1  1192704    7421 /usr/lib/php/20220829/mbstring.so
php     1635 www-data  mem       REG  254,1    55536  938701 /usr/lib/x86_64-linux-gnu/liblqr-1.so.0.3.2
php     1635 www-data  mem       REG  254,1   161896    5684 /usr/lib/php/20220829/mysqli.so
php     1635 www-data  mem       REG  254,1   579784    7415 /usr/lib/php/20220829/intl.so
php     1635 www-data  mem       REG  254,1  1273360  937533 /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7400.6
php     1635 www-data  mem       REG  254,1  2820040  939284 /usr/lib/x86_64-linux-gnu/libMagickCore-6.Q16.so.6.0.0
php     1635 www-data  mem       REG  254,1    34920    7434 /usr/lib/php/20220829/readline.so
php     1635 www-data  mem       REG  254,1  1203192  939292 /usr/lib/x86_64-linux-gnu/libMagickWand-6.Q16.so.6.0.0
php     1635 www-data  mem       REG  254,1    22728  938739 /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
php     1635 www-data  mem       REG  254,1    59552  936894 /usr/lib/x86_64-linux-gnu/libicuio.so.72.1
php     1635 www-data  mem       REG  254,1    81568  939018 /usr/lib/x86_64-linux-gnu/libXext.so.6.4.0
php     1635 www-data  mem       REG  254,1   398304  938695 /usr/lib/x86_64-linux-gnu/liblcms2.so.2.0.14
php     1635 www-data  mem       REG  254,1   290392  938261 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
php     1635 www-data  mem       REG  254,1    39016    7058 /usr/lib/php/20220829/posix.so
php     1635 www-data  mem       REG  254,1    34920    5688 /usr/lib/php/20220829/pdo_mysql.so
php     1635 www-data  mem       REG  254,1   452752    5738 /usr/lib/php/20220829/imagick.so
php     1635 www-data  mem       REG  254,1   669624  941304 /usr/lib/x86_64-linux-gnu/libyuv.so.0.0.1857
php     1635 www-data  mem       REG  254,1  6759376  941298 /usr/lib/x86_64-linux-gnu/libSvtAv1Enc.so.1.4.1
php     1635 www-data  mem       REG  254,1    76000    7972 /usr/lib/php/20220829/igbinary.so
php     1635 www-data  mem       REG  254,1    18504  973612 /usr/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
php     1635 www-data  mem       REG  254,1  2701536  941293 /usr/lib/x86_64-linux-gnu/librav1e.so.0.5.1
php     1635 www-data  mem       REG  254,1    39464  938707 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.2
php     1635 www-data  mem       REG  254,1    75880    7411 /usr/lib/php/20220829/gmp.so
php     1635 www-data  mem       REG  254,1    22544  941216 /usr/lib/x86_64-linux-gnu/libabsl_throw_delegate.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    47312  914538 /usr/lib/x86_64-linux-gnu/libmd.so.0.0.5
php     1635 www-data  mem       REG  254,1    22616  941186 /usr/lib/x86_64-linux-gnu/libabsl_int128.so.20220623.0.0
php     1635 www-data  mem       REG  254,1   100528  941218 /usr/lib/x86_64-linux-gnu/libabsl_time_zone.so.20220623.0.0
php     1635 www-data  mem       REG  254,1   120920  941212 /usr/lib/x86_64-linux-gnu/libabsl_strings.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    39080  941168 /usr/lib/x86_64-linux-gnu/libabsl_demangle_internal.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    22632  941167 /usr/lib/x86_64-linux-gnu/libabsl_debugging_internal.so.20220623.0.0
php     1635 www-data  mem       REG  254,1   948240  941287 /usr/lib/x86_64-linux-gnu/libgav1.so.1.0.0
php     1635 www-data  mem       REG  254,1  5643672  938662 /usr/lib/x86_64-linux-gnu/libaom.so.3.6.0
php     1635 www-data  mem       REG  254,1    18512  941213 /usr/lib/x86_64-linux-gnu/libabsl_strings_internal.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    84840  919502 /usr/lib/x86_64-linux-gnu/libbsd.so.0.11.7
php     1635 www-data  mem       REG  254,1    14392  941204 /usr/lib/x86_64-linux-gnu/libabsl_raw_logging_internal.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    14264  941206 /usr/lib/x86_64-linux-gnu/libabsl_spinlock_wait.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    18536  941158 /usr/lib/x86_64-linux-gnu/libabsl_base.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    18600  941190 /usr/lib/x86_64-linux-gnu/libabsl_malloc_internal.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    72088  941217 /usr/lib/x86_64-linux-gnu/libabsl_time.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    59640  941215 /usr/lib/x86_64-linux-gnu/libabsl_synchronization.so.20220623.0.0
php     1635 www-data  mem       REG  254,1   166824  938745 /usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0
php     1635 www-data  mem       REG  254,1  1630208  938668 /usr/lib/x86_64-linux-gnu/libdav1d.so.6.6.0
php     1635 www-data  mem       REG  254,1 16394944  938684 /usr/lib/x86_64-linux-gnu/libx265.so.199
php     1635 www-data  mem       REG  254,1    30808  941214 /usr/lib/x86_64-linux-gnu/libabsl_symbolize.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    14264  941207 /usr/lib/x86_64-linux-gnu/libabsl_stacktrace.so.20220623.0.0
php     1635 www-data  mem       REG  254,1    52312  938679 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
php     1635 www-data  mem       REG  254,1   452776  938673 /usr/lib/x86_64-linux-gnu/libde265.so.0.1.4
php     1635 www-data  mem       REG  254,1   149384  938202 /usr/lib/x86_64-linux-gnu/libdeflate.so.0
php     1635 www-data  mem       REG  254,1    66240  938206 /usr/lib/x86_64-linux-gnu/libjbig.so.0
php     1635 www-data  mem       REG  254,1   616464  938211 /usr/lib/x86_64-linux-gnu/libLerc.so.4
php     1635 www-data  mem       REG  254,1  1318408  939010 /usr/lib/x86_64-linux-gnu/libX11.so.6.4.0
php     1635 www-data  mem       REG  254,1   174216  929684 /usr/lib/x86_64-linux-gnu/libexpat.so.1.8.10
php     1635 www-data  mem       REG  254,1   163128  941309 /usr/lib/x86_64-linux-gnu/libavif.so.15.0.1
php     1635 www-data  mem       REG  254,1   637144  938690 /usr/lib/x86_64-linux-gnu/libheif.so.1.15.1
php     1635 www-data  mem       REG  254,1   453240  938217 /usr/lib/x86_64-linux-gnu/libwebp.so.7.1.5
php     1635 www-data  mem       REG  254,1   567304  938223 /usr/lib/x86_64-linux-gnu/libtiff.so.6.0.0
php     1635 www-data  mem       REG  254,1    81000  941316 /usr/lib/x86_64-linux-gnu/libXpm.so.4.11.0
php     1635 www-data  mem       REG  254,1   600128  938184 /usr/lib/x86_64-linux-gnu/libjpeg.so.62.3.0
php     1635 www-data  mem       REG  254,1   825256  938276 /usr/lib/x86_64-linux-gnu/libfreetype.so.6.18.3
php     1635 www-data  mem       REG  254,1   306112  938656 /usr/lib/x86_64-linux-gnu/libfontconfig.so.1.12.0
php     1635 www-data  mem       REG  254,1  7149728    7052 /usr/lib/php/20220829/fileinfo.so
php     1635 www-data  mem       REG  254,1    26712  941183 /usr/lib/x86_64-linux-gnu/libabsl_graphcycles_internal.so.20220623.0.0
php     1635 www-data  mem       REG  254,1   219056  938191 /usr/lib/x86_64-linux-gnu/libpng16.so.16.39.0
php     1635 www-data  mem       REG  254,1   417992  941322 /usr/lib/x86_64-linux-gnu/libgd.so.3.0.11
php     1635 www-data  mem       REG  254,1    51304    7055 /usr/lib/php/20220829/iconv.so
php     1635 www-data  mem       REG  254,1    14440    7054 /usr/lib/php/20220829/gettext.so
php     1635 www-data  mem       REG  254,1   108648    7407 /usr/lib/php/20220829/gd.so
php     1635 www-data  mem       REG  254,1   174184    7051 /usr/lib/php/20220829/ffi.so
php     1635 www-data  mem       REG  254,1    92264    5702 /usr/lib/php/20220829/exif.so
php     1635 www-data  mem       REG  254,1   194664    7594 /usr/lib/php/20220829/dom.so
php     1635 www-data  mem       REG  254,1    43480  914422 /usr/lib/x86_64-linux-gnu/libffi.so.8.1.2
php     1635 www-data  mem       REG  254,1    60328  915457 /usr/lib/x86_64-linux-gnu/libresolv.so.2
php     1635 www-data  mem       REG  254,1    83968  916144 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.3
php     1635 www-data  mem       REG  254,1  1261376  914428 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
php     1635 www-data  mem       REG  254,1   137376  936840 /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1.0.9
php     1635 www-data  mem       REG  254,1   113392  936815 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
php     1635 www-data  mem       REG  254,1    52224  919509 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
php     1635 www-data  mem       REG  254,1   183048  919514 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
php     1635 www-data  mem       REG  254,1  2197240  914444 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.34.3
php     1635 www-data  mem       REG  254,1    14496  938733 /usr/lib/x86_64-linux-gnu/libXau.so.6.0.0
php     1635 www-data  mem       REG  254,1    22448  919519 /usr/lib/x86_64-linux-gnu/libkeyutils.so.1.10
php     1635 www-data  mem       REG  254,1   888080  919532 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
php     1635 www-data  mem       REG  254,1   529216  914384 /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.1
php     1635 www-data  mem       REG  254,1   317544  914394 /usr/lib/x86_64-linux-gnu/libnettle.so.8.6
php     1635 www-data  mem       REG  254,1   292856  914411 /usr/lib/x86_64-linux-gnu/libhogweed.so.6.6
php     1635 www-data  mem       REG  254,1  1792040  916085 /usr/lib/x86_64-linux-gnu/libunistring.so.2.2.0
php     1635 www-data  mem       REG  254,1    47264  936841 /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1.0.9
php     1635 www-data  mem       REG  254,1   763816  916122 /usr/lib/x86_64-linux-gnu/libzstd.so.1.5.4
php     1635 www-data  mem       REG  254,1    63832  936823 /usr/lib/x86_64-linux-gnu/liblber-2.5.so.0.1.8
php     1635 www-data  mem       REG  254,1   380432  936824 /usr/lib/x86_64-linux-gnu/libldap-2.5.so.0.1.8
php     1635 www-data  mem       REG  254,1   334584  920636 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
php     1635 www-data  mem       REG  254,1    75616  936858 /usr/lib/x86_64-linux-gnu/libpsl.so.5.3.4
php     1635 www-data  mem       REG  254,1   260552  936870 /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
php     1635 www-data  mem       REG  254,1   122280  936864 /usr/lib/x86_64-linux-gnu/librtmp.so.1
php     1635 www-data  mem       REG  254,1   198776  913927 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.8
php     1635 www-data  mem       REG  254,1  3048992  922411 /usr/lib/locale/locale-archive
php     1635 www-data  mem       REG  254,1    18344  915614 /usr/lib/x86_64-linux-gnu/libcom_err.so.2.1
php     1635 www-data  mem       REG  254,1   190928  936851 /usr/lib/x86_64-linux-gnu/libnghttp2.so.14.24.1
php     1635 www-data  mem       REG  254,1   716216  936879 /usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0
php     1635 www-data  mem       REG  254,1   120936    7043 /usr/lib/php/20220829/curl.so
php     1635 www-data  mem       REG  254,1    74688  914303 /usr/lib/x86_64-linux-gnu/libbz2.so.1.0.4
php     1635 www-data  mem       REG  254,1  2190440  916113 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
php     1635 www-data  mem       REG  254,1    71784    7053 /usr/lib/php/20220829/ftp.so
php     1635 www-data  mem       REG  254,1    14440    5701 /usr/lib/php/20220829/ctype.so
php     1635 www-data  mem       REG  254,1    34920    5699 /usr/lib/php/20220829/calendar.so
php     1635 www-data  mem       REG  254,1    26728    7020 /usr/lib/php/20220829/bz2.so
php     1635 www-data  mem       REG  254,1    88168    5727 /usr/lib/php/20220829/apcu.so
php     1635 www-data  mem       REG  254,1    55400    8981 /usr/lib/php/20220829/xml.so
php     1635 www-data  mem       REG  254,1   227024    5685 /usr/lib/php/20220829/mysqlnd.so
php     1635 www-data  mem       REG  254,1  1022856    5691 /usr/lib/php/20220829/opcache.so
php     1635 www-data  mem       REG  254,1   353616  917972 /usr/lib/locale/C.utf8/LC_CTYPE
php     1635 www-data  mem       REG  254,1 31262256  936892 /usr/lib/x86_64-linux-gnu/libicudata.so.72.1
php     1635 www-data  mem       REG  254,1   129128    7056 /usr/lib/php/20220829/pdo.so
php     1635 www-data  mem       REG  254,1   125312  913925 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
php     1635 www-data  mem       REG  254,1    14480  915456 /usr/lib/x86_64-linux-gnu/libpthread.so.0
php     1635 www-data  mem       REG  254,1  2078888  936897 /usr/lib/x86_64-linux-gnu/libicuuc.so.72.1
php     1635 www-data  mem       REG  254,1  1922136  915443 /usr/lib/x86_64-linux-gnu/libc.so.6
php     1635 www-data  mem       REG  254,1  4713752  918285 /usr/lib/x86_64-linux-gnu/libcrypto.so.3
php     1635 www-data  mem       REG  254,1   190456  914329 /usr/lib/x86_64-linux-gnu/liblzma.so.5.4.1
php     1635 www-data  mem       REG  254,1    34880  918323 /usr/lib/x86_64-linux-gnu/libargon2.so.1
php     1635 www-data  mem       REG  254,1   363208  971023 /usr/lib/x86_64-linux-gnu/libsodium.so.23.3.0
php     1635 www-data  mem       REG  254,1   121280  916108 /usr/lib/x86_64-linux-gnu/libz.so.1.2.13
php     1635 www-data  mem       REG  254,1   629384  914545 /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.11.2
php     1635 www-data  mem       REG  254,1   696352  918286 /usr/lib/x86_64-linux-gnu/libssl.so.3
php     1635 www-data  mem       REG  254,1  1750104  936908 /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.14
php     1635 www-data  mem       REG  254,1   907784  915447 /usr/lib/x86_64-linux-gnu/libm.so.6
php     1635 www-data  mem       REG  254,1    39024    7016 /usr/lib/php/20220829/bcmath.so
php     1635 www-data  mem       REG  254,1    27028  929952 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
php     1635 www-data  mem       REG  254,1   210976  915440 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
php     1635 www-data    0u      CHR  136,2      0t0       5 /dev/pts/2
php     1635 www-data    1u      CHR  136,2      0t0       5 /dev/pts/2
php     1635 www-data    2u      CHR  136,2      0t0       5 /dev/pts/2
php     1635 www-data    3r      REG   0,39  1172794    2897 /var/www/nextcloud/updater/updater.phar
php     1635 www-data    4r      DIR   0,39       30     384 /var/www/nextcloud
php     1635 www-data    5r      DIR   0,39       56   28942 /var/www/nextcloud/data
php     1635 www-data    6r      DIR   0,39       13   25104 /var/www/nextcloud/data/appdata_oco6vecpv4ud
php     1635 www-data    7r      DIR   0,39       18   36498 /var/www/nextcloud/data/appdata_oco6vecpv4ud/preview
php     1635 www-data    8r      DIR   0,39       18   34623 /var/www/nextcloud/data/appdata_oco6vecpv4ud/preview/b
php     1635 www-data    9r      DIR   0,39       18   40364 /var/www/nextcloud/data/appdata_oco6vecpv4ud/preview/b/6
php     1635 www-data   10r      DIR   0,39       17   56852 /var/www/nextcloud/data/appdata_oco6vecpv4ud/preview/b/6/c
php     1635 www-data   11r      DIR   0,39        5  255604 /var/www/nextcloud/data/appdata_oco6vecpv4ud/preview/b/6/c/c
php     1635 www-data   12r      DIR   0,39        3 1041286 /var/www/nextcloud/data/appdata_oco6vecpv4ud/preview/b/6/c/c/4
php     1635 www-data   13   unknown                         /proc/1635/fd/13 (readlink: No such file or directory)
php     1635 www-data   14   unknown                         /proc/1635/fd/14 (readlink: No such file or directory)
php     1635 www-data   15   unknown                         /proc/1635/fd/15 (readlink: No such file or directory)
=======

Out of curiosity I counted the files in the preview folder. It is 1.514.811. Wow!

I did set soft limit to 1.000.000 and that still was no sufficient. I can not go higher because the hard limit is 1.048.576 and I can not do more in /etc/security/limits.conf.

Anyhow, I do not understand why the updater is actually concerned about the preview folder.

I am lost.

I wonder why the files all have to be open at the same time. Perhaps someone can explain this.

Something really odd is happening here. Some more findings:

First of all, it is not sufficient to set the soft limit for user www-data. When I check the limits of the updater process with cat /proc/PID/limits I still get the default of 1024. See here for PID 1633:

╭─[root@nxcloud] /root
╰─# ps -ef | grep upda   
root        1631    1630  0 09:09 pts/1    00:00:00 sudo -u www-data php --define apc.enable_cli=1 /var/www/nextcloud/./updater/updater.phar --no-backup
root        1632    1631  0 09:09 pts/4    00:00:00 sudo -u www-data php --define apc.enable_cli=1 /var/www/nextcloud/./updater/updater.phar --no-backup
www-data    1633    1632  1 09:09 pts/4    00:00:00 php --define apc.enable_cli=1 /var/www/nextcloud/./updater/updater.phar --no-backup
root        1667    1611  0 09:10 pts/3    00:00:00 grep --color upda
                                                                                                                                          
╭─[root@nxcloud] /root
╰─# cat /proc/1633/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    0                    bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             31541                31541                processes 
Max open files            1024                 1048576              files     
Max locked memory         1040805888           1040805888           bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       31541                31541                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

Somehow the process inherits the settings from root. Only if I change the root limit to lets say 1000000 it is applied to the updater script as well. See here for PID=2108

╰─# ps -ef | grep updat
root        2106    2105  0 10:41 pts/1    00:00:00 sudo -u www-data php --define apc.enable_cli=1 /var/www/nextcloud/./updater/updater.phar --no-backup
root        2107    2106  0 10:41 pts/2    00:00:00 sudo -u www-data php --define apc.enable_cli=1 /var/www/nextcloud/./updater/updater.phar --no-backup
www-data    2108    2107 22 10:41 pts/2    00:00:03 php --define apc.enable_cli=1 /var/www/nextcloud/./updater/updater.phar --no-backup
root        2120    1547  0 10:41 pts/4    00:00:00 grep --color updat
                                                                                                                                    
╭─[root@nxcloud] /root
╰─# cat /proc/2108/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    0                    bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             31541                31541                processes 
Max open files            1000000              1048576              files     
Max locked memory         1040805888           1040805888           bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       31541                31541                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

At the end I was setting the following limits in /etc/security/limits.conf just to be on the save side:

root soft nofile 1000000
*    soft nofile 1000000

Then I was wondering why lsof -p 2108 is only showing me a few hundred files. So I monitored all open files during the update process with a plain lsof

The maximum number of open files lsof is showing is 28.000. php has less than 500 open. This is way below any limits.

That leaves two questions:

  1. Why is the upgrade process complaining about too many open files, when the actual number of open files is below any limits?

  2. Why is the upgrade process descending into the preview folder? That does not seem necessary.

In the meantime I succeeded with the the update. I just moved the preview folder out of the way, to /var/www. Luckily this is the same partition so that it just took a second. With that the update process went through without issue.

This is only a dirty workaround from my point of view. And I am wondering if I am missing something important when doing that.

The questions from my previous post remain valid.

1 Like
  1. Why is the upgrade process complaining about too many open files, when the actual number of open files is below any limits?

It’s not. The OS is. That message is coming via PHP and being passed along from the underlying system call.

I agree something odd is going on here. That’s why I ask before: Can you share a bit more about your setup? What’s your underlying filesystem?

The only other odd thing I noticed is your run command:

# sudo -u www-data php --define apc.enable_cli=1 /var/www/nextcloud/./updater/updater.phar --no-backup

Is there a reason you are adding /./ in there? At best it’s a no-op, but I suppose it could create some weirdness with some of the “determine what directory I’m running in” logic.

  1. Why is the upgrade process descending into the preview folder? That does not seem necessary.

It’s not necessary. That’s the aforementioned known issue already previously linked to above. It’s a known problem. But that traversal is a performance matter. It does not create the situation you’re seeing (though it may exasperate it, but we don’t know what it is just yet in your case hah).

For the record, unless you’re on extremely limited hardware (CPU-wise), it is not necessary to use the optional previewgenerator app and your pre-generate previews (including all the variations) ahead of time. You may be able to just use the on demand generation (which will also still cache generated images).

It is a virtual machine (kvm, libvirt). The host is endeavouros, the guest is a fresh install of debian 12 and nextcloud. The nextcloud directory /var/www/nextcloud is on the hosts zfs RAID10, mounted by the debian VM with virtiofs.

There is not reason. I never saw this. This is probably a left over from a copy&paste activity. I will remove it.

I don’t know whether it helps or whether it makes sense. But you can also completely deactivate previews in config.php.

'enable_previews' => false,

Previews configuration

I also did a

sudo -u www-data php /var/www/nextcloud/occ files:scan-app-data

to ensure that the preview / cache is up to date. After several minutes it errors out with “Too many open files”

Scanning AppData for files

Exception during scan: opendir(/var/www/nextcloud/data/appdata_oco6vecpv4ud/preview/b/c/d/b/e): Failed to open directory: Too many open files
#0 [internal function]: OCA\Files\Command\ScanAppData->exceptionErrorHandler()
#1 /var/www/nextcloud/lib/private/Files/Storage/Local.php(164): opendir()
#2 /var/www/nextcloud/lib/private/Files/Storage/Common.php(903): OC\Files\Storage\Local->opendir()
#3 [internal function]: OC\Files\Storage\Common->getDirectoryContent()
#4 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(496): iterator_to_array()
#5 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(453): OC\Files\Cache\Scanner->handleChildren()
#6 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(458): OC\Files\Cache\Scanner->scanChildren()
#7 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(458): OC\Files\Cache\Scanner->scanChildren()
#8 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(458): OC\Files\Cache\Scanner->scanChildren()
#9 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(458): OC\Files\Cache\Scanner->scanChildren()
#10 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(458): OC\Files\Cache\Scanner->scanChildren()
#11 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(458): OC\Files\Cache\Scanner->scanChildren()
#12 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(359): OC\Files\Cache\Scanner->scanChildren()
#13 /var/www/nextcloud/lib/private/Files/Cache/LocalRootScanner.php(39): OC\Files\Cache\Scanner->scan()
#14 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(278): OC\Files\Cache\LocalRootScanner->scan()
#15 /var/www/nextcloud/apps/files/lib/Command/ScanAppData.php(121): OC\Files\Utils\Scanner->scan()
#16 /var/www/nextcloud/apps/files/lib/Command/ScanAppData.php(156): OCA\Files\Command\ScanAppData->scanFiles()
#17 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\ScanAppData->execute()
#18 /var/www/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()
#19 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run()
#20 /var/www/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()
#21 /var/www/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
#22 /var/www/nextcloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run()
#23 /var/www/nextcloud/console.php(102): OC\Console\Application->run()
#24 /var/www/nextcloud/occ(11): require_once('...')
#25 {main}

To me this looks like the files in the preview are opened but never closed.

I am running nextcloud 29.0.2 on debian 12. Today I wanted to upgrade to 29.0.3 but it fails with message “Too many open files

It is a virtual machine (kvm, libvirt). The host is endeavouros, the guest is a fresh install of debian 12 and nextcloud. The nextcloud directory /var/www/nextcloud is on the hosts zfs RAID10, mounted by the debian VM with virtiofs.

Then I was wondering why lsof -p 2108 is only showing me a few hundred files. So I monitored all open files during the update process with a plain lsof

The maximum number of open files lsof is showing is 28.000. php has less than 500 open. This is way below any limits.

To me this looks like the files in the preview are opened but never closed.

Or just the same underlying cause as whatever is triggering this in your environment with the Updater too. :slight_smile:

None of the evidence suggested that at the Linux kernel level (in your guest) that PHP even had that many files open in your environment with the Updater. If I had to guess, it’ll be similar for the scan.

At the moment my money is on something related to virtiofs or in that ballpark. That is, the errors aren’t coming from the guest, but upstream / due to the virtual filesystem layer.

Unless there has been some recent regression, this doesn’t happen in a standard Debian/Linux environment. I’ll try to get some time soon to do some manual testing, but I don’t know when that’ll be.

I finally know the root cause of this issue and how to fix it.

When answering the questions from @jtr about the underlying filesystem it came to my mind that the issue might have something to do with the virtiofs which I am using to import the /var/www directory in the nextcloud VM. And that it is, indeed.

The “Too many open files” error is not coming from the nextcloud VM. Thats the reason why it did not make any difference when I changed ulimit values in the nextcloud VM.

The “Too many open files” error is coming from the host. The host is reporting this error to the VM and the VM is killing the upgrade process accordingly with the proper message.

When running the upgrade process I see the list of open files on the host growing well above 1 million before it errors out. The reason is the virtiofsd on the host which is caching all the file requests from the VM by default.

The solution is, to turn off caching for the shared folders. This is easily done in the XML configuration of the VM by adding <cache mode="none"/> to the XML configuration. Like this:

  <binary path="/usr/lib/virtiofsd">
    <cache mode="none"/>
  </binary>

That is it. When I did this, the upgrade process was running fine until the end, including backup - although the “backup” and “delete old files” steps took roughly 90 min each to complete.

I hope other people with nextcloud on kvm find this info helpful.

1 Like

Awesome!

Glad to hear it is working! Definitely interesting.

Thanks for posting the solution you discovered.