Troubles with nextcloudcmd: "Too many open files"

đŸ“± Desktop & mobile clients đŸ’» Desktop
1

Background
The device is a Raspberry Pi 3 with armhf architecture (not arm64) running Debian Bookworm as a headless server and fully updated. It has a fully functioning Nextcloud server which is mounted on a 3TiB external drive as a ext4 Logical Volume at /var/www/html/nextcloud/data This cloud is successfully sync’ing about 80GiB from another device on my LAN. The cloud uses MySQL as the database.

Aim
On the same server at /var/www/html/joomla i have my website development work. I want to sync this work (about 108MiB) into the cloud at /var/www/html/nextcloud/data/joomla

Process
The first difficulty was finding nextcloudcmd. It is in the Debian package nextcloud-desktop-cmd Unfortunately that package is not in the Raspberry Pi repositories so a simple apt install was not possible. I had to download the suitable package from the Debian repository. Using dpkg -i to install the .deb file presented about 6 unresolved dependencies and suggested dpkg --fix-missing install as the solution. All that achieved was to remove the initial .deb from the cache. Eventually I was able to manually install the missing dependencies and persuade dpkg to install the deb and then install all the remaining dependencies (about 30 many of which look as if they are intended for a GUI environment). The end result was that was finally able to run nextcloudcmd.

Running nextcloudcmd
I should comment here that I do have group permissions into the www-data group for the /var/www/html branches, so there is not a permissions problem.
The command I ran was:

nextcloudcmd -u joomla -pxxxxx /var/www/html/joomla https://howard.id.au/nextcloud

This produced a massive amount of screen output (I was running ssh onto the server) but the upshot was that it only transferred 32MiB of the expected 108MiB. The last few lines of the output are:

04-20 12:12:48:946 [ warning nextcloud.sync.propagator.bulkupload ]: Could not prepare upload device: “Too many open files”
04-20 12:12:48:946 [ info nextcloud.sync.propagator.bulkupload ]: Item completed “libraries/src/Form/Field/CalendarField.php” OCC::SyncFileItem::NormalError CSyncEnums::CSYNC_INSTRUCTION_NEW “Too many open files”
04-20 12:12:48:946 [ warning nextcloud.sync.propagator ]: Could not complete propagation of “libraries/src/Form/Field/CalendarField.php” by OCC::BulkPropagatorJob(0x6c2e00) with status OCC::SyncFileItem::NormalError and error: “Too many open files”
04-20 12:12:48:947 [ info nextcloud.sync.propagator.bulkupload ]: modify final status NormalError OCC::SyncFileItem::NormalError OCC::SyncFileItem::NormalError
04-20 12:12:48:948 [ info nextcloud.sync.propagator.root.directory ]: OCC::SyncFileItem::NormalError slotSubJobsFinished OCC::PropagatorJob::Running pending uploads 0 subjobs state OCC::PropagatorJob::Finished
04-20 12:12:48:948 [ info nextcloud.sync.propagator ]: PropagateRootDirectory::slotSubJobsFinished emit finished OCC::SyncFileItem::NormalError
04-20 12:12:49:514 [ info nextcloud.sync.engine ]: Sync run took 72191 ms
04-20 12:12:51:251 [ warning default ]: Another sync is needed, but not done because restart count is exceeded 3
04-20 12:12:51:280 [ info nextcloud.sync.database ]: Closing DB “/var/www/html/joomla/.sync_191779cac248.db”

I notice that nextcloudcmb works with a SQLite database but only 2 tables in that database have much significance - uploadinfo and metadata. This makes me assume that the reference to “Too many open files” must be from the MySQL database.
I guess my question is twofold:

  1. Why am I getting this problem?
  2. How do I fix it?

TIA,

1 Like
2

It seems nextcloudcmd tries to ingest a huge amount of files at once, which is a terrible idea (especially on HDD):

# ps fauxwww |grep nextcloudcmd
st-at    2643300 95.3  0.1 1521300 59924 pts/1   Rl+  13:39   5:52              \_ nextcloudcmd --path ...
# ls -l /proc/2643300/fd|wc -l
3121

It also has abysmal error handling (the reaction to this kind of error should be to wait until the worker threads have freed file descriptors again).
I was able to get it to continue by increasing ulimit -n from 1024 to 65536, that got it to proceed, but it eventually aborted with OCC::SyncFileItem::NormalError CSyncEnums::CSYNC_INSTRUCTION_NEW "Network error: 99", which I guess is the other side running into the open file limit.

3

Try
‘bulkupload.enabled’ => false,
in config.php

4

I don’t have access to the nextcloud server, it’s a managed instance at hetzner, so I can’t edit the server config myself.
However I’m trying to build a client without bulkupload right now. Wasn’t easy to get it to compile at all - someone changed the default Qt to 6 but didn’t change the install & compile documentation, and with branch debian/dist/bookworm/stable-3.14 some dependencies were missing as well: libqt5webkit5-dev needs to be installed, and in /usr/local/src/nextcloud/desktop/src/libsync/networkjobs.cpp, “#include ” needs to be added.
Then I found that branch is totally obsolete and switched to stable-3.13 instead (because that still has Qt5). .
That required adding even more dependencies, libkf5archive-dev, libkf5guiaddons-dev and - just to generate some icons from svg - inkscape.
Now I’m letting it run before trying anything else.