AIO in VM: BORG backup breaks with [Errno 24] No File descriptors available

Support intro

Sorry to hear you’re facing problems. :slightly_frowning_face:

The community help forum (help.nextcloud.com) is for home and non-enterprise users. Support is provided by other community members on a best effort / “as available” basis. All of those responding are volunteering their time to help you.

If you’re using Nextcloud in a business/critical setting, paid and SLA-based support services can be accessed via portal.nextcloud.com where Nextcloud engineers can help ensure your business keeps running smoothly.

Getting help

In order to help you as efficiently (and quickly!) as possible, please fill in as much of the below requested information as you can.

Before clicking submit: Please check if your query is already addressed via the following resources:

(Utilizing these existing resources is typically faster. It also helps reduce the load on our generous volunteers while elevating the signal to noise ratio of the forums otherwise arising from the same queries being posted repeatedly).

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can. :heart:

The Basics

* Nextcloud Server version _(e.g., 29.x.x)_: 
  - **[Nextcloud Hub 9](https://nextcloud.com) (30.0.10)**
* Operating system and version _(e.g., Ubuntu 24.04)_:
  - Linux nextcloud 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
* Web server and version _(e.g, Apache 2.4.25)_: 
  - `replace me`
* Reverse proxy and version _(e.g. nginx 1.27.2)
  - `replace me`
* PHP version _(e.g, 8.3)_:
  - `replace me`
* Is this the first time you've seen this error? (Yes / No):
  - No
* When did this problem seem to first start?
   - long time - but did not check all logs 
* Installation method _(e.g. AlO, NCP, Bare Metal/Archive, etc.)_
   - host: Ubunut 22 + VM with AIO image
* Are you using CloudfIare, mod_security, or similar? _(Yes / No)_
  - no

Summary of the issue you are facing:

Borg backup breaks with: [Errno 24] No File descriptors available

I do not know how to further debug to find the root cause. My guess is that somewhere a ulimit is wrong. That could be on host level or inside VM (AIO runs in VM).

I only see errors in borg logfile. No other container shows errors or helpful messages. Also on syslog level (host or VM) I cannot find corresponding messages.

my setup is:
UBUNTU server + AIO in VM
Nextcloud directory configured at host and is mounted into VM

all disks are below 40% usage , inodes are below 1% usage

Is there a documentation available specifying system limits for AIO?

limits - inside AIO VM user www-data:

root@nextcloud:~# su www-data --shell /bin/bash --command "ulimit -aH"
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 256766
max locked memory           (kbytes, -l) 8230512
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1048576
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 256766
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

limits - inside AIO VM user root:

root@nextcloud:~# ulimit -aH
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 256766
max locked memory           (kbytes, -l) 8230512
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1048576
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 256766
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

Steps to replicate it (hint: details matter!):

  1. start backup at AIO interface

Log entries

borg log file:

...
B O 87.40 MB C 82.33 MB D 2 N nextcloud_aio_volu...ump/database-dump.sql
B O 90.39 MB C 84.91 MB D 2 N nextcloud_aio_volu...ump/database-dump.sql
ud_aio_volumes/nextcloud_aio_nextcloud_data/appdata_oc14i1aboikj/preview/1/9/c/a/1/4/e/36: scandir: [Errno 24] No file descriptors available: 17
2025-05-02T11:51:57.931977708Z /nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/appdata_oc14i1aboikj/preview/1/9/c/7/f/2/9/62129: scandir: [Errno 24] No file descriptors available: 17
2025-05-02T11:51:57.933326171Z /nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/appdata_oc14i1aboikj/preview/1/9/c/6/f/8/9/86269: scandir: [Errno 24] No file descriptors available: 17
...

some more lines like this and finally traceback crash:

2025-05-02T11:52:00.450273134Z /nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/appdata_oc14i1aboikj/preview/c/5/1/d/d/4/8/86270: scandir: [Errno 24] No file descriptors available: 17
2025-05-02T11:52:00.451799548Z /nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/appdata_oc14i1aboikj/preview/c/5/1/c/e/4/1/13/256-181-max.png: stat: [Errno 24] No file descriptors available: '256-181-max.png'
2025-05-02T11:52:00.761786181Z Local Exception
2025-05-02T11:52:00.761823601Z Traceback (most recent call last):
2025-05-02T11:52:00.761828901Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 5391, in main
2025-05-02T11:52:00.761833061Z     exit_code = archiver.run(args)
2025-05-02T11:52:00.761836832Z                 ^^^^^^^^^^^^^^^^^^
2025-05-02T11:52:00.761840552Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 5309, in run
2025-05-02T11:52:00.761844362Z     rc = func(args)
2025-05-02T11:52:00.761848032Z          ^^^^^^^^^^
2025-05-02T11:52:00.761851682Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 191, in wrapper
2025-05-02T11:52:00.761855472Z     return method(self, args, repository=repository, **kwargs)
2025-05-02T11:52:00.761859272Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02T11:52:00.761863012Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 687, in do_create
2025-05-02T11:52:00.761866772Z     create_inner(archive, cache, fso)
2025-05-02T11:52:00.761870432Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 619, in create_inner
2025-05-02T11:52:00.761874222Z     self._rec_walk(path=path, parent_fd=None, name=None,
2025-05-02T11:52:00.761877902Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 834, in _rec_walk
2025-05-02T11:52:00.761881652Z     self._rec_walk(
2025-05-02T11:52:00.761885282Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 834, in _rec_walk
2025-05-02T11:52:00.761898662Z     self._rec_walk(
2025-05-02T11:52:00.761902552Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 834, in _rec_walk
2025-05-02T11:52:00.761906282Z     self._rec_walk(
2025-05-02T11:52:00.761909902Z   [Previous line repeated 9 more times]
2025-05-02T11:52:00.761913552Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 797, in _rec_walk
2025-05-02T11:52:00.761917302Z     status = self._process_any(path=path, parent_fd=parent_fd, name=name, st=st, fso=fso, cache=cache,
2025-05-02T11:52:00.761921032Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02T11:52:00.761924752Z   File "/usr/lib/python3.12/site-packages/borg/archiver.py", line 699, in _process_any
2025-05-02T11:52:00.761928622Z     return fso.process_file(path=path, parent_fd=parent_fd, name=name, st=st, cache=cache, strip_prefix=strip_prefix)
2025-05-02T11:52:00.761932372Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02T11:52:00.761936122Z   File "/usr/lib/python3.12/site-packages/borg/archive.py", line 1426, in process_file
2025-05-02T11:52:00.761939912Z     with OsOpen(path=path, parent_fd=parent_fd, name=name, flags=flags, noatime=True) as fd:
2025-05-02T11:52:00.761943862Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02T11:52:00.761948022Z   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
2025-05-02T11:52:00.761951872Z     next(self.gen)
2025-05-02T11:52:00.761955492Z   File "/usr/lib/python3.12/site-packages/borg/archive.py", line 255, in OsOpen
2025-05-02T11:52:00.761959382Z     os.close(fd)
2025-05-02T11:52:00.761963022Z OSError: [Errno 9] Bad file descriptor
2025-05-02T11:52:00.761966712Z 
2025-05-02T11:52:00.761970372Z Platform: Linux 43824f97b366 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64
2025-05-02T11:52:00.761974152Z Linux: Unknown Linux  
2025-05-02T11:52:00.761977782Z Borg: 1.4.0  Python: CPython 3.12.10 msgpack: 1.0.8 fuse: llfuse 1.5.1 [pyfuse3,llfuse]
2025-05-02T11:52:00.761981492Z PID: 37  CWD: /
2025-05-02T11:52:00.761985332Z sys.argv: ['/usr/bin/borg', 'create', '-v', '--stats', '--compression', 'auto,zstd', '--progress', '--exclude', '/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/nextcloud.log*', '--exclude', '/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/audit.log', '--exclude', '/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/lost+found', '::20250502_115134-nextcloud-aio', '/nextcloud_aio_volumes/', '--exclude-from', '/borg_excludes']
2025-05-02T11:52:00.761989692Z SSH_ORIGINAL_COMMAND: None
2025-05-02T11:52:00.761993352Z 
2025-05-02T11:52:00.868185314Z Deleting the failed backup archive...

Nextcloud

Please provide the log entries from your Nextcloud log that are generated during the time of problem (via the Copy raw option from Administration settings->Logging screen or from your nextcloud.log located in your data directory). Feel free to use a pastebin/gist service if necessary.

nothing found in logs here

Web Browser

If the problem is related to the Web interface, open your browser inspector Console and Network tabs while refreshing (reloading) and reproducing the problem. Provide any relevant output/errors here that appear.

PASTE

Web server / Reverse Proxy

The output of your Apache/nginx/system log in /var/log/____:

PASTE HERE

Configuration

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):


dcf5067cdefd:/var/www/html# sudo -u www-data -E php occ config:list system
{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "huhn.hzs10.ovh",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "huhn.hzs10.ovh"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "30.0.10.1",
        "overwrite.cli.url": "https:\/\/huhn.hzs10.ovh\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "loglevel": "2",
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "htaccess.RewriteBase": "\/",
        "files_external_allow_create_new_local": false,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "allow_local_remote_servers": true,
        "maintenance": false,
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "updatedirectory": "\/nc-updater",
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "davstorage.request_timeout": 3600,
        "dbpersistent": false,
        "appsallowlist": false,
        "maintenance_window_start": 100,
        "preview_imaginary_key": "***REMOVED SENSITIVE VALUE***",
        "auth.bruteforce.protection.enabled": true,
        "ratelimit.protection.enabled": true,
        "documentation_url.server_logs": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/5425",
        "DOMAIN": "huhn.hzs10.ovh"
    }
}


Apps

The output of occ app:list (if possible).

Tips for increasing the likelihood of a response

  • Use the preformatted text formatting option in the editor for all log entries and configuration output.
  • If screenshots are useful, feel free to include them.
    • If possible, also include key error output in text form so it can be searched for.
  • Try to edit log output only minimally (if at all) so that it can be ran through analyzers / formatters by those trying to help you.

After deleting a large number of files and directories with special characters in the name the error is gone.

For non-UTF8 files names I have seen other errors before and it does not really makes sense to me.

As of now this particular error disappeared and I will close the ticket.