Vitual file download failed with code "403", Status "503"

Support intro

Sorry to hear you’re facing problems. :slightly_frowning_face:

The community help forum (help.nextcloud.com) is for home and non-enterprise users. Support is provided by other community members on a best effort / “as available” basis. All of those responding are volunteering their time to help you.

If you’re using Nextcloud in a business/critical setting, paid and SLA-based support services can be accessed via portal.nextcloud.com where Nextcloud engineers can help ensure your business keeps running smoothly.

Getting help

In order to help you as efficiently (and quickly!) as possible, please fill in as much of the below requested information as you can.

Before clicking submit: Please check if your query is already addressed via the following resources:

(Utilizing these existing resources is typically faster. It also helps reduce the load on our generous volunteers while elevating the signal to noise ratio of the forums otherwise arising from the same queries being posted repeatedly).

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can. :heart:

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    30.0.4 (Upgraded from 30.0.2)

  • Operating system and version (e.g., Ubuntu 24.04):
    Debian 12 ( Linux 6.1.0-27-amd64 x86_64)

  • Web server and version (e.g, Apache 2.4.25):
    Apache/2.4.62 (Unix) (fpm-fcgi)

  • Reverse proxy and version _(e.g. nginx 1.27.2)
    None

  • PHP version (e.g, 8.3):
    8.3.14

  • Is this the first time you’ve seen this error? (Yes / No):
    Yes

  • When did this problem seem to first start?
    After upgrading from 30.0.2 to 30.0.4 AIO via AIO Web interface.

  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    AIO

  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    No

Summary of the issue you are facing:

Users can download and sync files on their “Nextcloud” files, but they cannot for files accessed via external storage.

Upgrade threw a php memory allocation error at line 159. It appeared to be a re-indexing issue. I manually indexed using the
occ files:scan --all
After that Nextcloud came up. Users can see files in client and web from External storage, but cannot download via either method. Files not in external storage work fine.

We are checking permissions, but they look good. Configuration shows everything working. except WebDAV service.

Steps to replicate it (hint: details matter!):

Log entries

Nextcloud

{"reqId":"dC1F3JkJxEj793WSEtHU","level":3,"time":"2024-12-27T15:50:39+00:00","remoteAddr":"172.22.250.4","user":"Vizo","app":"webdav","method":"GET","url":"/remote.php/dav/files/Vizo/IT%20Share/3CX/Remove3CX1.ps1","message":"Could not open file","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0","version":"30.0.4.1","exception":{"Exception":"Sabre\\DAV\\Exception\\ServiceUnavailable","Message":"Could not open file","Code":0,"Trace":[{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":85,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/html/apps/dav/lib/Connector/Sabre/Server.php","line":43,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":371,"function":"start","class":"OCA\\DAV\\Connector\\Sabre\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":19,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":146,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","Line":463,"message":"Could not open file","exception":[],"CustomMessage":"Could not open file"},"id":"676ed03248d62"}

Web Browser

Firefox, Chrome, Edge : Windows/Linux

PASTE

Web server / Reverse Proxy

No proxy

Configuration

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):

{
    "system": {
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "check_data_directory_permissions": false,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "nextcloud.iccchem.com",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "nextcloud.iccchem.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "30.0.4.1",
        "overwrite.cli.url": "https:\/\/nextcloud.iccchem.com\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "updatedirectory": "\/nc-updater",
        "loglevel": 2,
        "app_install_overwrite": [
            "nextcloud-aio"
        ],
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": 10485760,
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": 2048,
        "preview_max_y": 2048,
        "jpeg_quality": 60,
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie",
            "7": "OC\\Preview\\Krita",
            "0": "OC\\Preview\\Imaginary",
            "23": "OC\\Preview\\ImaginaryPDF"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": 30,
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "maintenance_window_start": 100,
        "allow_local_remote_servers": true,
        "davstorage.request_timeout": 3600,
        "htaccess.RewriteBase": "\/",
        "dbpersistent": false,
        "auth.bruteforce.protection.enabled": true,
        "ratelimit.protection.enabled": true,
        "files_external_allow_create_new_local": false,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
        "preview_imaginary_key": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "documentation_url.server_logs": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/5425",
        "DOMAIN": "nextcloud.iccchem.com"
    }
}

Apps

The output of occ app:list (if possible).
Enabled:

  • activity: 3.0.0
  • admin_audit: 1.20.0
  • app_api: 4.0.3
  • bruteforcesettings: 3.0.0
  • circles: 30.0.0
  • cloud_federation_api: 1.13.0
  • comments: 1.20.1
  • contactsinteraction: 1.11.0
  • dashboard: 7.10.0
  • dav: 1.31.1
  • federatedfilesharing: 1.20.0
  • federation: 1.20.0
  • files: 2.2.0
  • files_antivirus: 5.6.1
  • files_downloadlimit: 3.0.0
  • files_external: 1.22.0
  • files_fulltextsearch: 30.0.0
  • files_pdfviewer: 3.0.0
  • files_reminders: 1.3.0
  • files_sharing: 1.22.0
  • files_trashbin: 1.20.1
  • files_versions: 1.23.0
  • firstrunwizard: 3.0.0
  • fulltextsearch: 30.0.0
  • fulltextsearch_elasticsearch: 30.0.0
  • logreader: 3.0.0
  • lookup_server_connector: 1.18.0
  • nextcloud-aio: 0.7.0
  • nextcloud_announcements: 2.0.0
  • notifications: 3.0.0
  • notify_push: 0.7.0
  • oauth2: 1.18.1
  • password_policy: 2.0.0
  • photos: 3.0.2
  • privacy: 2.0.0
  • provisioning_api: 1.20.0
  • recommendations: 3.0.0
  • related_resources: 1.5.0
  • richdocuments: 8.5.3
  • serverinfo: 2.0.0
  • settings: 1.13.0
  • sharebymail: 1.20.0
  • spreed: 20.1.1
  • support: 2.0.0
  • survey_client: 2.0.0
  • suspicious_login: 8.0.0
  • systemtags: 1.20.0
  • text: 4.1.0
  • theming: 2.5.0
  • twofactor_backupcodes: 1.19.0
  • twofactor_totp: 12.0.0-dev
  • user_ldap: 1.21.0
  • user_status: 1.10.0
  • viewer: 3.0.0
  • weather_status: 1.10.0
  • webhook_listeners: 1.1.0-dev
  • workflowengine: 2.12.0
    Disabled:
  • encryption: 2.18.0
  • twofactor_nextcloud_notification: 4.0.0

Tips for increasing the likelihood of a response

  • Use the preformatted text formatting option in the editor for all log entries and configuration output.
  • If screenshots are useful, feel free to include them.
    • If possible, also include key error output in text form so it can be searched for.
  • Try to edit log output only minimally (if at all) so that it can be ran through analyzers / formatters by those trying to help you.

A brute force workaround we have discovered is to remove any defined external storage, and re-create it. For some reason this appears to be working, but I have no idea why. My guess is this is related to editing of the storage that I performed (removed the case sensitive option). Recreating appears to resolve. That said, I have to backup any modified local files on the client before the change as it removes the original share, and creates a new one.

Upgrade threw a php memory allocation error at line 159. It appeared to be a re-indexing issue. I manually indexed using the
occ files:scan --all

Do you have the actual error message? An upgrade does not typically trigger a file scan.

We are checking permissions, but they look good. Configuration shows everything working. except WebDAV service.

Your problem seems mostly centered around External Storage but you didn’t really give us any info about your actual External Storage setup. :slight_smile: Perhaps you can share a bit more about what you’ve got setup (i.e. type you’re using, the configuration associated with it in Nextcloud, and what’s on the other end).

My guess is this is related to editing of the storage that I performed (removed the case sensitive option).

So your External Storage mount is SMB (that’s the only one with that option in Nextcloud unless you’re referring to an option you changed on the other end)? And you toggled case sensitivity right before your problems started? That seems pretty relevant. :wink:

Are you sure you don’t have other log messages appearing when attempting to either access files or manually run a scan? I would expect there to be an exception logged prior to the single log entry you posted (though I may be remembering the order of exception handling in that area of code incorrectly).

Thanks for the follow up.

External storage is SMB/CIFS. We use a single user ID for all access (at the admin level). The long index on startup appears to be full text indexing. Shutting that off got a fast restart.

We have tested with case sensitive and not case sensitive. It worked as case sensitive, but now does not work with either. It is now set for no case sensitive.

Removing and re-adding fixes “most” people, but the sync client is bombing out 100% of the time. Web access usually works during this time. I can provide log entries. All of the sync errors are some flavor of this…

[webdav] Error: Could not open file
GET /remote.php/dav/files/OLGA/EMK%20ICC%20FIles/Backup/CORONA-19/123.pdf
from 70.23.78.202 by OLGA at Jan 2, 2025, 4:50:27 PM

I have been working with a friend who uses NextCloud, and he says the issue may be the folder structure depth as well as the number of files. I can say that the people with less files tend to last longer before having issues. People with < 500 files have had no issues. We are thinking that this may be a scaling issue, but we use the AIO, and there is scant documentation on how to get the instance to use more than 4GB of RAM despite there being 64GB RAM and 8 CPU.

I guess some tuning help may be in order.

Jan 2, 2025, 4:10:47 PM
Error webdav
ServiceUnavailable Could not open file
Jan 2, 2025, 4:10:47 PM
Error webdav
ServiceUnavailable Could not open file
Jan 2, 2025, 4:10:46 PM
Error webdav
ServiceUnavailable Could not open file
Jan 2, 2025, 4:10:46 PM
Error webdav
ServiceUnavailable Could not open file

Level
Error
App
webdav
Time
Jan 2, 2025, 4:18:07 PM

ServiceUnavailable Could not open file

/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.phpLine 85

OCA\DAV\Connector\Sabre\File->get()

/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.phpLine 89

Sabre\DAV\CorePlugin->httpGet(
  {
    "__class__": "Sabre\\HTTP\\Request"
  },
  {
    "__class__": "Sabre\\HTTP\\Response"
  }
)

/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.phpLine 472

Sabre\DAV\Server->emit(
  "method:GET",
  [
    {
      "__class__": "Sabre\\HTTP\\Request"
    },
    {
      "__class__": "Sabre\\HTTP\\Response"
    }
  ]
)

/var/www/html/apps/dav/lib/Connector/Sabre/Server.phpLine 43

Sabre\DAV\Server->invokeMethod(
  {
    "__class__": "Sabre\\HTTP\\Request"
  },
  {
    "__class__": "Sabre\\HTTP\\Response"
  }
)

/var/www/html/apps/dav/lib/Server.phpLine 371

OCA\DAV\Connector\Sabre\Server->start()

/var/www/html/apps/dav/appinfo/v2/remote.phpLine 19

OCA\DAV\Server->exec()

/var/www/html/remote.phpLine 146

undefinedundefinedrequire_once(
  "/var/www/html/apps/dav/appinfo/v2/remote.php"
)

Raw log entry

{
“reqId”: “J6XPRN88eFbrvuZktgOn”,
“level”: 3,
“time”: “2025-01-02T21:18:07+00:00”,
“remoteAddr”: “178.85.130.111”,
“user”: “ICCTET”,
“app”: “webdav”,
“method”: “GET”,
“url”: “/remote.php/dav/files/ICCTET/TET%20Files/Documents/Saldibalans_ICC%20HOLDINGS%20BV_DIANNE.PDF”,
“message”: “Could not open file”,
“userAgent”: “Mozilla/5.0 (Windows) mirall/3.15.2 (build 20241216) (Nextcloud, windows-10.0.22631 ClientArchitecture: x86_64 OsArchitecture: x86_64)”,
“version”: “30.0.4.1”,
“exception”: {
“Exception”: “Sabre\DAV\Exception\ServiceUnavailable”,
“Message”: “Could not open file”,
“Code”: 0,
“Trace”: [
{
“file”: “/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php”,
“line”: 85,
“function”: “get”,
“class”: “OCA\DAV\Connector\Sabre\File”,
“type”: “->”,
“args”:
},
{
“file”: “/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php”,
“line”: 89,
“function”: “httpGet”,
“class”: “Sabre\DAV\CorePlugin”,
“type”: “->”,
“args”: [
{
class”: “Sabre\HTTP\Request”
},
{
class”: “Sabre\HTTP\Response”
}
]
},
{
“file”: “/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php”,
“line”: 472,
“function”: “emit”,
“class”: “Sabre\DAV\Server”,
“type”: “->”,
“args”: [
“method:GET”,
[
{
class”: “Sabre\HTTP\Request”
},
{
class”: “Sabre\HTTP\Response”
}
]
]
},
{
“file”: “/var/www/html/apps/dav/lib/Connector/Sabre/Server.php”,
“line”: 43,
“function”: “invokeMethod”,
“class”: “Sabre\DAV\Server”,
“type”: “->”,
“args”: [
{
class”: “Sabre\HTTP\Request”
},
{
class”: “Sabre\HTTP\Response”
}
]
},
{
“file”: “/var/www/html/apps/dav/lib/Server.php”,
“line”: 371,
“function”: “start”,
“class”: “OCA\DAV\Connector\Sabre\Server”,
“type”: “->”,
“args”:
},
{
“file”: “/var/www/html/apps/dav/appinfo/v2/remote.php”,
“line”: 19,
“function”: “exec”,
“class”: “OCA\DAV\Server”,
“type”: “->”,
“args”:
},
{
“file”: “/var/www/html/remote.php”,
“line”: 146,
“args”: [
“/var/www/html/apps/dav/appinfo/v2/remote.php”
],
“function”: “require_once”
}
],
“File”: “/var/www/html/apps/dav/lib/Connector/Sabre/File.php”,
“Line”: 463,
“message”: “Could not open file”,
“exception”: ,
“CustomMessage”: “Could not open file”
},
“id”: “67771a6c03a42”
}