Object storage as primary storage: cannot download files

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can. :heart:

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • 30.0.2
  • Operating system and version (e.g., Ubuntu 24.04):
    • N/A (Kubernetes)
  • Web server and version (e.g, Apache 2.4.25):
    • N/A
  • Reverse proxy and version _(e.g. nginx 1.27.2):
    • Nginx - ingress-nginx
  • PHP version (e.g, 8.3):
    • 8.2.25
  • Is this the first time you’ve seen this error? (Yes / No):
    • Yes
  • When did this problem seem to first start?:
    • Since first install
  • Installation method (e.g. AIO, NCP, Bare Metal/Archive, etc.)
    • Kubernetes
  • Are you using Cloudflare, mod_security, or similar? (Yes / No)
    • No

  • Helm chart:
  • Helm release version:
    • 6
  • Helm chart values:
    • persistence:
        enabled: true
        accessMode: ReadWriteMany
        size: 10Gi
        storageClass: ceph-filesystem
      cronjob:
        enabled: true
      redis:
        enabled: true
        master:
          persistence:
            enabled: false
        replica:
          persistence:
            enabled: false
      internalDatabase:
        enabled: false
      externalDatabase:
        enabled: true
        type: postgresql
        host: nextcloud-pooler
        existingSecret:
          enabled: true
          secretName: nextcloud-db-app
          # hostKey: host
          databaseKey: dbname
          usernameKey: user
          passwordKey: password
      nextcloud:
        host: ${host} # Needed for healthcheck
        objectStore:
          s3:
            enabled: true
            ssl: true
            port: 443
            # legacyAuth: false
            existingSecret: ${s3_secret_name}
            secretKeys:
              host: host
              accessKey: accessKey
              secretKey: secretKey
              bucket: bucket
              sse_c_key: sse_c_key
        configs:
          proxy.config.php: |-
            <?php
            $CONFIG = array (
              'trusted_proxies' => array(
                0 => '127.0.0.1',
                1 => '10.0.0.0/8',
              ),
              'forwarded_for_headers' => array('HTTP_X_FORWARDED_FOR'),
            );
        phpConfigs:
          zz-memory_limit.ini: |-
            memory_limit=512M
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
      phpClientHttpsFix:
        enabled: true
      

Summary of the issue you are facing:

I cannot download files but upload seems to work fine.

Images corrupted

Here is a corrupt image (to the bottom)


Another doesn’t even load:

But its preview is corrupted:
image


If I try to download an image it fails:
image

If I try to download an image with curl, this is the output:

>
* Request completely sent off
< HTTP/2 200
< date: Sat, 07 Dec 2024 11:56:56 GMT
< content-type: image/jpeg
< content-length: 1670520
< referrer-policy: no-referrer
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-permitted-cross-domain-policies: none
< x-robots-tag: noindex, nofollow
< x-xss-protection: 1; mode=block
< content-security-policy: default-src 'none';
< last-modified: Mon, 27 May 2024 20:38:15 GMT
< etag: "6753466650489"
< x-request-id: xxx
< oc-etag: "6753466650489"
< x-debug-token: xxx
< content-disposition: attachment; filename*=UTF-8''4r71funsgzu41.jpg; filename="4r71funsgzu41.jpg"
< strict-transport-security: max-age=31536000; includeSubDomains
< access-control-allow-origin: *
< access-control-allow-credentials: true
< access-control-allow-methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
< access-control-allow-headers: X-Forwarded-For
< access-control-max-age: 1728000
<
{ [3254 bytes data]
 60 1631k   60  992k    0     0   198k      0  0:00:08  0:00:04  0:00:04  198k* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
 60 1631k   60  992k    0     0   167k      0  0:00:09  0:00:05  0:00:04  188k
* Connection #0 to host ***REMOVED SENSITIVE VALUE*** left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

Steps to replicate it (hint: details matter!):

  1. Deploy the Helm chart with the secret poining to a S3 bucket (in my case using scaleway)
  2. Upload works but cannot download files (they are stuck in the download phase and images previews are corrupted)

Log entries

Nextcloud

Please provide the log entries from your Nextcloud log that are generated during the time of problem (via the Copy raw option from Administration settings->Logging screen or from your nextcloud.log located in your data directory). Feel free to use a pastebin/gist service if necessary.

{"reqId":"U0SdRjjQ7xFn2rQQceXj","level":3,"time":"2024-12-05T19:31:55+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Trying to access array offset on value of type null at /var/www/html/lib/private/App/AppStore/Fetcher/Fetcher.php#172","userAgent":"--","version":"30.0.2.2","data":{"app":"PHP"},"id":"67534e0fe71be"}

Configuration

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):

k exec -it nextcloud-7d7d8fb9c5-tnbn5 -- su -s /bin/sh www-data -c "./occ config:list system"
Defaulted container "nextcloud" out of: nextcloud, nextcloud-cron
{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "forwarded_for_headers": [
            "HTTP_X_FORWARDED_FOR"
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "***REMOVED SENSITIVE VALUE***",
                "region": "eu-west-1",
                "hostname": "s3.fr-par.scw.cloud",
                "port": "443",
                "storageClass": "STANDARD",
                "objectPrefix": "urn:oid:",
                "autocreate": false,
                "use_ssl": true,
                "use_path_style": false,
                "legacy_auth": false,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "sse_c_key": "***REMOVED SENSITIVE VALUE***" # This was not removed - WTF??
            }
        },
        "upgrade.disable-web": true,
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "30.0.2.2",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "overwriteprotocol": "https"
    }
}

Apps

The output of occ app:list (if possible).

k exec -it nextcloud-7d7d8fb9c5-tnbn5 -- su -s /bin/sh www-data -c "./occ app:list"
Defaulted container "nextcloud" out of: nextcloud, nextcloud-cron
Enabled:
  - activity: 3.0.0
  - app_api: 4.0.0
  - bruteforcesettings: 3.0.0
  - circles: 30.0.0
  - cloud_federation_api: 1.13.0
  - comments: 1.20.1
  - contactsinteraction: 1.11.0
  - dashboard: 7.10.0
  - dav: 1.31.1
  - federatedfilesharing: 1.20.0
  - federation: 1.20.0
  - files: 2.2.0
  - files_downloadlimit: 3.0.0
  - files_pdfviewer: 3.0.0
  - files_reminders: 1.3.0
  - files_sharing: 1.22.0
  - files_trashbin: 1.20.1
  - files_versions: 1.23.0
  - firstrunwizard: 3.0.0
  - logreader: 3.0.0
  - lookup_server_connector: 1.18.0
  - nextcloud_announcements: 2.0.0
  - notifications: 3.0.0
  - oauth2: 1.18.1
  - password_policy: 2.0.0
  - photos: 3.0.2
  - privacy: 2.0.0
  - provisioning_api: 1.20.0
  - recommendations: 3.0.0
  - related_resources: 1.5.0
  - serverinfo: 2.0.0
  - settings: 1.13.0
  - sharebymail: 1.20.0
  - support: 2.0.0
  - survey_client: 2.0.0
  - systemtags: 1.20.0
  - text: 4.1.0
  - theming: 2.5.0
  - twofactor_backupcodes: 1.19.0
  - updatenotification: 1.20.0
  - user_status: 1.10.0
  - viewer: 3.0.0
  - weather_status: 1.10.0
  - webhook_listeners: 1.1.0-dev
  - workflowengine: 2.12.0
Disabled:
  - admin_audit: 1.20.0
  - encryption: 2.18.0
  - files_external: 1.22.0
  - suspicious_login: 8.0.0
  - twofactor_nextcloud_notification: 4.0.0
  - twofactor_totp: 12.0.0-dev
  - user_ldap: 1.21.0
1 Like

This looks like an HTTP/2 matter.

In general INTERNAL_ERROR indicates a web server-side HTTP2 matter.

Either bug or weird interaction somewhere. I don’t see any evidence this is S3 related.

I would check your ingress itself (including versions/bug fixes) as well as the ingress<->apache/app container connection. Especially error logs.

I’d eliminate some variables. i.e. disable HTTP/2, etc.

Hi @jtr !
Thanks so much for getting back to me!

Unforutnately that’s not.
I’ve reviewed the logs and there’s nothing suggesting that.

I also did that by directly port-forwarding to the nextcloud pod.
Unfortunately the problem remained the same

Talking about logs, I also tried setting the logs of nextcloud to debug, but that was not very helpful

I also tried understanding if there was a correlation between the file size and the amount of bytes I was able to download before the download failed. I found that (for similiar enough file sizes) the downloaded bytes were the same. Eventually I tried uploading and downloading a 32MB file, and that finally triggered a log on nextcloud; The error was a 500 error from the S3 endpoint from Scaleway

I tried (just to exclude things) to use Backblaze B2 (instead of Scaleway) to test the object storage and magically everything started working like a charm!

So I suspect that the problem was (is) on Scaleway’s side.
I’ve opened a ticket to the support and in the meantime I’ll look to move the instance to another S3 provider (Backblaze is not Ok because of the single availability zone, so I’ll probably try OVH)

I also think that if the problem is on the S3’s side, perhaps Nextcloud should log an error or something.
I’d like to contribute to that but I’m not familiar with the codebase and I don’t know where to start with that

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.