Backblaze as external storage: almost stuck at upload end

Hello all,

I have troubles I was not able to manage myself after scrolling all the forum / github issues / internet. Some help would be appreciated :slight_smile:

Technical details
Nextcloud version (eg, 20.0.5): 26.0.0.11
Operating system and version (eg, Ubuntu 20.04): Alpine Linux v3.17 (Linux server docker container)
Apache or nginx version (eg, Apache 2.4.25): nginx/1.22.1
PHP version (eg, 7.4): 8.1.17

The issue you are facing:
When uploading a file using the web interface, to the external storage folder (Backblaze S3 compatible, known also as b2), the upload hang at the end with the status “file processing” (translated from the french message I get).
I tried to set new chunk upload values according to here S3 random storage problem on large files - #4 by T-House and according to B2 documentation Large Files. But the problem persists.

Probleme detected when uploading a file from my laptop and also when moving a nextcloud file to the external folder.

Is this the first time you’ve seen this error? (Y/N): Every time I upload a file, I have this error.

Steps to replicate it:

  1. Configure S3 external storage in nextcloud
  2. Upload a file. I tried with 6Mb, 200Mb and 2Gb, same problem.
  3. Try to move a nextcloud local file to the external storage, same probleme
  4. Try to change the uploadPartSize, same.

The output of your Nextcloud log in Admin > Logging: None at the time of the upload

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'filelocking.enabled' => false,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'uploadPartSize' => '131072000',
  'filesystem_check_changes' => '0',
  'tmpdirectory' => '/tmpdir',
  'tempdirectory' => '/tmpdir',
  'sys_temp_dir' => '/tmpdir',
  'upload_tmp_dir' => '/tmpdir',
  'datadirectory' => '/data',
  'default_language' => 'fr',
  'default_locale' => 'fr_FR',
  'default_phone_region' => 'FR',
  'remember_login_cookie_lifetime' => 1296000,
  'session_lifetime' => 86400,
  'auth.bruteforce.protection.enabled' => true,
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_sendmailmode' => 'smtp',
  'mail_smtphost' => 'smtp.sparkpostmail.com',
  'mail_smtpport' => '587',
  'mail_smtpsecure' => 'tls',
  'mail_from_address' => 'nextcloud',
  'trashbin_retention_obligation' => '30,31',
  'versions_retention_obligation' => '30,31',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'nextcloud-redis',
    'port' => 6379,
    'timeout' => 0.0,
    'password' => 'REDACTED',
    'dbindex' => 0,
  ),
  'hashingCost' => 12,
  'cipher' => 'AES-256-CFB',
  'secret' => 'REDACTED',
  'login_form_autocomplete' => false,
  'instanceid' => 'REDACTED',
  'passwordsalt' => 'REDACTED',
  'trusted_domains' =>
  array (
    0 => 'nextcloud.REDACTED.com',
  ),
  'trusted_proxies' =>
  array (
    0 => 'nextcloud.REDACTED.com',
    1 => 'nextcloud.REDACTED.co',
    2 => '127.0.0.1',
    3 => 'REDACTED',
  ),
  'dbtype' => 'mysql',
  'version' => '26.0.0.11',
  'overwrite.cli.url' => 'https://nextcloud.REDACTED.com',
  'dbname' => 'lmi-nextcloud',
  'dbhost' => 'nextcloud-db',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'REDACTED',
  'installed' => true,
  'overwriteprotocol' => 'https',
  'app_install_overwrite' =>
  array (
    0 => 'ocdownloader',
    1 => 'impersonate',
    2 => 'unsplash',
    3 => 'twofactor_gateway',
    4 => 'ransomware_protection',
  ),
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 1,
  'log_type' => 'file',
  'logfile' => 'nextcloud.log',
  'mail_domain' => 'REDACTED.co',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'SMTP_Injection',
  'mail_smtppassword' => 'REDACTED',
  'memories.exiftool' => '/config/www/nextcloud/apps/memories/lib/../exiftool-bin/exiftool-amd64-musl',
  'memcache.local' => '\\OC\\Memcache\\Redis',
);

The output of your Apache/nginx/system log in /var/log/____:

None at the time of the upload test (in error.log)

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

{"reqId":"XKEZVv8o1YBebJL6LLAm","level":1,"time":"2023-04-14T08:57:07+00:00","remoteAddr":"172.16.238.5","user":"lmilcent","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2Fbackblaze-files%2FMY%20FILES","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36","version":"26.0.0.11","data":[]}
{"reqId":"XKEZVv8o1YBebJL6LLAm","level":1,"time":"2023-04-14T08:57:07+00:00","remoteAddr":"172.16.238.5","user":"lmilcent","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2Fbackblaze-files%2FMY%20FILES","message":"Deprecated event type for OCA\\Files::loadAdditionalScripts: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36","version":"26.0.0.11","data":[]}

The upload of my 1.2Gb file takes only 1 minute, but after few hours the file processing ends with that error message and the upload fails.
image

(it says “Error when assembling blocks, code 504”)

Is there anything to configure to solve the chunk error and speedup uploads?
Is nextcloud capable to direct upload to s3 backend and avoid writing in temp files on the local instance?

Hello all,

Can anyone point me to the right path? What did I miss?