Big File uploading result in corrupted archive

ℹ️ Support
1

Nextcloud version: 19.0.2
Operating system and version:
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster

Linux 4.19.0-10-686-pae i686

Apache version:
Server version: Apache/2.4.38 (Debian)
Server built: 2019-10-15T19:53:42
PHP version:
Version: 7.4.9
Memory Limit: 1 GB
Max Execution Time: 3600
Upload max size: 2 GB
HW Info:
CPU: Intel® Core™2 Duo CPU T8300 @ 2.40GHz (2 cores)
RAM: Total: 3 GB
SWAP: Total: 4 GB

The issue you are facing:
Uploading a big file does not work, not with desktop client, not with flowupload
I did not try using normal upload on the webUI but I don’t think it would work

Is this the first time you’ve seen this error? Y

Steps to replicate it:

  1. create a big file (my file is a 7zip archive)
  2. try uploading it
  3. enjoy error

The file is named “NAS Folder.7z” and is 13GB
IF it uploads it gets around 1.3GB
Uploader and Server are on the same network and work fine for smaller files (also 1-2GB)

The output of your Nextcloud log in Admin > Logging:

Fatal	webdav	Sabre\DAV\Exception\BadRequest: Expected filesize of 100000000 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 89374720 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.

    /var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php - line 154:

    OCA\DAV\Connector\Sabre\File->put()

    /var/www/nextcloud/apps/dav/lib/Upload/UploadFolder.php - line 46:

    OCA\DAV\Connector\Sabre\Directory->createFile()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 1104:

    OCA\DAV\Upload\UploadFolder->createFile()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 527:

    Sabre\DAV\Server->createFile()

    /var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php - line 89:

    Sabre\DAV\CorePlugin->httpPut()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 474:

    Sabre\DAV\Server->emit()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 251:

    Sabre\DAV\Server->invokeMethod()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 319:

    Sabre\DAV\Server->start()

    /var/www/nextcloud/apps/dav/lib/Server.php - line 320:

    Sabre\DAV\Server->exec()

    /var/www/nextcloud/apps/dav/appinfo/v2/remote.php - line 35:

    OCA\DAV\Server->exec()

    /var/www/nextcloud/remote.php - line 167:

    require_once("/var/www/ne ... p")

	
2020-09-09T15:19:02+0200
Error	no app in context	Sabre\DAV\Exception\BadRequest: Expected filesize of 100000000 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 89374720 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.

    /var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php - line 154:

    OCA\DAV\Connector\Sabre\File->put()

    /var/www/nextcloud/apps/dav/lib/Upload/UploadFolder.php - line 46:

    OCA\DAV\Connector\Sabre\Directory->createFile()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 1104:

    OCA\DAV\Upload\UploadFolder->createFile()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 527:

    Sabre\DAV\Server->createFile()

    /var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php - line 89:

    Sabre\DAV\CorePlugin->httpPut()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 474:

    Sabre\DAV\Server->emit()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 251:

    Sabre\DAV\Server->invokeMethod()

    /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 319:

    Sabre\DAV\Server->start()

    /var/www/nextcloud/apps/dav/lib/Server.php - line 320:

    Sabre\DAV\Server->exec()

    /var/www/nextcloud/apps/dav/appinfo/v2/remote.php - line 35:

    OCA\DAV\Server->exec()

    /var/www/nextcloud/remote.php - line 167:

    require_once("/var/www/ne ... p")

	
2020-09-09T15:19:01+0200
Error	core	TypeError: rtrim() expects parameter 1 to be string, array given

    /var/www/nextcloud/lib/private/URLGenerator.php - line 247:

    rtrim()

    /var/www/nextcloud/lib/private/User/User.php - line 457:

    OC\URLGenerator->getAbsoluteURL()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 202:

    OC\User\User->getCloudId()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 123:

    OCA\Settings\BackgroundJobs\VerifyUserData->verifyViaLookupServer()

    /var/www/nextcloud/lib/private/BackgroundJob/Job.php - line 62:

    OCA\Settings\BackgroundJobs\VerifyUserData->run()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 104:

    OC\BackgroundJob\Job->execute()

    /var/www/nextcloud/cron.php - line 126:

    OCA\Settings\BackgroundJobs\VerifyUserData->execute()

	
2020-09-09T15:12:27+0200
Error	core	TypeError: rtrim() expects parameter 1 to be string, array given

    /var/www/nextcloud/lib/private/URLGenerator.php - line 247:

    rtrim()

    /var/www/nextcloud/lib/private/User/User.php - line 457:

    OC\URLGenerator->getAbsoluteURL()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 202:

    OC\User\User->getCloudId()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 123:

    OCA\Settings\BackgroundJobs\VerifyUserData->verifyViaLookupServer()

    /var/www/nextcloud/lib/private/BackgroundJob/Job.php - line 62:

    OCA\Settings\BackgroundJobs\VerifyUserData->run()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 104:

    OC\BackgroundJob\Job->execute()

    /var/www/nextcloud/cron.php - line 126:

    OCA\Settings\BackgroundJobs\VerifyUserData->execute()

	
2020-09-09T14:10:15+0200
Warning	appstoreFetcher	Could not connect to appstore: cURL error 28: Resolving timed out after 60000 milliseconds (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)	
2020-09-09T13:59:32+0200
Error	core	TypeError: rtrim() expects parameter 1 to be string, array given

    /var/www/nextcloud/lib/private/URLGenerator.php - line 247:

    rtrim()

    /var/www/nextcloud/lib/private/User/User.php - line 457:

    OC\URLGenerator->getAbsoluteURL()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 202:

    OC\User\User->getCloudId()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 123:

    OCA\Settings\BackgroundJobs\VerifyUserData->verifyViaLookupServer()

    /var/www/nextcloud/lib/private/BackgroundJob/Job.php - line 62:

    OCA\Settings\BackgroundJobs\VerifyUserData->run()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 104:

    OC\BackgroundJob\Job->execute()

    /var/www/nextcloud/cron.php - line 126:

    OCA\Settings\BackgroundJobs\VerifyUserData->execute()

	
2020-09-09T13:06:12+0200
Error	cron	TypeError: rtrim() expects parameter 1 to be string, array given

    /var/www/nextcloud/lib/private/URLGenerator.php - line 247:

    rtrim()

    /var/www/nextcloud/lib/private/User/User.php - line 457:

    OC\URLGenerator->getAbsoluteURL()

    /var/www/nextcloud/apps/lookup_server_connector/lib/BackgroundJobs/RetryJob.php - line 201:

    OC\User\User->getCloudId()

    /var/www/nextcloud/apps/lookup_server_connector/lib/BackgroundJobs/RetryJob.php - line 149:

    OCA\LookupServerConnector\BackgroundJobs\RetryJob->getUserAccountData()

    /var/www/nextcloud/lib/public/BackgroundJob/Job.php - line 80:

    OCA\LookupServerConnector\BackgroundJobs\RetryJob->run()

    /var/www/nextcloud/apps/lookup_server_connector/lib/BackgroundJobs/RetryJob.php - line 113:

    OCP\BackgroundJob\Job->execute()

    /var/www/nextcloud/cron.php - line 126:

    OCA\LookupServerConnector\BackgroundJobs\RetryJob->execute()

	
2020-09-09T12:56:08+0200
Error	core	TypeError: rtrim() expects parameter 1 to be string, array given

    /var/www/nextcloud/lib/private/URLGenerator.php - line 247:

    rtrim()

    /var/www/nextcloud/lib/private/User/User.php - line 457:

    OC\URLGenerator->getAbsoluteURL()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 202:

    OC\User\User->getCloudId()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 123:

    OCA\Settings\BackgroundJobs\VerifyUserData->verifyViaLookupServer()

    /var/www/nextcloud/lib/private/BackgroundJob/Job.php - line 62:

    OCA\Settings\BackgroundJobs\VerifyUserData->run()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 104:

    OC\BackgroundJob\Job->execute()

    /var/www/nextcloud/cron.php - line 126:

    OCA\Settings\BackgroundJobs\VerifyUserData->execute()


2020-09-09T12:00:40+0200
Error	core	TypeError: rtrim() expects parameter 1 to be string, array given

    /var/www/nextcloud/lib/private/URLGenerator.php - line 247:

    rtrim()

    /var/www/nextcloud/lib/private/User/User.php - line 457:

    OC\URLGenerator->getAbsoluteURL()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 202:

    OC\User\User->getCloudId()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 123:

    OCA\Settings\BackgroundJobs\VerifyUserData->verifyViaLookupServer()

    /var/www/nextcloud/lib/private/BackgroundJob/Job.php - line 62:

    OCA\Settings\BackgroundJobs\VerifyUserData->run()

    /var/www/nextcloud/apps/settings/lib/BackgroundJobs/VerifyUserData.php - line 104:

    OC\BackgroundJob\Job->execute()

    /var/www/nextcloud/cron.php - line 126:

    OCA\Settings\BackgroundJobs\VerifyUserData->execute()

	
2020-09-09T10:55:11+0200

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'instanceid' => 'A',
  'passwordsalt' => 'B',
  'secret' => 'C',
  'trusted_domains' =>
  array (
    0 => 'site.com',
  ),
  'datadirectory' => '/var/www/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '19.0.2.2',
  'overwrite.cli.url' =>
  array (
    0 => 'site.com/nextcloud',
  ),
  'dbname' => 'D',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'U',
  'dbpassword' => 'P',
  'installed' => true,
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpauth' => 1,
  'app_install_overwrite' =>
  array (
    0 => 'keeporsweep',
    1 => 'bookmarks_fulltextsearch',
    2 => 'dashboard',
    3 => 'caniupdate',
    4 => 'files_clipboard',
  ),
  'has_rebuilt_cache' => true,
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'filelocking.enabled' => 'true',
  'redis' =>
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'port' => 0,
    'timeout' => 0.0,
  ),
  'default_language' => 'en_GB',
  'default_locale' => 'en_IE',
  'maintenance' => false,
  'mail_smtpsecure' => 'tls',
  'mail_from_address' => 'address',
  'mail_domain' => 'gmail.com',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtphost' => 'smtp.gmail.com',
  'mail_smtpport' => '587',
  'mail_smtpname' => 'address',
  'mail_smtppassword' => 'password',
  'theme' => '',
  'loglevel' => 2,
);

The output of your Apache/nginx/system log in /var/log/apache2/nextcloud_access_log:
https://del.dog/ydulonalle

The output of your Apache/nginx/system log in /var/log/syslog:
https://del.dog/dylysurrad.txt

2

Sorry no idea. But perhaps you can bypass the problem until solution with this app.
https://apps.nextcloud.com/apps/flowupload

3

This should be done sparingly, but if you need a one-time large file upload, you can copy it with SFTP and then run occ files:scan. Be aware that it may sync back to you if you put it in a folder that you sync.

I have done large uploads by dragging and dropping folders as large as I think 36 GB directly into the browser, but the single largest file I’ve uploaded is probably only 3-4 GB.

4

Thank you, but I clearly stated that I tried it with flowupload and I get the same error

5

yeah, for direct browser uploads it depends on the amount of RAM and the PHP config
I have 32bit so I’m screwed :frowning:
Anyway this should not be a problem using desktop client or flowupload

6

after some retries I did manage to upload it using the SFTP workaround @KarlF12 suggested, but again, when syncing to PC it gets a 1.3GB file which is of course corrupted, the issue seem to occur when downloading the uploaded file

P.S. also the webUI shows the wrong size