Windows App Sync too slow in high volume files sync case using Amazon s3 external storage - HELP PLEASE!

tecnodaniel · January 30, 2021, 5:01pm

Support intro

Sorry to hear you’re facing problems

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can

Nextcloud version (eg, 20.0.5): 19.0.2
Operating system and version (eg, Ubuntu 20.04): CENTOS 7.9
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.46
PHP version (eg, 7.4): 7.4
MySQL version: 5.7.33
S3 storage provider: Amazon
Windows version: 10
Nextcloud Windows App version: 3.0.3 to 3.1.2

The issue you are facing:

Hi fellows,

I have been working with nextcloud for a few months now because I think it is a good platform to manage files and share them with my work team.

We handle a lot of information that I need to share with different departments or people in different geographical locations. The total size of my team so far is 43 people.

To facilitate the management of the files we are making use of the Windows app, so we can synchronize the files on the local machines of each team member. That is to say, a minimum of 43 machines synchronize the files to which each of them has access.

The data structure handles more than 3,000 folders and more than 15,000 files.

The project has 500 GB of data information. This data is in a structure of folders and sub-folders required for each department. That is, a folder can have 10 sub folders and each sub folder can have within it 20 more folders, so on and so forth if required.

Some machines can synchronize between 1 GB to 10 GB minimum. This depends on the level of access that each member of the company has.

At the beginning and to get to know nextcloud we started with a small volume of information and we saw that it worked as we wanted.

As the data volumes grew, I noticed that the synchronization took a little longer to review the remote folders, all this from the Windows App. However, everything synchronized correctly.

As the volume of data to manage with nextcloud is increasing we decided to migrate part of the file storage to Amazon S3 since in the future I hope to manage more than 1 TB of information, for this reason I thought it was convenient to migrate the files to Amazon S3 and connect both platforms using the external storage in nextcloud.

So, I currently have 2 external storages configured, a local one with data that I do not require to be uploaded to Amazon S3 and an external storage connected to Amazon S3, this is where I have the largest amount of files of my company.

The local external storage has 1.7 Gb.
The Amazon S3 storage has 498.3 Gb.

Now, the problem I have is that the information takes a long time to synchronize and in some cases it does not finish synchronizing. Since the Windows App when it detects some communication error, it restarts the sync verification process.

The server where I have nextcloud was optimized according to nextcloud recommendations for the use of APCU, Redis, OP cache and optimizing MySQL database as well as php-fpm. It has also 24GB RAM. However, the synchronization is still very slow, and every time it detects an error it restarts the process instead of skipping the file and trying to synchronize later.

For this reason I would like to ask you the following questions:

Is there any nextcloud configuration that allows me to synchronize files by exception? That is to say that it only synchronizes the modified ones.
How can I improve the performance of the Windows App?
If in the Amazon S3 external storage settings, I set it to “Check for changes” with value “Never” instead of “Once on every access”. How do I notify nextcloud that there are changes?

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '[ID]',
  'passwordsalt' => '[PW]',
  'secret' => '[SECRET]',
  'trusted_domains' => 
  array (
    0 => '[DOMAIN 1]',
    1 => '[DOMAIN 2]',
    2 => '[DOMAIN 3]',
  ),
  'datadirectory' => '/home/[USER]/public_html/nube/data',
  'dbtype' => 'mysql',
  'version' => '19.0.2.2',
  'overwrite.cli.url' => '[URL]',
  'dbname' => '[DBNAME]',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => '[USER]',
  'dbpassword' => '[PW]',
  'installed' => true,
  'filelocking.enabled' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => 'localhost',
    'port' => 6379,
  ),
  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => '[SMTP HOST]',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_from_address' => 'cloud',
  'mail_domain' => '[DOMAIN]',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtpport' => '465',
  'mail_smtpname' => '[SMTP]',
  'mail_smtppassword' => '[PASSWORD]',
  'app_install_overwrite' => 
  array (
    0 => 'gluusso',
  ),
  'default_language' => 'es',
  'force_locale' => 'es',
  'maintenance' => false,
  'signature' => 'v4',
  'region' => 'us-east-1',
  'theme' => '',
  'loglevel' => 2,
  'integrity.check.disabled' => true,
  'mysql.utf8mb4' => false, 
  'skeletondirectory' => '/home/[USER]/esqueleto',
);

The output of your Apache/nginx/system log in /var/log/____:

Warning	appstoreFetcher	Could not connect to appstore: cURL error 28: Failed to connect to apps.nextcloud.com port 443: Connection timed out (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)	
hace 2 horas
Fatal	webdav	Sabre\DAV\Exception\ServiceUnavailable: Could not open file	
hace 3 horas
Error	PHP	fopen(httpseek://): failed to open stream: &quot;OC\Files\Stream\SeekableHttpStream::stream_open&quot; call failed at /home/[USER]/public_html/nube/lib/private/Files/Stream/SeekableHttpStream.php#67	
hace 3 horas
Error	PHP	fopen(https://[BUCKET].s3.us-east-1.amazonaws.com/01_PROY/24_[PROJECT]/24_PALTO_01_REGISTROS/24_PALTO_01_ARQ/2019.03.15%20-%20Modificaciones%20Et%202%20y%203/hardcopy.log): failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable at /home/[USER]/public_html/nube/lib/private/Files/ObjectStore/S3ObjectTrait.php#73	
hace 3 horas
Error	PHP	filesize(): stat failed for /home/[USER]/public_html/nube/data/csaldarriaga/files/ADMIN/07_GH/00_SST/EDIFICACIONES/SST ACTUALIZACION AGOSTO Y SEPTIEMBRE/INFORMACION CONTRATIS/ACABADOS Y REFORMAS/Camilo Gamez/CCF.pdf.ocTransferId1883884888.part at /home/[USER]/public_html/nube/lib/private/Files/Storage/Local.php#224	
hace 3 horas
Error	PHP	filesize(): stat failed for /home/[USER]/public_html/nube/data/csaldarriaga/files/ADMIN/07_GH/00_SST/EDIFICACIONES/SST ACTUALIZACION AGOSTO Y SEPTIEMBRE/INFORMACION CONTRATIS/ACABADOS Y REFORMAS/Camilo Gamez/ARL.PDF.ocTransferId1328432758.part at /home/[USER]/public_html/nube/lib/private/Files/Storage/Local.php#224	
hace 3 horas
Error	PHP	filesize(): stat failed for /home/[USER]/public_html/nube/data/csaldarriaga/files/ADMIN/07_GH/00_SST/EDIFICACIONES/SST ACTUALIZACION AGOSTO Y SEPTIEMBRE/INFORMACION CONTRATIS/ACABADOS Y REFORMAS/Allan zapata/Curso Alturas.pdf.ocTransferId1059788207.part at /home/[USER]/public_html/nube/lib/private/Files/Storage/Local.php#224	
hace 3 horas
Error	PHP	filesize(): stat failed for /home/[USER]/public_html/nube/data/csaldarriaga/files/ADMIN/07_GH/00_SST/EDIFICACIONES/SST ACTUALIZACION AGOSTO Y SEPTIEMBRE/INFORMACION CONTRATIS/ACABADOS Y REFORMAS/Allan zapata/concepto_ALLAN YESID ZAPATA VELEZ_2020.pdf.ocTransferId321365193.part at /home/[USER]/public_html/nube/lib/private/Files/Storage/Local.php#224	
hace 3 horas
Fatal	webdav	Sabre\DAV\Exception\ServiceUnavailable: Could not open file	
hace 3 horas
Error	PHP	fopen(httpseek://): failed to open stream: &quot;OC\Files\Stream\SeekableHttpStream::stream_open&quot; call failed at /home/[USER]/public_html/nube/lib/private/Files/Stream/SeekableHttpStream.php#67	
hace 3 horas
Error	PHP	fopen(https://[BUCKET].s3.us-east-1.amazonaws.com/01_PROY/24_[PROJECT]/24_PALTO_01_REGISTROS/24_PALTO_01_ARQ/2019.03.15%20-%20Modificaciones%20Et%202%20y%203/hardcopy.log): failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable at /home/[USER]/public_html/nube/lib/private/Files/ObjectStore/S3ObjectTrait.php#73	
hace 3 horas
Fatal	webdav	Sabre\DAV\Exception\ServiceUnavailable: Could not open file	
hace 7 horas
Error	PHP	fopen(httpseek://): failed to open stream: &quot;OC\Files\Stream\SeekableHttpStream::stream_open&quot; call failed at /home/[USER]/public_html/nube/lib/private/Files/Stream/SeekableHttpStream.php#67	
hace 7 horas
Error	PHP	fopen(https://[BUCKET].s3.us-east-1.amazonaws.com/01_PROY/24_[PROJECT]/24_PALTO_01_REGISTROS/24_PALTO_01_ARQ/2019.03.15%20-%20Modificaciones%20Et%202%20y%203/hardcopy.log): failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable at /home/[USER]/public_html/nube/lib/private/Files/ObjectStore/S3ObjectTrait.php#73	
hace 7 horas
Fatal	webdav	Sabre\DAV\Exception\ServiceUnavailable: Could not open file	
hace 7 horas
Error	PHP	fopen(httpseek://): failed to open stream: &quot;OC\Files\Stream\SeekableHttpStream::stream_open&quot; call failed at /home/[USER]/public_html/nube/lib/private/Files/Stream/SeekableHttpStream.php#67	
hace 7 horas
Error	PHP	fopen(https://[BUCKET].s3.us-east-1.amazonaws.com/01_PROY/24_[PROJECT]/24_PALTO_01_REGISTROS/24_PALTO_01_ARQ/2019.03.15%20-%20Modificaciones%20Et%202%20y%203/hardcopy.log): failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable at /home/[USER]/public_html/nube/lib/private/Files/ObjectStore/S3ObjectTrait.php#73	
hace 7 horas
Error	PHP	Undefined index: name at /home/[USER]/public_html/nube/lib/private/Files/Cache/Scanner.php#418	
hace 10 horas
Error	PHP	Undefined index: permissions at /home/[USER]/public_html/nube/lib/private/Files/Storage/Wrapper/PermissionsMask.php#160	
hace 10 horas
Error	PHP	Trying to access array offset on value of type null at /home/[USER]/public_html/nube/lib/private/Files/Storage/Wrapper/PermissionsMask.php#159	
hace 10 horas
Error	files_external	Aws\S3\Exception\S3Exception: Error executing "HeadObject" on "https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11"; AWS HTTP error: Server error: `HEAD https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11` resulted in a `500 Internal Server Error` response (server): 500 Internal Server Error (Request-ID: 86B8AF353BB58BF1) - 	
hace 10 horas
Error	files_external	Aws\S3\Exception\S3Exception: Error executing "HeadObject" on "https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11"; AWS HTTP error: Server error: `HEAD https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11` resulted in a `500 Internal Server Error` response (server): 500 Internal Server Error (Request-ID: 5BF331ABBD28DE75) - 	
hace 10 horas
Error	files_external	Aws\S3\Exception\S3Exception: Error executing "HeadObject" on "https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11"; AWS HTTP error: Server error: `HEAD https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11` resulted in a `500 Internal Server Error` response (server): 500 Internal Server Error (Request-ID: 4V6HEH0V9K4X3PAM) - 	
hace 10 horas
Error	PHP	Undefined index: name at /home/[USER]/public_html/nube/lib/private/Files/Cache/Scanner.php#418	
hace 10 horas
Error	PHP	Undefined index: permissions at /home/[USER]/public_html/nube/lib/private/Files/Storage/Wrapper/PermissionsMask.php#160	
hace 10 horas
Error	PHP	Trying to access array offset on value of type null at /home/[USER]/public_html/nube/lib/private/Files/Storage/Wrapper/PermissionsMask.php#159	
hace 10 horas
Error	files_external	Aws\S3\Exception\S3Exception: Error executing "HeadObject" on "https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11"; AWS HTTP error: Server error: `HEAD https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11` resulted in a `500 Internal Server Error` response (server): 500 Internal Server Error (Request-ID: 9F26D91EB8DF3BEE) - 	
hace 10 horas
Error	PHP	Undefined index: name at /home/[USER]/public_html/nube/lib/private/Files/Cache/Scanner.php#418	
hace 10 horas
Error	PHP	Undefined index: permissions at /home/[USER]/public_html/nube/lib/private/Files/Storage/Wrapper/PermissionsMask.php#160	
hace 10 horas
Error	PHP	Trying to access array offset on value of type null at /home/[USER]/public_html/nube/lib/private/Files/Storage/Wrapper/PermissionsMask.php#159	
hace 10 horas
Error	files_external	Aws\S3\Exception\S3Exception: Error executing "HeadObject" on "https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11"; AWS HTTP error: Server error: `HEAD https://[BUCKET].s3.us-east-1.amazonaws.com/00_ADMIN/02_ENTIDADES/DIAN/PRISMA/01_RTE_FTE/2019/11` resulted in a `500 Internal Server Error` response (server): 500 Internal Server Error (Request-ID: 7C65AC710FEA79E4) - 	
hace 10 horas

simonspa · January 31, 2021, 4:28pm

Dear @tecnodaniel,

Your application sounds like a proper business use case of Nextcloud - have you considered to get professional support from Nextcloud GmbH or one of their partners? I think this would be the proper solution for you and I’m sure they’re engineering team will be able to solve your issues.

Simon

tecnodaniel · February 1, 2021, 12:00pm

Hi @simonspa,

In fact this is a project for a client who doesn’t have enough budget. They started with basic requests and then asked for the Amazon S3 external storage and then with the windows sync but with lot’s of big files and too many folders as well as multiple persons synchronizing. We have optimized MySQL Innodb parameters as well as the php-fpm parameters and cache tools but the real problem I guess is the way how the windows app syncs the files since each time it lost anything starts again over and over causing that all is too slow. What we need to understand is if this kind of use is really compatible with nextcloud or we made a mistake choosing it and now we are in trouble with the customer.