Slow Login and App switching after Migration to NC12

Nextcloud version (eg, 12.0.2): 12.0.4.3
Operating system and version (eg, Ubuntu 17.04): Ubuntu 14.04
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.7
PHP version (eg, 7.1): 7.0.26

The issue you are facing:
After migration from ownCloud 9.1.6 to Nextcloud 12.0.4 we face slow loading times for the login procedure and when switching between apps.
We noticed the strange phenomenon that when we click a second time (instead of waiting) on a link or button (login, app switching), the page is loaded smoothly and fast.
While navigating inside an app (f.e. files) everything runs fast. No slow loading.

We have two other Nextcloud servers with a similar setup with no issues. Difference: The problematic setup has a HAProxy with loadbalancing and multiple webservers.

We suspect that the new rate limiting (trottling) mechnism and Brute Force Protection (which is disabled) causes these delays.
We tried several things to solve the issue:

  • Disable Bruteforce app via config.php and restart Apache
  • Add the IP Addressess of the clients who experiance this issue
  • Add an IP address with Subnet 1 ( covering 128.0.0.0 - 255.255.255.254 )
  • Uninstall Bruteforce app

The disabling Brute Force Protection in config.php have solved part of the problem.

Is this the first time you’ve seen this error? (Y/N): Yes

Steps to replicate it:

  1. Navigate to Cloud-Service
  2. Enter Login Credentials and hit Login
  3. Enter TOTP-Code and hit Login Button
  4. Login takes approximately 17 seconds (second click on Login Button leeds to substantially faster loading time)
  5. Switching to Activity App takes approximately 17 seconds to load the App (Page) (second click on Activity app leeds to substantially faster loading time)
  6. Switching back Files app takes approximately 17 seconds to load the App (Page) (second click on Files app leeds to substantially faster loading time)

The output of your Nextcloud log in Admin > Logging:

CONTAINS SENSITIVE DATA. No indication of the problem mentioned

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'SENSITIVE DATA',
  'asset-pipeline.enabled' => false,
  'passwordsalt' => 'SENSITIVE DATA',
  'dbtype' => 'mysql',
  'version' => '12.0.4.3',
  'dbname' => 'owncloud',
  'dbhost' => '10.0.222.9:6033',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'SENSITIVE DATA',
  'dbpassword' => 'SENSITIVE DATA',
  'installed' => true,
  'integrity.check.disabled' => true,
  'appcodechecker' => false,
  'theme' => 'wolke',
  'forcessl' => true,
  'maintenance' => false,
  'enable_previews' => true,
  'preview_max_x' => 2048,
  'preview_max_y' => 2048,
  'session_lifetime' => 86400,
  'maxZipInputSize' => 2147483648,
  'allowZipDownload' => true,
  'trashbin_retention_obligation' => '30',
  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => 'SENSITIVE DATA',
  'appstoreenabled' => true,
  'activity_expire_days' => 21,
  'versions_retention_obligation' => 'auto',
  'enable_avatars' => false,
  'logtimezone' => 'Europe/Zurich',
  'loglevel' => 0,
  'log_rotate_size' => 104857600,
  'filesystem_check_changes' => 1,
  'trusted_domains' => 
  array (
    0 => 'SENSITIVE DATA',
    1 => 'SENSITIVE DATA',
    2 => 'SENSITIVE DATA',
    3 => 'SENSITIVE DATA',
  ),
  'datadirectory' => '/mnt/nfs/data',
  'auth.bruteforce.protection.enabled' => false,
  'skeletondirectory' => '/var/www/owncloud/themes/wolke/core/skeleton/',
  'remember_login_cookie_lifetime' => 1296000,
  'mail_from_address' => 'noreply',
  'mail_domain' => 'SENSITIVE DATA',
  'mail_smtpauthtype' => 'LOGIN',
  'blacklisted_files' => 
  array (
    0 => '.DS_Store',
  ),
  'secret' => 'SENSITIVE DATA',
  'appstore.experimental.enabled' => true,
  'singleuser' => false,
  'mail_smtpport' => '465',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'SENSITIVE DATA',
  'mail_smtppassword' => 'SENSITIVE DATA',
  'mail_smtpsecure' => 'ssl',
  'updatechecker' => false,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'filelocking.enabled' => 'true',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '10.0.222.148',
    'port' => 6379,
    'timeout' => 0,
    'dbindex' => 0,
  ),
  'cron_log' => true,
  'overwrite.cli.url' => 'SENSITIVE DATA',
  'trusted_proxies' => 
  array (
    0 => '10.0.222.89',
    1 => '10.0.222.38',
  ),
);

The output of your Apache/nginx/system log in /var/log/____:

CONTAINS SENSITIVE DATA. No indication of the problem mentioned
1 Like

Thanks for raising this issue. I now realize I seem to have the same issue as of late running it on a shared OVH webhost that is likely using similar load balancing tech.

You have the code check and app code checker disabled, are you sure to run only compatible apps? Did you adapt theming to NC 12, does it work faster with default theme?

Did you open a browser console during loading the page? Any resource which is delayed, scripting errors?

1 Like

Thanks @tflidd for your answer.

Yes we adapted the Theme from ownCloud and did the needed adjustments in CSS
I’ve double checked that we are running only Official Apps from the Nextcloud App Store.
We had a small App who is generating Reports about the user Storage load. But i removed the application and it has not change the game.

I tested the Login and Navigation with the default Nextcloud theme with exluding the
'theme' => 'wolke', parameter in the config.php.
After Excluding i restarted the Apache Server.
Unfortunately nothing has changed. I’ve still 17 Seconds for a normal Login process.

We noticed also that a couple Accounts are affected of a massive throttling with ~35 Seconds

So we can say that the delay or throttleing ist related to the user Acocunt.
Kind of strange since we cant finde any Records in the oc_bruteforce_attempts table or other indicators.

This is the Console output while loading the Page:

Error] Unrecognized Content-Security-Policy directive ‘manifest-src’.
[Log] JQMIGRATE: Migrate is installed, version 1.4.0 (core.js, line 7)
[Warning] Missing plural form in language file (merged-template-prepend.js, line 3650)
[Error] Failed to load resource: the server responded with a status of 404 (Not Found) (purify.min.js.map, line 0)

Could be the manifest-src the problem?

Thanks in advance for your help!

If you turned off the bruteforce protection, there shouldn’t be any entries in the database.

Can you bypass the load balancer and see if a direct login on the server is faster? For NC12, they did change quite a lot of the CSS stuff and with reverse proxies, there is perhaps something broken.

@juliushaertl do you have any ideas regarding these errors? Or you know who to ask?

Hello @tflidd
Thanks for comming back to this topic.

We got the same performance issue when bypass the loadbalancer directly. But we figured out somting went wrong with loading the Apps. TTFB (Time To First Byte) for all applications is very high. See screenshots:

Hope that helps.
Thanks in advance for your effort

Might be unrelated, but for me the issue seems more pronounced under the latest Firefox than with Chromium. Did you test that?

Hello @Krischan
Thanks for your answer.

We get the Same Results with all Browsers.

Here an Update:
We tried yesterday night to deploy a fresh Apache machine with Ubuntu 16.04.04.
We mountet the Data-Storage und connected it to the MySQL-Server but not to the REDIS. Even the Theming was not installed. The Peformance was super and we have seen the light at the end of the tunnel.

After connecting the Redis to the Appache it has started laging again.
Inspecting the REDIS Server has shown that it has no load or high I/O on the Network interface.

After removing this line:
'memcache.locking' => '\\OC\\Memcache\\Redis',
it has worked like expected. so we are wondering if there is somithing wrong with NC 12.0.4.3.

On our other Platforms, we migrated from OC 9.1.6 to 12.0.3 ant the to 12.0.4 without any Problems.

On the system whos affected we updatede from OC 9.1.6 to NC 12.0.4 directly

Update

So i did a Deep dive into REDIS with the MONITOR funktion and noticed that while loading a Page REDIS is going insane with the transactions.

I’ve got tousands of transactions with loading the Page once. Here just one block of the MONITOR log:

1515345445.254258 [0 127.0.0.1:33904] “EXEC”
1515345445.254432 [0 127.0.0.1:33904] “INCRBY” “0a05783b24c9cb834dd7430c1a79ac24/lockfiles/d6aad4cbc3df439e322896aebdcb520e” “1”
1515345445.254507 [0 127.0.0.1:33904] “EXPIRE” “0a05783b24c9cb834dd7430c1a79ac24/lockfiles/d6aad4cbc3df439e322896aebdcb520e” “3600”
1515345445.255699 [0 127.0.0.1:33904] “WATCH” “0a05783b24c9cb834dd7430c1a79ac24/lockfiles/d6aad4cbc3df439e322896aebdcb520e”
1515345445.255807 [0 127.0.0.1:33904] “GET” “0a05783b24c9cb834dd7430c1a79ac24/lockfiles/d6aad4cbc3df439e322896aebdcb520e”
1515345445.255878 [0 127.0.0.1:33904] “MULTI”
1515345445.255999 [0 127.0.0.1:33904] “DEL” “0a05783b24c9cb834dd7430c1a79ac24/lockfiles/d6aad4cbc3df439e322896aebdcb520e”

Memcache Loking is activated

‘memcache.locking’ => ‘\OC\Memcache\Redis’,

On all other Cloud Platforms do we have most of the time only GET requests in REDIS

may that helkps to isolate the problem

@MorrisJobke @nickvergessen Can you help with redis? Or does this look like a bug?

This is due to our locking. Maybe @icewind can have a look if this is all needed, but I guess so. :confused:

It shouldn’t be doing a large amount of locking operations per file load.

You can try installing xray which should be able to log what locking operations are done per request to give some in (but do note the “not for production” warnings).

Hello @icewind
Thanks for your answer

unfortunately i cant activate the xray app.

This is the Output of the Logfile:

Declaration of OCA\XRay\Source\StorageWrapper::copyFromStorage(OCP\Files\Storage\IStorage $sourceStorage, $sourceInternalPath, $targetInternalPath) must be compatible with OCP\Files\Storage::copyFromStorage(OCP\Files\Storage $sourceStorage, $sourceInternalPath, $targetInternalPath) at /var/www/owncloud/apps/xray/lib/Source/StorageWrapper.php#0

have i missed something?

Thanks in advance

33

Hello All

We updated our Instance to NC12.0.5 and have still facing the same issues.
Has anyone a idea where the Problem could sit?

Thanks in advance!

No idears what it could be?

There were already some developers involved, so the debugging goes a bit beyond the scope of this board. Feel free to open an issue on the bugtracker: https://github.com/nextcloud/server/issues

We started a disccousion here: https://github.com/nextcloud/server/issues/8559
and found the Solution here: https://github.com/nextcloud/server/issues/6459

Deleting the content of the preview folder and runf occ files:scan-app-data has solved the issue.

Thanks to everyone who was involved in this disscousion.