My oc_filecache is growing since 3 day

Nextcloud version (eg, 18.0.2): 18.0.8
Operating system and version (eg, Ubuntu 20.04): Debian 9
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.25
PHP version (eg, 7.1): 7.3 FPM

The issue you are facing:
In the last 3 day, the OC_filecache have grown uncontrollably.
My nextcloud instance have always (4Y) been running on a small HDD, and the /var/lib/mysql mount never outgrow a gigabit. 3 day ago i received a notification from netdata telling me that my disk was > 95%, i checked and though it might be normal. I added 5GB through LVM and watched.
in the next hours a couple hundred MB were writed to it, and in a day 2 new GB of data was there
It’s been 2 day now and i’m once again >90% with 7gb of storage

for information, yesterday ~ 4pm :

root@nxcld:/var/lib/mysql/nextcloud# du -sx * | sort -n | tail -5                                                                                                                                                                      
8196    oc_ocsms_smsdatas.ibd
12292   oc_audioplayer_albums.ibd
102404  oc_news_items.ibd
397324  oc_file_locks.ibd
2629700 oc_filecache.ibd

and today 11 am:

root@nxcld:/var/lib/mysql/nextcloud# du -sx * | sort -n | tail -5                                                                                                                                                                      
8196    oc_ocsms_smsdatas.ibd                                                                                                                                                                                                                 
12292   oc_audioplayer_albums.ibd                                                                                                                                                                                                             
102404  oc_news_items.ibd                                                                                                                                                                                                                     
720908  oc_file_locks.ibd                                                                                                                                                                                                                     
4694132 oc_filecache.ibd

i don’t have any error in log, and i upgrade after seeing the issue 3 day ago just to be sure (minor 18 upgrade)
i am a single user, with only cifs external share
the only recent change i did was to disable Atime on my nas

I don’t aprear to have duplicate in the db:

MariaDB [nextcloud]> SELECT * FROM oc_filecache where ( path not like Concat('%/',name) ) AND path <> name                                                                                                                                    
    -> ;                                                                                                                                                                                                                                      
+--------+---------+------+----------------------------------+--------+-------+----------+----------+---------------+------------+---------------+-----------+------------------+---------------+-------------+----------+                    
| fileid | storage | path | path_hash                        | parent | name  | mimetype | mimepart | size          | mtime      | storage_mtime | encrypted | unencrypted_size | etag          | permissions | checksum |                    
+--------+---------+------+----------------------------------+--------+-------+----------+----------+---------------+------------+---------------+-----------+------------------+---------------+-------------+----------+                    
| 214187 |       8 |      | d41d8cd98f00b204e9800998ecf8427e |     -1 | Films |        2 |        1 | 5254745590163 | 1560743536 |    1548500266 |         0 |                0 | 5d070e707d275 |          17 |          |
| 214188 |       9 |      | d41d8cd98f00b204e9800998ecf8427e |     -1 | jeux  |        2 |        1 | 1018794001926 | 1548590573 |    1548500269 |         0 |                0 | 5c4d9deda0add |          17 |          |
+--------+---------+------+----------------------------------+--------+-------+----------+----------+---------------+------------+---------------+-----------+------------------+---------------+-------------+----------+
2 rows in set (1 min 58.947 sec)

there is no cleanup to do

user@nxcld:/var/www/nextcloud$ sudo -u www-data php ./occ files:cleanup                                                                                                                                                           
0 orphaned file cache entries deleted
my nextcloud specific service/timer
# /etc/systemd/system/nextcloudnewsupdater.service                                                                                                                                                                                   [35/5286]
[Unit]                                                                                                                                                                                                                                        
After=default.target                                                                                                                                                                                                                          
Description=Nextcloud News Updater                                                                                                                                                                                                            
                                                                                                                                                                                                                                              
[Service]                                                                                                                                                                                                                                     
Type=simple                                                                                                                                                                                                                                   
User=www-data                                                                                                                                                                                                                                 
ExecStart=/usr/bin/nextcloud-news-updater -c /etc/nextcloud/news/updater.ini                                                                                                                                                                  
                                                                                                                                                                                                                                              
[Install]                                                                                                                                                                                                                                     
WantedBy=default.target                                                                                                                                                                                                                       
                                                                                                                                                                                                                                              
# /etc/systemd/system/nextcloudcron.service                                                                                                                                                                                                   
[Unit]                                                                                                                                                                                                                                        
Description=Nextcloud cron.php job                                                                                                                                                                                                            
                                                                                                                                                                                                                                              
[Service]                                                                                                                                                                                                                                     
User=www-data                                                                                                                                                                                                                                 
ExecStart=/usr/bin/php -f /var/www/nextcloud/cron.php                                                                                                                                                                                         
                                                                                                                                                                                                                                              
[Install]                                                                                                                                                                                                                                     
WantedBy=basic.target                                                                                                                                                                                                                         
                                                                                                                                                                                                                                              
# /etc/systemd/system/nextcloudcron.timer                                                                                                                                                                                                     
[Unit]                                                                                                                                                                                                                                        
Description=Run Nextcloud cron.php every 5 minutes                                                                                                                                                                                            
                                                                                                                                                                                                                                              
[Timer]                                                                                                                                                                                                                                       
OnBootSec=5min                                                                                                                                                                                                                                
OnUnitActiveSec=5min                                                                                                                                                                                                                          
Unit=nextcloudcron.service                                                                                                                                                                                                                    
                                                                                                                                                                                                                                              
[Install]                                                                                                                                                                                                                                     
WantedBy=timers.target

# /etc/systemd/system/nextcloudpreview.timer
[Unit]
Description=Run Nextcloud preview generator every night

[Timer]
OnCalendar=*-*-* 02:00:00
Unit=nextcloudpreview.service

[Install]
WantedBy=timers.target

i have honestly no idea what to do…

Is this the first time you’ve seen this error? (Y/N): Y

The output of your Nextcloud log in Admin > Logging:

### THIS WAS WHEN I TRIED TO DO  php ./occ log:tail
Declaration of Symfony\Component\Console\Style\OutputStyle::write($messages, bool $newline = false, int $type = self::OUTPUT_NORMAL) must be compatible with Symfony\Component\Console\Output\OutputInterface::write($messages, $newline = false, $options = 0) at /var/www/nextcloud/apps/bookmarks/vendor/symfony/console/Style/OutputStyle.php#24

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php                                                                                                                                                                                                                                         
$CONFIG = array (                                                                                                                                                                                                                             
  'instanceid' => 'TRUNCATED',                                                                                                                                                                                                             
  'passwordsalt' => 'TRUNCATED',                                                                                                                                                                                         
  'secret' => 'TRUNCATED',                                                                                                                                                                             
  'trusted_domains' =>                                                                                                                                                                                                                        
  array (                                                                                                                                                                                                                                     
    0 => 'nextcloud.TRUNCATED.com',                                                                                                                                                                                                             
  ),                                                                                                                                                                                                                                          
  'trusted_proxies' =>                                                                                                                                                                                                                        
  array (                                                                                                                                                                                                                                     
    0 => 'TRUNCATED',                                                                                                                                                                                                                      
    1 => 'TRUNCATED',                                                                                                                                                                                                                      
  ),                                                                                                                                                                                                                                          
  'forwarded_for_headers' =>                                                                                                                                                                                                                  
  array (                                                                                                                                                                                                                                     
    0 => 'HTTP_X_FORWARDED',                                                                                                                                                                                                                  
    1 => 'HTTP_FORWARDED_FOR',                                                                                                                                                                                                                
  ),                                                                                                                                                                                                                                          
  'datadirectory' => '/var/www/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '18.0.8.2',
  'overwrite.cli.url' => 'http://nextcloud-pp.TRUNCATED.com',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud_user',
  'dbpassword' => 'TRUNCATED',
  'installed' => true,
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'mail_smtpmode' => 'sendmail',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'nextcloud',
  'mail_domain' => 'TRUNCATED.com',
  'twofactor_enforced' => 'false',
  'twofactor_enforced_groups' =>
  array (
    0 => 'admin',
  ),
'twofactor_enforced_excluded_groups' =>
  array (
  ),
  'trashbin_retention_obligation' => 'auto, 15',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 0,
  'app_install_overwrite' =>
  array (
    0 => 'apporder',
  ),
  'enable_previews' => true,
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\GIF',
    3 => 'OC\\Preview\\HEIC',
    4 => 'OC\\Preview\\BMP',
    5 => 'OC\\Preview\\XBitmap',
    6 => 'OC\\Preview\\MP3',
    7 => 'OC\\Preview\\TXT',
    8 => 'OC\\Preview\\MarkDown',
    9 => 'OC\\Preview\\PDF',
    10 => 'OC\\Preview\\SVG',
    11 => 'OC\\Preview\\Movie',
  ),
  'updater.secret' => 'TRUNCATED',
);

The output of your Apache/nginx/system log in /var/log/____:

Nothing in here aside 302 from haproxy and davx5 request

Additional error now that the DB is full

Error	lib	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'INSERT INTO `oc_file_locks` (`key`, `lock`, `ttl`) VALUES(?, ?, ?)' with params ["files\/4e75bda0a32579541fcac4252eff2f39", 1, 1598794424]: SQLSTATE[HY000]: General error: 1114 The table 'oc_file_locks' is full	
2020-08-30T14:33:44+0200
Error	lib	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'INSERT INTO `oc_file_locks` (`key`, `lock`, `ttl`) VALUES(?, ?, ?)' with params ["files\/183603213827e71eb098207e09b82fe3", 1, 1598794424]: SQLSTATE[HY000]: General error: 1114 The table 'oc_file_locks' is full	
2020-08-30T14:33:44+0200
Error	PHP	session_write_close(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/sessions) at /var/www/nextcloud/lib/private/Session/Internal.php#211	
2020-08-30T14:33:44+0200
Error	PHP	session_write_close(): write failed: No space left on device (28) at /var/www/nextcloud/lib/private/Session/Internal.php#211	
2020-08-30T14:33:44+0200
Error	lib	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'INSERT INTO `oc_file_locks` (`key`, `lock`, `ttl`) VALUES(?, ?, ?)' with params ["files\/4e75bda0a32579541fcac4252eff2f39", 1, 1598794420]: SQLSTATE[HY000]: General error: 1114 The table 'oc_file_locks' is full	
2020-08-30T14:33:40+0200
Error	lib	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'INSERT INTO `oc_file_locks` (`key`, `lock`, `ttl`) VALUES(?, ?, ?)' with params ["files\/183603213827e71eb098207e09b82fe3", 1, 1598794420]: SQLSTATE[HY000]: General error: 1114 The table 'oc_file_locks' is full	
2020-08-30T14:33:40+0200
Error	PHP	session_write_close(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/sessions) at /var/www/nextcloud/lib/private/Session/Internal.php#211

This looks like a similar problem: Large oc_filecache

Weird i can’t see the message i have send yesterday :frowning:

I have tried to cleanup like i’ve seen in your post, also going over the lock table.
After 24H, it does not look like it’s growing again, which make no sens to me …
how can purging a db stop it from growning back up :astonished:

Anyway, stable for now. any idea what have caused this ?
I could recover because i was on lvm with dynamic partition, but if this was on a HDD, no way to make space in order to clean the db … it could be dramatic for a company.

And here we go again, not 2 day and my /var is almost full again

    4,0 GiB [##########]  oc_filecache.ibd
    1,2 GiB [##        ]  oc_file_locks.ibd
  100,0 MiB [          ]  oc_news_items.ibd

I have 7H in front of me before the server is down again…

Any idea ?

I have experienced the same issue.

We use external storages, and from what I can tell, whenever a user “touches” an external storage, NextCloud then proceeds to cache a file handle for every file in that storage, for the given user … the file handles all going into oc_filecache.

We also saw some problems with oc_storages, which I believe are related.

The question then to the Dev community – what is the exact behavior that is going on here? And for situations where we have external storages with millions of files, is there a way to turn off this file handle caching behavior, or control it in some manner?

Thanks. John

I found that my issue was caused by having the .snapshot suddenly present.
The number of file virtually did a x20 and no way to disable or exclude something form nextcloud.

Owncloud seam to have a php option to list file/folder to exclude but it didn’t work here so it might be from after to fork

1 Like

In which directory did you get this file?

Hello :slight_smile:
This isn’t a file, it’s not even a folder.
My file system support snapshot, and expose them through this .snapshot “folder”.
In them I can find the 20 last version of the entire nas.
It shows up at the Mount point (I used the external storage from the UI to Mount this cifs share.)

1 Like