Files amount after moving of data directory is wrong (much bigger)

Hello, since few month I was able to successfully move my data directory (as per HowTo: Change / Move data directory after installation) on long time ago installed system with Ubuntu and Nextcloud 15.

Some Server data

Ubuntu 16.04
Apache 2.4.39
php 7.0.33
Nextcloud 15.0.10

I found one strange behavior, after movement amount of reported files via api (https://MyNCDomain/ocs/v2.php/apps/serverinfo/api/v1/info) or Server Information page (https://MyNCDomain/index.php/settings/admin/serverinfo) is more then doubled.
E.g. I have currently 1022244 files reported, but If I count files in FS under data directroy:

find . -type f | wc -l

it will be only 322704 files reported.

I run command to do files re scan and scan-app-data, but without success:

sudo -u www-data php occ files:files:scan --all
sudo -u www-data php occ files:scan-app-data
sudo -u www-data php occ files:cleanup
sudo -u www-data php occ db:add-missing-indices

This graph showing files amount ramp up when data folder was moved and commands from above are run:

Any idea why it is wrongly reported and how to solve this issue?

Could you paste: mysql -e 'select * from nextcloud.oc_storages' to check if there is more than one local:: entry now.
On which dir did you run find . -type f | wc -l in?
E.g. a large amount (multiple preview files for each picture) is located in the appdata dir:

2019-08-01 19:40:47 root@micha:/mnt/sda/ncdata# find . -type f | wc -l
652051
2019-08-01 19:41:47 root@micha:/mnt/sda/ncdata# find ./appdata_<ID> -type f | wc -l
566406

I run it under data folder, here is an output for whole data and appdata_:

$ sudo find data/appdata_ID -type f | wc -l
204388
$ sudo find data/ -type f | wc -l
323014

Here is mysql output with a bit masking:

$ mysql -e 'select * from nextcloud.oc_storages'
+--------------------------------------------------------------+------------+-----------+--------------+
| id                                                           | numeric_id | available | last_checked |
+--------------------------------------------------------------+------------+-----------+--------------+
| local::/var/www/nextcloud/data/ |          1 |         1 |         | NULL |
| home::USER1 |          2 |         1 |         NULL |
| home::USER2 | |          4 |         1 |         NULL |
| home::USER |          5 |         | 1 |         NULL |
| home::USER3 |          6 |         1 |         NULL |
| home::USER4 |          8 |         1 |         NULL | 
| dropbox_external::8xxxxxxxxxxxxxxx// |         10 |         1 | 1537389174 |
| home::USER5 |         11 |         1 |         NULL | 
| dropbox_external::3xxxxxxxxxxxxxxx// |         15 |         1 | 1538729004 |
| home::USER6 |         16 |         1 |         NULL |
| home::USER8 |         | 17 |         1 |         NULL | 
| webdav::USER2@DOMAIN@webdav.magentacloud.de// |         18 | 0 |   1540213359 |
| webdav::USER@DOMAIN@webdav.magentacloud.de// | 19 |         1 |   1564667758 |
| local::/home/USER2/Video/ |         20 |         1 | 1564667757 |
| webdav::USER@DOMAIN@webdav.magentacloud.de// | 21 |         1 |   1564663497 |
| webdav::USER2@webdav.yandex.ru// |         22 |         1 | 1564667757 |
| webdav::USER@webdav.yandex.ru// |         23 |         1 | 1564667757 |
| webdav::USER2@webdav.cloud.mail.ru// |         25 |         0 | 1544089172 |
| webdav::USER2@DOMAIN@USER2@DOMAIN@webdav.cloud.mail.ru// | 27 |         0 |   1544089241 |
| yxyxyxyxyxyxyxyxyxyxyxyxyxyxyxy |         28 |         0 |   | 1544089324 |
| webdav::USER2@DOMAIN@webdav.cloud.mail.ru/// |         | 29 | 0 |   1544089690 |
| home::USER7 |         30 |         1 |         NULL |  |
|webdav::USER2@webdav.cloudme.com/USER2/// |         32 |         1|   1564667760 |
| local::/var/www/dokuiki/data/media/USER2/ |         34 |         1 | 1557903654 |
| local::/var/www/dokuiki/data/media/ |         35 |         1 | 1559661957 |
| local::/var/nextcloud/data/ |         36 |         1 |         NULL | 
| local::/var/www/dokuwiki/data/media/ |         37 |         1 | 1564667759 |
+--------------------------------------------------------------+------------+-----------+--------------+

I can see at least old data and new data folder locations under ID 1 and 36. The /var/www/nextcloud/data/ was old default location and /var/nextcloud/data/is a new one.

The funny thing is that e.g. Dropbox was disabled and deleted from the externals:

sudo -u www-data php /var/www/nextcloud/occ files_external:list
+----------+------------------------------+---------+-----------------------+---------------------+----------------------+------------------+-------------------+
| Mount ID | Mount Point                  | Storage | Authentication Type   | Configuration       | Options              | Applicable Users | Applicable Groups |
+----------+------------------------------+---------+-----------------------+---------------------+----------------------+------------------+-------------------+
| 8        | /MagentaCloud                | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 9        | /Externals/MagentaCloud      | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 10       | /Externals/Local-Videos      | Local   | None                  |                     | enable_sharing: true | USER             |                   |
| 11       | /Externals/Yandex-User1      | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 12       | /Externals/Yandex-User2      | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 14       | /Externals/CloudMe           | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 15       | /Externals/DokuWiki          | Local   | None                  |                     | enable_sharing: true | USER             |                   |
+----------+------------------------------+---------+-----------------------+---------------------+----------------------+------------------+-------------------+

Okay there are multiple local:: entries. Seems like you moved the Nextcloud install dir and data (at least attempted) multiple times?

local::/var/www/nextcloud/data/
local::/home/USER2/Video/
local::/var/www/dokuiki/data/media/USER2/
local::/var/www/dokuiki/data/media/
local::/var/nextcloud/data/
local::/var/www/dokuwiki/data/media/

To be true I am not sure if those include “local” external storages, so when you add a local directory via external storages app.

Also I am not sure how well entries are removed when you remove external storages from within Nextcloud UI, e.g. the two obsolete Dropbox entries.

So before you touch anything, and if downtime is okay, put your Nextcloud into maintenance mode, and do a database dump:

sudo -u www-data php /var/www/nextcloud/occ maintenance:mode --on
sudo mysqldump nextcloud > nextcloud_bak.sql

Then remove the entries that do not match any currently added data dir or external storage:

mysql -e ‘delete from nextcloud.oc_storages where numeric_id = 1’

e.g. to delete the first entry, if this is the old data dir.
You can do the same for the Dropbox entries and others that are definitely not added to Nextcloud anymore or e.g. if the linked directory does not exist anyway.
Use the numeric_id field as this is definitely unique and easiest.

Then disable maintenance mode and check if all users can access their data and external storages as before.
If so then perhaps a cleanup will reduce the file amount in the database but not 100% sure.

But what I am pretty sure about is that all external storage files are added to the database as well (indexed), so perhaps the 1M files matches if you count all files from all external storages as well? And also directories are indexed btw.

2 Likes

Yes, those are local folders added as external storages, funny, but Dropbox was deleted long time ago and still listed in mysql output.

This is an old data location.

This is a new one.

I will try your solution tomorrow. Thanks!

I have successfully did what you wrote about, with only difference: my MariaDB do not want to execute your command and give me a syntax error, so I login into DB and run it from then nextcloud table:

DELETE FROM nextcloud.oc_storages WHERE numeric_id = '1';
DELETE FROM nextcloud.oc_storages WHERE numeric_id = '10';
DELETE FROM nextcloud.oc_storages WHERE numeric_id = '15';

Then I check with command mysql -e 'select * from nextcloud.oc_storages' that wrong storage’s are gone and perform cleanup by command sudo -u www-data php occ maintenance:repair.

The output of repair

sudo -u www-data php occ maintenance:repair

  • Repair MySQL collation
    • All tables already have the correct collation -> nothing to do
  • Repair mime types
  • Clean tags and favorites
    • 0 tags of deleted users have been removed.
    • 0 tags for delete files have been removed.
    • 0 tag entries for deleted tags have been removed.
    • 0 tags with no entries have been removed.
  • Repair invalid shares
  • Remove shares of a users root folder
  • Move .step file of updater to backup location
  • Fix potential broken mount points
    • No mounts updated
  • Repair invalid paths in file cache
  • Add log rotate job
  • Clear frontend caches
    • Image cache cleared
    • SCSS cache cleared
    • JS cache cleared
  • Clear every generated avatar on major updates
  • Add preview background cleanup job
  • Queue a one-time job to cleanup old backups of the updater
  • Repair pending cron jobs
    • No need to repair pending cron jobs.
  • Extract the vcard uid and store it in the db
  • Remove potentially over exposing share links
    • No need to remove link shares.
  • Fix component of birthday calendars
    • 2 birthday calendars updated.
  • Fix broken values of calendar objects
    0 [>---------------------------]
  • Registering building of calendar search index as background job
    • Repair step already executed
  • Registering background jobs to update cache for webcal calendars
    • Added 0 background jobs to update webcal calendars
  • Clean up orphan event and contact data
    • 0 events without a calendar have been cleaned up
    • 0 properties without an events have been cleaned up
    • 0 changes without a calendar have been cleaned up
    • 0 contacts without an addressbook have been cleaned up
    • 0 properties without a contact have been cleaned up
    • 0 changes without an addressbook have been cleaned up
  • Remove activity entries of private events
    • Removed 0 activity entries
  • Delete orphaned ACL rules
  • Fix the share type of guest shares when migrating from ownCloud
  • Copy the share password into the dedicated column
  • Purify and migrate collected mail addresses
    0 [>---------------------------]
    1 [============================]
  • Update OAuth token expiration times
  • Add background job to check for backup codes

It is still reported about 1M of files and 24 Storages (what is less then before). I will run files:scan --all and files:cleanup to rescan the system and cleanup the cache, but this will take a while.

P.S. Seems it works somehow, during the rescan I can see that files count goes slowly down:

1 Like

Ah yeah sorry quoting required in mysql syntax for field values. But it shouldn’t be case sensitive, at least I never use capital letter.

Okay great that the file count went down now with the clean-up. Although it looks like going to zero? Perhaps the cache gets rebuild or something when waiting for or forcing the next scan --all.

It was just capital written in example that I took, I believe it is not case sensitive.

:slight_smile: It is only such kind of zoom (between 1M and 800K):

Ah lol yes I didn’t check the axis scale :smile:.

Everything goes good, Files amount is around 400K and DB is more then twice smaller as before. Seems this command helps to do cleanup at the end: