Files amount after moving of data directory is wrong (much bigger)

gas85 · July 31, 2019, 12:54pm

Hello, since few month I was able to successfully move my data directory (as per HowTo: Change / Move data directory after installation) on long time ago installed system with Ubuntu and Nextcloud 15.

Some Server data

Ubuntu 16.04
Apache 2.4.39
php 7.0.33
Nextcloud 15.0.10

I found one strange behavior, after movement amount of reported files via api (https://MyNCDomain/ocs/v2.php/apps/serverinfo/api/v1/info) or Server Information page (https://MyNCDomain/index.php/settings/admin/serverinfo) is more then doubled.
E.g. I have currently 1022244 files reported, but If I count files in FS under data directroy:

find . -type f | wc -l

it will be only 322704 files reported.

I run command to do files re scan and scan-app-data, but without success:

sudo -u www-data php occ files:scan --all
sudo -u www-data php occ files:scan-app-data
sudo -u www-data php occ files:cleanup
sudo -u www-data php occ db:add-missing-indices

This graph showing files amount ramp up when data folder was moved and commands from above are run:

Any idea why it is wrongly reported and how to solve this issue?

MichaIng · August 1, 2019, 5:43pm

Could you paste: mysql -e 'select * from nextcloud.oc_storages' to check if there is more than one local:: entry now.
On which dir did you run find . -type f | wc -l in?
E.g. a large amount (multiple preview files for each picture) is located in the appdata dir:

2019-08-01 19:40:47 root@micha:/mnt/sda/ncdata# find . -type f | wc -l
652051
2019-08-01 19:41:47 root@micha:/mnt/sda/ncdata# find ./appdata_<ID> -type f | wc -l
566406

gas85 · August 2, 2019, 7:27am

I run it under data folder, here is an output for whole data and appdata_:

$ sudo find data/appdata_ID -type f | wc -l
204388
$ sudo find data/ -type f | wc -l
323014

Here is mysql output with a bit masking:

$ mysql -e 'select * from nextcloud.oc_storages'
+--------------------------------------------------------------+------------+-----------+--------------+
| id                                                           | numeric_id | available | last_checked |
+--------------------------------------------------------------+------------+-----------+--------------+
| local::/var/www/nextcloud/data/ |          1 |         1 |         | NULL |
| home::USER1 |          2 |         1 |         NULL |
| home::USER2 | |          4 |         1 |         NULL |
| home::USER |          5 |         | 1 |         NULL |
| home::USER3 |          6 |         1 |         NULL |
| home::USER4 |          8 |         1 |         NULL | 
| dropbox_external::8xxxxxxxxxxxxxxx// |         10 |         1 | 1537389174 |
| home::USER5 |         11 |         1 |         NULL | 
| dropbox_external::3xxxxxxxxxxxxxxx// |         15 |         1 | 1538729004 |
| home::USER6 |         16 |         1 |         NULL |
| home::USER8 |         | 17 |         1 |         NULL | 
| webdav::USER2@DOMAIN@webdav.magentacloud.de// |         18 | 0 |   1540213359 |
| webdav::USER@DOMAIN@webdav.magentacloud.de// | 19 |         1 |   1564667758 |
| local::/home/USER2/Video/ |         20 |         1 | 1564667757 |
| webdav::USER@DOMAIN@webdav.magentacloud.de// | 21 |         1 |   1564663497 |
| webdav::USER2@webdav.yandex.ru// |         22 |         1 | 1564667757 |
| webdav::USER@webdav.yandex.ru// |         23 |         1 | 1564667757 |
| webdav::USER2@webdav.cloud.mail.ru// |         25 |         0 | 1544089172 |
| webdav::USER2@DOMAIN@USER2@DOMAIN@webdav.cloud.mail.ru// | 27 |         0 |   1544089241 |
| yxyxyxyxyxyxyxyxyxyxyxyxyxyxyxy |         28 |         0 |   | 1544089324 |
| webdav::USER2@DOMAIN@webdav.cloud.mail.ru/// |         | 29 | 0 |   1544089690 |
| home::USER7 |         30 |         1 |         NULL |  |
|webdav::USER2@webdav.cloudme.com/USER2/// |         32 |         1|   1564667760 |
| local::/var/www/dokuiki/data/media/USER2/ |         34 |         1 | 1557903654 |
| local::/var/www/dokuiki/data/media/ |         35 |         1 | 1559661957 |
| local::/var/nextcloud/data/ |         36 |         1 |         NULL | 
| local::/var/www/dokuwiki/data/media/ |         37 |         1 | 1564667759 |
+--------------------------------------------------------------+------------+-----------+--------------+

I can see at least old data and new data folder locations under ID 1 and 36. The /var/www/nextcloud/data/ was old default location and /var/nextcloud/data/is a new one.

The funny thing is that e.g. Dropbox was disabled and deleted from the externals:

sudo -u www-data php /var/www/nextcloud/occ files_external:list
+----------+------------------------------+---------+-----------------------+---------------------+----------------------+------------------+-------------------+
| Mount ID | Mount Point                  | Storage | Authentication Type   | Configuration       | Options              | Applicable Users | Applicable Groups |
+----------+------------------------------+---------+-----------------------+---------------------+----------------------+------------------+-------------------+
| 8        | /MagentaCloud                | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 9        | /Externals/MagentaCloud      | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 10       | /Externals/Local-Videos      | Local   | None                  |                     | enable_sharing: true | USER             |                   |
| 11       | /Externals/Yandex-User1      | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 12       | /Externals/Yandex-User2      | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 14       | /Externals/CloudMe           | WebDAV  | Username and password |                     | enable_sharing: true | USER             |                   |
| 15       | /Externals/DokuWiki          | Local   | None                  |                     | enable_sharing: true | USER             |                   |
+----------+------------------------------+---------+-----------------------+---------------------+----------------------+------------------+-------------------+

MichaIng · August 2, 2019, 4:55pm

Okay there are multiple local:: entries. Seems like you moved the Nextcloud install dir and data (at least attempted) multiple times?

local::/var/www/nextcloud/data/
local::/home/USER2/Video/
local::/var/www/dokuiki/data/media/USER2/
local::/var/www/dokuiki/data/media/
local::/var/nextcloud/data/
local::/var/www/dokuwiki/data/media/

To be true I am not sure if those include “local” external storages, so when you add a local directory via external storages app.

Also I am not sure how well entries are removed when you remove external storages from within Nextcloud UI, e.g. the two obsolete Dropbox entries.

So before you touch anything, and if downtime is okay, put your Nextcloud into maintenance mode, and do a database dump:

sudo -u www-data php /var/www/nextcloud/occ maintenance:mode --on
sudo mysqldump nextcloud > nextcloud_bak.sql

Then remove the entries that do not match any currently added data dir or external storage:

mysql -e ‘delete from nextcloud.oc_storages where numeric_id = 1’

e.g. to delete the first entry, if this is the old data dir.
You can do the same for the Dropbox entries and others that are definitely not added to Nextcloud anymore or e.g. if the linked directory does not exist anyway.
Use the numeric_id field as this is definitely unique and easiest.

Then disable maintenance mode and check if all users can access their data and external storages as before.
If so then perhaps a cleanup will reduce the file amount in the database but not 100% sure.

But what I am pretty sure about is that all external storage files are added to the database as well (indexed), so perhaps the 1M files matches if you count all files from all external storages as well? And also directories are indexed btw.

gas85 · August 6, 2019, 2:37pm

Yes, those are local folders added as external storages, funny, but Dropbox was deleted long time ago and still listed in mysql output.

This is an old data location.

This is a new one.

I will try your solution tomorrow. Thanks!

gas85 · August 7, 2019, 7:57am

I have successfully did what you wrote about, with only difference: my MariaDB do not want to execute your command and give me a syntax error, so I login into DB and run it from then nextcloud table:

DELETE FROM nextcloud.oc_storages WHERE numeric_id = '1';
DELETE FROM nextcloud.oc_storages WHERE numeric_id = '10';
DELETE FROM nextcloud.oc_storages WHERE numeric_id = '15';

Then I check with command mysql -e 'select * from nextcloud.oc_storages' that wrong storage’s are gone and perform cleanup by command sudo -u www-data php occ maintenance:repair.

The output of repair

sudo -u www-data php occ maintenance:repair

Repair MySQL collation
- All tables already have the correct collation -> nothing to do
Repair mime types
Clean tags and favorites
- 0 tags of deleted users have been removed.
- 0 tags for delete files have been removed.
- 0 tag entries for deleted tags have been removed.
- 0 tags with no entries have been removed.
Repair invalid shares
Remove shares of a users root folder
Move .step file of updater to backup location
Fix potential broken mount points
- No mounts updated
Repair invalid paths in file cache
Add log rotate job
Clear frontend caches
- Image cache cleared
- SCSS cache cleared
- JS cache cleared
Clear every generated avatar on major updates
Add preview background cleanup job
Queue a one-time job to cleanup old backups of the updater
Repair pending cron jobs
- No need to repair pending cron jobs.
Extract the vcard uid and store it in the db
Remove potentially over exposing share links
- No need to remove link shares.
Fix component of birthday calendars
- 2 birthday calendars updated.
Fix broken values of calendar objects
0 [>---------------------------]
Registering building of calendar search index as background job
- Repair step already executed
Registering background jobs to update cache for webcal calendars
- Added 0 background jobs to update webcal calendars
Clean up orphan event and contact data
- 0 events without a calendar have been cleaned up
- 0 properties without an events have been cleaned up
- 0 changes without a calendar have been cleaned up
- 0 contacts without an addressbook have been cleaned up
- 0 properties without a contact have been cleaned up
- 0 changes without an addressbook have been cleaned up
Remove activity entries of private events
- Removed 0 activity entries
Delete orphaned ACL rules
Fix the share type of guest shares when migrating from ownCloud
Copy the share password into the dedicated column
Purify and migrate collected mail addresses
0 [>---------------------------]
1 [============================]
Update OAuth token expiration times
Add background job to check for backup codes

It is still reported about 1M of files and 24 Storages (what is less then before). I will run files:scan --all and files:cleanup to rescan the system and cleanup the cache, but this will take a while.

P.S. Seems it works somehow, during the rescan I can see that files count goes slowly down:

MichaIng · August 7, 2019, 11:33am

Ah yeah sorry quoting required in mysql syntax for field values. But it shouldn’t be case sensitive, at least I never use capital letter.

Okay great that the file count went down now with the clean-up. Although it looks like going to zero? Perhaps the cache gets rebuild or something when waiting for or forcing the next scan --all.

gas85 · August 7, 2019, 11:36am

It was just capital written in example that I took, I believe it is not case sensitive.

It is only such kind of zoom (between 1M and 800K):

MichaIng · August 7, 2019, 11:49am

Ah lol yes I didn’t check the axis scale .

gas85 · August 9, 2019, 7:35am

Everything goes good, Files amount is around 400K and DB is more then twice smaller as before. Seems this command helps to do cleanup at the end: