Automate OCC Filescan

[K-MTG]

How can I use OCC (or something else?) to trigger a rescan of external storage whenever a change is made?

I have Nextcloud running on Docker on my Mac with this docker-compose. I made a small tweak to it so I can access my external storage (external hard drive):

  app:
build: ./app
restart: always
volumes:
  - nextcloud:/var/www/html
  **- /Volumes/LaCie:/Volume**
environment:
  - VIRTUAL_HOST=email@emaple.com
  - LETSENCRYPT_HOST=email@emaple.com
  - LETSENCRYPT_EMAIL=email@emaple.com
  - MYSQL_HOST=db
env_file:
  - db.env
depends_on:
  - db
  - redis
networks:
  - proxy-tier
  - default

I was able to successfully enable local storage:

Immediately after setting up local storage, I am able to access all of my files and the data size states pending. However, after sometime only the files I accessed remain. Hence, if I have a TV show folder and click on the show SpongeBob and select season 1. All episodes from season 1 will load and the data size will update. However, if I don’t select and open the other seasons (i.e season 2,3,4) those folders will disappear from nextcloud.
Also, if I create a new folder with the same name name (i.e Season 2). Nextcloud will state Season 2 already exist and the folder will show up.

Running the command php occ files:scan --all under the user www-data appears to have done the job…all of my external storage files are showing! However, is there a better way to implement this code somewhere? I think running this script every time will use a lot of resources…is there an alternative solution?

[gas85]

By the way files:scan --all will rescan whole file system together with externals… https://docs.nextcloud.com/server/13/admin_manual/configuration_server/occ_command.html#file-operations
I used this script to rescan external shares only. Script will fetch data about all external storages of all users and perform scan of them. I put it in cron like twice per day and it consumes much less resources in compare to “rescan all” command

[K-MTG]

Thanks for the info! Can you please provide the path where you stored this script and how you were able to call it with cron?
Also, I noticed some features on the script are commented. Can you please share what you un-commented to automate scanning on external drives.

Thanks

[gas85]

Basically it works out from the box. Only that you have to check you nextcloud path, log path and create a log file for php occ output.

I put it in

/usr/local/bin/

with chmod 755

I run it under nextcloud user (for me it is www-data) basically twice per day at 2:30 and 14:30. You can run it also hourly. This is my cron config:

30 2,14 * * * perl -e 'sleep int(rand(1800))' && /usr/local/bin/nextcloud-file-sync.sh #Nextcloud file sync

Here I add perl -e ‘sleep int(rand(1800))’ to inject some random start time within 30 Minutes, but since it scans externals only it is not necessary any more. Your cron job config to run it hourly could be:

* */1 * * * /usr/local/bin/nextcloud-file-sync.sh

Lets go through what it does (valid for commit 44d9d2f):

COMMAND=/var/www/nextcloud/occ ← This is where your nextcloud OCC command located
OPTIONS=“files:scan” ← This is “Command” to run, just live it as it is
LOCKFILE=/tmp/nextcloud_file_scan ← Lock file to not execute script twice, if already ongoing
LOGFILE=/var/www/nextcloud/data/nextcloud.log ← Location of Nextcloud LOG file, will put some logs in Nextcloud format
CRONLOGFILE=/var/log/next-cron.log ← location for bash log. In case when there is an output by command generated. AND IT IS GENERATED…

Line 22 will generate NC log input. You will see it in a GUI as:

From the line 26 starts the job, basically it is left from an older version of script and it is exactly what you done - scan all users, all shares, all locals with all folders. It takes ages to perform an a big installations, so I commented it.

Second option (line 31) is to scan for specific user, but as soon as I get more than one user with external shares it does not work also. Besides it is still scanning whole partition (local and remote) for specific user - commented.

From line 35 till 42 comments how to not forget how I get users from the NC, basically everything is happens in line 45, scipt will generate exactly path for external shares to be updated for all users (you can run it and test output). Here an example command:

sudo -u www-data php occ files_external:list | awk -F'|' '{print $8"/files"$3}'| tail -n +4 | head -n -1 | awk '{gsub(/ /, "", $0); print}'

and output:

user1/files/Dropbox-user1
user2/files/Dropbox-user2
user1/files/MagentaCloud
user2/files/MagentaCloud
user1/files/Local-Folder

Those lines will be read one by one and synced in line 49.

After this script will generate NC log output:

I have had some issues (like described here OCC files:cleanup - Does it delete the db table entries of the missing files?) in older NC versions, so I added workaround from line 60 till 67 as files:cleanup command, nut sure if it is needed now, but it does not harm anything.

[K-MTG]

I have a few docker containers for my nextcloud install. Which one should I use for this script?

Here is what my current cron.php looks like:

Where do I put this script:

30 2,14 * * * perl -e 'sleep int(rand(1800))' && /usr/local/bin/nextcloud-file-sync.sh #Nextcloud file sync

Just to confirm, my nextcloud-file-sync.sh file will be in the same directory?

And just to confirm…I don’t need to modify the nextcloud-file-sync.sh file?

So I have all of my external shared mounted under the admin user (and certain shares are shared with certain users). My users don’t have the ability to mount their own external storage.

Thanks

[gas85]

I do not know your installation. If you have few dockers as replicant of your cloud with DB inside, not sure if this script could help you at all.

If you have few dockers that connected to one independent db - theoretically you can run it on one of them and it will do the job for others. But container could be closed and job will be not finished.

If you have few dockers with own db each - run it everywhere where external shares are presented.

With Docker it is a bit more complicated to run a cron job (not related to cron.php from nextcloud, but to system cron). Try to read this https://stackoverflow.com/questions/37458287/how-to-run-a-cron-job-inside-a-docker-container and find your best way to do it.

As I wrote you also do not needed to run it with random time shift especially under docker, use simple line. Again, this is not cron.php, this is system cron.

Steps to do:

1. Put script in whatever place within you NC Docker container.
2. Adjust header of script if your NC installation folder is different from what is in the header. Like COMMAND=/Your/NC/Path and for LOGFILE also.
3. Read how to run cron job under Docker
4. Create cron job (not cron.php) for you NC user with simple line like:

* */1 * * * /path/to/the/nextcloud-file-sync.sh

5. Test it

[gas85]

I did update the script, it does not necessary now set LOG files path, it will drop all the output if logs are note presented or not writeable.
Also I add option -all to make it rescan whole partition in an old way. If you run it wihtout any input, it will do rescan of external shares only. You only need to check header and untouch the rest.

# Adjust to your NC installation
    # Your NC OCC Command path
COMMAND=/var/www/nextcloud/occ
    # Your NC log file path
LOGFILE=/var/www/nextcloud/data/nextcloud.log
    # Your log file path for other output if needed
CRONLOGFILE=/var/log/next-cron.log
    # If you want to perform cache cleanup, please change CACHE value to 1
CACHE=0

[K-MTG]

I basically copied my docker-setup from here for nextcloud: https://github.com/nextcloud/docker/tree/master/.examples/docker-compose/with-nginx-proxy/mariadb-cron-redis/apache

So the configuration already had a docker file inside the app folder…so I simply added the command to it (hopefully correctly):

And I copied the code of the nextcloud-file-sync.sh file and placed it in /usr/local/bin with vim (I created the file and copied/pasted the code.):

The only thing I changed in the nextcloud-file-sync.sh file is the file path:
# Adjust to your NC installation
# Your NC OCC Command path
COMMAND=/var/src/nextcloud/occ
# Your NC log file path
LOGFILE=/var/src/nextcloud/data/nextcloud.log
# Your log file path for other output if needed
CRONLOGFILE=/var/log/next-cron.log
# If you want to perform cache cleanup, please change CACHE value to 1
CACHE=0

I ran docker-compose up -d and waited an hour…but I don’t think the filescan occurred.

I believe I am creating the cron job incorrectly in the dockerfile. Any tips?

Thanks

[K-MTG]

@gas85 do you know if it’s possible to run the cron script directly on bash so I can confirm if it’s working? Currently I am not sure if the script isn’t working or if my call to run the script isn’t.

Thanks!

[gas85]

Of course run it under your NC user (e.g. www-data). Not sure how it is under docker, but on normal (e.g. Ubuntu) system it is something like:

sudo -u www-data nextcloud-file-sync.sh

[K-MTG]

I made the .sh file executable: chmod +x nextcloud-file-sync.sh

And had to adjust the file paths in the sh file:

# Adjust to your NC installation
	# Your NC OCC Command path
COMMAND=/var/www/html/occ
	# Your NC log file path
LOGFILE=/var/www/html/data/nextcloud.log
	# Your log file path for other output if needed
CRONLOGFILE=/var/www/html/data/next-cron.log
	# If you want to perform cache cleanup, please change CACHE value to 1
CACHE=0

When I just ran nextcloud-file-sync.sh in terminal. I got:

And when I ran sudo -u www-data nextcloud-file-sync.sh (had to install sudo in my docker container). I got:

WARNING - could not write to Log file /var/www/html/data/next-cron.log, will drop log messages. Is User Correct?
root@c2663b1e77ea:/usr/local/bin#

In both scenarios, my file path didn’t update. So I am assuming I need to edit something in the sh file so it scans my paths?

Thanks

[K-MTG]

@gas85 any idea why the script doesn’t update after the scan? Do I have to make any modifications (please see above post)

Thanks

[gas85]

Hey. It looks like file /var/www/html/data/next-cron.log does not exist, or not writable. Thats all, not a big issue. You will not see output of occ, but it will not stop because of it.

Regarding sudo, not sure if you need to install it. I thought that container user is your NC user, so you should be able to run your script directly under command line in container. Sorry, I did not work with containers directly, so I couldn’t help you how to run script there.

30 августа 2018 г. 21:02:02 GMT+02:00, K-MTG noreply@nextcloud.com пишет:

[K-MTG]

I think I have the script running. But it seems to be scanning the wrong thing…because it says scan completed but nothing updates.

[gas85]

Try to run (under NC user):

php occ files_external:list

you should see something similar to this:

This will output list of all external shares. Check that it is not empty.

Then try to run and check the output (it should be list of USERNAME/files/SHARENAME):

php occ files_external:list | awk -F'|' '{print $8"/files"$3}'| tail -n +4 | head -n -1 | awk '{gsub(/ /, "", $0); print}'

Then take one path (from the 1st table, or last adopted list) and run command to check what happens:

php occ files:scan --path="/USERNAME/files/Mount_Point"

Every thing must bus executed under your NC user. For Ubuntu I used sudo -u www-data command as www-data is my NC user. Hope it helps you in trouble shooting.

[Dagon84]

Dear Friend,

thank you for your script!

I have the same issue with external FTP upload and NextCloud, but i have a FTP PLAN on a shared server.

Is there any way to run this scxript on a Shared FTP ?

I do have a cPanel where i manage my server space, but I’m not sure to have the possibility to follow the instruction that you have wrote (uoload iun user/local/bin/ ecc).

Any help would be appreciated!

[gas85]

If you means

@gas85

this script to rescan external shares only

then yes, it works for all external shares independent of type (ftp, smb, local, etc.).

@Dagon84

I do have a cPanel where i manage my server space, but I’m not sure to have the possibility to follow the instruction that you have wrote (uoload iun user/local/bin/ ecc).

You can upload this script to anywhere and added it to the cron job.
The aim of this script is to reduce scanning time by focused scan of external shares only. Otherwise you can run (as cron job also) following commands:

1. if you have only 1 external storage then simply run command to rescan it:

php /var/www/nextcloud/occ files:scan --path="/USERNAME/files/FTP_Mount_Point"

you can find more info about and how to in this comment:

@gas85

Try to run (under NC user):

OR

2. You can simply try to rescan whole NC (with external storage also) by command:

php /var/www/nextcloud/occ files:scan --all

This is inefficient way to rescan external shares, but it simply works. More info you can find in official docu: Using the occ command — Nextcloud latest Administration Manual latest documentation

[hep]

Hi,

Thanks for that script.
I get this error:

.: nextcloud-scripts-config.conf: not found

If I remove this line, script seems to not working:

45: [: Illegal number:
stat: opérande manquant
Saisissez « stat --help » pour plus d’informations.
WARNING - could not write to Log file , will drop log messages. Is User Correct? Current log file owener is
stat: impossible d’évaluer ‘/var/log/next-cron.log’: Aucun fichier ou dossier de ce type
WARNING - could not write to Log file /var/log/next-cron.log, will drop log messages. Is User Correct? Current log file owener is
root@bigfoot2:/jukebox/Pictures/Hep_Familly/Vacance# joe /etc/cron.weekly/NextCloud_Hep

[gas85]

Hi, you can create this file from it nextcloud_scripts/etc/nextcloud-scripts-config.conf at master · GAS85/nextcloud_scripts · GitHub

@hep

If I remove this line

Then you have to configure variables in script:

# Adjust to your NC installation

# Your NC OCC Command path

COMMAND=/var/www/nextcloud/occ
# Your NC log file path

LOGFILE=/var/www/nextcloud/data/nextcloud.log

# Your log file path for other output if needed

CRONLOGFILE=/var/log/next-cron.log

# If you want to perform cache cleanup, please change CACHE value to 1

CACHE=0

# Your PHP location

PHP=/usr/bin/php

[johnwayne]

thank you @gas85 , I was using external storage as primary storage, and the quota usage would never update. you saved my whole operation.