# Automate OCC Filescan **URL:** \ **Category:** ℹ️ Support\ **Tags:** external\_storage\ **Created:** [August 18, 2018, 8:47pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282 "2018-08-18T20:47:15Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![K-MTG](https://help.nextcloud.com/letter_avatar/k-mtg/32/5_5575768a8748004e209b776fc1b2916d.png) [@K-MTG](https://help.nextcloud.com/u/K-MTG)\ **Post date:** [August 18, 2018, 8:47pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/1 "2018-08-18T20:47:16Z")
**How can I use OCC (or something else?) to trigger a rescan of external storage whenever a change is made?** I have Nextcloud running on Docker on my Mac with this [docker-compose](https://github.com/nextcloud/docker/tree/master/.examples/docker-compose/with-nginx-proxy/mariadb-cron-redis/apache). I made a small tweak to it so I can access my external storage (external hard drive): ``` app: build: ./app restart: always volumes: - nextcloud:/var/www/html **- /Volumes/LaCie:/Volume** environment: - VIRTUAL_HOST=email@emaple.com - LETSENCRYPT_HOST=email@emaple.com - LETSENCRYPT_EMAIL=email@emaple.com - MYSQL_HOST=db env_file: - db.env depends_on: - db - redis networks: - proxy-tier - default ``` I was able to successfully enable local storage: ![16%20PM](https://help.nextcloud.com/uploads/default/original/2X/5/5c98a294b864cb30622ed2aefcd3468de4582c27.png) Immediately after setting up local storage, I am able to access all of my files and the data size states pending. However, after sometime only the files I accessed remain. Hence, if I have a TV show folder and click on the show SpongeBob and select season 1. All episodes from season 1 will load and the data size will update. However, if I don’t select and open the other seasons (i.e season 2,3,4) those folders will disappear from nextcloud. Also, if I create a new folder with the same name name (i.e Season 2). Nextcloud will state Season 2 already exist and the folder will show up. **Running the command `php occ files:scan --all` under the user www-data appears to have done the job…all of my external storage files are showing! However, is there a better way to implement this code somewhere? I think running this script every time will use a lot of resources…is there an alternative solution?** ![24%20PM](https://help.nextcloud.com/uploads/default/original/2X/1/12a1766fb8918f7672876716dbb30bc906323653.jpg) ---
**Author:** ![gas85](https://help.nextcloud.com/user_avatar/help.nextcloud.com/gas85/32/7772_2.png) [@gas85](https://help.nextcloud.com/u/gas85)\ **Post date:** [August 20, 2018, 9:35am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/2 "2018-08-20T09:35:02Z")
By the way _files:scan --all_ will rescan **whole** file system together with externals… [https://docs.nextcloud.com/server/13/admin\_manual/configuration\_server/occ\_command.html#file-operations](https://docs.nextcloud.com/server/13/admin_manual/configuration_server/occ_command.html#file-operations) I used this [script](https://github.com/GAS85/nextcloud_scripts/blob/master/nextcloud-file-sync.sh) to rescan external shares only. Script will fetch data about all external storages of all users and perform scan of them. I put it in cron like twice per day and it consumes much less resources in compare to “rescan all” command ---
**Author:** ![K-MTG](https://help.nextcloud.com/letter_avatar/k-mtg/32/5_5575768a8748004e209b776fc1b2916d.png) [@K-MTG](https://help.nextcloud.com/u/K-MTG)\ **Post date:** [August 20, 2018, 7:32pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/3 "2018-08-20T19:32:03Z")
Thanks for the info! Can you please provide the path where you stored this script and how you were able to call it with cron? Also, I noticed some features on the script are commented. Can you please share what you un-commented to automate scanning on external drives. Thanks ---
**Author:** ![gas85](https://help.nextcloud.com/user_avatar/help.nextcloud.com/gas85/32/7772_2.png) [@gas85](https://help.nextcloud.com/u/gas85)\ **Post date:** [August 21, 2018, 8:34am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/4 "2018-08-21T08:34:21Z")
Basically it works out from the box. Only that you have to check you nextcloud path, log path and create a log file for `php occ` output. I put it in ``` /usr/local/bin/ ``` with `chmod 755` I run it under _nextcloud user_ (for me it is www-data) basically twice per day at 2:30 and 14:30. You can run it also hourly. This is my cron config: ``` 30 2,14 * * * perl -e 'sleep int(rand(1800))' && /usr/local/bin/nextcloud-file-sync.sh #Nextcloud file sync ``` Here I add _perl -e ‘sleep int(rand(1800))’_ to inject some random start time within 30 Minutes, but since it scans externals only it is not necessary any more. Your cron job config to run it hourly could be: ``` * */1 * * * /usr/local/bin/nextcloud-file-sync.sh ``` Lets go through what it does (valid for [commit 44d9d2f](https://github.com/GAS85/nextcloud_scripts/commit/44d9d2ffe1153130560c8039e1299483bc2a36a5)): > COMMAND=/var/www/nextcloud/occ **← This is where your nextcloud OCC command located** > OPTIONS=“files:scan” **← This is “Command” to run, _just live it as it is_** > LOCKFILE=/tmp/nextcloud\_file\_scan **← Lock file to not execute script twice, if already ongoing** > LOGFILE=/var/www/nextcloud/data/nextcloud.log **← Location of Nextcloud LOG file, will put some logs in Nextcloud format** > CRONLOGFILE=/var/log/next-cron.log **← location for bash log. In case when there is an output by command generated. AND IT IS GENERATED…** Line 22 will generate NC log input. You will see it in a GUI as: ![image](https://help.nextcloud.com/uploads/default/original/2X/e/ebd7635c409b67d3ee0144246e4ca93f2363540a.png) From the line 26 starts the job, basically it is left from an older version of script and it is exactly what you done - scan all users, all shares, all locals with all folders. It takes ages to perform an a big installations, so I commented it. Second option (line 31) is to scan for specific user, but as soon as I get more than one user with external shares it does not work also. Besides it is still scanning whole partition (local and remote) for specific user - commented. From line 35 till 42 comments how to not forget how I get users from the NC, basically everything is happens in line 45, scipt will generate exactly path for external shares to be updated for all users (you can run it and test output). Here an example command: ``` sudo -u www-data php occ files_external:list | awk -F'|' '{print $8"/files"$3}'| tail -n +4 | head -n -1 | awk '{gsub(/ /, "", $0); print}' ``` and output: ``` user1/files/Dropbox-user1 user2/files/Dropbox-user2 user1/files/MagentaCloud user2/files/MagentaCloud user1/files/Local-Folder ``` Those lines will be read one by one and synced in line 49. After this script will generate NC log output: ![image](https://help.nextcloud.com/uploads/default/original/2X/b/bfc2a6ad6de3d7af5d287776e87ffbcd5d6fcc18.png) I have had some issues (like described here [OCC files:cleanup - Does it delete the db table entries of the missing files?](https://help.nextcloud.com/t/occ-files-cleanup-does-it-delete-the-db-table-entries-of-the-missing-files/20253)) in older NC versions, so I added workaround from line 60 till 67 as `files:cleanup` command, nut sure if it is needed now, but it does not harm anything. ---
**Author:** ![K-MTG](https://help.nextcloud.com/letter_avatar/k-mtg/32/5_5575768a8748004e209b776fc1b2916d.png) [@K-MTG](https://help.nextcloud.com/u/K-MTG)\ **Post date:** [August 22, 2018, 1:33am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/5 "2018-08-22T01:33:47Z")
I have a few docker containers for my nextcloud install. Which one should I use for this script? ![04%20PM](https://help.nextcloud.com/uploads/default/original/2X/6/690ebb1b816a4fb5fad76cac45b890f136fb9e10.png) Here is what my current cron.php looks like: ![30%20PM](https://help.nextcloud.com/uploads/default/original/2X/4/44d8a9a7b5a731bafb2fcb593dbfad917b0f1315.png) Where do I put this script: ```auto 30 2,14 * * * perl -e 'sleep int(rand(1800))' && /usr/local/bin/nextcloud-file-sync.sh #Nextcloud file sync ``` Just to confirm, my nextcloud-file-sync.sh file will be in the same directory? And just to confirm…I don’t need to modify the nextcloud-file-sync.sh file? So I have all of my external shared mounted under the admin user (and certain shares are shared with certain users). My users don’t have the ability to mount their own external storage. Thanks ---
**Author:** ![gas85](https://help.nextcloud.com/user_avatar/help.nextcloud.com/gas85/32/7772_2.png) [@gas85](https://help.nextcloud.com/u/gas85)\ **Post date:** [August 22, 2018, 7:19am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/6 "2018-08-22T07:19:11Z")
I do not know your installation. If you have few dockers as replicant of your cloud with DB inside, not sure if this script could help you at all. If you have few dockers that connected to one independent db - theoretically you can run it on one of them and it will do the job for others. But container could be closed and job will be not finished. If you have few dockers with own db each - run it everywhere where external shares are presented. With Docker it is a bit more complicated to run a cron job ( **not related to _cron.php_ from nextcloud, but to system cron** ). Try to read this [https://stackoverflow.com/questions/37458287/how-to-run-a-cron-job-inside-a-docker-container](https://stackoverflow.com/questions/37458287/how-to-run-a-cron-job-inside-a-docker-container) and find your best way to do it. As I wrote you also do not needed to run it with random time shift especially under docker, use simple line. _Again, this is not cron.php, this is system cron_. Steps to do: 1. Put script in whatever place within you NC Docker container. 2. Adjust header of script if your NC installation folder is different from what is in the header. Like `COMMAND=/Your/NC/Path` and for `LOGFILE` also. 3. Read how to run cron job under Docker 4. Create cron job (not cron.php) for you NC user with simple line like: ```auto * */1 * * * /path/to/the/nextcloud-file-sync.sh ``` 1. Test it ---
**Author:** ![gas85](https://help.nextcloud.com/user_avatar/help.nextcloud.com/gas85/32/7772_2.png) [@gas85](https://help.nextcloud.com/u/gas85)\ **Post date:** [August 22, 2018, 11:33am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/7 "2018-08-22T11:33:59Z")
I did update the script, it does not necessary now set LOG files path, it will drop all the output if logs are note presented or not writeable. Also I add option _-all_ to make it rescan whole partition in an old way. If you run it wihtout any input, it will do rescan of external shares only. You only need to check header and untouch the rest. ``` # Adjust to your NC installation # Your NC OCC Command path COMMAND=/var/www/nextcloud/occ # Your NC log file path LOGFILE=/var/www/nextcloud/data/nextcloud.log # Your log file path for other output if needed CRONLOGFILE=/var/log/next-cron.log # If you want to perform cache cleanup, please change CACHE value to 1 CACHE=0 ``` ---
**Author:** ![K-MTG](https://help.nextcloud.com/letter_avatar/k-mtg/32/5_5575768a8748004e209b776fc1b2916d.png) [@K-MTG](https://help.nextcloud.com/u/K-MTG)\ **Post date:** [August 26, 2018, 7:00pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/8 "2018-08-26T19:00:24Z")
I basically copied my docker-setup from here for nextcloud: [https://github.com/nextcloud/docker/tree/master/.examples/docker-compose/with-nginx-proxy/mariadb-cron-redis/apache](https://github.com/nextcloud/docker/tree/master/.examples/docker-compose/with-nginx-proxy/mariadb-cron-redis/apache) So the configuration already had a docker file inside the app folder…so I simply added the command to it (hopefully correctly): ![13%20AM](https://help.nextcloud.com/uploads/default/original/2X/9/969a120f8dfd2e7a24a52f5356e078b4ededc40b.png) And I copied the code of the nextcloud-file-sync.sh file and placed it in /usr/local/bin with vim (I created the file and copied/pasted the code.): ![37%20AM](https://help.nextcloud.com/uploads/default/original/2X/1/1ecafe0924e182c8205b4088aeba26dd3e23950e.jpeg) The only thing I changed in the nextcloud-file-sync.sh file is the file path: # Adjust to your NC installation # Your NC OCC Command path COMMAND=/var/src/nextcloud/occ # Your NC log file path LOGFILE=/var/src/nextcloud/data/nextcloud.log # Your log file path for other output if needed CRONLOGFILE=/var/log/next-cron.log # If you want to perform cache cleanup, please change CACHE value to 1 CACHE=0 I ran docker-compose up -d and waited an hour…but I don’t think the filescan occurred. I believe I am creating the cron job incorrectly in the dockerfile. Any tips? Thanks ---
**Author:** ![K-MTG](https://help.nextcloud.com/letter_avatar/k-mtg/32/5_5575768a8748004e209b776fc1b2916d.png) [@K-MTG](https://help.nextcloud.com/u/K-MTG)\ **Post date:** [August 27, 2018, 7:32pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/9 "2018-08-27T19:32:58Z")
@gas85 do you know if it’s possible to run the cron script directly on bash so I can confirm if it’s working? Currently I am not sure if the script isn’t working or if my call to run the script isn’t. Thanks! ---
**Author:** ![gas85](https://help.nextcloud.com/user_avatar/help.nextcloud.com/gas85/32/7772_2.png) [@gas85](https://help.nextcloud.com/u/gas85)\ **Post date:** [August 28, 2018, 2:16pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/10 "2018-08-28T14:16:32Z")
Of course run it under your NC user (e.g. www-data). Not sure how it is under docker, but on normal (e.g. Ubuntu) system it is something like: ``` sudo -u www-data nextcloud-file-sync.sh ``` ---
**Author:** ![K-MTG](https://help.nextcloud.com/letter_avatar/k-mtg/32/5_5575768a8748004e209b776fc1b2916d.png) [@K-MTG](https://help.nextcloud.com/u/K-MTG)\ **Post date:** [August 28, 2018, 5:42pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/11 "2018-08-28T17:42:42Z")
I made the .sh file executable: `chmod +x nextcloud-file-sync.sh` And had to adjust the file paths in the sh file: ```auto # Adjust to your NC installation # Your NC OCC Command path COMMAND=/var/www/html/occ # Your NC log file path LOGFILE=/var/www/html/data/nextcloud.log # Your log file path for other output if needed CRONLOGFILE=/var/www/html/data/next-cron.log # If you want to perform cache cleanup, please change CACHE value to 1 CACHE=0 ``` When I just ran `nextcloud-file-sync.sh` in terminal. I got: ![55%20AM](https://help.nextcloud.com/uploads/default/original/2X/5/5b90e9748b1b723d6cadc9ae7b961a97d9b27cda.png) And when I ran `sudo -u www-data nextcloud-file-sync.sh` (had to install sudo in my docker container). I got: ```auto WARNING - could not write to Log file /var/www/html/data/next-cron.log, will drop log messages. Is User Correct? root@c2663b1e77ea:/usr/local/bin# ``` ![49%20AM](https://help.nextcloud.com/uploads/default/original/2X/d/df52f5b542e0d88c92a2fa6f969afee1bd7008d5.png) In both scenarios, my file path didn’t update. So I am assuming I need to edit something in the sh file so it scans my paths? Thanks ---
**Author:** ![K-MTG](https://help.nextcloud.com/letter_avatar/k-mtg/32/5_5575768a8748004e209b776fc1b2916d.png) [@K-MTG](https://help.nextcloud.com/u/K-MTG)\ **Post date:** [August 30, 2018, 6:51pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/12 "2018-08-30T18:51:59Z")
@gas85 any idea why the script doesn’t update after the scan? Do I have to make any modifications (please see above post) Thanks ---
**Author:** ![gas85](https://help.nextcloud.com/user_avatar/help.nextcloud.com/gas85/32/7772_2.png) [@gas85](https://help.nextcloud.com/u/gas85)\ **Post date:** [August 31, 2018, 10:00am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/13 "2018-08-31T10:00:45Z")
Hey. It looks like file /var/www/html/data/next-cron.log does not exist, or not writable. Thats all, not a big issue. You will not see output of occ, but it will not stop because of it. Regarding sudo, not sure if you need to install it. I thought that container user is your NC user, so you should be able to run your script directly under command line in container. Sorry, I did not work with containers directly, so I couldn’t help you how to run script there. 30 августа 2018 г. 21:02:02 GMT+02:00, K-MTG [noreply@nextcloud.com](mailto:noreply@nextcloud.com) пишет: ---
**Author:** ![K-MTG](https://help.nextcloud.com/letter_avatar/k-mtg/32/5_5575768a8748004e209b776fc1b2916d.png) [@K-MTG](https://help.nextcloud.com/u/K-MTG)\ **Post date:** [August 31, 2018, 6:58pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/14 "2018-08-31T18:58:28Z")
I think I have the script running. But it seems to be scanning the wrong thing…because it says scan completed but nothing updates. ---
**Author:** ![gas85](https://help.nextcloud.com/user_avatar/help.nextcloud.com/gas85/32/7772_2.png) [@gas85](https://help.nextcloud.com/u/gas85)\ **Post date:** [September 13, 2018, 11:55am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/15 "2018-09-13T11:55:27Z")
Try to run (under NC user): ``` php occ files_external:list ``` you should see something similar to this: ![image](https://help.nextcloud.com/uploads/default/original/2X/8/84273075445b1819d5e68a78b1e9580362a86f61.png) This will output list of _all_ external shares. Check that it is not empty. Then try to run and check the output (it should be list of `USERNAME/files/SHARENAME`): ``` php occ files_external:list | awk -F'|' '{print $8"/files"$3}'| tail -n +4 | head -n -1 | awk '{gsub(/ /, "", $0); print}' ``` ![image](https://help.nextcloud.com/uploads/default/original/2X/9/95bd4489fcad38b8c1bd9b64d8452c47afce963d.png) Then take one path (from the 1st table, or last adopted list) and run command to check what happens: ``` php occ files:scan --path="/USERNAME/files/Mount_Point" ``` Every thing must bus executed under your NC user. For Ubuntu I used `sudo -u www-data` command as `www-data` is my NC user. Hope it helps you in trouble shooting. ---
**Author:** ![Dagon84](https://help.nextcloud.com/user_avatar/help.nextcloud.com/dagon84/32/15510_2.png) [@Dagon84](https://help.nextcloud.com/u/Dagon84)\ **Post date:** [January 8, 2020, 8:50am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/16 "2020-01-08T08:50:48Z")
Dear Friend, thank you for your script! I have the same issue with external FTP upload and NextCloud, but i have a FTP PLAN on a shared server. Is there any way to run this scxript on a Shared FTP ? I do have a cPanel where i manage my server space, but I’m not sure to have the possibility to follow the instruction that you have wrote (uoload iun user/local/bin/ ecc). Any help would be appreciated! ---
**Author:** ![gas85](https://help.nextcloud.com/user_avatar/help.nextcloud.com/gas85/32/7772_2.png) [@gas85](https://help.nextcloud.com/u/gas85)\ **Post date:** [January 13, 2020, 8:43am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/17 "2020-01-13T08:43:22Z")
If you means > [@gas85](#): > > this [script](https://github.com/GAS85/nextcloud_scripts/blob/master/nextcloud-file-sync.sh) to rescan external shares only then yes, it works for all external shares independent of type (ftp, smb, local, etc.). > [@Dagon84](#): > > I do have a cPanel where i manage my server space, but I’m not sure to have the possibility to follow the instruction that you have wrote (uoload iun user/local/bin/ ecc). You can upload this script to anywhere and added it to the cron job. The aim of this script is to reduce scanning time by focused scan of external shares only. Otherwise you can run (as cron job also) following commands: 1. if you have only 1 external storage then simply run command to rescan it: ```auto php /var/www/nextcloud/occ files:scan --path="/USERNAME/files/FTP_Mount_Point" ``` you can find more info about and how to in this comment: > [@gas85](#): > > Try to run (under NC user): **OR** 1. You can simply try to rescan whole NC (with external storage also) by command: ```auto php /var/www/nextcloud/occ files:scan --all ``` This is **inefficient** way to rescan external shares, but it simply works. More info you can find in official docu: [Using the occ command — Nextcloud latest Administration Manual latest documentation](https://docs.nextcloud.com/server/17/admin_manual/configuration_server/occ_command.html#file-operations) ---
**Author:** ![hep](https://help.nextcloud.com/user_avatar/help.nextcloud.com/hep/32/24096_2.png) [@hep](https://help.nextcloud.com/u/hep)\ **Post date:** [July 19, 2020, 5:36am UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/18 "2020-07-19T05:36:38Z")
Hi, Thanks for that script. I get this error: > .: nextcloud-scripts-config.conf: not found If I remove this line, script seems to not working: > 45: [: Illegal number: > stat: opérande manquant > Saisissez « stat --help » pour plus d’informations. > WARNING - could not write to Log file , will drop log messages. Is User Correct? Current log file owener is > stat: impossible d’évaluer ‘/var/log/next-cron.log’: Aucun fichier ou dossier de ce type > WARNING - could not write to Log file /var/log/next-cron.log, will drop log messages. Is User Correct? Current log file owener is > root@bigfoot2:/jukebox/Pictures/Hep\_Familly/Vacance# joe /etc/cron.weekly/NextCloud\_Hep ---
**Author:** ![gas85](https://help.nextcloud.com/user_avatar/help.nextcloud.com/gas85/32/7772_2.png) [@gas85](https://help.nextcloud.com/u/gas85)\ **Post date:** [July 22, 2020, 1:49pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/19 "2020-07-22T13:49:44Z")
Hi, you can create this file from it [nextcloud\_scripts/etc/nextcloud-scripts-config.conf at master · GAS85/nextcloud\_scripts · GitHub](https://github.com/GAS85/nextcloud_scripts/blob/master/etc/nextcloud-scripts-config.conf) > [@hep](#): > > If I remove this line Then you have to configure variables in script: ```auto # Adjust to your NC installation # Your NC OCC Command path COMMAND=/var/www/nextcloud/occ # Your NC log file path LOGFILE=/var/www/nextcloud/data/nextcloud.log # Your log file path for other output if needed CRONLOGFILE=/var/log/next-cron.log # If you want to perform cache cleanup, please change CACHE value to 1 CACHE=0 # Your PHP location PHP=/usr/bin/php ``` ---
**Author:** ![johnwayne](https://help.nextcloud.com/letter_avatar/johnwayne/32/5_5575768a8748004e209b776fc1b2916d.png) [@johnwayne](https://help.nextcloud.com/u/johnwayne)\ **Post date:** [November 6, 2020, 4:08pm UTC](https://help.nextcloud.com/t/automate-occ-filescan/35282/20 "2020-11-06T16:08:56Z")
thank you @gas85 , I was using external storage as primary storage, and the quota usage would never update. you saved my whole operation. [Next page](https://help.nextcloud.com/t/automate-occ-filescan/35282.md?page=2)