Is there a safe and reliable way to move data directory out of web root?

MichaIng · July 19, 2017, 11:39am

Ah okay, this is also covered by the example configuration in admin manual:

location ~ ^/nextcloud/(?:build|tests|config|lib|3rdparty|templates|data)/ {
    deny all;
}
location ~ ^/nextcloud/(?:\.|autotest|occ|issue|indie|db_|console) {
    deny all;
}

But yeah, for every security step, there is a way to get around it. For my point of view, there is not enough you can do to protect your data .
Same for strong nextcloud password + internal brute force prevention + fail2ban as second brute force detection (okay I do not use that anymore actually )

Tom_Forge · July 29, 2017, 9:03pm

I am not skilled at linux or apache, etc. I just got NC 12.0.0 up and running following the manual install instructions and the default installation path when first starting nextcloud 192.168.1.xxx/nextcloud put the data in the webpath apparently.

I’d like to uninstall everything and start over.

I am guessing that I need to:

apt-get purge nextcloud
apt-get purge mysql-server

as to the non-web-root directory I was thinking of creating /home/share as the /home mount point has 3TB available and should do okay for my family sharing files

I don’t know what my chown command should look like?

And also is /home/share where both the nextcloud database and user created documents will be kept?

Thanks in advance, and any help much appreciated.

ReinisN · August 15, 2017, 7:47am

Thanks for support!
I have moved data directory to /opt/data and I can see data in the NextCoud App.
BUT I can’t upload any files, error is “Forbiden”.
In the Admin/Basic Settings I see error: Error occurred while checking server setup
In the Admin panel Logs never show up. I see file under /…/data directory file permissions are: -rw-r-----. 1 httpd httpd
Same as they where in the old location, where it worked fine…

When I upload files manually to home directory and run: sudo -u httpd php /var/www/nextcloud/occ files:scan --all
Files appear in my file list in web and application.
File permissions are the same as they where in original /…/data/ folder.

Strange thing. When I open new incognito window and try to open my nextcloud web page, I get page with error (Old session works):
“Your data directory is not writable
Permissions can usually be fixed by giving the webserver write access to the root directory.”

But in the console I can move files around with httpd user with no problem. For example:
sudo -u httpd cp /var/www/nextcloud/console.php /opt/data/

Where I might be forgetting to change something?
Thank You!Preformatted text

JasonBayton · August 15, 2017, 8:13am

Just to move your data you don’t need to reinstall. But how did you install it? Via apt-get or with the zip?

Don’t purge mySQL or Apache, there’s no need. You can instead make a new database as part of the reinstall.

Don’t put your data in /home/ as there can be issues with ownership, rather use /opt/ or /media/ or /var/ or some other area.

ReinisN · August 15, 2017, 1:47pm

I have tried to:
sudo chmod -R 777 /opt
all /opt is owned by httpd:httpd still no success…

Log says:

Aug 15 16:34:05 nextcloud httpd[3000]: [:error] [pid 3000] [client 192.168.20.125:42994] {"reqId":"WZL4TXCfAMRB0DF7mRLwegAAAAM","level":3,"time":"2017-08-15T13:34:05+00:00","remoteAddr":"192.168.20.125","user":"--","app":"PHP","method":"GET","url":"\\/index.php\\/js\\/core\\/merged-template-prepend.js?v=df00724434e5f06d1daca52134e36ea3-2","message":"Invalid argument supplied for foreach() at \\/var\\/www\\/html\\/nextcloud\\/lib\\/private\\/Template\\/JSCombiner.php#107","userAgent":"Mozilla\\/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko\\/20100101 Firefox\\/54.0","version":"12.0.0.29"}
Aug 15 16:34:05 nextcloud httpd[3000]: [:error] [pid 3000] [client 192.168.20.125:42994] PHP Warning:  fileperms(): stat failed for /opt/data/nextcloud.log in /var/www/html/nextcloud/lib/private/Log/File.php on line 122
Aug 15 16:34:05 nextcloud httpd[3000]: [:error] [pid 3000] [client 192.168.20.125:42994] {"reqId":"WZL4TXCfAMRB0DF7mRLwegAAAAM","level":3,"time":"2017-08-15T13:34:05+00:00","remoteAddr":"192.168.20.125","user":"--","app":"PHP","method":"GET","url":"\\/index.php\\/js\\/core\\/merged-template-prepend.js?v=df00724434e5f06d1daca52134e36ea3-2","message":"Invalid argument supplied for foreach() at \\/var\\/www\\/html\\/nextcloud\\/lib\\/private\\/Template\\/SCSSCacher.php#145","userAgent":"Mozilla\\/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko\\/20100101 Firefox\\/54.0","version":"12.0.0.29"}

Or with previously logged user - reinis:

Aug 15 16:36:50 nextcloud httpd[3031]: [:error] [pid 3031] [client 192.168.20.125:43117] PHP Warning:  fileperms(): stat failed for /opt/data/nextcloud.log in /var/www/html/nextcloud/lib/private/Log/File.php on line 122
Aug 15 16:36:50 nextcloud httpd[3031]: [:error] [pid 3031] [client 192.168.20.125:43117] {"reqId":"WZL48jmwqeJEN51EO-50fAAAAAo","level":3,"time":"2017-08-15T13:36:50+00:00","remoteAddr":"192.168.20.125","user":"reinis","app":"PHP","method":"GET","url":"\\/index.php\\/apps\\/theming\\/img\\/core\\/filetypes\\/application.svg?v=2","message":"fileperms(): stat failed for \\/opt\\/data\\/nextcloud.log at \\/var\\/www\\/html\\/nextcloud\\/lib\\/private\\/Log\\/File.php#122","userAgent":"Mozilla\\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\\/537.36 (KHTML, like Gecko) Chrome\\/60.0.3112.90 Safari\\/537.36","version":"12.0.0.29"}`
Aug 15 16:36:50 nextcloud httpd[3031]: [:error] [pid 3031] [client 192.168.20.125:43117] {"reqId":"WZL48jmwqeJEN51EO-50fAAAAAo","level":3,"time":"2017-08-15T13:36:50+00:00","remoteAddr":"192.168.20.125","user":"reinis","app":"PHP","method":"GET","url":"\\/index.php\\/apps\\/theming\\/img\\/core\\/filetypes\\/application.svg?v=2","message":"chmod(): Permission denied at \\/var\\/www\\/html\\/nextcloud\\/lib\\/private\\/Log\\/File.php#123","userAgent":"Mozilla\\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\\/537.36 (KHTML, like Gecko) Chrome\\/60.0.3112.90 Safari\\/537.36","version":"12.0.0.29"}`

Any ideas?

JasonBayton · August 15, 2017, 4:10pm

Can you also please create an issue on GitHub: github.com/nextcloud/server/issues

ReinisN · August 16, 2017, 7:29am

Created issue: Github #6145

Created simple PHP file in
/var/www/nextcloud/pi.php
Containig:
require('/opt/data/test.php');

It also doesn’t work

require(/opt/data/pi.php): failed to open stream: Permission denied in /var/www/html/nextcloud/pi.php

So problem is not in the NextCloud itself, but in my config somewhere somehow…
Keep on digging

ReinisN · August 16, 2017, 9:52am

Thnx comunity for support

Problem was in SELinux…

Tom_Forge · August 16, 2017, 4:30pm

Thank you for your response:

Actually I have no data to move as I have not done anything with Nextcloud itself other than install it. I have not placed any files anywhere other than the install files. I installed with apt-get per Installation on Linux — Nextcloud 12 Server Administration Manual 12 documentation since I use Linux Mint.

How about /nextcloud/data off the root then? And why would /home/share cause problems?

I’m really confused and didn’t find/understand the help on what is going on in Linux by referencing the admin manual. I am guessing that MySQL is going to put the database for Nextcloud where I designate the datapath during the initial setup of Nextcloud. But I am not sure about that. Also, I do not understand what happens if say I load a file into Nextcloud to share as far as the Linux file system goes. Does Nextcloud copy the file and share the copy? If so, where does it copy the file to – the Nextcloud datapath? Thanks.

JasonBayton · August 16, 2017, 5:00pm

As it happens I haven’t tried the apt install method but I think @jospoortvliet knows who maintains it and can answer those specific questions around how it installs and to where - much like Windows I guess the dev can decide if everything is in Program Files or %AppData%, etc.

So all the web interface is in practice for file management (and I’m simplifying this horribly, no lynching please) is a front-end to the hidden data folder on the linux system. Therefore when you upload a file you’re uploading it directly to the folder on linux. Nextcloud also registers the file in the database and can associate shares and other bits and pieces to it that way.

The database normally will be setup in mysql which in itself has a directory (/var/lib/mysql/… in ubuntu), but you don’t normally need to mess with directories for the database, instead something like phpmysql will offer a graphical interface for database management if you wish to install it.

Tom_Forge · August 16, 2017, 5:42pm

That helps straighten things out for me, plus I had a chance this morning to checkout the howto Tutorial: How to migrate mass data to a new NextCloud server so between your response and that article I think I have it squared away in my head and I am pleased to see the files are being stored on the server and maintained in a directory structure that the authorized linux users and especially the administrators can get to the raw user files. For me this was important in case I botch up my Nextcloud install. Thanks a ton Jason!

Tom_Forge · August 16, 2017, 9:03pm

My bad on recollecting how the install on linux documentation goes. As it turns out the installation instructions are apt-get for much of the dependencies but you install pretty much by unpacking and recursively copying the Nextcloud files.

JasonBayton · August 16, 2017, 9:12pm

I should have clicked the link, oops. That then is the standard install (though I believe an apt install exists, community-supported) and one I’m quite familiar with.

In which case your installation will likely reside in /var/www(/html)/nextcloud where /html is optional, if you’re happy working with your own Apache vhost files (assume that would be a no for you) you could omit /html

data is set by you, so can go basically anywhere. Linux has some rules around mounting things in home folders, which can lead to issues with permissions and other non-niceties; better to use a directory out of /home/user/.., and while root / could be used, it isn’t very often.

worric · August 21, 2017, 6:34pm

I wish I had read both the wonderful guide by @MichaIng found here as well as this thread before changing the data directory; it would have been an easy fix to manually update the database along with making the change to config.php.

Alas, here I am with a Nextcloud install that, in light of the information in the guide/thread, seems broken under the hood in some ways. I’ve been poking around in the database, and I do indeed see duplicate entries in the oc_filecache table pertaining - in a large part - to image previews, but I guess there could be other duplicate entries as well. I’ve also been experimenting a bit with occ files:scan --all in a Nextcloud install on a VM, but I don’t think it picks up all the preview files either. Which would probably also be strange, since the fileid of a previewable file corresponds to the name of the preview folder both in file system and database, and hence, with new fileid's generated by scanning, the old preview folders would be invalid.

For a manual cleanup I’ve been considering removing the outdated entries from the oc_filecache table, and also manually removing the generated preview folders by using find and filtering by mtime/ctime.

But is there a more correct and/or more comprehensive way to do it?

JasonBayton · August 21, 2017, 7:45pm

I’d suggest you create a new topic describing this as well as your environment and anything you’ve done so far not mentioned above.

MichaIng · August 21, 2017, 7:45pm

Yeah, previews have to be regenerated as you say: https://github.com/nextcloud/server/issues/5264
You can force this by using the preview generator app. But depending on the amount of data, this can take days.

The oc_filescache should be cleaned up automatically for obsolete entries. The previews were just removed, if you remove the related file from inside nextcloud (instead of manually). So if you want to get rid of the obsolete preview files (which are by the way also stored in the oc_filescache and can make a major part there), you need to clear the whole appdata_<instance_id>/previews folder, which would of course also remove in between recreated previews .

Something might have changed there or will change soon, but until 12.0.1 I cannot see any different behaviour about obsolete previews:

worric · August 22, 2017, 8:42am

I guess I should have created a new topic JasonBayton mentions, but, yeah, I’ll keep that in mind for next time. I’ll just leave a reply here and let it be the last of it.

And thanks for your input. I do think I can find all the outdated preview folders by using the find command in Linux. I played around with it a bit on my VM sandbox install of Nextcloud and found that if I deleted some preview folders followed by a manually triggered cron.php, the database was actually cleaned up as expected. I guess that will save some time on my production system as opposed to just deleting all preview folders.

MichaIng · August 22, 2017, 9:40am

Ah yes, going for creation date e.g. is of course an easy solution to clean up previews folder .

Cron.php runs occ files:cleanup, which cleans oc_filecache.

JasonBayton · September 24, 2017, 8:24am

A post was split to a new topic: Blank page on login