Ok I am new here so I am sorry if I have not got the time to learn the forum rules and formatting correctly; this is an urgency to me.
Dropbox stopped supporting my form of drive encryption on Ubuntu 16.04 which made me switch to Nextcloud. I got the Nextcloud server up and running and set it running as a client on my Android phone and Ubuntu 16.04 laptops. I have only now noticed that it appears to be deleting masses of my files! All of my .tex files, for example, are just vanished! This is horrifying to me and very damaging! How can I stop this from happening? What has happened?? I request urgent assistance.
I am seeing whole directories deleted, .tex file deleted. This should not happen. I could understand hiding dotfiles, but this mass deletion of files critical to how I work is crazy. Nextcloud should not be, by default, deleting .tex. files, for example. Because the directory is so huge I am having difficulty quantifying the damage but am searching for a visual tool that is a bit like a mix of dmap and tkdirdiff.
I will read your link but there must be something weird happening and I request expert assistance. I have used boring, standard configurations in setup of clients and server.
If it helps, I am seeing errors like this in the Nextcloud client on Ubuntu 16.04:
could you describe what you did?
did you move all your files from a local dropbox folder to a nextcloud folder and got stuck now while syncing? and the files are now deleted from your local folder before syncing to the server?
I can describe the server setup procedure precisely, but I did nothing unusual. It is Nextcloud 14 on Ubuntu 18.04 with PHP 7.1 and MySQL. Using the Nextcloud administrator account, I created my user with unlimited storage and did not grant the user administrator privileges.
On my Ubuntu 16.04 laptop, I disabled the Dropbox client, launched the Nextcloud client, moved the ~ 52 GB contents of the ~/Dropbox directory to the new ~/Nextcloud directory and let Nextcloud do its syncing. I also installed the Android Nextcloud client but didn’t do anything unusual there. Next I find Nextcloud wiping out vast numbers of critical files, including .tex and some.sh! I will try to provide more information on the damage it has caused but I need to fix other things this has caused first. I would really appreciate some guidance or even guesses at what might be going wrong because this is breaking my entire computing infrastructure. I really want out and away from Dropbox but this is really damaging to me.
Can I suggest copying the contents of the dropbox store to a temporary directory so that you have a backup (assuming it is not too late) Then deal with the nextcloud sync. If the files actually are being deleted, log into the nextcloud web interface and click the deleted files icon on the bottom left of the screen. They might be in there. If so you can undelete them.
I wouldn’t panic about those warning messages. When you are transferring a lot of files, that happens. Not sure why, but I suspect the error occurs when the file is locked while being copied to the client and the server is trying to upload the not yet copied file. Let it finish. It will resync and fix any errors that occurred the next time round.
ie let it do its thing. It will get it right in the end.
so they are deleted in your local folder or they are not synced (yet)?
did you check the folder on the server directly? not through the webgui or nextcloud clients.
ssh to the server and do “ls”.
@Reiner_Nippes Thanks for your suggestion. Here is my view of events:
I set up the Nextcloud server on Ubuntu 18.04 as described above. I moved the data directory of Nextcloud to a directory in /home to accommodate the large size.
I reconfigure the Nextcloud server so that it knows its data directory by editing the file /var/www/nextcloud/config/config.php such that it now contains the following line:
'datadirectory' => '/home/nextcloud_data',
I then set the correct filesystem permissions such that Apache has full access to the Nextcloud files and directories.
chown -R www-data: /home/nextcloud_data
I restart Apache and go through the web interface as administrator, set up an non-administrator user etc.
systemctl start apache2
On my local machine, I switch off Dropbox, delete Dropbox dot files etc. Now I just have a directory ~/Dropbox. I launch the Nextcloud client on Ubuntu 16.04 and create the directory ~/.Nextcloud. I move the contents of the old Dropbox directory to the new Nextcloud directory and the Nextcloud client seems to start off doing its syncing thing. Excellent. I let it do its thing overnight.
Next morning I begin to notice files are missing. Like, very important files like my .tex files. I am finding it difficult even to quantify the damage caused and I cannot understand why Nextcloud should act in this way.
Now, as per your suggestion (and thank you again for your continued assistance) I have checked the directory /home/nextcloud_data/wbm/files. I can see files being synced there, but there are no .tex files there either where there should be. So I am distraught and confused.
Do you have any idea what might have happened? Is there any recovery from this? I would welcome any even tentative suggestions. This is making my days really bad.
@Jessie Thank you for your suggestion. Since I am dealing with a lot of computers, the suggestion is non-trivial to implement, but I will try to do something like what you are suggesting.
I am hoping that the damage has resulted in loss of only a week of work. But this is making my days really bad right now.
Thank you for your comment on the warning messages, that calms me a bit.
I would really like to know why so much has been deleted by Nextcloud though and how to prevent this. Would you have any suggestions on this?
I haven’t really noticed any major issues. Certainly no missing files. If anything very reliable. The only quirk I’ve noticed is the client won’t display a folder with a name beginning with a number. eg a folder named “123”. The actual folder does get synced and you can verify it is there by accessing the web interface, but it will not display in the client program. Therefore you can’t uncheck it.
I wondered whether that might be by design.
eg you could have a folder starting with a number which can contain files which can’t be turned off. Or it’s a bug…
I’ve set up a number of nextcloud installs. Mainly small systems using an unraid platform and linuxserver’s docker, MySQL or MariaDB docker and the letsencrypt docker which provides SSL certificates and a reverse proxy.
I’ve also set up nextcloud on Linux mint platforms using Snap. All painless… (Well painless after I ironed out the installation procedures)
Whilst I don’t use .tex files, I created a txt file and renamed it to abc.tex and it synced fine.
Another platform I have used is Syncthing. Works well but I’ve had a few unexplained occurrences eg a file sync going the wrong direction, so the old file overwrote the new. Not so with nextcloud in my experience.
@Jessie Hey again. Thank you for thinking about this and for your response. I have wiped everything and set up all again from scratch. The Nextcloud server is running on Ubuntu 18.04 and the Nextcloud clients are running on Ubuntu 16.04, 18.04 and LineageOS. I have been slowly adding files and I am still seeing Nextcloud nuking files. This time I am seeing some shell scripts not syncing. I have done the obvious things like waited a long time, restarted clients etc., and manually checked the server Nextcloud files storage directory and confirmed that some of the files are not getting picked up by the server.
So I’ve looked at the permissions of the shell scripts but the results don’t show anything obvious. A specific example is adding three shell scripts to Nextcloud running on an Ubuntu 16.04 laptop (ext4 via ecryptfs) and syncing to the Nextcloud server running on a Ubuntu 18.04 (the default install of 18.04 on a Kimsufi server). The following shell script syncs fine:
-rwxrwxr-x 1 wbm wbm 316735 Mar 20 2018 script_1.sh
The following shell scripts do not sync:
-rwxr-xr-x 1 wbm wbm 87 Nov 1 19:10 script_2.sh
-rwxrwxr-x 1 wbm wbm 98 Nov 1 19:07 script_3.sh
This is really a worst case experience for a new Nextcloud user. For a more detailed analysis, we’d need some more information, e.g. the versions of Nextcloud and the client you are using (the template that is shown when creating a new topic is a good starting point).
To generate logs of the deleting procedure:
You can start the client with logging more information into a file: https://docs.nextcloud.com/desktop/2.3/troubleshooting.html#log-files
On the server side, there are as well a few logs:
/path/to/nextcloud/data/nextcloud.log
/var/log/apache2/access.log
/var/log/apache2/error.log
A first test could be as well to take the client completely out of the scenario and do uploads via web-interface and webdav. You can as well test the client only with one of the test instances (don’t use any sensitive data):
How are you uploading the files? Via the client or direct to the web interface?
What happens if you create 3 x .txt files and rename them to .sh extensions?
Do they sync?
ie
script_1.txt to script_1.sh
script_2.txt to script_2.sh
script_3.txt to script_3.sh
In the client under general/advanced, there is an ignored file list.
I did not see any reference to .sh files in mine.
Then again, the examples above in my system synced ok.
(the created files had data in them. I also tested them as empty files and they still synced)
More on this. I wondered whether there might have been an issue with the Linux workstation client. All my clients are on windows platforms. I had a Linux Mint (Mate)18.3vm lying around. Did the experiment. It worked synchronising .sh files. I have downloaded ubuntu 18.04. Will set up a vm and see if that replicates the problem. Which version of nextcloud are you running 13.5 or one of the v14 varieties?
Thinking further outside the square, consider setting up a temp nextcloud server on an unraid platform. Could be set up in an hour. Virtually any old hardware could be used plus a usb flash drive. No need to set up arrays. Can be done on a single hard drive. Get the demo version of unraid, copy to flash drive. Boot the machine. Get linuxserver dockers for MariaDB or MySQL, Nextcloud. Create users and connect the clients. This might assist you in ascertaining whether the issue is caused by the server or the client. You can also load the letsencrypt docker for ssl certs and a reverse proxy. In the past, I have seen the nginx reverse proxy in the letsencrypt docker restrict large files. (was easily fixed) Your example files are only a few bytes, so that shouldn’t be a factor. Not sure if you are using a reverse proxy. If you are, that might be worth investigating.
If you need to get up to speed with unraid quickly, there are tutorials on youtube by a person called Spaceinvader one. He has a lot of them but specifically there is one on nextcloud setup in unraid. There are a couple of tweaks needed to make it work.
