NC resets when server reboots

ℹ️ Support
1

Hello everyone,

5 days ago I switched from AJAX to CRON according to the manual, which means I updated the crontab file according to https://docs.nextcloud.com/server/11/admin_manual/configuration_server/background_jobs_configuration.html and selected cron on the settings page in the nextcloud web interface.

From then on, whenever I reboot my server (Raspberry Pi, Apache2), all files and settings that I changed in the meantime are reset. This means that I have been able to upload files to the cloud the normal way. However, after I restarted my Pi earlier today, all files that I uploaded in the last 5 days were deleted automatically and all files that have been modified were taken back to the versions that were up-to-date 5 days ago.

Another example: Yesterday I updated the ‘Theming’ app on my nextcloud web interface from 1.4.4 to 1.4.5. After my server rebooted today, I am back to version 1.4.4 again and the updater reminds me that 1.4.5 is available.

This is also true for all the settings I changed in the meantime. For example I deactivated 2-factor authentification and activated it again yesterday, because I switched my phone with the 2nd factor authentification app. I was able to login yesterday after I registered my new phone and revoked access permissions for my old phone on Nextcloud-Settings → Securty. After I restarted my Pi today, these changes have been reset and I am unable to log into my account, because Nextcloud won’t accept my the key-generator on my new app, but only the old one, which I do not have anymore. I was still able to log in via backup-keys though.
Because I already used 7/10 of these codes, I decided to generate a new set of backup codes. I just had to restart the Pi again for another application, and it seems I cannot use the new set of backup codes. When I try to enter one of these it says “Error while validating your second factor”. So now I am completely locked out from the interface.

Something is seriously broken with my setup. Can anybody assist me and tell me
1.) How can I get access back to the web interface,aka deactivate 2nd-factor-authentification from the server?
2.) How can I fix this issue that the server always reverts everything back to the state that was current 5 days ago?

Have a great day.

Attachment: crontab -l

pi@raspberrypi:~ $ sudo crontab -u www-data -l
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use ‘*’ in these fields (for ‘any’).#
# Notice that tasks will be started based on the cron’s system
# daemon’s notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command

*/15 * * * * php -f /var/www/nextcloud/cron.php

2

Hi,

Run the following command to disable the two-factor-authentication for the specified user:
sudo -u www-data php /var/www/nextcloud/occ twofactorauth:disable <username>

Regarding your issue that everything is reverted back:
It seems illogical to me, that this should be related to your switch to cron. I’m not aware that cron modifies files and app updates are performed on file base, meaning that after the app update there are modified files in /var/www/nextcloud/apps and reading this folder will Nextcloud tell which files are available and which version they have. The only thing in the database are the settings of that app and the information if the app is enabled. And when anything is written to the database that means a file modification as well and cron is not able to revert back this file modification afaik.
Is there really not anything else that you changed to your server?
It rather sounds like there is some kind of snapshot which is always activated at boot time.

OR: your database and data directory are stored in volatile memory (like RAM).

Not sure if it might also be possible, that there is a caching issue? Something is “caching” the write to disk requests and the actual writing does never happen?

3

Hi schmu

and thank you very much for your reply.
I deactivated 2FA with success, which is very nice!

Also, I investigated further and found out that not only my Nextcloud is resetting to a previous state, but also files that I create on ~/Desktop on my pi are gone, once I restart the Pi. Presumably, the issue is not the cloud here, which is a good thing! I am still wondering, what could cause this issue. All I can remember that I did was switching to cron and trying to switch to utf8 according to this page: https://docs.nextcloud.com/server/11/admin_manual/maintenance/mysql_4byte_support.html
But I reverted all changes I did to the file. Also I cannot imagine that this is the cause for the reset-problems I am experiencing with my pi.

Guess I’ll have to google for a solution, as it seems unlikely now, that this is related to nextcloud. However, if anyone ever experienced such behaviour and has a clue about what I could do to prevent the resetting on bootup, please write me a message!

Have a great day.

4

Hi @flashvoyager33

What kind of storage device are you using for your raspi (external hard disk, USB stick, SD card)?
I’m not very familiar with Raspi and so I’m not sure if it’s possible that (in case you use a SD card) this SD card was (accidentally) set write protected (by the little switch on the side) and that is why there is no real write operations going on.
In case of an external hard disk drive, do you see read/ write operations happening indicated by the blinking activity LED?

5

Hi @Schmu,

thanks again for getting back at me.

I do not need a lot of space so I am only having the Raspbian microSD card installed. Additionally I have an USB stick mounted for backups. I checked the card: there is no switch for write protection.

6

It might be possible that if the SDcard is bad or the filesystem has problems, that it’s being mounted in read only at boot, so any changes made are not actually written to the card. Can you read through this and see if any of it might apply to you?

https://www.raspberrypi.org/forums/viewtopic.php?p=988640

1 Like
7

Hi @linucksrox

your idea seems correct. I checked the link you provided and indeed, the person in the other topic has very similar issues.
I went ahead and plugged in the SD Card from my Pi into my Laptop and tried to run fsck on the ext4 partition.

# sudo fsck -fvn /dev/sdb7
fsck from util-linux 2.27.1
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 523682 has zero dtime. Fix? no

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (1603838, counted=781309).
Fix? no

Inode bitmap differences: -523630 -523682 -524992 -525029 -525074 -525099
Fix? no

Free inodes count wrong (706347, counted=679043).
Fix? no

root0: ********** WARNING: Filesystem still has errors **********

199413 inodes used (22.02%, out of 905760)
   572 non-contiguous files (0.3%)
  139 non-contiguous directories (0.1%)
     # of inodes with ind/dind/tind blocks: 0/0/0
    Extent depth histogram: 210079/58

2012674 blocks used (55.65%, out of 3616512)
0 bad blocks
1 large file

186435 regular files
23402 directories
  55 character device files
 25 block device files
  0 fifos
795 links

16762 symbolic links (16463 fast symbolic links)
23 sockets

227497 files

But fsck cannot repair the file system, because it is write-protected:

# sudo fsck -fvy /dev/sdb7
fsck from util-linux 2.27.1
Disk write-protected; use the -n option to do a read-only
check of the device.

It seems to me that this is not a Nextcloud issue, so I created another thread with all information on raspberrypi.stackexchange.com (just in case someone is trying to help me here and finds my other topic on the other forums).
If anyone has an idea on how to force fsck do check the file system (although it is write-protected), feel free to post below or at the other forum.

Have a great day and thanks for your efforts so far!

1 Like
8

It could be simply a broken sd card. Especially small ones break fast with regular write, database and log access.

Try to do a full image backup if still possible and write it back to a new one. You could also try to format and rewrite the old one, but to guarantee it works.

How much capacity has it and how long you already use it?

9

Went to a local electronics store today and bought a new SD card. Used dd to flash the image I have taken of my Raspi SD Card onto my new SD card. Inserted the new SD card into the Pi and it works like nothing happened. Indeed, I assume this was all about a broken SD Card. Lucky for me!
Maybe it is time to think about a smarter solution to limit the number of read/write processes onto the SD card to improve its life duration. Maybe I will start looking for an external HDD to put the Nextcloud /data folder on it and ease up the load that the SD card had to endure otherwise.

Thanks for your help everybody!

@MichaIng
The SD card has 16GB of capacity and has now been in use for about 18 months, 24/7.

3 Likes
10

Good to it then worked.

Btw. in case of SDcard failure in general, you could use a tool like Rufus to check the SDcard for bad blocks (broken storage parts) or worse errors. If a clean format does not help as well, check warranty of the card. Most manufacturers give way longer warranty than the card can possibly survive at 24/7 usage and many of the lower prizes lower capacity ones break earlier even on usual camera etc. usage. No idea why the warranty is often so high, maybe it’s trusted that most customers never make use of warranty. I have lifetime warranty here on my 64G SDcard :thinking:. So depending on the shop where it was bough, you can get a new one even without shipping costs or for “just” shipping costs to manufacturer. Might be worth or not, just as a hint :wink:.

Jep, definitely it makes sense to actively reduce SDcard write operations. Nextcloud data storage, but possibly also logging location (/var/log), swapfile (often /var/swap), database folder (/var/lib/mysql). But depending on drive speed, at least if it’s about access time an external HDD (if not SSD) can have negative performance impact. But clearly if it’s about your personal data, reliability/stability should have priority over performance. That’s the trade-off when using a low/mid range SBC.

11

Thanks again for your hints @MichaIng.

I tried to format the (presumably) broken SD card with GParted and even fdisk, however both programs are not able to format the SD card.

#sudo fdisk /dev/sdb

Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

fdisk: cannot open /dev/sdb: Read-only file system

Currently I have no other idea on how to format this thing. I assume your tool would behave the same way, maybe I will give it a try later though!

Good idea about changing the location of the logging file (and maybe swap too, if performance does not drop to the basement). I will definitely keep this in mind.

12

What’s the goal in formatting the bad SD card? Of course you can’t write to the card if it’s mounted read only. You can try to remount in read/write mode, and you might possibly have to specify what type of filesystem it uses (probably ext3/4).

Another way to do a “disk initialize” type of thing is using dd. I think you can do this regardless of the mount parameters, assuming there is not a hardware problem with the device. This wipes the MBR by writing all zeroes to the first 512 bytes of the disk.
sudo dd if=/dev/zero of=/dev/sdb bs=512 count=1

1 Like
13

The idea was just, that an external format tool might be able to recover write-ability. Could be tested e.g. on a windows system as well, perhaps it handles the corruption differently. Although it’s most likely a very bad idea to reuse an SD that had such an error once, at least a careful bad block test or something should be done. Not sure if it’s possible that the error can be caused by the system instead of the SD being broken.