This is what really bothers me about Nextcloud. Sneeze at it and it dies on you and all your data is lost! Hardly a safe place for all my data.
Case in point. I reinstalled NC a while ago and had been using it with sqlite. Leary of using it that much because this has happened to me before, I decided just to treat NC as a Dropbox replacement and leave it with sqlite and not screw with any of the other things like calendar, contacts, etc. For a while it seemed OK so I recently moved forward and went through the effort of converting sqlite → Mariadb (I posted here and managed to get that done).
Next figured out how to mount the webdav to my Linux desktop. Also posted here and worked it out. It’s slow but it seems to work. Then I tried moving over my .git directory for the folder I was working on - /opt/songbook - I was keeping my songs in git. After moving my .git folder there I was having all sorts of problems with git complaining about this and that so I decided to remove the .git folder and do a git init. I did that and all of my song files were showing up as not tracked. Tried a git add . and that never seemed to finish. So I tried adding one file. That worked. So I set up a loop adding each file one by one - there are about 360 of them or so. I know I would be losing past history but I figured I could at least have history going forward. Some files failed to add but I left that running.
Now I run NC and MariaDB in docker containers on my Synology. Over on Synology, the new Container manager informed me that some of my docker containers were out of date and needed to be updated. One of them was MariaDB. it seems after updating Mariadb my loop of git adds all crashed. I figured perhaps that happened because the MariaDB container was updated and thus recreated. So I started the Mariadb container. No dice. Could not access my NC. I stopped the Nextcloud container and the MariaDB container and then restarted MariaDB and then Nextcloud. Still no dice. Said there was some sort of error - uh oh!
Next I tried recreating the Nextcloud container based on an export of the container that Synology does to allow you to export and import such containers to recreate them. It was a JSON file. But the import said the format was invalid. Crap!
So I deleted the NC container and image and decided to download the image again and recreate it by hand. Pulled the image and went to create the container using the JSON file as a guide as to what volume mounts and env variables were supposed to be. Now when I go to NC it’s as if I have a new installation, it’s using sqlite and doesn’t know about any of my prior files or shares. IOW I need to recreate all of this again for about the 4th time
Trying to get back up here, I selected Storage & database, then MariaDB, and tried using my nextcloud user but it would not work. I tried using my root user for MariaDB. Didn’t work. Don’t know what happened, my Synology said this website was not available.
Refreshed the page and it’s asking me to install it again. Jumped into my docker mariadb user. Don’t see my nextcloud user but I do see my nextcloud database. Added nextcloud user and granted permissions.
Back to the Install dialog specifying the nextcloud credentials. Seemed to have work.
OK, login dialog. Enter my admin credentials and it’s Logging in… for the last 5 minutes or so… Never changed. Started another browser tab. Hey, look I’m logged in. But a very plain set of files. Where did all my files go?!?
I installed the nextcloud server from linuxserver and thought I’d go back to the default nextcloud image. After recreating this I now get:
Internal Server Error
The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.
Not exactly sure what you mean by “backup of previous install”. If by that you mean the Synology export of the docker container, yes I had a backup. That’s what I exported and when I tried to import it it said the file was not in the correct format.
My files are still around but not known to NC. But then again I can’t even log in to NC at this point.
Do not get me wrong. I first tried to just restart the already running and correctly configured docker container. That didn’t work. It was then that I tried to recreate the container using the export/import feature of Synology. That failed with invalid file format. Then I resorted to removing the container and image and reinstalling that. I figured that since most of my docker images were from linuxserver I might as well try that. I did not go to recreating the whole environment just for the hell of it, rather I took a series of steps that failed to get deep in the mud. Right now I just want to get back up.
Again, I have no “previous Nextcloud installation from backup” to restore. I have a docker image. I have a docker container. I have an export of the docker container’s configuration. I can re-pull the docker image… I have re-pulled the docker image. I can recreate the docker container. Alas, I cannot recreate the docker container from the export because it says the import file is an invalid format. I can re-create the docker container from the docker image using the export JSON file as a guide to re-set all of the volumes, env vars, ports, etc. When I do I get an internal error that I reported above.
I did not cause this problem. NC was running A-OK. It was Synology who said my MariaDB container (tied to NC) needed updating. I updated it and all hell broke loose.
Last I checked my NC installation has about 256G of used space. I do not have the space to back up 1/4 of a TB. I still have the files though. They are in a safe place and can be mounted to the NC container. However, the NC container has an internal error so I cannot get in there to get things to work.
I have a script which goes into the docker container and runs:
Inconsistent and misleading error messages are not helpful.
Wonder what happens if I try the update… Seems to have updated!
Logging into NC I see a pretty plain set of files. Time to run that php console.php scan again… Dang, all my previous share links are gone! Maybe a full rescan will get it back but I doubt it as it hasn’t done it before. This means when I hand out those links to my tech illiterate friends they now have broken links that don’t work.
Then it is understandable why you are having such trouble. The way docker works is you must map your data to an actual location, which is known as a volume (or physical location on disk). Images and containers are ephemeral and not mounting for storing anything.
Until you begin making backups, and can know how to restore them, you will continue to fail with Nextcloud as a service. Your data should live on a volume on your physical disk, so try checking back through your run scripts or volume mounts in docker-compose to understand where it is.
Also, you can use Docker to spin up more than one Nextcloud instance. There is no need to fight through “ProductionInstance” when you could just as easily spin up “TestInstanceA” and “MigrationInstanceB” to safely work through all of these experiments. But, that is Docker specific discussion as opposed to this forum.
I have volumes, trust me. For example, I map /volume1/System/NextCloud to /var/www/html/data in the container and I map /volume1/docker/nextcloud to /var/www/html/config. And while my data is “safe” residing on my Synology in a volume that’s mounted to the container, I don’t consider this a backup.
Stated differently, the apparent fact that updating the MariaDB docker container made it such that NC was inaccessible is a problem. What we have here is the separation of code (NC code running in the container) and data (mounted as volumes in the NC container). Now if NC in the container just needed to be restarted or reinstalled/updated, start it up and wham everything is good I wouldn’t mind. But it wouldn’t restart and it wouldn’t let me in making me resort to trying to re-install the image and re-create the container. Now if all the data and config remained the same (remember these we safe in their volumes) one would expect that NC would come up and say “Oh, OK. Here’s where all this stuff is. I can just continue on as before”. But it didn’t, it came up and said, “New installation? OK configure me”.
This then led to re-configuring the NC installation with a bare-bones/default set of files. But my data volume was mounted? Apparently, NC knew nothing about all the file information stored in the MariaDB database. Ugh.
Luckily I know about php console.php files:scan and ran my script which rescans and updates all the metadata in MariaDB. This takes about an hour and a half! Luckily I got back up and managed to eventually see my files in the NC interface as well as davfs2 mount my SongBook to /opt/songbook on my desktop. But it’s still slow as anything and emits strange errors like:
Earth:rm index
rm: cannot remove 'index': Invalid argument
Earth:
It is simply a file and an rm returns “Invalid argument”???
Also:
Earth:time cd /opt/songbook
real 0m26.570s
user 0m0.023s
sys 0m0.013s
Earth:
27 seconds just to cd into the directory?!?
This is why I said if you sneeze at NC you may be spending hours recreating your instance.
This wall of text approach makes it almost impossible to help you. Since you are having trouble running a command like cd it might be best to look at your underlying server infra. I have none of these issues and cannot help you further. Good luck.
I’m sorry. Perhaps I should endeavor to not explain things fully and leave out choice pieces of data points so that I don’t overwhelm you with a “wall of text” and provide you with opportunities to guess those missing pieces. Please forgive me…
My server infrastructure is just fine. It’s all on a local LAN and accessing remote file systems from my desktop over NFS mounts works just fine and quickly.
Earth:time cd /Music
real 0m0.060s
user 0m0.009s
sys 0m0.006s
Earth:df .
Filesystem 1K-blocks Used Available Use% Mounted on
jupiter:/volume1/Media 28113245312 19769732224 8343513088 71% /nas
Earth:
I have been having my issues with php dependencies which kept on breaking Nextcloud out of the blue, but have to say that ever since I moved my instances to docker it’s been smooth sailing.
Said this, one thing I keep an eye on is database versions. I use Postgres and have learned that one can not simple move from e.g. v14 to v15, no matter if docker or not. You need to migrate.
And I think I remember that mysql requently had such a breaking change as well, this is likely what hit you as well.
As annyoing as it may be, one could argue that it’s not really NCs fault if the db vendor decides that a db version update requires manual intervention.
In any case, my recommendation would be the start with a fresh install, import your files and run a occ scan, like you did before.
Once things are to your liking, think about your backup strategy.
I do multiple levels:
File system: shut down docker containers, run borg backup to an external disk, restart
DB dump: run a sql dump into file ( which gets picked up by Borg)
That approach gives me multiple ways and approaches to restore if I ever need to.
A few things here. I use MariaDB in a Docker container. In a Docker context, I would assume pulling a Docker image where MariaDB’s version goes from say X to X+1 and if data migration is required for the new X+1 image then it should contain the necessary start-up to convert from X → X+1. IIRC other Docker images like Plex do this. I would hope Postgres would do likewise but I know nothing about Postgress. In fact, WRT the Plex Docker image, if the contain is restarted and there’s a new version it will pull that image and update the Docker container. IMHO that’ shows it should be (unless impossible). IMHO, at the very least, if a migration is required you should be notified that a migration is required first and not be allowed to continue unless you approve (or a switch for those bold enough to say “Always install new versions”). There was no notification in my case.
Fresh installs I’ve done many times. It’s a step backward that I really wish to avoid.
To be clear I did not “Import my files” rather I pointed the Docker container to my set of files contained in a directory in my large filesystem on my Synology, and re-ran occ scan. That caused NC to learn of the existence of all these files but alas it lost all the share links and other customizations which I then recreated, only to be told to fresh install yet again?!? This is madness!
I’m not sure of these “Borg” you speak of, nor what exactly I’m trying to back up. The files remain safe outside of NC in a volume that is mounted and I don’t have enough backup space to backup my whole Synology.
In any event, I did manage to convert to MariaDB now but I’m still skeptical that this will not happen again. One thing I will not do is allow the Container Manager in Synology to update either the MariaDB image or NextCloud image without consulting here next time.
Well, they don’t. The DBs I know and use offer an image of a version and don’t know and care what, if any, version this image is going to replace. ANd I’d say that’s the right thing to do because images should be completely stateless.
That’s the way! You should decide when and how to update. I don’t know how it’s handled in Synology but for me it’s quite simple, whenever NC notifies me that a new version is available a will wait a few days, change the image string in my docker compose from let’s say 27.1.2 to 27.1.3 and do a docker-compose down --rmi all / up -d
That’s my upgrade process, and it served me well for years.
As far as the db is concerned I would strongly advise to not use the “latest” image, because that way you might end up with an unwanted and unplanned migration.
I use “image: postgres:15-alpine” which will give me that latest within the v15 line, because I know that as long as I stay on the same major version I won’t have to migrate. If this was like image: postgres:latest-alpine trouble is ahead whenever Postgres releases a v16 without me knowing. Maybe they did already?
The same applies to MariaDB as far as I know.
Well, I disagree. It should “just work” especially with a little bit of effort it can just work as evidenced by others where it just works. Additionally failing to put in a warning that a migration is needed is just dumb. The decision and desire to be on the latest version, especially considering security issues and concerns in this day and age is the smart way to go IMHO and YMMV. Software needs to adopt the “First do no harm” principle as well as “make sure the patient’s prescriptions are up to date”. Again this is my opinion and like yours I am allowed to have it.
No, it’s not, at least not in the case of MariaDB. The smart thing to do is to use an LTS release and then stay with it until your application requires a newer version.
The latestst LTS release 10.11 will be maintained until at least February 2028, and the previous LTS, 10.6, which is still supported by NC btw, will be maintaind until July 2026: https://mariadb.org/about/#maintenance-policy
So there is absolutely no need to jump to a new major release every year because of potential security issues.
Suffice to say, I disagree. Potential security threats are often real. This has been shown to be the case many times before. It’s not always the case but then again you only need one.
Sorry, but I don’t think you understood what LTS means or how the general release maintenance periods work with MariaDB. Maybe you should have clicked on that link I provided
TL;DR: Long-term maintenance releases (LTS) will be maintained for at least 5 years, and short-term maintenance releases for at least one year. During these maintainance periods, you will get bug fixes and security updates, but no new features that might require a migration/upgrade of the database itself.
@bb77 You are wrong. I know what LTS means. I didn’t need to click the link. Still, I disagree with you just like you disagree with me.
@tomz I don’t need you to dispute something for me to simply assert a fact. Sorry, that seems to trigger you and cause you to mischaracterize me to be semi-aggressive in tone. Such quick jumps to judgment without any sight of confirmation must often cause you great pain.
You can disagree with me all day long, but that doesn’t change the fact that you are factually wrong.
There might be a multitude of good reasons to upgrade to a newer version of MariaDB, but security isn’t one of them, as long as the version you’re running is still maintained by MariaDB.
