I mentioned elsewhere that I my pi lost power and isn’t behaving right (docker says it’s running but I cannot communicate via local network, and cannot get X started to attempt from localhost). At first I was going to try to fix things, but after noticing some other issues (such as not being able to start X) I’m wondering if the USB that I was booting from isn’t a POS (and part of the problem) and am thinking that re-installing might be my best option. While it wouldn’t be the end of the world if I started from scratch as I’ve only been using nextcloud for about 3 weeks and I have other copies of all the data, I’d like to try to salvage my user configurations and data if possible. When I installed I used the new feature to store the data on a specific and separate drive.
Please ignore the port translations there, I’m doing some network trickery to get other services running as well.
I have been able to mount that /media/USB/NCstorage/ncdata/data directory on a newly installed Raspbian OS, however I noticed there were some permissions issues when first trying to mount. I was originally thinking that if I did a fresh install of docker the exact same way I did on my first run-through and then powered it off and replaced it’s data directory with a copy from my old one I might get away with it. Will this work? Will I have to do any work to make permissions right (both linux and NextCloud user permissions)?
Thanks for any help. This was exactly the sort of thing I was hoping to be able to do by storing the data elsewhere, though I was hoping to not have to do it.
Well, you might get some way into it, but you shouldn’t.
Instead, backup the data from within your user(s) in the data directory (eg /var/www/html/nextcloud/data/files/jim/) then copy/rsync those into the new user folder on the new install. Check the ownership of the directories on the new install before copying over, then match them when the copy is finished, before doing an occ files scan.
Thanks for your help. I’m more of a networking+windows guy than a linux sysad, so take it easy on me here as I’ve got a bunch of questions
I’m thinking that your example is referencing a normal install of Nextcloud as opposed to the docker version - just wanted to make sure you knew I was using the docker image. For what it’s worth it looks like my user files are under /media/USB/NCstorage/ncdata/app/data/[username] and the database is under the ncdata directory in that filepath. So you’re saying I should just copy the user data to the new install? Do I need to first create the users? If that were the case no user configurations such as passwords, previous sharing, or storage limits would be migrated, correct?
Alright, I know chown is probably going to be involved here, but I’m not sure exactly how to check permissions. Would you mind giving me some more specific commands as examples? I could probably look some of this up, but I don’t want to miss anything that would involve knowledge I don’t have such as perhaps needing to force recursion on these or something. Also what is an occ files scan?
I was hoping, if possible, to migrate the user settings (as mentioned above such as passwords and sharing) as well. It this isn’t possible that’s fine. At this point I just want to make sure I’m understanding this right. Or a little more than the nearly-not-at-all that I’m at right now.
Thanks nachoparker. Yes this is the own-your-own-bits version, so that is excellent news. Thanks so much! I’ll try to give this a shot tonight, but it turns out a 9 month old child takes a lot of your time so we’ll see if I get a chance. I’ll report my findings whenever I get it done.
Have any recommendations on how I should re-use the volume? Do I need to copy it with any commands to make sure permissions are right, or can I just mount it to the location that the container is expecting its data? If that works I’m thinking I’d actually probably disk dump it to a new volume and mount that as if something goes wrong I won’t want to mess up the original data.
Web interface isn’t working (nether the 443 one, nor the 4443 admin one). That’s actually the main problem.
Unfortunately I had no success with that. Drat! Worth a try though. To be clear, my method was to:
make a copy of my original data volume
install the own-your-bits image of nextcloudpi
stop nextcloud and docker
unmount the nextcloudpi data volume
mount my copy of the data volume at the same mount point as what I just unmounted
start docker and nextcloudpi
Perhaps there is a version issue with the image I just installed?
I think this has been the part I’ve been missing on how to troubleshoot docker containers. So basically you can run any commands you’d normally be able to with the program, you just need to prepend docker exec [containername] to them? Once I do that does ncp-config give me a cli method to backup my database? How? Maybe this is the route I should be going.
EDIT: I ran sudo docker exec nextcloudpi ncp-config and it said “TERM environment variable needs set.” I have no idea what this means, but it did not bring up a menu.
Alright, well I’m making progress and learning a lot here. But I’m still stuck. I was able to run ncp-config. I figured, hey why not try to backup my old data. So I booted into my original install of raspbian with the original docker setup and ran ncp-config and select nc-backup (I had mounted a USB drive before doing this). it seemed to run, but nothing showed up on the USB drive. Is this because it’s containerized and just going… nowhere? Here’s what the output was:
Launching nc-backup
check free space...
Maintenance mode enabled
backup database...
backup base files...
backup /media/USB/backup/nc-backups/nextcloud-bkp_20180612_1528838650.tar generated
Nextcloud is in maintenance mode - no apps have been loaded
Maintenance mode disabled
Done. Press any key...
pi@hostname:~ $ ls /media/USB/backup/
lost+found nc-backups
pi@hostname:~ $ ls /media/USB/backup/nc-backups/
It clearly states that it made a backup at /media/USB/backup/nc-backups/[filename] but when I ls nothing is there.
Keep in mind I’m a serious noob here. Where might I find associated logs?
Also, as far as doing a copy/rsync - I’m still unclear as to if this will work without first creating the users? Does an occ files scan create user accounts for users that didn’t already exist?
Also, one thing I didn’t mention - given that this is the own-your-bits nextcloudpi (nextcloudplus) it includes some features like notes and calenders that I’m wondering if they might not be part of the standard base nextcloud install? Either way, I did create a calendar originally and I don’t see it in my user’s data folder. Anyone know where that gets stored?
in the docker container, the default location for backups is /data/backups. Your /data folder inside the container corresponds to your external folder /media/USB/NCstorage/ncdata
Not familiar enough with NCP’s changes, but I’d try running the backup again from their interactive tooling:
If that wiki doesn’t cover it, please consider commenting on the linked Issue so that it can be properly documented.
Also, as far as doing a copy/rsync - I’m still unclear as to if this will work without first creating the users? Does an occ files scan create user accounts for users that didn’t already exist?
It does not. You will need to create the users in the new install, then copy the files into place after that. I’d recommend only doing this to the [username]/files/* folders first if you go this route rather than the backup. Doing it on the other folders within each user account could work, but I’d try them out one at a time in case there are issues.
Also, one thing I didn’t mention - given that this is the own-your-bits nextcloudpi (nextcloudplus) it includes some features like notes and calenders that I’m wondering if they might not be part of the standard base nextcloud install? Either way, I did create a calendar originally and I don’t see it in my user’s data folder. Anyone know where that gets stored?
If it’s the standard Nextcloud Notes app (as opposed to OwnNotes[?] which requires an API endpoint), then it actually stores the notes themselves in [username]/files/Notes, with only redundant metadata elsewhere. No problems with that one. The calendar relies on the database, though, so again the full backup/restore is your best bet.
Is there any way to change that to something outside of the container? If not I can make it work for now, but ideally I’ll be setting an auto-backup and I’d rather this be on a separate partition or drive all together.
That’s actually what I was doing. The output I posted was what it displayed after exiting the menu screen. I’m not sure if it’s a bug or not yet, but the whole system is responding poorly so I’m not surprised honestly, just hoping to find out whats tripping it up. My first thought was, because I don’t know how docker works, maybe it’s not being allowed to save at the location I chose because I chose something outside of the container.
Alright, well it’s been a long and strange journey for me. But right now I’m watching a putty screen saying that my backup is being restored! It did appear that I was unable to save anything outside of the docker data directory for backups. Conversely I was unable to restore anything outside of the docker data directory. So backups needed to be copied out of and into the docker container directories, but once I figured that out I was good.
This seems less-than ideal. Is this a limitation of being installed as a docker instance or could this be a feature request for future improvement? I certainly don’t mean to complain. I’ve been happy using everything thus far and appreciate the help I’ve been getting for support. Nachoparker you seem like a busy guy so I can’t say thanks enough.
Is there anything I need to be aware of after the restore function? I realize I may need to reconfigure some settings, one of my biggest questions is: will clients that were syncing automatically reconnect, or will they see this as a different instance and get upset about that?
Hey, thanks again guys. NC and docker are both completely new to me and before I started my first post last weekend I hadn’t even seen the nc command reference sheet less likely know how to invoke any of the commands in a docker container.
EDIT: Ok well that took a while and I thought we were good, but I was wrong. After the restore finished I don’t see my users existing in the data file. The output on the restore looked like this:
Launching nc-restore
INFO: overwriting old instance
extracting backup file /data/backups/nextcloud-bkp_1.tar...
restore files...
redis: unrecognized service
Cleanup...
Done. Press any key...
So I guess the unrecognized service threw some stuff off? Any ideas on how to find out what’s going on there?
/media/USB/NCstorage/ncdata is both inside and outside of the container
if you restored it is the same instance, they won’t see a change.
Seems like you found a bug. Sorry about that. I fixed it in v0.56.22. If you update through ncp-update or ncp-config or nc-web it should work this time.
It is true that the docker version is intended for a more advanced user, compared to the SD card version. There are plans to make it easier to download and upload the backups through the web interface, but little time to do it.
Alright, that made me laugh at myself in shame! I preach staying up-to-date all the time, but knowing that something isn’t working right I’ve been purposefully not updating until after the restore for fear that that process will damage the data or that perhaps the restore might not work with different versions of a running system vs. the backed up system. I can see where that got me! Well here goes one more try!
Just an update. The restore completed successfully and I’m now using my NC server again. As expected all data was there and my devices synced up perfectly. It’s worth noting that I’m getting a message on the web gui that says “There are problems with the code integrity check”. I looked into this a little and found it was a lot of tiles in the “data-6-14-18” directory. I kind of assumed this was a backup or something that was created during the install process and just moved it out of the data directory and have not noticed any adverse affects. This removed 99% of the items listed as being a problem, but there is still one. When I select the list of invalid files I get this:
Technical information
=====================
The following list covers which files have failed the integrity check. Please read
the previous linked documentation to learn more about the errors and how to fix
them.
Results
=======
- core
- INVALID_HASH
- .user.ini
Raw output
==========
Array
(
[core] => Array
(
[INVALID_HASH] => Array
(
[.user.ini] => Array
(
[expected] => 467d4f028c447895716a2b7859ed6e569f8ee34b87b51d73dab2e6a9ca4fbe63172c7be3e365ae864a60408286afcce852dd38ee544b61685ed4ee5e021fecb0
[current] => 42eca5f14f21e928e8fa2ce3db4ca2ae1c6b055fbc02e52775e785822292c833bdc2ab832110fcc542ffe41a480c294dafc3dac4d20538f0f50c9aa7c662f00f
)
)
)
)
Not sure what that was about. I saw another update from Nextcloud 13.0.2 to 13.0.4 and ran that and it appeared to solve everything. I’m not sure if I should have done that because on the https://[server]:4443 interface the newest version it sees is 13.0.2. I’m not sure what to call that page vs the regular 443 login is.
Either way, I think I’m comfortable closing this at this point as long as you don’t think updating to 13.0.4 is a major issue?
Worth a try though. To be clear, my method was to:
