An almost complete migration (borked 'external storages')

[dietert]

HI all,

i’ve just completed (almost) migrating from the Ubuntu 18.04LST based version of the VM to the 20.04LTS version. see :

Regarding Nextcloud VM and OS versions

Since we upgraded the OS to Ubuntu 20.04, there have been some confusion regarding what’s what. There’s also been several bug reports regarding that the VMs doesn’t work, script fails, and so on… In most cases that’s not correct (related to downloading an old VM), but for some users the ZFS mount failed to import on newer VMs, causing .ocdata to go missing, which resulted in that the script failed to run the startup script at all. That’s now fixed, and new versions are uploaded. (2020-05-03). I…

A few days ago i noticed that 20.0.2 installed correctly on the 18.04 version so i took this as i sign that i should migrate now. I followed the migration procedure outlined in the manual (tar balled /var/www/nextcloud/ and rsync’d it over the 20.04 machine + backed up and restored the databases + edited config.php to have all the old passwords of for all the stuff (except the database which has a auto-generated password for at the time of VM setup) as per the old machine. (anyway i followed the manual’s guidance)

the only thing that seems complete broken is the ‘external storages’ admin interface which yeilds an internal server error every time i click on it. in my old machine i had ldap users configured and they mounted smb shares using save credentials in database. I’m happy to re-enter the config if i can get back in but i’m not sure how to reset ‘external storages’ so that i can edit things.

here is what the log says when i grep’d for the error code:
{"reqId":"4sLZUOfSJ4j6DwSM0Fbv","level":3,"time":"2020-12-09T21:30:48-06:00","remoteAddr":"192.168.1.196","user":"ncadmin","app":"index","method":"GET","url":"/settings/admin/externalstorages","message":{"Exception":"Exception","Message":"HMAC does not match.","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Security/CredentialsManager.php","line":101,"function":"decrypt","class":"OC\\Security\\Crypto","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/apps/files_external/lib/Lib/Auth/Password/GlobalAuth.php","line":56,"function":"retrieve","class":"OC\\Security\\CredentialsManager","type":"->"},{"file":"/var/www/nextcloud/apps/files_external/lib/Settings/Admin.php","line":72,"function":"getAuth","class":"OCA\\Files_External\\Lib\\Auth\\Password\\GlobalAuth","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/Controller/CommonSettingsTrait.php","line":141,"function":"getForm","class":"OCA\\Files_External\\Settings\\Admin","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/Controller/AdminSettingsController.php","line":83,"function":"formatSettings","class":"OCA\\Settings\\Controller\\AdminSettingsController","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/Controller/CommonSettingsTrait.php","line":152,"function":"getSettings","class":"OCA\\Settings\\Controller\\AdminSettingsController","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/Controller/AdminSettingsController.php","line":68,"function":"getIndexResponse","class":"OCA\\Settings\\Controller\\AdminSettingsController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":169,"function":"index","class":"OCA\\Settings\\Controller\\AdminSettingsController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":100,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":152,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":308,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/lib/base.php","line":1008,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/index.php","line":37,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/nextcloud/lib/private/Security/Crypto.php","Line":139,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) xxxxxxxxxx","version":"20.0.2.2"}

p.s. the plural of storage is storage not ‘storages’ (I’m a native English speaker for what its worth)

p.p.s. if i’ve done something stupid with my migration i can always try again. i’ve snapshot’d the new VM in a just after basic setup state…

[dietert]

so googleing “HMAC does not match” i get

https://help.nextcloud.com/t/hmac-does-not-match/56900

which leads me too :

Server Migration · Wiki · nextcloud / passwords

trying this i get

sudo -u www-data php /var/www/nextcloud/occ passwords:backup:restore migration
This backup file will be used: migration

The following will be restored:
 - The Nextcloud server secret
 - Server and user encryption keys
 - User passwords, folder, tags and shares
 - Application settings
 - User settings
 - Third party client settings

Restoring user data means that the current user data will be wiped.
❗❗❗ The backup "migration" will now be restored ❗❗❗
Type "yes" to confirm this: yes

Restoring backup ... done
An unhandled exception has been thrown:
TypeError: Return value of "OCA\Passwords\Command\BackupRestoreCommand::execute()" must be of the type int, "null" returned. in /var/www/nextcloud/apps/mail/vendor/symfony/console/Command/Command.php:261
Stack trace:
#0 /var/www/nextcloud/apps/mail/vendor/symfony/console/Application.php(920): Symfony\Component\Console\Command\Command->run()
#1 /var/www/nextcloud/apps/mail/vendor/symfony/console/Application.php(266): Symfony\Component\Console\Application->doRunCommand()
#2 /var/www/nextcloud/apps/mail/vendor/symfony/console/Application.php(142): Symfony\Component\Console\Application->doRun()
#3 /var/www/nextcloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run()
#4 /var/www/nextcloud/console.php(100): OC\Console\Application->run()
#5 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')
#6 {main}

any thoughts?

[dietert]

so prior to this i was able to at least log in (i had had to disable uf2 second factor auth since it also was broken) now i get an error when i try to log in too (before i get anywhere at all)…

going to try two things:

1. revert my new vm to the before i fu#ked with the passwords DB state and try :
   External Storage - HMAC does not match
2. going to retry the do-release-upgrade method at this this my 5th time thought trying to migrate this VM - this whole process not very DYI home lab friendly - sorta undermines the whole ‘data autonomy for everyone’ promise of next cloud… i tried this once already but now and it broke everything i have a fresh install of the 20.04 version of the vm so maybe i can compare the config file that ubuntu wants to modify as it does the upgrade to 20.04…

[dietert]

for options 1 when i do :

\dt and list all the tables inside of nextcloud_db (and grep the list) i get

public | oc_storages_credentials | table | ncadmin

which close to

@space-cadet

oc_credentials

from the link i found about about HMAC problems and external storage
trying to truncate oc_storages_credentials still gives and internal server error and the log looks like this:

root@nextcloud:~# cat /var/log/nextcloud/nextcloud.log | grep J1oJczjPAOTkDReb2y3B
{“reqId”:“J1oJczjPAOTkDReb2y3B”,“level”:3,“time”:“2020-12-22T13:15:11-06:00”,“remoteAddr”:“192.168.1.196”,“user”:“ncadmin”,“app”:“index”,“method”:“GET”,“url”:“/settings/admin/externalstorages”,“message”:{“Exception”:“Exception”,“Message”:“HMAC does not match.”,“Code”:0,“Trace”:[{“file”:“/var/www/nextcloud/lib/private/Security/CredentialsManager.php”,“line”:101,“function”:“decrypt”,“class”:“OC\Security\Crypto”,“type”:“->”,“args”:[“*** sensitive parameters replaced ***”]},{“file”:“/var/www/nextcloud/apps/files_external/lib/Lib/Auth/Password/GlobalAuth.php”,“line”:56,“function”:“retrieve”,“class”:“OC\Security\CredentialsManager”,“type”:“->”},{“file”:“/var/www/nextcloud/apps/files_external/lib/Settings/Admin.php”,“line”:72,“function”:“getAuth”,“class”:“OCA\Files_External\Lib\Auth\Password\GlobalAuth”,“type”:“->”},{“file”:“/var/www/nextcloud/apps/settings/lib/Controller/CommonSettingsTrait.php”,“line”:141,“function”:“getForm”,“class”:“OCA\Files_External\Settings\Admin”,“type”:“->”},{“file”:“/var/www/nextcloud/apps/settings/lib/Controller/AdminSettingsController.php”,“line”:83,“function”:“formatSettings”,“class”:“OCA\Settings\Controller\AdminSettingsController”,“type”:“->”},{“file”:“/var/www/nextcloud/apps/settings/lib/Controller/CommonSettingsTrait.php”,“line”:152,“function”:“getSettings”,“class”:“OCA\Settings\Controller\AdminSettingsController”,“type”:“->”},{“file”:“/var/www/nextcloud/apps/settings/lib/Controller/AdminSettingsController.php”,“line”:68,“function”:“getIndexResponse”,“class”:“OCA\Settings\Controller\AdminSettingsController”,“type”:“->”},{“file”:“/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php”,“line”:169,“function”:“index”,“class”:“OCA\Settings\Controller\AdminSettingsController”,“type”:“->”},{“file”:“/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php”,“line”:100,“function”:“executeController”,“class”:“OC\AppFramework\Http\Dispatcher”,“type”:“->”},{“file”:“/var/www/nextcloud/lib/private/AppFramework/App.php”,“line”:152,“function”:“dispatch”,“class”:“OC\AppFramework\Http\Dispatcher”,“type”:“->”},{“file”:“/var/www/nextcloud/lib/private/Route/Router.php”,“line”:308,“function”:“main”,“class”:“OC\AppFramework\App”,“type”:“::”},{“file”:“/var/www/nextcloud/lib/base.php”,“line”:1008,“function”:“match”,“class”:“OC\Route\Router”,“type”:“->”},{“file”:“/var/www/nextcloud/index.php”,“line”:37,“function”:“handleRequest”,“class”:“OC”,“type”:“::”}],“File”:“/var/www/nextcloud/lib/private/Security/Crypto.php”,“Line”:139,“CustomMessage”:“–”},“userAgent”:“Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0”,“version”:“20.0.4.0”}

[dietert]

so now i’m trying the do-relaese-update road:
so far theese are the config files ubuntu is highlighting as haing config changes:

• Something i didn’t write down

• modsecurity (/etc/modsecurity/crs/crs-setup.conf) (used package mantianers version since it seems like the 20.04 version doesn’t have that anymor)

• systemctl (/etc/sysctl.conf) (used exsiting version - will cross check against the 20.04 version of the VM…)

• samba (/etc/samba/smb.conf) kept the currently customized version…

• redis (/etc/redis/redis.conf) this one makes sense - there has been some nomenclature changes so i accepted the PM version will re-edit in the config at the beginning about sockets, permissions, etc… (i save the old one for reference and creating a diff manually)

• postgressSQL need upgrading from 10 to 12… there is a splash screen with info one how to do this…

• the config for coturn - i think this is an artifact of my efforts to get NC talk working. I’m going to purge coturn from this server anyhow so it didn’t matter to me…

• http2.conf (/etc/apache2/mods-available/http2.conf’) this is an important one… i’ve saved the old one and will compare with the fresh version of the VM and the new defult config… the major change seem to be :
  @@ -1,4 +1,34 @@

- <IfModule http2_module>
+
+# mod_http2 doesn’t work with mpm_prefork
+<IfModule !mpm_prefork>
Protocols h2 h2c http/1.1
- H2Direct on

• modsecurity.conf-recommended (/etc/modsecurity/modsecurity.conf-recommended) again mod secuity doesn’t seem to be used on 20.04 however all the changes seem to be additions so i don’t think its a problem to use the PM version. i backed the old one up just in case…

[dietert]

things are working again!!!

the migration path didn’t work (again) - my theory is something i breaking because i’m using the u2f 2nd factor and passwords plugin was causing lots of weirdness - but i could be totally wrong. after much head scratching here is what i did:

do-release-updated as per the post above this then used my half working ‘new’ 20.04 as a donar and transplanted config files, scripts and installed apps…

1. I upgraded my postgressql to 12 from 10

2. created tar ball(s) of the following configs and script from the ‘new’ VM

   /etc/php/7.4/*
   /etc/apache2/*
   /var/scripts/*

3. i then scp them over to the ‘old’ do-release-upgraded version of the VM and rscyn’d them to the correct places.

4. i used :
   4.1 sudo dpkg-query -f ‘${binary:Package}\n’ -W > packages_list.txt
   on the ‘new’ VM to create a txt file that had all the packages and then used
   4.2 scp to move the file to the ‘old’ vm an then used :
   4.3 sudo xargs -a packages_list.txt apt install to make sure that all the packages on the ‘new’ VM were installed on the ‘old’ VM

this list is from memory and there was hours of googling and back and forth and trying things and reverting them. Sorry if i forgotten something if you are trying to follow in my foot steps…