Return on experience, migration from MySQL to Postgresql

Hello,

I have migrated my nextcloud 22 intance from MySQL to Postgreql
While this activity could seem trivial using the built-in migration command

occ db:convert-type --all-apps --clear-schema --password <password> pgsql <user> <pg host> <db name>

it turns out in my case that the migration of data was real slow.
activity tables (~40Mo) would take 2hrs, filescache (1G) would take 4 days !

So I came with an alternative method that I describe here, for the sake of anyone needing this …

I first created the schema with the occ db:convert-type, hiting [n] when asking to start.
This made my pgsql database with a clean schema
I also noted the tables that would not be converted by the db:convert-type.

I then installed pgloader

create a loader.load file with bellow content

LOAD DATABASE
        FROM mysql://<mysqluser>:<mysqlpass>@<mysql host>/<mysql db name>
        INTO pgsql://<pg user>@<pg host>/<pg db name>
        WITH create no tables, disable triggers, preserve index names, include no drop, truncate, create indexes

        CAST type datetime to timestamptz drop default drop not null using zero-dates-to-null,
         type date drop not null drop default using zero-dates-to-null
EXCLUDING TABLE NAMES MATCHING  ~/^audioplayer_.+/,~/^contacts_cards_properties$/, ~/^fulltextsearch_.+$/,~/^news_.+$/,~/^ocsms_.+$/,~/^passman_.+$/,~/^phonetrack_.+$/,~/^privatedata$/,~/^radio_stations$/
;

the Exludeing table names regex I set to not migrate the tables that would not be migrated by the occ db:convert-type

the command execution took only 6 minutes

My setup is Nextcloud running docker, so:

  • I added a container for target pgsql
  • I added a container to run pgloader

Note: pgloader seems to have a bug with pgsql database password, so I had to edit pg_hba.conf to add the host where pgloader was running to trust mode so that it would work without a password

That seems like an absurdly long time. You sure it would have actually taken that long and that it wasn’t just a bad estimate?

And if you don’t mind my asking, what’s the motivation for switching to postgresql?

wouldn*t that make more sense to be filed under Howto and make it a Wiki?

Hi Larry,
No the estimates where rigth, Ieft the migration go on for a full day.
What bother me is the maintenance window beeing to long where I could not use the server

As for the motivation to switch to pg, I read several articles mentioning the better performance of pg

not to mention that I use pg for work on a daily basis …

yes this is what I was willing to do, but for some reason I was not allowed to post over there…

done :wink: :wink:

1 Like

I try you config but I got an error…

KABOOM!
FATAL error: pgloader failed to find schema "wolke7" in target catalog.
An unhandled error condition has been signalled:
   pgloader failed to find schema "wolke7" in target catalog.




What I am doing here?

pgloader failed to find schema "wolke7" in target catalog.

to solve schema error, need to alter old name to default one.
after WITH …
ALTER SCHEMA 'wrong_name' RENAME TO 'public'

Exclution regexp look not working for now, decide to remove it from code.

Also nice guide here How To Migrate a MySQL Database to PostgreSQL Using pgLoader | DigitalOcean

Just to show some data.

Clearing schema in new database
Creating schema in new database
 - files
 - activity
 - app_api
 - bruteforcesettings
 - calendar
 - circles
 - cloud_federation_api
 - comments
 - contacts
 - contactsinteraction
 - dav
 - federatedfilesharing
 - files_accesscontrol
 - files_downloadlimit
 - files_pdfviewer
 - files_reminders
 - files_sharing
 - files_trashbin
 - files_versions
 - firstrunwizard
 - integration_google
 - keeporsweep
 - logreader
 - lookup_server_connector
 - nextcloud_announcements
 - notes
 - notifications
 - oauth2
 - password_policy
 - photos
 - privacy
 - provisioning_api
 - quota_warning
 - recommendations
 - related_resources
 - richdocuments
 - serverinfo
 - settings
 - sharebymail
 - support
 - survey_client
 - suspicious_login
 - text
 - theming
 - twofactor_backupcodes
 - updatenotification
 - user_status
 - viewer
 - webhook_listeners
 - workflowengine
The following tables will not be converted:
oc_audioplayer_albums
oc_audioplayer_artists
oc_audioplayer_genre
oc_audioplayer_playlists
oc_audioplayer_playlist_tracks
oc_audioplayer_stats
oc_audioplayer_streams
oc_audioplayer_tracks
oc_carnet_metadata
oc_cookbook_categories
oc_cookbook_keywords
oc_cookbook_names
oc_cospend_bills
oc_cospend_bill_owers
oc_cospend_currencies
oc_cospend_members
oc_cospend_projects
oc_cospend_project_categories
oc_cospend_shares
oc_files_antivirus
oc_files_avir_status
oc_file_metadata
oc_music_albums
oc_music_ampache_sessions
oc_music_ampache_users
oc_music_artists
oc_music_cache
oc_music_playlists
oc_music_tracks
oc_ocdownloader_adminsettings
oc_ocdownloader_personalsettings
oc_ocdownloader_queue
oc_passwords_challenge
oc_passwords_folder
oc_passwords_folder_rv
oc_passwords_keychain
oc_passwords_password
oc_passwords_password_rv
oc_passwords_pw_tag_rel
oc_passwords_registration
oc_passwords_session
oc_passwords_share
oc_passwords_tag
oc_passwords_tag_rv
oc_phonetrack_devices
oc_phonetrack_filtersb
oc_phonetrack_geofences
oc_phonetrack_points
oc_phonetrack_proxims
oc_phonetrack_pubshares
oc_phonetrack_sessions
oc_phonetrack_shares
oc_phonetrack_tileserver
oc_radio_stations
oc_reader_bookmarks
oc_reader_preferences
oc_spreedme_tps
oc_talk_bridges
oc_talk_commands
oc_talk_guests
oc_talk_participants
oc_talk_rooms
oc_talk_signaling
oc_trusted_servers
Please note that tables belonging to disabled (but not removed) apps
can be included by specifying the --all-apps option.
Continue with the conversion (y/n)? [n] y
 - oc_oauth2_clients
    0 [>---------------------------] < 1 sec
 - oc_oauth2_access_tokens
    0 [>---------------------------] < 1 sec
 - oc_photos_albums
    0 [>---------------------------] < 1 sec
 - oc_photos_albums_files
    0 [>---------------------------] < 1 sec
 - oc_photos_albums_collabs
    0 [>---------------------------] < 1 sec
 - oc_privacy_admins
    0 [>---------------------------] < 1 sec
 - oc_richdocuments_wopi
 1/1 [============================] 100% < 1 sec/< 1 sec
 - oc_richdocuments_direct
    0 [>---------------------------] < 1 sec
 - oc_richdocuments_assets
    0 [>---------------------------] < 1 sec
 - oc_richdocuments_template
    0 [>---------------------------] < 1 sec
 - oc_login_address
chunked query, 3 chunks
 2072/2072 [============================] 100% 2 secs/2 secs
 - oc_suspicious_login_model
 340/340 [============================] 100%  1 sec/1 sec
 - oc_suspicious_login
chunked query, 5 chunks
 4213/4213 [============================] 100% 3 secs/3 secs
 - oc_login_ips_aggregated
 363/363 [============================] 100%  1 sec/1 sec
 - oc_text_documents
 4/4 [============================] 100% < 1 sec/< 1 sec
 - oc_text_sessions
 4/4 [============================] 100% < 1 sec/< 1 sec
 - oc_text_steps
 57/57 [============================] 100% < 1 sec/< 1 sec
 - oc_twofactor_backupcodes
    0 [>---------------------------] < 1 sec
 - oc_user_status
 5/5 [============================] 100% < 1 sec/< 1 sec
 - oc_webhook_listeners
    0 [>---------------------------] < 1 sec
 - oc_flow_checks
 3/3 [============================] 100% < 1 sec/< 1 sec
 - oc_flow_operations
 1/1 [============================] 100% < 1 sec/< 1 sec
 - oc_flow_operations_scope
 1/1 [============================] 100% < 1 sec/< 1 sec
 - oc_notifications_settings
 7/7 [============================] 100% < 1 sec/< 1 sec
 - oc_migrations
Skipping migrations table because it was already filled by running the migrations
 - oc_storages
 11/11 [============================] 100% < 1 sec/< 1 sec
 - oc_mounts
 13/13 [============================] 100% < 1 sec/< 1 sec
 - oc_mimetypes
 143/143 [============================] 100% < 1 sec/< 1 sec
 - oc_filecache
chunked query, 6416 chunks
 3472000/6415044 [===============>------------]  54% 2 days, 10 hrs/4 days, 11 hrscommand terminated with exit code 137

As this took me days (possibly a couple weeks) to figure out, I think I’ll lend my settings. I had the same issue with db:convert-type and figured this was the only way to get something going. My issue was that pgloader was having a heap exception a lot. I could do all of the tables individually but oc_filecache would never work. The following is what I used to finally get it to fully run successfully.

loader.load

LOAD DATABASE
        FROM mysql://<user>:<pass>@<host>/<database>
        INTO pgsql://<user>:<pass>@<host>/<database>
WITH create no tables, disable triggers, preserve index names, include no drop, truncate, create indexes,

    -- New/Adjusted settings for batching and concurrency:
    workers = 2,                 -- Default is 4. For parallel schema processing.
    concurrency = 2,             -- Default is 8. For parallel data loading and index creation.
    batch rows = 2500,           -- Drastically reduced from default 25000.
    batch size = 2MB,            -- Drastically reduced from default 20MB.
    prefetch rows = 500          -- Reduced from default 1000. For reader threads.

CAST type datetime to timestamptz drop default drop not null using zero-dates-to-null,
         type date drop not null drop default using zero-dates-to-null

SET work_mem TO '128MB', maintenance_work_mem TO '512 MB'
;

Take the conversion at your own risk.

# Sets the memory space ridiculously high. Not likely needed this high as I never saw it use that much.
SBCL_OPTIONS="--dynamic-space-size 16096" pgloader --verbose loader.load

For the output: Ignore errors about target table not found. If db:convert-type didn’t setup the table, I don’t think it’s needed. I also checked the conversion warnings and I think they’re okay. Maybe someone on here can write all the custom rules to handle it, but I just ignored them.

Output:

2025-05-21T14:06:00.004000Z NOTICE Starting pgloader, log system is ready.
2025-05-21T14:06:00.016001Z LOG pgloader version "3.6.7~devel"
2025-05-21T14:06:00.211003Z LOG Migrating from #<MYSQL-CONNECTION mysql://nextcloud@10.43.148.151:3306/nextcloud {1006B6B3E3}>
2025-05-21T14:06:00.212003Z LOG Migrating into #<PGSQL-CONNECTION pgsql://nextcloud@10.43.219.157:5432/nextcloud {1006B6B823}>
2025-05-21T14:06:01.677022Z NOTICE Prepare PostgreSQL database.
...
Errors and warnings that you should double check.
...
Tables that are skipped.
...
Table COPY
...
2025-05-21T14:31:40.142905Z NOTICE Completing PostgreSQL database.
2025-05-21T14:31:40.142905Z NOTICE Reset sequences
2025-05-21T14:31:45.754977Z LOG report summary reset
                           table name     errors       read   imported      bytes      total time       read      write
-------------------------------------  ---------  ---------  ---------  ---------  --------------  ---------  ---------
                      fetch meta data          0        198        198                     0.201s
                    Drop Foreign Keys          0          0          0                     0.000s
                             Truncate          0        134        134                     1.231s
-------------------------------------  ---------  ---------  ---------  ---------  --------------  ---------  ---------
               nextcloud.oc_filecache          0    6415044    6415044     5.7 GB      24m50.335s  24m47.196s  24m20.399s
      nextcloud.oc_filecache_extended          0     141557     141557     3.3 MB          3.513s     1.168s     0.324s
    nextcloud.oc_files_metadata_index          0      63215      63215     3.1 MB          2.336s     1.801s     1.397s
          nextcloud.oc_files_metadata          0      73468      73468    74.7 MB          9.333s     9.039s     7.109s
             nextcloud.oc_files_trash          0      45496      45496     5.6 MB          2.945s     1.572s     1.299s
              nextcloud.oc_properties          0      27797      27797    20.1 MB          4.468s     3.480s     2.850s
              nextcloud.oc_file_locks          0       8457       8457   503.8 kB          0.338s     0.218s     0.029s
                nextcloud.oc_activity          0       5336       5336     1.5 MB          0.559s     0.301s     0.170s
        nextcloud.oc_suspicious_login          0       4213       4213   441.1 kB          0.774s     0.145s     0.020s
          nextcloud.oc_files_versions          0       3544       3544   176.8 kB          0.329s     0.190s     0.031s
           nextcloud.oc_login_address          0       2072       2072    79.5 kB          0.239s     0.147s     0.004s
         nextcloud.oc_calendarchanges          0        562        562    35.8 kB          0.219s     0.110s
             nextcloud.oc_preferences          0        522        522    29.4 kB          0.218s     0.084s     0.002s
        nextcloud.oc_cards_properties          0        506        506    19.2 kB          0.206s     0.096s     0.001s
    nextcloud.oc_login_ips_aggregated          0        363        363    18.3 kB          0.965s     0.016s     0.001s
               nextcloud.oc_appconfig          0        366        366    19.0 kB          0.256s     0.009s
  nextcloud.oc_suspicious_login_model          0        340        340    43.6 kB          0.229s     0.030s     0.006s
              nextcloud.oc_migrations          0        302        302     9.3 kB          0.269s     0.013s     0.001s
      nextcloud.oc_addressbookchanges          0        249        249    14.0 kB          1.041s     0.024s
   nextcloud.oc_calendarobjects_props          0        232        232    10.5 kB          0.227s     0.017s     0.001s
               nextcloud.oc_mimetypes          0        143        143     3.8 kB          0.265s     0.012s
         nextcloud.oc_calendarobjects          0        126        126   149.6 kB          0.254s     0.025s     0.006s
                    nextcloud.oc_jobs          0        110        110    18.0 kB          0.293s     0.017s
                   nextcloud.oc_cards          0         79         79   269.6 kB          0.283s     0.056s     0.009s
              nextcloud.oc_text_steps          0         57         57    12.0 kB          0.946s     0.013s     0.001s
                nextcloud.oc_comments          0         53         53     7.9 kB          0.271s     0.015s     0.001s
               nextcloud.oc_whats_new          0         53         53     2.9 kB          0.222s     0.011s
           nextcloud.oc_accounts_data          0         36         36     0.9 kB          0.214s     0.011s
               nextcloud.oc_authtoken          0         15         15    69.5 kB          0.247s     0.030s     0.003s
               nextcloud.oc_calendars          0         13         13     1.1 kB          0.212s     0.010s
                  nextcloud.oc_mounts          0         13         13     1.0 kB          0.903s     0.013s
              nextcloud.oc_notes_meta          0         13         13     1.7 kB          0.199s     0.009s
           nextcloud.oc_notifications          0         10         10     1.0 kB          0.275s     0.014s
      nextcloud.oc_calendar_reminders          0         12         12     1.8 kB          0.198s     0.007s
                   nextcloud.oc_share          0         12         12     1.6 kB          0.177s     0.007s
          nextcloud.oc_circles_member          0         11         11     2.6 kB          0.173s     0.010s
          nextcloud.oc_circles_circle          0         10         10     1.2 kB          0.449s     0.013s
      nextcloud.oc_circles_membership          0         10         10     1.6 kB          0.168s     0.016s
                nextcloud.oc_storages          0         11         11     0.3 kB          0.533s     0.007s
                nextcloud.oc_accounts          0          7          7     4.0 kB          0.220s     0.014s
  nextcloud.oc_notifications_settings          0          7          7     0.1 kB          0.210s     0.007s
                   nextcloud.oc_users          0          7          7     0.9 kB          0.194s     0.006s
            nextcloud.oc_addressbooks          0          6          6     0.4 kB          0.195s     0.005s
     nextcloud.oc_twofactor_providers          0          6          6     0.1 kB          0.153s     0.003s
             nextcloud.oc_user_status          0          5          5     0.3 kB          0.182s     0.004s
             nextcloud.oc_flow_checks          0          3          3     0.3 kB          0.157s     0.004s
          nextcloud.oc_text_documents          0          4          4     0.3 kB          0.124s     0.006s
           nextcloud.oc_text_sessions          0          4          4     1.3 kB          0.152s     0.004s
             nextcloud.oc_direct_edit          0          2          2     0.2 kB          0.157s     0.007s
  nextcloud.oc_notifications_pushhash          0          2          2     1.7 kB          0.161s     0.007s     0.001s
          nextcloud.oc_profile_config          0          2          2     0.8 kB          0.139s     0.004s
     nextcloud.oc_vcategory_to_object          0          2          2     0.0 kB          0.128s     0.004s
             nextcloud.oc_activity_mq          0          0          0                     0.091s     0.004s
            nextcloud.oc_appconfig_ex          0          0          0                     0.116s     0.005s
       nextcloud.oc_authorized_groups          0          0          0                     0.127s     0.009s
     nextcloud.oc_bruteforce_attempts          0          0          0                     0.080s     0.011s
   nextcloud.oc_calendarsubscriptions          0          0          0                     0.117s     0.004s
  nextcloud.oc_calendar_appt_bookings          0          0          0                     0.116s     0.007s
   nextcloud.oc_calendar_appt_configs          0          0          0                     0.108s     0.012s
    nextcloud.oc_calendar_invitations          0          0          0                     0.084s     0.011s
      nextcloud.oc_calendar_resources          0          0          0                     0.152s     0.004s
   nextcloud.oc_calendar_resources_md          0          0          0                     0.081s     0.009s
          nextcloud.oc_calendar_rooms          0          0          0                     0.115s     0.004s
       nextcloud.oc_calendar_rooms_md          0          0          0                     0.104s     0.003s
           nextcloud.oc_circles_event          0          0          0                     0.127s     0.014s
           nextcloud.oc_circles_mount          0          0          0                     0.094s     0.006s
      nextcloud.oc_circles_mountpoint          0          0          0                     0.096s     0.008s
          nextcloud.oc_circles_remote          0          0          0                     0.105s     0.004s
      nextcloud.oc_circles_share_lock          0          0          0                     0.102s     0.003s
           nextcloud.oc_circles_token          0          0          0                     0.123s     0.024s
     nextcloud.oc_collres_accesscache          0          0          0                     0.128s     0.003s
     nextcloud.oc_collres_collections          0          0          0                     0.107s     0.004s
       nextcloud.oc_collres_resources          0          0          0                     0.194s     0.005s
   nextcloud.oc_comments_read_markers          0          1          1     0.0 kB          0.086s     0.016s
             nextcloud.oc_dav_absence          0          0          0                     0.116s     0.006s
           nextcloud.oc_dav_cal_proxy          0          0          0                     0.130s     0.015s
              nextcloud.oc_dav_shares          0          0          0                     0.110s     0.012s
              nextcloud.oc_directlink          0          0          0                     0.113s     0.004s
                 nextcloud.oc_ex_apps          0          0          0                     0.127s     0.010s
         nextcloud.oc_ex_apps_daemons          0          0          0                     0.113s     0.013s
          nextcloud.oc_ex_apps_routes          0          0          0                     0.135s     0.005s
       nextcloud.oc_ex_event_handlers          0          0          0                     0.129s     0.004s
         nextcloud.oc_ex_occ_commands          0          0          0                     0.181s     0.010s
       nextcloud.oc_ex_settings_forms          0          0          0                     0.107s     0.012s
       nextcloud.oc_ex_speech_to_text          0          0          0                     0.170s     0.013s
     nextcloud.oc_ex_speech_to_text_q          0          0          0                     0.115s     0.013s
      nextcloud.oc_ex_task_processing          0          0          0                     0.187s     0.014s
      nextcloud.oc_ex_text_processing          0          0          0                     0.118s     0.012s
    nextcloud.oc_ex_text_processing_q          0          0          0                     0.232s     0.013s
          nextcloud.oc_ex_translation          0          0          0                     0.124s     0.013s
        nextcloud.oc_ex_translation_q          0          0          0                     0.210s     0.014s
     nextcloud.oc_ex_ui_files_actions          0          0          0                     0.109s     0.012s
           nextcloud.oc_ex_ui_scripts          0          0          0                     0.185s     0.016s
            nextcloud.oc_ex_ui_states          0          0          0                     0.103s     0.016s
            nextcloud.oc_ex_ui_styles          0          0          0                     0.186s     0.015s
          nextcloud.oc_ex_ui_top_menu          0          0          0                     0.101s     0.010s
      nextcloud.oc_federated_reshares          0          0          0                     0.207s     0.010s
         nextcloud.oc_files_reminders          0          0          0                     0.113s     0.011s
         nextcloud.oc_flow_operations          0          1          1     0.1 kB          0.206s     0.009s
   nextcloud.oc_flow_operations_scope          0          1          1     0.0 kB          0.408s     0.012s
                  nextcloud.oc_groups          0          1          1     0.0 kB          0.150s     0.004s
             nextcloud.oc_group_admin          0          0          0                     0.101s     0.003s
              nextcloud.oc_group_user          0          1          1     0.0 kB          0.129s     0.005s
             nextcloud.oc_known_users          0          0          0                     0.104s     0.004s
           nextcloud.oc_login_flow_v2          0          0          0                     0.108s     0.009s
    nextcloud.oc_oauth2_access_tokens          0          0          0                     0.096s     0.004s
          nextcloud.oc_oauth2_clients          0          0          0                     0.159s     0.015s
       nextcloud.oc_open_local_editor          0          0          0                     0.118s     0.005s
           nextcloud.oc_photos_albums          0          0          0                     0.135s     0.014s
   nextcloud.oc_photos_albums_collabs          0          0          0                     0.094s     0.014s
     nextcloud.oc_photos_albums_files          0          0          0                     0.125s     0.004s
          nextcloud.oc_preferences_ex          0          0          0                     0.118s     0.004s
          nextcloud.oc_privacy_admins          0          0          0                     0.132s     0.013s
       nextcloud.oc_ratelimit_entries          0          0          0                     0.107s     0.012s
               nextcloud.oc_reactions          0          0          0                     0.144s     0.004s
    nextcloud.oc_richdocuments_assets          0          0          0                     0.110s     0.005s
    nextcloud.oc_richdocuments_direct          0          0          0                     0.149s     0.006s
  nextcloud.oc_richdocuments_template          0          0          0                     0.098s     0.016s
      nextcloud.oc_richdocuments_wopi          0          1          1     0.1 kB          0.157s     0.004s
       nextcloud.oc_schedulingobjects          0          0          0                     0.094s     0.004s
           nextcloud.oc_shares_limits          0          0          0                     0.114s     0.005s
          nextcloud.oc_share_external          0          0          0                     0.108s     0.004s
    nextcloud.oc_storages_credentials          0          0          0                     0.117s     0.013s
               nextcloud.oc_systemtag          0          0          0                     0.099s     0.005s
         nextcloud.oc_systemtag_group          0          0          0                     0.147s     0.004s
nextcloud.oc_systemtag_object_mapping          0          0          0                     0.115s     0.006s
    nextcloud.oc_taskprocessing_tasks          0          0          0                     0.114s     0.004s
        nextcloud.oc_text2image_tasks          0          0          0                     0.094s     0.019s
    nextcloud.oc_textprocessing_tasks          0          0          0                     0.106s     0.005s
   nextcloud.oc_twofactor_backupcodes          0          0          0                     0.117s     0.005s
     nextcloud.oc_user_transfer_owner          0          0          0                     0.121s     0.004s
               nextcloud.oc_vcategory          0          1          1     0.0 kB          0.197s     0.011s
                nextcloud.oc_webauthn          0          0          0                     0.133s     0.013s
       nextcloud.oc_webhook_listeners          0          0          0                     0.103s     0.012s
-------------------------------------  ---------  ---------  ---------  ---------  --------------  ---------  ---------
              COPY Threads Completion          0          2          2                 25m36.069s
                      Reset Sequences          0        113        113                     4.818s
                  Create Foreign Keys          0          0          0                     0.000s
                     Install Comments          0         21         21                     0.309s
-------------------------------------  ---------  ---------  ---------  ---------  --------------  ---------  ---------
                    Total import time          ✓    6794492    6794492     5.8 GB      25m41.196s