Convert Postgres to Maria fails with duplicate entries in oc_filecache

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 29.0.5): 27.1.11.3
Operating system and version (eg, Ubuntu 24.04): docker compose stack
Apache or nginx version (eg, Apache 2.4.25): 2.4.61
PHP version (eg, 8.3): 4.2.22

The issue you are facing:

I’m trying to convert from Postgres to MariaDB:

 php occ db:convert-type -vvv --no-interaction --clear-schema --password "${MYSQL_PASSWORD}" mysql "${MYSQL_DB_USER}" "${MYSQL_HOST}-maria" "${MYSQL_DB_NAME}"

which fails with the following output:

 - oc_filecache                                                                                                                                                               
 540/665 [======================>-----]  81%  1 sec/1 sec                                                                                                                     
In ExceptionConverter.php line 64:                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                            
  [Doctrine\DBAL\Exception\UniqueConstraintViolationException (1062)]                                                                                                                                                                                                                                                                                       
  An exception occurred while executing a query: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '2-d41d8cd98f00b204e9800998ecf8427e' for key 'fs_storage_path_hash'                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

Looking into the table this is not the only duplicate key. Showing the top ones, there are much more:

ncfvbs=# SELECT fileid, storage, path, name, path_hash, COUNT(*) OVER (PARTITION BY path_hash) as count FROM oc_filecache WHERE path_hash IN (     SELECT path_hash     FROM oc_filecache     GROUP BY path_hash     HAVING COUNT(*) > 1 ) ORDER BY count DESC, path_hash;                                                                                  
 fileid | storage |                             path                             |              name               |            path_hash             | count                                                                                                                                                                                               
--------+---------+--------------------------------------------------------------+---------------------------------+----------------------------------+-------                                                                                                                                                                                              
    533 |       6 |                                                              |                                 | d41d8cd98f00b204e9800998ecf8427e |     9                                                                                                                                                                                               
    253 |       3 |                                                              |                                 | d41d8cd98f00b204e9800998ecf8427e |     9                                                                                                                                                                                               
    551 |       8 |                                                              |                                 | d41d8cd98f00b204e9800998ecf8427e |     9                                                                                                                                                                                               
    527 |       5 |                                                              |                                 | d41d8cd98f00b204e9800998ecf8427e |     9                                                                                                                                                                                               
    894 |       2 |                                                              |                                 | d41d8cd98f00b204e9800998ecf8427e |     9                                                                                                                                                                                               
    539 |       7 |                                                              |                                 | d41d8cd98f00b204e9800998ecf8427e |     9                                                                                                                                                                                               
      1 |       1 |                                                              |                                 | d41d8cd98f00b204e9800998ecf8427e |     9                                                                                                                                                                                               
     38 |       2 |                                                              |                                 | d41d8cd98f00b204e9800998ecf8427e |     9                                                                                                                                                                                               
    383 |       4 |                                                              |                                 | d41d8cd98f00b204e9800998ecf8427e |     9                                                                                                                                                                                               
     13 |       1 | files/Vorlagen/Meeting notes.md                              | Meeting notes.md                | 08e272567d63a2d063c00109556a430a |     4                                                                                                                                                                                               
    270 |       3 | files/Vorlagen/Meeting notes.md                              | Meeting notes.md                | 08e272567d63a2d063c00109556a430a |     4                                                                                                                                                                                               
    568 |       8 | files/Vorlagen/Meeting notes.md                              | Meeting notes.md                | 08e272567d63a2d063c00109556a430a |     4                                                                                                                                                                                               
    400 |       4 | files/Vorlagen/Meeting notes.md                              | Meeting notes.md                | 08e272567d63a2d063c00109556a430a |     4                                                                                                                                                                                               
      5 |       1 | files/Documents                                              | Documents                       | 0ad78ba05b6961d92f7970b2b3922eca |     4                                                                                                                                                                                               
    258 |       3 | files/Documents                                              | Documents                       | 0ad78ba05b6961d92f7970b2b3922eca |     4                                                                                                                                                                                               
    556 |       8 | files/Documents                                              | Documents                       | 0ad78ba05b6961d92f7970b2b3922eca |     4                                                                                                                                                                                               
    388 |       4 | files/Documents                                              | Documents                       | 0ad78ba05b6961d92f7970b2b3922eca |     4                                                                                                                                                                                               
     34 |       1 | files/Photos/Library.jpg                                     | Library.jpg                     | 0b785d02a19fc00979f82f6b54a05805 |     4                                                                                                                                                                                               
    427 |       4 | files/Photos/Library.jpg                                     | Library.jpg                     | 0b785d02a19fc00979f82f6b54a05805 |     4                                                                                                                                                                                               
    595 |       8 | files/Photos/Library.jpg                                     | Library.jpg                     | 0b785d02a19fc00979f82f6b54a05805 |     4                                                                                                                                                                                               
    297 |       3 | files/Photos/Library.jpg                                     | Library.jpg                     | 0b785d02a19fc00979f82f6b54a05805 |     4                                                                                                                                                                                               

Does anyone have an idea how to resolve this?

The storage column seems to give a clue here. Anyone an idea, what the semantics are here?

I ran all the files:cleanup/scan commands, but this did not change anything.

1 Like

Just updated to version 28, the same problem still occurs. (Later versions disable the db:convert-type command.)

Digged a bit into the storage column and learned that it refers to the oc_storages table.

So it looks like the collisions stem from files being present by default on first login. So it would actually be okay to have a hash present several times. But why then is this a constraint on mysql?

1 Like

Answering my own question. :slight_smile:

I finally succeeded doing the migration. I noticed that the duplicate was due to the concatenation of storage and path_hash which reduced the conflicts to three rows. These were explicable, as I hade some issues with the data folder earlier which I resolved by fiddling around manually. I could just remove the entries with the lower fileid and everything went smooth.

1 Like

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.