Test returns "Index does not exist exception"

🍱 Features & apps fulltextsearch
1

Hello all,

using NextCloud 14.0.12, Fulltextsearch 1.1.2. After several attempts to get indexing done, the indexing has now completed. With the ‘errors: reset’ parameter, I did a last indexing run, to repeat any failed documents.

But as the count of indexed documents does not really reflect the expected count, I decided to restart indexing from scratch. Following the advice of another thread (do occ fulltextsearch:reset and then occ fulltextsearch:test) I now get the result:

.Testing your current setup:  
Creating mocked content provider. ok  
Testing mocked provider: get indexable documents. (2 items) ok  
Loading search platform. (Elasticsearch) ok  
Testing search platform. ok  
Locking process ok  
Removing test. ok  
Pausing 3 seconds 1 2 3 ok  
Initializing index mapping. ok  
Indexing generated documents. fail  
Error detected, unlocking process ok 
In ProviderIndexes.php line 59:
                                                              
  [OCA\FullTextSearch\Exceptions\IndexDoesNotExistException]

I have seen that occ fulltextsearch:reset deletes the given index (i see it in ElasticSearch’s logs, and a /_cat/indices query confirms it) and that occ fulltextsearch:test creates the index. Can someone explain what the error message means (what is missing or not found here?) and how to get it working again?

Thanks.

2

version is Elasticsearch used ?

(might be related: https://github.com/nextcloud/fulltextsearch_elasticsearch/issues/61)

3

Hi @Cult,

   "version" : {
    "number" : "6.6.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "3bd3e59",
    "build_date" : "2019-03-06T15:16:26.864148Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },

so the linked issue doesn’t seem to be related.

4

Wow, sorry about my syntax :slight_smile:

Issue might still be related to your version. Please downgrade ES to 6.x but < 6.6.x or upgrade your nextcloud.

5

Hi @Cult,

We’re talking about a production system, and the chances to choose versions at will are somewhat limited. Even upgrading Nextcloud to 15.x isn’t in the planning yet. So bear with me if I don’t follow your proposal right now.

Instead of experimenting with different version combinations, would you think that clearing all traces of the current index (in ES and the DB) could make the ‘first indexing’ work again, just as if it was installed for the first time? And could you point me in the right direction how to achieve this?

6

occ fulltextsearch:reset does what you are looking for; resets everything: from the mapping on ES to some data about indexed document in Nextcloud.

As a developer, I am also facing some limitation:

  • NC14 is PHP 5.6.
  • elasticsearch-php v6.x is PHP 7.
  • Elasticsearch 6.6.x started to implement stuff from ES v7.x in order to perform smooth migration from 6.x to 7.x

As you can see on the version matrix, even earlier version of elasticsearch-php v6.x were not compatible with ES 6.6.x

Please note that it seems that once the mapping is done, there is no real issues when using up-to-date version of ES.

Was the ‘first indexing’ done on this exact same version of ElasticSearch, or a older one and upgraded to 6.6.x since then ?

7

Yes we upgraded in between. Our installation suffered from compareWithCurrentIndex can take 9-12 hours per user for 100k PDFs from files_external · Issue #408 · nextcloud/fulltextsearch · GitHub, and when we upgraded FTS to 1.1.2 (optimization backport) there was a newer ES version available. It was all inside the 6.6.x range, so we did not expect fatal incompatibilities.

Initially, yes. But the self-diagnosing module of NC strongly suggests using 7.x, so our installation is already on 7.0

hmm, it didn’t :frowning: that’s why I’m here asking questions. After occ fulltextsearch:reset, the occ fulltextsearch:test returned the error cited above, and occ fulltextsearch:index crawls through all users’ stores and does not do anything.

What can I do besides the reset, to achieve a clean start? I’m tempted to do a full uninstall-delete-install cycle if there are no more official procedures.

8

I might not made myself clear enough. The :reset made what you wanted, index and mapping on ES is gone. Pouf. And the data about anything related to your index within Nextcloud is also deleted.

Now, you upgraded ES to v6.6.x. And after a :reset (mapping is removed from ES) the first :index or :test could not recreate the mapping as you are using ES v6.6.x with an old version of elasticsearch-php (that is supposed to work on PHP 5.6 with Elasticsearch < 6.6.x).

Now, the best solution I can advice (again) would be to upgrade Nextcloud to 16: indexing is really (really) improved

The other solution would be to upgrade the lib Elasticsearch-PHP within the app:

  • navigate to the root of the app fulltextsearch_elasticsearch
  • run the command:
    composer require elasticsearch/elasticsearch:6.7.1

Please note that I can only assume that this will work !

9

As long as I can avoid it, I won’t use development tools on a production machine. Your advice adds to my resistance against experiments.

I’m now aware of the version matrix. Unfortunately, it does not speak about backwards compatibility. When we first created the index, ES was on 6.6.0 (you misunderstood that) and FTS at 1.1.1, bundled with elasticsearch-php 5.3.2. and it worked. On another instance, NC 15, FTS 1.2.1, elasticsearch-php 6.1.0 and ES 6.8.0 and it works too. This matrix is obviously to be taken with a grain of salt.

As the root cause of the exception seems to be found (and a fix is in the works) I currently don’t need no more help on this issue.

10

Welp, I totally missed that; in this thread, I though it was an issue with the mapping creation, not a regression from the index comparison. My bad.

Glad it is fixed, I will release a new version of fulltextsearch for NC14 at the end of this week, or beginning of next week.

11

Wonderful, thanks. I’ll keep you posted about the results.

12

Hi @cult, I could not resist patching my system using https://github.com/nextcloud/fulltextsearch/pull/519 manually. Many thanks for the ‘official’ fix in 1.1.3. In the meantime, ES was updated to 6.8.1 and continues to work as expected.

1 Like