Fulltextsearch 22 is broken

🍱 Features & apps fulltextsearch
1

Hi, after updating fulltextsearch to version 22 it does not work anymore:

root@nuc:/var/www/html/nextcloud# sudo -u www-data php occ fulltextsearch:test

.Testing your current setup:
Creating mocked content provider. ok
Testing mocked provider: get indexable documents. (2 items) ok
Loading search platform. (Elasticsearch) ok
Testing search platform. ok
Locking process ok
Removing test. ok
Pausing 3 seconds 1 2 3 ok
Initializing index mapping. ok
Indexing generated documents. ok
Pausing 3 seconds 1 2 3 ok
Retreiving content from a big index (license). (size: 32386) ok
Comparing document with source. ok
Searching basic keywords:

  • ‘test’ (result: 1, expected: [“simple”]) ok
  • ‘document is a simple test’ (result: 2, expected: [“simple”,“license”]) ok
  • ‘“document is a test”’ (result: 0, expected: ) ok
  • ‘“document is a simple test”’ (result: 1, expected: [“simple”]) ok
  • ‘document is a simple -test’ (result: 1, expected: [“license”]) ok
  • ‘document is a simple +test’ (result: 1, expected: [“simple”]) ok
  • ‘-document is a simple test’ (result: 0, expected: ) ok
  • ‘document is a simple +test +testing’ (result: 1, expected: [“simple”]) ok
  • ‘document is a simple +test -testing’ (result: 0, expected: ) ok
  • ‘document is a +simple -test -testing’ (result: 0, expected: ) ok
  • ‘+document is a simple -test -testing’ (result: 1, expected: [“license”]) ok
  • ‘document is a +simple -license +testing’ (result: 1, expected: [“simple”]) ok
    Updating documents access. ok
    Pausing 3 seconds 1 2 3 ok
    Searching with group access rights:
  • ‘license’ - - (result: 0, expected: ) ok
  • ‘license’ - [“group_1”] - (result: 0, expected: [“license”]) fail
    Error detected, unlocking process ok
    In Test.php line 675:

Unexpected SearchResult: {“provider”:{“id”:“test_provider”,“name”:“Test Provider”},“platform”:{“id”:“elastic_search”,“name”:“Elasticsearch”},“documents”:,“info”:,“meta”:{“timedOut”:false,“time”:10,“count”:0,“total”:0,"maxScore
":0}}

fulltextsearch:test [–output [OUTPUT]] [-j|–json] [-d|–platform_delay PLATFORM_DELAY]

Is this a known error? I already uninstalled the whole stuff, but it did not solve the problem.

2

do you have more info from nextcloud.log ?

3

Sure, attached the error.log File

"reqId":"AiVYk5MY8pGrm0n3bk4v","level":0,"time":"2021-07-21T20:14:20+00:00","remoteAddr":"","user":"--","app":"fulltextsearch_elasticsearch","method":"","url":"--","message":"Filled SearchResult Model: {\"provider\":{\"id\":\"test_provider\",\"name\":\"Test Provider\"},\"platform\":{\"id\":\"elastic_search\",\"name\":\"Elasticsearch\"},\"documents\":[{\"id\":\"simple\",\"providerId\":\"test_provider\",\"access\":{\"ownerId\":\"\",\"viewerId\":\"user1\",\"users\":[],\"groups\":[],\"circles\":[],\"links\":[]},\"modifiedTime\":0,\"title\":\"\",\"link\":\"\",\"index\":null,\"source\":\"\",\"info\":[],\"hash\":\"dc5617141771b9472dcc0739960bf07a\",\"contentSize\":0,\"tags\":[],\"metatags\":[],\"subtags\":[],\"more\":[],\"excerpts\":[{\"source\":\"content\",\"excerpt\":\"testing document is a simple test\"}],\"score\":\"0\"}],\"info\":[],\"meta\":{\"timedOut\":false,\"time\":20,\"count\":1,\"total\":1,\"maxScore\":20}}","userAgent":"--","version":"22.0.0.11"}
{"reqId":"AiVYk5MY8pGrm0n3bk4v","level":0,"time":"2021-07-21T20:14:23+00:00","remoteAddr":"","user":"--","app":"fulltextsearch_elasticsearch","method":"","url":"--","message":"New Search Request; SearchResult Model: {\"provider\":{\"id\":\"test_provider\",\"name\":\"Test Provider\"},\"platform\":{\"id\":\"elastic_search\",\"name\":\"Elasticsearch\"},\"documents\":[],\"info\":[],\"meta\":{\"timedOut\":false,\"time\":0,\"count\":0,\"total\":0,\"maxScore\":0}}","userAgent":"--","version":"22.0.0.11"}
{"reqId":"AiVYk5MY8pGrm0n3bk4v","level":0,"time":"2021-07-21T20:14:23+00:00","remoteAddr":"","user":"--","app":"fulltextsearch_elasticsearch","method":"","url":"--","message":"Searching ES: {\"index\":\"cloud1\",\"size\":10,\"from\":0,\"body\":{\"query\":{\"bool\":{\"must\":{\"bool\":{\"should\":[{\"match_phrase_prefix\":{\"content\":\"license\"}},{\"match_phrase_prefix\":{\"title\":\"license\"}}]}},\"filter\":[{\"bool\":{\"must\":{\"term\":{\"provider\":\"test_provider\"}}}},{\"bool\":{\"should\":[{\"term\":{\"owner\":\"notuser\"}},{\"term\":{\"users\":\"notuser\"}},{\"term\":{\"users\":\"__all\"}}]}},{\"bool\":{\"should\":[]}},{\"bool\":{\"must\":[]}},{\"bool\":{\"must\":[]}}]}},\"highlight\":{\"fields\":{\"content\":{}},\"pre_tags\":[\"\"],\"post_tags\":[\"\"]}}}","userAgent":"--","version":"22.0.0.11"}
{"reqId":"AiVYk5MY8pGrm0n3bk4v","level":0,"time":"2021-07-21T20:14:23+00:00","remoteAddr":"","user":"--","app":"fulltextsearch_elasticsearch","method":"","url":"--","message":"Result from ES: {\"took\":9,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\":5,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null,\"hits\":[]}}","userAgent":"--","version":"22.0.0.11"}
{"reqId":"AiVYk5MY8pGrm0n3bk4v","level":0,"time":"2021-07-21T20:14:23+00:00","remoteAddr":"","user":"--","app":"fulltextsearch_elasticsearch","method":"","url":"--","message":"Filled SearchResult Model: {\"provider\":{\"id\":\"test_provider\",\"name\":\"Test Provider\"},\"platform\":{\"id\":\"elastic_search\",\"name\":\"Elasticsearch\"},\"documents\":[],\"info\":[],\"meta\":{\"timedOut\":false,\"time\":9,\"count\":0,\"total\":0,\"maxScore\":0}}","userAgent":"--","version":"22.0.0.11"}
{"reqId":"AiVYk5MY8pGrm0n3bk4v","level":0,"time":"2021-07-21T20:14:23+00:00","remoteAddr":"","user":"--","app":"fulltextsearch_elasticsearch","method":"","url":"--","message":"New Search Request; SearchResult Model: {\"provider\":{\"id\":\"test_provider\",\"name\":\"Test Provider\"},\"platform\":{\"id\":\"elastic_search\",\"name\":\"Elasticsearch\"},\"documents\":[],\"info\":[],\"meta\":{\"timedOut\":false,\"time\":0,\"count\":0,\"total\":0,\"maxScore\":0}}","userAgent":"--","version":"22.0.0.11"}
{"reqId":"AiVYk5MY8pGrm0n3bk4v","level":0,"time":"2021-07-21T20:14:23+00:00","remoteAddr":"","user":"--","app":"fulltextsearch_elasticsearch","method":"","url":"--","message":"Searching ES: {\"index\":\"cloud1\",\"size\":10,\"from\":0,\"body\":{\"query\":{\"bool\":{\"must\":{\"bool\":{\"should\":[{\"match_phrase_prefix\":{\"content\":\"license\"}},{\"match_phrase_prefix\":{\"title\":\"license\"}}]}},\"filter\":[{\"bool\":{\"must\":{\"term\":{\"provider\":\"test_provider\"}}}},{\"bool\":{\"should\":[{\"term\":{\"owner\":\"notuser\"}},{\"term\":{\"users\":\"notuser\"}},{\"term\":{\"users\":\"__all\"}},{\"term\":{\"groups\":\"group_1\"}}]}},{\"bool\":{\"should\":[]}},{\"bool\":{\"must\":[]}},{\"bool\":{\"must\":[]}}]}},\"highlight\":{\"fields\":{\"content\":{}},\"pre_tags\":[\"\"],\"post_tags\":[\"\"]}}}","userAgent":"--","version":"22.0.0.11"}
{"reqId":"AiVYk5MY8pGrm0n3bk4v","level":0,"time":"2021-07-21T20:14:23+00:00","remoteAddr":"","user":"--","app":"fulltextsearch_elasticsearch","method":"","url":"--","message":"Result from ES: {\"took\":4,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\":5,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null,\"hits\":[]}}","userAgent":"--","version":"22.0.0.11"}
{"reqId":"AiVYk5MY8pGrm0n3bk4v","level":0,"time":"2021-07-21T20:14:23+00:00","remoteAddr":"","user":"--","app":"fulltextsearch_elasticsearch","method":"","url":"--","message":"Filled SearchResult Model: {\"provider\":{\"id\":\"test_provider\",\"name\":\"Test Provider\"},\"platform\":{\"id\":\"elastic_search\",\"name\":\"Elasticsearch\"},\"documents\":[],\"info\":[],\"meta\":{\"timedOut\":false,\"time\":4,\"count\":0,\"total\":0,\"maxScore\":0}}","userAgent":"--","version":"22.0.0.11"}
4

did you have Group Folder Apps?

5

Yes, Group Folders 9.0.2 which I disabled without any effect.
Here are my apps:
Enabled:

  • accessibility: 1.7.0
  • activity: 2.15.0
  • admin_notifications: 1.0.2
  • bruteforcesettings: 2.2.0
  • calendar: 2.3.1
  • circles: 22.0.0
  • cloud_federation_api: 1.4.0
  • comments: 1.11.0
  • contacts: 4.0.0
  • contactsinteraction: 1.2.0
  • dav: 1.18.0
  • deck: 1.5.0
  • federatedfilesharing: 1.11.0
  • federation: 1.11.0
  • files: 1.16.0
  • files_accesscontrol: 1.12.0
  • files_downloadactivity: 1.11.1
  • files_external: 1.12.1
  • files_fulltextsearch: 22.0.0
  • files_pdfviewer: 2.3.0
  • files_retention: 1.11.0
  • files_rightclick: 1.1.0
  • files_sharing: 1.13.2
  • files_trackdownloads: 1.10.0
  • files_trashbin: 1.11.0
  • files_versions: 1.14.0
  • files_videoplayer: 1.11.0
  • firstrunwizard: 2.11.0
  • forms: 2.2.4
  • fulltextsearch: 22.0.0
  • fulltextsearch_elasticsearch: 22.0.0
  • groupfolders: 9.0.2
  • logreader: 2.7.0
  • lookup_server_connector: 1.9.0
  • mail: 1.10.1
  • nextcloud_announcements: 1.11.0
  • notes: 4.1.0
  • notifications: 2.10.1
  • notify_push: 0.2.1
  • oauth2: 1.9.0
  • password_policy: 1.12.0
  • photos: 1.4.0
  • polls: 3.0.0
  • privacy: 1.6.0
  • provisioning_api: 1.11.0
  • quota_warning: 1.11.0
  • recommendations: 1.1.0
  • richdocuments: 4.2.0
  • richdocumentscode: 6.4.1004
  • serverinfo: 1.12.0
  • settings: 1.3.0
  • sharebymail: 1.11.0
  • support: 1.5.0
  • survey_client: 1.10.0
  • systemtags: 1.11.0
  • tasks: 0.14.0
  • text: 3.3.0
  • theming: 1.12.0
  • twofactor_backupcodes: 1.10.1
  • updatenotification: 1.11.0
  • user_status: 1.1.1
  • viewer: 1.6.0
  • weather_status: 1.1.0
  • workflowengine: 2.3.0
    Disabled:
  • admin_audit
  • dashboard
  • encryption
  • passwords
  • user_ldap
6

If you have a lot of files, and want to do some testing before reset, can you assign a new index name in your settings and run the testing command again.

If you do not have that many files, try reseting the index first and run the testing:

$ ./occ fulltextsearch:reset
$ ./occ fulltextsearch:test
7

Tried with new index:

sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:test

.Testing your current setup:
Creating mocked content provider. ok
Testing mocked provider: get indexable documents. (2 items) ok
Loading search platform. (Elasticsearch) ok
Testing search platform. ok
Locking process ok
Removing test. ok
Pausing 3 seconds 1 2 3 ok
Initializing index mapping. ok
Indexing generated documents. ok
Pausing 3 seconds 1 2 3 ok
Retreiving content from a big index (license). (size: 32386) ok
Comparing document with source. ok
Searching basic keywords:
 - 'test' (result: 1, expected: ["simple"]) ok
 - 'document is a simple test' (result: 2, expected: ["simple","license"]) ok
 - '"document is a test"' (result: 0, expected: []) ok
 - '"document is a simple test"' (result: 1, expected: ["simple"]) ok
 - 'document is a simple -test' (result: 1, expected: ["license"]) ok
 - 'document is a simple +test' (result: 1, expected: ["simple"]) ok
 - '-document is a simple test' (result: 0, expected: []) ok
 - 'document is a simple +test +testing' (result: 1, expected: ["simple"]) ok
 - 'document is a simple +test -testing' (result: 0, expected: []) ok
 - 'document is a +simple -test -testing' (result: 0, expected: []) ok
 - '+document is a simple -test -testing' (result: 1, expected: ["license"]) ok                                                                                       
 - 'document is a +simple -license +testing' (result: 1, expected: ["simple"]) o                                                                                      k
Updating documents access. ok
Pausing 3 seconds 1 2 3 ok
Searching with group access rights:
 - 'license' - [] -  (result: 0, expected: []) ok
 - 'license' - ["group_1"] -  (result: 0, expected: ["license"]) fail
Error detected, unlocking process ok
In Test.php line 675:

  Unexpected SearchResult: {"provider":{"id":"test_provider","name":"Test Pro
  vider"},"platform":{"id":"elastic_search","name":"Elasticsearch"},"document
  s":[],"info":[],"meta":{"timedOut":false,"time":2,"count":0,"total":0,"maxS
  core":0}}


fulltextsearch:test [--output [OUTPUT]] [-j|--json] [-d|--platform_delay PLATFORM_DELAY]

So no success

8

Version of your elasticsearch ?

9

Version: 6.8.17, Build: default/deb/206f6a2/2021-07-01T18:51:20.391869Z, JVM: 1.8.0_232

10

With Fulltextsearch <22.0 everything worked …

11

You will need to upgrade your ES to 7.x

12

You are my hero of the day - after upgrading to elasticsearch 7 connection works again, BUT:
indexing aborts

sudo -u www-data php /var/www/html/nextcloud/occ fulltextsearch:index

## x:first result ## c/v:prec/next result ## b:last result
## f:first error ## h/j:prec/next error ## d:delete error ## l:last error
## q:quit ## p:pause
An unhandled exception has been thrown:
TypeError: Return value of OCA\Files_FullTextSearch\Model\MountPoint::isGlobal() must be of the type bool, null returned in /mnt/EXTUSB/www/html/nextcloud/apps/files_fulltextsearch/lib/Model/MountPoint.php:103
Stack trace:
#0 /mnt/EXTUSB/www/html/nextcloud/apps/files_fulltextsearch/lib/Model/MountPoint.php(175): OCA\Files_FullTextSearch\Model\MountPoint->isGlobal()
#1 [internal function]: OCA\Files_FullTextSearch\Model\MountPoint->jsonSerialize()
#2 /mnt/EXTUSB/www/html/nextcloud/apps/fulltextsearch/vendor/artificial-owl/my-small-php-tools/lib/Traits/Nextcloud/nc22/TNC22Logger.php(192): json_encode()
#3 /mnt/EXTUSB/www/html/nextcloud/apps/fulltextsearch/vendor/artificial-owl/my-small-php-tools/lib/Traits/Nextcloud/nc22/TNC22Logger.php(177): OCA\Files_FullTextSearch\Service\GroupFoldersService->log()
#4 /mnt/EXTUSB/www/html/nextcloud/apps/files_fulltextsearch/lib/Service/GroupFoldersService.php(133): OCA\Files_FullTextSearch\Service\GroupFoldersService->debug()
#5 /mnt/EXTUSB/www/html/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(379): OCA\Files_FullTextSearch\Service\GroupFoldersService->initGroupSharesForUser()
#6 /mnt/EXTUSB/www/html/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(202): OCA\Files_FullTextSearch\Service\FilesService->initFileSystems()
#7 /mnt/EXTUSB/www/html/nextcloud/apps/files_fulltextsearch/lib/Provider/FilesProvider.php(249): OCA\Files_FullTextSearch\Service\FilesService->getChunksFromUser()
#8 /mnt/EXTUSB/www/html/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(182): OCA\Files_FullTextSearch\Provider\FilesProvider->generateChunks()
#9 /mnt/EXTUSB/www/html/nextcloud/apps/fulltextsearch/lib/Command/Index.php(410): OCA\FullTextSearch\Service\IndexService->indexProviderContentFromUser()
#10 /mnt/EXTUSB/www/html/nextcloud/apps/fulltextsearch/lib/Command/Index.php(273): OCA\FullTextSearch\Command\Index->indexProvider()
#11 /mnt/EXTUSB/www/html/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OCA\FullTextSearch\Command\Index->execute()
#12 /mnt/EXTUSB/www/html/nextcloud/core/Command/Base.php(168): Symfony\Component\Console\Command\Command->run()
#13 /mnt/EXTUSB/www/html/nextcloud/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run()
#14 /mnt/EXTUSB/www/html/nextcloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand()
#15 /mnt/EXTUSB/www/html/nextcloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#16 /mnt/EXTUSB/www/html/nextcloud/lib/private/Console/Application.php(209): Symfony\Component\Console\Application->run()
#17 /mnt/EXTUSB/www/html/nextcloud/console.php(99): OC\Console\Application->run()
#18 /mnt/EXTUSB/www/html/nextcloud/occ(11): require_once('/mnt/EXTUSB/www...')
#19 {main}root@nuc:/home/benjamin#
13

So, here my setup:
Nextcloud running on NUC, datadir is stored on NFS mount.
I also use external Storeage - SMB/CIFS where videos are stored.
I also use Groupfolder to share two folder on my home with other members.

1 Like
14

Ich can confirm the error above with NC 22.
Everything was ok with ES 7 and NC 21.03

Log:
root@kronos:/var/www/nextcloud# sudo -u www-data php occ fulltextsearch:stop
stopping all running indexes
root@kronos:/var/www/nextcloud# sudo -u www-data php occ fulltextsearch:reset
reset.
root@kronos:/var/www/nextcloud# sudo -u www-data php occ fulltextsearch:index

Options:
Memory: 49 MB
┌─ Indexing ────
│ Action: generateIndexFiles
│ Provider: Files Account: Anna
│ Document: 3
│ Info:
│ Title:
│ Content size: 0
│ Chunk: 0/0
│ Progress: 0/0
└──
┌─ Results ────
│ Result: 20/20
│ Index: deck:3
│ Status: ok
│ Message: {"_index":“nextcloud_index”,"_type":"_doc","_id":“deck:3”,"_version":1,“result”:“created”,"_shards":{“total”:2,“successful”:1,“failed”:0},"_seq_no":19,"_primary_term":1}
│
│
└──
┌─ Errors ────
│ Error: 0/0
│ Index:
│ Exception:
│ Message:
│
│
└──

x:first result ## c/v:prec/next result ## b:last result

f:first error ## h/j:prec/next error ## d:delete error ## l:last error

q:quit ## p:pause

An unhandled exception has been thrown:
TypeError: Return value of OCA\Files_FullTextSearch\Model\MountPoint::isGlobal() must be of the type bool, null returned in /var/www/nextcloud/apps/files_fulltextsearch/lib/Model/MountPoint.php:103
Stack trace:
#0 /var/www/nextcloud/apps/files_fulltextsearch/lib/Model/MountPoint.php(175): OCA\Files_FullTextSearch\Model\MountPoint->isGlobal()
#1 [internal function]: OCA\Files_FullTextSearch\Model\MountPoint->jsonSerialize()
#2 /var/www/nextcloud/apps/fulltextsearch/vendor/artificial-owl/my-small-php-tools/lib/Traits/Nextcloud/nc22/TNC22Logger.php(192): json_encode()
#3 /var/www/nextcloud/apps/fulltextsearch/vendor/artificial-owl/my-small-php-tools/lib/Traits/Nextcloud/nc22/TNC22Logger.php(177): OCA\Files_FullTextSearch\Service\GroupFoldersService->log()
#4 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/GroupFoldersService.php(133): OCA\Files_FullTextSearch\Service\GroupFoldersService->debug()
#5 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(379): OCA\Files_FullTextSearch\Service\GroupFoldersService->initGroupSharesForUser()
#6 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(202): OCA\Files_FullTextSearch\Service\FilesService->initFileSystems()
#7 /var/www/nextcloud/apps/files_fulltextsearch/lib/Provider/FilesProvider.php(249): OCA\Files_FullTextSearch\Service\FilesService->getChunksFromUser()
#8 /var/www/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(182): OCA\Files_FullTextSearch\Provider\FilesProvider->generateChunks()
#9 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Index.php(410): OCA\FullTextSearch\Service\IndexService->indexProviderContentFromUser()
#10 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Index.php(273): OCA\FullTextSearch\Command\Index->indexProvider()
#11 /var/www/nextcloud/apps/bookmarks/vendor/symfony/console/Command/Command.php(255): OCA\FullTextSearch\Command\Index->execute()
#12 /var/www/nextcloud/core/Command/Base.php(168): Symfony\Component\Console\Command\Command->run()
#13 /var/www/nextcloud/apps/bookmarks/vendor/symfony/console/Application.php(1009): OC\Core\Command\Base->run()
#14 /var/www/nextcloud/apps/bookmarks/vendor/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand()
#15 /var/www/nextcloud/apps/bookmarks/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#16 /var/www/nextcloud/lib/private/Console/Application.php(209): Symfony\Component\Console\Application->run()
#17 /var/www/nextcloud/console.php(99): OC\Console\Application->run()
#18 /var/www/nextcloud/occ(11): require_once(’/var/www/nextcl…’)

1 Like
15

So who can help us here?

16

Not sure if this is related to your issue, but fulltextsearch doesn’t seem to work if the circles app is enabled.

17

Last version of files_fulltextsearch should fix this issue. Please confirm.

2 Likes
18

Dear @Cult thank you very much indexing is now running again.

1 Like
19

@Cult - is it necessary to update elasticsearch to 7.x for NC v22?

*I’m currently running NC 21 with ES 6.8.14

20

Yes I do not think the lib from elasticsearch will work on older version

1 Like