[Nextant] Full-Text Search (old thread)

Cult · September 4, 2016, 5:00pm

  Navigate through your cloud using Solr

0.5.0: Alpha is over, Nextant is now in Beta.

Well, still bugged but at least some neat features are here !

This update resets the current setup of your Nextant.
Meaning that, after upgrading, you will need to visit the administration page to configure your Nextant (again) and index your files (again).

It is recommended to delete/recreate the core from your Solr Servlet before configuring your Nextant.

$ sudo -u solr /opt/solr/bin/solr delete -c nextant
$ sudo /etc/init.d/solr restart
$ sudo -u solr /opt/solr/bin/solr create -c nextant

Note: Indexing requiert a descent amount of ressources.
I am aware that some of you are running nextcloud on single-board computer (Raspberry Pi, oDroid, …)
and I have no way to tell you if Solr is running fine on that kind of equipment.

Nextant

Indexes your files,
Indexes your shared documents and deleted files (trash bin)
If needed, will OCR your stuff.
perform fast and concise full-text search
display results integrated to the Files apps.

Recognized file format:

plain text,
rtf,
pdf (old pdf will requiert Tesseract installed)
jpeg & tiff (will also requiert Tesseract)
html,
openoffice,
microsoft office,
…

Installation

You first need to install a Solr servlet
Download the .zip from the appstore, unzip and place this app in nextcloud/apps/ (or clone the github and build the app yourself)
Enable the app in the app list,
Edit the settings in the administration page.
Extract the current files from your cloud using the ./occ nextant:index command
Have a look to this explanation on how Nextant works
(Optional) Installing Tesseract (Optical Character Recognition (OCR) Engine) will allow Nextant to extract text from image file and old pdf.

Also, more docs are now available on the Wiki:

Cult · September 13, 2016, 2:48pm

v0.5.1

bugfix: search allowed to non-admin
bugfix: warning on deleted file with no extension

v0.5.0

Beta release
minor bugfixes
new Admin UI
new display on search result
search within trash bin
OCR/index jpeg/tiff file (needs Tesseract installed)

earth · September 22, 2016, 3:15am

there is no specific how to do for nginx,
Please guide
thanks

Cult · September 22, 2016, 2:56pm

Unfortunately, I am not able to help on nginx configuration issue. However, as specified in the wiki, it is not mandatory to edit your HTTP server configuration. Nextant will work the same way.

earth · October 5, 2016, 2:58am

getting error #21 while extracting
when my solr add is
http://127.0.0.1:8983/solr/
This error is shown when tyring to test and save from admin(nextcloud)
Please guide

Cult · October 5, 2016, 7:40am

You should not get an error 21 (Http Exception) just after a ping.

Can you retry to Save your configuration or is the issue permanent ?

earth · October 6, 2016, 3:26am

This issue is permanent,
Please check screenshots

support

manches · October 6, 2016, 8:42am

Hi,

I am trying to an ./occ nextant:index but after a while i am getting a timeout from solr
Anyone had this issue?

Extracting files to Solr:
Lueas/files: 13390/28585 [=============>--------------] 46%
[scanning] - Solr memory: 407 MB (%82.9)

[Solarium\Exception\HttpException]
Solr HTTP error: HTTP request failed, Operation timed out after 5000 milliseconds with 0 bytes received

Thanks

Cult · October 6, 2016, 10:25am

@manches: have a look at the last part of https://github.com/daita/nextant/wiki/Extracting-&-Live-Update about increasing memory allocation

Cult · October 6, 2016, 10:39am

@earth: Can you have a look to your Solr logs ? You should not have an error #21 here, as it means you cannot reach the Solr Servlet, and before doing this test we check that you can reach the Solr Servlet.

earth · October 7, 2016, 3:05am

Logs updated

2016-10-07 08:26:23.480 ERROR (qtp30621981-9) [ x:nextant] o.a.s.h.RequestHandlerBase java.nio.file.NoSuchFileException: /var/solr/data/nextant/data/index/write.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:177)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:108)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:83)
at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:331)
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:368)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:279)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:211)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:166)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:335)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:939)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1094)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:720)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:91)
at org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
at org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
at org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

2016-10-07 08:26:23.527 ERROR (qtp30621981-9) [ x:nextant] o.a.s.s.HttpSolrCall null:java.nio.file.NoSuchFileException: /var/solr/data/nextant/data/index/write.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:177)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:108)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:83)
at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:331)
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:368)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:279)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:211)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:166)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:335)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:939)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1094)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:720)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:91)
at org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:126)
at org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:131)
at org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:237)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

2016-10-07 08:26:28.994 ERROR (Finalizer) [ ] o.a.s.u.SolrIndexWriter SolrIndexWriter was not closed prior to finalize(), indicates a bug – POSSIBLE RESOURCE LEAK!!!
2016-10-07 08:26:28.998 ERROR (Finalizer) [ ] o.a.s.u.SolrIndexWriter Error closing IndexWriter
java.lang.NullPointerException
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3371)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3333)
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1117)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1162)
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:142)
at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:195)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)

earth · October 7, 2016, 4:38am

now after re indexing got error 41

manches · October 7, 2016, 5:19am

@Cult i have changed it to 2048 and also i have change the timeout limit to 5000 on the jetty.xml file and still will issues

Extracting files to Solr:
Lueas/files: 20060/28883 [===================>--------] 69%
[scanning] - Solr memory: 731.3 MB (%37.3)

[Solarium\Exception\HttpException]
Solr HTTP error: HTTP request failed, Operation timed out after 5001 milliseconds with 0 bytes received

Cult · October 7, 2016, 10:08am

@earth: This is a really strange issue; you should delete your Solr Core, restart your Solr servlet and create a new Core:

 $ sudo -u solr /opt/solr/bin/solr delete -c nextant
 $ sudo /etc/init.d/solr restart
 $ sudo -u solr /opt/solr/bin/solr create -c nextant

@manches: Can you launch the index with --debug option and send me a PM with the last 20 entries of your nextcloud log file ?

 ./occ nextant:index --debug

manches · October 7, 2016, 11:39am

@Cult i have try with the debug option and same result. I have check the nextcloud.log and no entries. I have change in the nextcloud logging option to ALL and relaunch it. Same result

earth · October 7, 2016, 1:20pm

i have to use ./occ nextant:index --debug --force

logs

“zbxj2P12ENFU6hV99dF3”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for local cache”,“level”:1,“time”:“2016-10-07T18:42:48+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“zbxj2P12ENFU6hV99dF3”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for distributed cache”,“level”:1,“time”:“2016-10-07T18:42:48+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for local cache”,“level”:1,“time”:“2016-10-07T18:45:01+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for distributed cache”,“level”:1,“time”:“2016-10-07T18:45:01+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Federation\SyncJob job with ID 3”,“level”:0,“time”:“2016-10-07T18:45:02+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Federation\SyncJob job with ID 3”,“level”:0,“time”:“2016-10-07T18:45:02+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Files_Trashbin\BackgroundJob\ExpireTrash job with ID 4”,“level”:0,“time”:“2016-10-07T18:45:02+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Files_Trashbin\BackgroundJob\ExpireTrash job with ID 4”,“level”:0,“time”:“2016-10-07T18:45:02+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Files_Versions\BackgroundJob\ExpireVersions job with ID 7”,“level”:0,“time”:“2016-10-07T18:45:03+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Files_Versions\BackgroundJob\ExpireVersions job with ID 7”,“level”:0,“time”:“2016-10-07T18:45:03+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Files_Sharing\DeleteOrphanedSharesJob job with ID 9”,“level”:0,“time”:“2016-10-07T18:45:03+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“DeleteOrphanedSharesJob”,“message”:“0 orphaned share(s) deleted”,“level”:0,“time”:“2016-10-07T18:45:03+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Files_Sharing\DeleteOrphanedSharesJob job with ID 9”,“level”:0,“time”:“2016-10-07T18:45:03+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Files_Sharing\ExpireSharesJob job with ID 10”,“level”:0,“time”:“2016-10-07T18:45:03+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Files_Sharing\ExpireSharesJob job with ID 10”,“level”:0,“time”:“2016-10-07T18:45:03+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Files\BackgroundJob\ScanFiles job with ID 11”,“level”:0,“time”:“2016-10-07T18:45:03+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Files\BackgroundJob\ScanFiles job with ID 11”,“level”:0,“time”:“2016-10-07T18:45:03+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Activity\BackgroundJob\EmailNotification job with ID 5”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Activity\BackgroundJob\EmailNotification job with ID 5”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Activity\BackgroundJob\ExpireActivities job with ID 6”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Activity\BackgroundJob\ExpireActivities job with ID 6”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Files\BackgroundJob\DeleteOrphanedItems job with ID 12”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Files\BackgroundJob\DeleteOrphanedItems job with ID 12”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\Files\BackgroundJob\CleanupFileLocks job with ID 13”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\Files\BackgroundJob\CleanupFileLocks job with ID 13”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OC\Authentication\Token\DefaultTokenCleanupJob job with ID 14”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“no app in context”,“message”:“Invalidating tokens older than 2016-10-06T13:15:04+00:00”,“level”:1,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OC\Authentication\Token\DefaultTokenCleanupJob job with ID 14”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\OwnBackup\Jobs\BackupJob job with ID 15”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\OwnBackup\Jobs\BackupJob job with ID 15”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\UpdateNotification\Notification\BackgroundJob job with ID 2”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\UpdateNotification\Notification\BackgroundJob job with ID 2”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Run OCA\UpdateNotification\ResetTokenBackgroundJob job with ID 19”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“kmjhN22efzLBVGe+htFX”,“remoteAddr”:"",“app”:“cron”,“message”:“Finished OCA\UpdateNotification\ResetTokenBackgroundJob job with ID 19”,“level”:0,“time”:“2016-10-07T18:45:04+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“aFn+g6s+7Nz5R+SABLa3”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for local cache”,“level”:1,“time”:“2016-10-07T18:45:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“aFn+g6s+7Nz5R+SABLa3”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for distributed cache”,“level”:1,“time”:“2016-10-07T18:45:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“wvZiuPHEr/XEOvG3HZGm”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for local cache”,“level”:1,“time”:“2016-10-07T18:46:20+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“wvZiuPHEr/XEOvG3HZGm”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for distributed cache”,“level”:1,“time”:“2016-10-07T18:46:20+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for local cache”,“level”:1,“time”:“2016-10-07T18:46:28+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“cli”,“message”:“Memcache \OC\Memcache\APCu not available for distributed cache”,“level”:1,“time”:“2016-10-07T18:46:28+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“Init Extracting new files for user JN”,“level”:1,“time”:“2016-10-07T18:46:29+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:"(JN) - Init Filesystem",“level”:1,“time”:“2016-10-07T18:46:29+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:"(JN//files) - found root folder",“level”:1,“time”:“2016-10-07T18:46:30+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:"(JN) - found 333 files",“level”:1,“time”:“2016-10-07T18:46:50+05:30”,“method”:"–",“url”:"–",“user”:"–"}

Cult · October 7, 2016, 2:18pm

@earth: {“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:"(JN)
_ - found 333 _
files",“level”:1,“time”:“2016-10-07T18:46:50+05:30”,“method”:"–",“url”:"–",“user”:"–"} is really the last entry in your log ? did you delete/recreate your Solr Core ?

Cult · October 7, 2016, 2:28pm

@manches: it is really odd that you have no entry in your log. Do you have logreader installed ?
When you’re saying you got the same result, does it mean it crashed on the same file during extraction ? You might have more information in the logs of your Solr Servlet like this:

[ x:nextant] o.a.s.u.p.LogUpdateProcessorFactory [nextant] webapp=/solr path=/update/extract params={literal.nextant_mtime=1474160212&json.nl=flat&fmap.p=ignored&fmap.h2=ignored_&fmap.style=ignored_&fmap.h3=ignored_&resource.name=pdf_030.pdf&literal.id=3144&fmap.h1=ignored_&commit=true&fmap.link=ignored_&extractOnly=false&uprefix=nextant_attr_&literal.nextant_path=/test/files/pdfs/010g/013/file.pdf&omitHeader=true&fmap.a=ignored_&fmap.content=text&literal.nextant_owner=test&captureAttr=true&fmap.html=ignored_&fmap.div=ignored_&ignoreTikaException=true&wt=json&fmap.span=ignored_}{add=[3144 (1547436286195793920)],commit=} 0 2372_

Also, v0.5.0 is on its way with few improvements on the indexing

earth · October 7, 2016, 6:40pm

{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“Pinu update failed”,“level”:1,“time”:“2016-10-08T00:03:38+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“Init Updating documents for user SJ”,“level”:1,“time”:“2016-10-08T00:03:38+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“Init filesystem for user SJ”,“level”:1,“time”:“2016-10-08T00:03:39+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:"(SJ) - found 2 files",“level”:1,“time”:“2016-10-08T00:03:39+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“updating Files array (\n 0 => \n array (\n ‘fileid’ => ‘267904’,\n ‘options’ => \n array (\n ),\n ‘path’ => ‘/SJ/files/Documents/About.txt’,\n ),\n)”,“level”:1,“time”:“2016-10-08T00:03:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“updateDocument error #61”,“level”:1,“time”:“2016-10-08T00:03:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“SJ update failed”,“level”:1,“time”:“2016-10-08T00:03:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“updating Files array (\n 0 => \n array (\n ‘fileid’ => ‘267905’,\n ‘options’ => \n array (\n ),\n ‘path’ => ‘/SJ/files/Documents/About.odt’,\n ),\n)”,“level”:1,“time”:“2016-10-08T00:03:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“updateDocument error #61”,“level”:1,“time”:“2016-10-08T00:03:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“SJ update failed”,“level”:1,“time”:“2016-10-08T00:03:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“Init Updating documents for user Sangita”,“level”:1,“time”:“2016-10-08T00:03:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“Init filesystem for user Sangita”,“level”:1,“time”:“2016-10-08T00:03:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:"(Sangita) - found 2 files",“level”:1,“time”:“2016-10-08T00:03:44+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“updating Files array (\n 0 => \n array (\n ‘fileid’ => ‘221934’,\n ‘options’ => \n array (\n ),\n ‘path’ => ‘/Sangita/files/Documents/About.txt’,\n ),\n)”,“level”:1,“time”:“2016-10-08T00:03:49+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“updateDocument error #61”,“level”:1,“time”:“2016-10-08T00:03:50+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“Sangita update failed”,“level”:1,“time”:“2016-10-08T00:03:50+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“updating Files array (\n 0 => \n array (\n ‘fileid’ => ‘221935’,\n ‘options’ => \n array (\n ),\n ‘path’ => ‘/Sangita/files/Documents/About.odt’,\n ),\n)”,“level”:1,“time”:“2016-10-08T00:03:50+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“updateDocument error #61”,“level”:1,“time”:“2016-10-08T00:03:50+05:30”,“method”:"–",“url”:"–",“user”:"–"}
{“reqId”:“CWv53gnjAl60yJBUp0+E”,“remoteAddr”:"",“app”:“nextant”,“message”:“Sangita update failed”,“level”:1,“time”:“2016-10-08T00:03:50+05:30”,“method”:"–",“url”:"–",“user”:"–"}

Cult · October 7, 2016, 7:37pm

Well, i am confuse…

As much as I want to fix any bug, your first post was about an issue on the setup while testing the extract, then a log showing the extract (step 1 of the indexing) but with no errors and the last log is about some update that fail (step 2 of the indexing).

So it seems that you manage to setup nextant, but did you reset the core as I asked ? Give me some details about the memory allocated to Solr, the number of files on your cloud, are you using the last version (0.4.2) ? how did you get that error ?