Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DBPTK Enterprise Crash #13

Open
Laurira opened this issue Feb 2, 2021 · 7 comments
Open

DBPTK Enterprise Crash #13

Laurira opened this issue Feb 2, 2021 · 7 comments

Comments

@Laurira
Copy link

Laurira commented Feb 2, 2021

I installed DBPTK Enterprise on Ubuntu. Successfully loaded one siard and used it.
Then tried to load another siard (that worked nicely on DBPTK Desktop) then in the middle of loading there were some error message and after that DBPTK Enterprise is useless.
image

I am not sure where to find logs via Docker so I used command:
sudo docker logs -t --since 10m a4e098dcf0ac
(I think these should be last 10 minute logs)

There are something like:
2021-02-02T08:12:23.088931335Z org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://XXX.YY.ZZ.ZZ:8983/solr/dbv-database-dd08a065-9af7-4ee0-b859-e894ad4500ba: Server error writing document id 23676501 to the index
2021-02-02T08:12:23.088938208Z at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:665)
2021-02-02T08:12:23.088944490Z at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
2021-02-02T08:12:23.088950702Z at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
2021-02-02T08:12:23.088956673Z at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
2021-02-02T08:12:23.088962674Z at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
2021-02-02T08:12:23.088968495Z at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1143)
2021-02-02T08:12:23.088974477Z at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906)
2021-02-02T08:12:23.088980748Z at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838)
2021-02-02T08:12:23.088986730Z at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
2021-02-02T08:12:23.088992701Z at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177)
2021-02-02T08:12:23.088998462Z at com.databasepreservation.common.server.index.DatabaseRowsSolrManager.insertDocument(DatabaseRowsSolrManager.java:380)
2021-02-02T08:12:23.089017257Z at com.databasepreservation.common.server.index.DatabaseRowsSolrManager.addRow(DatabaseRowsSolrManager.java:153)
2021-02-02T08:12:23.089023549Z at com.databasepreservation.modules.viewer.DbvtkExportModule.handleDataRow(DbvtkExportModule.java:127)
2021-02-02T08:12:23.089029640Z at com.databasepreservation.model.modules.filters.IdentityFilter.handleDataRow(IdentityFilter.java:88)
2021-02-02T08:12:23.089035652Z at com.databasepreservation.model.modules.filters.ObservableFilter.handleDataRow(ObservableFilter.java:123)
2021-02-02T08:12:23.089041583Z at com.databasepreservation.modules.siard.in.content.SIARD2ContentImportStrategy.endElement(SIARD2ContentImportStrategy.java:374)
2021-02-02T08:12:23.089048736Z at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
2021-02-02T08:12:23.089054858Z at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.endElement(XMLSchemaValidator.java:859)
2021-02-02T08:12:23.089060789Z at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
2021-02-02T08:12:23.089066790Z at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2967)
2021-02-02T08:12:23.089072832Z at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
2021-02-02T08:12:23.089078703Z at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
2021-02-02T08:12:23.089084704Z at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
2021-02-02T08:12:23.089090705Z at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
2021-02-02T08:12:23.089096486Z at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
2021-02-02T08:12:23.089102427Z at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
2021-02-02T08:12:23.089108208Z at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
2021-02-02T08:12:23.089114009Z at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
2021-02-02T08:12:23.089119920Z at com.databasepreservation.modules.siard.in.content.SIARD2ContentImportStrategy.importContent(SIARD2ContentImportStrategy.java:184)
2021-02-02T08:12:23.089125981Z at com.databasepreservation.modules.siard.in.input.SIARDImportDefault.migrateDatabaseTo(SIARDImportDefault.java:64)
2021-02-02T08:12:23.089131853Z at com.databasepreservation.DatabaseMigration.migrate(DatabaseMigration.java:123)
2021-02-02T08:12:23.089137663Z at com.databasepreservation.common.server.controller.SIARDController.convertSIARDtoSolr(SIARDController.java:697)
2021-02-02T08:12:23.089143615Z at com.databasepreservation.common.server.controller.SIARDController.loadFromLocal(SIARDController.java:660)
2021-02-02T08:12:23.089155186Z at com.databasepreservation.common.api.v1.CollectionResource.createCollection(CollectionResource.java:188)
2021-02-02T08:12:23.089161338Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-02-02T08:12:23.089167780Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-02-02T08:12:23.089173751Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-02-02T08:12:23.089179572Z at java.lang.reflect.Method.invoke(Method.java:498)
2021-02-02T08:12:23.089185293Z at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
2021-02-02T08:12:23.089191405Z at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
2021-02-02T08:12:23.089197376Z at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
2021-02-02T08:12:23.089203347Z at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
2021-02-02T08:12:23.089209408Z at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
2021-02-02T08:12:23.089215440Z at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
2021-02-02T08:12:23.089221491Z at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
2021-02-02T08:12:23.089227412Z at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
2021-02-02T08:12:23.089233253Z at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
2021-02-02T08:12:23.089239154Z at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
2021-02-02T08:12:23.089244875Z at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
2021-02-02T08:12:23.089250676Z at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
2021-02-02T08:12:23.089256427Z at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
2021-02-02T08:12:23.089262138Z at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
2021-02-02T08:12:23.089267858Z at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
2021-02-02T08:12:23.089273649Z at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
2021-02-02T08:12:23.089279360Z at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
2021-02-02T08:12:23.089285141Z at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:392)
2021-02-02T08:12:23.089290811Z at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:385)
2021-02-02T08:12:23.089297484Z at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:560)
2021-02-02T08:12:23.089309066Z at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:501)
2021-02-02T08:12:23.089315087Z at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:438)
2021-02-02T08:12:23.089320848Z at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
2021-02-02T08:12:23.089326829Z at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
2021-02-02T08:12:23.089332630Z at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
2021-02-02T08:12:23.089338591Z at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
2021-02-02T08:12:23.089344392Z at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
2021-02-02T08:12:23.089350233Z at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
2021-02-02T08:12:23.089356084Z at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:114)
2021-02-02T08:12:23.089362035Z at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:104)
2021-02-02T08:12:23.089367937Z at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
2021-02-02T08:12:23.089373757Z at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
2021-02-02T08:12:23.089379598Z at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

So if you want to understand in more detail what happened then you should tell me where to get you more information.

Tomorrow I will clean up this mess and try again, so all logs will be vanished in 24 hours.
@Laurira
Copy link
Author

Laurira commented Feb 2, 2021

I examined this issue further and seems like DBPTK Enterprise is quite chatty... it did create >300GB log file so that it run out of disk space.
I am not sure that this application needs to log so much information, but when it is ok, then close this issue.

@Laurira
Copy link
Author

Laurira commented Feb 2, 2021

And maybe only fix for this issue could be that you create proper error message to the user that "There is no space left on disk"

@Laurira
Copy link
Author

Laurira commented Feb 3, 2021

Second attempt:
image

Now I am sure there is no disk space issue. But error in log is related to heap space
solr_1 | Caused by: java.lang.OutOfMemoryError: Java heap space

@Laurira
Copy link
Author

Laurira commented Feb 5, 2021 •
edited
Loading

My docker gets maximum amount of memory so it can not be docker related RAM issue:
image

But I do not know how much memory is assigned to Java inside the docker.

@hmiguim
Copy link
Member

hmiguim commented Feb 18, 2021

Can you post your docker-compose file or sent it to me to check the available options?

@Laurira
Copy link
Author

Laurira commented Aug 5, 2021 •
edited
Loading

The same problem occurred. This is the docker-compose file
version: "3.2" services: dbptke: image: keeps/dbvtk:latest ports: - 8080:8080 - 5005:5005 environment: - SOLR_ZOOKEEPER_HOSTS=solr:9983 - CAS_ENABLED=true - CAS_INTERNAL_URL=http://cas:8080 - CAS_EXTERNAL_URL=http://192.168.65.64:8181 - DBVTK_INTERNAL_URL=http://dbptke:8080 - DBVTK_EXTERNAL_URL=http://192.168.65.64:8080 - DBPTKE_SHARED_FOLDER=/volumes/siards volumes: # configuration overrides - ./dbptke/config/dbvtk-viewer.properties:/dbvtk/config/dbvtk-viewer.properties:ro # map /volumes/siards to the desired folder depends_on: - solr entrypoint: java -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -jar /app.war solr: image: solr:8.4 ports: - 8983:8983 command: -c # Use cloud mode cas: image: cas:latest build: context: ./image/cas ports: - 8181:8080 volumes: - ./cas/attribute-repository.json:/etc/cas/attribute-repository.json - ./cas/users.txt:/etc/cas/config/users.txt

@Laurira
Copy link
Author

Laurira commented Aug 9, 2021

I tried with another bigger database and got the same error.

solr_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
solr_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
solr_1 | at java.base/java.lang.Thread.run(Unknown Source)
solr_1 | Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
solr_1 | at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
solr_1 | at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
solr_1 | at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1591)
solr_1 | at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
solr_1 | at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:977)
solr_1 | at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:345)
solr_1 | at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:292)
solr_1 | at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:239)
solr_1 | ... 82 more
solr_1 | Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1856833 actual=347a8e6 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/solr/data/dbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2_shard1_replica_n1/data/index/_2i4.cfs") [slice=_2i4.fdt]))
solr_1 | at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
solr_1 | at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526)
solr_1 | at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.checkIntegrity(CompressingStoredFieldsReader.java:664)
solr_1 | at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:574)
solr_1 | at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:229)
solr_1 | at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
solr_1 | at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4463)
solr_1 | at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4057)
solr_1 | at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:201)
solr_1 | at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
solr_1 | at org.apache.lucen...
dbptke_1 | 2021-08-09 07:36:02.682 ERROR 1 --- [nio-8080-exec-2] o.a.s.c.solrj.impl.BaseCloudSolrClient : Request to collection [dbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2] failed due to (500) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.0.3:8983/solr/dbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2: Server error writing document id 3381177 to the index, retry=0 commError=false errorCode=500
dbptke_1 | 2021-08-09 07:36:02.683 WARN 1 --- [nio-8080-exec-2] c.d.c.s.index.DatabaseRowsSolrManager : Could not insert a document batch in collectiondbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2. Last response (if any): null
dbptke_1 |
dbptke_1 | org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.0.3:8983/solr/dbv-database-3d71ac32-908f-444d-be82-7fc7dfd607a2: Server error writing document id 3381177 to the index
dbptke_1 | at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:665)
dbptke_1 | at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
dbptke_1 | at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
dbptke_1 | at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
dbptke_1 | at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
dbptke_1 | at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1143)
dbptke_1 | at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906)
dbptke_1 | at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838)
dbptke_1 | at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
dbptke_1 | at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177)
dbptke_1 | at com.databasepreservation.common.server.index.DatabaseRowsSolrManager.insertDocument(DatabaseRowsSolrManager.java:387)
dbptke_1 | at com.databasepreservation.common.server.index.DatabaseRowsSolrManager.addRow(DatabaseRowsSolrManager.java:160)
dbptke_1 | at com.databasepreservation.modules.viewer.DbvtkExportModule.handleDataRow(DbvtkExportModule.java:134)
dbptke_1 | at com.databasepreservation.model.modules.filters.IdentityFilter.handleDataRow(IdentityFilter.java:88)
dbptke_1 | at com.databasepreservation.model.modules.filters.ObservableFilter.handleDataRow(ObservableFilter.java:123)
dbptke_1 | at com.databasepreservation.modules.siard.in.content.SIARD2ContentImportStrategy.endElement(SIARD2ContentImportStrategy.java:374)
dbptke_1 | at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
dbptke_1 | at org.apache.xerces.impl.xs.XMLSchemaValidator.endElement(Unknown Source)
dbptke_1 | at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
dbptke_1 | at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
dbptke_1 | at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
dbptke_1 | at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
dbptke_1 | at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hmiguim @Laurira