Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v7.4 Is creating too many Index shards #4461

Open
stroomdev10 opened this issue Sep 18, 2024 · 2 comments
Open

v7.4 Is creating too many Index shards #4461

stroomdev10 opened this issue Sep 18, 2024 · 2 comments
Assignees
@stroomdev66
Labels
bug f:pipeline-processing Issues relating to pipelines and processing p:high High priority
Milestone
v7.5

Comments

@stroomdev10
Copy link
Member

I have an new index with 1 shard per partition, sending data to that index has created 2 or 3 Index shards per partition per node.
@stroomdev10 stroomdev10 added bug p:high High priority f:pipeline-processing Issues relating to pipelines and processing labels Sep 18, 2024
@stroomdev10 stroomdev10 added this to the v7.5 milestone Sep 18, 2024
@stroomdev10
Copy link
Member Author 

Error opening index shard 59556 
stroom.index.shared.IndexException: Unable to set index shard status to OPENING
	at stroom.index.impl.IndexShardWriterCacheImpl.lambda$openWriter$9(IndexShardWriterCacheImpl.java:150)
	at stroom.security.impl.SecurityContextImpl.asUserResult(SecurityContextImpl.java:338)
	at stroom.security.impl.SecurityContextImpl.asProcessingUserResult(SecurityContextImpl.java:369)
	at stroom.index.impl.IndexShardWriterCacheImpl.openWriter(IndexShardWriterCacheImpl.java:135)
	at stroom.index.impl.IndexShardWriterCacheImpl.lambda$open$2(IndexShardWriterCacheImpl.java:128)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
	at stroom.index.impl.IndexShardWriterCacheImpl.open(IndexShardWriterCacheImpl.java:124)
	at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$2(LocalCache.java:167)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2688)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669)
	at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
	at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
	at stroom.cache.impl.AbstractStroomCache.get(AbstractStroomCache.java:170)
	at stroom.index.impl.IndexShardWriterCacheImpl.getOrOpenWriter(IndexShardWriterCacheImpl.java:116)
	at stroom.index.impl.ActiveShardsCacheImpl$ActiveShards.addDocument(ActiveShardsCacheImpl.java:210)
	at stroom.index.impl.ActiveShardsCacheImpl$ActiveShards.addDocument(ActiveShardsCacheImpl.java:193)
	at stroom.index.impl.ActiveShardsCacheImpl$ActiveShards.addDocument(ActiveShardsCacheImpl.java:148)
	at stroom.index.impl.IndexerImpl.addDocument(IndexerImpl.java:41)
	at stroom.index.impl.DynamicIndexingFilter.processFields(DynamicIndexingFilter.java:229)
	at stroom.search.extraction.AbstractFieldFilter.endElement(AbstractFieldFilter.java:96)
	at stroom.pipeline.filter.XMLFilterFork.endElement(XMLFilterFork.java:527)
	at stroom.pipeline.filter.AbstractXMLFilter.endElement(AbstractXMLFilter.java:326)
	at net.sf.saxon.event.ContentHandlerProxy.endElement(ContentHandlerProxy.java:412)
	at net.sf.saxon.event.ProxyReceiver.endElement(ProxyReceiver.java:185)
	at net.sf.saxon.event.SequenceNormalizer.endElement(SequenceNormalizer.java:230)
	at net.sf.saxon.event.NamespaceReducer.endElement(NamespaceReducer.java:235)
	at net.sf.saxon.event.ComplexContentOutputter.endElement(ComplexContentOutputter.java:559)
	at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:356)
	at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:300)
	at net.sf.saxon.expr.instruct.TemplateRule.applyLeavingTail(TemplateRule.java:352)
	at net.sf.saxon.trans.Mode.applyTemplates(Mode.java:533)
	at net.sf.saxon.expr.instruct.ApplyTemplates.apply(ApplyTemplates.java:300)
	at net.sf.saxon.expr.instruct.ApplyTemplates.processLeavingTail(ApplyTemplates.java:255)
	at net.sf.saxon.expr.instruct.Block.processLeavingTail(Block.java:735)
	at net.sf.saxon.expr.instruct.Instruction.process(Instruction.java:132)
	at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:353)
	at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:300)
	at net.sf.saxon.expr.instruct.TemplateRule.applyLeavingTail(TemplateRule.java:352)
	at net.sf.saxon.trans.Mode.applyTemplates(Mode.java:533)
	at net.sf.saxon.trans.rules.TextOnlyCopyRuleSet.process(TextOnlyCopyRuleSet.java:71)
	at net.sf.saxon.trans.Mode.applyTemplates(Mode.java:495)
	at net.sf.saxon.trans.XsltController.applyTemplates(XsltController.java:746)
	at net.sf.saxon.s9api.AbstractXsltTransformer.applyTemplatesToSource(AbstractXsltTransformer.java:347)
	at net.sf.saxon.s9api.XsltTransformer.transform(XsltTransformer.java:349)
	at net.sf.saxon.jaxp.TransformerImpl.transform(TransformerImpl.java:71)
	at net.sf.saxon.jaxp.TransformerHandlerImpl.endDocument(TransformerHandlerImpl.java:173)
	at stroom.pipeline.filter.XsltFilter.endDocument(XsltFilter.java:305)
	at stroom.pipeline.filter.AbstractXMLFilter.endDocument(AbstractXMLFilter.java:158)
	at stroom.pipeline.filter.AbstractXMLFilter.endDocument(AbstractXMLFilter.java:158)
	at stroom.pipeline.filter.SplitFilter.fireRemainingEvents(SplitFilter.java:440)
	at stroom.pipeline.filter.SplitFilter.endDocument(SplitFilter.java:199)
	at stroom.pipeline.filter.AbstractXMLFilter.endDocument(AbstractXMLFilter.java:158)
	at org.apache.xerces.parsers.AbstractSAXParser.endDocument(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)
	at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)
	at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
	at org.apache.xerces.impl.XML11EntityScanner.skipSpaces(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
	at stroom.pipeline.parser.AbstractParser.parse(AbstractParser.java:185)\n	at stroom.pipeline.parser.AbstractParser.lambda$createProcessors$0(AbstractParser.java:86)
	at stroom.pipeline.factory.PipelineImpl.process(PipelineImpl.java:129)
	at stroom.core.dataprocess.AbstractProcessorTaskExecutor.processNestedStreams(AbstractProcessorTaskExecutor.java:459)
	at stroom.core.dataprocess.AbstractProcessorTaskExecutor.process(AbstractProcessorTaskExecutor.java:340)
	at stroom.core.dataprocess.AbstractProcessorTaskExecutor.exec(AbstractProcessorTaskExecutor.java:239)
	at stroom.processor.impl.DataProcessorTaskHandler.exec(DataProcessorTaskHandler.java:164)
	at stroom.processor.impl.DataProcessorTaskHandler.lambda$exec$0(DataProcessorTaskHandler.java:92)
	at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$2(TaskContextFactoryImpl.java:253)
	at stroom.util.logging.LocationAwareLambdaLogger.logDurationIfDebugEnabled(LocationAwareLambdaLogger.java:307)
	at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$4(TaskContextFactoryImpl.java:253)
	at stroom.util.pipeline.scope.PipelineScopeRunnable.scopeResult(PipelineScopeRunnable.java:39)
	at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$5(TaskContextFactoryImpl.java:250)
	at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$6(TaskContextFactoryImpl.java:262)
	at stroom.security.impl.SecurityContextImpl.useAsReadResult(SecurityContextImpl.java:406)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
	at jdk.proxy2/jdk.proxy2.$Proxy122.useAsReadResult(Unknown Source)
	at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$7(TaskContextFactoryImpl.java:262)
	at stroom.security.impl.SecurityContextImpl.asUserResult(SecurityContextImpl.java:338)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
	at jdk.proxy2/jdk.proxy2.$Proxy122.asUserResult(Unknown Source)
	at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$11(TaskContextFactoryImpl.java:260)
	at stroom.processor.impl.DataProcessorTaskHandler.lambda$exec$1(DataProcessorTaskHandler.java:92)
	at stroom.security.impl.SecurityContextImpl.useAsReadResult(SecurityContextImpl.java:406)
	at stroom.processor.impl.DataProcessorTaskHandler.lambda$exec$2(DataProcessorTaskHandler.java:87)
	at stroom.security.impl.SecurityContextImpl.asUserResult(SecurityContextImpl.java:338)
	at stroom.processor.impl.DataProcessorTaskHandler.exec(DataProcessorTaskHandler.java:87)
	at stroom.processor.impl.DataProcessorTaskFactory$RunnableFactoryImpl.lambda$create$0(DataProcessorTaskFactory.java:166)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

@stroomdev10 stroomdev10 changed the title v7.5 Is creating too many Index shards v7.4 Is creating too many Index shards Sep 18, 2024
@stroomdev66 stroomdev66 self-assigned this Sep 20, 2024
@stroomdev10
Copy link
Member Author

This seems to relate to old shards bering created and then quickly deleted.

DEBUG  [2024-09-23T06:45:33.237Z] [ForkJoinPool.commonPool-worker-21] stroom.index.impl.ActiveShardsCacheImpl -  Creating ActiveShards for node: NodeInfoImpl{nodeName=node1}, indexShardKey: IndexShardKey{indexUuid='xxxxxx-xxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxx', partition='TimePartition{partitionBy=DAY, partitionSize=1, partitionFromTime=1285200000000, partitionToTime=1285286400000, label='2010-09-23'}'}
DEBUG  [2024-09-23T06:45:33.238Z] [Data Processor# #23668] stroom.index.impl.ActiveShardsCacheImpl$ActiveShards -  Unable to find index shard with id = 98696
stroom.index.shared.IndexException: Unable to find index shard with id = 98696
        at stroom.index.impl.IndexShardWriterCacheImpl.lambda$openWriter$9(IndexShardWriterCacheImpl.java:138)
        at stroom.security.impl.SecurityContextImpl.asUserResult(SecurityContextImpl.java:337)
        at stroom.security.impl.SecurityContextImpl.asProcessingUserResult(SecurityContextImpl.java:368)
        at stroom.index.impl.IndexShardWriterCacheImpl.openWriter(IndexShardWriterCacheImpl.java:135)
        at stroom.index.impl.IndexShardWriterCacheImpl.lambda$open$2(IndexShardWriterCacheImpl.java:128)
        at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
        at stroom.index.impl.IndexShardWriterCacheImpl.open(IndexShardWriterCacheImpl.java:124)
        at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$2(LocalCache.java:167)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2688)
        at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
        at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
        at stroom.cache.impl.AbstractStroomCache.get(AbstractStroomCache.java:170)
        at stroom.index.impl.IndexShardWriterCacheImpl.getOrOpenWriter(IndexShardWriterCacheImpl.java:116)
        at stroom.index.impl.ActiveShardsCacheImpl$ActiveShards.addDocument(ActiveShardsCacheImpl.java:210)
        at stroom.index.impl.ActiveShardsCacheImpl$ActiveShards.addDocument(ActiveShardsCacheImpl.java:193)
        at stroom.index.impl.ActiveShardsCacheImpl$ActiveShards.addDocument(ActiveShardsCacheImpl.java:148)
        at stroom.index.impl.IndexerImpl.addDocument(IndexerImpl.java:41)
        at stroom.index.impl.IndexingFilter.processDocument(IndexingFilter.java:237)
        at stroom.index.impl.IndexingFilter.endElement(IndexingFilter.java:201)
        at stroom.pipeline.filter.AbstractXMLFilter.endElement(AbstractXMLFilter.java:326)
        at stroom.pipeline.filter.RecordCountFilter.endElement(RecordCountFilter.java:192)
        at net.sf.saxon.event.ContentHandlerProxy.endElement(ContentHandlerProxy.java:412)
        at net.sf.saxon.event.ProxyReceiver.endElement(ProxyReceiver.java:185)
        at net.sf.saxon.event.SequenceNormalizer.endElement(SequenceNormalizer.java:230)
        at net.sf.saxon.event.NamespaceReducer.endElement(NamespaceReducer.java:235)
        at net.sf.saxon.event.ComplexContentOutputter.endElement(ComplexContentOutputter.java:559)
        at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:356)
        at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:300)
        at net.sf.saxon.expr.instruct.TemplateRule.applyLeavingTail(TemplateRule.java:352)
        at net.sf.saxon.trans.Mode.applyTemplates(Mode.java:533)
        at net.sf.saxon.expr.instruct.ApplyTemplates.apply(ApplyTemplates.java:300)
        at net.sf.saxon.expr.instruct.ApplyTemplates.processLeavingTail(ApplyTemplates.java:255)
        at net.sf.saxon.expr.instruct.Block.processLeavingTail(Block.java:735)
        at net.sf.saxon.expr.instruct.Instruction.process(Instruction.java:132)
        at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:353)
        at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:300)
        at net.sf.saxon.expr.instruct.TemplateRule.applyLeavingTail(TemplateRule.java:352)
        at net.sf.saxon.trans.Mode.applyTemplates(Mode.java:533)
        at net.sf.saxon.trans.rules.TextOnlyCopyRuleSet.process(TextOnlyCopyRuleSet.java:71)
        at net.sf.saxon.trans.Mode.applyTemplates(Mode.java:495)
        at net.sf.saxon.trans.XsltController.applyTemplates(XsltController.java:746)
        at net.sf.saxon.s9api.AbstractXsltTransformer.applyTemplatesToSource(AbstractXsltTransformer.java:347)
        at net.sf.saxon.s9api.XsltTransformer.transform(XsltTransformer.java:349)
        at net.sf.saxon.jaxp.TransformerImpl.transform(TransformerImpl.java:71)
        at net.sf.saxon.jaxp.TransformerHandlerImpl.endDocument(TransformerHandlerImpl.java:173)
        at stroom.pipeline.filter.XsltFilter.endDocument(XsltFilter.java:305)
        at stroom.pipeline.filter.AbstractXMLFilter.endDocument(AbstractXMLFilter.java:158)
        at stroom.pipeline.filter.AbstractXMLFilter.endDocument(AbstractXMLFilter.java:158)
        at stroom.pipeline.filter.AbstractXMLFilter.endDocument(AbstractXMLFilter.java:158)
        at stroom.pipeline.filter.SplitFilter.fireRemainingEvents(SplitFilter.java:440)
        at stroom.pipeline.filter.SplitFilter.endDocument(SplitFilter.java:199)
        at org.apache.xerces.parsers.AbstractSAXParser.endDocument(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)
        at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)
        at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
        at org.apache.xerces.impl.XML11EntityScanner.skipSpaces(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
        at stroom.pipeline.parser.AbstractParser.parse(AbstractParser.java:185)
        at stroom.pipeline.parser.AbstractParser.lambda$createProcessors$0(AbstractParser.java:86)
        at stroom.pipeline.factory.PipelineImpl.process(PipelineImpl.java:129)
        at stroom.core.dataprocess.AbstractProcessorTaskExecutor.processNestedStreams(AbstractProcessorTaskExecutor.java:459)
        at stroom.core.dataprocess.AbstractProcessorTaskExecutor.process(AbstractProcessorTaskExecutor.java:340)
        at stroom.core.dataprocess.AbstractProcessorTaskExecutor.exec(AbstractProcessorTaskExecutor.java:239)
        at stroom.processor.impl.DataProcessorTaskHandler.exec(DataProcessorTaskHandler.java:164)
        at stroom.processor.impl.DataProcessorTaskHandler.lambda$exec$0(DataProcessorTaskHandler.java:92)
        at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$2(TaskContextFactoryImpl.java:253)
        at stroom.util.logging.LocationAwareLambdaLogger.logDurationIfDebugEnabled(LocationAwareLambdaLogger.java:307)
        at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$4(TaskContextFactoryImpl.java:253)
        at stroom.util.pipeline.scope.PipelineScopeRunnable.scopeResult(PipelineScopeRunnable.java:39)
        at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$5(TaskContextFactoryImpl.java:250)
        at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$6(TaskContextFactoryImpl.java:262)
        at stroom.security.impl.SecurityContextImpl.useAsReadResult(SecurityContextImpl.java:405)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
        at jdk.proxy2/jdk.proxy2.$Proxy122.useAsReadResult(Unknown Source)
        at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$7(TaskContextFactoryImpl.java:262)
        at stroom.security.impl.SecurityContextImpl.asUserResult(SecurityContextImpl.java:337)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
        at jdk.proxy2/jdk.proxy2.$Proxy122.asUserResult(Unknown Source)
        at stroom.task.impl.TaskContextFactoryImpl.lambda$wrap$11(TaskContextFactoryImpl.java:260)
        at stroom.processor.impl.DataProcessorTaskHandler.lambda$exec$1(DataProcessorTaskHandler.java:92)
        at stroom.security.impl.SecurityContextImpl.useAsReadResult(SecurityContextImpl.java:405)
        at stroom.processor.impl.DataProcessorTaskHandler.lambda$exec$2(DataProcessorTaskHandler.java:87)
        at stroom.security.impl.SecurityContextImpl.asUserResult(SecurityContextImpl.java:337)
        at stroom.processor.impl.DataProcessorTaskHandler.exec(DataProcessorTaskHandler.java:87)
        at stroom.processor.impl.DataProcessorTaskFactory$RunnableFactoryImpl.lambda$create$0(DataProcessorTaskFactory.java:166)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
DEBUG  [2024-09-23T06:45:33.238Z] [Data Processor# #23668] stroom.index.impl.ActiveShardsCacheImpl$ActiveShards -  Trying again under lock
DEBUG  [2024-09-23T06:45:33.238Z] [Data Processor# #23668] stroom.index.impl.ActiveShardsCacheImpl$ActiveShards -  ensureShards, indexShards size before: 0
DEBUG  [2024-09-23T06:45:33.238Z] [Data Processor# #23668] stroom.index.impl.ActiveShardsCacheImpl$ActiveShards -  Creating shard for key IndexShardKey{indexUuid='xxxxxxxxx-xxxxxxxxxxxxxxxx-xxxxxxxx', partition='TimePartition{partitionBy=DAY, partitionSize=1, partitionFromTime=1285200000000, partitionToTime=1285286400000, label='2010-09-23'}'} on node1
DEBUG  [2024-09-23T06:45:33.238Z] [ForkJoinPool.commonPool-worker-21] stroom.index.impl.ActiveShardsCacheImpl$ActiveShards -  getExistingShards(), found 0 un-filtered shards, maxDocsPerShard: 1000000000
DEBUG  [2024-09-23T06:45:33.238Z] [ForkJoinPool.commonPool-worker-21] stroom.index.impl.ActiveShardsCacheImpl$ActiveShards -  getExistingShards(), indexShards size: 0
DEBUG  [2024-09-23T06:45:33.238Z] [ForkJoinPool.commonPool-worker-21] stroom.index.impl.ActiveShardsCacheImpl$ActiveShards -  ensureShards, indexShards size before: 0
DEBUG  [2024-09-23T06:45:33.238Z] [ForkJoinPool.commonPool-worker-21] stroom.index.impl.ActiveShardsCacheImpl$ActiveShards -  Creating shard for key IndexShardKey{indexUuid='xxxxxxxxx-xxxxxxxxxxxxxxxx-xxxxxxxx', partition='TimePartition{partitionBy=DAY, partitionSize=1, partitionFromTime=1285200000000, partitionToTime=1285286400000, label='2010-09-23'}'} on node1
INFO   [2024-09-23T06:45:33.240Z] [Data Processor# #23668] stroom.index.impl.IndexShardCreatorImpl -  Created index shard for index: MY-INDEX (xxxxxxxxx-xxxxxxxxxxxxxxxx-xxxxxxxx), partition: '2010-09-23', ownerNodeName: node1, index version: 9.8.0, volume group: 'xxxxxxxxx Group', volume state: ACTIVE, path: /xxxx/xxxxxxxx1
DEBUG  [2024-09-23T06:45:33.240Z] [Data Processor# #23668] stroom.index.impl.ActiveShardsCacheImpl$ActiveShards -  ensureShards, indexShards size after: 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stroomdev66 stroomdev66

Labels
bug f:pipeline-processing Issues relating to pipelines and processing p:high High priority
Projects
None yet
Milestone
v7.5
Development

No branches or pull requests
2 participants
@stroomdev66 @stroomdev10