activity-aggregate-updater-job is not working

[manojkumarsah13]

Dear All,

activity-aggregate-updater-job is not working and it is failing.

2023-09-19 05:04:13,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - consumption-dedup-process (4/4) (a0f6400a2b8fbf7b5e3e465d3b51b1ef) switched from RUNNING to CANCELING.
2023-09-19 05:04:13,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - activity-aggregate-updater-fn -> (progress-update-process, collection-completion-process) (1/4) (59581ab19a0a73bfee742a0ca3e966d4) switched from RUNNING to CANCELING.
2023-09-19 05:04:13,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - activity-aggregate-updater-fn -> (progress-update-process, collection-completion-process) (2/4) (6e4b9472af6ad8e2eed2950bb499309f) switched from RUNNING to CANCELING.
2023-09-19 05:04:13,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - activity-aggregate-updater-fn -> (progress-update-process, collection-completion-process) (3/4) (393b05186e97e188e4f3c7f5e84ce74b) switched from RUNNING to CANCELING.
2023-09-19 05:04:13,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: certificate-issue-event-producer (1/1) (88e047cd3d11f45d6a808fae47f88ea2) switched from RUNNING to CANCELING.
2023-09-19 05:04:13,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: enrolment-complete-audit-sink (1/1) (57132549599b7e7af25c841966b460f3) switched from RUNNING to CANCELING.
2023-09-19 05:04:13,334 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: activity-aggregate-updater-audit-events-sink (1/1) (ea020aba405cec8ef045974b8c728ee3) switched from RUNNING to CANCELING.
2023-09-19 05:04:13,334 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: activity-aggregate-updater-failed-sink (1/1) (520a770342b4a2577ec7c74f9235f147) switched from RUNNING to CANCELING.
2023-09-19 05:04:13,352 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: activity-aggregate-updater-consumer (4/4) (d3a1a78e902c8bd3d606b7a50d5890de) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,353 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - consumption-dedup-process (4/4) (a0f6400a2b8fbf7b5e3e465d3b51b1ef) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,353 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 8fdb99b9e16c98fa22831bd8f6d77eb2: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=3}]
2023-09-19 05:04:13,369 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: activity-aggregate-updater-consumer (3/4) (8a47d5a1a0dedb2b44a89bca13122618) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,384 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - consumption-dedup-process (2/4) (7926f1fdde6db753dc1c184d17d05902) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,385 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: activity-aggregate-updater-consumer (2/4) (e2e02eb353863e4b585d2a05a2ede55d) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,385 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - activity-aggregate-updater-fn -> (progress-update-process, collection-completion-process) (2/4) (6e4b9472af6ad8e2eed2950bb499309f) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,386 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 8fdb99b9e16c98fa22831bd8f6d77eb2: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=2}]
2023-09-19 05:04:13,386 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - consumption-dedup-process (1/4) (66ee220684de8ee2a3048e5cffab3cb7) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,387 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - activity-aggregate-updater-fn -> (progress-update-process, collection-completion-process) (1/4) (59581ab19a0a73bfee742a0ca3e966d4) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,387 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: activity-aggregate-updater-consumer (1/4) (40a0f57b9ce7467b3a727b74e165761d) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,387 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: certificate-issue-event-producer (1/1) (88e047cd3d11f45d6a808fae47f88ea2) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,388 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: activity-aggregate-updater-audit-events-sink (1/1) (ea020aba405cec8ef045974b8c728ee3) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,388 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - consumption-dedup-process (3/4) (028166877ed7474931159bdb0c4208f1) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,388 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: enrolment-complete-audit-sink (1/1) (57132549599b7e7af25c841966b460f3) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,404 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - activity-aggregate-updater-fn -> (progress-update-process, collection-completion-process) (3/4) (393b05186e97e188e4f3c7f5e84ce74b) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,405 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 8fdb99b9e16c98fa22831bd8f6d77eb2: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2023-09-19 05:04:13,407 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: activity-aggregate-updater-failed-sink (1/1) (520a770342b4a2577ec7c74f9235f147) switched from CANCELING to CANCELED.
2023-09-19 05:04:13,408 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Clearing resource requirements of job 8fdb99b9e16c98fa22831bd8f6d77eb2
2023-09-19 05:04:13,408 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job activity-aggregate-updater (8fdb99b9e16c98fa22831bd8f6d77eb2) switched from state FAILING to FAILED.
org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=240000)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:216) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:206) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:197) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:682) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435) ~[notification-job-1.0.0.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[notification-job-1.0.0.jar:?]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [notification-job-1.0.0.jar:?]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [notification-job-1.0.0.jar:?]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) [activity-aggregate-updater-1.0.0.jar:?]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) [activity-aggregate-updater-1.0.0.jar:?]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [notification-job-1.0.0.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) [activity-aggregate-updater-1.0.0.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [activity-aggregate-updater-1.0.0.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [activity-aggregate-updater-1.0.0.jar:?]
at akka.actor.Actor.aroundReceive(Actor.scala:517) [notification-job-1.0.0.jar:?]
at akka.actor.Actor.aroundReceive$(Actor.scala:515) [notification-job-1.0.0.jar:?]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [notification-job-1.0.0.jar:?]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [notification-job-1.0.0.jar:?]
at akka.actor.ActorCell.invoke(ActorCell.scala:561) [notification-job-1.0.0.jar:?]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [notification-job-1.0.0.jar:?]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) [notification-job-1.0.0.jar:?]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [notification-job-1.0.0.jar:?]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [notification-job-1.0.0.jar:?]
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [notification-job-1.0.0.jar:?]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [notification-job-1.0.0.jar:?]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [notification-job-1.0.0.jar:?]
Caused by: java.lang.Exception: leaf nodes are not available for a published collection: do_3138840147146342401947
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.courseActivityAgg(ActivityAggregatesFunction.scala:152) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.$anonfun$process$6(ActivityAggregatesFunction.scala:94) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.collection.immutable.List.flatMap(List.scala:338) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.process(ActivityAggregatesFunction.scala:91) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.process(ActivityAggregatesFunction.scala:27) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.sunbird.job.WindowBaseProcessFunction.process(BaseProcessFunction.scala:71) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.apache.flink.streaming.api.scala.function.util.ScalaProcessWindowFunctionWrapper.process(ScalaProcessWindowFunctionWrapper.scala:63) ~[flink-streaming-scala_2.12-1.13.5.jar:1.13.5]
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:57) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:32) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents(WindowOperator.java:577) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:434) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[notification-job-1.0.0.jar:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-09-19 05:04:13,410 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping checkpoint coordinator for job 8fdb99b9e16c98fa22831bd8f6d77eb2.
2023-09-19 05:04:13,411 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Shutting down
2023-09-19 05:04:13,412 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job 8fdb99b9e16c98fa22831bd8f6d77eb2 reached terminal state FAILED.
org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=240000)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:216)
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:206)
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:197)
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:682)
at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435)
at jdk.internal.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: leaf nodes are not available for a published collection: do_3138840147146342401947
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.courseActivityAgg(ActivityAggregatesFunction.scala:152)
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.$anonfun$process$6(ActivityAggregatesFunction.scala:94)
at scala.collection.immutable.List.flatMap(List.scala:338)
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.process(ActivityAggregatesFunction.scala:91)
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.process(ActivityAggregatesFunction.scala:27)
at org.sunbird.job.WindowBaseProcessFunction.process(BaseProcessFunction.scala:71)
at org.apache.flink.streaming.api.scala.function.util.ScalaProcessWindowFunctionWrapper.process(ScalaProcessWindowFunctionWrapper.scala:63)
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:57)
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:32)
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents(WindowOperator.java:577)
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:434)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.base/java.lang.Thread.run(Unknown Source)
2023-09-19 05:04:13,414 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Stopping the JobMaster for job activity-aggregate-updater(8fdb99b9e16c98fa22831bd8f6d77eb2).
2023-09-19 05:04:13,417 INFO org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] - Releasing slot [38568e9ab82d1919263f68460fc6aca0].
2023-09-19 05:04:13,419 INFO org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] - Releasing slot [63436d5699e53e8db9d70a0bb156e209].
2023-09-19 05:04:13,419 INFO org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] - Releasing slot [d394085edd2fd67221c24065585c51dc].
2023-09-19 05:04:13,420 INFO org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] - Releasing slot [a7645a74bc00b258bf5d57bd3b5eddca].
2023-09-19 05:04:13,420 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Close ResourceManager connection 9441ce5991cce971a6df4db812466a02: Stopping JobMaster for job activity-aggregate-updater(8fdb99b9e16c98fa22831bd8f6d77eb2)..
2023-09-19 05:04:13,421 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Disconnect job manager [email protected]://flink@activity-aggregate-updater-jobmanager:6123/user/rpc/jobmanager_2 for job 8fdb99b9e16c98fa22831bd8f6d77eb2 from the resource manager.
2023-09-19 05:04:14,993 INFO org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap [] - Application FAILED:
java.util.concurrent.CompletionException: org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: Application Status: FAILED
at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$unwrapJobResultException$4(ApplicationDispatcherBootstrap.java:304) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
at org.apache.flink.client.deployment.application.JobStatusPollingUtils.lambda$null$2(JobStatusPollingUtils.java:101) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237) ~[notification-job-1.0.0.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081) ~[notification-job-1.0.0.jar:?]
at akka.dispatch.OnComplete.internal(Future.scala:264) ~[notification-job-1.0.0.jar:?]
at akka.dispatch.OnComplete.internal(Future.scala:261) ~[notification-job-1.0.0.jar:?]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191) ~[notification-job-1.0.0.jar:?]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188) ~[notification-job-1.0.0.jar:?]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73) ~[notification-job-1.0.0.jar:?]
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288) ~[activity-aggregate-updater-1.0.0.jar:?]
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:573) ~[notification-job-1.0.0.jar:?]
at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22) ~[notification-job-1.0.0.jar:?]
at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21) ~[notification-job-1.0.0.jar:?]
at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:536) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) ~[activity-aggregate-updater-1.0.0.jar:?]
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) ~[notification-job-1.0.0.jar:?]
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91) ~[notification-job-1.0.0.jar:?]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [activity-aggregate-updater-1.0.0.jar:?]
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) [activity-aggregate-updater-1.0.0.jar:?]
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91) [notification-job-1.0.0.jar:?]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) [notification-job-1.0.0.jar:?]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) [notification-job-1.0.0.jar:?]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [notification-job-1.0.0.jar:?]
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [notification-job-1.0.0.jar:?]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [notification-job-1.0.0.jar:?]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [notification-job-1.0.0.jar:?]
Caused by: org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: Application Status: FAILED
at org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
... 43 more
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:60) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
... 43 more
Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=240000)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:216) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:206) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:197) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:682) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435) ~[notification-job-1.0.0.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[notification-job-1.0.0.jar:?]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[notification-job-1.0.0.jar:?]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[notification-job-1.0.0.jar:?]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) ~[activity-aggregate-updater-1.0.0.jar:?]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[notification-job-1.0.0.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) ~[activity-aggregate-updater-1.0.0.jar:?]
at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[notification-job-1.0.0.jar:?]
at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[notification-job-1.0.0.jar:?]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[notification-job-1.0.0.jar:?]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[notification-job-1.0.0.jar:?]
at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[notification-job-1.0.0.jar:?]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[notification-job-1.0.0.jar:?]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[notification-job-1.0.0.jar:?]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[notification-job-1.0.0.jar:?]
... 4 more
Caused by: java.lang.Exception: leaf nodes are not available for a published collection: do_3138840147146342401947
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.courseActivityAgg(ActivityAggregatesFunction.scala:152) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.$anonfun$process$6(ActivityAggregatesFunction.scala:94) ~[activity-aggregate-updater-1.0.0.jar:?]
at scala.collection.immutable.List.flatMap(List.scala:338) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.process(ActivityAggregatesFunction.scala:91) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.sunbird.job.aggregate.functions.ActivityAggregatesFunction.process(ActivityAggregatesFunction.scala:27) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.sunbird.job.WindowBaseProcessFunction.process(BaseProcessFunction.scala:71) ~[activity-aggregate-updater-1.0.0.jar:?]
at org.apache.flink.streaming.api.scala.function.util.ScalaProcessWindowFunctionWrapper.process(ScalaProcessWindowFunctionWrapper.scala:63) ~[flink-streaming-scala_2.12-1.13.5.jar:1.13.5]
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:57) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:32) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents(WindowOperator.java:577) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:434) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[notification-job-1.0.0.jar:?]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[notification-job-1.0.0.jar:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-09-19 05:04:14,996 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting StandaloneApplicationClusterEntryPoint down with application status FAILED. Diagnostics null.
2023-09-19 05:04:14,997 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting down rest endpoint.
2023-09-19 05:04:15,006 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing cache directory /tmp/flink-web-354aa470-ae20-47ef-995b-5f23900b3cf3/flink-web-ui
2023-09-19 05:04:15,007 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - http://activity-aggregate-updater-jobmanager:8081 lost leadership
2023-09-19 05:04:15,007 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut down complete.
2023-09-19 05:04:15,008 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Shut down cluster because application is in FAILED, diagnostics null.
2023-09-19 05:04:15,008 INFO org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent [] - Closing components.
2023-09-19 05:04:15,008 INFO org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner [] - DefaultDispatcherRunner was revoked the leadership with leader id 00000000-0000-0000-0000-000000000000. Stopping the DispatcherLeaderProcess.
2023-09-19 05:04:15,009 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Stopping SessionDispatcherLeaderProcess.
2023-09-19 05:04:15,009 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping dispatcher akka.tcp://flink@activity-aggregate-updater-jobmanager:6123/user/rpc/dispatcher_1.
2023-09-19 05:04:15,009 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping all currently running jobs of dispatcher akka.tcp://flink@activity-aggregate-updater-jobmanager:6123/user/rpc/dispatcher_1.
2023-09-19 05:04:15,009 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Closing the slot manager.
2023-09-19 05:04:15,009 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Suspending the slot manager.
2023-09-19 05:04:15,009 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopped dispatcher akka.tcp://flink@activity-aggregate-updater-jobmanager:6123/user/rpc/dispatcher_1.
2023-09-19 05:04:15,012 INFO org.apache.flink.runtime.blob.BlobServer [] - Stopped BLOB server at 0.0.0.0:6124
2023-09-19 05:04:15,013 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopping Akka RPC service.
2023-09-19 05:04:15,017 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopping Akka RPC service.
2023-09-19 05:04:15,024 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting down remote daemon.
2023-09-19 05:04:15,024 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting down remote daemon.
2023-09-19 05:04:15,026 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote daemon shut down; proceeding with flushing remote transports.
2023-09-19 05:04:15,026 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote daemon shut down; proceeding with flushing remote transports.
2023-09-19 05:04:15,047 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting shut down.
2023-09-19 05:04:15,048 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting shut down.
2023-09-19 05:04:15,064 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopped Akka RPC service.
2023-09-19 05:04:15,066 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopped Akka RPC service.
2023-09-19 05:04:15,066 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Terminating cluster entrypoint process StandaloneApplicationClusterEntryPoint with exit code 1443.

We also Checked in DB and API, we are getting data, but in Redis we dont have data available for this do_ids. content cache update is running fine. Please suggest on this.

Thanks,
Manoj

[pallakartheekreddy]

@manojkumarsah13 activity-aggregate-updater-job is part of Lern building block. Can you create this bug in the below discussion forum
https://github.com/orgs/Sunbird-Lern/discussions

@AmiableAnil For your rerence.

[Krishnaj20]

@manojkumarsah13 - Pls raise this in the Lern discussion as requested by Kartheek. For now closing this discussion.