observation loading performance issue

[zhouji2013]

There is an obvious delay between the loading of two submissions. Specifically, the gap between the executions of the last line of method write of ObservationDataWriter and the first line of method mapFieldSet of ObservationDataFieldSetMapper can be very long. In the current case of 689 submissions, the gap is smaller at the beginning, but for later submissions, say, at 500th submission, this gap becomes longer than 10 seconds. I should investigate what is happening during the gap. It could be a chance to improve the loading performance.

[zhouji2013]

With the latest data, the last gap is as long as about 18 seconds. Nothing is apparently supposed to happen during that gap.

[zhouji2013]

It looks like that all the gaps are caused by Spring Batch code instead of the application's own code.

1. After I commented out all the content of ObservationDataFieldSetMapper.mapFieldSet() and ObservationDataWriter.write(), the duration and pattern of the gaps stayed the same.
2. When observing with VisualVM, all the CPU usages are from org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep (), which includes about half by org.springframework.batch.core.job.SimpleStepHandler.handleStep () and the other half by org.springframework.batch.core.job.flow.JobFlowExecutor.isStepRestart ().

[zhouji2013]

Maybe we could not do much with this, but the reason that this got my attention is that this step used to take about 15 minutes for 122 submissions, but now it takes 90 minutes for 689 submissions.