Just-In-Time tabularization take very long time ?

[joejoe2]

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Checked the Answered Questions on the Github Discussion board: https://github.com/ourownstory/neural_prophet/discussions
    If you have the same question but the Answer does not solve your issue, please continue the conversation there.
  • Checked that your issue isn't already filed: https://github.com/ourownstory/neural_prophet/issues
    If you have the same issue but there is a twist to your situation, please add an explanation there.
  • Considered whether your issue might need further discussing before being defined as a feature request:
    Please post an idea or feedback

Is your feature request related to a problem? Please describe.

I have upgraded to 1.0.0rc10 from 0.8.0. The dataloader takes 4 minute before training start in 0.8.0 with 3000 time series. But now it takes more than 20 minute (still running ... ). I think this is related to #1469 and #1529.

Describe the solution you'd like

Maybe we can make the Just-In-Time tabularization optional with a flag ?

[ourownstory]

Hi @joejoe2 You are right - this move is currently increasing the computation times. Please bear with us - It is a larger move that we are working on in multiple steps. The type of change is of a more fundamental nature that is not compatible with an optional flag. In the next workstep we will focus on improving computation times. Overall, the expected outcome is to achieve dramatically lower memory usage (allowing for larger datasets) and better parallelization, requiring comparable compute (depending on data/model/hardware setup).

We did a regular release of NP 0.9.0 before merging this change. In this transitory phase, please stick to 0.9.0 (or future regular releases) and avoid main (or the 1.0.0rc release candidates) for stable/production environments.