Polars dataframe compatibility

[jonimatix]

Hi,

Are there plans to have TsFresh compatible with Polars data frames?

Thanks!

[nils-braun]

Hi @jonimatix!
Depends on what you mean with "compatible".

Do you mean if tsfresh should accept a polars data frame as input and convert to a pandas data frame (our internal representation) automatically? Not sure if this is really needed as this can be done with a single API call in polars and we do not need to add a (loose) dependency on polars just for this I think.

Or do you mean if tsfresh should use polars internally to speed up the feature computations? That would be nice, but currently most calculators need the time series as numpy array, so there would not be any speed improvements unfortunately.

[jonimatix]

Hello @nils-braun,

Thanks for your reply. I was referring at exactly the same 2 scenarios, where taking the advantages of Polars to perform feature computation (perhaps at a much faster speed?), and also taking advantage of Polars' groupby API to continue to work with a Polars df (without having to convert between pl and pd), whilst using multi-CPU processing ability of Polars.

May I ask you for a working example on how best to apply tsfresh to a very large Polars df?

Appreciate any thoughts.
Thank you!

[nils-braun]

I do not have a full example for using a polars data frame. If it fits into memory, you could probably just convert it into a pandas data frame. This is definitely the easiest way.
If this does not work, you might want to use dask (https://tsfresh.readthedocs.io/en/stable/text/large_data.html) for distributing it either on a single machine (out of core) or on multiple machines. You could e.g. write out the data into parquet in pandas and then start from there using dask or spark (or just read in the files in chunks and run extraction on each separately). I have written a two-part medium article which might be useful (https://towardsdatascience.com/time-series-feature-extraction-on-really-large-data-samples-b732f805ba0e)
If this works depends on whether you just have many timeseries and it is therefore a lot of data (good) or whether it is just one big timeseries (bad).

In the latter case, I would recommend chunking it up, extract features on each chunk separately and then build a timeseries out of the features (and maybe run extraction on them again?).