[destination-motherduck] not loading entire dataset only keep last batch

[marcosmarxm]

Connector Name

destination-motherduck

Connector Version

0.1.16

What step the error happened?

During the sync

Relevant information

I'm syncing a source has more than 50k (batch size) used by Destination Motherduck. The result end up with only the last batch.

In my example I'm using https://huggingface.co/datasets/rajpurkar/squad dataset it has ~ 87k rows.

This is the result in motherduck database:

Relevant log output

Contribute

• Yes, I want to contribute