You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Nov 11, 2022. It is now read-only.
Since SDK 1.9:
Changed FileBasedSource to throw an exception when reading from a file pattern that has no matches. Pipelines will now fail at runtime rather than silently reading no data in this case.
In some of our pipelines we read from multiple buckets, and some of them could be empty in some cases (we still process data from the other buckets).
For this reason, while trying to upgrade to 1.9 we have to change our code so that if will check first if there is files in the bucket and return empty PCollection in case there is no files that match to the pattern.
It will be very useful if TextIO will have a flag allowEmpty() that will allow silently reading no data (like withoutValidation() in the init time).
The text was updated successfully, but these errors were encountered:
Since SDK 1.9:
Changed FileBasedSource to throw an exception when reading from a file pattern that has no matches. Pipelines will now fail at runtime rather than silently reading no data in this case.
In some of our pipelines we read from multiple buckets, and some of them could be empty in some cases (we still process data from the other buckets).
For this reason, while trying to upgrade to 1.9 we have to change our code so that if will check first if there is files in the bucket and return empty PCollection in case there is no files that match to the pattern.
It will be very useful if
TextIO
will have a flagallowEmpty()
that will allow silently reading no data (likewithoutValidation()
in the init time).The text was updated successfully, but these errors were encountered: