Skip to content

Commit

Permalink
Update triggering frequency doc (#30457)
Browse files Browse the repository at this point in the history
  • Loading branch information
ahmedabu98 authored Mar 4, 2024
1 parent b776d70 commit 6a03f9b
Show file tree
Hide file tree
Showing 3 changed files with 7 additions and 6 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2949,8 +2949,8 @@ public Write<T> withLoadJobProjectId(ValueProvider<String> loadJobProjectId) {
/**
* Choose the frequency at which file writes are triggered.
*
* <p>This is only applicable when the write method is set to {@link Method#FILE_LOADS}, and
* only when writing an unbounded {@link PCollection}.
* <p>This is only applicable when the write method is set to {@link Method#FILE_LOADS} or
* {@link Method#STORAGE_WRITE_API}, and only when writing an unbounded {@link PCollection}.
*
* <p>Every triggeringFrequency duration, a BigQuery load job will be generated for all the data
* written since the last load job. BigQuery has limits on how many load jobs can be triggered
Expand All @@ -2975,9 +2975,9 @@ public Write<T> withNumFileShards(int numFileShards) {
}

/**
* Control how many parallel streams are used when using Storage API writes. Applicable only
* when also setting {@link #withTriggeringFrequency}. To let runner determine the sharding at
* runtime, set this to zero, or {@link #withAutoSharding()} instead.
* Control how many parallel streams are used when using Storage API writes. Applicable only for
* streaming pipelines, and when {@link #withTriggeringFrequency} is also set. To let runner
* determine the sharding at runtime, set this to zero, or {@link #withAutoSharding()} instead.
*/
public Write<T> withNumStorageWriteApiStreams(int numStorageWriteApiStreams) {
return toBuilder().setNumStorageWriteApiStreams(numStorageWriteApiStreams).build();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ public interface BigQueryOptions
void setUseStorageWriteApiAtLeastOnce(Boolean value);

@Description(
"If set, then BigQueryIO.Write will default to using this number of Storage Write API streams. ")
"When writing with a streaming pipeline, the BigQueryIO.Write will default to using this number of Storage Write API streams. ")
@Default.Integer(0)
Integer getNumStorageWriteApiStreams();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -858,6 +858,7 @@ pipeline uses. You can set it explicitly on the transform via
[`withNumStorageWriteApiStreams`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withNumStorageWriteApiStreams-int-)
or provide the `numStorageWriteApiStreams` option to the pipeline as defined in
[`BigQueryOptions`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.html).
Please note this is only supported for streaming pipelines.

Triggering frequency determines how soon the data is visible for querying in
BigQuery. You can explicitly set it via
Expand Down

0 comments on commit 6a03f9b

Please sign in to comment.