Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace storage_v1_client with GCS client #25676

Closed
1 of 15 tasks
BjornPrime opened this issue Mar 1, 2023 · 10 comments · Fixed by #29360
Closed
1 of 15 tasks

Replace storage_v1_client with GCS client #25676

BjornPrime opened this issue Mar 1, 2023 · 10 comments · Fixed by #29360
Assignees
@shunping @BjornPrime
Labels
done & done Issue has been reviewed after it was closed for verification, followups, etc. new feature P2 python
Milestone
2.53.0 Release

Comments

@BjornPrime
Copy link
Contributor

BjornPrime commented Mar 1, 2023 •
edited
Loading

Replace references to the apitools-generated storage client (sdks/python/apache_beam/io/gcp/internal/clients/storage) with analogous calls to the Google Cloud Storage client, supplementing with other Google Cloud Python clients if necessary.

Please ensure that all client references are being thoroughly tested before swapping the GCS client in for the old client. The job is done when the old client can be safely deleted from Beam without impacting functionality.

Background:

apitools is deprecated and continued reliance on it is complicating Beam's dependency management. In order to remove it we need to replace the clients that it currently generates with Google Cloud Python clients, which are being actively maintained (and should be reasonably up-to-date in google3, though please confirm for any particular libraries you end up using.)

Tracking issue: #19073

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@BjornPrime BjornPrime mentioned this issue Mar 1, 2023
Open
@BjornPrime
Copy link
Contributor Author

.take-issue

@BjornPrime BjornPrime mentioned this issue Mar 1, 2023
Open
15 tasks
This was referenced Mar 20, 2023
Closed
Merged
@liferoad liferoad mentioned this issue Apr 11, 2023
Open
@BjornPrime
Copy link
Contributor Author

Closing now that PR #25965 has been merged.

@github-actions github-actions bot added this to the 2.50.0 Release milestone Jul 5, 2023
@damccorm
Copy link
Contributor

damccorm commented Jul 11, 2023 •
edited
Loading

Reopening since this got reverted in #27388

@damccorm damccorm reopened this Jul 11, 2023
@damccorm damccorm removed this from the 2.50.0 Release milestone Jul 11, 2023
@BjornPrime BjornPrime mentioned this issue Jul 12, 2023
Closed
3 tasks
@tvalentyn tvalentyn mentioned this issue Jul 24, 2023
Open
15 tasks
@BjornPrime
Copy link
Contributor Author

Running into an issue with tar.gz objects (such as staged workflow files) being implicitly decompressed when retrieved from GCS (see #12619).

The way this was handled before seems to have possibly been apitools-specific.

@BjornPrime BjornPrime mentioned this issue Aug 7, 2023
Merged
3 tasks
@liferoad liferoad mentioned this issue Aug 11, 2023
Merged
3 tasks
This was referenced Aug 21, 2023
Merged
Merged
@liferoad liferoad added this to the 2.51.0 Release milestone Sep 15, 2023
@kennknowles
Copy link
Member

What is the status of this now?

@BjornPrime
Copy link
Contributor Author

#28079 should resolve this. We're currently running some internal tests to ensure we don't break any internal users before merging again. If all goes well, that should be done by the end of this week, possibly even today.

@tvalentyn
Copy link
Contributor

It sounds like this will go to next release.

@kennknowles kennknowles removed this from the 2.51.0 Release milestone Sep 25, 2023
@kennknowles kennknowles added this to the 2.52.0 Release milestone Sep 25, 2023
@BjornPrime BjornPrime mentioned this issue Sep 29, 2023
Draft
3 tasks
@jrmccluskey
Copy link
Contributor

Do we have any updates on this ahead of the 2.53.0 branch cut?

@damccorm
Copy link
Contributor

damccorm commented Dec 6, 2023

@shunping was this completed in #29360 or is there still remaining work?

@shunping
Copy link
Contributor

shunping commented Dec 6, 2023

It was completed. We should just close it.

@damccorm damccorm closed this as completed Dec 6, 2023
@tvalentyn tvalentyn added the done & done Issue has been reviewed after it was closed for verification, followups, etc. label Dec 7, 2023
@lgeiger lgeiger mentioned this issue Jan 31, 2024
Closed
16 tasks
@shunping shunping mentioned this issue Aug 13, 2024
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shunping shunping

@BjornPrime BjornPrime

Labels
done & done Issue has been reviewed after it was closed for verification, followups, etc. new feature P2 python
Projects
None yet
Milestone
2.53.0 Release
Development

Successfully merging a pull request may close this issue.

Gcsio migration shunping/beam
7 participants
@kennknowles @shunping @liferoad @tvalentyn @BjornPrime @jrmccluskey @damccorm