[HW] Apache Beam By Joshua B. Turner

[joshprojekt202]

Name: Joshua B. Turner
Email: [email protected]
LinkedIn: https://www.linkedin.com/in/taztech/

Homework:

I will provide key steps and output here.

Uploaded secure bundle to Gitpod:

Environmental variables used throughout workshop:

ASTRA_DB_APPLICATION_TOKEN=AstraCS:{my_unique_organization_adminstrator_token}
ASTRA_DB_GRAPHQL_URL_ADMIN=https://4d8c7097-4759-427f-a9a5-d52d98757ea8-us-east1.apps.astra.datastax.com/api/graphql-admin
ASTRA_DB_SECURE_BUNDLE_PATH=/workspace/workshop-beam/secure-connect-workshop-beam.zip
ASTRA_DB_REGION=us-east1
ASTRA_DB_GRAPHQL_URL_PLAYGROUND=https://4d8c7097-4759-427f-a9a5-d52d98757ea8-us-east1.apps.astra.datastax.com/api/playground
ASTRA_DB_SECURE_BUNDLE_URL=https://datastax-cluster-config-prod.s3.us-east-2.amazonaws.com/4d8c7097-4759-427f-a9a5-d52d98757ea8-1/secure-connect-workshop-beam.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2AIQRQ76S2JCB77W%2F20230809%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20230809T123526Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=ba82c4379dd3b13454d2bbee80a5ae2c2af9d57e02b3f78e8170a23d7feabf18
ASTRA_ORG_TOKEN=AstraCS:{my_unique_organization_adminstrator_token}
ASTRA_DB_GRAPHQL_URL=https://4d8c7097-4759-427f-a9a5-d52d98757ea8-us-east1.apps.astra.datastax.com/api/graphql/beam
ASTRA_ORG_ID=f9460f14-9879-4ebe-83f2-48d3f3dce13c
ASTRA_DB_REST_URL=https://4d8c7097-4759-427f-a9a5-d52d98757ea8-us-east1.apps.astra.datastax.com/api/rest
ASTRA_DB_ID=4d8c7097-4759-427f-a9a5-d52d98757ea8
ASTRA_DB_GRAPHQL_URL_SCHEMA=https://4d8c7097-4759-427f-a9a5-d52d98757ea8-us-east1.apps.astra.datastax.com/api/graphql-schema
[email protected]
ASTRA_DB_KEYSPACE=beam
ASTRA_DB_REST_URL_SWAGGER=https://4d8c7097-4759-427f-a9a5-d52d98757ea8-us-east1.apps.astra.datastax.com/api/rest/swagger-ui/

Ran the following in the terminal:

mvn clean compile exec:java \
 -Dexec.mainClass=com.datastax.astra.beam.genai.GenAI_01_ImportData \
 -Dexec.args="\
 --astraToken=${ASTRA_DB_APPLICATION_TOKEN} \
 --astraSecureConnectBundle=${ASTRA_DB_SECURE_BUNDLE_PATH} \
 --astraKeyspace=${ASTRA_DB_KEYSPACE} \
 --csvInput=`pwd`/src/main/resources/fables_of_fontaine.csv"

Ran the following in CQL Console:

SELECT * FROM fable;

Results from previous command:

document_id | document                                   | title
-------------+--------------------------------------------+----------------------
           6 |              Of certain gormandizing cits. | THE FLY AND THE GAME
          16 |      The lady thought the creatures prime, | THE FLY AND THE GAME
          32 |      Such game as this may suit the dogs.' | THE FLY AND THE GAME
          26 |               Was anything but fit to eat. | THE FLY AND THE GAME
          19 |          For dinner she could hardly wait. | THE FLY AND THE GAME
          13 |        Pronounced it racy, rich, and rare, | THE FLY AND THE GAME
          34 |        His soul possess'd of this surmise, | THE FLY AND THE GAME
          21 |          Just as the buyer drew his purse, | THE FLY AND THE GAME
           7 |           With merry heart the fellow went | THE FLY AND THE GAME
          17 |          And for their dinner just in time | THE FLY AND THE GAME
           9 |         Who loved, as well was understood, | THE FLY AND THE GAME
          15 |       About its purchase for their dishes. | THE FLY AND THE GAME
          10 |           Whatever game was nice and good. | THE FLY AND THE GAME
           4 |           By poverty were sorely press'd-- | THE FLY AND THE GAME
           3 |           Unless the feelings of my breast | THE FLY AND THE GAME
          30 |             He knows a thing or two I ween | THE FLY AND THE GAME
           5 |     With birds and squirrels for the spits | THE FLY AND THE GAME
          36 |           A filthy taint they soonest find | THE FLY AND THE GAME
          25 |        But here his words I won't repeat-- | THE FLY AND THE GAME
          35 |          About some men, as well as flies: | THE FLY AND THE GAME
          18 |          So sweet they were, and delicate, | THE FLY AND THE GAME
          28 |                 I never knew to tell a lie | THE FLY AND THE GAME
          14 |    And call'd his wife, to know her wishes | THE FLY AND THE GAME
           8 |                 Direct to Mr. Centpercent, | THE FLY AND THE GAME
          31 |            My dear, I beg you, do not buy: | THE FLY AND THE GAME
          20 | But now there came--could luck be worse?-- | THE FLY AND THE GAME
           2 |       Once fill'd his bag--as I would not, | THE FLY AND THE GAME
          22 |             A bulky fly, with solemn buzz, | THE FLY AND THE GAME
          12 |         Survey'd the dainty lot with care, | THE FLY AND THE GAME
          23 |           And smelt, as an inspector does, | THE FLY AND THE GAME
          24 |    This bird and that, and said the meat-- | THE FLY AND THE GAME
          33 |         So on our peddling sportsman jogs, | THE FLY AND THE GAME
          29 |         His coat, you see, is bottle-green | THE FLY AND THE GAME
          37 |          Who are to relish filth inclined. | THE FLY AND THE GAME
          11 |          This gentleman, with knowing air, | THE FLY AND THE GAME
           1 |           A knight of powder-horn and shot | THE FLY AND THE GAME
          27 |       'Ah!' cried the lady, 'there's a fly | THE FLY AND THE GAME

(37 rows)

Attempted the optional portion and notified current quota exceeded.

Presently on waitlist to upgrade plan:

Additional settings used in final flow:

export GCP_PROJECT_ID=workshop-beam
export GCP_PROJECT_CODE={my_unique_project_code}
export [email protected]
export GCP_COMPUTE_ENGINE={my_unique_project_code}[email protected]
export ASTRA_SECRET_TOKEN=projects/{my_unique_project_code}/secrets/astra-token/versions/2
export ASTRA_SECRET_SECURE_BUNDLE=projects/{my_unique_project_code}/secrets/secure-connect-bundle-demo/versions/1
export GOOGLE_APPLICATION_CREDENTIALS=/workspace/workshop-beam/workshop-beam-f790f7cd5c28.json
export ASTRA_DB_TABLE=fable

Created secrets:

gcloud secrets create astra-token \
   --data-file <(echo -n "${ASTRA_DB_APPLICATION_TOKEN}") \
   --replication-policy="automatic"

gcloud secrets create secure-connect-bundle-demo \
   --data-file ${ASTRA_DB_SECURE_BUNDLE_PATH} \
   --replication-policy="automatic"

gcloud secrets add-iam-policy-binding secure-connect-bundle-demo \
    --member="serviceAccount:${GCP_COMPUTE_ENGINE}" \
    --role='roles/secretmanager.secretAccessor'

gcloud secrets add-iam-policy-binding astra-token \
    --member="serviceAccount:${GCP_COMPUTE_ENGINE}" \
    --role='roles/secretmanager.secretAccessor'
    
gcloud secrets list

Uploaded JSON key file:

Granted permissions:

gcloud projects add-iam-policy-binding workshop-beam \
  --member="serviceAccount:{my_unique_project_code}[email protected]" \
  --role=roles/dataflow.worker

Ran the following in the terminal:

mvn compile exec:java \
 -Dexec.mainClass=com.datastax.astra.dataflow.AstraDb_To_BigQuery_Dynamic \
 -Dexec.args="\
 --astraToken=${ASTRA_SECRET_TOKEN} \
 --astraSecureConnectBundle=${ASTRA_SECRET_SECURE_BUNDLE} \
 --astraKeyspace=${ASTRA_DB_KEYSPACE} \
 --table=${ASTRA_DB_TABLE} \
 --runner=DataflowRunner \
 --project=${GCP_PROJECT_ID} \
 --region=us-central1"

Ran the following in the terminal:

bq head -n 10 ${ASTRA_DB_KEYSPACE}.${ASTRA_DB_TABLE}

Results from previous command:

+-------------+----------------------------------------+------------------+---------------+----------------------+
| document_id |                document                | embedding_vector | metadata_blob |        title         |
+-------------+----------------------------------------+------------------+---------------+----------------------+
| 26          | Was anything but fit to eat.           | NULL             | NULL          | THE FLY AND THE GAME |
| 36          | A filthy taint they soonest find       | NULL             | NULL          | THE FLY AND THE GAME |
| 25          | But here his words I won't repeat--    | NULL             | NULL          | THE FLY AND THE GAME |
| 35          | About some men, as well as flies:      | NULL             | NULL          | THE FLY AND THE GAME |
| 18          | So sweet they were, and delicate,      | NULL             | NULL          | THE FLY AND THE GAME |
| 27          | 'Ah!' cried the lady, 'there's a fly   | NULL             | NULL          | THE FLY AND THE GAME |
| 5           | With birds and squirrels for the spits | NULL             | NULL          | THE FLY AND THE GAME |
| 16          | The lady thought the creatures prime,  | NULL             | NULL          | THE FLY AND THE GAME |
| 32          | Such game as this may suit the dogs.'  | NULL             | NULL          | THE FLY AND THE GAME |
| 9           | Who loved, as well was understood,     | NULL             | NULL          | THE FLY AND THE GAME |
+-------------+----------------------------------------+------------------+---------------+----------------------+

[hemidactylus]

@clun here's to a true workshop hero! Congrats, Josh!