Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SYNPY-1322] Object Orientated Programming Interfaces #1013

Merged
merged 34 commits into from
Jan 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
625b0ba
OTEL Additions to what is/isn't a span
BryanFauble Nov 10, 2023
6105a71
>= dependencies
BryanFauble Nov 10, 2023
cebe138
Add in document and script around benchmarking
BryanFauble Nov 14, 2023
89cd7c7
Code review feedback
BryanFauble Nov 16, 2023
40250fb
Link to tutorial section
BryanFauble Nov 16, 2023
dd0eeb0
Adjust OTEL dependencies
BryanFauble Nov 16, 2023
03cf453
Mark test as flaky
BryanFauble Nov 16, 2023
dcc29f0
Publish branch to remote
BryanFauble Nov 16, 2023
77a8d6b
Import File earlier
BryanFauble Nov 16, 2023
b27cf12
Merge branch 'develop' into SYNPY-1322-OOP-POC
BryanFauble Nov 16, 2023
244c69f
Support for annotations,cache syn client instance
BryanFauble Nov 20, 2023
55ee852
Remove httpx
BryanFauble Nov 20, 2023
65a1c70
Merge branch 'develop' into SYNPY-1322-OOP-POC
BryanFauble Nov 28, 2023
57da4e1
Adding support for Table
BryanFauble Nov 30, 2023
5bdf4c4
Adding delete row note
BryanFauble Nov 30, 2023
1a11b17
Adding delete table
BryanFauble Nov 30, 2023
f988b5c
More example scripts
BryanFauble Nov 30, 2023
5d5f9f5
Remove comment
BryanFauble Nov 30, 2023
6b5b9af
Prevent creating instance for calling class method
BryanFauble Nov 30, 2023
86a474e
Correcting OTEL context propogation
BryanFauble Dec 1, 2023
dfbddcc
Add note about assuming annotation type
BryanFauble Dec 1, 2023
6b74371
Merge branch 'develop' into SYNPY-1322-OOP-POC
BryanFauble Dec 7, 2023
84a5369
Adding more verbose examples for current project interface
BryanFauble Dec 8, 2023
d370b46
simplify the annotation interface (#1022)
BryanFauble Dec 12, 2023
3aec4ff
[SYNPY-1345] Migrate to mkdocstrings (#1023)
BryanFauble Dec 12, 2023
93bacca
Making sticky nav tabs per team meeting
BryanFauble Dec 12, 2023
bbe905f
Merge branch 'develop' into SYNPY-1322-OOP-POC
BryanFauble Jan 16, 2024
410de22
Remove content not needed
BryanFauble Jan 16, 2024
be9f193
Remove content that is not needed
BryanFauble Jan 16, 2024
cc049b0
Merge develop and consistency changes
BryanFauble Jan 16, 2024
53e4c22
Merge branch 'develop' into SYNPY-1322-OOP-POC
BryanFauble Jan 17, 2024
8fef247
Correct for some changes
BryanFauble Jan 17, 2024
a19d77a
Clean up doc structure
BryanFauble Jan 22, 2024
377df74
Code review updates
BryanFauble Jan 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions docs/reference/oop/models.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
Contained within this file are experimental interfaces for working with the Synapse Python
Client. Unless otherwise noted these interfaces are subject to change at any time. Use
at your own risk.

These APIs also introduce [AsyncIO](https://docs.python.org/3/library/asyncio.html) to
the client.

## API reference

::: synapseclient.models.Project
options:
members:
- get
- store
- delete
---
::: synapseclient.models.Folder
options:
members:
- get
- store
- delete
---
::: synapseclient.models.File
options:
members:
- get
- store
- delete
---
::: synapseclient.models.Table
options:
members:
- get
- store_schema
- store_rows_from_csv
- delete_rows
- query
- delete
---

## Sample Scripts:

<details class="quote">
<summary>Working with a project</summary>

```python
{!docs/scripts/object_orientated_programming_poc/oop_poc_project.py!}
```
</details>

<details class="quote">
<summary>Working with folders</summary>

```python
{!docs/scripts/object_orientated_programming_poc/oop_poc_folder.py!}
```
</details>

<details class="quote">
<summary>Working with files</summary>

```python
{!docs/scripts/object_orientated_programming_poc/oop_poc_file.py!}
```
</details>

<details class="quote">
<summary>Working with tables</summary>

```python
{!docs/scripts/object_orientated_programming_poc/oop_poc_table.py!}
```
</details>


<details class="quote">
<summary>Current Synapse interface for working with a project</summary>

```python
{!docs/scripts/object_orientated_programming_poc/synapse_project.py!}
```
</details>
148 changes: 148 additions & 0 deletions docs/scripts/object_orientated_programming_poc/oop_poc_file.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
"""The purpose of this script is to demonstrate how to use the new OOP interface for files.
The following actions are shown in this script:
1. Creating a file
2. Storing a file to a project
3. Storing a file to a folder
4. Getting metadata about a file
5. Downloading a file
6. Deleting a file
"""
import asyncio
import os

from synapseclient.models import (
File,
Folder,
)
from datetime import date, datetime, timedelta, timezone
import synapseclient

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(
TracerProvider(resource=Resource(attributes={SERVICE_NAME: "oop_table"}))
)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
tracer = trace.get_tracer("my_tracer")

PROJECT_ID = "syn52948289"

syn = synapseclient.Synapse(debug=True)
syn.login()


def create_random_file(
path: str,
) -> None:
"""Create a random file with random data.

:param path: The path to create the file at.
"""
with open(path, "wb") as f:
f.write(os.urandom(1))


@tracer.start_as_current_span("File")
async def store_file():
# Creating annotations for my file ==================================================
annotations_for_my_file = {
"my_single_key_string": "a",
"my_key_string": ["b", "a", "c"],
"my_key_bool": [False, False, False],
"my_key_double": [1.2, 3.4, 5.6],
"my_key_long": [1, 2, 3],
"my_key_date": [date.today(), date.today() - timedelta(days=1)],
"my_key_datetime": [
datetime.today(),
datetime.today() - timedelta(days=1),
datetime.now(tz=timezone(timedelta(hours=-5))),
datetime(2023, 12, 7, 13, 0, 0, tzinfo=timezone(timedelta(hours=0))),
datetime(2023, 12, 7, 13, 0, 0, tzinfo=timezone(timedelta(hours=-7))),
],
}

name_of_file = "my_file_with_random_data.txt"
path_to_file = os.path.join(os.path.expanduser("~/temp"), name_of_file)
create_random_file(path_to_file)

# Creating and uploading a file to a project =========================================
file = File(
path=path_to_file,
name=name_of_file,
annotations=annotations_for_my_file,
parent_id=PROJECT_ID,
description="This is a file with random data.",
)

file = await file.store()

print(file)

# Updating and storing an annotation =================================================
file_copy = await File(id=file.id).get()
file_copy.annotations["my_key_string"] = ["new", "values", "here"]
stored_file = await file_copy.store()
print(stored_file)

# Downloading a file =================================================================
downloaded_file_copy = await File(id=file.id).get(
download_location=os.path.expanduser("~/temp/myNewFolder")
)

print(downloaded_file_copy)

# Get metadata about a file ==========================================================
non_downloaded_file_copy = await File(id=file.id).get(
download_file=False,
)

print(non_downloaded_file_copy)

# Creating and uploading a file to a folder =========================================
folder = await Folder(name="my_folder", parent_id=PROJECT_ID).store()

file = File(
path=path_to_file,
name=name_of_file,
annotations=annotations_for_my_file,
parent_id=folder.id,
description="This is a file with random data.",
)

file = await file.store()

print(file)

downloaded_file_copy = await File(id=file.id).get(
download_location=os.path.expanduser("~/temp/myNewFolder")
)

print(downloaded_file_copy)

non_downloaded_file_copy = await File(id=file.id).get(
download_file=False,
)

print(non_downloaded_file_copy)

# Uploading a file and then Deleting a file ==========================================
name_of_file = "my_file_with_random_data_to_delete.txt"
path_to_file = os.path.join(os.path.expanduser("~/temp"), name_of_file)
create_random_file(path_to_file)

file = await File(
path=path_to_file,
name=name_of_file,
annotations=annotations_for_my_file,
parent_id=PROJECT_ID,
description="This is a file with random data I am going to delete.",
).store()

await file.delete()


asyncio.run(store_file())
143 changes: 143 additions & 0 deletions docs/scripts/object_orientated_programming_poc/oop_poc_folder.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
"""The purpose of this script is to demonstrate how to use the new OOP interface for folders.
The following actions are shown in this script:
1. Creating a folder
2. Storing a folder to a project
3. Storing several files to a folder
4. Storing several folders in a folder
5. Getting metadata about a folder
6. Updating the annotations in bulk for a number of folders and files
7. Deleting a folder
"""
import asyncio
import os
from synapseclient.models import (
File,
Folder,
)
import synapseclient
from datetime import date, datetime, timedelta, timezone

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(
TracerProvider(resource=Resource(attributes={SERVICE_NAME: "oop_folder"}))
)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
tracer = trace.get_tracer("my_tracer")

PROJECT_ID = "syn52948289"

syn = synapseclient.Synapse(debug=True)
syn.login()


def create_random_file(
path: str,
) -> None:
"""Create a random file with random data.

:param path: The path to create the file at.
"""
with open(path, "wb") as f:
f.write(os.urandom(1))


@tracer.start_as_current_span("Folder")
async def store_folder():
# Creating annotations for my folder ==================================================
annotations_for_my_folder = {
"my_single_key_string": "a",
"my_key_string": ["b", "a", "c"],
"my_key_bool": [False, False, False],
"my_key_double": [1.2, 3.4, 5.6],
"my_key_long": [1, 2, 3],
"my_key_date": [date.today(), date.today() - timedelta(days=1)],
"my_key_datetime": [
datetime.today(),
datetime.today() - timedelta(days=1),
datetime.now(tz=timezone(timedelta(hours=-5))),
datetime(2023, 12, 7, 13, 0, 0, tzinfo=timezone(timedelta(hours=0))),
datetime(2023, 12, 7, 13, 0, 0, tzinfo=timezone(timedelta(hours=-7))),
],
}

# Creating a folder ==================================================================
folder = Folder(
name="my_new_folder_for_this_project",
annotations=annotations_for_my_folder,
parent_id=PROJECT_ID,
description="This is a folder with random data.",
)

folder = await folder.store()

print(folder)

# Updating and storing an annotation =================================================
folder_copy = await Folder(id=folder.id).get()
folder_copy.annotations["my_key_string"] = ["new", "values", "here"]
stored_folder = await folder_copy.store()
print(stored_folder)

# Storing several files to a folder ==================================================
files_to_store = []
for loop in range(1, 10):
name_of_file = f"my_file_with_random_data_{loop}.txt"
path_to_file = os.path.join(os.path.expanduser("~/temp"), name_of_file)
create_random_file(path_to_file)

file = File(
path=path_to_file,
name=name_of_file,
)
files_to_store.append(file)
folder.files = files_to_store
folder = await folder.store()

# Storing several folders in a folder ==================================================
folders_to_store = []
for loop in range(1, 10):
folder_to_store = Folder(
name=f"my_new_folder_for_this_project_{loop}",
)
folders_to_store.append(folder_to_store)
folder.folders = folders_to_store
folder = await folder.store()

# Getting metadata about a folder =====================================================
folder_copy = await Folder(id=folder.id).get(include_children=True)

print(folder_copy)
for file in folder_copy.files:
print(f"File: {file.name}")

for folder in folder_copy.folders:
print(f"Folder: {folder.name}")

# Updating the annotations in bulk for a number of folders and files ==================
new_annotations = {
"my_new_key_string": ["b", "a", "c"],
}

for file in folder_copy.files:
file.annotations = new_annotations

for folder in folder_copy.folders:
folder.annotations = new_annotations

await folder_copy.store()

# Deleting a folder ==================================================================
folder_to_delete = await Folder(
name="my_new_folder_for_this_project_I_want_to_delete",
parent_id=PROJECT_ID,
).store()

await folder_to_delete.delete()


asyncio.run(store_folder())
Loading
Loading