Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hotfix/stability #10

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Hotfix/stability #10

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e0dab5a
Stability fixes
steffencruz Aug 18, 2023
29e0f00
Stability fixes
steffencruz Aug 18, 2023
afadc94
Update multigraph.py
steffencruz Aug 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions multigraph.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,16 +116,18 @@ def parse_arguments():

if not args.no_dataframe:
save_path = f'data/metagraph/{netuid}/df.parquet'
blocks = range(start_block, end_block, step_size)
blocks = sorted(range(start_block, end_block, -step_size))
df_loaded = None
if os.path.exists(save_path):
df_loaded = pd.read_parquet(save_path)
blocks = [block for block in blocks if block not in df_loaded.block.unique()]
print(f'Loaded dataframe from {save_path!r}. {len(df_loaded)} rows. {len(blocks)} blocks to process.')
if len(blocks)==0:
print('No blocks to process.')
sys.exit(0)


if len(blocks)==0:
steffencruz marked this conversation as resolved.
Show resolved Hide resolved
print('No blocks to process.')
sys.exit(0)

print(f'Loading metagraphs from {blocks[0]}-{blocks[-1]} ({len(blocks)}) to create dataframe.')
df = load_metagraphs(blocks[0], blocks[-1], block_step=step_size, datadir=datadir)
if df_loaded is not None:
df = pd.concat([df, df_loaded], ignore_index=True)
Expand Down
13 changes: 12 additions & 1 deletion multistats.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,17 @@
api= wandb.Api(timeout=60)
wandb.login(anonymous="allow")

import json
# Store the original json.loads
original_json_loads = json.loads

def relaxed_json_loads(s, *args, **kwargs):
kwargs.pop("strict", None) # remove the strict argument if it exists
return original_json_loads(s, strict=False, *args, **kwargs)
Copy link
Contributor

@eduardogr eduardogr Aug 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not call it explicitly like this in every json.loads call ?? or use this relaxed_json_loads.
Even though the approach is very nice in shape. I think it obscures what's really happening, and that's important for maintenance and readability of the code.

From my point of view, I'd say, let's make the call explicit as it is in this line, or use the function relaxed_json_loads, so we know something different is happening

What do you think about this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea it's not my preferred solution at all, however the issue actual originates inside the wandb codebase - they enforce strict mode when pulling assets from their databases. We never actually call json decide ourselves..
I propose we put it either here or shamefully in a Util file for now

Copy link
Contributor Author

@steffencruz steffencruz Aug 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it's a way of overriding json globally.
Floppy actually came up with this hack because anyone who wants to analyze wandb data has to do something similar


# Patch the library's json.loads
json.loads = relaxed_json_loads

def pull_wandb_runs(project='openvalidators', filters=None, min_steps=50, max_steps=100_000, ntop=10, netuid=None, summary_filters=None ):
# TODO: speed this up by storing older runs

Expand Down Expand Up @@ -264,7 +275,7 @@ def parse_arguments():
print(args)

filters = None# {"tags": {"$in": [f'1.1.{i}' for i in range(10)]}}
# filters={'tags': {'$in': ['5F4tQyWrhfGVcNhoqeiNsR6KjD4wMZ2kfhLj4oHYuyHbZAc3']}} # Is foundation validator

if args.load_runs and os.path.exists('data/wandb.csv'):
df_runs = pd.read_csv('data/wandb.csv')
assert len(df_runs) >= args.ntop, f'Loaded {len(df_runs)} runs, but expected at least {args.ntop}'
Expand Down