Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

implemented multithreading using the multiprocessing module #1116

Open
wants to merge 10 commits into
base: 2.18-Beta
Choose a base branch
from
Open

implemented multithreading using the multiprocessing module #1116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
805706f
implement multiprocessing in auth.py
virophagesp May 14, 2024
e3189a6
implement multiprocessing in logging.py
virophagesp May 14, 2024
2bf28fd
implement multiprocessing in operations.py
virophagesp May 14, 2024
b11a847
implement multiprocessing in utils.py
virophagesp May 14, 2024
68fd3d4
implement multiprocessing in validation.py
virophagesp May 14, 2024
6b01318
add comment in auth.py regarding multithreading
virophagesp May 14, 2024
29aa5ef
adding commet in operations.py regarding multithreading
virophagesp May 14, 2024
1305825
added more multithreading documentation auth.py
virophagesp May 14, 2024
010a6e7
added shared dictionary, but now the ram usage is extreme
May 24, 2024
8cc8d2a
fixed memory leak
May 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions Scripts/auth.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@
SALT_BYTES = 64
SCRYPT_N = 2**18

YOUTUBE = None
# the second element will be used for multithreading only
YOUTUBE = [None, None]
CURRENTUSER = None


Expand Down Expand Up @@ -100,7 +101,10 @@ def get_authenticated_service():
else:
with open(TOKEN_FILE_NAME, 'w') as token:
token.write(creds.to_json())
YOUTUBE = build(API_SERVICE_NAME, API_VERSION, credentials=creds, discoveryServiceUrl=DISCOVERY_SERVICE_URL, cache_discovery=False, cache=None)

# the program crashes if i used the same build for multithreading so i made a list of 2
YOUTUBE[0] = build(API_SERVICE_NAME, API_VERSION, credentials=creds, discoveryServiceUrl=DISCOVERY_SERVICE_URL, cache_discovery=False, cache=None)
YOUTUBE[1] = build(API_SERVICE_NAME, API_VERSION, credentials=creds, discoveryServiceUrl=DISCOVERY_SERVICE_URL, cache_discovery=False, cache=None)
return YOUTUBE


Expand Down Expand Up @@ -143,7 +147,7 @@ def get_current_user(config):

#Define fetch function so it can be re-used if issue and need to re-run it
def fetch_user():
results = YOUTUBE.channels().list(
results = YOUTUBE[0].channels().list(
part="snippet", #Can also add "contentDetails" or "statistics"
mine=True,
fields="items/id,items/snippet/title"
Expand Down
4 changes: 2 additions & 2 deletions Scripts/logging.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -656,7 +656,7 @@ def get_extra_json_data(channelIDs, jsonSettingsDict):

def fetch_data(channelIdGroup):
try:
response = auth.YOUTUBE.channels().list(part="snippet,statistics", id=channelIdGroup, fields=fieldsToFetch).execute()
response = auth.YOUTUBE[0].channels().list(part="snippet,statistics", id=channelIdGroup, fields=fieldsToFetch).execute()
if response['items']:
for infoDict in response['items']:
tempDict = {}
Expand Down Expand Up @@ -687,7 +687,7 @@ def fetch_data(channelIdGroup):
pass

# Get info about uploader
response = auth.YOUTUBE.channels().list(part="snippet,statistics", id=channelOwnerID, fields=fieldsToFetch).execute()
response = auth.YOUTUBE[0].channels().list(part="snippet,statistics", id=channelOwnerID, fields=fieldsToFetch).execute()
if response['items']:
tempDict = {}
tempDict['PublishedAt'] = response['items'][0]['snippet']['publishedAt']
Expand Down
Loading