Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix#587: index list index out of range in transcript_enrich_bucket.py #588

Merged
merged 3 commits into from
Sep 17, 2024
Merged

Fix#587: index list index out of range in transcript_enrich_bucket.py #588

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
88fb41f
Fix#542: improve PS Script, Move-Item -Force if file is already there
bmerkle Sep 9, 2024
342d9cb
Merge branch 'Fix#542'
bmerkle Sep 9, 2024
7a1030c
fix issue 587: list index out of range
bmerkle Sep 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions 08-building-search-applications/scripts/prepare_transcripts_ai_show.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,10 @@ python transcript_enrich_lite.py -f $TRANSCRIPT_FOLDER

# Check if master_enriched.json file exists then rename it to include segment minutes
if (Test-Path "$TRANSCRIPT_FOLDER\output\master_enriched.json") {
Move-Item -Path "$TRANSCRIPT_FOLDER\output\master_enriched.json" -Destination "$TRANSCRIPT_FOLDER\output\embedding_index_full_${TRANSCRIPT_BUCKET_MINUTES}m.json"
Move-Item -Path "$TRANSCRIPT_FOLDER\output\master_enriched.json" -Destination "$TRANSCRIPT_FOLDER\output\embedding_index_full_${TRANSCRIPT_BUCKET_MINUTES}m.json" -Force
}

# Check if master_enriched_lite.json file exists then rename it to include segment minutes
if (Test-Path "$TRANSCRIPT_FOLDER\output\master_enriched_lite.json") {
Move-Item -Path "$TRANSCRIPT_FOLDER\output\master_enriched_lite.json" -Destination "$TRANSCRIPT_FOLDER\output\embedding_index_${TRANSCRIPT_BUCKET_MINUTES}m.json"
Move-Item -Path "$TRANSCRIPT_FOLDER\output\master_enriched_lite.json" -Destination "$TRANSCRIPT_FOLDER\output\embedding_index_${TRANSCRIPT_BUCKET_MINUTES}m.json" -Force
}
25 changes: 14 additions & 11 deletions 08-building-search-applications/scripts/transcript_enrich_bucket.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,20 +172,23 @@ def parse_json_vtt_transcript(vtt, metadata):

# Append the last text segment to the last segment in segments dictionary
if seg_begin_seconds and text != "":
previous_segment_tokens = len(tokenizer.encode(segments[-1]["text"]))
current_segment_tokens = len(tokenizer.encode(text))

if previous_segment_tokens + current_segment_tokens < MAX_TOKENS:
segments[-1]["text"] += text
if segments:
previous_segment_tokens = len(tokenizer.encode(segments[-1]["text"]))
current_segment_tokens = len(tokenizer.encode(text))

if previous_segment_tokens + current_segment_tokens < MAX_TOKENS:
segments[-1]["text"] += text
else:
if not first_segment:
# append PERCENTAGE_OVERLAP text to the previous segment
# to smooth context transition
append_text_to_previous_segment(text)
first_segment = False
add_new_segment(metadata, text, seg_begin_seconds)
else:
if not first_segment:
# append PERCENTAGE_OVERLAP text to the previous segment
# to smooth context transition
append_text_to_previous_segment(text)
first_segment = False
# If segments list is empty, add the text as a new segment
add_new_segment(metadata, text, seg_begin_seconds)


def get_transcript(metadata):
"""get the transcript from the .vtt file"""
global total_files
Expand Down
Loading