Orcasound extra .ts files in bucket

[Molkree]

While working on Orcasound data I've noticed that buckets contain some "extra" files if you can call them so.
Examples:

• last file in bush_point/1624170621 is live2158.ts while live.m3u8 lists live2150.ts as the last one.
• similar with bush_point/1624192222, last one is live2158.ts while live.m3u8 lists live2151.ts as the last one.

The problem with these very last files (live2158.ts) is that they are empty (0 bytes). Honestly only noticed them because ffmpeg failed when processing them, I wasn't even looking at live.m3u8 before.

Files live2151-2157 seem to be valid ones.

@paulcretu, I think you can help with this/did the setup of this? For now I just drop the last file (will probably read only files listed in the playlist eventually) but we probably shouldn't store these files at all or should include them in .m3u8 (except the last one)?

[paulcretu]

Good find! The reason these streams end is because the node gets shut down, so I assume this has something to do with that, especially that very last file being empty.

I'm not sure why the last 8 files aren't in the m3u8. I did set up the streaming, but that was a while ago and @mcshicks has made a lot of changes since then. He wrote this python script to handle uploads, but it's not obvious to me looking at it why it would cause the missing entries. I know the older versions of the code used to intentionally cut out the last few files, it seems likely that Bush Point is still running the older code.

So for now @Molkree I agree with just dropping the last file as you're doing. @scottveirs or @mcshicks do you know what version of the code Bush Point is currently running? I think figuring that out is the next step here.

[mcshicks]

I'm pretty sure it's pre upload script. I think upgrading to a more recent versions was one of @scottveirs goals for his hydrophone updates, which was why we wanted to get a working image in docker hub. But I since it's only a few files at the end it certainly seems like it's probably related to the cron restart. I'm not sure it won't happen with the new script since we still I think we are still restarting, but I think since we guarantee in order writes i'm suspecting that ffmpeg closes the .ts file before doing updating the m3u8. With the s3fs upload we can't guarantee the uploads are in order.

[Molkree]

For now this was solved on my end by reading playlist .m3u8 file and processing only files in it.