Multi period for DASH #78

mariocynicys · 2021-06-21T18:38:44Z

Changes:

ConflictingFields error raised when inputs and multiperiod_inputs_list fields are present at the same time in the input_config.yaml.
MissingRequiredExclusiveFields error raised when neither inputs field nor multiperiod_inputs_list field is present.
InputConfig.__init__() pre-checks for the presence or absence of these fields mentioned above and raises error accordingly.
SinglePeriod class added.
controller._append_nodes_for_inputs_list() performs as what controller.start() was performing, controller.start() now calls controller._append_nodes_for_inputs_list() once or multiple time (based on whether the input is multiperiod or not) to append more Transcoder and Packager nodes. #change the name of this method.
In ProcessStatus Running is assigned to a greater value than Finished.
periodconcat_node.py added, it contains a new ThreadedNode that is appended after all the Transcoder and Packager nodes here, it checks whether all the Packager nodes have finished or not every while (3s) in the PeriodConcatNode._thread_single_pass(), if all of them are finished, it starts the period concatenation. For DASH, xml module is used to parse the .mpd files and extract the periods and duration information, for HLS we might use m3u8.

adding period concatenation for dash, no tests are done yet.

mariocynicys · 2021-06-21T19:15:52Z

@joeyparrish
For the last point mentioned above, when doing periods in LIVE, do we need to mention the 'start' attribute for each Period for dash players to play it correctly, or we can just omit it safely.

TheModMaker · 2021-06-21T19:33:21Z

run_end_to_end_tests.py

@@ -189,7 +189,7 @@ def stop():
  resp = createCrossOriginResponse()
  if controller is not None:
    # Check status to see if one of the processes exited.
-    if controller.check_status() == node_base.ProcessStatus.Errored:
+    if controller.check_status() == node_base.ProcessStatus.ERRORED:


This kind of change would be great as its own change. It would be small and easy to review.

This kind of change would be great as its own change. It would be small and easy to review.

I agree completely. Please send these non-functional or cosmetic changes separately.

TheModMaker · 2021-06-21T19:35:03Z

streamer/bitrate_configuration.py

@@ -368,7 +368,8 @@ class BitrateConfig(configuration.Base):
  """

  video_resolutions = configuration.Field(
-      Dict[str, VideoResolution], default=DEFAULT_VIDEO_RESOLUTIONS).cast()
+      Dict[str, VideoResolution],
+      default=DEFAULT_VIDEO_RESOLUTIONS).cast()


I would avoid whitespace-only changes, especially in larger changes like these. If you think this is better, I'd suggest having all these style changes as their own change.

sorry for that.

TheModMaker · 2021-06-21T19:35:22Z

streamer/cloud_node.py

-
-Additional output from gsutil:
-  {}""".format(bucket_url, status.stderr)
+      message = "Unable to write to cloud storage URL: {}\n\nPlease double-check that the URL is correct, that you are signed into the\nGoogle Cloud SDK or Amazon AWS CLI, and that you have access to the\ndestination bucket.\n\nAdditional output from gsutil:\n  {}".format(bucket_url, status.stderr)


Please revert. Limit lines to 80 characters.

TheModMaker · 2021-06-21T19:40:28Z

streamer/configuration.py

+  # NOTE: ignore[override] is needed to suppress the following the mypy
+  # error: "This violates the Liskov substitution principle,
+  # See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides"
+  def __eq__(self, other: 'RuntimeMap') -> bool: # type: ignore[override]


I don't think this change is correct. We could compare with any type, so it should stay Any.

Well, that Any must have a _sortable_properties method for it to be comparable with self: RuntimeMap, so that Any is a RuntimeMap.
These magical methods are used to compare objects of type VideoResolution & AudioChannelLayout which are RuntimeMaps.

I think this change is fine, but not obviously related to multiperiod support. We'd prefer this kind of unrelated cleanup separately, because it makes reviews easier and faster when a PR is focused on one thing.

TheModMaker · 2021-06-21T19:51:16Z

streamer/controller_node.py

+      if isinstance(node, TranscoderNode):
+        status = self._nodes[i+1].check_status()
+      elif isinstance(node, PackagerNode):
+        status = self._nodes[i-1].check_status()


This required ordering is fragile and confusing. If you want to get a specific node or its status, I'd suggest storing the nodes in separate fields instead of in a list. Something like self._packager_node.

It's no more the case that there is only one transcoder and packager node, i think putting each type in a separate list will be less confusing, right?

I don't think this change is necessary. stop() is only called on the controller when we're stopping everything. There are only two cases for this:

check_status() on controller is Finished, meaning everything is finished, so the pairs don't matter because all status are the same

check_status() on controller is Running or Errored, meaning something failed or we shut down early, so the pairs don't matter because we want to force-kill anything still running

TheModMaker · 2021-06-21T20:10:15Z

streamer/periodconcat_node.py

+
+        # If all the time containers were zero, write zero seconds
+        if self.dur == 'PT':
+          self.dur += '0S'


First, you can have 0 in these fields, so you can just have every value. Second, since we target Python3, you can use format strings for this.

return f'P{self.PY}Y{self.PM}M{self.PW}W{self.PD}DT{self.TH}H{self.TM}M{self.TS}S'

yes i just wanted the output to look cleaner :).
i looked up for f strings and they where supported for >= python3.6, is that ok?

TheModMaker · 2021-06-21T20:13:48Z

streamer/periodconcat_node.py

+          self.dur += '0S'
+
+      def parse(self, duration: str) -> None:
+        """Parses duration string into different containers."""


This whole thing can be done with regexes. See:

https://github.com/google/shaka-player/blob/0146ccd69d84601fe160d608e6954bfd430360c4/lib/util/xml_utils.js#L193-L208

TheModMaker · 2021-06-21T20:14:51Z

streamer/periodconcat_node.py

+                   duration: Optional[str] =None):
+        """If given a duration, it uses it to set the time containers."""
+
+        self.PY: int = Y # Years


I would suggest full names for these. e.g. years, months.

TheModMaker · 2021-06-21T20:16:55Z

streamer/periodconcat_node.py

+      """A helper class represeting the iso8601:2004 duration format."""
+
+      def __init__(self, Y=0, PM=0, W=0, D=0, H=0, TM=0, S=0,
+                   duration: Optional[str] =None):


Instead of having both the string and number versions, I would suggest a static method to parse from string.

@staticmethod def parse(value: str) -> ISO8601: ... return ISO8601(...)

TheModMaker · 2021-06-21T20:22:04Z

streamer/periodconcat_node.py

+from streamer.packager_node import PackagerNode
+from streamer.input_configuration import Input
+from streamer.pipeline_configuration import PipelineConfig, ManifestFormat, StreamingMode
+from typing import List, Tuple, Optional


Imports go in sections, each alphabatized. Sections go: __future__, system, third-party, ours. So it should be:

import os from typing import List, Tuple, Optional from xml.etree import ElementTree from streamer import __version__ ...

mariocynicys · 2021-06-22T14:53:46Z

streamer/cloud_node.py

-
-Additional output from gsutil:
-  {}""".format(bucket_url, status.stderr)
+      message = ("Unable to write to cloud storage URL: {}\n\n"


@TheModMaker
Would that be ok?, i am trying to keep the indentation because it causes visual problems in vscode and i think other text editors as well.

Would that be ok?, i am trying to keep the indentation because it causes visual problems in vscode and i think other text editors as well.

Yes, this okay. But please make non-functional or cosmetic-only changes in a separate PR.

In fact, I don't see any of the changes in this file as being related to multi-period. (At least, not obviously to me.)
You've renamed a member variable, reformatted this error message, and changed the way we read from a file while waiting to upload.

I'm not opposed to all of those changes, but they are unrelated and make the review of this PR more difficult.

joeyparrish · 2021-06-22T17:14:20Z

config_files/input_multiperiod.yaml

+
+multiperiod_inputs_list:
+
+  # List of inputs.


To help the reader understand the config at a glance, would you please add to this comment to say that this list of inputs is the first period? And on the second list below, add a comment saying that these inputs are for the second period?

joeyparrish · 2021-06-22T17:14:44Z

run_end_to_end_tests.py

@@ -189,7 +189,7 @@ def stop():
  resp = createCrossOriginResponse()
  if controller is not None:
    # Check status to see if one of the processes exited.
-    if controller.check_status() == node_base.ProcessStatus.Errored:
+    if controller.check_status() == node_base.ProcessStatus.ERRORED:


This kind of change would be great as its own change. It would be small and easy to review.

I agree completely. Please send these non-functional or cosmetic changes separately.

joeyparrish · 2021-06-22T17:19:16Z

streamer/cloud_node.py

-
-Additional output from gsutil:
-  {}""".format(bucket_url, status.stderr)
+      message = ("Unable to write to cloud storage URL: {}\n\n"


Would that be ok?, i am trying to keep the indentation because it causes visual problems in vscode and i think other text editors as well.

Yes, this okay. But please make non-functional or cosmetic-only changes in a separate PR.

In fact, I don't see any of the changes in this file as being related to multi-period. (At least, not obviously to me.)
You've renamed a member variable, reformatted this error message, and changed the way we read from a file while waiting to upload.

I'm not opposed to all of those changes, but they are unrelated and make the review of this PR more difficult.

joeyparrish · 2021-06-22T17:30:53Z

streamer/cloud_node.py

      with open(manifest_path, 'rb') as f:
-        contents = f.read()
-
-      while (not contents and
-             self.check_status() == ProcessStatus.Running):
-        time.sleep(0.1)
-
-        with open(manifest_path, 'rb') as f:
+        while (self.check_status() == ProcessStatus.RUNNING):
+          f.seek(0)
          contents = f.read()
+          if contents:
+            break
+          time.sleep(0.1)


Before, we would close and reopen the file. Now, you have it keeping the file handle open while polling for content. There is a subtle difference between these which may cause issues, depending on what Shaka Packager does with the files it's writing. It's possible to write them in a way that breaks with your changes. To demonstrate, run these at roughly the same time, in this order:

App 1

import os import time print('Writing!') f = open('foo.mpd', 'w') # Creates an inode on disk that maps to this FS name f.write('old data') f.close() print('Sleeping!') time.sleep(10) # Let app 2 open the file now print('Deleting and re-writing!') os.unlink('foo.mpd') # Unlinks the old inode from the FS f = open('foo.mpd', 'w') # Creates a new inode on disk mapping to the same FS name f.write('new data') f.close() print('Done!')

App 2

import time print('Sleeping!') time.sleep(3) # Let app 1 write the file print('Reading!') f = open('foo.mpd', 'r') # Open the original inode data = f.read() while data != 'new data': # This loop will never terminate, because you're re-reading # the same inode that app 1 unlinked from the file system. # The inode persists on disk until all handles are closed. # foo.mpd, if opened again as a new handle, would reference # a different inode and you would see the new data. print('Trying again!') time.sleep(1) f.seek(0) data = f.read() print('Done!')

Finally, you wouldn't know this, but we're working with another contributor on a plan to remove cloud_node completely. We're replacing it with a new HTTP upload capability in Shaka Packager, combined with an authentication proxy, such that Packager can write files directly to cloud storage via HTTP without going through the filesystem, and without polling here. So changes to this class are not going to be relevant for much longer.

joeyparrish · 2021-06-22T17:42:51Z

streamer/configuration.py

@@ -223,7 +232,7 @@ def get_subtypes(

    underlying = Field.get_underlying_type(type)
    if underlying is dict:
-      return cast(Tuple[Optional[Type], Optional[Type]], args)
+      return (args[0], args[1])


I have mypy 0.812 on my workstation as well.

But I found the difference. return args is an error, because args is Tuple[Any, ...], and the return value must be a 2-tuple. So returning those two specific args seems to work without the cast.

I dug a little deeper and found that args[0] and args[1] are seen as Any, and a Tuple[Any, Any] seems to be accepted for a return type of Tuple[Optional[Type], Optional[Type]], presumably converting Any to Optional[Type].

So that explains how it's working... but I don't see why you've made this change at all. How does it relate to the point of the PR?

joeyparrish · 2021-06-22T18:34:43Z

streamer/periodconcat_node.py

+    for node in all_nodes:
+      if isinstance(node, TranscoderNode):
+        self._transcoder_nodes.append(node)
+        self._period_inputs_list.append(node._inputs)


It's poor form to access a private member of another class. Also, I don't see this being used anywhere.

I thought i might need it for the HLS, but i will remove it for now.

joeyparrish · 2021-06-22T18:35:42Z

streamer/periodconcat_node.py

+          "PeriodConcatNode: 'PackagerNode#{} Errored, Concatenation is stopped.'".format(i)
+        )
+
+    if(ManifestFormat.DASH in self._pipeline_config.manifest_format):


style nit: Don't use parens unless you need to span two lines

joeyparrish · 2021-06-22T18:36:29Z

streamer/periodconcat_node.py

+    """Concatenates multiple single-period DASH manifests into one multi-period DASH manifest."""
+
+    class ISO8601:
+      """A helper class represeting the iso8601:2004 duration format."""


typo: representing

joeyparrish · 2021-06-22T18:39:29Z

streamer/periodconcat_node.py

+    # We will add the time of each period on this container to set that time later
+    # as 'mediaPresentationDuration' for the MPD tag for VOD.
+    # For LIVE we will use this object to determine the start time of the next period.
+    total_time: ISO8601 = ISO8601()


You don't need to do this. If you move the mediaPresentationDuration of each mpd into the Period element's duration attribute, you can omit mediaPresentationDuration from the combined mpd element. Players must be able to handle this, so you can skip all the duration parsing above.

streamer/periodconcat_node.py

joeyparrish · 2021-06-22T18:43:52Z

streamer/periodconcat_node.py

+    # We will start the concatenation once all the PackagerNode(s) are finished.
+    for node in all_nodes:
+      if isinstance(node, TranscoderNode):
+        self._transcoder_nodes.append(node)


This also appears to be unused

joeyparrish · 2021-06-22T18:44:23Z

streamer/periodconcat_node.py

+      Otherwise it will raise an AssertionError."""
+      full_path = ''
+      for tag in args:
+        full_path+='{'+dash_namespace+'}'+tag+'/'


style nit: add spaces around operators

Also, a more pythonic suggestion for this:

full_path = '/'.join([ '{' + dash_namespace + '} + tag for tag in args ]) child_elem = elem.find(full_path)

Finally, are you sure you need the explicit namespace in this helper at all? It might be more resilient not to rely on a hard-coded namespace with "2011" in the name, in case a newer version of the DASH spec changes this in the future.

okay , I rather be reading it from the sample dash file generated by packager.

joeyparrish · 2021-06-22T18:44:38Z

streamer/periodconcat_node.py

+      full_path = ''
+      for tag in args:
+        full_path+='{'+dash_namespace+'}'+tag+'/'
+      child_elem =  elem.find(full_path[:-1])


style nit: remove double space after equals

joeyparrish · 2021-06-22T18:51:32Z

streamer/periodconcat_node.py

+        period.attrib['duration'] = mpd.attrib['mediaPresentationDuration']
+        total_time += ISO8601.parse(period.attrib['duration'])
+      else:
+        # For LIVE, set the start attribute.


Since this only runs when all packagers have completed, I'm not convinced it can work for live at all. I'm okay with restricting it to VOD for now.

I don't think Shaka Streamer's config file approach is well-suited to a truly open-ended multi-period live stream, because a streaming service will generally want to add periods dynamically over time, rather than configuring a fixed number of them upfront.

joeyparrish · 2021-06-22T18:54:09Z

streamer/periodconcat_node.py

+        'AdaptationSet',
+        'Representation',
+        'SegmentTemplate'):
+        path_to_segement = os.path.relpath(packager_node._output_dir, self._output_dir)


It looks like you are updating the segment URLs to point into the subdirectories. You can simplify this greatly by adding BaseURL to the start of each new Period element instead. BaseURLs are resolved across the whole XML hierarchy, so you can turn this:

<Period> <AdaptationSet> <Representation> <BaseURL>foo.mp4</BaseURL>

Into this:

<Period> <BaseURL>subdir/</BaseURL> <AdaptationSet> <Representation> <BaseURL>foo.mp4</BaseURL>

Now you don't need to know or modify all the places in an MPD where a URL could show up.

that's way neater, thanks.

streamer/periodconcat_node.py

joeyparrish · 2021-06-22T19:08:36Z

streamer/periodconcat_node.py

+  def __init__(self,
+               pipeline_config: PipelineConfig,
+               all_nodes: List[NodeBase],
+               outpud_dir: str) -> None:


Typo: output_dir

joeyparrish · 2021-06-22T19:09:51Z

streamer/periodconcat_node.py

+    for i, packager_node in enumerate(self._packager_nodes):
+
+      dash_file_path = os.path.join(
+        packager_node._output_dir,


Make output_dir public on PackagerNode (remove the leading underscore)

joeyparrish · 2021-06-22T19:10:56Z

streamer/controller_node.py

+    for node in self._nodes:
+      status: ProcessStatus = node.check_status()
+      if isinstance(node, (TranscoderNode, PackagerNode, PeriodConcatNode)):
+        values.add(status.value)


Here's a suggestion:

Pass all packager nodes to both PeriodConcatNode and CloudNode constructors.

Add a static method to NodeBase to query the combined status of a list of nodes, based on the max() comprehension.

Make ControllerNode use the new static method here to compute status of all nodes.

PeriodConcatNode can use the new static method to wait for all PackagerNodes to have a Finished status, then combine manifests/playlists, then shut down.

If CloudNode sees Finished across PackagerNodes, it can make one additional upload pass, then shut down.

joeyparrish · 2021-06-22T19:11:59Z

streamer/periodconcat_node.py

+
+  def __init__(self,
+               pipeline_config: PipelineConfig,
+               all_nodes: List[NodeBase],


Instead of all_nodes, could this just be packager_nodes?

joeyparrish

Looks great otherwise. Thanks!

streamer/input_configuration.py

joeyparrish · 2021-06-23T18:48:23Z

streamer/input_configuration.py

+
+    # Because these fields are not marked as required at the class level
+    # , we need to check ourselves that one of them is provided.
+    if dictionary.get('inputs') is None and dictionary.get('multiperiod_inputs_list') is None:


This uses "is None", so empty lists will not be matched. This means this condition would allow:

inputs: []

I would prefer to treat this as missing fields, so let's drop use "is None" here and just use truthiness.

Combined with the above suggestion, we would require exactly one field, and that the exactly one field must be non-empty.

We should add tests for these various cases, too.

I did want such malformed field error or other types of errors to be caught in the Base class constructor instead, that would provide more descriptive error messages about the cause of the error.
But just found out that empty lists between all other types indeed pass the Base class error checking.

Can we add some check in Base._check_and_convert_type that the list passed by the user must not be empty ??

I tried adding this error for the length of the user-inputted list and ran the tests, some resolution fields were given as an empty list, after putting some dummy resolutions inside them, the tests passed.

What do you think about this approach?

I think you're right. We should check for empty lists generally. Possibly empty strings, too. But let's put ~~that~~ those in a separate PR.

I'm thinking of a property on Field that indicates whether empty is acceptable, defaulting to False.

mariocynicys · 2021-06-26T15:20:43Z

tests/tests.js

+
+    await startStreamer(inputConfigDict, pipelineConfigDict);
+    await player.load(manifestUrl);
+    const duration = player.seekRange().end


@joeyparrish should i be using this to determine the time??
also is there any other way to check for this, like some function that tells the number of periods in the manifest?

There is no way to check the number of periods in Shaka v3. In Shaka v2, we kept periods separate, but now we combine them in the player when we parse DASH.

joeyparrish · 2021-06-28T13:31:46Z

tests/tests.js

+  });
+
+  it('fails when neither "inputs" nor "multiperiod_inputs_list" is given', async() =>{
+    const inputConfig = {}


style nit: add a semicolon at the end of this statement

joeyparrish · 2021-06-28T13:32:00Z

tests/tests.js

+    inputConfig.multiperiod_inputs_list = [
+      getBasicInputConfig(),
+      getBasicInputConfig(),
+    ]


style nit: add a semicolon at the end of this statement

joeyparrish · 2021-06-28T13:32:15Z

tests/tests.js

@@ -316,6 +318,38 @@ function errorTests() {
          field_name: 'content_id',
        }));
  });
+
+  it('fails when both "inputs" and "multiperiod_inputs_list" are given', async() =>{


style nit: add a space between => and {

joeyparrish · 2021-06-28T13:32:31Z

tests/tests.js

+        }));
+  });
+
+  it('fails when neither "inputs" nor "multiperiod_inputs_list" is given', async() =>{


style nit: add a space between => and {

joeyparrish · 2021-06-28T13:33:55Z

tests/tests.js

+
+    await startStreamer(inputConfigDict, pipelineConfigDict);
+    await player.load(manifestUrl);
+    const duration = player.seekRange().end


There is no way to check the number of periods in Shaka v3. In Shaka v2, we kept periods separate, but now we combine them in the player when we parse DASH.

joeyparrish

Looks pretty good! Just a few nits, mostly style.

joeyparrish · 2021-06-28T13:34:24Z

tests/tests.js

+
+    await startStreamer(inputConfigDict, pipelineConfigDict);
+    await player.load(manifestUrl);
+    const duration = player.getManifest().presentationTimeline.getDuration()


Here you can use video.duration.

joeyparrish · 2021-06-28T13:34:42Z

tests/tests.js

+
+    // Since we processed only 0:01s, the total duration shoud be 2s.
+    // Be more tolerant with float comparison, (D > 1.9 * length) instead of (D == 2 * length).
+    expect(duration).toBeGreaterThan(1.9 * 1)


I think you should drop the * 1 part.

joeyparrish

Well done! Congrats!

This (manually) reverts the non-refactoring parts of commit e6152fd. For the v0.4.0 release, we will remove the multi-period DASH feature. It will then be restored and released together with multi-period HLS in v0.5.0. Change-Id: I450ca7b78d93f781d85a9561a6b747e4389d0126

This reverts commit ea9f1f2, restoring the multi-period DASH feature in anticipation of v0.5.0. Change-Id: Ib43fd6d8d3dc1a025ffd40bfbe0834951ddce169

mariocynicys added 4 commits June 9, 2021 09:04

multi-input & multi-period concat proposal

f3d5d93

doc string for inputs_list and single_inputs

d562720

adding period concatenation for dash

8fa9720

adding period concatenation for dash, no tests are done yet.

adding period concat to LIVE for DASH

e806d66

TheModMaker reviewed Jun 21, 2021

View reviewed changes

using regex for iso8601 duration parsing

f239954

mariocynicys commented Jun 22, 2021

View reviewed changes

joeyparrish requested changes Jun 22, 2021

View reviewed changes

mariocynicys added 8 commits June 23, 2021 09:10

reverting unrelated changes

0ce17f6

limiting to 2 conflicting/required-exclusive fields

3f0aaef

using BaseURL to specify the nested period directories

71e50b0

minor edits

6b51750

minor edits2

5a91827

minor edits3

9c5d4c1

minor edits4

e9cc4da

so to not to stuck Running if one packager node errors

17ca8c2

joeyparrish requested changes Jun 23, 2021

View reviewed changes

adding tests

95d7510

mariocynicys commented Jun 26, 2021

View reviewed changes

using getDuration() instead of seekRange()

d1dd0cf

joeyparrish reviewed Jun 28, 2021

View reviewed changes

joeyparrish requested changes Jun 28, 2021

View reviewed changes

style code

11ea3a2

joeyparrish approved these changes Jun 28, 2021

View reviewed changes

joeyparrish merged commit e6152fd into shaka-project:master Jun 28, 2021

shaka-bot pushed a commit that referenced this pull request Aug 26, 2021

Restore "Multi period for DASH (#78)"

923ceb9

This reverts commit ea9f1f2, restoring the multi-period DASH feature in anticipation of v0.5.0. Change-Id: Ib43fd6d8d3dc1a025ffd40bfbe0834951ddce169

github-actions bot added the status: archived Archived and locked; will not be updated label Jul 25, 2023

github-actions bot locked as resolved and limited conversation to collaborators Jul 25, 2023

Multi period for DASH #78

Multi period for DASH #78

Conversation

mariocynicys commented Jun 21, 2021 • edited Loading

Changes:

mariocynicys commented Jun 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

App 1

App 2

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joeyparrish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariocynicys Jun 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joeyparrish Jun 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joeyparrish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joeyparrish left a comment

Choose a reason for hiding this comment

mariocynicys commented Jun 21, 2021 •

edited

Loading

mariocynicys Jun 24, 2021 •

edited

Loading

joeyparrish Jun 28, 2021 •

edited

Loading