Adds Mastodon social network #10

jpgoldberg · 2023-11-27T21:30:02Z

The purpose of this PR is to add Mastodon as a social network, resolving issue #11.

Unlike the other social networks supported, there is no fixed hostname for the URL. Instead the hostname for the URL must be constructed from the full Mastodon address.

That is, a mastodon address may look like "[email protected]" which should then have the associated url be "https://social.example/@a_tooter"

Why is this PR more complicated than one might expect?

This really seems like it would be a simple fix.

Mastodon specs and DNS hostnames

Unfortunately there is no formal specification for these things. I am grateful @ThisIsMissEm for pointing me to the relevant lines of Mastodon reference implementation.

Unfortunately, what I glean from that would allow syntactically invalid hostnames. And so I included stricter validation of hostnames. Perhaps I failed to understand the Pydantic and dnspython documentation packages, but I found no obvious way to validate hostnames properly. So I resorted to a stack exchange answer.

I wasn't sure where to put my Mastodon address handling methods, so they are static methods in data_model.Connections.

Getting things to run

I need to tinker with pyproject.toml to get even the original to build on my system. I do not believe that I've done any harm, but testers should make sure that I haven't messed up build or packaging process.

I also added tests, and to make it easier for me to run tests tinkered with tests/test_data_model.py. Again, I don't think I did anything that should interfere with other testing.

but there are tests

sinaatalay · 2023-11-27T21:59:46Z

Thank you! I will review it thoroughly this weekend.

ThisIsMissEm · 2023-11-28T07:01:43Z

Whilst I cannot review this, @jpgoldberg I think what you'll actually want to do is make a webfinger request:

https://host.example/.well-known/webfinger?resource=acct:[email protected]

this'll either succeed or fail, and gives the URLs to any such profile.

e.g., mine:

{
  "subject" : "acct:[email protected]",
  "aliases" : [
    "https://hachyderm.io/@thisismissem",
    "https://hachyderm.io/users/thisismissem"
  ],
  "links" : [
    {
      "rel" : "http://webfinger.net/rel/profile-page",
      "type" : "text/html",
      "href" : "https://hachyderm.io/@thisismissem"
    },
    {
      "rel" : "self",
      "type" : "application/activity+json",
      "href" : "https://hachyderm.io/users/thisismissem"
    },
    {
      "rel" : "http://ostatus.org/schema/1.0/subscribe",
      "template" : "https://hachyderm.io/authorize_interaction?uri={uri}"
    },
    {
      "rel" : "http://webfinger.net/rel/avatar",
      "type" : "image/jpeg",
      "href" : "https://media.hachyderm.io/accounts/avatars/109/296/581/037/186/439/original/b55ac3f3a27e171e.jpeg"
    }
  ]
}

jpgoldberg · 2023-12-02T18:58:18Z

Whilst I cannot review this, @jpgoldberg I think what you'll actually want to do is make a webfinger request:
https://host.example/.well-known/webfinger?resource=acct:[email protected]

Oh, that is a very cool thing. I did not know about it. (As you see, I really know very little about the protocols associated with Mastodon.)

But I do not want to take that approach for two (related) reasons

I do not want to make a query to https://host.example/ before knowing that "host.example is syntactically valid. Attempting to use malformed data to see if it breaks as a way to test for its validity is the road to many security bugs. Instead, we should always check that something is syntactically valid before attempting to use it. Making a network call with potentially malformed data is a use, even if the use is intended as a validity check.
I would like to be able to construct the CV without a network connection. The CV should build off-line.

I really am a stickler for checking syntactic validity before any use. And, as you see, I do not pass up the opportunity to rant/lecture about it. Checking strict syntactic validity before any use is a really good habit to get into, and it prevents security issues that might otherwise arise years later.

sinaatalay

Hi, this looks great. I would like to merge it. Before the merge, it would be great if we could change a few small things I mentioned in the comments. Let me know if there is something you disagree with. I might have missed something.

sinaatalay · 2023-12-03T16:38:34Z

rendercv/data_model.py

+    @staticmethod
+    def is_valid_hostname(hostname: str) -> bool:
+        """Is hostname a valid hostname by RFCs 952 and 1123"""
+
+        # slightly modified from
+        # https://stackoverflow.com/a/33214423/1304076
+        if hostname[-1] == ".":
+            # strip exactly one dot from the right, if present
+            hostname = hostname[:-1]
+        if len(hostname) > 253:
+            return False
+
+        labels = hostname.split(".")
+
+        # the last label must be not all-numeric
+        if re.match(r"[0-9]+$", labels[-1]):
+            return False
+
+        # labels cannot begin with a hyphen
+        # labels must have at least character
+        # labels may not have more than 63 characters
+        allowed = re.compile(r"(?!-)[a-z0-9-]{1,63}(?<!-)$", re.IGNORECASE)
+        return all(allowed.match(label) for label in labels)
+


If we need to verify Mastodon usernames, we should also verify other networks. Is it necessary to validate usernames? Moreover, if we want to do a validation, we should use the @field_validator decorator of Pydantic. This will help exception handling since we will know it's a validation error.

There are two separate suggestions. The reason that Mastodon needs special validation as opposed to, say, LinkedIn is that we know that know that www.linkedin.com ia syntactically valid hostname. We aren't going to create a malicious PDF by using it.

There may be value in syntactically validating other usernames. But I do think it would be a mistake to actually perform any network calls while running RenderCV. So the validation is only syntactic.

I very much suspect you are correct about @field_validator, but the be perfectly honest, I don't fully grok what Pydantic is doing. However, I am happy to learn. So I will work on this.

Actually, (if you haven’t started working on it), we can keep it like this for now because the @field_validator check should be done on the SocialNetwork class since the Connection class is not used by the users directly. But if we move it to SocialNetwork, we will need to do something about other social networks, too. Therefore, we can address this validation problem later. Whatever you prefer.

sinaatalay · 2023-12-03T16:39:05Z

rendercv/data_model.py

+    @staticmethod
+    def MastodonUname2Url(address: str) -> Optional[HttpUrl]:
+        """returns profile url from a mastodon user address.
+
+        Args:
+            address (str): A Mastodon user address. E.g., "[email protected]"
+
+        Returns:
+            A pydantic HttpUrl object with the https URL for the user profile
+
+        Example:
+            ```
+            url = MastodonUname2Url("[email protected]")
+            assert(url == HttpUrl(http://social.example/@user))
+            ```
+
+        Exceptions:
+            ValueError if the address is malformed.
+            Note that well-formed addresses should never yield
+            syntactically invalid URLs.
+        """
+
+        # The closest thing to a formal spec of Mastodon usernames
+        # where these regular expressions from a (reference?)
+        # implementation
+        #
+        # https://github.com/mastodon/mastodon/blob/f1657e6d6275384c199956e8872115fdcec600b0/app/models/account.rb#L68
+        #
+        # USERNAME_RE   = /[a-z0-9_]+([a-z0-9_.-]+[a-z0-9_]+)?/i
+        # MENTION_RE    = %r{(?<![=/[:word:]])@((#{USERNAME_RE})(?:@[[:word:].-]+[[:word:]]+)?)}i
+        #
+        # `[[:word:]]` in Ruby includes lots of things that could never be in a # domain name. As my intent here is to construct an HTTPS URL,
+        # What we need are valid hostnames,
+        # and so need to satisfy the constraints of RFC 952 and and 1123.
+
+        pattern = re.compile(
+            r"""
+            ^\s*                    # ignore leading spaces
+            @?                      # Optional @ prefix
+            (?P<uname>[a-z0-9_]+([a-z0-9_.-]+[a-z0-9_]+)?)  # username part
+            @                       # separator
+            (?P<domain>[a-z0-9]+([a-z0-9.-]+)?) # domain part
+            \s*$                    # ignore trailing whitespace
+        """,
+            re.VERBOSE | re.IGNORECASE,
+        )
+
+        m = pattern.match(address)
+        if m is None:
+            raise ValueError("Invalid mastodon address")
+        uname = m.group("uname")
+        domain = m.group("domain")
+
+        # the domain part of pattern allows some things that are not
+        # valid names. So we run a stricter check
+        if not Connection.is_valid_hostname(domain):
+            raise ValueError("Invalid hostname in mastodon address")
+
+        url = HttpUrl(f"https://{domain}/@{uname}")
+        return url


I don't see this function being used anywhere other than the url method. Maybe we can move the contents inside the url method for clarity?

We have different opinions about "clarity", but it is your project, so I will follow your suggestion.

sinaatalay · 2023-12-03T16:39:32Z

tests/test_data_model.py

+    def test_mastodon_parsing(self):
+        mastodon_name = "[email protected]"
+        expected = HttpUrl("https://example.exchange/@a_tooter")
+        result = data_model.Connection.MastodonUname2Url(mastodon_name)
+        with self.subTest("Without '@' prefix"):
+            self.assertEqual(result, expected)
+
+        mastodon_name = "@[email protected]"
+        expected = HttpUrl("https://example.exchange/@a_tooter")
+        result = data_model.Connection.MastodonUname2Url(mastodon_name)
+        with self.subTest("With '@' prefix"):
+            self.assertEqual(result, expected)
+
+        mastodon_name = "@too@many@symbols"
+        with self.subTest("Too many '@' symbols"):
+            with self.assertRaises(ValueError):
+                data_model.Connection.MastodonUname2Url(mastodon_name)
+
+        mastodon_name = "@not_enough_at_symbols"
+        with self.subTest("Missing '@' separator"):
+            with self.assertRaises(ValueError):
+                data_model.Connection.MastodonUname2Url(mastodon_name)
+
+        mastodon_name = "user@bad_domain.example"
+        with self.subTest("Underscore in domain portion"):
+            with self.assertRaises(ValueError):
+                data_model.Connection.MastodonUname2Url(mastodon_name)
+
+        mastodon_name = "[email protected]"
+        with self.subTest("All digit TLD"):
+            with self.assertRaises(ValueError):
+                data_model.Connection.MastodonUname2Url(mastodon_name)
+
+        mastodon_name = "[email protected]."
+        expected = HttpUrl("https://example.exchange./@a_tooter")
+        result = data_model.Connection.MastodonUname2Url(mastodon_name)
+        with self.subTest("With FQDN root '.'"):
+            self.assertEqual(result, expected)


This makes sense, and I will consider creating a new test for each social network. However, instead of using the MastodonUname2Url method directly, we can create a Connection object and check its url field (since we plan to move MastodonUname2Url to inside the url method.)

Makes sense. Note that we can use Pydantic's HttpUrl class for the urls created for the other social networks.

Indeed, that offers use a simpler way of validating Mastodon domain names, as Pydantic does (mostly) check that the host part of a URL is syntactically valid.

sinaatalay · 2024-01-14T18:06:28Z

Hello, if you're not working on this PR, I would like to merge it as it is.

jpgoldberg and others added 30 commits November 26, 2023 15:37

WIP: start on mastodon

222026d

git ignore my CV

44ffc0f

WIP: more mast parsing

2d315dc

gitingnor venv stuff

c4460bd

unittest

513d507

Needed to build

76de600

! some tests fail

03b97cd

but there are tests

\w is not whitespace. Derp

c95625f

Mastodon in connection tests

125a1bb

update version tag

d9e374b

remove requirements.txt

37dc7c8

fix YYYY date issue (sinaatalay#5)

830dcf3

update CI.yaml

c38d293

remove spell checking feature

fc6a3f7

fix escape characters test in test_data_model.py

682bce0

add more sections

17a1122

improve classic theme

94d2f56

fix the reference latex file

1af54eb

support markdown_to_latex in more areas

7a3be88

fix run_rendercv.py

243f7b4

update .gitignore

69ac7fd

add lastpage package

42ee8d9

run latex twice for cross-referencing

977a4a5

make the highlighted author bold and italic

9eb0467

add page numbering

5c76e9d

suppress lualatex output

08b262d

improve the classic theme

71a02ab

update tests

1df1fdd

fix tests

c9b3b0f

create a reference pdf file

a74cef7

sinaatalay and others added 13 commits November 27, 2023 14:53

fix an encoding bug

aa8d5e8

allow division of section contents

c0bddbc

don't escape backslashes

0c83a37

fix tests

237d791

add text alignment options

711a9a8

update reference files

ce34076

add header margin options

bd3cb64

add header_font_size option

9c3073d

rename Code Documentation to API Reference

af50c7c

upload licenses for 3rd party fonts (sinaatalay#7)

085d222

don't allow line break between first and last name in publications

68c1640

add university_projects field

2b7555e

Merge branch 'main' into jpg/mastodon

a5babe5

jpgoldberg mentioned this pull request Nov 27, 2023

Support Mastodon as a Social Connections #11

Closed

Ruff whitespace linting

161b0d6

Merge branch 'main' into jpg/mastodon

7283c9d

jpgoldberg mentioned this pull request Dec 2, 2023

Provide something like is_dnsname(s: str) -> bool and is_fqdn(s: str) -> bool rthalley/dnspython#1019

Closed

jpgoldberg added 2 commits December 3, 2023 01:09

Merge branch 'main' into jpg/mastodon

795a98d

hostname validity (not domain name)

08902b7

sinaatalay reviewed Dec 3, 2023

View reviewed changes

sinaatalay and others added 5 commits December 3, 2023 18:09

update CI

e072721

improve issue templates

60672db

gitignore some jpgoldberg files

4ac7e75

doc mention mastodon

b26e232

improve data_model tests

2454cc7

sinaatalay merged commit 5456a8b into sinaatalay:main Jan 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds Mastodon social network #10

Adds Mastodon social network #10

jpgoldberg commented Nov 27, 2023 •

edited

Loading

sinaatalay commented Nov 27, 2023

ThisIsMissEm commented Nov 28, 2023

jpgoldberg commented Dec 2, 2023

sinaatalay left a comment

sinaatalay Dec 3, 2023 •

edited

Loading

jpgoldberg Dec 5, 2023

sinaatalay Dec 7, 2023

sinaatalay Dec 3, 2023

jpgoldberg Dec 5, 2023

sinaatalay Dec 3, 2023

jpgoldberg Dec 5, 2023

sinaatalay commented Jan 14, 2024

Adds Mastodon social network #10

Adds Mastodon social network #10

Conversation

jpgoldberg commented Nov 27, 2023 • edited Loading

Why is this PR more complicated than one might expect?

Mastodon specs and DNS hostnames

Getting things to run

sinaatalay commented Nov 27, 2023

ThisIsMissEm commented Nov 28, 2023

jpgoldberg commented Dec 2, 2023

sinaatalay left a comment

Choose a reason for hiding this comment

sinaatalay Dec 3, 2023 • edited Loading

Choose a reason for hiding this comment

jpgoldberg Dec 5, 2023

Choose a reason for hiding this comment

sinaatalay Dec 7, 2023

Choose a reason for hiding this comment

sinaatalay Dec 3, 2023

Choose a reason for hiding this comment

jpgoldberg Dec 5, 2023

Choose a reason for hiding this comment

sinaatalay Dec 3, 2023

Choose a reason for hiding this comment

jpgoldberg Dec 5, 2023

Choose a reason for hiding this comment

sinaatalay commented Jan 14, 2024

jpgoldberg commented Nov 27, 2023 •

edited

Loading

sinaatalay Dec 3, 2023 •

edited

Loading