Task/reference sitemap when crawling frontend /cdd 2260

[A-Ashiq]

Description

This PR includes the following:

• Traverses the frontend by making use of the sitemap instead of trying to build the URLs ourselves. Which we had to do originally before the sitemap was in place

Fixes #CDD-2260

---

Type of change

Please select the options that are relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Tech debt item (this is focused solely on addressing any relevant technical debt)

---

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests at the right levels to prove my change is effective
• I have added screenshots or screen grabs where appropriate
• I have added docstrings in the correct style (google)

[sonarcloud]

Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarCloud