[FEAT]: Allow analysis of all branches

[emil-wire]

Request Description

Hi guys,

thanks for your project! I'm currently testing it out and was wondering if you would consider adding a feature that allows for analysis of PRs for all branches and not just on merge for the default branch?
Catching vulnerable dependencies early on would be a very valuable feature. Of course resource consumption is something to keep in mind.

Additional Information

No response

[heubeck]

Hi @emil-wire,

great to hear from you.
Yes, of cause we can enable an analysis for any PR.

Would you require any base branch, or prefer a pattern to configure base branches of PRs that should be analyzed?

We could even get more open and make the branches to be analyzed configurable - what would result in one project per branch imported in dependency-track.

WDYT?

[emil-wire]

First: thanks for being open to this!
For the matter itself: hmm, really good question! I'd like to have every PR analyzed and a per branch project in dependency-track - where the parent project is the default branch.
Since I haven't been able to run technolinator successfully yet, could you tell me if you transmit additional metadata to dependency-track? Like versions, tags, etc.?

[heubeck]

First: thanks for being open to this!

always. to everything ;)

For the matter itself: hmm, really good question! I'd like to have every PR analyzed and a per branch project in dependency-track - where the parent project is the default branch.

interesting idea, need to test around a bit, but that's definite valuable.

Since I haven't been able to run technolinator successfully yet, could you tell me if you transmit additional metadata to dependency-track? Like versions, tags, etc.?

Yes, we do:
dtrack:

github:

atm, the (default) branch name is used as version in dtrack, what makes it easy to have any branch as separate entry in dtrack - but that's because of the leak of a good idea, how to version properly. maybe we can stay with this, or use also labels for detected latest versions...

[heubeck]

Note: There should be a cleanup of dependency-track projects from branches that got deleted.

We've a similar issue open for deleted or archived repos, guess we can implement it in a way that fits both use-cases: #96

• don't forget about that ;)

[emil-wire]

always. to everything ;)

awesome :)

Maybe one could use the commit hash as versioning information? Within our org, that would be the lowest common denominator.

Note: There should be a cleanup of dependency-track projects from branches that got deleted.
absolutely. but I think that could be achievable with a little scripting from our side :)

[heubeck]

Maybe one could use the commit hash as versioning information? Within our org, that would be the lowest common denominator.

ok, so always put a new version to dtrack with every commit? would you disable all previous version in dtrack with a new one?

I'm always thinking about something like: listening on new GH releases and update the according dtrack project with that version - not sure how practical that would be...

[emil-wire]

would you disable all previous version in dtrack with a new one?

I don't think I would tbh. But I'm also not 100% sure how to integrate that into our workflow... Do you have an idea?

[heubeck]

Can you briefly describe your workflow, @emil-wire ?

Is there a verification about the dependency-track scan results included somewhere (and if: how)?

I'm not quite confident, that dependency-track will be able to deal with that high amount of projects, if a new one is created for each commit (even if only on default branch), but we need to test out.
Maybe that objection also disappears with project hyades that's build to scale out to a really high number of projects.

Just to sort it out, what are your most important demands for now, @emil-wire, did I recap it correct?:

1. analyzing every PR, no matter of the base branch
2. BOM creation and upload to dtrack for every branch
3. inactivation of dtrack projects on deletion of branches (and archiving/deletion of entire repos)
4. further dtrack project versioning strategies - tbd.

That's also from easy to harder, and maybe we get some nice idea about the versioning, when we realized 1-3 ;)

[emil-wire]

I don't have demands, only wishes :) Not my place to demand stuff from people who have made my life already much easier (again, can't thank you enough!). But 1. and 2. on the list are spot on.

Hyades sounds amazing, and very sensible. Can't wait for it to be released

Is there a verification about the dependency-track scan results included somewhere (and if: how)?

Verification: currently manual. I triage the boms and am looking for critical and highs only and after validation, create tickets.

we have the usual gitflow at Wire (see https://github.com/wireapp/wire-webapp as an example).
Because we have so many different technologies, environments and programming languages, no commercial tool out there covers our needs, not even 60%. This is where technolinator and dependency track come in. I'd like every PR to any branch to be analyzed because this is where new features come in or significant changes to features are being made and I'd like our devs to be aware of outdated dependencies early on in the development process and fix them during dev - classic SSDLC stuff.

I don't think, that we need to store the sbom for each and every branch in the long term and ideally (once we operationalize this stack across all of our repos and techstack) a simple PR decoration giving the devs feedback would be enough. But we're not there yet, because there is a LOT of heterogeneity in our dev process (I'm tackling this, but it takes time...) - there isn't a one size fits all solution currently. But, because some of our institutional customers lag a couple of versions behind our main branches, there's a need to have a historic record of some kind somewhere and provide a fix on a path of least resistance. Since github is pretty public, I can't go into more details here, but we can setup a call if you'd like and get to know each other :)

3. inactivation of dtrack projects on deletion of branches (and archiving/deletion of entire repos)

deletion/inactivation could be handled by me building some kind of script that runs once every week or so. no need for you to worry about that, I think the complexity of technolinator would explode and I'm sure most orgs who'd adopt this feature would have some very specific requirements that would be better served by building their own automation around dependency track.

4. further dtrack project versioning strategies - tbd.

After thinking about this some more, I can see a good way to solve this for us. Each time we cut a release on github, run an action that looks at the PRs that made it into the release, get the bom for that release, attach it to the release artifacts and remove all versions from dtrack that are tied to non release PRs. Sounds simple enough to automate and again highly specific to each org, where I wouldn't want to burden technolinator with that task.

What do you think?

[heubeck]

Thx for your thoughts and insights, @emil-wire

After thinking about this some more, I can see a good way to solve this for us. Each time we cut a release on github, run an action that looks at the PRs that made it into the release, get the bom for that release, attach it to the release artifacts and remove all versions from dtrack that are tied to non release PRs. Sounds simple enough to automate and again highly specific to each org, where I wouldn't want to burden technolinator with that task.

That's an idea we also discussed for version handling using technolinator. react on "onRelease" event in the GH app, and either create/upload a new bom of that commit, versioned with the git tag name, or label or re-version (if that's) possible the according dtrack project.

Will realize something when hitting topic 4 of that list, wanting this feature myself for quite some time ;)

---

are you located in Switzerland, @emil-wire?
Would you like to elaborate on your work or workflows or technologies or learnings maybe on a conference that I'm part of?: https://www.cloudland.org/en/speakers/

[emil-wire]

Wires development company is located in Berlin, as am I.

Regarding the conference: I'm a security engineer and am on a spree to shift security left at Wire. Generally building out a somewhat automated security program here of which dependency tracking is a priority. Not sure if I have something to contribute to such a conference as currently, we're implementing the basics here...

[heubeck]

ok.

I'm a security engineer and am on a spree to shift security left at Wire. 
Generally building out a somewhat automated security program here of which dependency tracking is a priority.

sounds already like a great abstract.

just think about, we're looking for "user stories" of any kind, so how to shift security left from someone actually doing it is much more desired than from some consultant who just always talks about ;)

[heubeck]

1. analyzing every PR, no matter of the base branch

🤣 that's already working, there's no check on the base branch of a PR, @emil-wire - would have bet my ass that's different.
#317

[heubeck]

2. BOM creation and upload to dtrack for every branch

atm, this would trigger a parallel bom creation, if there are PRs with the branch to be analyzed as head.
That's fine for a first quick win, but we should optimize that.

Edit: ok. will refactor technolinator to only react on onPush events, and query GH for pull-requests with the push target as head branch. that's a nice simplification in general.

Edit2: onPR is still needed, but need to check, whether there's a analysis of it's head commit already running from a onPush.

[emil-wire]

sounds already like a great abstract.

just think about, we're looking for "user stories" of any kind, so how to shift security left from someone actually doing it is much more desired than from some consultant who just always talks about ;)

I'll look into it :)

1. analyzing every PR, no matter of the base branch

🤣 that's already working, there's no check on the base branch of a PR, @emil-wire - would have bet my ass that's different. #317

you sure? I have a bunch of these in my logs:
technolinator | {"timestamp":"2024-01-15T16:06:08.355781364Z","sequence":654,"loggerClassName":"org.jboss.logging.Logger","loggerName":"com.mediamarktsaturn.technolinator.events.OnPushDispatcher","level":"INFO","message":"Ref refs/heads/fix/background-task-cherry-pick of repository https://github.com/wireapp/wire-ios not eligible for analysis, ignoring.","threadName":"executor-thread-112","threadId":290,"mdc":{"ref":"refs/heads/fix/background-task-cherry-pick","commitSha":"f627a226033fdf21e0218af431f82031a7e473f1","repository":"wireapp/wire-ios","traceId":"be4744e5"},"ndc":"","hostName":"954b0fa69dfb","processName":"quarkus-run.jar","processId":1}

[heubeck]

you sure? I have a bunch of these in my logs:

yes, push to non-default branches are ignored.
but all PRs are analyzed, just do a PR to a non-default branch and see ;)

[heubeck]

this issue is not forgotten, but need some time to test a proper setup with dtrack and the archiving/cleanup function in place upfront.