Document not found (404)
+This URL is invalid, sorry. Please use the navigation bar or search to continue.
+ +This URL is invalid, sorry. Please use the navigation bar or search to continue.
+ +The preferred workflow is to fork the quay/claircore repository, push a feature branch to the new fork, then open a pull request.
+All pull requests should be targeted to the main branch outside of exceptional circumstances.
As many tests as possible should run with the standard go test invocations.
+Adding the special tag integration (e.g. go test -tags integration ./...) will also run "integration" tests.
+The project interprets "integration" tests to mean any test that would need external resources, such as:
After at least one run with the integration tag, the tests should cache needed resources and run as many tests as possible.
+See also the test/integration package.
The Pull Request (PR) is the unit of code review. +Claircore's review flow treats a feature branch as a stack of patches to be applied. +That is to say, the feature branch should be rebased onto the target branch and have well-organized commits. +Merge commits are disallowed. +If the author would prefer to not rewrite commit history while working through reviews, fixup commits are the suggested way to achieve that. +As many requirements as possible are enforced by CI, like:
+Please use the "draft" option if the branch is not ready. +Please enable the "allow edits by maintainers" option.
+The maintainers may rebase, push, and merge contributors' branches.
+This may necessitate doing a git reset <remote>/<branch> to update a local branch.
Git commits should be formatted like "subject: summary" and avoid going over 80 characters per line.
+The "subject" is usually the package affected by the commit (like jar or rhel -- the relative path isn't needed) but sometimes a broader category (like docs, all, or cicd) is OK.
All the helper scripts should handle the "normal" convention (origin is quay/claircore and fork is one's personal fork) and the "British" convention (origin is one's personal fork and upstream is quay/claircore).
More detailed contributor documentation can be found in the project documentation.
+ +The following sections give a conceptual overview of how Claircore works internally.
+ + +claircore/indexer
The Indexer package performs Libindex's heavy lifting. It is responsible for retreiving Manifest layers, parsing the contents of each layer, and computing an IndexReport.
+To perform this action in incremental steps the Indexer is implemented as a finite state machine. At each state transition the Indexer persists an updated IndexReport to its datastore.
+The following diagram expresses the possible states of the Indexer:
++stateDiagram-v2 + state if_indexed <<choice>> + [*] --> CheckManifest + CheckManifest --> if_indexed + if_indexed --> [*]: Indexed + if_indexed --> FetchLayers: Unindexed + FetchLayers --> ScanLayers + ScanLayers --> Coalesce + Coalesce --> IndexManifest + IndexManifest --> IndexFinished + IndexFinished --> [*] +%% These notes make the diagram unreadable :/ +%% note left of CheckManifest: Determine if this manifest has been indexed previously. +%% note right of FetchLayers: Determine which layers need to be indexed and fetch them. +%% note right of ScanLayers: Concurrently run needed Indexers on layers. +%% note right of Coalesce: Compute the final contents of the container image. +%% note right of IndexManifest: Associate all the discoved data. +%% note right of IndexFinished: Persist the results. ++
The Indexer data model focuses on content addressable hashes as primary keys, the deduplication of package/distribution/repostitory information, and the recording of scan artifacts. +Scan artifacts are unique artifacts found within a layer which point to a deduplicated general package/distribution/repository record.
+The following diagram outlines the current Indexer data model.
+
+%%{init: {"er":{"layoutDirection":"RL"}} }%%
+erDiagram
+ ManifestLayer many to 1 Manifest: ""
+ ManifestLayer many to 1 Layer: ""
+ ScannedLayer many to 1 Layer: ""
+ ScannedLayer many to 1 Scanner: ""
+ ScannedManifest many to 1 Manifest: ""
+ ScannedManifest many to 1 Scanner: ""
+
+ TYPE_ScanArtifact 1 to 1 Layer: ""
+ TYPE_ScanArtifact 1 to 1 Scanner: ""
+ TYPE_ScanArtifact 1 to 1 TYPE: ""
+
+ ManifestIndex many to 1 Manifest: ""
+ ManifestIndex 1 to zero or one TYPE: ""
+
+ IndexReport 1 to 1 Manifest: "cached result"
+
+Note that TYPE stands in for each of the Indexer types (i.e. Package, Repository, etc.).
Indexers as currently built may make network requests. +This is an outstanding issue. +The following are the URLs used.
+claircore/internal/matcher
+claircore/libvuln/driver
The Matcher architecture is based on a data flow application.
++graph TD + start[Libvuln.Scan] + finish[Merge into VulnerabilityReport] + start ---> RemoteMatcher & Matcher ---> finish + subgraph RemoteMatcher + ra[Filter interested packages] + api[Make API calls] + rv[Decide vulnerability] + ra --> api --> rv + end + subgraph Matcher + dbv[Check versions in-database] + ma[Filter interested packages] + adv[Retrive vulnerabilty information] + mv[Deicide vulnerability] + ma --> adv --> mv + adv -.-> dbv -.-> mv + end ++
When Libvuln's Scan method is called with an IndexReport it will begin the process of matching container contents with vulnerabilities.
+Each configured Matcher will be instantiated concurrently. Depending on the interfaces the Matcher implements, one of the possible data flows will occur in the diagram above.
+The provided IndexReport will be unpacked into a stream of IndexRecord structs. Each Matcher will evaluate each record in the stream and determine if the IndexRecord is vulnerable to a security advisory in their responsible databases.
+Once each Matcher returns the set of vulnerabities, Libvuln will merge the results into a VulnerabilityReport and return this to the client.
+NOTE: Remote matchers are being considered for removal.
+"Remote matchers" may make HTTP requests during the matcher flow. +These requests are time-bound and errors are not reported. +The following are the URLs used.
+ +Claircore will normalize a security databases's severity string to a set of defined values.
+Clients may use the NormalizedSeverity field on a claircore.Vulnerability to react to vulnerability severities without needing to know each security database's severity strings.
+All strings used in the mapping tables are identical to the strings found within the relevant security database.
The following are severity strings Claircore will normalize others to. +Clients can guarantee one of these strings will be associated with a claircore.Vulnerability.
+The Alpine SecDB database does not provide severity information. +All vulnerability severities will be Unknown.
+| Alpine Severity | Claircore Severity |
|---|---|
| * | Unknown |
The AWS UpdateInfo database provides severity information.
+| AWS Severity | Claircore Severity |
|---|---|
| low | Low |
| medium | Medium |
| important | High |
| critical | Critical |
The Debian security tracker data provides severity information.
+| Debian Severity | Claircore Severity |
|---|---|
| unimportant | Low |
| low | Medium |
| medium | High |
| high | Critical |
| * | Unknown |
The Oracle OVAL database provides severity information.
+| Oracle Severity | Claircore Severity |
|---|---|
| N/A | Unknown |
| LOW | Low |
| MODERATE | Medium |
| IMPORTANT | High |
| CRITICAL | Critical |
The RHEL OVAL database provides severity information.
+| RHEL Severity | Claircore Severity |
|---|---|
| None | Unknown |
| Low | Low |
| Moderate | Medium |
| Important | High |
| Critical | Critical |
The SUSE OVAL database provides severity information.
+| SUSE Severity | Claircore Severity |
|---|---|
| None | Unknown |
| Low | Low |
| Moderate | Medium |
| Important | High |
| Critical | Critical |
The Ubuntu OVAL database provides severity information.
+| Ubuntu Severity | Claircore Severity |
|---|---|
| Untriaged | Unknown |
| Negligible | Negligible |
| Low | Low |
| Medium | Medium |
| High | High |
| Critical | Critical |
The Photon OVAL database provides severity information.
+| Photon Severity | Claircore Severity |
|---|---|
| Low | Low |
| Moderate | Medium |
| Important | High |
| Critical | Critical |
OSV provides severity information via CVSS vectors, when applicable. +These are normalized according to the NVD qualitative rating scale. +If both v3 and v2 vectors are present, v3 is preferred.
+| Base Score | Claircore Severity |
|---|---|
| 0.0 | Negligible |
| 0.1-3.9 | Low |
| 4.0-6.9 | Medium |
| 7.0-8.9 | High |
| 9.0-10.0 | Critical |
| Base Score | Claircore Severity |
|---|---|
| 0.0-3.9 | Low |
| 4.0-6.9 | Medium |
| 7.0-10.0 | High |
The default updaters are tracked in updater/defaults/defaults.go.
The following are the HTTP hosts and paths that Clair will attempt to +talk to in a default configuration. This list is non-exhaustive, as +some servers will issue redirects and some request URLs are constructed +dynamically.
+The following describes a successful scan.
+These topics cover helpful tips for contributing to Claircore.
+The claircore project has switched from a git log based changelog to a
+git notes based changelog.
This has the benefit of making the changelog more human-friendly, as it can have
+prose describing changes now, but makes adding entries a bit more involved. A
+full understanding of git notes is helpful for working with the changelog, but
+not required. If the reader has worked with the notes feature before, the
+changelog entries are stored under the changelog ref. For other users, there
+are some helper scripts in .github/scripts.
git notesGit notes is a mechanism for storing additional information alongside commits
+without modifying the commits. It does this by creating a ref full of files
+named after the commits, with their contents being the notes. This scheme
+requires some special care and tooling -- see the documentation for
+more information.
The primary helper script is changelog-edit. It allows a user to sync down
+notes, edit an entry, or both. See the output of the h flag for more
+information.
The other script of interest is changelog-render, which can be used to render
+out the changelog on demand, assuming the changelog notes have been pulled
+locally.
The changelog-update script uses changelog-render to add to the
+CHANGELOG.md file in the repository root.
Broadly, changelog entries should be formatted like commit messages without any
+trailers. Entries are turned into list items, with the subject being the bullet
+point and the body of the entry being the "body" of the item, or hidden behind
+details elements when using HTML-enabled output.
The entries are almost always rendered as markdown, so using minimal markdown is +OK. Anything requiring excessive markdown is probably better served as +documentation proper, rather than a changelog entry.
+ +A local development environment is implemented via docker-compose.
+Several make targets are defined for working with the local development environment.
+claircore-db-up - creates just the claircore database useful for running integration tests without test servers
+claircore-db-restart - destroys and recreates a fresh database. localhost:5434
+Several make targets are defined for working with tests.
+integration - run the integration test suite. requires the claircore-db to be up. run `make clair-db-up` before this target
+unit - run the unit test suite.
+bench - runs the benchmarks
+integration-v - runs the integration test suite with verbose
+unit-v - runs the unit test suite with verbose
+All the logging in claircore is done with zerolog via context.Context
+values. The zlog package takes OpenTelemetry labels and attaches them to
+zerolog events.
This allows for claircore's logging to be used consistently throughout all the +packages without having unintended prints to stderr.
+In a function, use zlog to add key-value pairs of any relevant context:
ctx = zlog.ContextWithValues(ctx,
+ "component", "Example.Logger")
+Alternatively, the go.opentelemetry.io/otel/baggage package can be used for
+more explicit control around the baggage values.
Zerolog emits lines when the Msg or Msgf methods are called. Project style
+is to not use Msgf. Any variable data should be set as key-value pairs on
+the Event object.
For example, don't do this:
+ zlog.Info(ctx).Msgf("done at: %v", time.Now())
+Do this instead:
+ zlog.Info(ctx).
+ Time("time", time.Now()).
+ Msgf("done")
+When noting the change during a chunk of work, make sure that the +log messages scan as visually similar. Usually, this means formatting messages +into "${process} ${event}". For example:
+frob start
+frob initialized
+frob ready
+frob success
+frob done
+Is much easier to scan than:
+starting to frob
+initialized frobber
+ready for frobbing
+did frob
+done with frobing
+When handling an error, code should only log it if it does not propagate it. The +code that ultimately handles the error is responsible for deciding what to do +with it. Logging and returning ends up with the same message repeated multiple +times in the logs.
+Claircore attempts to have consistent leveled logging. The rules for figuring +out what level to use is:
+Panic
+There's some occurrence that means the process won't work correctly.
+Fatal
+Unused, because it prevents defers from running.
+Error
+Something unexpected occurred and the process can continue, but a +human needs to be notified. An error will be returned for this request.
+Warn
+Something unexpected occurred and the process can continue. An error will be +returned for this request.
+Info
+Some information that may be useful to an operator. Examples include +a timer-based process starting and ending, a user request starting and +ending, or a summary of work done.
+Debug
+Some information that may be useful to a developer. Examples include entering +and exiting functions, stepping through a function, or specific file paths +used while work is being done.
+Here's various codebase conventions that don't have dedicated pages:
+//doc:url directive comment. See
+the the internal/cmd/mdbook-injecturls command for documentation on how the
+preprocessor works. The list of keywords isn't an allowlist, so an invocation
+like the following should list the ones actually used in the documentation
+using a command like git grep injecturls -- :/docs/*.md.Tests in the claircore module may use various helpers underneath the test directory.
+Using these packages outside of testing code is disallowed.
+Assert packages are disallowed;
+the go-cmp package is the only external package helper allowed.
Tests that use external resources or generate test fixtures should be annotated according to the integration package.
Tests using the integration package cache generated and downloaded assets into a directory named clair-testing inside the directory reported by os.UserCacheDir.
+For example, on a Linux system, the cache directory will be (in sh notation) ${XDG_CACHE_HOME-$HOME/.cache}/clair-testing.
${p.children.map(f).join("")}
`:`Unsupported markdown: ${p.type}`}return a.map(f).join("")}function Nje(i){return Intl.Segmenter?[...new Intl.Segmenter().segment(i)].map(a=>a.segment):[...i]}function Pje(i,a){const f=Nje(a.content);return aBt(i,[],f,a.type)}function aBt(i,a,f,p){if(f.length===0)return[{content:a.join(""),type:p},{content:"",type:p}];const[v,...m]=f,b=[...a,v];return i([{content:b.join(""),type:p}])?aBt(i,b,m,p):(a.length===0&&v&&(a.push(v),f.shift()),[{content:a.join(""),type:p},{content:f.join(""),type:p}])}function Fje(i,a){if(i.some(({content:f})=>f.includes(` +`)))throw new Error("splitLineToFitWidth does not support newlines in the line");return C0t(i,a)}function C0t(i,a,f=[],p=[]){if(i.length===0)return p.length>0&&f.push(p),f.length>0?f:[];let v="";i[0].content===" "&&(v=" ",i.shift());const m=i.shift()??{content:" ",type:"normal"},b=[...p];if(v!==""&&b.push({content:v,type:"normal"}),b.push(m),a(b))return C0t(i,a,f,b);if(p.length>0)f.push(p),i.unshift(m);else if(m.content){const[E,C]=Pje(a,m);f.push([E]),C.content&&i.unshift(C)}return C0t(i,a,f)}function Bje(i,a){a&&i.attr("style",a)}function Rje(i,a,f,p,v=!1){const m=i.append("foreignObject"),b=m.append("xhtml:div"),E=a.label,C=a.isNode?"nodeLabel":"edgeLabel";b.html(` + "+E+""),Bje(b,a.labelStyle),b.style("display","table-cell"),b.style("white-space","nowrap"),b.style("max-width",f+"px"),b.attr("xmlns","http://www.w3.org/1999/xhtml"),v&&b.attr("class","labelBkg");let A=b.node().getBoundingClientRect();return A.width===f&&(b.style("display","table"),b.style("white-space","break-spaces"),b.style("width",f+"px"),A=b.node().getBoundingClientRect()),m.style("width",A.width),m.style("height",A.height),m.node()}function oBt(i,a,f){return i.append("tspan").attr("class","text-outer-tspan").attr("x",0).attr("y",a*f-.1+"em").attr("dy",f+"em")}function jje(i,a,f){const p=i.append("text"),v=oBt(p,1,a);cBt(v,f);const m=v.node().getComputedTextLength();return p.remove(),m}function $je(i,a,f,p=!1){const m=a.append("g"),b=m.insert("rect").attr("class","background"),E=m.append("text").attr("y","-10.1");let C=0;for(const A of f){const P=j=>jje(m,1.1,j)<=i,F=P(A)?[A]:Fje(A,P);for(const j of F){const R=oBt(E,C,1.1);cBt(R,j),C++}}if(p){const A=E.node().getBBox(),P=2;return b.attr("x",-P).attr("y",-P).attr("width",A.width+2*P).attr("height",A.height+2*P),m.node()}else return E.node()}function cBt(i,a){i.text(""),a.forEach((f,p)=>{const v=i.append("tspan").attr("font-style",f.type==="emphasis"?"italic":"normal").attr("class","text-inner-tspan").attr("font-weight",f.type==="strong"?"bold":"normal");p===0?v.text(f.content):v.text(" "+f.content)})}const vK=(i,a="",{style:f="",isTitle:p=!1,classes:v="",useHtmlLabels:m=!0,isNode:b=!0,width:E=200,addSvgBackground:C=!1}={})=>{if(Wt.info("createText",a,f,p,v,m,b,C),m){const A=Oje(a),P={isNode:b,label:nP(A).replace(/fa[blrs]?:fa-[\w-]+/g,j=>``),labelStyle:f.replace("fill:","color:")};return Rje(i,P,E,v,C)}else{const A=Ije(a);return $je(E,i,A,C)}},Od=async(i,a,f,p)=>{let v;const m=a.useHtmlLabels||u1(De().flowchart.htmlLabels);f?v=f:v="node default";const b=i.insert("g").attr("class",v).attr("id",a.domId||a.id),E=b.insert("g").attr("class","label").attr("style",a.labelStyle);let C;a.labelText===void 0?C="":C=typeof a.labelText=="string"?a.labelText:a.labelText[0];const A=E.node();let P;a.labelType==="markdown"?P=vK(E,v0(nP(C),De()),{useHtmlLabels:m,width:a.width||De().flowchart.wrappingWidth,classes:"markdown-node-label"}):P=A.appendChild(dp(v0(nP(C),De()),a.labelStyle,!1,p));let F=P.getBBox();const j=a.padding/2;if(u1(De().flowchart.htmlLabels)){const R=P.children[0],H=xr(P),Y=R.getElementsByTagName("img");if(Y){const rt=C.replace(/