Fetch CPEs from Repology #5352
Comments
|
So I've looked more into it and i have some questions:
Example of what i mean (this is the infoblox WAPI)
|
|
Added macOS CPE's for a test and to discuss with my colleague tomorrow how the KQL Queries will look like and other things |
|
Using the NVD dataset is a dead-end - I wouldn't go via that route. There's some attempts at making a PURL/CPE mapping, but none of them are as accurate as the approach the repology already takes: Use NVD CVEs (which link to CPEs), and if the same CVE is noticed on a product from another source (such as a distro CVE), link the CPE.
for many products on endoflife.date, we have a repology identifier in the metadata. Here's a complete mapping that you can use. the first column is the filename, and the second column is the corresponding repology project. So apache-http-server.md links to https://repology.org/project/apache/information as an example.
We have them in the same markdown files under the That said, there is no such thing as an official PURL. PURL are just an identifier that anyone can generate. In our case, we add them to our pages as and when we find them. See #5302 for an example, where I went through various installation methods of the product, generated PURLs for them, and added them to the product.
Assuming you use the repology dataset from https://dumps.repology.org/ (which is the point of this issue), this should not matter.
The API is light enough that I don't see the use for this. If you want to get data for multiple products in a single call, we don't support that in the current API, but the next version of the API has an endpoint for this: https://deploy-preview-2080--endoflife-date.netlify.app/api/v1/products/full/. See docs at https://deploy-preview-2080--endoflife-date.netlify.app/docs/api/v1/#/default/get_products_full_ and PR at #2080 |
This means I shouldn't use the API with a regex and instead use the API to look for CVE's and their linked CPE's? (see https://documenter.getpostman.com/view/16438573/UzXKWe99 )
So basically the repology identifier is just how the project is called on repology (if it exists)?
So for now I should just focus on the repology and CPE identifiers?
This means that for the repology identifiers i can use this DB that I can import right? Do you know by any chance which DBMS it is?
I meant like getting only one single field with an API call (like just get all identifiers), but thanks for the update about the API, this will be great ! |
Yes, but repology has done this hard work, so we need not reproduce.
Yes
Yes
postgres, with a few extensions. There's a README: https://dumps.repology.org/README.txt which links to https://github.com/repology/repology-updater#creating-the-database I have a WIP dockerfile which might be of help: FROM opensuse/leap:15.6
RUN zypper --non-interactive install postgresql14 postgresql14-libversion postgresql14-contrib zstd
RUN mkdir -p /var/run/postgresql && chown -R postgres /var/run/postgresql
USER postgres
ADD repology-database-dump-latest.sql.zst /tmp/repology-database-dump-latest.sql.zst
RUN /usr/lib/postgresql14/bin/initdb -D /var/lib/pgsql/data
RUN pg_ctl --wait --mode immediate -D /var/lib/pgsql/data start && \
psql -c "CREATE DATABASE repology" && \
psql -c "CREATE USER repology WITH PASSWORD 'repology'" && \
psql -c "GRANT ALL ON DATABASE repology TO repology" && \
psql --dbname repology -c "GRANT CREATE ON SCHEMA public TO PUBLIC" && \
psql --dbname repology -c "CREATE EXTENSION pg_trgm" && \
psql --dbname repology -c "CREATE EXTENSION libversion" && \
echo "host all all 0.0.0.0/0 trust" >> /var/lib/pgsql/data/pg_hba.conf && \
zstd -d /tmp/repology-database-dump-latest.sql.zst | psql --dbname repology -v ON_ERROR_STOP=1
CMD postgres -D /var/lib/pgsql/data
EXPOSE 5432 |
|
The table we need is Which results in this table (with a
Hopefully this should be good enough for you to file some more PRs. I've also updated the issue's description with a TODO. |
|
thanks for the examples, the files and the TODO's! i'm currently setting up the VM for the docker instance and i'll create the PR's soon :) |
|
Hello @captn3m0 and with the ansible example, there are the same CPEs but for different repology projects repology | cpe Add it to both files right? |
|
The ansible package includes the ansible-core package internally. We can add to just ansible-core for now, need to think a bit more on this. |
Added identifiers to all the products which filenames starts with an A. Not everything was in the Repology DB, so some had to be manually filled. See also #5352. --------- Co-authored-by: Marc Wrobel <[email protected]>
More info on #5352. --------- Co-authored-by: Marc Wrobel <[email protected]>
|
hello @captn3m0 and @marcwrobel so I've added more CPEs and I some questions: what are the next steps here? some products apparently don't have a CPE and some products I'm not sure which one to use. and for example, nokia and google phones have a cpe for each version, what should I do with these? is there anything to do for the API, so that we could get the CPE via EOL-API ? |
Here's a simpler TODO:
cpe:<cpe_version>:<part>:<vendor>:<product>, since we are linking to the product, and not a specific version or variant.The text was updated successfully, but these errors were encountered: