grype db diff consumes lots of memory

[tomerse-sg]

What happened:
I am running grype inside a container and try to run grype db diff.
I allocated 4gb of memory and it fails on "signal killed".
If allocate more memory it doesn't happen (more than 8gb).

What you expected to happen:
use less memory or have the modified-time in the schema db.
How to reproduce it (as minimally and precisely as possible):
I attached my Dockerfile and configuration & run the memory locally (in the docker it fails).

config:

  log:
      quiet: false
      level: trace
      file: ""
  dev:
      profile: mem

memory:

Type: inuse_space
Time: Jun 4, 2024 at 4:47pm (IDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 2400.30MB, 99.29% of 2417.40MB total
Dropped 210 nodes (cum <= 12.09MB)
Showing top 10 nodes out of 37
      flat  flat%   sum%        cum   cum%
 1041.46MB 43.08% 43.08%  1041.46MB 43.08%  github.com/glebarez/go-sqlite.(*conn).columnText
  605.45MB 25.05% 68.13%   605.45MB 25.05%  github.com/anchore/grype/grype/db/v5/store.buildVulnerabilityPkgsMap
  330.11MB 13.66% 81.78%   330.11MB 13.66%  reflect.growslice
  151.29MB  6.26% 88.04%  1670.59MB 69.11%  github.com/anchore/grype/grype/db/v5/store.(*store).GetAllVulnerabilityMetadata
  142.22MB  5.88% 93.92%   154.57MB  6.39%  encoding/json.(*decodeState).literalStore
   61.70MB  2.55% 96.48%    73.90MB  3.06%  encoding/json.(*decodeState).objectInterface
   30.20MB  1.25% 97.73%    30.20MB  1.25%  reflect.New
   29.20MB  1.21% 98.93%    29.20MB  1.21%  modernc.org/libc.GoString (inline)
    7.15MB   0.3% 99.23%  1077.81MB 44.59%  github.com/glebarez/go-sqlite.(*rows).Next
    1.51MB 0.062% 99.29%   239.42MB  9.90%  reflect.Value.extendSlice

Dockerfile:

# Use an official lightweight image as a base
FROM alpine:3.18

# Set environment variables
ENV GRYPE_VERSION=v0.74.4
ENV DESTINATION_DIR=/usr/local/bin

# Install dependencies
RUN apk add --no-cache curl

# Download and install Grype
RUN curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b ${DESTINATION_DIR} ${GRYPE_VERSION}

# Set permissions
RUN chmod 777 ${DESTINATION_DIR}/grype

# Copy the config file
COPY config.yaml /config.yaml

# Set the working directory
WORKDIR ${DESTINATION_DIR}

# Run the grype command and output the results
CMD ["./grype", "db", "diff", "-o", "json", "-c", "/config.yaml"]

Anything else we need to know?:
do you have an idea how to utilize this process?

Environment:

• Output of grype version: v0.74.4
• OS (e.g: cat /etc/os-release or similar):

[tomersein]

it sometimes happen also when you try to download grype-db and scan a regular scan, not only in grype db diff

[willmurphyscode]

This would not be relevant if we do #2129. However there has been some feedback on that decision, so I've added needs-discussion over on that issue.

[willmurphyscode]

I'm closing this issue in favor of #2129