[Bug]: memory leak with cosmos mainnet tendermint full node

[jay-ginco]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

We have observed this since few months, our cosmos mainnet node, currently on latest gaia version v21.0.0, is continously reaching the memory limits set, leading to frequent restarts and db corruption. Initially it was running on 40 GB, then 64 then 98 GB, but its reaching the limits, kindly explain whats the recommended memory for the some, and if any config is leading to this behaviour.

Just for info, we are running on default pruning configs

Gaia Version

v21.0.0

How to reproduce?

No response

[MSalopek]

Hello! Thank you for opening the issue.

Would you mind providing your config.toml and app.toml?

Is this node used as an RPC node with all query endpoints enabled?

[jay-ginco]

Hi @MSalopek yes we are using it as RPC node with query endpoints

• config.toml

    ###                   Main Base Config Options                      ###
    #######################################################################

    # TCP or UNIX socket address of the ABCI application,
    # or the name of an ABCI application compiled in with the CometBFT binary
    proxy_app = "tcp://127.0.0.1:26658"

    # A custom human readable name for this node
    moniker = "{{ $.Values.cosmos.args.moniker }}"

    # If this node is many blocks behind the tip of the chain, FastSync
    # allows them to catchup quickly by downloading blocks in parallel
    # and verifying their commits
    fast_sync = true

    # Database backend: goleveldb | cleveldb | boltdb | rocksdb | badgerdb
    # * goleveldb (github.com/syndtr/goleveldb - most popular implementation)
    #   - pure go
    #   - stable
    # * cleveldb (uses levigo wrapper)
    #   - fast
    #   - requires gcc
    #   - use cleveldb build tag (go build -tags cleveldb)
    # * boltdb (uses etcd's fork of bolt - github.com/etcd-io/bbolt)
    #   - EXPERIMENTAL
    #   - may be faster is some use-cases (random reads - indexer)
    #   - use boltdb build tag (go build -tags boltdb)
    # * rocksdb (uses github.com/tecbot/gorocksdb)
    #   - EXPERIMENTAL
    #   - requires gcc
    #   - use rocksdb build tag (go build -tags rocksdb)
    # * badgerdb (uses github.com/dgraph-io/badger)
    #   - EXPERIMENTAL
    #   - use badgerdb build tag (go build -tags badgerdb)
    db_backend = "goleveldb"

    # Database directory
    db_dir = "data"

    # Output level for logging, including package level options
    log_level = "info"

    # Output format: 'plain' (colored text) or 'json'
    log_format = "plain"

    ##### additional base config options #####

    # Path to the JSON file containing the initial validator set and other meta data
    genesis_file = "config/genesis.json"

    # Path to the JSON file containing the private key to use as a validator in the consensus protocol
    priv_validator_key_file = "config/priv_validator_key.json"

    # Path to the JSON file containing the last sign state of a validator
    priv_validator_state_file = "data/priv_validator_state.json"

    # TCP or UNIX socket address for CometBFT to listen on for
    # connections from an external PrivValidator process
    priv_validator_laddr = ""

    # Path to the JSON file containing the private key to use for node authentication in the p2p protocol
    node_key_file = "config/node_key.json"

    # Mechanism to connect to the ABCI application: socket | grpc
    abci = "socket"

    # If true, query the ABCI app on connecting to a new peer
    # so the app can decide if we should keep the connection or not
    filter_peers = false


    #######################################################################
    ###                 Advanced Configuration Options                  ###
    #######################################################################

    #######################################################
    ###       RPC Server Configuration Options          ###
    #######################################################
    [rpc]

    # TCP or UNIX socket address for the RPC server to listen on
    laddr = "tcp://0.0.0.0:26657"

    # A list of origins a cross-domain request can be executed from
    # Default value '[]' disables cors support
    # Use '["*"]' to allow any origin
    cors_allowed_origins = ["*"]

    # A list of methods the client is allowed to use with cross-domain requests
    cors_allowed_methods = ["HEAD", "GET", "POST", ]

    # A list of non simple headers the client is allowed to use with cross-domain requests
    cors_allowed_headers = ["Origin", "Accept", "Content-Type", "X-Requested-With", "X-Server-Time", ]

    # TCP or UNIX socket address for the gRPC server to listen on
    # NOTE: This server only supports /broadcast_tx_commit
    grpc_laddr = ""

    # Maximum number of simultaneous connections.
    # Does not include RPC (HTTP&WebSocket) connections. See max_open_connections
    # If you want to accept a larger number than the default, make sure
    # you increase your OS limits.
    # 0 - unlimited.
    # Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}
    # 1024 - 40 - 10 - 50 = 924 = ~900
    grpc_max_open_connections = 900

    # Activate unsafe RPC commands like /dial_seeds and /unsafe_flush_mempool
    unsafe = false

    # Maximum number of simultaneous connections (including WebSocket).
    # Does not include gRPC connections. See grpc_max_open_connections
    # If you want to accept a larger number than the default, make sure
    # you increase your OS limits.
    # 0 - unlimited.
    # Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}
    # 1024 - 40 - 10 - 50 = 924 = ~900
    max_open_connections = 900

    # Maximum number of unique clientIDs that can /subscribe
    # If you're using /broadcast_tx_commit, set to the estimated maximum number
    # of broadcast_tx_commit calls per block.
    max_subscription_clients = 100

    # Maximum number of unique queries a given client can /subscribe to
    # If you're using GRPC (or Local RPC client) and /broadcast_tx_commit, set to
    # the estimated # maximum number of broadcast_tx_commit calls per block.
    max_subscriptions_per_client = 5

    # Experimental parameter to specify the maximum number of events a node will
    # buffer, per subscription, before returning an error and closing the
    # subscription. Must be set to at least 100, but higher values will accommodate
    # higher event throughput rates (and will use more memory).
    experimental_subscription_buffer_size = 200

    # Experimental parameter to specify the maximum number of RPC responses that
    # can be buffered per WebSocket client. If clients cannot read from the
    # WebSocket endpoint fast enough, they will be disconnected, so increasing this
    # parameter may reduce the chances of them being disconnected (but will cause
    # the node to use more memory).
    #
    # Must be at least the same as "experimental_subscription_buffer_size",
    # otherwise connections could be dropped unnecessarily. This value should
    # ideally be somewhat higher than "experimental_subscription_buffer_size" to
    # accommodate non-subscription-related RPC responses.
    experimental_websocket_write_buffer_size = 200

    # If a WebSocket client cannot read fast enough, at present we may
    # silently drop events instead of generating an error or disconnecting the
    # client.
    #
    # Enabling this experimental parameter will cause the WebSocket connection to
    # be closed instead if it cannot read fast enough, allowing for greater
    # predictability in subscription behaviour.
    experimental_close_on_slow_client = false

    # How long to wait for a tx to be committed during /broadcast_tx_commit.
    # WARNING: Using a value larger than 10s will result in increasing the
    # global HTTP write timeout, which applies to all connections and endpoints.
    # See https://github.com/tendermint/tendermint/issues/3435
    timeout_broadcast_tx_commit = "10s"

    # Maximum size of request body, in bytes
    max_body_bytes = 1000000

    # Maximum size of request header, in bytes
    max_header_bytes = 1048576

    # The path to a file containing certificate that is used to create the HTTPS server.
    # Might be either absolute path or path related to CometBFT's config directory.
    # If the certificate is signed by a certificate authority,
    # the certFile should be the concatenation of the server's certificate, any intermediates,
    # and the CA's certificate.
    # NOTE: both tls_cert_file and tls_key_file must be present for CometBFT to create HTTPS server.
    # Otherwise, HTTP server is run.
    tls_cert_file = ""

    # The path to a file containing matching private key that is used to create the HTTPS server.
    # Might be either absolute path or path related to CometBFT's config directory.
    # NOTE: both tls-cert-file and tls-key-file must be present for CometBFT to create HTTPS server.
    # Otherwise, HTTP server is run.
    tls_key_file = ""

    # pprof listen address (https://golang.org/pkg/net/http/pprof)
    pprof_laddr = "localhost:6060"

    #######################################################
    ###           P2P Configuration Options             ###
    #######################################################
    [p2p]

    # Address to listen for incoming connections
    laddr = "tcp://0.0.0.0:26656"

    # Address to advertise to peers for them to dial
    # If empty, will use the same port as the laddr,
    # and will introspect on the listener or use UPnP
    # to figure out the address. ip and port are required
    # example: 159.89.10.97:26656
    external_address = ""

    # Comma separated list of seed nodes to connect to
    seeds = "{{ $.Values.cosmos.args.seeds }}"

    # Comma separated list of nodes to keep persistent connections to
    persistent_peers = "{{ $.Values.cosmos.args.persistent_peers }}"

    # UPNP port forwarding
    upnp = false

    # Path to address book
    addr_book_file = "config/addrbook.json"

    # Set true for strict address routability rules
    # Set false for private or local networks
    addr_book_strict = true

    # Maximum number of inbound peers
    max_num_inbound_peers = 40

    # Maximum number of outbound peers to connect to, excluding persistent peers
    max_num_outbound_peers = "{{ $.Values.cosmos.args.max_num_outbound_peers }}"

    # List of node IDs, to which a connection will be (re)established ignoring any existing limits
    unconditional_peer_ids = ""

    # Maximum pause when redialing a persistent peer (if zero, exponential backoff is used)
    persistent_peers_max_dial_period = "600s"

    # Time to wait before flushing messages out on the connection
    flush_throttle_timeout = "100ms"

    # Maximum size of a message packet payload, in bytes
    max_packet_msg_payload_size = 1024

    # Rate at which packets can be sent, in bytes/second
    send_rate = 5120000

    # Rate at which packets can be received, in bytes/second
    recv_rate = 5120000

    # Set true to enable the peer-exchange reactor
    pex = true

    # Seed mode, in which node constantly crawls the network and looks for
    # peers. If another node asks it for addresses, it responds and disconnects.
    #
    # Does not work if the peer-exchange reactor is disabled.
    seed_mode = false

    # Comma separated list of peer IDs to keep private (will not be gossiped to other peers)
    private_peer_ids = ""

    # Toggle to disable guard against peers connecting from the same ip.
    allow_duplicate_ip = false

    # Peer connection configuration.
    handshake_timeout = "20s"
    dial_timeout = "3s"

    #######################################################
    ###          Mempool Configuration Option          ###
    #######################################################
    [mempool]

    # Mempool version to use:
    #   1) "v0" - (default) FIFO mempool.
    #   2) "v1" - prioritized mempool.
    version = "v0"

    # Recheck (default: true) defines whether CometBFT should recheck the
    # validity for all remaining transaction in the mempool after a block.
    # Since a block affects the application state, some transactions in the
    # mempool may become invalid. If this does not apply to your application,
    # you can disable rechecking.
    recheck = true
    broadcast = true
    wal_dir = ""

    # Maximum number of transactions in the mempool
    size = 5000

    # Limit the total size of all txs in the mempool.
    # This only accounts for raw transactions (e.g. given 1MB transactions and
    # max_txs_bytes=5MB, mempool will only accept 5 transactions).
    max_txs_bytes = 1073741824

    # Size of the cache (used to filter transactions we saw earlier) in transactions
    cache_size = 10000

    # Do not remove invalid transactions from the cache (default: false)
    # Set to true if it's not possible for any invalid transaction to become valid
    # again in the future.
    keep-invalid-txs-in-cache = false

    # Maximum size of a single transaction.
    # NOTE: the max size of a tx transmitted over the network is {max_tx_bytes}.
    max_tx_bytes = 1048576

    # Maximum size of a batch of transactions to send to a peer
    # Including space needed by encoding (one varint per transaction).
    # XXX: Unused due to https://github.com/tendermint/tendermint/issues/5796
    max_batch_bytes = 0

    # ttl-duration, if non-zero, defines the maximum amount of time a transaction
    # can exist for in the mempool.
    #
    # Note, if ttl-num-blocks is also defined, a transaction will be removed if it
    # has existed in the mempool at least ttl-num-blocks number of blocks or if it's
    # insertion time into the mempool is beyond ttl-duration.
    ttl-duration = "0s"

    # ttl-num-blocks, if non-zero, defines the maximum number of blocks a transaction
    # can exist for in the mempool.
    #
    # Note, if ttl-duration is also defined, a transaction will be removed if it
    # has existed in the mempool at least ttl-num-blocks number of blocks or if
    # it's insertion time into the mempool is beyond ttl-duration.
    ttl-num-blocks = 0

    #######################################################
    ###         State Sync Configuration Options        ###
    #######################################################
    [statesync]
    # State sync rapidly bootstraps a new node by discovering, fetching, and restoring a state machine
    # snapshot from peers instead of fetching and replaying historical blocks. Requires some peers in
    # the network to take and serve state machine snapshots. State sync is not attempted if the node
    # has any local state (LastBlockHeight > 0). The node will have a truncated block history,
    # starting from the height of the snapshot.
    enable = "{{ $.Values.cosmos.args.state_sync.enable }}"

    # RPC servers (comma-separated) for light client verification of the synced state machine and
    # retrieval of state data for node bootstrapping. Also needs a trusted height and corresponding
    # header hash obtained from a trusted source, and a period during which validators can be trusted.
    #
    # For Cosmos SDK-based chains, trust_period should usually be about 2/3 of the unbonding time (~2
    # weeks) during which they can be financially punished (slashed) for misbehavior.
    rpc_servers = "{{ $.Values.cosmos.args.state_sync.rpc_servers }}"
    trust_height = {{ $.Values.cosmos.args.state_sync.trust_height}}
    trust_hash = "{{ $.Values.cosmos.args.state_sync.trust_hash }}"
    trust_period = "168h0m0s"

    # Time to spend discovering snapshots before initiating a restore.
    discovery_time = "15s"

    # Temporary directory for state sync snapshot chunks, defaults to the OS tempdir (typically /tmp).
    # Will create a new, randomly named directory within, and remove it when done.
    temp_dir = ""

    # The timeout duration before re-requesting a chunk, possibly from a different
    # peer (default: 1 minute).
    chunk_request_timeout = "10s"

    # The number of concurrent chunk fetchers to run (default: 1).
    chunk_fetchers = "4"

    #######################################################
    ###       Fast Sync Configuration Connections       ###
    #######################################################
    [fastsync]

    # Fast Sync version to use:
    #   1) "v0" (default) - the legacy fast sync implementation
    #   2) "v1" - refactor of v0 version for better testability
    #   2) "v2" - complete redesign of v0, optimized for testability & readability
    version = "v0"

    #######################################################
    ###         Consensus Configuration Options         ###
    #######################################################
    [consensus]

    wal_file = "data/cs.wal/wal"

    # How long we wait for a proposal block before prevoting nil
    timeout_propose = "3s"
    # How much timeout_propose increases with each round
    timeout_propose_delta = "500ms"
    # How long we wait after receiving +2/3 prevotes for “anything” (ie. not a single block or nil)
    timeout_prevote = "1s"
    # How much the timeout_prevote increases with each round
    timeout_prevote_delta = "500ms"
    # How long we wait after receiving +2/3 precommits for “anything” (ie. not a single block or nil)
    timeout_precommit = "1s"
    # How much the timeout_precommit increases with each round
    timeout_precommit_delta = "500ms"
    # How long we wait after committing a block, before starting on the new
    # height (this gives us a chance to receive some more precommits, even
    # though we already have +2/3).
    timeout_commit = "5s"

    # How many blocks to look back to check existence of the node's consensus votes before joining consensus
    # When non-zero, the node will panic upon restart
    # if the same consensus key was used to sign {double_sign_check_height} last blocks.
    # So, validators should stop the state machine, wait for some blocks, and then restart the state machine to avoid panic.
    double_sign_check_height = 0

    # Make progress as soon as we have all the precommits (as if TimeoutCommit = 0)
    skip_timeout_commit = false

    # EmptyBlocks mode and possible interval between empty blocks
    create_empty_blocks = true
    create_empty_blocks_interval = "0s"

    # Reactor sleep duration parameters
    peer_gossip_sleep_duration = "100ms"
    peer_query_maj23_sleep_duration = "2s"

    #######################################################
    ###         Storage Configuration Options           ###
    #######################################################
    [storage]

    # Set to true to discard ABCI responses from the state store, which can save a
    # considerable amount of disk space. Set to false to ensure ABCI responses are
    # persisted. ABCI responses are required for /block_results RPC queries, and to
    # reindex events in the command-line tool.
    discard_abci_responses = false

    #######################################################
    ###   Transaction Indexer Configuration Options     ###
    #######################################################
    [tx_index]

    # What indexer to use for transactions
    #
    # The application will set which txs to index. In some cases a node operator will be able
    # to decide which txs to index based on configuration set in the application.
    #
    # Options:
    #   1) "null"
    #   2) "kv" (default) - the simplest possible indexer, backed by key-value storage (defaults to levelDB; see DBBackend).
    # 		- When "kv" is chosen "tx.height" and "tx.hash" will always be indexed.
    #   3) "psql" - the indexer services backed by PostgreSQL.
    # When "kv" or "psql" is chosen "tx.height" and "tx.hash" will always be indexed.
    indexer = "kv"

    # The PostgreSQL connection configuration, the connection format:
    #   postgresql://<user>:<password>@<host>:<port>/<db>?<opts>
    psql-conn = ""

    #######################################################
    ###       Instrumentation Configuration Options     ###
    #######################################################
    [instrumentation]

    # When true, Prometheus metrics are served under /metrics on
    # PrometheusListenAddr.
    # Check out the documentation for the list of available metrics.
    prometheus = false

    # Address to listen for Prometheus collector(s) connections
    prometheus_listen_addr = ":26660"

    # Maximum number of simultaneous connections.
    # If you want to accept a larger number than the default, make sure
    # you increase your OS limits.
    # 0 - unlimited.
    max_open_connections = 3

    # Instrumentation namespace
    namespace = "cometbft"

• app.toml

  app.toml: |
    ###############################################################################
    ###                        Custom Gaia Configuration                        ###
    ###############################################################################
    # bypass-min-fee-msg-types defines custom message types the operator may set that
    # will bypass minimum fee checks during CheckTx.
    # NOTE:
    # bypass-min-fee-msg-types = [] will deactivate the bypass - no messages will be allowed to bypass the minimum fee check
    # bypass-min-fee-msg-types = [<MsgType>...] will allow messages of specified type to bypass the minimum fee check
    # removing bypass-min-fee-msg-types from the config file will apply the default values:
    # ["/ibc.core.channel.v1.MsgRecvPacket", "/ibc.core.channel.v1.MsgAcknowledgement", "/ibc.core.client.v1.MsgUpdateClient"]
    #
    # Example:
    # bypass-min-fee-msg-types = ["/ibc.core.channel.v1.MsgRecvPacket", "/ibc.core.channel.v1.MsgAcknowledgement", "/ibc.core.client.v1.MsgUpdateClient"]
    bypass-min-fee-msg-types = ["/ibc.core.channel.v1.MsgRecvPacket", "/ibc.core.channel.v1.MsgAcknowledgement", "/ibc.core.client.v1.MsgUpdateClient", ]

    ###############################################################################
    ###                           Base Configuration                            ###
    ###############################################################################

    # The minimum gas prices a validator is willing to accept for processing a
    # transaction. A transaction's fees must meet the minimum of any denomination
    # specified in this config (e.g. 0.25token1;0.0001token2).
    minimum-gas-prices = "0.0025uatom"

    # default: the last 100 states are kept in addition to every 500th state; pruning at 10 block intervals
    # nothing: all historic states will be saved, nothing will be deleted (i.e. archiving node)
    # everything: all saved states will be deleted, storing only the current and previous state; pruning at 10 block intervals
    # custom: allow pruning options to be manually specified through 'pruning-keep-recent', 'pruning-keep-every', and 'pruning-interval'
    pruning = "default"

    # These are applied if and only if the pruning strategy is custom.
    pruning-keep-recent = "0"
    pruning-keep-every = "0"
    pruning-interval = "0"

    # HaltHeight contains a non-zero block height at which a node will gracefully
    # halt and shutdown that can be used to assist upgrades and testing.
    #
    # Note: Commitment of state will be attempted on the corresponding block.
    halt-height = 0

    # HaltTime contains a non-zero minimum block time (in Unix seconds) at which
    # a node will gracefully halt and shutdown that can be used to assist upgrades
    # and testing.
    #
    # Note: Commitment of state will be attempted on the corresponding block.
    halt-time = 0

    # MinRetainBlocks defines the minimum block height offset from the current
    # block being committed, such that all blocks past this offset are pruned
    # from Tendermint. It is used as part of the process of determining the
    # ResponseCommit.RetainHeight value during ABCI Commit. A value of 0 indicates
    # that no blocks should be pruned.
    #
    # This configuration value is only responsible for pruning Tendermint blocks.
    # It has no bearing on application state pruning which is determined by the
    # "pruning-*" configurations.
    #
    # Note: Tendermint block pruning is dependant on this parameter in conunction
    # with the unbonding (safety threshold) period, state pruning and state sync
    # snapshot parameters to determine the correct minimum value of
    # ResponseCommit.RetainHeight.
    min-retain-blocks = 0

    # InterBlockCache enables inter-block caching.
    inter-block-cache = true

    # IndexEvents defines the set of events in the form {eventType}.{attributeKey},
    # which informs Tendermint what to index. If empty, all events will be indexed.
    #
    # Example:
    # ["message.sender", "message.recipient"]
    index-events = []

    # IavlCacheSize set the size of the iavl tree cache. 
    # Default cache size is 50mb.
    iavl-cache-size = 100000000

    # IAVLDisableFastNode enables or disables the fast node feature of IAVL. 
    # Default is true.
    iavl-disable-fastnode = false

    ###############################################################################
    ###                         Telemetry Configuration                         ###
    ###############################################################################

    [telemetry]

    # Prefixed with keys to separate services.
    service-name = ""

    # Enabled enables the application telemetry functionality. When enabled,
    # an in-memory sink is also enabled by default. Operators may also enabled
    # other sinks such as Prometheus.
    enabled = false

    # Enable prefixing gauge values with hostname.
    enable-hostname = false

    # Enable adding hostname to labels.
    enable-hostname-label = false

    # Enable adding service to labels.
    enable-service-label = false

    # PrometheusRetentionTime, when positive, enables a Prometheus metrics sink.
    prometheus-retention-time = 0

    # GlobalLabels defines a global set of name/value label tuples applied to all
    # metrics emitted using the wrapper functions defined in telemetry package.
    #
    # Example:
    # [["chain_id", "cosmoshub-1"]]
    global-labels = [
    ]

    ###############################################################################
    ###                           API Configuration                             ###
    ###############################################################################

    [api]

    # Enable defines if the API server should be enabled.
    enable = true

    # Swagger defines if swagger documentation should automatically be registered.
    swagger = false

    # Address defines the API server to listen on.
    address = "tcp://0.0.0.0:1317"

    # MaxOpenConnections defines the number of maximum open connections.
    max-open-connections = 1000

    # RPCReadTimeout defines the Tendermint RPC read timeout (in seconds).
    rpc-read-timeout = 10

    # RPCWriteTimeout defines the Tendermint RPC write timeout (in seconds).
    rpc-write-timeout = 0

    # RPCMaxBodyBytes defines the Tendermint maximum response body (in bytes).
    rpc-max-body-bytes = 1000000

    # EnableUnsafeCORS defines if CORS should be enabled (unsafe - use it at your own risk).
    enabled-unsafe-cors = false

    ###############################################################################
    ###                           Rosetta Configuration                         ###
    ###############################################################################

    [rosetta]

    # Enable defines if the Rosetta API server should be enabled.
    enable = false

    # Address defines the Rosetta API server to listen on.
    address = ":8080"

    # Network defines the name of the blockchain that will be returned by Rosetta.
    blockchain = "app"

    # Network defines the name of the network that will be returned by Rosetta.
    network = "network"

    # Retries defines the number of retries when connecting to the node before failing.
    retries = 3

    # Offline defines if Rosetta server should run in offline mode.
    offline = false

    ###############################################################################
    ###                           gRPC Configuration                            ###
    ###############################################################################

    [grpc]

    # Enable defines if the gRPC server should be enabled.
    enable = false

    # Address defines the gRPC server address to bind to.
    address = "0.0.0.0:9090"

    ###############################################################################
    ###                        gRPC Web Configuration                           ###
    ###############################################################################

    [grpc-web]

    # GRPCWebEnable defines if the gRPC-web should be enabled.
    # NOTE: gRPC must also be enabled, otherwise, this configuration is a no-op.
    enable = false

    # Address defines the gRPC-web server address to bind to.
    address = "0.0.0.0:9091"

    # EnableUnsafeCORS defines if CORS should be enabled (unsafe - use it at your own risk).
    enable-unsafe-cors = false

    ###############################################################################
    ###                        State Sync Configuration                         ###
    ###############################################################################

    # State sync snapshots allow other nodes to rapidly join the network without replaying historical
    # blocks, instead downloading and applying a snapshot of the application state at a given height.
    [state-sync]

    # snapshot-interval specifies the block interval at which local state sync snapshots are
    # taken (0 to disable). Must be a multiple of pruning-keep-every.
    snapshot-interval = 1000

    # snapshot-keep-recent specifies the number of recent snapshots to keep and serve (0 to keep all).
    snapshot-keep-recent = 10

    ###############################################################################
    ###                         Store / State Streaming                         ###
    ###############################################################################

    [store]
    streamers = []

    [streamers]
    [streamers.file]
    keys = ["*", ]
    write_dir = ""
    prefix = ""

    # output-metadata specifies if output the metadata file which includes the abci request/responses 
    # during processing the block.
    output-metadata = "true"

    # stop-node-on-error specifies if propagate the file streamer errors to consensus state machine.
    stop-node-on-error = "true"

    # fsync specifies if call fsync after writing the files.
    fsync = "false"

[MSalopek]

Thank you for providing your configs. You seem to have an older config (used for gaia up to v16.0.0). Please try the following settings.

After the settings I've written some questions about your proxy config (nginx and similar).

• config.toml

    #######################################################
    ###       RPC Server Configuration Options          ###
    #######################################################
    [rpc]

    cors_allowed_origins = ["*"] # allow only origins you wish to support

    
    #######################################################
    ###           P2P Configuration Options             ###
    #######################################################
    [p2p]

    # Maximum number of inbound peers
    max_num_inbound_peers = 10


    #######################################################
    ###          Mempool Configuration Option          ###
    #######################################################
    [mempool]

    recheck = false # if you are not accepting Txs from the network, set to false

    # Maximum number of transactions in the mempool
    size = 100

    max_txs_bytes = 104857600 # 100MB

    cache_size = 1000
    keep-invalid-txs-in-cache = false
    max_tx_bytes = 1048576

    #######################################################
    ###   Transaction Indexer Configuration Options     ###
    #######################################################
    [tx_index]

    indexer = "null" # turn off indexing if you don't need it

• app.toml -> nothing to suggest

This has been reported before and discussed in various issues. We've already answered a similar request here: #2415 (comment).

Some of the staking queries are quite expensive and can degrade node performance after a while. Related discussion is here: #2726. The issue is on cosmos-sdk side and cannot be resolved on gaia repo itself.

If you are able, please consider adding a proxy in front of your RPC node that would either cache the reponses for staking endpoints or rate limit them.

To confirm which endpoints are killing your node, try disabling access to all staking endpoints (using a proxy or an API gateway).
Additionally, only allow origins that you trust and don't use cors_allowed_origins = ["*"] unless you indent to operate a publicly available node.

[jay-ginco]

Hi @MSalopek thanks for response, few inputs

1. We are not running our full node as a validator, but as a RPC full node with default pruning providing api endpoints

2. Whats the recommended memory in above case?

3. We have enabled sate-sync, just a doubt, if I recover my node from snapshot available from polkachu, but my state-sync settings are having older block height, does that make any difference? As I am not bootstrapping my node so I beilieve to will not use state-sync

[MSalopek]

Thanks for your reply.

We are not running our full node as a validator, but as a RPC full node with default pruning providing api endpoints

You can definitely reduce your mempool size and peers to the values provided above.

Whats the recommended memory in above case?

The 32GB should suffice. We've run nodes with 16GB without much problem.

We have enabled sate-sync, just a doubt, if I recover my node from snapshot available from polkachu, but my state-sync settings are having older block height, does that make any difference? As I am not bootstrapping my node so I beilieve to will not use state-sync

If you are loading from snapshot, statetsync will not be attempted (you already have local blockchain state).

Additionally, you may attempt to run pprof to see what your node is doing if the proxy configuration seems difficult.

[jay-ginco]

Hi @MSalopek setting the mempool configs as you suggested did not work, there is still memory leak issue persisting, quick ques, what affect does disabling indexer does, does it degrades performance for some queries or any other affect like some methods will not work

pprof metrics for your reference

curl -o heap.pprof http://localhost:6060/debug/pprof/heap

(pprof) top
Showing nodes accounting for 45.66GB, 94.75% of 48.19GB total
Dropped 1117 nodes (cum <= 0.24GB)
Showing top 10 nodes out of 91
      flat  flat%   sum%        cum   cum%
   15.58GB 32.32% 32.32%    20.39GB 42.32%  github.com/cosmos/iavl.MakeNode
   12.08GB 25.06% 57.38%    13.13GB 27.24%  github.com/syndtr/goleveldb/leveldb/table.(*Reader).find
    6.50GB 13.49% 70.87%    11.32GB 23.50%  github.com/cosmos/iavl/cache.(*lruCache).Add
    4.87GB 10.11% 80.98%     4.87GB 10.11%  container/list.(*List).insertValue (inline)
    1.68GB  3.48% 84.46%     1.68GB  3.48%  github.com/cosmos/iavl.(*NodeKey).GetKey (inline)
    1.64GB  3.39% 87.86%     1.64GB  3.39%  crypto/sha256.(*digest).Sum
    1.57GB  3.26% 91.11%     1.57GB  3.26%  github.com/cosmos/iavl.GetNodeKey (inline)
    0.82GB  1.70% 92.81%     0.82GB  1.70%  github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter
    0.54GB  1.13% 93.94%     0.57GB  1.19%  github.com/cosmos/gogoproto/proto.Marshal
    0.39GB  0.81% 94.75%     0.65GB  1.35%  github.com/cosmos/iavl.(*Node).clone

curl -o heap.pprof http://localhost:6060/debug/pprof/allocs

File: gaiad
Build ID: eb246ac98debe7855ea340e947b088262dfc1129
Type: alloc_space
Time: Nov 22, 2024 at 8:13am (UTC)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 153.86GB, 46.73% of 329.25GB total
Dropped 2609 nodes (cum <= 1.65GB)
Showing top 10 nodes out of 250
      flat  flat%   sum%        cum   cum%
   52.63GB 15.98% 15.98%    52.65GB 15.99%  github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter
   16.11GB  4.89% 20.88%    28.91GB  8.78%  github.com/cosmos/iavl.MakeNode
   15.33GB  4.66% 25.53%    15.33GB  4.66%  github.com/syndtr/goleveldb/leveldb/cache.(*mBucket).get
   12.72GB  3.86% 29.40%   109.65GB 33.30%  github.com/syndtr/goleveldb/leveldb/table.(*Reader).find
   12.44GB  3.78% 33.18%    19.08GB  5.79%  github.com/cosmos/iavl/cache.(*lruCache).Add
    9.85GB  2.99% 36.17%     9.85GB  2.99%  container/list.(*List).insertValue (inline)
       9GB  2.73% 38.90%        9GB  2.73%  github.com/syndtr/goleveldb/leveldb/table.(*blockIter).Next
    8.88GB  2.70% 41.60%     8.88GB  2.70%  cosmossdk.io/store/rootmulti.(*Store).Snapshot.func2
    8.65GB  2.63% 44.23%     8.65GB  2.63%  github.com/syndtr/goleveldb/leveldb/table.(*Writer).writeBlock
    8.25GB  2.50% 46.73%    12.48GB  3.79%  github.com/syndtr/goleveldb/leveldb/cache.(*lru).Promote

Also,

One obeservation while investigating memory for our node, we see below metrics

/ # cat /proc/$(pgrep gaiad)/status
Name:	gaiad
VmHWM:	104249408 kB
VmRSS:	102533984 kB
RssAnon:	102417264 kB
RssFile:	  116720 kB
RssShmem:	       0 kB

Seems like its using ~102 GB as resident memory in RAM, does that mean its storing the block or state data in memory, and not on disk, as for disk usage I see its only 50 GB any config that may apply here to change the same, kindly advice

[jay-ginco]

@MSalopek seems like tweaking iavl-cache-size = 100000000 value in app.toml to lower value makes the memory usage more stable, though its still linearly increasing, it reached to 40 GB in 3 days

[MSalopek]

Hey, thanks for reporting back.

As I'm looking at this, I think this should be escalated to the cosmos-sdk. We have reports from other networks where similar behavior was noticed.

Hi @MSalopek setting the mempool configs as you suggested did not work, there is still memory leak issue persisting, quick ques, what affect does disabling indexer does, does it degrades performance for some queries or any other affect like some methods will not work

This would prevent you from querying transaction data (by height, events and such).

EDIT:
Opened issue on cosmos-sdk: cosmos/cosmos-sdk#22657

[MSalopek]

@jay-ginco Btw, could you let us know if this problem persists when you use statesync or some other snapshot node?

I have a vague recollection that this was somehow related to cosmwasm in the past, but I'm trying to find the relevant issues and threads.

[faddat]

@jay-ginco cosmwasm leaks ram very badly.

I think there's one in the SDK too, but much smaller.

I'm basing this on running a lot of nodes and watching their instrumentation.