Cadence storage interaction limit exceeded when retrieving large collections

[LibruaryNFT]

Current Behavior

When attempting to retrieve a large collection of NFTs using Cadence on mainnet, the transaction fails due to exceeding the storage interaction limit. This issue occurs both when trying to access the ownedNFTs dictionary and when using the forEachID method to paginate over the NFT IDs.

The following error is encountered:

[Error Code: 1106] max interaction with storage has exceeded the limit (used: 20000158 bytes, limit 20000000 bytes)

Even small batches of IDs fail to retrieve due to the interaction limit being exceeded.

Expected Behavior

I expect to be able to retrieve large collections of NFTs, either by using pagination or some other strategy without exceeding the storage interaction limit.

Steps To Reproduce

1. Write a Cadence script that retrieves NFT IDs from a large ownedNFTs collection using either direct access (ownedNFTs.keys) or forEachID.
2. Run the script against an account that owns a large number of NFTs.
3. Observe that the script fails with the storage interaction limit error.

Here are three different scripts which all fail on mainnet when trying to get the NFT IDs:

import NonFungibleToken from 0x1d7e57aa55817448
import Pinnacle from 0xedf9df96c92f4595

access(all) fun main(startIndex: Int, batchSize: Int): [UInt64] {

    let account = getAccount(0xedf9df96c92f4595)

    let collectionRef = account
        .capabilities
        .borrow<&Pinnacle.Collection>(/public/PinnacleCollection)
        ?? panic("Could not borrow a reference to the collection")

    var allNftIDs: [UInt64] = []
    var count: Int = 0
    var currentIndex: Int = 0

    // Define the callback function for forEachID
    let callback = fun (id: UInt64): Bool {
        // Skip NFTs until we reach the startIndex
        if currentIndex < startIndex {
            currentIndex = currentIndex + 1
            return true // Continue iterating
        }

        // Once past the startIndex, add to the current batch
        if count < batchSize {
            allNftIDs.append(id)
            count = count + 1
            currentIndex = currentIndex + 1
            return true // Continue iterating until the batch is full
        } else {
            return false // Stop iteration once we reach batchSize
        }
    }

    // Iterate over the IDs starting at startIndex
    collectionRef.forEachID(callback)

    return allNftIDs
}

import NonFungibleToken from 0x1d7e57aa55817448
import Pinnacle from 0xedf9df96c92f4595

access(all) fun main(): [UInt64] {

    let account = getAccount(0xedf9df96c92f4595)

    let collectionRef = account
        .capabilities
        .borrow<&Pinnacle.Collection>(/public/PinnacleCollection)
        ?? panic("Could not borrow a reference to the collection")


    // Get and return all the NFT IDs in the collection
    return collectionRef.getIDs()
}

import NonFungibleToken from 0x1d7e57aa55817448
import Pinnacle from 0xedf9df96c92f4595

access(all) fun main(accountAddress: Address, startIndex: Int, batchSize: Int): [UInt64] {
    let account = getAccount(accountAddress)

    let collectionRef = account
        .capabilities
        .borrow<&Pinnacle.Collection>(/public/PinnacleCollection)
        ?? panic("Could not borrow a reference to the collection")

    let ownedNFTs = collectionRef.ownedNFTs

    var batchedNftIDs: [UInt64] = []
    var currentIndex = 0

    // Callback function for iterating over the keys
    let callback = fun (key: UInt64): Bool {
        // Skip until we reach the startIndex
        if currentIndex < startIndex {
            currentIndex = currentIndex + 1
            return true // Continue iteration
        }

        // Add NFTs to the batch until the batchSize is reached
        if currentIndex >= startIndex && currentIndex < (startIndex + batchSize) {
            batchedNftIDs.append(key)
            currentIndex = currentIndex + 1
        }

        // Stop if we have reached the batch size
        return batchedNftIDs.length < batchSize
    }

    // Use `forEachKey` to iterate over the owned NFTs
    ownedNFTs.forEachKey(callback)

    return batchedNftIDs
}

Environment

- Cadence version: 2.0.1
- Network: Mainnet

[LibruaryNFT]

This is related as well:
https://github.com/dapperlabs/nba-smart-contracts/blob/master/contracts/TopShotShardedCollection.cdc

[j1010001]

related: #3544

[j1010001]

We need Investigate potential solutions before we can estimate size.