Partial read of (binary) JSON file

[Chrismarsh]

I have text JSON files that have GB of data in them, stored as arrays of numbers. Further, I need to do distributed parallel processing (via MPI) on these data and would really like a way to read a sub-set of the arrays. That is, be able to read some sub-set of the data in memory without having to load the entire JSON file on each MPI process.

Is such a feature available or planned for either text or binary JSON files?

[nlohmann]

What do you know about the arrays? Meaning - how do you decide what to skip? Is there something like a byte offset?

[Chrismarsh]

I have metadata regarding the length of the arrays. Then I would, ideally, just read some subset of that array. For example MPI process 0 would read indexes 0-20000 and and MPI process 1 would read indexes 20001-50000.

If I had it stored as binary, it could be a byte offset, yes.

If you want more details, the schema I use is described here
https://mesher-hydro.readthedocs.io/en/latest/output.html#mesh
The 3 arrays under the 'mesh' key would be what I would read subsets of.

[nlohmann]

You can use the SAX interface and use sax_parse to define your desired behavior.

[Chrismarsh]

As I understand it from looking at that interface I could just discard the first n streamed values from my array until I got to, e.g., n=20001?

[igormcoelho]

@Chrismarsh I don't know if this is answered already, but I think I have a very similar problem. This could be adapted into your problem, just by pre-reading all "small" info (fields "mesh": { and "nvertex": int,), and leaving the following strategy to the "big" part (the 3 big arrays). Me too, I have quite large arrays, organized such as (simplified example... there can be other json structures inside array elements):

[[1,2,3], [4,5], [6,7,8,9]]

I want to partially read the first array (located at offset 1), but in my case, I don't know where it ends... and it's quite large, so I don't want to parse the whole thing.
If I skip to first position, and try to read, nlohmann::json will complain (because of token ,):

    std::istringstream is8("[1,2,3], [4,5], [6,7,8,9]]");
    nlohmann::json jj8 = nlohmann::json::parse(is8);

terminate called after throwing an instance of 'nlohmann::detail::parse_error'
  what():  [json.exception.parse_error.101] parse error at line 1, column 8: syntax error while parsing value - unexpected ','; expected end of input

I tried to try-catch, but exception handling won't give me a partial result, or even the string consumed...
Now it comes the "solution" I found, sorry if it looks like some heresy @nlohmann!
I created some "custom adapter" that requires method get_character from any class T, while keeping track of consumed chars (by nlohmann::json), and then, I redo the process (yes, twice!) only within the correct segment (now I know where it ends!). Maybe there's some better way... Anyway, here it goes (it works!):

// this is my code, before #include<nlohmann/json>
namespace nlohmann
{
namespace detail
{
template<class T>
class custom_stream_adapter
{
  public:
    using char_type = typename T::char_type;
    std::shared_ptr<T> adapter;
    
    ~custom_stream_adapter()
    {
        adapter = nullptr;
    }

    explicit custom_stream_adapter(std::shared_ptr<T> _adapter)
        : adapter(_adapter)
    {}

    // delete because of pointer members
    custom_stream_adapter(const custom_stream_adapter&) = delete;
    custom_stream_adapter& operator=(custom_stream_adapter&) = delete;
    custom_stream_adapter& operator=(custom_stream_adapter&&) = delete;

    custom_stream_adapter(custom_stream_adapter&& rhs) noexcept
        : adapter(rhs.adapter)
    {
        rhs.adapter = nullptr;
    }

    // std::istream/std::streambuf use std::char_traits<char>::to_int_type, to
    // ensure that std::char_traits<char>::eof() and the character 0xFF do not
    // end up as the same value, eg. 0xFFFFFFFF.
    std::char_traits<char>::int_type get_character()
    {
        return adapter->get_character();
    }
};

template<class T>
inline custom_stream_adapter<T> input_adapter(std::shared_ptr<T>& stream_ptr)
{
    return custom_stream_adapter<T>(stream_ptr);
}

} // namespace
} // namespace


#include<nlohmann/json.hpp>


// this is my custom stream with 'cache'
struct CacheStream
{
    using char_type = char;
    std::istringstream* is;
    std::string cache;

    CacheStream(std::istringstream* _is) :
       is(_is)
    {
    }

    std::char_traits<char>::int_type get_character() {
        int c = is->get();
        cache += c;
        return c;
    }
};

int main() {
std::istringstream is6("{\"A\": 10},{\"B\":20}");
    auto sptr = std::make_shared<CacheStream>(CacheStream(&is6));
    nlohmann::json jj6;
    std::string again;
    try {
        jj6 = nlohmann::json::parse(sptr);
    } 
    catch(std::exception& e) {
        //std::cout << "BAD READ! TRY AGAIN..." << std::endl;
        again = sptr->cache;
        again.pop_back();
    }
    try {
        jj6 = nlohmann::json::parse(again);
    } 
    catch(std::exception& e) {
        //std::cout << "REALLY BAD READ! NO TRY AGAIN..." << std::endl;
    }

    // now it works!
    std::cout << jj6 << std::endl;
}

I know it's ugly, but it works #:joy:

[Chrismarsh]

Ah interesting! We ended up spinning out wheels on this for a bit. Eventually we punting by moving to a HDF5 data structure where the binary offsets are clear. Some of our thinking changed and once we were looking at O(100GB)+ we decided the HDF5 option was likely better than trying to shoe-horn json into it.

I have other work with geojson files that partial indesing could be quite usful. I will have to come back to that later though!