Support JSON streaming formats besides application/stream+json

[spring-projects-issues]

Sebastiaan van Erk opened SPR-16742 and commented

The mime type used by Spring Web Reactive for streaming json does not seem to be correct. The mime type "application/stream+json" is for an obsolete standard for activity streams:

https://tools.ietf.org/id/draft-snell-activity-streams-type-01.html

It has been superseded by the standard:

https://www.w3.org/TR/activitystreams-core/

with mime type "application/activity+json".

However, an activity stream is not the same as streaming json, so neither of these mime types should be used.

I was unable to find any standard mime type for json streaming, so I'm not sure what the resolution should be. The mime type should not be in the standards tree (since it is not a standard mime type), and the "x-" tree seems to be deprecated (since 1996) (see https://en.wikipedia.org/wiki/Media_type).

But I'm not sure that using a deprecated media type in the standards tree which is not officially registered, was invented for a different purpose, and has been superseded, is the way to go.

---

Affects: 5.0.5

Issue Links:

• Support Protobuf serialization in WebFlux [SPR-15776] #20331 Support Protobuf serialization in WebFlux

1 votes, 6 watchers

[spring-projects-issues]

Rossen Stoyanchev commented

Indeed the "application/stream+json" seems to have originated from Activity Streams and is not the same as streaming JSON. Sébastien Deleuze do you have any further insight? 

Technically, we could stream via "application/json", effectively producing or consuming a JSON array. However it is useful for both client and server to differentiate between a finite array and a long running stream. Perhaps all we need is a custom parameter on "application/json"?

Note that we have a similar question for streaming with Protobuf in #20331. 

 

[spring-projects-issues]

Sebastiaan van Erk commented

I also checked application/json spec (https://tools.ietf.org/html/rfc4627) and the spec says that it must contain a valid JSON-text which is an object or an array according to the grammer defined in the RFC. Unfortunately a sequence of 2 json objects/arrays is not a valid JSON-text anymore. :( 

It's suprising to me that there isn't anything standard for this, seems like a very common use case. I've seen some attempts at standardization (e.g., http://ndjson.org/), but they don't seem to be used a lot. The ndjson seems nice (the fact that it allows any json value instead of only objects or arrays is nice), but I do find the single line per value rather restrictive.

[spring-projects-issues]

Sébastien Deleuze commented

I agree that current mime type is not ideal.

I am not sure about just using application/json since for streaming we are not producing valid JSON as pointed out by Sebastiaan van Erk.

In fact there is an official RFC for JSON streams: JavaScript Object Notation (JSON) Text Sequences aka RFC7464 which defines application/json-seq media type, where record separators can be used to delimit JSON sequences.
Supporting it would make sense even if we can't use that as a default because it would be a breaking change.

Our current implementation supports both line-delimited JSON and concatenated JSON.
ndjson has indeed started and RFC draft, but not it seems it has not been submitted.

As pointed out by Rossen Stoyanchev, we have similar question for Protobuf in #20331, but also for Smile streaming support introduced via #20699 and I am pretty sure that the question will arise for other formats as well.

Even if it does not exist yet, I think a flexible solution would something like be a +stream media type suffix with a boundary parameter that could take following values:

• self for self delimiting like concatenated JSON
• length like what we need for Protobuf
• separator for specifying a custom separator via additional custom prefix and suffix parameters (by default it could be ASCII Record Separator (0x1E) prefix and LF suffix like in application/json-seq).

Given +zip official support that does not seem unreasonable to me. We could optionally submit this as an addition to RFC6839.

So in a nutshell in this proposal, we would use application/json+stream; boundary=self, application/x-jackson-smile+stream; boundary=self and application/x-protobuf+stream; boundary=length as defaults for streaming, add support for application/json-seq (equivalent to application/json+stream; boundary=separator; prefix=%1E; suffix=%0A), keep support for application/stream+json and application/x-jackson-smile+json as non defaults for compatibility and deprecate MediaType#APPLICATION_STREAM_JSON + MediaType#APPLICATION_STREAM_JSON_VALUE.

One of the benefits of that approach is that we could implement streaming support in a generic way.

Any thoughts?

[spring-projects-issues]

Sébastien Deleuze commented

As suggested by Rossen Stoyanchev (we discussed that yesterday more in detail), another solution would be to serialize JSON streams as arrays since our JSON decoder is able to decode them in a streaming way too. That would make the output valid from a JSON POV (even if the closing bracket will never be reached for infinite streams), allow usage of application/json with an additional parameter. Same is also applicable to Smile.

Brian Clozel rightfully noticed that RFC7464 has been mainly introduced to help non streaming parser/encoder to support JSON streaming that were not widely available at the time the spec was introduced, so Spring WebFlux is maybe not the typical target for such support since we have streaming support out of the box. Based on this information, we can probably wait somebody has a real use case before supporting it.

It seems we agree on deprecating MediaType#APPLICATION_STREAM_JSON + MediaType#APPLICATION_STREAM_JSON_VALUE.

Next step is deciding what parameter to use for JSON, Smile and Protobuf streaming.

[spring-projects-issues]

Sébastien Deleuze commented

We have a working solution for Protobuf and no clear industry standard for JSON streaming yet, so I prefer to keep current arrangement for 5.1.

[spring-projects-issues]

Rossen Stoyanchev commented

A few APIs use "application/json" but return line-delimited JSON data, e.g. Oanda and Flowdock.

Neo4j API expects an "X-Stream: true" request header and responds with "Content-Type: application/json; stream=true".

JS client libraries support parsing in streaming mode. JSONStream takes a JSON path like JSONStream.parse('rows.*.doc') where "rows" is a top-level Object in the JSON with an array where each value has a "doc" element and provides the doc elements in streaming fashion. Similar feature in Oboe.js with something like .node('foods.*', function(foodThing ){ ... }). This shows clients can process a JSON array as a stream.

[membersound]

Is there a decision if application/json+stream will be deprecated in future? If so, what stream mediaType should I use if I'd want to output json as stream?

{"id":5, "name":"john"}
{"id":6, "name":"doe"}
{"id":7, "name":"jane"}
...

[rstoyanchev]

@sdeleuze maybe we should take another look for 5.2?

I see JSON Lines and ndjson (with "application/x-ndjson" used in Elasticsearch). There is also JSON Sequences RFC.

[makdeniss]

Any decision made on this? FluentD f.e. defaults to ndjson now...

[raman-nbg]

Is there any update on this? What would be the best way to do JSON streaming right now?

[rstoyanchev]

Scheduling tentatively for 5.3 M1 to try and add support for additional JSON streaming formats. We could also deprecate "application/stream+json".

[rstoyanchev]

I've deprecated "application/stream+json" and added "application/x-ndjson". The latter is now also supported out of the box in the Jackson Encoder and Decoder. While it is not an officially registered media type, at least it's a common convention that seems to be in use while JSON lines doesn't even have that. There is also JSON Text Sequences and that is possible to use still with newline as the separator but "application/json-seq" would need to be registered with the Jackson Encoder and Decoder. As for separators other than newline, I'm not sure that Jackson's non-blocking parser supports that.

In summary, use line-delimited JSON with "application/x-ndjson" supported as a media type out of the box and used as a replacement for "application/stream+json". Alternatively, register any other mime type and as long as it's a line-delimited JSON format it'll work the same. The mime type itself is just a convention for describing the format and as long as both sides understand it, it's enough.