Handling of Null or Empty Fields in JSON Serialization for gRPC Responses

[cubxxw]

Problem Description

In our current implementation using protocol buffers with gRPC in Go, we've encountered an issue where JSON serialization omits fields with null or empty values due to the omitempty option. This behavior is not always desirable, especially when the client expects all fields, regardless of whether they hold a value or not.

Impact

This behavior can lead to inconsistent API responses and potential misunderstandings on the client side, as the client might expect all defined fields to be present in the response.

Possible Solution

We are looking for a solution that allows us to include all defined fields in the JSON response, even if their value is null or empty, without modifying the default behavior set by the omitempty option in the protocol buffer compiler for Go.

Considerations

• We would like to maintain the integrity of the protocol buffer specifications and avoid custom plugins if possible.
• It's essential that the solution aligns with best practices for JSON serialization in gRPC while ensuring backward compatibility.

References

We've noted that similar issues have been discussed in the context of the Go-Kratos framework (e.g., go-kratos/kratos#1539 and go-kratos/kratos#1952). We're looking for advice or solutions that may have been identified in these or other discussions.

Request

We kindly request guidance on how to approach this serialization concern while adhering to the gRPC and protocol buffer standards in Go. Any suggestions or recommendations for practices or tools to resolve this issue would be greatly appreciated.

[kubbot]

This issue is available for anyone to work on. Make sure to reference this issue in your pull request. ✨ Thank you for your contribution! ✨
Join slack 🤖 to connect and communicate with our developers.
If you wish to accept this assignment, please leave a comment in the comments section: /accept.🎯

[kvii]

apiresp.ApiSuccess can use this code:

var marshalOptions = protojson.MarshalOptions{
	EmitUnpopulated: true,
	UseEnumNumbers:  true,
}

func ApiSuccess(data any) *ApiResponse {
	if format, ok := data.(ApiFormat); ok {
		format.ApiFormat()
	}
	if isAllFieldsPrivate(data) {
		return &ApiResponse{}
	}
	if m, ok := data.(proto.Message); ok {
		b, err := marshalOptions.Marshal(m)
		if err == nil {
			return &ApiResponse{
				Data: json.RawMessage(b),
			}
		}
	}

	return &ApiResponse{
		Data: data,
	}
}

Note: protojson marshal int64 field to string (1 -> "1"). Cause json not support number > 2^53. We can use double type in field to avoid this.

[kubbot]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[kubbot]

This issue was closed because it has been stalled for 7 days with no activity.

[kubbot]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.