Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functions for stream encoding/decoding of big strings #133

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

xakep666
Copy link

This may be useful i.e. when your service should download file and send it inside json to other service without storing whole file in memory.
In my case service downloads a file from S3-like service and makes request to our email provider to send it as attachment (base64 encoded).
Currently I'm using this snippet:

func (a *Attach) MarshalJSONObject(e *gojay.Encoder) {
	e.StringKey("name", a.Name)
	e.StringKeyOmitEmpty("encoding", a.Encoding)

	// attach content handling
	if a.Content != nil {
		e.AppendByte(',')                                 // begin new field
		e.AppendBytes([]byte(`"content"`))                // appends "content" key
		e.AppendByte(':')                                 // delimiter between key and value
		e.AppendByte('"')                                 // string begins
		_, err := io.Copy((*encoderWriter)(e), a.Content) // big string (unsafe, no escaping)
		if err != nil {
			panic(err) // caught by caller
		}
		e.AppendByte('"') // string ends
	}
}

My changes allows to shorten it to this:

func (a *Attach) MarshalJSONObject(e *gojay.Encoder) {
	e.StringKey("name", a.Name)
	e.StringKeyOmitEmpty("encoding", a.Encoding)
        e.ReaderToBase64Key("content", a.Content, base64.StdEncoding)
}

This PR adds similar methods for decoding (may be useful i.e. when you get response where file goes as base64-encoded string and this file should be uploaded to somewhere).

Side changes:

  • decoder: better buffer reusability when parsing escape sequences
  • encoder: flush buffer when it's size reaches a limit provided in SetBufFlushThreshold if target writer provided (to reduce memory consumption). Should close Periodically flush the buf to the Writer. #78

This made to reuse existing data buffer without (re)allocations
when decoder tries to parse escape sequence
Buffer flushing happens when target writer provided
and buffer size reaches value provided in SetBufFlushThreshold
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Periodically flush the buf to the Writer.
1 participant