Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bzip2: merge into Go standard library #58

Open
darren opened this issue Mar 17, 2017 · 12 comments
Open

bzip2: merge into Go standard library #58

darren opened this issue Mar 17, 2017 · 12 comments

Comments

@darren
Copy link

darren commented Mar 17, 2017

it hasn't been updated for a while, is it stable yet?
if it is, hope it can merged upstream and fix golang/go#4828

@dsnet dsnet changed the title what's the status of the bzip2 writer bzip2: merge into Go standard library Mar 17, 2017
@dsnet
Copy link
Owner

dsnet commented Mar 17, 2017

Hi, the version of bzip2.Writer in the development branch is very stable has gone through many months of fuzzing comparing it's implementation with that of the C version. I have not merged the development branch into master yet, but plan to do so in the near future.

In regards to golang/go#4828, the main roadblock is finding someone to be a suitable reviewer for the implementation. I'm not aware of anyone else who has a deep knowledge of how bzip2 works. @mdempsky and @nigeltao have reviewed a number of my compression related CLs (showing knowledge in LZ77 and Huffman encoding), but I don't know if they're knowledgeable about BWT. It also depends on the availability of free time on my behalf and those of the reviewers.

@nigeltao
Copy link

Happy to learn about BWT, but I'm really low on free time for the forseeable future. Sorry.

@mkaczanowski
Copy link

Hey, @dsnet any progress with merging bzip2 compressor into standard library? What help do you need to make it or what are the blockers?

@dsnet
Copy link
Owner

dsnet commented Sep 6, 2017

I am not currently working on merging it into the standard library. I would love to see that happen in the near future, but I am still blocked on finding appropriate reviewers.

The master branch of this repo has a stable implementation that has gone through CPU-years of fuzz testing.

cmarcelo added a commit to cmarcelo/mixer-tools that referenced this issue Dec 11, 2017
This package contains the code to do bzip2 compression. The Go
standard library only have a decompressor for this type of file.

While the repository has no releases, there's evidence the code is
stable enough. See

    "bzip2: merge into Go standard library"
    dsnet/compress#58

for details. The license for the package is a "BSD 3-clause" type.

Signed-off-by: Caio Marcelo de Oliveira Filho <[email protected]>
cmarcelo added a commit to cmarcelo/mixer-tools that referenced this issue Dec 12, 2017
This package contains the code to do bzip2 compression. The Go
standard library only have a decompressor for this type of file.

While the repository has no releases, there's evidence the code is
stable enough. See

    "bzip2: merge into Go standard library"
    dsnet/compress#58

for details. The license for the package is a "BSD 3-clause" type.

Note: the blank import was used to signal 'dep ensure' that it should
download this package.

Signed-off-by: Caio Marcelo de Oliveira Filho <[email protected]>
cmarcelo added a commit to cmarcelo/mixer-tools that referenced this issue Dec 20, 2017
This package contains the code to do bzip2 compression. The Go
standard library only have a decompressor for this type of file.

While the repository has no releases, there's evidence the code is
stable enough. See

    "bzip2: merge into Go standard library"
    dsnet/compress#58

for details. The license for the package is a "BSD 3-clause" type.

Note: the blank import was used to signal 'dep ensure' that it should
download this package.

Signed-off-by: Caio Marcelo de Oliveira Filho <[email protected]>
cmarcelo added a commit to cmarcelo/mixer-tools that referenced this issue Dec 23, 2017
This package contains the code to do bzip2 compression. The Go
standard library only have a decompressor for this type of file.

While the repository has no releases, there's evidence the code is
stable enough. See

    "bzip2: merge into Go standard library"
    dsnet/compress#58

for details. The license for the package is a "BSD 3-clause" type.

Note: the blank import was used to signal 'dep ensure' that it should
download this package.

Signed-off-by: Caio Marcelo de Oliveira Filho <[email protected]>
cmarcelo added a commit to cmarcelo/mixer-tools that referenced this issue Dec 23, 2017
This package contains the code to do bzip2 compression. The Go
standard library only have a decompressor for this type of file.

While the repository has no releases, there's evidence the code is
stable enough. See

    "bzip2: merge into Go standard library"
    dsnet/compress#58

for details. The license for the package is a "BSD 3-clause" type.

Note: the blank import was used to signal 'dep ensure' that it should
download this package.

Signed-off-by: Caio Marcelo de Oliveira Filho <[email protected]>
@retnuh
Copy link

retnuh commented Apr 16, 2018

As an outside observer (i.e. someone who would like to use the library), does this really need a reviewer with deep knowledge of compressors? As a user of the library, all I really care about is that the outputs are compatible with other standard bzip2 implementations out there, and that it doesn't do anything crazy. Wouldn't a test suite would cover that?

Surely it's more important that there is a decent implementation available in the standard library now, rather than a potential future "best" implementation?

@dsnet
Copy link
Owner

dsnet commented Apr 16, 2018

outputs are compatible with other standard bzip2 implementations out there

Review by a another person doesn't guarantee this, but history has shown that programmers are faulty and the number of extra eyeballs tend to decrease the occurrence of bugs.

Wouldn't a test suite would cover that?

History has also demonstrated that test suites are often sub-par at detecting correctness bugs.

@dmitshur
Copy link

Surely it's more important that there is a decent implementation available in the standard library now, rather than a potential future "best" implementation?

If this package has the needed functionality, why is its name or location important? If API stability is important, it can/should be vendored.

i.e. someone who would like to use the library

Is there something that's preventing the package from being used at its current location?

@retnuh
Copy link

retnuh commented Apr 17, 2018

@shurcooL yes some people want only things in the base go distribution. The issue is about including/merging bzip2 in the standard library - see the original issue description.

@dsnet sure I get that and am all in favor of reviews but there's an argument to be made that something that seems to be working quite well is better than nothing, right?

@darren
Copy link
Author

darren commented Apr 17, 2018

@shurcooL

If this package has the needed functionality, why is its name or location important? If API stability is important, it can/should be vendored.

It seems that you're a big fan of the amazing vendoring feature of go, so can you please remove net/http, encoding, compress, image etc from the standard library? If somebody needs then, JUST vendoring them, so simple and easy! I just don't need them while keep the go tool distribution bloated on my disk, pretty annoying:

du -sch  go1.10.1
357M	total

Oh, yes there's go 1.0 compatibility promise, can you PLEASE do this in go2.0?

@dsnet
Copy link
Owner

dsnet commented Apr 17, 2018

This is not the right place to discuss vendoring or whether the standard library should be split apart and versioned separately or even distributed. All of that is being discussed with the vgo proposal.

@dmitshur
Copy link

@darren I see your point.

I expect/hope this issue will eventually be resolved, when the conditions for it are right.

@kortschak
Copy link

Having put off work that needed a bzip2 compressor and only just recently having randomly stumbled across this, a strong argument for merging into the stdlib is the aspect of discoverability.

Has the path towards merging this into the standard library been walked any further along?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants