How about recode it to C/C++/Java

[billzt]

The python script is too slow (several hours) and use high memeory (~70G)

[ylxdzsw]

I have an optimized fork of an old version which features multi process and a C extension and is expected to be slightly faster. You can try it at https://github.com/ylxdzsw/dedup . run make before running to utilize the C extension.

[billzt]

Thanks, I'll have a try