Large output files truncated to 64K when using node-sass

[asendecka]

When we tried to compile the css files using node-sass and the output file was bigger than 64K, the generated file was truncated exactly after reaching 64K, which messed our css completely.

The reason for it turned out to be using Popen (in webassets/filter/node_sass.py) with subprocess.PIPE for standard input and output in these lines:

proc = subprocess.Popen(args,
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    # shell: necessary on windows to execute
    # ruby files, but doesn't work on linux.
    shell=(os.name == 'nt'))
stdout, stderr = proc.communicate(_in.read().encode('utf-8'))

When we took a look at the Python docs for communicate we found that:

The data read is buffered in memory, so do not use this method if the data size is large or unlimited.

We managed to solve our issue with this commit, where instead of reading in memory we use temporary files.

However, we noticed that Popen with subprocess.PIPE is used all over the place in webassets, so it is worth to consider if it's a good idea to refactor the code to not limit output to 64K everywhere.

Happy to hear your thoughts on the issue. Thank you!