Skip to content

Commit

Permalink
Optimize input reading and output writing
Browse files Browse the repository at this point in the history
This commit aims to fix issue #20.

Use the Emscripten FS.writeFile API for accepting XML input files,
instead of the createDataFile and especially the intArrayFromString
function. Those were inherited from the parent upstream project, but
this writeFile API seems to be simpler to use and performs better.

The bigger fix, though, is in the output side, as pushing one piece of
stdout (I guess it was pushing one byte at a time?) caused the
stdoutBuffer array to eventually grow so large that it'd throw

> RangeError [Error]: Invalid array length

when the output was very big, like when normalizing a big input XML, as
described in #20.

Here, too, we can switch to the print/printErr APIs, which seem to be not
only simpler but also more resilient to the input size growing.
  • Loading branch information
noppa committed Mar 18, 2024
1 parent c3d38ba commit a639621
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 11 deletions.
25 changes: 15 additions & 10 deletions src/worker-post.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,14 @@
const {parentPort} = require('worker_threads');
// #endif

function bytesToUtf8(buffer) {
return new TextDecoder().decode(Uint8Array.from(buffer));
}

const stdoutBuffer = [];
const stderrBuffer = [];
let stdout = '';
let stderr = '';

function onExit(exitCode) {
const message = {
exitCode,
stdout: bytesToUtf8(stdoutBuffer),
stderr: bytesToUtf8(stderrBuffer),
stdout,
stderr,
};
// #ifdef node
parentPort.postMessage(message);
Expand All @@ -37,8 +33,17 @@
Module({
inputFiles: data.inputFiles,
arguments: data.args,
stderr: stderrBuffer.push.bind(stderrBuffer),
stdout: stdoutBuffer.push.bind(stdoutBuffer),
// TODO: We could eagerly start sending stdout to the parent thread while
// waiting for more. Or we could probably use some other, more efficient
// Emscripten API for output communication in the first place.
// But this seems to work fine for now, better than pushing the stdout
// values to an array.
print(text) {
stdout += text;
},
printErr(text) {
stderr += text;
},
onExit,
wasmMemory,
// #ifdef browser
Expand Down
2 changes: 1 addition & 1 deletion src/worker-pre.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Module['preRun'] = function () {
Module['inputFiles'].forEach(function(inputFile) {
FS.createDataFile('/', inputFile['fileName'], intArrayFromString(inputFile['contents']), true, true);
FS.writeFile('/' + inputFile['fileName'], inputFile['contents']);
});
};

0 comments on commit a639621

Please sign in to comment.