I'm having an issue with Node.js Streams/Buffers where they aren't being closed/flushed following first use. I have a read stream created from fs.createReadStream that I am piping to a custom write stream. The highWaterMark for each chunk is ~2MB (this is important). When I initially stream through a ~3MB file, it is handled in 2 chunks by my stream, first a ~2MB chunk, then a ~1MB chunk. This is expected.
On the second file that I pass through, the first chunk is only ~1MB. This is an issue. When I add up the bytes piped, I can see that clearly after the first file, the streams/buffers associated were not cleaned up properly. This can be shown with the following math:
The final chunk that is transferred of the first file is 875,837 bytes. The first chunk of the next file that is transferred is 1,221,316 bytes (expected: 2,097,152 bytes). When you add 875,837 to 1,221,316, you get 2,097,153, which is the high water mark I mentioned earlier (with an off by one error).
Here is the code I've got:
return new Promise(async (resolve, reject) => {
const maximalChunkedTransferSize = 2*1024*1024; // 2MB (Minimum amount, Autodesk recommends 5MB).
const pathToFile = await bucketManagement.locationOfBucketZip(bucketEntity.name);
let readFileStream = fs.createReadStream(pathToFile, { highWaterMark: maximalChunkedTransferSize });
let writeStream = new HttpAutodeskPutBucketObjectWriteStream(accessToken, bucketEntity);
readFileStream.pipe(writeStream);
writeStream.on("finish", () => {
resolve(writeStream.urn);
});
writeStream.on("error", err => reject("Putting the file into the bucket failed. " + err.message));
});
I've tried calling .destroy() on both the read and write stream. I've tried calling .end(), .unpipe(). None of these have worked. How can I destroy/flush the stream and/or underlying buffer so that the first chunk of the next file is the expected 2MB?