how to change skipper maxTimeToBuffer attribute

Question

I am using skipper and skipper-azure to upload multiple files at a time, if I upload small number of files lets say 20 to 30 at a time every thing works fine but if I upload more files like 200 or 300 I start receiving the following error for some of the files i.e only for two or three files not all files.

An Upstream timed out before it was plugged into a receiver

In sails>node_modules>skipper>Standalone>Upstream>Upstream.js file there is an attribute maxTimeToBuffer and its default value is 4500 when i change this to 10000 my code works fine I tested it over a 100 times.

My questions are

• What is the impact of this change ?
• Is there any place where I can override this configuration instead of changing it main file i don't want this configuration to go away with every update.

Thanking you guys for any help in advance

Trestle

You should normally be able to configure skipper in sails.config.http.middleware.bodyParser as described here, but I'm not sure maxTimeToBuffer can be configured from skipper's contructor (perhaps maxBufferTime corresponds to that setting?) – Bestrew

Answer 1

I had the same problem... You have to modify the http.js file located in the config folder... you have to do something like this:

passportInit: require('passport').initialize(),
passportSession: require('passport').session(),

bodyParser: (function _configureBodyParser(){
  var skipper = require('skipper');
  var middlewareFn = skipper({
    strict: true,
    maxTimeToBuffer: 100000,
  });
  return middlewareFn;
})(),

order: [
  'cookieParser',
  'session',
  'passportInit',
  'passportSession',
  'bodyParser',
  'compress',
  'poweredBy',
  '$custom',
  'router',
  'www',
  'favicon',
]

}

Sweatshop

Answer 2

In sails.js v0.12, the property maxTimeToBuffer is a customizable property that can be added to the config/http.js file to overwrite the default value of 4500ms i.e. The maximum number of miliseconds to wait for any given live upstream to be plugged in to a receiver after it begins receiving an incoming file upload.

To use customizable properties, you must first uncomment the direct call to require skipper.

// bodyParser: require('skipper')({strict: true})

With regards to the impact of this change, this is clarified in the official sails documentation:

The maximum number of miliseconds to wait for any given live upstream to be plugged in to a receiver after it begins receiving an incoming file upload. Skipper pauses upstreams to allow custom code in your app's policies and controller actions to run (e.g. doing database lookups) before you "plug in" the incoming file uploads (e.g. req.file('avatar').upload(...)) into your desired upload target (local disk, S3, gridfs, etc). Incoming bytes are managed using a combination of buffering and TCP backpressure built in to Node.js streams. The max buffer time is a configurable layer of defense to protect against denial of service attacks that attempt to flood servers with pending file uploads. If the timeout is exceeded, an EMAXBUFFER error will fire. The best defense against these types of attacks is to plug incoming file uploads into receivers as early as possible at the top of your controller actions.

Ar