Meteor and the /private directory

Question

I'm using the /private directory in Meteor 1.0.3 at the moment to store and serve up pdf documents to the browser.

As an example, I have a folder structure like so:

/application-name
 /private
  /files
   /users
    /user-name
     /pdf-file1.pdf 

I have a template with a button click event. In this event I make a couple of calls to Meteor methods and finally a server side Iron Router go('render-pdf') method. In these Meteor methods I use fs node.js to:

(1) check if the /user-name directory exists, and if it doesn't I create it.

(2) create the pdf-file.pdf file

Then in the server side Iron Router go('render-pdf') route, again using fs node.js to:

(3) read the created pdf-file.pdf and

(4) finally render it to the browser

The problem is in step (1), when creating the /user-name directory, Meteor server restarts. In step (2), again Meteor server restarts.

But most importantly, the first time my code runs, and the directory does not exist (step (1)), I get an error.

I can then call the button event again, this time after the directory has been created, and the pdf is rendered fine.

The error looks like so:

Error: ENOENT, no such file or directory '/Users/myname/meteor/meteor-application/private/files/users/user-name/pdf-file.pdf' at Object.fs.openSync (fs.js:438:18) at Object.fs.readFileSync (fs.js:289:15) at [object Object].Router.route.name (meteor-application/both/routes.js:225:17) at boundNext (packages/iron:middleware-stack/lib/middleware_stack.js:251:1) at runWithEnvironment (packages/meteor/dynamics_nodejs.js:108:1) at packages/meteor/dynamics_nodejs.js:121:1 at [object Object].urlencodedParser (/Users/myname/.meteor/packages/iron_router/.1.0.7.15dqor4++os+web.browser+web.cordova/npm/node_modules/body-parser/lib/types/urlencoded.js:72:36) at packages/iron:router/lib/router.js:277:1 at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1) at [object Object].hookWithOptions (packages/iron:router/lib/router.js:276:1)

It's probably that when I get to the point step (4) of trying to render the file, it either doesn't exist yet or the application is restarting. The next time I try the application has already restarted and files exist.

I was under the impression that the /private directory provides a place to handle files that do not affect the execution of the application? To me this means, at runtime I can add whatever I want without the application restarting.

Little history

At first I used the /server directory with a ./folder-name subdirectory. This worked as when I added folder and files the application didn't restart. The downside is when I deployed Meteor using the great Meteor-up package (mup), the deployment bundle ignored these files unless I added a *.js file somewhere inside. And further, if I created the 'hidden' folder structure on my EC2 instance, the deployment would remove the directory.

So using /private folder solved this issue, or so I thought. The folder structure and 'assets' deployed. But the downside to this approach is when I add 'assets' to it, it seems to restart -- even though I though this wasn't something that was suppose to happen.

Question

How can I add 'assets' (in the form of directories and files) under the /private directory without the Meteor application restarting? If this can't be done, how can I add 'assets' anywhere only server side without the application restarting?

Please note

When I deploy to production, I'd like some of the folder structure to stay in place, for example:

/private/files/users

should say there, while the

/user-name 

directory can be dynamic. I only mention this because I've read if you do a /.directory-name, Meteor ignores the folder and its contents. But this includes deployments as well.

What I really need

A server side only folder that gets included in the deployment bundle, and when I add 'stuff' to it at runtime, doesn't restart my application...

Either a way to include /.hidden-folder in my mup deployment bundle or have the /private folder not restart every time I add stuff to it at runtime.

Brazell

Using the private directory to dynamically store and server stuff to users is riddled with complications - it's much better to use something like CollectionFS. – Gulgee

All I need is a server side only folder that I can deploy, and when I add stuff to it my application at runtime doesn't restart my application -- that's all really. – Brazell

@richsilv, thanks for the feedback. What complications? Why is using CollectionFS better (easier)? Creating dirs and files is so easy. I actually considered CollectionFS, but after doing a search for pdf issues (open or not) on their github page, what I found make me pause for now. Again, what I'm trying to do should be straight forward. I'm sure there's a simple answer waiting to be found. – Brazell

yes, i agree that just working with the filesystem should be easy, it's just that exactly the sort of problems you're having pushed me away from this approach in the past. and yes, CollectionFS is not a panacea, but it's worked for me. – Gulgee

thanks @richsilv. i understand where you're coming from. the thing is, for the pdf rendering stuff i'm trying to do, i actually read on the CollectionFS issues page about problems people were having trying to do exactly the same thing as me... so i thought to go another route for now. besides, i don't want to take on another big dependency (for a # of reasons) if I don't have to. again, I'm only talking about creating a directory and file in an application here. practically speaking, i wouldn't think a package would be needed in this case as this is nodejs's specialty. – Brazell

yes, i understand where you're coming from. there does seem to be a bit of a pdf problem that's waiting to be solved, not that i've ever been involved in it or had need for a solution. – Gulgee

Answer 1

In order to keep from:

(1) overwriting/removing the directory structure every time I deployed and,

(2) restarting the Meteor application every time I created a directory or file.

I decided in my case it just made sense to use a directory structure outside of the Meteor project instead of inside as before.

Something like Dropbox/users/user-name, or anything really.

I now believe that the /private and /public folders are more for static content than anything else.

I'm not really storing that many files yet, and some of them are only temporary anyhow, so this method will hold me over until I move to something like S3.

Please note:

(1) You need to give your Meteor user permissions to access the outside-the-project directory.

(2) Consider that this will take up space on your OS Instance HD.

(3) You'll need to use Node.js for file system calls. These calls are not wrapped in Meteor Fibers, so you're on your own in terms of async/sync programming.

Brazell