Architecture: Combine several JavaScript projects (modular) [closed]

Question

I would like to create a clean archtecture for my JavaScript project. The project consists of one Node.js server and two separate Angular.js front-ends with different purposes. For building the front-ends I use a custom grunt build each. The build results in one single HTML file per project and two minified/uglified CSS and JavaScript files. Each front-end is then running on a separate minimal version of a Node server (serves only the static files).

So far, so clear. The goal is now to make it possible to add plugin modules to each one of the three core projects. A module should extend the JavaScript of either one of the projects. This means e.g. in case of one front-end to add an additional angular module to the angular configuration. I already know where and how to add the angular module code into the core app.

The problem is now: How do you create a reasonable build process over multiple projects which is also depending on plugin modules? I came up with two solutions.

1. I could add a plugin as an NPM dependency to one of the core projects. This has however the drawback, that every change in the module needs to be pushed to NPM and it is not very convenient to develop the plugin module like this. A plugin is not runnable on its own.
2. I could have a list of plugins inside the Gruntfile of one of the core projects. This list would contain local file paths to the modules. In the build of the core module, the builds of the plugins will be executed. This would however include to watch changes of the built files of the plugins. It is an unclean solution.
3. I could have another project which contains dependencies to the core project and all the plugins and builds it all together. The question of how to add the dependency remains (case 1 or 2)

How would you solve that problem?

Dovap

Probably want to look into using webpack instead of grunt. – Borszcz

@Borszcz Could you explain how you would achieve an architecture and a nice development environment with webpack over multiple projects? – Dovap

Am afraid i really am not well versed in using it as it doesn't fit into most of what i do. i only through it out there as I believe it would help since it pulls from node modules – Borszcz

For solution #1 I have similar problem while developing modules pulled via composer. My solution is that for development I symlink those modules. So I have realtime updated dependencies, but they are located like if were pulled via package manager. – Braddock

@Borszcz what benefit does grunt provide over webpack? – Lorusso

you can also use "some-local-package": "file:./to/the/pkg/dir" if I'm not mistaken. This way your local changes can be included but for a production / ci build you would need to publish those changes to whatever registry you are using. – Neela

Answer 1

IMHO, it's not that related to task manager work (aka gulp, grunt or even webpack, it doesn't really matter here). The community goes to a place where you own lots of (relatively) tiny node modules that do one thing and do it well, so it's pretty connected to your first suggestion.

A VCS repo could look like this structure:

my-repo/
  package-1/
    package.json
  package-2/
    package.json
  package-3/
    package.json
  ...

... then you work with npm link to make symlinks inside the modules themselves, that way you don't have to publish the modules for getting updates.

There's a pretty new package called lerna that does exactly this npm link thingy, automatically (it detects your dependencies graph in your "local" modules and make a link between them), all you need to do is follow their structure. In addition, it makes publish smartly, you can run commands in all packages and bunch of other related things.

Babel, React are great examples of this modules-separation-of-concern. Babel works with lerna to automate the linking between the packages, here are their reasons to have a monorepo.

Chockfull

Thanks for your answer! I took a look at Lerna and it seems pretty promising. Could you elaborate, how one can use this? The official docs are pretty bad. As far as I understood: you have several projects, which are then copied/linked to the ./packages/ folder. How does Lerna recognize, that your local copy of a repository should be linked instead of taking the one from NPM? I also do not really get the link between the two terms modules-separation-of-concern and monorepo, as it sounds like the opposite? – Dovap

Lerna maps the modules you currently have in your repo and link accordingly. Say you have modules a, b and a depends on b, it knows to run npm link in b, then npm link b in a. Regarding the two terms, they aggregate their modules in one repo, the repo is just a way to organize them, the modules themselves are still separated. Here's another resource about lerna btw (just published): blog.cloudflare.com/cf-ui/… – Chockfull

Answer 2

I usually do exactly what you have described under 1. But I use local dependencies that will be concatenated by my build chain.

npm install ~/project_root/sub_project --save-dev

this way you do not need to push it all the time. you simply delete the node_module/sub_project folder and run npm install again inside your build chain.

with browserify you can then add your dependencies to your angular project.

A plugin is not runnable on its own.

I usually develop my services or directives as regular JavaScript classes and wrap them with angular.

var Service = require("./path/to/service.js")
module.exports = [function() {
    return {
        restrict: 'AC',
        link: function($scope, $element) {
            var someService = new Service();
            $scope.$watch("whatever",function(value){
                service.doSomething();
            })
        }
    };
}];

This way you can run them and test them independently from angular. This also comes in advantage when working with WebWorkers. Since they can not really(fully) be used with angular 1.X.

Devi

Thank you for your answer. I already found the solution with local paths in NPM but wasn't that satisfied with the manual deletion the node modules and rerunning npm install, because I'm usually using a livereloading build process.. Do you know any way to still use autoreloading while developing a plugin? – Dovap

If you use gulp-watch you could only scan your final builds for autoreloading. Gulp watch kind of forwards this functionality to your reloader. – Devi

The problem/advantage of the build chains is that they are made for test driven development. I know this is not satisfying your question but worth thinking about. – Devi

Answer 3

I had a similar problem a few months ago. Installing dynamic dependencies could be solved fairly easy since you are familiar with streaming building systems like Grunt or Gulp.

1. You may allocate a property inside your package.json like dependencies or devdependencies which will hold all additional info you need, bower.json or even a custom JSON file will do the trick too. I prefer NPM packaging manifests or Bower manifests as they provide a valid entry point for each module.
2. Using grunt-auto-install you can dynamically install any additional module dependencies for your building process, the ones that you can parse from your project's main package.json file.
3. Using each module's main property from each additional package.json will give you a file list to initialize an uglify or concatenating task .

Brethren

Thanks for your answer! So do you propose a link to the NPM modules like this: "project_a": "file:../project_a" or do you propose not to use NPM to retrieve the dependencies, but Grunt instead? How and where do you read the main property of each package.json? In the Gruntfile? How would you watch changes in plugin modules? – Dovap

Hello there, I would suggest using NPM or Bower to grab dependencies. Using Grunt you may install them, version numbers will keep track on your changes. Afterwards, a new Grunt task will loop each additional package.json or bower.json from the installed properties. Using main property of each will give you the actual entry point of your JavaScript files. More about main property could be found on NPM docs: docs.npmjs.com/files/package.json#main – Brethren

But how would you link the NPM packages to your local repositories to e.g. live-reload the local files during the development? If you link the projects to git {"dependencies": { "project_a": "http://github.com/...." } } you can only update the project, if you push another version. This is cumbersome during development.. – Dovap

Use Bower local dependencies: https://mcmap.net/q/319959/-how-to-register-a-local-git-package-in-bower Or NPM local dependencies :https://mcmap.net/q/63388/-local-dependency-in-package-json You can keep a separate file with the versioning though and scale your project with ease. – Brethren

Answer 4

You're adding an unecessary complexity to your MEAN project.
Why do you concatenate the two html parts via grunt ???
You should be using the SPA architecture of AngularJS , there is a reason why they adopted that..one of them is not running in such problems.
Instead of concatenating the files , just configure the states of your SPA to call the required partial template from a html file.THAT'S IT !
And use webservices on your backend (I think you're doing that though) that way you're independent from the origin of data since you're consuming the URI regardless of which server it came from.
About the dependencies , you would use npm seamlessly either on front or back ends with the --save flag in order to add them to the package.json file,and then copy that file to the respective apps dirs and npm install .
But I really don't see where you're going with all this , if you'd explain a bit more about the functional requirements that would help in making a sense out of it..

Pincenez

The question is about the build process and not about running it in angular. Running an angular single page application is not the problem. Furthermore, I don't concat the HTML files into one HTML file. One tiny index.html results, the main HTML components are built together to minified JavaScript strings with Grunt (grunt-html2js and grunt-contrib-htmlmin). This way, the HTML components are always available after the initial page loading and the application reacts the fastest way possible. – Dovap

I guess, you didn't understand the main question. To simplify the problem, there is one core GitHub project which is depending on multiple "plugin" GitHub projects. Those projects are not required, but can be added to the core project and will then be added as a regular angular module. But first, the code of the several projects has to be merged (concatted or copied to the core project, that angular can pick the module up). This can be done with NPM, like described in the question, but it has some drawbacks and I'm looking for a better way to do this. – Dovap