Node npm windows file paths are too long to install packages

Question

Situation

I want to use gulp and related front-end tool chains in Windows-hosted development environments. I'm hitting a wall trying to use gulp plug-ins like Browser-Sync, because the node_modules folder graph fans out making the windows file paths too long to copy the files. I'd like a pragmatic approach for handling this problem right now on Windows, irrespective of what the Node community may or may not provide to improve npm usability on the Windows in the future.

2 Questions

1. Is there an npm workflow for Windows that just works the way it was intended? "run the command and the files install" (e.g. comparable to npm on OSX, npm on Linux, ruby gems or even nuget) I don't want to fiddle with a bunch of manual file edits, symlinks, etc. every time I use npm on Windows.

2. Is there a well-documented, stable Cygwin workflow for npm and node execution to workaround the Windows API file path limits?

Gory details listed below...

General Problem

• Running npm install from a standard Windows command prompt fails on deeply nested node_modules hierarchies.
• Per Joyent's github repo thread, this is an acknowledged issue with no palatable workarounds for developers in Windows-centric environments. (Really?)
• NT Kernel supports file path lengths up to 32,767 characters.
• Windows API's MAXPATH is limited to 260 characters.
• Windows API handles file operations for all major Windows shells and whatnot including: Explorer, CMD, Powershell,MYSgit bash, etc. (MS really? How long has NTFS been around?)
• Cygwin supports long file paths, but npm.cmd doesn't work out-of-box due to crlf formatting. I tried the DOS2Unix transform on npm to get it working with Cygwin, but there seem to be other issues with this.

My Current Hack

• Create an "n" folder as a staging area on the root of C:\, because this shortens my folder path.
• Run npm inside the "n" folder to install modules for whatever I need.
• Fire up Cygwin and use cp to copy the node_modules folder into a destination project.
• Rinse and repeat when dependencies change or when I need to spin up a new project.

Other Unpalatable Workarounds

Symbolic Links can be used to shorten file paths, but these are kludgy hacks. As the npm ecosystem grows, nested dependency chains will become too long and this workaround become unusable.

Adding ALL dependences to the root folder's package.json file was mentioned in one thread I came across. Although this approach will flatten the folder structure and prevent loading of duplicate modules, this workaround feels unnatural. It also kills the usability, durability, and productivity of npm, because you have to fiddle with files and folders post-install either manually or with some hacky scripts. The approach is also vulnerable to the same fate that Symbolic Links approach may eventually suffer.

Blanding

I almost thought I had this solved. I got Cygwin working with npm by running dos2unix util on the following 2 files: npm.cmd and npm – Blanding

Windows API path limitations make npm unusable, because some npm modules use Visual Studio to build files. This is the error I receive when I npm Browser-Sync: C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppBuild.targets(301,5): error MS B3491: Could not write lines to file "Release\obj\validation\validation.tlog\validation.lastbuilds tate". The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters. – Blanding

I may have a "taffy-pulling" approach to get node modules loaded with npm on windows. It involves a few rounds of the following: npm install, npm dedupe, npm shrink, and rm -r node_modules. Doing this repeatedly seems to iron out the long file paths to some degree, but it's kind of like pulling taffy (e.g. not done until you're done). Anyone codified this or written an automated tool to make this more turnkey? – Blanding

Speaking of "hacky scripts", I wrote one that I don't find to be TERRIBLY hacky. I've created a tool called fenestrate which you can use to programmatically flatten the directory structure of your modules after installation. You can install it as a global npm postinstall hook. – Trigonal

Thanks. I'll give fenestrate a try. I don't see much input on this post and it's nearly 2 weeks old. Guessing the 260 char file path limitation is just going to be a challenge for Windows node development for the time being with no official workflow guidance per se. – Blanding

@Trigonal For fenestrate to work, I should fork each of my dependencies and run fenestrate make on them? – Harass

@Harass For personal use, and to get started quickly, fenestrate should recursively walk your node_modules folder, so you shouldn't need to run it manually on deep dependencies. However, it would be great to fork those dependencies--I think that a lot of forked modules with simple fenestrate configs would send a great message to npm maintainers. – Trigonal

Answer 1

The problem with deeply nested folders on Windows has been mostly solved starting with npm version 3.x.

According to npm:

.npm@3 makes the install "maximally flat" by hoisting everything it can to the top level node_modules. This means nesting only occurs on conflicts and as such, trees should never get very deep. As such, the windows path length limitation shouldn't be run into.

I have just installed npm 3.1.0 and tried it out on a package that was throwing the dreaded The specified path, file name, or both are too long error.

The problem went away.

You can get the latest npm builds from here : npm releases

Sticktight

I had also a success with the npm 3.x update on Windows machine. Shameless plug: I wrote an article about npm 3 on Windows triplet.fi/blog/… – Clawson

Answer 2

Windows 8.1 and 10 have an option to increase the Win32 path limit:

• Open Group Policy Editor (Press Windows+R and type gpedit.msc and hit Enter)
• Navigate to the following directory: Local Computer Policy\Computer Configuration\Administrative Templates\System\Filesystem
• Doubleclick on Enable Win32 long paths option and enable it.

Savanna

option wasn't available for me, and fwiw, i upgraded from win 7 pro so that's a possible cause – Roseroseann

@EvanMorrison "Filesystem\NTFS\Enable NTFS long paths" was renamed to "FileSystem\Enable Win32 long paths" in later win10 builds. I updated the answer for future reference. – Savanna

any idea for Win Server 2012 R2 – Welldefined

@Welldefined according to superuser.com/questions/1528789/…, it is unfortunately not possible – Pitiful

Answer 3

This is a work around solution.

There are some node modules that flattens your dependencies for you.
Links are here:

• npm-flatten
• npm-dedupe

What these modules are doing can be done manually as well. This is the only real solution exists as of now, i.e to have all your modules at a single level, requiring each other, instead of all having private copies of their dependencies nested deeply.

Kierkegaardian

I found flatten-packages to be well-documented and easy to use. – Anhydride

The link above for npm-dedupe is not valid anymore. I found the current existing one here: docs.npmjs.com/cli/v8/commands/npm-dedupe – Recover

Answer 4

Allan -

From the github issue you linked,

npm will add dedupe-at-install-time by default. This is significantly more feasible than Node's module system changing, but it is still not exactly trivial, and involves a lot of reworking of some long-entrenched patterns.

This is (finally) currently in the works at npm, going by the name multi-stage-install, and is targeted for npm@3. npm development lead Forrest Norvell is going to spend some time running on Windows in the new year, so please do create windows-related issues on the npm issue tracker < https://github.com/npm/npm/issues >

Rock

Answer 5

I have the same issue. Flattening the dependencies isn't a complete solution, since you might be using modules that depend on different versions of the same dependent module. I discovered the gulp-run module stopped working after flattening (related to module assumptions about bin/.bin directories, I suspect). Drat!

There's lots of discussion about the problem, but no solution in sight: https://github.com/joyent/node/issues/6960

https://github.com/npm/npm/issues/3697

A workaround that's working for me is to manually add dependencies my project doesn't explicitly need.

If you want to identify which packages are giving you problems, I found PathLengthChecker quite useful. Just extract the EXE and run the GUI or command line app. The other way I've uncovered the problem is to try to build in Visual Studio, but it fails without telling you which directory name is too long.

Here's a command line example of my workaround:

mkdir c:\reallylongdirectorywillbreakinwindows
cd c:\reallylongdirectorywillbreakinwindows
npm init
npm install --save-dev grunt-bower-task
PathLengthChecker.exe RootDirectory="C:\reallylongdirectorywillbreakinwindows" MinLength=260

I got back:

261: C:\reallylongdirectorywillbreakinwindows\node_modules\grunt-bower-task\node_modules\bower\node_modules\update-notifier\node_modules\latest-version\node_modules\package-json\no de_modules\registry-url\node_modules\npmconf\node_modules\config-chain\readme.markdown

[snip - there were 12 of them]

According to the npm ls command:

└─┬ [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├─┬ [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ └─┬ [email protected]
  │ │ │   └─┬ [email protected]
  │ │ │     └─┬ [email protected]
  │ │ │       ├─┬ [email protected]
  │ │ │       │ └── [email protected]

Let's go with npmconf - it's the container for all the over-length files that are causing issues. We need npmconf 2.1.1.

npm install --save-dev [email protected]
(now delete the node_modules directory - you may have to use Windows Explorer if you can't do it with rmdir /s)
npm install
PathLengthChecker.exe RootDirectory="C:\reallylongdirectorywillbreakinwindows" MinLength=260

No results - all files are within limits!

The obvious caveat here is that it only works once per package - dependencies on different versions of the same module can't be installed at the root node_modules level because node doesn't account for versions in the directory structure.

This workaround isn't perfect, but it solves my main goals of having node work on Windows, and since the resolution is right in package.json, the workaround works for other developers and build servers without any manual or global fussing.

Malony

Answer 6

If you are OK with installing it globally this could be a work-around:

You can adjust the path where npm is installing the global modules to something very short (usually it is: c:\users\\{username}\AppData\Roaming\npm\npm_modules) which already takes a lot of characters.

To adjust it see here: Change default global installation directory for node.js modules in Windows?

If you adjust it to, e.g., c:\n\ in some cases it might solve the issue.

Citadel

Answer 7

This is what finally fixed it for me...

After installing gulp and receiving errors, run... gulp

When you see a package failing, install it manually with --no-bin-link.

sudo npm install {package} --no-bin-link

Where {package} is the package that is having problems.

After all of this I was receiving an Error in plugin 'gulp-notify' Message: not found: notify-send.

This was due to a plugin issue with Vagrant. You can either turn off notifications..

export DISABLE_NOTIFIER=true;

Or install the plugin with Vagrant.

Best of luck.. I spent a long time on this, even after following a lot of people's recommendations.

Brandon

Urticaceous

Answer 8

In windows:

1. Using your windows explorer, Navigate to your vagrant shared folder (I am using scotchbox by the way) e.g C:\scotchbox/public/gulpProject
2. In the address bar of the folder, type cmd and press Enter
3. Do your gulp installation npm install

Iover

Avoid copy-pasting the same answer. You should flag as duplicate instead. Additionally, don't swear in your post. – Radiolocation

Answer 9

npm install --no-bin-link. You will have a entire flattened node_modules

Patty