Node.js recursively list full path of files [duplicate]

Question

Good night everyone. I'm having trouble with probably some simple recursive function. The problem is to recursively list all files in a given folder and its subfolders.

For the moment, I've managed to list files in a directory using a simple function:

fs.readdirSync(copyFrom).forEach((file) => {
  let fullPath = path.join(copyFrom, file);

  if (fs.lstatSync(fullPath).isDirectory()) {
    console.log(fullPath);
  } else {
    console.log(fullPath);
  }
});

I've tried various methods like do{} ... while() but I can't get it right. As I'm a beginner in javascript, I finally decided to ask for help from you guys.

Bordelaise

You can use recursive-readdir package for this. npmjs.com/package/recursive-readdir – Fredette

I would say Good day to you from another timezone and that you should maybe also read this: meta.stackexchange.com/questions/2950/… – Christoforo

Thanks @Kirill, I'll give it a try but as recursive functions seemed to be quite common I would have liked to understand how to do it :) – Bordelaise

Answer 1

Just add a recursive call and you are done:

 function traverseDir(dir) {
   fs.readdirSync(dir).forEach(file => {
     let fullPath = path.join(dir, file);
     if (fs.lstatSync(fullPath).isDirectory()) {
        traverseDir(fullPath);
     }  
     console.log(fullPath);
   });
 }

Necolenecro

Thanks, it seems to do the trick. However, I'm not having any console messages when executing the function... I must have missed something. – Bordelaise

@YouDeserveThat, maybe a dumb question, but are you actually calling the function and not just defining it? – Battlement

Hm. Dumb questions often help beginners :D. Thanks @FissureKing ! – Bordelaise

@YouDeserveThat Just call the function like: traverseDir(copyFrom);. Also, accept the answer if it solves your problem :D – Gussiegussman

Thanks so much all of you, you saved my night ;) – Bordelaise

@JonasW. a perfect use case for unfold <3 – Deathlike

Don't forget to import path var path = require('path'); – Cannon

Answer 2

Using console.log in this way displays the path and that's great, but what if you want to do something more meaningful with the paths? For example, maybe you want to collect all of them in an array and pass them off for processing elsewhere...

This process of beginning with a seed state and expanding a sequence of values while the state changes is called an unfold.

const { join } =
  require ('path')

const { readdirSync, statSync } =
  require ('fs')

const unfold = (f, initState) =>
  f ( (value, nextState) => [ value, ...unfold (f, nextState) ]
    , () => []
    , initState
    )

const None =
  Symbol ()

const relativePaths = (path = '.') =>
  readdirSync (path) .map (p => join (path, p))

const traverseDir = (dir) =>
  unfold
    ( (next, done, [ path = None, ...rest ]) =>
        path === None
          ? done ()
          : next ( path
                 , statSync (path) .isDirectory ()
                     ? relativePaths (path) .concat (rest)
                     : rest
                 )
    , relativePaths (dir)
    )

console.log (traverseDir ('.'))
// [ a, a/1, a/1/1, a/2, a/2/1, a/2/2, b, b/1, ... ]

If this is your first time seeing a program like this, unfold will feel very overwhelming. Below, is a simplified example of unfold used to generate an array of the lowercase alphabet

const unfold = (f, init) =>
  f ( (x, next) => [ x, ...unfold (f, next) ]
    , () => []
    , init
    )

const nextLetter = c =>
  String.fromCharCode (c.charCodeAt (0) + 1)

const alphabet =
  unfold
    ( (next, done, c) =>
        c > 'z'
          ? done ()
          : next ( c              // value to add to output
                 , nextLetter (c) // next state
                 )
    , 'a' // initial state
    )

console.log (alphabet)
// [ a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z ]

If you're still stuck, the techniques I've demonstrated here are explained in greater detail in answers to similar questions

• Functional way to create an array of numbers
• Loop until... with Ramda

Generally, it's preferred to use the asynchronous functions in the fs module as this prevents the program from hanging in the event of long disk read times or network delay. Unfolding plays nicely with asynchrony as demonstrated in these other Q&A's

• Recursive Promises Not Returning
• Recursion call async func with promises gets Possible Unhandled Promise Rejection
• Recursive JS function with setTimeout
• Promise with recursion

Deathlike

Thanks for the really detailed anwser, I'll give it a try tomorrow :) – Bordelaise

@YouDeserveThat my pleasure. I've recently learned about anamorphisms and I see use cases for them everywhere now! Let me know if you have any questions, I'm happy to help :D – Deathlike

Answer 3

I am using following getFilesTree function. This function recursively list all files in a directory and its subdirectories except hidden folders and files (start's with .).

import {readdir} from 'node:fs/promises';
import {join, resolve} from 'node:path';
import {parse} from 'node:path';

export async function getFilesTree(dir) {
    return await Promise.all(
        (await readdir(dir, {withFileTypes: true}))
            .filter(child => !child.name.startsWith('.')) // skip hidden
            .map(async (child) => {
                const base = parse(child.name).base;
                const path = resolve(dir, child.name);
                return child.isDirectory() ?
                    {base, path, children: await getFilesTree(join(dir, child.name))} :
                    {base, path};
            }),
    );
}

Function itself is very similar to the recursive-readdir library. The results then look something like this:

[
    {
        "base": "file.js",
        "path": "/Volumes/Work/file.js"
    },
    {
        "base": "css",
        "path": "/Volumes/Work/css",
        "children": [
            {
                "base": "index.css",
                "path": "/Volumes/Work/css/index.css"
            },
            {
                "base": "code.css",
                "path": "/Volumes/Work/css/code.css"
            }
        ]
    }
]

Sometimes there is no need to have the structured data, then you can use generator instead:

import {readdir} from 'node:fs/promises';
import {resolve} from 'node:path';

async function * getFiles(dir) {
    for (const dirent of await readdir(dir, {withFileTypes: true})) {
        const res = resolve(dir, dirent.name);
        if (dirent.isDirectory()) {
            yield * getFiles(res);
        } else {
            yield res;
        }
    }
}

for await (const file of getFiles('content')) {
    console.log(file);
}

Yokoyama

Answer 4

Piggybacking off Jonas Wilms answer - that turned out to be really slow for a large number of files. This is faster (though it is async).

const getFilesnamesRecursive = (dir: string, foundFiles: string[]) => {
const files = fs.readdirSync(dir, {withFileTypes: true});
files.forEach((entry: any) =>{        
    if (entry.isDirectory()){                    
        getFilesnamesRecursive(path.join(dir, entry.name)), foundFiles, logger);
    }
    else{            
        foundFiles.push(path.join(dir, entry.name);
    }
});

}

The other is slower because it was using readdir to get each filename then checking if it was a directory. This way is faster because it users the option withFileTypes: true to return each entry as a Dirent object, instead of just the filename.

Endoscope

Answer 5

Update v20.1.0, v18.17.0 (May 2023+)

This is now natively supported like this:

const fs = require('fs').promises;

const files = await fs.readdir(".", { recursive: true })

Futher Reading:

• Docs on fs readdir options
• PR #41439 - fs: add recursive opendir/readdir
• Node.js fs.readdir recursive directory search
• Get all files recursively in directories NodeJS

Leniency