Node modules and NPM in JS

Node modules for JS devs

One of the most debatable topics in the JS ecosystem. Node modules have been created by Node to keep track of dependencies and leverage the ability to share code between JS developers. But with the complexity of modern applications, it's not simple, as we usually have real dependencies, development dependencies, peer dependencies, and so on. Sure, Node makes it easy to develop software, but are you curious about how NodeJS works behind the scenes? How do we simply run node app.js while the program is running? The answer is that it's not always simple, so with that in mind,let's get started. And also, how does NPM actually download packages and link packages together? What is the best way to dedupe dependencies and so on? In this blog you and I will find the mystery answers to all the problems above.

Node module resolution algorithm

We will talk about all the ways Node.js checks to find the module that you imported.

  • core modules

  • relative files/folders

  • imports and exports alias

  • node_modules

Core modules

First, Node.js checks if your import is a core Node.js module, anything like, os, node:asserts, fs, etc.

import {join} from 'node:path';
import {writeFile} from 'fs';

Relative modules

Secondly, Node.js checks if our imports start with

  • / , ./ , ../

  • If it does, Node.js checks this path for files and directories and treats the imports as a path

  • If the path starts, / Node.js will look for the module from the root of the system

  • If the path begins with ./ of ../ Node.js will search for the module relative to the current's file location

💡
Note that if one of these checks passes, the Node will not continue the next step

Handle filename

  • If button.js exists, execute as a javascript module

  • If button.json exists, parse json and return js object

  • if button.node exists, runs as a binary add-on

If we walk into a folder, we need the folder to tell us where to look (main field)

  • If button/package.json exits, Node.js will search for the main field. If present, Nodejs will run steps 1-4 and 6-8 for the given path in main

Load button as a folder

  • if button/index.js exists, execute as a javascript module

  • if button/index.json exists, parse json and return js object

  • if button/index.node exists, runs as a binary add-on

Filename auto-translate
button.js
button.json
button.node

button/package.json

// Load as a folder and prefix with index as a name
button/index.js
button/index.json
button/index.node
💡
Most of the time, we'll want the .js file

Using someone's else code

import Button from "button";

With this import, nodeJS will try to resolve it as node_modules (folder)

Assuming we are on this location
project
    packages
         app
            components
                button

Node will check the following path 
/project/packages/app/components/node_modules
/project/packages/app/node_modules
/project/packages/node_modules
/project/node_modules
/node_modules

This technique will help to deduplicate dependencies

In general, Node.js will check every parent folder in the tree until they get to the root, meaning every package installed on the roots node_moduleswill be available to all the Node.js programs in your machine

import {Input} from 'antd';

Nodejs will look at package.json to get instructions on where it should look next.In our JS ecosystem, we have different module mechanisms to load the file. Usually for web browser, it'll look at the module key and for NodeJS, it will look at the main key

There you go. We'll have the list of exported components from ant

Override node resolution with webpack

In modern app today, we have more flexibility than we did in the past

We can support different extension like.js,.jsx,.ts, and.tsx as well

resolve: {
      extensions: ['.js', '.jsx', '.ts', '.tsx'],
      alias: {
        'is-plain-object': path.resolve(
          __dirname,
          'src/app/utils/is-plain-object.js'
        )
},

Duplicated deps with node_modules

Image that we have a modal dialog version 3.0.0 as our dependencies and this modia log has another deps

import { ModalDialog } from 'Modal';
import {Editor} from 'Editor'; 

// ModalDialog deps
import {Button} from 'Button' version 1.0.0

// Editor deps
import {Button} from 'Button' version 1.3.0

//Now we have 2 duplicated depedencies

Solving bundle duplicates with webpack and yarn

https://www.atlassian.com/engineering/performance-in-jira-front-end-solving-bundle-duplicates-with-webpack-and-yarn

Flat dependency

If our application depends on A , and A depends on B

Can we say that our app depends on both A and B?

Yes we can, but things gets bit more complicated with this

And another strange situation

💡
There's pretty much nothing we can do to deduplicate dependency in this situation. Maybe we can upgrade A v2.0 so it can use B v2.0?

npm dedupe

Yarn lock

Because of the power of flat-dependency, we can dedupe

Mono repo with node_modules

There are two major advantages to using a monorepo for a large-scale codebase. first of all, shared packages can be used in multiple applications on a local machine without an online registration (npm)

A shared package can be used across multiple applications on a local machine

Improve collaboration across codebases

Reunited ESLINT,TSCONFIG,BABEL

We can have a single configuration at root of our files and share that configuration with other repos

💡
More challenging than a single repo

Lerna

Lerna is a mono-repo tool for TS and JS. It has been around for many years and is used by tens of thousands of projects, including React, Jest and Babel

  • Lerna links different projects within the repo, so they can import each other without having to publish to npm (using symlink under the hook)

  • I have an example in my project using Lerna

  • Lerna runs a command against any number of projects; this is great because we can run download dependencies of every repository and link them together

  •         "update-deps":"yarn install && lerna bootstrap",
            // Running lerna bootstrap will invoke npm install 
            // in each of the packages, and will link local packages together
    

    Lerna manages your publishing process, from version management to publishing to NPM, and it provides a variety of options to make sure any workflow can be accommodated.

NPM

Npm command is inside /Users/{userName}/.nvm/versions/node/v16.19/bin

Workflow from NPM

Semantic versioning

Sematic of npm

Package-lock.json

Problem with ^

Package-lock.json

To actually lock our version despite the way npm automatically updates our minor-version

Reduce dependencies

Peer dependencies for React library

💡
But be careful; some libraries would actually request the higher version of, let's say, React to use their library correctly. This is a very traditional way of software world

NPM run scripts from bin

When you install a package locally, npm creates a .bin directory inside the node_modules folder. This directory contains binary executables for the installed packages.

In this caseeslint, the binary executable is placed innode_modules/.bin/eslint.

{
    "script": {
        "lint":"eslint ."
    }
}

npm run lint
/** Eslint syntax */ 
/* we use . to pass ourfile as argument to eslint script */
./node_modules/.bin/eslint yourfile.js

In the end, those packages won't be imported into our project. It's there, sitting in node_modules/bin and whenever we need it to do testing or formatting in our code- base, we simply call it from our project with npm script

If we have a different script name in NPM, we'd better give an instruction for NPM to run our code

//Different arguments to script
{
  "scripts" : {
    "lint:check":"eslint .",
    "lint:fix":"eslint . --fix"
   } 
}'

Executes arguments to scripts

console.log(process.argv)

// node node.js
[
  '/Users/vincenguyen/.nvm/versions/node/v16.19.0/bin/node',
  '/Users/vincenguyen/Desktop/node.js'
];'
  • process.argv[0]: contains the path to Node.js executable

  • process.argv[1]: contains the path to your script

node node.js --myname="vince" --job javascript-developers

[ // root
  '/Users/vincenguyen/.nvm/versions/node/v16.19.0/bin/node', //node 
  '/Users/vincenguyen/Desktop/node.js', //location
  '--myname=vince',
  '--job',
  'javascript-developers'
]

/** As we can see that, we store an arguments in an array **/

Let's try to handle our arguments nicely

const args = process.argv.slice(2);
const keyValuePairs = {};
// [--myname="vince","--job","javascript--developers]

args.forEach((arg)=>{
    const [key,value] = arg.split("=");
    keyValueParis[key.slice(2)] = value || true 
});

// API proposal
{
  myName : "Vince",
  job:true
  javascript-developers:true
}

An example

#!/usr/bin/env node

'use strict';

var util = require('util');
var path = require('path');
var fs = require('fs');

var getStdin = require('get-stdin');

var args = require('minimist')(process.argv.slice(2), {
  boolean: ['help', 'in'],
  string: ['file'],
});

var BASE_PATH = path.resolve(process.env.BASE_PATH || __dirname);

if (args.help) {
  printHelp();
} else if (args.in || args._.includes('-')) {
  getStdin()
    .then(processFile)
    .catch(error);
} else if (args.file) {
  fs.readFile(path.join(BASE_PATH, args.file), function onContents(
    err,
    contents
  ) {
    if (err) {
      error(err.toString());
    } else {
      processFile(contents.toString());
    }
  });
} else {
  error('Incorrect usage.', true);
}

// **********************

function processFile(contents) {
  contents = contents.toUpperCase();
  process.stdout.write(contents);
  // console.log(contents);
}

function error(msg, includeHelp = false) {
  console.error(msg);
  if (includeHelp) {
    console.log('');
    printHelp();
  }
}

function printHelp() {
  console.log('ex1 usage:');
  console.log(' ex1.js --file={FILENAME}');
  console.log('');
  console.log('--help                 print this help');
  console.log('--file={FILENAME}      process the file');
  console.log('--in, -                process stdin');
  console.log('');
}

NPM audit broken by design

Shout out this blog: https://overreacted.io/npm-audit-broken-by-design/

💡
When we run the npm audit, it will check all of our dependencies in package-lock or yarn-lock and scan them through the security database in NPM to see if the packages we're using have vulnerabilities.

audit fix --force

{
   "dependencies" : {
     "network-utility": 1.0.0; // strictly depends on 1.0.0  
   }
}
// fix-force can update it to a version 1.x.y but can potentially break
// things

Finally, if there is no way to gracefully upgrade the tree, you could try npm audit fix --force. This is supposed to be used if database-layer doesn’t accept the new version of network-utility and also doesn’t release an update to accept it. So you’re kind of taking matters into your own hands, potentially risking breaking changes. It seems like a reasonable option to have.

Consume NPM package

Remember that we have CJS and ESM module formats for both browser and node application

  • CommonJS require useScroller from "vince-scroller"

  • ESM import useScroller from "vince-scroller"

const scroller = require("scroller");
// npm will read the package.json instructions
//  and go to point to the main field

import { scroller } from "scroller";
// npm will read the package.json instructions 
// and go to the module field

Deploy to NPM

.npmignore

Yarn uses the command yarn publish to push our package to the npm package registry

💡
A package is a directory containing a package.json file

Yarn publish will push all file to the registry, except in two cases

  • .gitignore

  • .npmignore

  • Use .gitignore to specify files and directories you want Git to ignore when tracking changes in your repository.

  • Use npmignore to specify files and directories you want npm to exclude when publishing your package to the npm registry.