Node modules and NPM in JS

One of the most debatable topics in the JS ecosystem. Node modules have been created by Node to keep track of dependencies and leverage the ability to share code between JS developers. But with the complexity of modern applications, it's not simple, as we usually have real dependencies, development dependencies, peer dependencies, and so on. Sure, Node makes it easy to develop software, but are you curious about how NodeJS works behind the scenes? How do we simply run node app.js while the program is running? The answer is that it's not always simple, so with that in mind,let's get started. And also, how does NPM actually download packages and link packages together? What is the best way to dedupe dependencies and so on? In this blog you and I will find the mystery answers to all the problems above.

Node module resolution algorithm

We will talk about all the ways Node.js checks to find the module that you imported.

• core modules

• relative files/folders

• imports and exports alias

• node_modules

Core modules

First, Node.js checks if your import is a core Node.js module, anything like, os, node:asserts, fs, etc.

import {join} from 'node:path';
import {writeFile} from 'fs';

Location

Using our code, the import location starts with "/"

Handle filename

• If button.js exists, execute as a javascript module

• If button.json exists, parse json and return js object

• if button.node exists, runs as a binary add-on

If we walk into a folder, we need the folder to tell us where to look (main field)

• If button/package.json exits, Node.js will search for the main field. If present, Nodejs will run steps 1-4 and 6-8 for the given path in main

Load button as a folder

• if button/index.js exists, execute as a javascript module

• if button/index.json exists, parse json and return js object

• if button/index.node exists, runs as a binary add-on

Filename auto-translate
button.js
button.json
button.node

button/package.json

// Load as a folder and prefix with index as a name
button/index.js
button/index.json
button/index.node

Using someone's else code (third-party)

import Button from "button";

With this import, nodeJS will try to resolve it as node_modules (folder)

Assuming we are on this location
project
    packages
         app
            components
                button

Node will check the following path 
/project/packages/app/components/node_modules
/project/packages/app/node_modules
/project/packages/node_modules
/project/node_modules
/node_modules

This technique will help to deduplicate dependencies

But why do we deduplicate dependencies ?

Image this scenerio

Project
node_modules
folder A
    node_modules
    package.json
    folder B
    // have some code here it can use node_modules of folder A
    // instead of having another node_modules
package.json

In general, Node.js will check every parent folder in the tree until they get to the root, meaning every package installed on the roots node_moduleswill be available to all the Node.js programs in your machine

import {Input} from 'antd';

Nodejs will look at package.json to get instructions on where it should look next.In our JS ecosystem, we have different module mechanisms to load the file. Usually for web browser, it'll look at the module key and for NodeJS, it will look at the main key

There you go. We'll have the list of exported components from ant

Type lookup for VSCode(not for bundle)

When you import a JavaScript file from a TypeScript file, TypeScript follows an algo‐ rithm that looks like this to look up type declarations for your JavaScript code

Look for a sibling .d.ts file with the same name as your .js file. If it exists, use it as the type declaration for the .js file.

my-app
    src
       index.ts
    legacy
        old-file.js
        old-file.d.ts

//index.ts
import "./legacy/old-file"

TS will use src/legacy/old-file.d.ts as the source of type-declaration and VS code will help us visualize the data type

Import an NPM packages (most commonly used)

my-app
    node_modules
        foo
    src
        index.ts

import { StyleSheet, Text, View } from "react-native";
// It will look at the package.json
// look for the files cakked tyoes ir typing

node_modules/react-native/types

If the package.json doesn't specify the types field, it will traverse out a directory at a time and look for node_modules/@types directory at the last hope

We can tell TS in our project to look at the other places

{
"compilerOptions": {
    "typeRoots" : ["./typings", "./node modules/@types"] }
} // tell TS to look for type declarations at ./typings as well as node

Override node resolution with webpack

In modern app today, we have more flexibility than we did in the past

We can support different extension like.js,.jsx,.ts, and.tsx as well

resolve: {
      extensions: ['.js', '.jsx', '.ts', '.tsx'],
      alias: {
        'is-plain-object': path.resolve(
          __dirname,
          'src/app/utils/is-plain-object.js'
        )
},

Duplicated deps with node_modules

Image that we have a modal dialog version 3.0.0 as our dependencies and this modia log has another deps

// Our code
import { ModalDialog } from 'Modal';
import {Editor} from 'Editor'; 

// ModalDialog deps
import {Button} from 'Button' version 1.0.0

// Editor deps
import {Button} from 'Button' version 1.3.0

//Now we have 2 duplicated depedencies

Solving bundle duplicates with webpack and yarn

https://www.atlassian.com/engineering/performance-in-jira-front-end-solving-bundle-duplicates-with-webpack-and-yarn

Flat dependency

If our application depends on A , and A depends on B

Can we say that our app depends on both A and B?

Yes we can, but things gets bit more complicated with this

And another strange situation

npm dedupe

Yarn lock

Because of the power of flat-dependency, we can dedupe

Mono repo with node_modules

There are two major advantages to using a monorepo for a large-scale codebase. first of all, shared packages can be used in multiple applications on a local machine without an online registration (npm)

A shared package can be used across multiple applications on a local machine

Improve collaboration across codebases

Reunited ESLINT,TSCONFIG,BABEL

We can have a single configuration at root of our files and share that configuration with other repos

Lerna

Lerna is a mono-repo tool for TS and JS. It has been around for many years and is used by tens of thousands of projects, including React, Jest and Babel

• Lerna links different projects within the repo, so they can import each other without having to publish to npm (using symlink under the hook)

• I have an example in my project using Lerna

• 

  Lerna runs a command against any number of projects; this is great because we can run download dependencies of every repository and link them together

•           "update-deps":"yarn install && lerna bootstrap",
            // Running lerna bootstrap will invoke npm install 
            // in each of the packages, and will link local packages together
  

  Lerna manages your publishing process, from version management to publishing to NPM, and it provides a variety of options to make sure any workflow can be accommodated.

• 

NPM

Npm command is inside /Users/{userName}/.nvm/versions/node/v16.19/bin

Workflow from NPM

Semantic versioning

Sematic of npm

Package-lock.json

Problem with ^

Package-lock.json

To actually lock our version despite the way npm automatically updates our minor-version

Reduce dependencies

Peer dependencies for React library

NPM run scripts from bin

When you install a package locally, npm creates a .bin directory inside the node_modules folder. This directory contains binary executables for the installed packages.

In this caseeslint, the binary executable is placed innode_modules/.bin/eslint.

{
    "script": {
        "lint":"eslint ."
    }
}

npm run lint
/** Eslint syntax */ 
/* we use . to pass ourfile as argument to eslint script */
./node_modules/.bin/eslint yourfile.js

In the end, those packages won't be imported into our project. It's there, sitting in node_modules/bin and whenever we need it to do testing or formatting in our code- base, we simply call it from our project with npm script

If we have a different script name in NPM, we'd better give an instruction for NPM to run our code

//Different arguments to script
{
  "scripts" : {
    "lint:check":"eslint .",
    "lint:fix":"eslint . --fix"
   } 
}'

Executes arguments to scripts

console.log(process.argv)

// node node.js
[
  '/Users/vincenguyen/.nvm/versions/node/v16.19.0/bin/node',
  '/Users/vincenguyen/Desktop/node.js'
];'

• process.argv[0]: contains the path to Node.js executable

• process.argv[1]: contains the path to your script

node node.js --myname="vince" --job javascript-developers

[ // root
  '/Users/vincenguyen/.nvm/versions/node/v16.19.0/bin/node', //node 
  '/Users/vincenguyen/Desktop/node.js', //location
  '--myname=vince',
  '--job',
  'javascript-developers'
]

/** As we can see that, we store an arguments in an array **/

Let's try to handle our arguments nicely

const args = process.argv.slice(2);
const keyValuePairs = {};
// [--myname="vince","--job","javascript--developers]

args.forEach((arg)=>{
    const [key,value] = arg.split("=");
    keyValueParis[key.slice(2)] = value || true 
});

// API proposal
{
  myName : "Vince",
  job:true
  javascript-developers:true
}

An example

#!/usr/bin/env node

'use strict';

var util = require('util');
var path = require('path');
var fs = require('fs');

var getStdin = require('get-stdin');

var args = require('minimist')(process.argv.slice(2), {
  boolean: ['help', 'in'],
  string: ['file'],
});

var BASE_PATH = path.resolve(process.env.BASE_PATH || __dirname);

if (args.help) {
  printHelp();
} else if (args.in || args._.includes('-')) {
  getStdin()
    .then(processFile)
    .catch(error);
} else if (args.file) {
  fs.readFile(path.join(BASE_PATH, args.file), function onContents(
    err,
    contents
  ) {
    if (err) {
      error(err.toString());
    } else {
      processFile(contents.toString());
    }
  });
} else {
  error('Incorrect usage.', true);
}

// **********************

function processFile(contents) {
  contents = contents.toUpperCase();
  process.stdout.write(contents);
  // console.log(contents);
}

function error(msg, includeHelp = false) {
  console.error(msg);
  if (includeHelp) {
    console.log('');
    printHelp();
  }
}

function printHelp() {
  console.log('ex1 usage:');
  console.log(' ex1.js --file={FILENAME}');
  console.log('');
  console.log('--help                 print this help');
  console.log('--file={FILENAME}      process the file');
  console.log('--in, -                process stdin');
  console.log('');
}

NPM audit broken by design

Shout out this blog: https://overreacted.io/npm-audit-broken-by-design/

audit fix --force

{
   "dependencies" : {
     "network-utility": 1.0.0; // strictly depends on 1.0.0  
   }
}
// fix-force can update it to a version 1.x.y but can potentially break
// things

Finally, if there is no way to gracefully upgrade the tree, you could try npm audit fix --force. This is supposed to be used if database-layer doesn’t accept the new version of network-utility and also doesn’t release an update to accept it. So you’re kind of taking matters into your own hands, potentially risking breaking changes. It seems like a reasonable option to have.

Consume NPM package

Remember that we have CJS and ESM module formats for both browser and node application

• CommonJS require useScroller from "vince-scroller"

• ESM import useScroller from "vince-scroller"

const scroller = require("scroller");
// npm will read the package.json instructions
//  and go to point to the main field

import { scroller } from "scroller";
// npm will read the package.json instructions 
// and go to the module field

Deploy to NPM

.npmignore

Yarn uses the command yarn publish to push our package to the npm package registry

Yarn publish will push all file to the registry, except in two cases

• .gitignore

• .npmignore

• Use .gitignore to specify files and directories you want Git to ignore when tracking changes in your repository.

• Use npmignore to specify files and directories you want npm to exclude when publishing your package to the npm registry.