Mastering Native Node.js Addons with node-addon-api: A Comprehensive Guide

Introduction#

Welcome to the second installment of our deep dive into native Node.js modules! If you’ve been following along, you’ve already learned the basics of what native modules are, how they work, and why they are such a vital part of the Node.js ecosystem. If you missed it, you might want to check out the first post in the series: "Unlocking the Power of Native Modules in JavaScript: An Introductory Guide"

Today, we’re rolling up our sleeves to get into the nitty-gritty: building a real-world native module from scratch. But we’re not just going to build any module — we’re creating a "Fast Image Resizer" using C++ and integrating it into a Node.js application. The primary objective here is to offer a hands-on demonstration of the performance benefits that native modules can bring to computationally intensive tasks.

Now, you might be asking, "Why image resizing?" The answer lies in its practical applicability and the intense computation it demands, making it a perfect case study for native modules. JavaScript alone can be painfully slow for such operations, giving us an excellent backdrop against which to showcase the raw speed of a native module. By the end of this post, you’ll not only have a fully functional image resizer but also a robust understanding of node-gyp, the build tool that turns our C++ code into a native Node.js module.

Here’s what we’ll cover:

Setting up your development environment to use node-gyp.
Writing the C++ code that will form the core of our Fast Image Resizer.
Crafting the essential binding.gyp file, the blueprint for our native module.
Building the module and integrating it into a Node.js application.
Troubleshooting and overcoming common challenges.

☕

So grab a cup of coffee, fire up your code editor, and let’s get building!

Setting the Stage: Tools and Environment Setup 🛠️#

Before diving into the code, it’s essential to ensure that our development environment is well-equipped with the tools we’ll be using throughout this hands-on tutorial. The right tools not only make the development process smoother but also allow us to focus more on coding and less on fixing environment-related issues. Here, we’ll cover how to set up everything you need, based on the operating system you’re using.

The Tools You’ll Need 🛠️#

Here’s a list of tools that you’ll require for developing native Node.js modules:

Node.js: To run your JavaScript code.
npm: For package management and distribution.
C/C++ Development Tools: For writing native code.
Python: Required by node-gyp.
Node and npm

Ensure you have Node.js and npm installed on your machine. If you haven’t, download the appropriate installer from Node.js website. It’s recommended to use the LTS (Long Term Support) version for stability. The installer will also include npm.

Terminal

# Verify Node and npm installation
node --version
npm --version

🥸

I will actually be using pnpm as a personal preference. You can use npm or yarn as well. (Or even bun 🍞 if you are feeling adventurous!)

C/C++ Development Tools and Python

The requirements for C/C++ development tools and Python differ based on your operating system:

Windows: Use npm to install windows-build-tools which includes the required C/C++ compilers and Python.
Terminal
```
npm install --global --production windows-build-tools
```
Mac: Use xcode-select to install the necessary C/C++ tools.
Terminal
```
xcode-select --install
```
Python usually comes pre-installed on macOS.
Linux: The required C/C++ and Python tools are generally pre-installed. If not, consult your distribution’s package manager for installation.

Terminal

# Verify Python installation
python --version
 
# Verify C/C++ compiler
cc --version
 
# Verify make utility
make --version

Other Tools

You’ll also need a shell program and a code editor of your choice. macOS and Linux generally come with a shell pre-installed. For Windows, you might consider using PowerShell.

Initializing Your Project 🚀#

Create a new directory for your project and initialize it with pnpm:

Terminal

mkdir fast-image-resizer
cd fast-image-resizer
pnpm init # npm init -y

This action will generate a package.json file, your project’s manifest.

Setting Up node-gyp 🛠️🔧#

Before we dive into the actual setup, it’s crucial to understand what node-gyp is and why it’s an essential tool for native Node.js module development. Node-gyp is a build automation tool used to compile native addon modules for Node.js. It provides a cross-platform interface for native module compilation and takes care of the heavy lifting around configuring the build environment for various operating systems. Node-gyp uses binding.gyp files written in JSON format to describe the configuration for building your native addons.

Now that you have an idea of what node-gyp is, let’s get it set up so we can proceed with creating a native module.

Installation 💾#

Installing node-gyp is straightforward. You can either install it globally on your machine or as a development dependency in your specific project.

Terminal

pnpm i -D node-gyp # npm install --save-dev node-gyp

Verifying Installation ✅#

To ensure that node-gyp is correctly installed, run the following command:

Terminal

pnpm node-gyp --version # Mine is v9.4.0

If you see the version number displayed, you are good to go.

That’s it for the setup! You’re now well-equipped to tackle native Node.js module development.

Your First Native Module: Hello, World! 🌍#

Before we start with the actual implementation, let’s create a basic "Hello, World!" native module to get a taste of the process. This simple example will demonstrate the basics of setting up, compiling, and importing a native module.

The C++ Code 📝#

Create a file named hello.cc and add the following C++ code:

hello.cc

#include <node.h>
 
void HelloWorld(const v8::FunctionCallbackInfo<v8::Value>& args) {
  v8::Isolate* isolate = args.GetIsolate();
  args.GetReturnValue().Set(v8::String::NewFromUtf8(isolate, "Hello, world from C++!").ToLocalChecked());
}
 
void Initialize(v8::Local<v8::Object> exports) {
  NODE_SET_METHOD(exports, "hello", HelloWorld);
}
 
NODE_MODULE(NODE_GYP_MODULE_NAME, Initialize)

The Binding Configuration 📐#

Create a binding.gyp file with the following content:

binding.gyp

{
  "targets": [
    {
      "target_name": "hello",
      "sources": ["hello.cc"]
    }
  ]
}

Compilation 🔨#

Navigate to your project folder and run:

package.json

{
  "scripts": {
    "build": "node-gyp configure build"
  }
}

If everything goes as planned, you should see a build/ directory containing your compiled native module.

Using the Module in Node.js 📦#

Finally, let’s use this native module in a Node.js script. Create a test.js file with the following content:

test.js

const addon = require("./build/Release/hello.node");
console.log(addon.hello()); // Outputs "Hello, world from C++!"

Run the script:

package.json

{
  "scripts": {
    "build": "node-gyp configure build",
    "dev": "node test.js"
  }
}

You should see "Hello, world from C++!" printed to the console. Congratulations, you’ve just created and used your first native Node.js module!

Setting Up Language Server Protocol (LSP) 🧝#

When developing native modules, it’s extremely beneficial to have an IDE or text editor that understands C/C++ for features like IntelliSense, refactoring, and error checking. LSP can help with this, and luckily, we can generate a compile_commands.json file to make this happen. Below is a helper script that can automate this process for you. This script is adapted from a GitHub issue discussion.

First, create a script named lsp.sh and paste the following code into it:

lsp.sh

#!/bin/bash
 
# Configure node-gyp with pnpm
if ! pnpm node-gyp configure --release -- -f gyp.generator.compile_commands_json.py >/dev/null 2>&1; then
 echo "Error configuring node-gyp."
 exit 1
fi
 
# Move compile_commands.json and remove 'Release' directory
if ! mv Release/compile_commands.json ./ >/dev/null 2>&1 || ! rm -r Release >/dev/null 2>&1; then
 echo "Error moving compile_commands.json or removing 'Release' directory."
 exit 1
fi
 
# Remove extra compile_commands.json and their directories
EXTRA=$(find . -mindepth 2 -name 'compile_commands.json')
if ! echo "$EXTRA" | xargs rm -f >/dev/null 2>&1; then
 echo "Error removing extra compile_commands.json files."
 exit 1
fi
 
# Remove empty directories
echo "$EXTRA" | xargs -n1 dirname | while read -r dir; do
 if [ -z "$(ls -A "$dir")" ]; then
  if ! rm -r "$dir" >/dev/null 2>&1; then
   echo "Error removing empty directory: $dir"
   exit 1
  fi
 fi
done
 
# Retrieve and escape compiler flags
ESCAPED_FLAGS=$(echo | cc -Wp,-v -x c++ - -fsyntax-only 2>&1 |
 grep -E '^ /' | sed 's/^ /-isystem /' | sed 's/\//\\\//g' | tr '\n' ' ')
 
if [ -z "$ESCAPED_FLAGS" ]; then
 echo "Error retrieving compiler flags."
 exit 1
fi
 
# Update compile_commands.json
if ! awk -i inplace -v flags="$ESCAPED_FLAGS" '{gsub(/ -c /, " " flags " -c "); print}' compile_commands.json >/dev/null 2>&1; then
 echo "Error updating compile_commands.json."
 exit 1
fi

Make sure to give execute permission to the script:

Terminal

chmod +x lsp.sh

Now, add the following line to your package.json under the “scripts” section:

package.json

{
  "scripts": {
    "build": "node-gyp configure build && ./lsp.sh",
    "dev": "node test.js"
  }
}

This way, the lsp.sh script will run automatically when you execute npm run build or pnpm run build.

With this setup, you can enjoy a more interactive and responsive development environment while working on your native Node.js modules. Feel free to integrate this with your favorite text editor or IDE that supports LSP for C/C++.

This can be a boon for your productivity, especially if you’re already accustomed to such features while working in TypeScript or Rust.

This LSP setup will provide you with rich coding features such as autocompletion, go to definition, and real-time error checking, making your development process much smoother. 🌟

The Complexity of Direct Node API 😵‍💫#

The Direct Node API, often referred to as Node-API or N-API, is robust and extremely flexible but can be verbose and intricate. It exposes a set of primitive APIs that deal directly with JavaScript objects and values. Though it allows for high customization and deep integration, it requires you to handle several underlying complexities like object lifetimes, scopes, and type conversions manually. This can be cumbersome for those accustomed to more abstracted or user-friendly interfaces, especially when you just want to get a project off the ground.

For example, the previous allegedly simple "Hello, World!" example using the Direct Node API would look like this:

hello.cc

#include <node.h>
 
void HelloWorld(const v8::FunctionCallbackInfo<v8::Value>& args) {
  v8::Isolate* isolate = args.GetIsolate();
  args.GetReturnValue().Set(v8::String::NewFromUtf8(isolate, "Hello, World!"));
}
 
void Initialize(v8::Local<v8::Object> exports) {
  NODE_SET_METHOD(exports, "hello", HelloWorld);
}
 
NODE_MODULE(NODE_GYP_MODULE_NAME, Initialize)

As you can see, even for a simple "Hello, World!", the code has a fair bit of boilerplate.

Why I Prefer node-addon-api 🌟#

For better usability, we will use node-addon-api, an abstraction layer on top of the Direct Node API. It provides an object-oriented C++ API, which considerably simplifies the complexity and makes it easier to manage. It’s essentially the C++ sugar on top of the Direct Node API’s C-based interface.

To give you a contrast, let’s rewrite our "Hello, World!" example using node-addon-api:

#include <napi.h>
 
Napi::String HelloWorld(const Napi::CallbackInfo& info) {
  Napi::Env env = info.Env();
  return Napi::String::New(env, "Hello, World from C++ [with NAPI]!");
}
 
Napi::Object Init(Napi::Env env, Napi::Object exports) {
  exports.Set("hello", Napi::Function::New(env, HelloWorld));
  return exports;
}
 
NODE_API_MODULE(NODE_GYP_MODULE_NAME, Init)

As you can see, this version of "Hello, World!" is cleaner and more straightforward. This is why we’ll proceed with node-addon-api for our deeper exploration and for building our example application. With its simplified API and user-friendly features, node-addon-api is an excellent choice for developers looking to create native Node.js addons without diving deep into the intricacies of the Direct Node API.

Required Packages and Adapted `binding.gyp` 📦#

Before diving into the code, let’s first install the necessary package for node-addon-api. Run the following command in your project directory:

pnpm add node-addon-api

Adapted `binding.gyp` 🛠️#

You’ll also need to modify your binding.gyp file to include node-addon-api’s header files. Here’s how your binding.gyp should look:

binding.gyp

{
  "targets": [
    {
      "target_name": "hello",
      "sources": ["hello.cc"],
      "include_dirs": ["<!@(node -p \"require('node-addon-api').include\")"],
      "dependencies": ["<!(node -p \"require('node-addon-api').gyp\")"],
      "cflags!": ["-fno-exceptions"],
      "cflags_cc!": ["-fno-exceptions"],
      "defines": ["NAPI_DISABLE_CPP_EXCEPTIONS"]
    }
  ]
}

There is always cases of platform specific configurations like the following, but I wont be covering them here:

binding.gyp

{
  "targets": [
    {
      "target_name": "hello",
      "sources": ["hello.cc"],
      "include_dirs": ["<!@(node -p \"require('node-addon-api').include\")"],
      "dependencies": ["<!(node -p \"require('node-addon-api').gyp\")"],
      "cflags!": ["-fno-exceptions"],
      "cflags_cc!": ["-fno-exceptions"],
      "defines": ["NAPI_DISABLE_CPP_EXCEPTIONS"],
      "xcode_settings": {
        "GCC_ENABLE_CPP_EXCEPTIONS": "YES",
        "CLANG_CXX_LIBRARY": "libc++",
        "MACOSX_DEPLOYMENT_TARGET": "10.7"
      },
      "msvs_settings": {
        "VCCLCompilerTool": {
          "ExceptionHandling": 1
        }
      }
    }
  ]
}

Understanding the `<!@()` and `<!()` Syntax in `binding.gyp` 🔍#

In your binding.gyp file, you might have noticed some unfamiliar syntax: <!@() and <!(). These are special directives that help with dynamic evaluation within the .gyp files. Let’s break down what each does:

<!(): This directive allows you to execute a shell command. The output of this command is then captured and used in the .gyp file.
For instance, <!(node -p "require('node-addon-api').gyp") executes the node command that fetches the path to the .gyp file of the node-addon-api package.
<!@(): Similar to the previous one, this directive also executes a shell command. However, the output is tokenized by whitespace, allowing you to capture multiple output entries.
In <!@(node -p "require('node-addon-api').include"), the command fetches the include directories for node-addon-api, and because we use <!@(), multiple directories would be handled correctly if present.

Understanding these directives gives you the flexibility to perform more complex operations dynamically, right within your binding.gyp configuration.

Re-Structuring the File Structure for Better Organization 🗂️#

Now that we’ve got a basic "Hello World" example up and running, let’s go a step further and reorganize our project structure for better maintainability and scalability.

Target Structure#

./addon/src: This is where all your C++ .cc files will reside.
./addon/include: This folder will contain all your header .hpp files.
./lib/binding.js: A JavaScript wrapper around the native node bindings.
./tests: This will contain all your test files.

addon/include/fast-image-resizer/hello.hpp

#pragma once
 
#include <napi.h>
 
namespace fast_image_resizer {
  Napi::String hello_world(const Napi::CallbackInfo &info);
}

addon/include/fast-image-resizer/exports.hpp

#pragma once
 
#include <napi.h>
 
namespace fast_image_resizer {
  void exports(Napi::Env env, Napi::Object exports);
}

addon/src/fast-image-resizer/hello.cc

#include <fast-image-resizer/hello.hpp>
 
namespace fast_image_resizer {
  Napi::String hello_world(const Napi::CallbackInfo &info) {
    Napi::Env env = info.Env();
    return Napi::String::New(env, "Hello, World from C++ [with NAPI]!");
  }
}  // namespace fast_image_resizer

addon/src/fast-image-resizer/exports.cc

#include <fast-image-resizer/exports.hpp>
#include <fast-image-resizer/hello.hpp>
 
namespace fast_image_resizer {
  void exports(Napi::Env env, Napi::Object exports) {
    exports.Set("helloWorld", Napi::Function::New(env, fast_image_resizer::hello_world));
  }
}  // namespace fast_image_resizer

addon/src/entrypoint.cc

#include <napi.h>
 
#include <fast-image-resizer/exports.hpp>
 
Napi::Object Init(Napi::Env env, Napi::Object exports) {
  fast_image_resizer::exports(env, exports);
 
  return exports;
}
 
NODE_API_MODULE(NODE_GYP_MODULE_NAME, Init)

Resulting in the following tree:

Terminal

tree addon
addon
├── include
│   └── fast-image-resizer
│       ├── exports.hpp
│       └── hello.hpp
└── src
    ├── entrypoint.cc
    └── fast-image-resizer
        ├── exports.cc
        └── hello.cc
 
5 directories, 5 files

Updated `binding.gyp`#

To facilitate this restructured file hierarchy, we’ll need to modify our binding.gyp file. We’ll configure it to automatically detect all .cc and .hpp files in their respective directories. Here is how to do it:

binding.gyp

{
  "targets": [
    {
      "target_name": "fast-image-resizer",
      "sources": ["<!@(find addon/src -name *.cc)"],
      "include_dirs": [
        "<!(node -p \"require('node-addon-api').include_dir\")",
        "/usr/local/include/",
        "<(module_root_dir)/addon/include"
      ],
      "dependencies": ["<!(node -p \"require('node-addon-api').gyp\")"],
      "cflags!": ["-fno-exceptions"],
      "cflags_cc!": ["-fno-exceptions"],
      "defines": ["NAPI_DISABLE_CPP_EXCEPTIONS"]
    }
  ]
}

Here, the find shell command recursively looks for .cc and .hpp files within ./addon/src and ./addon/include, respectively. This way, as you add more source and header files to these folders, you won’t need to manually update binding.gyp.

JavaScript Wrapper (`./lib/binding.ts`)#

Having a JavaScript wrapper around your native module can provide a cleaner and more JavaScript-friendly API. You can simply require your native addon and wrap its functionalities in this file.

lib/binding.ts

import { helloWorld } from "../build/Release/fast-image-resizer.node";
 
export { helloWorld };

Test Directory (`./tests`) 🧪#

To ensure the reliability of our native module, it’s essential to include automated tests. We’ll be using the Node.js Test API for this purpose.

First, add the tsx library as a dev dependency, as we’ll use it to transpile and run our TypeScript test files.

Terminal

pnpm i -D tsx

Next, update your package.json file to include the following test command:

package.json

{
  "scripts": {
    "build": "node-gyp configure build && ./scripts/lsp.sh",
    "test": "tsx --test tests/*"
  }
}

Btw, I just moved lsp.sh to a scripts directory.

This command will transpile and execute all test files located in the tests/ directory.

Now, let’s write our test using the Node.js Test API. Create a TypeScript file in your ./tests directory with the following code:

tests/dummy.test.ts

import { describe, it } from "node:test";
import assert from "node:assert";
 
import { helloWorld } from "../lib/binding";
 
describe("helloWorld", () => {
  it("should return 'Hello, World from C++ [with NAPI]!'", () => {
    assert.strictEqual(helloWorld(), "Hello, World from C++ [with NAPI]!");
  });
});

This test uses the describe and it functions from the Node.js Test API along with the assert module to ensure that the helloWorld function behaves as expected.

To run the tests, execute the following command from the root of your project:

Terminal

npm test

> tsx --test tests/*
 
▶ helloWorld
  should return 'Hello, World from C++ [with NAPI]!' (0.295912ms)
  helloWorld (1.314167ms)
 
ℹ duration_ms 92.176873

The tsx --test tests/* command will take care of transpiling the TypeScript files and running the tests. Ensure that all tests pass to verify the functionality of your native module.

Profit 📈.

Finalizing the Implementation: Image Resizing with OpenCV 🖼️🛠️#

Having set up the fundamental aspects of our environment, we’re ready to tackle a more intricate task — building a native module to resize images. In this endeavor, we will employ the powerful OpenCV C++ library.

Installing OpenCV Dependency#

The steps to install OpenCV may vary depending on your operating system:

Ubuntu/Linux
Terminal
```
sudo apt-get install libopencv-dev
```

Terminal (I use arch, btw)

pacman -S opencv

Note: For Arch users, installing hdf5 and vtk may also be necessary.

macOS
```
brew install opencv
```

Windows Download and install from OpenCV’s website.

Update binding.gyp#

Next, tweak your binding.gyp file to accommodate OpenCV headers and libraries. Take note that if you encounter issues with pkg-config --cflags, specifying the full path to the headers might resolve the issue.

binding.gyp

{
  "targets": [
    {
      "target_name": "addon",
      "sources": [
        "./addon/src/*.cc"
      ],
      "include_dirs": [
        "<!(node -p \"require('node-addon-api').include_dir\")",
        "<(module_root_dir)/addon/include",
        "/usr/local/include/",
        "/usr/include/opencv4" # In my case it was located here
        # you can also try `<!@(pkg-config --cflags opencv4)` but
        # for me it was not working properly
      ],
      "libraries": [
        "<!(pkg-config --libs opencv4 2>/dev/null || echo '')" # In case of errors
      ],
      "dependencies": [
        "<!(node -p \"require('node-addon-api').gyp\")",
      ],
      "cflags!": [ "-fno-exceptions", "-std=c++20" ],
      "cflags_cc!": [ "-fno-exceptions", "-std=c++20" ],
      'defines': [ 'NAPI_DISABLE_CPP_EXCEPTIONS' ]
    }
  ]
}

C++ Code#

Start by declaring new functions in addon/include/fast-image-resizer/resize.hpp. The Napi::Value get_file_dimensions(const Napi::CallbackInfo &info) function will serve as a helper for testing, negating the need to worry about its JavaScript implementation.

addon/include/fast-image-resizer/resize.hpp

#pragma once
 
#include <napi.h>
 
namespace fast_image_resizer {
  Napi::Value resize(const Napi::CallbackInfo &info);
  Napi::Value get_file_dimensions(const Napi::CallbackInfo &info);
}  // namespace fast_image_resizer

Then, proceed to flesh out the logic in addon/src/fast-image-resizer/resize.cc. First, we import the required headers, followed by parameter validation based on the API we’re targeting.