AI Node (Edge AI Nodes)¶

Edge AI Nodes brings NPU-accelerated computer-vision inference (object detection) into a Node-RED flow — "is there a person in this frame," "how many hard hats are missing," "count the trucks" — without Node-RED ever touching a pixel or the NPU directly. Three pieces work together:

Node-RED messages never carry image pixels directly (with two narrow exceptions, see Getting pixels in and out below) — they carry a small frame handle pointing at pixels living in the engine's shared memory. This keeps large images off the Node.js event loop and out of the flow's message history.

Node catalogue¶

Filter the palette for "npu" to find these, under an NPU category (plus the npu-engine config node, filed under config):

Every function node has two outputs — (1) result, (2) error — and shows connection/health status (fps, drops, offline/degraded) on the node itself.

Connecting to the engine¶

Double-click any npu-* node and click the pencil next to Engine to open the shared npu-engine config node:

All nodes sharing one npu-engine config node reuse the same pair of socket connections, reconnecting automatically (with backoff) if the daemon restarts.

Labels¶

Also on the npu-engine config node, since only one model realistically runs at a time on a single-NPU-core board (RK3566 — see Only one model at a time below):

• Use the model's built-in labels.txt (default) — class names come straight from whatever .npumodel bundle is loaded
• Preset list — a built-in list (currently just coco80, the standard 80-class COCO ordering shared by the default model families and COCO-trained YOLOv8/v11)
• Custom list — paste your own names, one per line or comma-separated, in classId order

This relabels every detection from every npu-detect/npu-watch node on that engine by numeric classId — it's a display-layer override only, and doesn't change what the model actually detects.

Choosing a model¶

npu-detect needs to know which model to run. Either point it at a Model config node (built from an npu-model config node — pick an already-installed bundle from a dropdown showing a [loaded] marker for whatever's currently resident, plus a Load now button to warm it up eagerly instead of waiting for the first detection request), or skip the config node and just type a bare model name/version into the or model name field — this is what both of the gateway's own example flows below actually do, e.g. hardhat-detect or yolox-s-coco.

Installing a new model bundle happens the same way, from an npu-model config node's Upload new section: pick a .npumodel bundle zip (the output of npu-forge convert, run separately on an x86+GPU host — the editor does not accept raw .onnx/.rknn files) and click Upload & install. The engine validates the bundle (manifest schema, per-variant checksum) before accepting it.

Only one model at a time on RK3566¶

RK3566 (EPG-002S) has exactly one NPU core, so loading a different model automatically evicts whatever was loaded before — there's no way around this on that hardware. RK3588-based gateways (EPG-004S/EPG-AI) have multiple NPU cores and can keep more than one model resident. Plan flows accordingly: two demo flows on the same RK3566 gateway that each want a different model loaded will fight over which one is actually resident, each detection call implicitly reloading the other's model out.

npu-detect: running inference¶

Feed it a frame handle and (optionally) a model reference:

msg.payload comes back as:

{
  model: "[email protected]",
  inferMs: 191.9, preMs: 3.1, postMs: 1.8,
  detections: [
    { label: "Hardhat", classId: 0, score: 0.91,
      box: { x: 412, y: 130, w: 220, h: 480 },
      boxNorm: { x: 0.2146, y: 0.1204, w: 0.1146, h: 0.4444 } }
  ]
}

Per-message overrides are available on msg.npu.model/threshold/ iou/classes/roi. Output 2 fires on a missing model, an expired frame handle, or a model that isn't installed — with a specific status ("model not found", "frame expired") rather than a generic error.

npu-watch: "is X present, how many, show me a picture"¶

Sits downstream of npu-detect — it doesn't call the model itself, it just filters an existing detections list for one label:

Output: { label, present, count, detections: [...matching only], snapshot: "data:image/jpeg;base64,..." } — snapshot drops straight into a dashboard image widget's src, and msg.frame carries a new handle with just the matching boxes drawn (same convention as npu-draw).

A single npu-detect can feed several npu-watch nodes in parallel, each watching a different label — see Tutorial: watching for several classes at once below.

Getting pixels in and out¶

Two nodes are the only place pixel bytes actually cross the Node-RED/ engine boundary:

• npu-buffer-to-frame — in: takes a msg.payload Buffer (JPEG/PNG, auto-detected by magic bytes, or raw rgb24 with a configured width/height) and registers it with the engine, emitting a frame handle
• npu-frame-to-jpeg — out: takes a frame handle and materializes it as JPEG/PNG bytes in msg.payload, either a raw Buffer or (if output a data: URL string is checked) a ready-to-use data:image/jpeg;base64,... string for a dashboard image widget

Cropping and drawing¶

• npu-roi crops a frame either per-detection (msg.payload.detections[].box — one crop per box) or to one fixed rectangle you configure, returning new derived-frame handle(s). Emit as one message with an array of handles, or one message per crop. Padding and square-crop options are available. Polygon ROI isn't supported — rectangles only.
• npu-draw takes a frame plus a detections list and burns boxes/labels/scores onto a new frame handle — the original is left untouched. Feed the result to npu-frame-to-jpeg to actually see it.
• npu-filter is the "no-code if" for a detections list: label allow-list, min/max score, minimum box width/height, min/max aspect ratio. Pure JavaScript, no engine call — it keeps working even if the engine itself is offline.

npu-engine-status¶

On each input message (or on a repeat interval), calls the engine's health/capability check and emits it as msg.payload: { soc, npuCores, driver, runtime, apiVersion, backend, uptime, models, sources, health, error }. Feed it from an inject node for a dashboard tile or an MQTT health-check topic.

Tutorial: hard-hat safety watch¶

The gateway's own PPE-compliance example flow — npu-camera → npu-detect (a hard-hat/no-hard-hat model) → npu-watch (watching "Hardhat") → a debug-sidebar snapshot preview:

The npu-camera node here is a folder source (a directory of walkthrough images/video frames the engine cycles through) — see npu-camera above for the other source kinds. The node statuses shown are real, live numbers from a running gateway: frames streaming, detections per cycle, inference time, and how many times the watched label showed up.

Tutorial: watching for several classes at once¶

One npu-detect can fan out to several npu-watch nodes, each watching a different label, without running the model more than once per frame — the gateway's second example flow reuses a general COCO detection model (yolox-s-coco — no new model needed, since COCO's 80 classes already include vehicle types) and fans out to four parallel watches:

npu-draw (drawing every vehicle box for an overview snapshot) and each npu-watch node all read the same upstream detections list — the model only runs once per frame regardless of how many labels you're watching for.

Known limitations¶

Carried over honestly from the project's own engineering notes — don't assume any of the following works just because a field exists for it:

• RTSP and USB/V4L2 live cameras are not supported. npu-camera's Kind is limited to folder (image sequence), file (a video file, software-decoded), and push (fed by npu-buffer-to-frame or an external caller). Live network/USB cameras are a planned future milestone, not available today.
• npu-camera's drop count is approximate, inferred client-side from gaps in the frame sequence number — the engine doesn't yet expose an exact per-source drop counter.
• A frame handle expires 5 seconds after it's emitted if nothing consumes it. A flow that's too slow for its camera's frame rate will see E_FRAME_EXPIRED errors on output 2 of whichever node tried to use a stale handle — this is expected backpressure behavior, not a bug to route around.
• Only one model resident at a time on RK3566 (EPG-002S) — see Only one model at a time above.
• Default starter model bundles use permissively-licensed families (YOLOX, PP-YOLOE, MobileNetV3, PP-OCRv4). YOLOv8/v11 architectures are supported by the engine if you bring your own weights, but are AGPL-3.0-licensed upstream and are never the default offered — that's a licensing choice, not a technical limitation.

Next: MQTT broker (Aedes), or back to Modbus / PLC integration for PLC connectivity instead of computer vision.