sample_face_ae¶

This guide explains how to build the K230 sample_face_ae application using CMake out-of-tree build. This sample uses a face detection model (MobileRetinaface) to detect faces and maps the detected face regions to the ISP AE (Auto Exposure) ROI, achieving face-optimized exposure control.

Prerequisites¶

K230 SDK must be built (toolchain extracted, MPP libraries compiled)
SDK placed at k230_sdk/ in the repository root
Face detection kmodel file (mobile_retinaface.kmodel)
Host OS: x86_64 Linux
CMake 3.16 or later

Building the SDK

For K230 SDK build instructions, see SDK Build.

Overview¶

sample_face_ae is based on the K230 SDK face detection sample (sample_face_ae), combining AI inference with ISP AE ROI control. It demonstrates:

Configuring camera sensors via the VICAP API (2 channels: YUV420 for display + RGB888P for AI inference)
Loading kmodel and AI2D preprocessing with the NNCASE runtime
Face detection using the MobileRetinaface model (bounding boxes + landmarks)
Mapping detected face regions to ISP AE ROI (area-weighted)
Real-time preview on VO display with face bounding box overlay

Source Files¶

File	Description
`main.cc`	Main application — VB/VICAP/VO initialization, AI inference loop, cleanup
`model.h` / `model.cc`	`Model` abstract base class — kmodel loading and inference pipeline
`mobile_retinaface.h` / `mobile_retinaface.cc`	`MobileRetinaface` class — face detection model (AI2D preprocessing, anchor decoding, NMS)
`face_ae_roi.h` / `face_ae_roi.cc`	`FaceAeRoi` class — maps face coordinates to ISP AE ROI
`util.h` / `util.cc`	Utility types (`box_t`, `face_coordinate`) and helpers
`anchors_320.cc`	Pre-computed anchor boxes for 320x320 input
`vo_test_case.h`	VO layer helper type (`layer_info`) declarations

Processing Flow¶

The application follows this data flow:

Sensor (OV5647)
  |
  +-- CHN0 (1920x1080 YUV420) --> VO Layer --> HDMI Display
  |                                    ^
  |                              Face box drawing (kd_mpi_vo_draw_frame)
  |
  +-- CHN1 (1280x720 RGB888P) --> AI Inference
                                    |
                              +-----+-----+
                              | AI2D      |
                              | Preprocess|
                              | (resize+pad) |
                              +-----+-----+
                                    |
                              +-----+-----+
                              | KPU       |
                              | Inference |
                              | (MobileRetinaface) |
                              +-----+-----+
                                    |
                              +-----+-----+
                              | Postprocess|
                              | (decode+NMS) |
                              +-----+-----+
                                    |
                              +-----+-----+
                              | AE ROI    |
                              | Update    |
                              | (FaceAeRoi) |
                              +-----+-----+
                                    |
                                    v
                              ISP AE Engine

Class Reference¶

Model — Abstract Base Class¶

Source: model.h L13–L39 / model.cc

Abstract base class that manages kmodel loading and the inference pipeline. Subclasses implement Preprocess() and Postprocess().

Method	Description
`Model(model_name, kmodel_file)`	Loads the kmodel file and creates input tensors
`Run(vaddr, paddr)`	Executes the inference pipeline: `Preprocess` → `KpuRun` → `Postprocess`
`Preprocess(vaddr, paddr)`	Pure virtual — preprocesses input data
`KpuRun()`	Runs the kmodel on the KPU
`Postprocess()`	Pure virtual — postprocesses inference results
`InputTensor(idx)` / `OutputTensor(idx)`	Access input/output tensors
`InputShape(idx)` / `OutputShape(idx)`	Get input/output shapes

Member variables:

Variable	Description
`ai2d_builder_`	AI2D preprocessing pipeline builder
`ai2d_in_tensor_` / `ai2d_out_tensor_`	AI2D input/output tensors
`interp_`	NNCASE runtime interpreter

MobileRetinaface — Face Detection Model¶

Source: mobile_retinaface.h L13–L48 / mobile_retinaface.cc

Inherits from Model and implements preprocessing and postprocessing for the MobileRetinaface face detection model.

Method	Description
`MobileRetinaface(kmodel_file, channel, height, width)`	Builds the AI2D preprocessing pipeline (resize + padding)
`GetResult()`	Returns detection results (bounding boxes + landmarks)
`Preprocess(vaddr, paddr)`	Creates AI2D tensor from VICAP frame and runs preprocessing
`Postprocess()`	Decodes 9 output tensors and filters with NMS

Postprocessing flow:

Confidence decoding (DealConfOpt) — Processes confidence scores from 3 scales with softmax, selects objects above threshold (obj_threshold_ = 0.6)
Location decoding (DealLocOpt) — Decodes bounding box positions using anchor boxes
Landmark decoding (DealLandmsOpt) — Decodes 5-point facial landmarks
NMS — Removes overlapping boxes using IoU threshold (nms_threshold_ = 0.5)
Coordinate conversion — Converts from model coordinates to camera input coordinates

FaceAeRoi — AE ROI Control¶

Source: face_ae_roi.h L9–L21 / face_ae_roi.cc

Converts detected face coordinates into ISP AE ROI windows, enabling face-optimized auto exposure control.

Method	Description
`FaceAeRoi(dev, model_w, model_h, sensor_w, sensor_h)`	Sets the ISP device and model/sensor resolutions
`SetEnable(enable)`	Enables/disables the ISP AE ROI feature (`kd_mpi_isp_ae_roi_set_enable`)
`Update(boxes)`	Converts face bounding boxes to AE ROI windows and applies them

Update() processing:

Scales face coordinates from model resolution to sensor resolution
Sets up to 8 ROI windows
Calculates each ROI weight by area ratio (larger face = higher weight)
Applies to ISP via kd_mpi_isp_ae_set_roi()

Build Steps¶

1. Configure¶

cmake -B build/sample_face_ae -S apps/sample_face_ae \
  -DCMAKE_TOOLCHAIN_FILE="$(pwd)/cmake/toolchain-k230-rtsmart.cmake"

2. Build¶

cmake --build build/sample_face_ae

3. Verify¶

file build/sample_face_ae/sample_face_ae

Expected output:

sample_face_ae: ELF 64-bit LSB executable, UCB RISC-V, RVC, double-float ABI, version 1 (SYSV), statically linked, ...

CMakeLists.txt Details¶

The apps/sample_face_ae/CMakeLists.txt handles:

MPP include paths: Headers from mpp/include/, mpp/include/comm/, mpp/include/ioctl/, and mpp/userapps/api/
NNCASE include paths: Headers from nncase/include/, nncase/include/nncase/runtime/, and rvvlib/include/
MPP static libraries: All MPP libraries linked with --start-group / --end-group to resolve circular dependencies
NNCASE libraries: Nncase.Runtime.Native, nncase.rt_modules.k230, functional_k230, rvv
C++20: Requires C++20 via target_compile_features

Command-Line Arguments¶

./sample_face_ae <kmodel> <roi_enable>

Argument	Description
`<kmodel>`	Path to the face detection kmodel file (e.g., `/sharefs/mobile_retinaface.kmodel`)
`<roi_enable>`	Enable AE ROI: `1` = enabled, `0` = disabled

Transferring and Running on K230¶

The CMake deploy / run targets handle transfer and execution in one command (see CMake Targets for details):

cmake --build build/sample_face_ae --target deploy   # build + SCP transfer
cmake --build build/sample_face_ae --target run      # run via serial (Ctrl+C to disconnect)

Manual Transfer and Execution¶

Manual operation via SCP + minicom

Transfer via SCP¶

scp build/sample_face_ae/sample_face_ae root@<K230_IP_ADDRESS>:/sharefs/sample_face_ae/
scp mobile_retinaface.kmodel root@<K230_IP_ADDRESS>:/sharefs/sample_face_ae/

Run on the K230 bigcore (msh)¶

On the K230 serial console (ACM1), run:

msh /> /sharefs/sample_face_ae/sample_face_ae /sharefs/sample_face_ae/mobile_retinaface.kmodel 1

To run with AE ROI disabled:

msh /> /sharefs/sample_face_ae/sample_face_ae /sharefs/sample_face_ae/mobile_retinaface.kmodel 0

Serial Connection¶

Bigcore (RT-Smart msh): /dev/ttyACM1 at 115200 bps

minicom -D /dev/ttyACM1 -b 115200

CMake Targets¶

Configuration¶

cmake -B build/sample_face_ae -S apps/sample_face_ae \
  -DCMAKE_TOOLCHAIN_FILE="$(pwd)/cmake/toolchain-k230-rtsmart.cmake"

Target List¶

Target	Command	Description
(default)	`cmake --build build/sample_face_ae`	Build C++ binary
`deploy`	`cmake --build build/sample_face_ae --target deploy`	Build + SCP transfer to K230
`run`	`cmake --build build/sample_face_ae --target run`	Run on K230 via serial (Ctrl+C to disconnect)

deploy¶

Builds the binary and transfers it to K230 via SCP:

cmake --build build/sample_face_ae --target deploy

Files transferred:

Local	Path on K230
`build/sample_face_ae/sample_face_ae`	`/sharefs/sample_face_ae/sample_face_ae`
`apps/face_detect/scripts/output/dump/mobile_retinaface.kmodel`	`/sharefs/sample_face_ae/mobile_retinaface.kmodel`

run¶

Sends a command to K230 bigcore (msh) via serial port and displays output in real time:

cmake --build build/sample_face_ae --target run

Keyboard input is forwarded to K230 (q + Enter to quit the app)
Ctrl+C to disconnect from serial

K230 Connection Settings¶

Customize connection parameters via CMake cache variables:

Variable	Default	Description
`K230_IP`	(empty = auto-detect)	Littlecore IP address
`K230_USER`	`root`	SSH user
`K230_SERIAL`	`/dev/ttyACM1`	Bigcore serial port (for run)
`K230_SERIAL_LC`	`/dev/ttyACM0`	Littlecore serial (for IP auto-detect)
`K230_BAUD`	`115200`	Baud rate