Native AI on NPUs: Constructing a zero-cloud picture deduplicator with OpenCLIP

$ python export_ov.py --batch 1

exporting picture encoder to OpenVINO IR (static batch=1) ...
IR: ~/photo-dedup/dedup/fashions/clip-vitb32-b1.xml
enter : [1,3,224,224] 
output: [1,512] 

The one-time export bakes the PyTorch encoder right into a static-batch OpenVINO IR—one construct per batch dimension, loaded by the NPU runs that comply with.

$ sudo ./.venv/bin/python embed.py ./scan --backend npu

backend: NPU (compiled clip-vitb32-b1.xml, trace=LATENCY, batch=1)
preprocessing: /usr/sbin/magick (parallel, 4 staff)
ready 373 crop(s) in 0.0s
[1/373] photo0000.png     26.1 ms -> photo0000.vitb32.npy
[2/373] photo0002.png     25.6 ms -> photo0002.vitb32.npy
[3/373] photo0020.png     24.9 ms -> photo0020.vitb32.npy
…
embedded 373 in 5.7s (65.3 img/s) [backend=npu, batch=1], 0 skipped (cached), avg energy 27.3 W

A naive, synchronous NPU loop improves throughput to ~25 milliseconds per picture, however leaves the accelerator idle between dispatch calls. Energy consumption considerably decreases to 27W

To unlock the silicon’s true potential, I restructured the script to make use of batched inference and utilized OpenVINO’s THROUGHPUT efficiency trace. By feeding the NPU batches of 16 photos concurrently, I saturated its execution items and eradicated the dispatch bottleneck. Per-photo inference time dropped from ~36 ms on the CPU to about 13 ms on the NPU, and end-to-end throughput climbed from roughly 28 photographs per second (a serial, single-image go) to over 92—a 3x win. However uncooked pace wasn’t the true victory—effectivity was. The NPU silently chewed by way of hundreds of photos within the background, drawing a mere 22W, permitting processing of a large media archive on battery energy with out triggering the followers.