C++

This guide targets Optimium Runtime 4.4.0.

Breaking change since 4.3.x: InferRequest::infer() now takes inputs only. The runtime allocates and owns the output buffers; retrieve them with getOutput() after wait(). The old infer(inputs, outputs) overloads are gone.

1. Install prerequisites

To build an app against Optimium Runtime you need CMake 3.23 or above and a C++ compiler that supports C++14 (tested on MSVC v14.43, GCC 9, Clang 11 or above).

# debian-based distros
sudo apt-get install build-essential cmake ninja-build

2. Install Optimium Runtime

Please click here to install the runtime.

3. Add Optimium Runtime as a dependency

Optimium Runtime ships a CMake package config (share/cmake/Optimium-Runtime). Point CMAKE_PREFIX_PATH at the install prefix and use find_package.

find_package(Optimium-Runtime REQUIRED)
target_link_libraries(MyExecutable PRIVATE Optimium::Runtime)

# Use C++14
set(CMAKE_CXX_STANDARD 14)

Optimium Runtime requires C++14 to compile correctly. Use set(CMAKE_CXX_STANDARD 14) to set the language version globally, or set_target_properties(<TARGET> PROPERTIES CXX_STANDARD 14) to apply it to a single target.

If you also want remote inference from C++, link the separate Optimium-Remote package:

find_package(Optimium-Remote REQUIRED)
target_link_libraries(MyExecutable PRIVATE Remote::Client)

Note the different namespace: the remote client target is Remote::Client, not Optimium::Remote.

IMPORTANT! If you're targeting Android, add the following:

set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE BOTH)
find_package(Optimium-Runtime REQUIRED)
target_link_libraries(MyExecutable PRIVATE Optimium::Runtime)

Plus, on Android you must add android:extractNativeLibs="true" in your AndroidManifest.xml file.

 <application ...
              android:extractNativeLibs="true"
              ...>

4. Initialize the runtime

Before loading a model you must initialize the runtime. Logging is configured before initialization with logging::setLogLevel() and logging::addLogWriter().

#include <Optimium/Runtime.h>
#include <Optimium/Runtime/Logging/LogSettings.h>
#include <Optimium/Runtime/Logging/ConsoleWriter.h>
#include <Optimium/Runtime/Logging/FileWriter.h>
#if defined(__ANDROID__)
#include <Optimium/Runtime/Logging/AndroidLogWriter.h>
#endif

// Stream operators for runtime types (TensorInfo, TensorShape, DeviceID,
// ElementType, InferStatus, ...) are NOT pulled in by <Optimium/Runtime.h>.
#include <Optimium/Runtime/Utils/StreamHelper.h>

namespace rt = optimium::runtime;

int main(...) {
    // change verbosity to debug level.
    // levels: Debug, Verbose, Info, Warning, Error
    rt::logging::setLogLevel(rt::LogLevel::Debug);

    // add console log writer.
    rt::logging::addLogWriter(std::make_unique<rt::logging::ConsoleWriter>());

    // add file log writer to "output.log" file.
    rt::logging::addLogWriter(std::make_unique<rt::logging::FileWriter>("output.log"));

#if defined(__ANDROID__)
    // add Android Logcat log writer. this class only exists on Android.
    rt::logging::addLogWriter(std::make_unique<rt::logging::AndroidLogWriter>());
#endif

    // Explicitly initialize and finalize the runtime.
    rt::initialize();

    // ... (load model, run inference, etc.)

    rt::finalize();
}

AndroidLogWriter is declared inside #ifdef __ANDROID__. Guard both the include and the use, or the code will not compile on other platforms.

Likewise, printing runtime types with operator<< requires Optimium/Runtime/Utils/StreamHelper.h, which the umbrella header does not include. All the std::cout << ... samples below assume it.

Initialization lifecycle

The runtime must be initialized before any model loading or inference, and finalized after all work is done. There are two approaches.

Approach 1: explicit initialize() / finalize() calls

int main(...) {
    rt::initialize();

    // ... load models, run inference ...

    rt::finalize();
}

Approach 2: rt::AutoInit (RAII)

rt::AutoInit calls rt::initialize() in its constructor and rt::finalize() in its destructor. Because rt::finalize() shuts the runtime down entirely, AutoInit must outlive every model, request and inference. If declared inside a narrow scope, the runtime is finalized when the variable goes out of scope and any in-flight inference will fail.

// Recommended: static global — runtime lives for the entire process lifetime.
static rt::AutoInit Init;

int main(...) {
    // ... load models, run inference ...
    // rt::finalize() is called automatically when the process exits.
}

// Also OK: top of main() — runtime lives until main() returns.
int main(...) {
    rt::AutoInit Init;

    // ... load models, run inference ...
}

Warning: Do NOT declare rt::AutoInit inside a loop or a helper function. If the destructor runs while requests are still alive, the runtime is finalized prematurely, causing undefined behavior or crashes.

Version information

#include <Optimium/Runtime/Version.h>

rt::Version V = rt::getVersion();
std::cout << V.Major << "." << V.Minor << "." << V.Patch << V.Tag << "\n"
          << "commit: " << V.CommitID << "\n"
          << "build:  " << V.BuildInfo << std::endl;

// Version the headers were compiled against.
constexpr rt::Version Compiled = OPTIMIUM_RT_CURRENT_VERSION;
if (V != Compiled)
    std::cerr << "warning: runtime/header version mismatch" << std::endl;

getVersion() (the loaded shared library) may differ from OPTIMIUM_RT_CURRENT_VERSION (the headers). A mismatch means a broken installation.

Check available devices

It is recommended to check available devices before loading a model. A missing device is the most common cause of DeviceError at load time.

int main(...) {
    // ...

    rt::HostInfo Local = rt::getLocalInfo();

    bool Found = false;
    for (rt::DeviceID ID : Local.Devices) {
        if (ID.getPlatform() == rt::PlatformKind::Native) {
            Found = true;
            break;
        }
    }

    if (!Found)
        std::cout << "error: cannot find needed device." << std::endl;
}

HostInfo contains:

• int ID — host identifier (0 is always localhost)
• StringRef Name — host name
• DeviceKind Architecture — host CPU architecture
• OSKind OS — Linux, Android, Windows, MacOS, IOS
• ArrayRef<DeviceID> Devices — available devices on this host

DeviceID provides:

• PlatformKind getPlatform() — Native, XNNPack, CUDA, Vulkan, OpenCL, SNPE, QNN, Synaptics, Hexagon
• DeviceKind getDeviceKind() — x86, x64, ARM, ARM64, RISCV32, RISCV64, AnyCPU, NVIDIA, AMDGPU, IntelGPU, Mali, Adreno, AnyGPU, Hexagon, Myriad, ARA1, GoogleTPU, CV22, SynapNPU, AnyNPU, Neutral, Unspecified
• uint32_t getIndex() — device index among the same kind on the host
• uint32_t getHostID() — host ID
• HostInfo getHostInfo() — host info for this device
• const Capability &getCapability() — hardware capabilities (X86Capability, ARMCapability, SPIRVCapability, CUDACapability, RISCVCapability, HexagonCapability, VoidCapability)
• std::string toString() — string representation
• static DeviceID from(PlatformKind, DeviceKind, uint8_t Index, uint8_t Host) — construct a DeviceID

Free helpers isDeviceCPU(), isDeviceGPU(), isDeviceNPU() and isDeviceNeutral() classify a DeviceKind.

5. Load a model

Model represents an ML model. Load one via rt::loadModel(), either from a path or from an in-memory buffer.

OPTIMIUM_RT_API Model loadModel(const std::string &Path, const ModelLoadOptions &Options = {});
OPTIMIUM_RT_API Model loadModel(const uint8_t *Data, size_t Size, const ModelLoadOptions &Options = {});

Unlike other inference engines such as TensorFlow Lite, an Optimium model may be a folder. The folder itself is the model, so pass the path to the folder and always copy the model together with its folder.

Warning: the in-memory overload does not copy the buffer. Keep it alive for the lifetime of the Model and every InferRequest created from it.

ModelLoadOptions

ModelLoadOptions is composed of two reusable bases plus the scheduler choice:

ModelLoadOptions : PerModelOptions, PoolTuning { SchedulerType Scheduler; }

PerModelOptions — per-model attributes

PoolTuning — shared thread-pool attributes (one per pool)

Scheduler

An explicit Exclusive / Pipeline / Sequential choice is always honored and never overridden.

int main(...) {
    // load a model with default options (auto-detected devices, Auto scheduler).
    rt::Model Model = rt::loadModel("path/to/model");

    // load a model with manual configuration.
    rt::ModelLoadOptions Options;
    Options.Devices = { rt::DeviceID::from(rt::PlatformKind::Native, rt::DeviceKind::ARM64) };
    Options.Strict = false;
    Options.EnableMemoryOptimization = true;
    Options.DisableDenormals = true;
    Options.Passphrase = "";
    Options.ThreadCount = 4;
    Options.Cores = {0, 1, 2, 3};
    Options.SpinIterations = rt::kDefaultSpinIterations;
    Options.SpinYieldIterations = rt::kDefaultSpinYieldIterations;
    Options.EnableWorkStealing = true;   // default
    Options.Scheduler = rt::SchedulerType::Auto;

    rt::Model Model2 = rt::loadModel("path/to/model", Options);
}

Tuning latency: spin vs. park

SpinIterations is the main latency knob for the intra-op CPU pool.

• 0 — workers park immediately. Lowest CPU usage while idle, but each dispatch after an idle gap pays a ~ms cold wake.
• N > 0 (default) — bounded spin then park. A warm window that cannot peg cores indefinitely.
• -1 (any negative) — spin while armed: the pool stays warm across inter-op gaps, giving the lowest per-op dispatch latency. Unbounded, so it can throttle big cores at high thread counts on mobile SoCs. The pool only spins while a request is actually running, so an idle loaded model does not peg cores.

All schedulers honor this setting.

Work stealing

EnableWorkStealing is on by default, matching upstream Google pthreadpool (which always steals). Each worker consumes its own contiguous chunk of a kernel fan-out from the front, and once out of work takes unclaimed items from the tail of busy workers' chunks.

Measured on a Snapdragon 778G (A78 + A55), it is win-or-neutral across configurations: ~2.1–2.5× on big.LITTLE, ~1.06–1.21× on homogeneous cores, no regression on small/cheap ops. The cost is one atomic claim per item, and dispatches too small to benefit (item count ≤ worker count) keep the static split regardless.

Set it to false only if you need a deterministic, static contiguous partition — for example when bisecting a numerical difference. Optimium's own kernels cannot depend on the partition (the launch path uses only the coordinate parallelize variants, never the per-worker-index ones), and XNNPack tolerates stealing by design.

Work stealing is a pool-level knob, so in a ModelGroup it belongs to ModelGroupOptions, not to individual members.

6. Listing model information

int main(...) {
    // print model name
    std::cout << "model name: " << Model.getName() << std::endl;

    // print list of input tensor info
    std::cout << "input tensors" << std::endl;
    for (rt::StringRef Name : Model.getInputNames())
        std::cout << Model.getTensorInfo(Name) << std::endl;

    // print list of output tensor info
    std::cout << "output tensors" << std::endl;
    for (rt::StringRef Name : Model.getOutputNames())
        std::cout << Model.getTensorInfo(Name) << std::endl;

    // access by index
    const rt::TensorInfo &FirstInput = Model.getInputTensorInfo(0);
    const rt::TensorInfo &FirstOutput = Model.getOutputTensorInfo(0);

    // every tensor in the model (inputs, outputs, intermediates)
    for (const rt::TensorInfo &Info : Model.getTensors())
        std::cout << Info << std::endl;

    // list operations
    for (const rt::OpInfo &Op : Model.getOperations())
        std::cout << "op: " << Op.Name << " on device: " << Op.Device << std::endl;
}

TensorInfo describes a single tensor:

const rt::TensorInfo &Info = Model.getTensorInfo("input_0");

std::cout << "name:      " << Info.Name << "\n"
          << "shape:     " << Info.Shape << "\n"
          << "alignment: " << Info.Alignment << "\n"
          << "type:      " << Info.Type << "\n"
          << "padding:   " << Info.Padding << "\n"
          << "bytes:     " << Info.getTensorSize() << "\n"
          << "resizable: " << Info.Resizable << std::endl;

if (Info.Scheme)
    std::cout << "scheme: " << *(Info.Scheme) << std::endl;

Model attributes

The compiler can attach attributes to the model's top-level (main) subgraph — for example decoder metadata. Read them with getAttribute():

if (Model.hasAttribute("model_type")) {
    rt::Attribute Attr = Model.getAttribute("model_type");
    if (Attr.isa<rt::StringAttr>())
        std::cout << "type: " << Attr.cast<rt::StringAttr>().value() << std::endl;
}

for (rt::StringRef Key : Model.getAttributeKeys())
    std::cout << "attribute: " << Key << std::endl;

getAttribute() returns an empty Attribute (its operator bool is false) when the key does not exist. Concrete attribute classes are I8Attr … I64Attr, U8Attr … U64Attr, F16Attr, F32Attr, F64Attr, BF16Attr, TF32Attr, QS8Attr … QS32Attr, BoolAttr, StringAttr, ListAttr and MapAttr.

Dynamic (symbolic) shape models

Models with symbolic shapes support shape inference through Model.inferShape():

// infer output shape from input shapes (by name)
std::map<std::string, rt::TensorShape> InputShapes;
InputShapes["input_0"] = rt::TensorShape({1, 3, 224, 224});
rt::TensorShape OutputShape = Model.inferShape("output_0", InputShapes);

// infer output shape from input shapes (by index)
std::vector<rt::TensorShape> InputShapeList = { rt::TensorShape({1, 3, 224, 224}) };
rt::TensorShape OutputShape2 = Model.inferShape("output_0", rt::make_array(InputShapeList));

TensorShape provides getRank(), getElementCount(), getStride(i), getStrides(), isDynamic(), isCompatible(other), resolve(variables), iteration and operator[]. A symbolic dimension is an rt::Expr, built from Expr(42) (constant) or Expr("batch") (symbol), and supports +, -, *, /, %, min() and max().

Models may also contain control-flow operations (if / while / parallel / switch). These are handled entirely inside the runtime — no extra API calls are needed — but a switch whose selector matches no case and that has no default case raises an error at run time.

7. Creating a request

InferRequest represents a single inference the model runs. You can create multiple requests and run the same model without them interfering with each other, and queue several requests to reach a target throughput.

rt::InferRequest Request = Model.createRequest();

8. Preparing inputs

Create tensors with rt::tensor().

// Create float32 tensor shaped 32x32.
rt::TypedTensor<float> f32_tensor = rt::tensor<float>({32, 32});

// Create generic tensor shaped 8.
rt::Tensor i16_tensor = rt::tensor(rt::ElementType::I16, {8});

// Create float16 tensor shaped 32x32 with a user-provided buffer.
rt::float16 *Data = new rt::float16[32 * 32];
rt::TypedTensor<rt::float16> f16_tensor = rt::tensor<rt::float16>({32, 32}, Data);

Both overloads take a trailing bool Padding = true argument.

When you create a tensor with a user-provided buffer, take extra care.

Optimium Runtime does not take ownership of the buffer, so it must not be freed before the tensor is finalized.

The runtime always assumes the buffer is valid. An invalid buffer (too small, bad pointer) can cause severe errors.

Keep Padding at true unless you know better — some models assume padding exists and will corrupt memory without it.

Access tensor memory with Tensor.data(); fill it with memcpy, std::copy, or TypedTensor.fill().

// Load from existing data.
std::vector<float> Data;
rt::TypedTensor<float> Tensor = rt::tensor<float>({32, 32});
std::copy(Data.begin(), Data.end(), Tensor.data());

// Load from a file. <fstream> is required.
std::ifstream File("path/to/file", std::ios::in | std::ios::binary);
File.read(reinterpret_cast<char *>(Tensor.data()), Tensor.getTensorSize());

// Fill with a scalar. 'Tensor' must be a TypedTensor.
Tensor.fill(1.0f);

// Save / load tensor contents.
Tensor.save("tensor.bin");
Tensor.load("tensor.bin");

// Copy into another tensor.
rt::Tensor Other = rt::tensor(rt::ElementType::F32, {32, 32});
rt::Tensor(Tensor).copyTo(Other);

// Release tensor data explicitly.
Tensor.release();

Optimium Runtime provides types that C++ does not support natively; they live under Optimium/Runtime/Types.

The remaining element types map to plain C++ types: I8, U8, I16, U16, I32, U32, I64, U64, F32, F64, Bool, String.

The runtime only recognizes these C++ types for the corresponding tensor type. Other data types are not recognized and result in a compilation error.

You can also tune how tensors print:

rt::config::setPrintThreshold(10);  // max elements per rank; < 0 = unlimited
rt::config::setPrintPrecision(4);   // decimal precision for floats

9. Running an inference

Inference is two calls: infer() starts it and returns immediately, wait() blocks until it finishes (successfully or not). Only inputs are passed in — output buffers are owned by the runtime and read back with getOutput().

infer() has two overloads: one takes rt::ArrayRef<Tensor> (positional, must match the model's input order) and one takes std::map<std::string, Tensor> (by name).

int main(...) {
    // running inference with an input list
    std::vector<rt::Tensor> Inputs;
    for (rt::StringRef Name : Model.getInputNames()) {
        const rt::TensorInfo &Info = Model.getTensorInfo(Name);
        Inputs.push_back(rt::tensor(Info.Type, Info.Shape));
    }

    // rt::make_array is a helper that creates an rt::ArrayRef.
    Request.infer(rt::make_array(Inputs));
    Request.wait();

    // read the outputs back
    rt::Tensor Out0 = Request.getOutput(0);
    rt::Tensor OutByName = Request.getOutput("output_0");
}

int main(...) {
    // running inference with an input map
    std::map<std::string, rt::Tensor> Inputs;
    for (rt::StringRef Name : Model.getInputNames()) {
        const rt::TensorInfo &Info = Model.getTensorInfo(Name);
        Inputs[Name] = rt::tensor(Info.Type, Info.Shape);
    }

    Request.infer(Inputs);
    Request.wait();
}

Output lifetime. getOutput() returns a Tensor that shares the runtime's internal buffer through reference counting: the memory stays valid as long as your Tensor is alive, even if a later infer() reallocates internal buffers. However, for static-shape models the buffer is reused across runs — a held Tensor is overwritten in place by the next infer(). Use Tensor::copyTo() if you need a snapshot.

You do not need to create input tensors for every inference. Tensors can be reused between requests and models as long as they are not used simultaneously. (You cannot feed request A's output tensor into request B while A is still running; it is fine once A has finished.)

Request.wait() takes an optional timeout. It returns false when the inference finished before the timeout, and true when the timeout was reached first. Called with no argument (or zero) it waits indefinitely.

Always call Request.wait() — it is where errors raised during inference surface. Starting an inference on a request that is already in a fault state is undefined behavior.

Request.getStatus() returns InferStatus::Ready, InferStatus::Running or InferStatus::Fault.

using namespace std::chrono_literals;

// wait until inference is finished.
Request.infer(rt::make_array(Inputs));
Request.wait();

// wait 500 milliseconds
Request.infer(rt::make_array(Inputs));
if (Request.wait(500ms))
    std::cout << "inference not finished after 500ms" << std::endl;
else
    std::cout << "inference finished within 500ms" << std::endl;

std::cout << "status of request: " << Request.getStatus() << std::endl;

InferRequest also mirrors the model's metadata: getModelName(), getModelOperations(), getModelTensors(), getTensorInfo(name), getInputTensorInfo(i), getOutputTensorInfo(i).

Callbacks

Request.addCallback([](rt::InferStatus Status, std::exception_ptr Err) {
    if (Status == rt::InferStatus::Fault) {
        try { std::rethrow_exception(Err); }
        catch (const std::exception &E) {
            std::cerr << "Inference error: " << E.what() << std::endl;
        }
    }
});

Request.infer(rt::make_array(Inputs));
Request.wait();

Cancellation

Request.infer(rt::make_array(Inputs));
// ... later:
Request.cancel();

10. Resizable tensors (runtime shape changes)

New in 4.4.0. A tensor marked Resizable in the model metadata has a concrete base shape but may be resized at run time — the typical case is a shared KV-cache that tracks the active sequence length. This is not the same as a symbolic dynamic shape.

void InferRequest::setTensorShape(StringRef Name, const TensorShape &Shape);
void InferRequest::applyShapeChanges(bool ShrinkToFit = false);
void InferRequest::releaseStaleShapeMemory();
void InferRequest::setResizeMigration(StringRef Name, ResizeMigrateFn Fn);

setTensorShape() records a new concrete shape; it takes effect on applyShapeChanges() or on the next infer().

Capacity planning

Declare a capacity per tensor at load time so resizes are cheap:

rt::ModelLoadOptions Options;
Options.ResizableCapacities.emplace("cache", rt::TensorShape({2048}));
rt::Model Model = rt::loadModel("path/to/model", Options);

rt::InferRequest Request = Model.createRequest();

// Within capacity: metadata-only. Same backing pointer, content stays in place,
// no re-plan, no reallocation, no device re-registration.
Request.setTensorShape("cache", rt::TensorShape({1536}));
Request.applyShapeChanges();

// Beyond a DECLARED capacity: hard cap.
// throws InvalidArgumentError
// Request.setTensorShape("cache", rt::TensorShape({4096}));

Behavior summary:

Validation at load: the tensor must exist and be Resizable, the capacity must be concrete with the same rank as the base shape, and its element count must not be smaller than the base shape's. Capacities are rejected for symbolic-dynamic models (ModelError).

Eager application

applyShapeChanges() applies pending resizes immediately instead of lazily on the next infer(). Within capacity it is a metadata-only update. On an overflow it re-plans and reallocates — the Resizable pool alone when every overflowing tensor is device-owned, the full variable pools otherwise — and re-registers the affected device buffers; afterwards getTensor() / getOutput() pointers reflect the new allocation. It is a no-op when nothing changed.

It throws InvalidStateError when an inference is currently running. On the reallocating path it can also propagate whatever the memory planner or the device backend raises (for example OutOfResourceError under memory pressure); the request is marked dirty in that case so the next infer() re-plans cleanly.

Changed in 4.4.0: an eager host-owned overflow no longer throws InvalidOperationError. It now takes the full reallocation path, the same one the lazy infer() recalculation uses.

Cached content survives the reallocation on every buffer policy. Host-owned and MemoryMappable buffers are migrated by pointer; a pure device-resident buffer is read back to host memory and written into the fresh buffer (a device→host→device round-trip, the same path inputs already use). You no longer have to hand-migrate anything, so releaseStaleShapeMemory() is now purely a memory-release call — see below.

Reclaiming capacity with ShrinkToFit

New in 4.4.0. Capacity used to be monotonic: it grew on overflow and never shrank, so a grow-then-shrink KV-cache kept its high-water footprint for the life of the request. applyShapeChanges(true) releases that headroom.

// Long prompt: the cache overflows its base capacity and the pool grows.
Request.setTensorShape("cache", rt::TensorShape({8192}));
Request.applyShapeChanges();

// ... decode the long sequence ...

// Short conversation next. Drop back down AND give the memory back.
Request.setTensorShape("cache", rt::TensorShape({512}));
Request.applyShapeChanges(/*ShrinkToFit=*/true);

What it does: for every Resizable tensor, re-freeze the capacity down to whichever of its current logical shape and its floor holds more elements, reallocate the affected pools fresh, and release the retained old pools so the footprint actually drops.

• The floor is the tensor's base shape, or its ResizableCapacities entry when one was declared. A declared capacity is a contract that resizes within it never reallocate, so it is never lowered.
• The comparison is on element count, not per axis. (A per-axis max of [8,2] and [4,4] would be [8,4] — larger than both, which would grow the pool during a reclaim.)
• Cached content is preserved exactly as for any other resize: the index-preserving overlap is migrated and any setResizeMigration callback runs. Only elements beyond the reclaimed slot are dropped.
• It is a no-op only when nothing changes. A pending setTensorShape that still exceeds its floor is applied, and may itself grow a capacity.
• On a symbolic-dynamic model it is a no-op — those capacities are managed lazily at the next infer().

The asymmetry is deliberate and mirrors std::vector: growth is implicit (an overflow forces it), release is explicit (shrink_to_fit). A long-running server that alternates between long and short sequences is the case this exists for.

releaseStaleShapeMemory()

After a reallocating applyShapeChanges() the previous variable pools are retained. releaseStaleShapeMemory() frees them, and is a no-op when nothing is retained.

Since content migration is now automatic for every buffer policy, you no longer need this call to stage a hand-written migration between the old and new buffers. It remains available on the eager grow path for callers that want to do their own device-to-device copy before the old pools go away. Do not infer() between applyShapeChanges() and releaseStaleShapeMemory() if you are using it that way — the old buffers' device registrations are already dropped.

applyShapeChanges(true) calls it as its final step — but only when the reclaim actually reallocates. If every capacity is already at its target, ShrinkToFit returns early and retained pools stay held. So after a grow, a bare applyShapeChanges(true) with no pending shrink will not free anything:

Request.setTensorShape("cache", rt::TensorShape({8192}));
Request.applyShapeChanges();           // grows; old pools retained

Request.applyShapeChanges(true);       // capacity already 8192 -> early return,
                                       // old pools still held
Request.releaseStaleShapeMemory();     // this is what frees them

// Whereas an actual shrink does release them itself:
Request.setTensorShape("cache", rt::TensorShape({512}));
Request.applyShapeChanges(true);       // reallocates down AND releases

Custom migration callbacks

By default the runtime performs an index-preserving row-major overlap copy (grown region zeroed) when a Preserved (cached) tensor is remapped. Override it per tensor:

Request.setResizeMigration("cache", [](const rt::Tensor &Old, rt::Tensor &New) {
    const auto *O = static_cast<const float *>(Old.data());
    auto *N = static_cast<float *>(New.data());
    const uint32_t OldN = Old.getShape()[0].value();
    const uint32_t NewN = New.getShape()[0].value();
    const uint32_t Count = OldN < NewN ? OldN : NewN;
    // e.g. a ring-buffer shift, a non-outer growth axis, or a custom fill.
    for (uint32_t I = 0; I < Count; ++I)
        N[I] = O[I];
});

// pass an empty function to restore the default migration
Request.setResizeMigration("cache", nullptr);

Old is a read-only view laid out for the pre-resize shape; New is writable and zero-initialized for the new shape — the runtime then writes New out to every destination buffer. The callback fires wherever the default copy would run: a within-capacity in-place restride, a reallocating overflow, and a ShrinkToFit reclaim. As of 4.4.0 it also fires for purely device-resident tensors, which receive the host read-back view. A per-step growing cache restrides every step, so keep the callback cheap. It throws InvalidArgumentError for an unknown or non-Preserved tensor.

11. Model groups

New in 4.4.0. Models that never run concurrently — alternating pipeline stages, an encoder/decoder pair, a prefill/decode split — can share one thread pool and one scheduler instead of owning N pools each.

A shared pool is re-armed by each member in turn, so it never sits idle across the alternation: the park/wake penalty every per-model pool pays when models alternate disappears, and one worker set replaces N.

#include <Optimium/Runtime.h>

int main(...) {
    rt::ModelGroupOptions GroupOptions;   // PoolTuning knobs only
    GroupOptions.Cores = {4, 5, 6, 7};
    GroupOptions.SpinIterations = -1;     // stay warm across the alternation
    GroupOptions.DisableDenormals = true;
    GroupOptions.EnableWorkStealing = true;   // default

    rt::ModelGroup Group = rt::createModelGroup(GroupOptions);

    rt::GroupMemberOptions MemberOptions;  // PerModelOptions only
    MemberOptions.ThreadCount = 4;

    rt::GroupMember Encoder = Group.add("path/to/encoder", MemberOptions);
    rt::GroupMember Decoder = Group.add("path/to/decoder", MemberOptions);

    std::vector<rt::Tensor> EncIn = /* ... */;
    std::vector<rt::Tensor> DecIn = /* ... */;

    // run() is BLOCKING: dispatch + wait folded into one call.
    Encoder.run(rt::make_array(EncIn));
    rt::Tensor EncOut = Encoder.getOutput(0);

    for (int Step = 0; Step < Steps; ++Step) {
        Decoder.run(rt::make_array(DecIn));
        rt::Tensor Logits = Decoder.getOutput("logits");
    }
}

Key points:

• GroupMember is deliberately not a Model. There is no createRequest() and no async InferRequest; the group owns execution, and the "members never run concurrently" contract is expressed in the call shape. run() blocks.
• A concurrent run() on another member from another thread blocks on the group's shared scheduler until the current one completes. The mutual exclusion is enforced and visible, never a silent async queue.
• The member's internal request is reused across calls, so cached/preserved tensors (KV-style buffers) persist between runs.
• Membership seals on the first run() of any member. A later add() throws InvalidStateError.
• The shared pool auto-sizes to the largest member's ThreadCount. Each member keeps its own ThreadCount for its compiled parallel ranges.
• Grouped members always run on the shared Exclusive scheduler — Scheduler is not part of ModelGroupOptions.
• Options are split by ownership so they cannot conflict: pool knobs (Cores, SpinIterations, SpinYieldIterations, DisableDenormals, EnableWorkStealing) live on ModelGroupOptions; everything per-model lives on GroupMemberOptions.

GroupMember API: run(inputs), getOutput(index), getOutput(name), getTensor(name), setTensorShape(name, shape), getInputNames(), getOutputNames(), getTensorInfo(name), getInputTensorInfo(i), getOutputTensorInfo(i), getName().

ModelGroup API: add(path[, options]), add(data, size, options), size(), sealed().

Warning: getOutput() points into the member's reused variable memory; the next run() overwrites it in place. Copy out with Tensor::copyTo for a snapshot.

12. Profiling

InferRequest::profile() runs warm-up plus measured iterations and records timestamped events. Unlike infer(), profile() is blocking.

int main(...) {
    rt::ProfileOptions Options;
    Options.Repeat = 100;                                     // measured iterations
    Options.WarmUp = 10;                                      // warm-up count
    Options.WarmUpTime = std::chrono::microseconds(1000000);  // warm-up duration cap
    Options.StopThreshold = std::chrono::microseconds(0);     // early-stop threshold
    Options.CheckPeriod = 0;                                  // threshold check period
    Options.EventBufferSize = rt::kDefaultEventBufferSize;    // event capacity (default 4,194,304 events)

    // optional cooperative stop flag
    auto StopFlag = std::make_shared<std::atomic_bool>(false);

    Request.profile(rt::make_array(Inputs), Options, StopFlag);

    for (const rt::ProfileEvent &Event : Request.getProfileEvents())
        std::cout << "Event: " << rt::toString(Event.Kind)
                  << " at " << Event.TimeStamp.time_since_epoch().count()
                  << std::endl;

    std::cout << "Model: " << Request.getModelName() << std::endl;
    for (const rt::OpInfo &Op : Request.getModelOperations())
        std::cout << "  Op: " << Op.Name << std::endl;
}

profile() also has a std::map<std::string, Tensor> overload.

For custom lock-free event recording, install your own recorder before profiling:

auto Recorder = std::make_shared<rt::ProfileEventRecorder>(1024 * 1024);
Request.setRecorder(Recorder);
Request.profile(rt::make_array(Inputs), Options);

setRecorder() does nothing if a recorder is already set.

EventBufferSize and the ProfileEventRecorder constructor argument are a number of events, not a byte count. kDefaultEventBufferSize is 1024 * 1024 * 4 = 4,194,304 events (each ProfileEvent is at most 32 bytes).

Profile event kinds: ModelExecuteBegin/End, LayerExecuteBegin/End, LaunchBegin/End, DeviceExecuteBegin/End, CopyBegin/End, QueueBegin/End, WaitBegin/End.

Intermediate tensor access

Disable memory optimization (or set IntermediateSavePath) to inspect intermediates:

rt::ModelLoadOptions Options;
Options.EnableMemoryOptimization = false;
rt::Model Model = rt::loadModel("path/to/model", Options);

auto Request = Model.createRequest();
Request.infer(rt::make_array(Inputs));
Request.wait();

rt::Tensor IntermediateTensor = Request.getTensor("intermediate_tensor_name");

With IntermediateSavePath set, add IndexedIntermediateSave = true to prefix each dump with the run index (0_<name>, 1_<name>, …) instead of overwriting.

13. Error handling

Optimium Runtime uses exception-based error handling. Every exception derives from optimium::runtime::Exception, which derives from std::runtime_error.

• InvalidArgumentError — invalid argument passed
• InvalidStateError — unexpected internal state
• InvalidOperationError — operation not allowed in the current state
• TypeError — type mismatch
• ShapeError — shape mismatch or incompatible shapes
• ExtensionError — extension loading or initialization failure
• DeviceError — device not found or operation failure
• ModelError — model loading or compilation failure
• RequestError — request operation error
• InferError — inference execution failure
• OutOfResourceError — resource allocation failure
• ContainerError — model container is invalid or corrupted
• RemoteError — remote communication error
• IOError — I/O operation error
• NetworkError — network communication error
• OSError — operating system error
• NotImplementedError — feature not yet implemented

Internally the runtime propagates errors with the monadic Result<T> template and converts them to exceptions at the API boundary.

14. Extensions

Hardware backends ship as separate shared libraries named liboptimium-runtime-<backend>.so (.dll on Windows).

At initialize() the runtime scans its own directory and auto-loads any of these known names it finds: xnnpack, cuda, vulkan, opencl, snpe, qnn. Failures during auto-load are logged as warnings, not thrown.

Anything not on that list must be loaded explicitly:

rt::loadExtension("path/to/liboptimium-runtime-hexagon.so");

loadExtension() throws InvalidArgumentError if the path is not a file, and ExtensionError if the extension is already loaded for that platform or fails to register.

Extensions actually built from this tree in 4.4.0:

CUDA, OpenCL and SNPE still exist as PlatformKind values and as auto-load names, and ENABLE_OPENCL / ENABLE_SNPE options remain, but no extension is built for them in this tree — they are carried over from 0.3.x and not yet ported.

15. Remote inference (C++)

Remote inference lives in the separate Optimium-Remote package.

#include <Optimium/Remote.h>

namespace rt = optimium::runtime;

int main(...) {
    rt::ConnectOptions ConnOpts;
    ConnOpts.EnableSecureConnection = true;
    ConnOpts.EnableCompression = true;

    rt::RemoteSession Session =
        rt::connect("192.168.1.100", rt::kDefaultRemotePort, ConnOpts);

    std::cout << Session.getHostInfo().toString() << std::endl;

    rt::ModelLoadOptions Options;
    Options.ThreadCount = 4;
    rt::RemoteModel Model = Session.loadModel(Options, "path/on/remote/model");

    std::vector<rt::Tensor> Inputs = /* ... */;
    Model.infer(rt::make_array(Inputs));   // asynchronous
    Model.wait();

    rt::Tensor Out = Model.getTensor("output_0");
}

RemoteModel exposes infer() (map and array overloads), profile(), wait(timeout), cancel(), setTensor(), getTensor(), getProfileEvents(), inferShape(), getInputTensorInfos(), getOutputTensorInfos(), getTensorInfos(), getOperationList(), getModelName(), isDynamic(), isSavedTensor(), getAttribute(), getAttributeKeys() and hasAttribute().

Model groups work remotely too: Session.createModelGroup(ModelGroupOptions) returns a RemoteModelGroup, whose add(GroupMemberOptions, Path) returns a RemoteGroupMember with the same blocking run() contract. The server owns a real ModelGroup, so grouped members share one server-side pool.

The server-side binary is optimium-remote-server; the default port is 32264.

Python

This guide targets Optimium Runtime 4.4.0.

Breaking change since 4.3.x: InferRequest.infer() now takes inputs only. The runtime allocates the output buffers; read them back with get_output() after wait().

Do not put Optimium Runtime objects in the global scope and do not create circular references to them.

This can lead to memory leaks or undefined behavior, because the C++ and Python memory-management models differ.

1. Install Optimium Runtime

Please click here to install the runtime.

2. Import Optimium Runtime

Import the optimium.runtime package. Unlike C++, initialization happens at import time.

import optimium.runtime as rt

def main():
    # change verbosity to debug level
    rt.logging.set_loglevel(rt.logging.LogLevel.DEBUG)

    # enable logger that writes logs to the console (stderr)
    rt.logging.enable_console_log()

    # enable logger that writes logs to a file
    rt.logging.enable_file_log("output.log")

To defer initialization, set OPTIMIUM_RT_DEFER_INIT before importing, then call rt.initialize().

import os
os.environ["OPTIMIUM_RT_DEFER_INIT"] = "TRUE"

import optimium.runtime as rt
rt.initialize()  # must be called before using any runtime component

Environment variables

Version information

version = rt.get_version()
print(f"Optimium Runtime v{version.major}.{version.minor}.{version.patch}{version.tag}")
print(f"Commit: {version.commit_id}")
print(f"Build:  {version.build_info}")

# rt.__version__ is the same information as a string
print(rt.__version__)

Version attributes: major, minor, patch, tag, commit_id, build_info.

Check available devices

def main():
    local = rt.get_local_info()

    found = any(dev.platform == rt.PlatformKind.NATIVE for dev in local.devices)
    if not found:
        print("cannot find needed device")

HostInfo properties: id, name, architecture (DeviceKind), os (OSKind: LINUX, ANDROID, WINDOWS, MACOS, IOS), devices.

DeviceID properties:

• platform — PlatformKind: NATIVE, XNNPACK, CUDA, VULKAN, OPENCL, SNPE, QNN, SYNAPTICS, HEXAGON
• device_kind — DeviceKind: UNSPECIFIED, NEUTRAL, X86, X64, ARM, ARM64, RISCV32, RISCV64, ANY_CPU, NVIDIA, AMDGPU, INTEL_GPU, MALI, ADRENO, ANY_GPU, HEXAGON, MYRIAD, ARA1, GOOGLE_TPU, CV22, SYNAP_NPU, ANY_NPU
• index, host_id, host_info
• capability — X86Capability, ARMCapability, SPIRVCapability, CUDACapability, RISCVCapability, HexagonCapability, VoidCapability

The Python enums now mirror the C++ ones completely. Mind the underscored spellings — ANY_CPU, INTEL_GPU, ANY_GPU, ANY_NPU, GOOGLE_TPU, SYNAP_NPU — which differ from the C++ AnyCPU / IntelGPU / AnyGPU / AnyNPU / GoogleTPU / SynapNPU.

3. Load a model

def main():
    # auto-detected devices, Auto scheduler
    model = rt.load_model("path/to/model")

    # manual configuration
    model = rt.load_model(
        "path/to/model",
        devices=None,                  # target devices; None = automatic
        strict=False,                  # fail instead of falling back to localhost
        memory_optimization=True,      # share buffers between non-overlapping tensors
        disable_denormals=True,        # flush denormals to zero (FTZ/DAZ)
        intermediate_save_path=None,   # debug: dump intermediates here
        passphrase=None,               # for encrypted models
        threads=4,                     # worker threads for this model
        cores=[0, 1, 2, 3],            # CPU cores to pin workers to
        scheduler_type=rt.SchedulerType.AUTO,
        spin_iterations=1000,          # worker spin budget (see below)
        spin_yield_iterations=0,       # sched_yield tail before parking
        enable_work_stealing=True,     # iteration-level work stealing (default on)
        resizable_capacities={},       # {tensor_name: shape} capacity planning
    )

Unlike other inference engines such as TensorFlow Lite, an Optimium model may be a folder. Pass the path to the folder, and always copy the model together with its folder.

Scheduler types

Thread-pool tuning

• spin_iterations — 0 parks workers immediately (lowest idle CPU, ~ms cold wake per dispatch); N > 0 (default: 100000 on Android, 1000 elsewhere) spins a bounded budget then parks; -1 spins while armed for the lowest dispatch latency, but is unbounded and can throttle big cores at high thread counts on mobile SoCs.
• spin_yield_iterations — sched_yield calls after the bounded budget, before parking (default 10 on Android, 0 elsewhere). Only applies when spin_iterations > 0.
• enable_work_stealing — on by default. Workers that finish their chunk of a kernel fan-out take unclaimed items from busy workers' tails. Measured win-or-neutral on a Snapdragon 778G: ~2.1–2.5× on big.LITTLE, ~1.06–1.21× on homogeneous cores, no regression on small ops. Costs one atomic claim per item. Set False for a deterministic static split.

4. Listing model information

def main():
    print(f"model name: {model.name}")

    print("input tensors")
    for name in model.input_names:
        print(model.get_tensor(name))

    print("output tensors")
    for name in model.output_names:
        print(model.get_tensor(name))

    # access by index
    first_input = model.get_input_tensor(0)
    first_output = model.get_output_tensor(0)

    # every tensor in the model
    for info in model.tensors:
        print(info)

    # list operations
    for op in model.operations:
        print(f"op: {op.name} on device: {op.device}")

    print(f"dynamic: {model.is_dynamic}")

TensorInfo properties: name, shape, type, alignment, padding, size (bytes), scheme, opt_out, constant, resizable.

def main():
    info = model.get_tensor("input_0")

    print(f"name:      {info.name}")
    print(f"shape:     {info.shape}")
    print(f"alignment: {info.alignment}")
    print(f"type:      {info.type}")
    print(f"padding:   {info.padding}")
    print(f"bytes:     {info.size}")
    print(f"opt out:   {info.opt_out}")
    print(f"constant:  {info.constant}")
    print(f"resizable: {info.resizable}")

    if info.scheme:
        if info.scheme.per_channel:
            print(f"per-channel on axis: {info.scheme.axis}")
            for i in range(len(info.scheme)):
                p = info.scheme[i]
                print(f"  channel {i}: scale={p.scale}, zero_point={p.zero_point}")
        else:
            p = info.scheme.get_param()    # index defaults to 0
            print(f"per-tensor: scale={p.scale}, zero_point={p.zero_point}")

Model attributes

New in 4.4.0. Attributes on the model's top-level (main) subgraph:

if model.has_attribute("model_type"):
    print(model.get_attribute("model_type"))

for key in model.attribute_keys:
    print(key, "=", model.get_attribute(key))

Values are converted to native Python types (int, float, bool, str, list, dict). get_attribute() returns None for a missing key, and raises TypeError for the quantized scalar attribute types, which have no Python representation.

Dynamic (symbolic) shape models

def main():
    # by-name dict
    output_shape = model.infer_shape("output_0", {
        "input_0": rt.TensorShape(1, 3, 224, 224)
    })

    # by-index list
    output_shape = model.infer_shape("output_0", [
        rt.TensorShape(1, 3, 224, 224)
    ])

TensorShape: rank, dynamic, size, strides, get_stride(dim), is_compatible(other), shape[i], len(shape).

Expr represents a symbolic dimension: Expr(42) (constant), Expr("batch") (symbol), is_const(), is_symbol(), value, symbol, and arithmetic +, -, *, /, %, min(), max().

5. Creating a request

request = model.create_request()

Multiple requests can exist for the same model and run without interfering with each other.

6. Preparing inputs

import numpy as np

def main():
    # uninitialized float32 tensor shaped 32x32
    f32_tensor = rt.tensor(shape=(32, 32), dtype=rt.ElementType.F32)

    # numpy-style alias for dtype
    f32_tensor = rt.tensor(shape=(32, 32), dtype=rt.float32)

    # int16 tensor filled with 123
    i16_tensor = rt.tensor(123, shape=(8,), dtype=rt.int16)

    # from a nested list (default dtype: float32)
    tensor_from_list = rt.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

    # from numpy (copies by default)
    arr = np.random.random((32, 32)).astype(np.float16)
    tensor_from_np = rt.tensor(arr)

    # zero-copy from numpy (shares memory; keep `arr` alive)
    tensor_zero_copy = rt.tensor(arr, copy=False)

Creation modes:

• rt.tensor(shape=(3, 4), dtype=rt.float32) — uninitialized
• rt.tensor(0.0, shape=(3, 4), dtype=rt.float32) — filled with a scalar
• rt.tensor([[1, 2], [3, 4]]) — from a nested list
• rt.tensor(numpy_array) — from numpy (copy)
• rt.tensor(numpy_array, copy=False) — zero-copy from numpy

dtype aliases:

Convert between ElementType and numpy dtypes:

np_dtype = rt.ElementType.F32.to_dtype()

# from_dtype takes a numpy *dtype*, not a scalar type
elem_type = rt.ElementType.from_dtype(np.dtype(np.float32))

rt.Tensor does not expose element access directly — go through numpy.

to_numpy() returns a copy. Writing into the returned array does not change the tensor. That makes it a safe snapshot of an output, but it is not a way to fill an input.

tensor = rt.tensor(shape=(32, 32), dtype=rt.float32)

snapshot = tensor.to_numpy()   # a copy — safe to keep across the next infer()

tensor.fill(0.0)               # fill with a scalar
print(tensor.shape, tensor.type)

To write data in, build the numpy array first and either wrap it zero-copy or hand it straight to infer():

arr = np.zeros((32, 32), dtype=np.float32)
arr[0, 0] = 1.0

tensor = rt.tensor(arr, copy=False)   # tensor shares arr's memory
# ... or skip the tensor entirely:
request.infer([arr])

rt.Tensor properties: shape, type. Methods: fill(value), to_numpy().

Tensor print configuration

rt.config.set_print_threshold(10)  # max elements per dimension (default 8, -1 = unlimited)
rt.config.set_print_precision(4)   # decimal precision for floats (default 6, range 0-15)

7. Running an inference

request.infer() starts the inference and returns immediately; request.wait() blocks until it finishes. Only inputs are passed — read outputs back with get_output().

Both a sequence (positional, matching the model's input order) and a dict (by name) are accepted, and both rt.Tensor and numpy.ndarray work as elements.

def main():
    # by index
    inputs = []
    for name in model.input_names:
        info = model.get_tensor(name)
        inputs.append(rt.tensor(shape=info.shape, dtype=info.type))

    request.infer(inputs)
    request.wait()

    out0 = request.get_output(0)
    out_named = request.get_output("output_0")
    print(out0.to_numpy())

def main():
    # by name
    inputs = {}
    for name in model.input_names:
        info = model.get_tensor(name)
        inputs[name] = rt.tensor(shape=info.shape, dtype=info.type)

    request.infer(inputs)
    request.wait()

def main():
    # numpy arrays work directly
    inputs = [
        np.random.random((32, 32)).astype(np.float32),
        np.random.random((32, 32)).astype(np.float32),
    ]

    request.infer(inputs)
    request.wait()

    result = request.get_output(0).to_numpy()

You do not need to create input tensors for every inference — reuse them between requests and models as long as they are not used simultaneously. (You cannot feed request A's output into request B while A is still running; it is fine once A has finished.)

request.wait() takes an optional timeout as an integer number of microseconds. It returns False when the inference finished before the timeout, and True when the timeout was reached first. With no argument (or 0) it waits indefinitely.

Always call request.wait() — it is where errors raised during inference surface. Starting an inference on a request already in a fault state is undefined behavior.

def main():
    # wait until finished
    request.infer(inputs)
    request.wait()

    # wait 500000 microseconds (500 ms)
    request.infer(inputs)
    if request.wait(500000):
        print("inference not finished after 500ms")
    else:
        print("inference finished within 500ms")

    print(f"current state of request: {request.status}")

request.status is InferStatus.READY, InferStatus.RUNNING or InferStatus.FAULT.

The GIL is released during infer(), profile() and wait(), so other Python threads keep running.

Callbacks

def on_complete(status, error):
    if status == rt.InferStatus.FAULT:
        print(f"Inference error: {error}")

request.set_callback(on_complete)
request.infer(inputs)
request.wait()

Cancellation

request.infer(inputs)
# ... later:
request.cancel()

Cancellation is cooperative: the runtime checks the flag between operations, and status becomes FAULT afterwards.

8. Resizable tensors (runtime shape changes)

New in 4.4.0. A tensor marked resizable in the model metadata has a concrete base shape but can be resized at run time — typically a shared KV-cache tracking the active sequence length. This is not a symbolic dynamic shape.

model = rt.load_model(
    "path/to/model",
    resizable_capacities={"cache": (2048,)},   # plan the slot once at capacity
)
request = model.create_request()

# Within capacity: metadata-only. No reallocation, content stays in place.
request.set_tensor_shape("cache", (1536,))
request.infer(inputs)
request.wait()

# Beyond a DECLARED capacity: hard cap -> ValueError
# request.set_tensor_shape("cache", (4096,))

resizable_capacities is validated at load: the tensor must exist and be resizable, the shape must be concrete with the same rank as the base shape, and no smaller in element count. Symbolic-dynamic models reject it with ModelError.

Check whether a tensor is resizable at all with model.get_tensor(name).resizable.

Applying eagerly and reclaiming memory

apply_shape_changes() applies pending resizes immediately instead of waiting for the next infer(). Within capacity it is a metadata-only update; on an overflow it re-plans and reallocates, preserving cached content. It raises InvalidStateError if an inference is running.

New in 4.4.0: shrink_to_fit=True additionally gives memory back. Capacity otherwise only ever grows, so a grow-then-shrink KV-cache keeps its high-water footprint for the life of the request.

# Long prompt: the cache overflows its base capacity and the pool grows.
request.set_tensor_shape("cache", (8192,))
request.apply_shape_changes()

# ... decode the long sequence ...

# Short conversation next. Drop back down AND release the memory.
request.set_tensor_shape("cache", (512,))
request.apply_shape_changes(shrink_to_fit=True)

Every Resizable capacity is re-frozen down to whichever of its current logical shape and its floor holds more elements — the floor being the base shape, or a declared resizable_capacities entry, which is never lowered. Cached content is preserved as for any resize; only elements past the reclaimed slot are dropped. It is a no-op on symbolic-dynamic models.

Growth is implicit, release is explicit — the same contract as list/std::vector capacity.

Custom migration callbacks

By default the runtime performs an index-preserving copy (grown region zeroed) when a cached tensor is remapped across a resize. Override it per tensor:

def migrate(old, new):
    o = old.to_numpy()
    n = new.to_numpy()
    count = min(o.shape[0], n.shape[0])
    n[:count] = o[:count]      # e.g. a ring-buffer shift or custom fill

request.set_resize_migration("cache", migrate)

# clear the callback to restore the default migration
request.set_resize_migration("cache", None)

old is a read-only view laid out for the pre-resize shape; new is writable and zero-initialized for the new shape. The callback fires wherever the runtime remaps the tensor — a within-capacity in-place restride, a reallocating overflow, or a shrink_to_fit reclaim — and as of 4.4.0 that includes purely device-resident tensors, which receive a host read-back view. A per-step growing cache restrides every step, so keep it cheap. An unknown or non-cached tensor raises ValueError.

InferRequest methods for resizing: set_tensor_shape(name, shape), apply_shape_changes(shrink_to_fit=False), set_resize_migration(name, callback).

The C++ releaseStaleShapeMemory() is not bound in Python, and shrink_to_fit=True only releases the retained pools when the reclaim actually reallocates. Pair it with a real shrink (as in the example above) rather than calling it with nothing pending — a bare apply_shape_changes(shrink_to_fit=True) right after a grow returns early and leaves the previous pools held, with no Python-side way to free them.

9. Model groups

New in 4.4.0. Models that never run concurrently can share one thread pool and one scheduler instead of owning one pool each, removing the park/wake penalty they otherwise pay when alternating.

import optimium.runtime as rt

group = rt.create_model_group(
    cores=[4, 5, 6, 7],
    spin_iterations=-1,            # stay warm across the alternation
    spin_yield_iterations=0,
    disable_denormals=True,
    enable_work_stealing=True,     # default
)

encoder = group.add("path/to/encoder", threads=4)
decoder = group.add("path/to/decoder", threads=4)

# run() is BLOCKING and returns the outputs in output_names order.
enc_out = encoder.run(encoder_inputs)

for step in range(steps):
    logits, = decoder.run(decoder_inputs)

print(group.size, group.sealed)

Key points:

• GroupMember is not a Model: there is no create_request(). The group owns execution and exposes it through the single blocking run(), so members are visibly one-at-a-time and never race on the shared pool.
• The member's internal request is reused across calls, so cached/preserved tensors persist between runs.
• Membership seals on the first run(). A later add() raises InvalidStateError.
• The shared pool auto-sizes to the largest member's threads.
• Grouped members always run on the shared Exclusive scheduler.
• Shared-pool knobs (cores, spin_iterations, spin_yield_iterations, disable_denormals, enable_work_stealing) belong to create_model_group(). Per-member options (devices, strict, memory_optimization, intermediate_save_path, passphrase, threads) belong to group.add().

GroupMember also exposes name, input_names, output_names, get_tensor(name), get_input_tensor(i), get_output_tensor(i).

Runtime resize and migration are exposed on InferRequest (set_tensor_shape / set_resize_migration), not on GroupMember.

10. Profiling

request.profile() is blocking (unlike infer()).

def main():
    request.profile(
        inputs,
        repeat=100,
        warmup=10,
        warmup_time=1000000,   # microseconds
        stop_threshold=0,      # microseconds; 0 disables early stop
        check_period=0,
        event_buffer_size=0,   # event capacity; 0 = default (4,194,304 events)
    )

    # raw events
    for event in request.get_profile_events():
        print(f"Event: {event.kind} at {event.timestamp}")

    # computed durations (nanoseconds, grouped by model/layer/copy)
    durations = request.get_profile_durations()
    print(f"Model: {durations.model_name}")
    print(f"Model durations (ns): {durations.model_durations}")
    for name, layer_dur in durations.layer_durations.items():
        print(f"  Layer '{name}': {layer_dur}")
    for name, copy_dur in durations.copy_durations.items():
        print(f"  Copy '{name}': {copy_dur}")

inputs may be a sequence or a dict, with rt.Tensor or numpy arrays. warmup_time and stop_threshold are integer microseconds, and none of these parameters are keyword-only — passing them positionally works too.

ProfileEvent

Event kinds (ProfileEventKind): MODEL_EXECUTE_BEGIN/END, LAYER_EXECUTE_BEGIN/END, LAUNCH_BEGIN/END, DEVICE_EXECUTE_BEGIN/END, COPY_BEGIN/END, QUEUE_BEGIN/END, WAIT_BEGIN/END.

Properties:

• kind — ProfileEventKind
• timestamp — raw std::chrono::high_resolution_clock ticks since epoch (int; nanoseconds on Linux/macOS). Use get_profile_durations() if you want normalized values.
• operation — operation index (LAYER_EXECUTE and WAIT events)
• thread_id — thread ID (LAYER_EXECUTE events)
• tensor_id — tensor ID (COPY events)
• source_device / dest_device — DeviceID (COPY events)

ProfileDurations

request.get_profile_durations() pairs begin/end events and resolves indices to names:

• model_name — model name (str)
• model_durations — model-level durations in nanoseconds (list[int])
• layer_durations — dict[str, list[int]] keyed by operation name
• copy_durations — dict[str, list[int]] keyed by tensor name

BasicProfiler / BatchProfiler

The optimium package (the compiler SDK, installed alongside the runtime) ships convenience profilers built on these events.

from optimium.runtime.profile import BasicProfiler

profiler = BasicProfiler(
    "path/to/model",
    memory_optimization=True,
    disable_denormals=False,
    threads=1,
    cores=None,
    intermediate_save_path="",
)

profiler.profile(repeat=100, warmup=10, warmup_time=1000000)

print(f"Model: min={profiler.model_stat.min}, "
      f"max={profiler.model_stat.max}, "
      f"mean={profiler.model_stat.mean:.2f}")

for stat in profiler.layer_stats:
    print(f"  Layer {stat.name}: mean={stat.mean:.2f}")

profiler.dump_json("profile_results.json")

BasicProfiler can also profile against a remote server:

profiler = BasicProfiler(
    "path/to/model",
    remote_address="192.168.1.100",
    remote_port=32264,
)

For multiple input batches, use BatchProfiler:

from optimium.runtime.profile import BatchProfiler

profiler = BatchProfiler("path/to/model")
profiler.profile(100, inputs=[batch1_inputs, batch2_inputs])

for i, stat in enumerate(profiler.batch_stat):
    print(f"Batch {i}: mean={stat.model_stat.mean:.2f}")
print(f"Total: mean={profiler.total_stat.model_stat.mean:.2f}")

Intermediate tensor access

model = rt.load_model("path/to/model", memory_optimization=False)
request = model.create_request()
request.infer(inputs)
request.wait()

intermediate = request.get_tensor("intermediate_tensor_name")
print(intermediate.to_numpy())

11. Remote inference

import optimium.runtime as rt

session = rt.connect(
    "192.168.1.100",
    port=32264,
    enable_secure_connection=True,
    enable_compression=True,
)

print(f"Remote host: {session.host_info}")

model = session.load_model(
    "path/on/remote/model",
    threads=4,
    cores=[0, 1, 2, 3],
    scheduler_type=rt.SchedulerType.AUTO,
)

print(f"Model: {model.name}")
for info in model.input_tensor_infos:
    print(f"  Input: {info.name} {info.shape} {info.type}")
for info in model.output_tensor_infos:
    print(f"  Output: {info.name} {info.shape} {info.type}")

inputs = {"input_0": rt.tensor(shape=(1, 3, 224, 224), dtype=rt.float32)}

# infer() and profile() are asynchronous — call wait() afterwards.
model.infer(inputs)
model.wait()

result = model.get_output(0)
# or by name
result = model.get_tensor("output_0")

The server-side request is created once and reused, so cached (KV-cache) tensors keep their state between successive infer() calls — which makes remote auto-regressive decoding work.

RemoteModel properties: name, input_tensor_infos, output_tensor_infos, tensor_infos, operations, is_dynamic.

RemoteModel methods:

• infer(inputs) — one asynchronous run (dict or sequence, Tensor or numpy)
• profile(inputs, repeat=1, warmup=0, warmup_time=0, stop_threshold=0, check_period=0, event_buffer_size=0) — asynchronous profiling
• wait(timeout=0) — wait for completion; integer microseconds, 0 waits indefinitely
• cancel() — cancel the in-progress operation
• get_output(index) / get_tensor(name) — fetch results
• set_tensor(name, value) — write any tensor by name; for a non-input (intermediate or cached) tensor the value lands in the persistent request buffer, so you can seed or update a KV-cache between runs
• get_profile_events() / get_profile_durations()
• is_saved_tensor(tensor_name), infer_shape(output_name, input_shapes)
• get_attribute(key), has_attribute(key), attribute_keys

session.load_model() accepts: devices, strict, memory_optimization, disable_denormals, intermediate_save_path, passphrase, threads, cores, scheduler_type, spin_iterations, spin_yield_iterations, enable_work_stealing.

Remote model groups

New in 4.4.0. The remote analogue of ModelGroup — members share one server-side pool and scheduler.

group = session.create_model_group(
    cores=[4, 5, 6, 7],
    spin_iterations=-1,
    disable_denormals=True,
    enable_work_stealing=True,     # default
)

# path is a LOCAL container that gets uploaded to the server
encoder = group.add("path/to/encoder", threads=4)
decoder = group.add("path/to/decoder", threads=4)

outputs = encoder.run(encoder_inputs)   # blocking round-trip

RemoteGroupMember exposes run(inputs), name, input_tensor_infos, output_tensor_infos.

12. Error handling

Optimium Runtime maps C++ exceptions to Python exceptions. rt.RuntimeException is the base class and inherits from RuntimeError.

Runtime-specific exceptions:

• rt.InvalidStateError
• rt.InvalidOperationError
• rt.ShapeError
• rt.ExtensionError
• rt.DeviceError
• rt.ModelError
• rt.RequestError
• rt.InferError
• rt.OutOfResourceError
• rt.ContainerError
• rt.RemoteError

These C++ errors are raised as Python builtins instead:

try:
    model = rt.load_model("nonexistent/path")
except rt.ContainerError as e:
    print(f"Container error: {e}")
except rt.ModelError as e:
    print(f"Model error: {e}")

13. Loading extensions

The runtime auto-loads the known extensions it finds next to the runtime library (xnnpack, cuda, vulkan, opencl, snpe, qnn) at initialization. Load anything else — currently hexagon and synap — explicitly:

rt.load_extension("path/to/liboptimium-runtime-hexagon.so")

Raises rt.ExtensionError if the extension cannot be loaded, ValueError if the path is not a file.