Optimium Runtime Guide

Optimium Runtime is the C++ inference engine that executes models compiled by Optimium. Optimium does the ahead-of-time work — operator selection, kernel tuning, memory planning — and emits a model container; the runtime loads that container and runs it, on the same machine or on a remote one.

  your model (PyTorch / TFLite / ONNX)
        │
        ▼  Optimium (compiler + auto-tuning)
  model container
        │
        ▼  Optimium Runtime  ── C++ / Python / Java
  inference

This section documents how to integrate the runtime into an application: loading a model, preparing tensors, running and profiling inference, and tuning latency. It does not cover compiling a model — see the Optimium compiler documentation for that.

Pick your version

The pages below are complete guides, one per runtime release line. Read the one that matches the runtime you have installed — the API differs between lines in ways that will not compile if you mix them up.

The 4.x numbering is a renumbering of the same product, not a rewrite: 0.4.1 (April 2026) was followed by 4.1.0 a week later. The API did change across that boundary, though — see Moving from 0.4 to 4.x.

Not sure which runtime you have?

import optimium.runtime as rt
print(rt.get_version())

#include <Optimium/Runtime/Version.h>
auto V = optimium::runtime::getVersion();   // V.Major, V.Minor, V.Patch, V.Tag

The model decides your runtime version

Version compatibility is enforced at load time, and it is the model that sets the constraint — not the other way round. loadModel() rejects the model with a ModelError when:

1. Metadata version mismatch. The container's schema version must match the runtime's exactly (current runtime: metadata version 6). A newer or older schema is refused outright — this is the usual reason a model from one release line will not load on another.
2. Model requires a newer runtime. Each container records the runtime version it was built for. A runtime older than that value is refused. The reverse is fine: a newer runtime runs older models, as long as rule 1 holds.
3. Model expired. Containers may carry an optional expiry date.

Practical consequence: recompile the model when you move to a new runtime line. Upgrading the runtime alone is not enough.

30-second quick start

Both snippets target 4.4. On 0.4 the infer() call takes outputs as a second argument — see that guide.

Python

import numpy as np
import optimium.runtime as rt

model = rt.load_model("path/to/model")     # a folder, not a single file
request = model.create_request()

request.infer([np.random.rand(1, 3, 224, 224).astype(np.float32)])
request.wait()

print(request.get_output(0).to_numpy())

C++

#include <Optimium/Runtime.h>

namespace rt = optimium::runtime;

static rt::AutoInit Init;   // must outlive every model and request

int main() {
    rt::Model Model = rt::loadModel("path/to/model");
    rt::InferRequest Request = Model.createRequest();

    std::vector<rt::Tensor> Inputs;
    for (rt::StringRef Name : Model.getInputNames()) {
        const rt::TensorInfo &Info = Model.getTensorInfo(Name);
        Inputs.push_back(rt::tensor(Info.Type, Info.Shape));
    }

    Request.infer(rt::make_array(Inputs));
    Request.wait();

    rt::Tensor Output = Request.getOutput(0);
}

An Optimium model is normally a directory, not a single file. Pass the path to the directory, and always copy the model together with its folder.