Guides

Migration Guide: Optimium Runtime 0.3.x → 0.4.x

In this page, describes how to migrate code written for prior Optimium Runtime 0.4.2

Changes on Context

Starting from Optimium Runtime 0.4.0, the Context class has been removed. As a result, several APIs have changed.

C++

The runtime must be explicitly initialized before use. You can either call rt::initialize() / rt::finalize() directly, or use rt::AutoInit which calls them automatically via RAII.

Warning: rt::AutoInit calls rt::finalize() in its destructor, which shuts down the runtime entirely. If AutoInit is declared as a local variable in a narrow scope, the runtime will be finalized when that variable goes out of scope — causing any in-progress inference to fail. For this reason, declare rt::AutoInit as a static global variable, or at the very top of main().

// old:
/*
rt::Context Context = rt::Context::create(...);
 */

// new (option 1): explicit initialize / finalize
int main(...) {
    rt::initialize();

    // ... load models, run inference ...

    rt::finalize();
}

// new (option 2): static AutoInit — runtime lives for the entire process lifetime.
static rt::AutoInit Init;

int main(...) {
    // ... load models, run inference ...
    // rt::finalize() is called automatically when the process exits.
}

Context::loadModel() has been replaced by the free function loadModel(). The Devices argument has been moved into the ModelLoadOptions struct.

int main() {
    // old:
    /*
    std::vector<Device> Devices{ ... };
    rt::Model Model = Context.loadModel("path/to/model", Devices).value();
    */

    // new:
    rt::ModelLoadOptions Options;
    std::vector<DeviceID> Devices{ ... };

    Options.Devices = Devices;

    rt::Model Model = rt::loadModel("path/to/model", Options);
}

Context::getAvailableDevices() has been replaced by the free function getLocalInfo(), which provides richer information about the local host.

int main(...) {
    // old:
    /*
    std::vector<Device> Devices = Context.getAvailableDevices().value();
     */

    // new:
    rt::HostInfo LocalInfo = rt::getLocalInfo();
    rt::ArrayRef<DeviceID> Devices = LocalInfo.Devices;
}

Context::loadExtension() has been replaced by the free function loadExtension().

int main(...) {
    // old:
    /*
    Context.loadExtension("path/to/extension.so");
    */

    // new:
    rt::loadExtension("path/to/extension.so");
}

Context::connectRemote() has been replaced by the free function connect(). RemoteContext has also been removed; the function now returns a RemoteSession object.

This API requires the Remote component header (Optimium/Remote.h).

#include <Optimium/Remote.h>

int main(...) {
    // old:
    /*
    rt::RemoteContext Remote = Context.connectRemote("addr").value();
     */

    // new:
    rt::RemoteSession Session = rt::connect("addr");
}
Python

Context.load_model() has been replaced by the free function load_model().

def main():
    # old:
    # context = rt.Context(...)
    # devices = [...]
    # model = context.load_model("path/to/model", devices)

    # new:
    # 'devices' is a keyword-only argument now.
    devices = [...]
    model = rt.load_model("path/to/model", devices=devices)

Context.available_devices has been replaced by the get_local_info() function, which provides richer information about the local host.

def main():
    # old:
    # devices = context.available_devices

    # new:
    local_info = rt.get_local_info()
    devices = local_info.devices

Context.load_extension() has been replaced by the free function load_extension().

def main():
    # old:
    # context.load_extension("path/to/extension.so")

    # new:
    rt.load_extension("path/to/extension.so")

Context.connect_remote() has been replaced by the free function connect(). RemoteContext has also been removed; the function now returns a RemoteSession object.

def main():
    # old:
    # remote = context.connect_remote("addr")

    # new:
    session = rt.connect("addr")

Changes on logging API

Logging API changes differ by language. Please choose your language below.

C++

Starting from Optimium Runtime 0.4.0, the LogSettings class has been replaced by the logging namespace.

int main(...) {
    // old:
    /*
    rt::LogSettings::setLogLevel(rt::LogLevel::Debug);
     */

    // new:
    rt::logging::setLogLevel(rt::LogLevel::Debug);

    // old:
    /*
    rt::LogSettings::addWriter(rt::WriterOption::ConsoleWriter());
    rt::LogSettings::addWriter(rt::WriterOption::FileWriter("path/to/log"));
    rt::LogSettings::addWriter(rt::WriterOption::AndroidWriter());
     */

    // new:
    rt::logging::addLogWriter(std::make_unique<rt::logging::ConsoleWriter>());
    rt::logging::addLogWriter(std::make_unique<rt::logging::FileWriter>("path/to/log"));

    // AndroidLogWriter is available only on Android.
    // Previously this was a no-op on other platforms.
    rt::logging::addLogWriter(std::make_unique<rt::logging::AndroidLogWriter>());
}

You can also add a custom LogWriter to process logs in your own way.

#include <Optimium/Runtime/Logging/LogWriter.h>

namespace rt = optimium::runtime;

class CustomWriter final : public rt::logging::LogWriter {

public:
    void onWriteMessage(rt::LogLevel Level,
                        std::chrono::system_clock::time_point Time,
                        rt::StringRef Tag,
                        rt::StringRef Message) override {
        // do something
    }
};

int main(...) {
    // ...

    // add custom writer
    rt::logging::addLogWriter(std::make_unique<CustomWriter>());
}
Python

Starting from Optimium Runtime 0.4.0, the logging submodule has been introduced to configure logger settings. This module is automatically imported when optimium.runtime is imported.

import optimium.runtime as rt

def main():
    # old:
    # ctx = rt.Context(verbosity=rt.LogLevel.Debug,
    #                  log_path="path/to/log")

    # new:
    # - Context(verbosity=...) has been replaced by
    #   logging.set_loglevel().
    # - rt.LogLevel enum values are now UPPER_SNAKE_CASE
    #   to follow Python conventions.
    rt.logging.set_loglevel(rt.logging.LogLevel.DEBUG)

    # - Context(log_path=...) has been replaced by
    #   logging.enable_*() functions.
    # - Starting from Optimium Runtime 0.4.0,
    #   logging is disabled by default.
    #   To enable logging, use the functions below:

    # enable console logging
    rt.logging.enable_console_log()
    # enable file logging
    rt.logging.enable_file_log("path/to/log")

Changes on Model

Starting from Optimium Runtime 0.4.0, some APIs have changed.

This part is subject to change.

Several member functions (properties and methods) have been replaced.

C++
  • getInputTensorsInfo() has been replaced by getInputNames()
  • getOutputTensorsInfo() has been replaced by getOutputNames()
  • getInputTensorInfo() and getOutputTensorInfo() have been removed. Use getTensorInfo() with the tensor name instead.
int main(..) {
  // old:
  /*
  std::vector<rt::TensorInfo> Infos = Model.getInputTensorsInfo().value();
   */

  // new:
  rt::ArrayRef<rt::StringRef> Names = Model.getInputNames();
  std::vector<rt::TensorInfo> Infos;
  for (rt::StringRef Name : Names)
      Infos.push_back(Model.getTensorInfo(Name));
}
Python
  • input_tensors_info has been replaced by input_names
  • output_tensors_info has been replaced by output_names
  • get_input_tensor_info() and get_output_tensor_info() have been removed. Use get_tensor() with the tensor name instead.
def main():
    # old:
    # infos = model.input_tensors_info

    # new:
    infos = [model.get_tensor(name) for name in model.input_names]

Changes on InferRequest

Starting from Optimium Runtime 0.4.0, users must prepare input and output tensors themselves.

Prior to 0.4.0, input and output tensors were provided by the runtime. However, this required copying data from outputs to inputs when models were executed sequentially. To eliminate this inefficiency, Optimium Runtime now requires users to provide their own input and output tensors.

C++
int main(...) {

    // old:
    /*
    rt::InferRequest Request = Model.createRequest().value();
    rt::Tensor Input0 = Request.getInputTensor("input_0").value();
    Input0.copyFrom(...);

    Request.infer();
    Request.wait();

    rt::Tensor Output = Request.getOutputTensor("output").value();
    Output.copyTo(...);
     */

    // new:
    rt::InferRequest Request = Model.createRequest();
    rt::TypedTensor<float> Input0 = rt::tensor<float>({/* shape of input */});
    rt::TypedTensor<float> Output = rt::tensor<float>({/* shape of output */});

    // using array - tensors must be in the same order
    // as the model's inputs or outputs.
    std::vector<rt::Tensor> Inputs;
    Inputs.push_back(Input0);

    std::vector<rt::Tensor> Outputs;
    Outputs.push_back(Output);

    Request.infer(rt::make_array(Inputs), rt::make_array(Outputs));
    Request.wait();

    // using map - keys must match the names of
    // the model's inputs or outputs.
    std::map<std::string, rt::Tensor> InputMap;
    std::map<std::string, rt::Tensor> OutputMap;

    InputMap["input_0"] = Input0;
    OutputMap["output"] = Output;

    Request.infer(InputMap, OutputMap);
    Request.wait();
}
Python
def main(...):
    # old:
    # request = model.create_request()
    # request.set_inputs(...)
    # request.infer()
    # request.wait()
    # outputs = request.get_outputs()

    # new:
    request = model.create_request()

    input_tensor = rt.tensor(shape=(...))  # default dtype is float32
    output_tensor = rt.tensor(shape=(...))

    # using sequence - tensors must be in the same order
    # as the model's inputs or outputs.
    request.infer([input_tensor], [output_tensor])
    request.wait()

    # using dict - keys must match the names of
    # the model's inputs or outputs.
    request.infer({"input_0": input_tensor},
                  {"output": output_tensor})
    request.wait()

    # numpy.ndarray can also be used directly as tensors.
    input_tensor = np.random.random((...)).astype(np.float32)
    output_tensor = np.zeros((...), dtype=np.float32)

    request.infer([input_tensor], [output_tensor])
    request.wait()

Changes on remote API

Starting from Optimium Runtime 0.4.0, the Remote API is provided as a separate component (Optimium/Remote.h in C++, optimium.runtime.connect in Python). The RemoteContext class has been removed and replaced by RemoteSession, which is returned by the connect() function.

Name changes

  • Device has been renamed to DeviceID.
  • Shape has been renamed to TensorShape.

C++ specific changes

  • rt::Result has been removed and exceptions have been introduced. Based on long-term experience, users tended not to check error results. To prevent segfaults caused by unchecked errors, the runtime now throws exceptions instead of returning error codes.

Python specific changes

  • Enum values (e.g. for PlatformKind, DeviceKind, ElementType) have been changed to UPPER_SNAKE_CASE to follow Python conventions.

Kotlin specific changes

  • The Kotlin binding has been replaced by a Java binding to reduce dependencies and avoid version conflicts.

Updated about 1 month ago