The custom device manager is available exclusively to on-premise plan users
Custom Device Manager
By managing devices via device managers, OwLite can benchmark your model on any deployment environment of your choice, even if it differs from the environment where you have trained your model. The device managers are the host machines where one or more devices reside. Once you have added the URL to a device manager, you can connect to the devices that belong to the device manager.
1. Setting up a custom device manager
Custom device manager is available as a Docker image in a private repository. For more straightforward integration, both a Docker compose script and a Kubernetes deployment corresponding to this image are supplied.
git clone https://{authoriation_key}@bitbucket.org/squeezebits/{authorized_bucket}
cd {authorized_bucket}
docker compose up -d
The device manager is a premium feature of OwLite, providing an instance that manages connected OwLite Runners equipped with GPU devices. It oversees the benchmark job queue and relays incoming requests for building TensorRT engines from users to the Runners. The custom device manager enables our users to effectively and safely expand their GPU resources to accommodate the growing demands of internal OwLite users.
Please note that the device manager is not available for free-plan users. Therefore, free-plan users won't be able to utilize the TensorRT engine execution and benchmark report on a deployment environment different from your local environment. However, free users can still enjoy various features offered by OwLite. These include:If you encounter a
- Creating a baseline and generating a compression configuration on the OwLite Web UI
- Applying the generated compression configuration to create a quantized model
- Deploying the quantized ONNX directly as a TensorRT engine on your local environment
- Viewing the log of each experiment on the OwLite Web UI
Device not found
error while usingowl.benchmark
, it is likely due to the absence of the device manager. Note that other features are still accessible without any device manager.
2. Adding a new device manager
You can utilize your dedicated device manager.
owlite device add --name --url
# or equivalently
owlite device add -n -u
3. Listing device managers
You can check the list of connected devices by running
owlite device ls
You'll be able to see the outputs similar to the following:
OwLite [INFO] Available device managers:
NEST : https://nest.owlite.ai
myDevice : http://173.22.1.31:1000
4. Connecting to a device in an existing device manager
owlite device connect --name <device_manager_name>
Enter the index of one of the devices connected to the device manager.
For example:
OwLite [INFO] Available devices:
0: NVIDIA RTX A6000
Enter the index of the device you want to connect to: 0
OwLite [INFO] Connected to the device 'NVIDIA RTX A6000' at 'myDevice' (http://173.22.1.31:1000)
5. Removing an existing device manager
owlite device rm --name <device_manager_name>