Install InferNex Bridge

This guide describes enabling Alauda Build of InferNex Bridge on arm64 clusters through the existing default AmlCluster. Alauda AI Platform manages installation; users do not select an OLM channel or install the operator manually. See the component matrix for the version included in your platform release.

Prerequisites

Before installing Alauda Build of InferNex Bridge, ensure the target cluster has the required platform and inference dependencies.

Required Dependencies

DependencyTypeDescription
Kubernetes clusterPlatformA running cluster with administrator access.
KServeOperatorRequired when using the KServe LLMInferenceService entry point. No specific Alauda AI Platform Inference Management version is prescribed here.
Envoy Gateway and Gateway APIOperator / CRDsRequired when exposing inference services through Gateway API resources.
Gateway API Inference ExtensionCRDsRequired for InferencePool based intelligent routing.
Alauda Build of LeaderWorkerSetOperator (managed by AmlCluster)Required for InferNex Bridge. The controller watches LeaderWorkerSet resources and uses them for multi-node inference workloads. It is normally enabled through the lws component in the default AmlCluster; see Install Alauda AI Platform.
Alauda AI Platform NPU Serving RuntimeOperator (managed by AmlCluster)Recommended for Ascend NPU inference. It provides the vLLM-Ascend image, ClusterServingRuntime, and LLMInferenceServiceConfig resources. Enable servingRuntimeNPUOperator in the default AmlCluster; see NPU Serving Runtime setup.
INFO

InferNex Bridge is installed and managed by Alauda AI Platform through the infernexBridge component in the default AmlCluster; it does not require installing the InferNex main chart. For Ascend inference, enable servingRuntimeNPUOperator as well. The Alauda AI Platform NPU Serving Runtime supplies the vLLM-Ascend image, ClusterServingRuntime, and LLMInferenceServiceConfig; model files and any feature-specific dependencies must still be prepared separately.

CRDs Installed by This Operator

The Alauda Build of InferNex Bridge OLM bundle installs the two product APIs and three CRDs used by its managed elastic orchestration components:

CRDInstalled by this operator
infernexservices.infernex.infernex.ioYes
infernexserviceconfigs.infernex.infernex.ioYes
resourcescalinggroups.autoscaling.openfuyao.comYes
elasticscalers.elasticscaler.ioYes
tidals.tidal.ioYes

The following CRDs are not installed by this OLM bundle. Install them separately according to the requirement shown below:

CRDSource ProjectWhen RequiredHow to Install
leaderworkersets.leaderworkerset.x-k8s.ioAlauda Build of LeaderWorkerSetInferNex Bridge controller and multi-node inference workloadsEnsure LeaderWorkerSet is installed and running. It is normally managed through the lws component in the default AmlCluster; see Install Alauda AI Platform.
rolebasedgroups.workloads.x-k8s.ioRoleBasedGroup (RBG)ResourceScalingGroup targets that explicitly use RoleBasedGroupInstall from the upstream RBG v0.7.0 release manifest.

The rolebasedgroups.workloads.x-k8s.io CRD is provided by the upstream RoleBasedGroup (RBG) project. It is a conditional dependency and is not required for Deployment or LeaderWorkerSet targets. To install the upstream RBG v0.7.0 controller and CRDs, use the versioned release manifest:

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/sgl-project/rbg/v0.7.0/deploy/kubectl/manifests.yaml

kubectl wait deployment/rbgs-controller-manager \
  --namespace rbgs-system \
  --for=condition=Available \
  --timeout=5m

kubectl get crd rolebasedgroups.workloads.x-k8s.io \
  -o jsonpath='{range .spec.versions[?(@.name=="v1alpha2")]}{.name}{" served="}{.served}{" storage="}{.storage}{"\n"}{end}'

The expected result is v1alpha2 served=true storage=true. The manifest uses docker.io/rolebasedgroup/rbgs-controller:v0.7.0-888ade7, which provides an arm64 image. For a disconnected cluster, mirror this image to a registry accessible from the cluster and update the manifest image reference before applying it.

See the upstream RBG releases when selecting a different version.

Enabling the managed Tidal, ElasticScaler, or ResourceScalingGroup components deploys their controller workloads. Scaling policy resources such as Tidal and ElasticScaler must still be created separately by the user or platform workflow.

Runtime Templates and Images

For Ascend inference, use Alauda AI Platform NPU Serving Runtime rather than manually installing the vLLM-Ascend runtime image or creating its ClusterServingRuntime and LLMInferenceServiceConfig. Enable servingRuntimeNPUOperator as described in NPU Serving Runtime setup; the component manages these resources.

InferNex Bridge's own runtime templates and component images may still reference their upstream repositories. In restricted or disconnected environments, verify that the package-generated ImageWhiteList contains the required image references and rewrite rules, and mirror any additional InferNex Bridge images to a registry accessible by the cluster.

Configure ImageWhiteList

The platform creates an operator-managed ImageWhiteList when the InferNex Bridge package is uploaded. Do not create or delete this object manually. Operator-managed ImageWhiteList objects are reconciled by olm-registry and may be recreated after deletion.

Before creating an inference service, verify that the generated object contains the required source image references and source-to-target rewrite rules. Keep unrelated entries that are already present.

The following table shows the expected repository mappings. Each rule preserves the original image tag.

Source repositoryTarget repository under <target-registry>
cr.openfuyao.cn/openfuyao/cache-indexer3rdparty/openfuyao/cache-indexer
cr.openfuyao.cn/openfuyao/proxy-server3rdparty/openfuyao/proxy-server
cr.openfuyao.cn/openfuyao/hermes-router3rdparty/openfuyao/hermes-router
cr.openfuyao.cn/openfuyao/hermes-router-tokenizer3rdparty/openfuyao/hermes-router-tokenizer
cr.openfuyao.cn/openfuyao/hermes-router-prediction3rdparty/openfuyao/hermes-router-prediction
hub.oepkgs.net/openfuyao/redis3rdparty/openfuyao/redis
cr.openfuyao.cn/openfuyao/tidal3rdparty/openfuyao/tidal
cr.openfuyao.cn/openfuyao/resource-scaling-group3rdparty/openfuyao/resource-scaling-group
cr.openfuyao.cn/openfuyao/elastic-scaler3rdparty/openfuyao/elastic-scaler
cr.openfuyao.cn/openfuyao/eagle-eye-hardware-monitor3rdparty/openfuyao/eagle-eye-hardware-monitor
cr.openfuyao.cn/openfuyao/eagle-eye-hardware-diagnosis3rdparty/openfuyao/eagle-eye-hardware-diagnosis
cr.openfuyao.cn/openfuyao/eagle-eye-network-performance-exporter3rdparty/openfuyao/eagle-eye-network-performance-exporter
hub.oepkgs.net/openfuyao/alpine/kubectl3rdparty/openfuyao/alpine/kubectl

The relevant part of the final object should look like this. The complete object contains additional package images and platform-managed entries.

apiVersion: app.alauda.io/v1alpha1
kind: ImageWhiteList
metadata:
  name: infernex-bridge
  namespace: cpaas-system
spec:
  repoList:
    - cr.openfuyao.cn/openfuyao/cache-indexer:26.6.0
    - busybox:1.36.1
    - docker.io/library/busybox:1.36.1
  rewriteRules:
    - regexp: ^cr\.openfuyao\.cn/openfuyao/cache\-indexer:(.*)$
      replacement: <target-registry>/3rdparty/openfuyao/cache-indexer:${1}

For BusyBox, retain both busybox:1.36.1 and docker.io/library/busybox:1.36.1 in repoList, but do not add a custom BusyBox rewrite rule. The platform _default rule handles the short name and Docker Hub normalization.

You can back up and edit the operator-managed object with these commands. If the object has a different name, locate it with kubectl get imagewhitelist -A.

kubectl get imagewhitelist -n cpaas-system infernex-bridge -o yaml \
  > imagewhitelist-infernex-bridge-backup.yaml

kubectl edit imagewhitelist -n cpaas-system infernex-bridge

The workload specification may continue to show the community source image. The rewrite is successful only when the actual Pod imageID points to the target registry. For example, an InferNex Bridge component should resolve as follows:

cache-indexer=<target-registry>/3rdparty/openfuyao/cache-indexer@sha256:...

Verify that the InferNex Bridge component images resolve to the intended registry by inspecting Pod image IDs:

kubectl get pods -n <inference-service-namespace> \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .status.initContainerStatuses[*]}  init:{.name}{"="}{.imageID}{"\n"}{end}{range .status.containerStatuses[*]}  {.name}{"="}{.imageID}{"\n"}{end}{end}'

Prepare the Model

Model artifacts are not included in the InferNex Bridge operator package. Use the Alauda AI Platform model catalog to prepare a ModelCar when OCI model storage is selected. See Model Storage.

INFO

The Alauda OLM bundle registers the InferNex Bridge admission webhook for the KServe LLMInferenceService API versions used by the release examples, including serving.kserve.io/v1alpha2. The webhook is used for admission-time compatibility patches when infernex.io/runtime: "true" is set on a KServe LLMInferenceService; it does not create or reconcile the LLMInferenceService resource itself.

Optional Dependencies

DependencyRequired ForDescription
NATSEagle-EyeRequired when enabling Eagle-Eye hardware monitoring, diagnosis, or network performance collection. The Service must be reachable as nats.nats:4222.
kube-prometheus-stackEagle-EyeRequired when platform monitoring must scrape and display Eagle-Eye metrics.

Optional EagleEye Observability

EagleEye is disabled by default in the Alauda OLM templates. Disabling EagleEye does not affect the inference engine, aggregate or prefill-decode deployment modes, Mooncake, cache-indexer, or PD-Orchestrator. It disables only the following optional observability workloads:

ComponentFunction
Hardware MonitorRuns on accelerator nodes and collects hardware status information.
Hardware DiagnosisProvides hardware diagnosis processing and diagnostic logs.
Network Performance ExporterCollects network performance data and exposes metrics on port 8222.

Before enabling EagleEye, verify that NATS has a ready Service and endpoint:

kubectl get service -n nats nats
kubectl get endpointslice -n nats -l kubernetes.io/service-name=nats

The EagleEye init containers connect to nats.nats:4222. If this endpoint is unavailable, the enabled EagleEye Pods remain in the init phase.

Enable EagleEye explicitly in an InferNexService or a referenced InferNexServiceConfig:

spec:
  components:
    eagleEye:
      hardwareMonitor:
        enabled: true
      hardwareDiagnosis:
        enabled: true
      networkPerformanceExporter:
        enabled: true

After applying the resource, verify the optional workloads in the inference service namespace:

kubectl get deployment,daemonset,service -n <inference-service-namespace> \
  -l infernex.io/owner=<infernex-service-name>

Expected workloads include the EagleEye hardware diagnosis Deployment, hardware monitor DaemonSet, and network performance exporter DaemonSet. If EagleEye is not required, keep all three child switches set to false.

Enable InferNex Bridge and NPU Serving Runtime

Alauda AI Platform installs and manages InferNex Bridge through the infernexBridge component of the existing default AmlCluster. For Ascend NPU inference, enable the Alauda AI Platform NPU Serving Runtime through servingRuntimeNPUOperator as well. Merge the required settings into the existing AmlCluster; do not create another instance:

spec:
  components:
    infernexBridge:
      managementState: Managed
    servingRuntimeNPUOperator:
      managementState: Managed

Save the AmlCluster and wait for reconciliation. The NPU Serving Runtime component installs the vLLM-Ascend image and provides the ClusterServingRuntime and LLMInferenceServiceConfig resources required by serving workloads. There is no need to install the InferNex Bridge or NPU Serving Runtime operators manually through OperatorHub.

Verification

Check the AmlCluster status and confirm the managed components and their workloads are ready:

kubectl get amlcluster default
kubectl get pods -n infernex-system
kubectl get crd \
  infernexservices.infernex.infernex.io \
  infernexserviceconfigs.infernex.infernex.io \
  resourcescalinggroups.autoscaling.openfuyao.com \
  elasticscalers.elasticscaler.io \
  tidals.tidal.io
kubectl get clusterservingruntime
kubectl get llminferenceserviceconfig -A

The AmlCluster should report Ready / Reconciled, the InferNex Bridge controller should be running, all five InferNex Bridge CRDs should be installed, and the LLMInferenceService mutating webhook should include v1alpha2. When servingRuntimeNPUOperator is managed, confirm the NPU ClusterServingRuntime and LLMInferenceServiceConfig are present.

Community Examples

For community-maintained examples, see InferNex Bridge v26.6.0 examples.

Upgrading InferNex Bridge

Alauda AI Platform manages the InferNex Bridge lifecycle through the infernexBridge component in the default AmlCluster. Follow your platform's Alauda AI Platform component upgrade procedure; do not upgrade the operator independently through OperatorHub. Before upgrading, back up the existing custom resources:

kubectl get infernexservice,infernexserviceconfig -A -o yaml > infernex-bridge-backup.yaml

Verification

After the AmlCluster reconciliation completes, verify the controller and CRD status:

kubectl get pods -n infernex-system
kubectl get crd \
  infernexservices.infernex.infernex.io \
  infernexserviceconfigs.infernex.infernex.io \
  resourcescalinggroups.autoscaling.openfuyao.com \
  elasticscalers.elasticscaler.io \
  tidals.tidal.io

Rollback

Use the platform's supported component rollback procedure and confirm that the target version's CRDs are compatible with the custom resources already stored in the cluster.

After rollback, verify the controller, CRDs, inference services, runtime image IDs, and one representative inference request. Do not delete the five bundled CRDs while custom resources still exist; deleting them also deletes the stored custom resources.