Training Guides

End-to-end recipes for training and fine-tuning models on Alauda AI Platform.

Pick a path

When you want…UseGuide
Fine-tune or train on GPU and serve a compatible model on Ascend NPUCross-accelerator workflow indexTrain on GPU and Run Inference on NPU
Fine-tune a YOLOv5 object-detection model and deploy it with TritonKubeflow Trainer v2 + Triton Inference ServerTrain, Fine-Tune, and Deploy YOLOv5
Reusable templates, repeatable runs, optional Kueue quotasKubeflow Trainer v2 + LlamaFactoryFine-Tuning with Kubeflow Trainer v2
Survive crashes, preemption, or a pause without losing hours of trainingCheckpoint PVC + resume_from_checkpoint + maxRestartsCheckpointing and Resuming TrainJobs
Survive crashes, preemption, or a resubmit of a KubeRay training jobCheckpoint PVC + Ray Train storage_path + TorchTrainer.restoreCheckpointing and Resuming RayJobs
Mix training with online inference, yield GPU back on demandKueue cohort + preemption + checkpoint resumePreemptible TrainJobs with Kueue
Run a fine-tune on a slice of a GPU instead of a whole cardDynamic Resource Allocation (DRA) MIG slice + Trainer v2GPU Slicing with Dynamic Resource Allocation (DRA)
A curated set of TrainingRuntime images (CUDA / CANN)Trainer v2 runtime catalogTraining Runtime Images
One-shot quick start of distributed PyTorch on Trainer v2ClusterTrainingRuntime + MNISTKubeflow Trainer Quick Start
Run a two-node PyTorch DDP job directly with Kubernetes JobSetIndexed Job + stable pod DNS + torchrunRun Distributed PyTorch Training with JobSet
Run an OpenMPI workload on Trainer v2MPI-enabled TrainingRuntime + TrainJobRun MPI Training with Kubeflow Trainer v2
Production SFT / OSFT with automatic memory managementtraining_hubFine-tuning LLMs with Training Hub
Reinforcement learning from programmatically verifiable rewardsTraining Hub lora_grpo + verlRun RL with verifiable rewards using Training Hub
Interactive exploration, custom scripts, VolcanoJob submissionWorkbench NotebookFine-tuning LLMs using Workbench
Turn PDFs into SFT JSONL from a WorkbenchDocling + SDG Hub + OpenAI-compatible teacherPrepare Fine-Tuning Data with Docling and SDG Hub
Full-parameter SFT / pretraining on Ascend NPUWorkbench PyTorch CANN / MindSpore CANNFine-tune and Pretrain on Ascend NPU
Track a KFP run's parameters and metrics in MLflowKFP component + MLflow SDKKubeflow Pipeline + MLflow Integration
Build features offline, materialize them online, and serve a model with FeastFeast Operator + KServeFeast Offline-to-Online Inference
Run Feast's Ray offline store on a KubeRay clusterFeast Ray offline store + KubeRay OperatorFeast with Ray Offline Store
Run a distributed training job on KubeRay and compare runs in MLflowKubeRay RayJob + Ray Train + MLflowTrain on KubeRay and Track Experiments with MLflow
Submit KubeRay training jobs through Kueue and scale workers on demandKueue admission + Ray Autoscaler v2KubeRay Training with Kueue and Autoscaling
Train tabular or time-series models with reusable AutoGluon assetsManaged KFP pipelines or composable componentsUse Reusable Kubeflow Pipeline Components
Use KFP caching, parallel loops, and persistent typed artifactsKFP 2.16.1 execution mechanismsKubeflow Pipelines Execution and Storage Behavior
Daily fine-tune → evaluate → compare loop with MLflow + TrustyAIKFP Recurring Run + MLflow Model Registry + LMEvalJobDaily Fine-Tuning Pipeline with MLflow and TrustyAI
Parse thousands of PDFs with Ray Data and optional Milvus ingestKFP parse_and_chunk RayJob from pipelines-componentsRun Ray Data Pipelines from pipelines-components
Search RAG chunking, embeddings, and retrieval settings for a document corpusKFP AutoRAG pipelines + OGX (Llama Stack)Optimize a RAG Application with AutoRAG Pipelines
NOTE

Several training notebooks load source code and manifests from their adjacent assets/ directories. Clone each notebook and its assets together instead of downloading the .ipynb alone:

git clone --depth 1 --filter=blob:none --sparse https://github.com/alauda/aml-docs.git
git -C aml-docs sparse-checkout set --no-cone \
  /docs/en/train/guides/feast-offline-to-online-inference.ipynb \
  /docs/en/train/guides/assets/feast-offline-to-online-inference/ \
  /docs/en/train/guides/feast-with-ray-offline-store.ipynb \
  /docs/en/train/guides/assets/feast-with-ray-offline-store/ \
  /docs/en/train/guides/kuberay-training-with-mlflow.ipynb \
  /docs/en/train/guides/assets/kuberay-training-with-mlflow/ \
  /docs/en/train/guides/kuberay-checkpointing-and-resuming.ipynb \
  /docs/en/train/guides/assets/kuberay-checkpointing/ \
  /docs/en/train/guides/prepare-finetune-data-with-docling-and-sdg.ipynb \
  /docs/en/train/guides/assets/docling-sdg/ \
  /docs/en/train/guides/run-ray-data-pipelines.ipynb \
  /docs/en/train/guides/assets/ray-data-pipelines/ \
  /docs/en/train/guides/run-distributed-training-with-jobset.ipynb \
  /docs/en/train/guides/assets/jobset-pytorch-ddp/ \
  /docs/en/train/guides/run-mpi-training-with-trainer-v2.ipynb \
  /docs/en/train/guides/assets/mpi-trainer-v2/
cd aml-docs/docs/en/train/guides