Install
Requirements
Hardware
- At least two nodes with 16 cores and 32 GB of memory in total.
- Additional resources for runtime serving are determined by the actual business scale: 10x7B-sized LLM inference instances at the same time, requires at least 10 GPUs and corresponding CPU, memory, disk storage, and object storage.
- 200G free disk space for each worker node.
Software
- CUDA Toolkit Version: 12.6 or higher.
INFO
If your GPU does not support CUDA 12.6, you may still use lower versions of the CUDA Toolkit.However, after deploying Alauda AI Platform, it is necessary to add a custom inference runtime that is adapted for older CUDA versions. This can be done by referring to Extend LLM Inference Runtimes,since the built-in vLLM inference runtime only supports CUDA 12.6 or later versions.
Installing
Installing Alauda AI Platform involves the following high-level tasks:
- Confirm and configure your cluster to meet all requirements. Refer to Pre-installation Configuration.
- Configure the platform entry, authentication resources, and optional UI branding for Alauda AI Platform. Refer to Configure Alauda AI Platform Entry and Branding.
- Install Alauda AI Platform. Refer to Install Alauda AI Platform.
Then, the core capabilities of Alauda AI Platform have been successfully deployed. If you want to quickly experience the product, please refer to the Quick Start.