AI Inference

AI Inference

AI inference within the Cortensor network is at the core of the platform’s capabilities, enabling efficient and scalable AI computations through a decentralized architecture. This section delves into the mechanisms and processes that facilitate AI inference, ensuring high performance, inclusivity, and security.

Overview

Cortensor's AI inference leverages a distributed network of miner nodes to perform computations using advanced AI models. The system supports a diverse range of hardware, from low-end devices to high-end GPUs, ensuring broad participation and inclusivity. The primary AI models currently supported include Llama 3, available in both quantized and regular versions, allowing even lower-end devices to contribute effectively.

AI Inference Process

Task Initiation:

• Users create sessions and submit prompts through router nodes.

• Router nodes verify session parameters, including payment and model specifications, before processing the request.

Task Allocation:

• Router nodes dynamically allocate inference tasks to suitable miner nodes.

• Allocation algorithms consider node performance, current workload, and specific task requirements to optimize resource utilization.

Inference Execution:

• Miner nodes perform the assigned AI inference tasks.

• Tasks are segmented into smaller subtasks to enhance processing efficiency and balance the workload.

• Model quantization allows lower-end devices to handle inference tasks, promoting inclusivity.

Result Submission:

• Miner nodes submit the results securely through encrypted channels.

• Results are sent to the router nodes for initial aggregation and verification.

Validation:

• Validation nodes or other miner nodes verify the inference results.

• Validation methods include semantic checks, embedding comparisons, and checksum verifications.

• Users can configure the level of validation required, balancing between cost and accuracy.

Result Delivery:

• Validated results are delivered to users through their preferred channels.

• The router node ensures secure and efficient result delivery while maintaining user privacy.

Security and Privacy

Encrypted Communication:

• All communications within the network are encrypted to ensure data privacy and integrity.

• Router nodes manage encryption and decryption, ensuring secure interactions between clients and miner nodes.

Validation and Verification:

• Validation nodes verify the accuracy of AI inference results.

• Configurable validation processes allow users to specify the required level of accuracy, influencing costs and ensuring reliable outputs.

Inclusivity through Quantization

Model Quantization:

• Cortensor employs model quantization to support a diverse range of hardware, including lower-end devices.

• This inclusivity allows devices with limited computational power to perform inference tasks, enhancing the network's scalability and resource utilization.

• The focus on supporting Llama 3 models, both quantized and regular, ensures wide participation and efficient task execution across different hardware capabilities.