Home Technology peripherals AI Sangfor and TrendForce join forces to use high-performance storage to support the development of large AI models

Sangfor and TrendForce join forces to use high-performance storage to support the development of large AI models

Aug 28, 2023 pm 06:53 PM
Convinced Trend Technology High performance storage

Recently, Beijing Trendong Technology Co., Ltd. (hereinafter referred to as " Trendong Technology ") and Sangfor officially launched a joint solution. This solution organically combines Sangfor EDS's high-performance storage with OrionX AI computing resource pooling software and Gemini AI training platform. It aims to integrate storage and computing resources to help users build efficient artificial intelligence platforms and effectively manage them. and utilizing artificial intelligence resources

Sangfor and TrendForce join forces to use high-performance storage to support the development of large AI models

Specifically, the launch of joint solutions will bring the following changes to the construction of user infrastructure in the field of artificial intelligence

The high-performance joint solution is ready to create a more efficient training platform

As the construction of large-scale artificial intelligence models accelerates, users have increasingly higher requirements for the efficiency of artificial intelligence model training. However, due to problems such as insufficient GPU computing resources and insufficient read and write performance of small files in the underlying storage, a large number of training tasks in the training platform have to be queued. Insufficient computing power and storage capacity slow down the training efficiency of the entire artificial intelligence training platform

To solve this problem, the joint solution has been fully optimized. In response to the efficiency issue of the upper-layer training platform, OrionX AI computing resource pooling software of Trend Technology helps users flexibly allocate GPU resources according to task conditions by creating a computing resource pool to achieve resource segmentation, aggregation, remote calling, and super-scoring. , task queuing, dynamic mounting and release, and domestic chip heterogeneous pooling and other capabilities to fully meet the computing power requirements of various training tasks and accelerate the progress of tasks. At the same time, the scheduling capability provided by the Gemini AI training platform optimizes the management mechanism of the training platform. Under unified scheduling, the training of AI models is more efficient

Through EDS's self-developed heuristic read-ahead mechanism and multi-live metadata service, we can effectively solve the performance problems of underlying storage. Even with tens of billions of data sets, we can still provide high-speed reading and writing capabilities. In this way, it can not only reduce the waiting time of the GPU, but also improve the throughput and training efficiency of short-term loop training

Sangfor and TrendForce join forces to use high-performance storage to support the development of large AI models

2. Capacity and performance can be expanded simultaneously to create a cost-effective storage solution

In the daily AI data set training process, in order to make the model more accurate, it is often necessary to use massive images, text and other data to train the AI ​​model. The rapid growth of data has put tremendous pressure on the capacity and performance of underlying storage. The high-cost and inefficient expansion model of traditional storage has become increasingly difficult to meet performance and capacity needs.

Relying on fully self-developed technologies such as matrix storage algorithms, EDS can effectively solve the space waste problem caused by write amplification of small files in the process of storing small files such as images, texts, and videos, maximizing the use of storage space, and is composed of three nodes. A cluster can meet the storage needs of a medium-sized AI training team. In terms of performance expansion, also benefiting from the architectural advantages of software-defined storage, EDS can achieve simultaneous expansion of capacity and performance during expansion, flexibly responding to the rapidly growing performance requirements of AI business.

3. Unified management and deep mining of the value of data

Before releasing the solution, EDS had successfully achieved seamless integration with Trend Micro’s Gemini AI training platform through protocols such as NFS CSI and S3. Through deep adaptation, the Kubernetes container orchestration platform can complete dynamic allocation of storage resources faster. Users can directly skip the feasibility study of the solution during deployment and quickly launch AI training tasks. At the same time, EDS also supports data interoperability between multiple protocols. Multiple types of clients can share a storage system. The result data of each stage does not need to be copied across storages, thereby achieving efficient flow and ensuring that users can Effectively use data results to more conveniently mine the value of data

In the future, the two parties will also strengthen cooperation in the technical field and help users accelerate the upgrade and construction of AI training platforms by providing joint solutions with higher storage performance, so that more users can achieve greater results on the road to AI training. Faster and more stable progress

About Trend Technology: Trend Technology is committed to providing users with the world's leading data center-level AI computing power virtualization and resource pooling solutions. At present, many leading companies and users in artificial intelligence, Internet, operators, finance, automobile and autonomous driving, education and other industries have used OrionX AI computing resource pooling solution

The above is the detailed content of Sangfor and TrendForce join forces to use high-performance storage to support the development of large AI models. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Best AI Art Generators (Free & Paid) for Creative Projects Best AI Art Generators (Free & Paid) for Creative Projects Apr 02, 2025 pm 06:10 PM

The article reviews top AI art generators, discussing their features, suitability for creative projects, and value. It highlights Midjourney as the best value for professionals and recommends DALL-E 2 for high-quality, customizable art.

Getting Started With Meta Llama 3.2 - Analytics Vidhya Getting Started With Meta Llama 3.2 - Analytics Vidhya Apr 11, 2025 pm 12:04 PM

Meta's Llama 3.2: A Leap Forward in Multimodal and Mobile AI Meta recently unveiled Llama 3.2, a significant advancement in AI featuring powerful vision capabilities and lightweight text models optimized for mobile devices. Building on the success o

Best AI Chatbots Compared (ChatGPT, Gemini, Claude & More) Best AI Chatbots Compared (ChatGPT, Gemini, Claude & More) Apr 02, 2025 pm 06:09 PM

The article compares top AI chatbots like ChatGPT, Gemini, and Claude, focusing on their unique features, customization options, and performance in natural language processing and reliability.

Is ChatGPT 4 O available? Is ChatGPT 4 O available? Mar 28, 2025 pm 05:29 PM

ChatGPT 4 is currently available and widely used, demonstrating significant improvements in understanding context and generating coherent responses compared to its predecessors like ChatGPT 3.5. Future developments may include more personalized interactions and real-time data processing capabilities, further enhancing its potential for various applications.

Top AI Writing Assistants to Boost Your Content Creation Top AI Writing Assistants to Boost Your Content Creation Apr 02, 2025 pm 06:11 PM

The article discusses top AI writing assistants like Grammarly, Jasper, Copy.ai, Writesonic, and Rytr, focusing on their unique features for content creation. It argues that Jasper excels in SEO optimization, while AI tools help maintain tone consist

Top 7 Agentic RAG System to Build AI Agents Top 7 Agentic RAG System to Build AI Agents Mar 31, 2025 pm 04:25 PM

2024 witnessed a shift from simply using LLMs for content generation to understanding their inner workings. This exploration led to the discovery of AI Agents – autonomous systems handling tasks and decisions with minimal human intervention. Buildin

Choosing the Best AI Voice Generator: Top Options Reviewed Choosing the Best AI Voice Generator: Top Options Reviewed Apr 02, 2025 pm 06:12 PM

The article reviews top AI voice generators like Google Cloud, Amazon Polly, Microsoft Azure, IBM Watson, and Descript, focusing on their features, voice quality, and suitability for different needs.

Selling AI Strategy To Employees: Shopify CEO's Manifesto Selling AI Strategy To Employees: Shopify CEO's Manifesto Apr 10, 2025 am 11:19 AM

Shopify CEO Tobi Lütke's recent memo boldly declares AI proficiency a fundamental expectation for every employee, marking a significant cultural shift within the company. This isn't a fleeting trend; it's a new operational paradigm integrated into p

See all articles