Table of Contents
humanoiddialogueinteraction
Active environment perception
Autonomous Control
Home Technology peripherals AI NPU launches an innovative UAV control framework: enabling group chat-style interaction, active perception of the environment, and autonomous control of UAVs

NPU launches an innovative UAV control framework: enabling group chat-style interaction, active perception of the environment, and autonomous control of UAVs

Nov 02, 2023 pm 02:25 PM
language Model biological intelligence

Super generalization ability makes large models a ray of hope for "general artificial intelligence".

However, reading thousands of books is not as good as traveling thousands of miles. In an open environment, large models need to truly "walk" into the physical world in order to truly understand complex tasks and solve practical problems.

Recently, Professor Li Xuelong’s team conducted innovative research on autonomous drone swarms in an open environment. They used domestic large-scale models to successfully realize human-computer and multi-machine dialogue interaction in an open environment, breaking the interaction barriers between humans and machines. This research further expands the application scenarios of local security, allowing large drones to soar in real life

Inspired by human cognitive models, our team summarized the highly autonomous cognitive process as " The three-dimensional interaction of "Thinking Computing-Entity Control-Environment Perception" has been established, and a "group chat-style" control framework for autonomous drones driven by the "Scholar·Puyu" open source large model has been established. We equip each drone with an intelligent brain, allowing the drone group to dynamically collaborate through language communication to achieve intelligent interaction, active perception and autonomous control in open environments and complex tasks. This move improves the autonomy of drone mission execution

In general, the main capabilities of autonomous drone swarms include human-like conversational interaction, active environment perception and autonomous entity control

humanoiddialogueinteraction

NPU launches an innovative UAV control framework: enabling group chat-style interaction, active perception of the environment, and autonomous control of UAVsFigure 1 Drone group chat communication

Explore the interaction between human users and drones, allowing drones to understand complex The user needs in the mission are the prerequisites for realizing autonomous drones.

In response to this, the team proposed a "group chat-style" dialogue interaction method, which converts various information such as sounds, images, and the drone's own status into a natural language dialogue form through a large model, realizing the dialogue between users and Drones, and autonomous and intuitive ways to interact with drones.

In order to improve the stability and safety of complex tasks, the team designed an efficient real-time feedback mechanism. This mechanism enables the drone to report its status through dialogue and seek user confirmation at key nodes in mission execution. At the same time, this mechanism can also greatly improve the efficiency of task execution

Active environment perception

NPU launches an innovative UAV control framework: enabling group chat-style interaction, active perception of the environment, and autonomous control of UAVsFigure 2 Actively discover and approach the target

NPU launches an innovative UAV control framework: enabling group chat-style interaction, active perception of the environment, and autonomous control of UAVsFigure 3 Dynamic environment obstacle avoidance

During flight, the drone actively senses the external environment and adjusts the mission plan in real time, which is a key link in completing complex tasks.

In order to deal with this problem, the team developed a task-guided active perception mechanism and proposed multi-sensor fusion low-altitude search, dynamic obstacle avoidance and visual positioning algorithms

In actual mission execution During the process, we can dynamically adjust the flight path and observation posture of the drone based on the perceived information and mission goals. We can try to perceive the world around us from different angles and positions, and gradually reduce the uncertainty in the environment to achieve efficient information collection and task execution

Autonomous Control

NPU launches an innovative UAV control framework: enabling group chat-style interaction, active perception of the environment, and autonomous control of UAVsFigure 4 Autonomous target grabbing

NPU launches an innovative UAV control framework: enabling group chat-style interaction, active perception of the environment, and autonomous control of UAVsFigure 5 Heterogeneous drone cluster collaborative control

The key research is to explore the form of composite agents , to enhance its ability to handle complex tasks. In the era of large models, this is a key area for new intelligent agents

In order to solve this problem, the R&D team used the drone platform to design end effectors such as grippers, upgrading traditional drones to " "Flying Robot", endows it with grabbing capabilities

At the same time, a heterogeneous UAV cluster collaborative control mechanism is also established, and combined with environmental perception feedback, the flight status of the UAV formation is adjusted in real time so that the cluster can Division of labor and cooperation to perform tasks such as regional search, target positioning and crawling

The team successfully tried to apply the three-dimensional interaction model of biological intelligence "thinking computing-entity control-environment perception" to autonomous agents, forming a large-scale autonomous drone cluster. This kind of cluster uses large-scale language models, drone platforms and a variety of sensors to achieve conversational interaction, active perception and autonomous control. This technology is of great significance for the application in on-site security scenarios such as security inspections, disaster rescue, and air logistics

References: Li Xuelong, Vicinagearth security, Communications of the Computer Society of China, 18(11) ), 44-52, 2022

The above is the detailed content of NPU launches an innovative UAV control framework: enabling group chat-style interaction, active perception of the environment, and autonomous control of UAVs. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
1 months ago By 尊渡假赌尊渡假赌尊渡假赌
Two Point Museum: All Exhibits And Where To Find Them
1 months ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

The world's most powerful open source MoE model is here, with Chinese capabilities comparable to GPT-4, and the price is only nearly one percent of GPT-4-Turbo The world's most powerful open source MoE model is here, with Chinese capabilities comparable to GPT-4, and the price is only nearly one percent of GPT-4-Turbo May 07, 2024 pm 04:13 PM

Imagine an artificial intelligence model that not only has the ability to surpass traditional computing, but also achieves more efficient performance at a lower cost. This is not science fiction, DeepSeek-V2[1], the world’s most powerful open source MoE model is here. DeepSeek-V2 is a powerful mixture of experts (MoE) language model with the characteristics of economical training and efficient inference. It consists of 236B parameters, 21B of which are used to activate each marker. Compared with DeepSeek67B, DeepSeek-V2 has stronger performance, while saving 42.5% of training costs, reducing KV cache by 93.3%, and increasing the maximum generation throughput to 5.76 times. DeepSeek is a company exploring general artificial intelligence

KAN, which replaces MLP, has been extended to convolution by open source projects KAN, which replaces MLP, has been extended to convolution by open source projects Jun 01, 2024 pm 10:03 PM

Earlier this month, researchers from MIT and other institutions proposed a very promising alternative to MLP - KAN. KAN outperforms MLP in terms of accuracy and interpretability. And it can outperform MLP running with a larger number of parameters with a very small number of parameters. For example, the authors stated that they used KAN to reproduce DeepMind's results with a smaller network and a higher degree of automation. Specifically, DeepMind's MLP has about 300,000 parameters, while KAN only has about 200 parameters. KAN has a strong mathematical foundation like MLP. MLP is based on the universal approximation theorem, while KAN is based on the Kolmogorov-Arnold representation theorem. As shown in the figure below, KAN has

Hello, electric Atlas! Boston Dynamics robot comes back to life, 180-degree weird moves scare Musk Hello, electric Atlas! Boston Dynamics robot comes back to life, 180-degree weird moves scare Musk Apr 18, 2024 pm 07:58 PM

Boston Dynamics Atlas officially enters the era of electric robots! Yesterday, the hydraulic Atlas just "tearfully" withdrew from the stage of history. Today, Boston Dynamics announced that the electric Atlas is on the job. It seems that in the field of commercial humanoid robots, Boston Dynamics is determined to compete with Tesla. After the new video was released, it had already been viewed by more than one million people in just ten hours. The old people leave and new roles appear. This is a historical necessity. There is no doubt that this year is the explosive year of humanoid robots. Netizens commented: The advancement of robots has made this year's opening ceremony look like a human, and the degree of freedom is far greater than that of humans. But is this really not a horror movie? At the beginning of the video, Atlas is lying calmly on the ground, seemingly on his back. What follows is jaw-dropping

AI subverts mathematical research! Fields Medal winner and Chinese-American mathematician led 11 top-ranked papers | Liked by Terence Tao AI subverts mathematical research! Fields Medal winner and Chinese-American mathematician led 11 top-ranked papers | Liked by Terence Tao Apr 09, 2024 am 11:52 AM

AI is indeed changing mathematics. Recently, Tao Zhexuan, who has been paying close attention to this issue, forwarded the latest issue of "Bulletin of the American Mathematical Society" (Bulletin of the American Mathematical Society). Focusing on the topic "Will machines change mathematics?", many mathematicians expressed their opinions. The whole process was full of sparks, hardcore and exciting. The author has a strong lineup, including Fields Medal winner Akshay Venkatesh, Chinese mathematician Zheng Lejun, NYU computer scientist Ernest Davis and many other well-known scholars in the industry. The world of AI has changed dramatically. You know, many of these articles were submitted a year ago.

Google is ecstatic: JAX performance surpasses Pytorch and TensorFlow! It may become the fastest choice for GPU inference training Google is ecstatic: JAX performance surpasses Pytorch and TensorFlow! It may become the fastest choice for GPU inference training Apr 01, 2024 pm 07:46 PM

The performance of JAX, promoted by Google, has surpassed that of Pytorch and TensorFlow in recent benchmark tests, ranking first in 7 indicators. And the test was not done on the TPU with the best JAX performance. Although among developers, Pytorch is still more popular than Tensorflow. But in the future, perhaps more large models will be trained and run based on the JAX platform. Models Recently, the Keras team benchmarked three backends (TensorFlow, JAX, PyTorch) with the native PyTorch implementation and Keras2 with TensorFlow. First, they select a set of mainstream

FisheyeDetNet: the first target detection algorithm based on fisheye camera FisheyeDetNet: the first target detection algorithm based on fisheye camera Apr 26, 2024 am 11:37 AM

Target detection is a relatively mature problem in autonomous driving systems, among which pedestrian detection is one of the earliest algorithms to be deployed. Very comprehensive research has been carried out in most papers. However, distance perception using fisheye cameras for surround view is relatively less studied. Due to large radial distortion, standard bounding box representation is difficult to implement in fisheye cameras. To alleviate the above description, we explore extended bounding box, ellipse, and general polygon designs into polar/angular representations and define an instance segmentation mIOU metric to analyze these representations. The proposed model fisheyeDetNet with polygonal shape outperforms other models and simultaneously achieves 49.5% mAP on the Valeo fisheye camera dataset for autonomous driving

Tesla robots work in factories, Musk: The degree of freedom of hands will reach 22 this year! Tesla robots work in factories, Musk: The degree of freedom of hands will reach 22 this year! May 06, 2024 pm 04:13 PM

The latest video of Tesla's robot Optimus is released, and it can already work in the factory. At normal speed, it sorts batteries (Tesla's 4680 batteries) like this: The official also released what it looks like at 20x speed - on a small "workstation", picking and picking and picking: This time it is released One of the highlights of the video is that Optimus completes this work in the factory, completely autonomously, without human intervention throughout the process. And from the perspective of Optimus, it can also pick up and place the crooked battery, focusing on automatic error correction: Regarding Optimus's hand, NVIDIA scientist Jim Fan gave a high evaluation: Optimus's hand is the world's five-fingered robot. One of the most dexterous. Its hands are not only tactile

Single card running Llama 70B is faster than dual card, Microsoft forced FP6 into A100 | Open source Single card running Llama 70B is faster than dual card, Microsoft forced FP6 into A100 | Open source Apr 29, 2024 pm 04:55 PM

FP8 and lower floating point quantification precision are no longer the "patent" of H100! Lao Huang wanted everyone to use INT8/INT4, and the Microsoft DeepSpeed ​​team started running FP6 on A100 without official support from NVIDIA. Test results show that the new method TC-FPx's FP6 quantization on A100 is close to or occasionally faster than INT4, and has higher accuracy than the latter. On top of this, there is also end-to-end large model support, which has been open sourced and integrated into deep learning inference frameworks such as DeepSpeed. This result also has an immediate effect on accelerating large models - under this framework, using a single card to run Llama, the throughput is 2.65 times higher than that of dual cards. one

See all articles