Drones are getting smarter! Li Xuelong's team creates a new era of machine speaking

Language is the most important symbol system for human communication and thinking, and an important force in promoting human civilization. So, can machines interact with language to express what they see, hear, and think, and become a real What about intelligent robots? Recently, Professor Li Xuelong from the Institute of Optoelectronics and Intelligence of Northwestern Polytechnical University and his colleagues have made innovative progress in machine interaction: based on a domestically produced large model, they developed a "group chat" drone The control framework equips each drone with a brain, allowing the drone cluster to dynamically collaborate in language communication, realizing the open environment of "human-machine" and "multiple "Machine" dialogue interaction breaks the interaction barriers between humans and machines and further expands the application scenarios of local security.

Large models have excellent generalization capabilities, which makes them ideal for achieving "general artificial intelligence" A ray of hope. However, just reading a lot of books is far less effective than hands-on practice. In an open environment, large models need to be truly integrated into the physical world to truly understand complex tasks and solve practical problems

Recently, Professor Li Xuelong’s team has carried out innovations in autonomous drone clusters in an open environment Research, let big models give wings and fly into our real life.

Inspired by human cognitive models, the team condensed the high degree of autonomy in cognitive formation into "Thinking Computing—Entity Control— Environmental awareness"'s three-dimensional interaction has established a "group chat-style" control framework for autonomous drones driven by the "ScholarPuyu" open source large model, achieving Intelligent interaction, active perception and autonomous control in open environments and complex tasks improve the autonomy of UAV mission execution.

In general, Human-like dialogue interaction, active environment perception, and autonomous entity control are the main capabilities of autonomous drone clusters.

1. Human-like dialogue interaction

##Figure 1 Drone Group Chat Communication

Exploring the interaction between human users and drones, allowing drones to understand user needs in complex tasks, is a prerequisite for realizing autonomous drones .

In response to this, the team proposed a "group chat" dialogue interaction method, which converts various information such as sounds, images, and the drone's own status into a natural language dialogue form through a large model,

It provides an autonomous and intuitive way of interaction between users and drones, as well as drones and drones. At the same time, the team designed an efficient real-time feedback mechanism that enables drones to report their status through dialogue and seek user confirmation at key nodes in mission execution, greatly improving the stability and safety of complex mission execution.

2. Active environment perception

##Figure 2

Actively discover and approach the target

##Figure 3

Dynamic Environment Obstacle Avoidance During flight, the drone actively senses the external environment and adjusts the mission plan in real time, which is a key link in completing complex tasks.

In response to this, the team designed an active perception mechanism for task guidance and proposed multi-sensor fusion low-altitude search, dynamic obstacle avoidance and visual positioning algorithms. In actual mission execution, the drone flight path and observation posture are dynamically adjusted based on the perceived information and mission goals,

attempts to perceive the surrounding world from different angles and positions, gradually reducing the uncertainty in the environment, and achieving efficient Information collection and task execution

.

3.

Autonomous control

Figure 4 Autonomous target capture

Figure 5 Heterogeneous UAV Cluster Collaborative Control

Explore the form of composite agents and enhance complex task processing capabilities. It is a large model The research focus of new intelligent agents in the era.

In response to this, the team relied on the drone platform to design end effectors such as grippers, expanding traditional drones into "flying robots”, grows “hands” and has the ability to grasp. At the same time, a heterogeneous UAV cluster collaborative control mechanism was constructed, which combined with environmental perception feedback to adjust the flight status of the UAV formation in real time, allowing the cluster to perform tasks such as regional search, target positioning and grabbing.

The large-model autonomous drone cluster is a successful attempt by the team to apply the three-dimensional interaction model of biological intelligence "thinking calculation-entity control-environment perception" to autonomous agents. Relying on the large language model, unmanned aerial vehicle It uses a machine platform and a variety of sensors to achieve dialogue interaction, active perception and autonomous control, which is of great significance for applications in security inspections, disaster rescue, air logistics and other local security scenarios.

Extended reading: Li Xuelong, Vicinagearth security, Communications of the China Computer Federation, 18(11), 44-52, 2022.

Full text download:

https://dl.ccf.org.cn/article/articleDetail.html?type=xhtx_thesis&_ack=1&id=6219452051015680

The above is the detailed content of Drones are getting smarter! Li Xuelong's team creates a new era of machine speaking. For more information, please follow other related articles on the PHP Chinese website!