Complete data analysis in one sentence, Zhejiang University's new large model data assistant eliminates the need for collection

To process data, just use this AI tool!

Relying on the large language model (LLM) behind it, you only need to describe the data you want to see in one sentence, and leave the rest to it!

Processing, analysis, and even visualization can all be done easily. You don’t even need to do the collection yourself.

Picture

This LLM-based AI data assistant is called Data-Copilot and was developed by a team from Zhejiang University.

The preprint of the relevant paper has been released.

The following content is provided by the contributor

Finance, meteorology, energy and other industries generate a large amount of heterogeneous data every day. There is an urgent need for a tool to effectively manage, process and display this data.

DataCopilot autonomously manages and processes massive data by deploying large language models to meet diverse user query, calculation, prediction, visualization and other needs.

You only need to enter text to tell DataCopilot the data you want to see, without tedious operations, No need to write your own code, DataCopilot autonomously transforms the original data into a visualization result that best meets the user's intention.

In order to achieve a universal framework that covers various forms of data-related tasks, the research team proposed Data-Copilot.

This model solves the problems of data leakage risk, poor computing power, and inability to handle complex tasks caused by simply using LLM.

Picture

When receiving a complex request, Data-Copilot will independently design and schedule an independent interface to build a work process to satisfy user intent.

Without human assistance, it can skillfully transform raw data from different sources and in different formats into humanized output such as graphics, tables and text.

Picture

The main contributions of the Data-Copilot project include:

Connecting data sources and diversification in different fields user needs, reducing tedious labor and professional knowledge.
• Achieves autonomous management, processing, analysis, prediction and visualization of data, and can transform raw data into informative results that best meet user intentions.
• It has the dual
• identities of designer and scheduler, including two processes: the interface tool design process (designer) and Scheduling process (scheduler). Data-Copilot Demo was built based on Chinese financial market data.
• Design and execute the workflow independently

Let’s take the following example to see the performance of Data-Copilot:

All components of the Shanghai Composite 50 Index in the first quarter of this year What is the year-on-year growth rate of the stock’s net profit?

Data-Copilot independently designed such a workflow:

PictureAimed at this For complex problems, Data-Copilot uses the loop_rank interface to implement multiple loop queries.

Data-Copilot got this result after executing this workflow:

The abscissa is the name of each component stock, and the ordinate is the year-on-year growth rate of net profit in the first quarter

PictureIn addition to the general data processing process, Data-Copilot can also generate a wide variety of workflows.

The research team tested Data-Copilot in two workflow modes: predictive and parallel.

Forecasting workflow

For parts other than known data, Data-Copilot can also predict, for example, enter the following question:

Predict the next four quarters China Quarterly GDP

Data-Copilot deploys such a workflow:

Get historical GDP data→Use linear regression model to predict the future→Output table

PictureThe results after execution are as follows:

Picture

Parallel Workflow

I want to see the price-earnings ratios of CATL and Kweichow Moutai in the last three years

corresponding The workflow is:

Get stock price data→Calculate related index→Generate chart

Picture

The related work of the two stocks is at the same time In parallel, the following chart is finally obtained:

Picture

Main method

Data-Copilot is a general large language model The system has two main stages: interface design and interface scheduling.

• Interface design: The research team designed a self-request process to enable LLM to autonomously generate enough requests from a small number of seed requests. Then, LLM iteratively designs and optimizes the interface based on the generated requests. These interfaces are described using natural language, making them easy to extend and transfer between different platforms.
• Interface scheduling: After receiving user requests, LLM plans and calls interface tools based on self-designed interface descriptions and in context demonstrations, deploys a workflow that meets user needs, and presents the results in multiple forms to users.

Data-Copilot achieves highly automated data processing and visualization by automatically generating requests and independently designing interfaces to meet user needs and display results to users in multiple forms.

Picture

Interface design

As shown in the figure above, data management must first be implemented, and the first step requires interface tools.

Data-Copilot will design a large number of interfaces as a data management tool. The interface is a module composed of natural language (functional description) and code (implementation), which is responsible for data acquisition, processing and other tasks.

• First of all, LLM uses a small number of seed requests and independently generates a large number of requests (explore data by self-request) to cover various application scenarios as much as possible.
• Then, LLM designs corresponding interfaces for these requests (interface definition: only includes description and parameters), and gradually optimizes the interface design (interface merge) in each iteration.
• Finally, the researchers used LLM's powerful code generation capabilities to generate specific code (interface implementation) for each interface in the interface library. This process separates the design of the interface from the specific implementation, creating a versatile set of interface tools that can satisfy most requests.

As shown below: Data-Copilot’s self-designed interface tool is used for data processing

Picture

Interface Scheduling

In the previous stage, researchers obtained various common interface tools for data acquisition, processing and visualization. Each interface has a clear and explicit functional description. As shown in the figure above for the two queries, Data-Copilot forms a workflow from data to results in multiple forms through planning and calling different interfaces in real-time requests.

• Data-Copilot first performs intent analysis to accurately understand the user's request.
• Once the user's intention is accurately understood, Data-Copilot will plan a reasonable workflow to handle the user's request. Data-Copilot will generate a fixed-format JSON that represents each step of scheduling, such as step={“arg”:””, “function”:””, “output”:””,”description”:””} .

Guided by interface descriptions and examples, Data-Copilot orchestrates the scheduling of interfaces within each step, either sequentially or in parallel.

Data-Copilot significantly reduces dependence on tedious labor and expertise by integrating LLMs into every stage of data-related tasks, automatically transforming raw data into user-friendly visualizations based on user requests .

GitHub project page: https://github.com/zwq2018/Data-Copilot

Paper address: https://arxiv.org/abs /2306.07209

HuggingFace DEMO:https://huggingface.co/spaces/zwq2018/Data-Copilot

The above is the detailed content of Complete data analysis in one sentence, Zhejiang University's new large model data assistant eliminates the need for collection. For more information, please follow other related articles on the PHP Chinese website!