Coders are ecstatic! Microsoft proposed CodePlan to automate LLM coding tasks across 168 code bases

For large models, it is good at localized encoding tasks.

If the task involves multiple interdependent files, LLM cannot solve this problem

Microsoft researchers designed a tool called CodePlan for this purpose Task-independent neural network framework

Picture

Paper address: https://arxiv.org/pdf/2309.12499. pdf

In the paper, CodePlan integrates a multi-step chain-of-edits, which is a new method that combines program analysis, planning and LLM.

Let’s take a closer look at how CodePlan is designed?

CodePlan: Large model planning

In software engineering activities such as package migration, fixing bug reports for static analysis or testing, and reporting to Adding type hints or other conventions to a code base involves pervasive editing of the entire code repository.

Researchers plan these activities to enable "repository-level coding tasks"

Coding tools such as GitHub Copilot, Code Whisperer have been obtained With the support of large model capabilities, it provides coders with solutions to localized coding problems

However, the fact is that "repository-level coding tasks" are more complex and cannot be directly Solved by LLM, because the code in the repository is interdependent, the entire repository may be too large to be included in the prompt

In this study, the Microsoft team used the library-level coding framework as a planning problem and propose a task-agnostic framework called CodePlan.

CodePlan combines a multi-step edit chain (plan), each step of which calls LLM at the code location. The context at that code location comes from the entire repository, previous code changes, and task-specific instructions.

CodePlan is a new combination based on incremental dependency analysis, possible impact analysis of changes and adaptive planning algorithms

Picture

The picture below shows the changes in the plural library API. Microsoft researchers are tasked with migrating the code base according to this change.

Picture

The left side of Figure 3 shows the parts of the code base related to using the plural library

Specifically, the method func in the Create.cs file calls the create_complex method in the library, and the method Process.cs in the Process.cs file calls func.

Picture

The researcher passes the task description and func body from Figure 1 to LLM to generate the modified func code , as shown on the right side of Figure 3

You can see that LLM has correctly edited the call to the create_complex API so that it returns a Complex type object instead of two floats. Tuple of point values.

Note that this edit resulted in a change in the signature of method func - it now returns an object of type Complex.

What needs to be modified is: the caller of method func needs to be modified, such as the process method in the Process.cs file, as shown in the lower left corner of Figure 3. Without appropriate modifications to the body of the process method, the code will not build!

The lower right of Figure 3 shows appropriate modifications to the process method, which can bring the repository to a consistent state so that there will be no errors during compilation.

The main task of the researchers is to establish a "repository-level encoding system" that can automatically generate the derivation specifications required for editing

LLM-driven library-level coding tasks are defined as follows:

Picture

In the overall framework of CodePlan, inputs include a repository, a seed specification expressed through natural language instructions or a set of initial code edits. Tasks, a correctness oracle, and an LLM

CodePlan creates a plan graph where each node represents the code editing task that the LLM needs to perform, and the edges represent the target nodes that need to be in the source The node then executes

CodePlan to monitor code editing and adaptively expand the plan graph.

Once all steps in the plan have been completed, the repository will be analyzed by Oracle. If Oracle validates the repository, the task is complete. If an error is found, the error report serves as a seed specification for the next round of plan generation and execution.

Picture

In addition, the CodePlan algorithm also maintains a dependency graph. Figure 4 illustrates the structure of the dependency graph.

Picture

Refresh SOTA

Researchers on CodePlan in two storage Effectiveness was evaluated on library-level tasks: package migration (C#) and temporal code editing (Python)

Each task was evaluated on multiple code bases, each The code base all requires interdependent changes to multiple files (2-97 files).

Picture

Never before has LLM been used to automate a coding task of this level of complexity

Research results show that CodePlan is more consistent with the ground truth than the baseline, enabling 5/6 repositories to pass validity checks such as error-free builds and correct code edits

Overall, CodePlan provides a promising approach to automating complex library-level coding tasks, improving both productivity and accuracy

It successfully addresses many challenges and opens up new possibilities for efficient and reliable software engineering practices

Reference:https:// www.php.cn/link/23b1f559d8bd157d0d741c83957ed4f2

The above is the detailed content of Coders are ecstatic! Microsoft proposed CodePlan to automate LLM coding tasks across 168 code bases. For more information, please follow other related articles on the PHP Chinese website!