Table of Contents
Concepts
Pipeline
Component
Recipe
Home Backend Development Golang From Zero to Merge: Building a JSON Renaming Field Component in Go

From Zero to Merge: Building a JSON Renaming Field Component in Go

Nov 13, 2024 pm 03:43 PM

From Zero to Merge: Building a JSON Renaming Field Component in Go

Introduction to Instill-ai

Working on Instill’s pipeline-backend project was like solving a jigsaw ? puzzle—except some pieces kept changing names! My mission? To create a component that could rename JSON fields without creating conflicts. Join me as I share my journey through learning Go, studying Instill’s well-organized docs, and creating a solution that’s now merged and ready to roll! ?


The Challenge

Instill needed a way to rename fields in JSON data structures dynamically. The twist? We had to handle cases where a renamed field might clash with an existing field. Without a conflict resolution system, chaos would reign supreme!

From Zero to Merge: Building a JSON Renaming Field Component in Go instill-ai / pipeline-backend

⇋ A REST/gRPC server for Instill VDP API service

pipeline-backend

From Zero to Merge: Building a JSON Renaming Field Component in Go

pipeline-backend manages all pipeline resources within Versatile Data Pipeline (VDP) to streamline data from the start component, through AI/Data/Application components and to the end component.

Concepts

Pipeline

In ? Instill VDP, a pipeline is a DAG (Directed Acyclic Graph) consisting of multiple components.

flowchart LR
s[Trigger] --> c1[OpenAI Component]
c1 --> c2[Stability AI Component]
c1 --> c3[MySQL Component]
c1 --> e[Response]
c2 --> e
Copy after login
Copy after login
Copy after login
Loading

Component

A Component serves as an essential building block within a Pipeline.

See the component package documentation for more details.

Recipe

A pipeline recipe specifies how components are configured and how they are interconnected.

Recipes are defined in YAML language:

flowchart LR
s[Trigger] --> c1[OpenAI Component]
c1 --> c2[Stability AI Component]
c1 --> c3[MySQL Component]
c1 --> e[Response]
c2 --> e
Copy after login
Copy after login
Copy after login
Enter fullscreen mode Exit fullscreen mode
View on GitHub

To be honest, I was starting to doubt if I could solve this issue, but then Anni dropped the perfect message that kept me going.

From Zero to Merge: Building a JSON Renaming Field Component in Go

Once I got comfortable, ChunHao, who had crafted a JSON schema for this task, gave me the green light? to start coding. And so, the journey began!


Step 2️⃣: Designing the Solution

The key requirements were:

  1. Dynamic Renaming: Fields should be renamed without disturbing the JSON structure.
  2. Conflict Detection: We needed to spot any conflicts between the original and renamed fields.
  3. Conflict Resolution: A smooth solution, like appending a suffix, would prevent name clashes.

From Zero to Merge: Building a JSON Renaming Field Component in Go


Step 3️⃣: Building the Component

Armed with coffee☕ and courage?, I got down to coding. Here’s a sneak peek at the core logic:

Mapping Fields

First, I created a mapping system to track old and new field names. This was key to detecting conflicts.

flowchart LR
s[Trigger] --> c1[OpenAI Component]
c1 --> c2[Stability AI Component]
c1 --> c3[MySQL Component]
c1 --> e[Response]
c2 --> e
Copy after login
Copy after login
Copy after login

Any time a conflict was detected, the function added "_conflict" to the new name. It’s a simple trick that ensures our JSON fields stay unique and, most importantly, friendly to each other! ✌️

Renaming Fields

Once the field mappings were in place, the next step was applying them to our JSON data.

variable
  <span># pipeline input fields</span>
output:
  <span># pipeline output fields</span>
component:
  <component-id>:
    type: <component-definition-id>
    task: <task-id>
    input:
      <span># values for the input fields</span>
    condition: <condition> <span># conditional statement to execute or bypass the</span>
Copy after login

Here’s the logic that applies the mapped names to our JSON data. The result? Our data’s neatly renamed, conflicts resolved, and structure intact. ?

After creating the component dropped the draft PR & got a comment:

From Zero to Merge: Building a JSON Renaming Field Component in Go


Step 4️⃣: Testing and Refinement

After familiarizing myself with Instill's testing methods and learning how to create effective test cases, I proceeded further.

From Zero to Merge: Building a JSON Renaming Field Component in Go

Testing time! ? I wrote tests covering everything from simple renames to complex edge cases with nested JSON fields. Each round of testing led to further refinements.

func mapFields(fields map[string]string) map[string]string {
    newFieldMap := make(map[string]string)
    for oldName, newName := range fields {
        // Check for conflict
        if _, exists := newFieldMap[newName]; exists {
            newName += "_conflict" // Add suffix for conflicts
        }
        newFieldMap[oldName] = newName
    }
    return newFieldMap
}
Copy after login

Here’s where I’d love to share a personal reflection: Testing was the hardest part of this project ?‍?. There were times when I thought, "Is this test even doing what it’s supposed to?"

Just then, I ran into a lint issue

From Zero to Merge: Building a JSON Renaming Field Component in Go

He pointed out the problem and even provided the solution. All I had to do was implement it, but it was a reminder that even the smallest details matter in making the code work smoothly.

Once I got past those initial hurdles, testing became my safety net. It gave me the confidence to know that my code would work across different scenarios ?️‍♂️. It also showed me that testing isn’t just a step to check off—it’s a way to ensure my code is reliable and resilient.


Step 5️⃣: CI Check and Final Adjustments

After completing my tests, I pushed my code, ready for the review process. However, our CI (Continuous Integration) checks didn’t pass. Anni’s comment gave me a gentle reminder to double-check my test cases:

“Hey @AkashJana18, could you check your test cases? Our CI check shows it has not passed here. Please test it locally first before pushing it to the PR. Whenever you push your commit, we’ll trigger the checks so you can spot any issues before our engineers review your code. Thanks!”

From Zero to Merge: Building a JSON Renaming Field Component in Go

That’s when I realized I had to run the tests locally before submitting. ChunHao also added:

"Please run and pass it before you request the review. Run $ go test ./pkg/component/operator/json/v0/... to check it locally."

I quickly ran the tests locally, identified the issues, and fixed them.

From Zero to Merge: Building a JSON Renaming Field Component in Go

From Zero to Merge: Building a JSON Renaming Field Component in Go
A little moment of celebration ?

This process made me appreciate the importance of local testing even more, as it ensured everything was solid before submitting for review.

Before merging, ChunHao did a final review, made a few tweaks, QAed Test Recipe and updated the documentation to reflect the new changes. Big thanks to Anni for her ongoing support throughout the process—it made a huge difference. ?


? Reflection on the Collaborative Process ??‍??

One of the biggest lessons I learned was how collaboration and mentorship can make or break a project. Instill's moderators, Anni and ChunHao, provided me with the guidance I needed when I was lost in Go syntax or struggling with the right approach. Working together, we turned a complex problem into a clean, functional solution.

I’ll be honest, there were moments I felt like I had bitten off more than I could chew. But the constant encouragement from Anni, combined with the clear direction from ChunHao, kept me on track.


⏭️ Next Steps and Future Improvements

Another step could be expanding this approach to other parts of the pipeline that require dynamic field name handling—because who doesn’t love a little bit of automation⚙️?


?️ Tools & Resources ?

  1. Go Documentation: For diving into Go syntax and understanding core concepts.
  2. Instill Docs: A goldmine of well-organized resources to understand the Instill pipeline.
  3. Go Testing Framework: The built-in testing package in Go for writing unit tests, ensuring everything works as expected, and integrating with CI tools.
  4. Golangci-lint: A Go linters aggregator to identify issues and enforce code quality during development and CI checks.

??‍??️ My Learning

With Instill’s rock-solid documentation, guidance from ChunHao, and Anni's moral support, this project became a fantastic learning experience. I went from knowing nothing about Go to implementing a fully functional feature ready for production (and I have the merged PR to prove it ?).

Proof:

From Zero to Merge: Building a JSON Renaming Field Component in Go

The above is the detailed content of From Zero to Merge: Building a JSON Renaming Field Component in Go. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1664
14
PHP Tutorial
1268
29
C# Tutorial
1242
24
Golang's Purpose: Building Efficient and Scalable Systems Golang's Purpose: Building Efficient and Scalable Systems Apr 09, 2025 pm 05:17 PM

Go language performs well in building efficient and scalable systems. Its advantages include: 1. High performance: compiled into machine code, fast running speed; 2. Concurrent programming: simplify multitasking through goroutines and channels; 3. Simplicity: concise syntax, reducing learning and maintenance costs; 4. Cross-platform: supports cross-platform compilation, easy deployment.

Golang vs. Python: Performance and Scalability Golang vs. Python: Performance and Scalability Apr 19, 2025 am 12:18 AM

Golang is better than Python in terms of performance and scalability. 1) Golang's compilation-type characteristics and efficient concurrency model make it perform well in high concurrency scenarios. 2) Python, as an interpreted language, executes slowly, but can optimize performance through tools such as Cython.

Golang and C  : Concurrency vs. Raw Speed Golang and C : Concurrency vs. Raw Speed Apr 21, 2025 am 12:16 AM

Golang is better than C in concurrency, while C is better than Golang in raw speed. 1) Golang achieves efficient concurrency through goroutine and channel, which is suitable for handling a large number of concurrent tasks. 2)C Through compiler optimization and standard library, it provides high performance close to hardware, suitable for applications that require extreme optimization.

Golang's Impact: Speed, Efficiency, and Simplicity Golang's Impact: Speed, Efficiency, and Simplicity Apr 14, 2025 am 12:11 AM

Goimpactsdevelopmentpositivelythroughspeed,efficiency,andsimplicity.1)Speed:Gocompilesquicklyandrunsefficiently,idealforlargeprojects.2)Efficiency:Itscomprehensivestandardlibraryreducesexternaldependencies,enhancingdevelopmentefficiency.3)Simplicity:

Golang vs. Python: Key Differences and Similarities Golang vs. Python: Key Differences and Similarities Apr 17, 2025 am 12:15 AM

Golang and Python each have their own advantages: Golang is suitable for high performance and concurrent programming, while Python is suitable for data science and web development. Golang is known for its concurrency model and efficient performance, while Python is known for its concise syntax and rich library ecosystem.

Golang vs. C  : Performance and Speed Comparison Golang vs. C : Performance and Speed Comparison Apr 21, 2025 am 12:13 AM

Golang is suitable for rapid development and concurrent scenarios, and C is suitable for scenarios where extreme performance and low-level control are required. 1) Golang improves performance through garbage collection and concurrency mechanisms, and is suitable for high-concurrency Web service development. 2) C achieves the ultimate performance through manual memory management and compiler optimization, and is suitable for embedded system development.

Golang and C  : The Trade-offs in Performance Golang and C : The Trade-offs in Performance Apr 17, 2025 am 12:18 AM

The performance differences between Golang and C are mainly reflected in memory management, compilation optimization and runtime efficiency. 1) Golang's garbage collection mechanism is convenient but may affect performance, 2) C's manual memory management and compiler optimization are more efficient in recursive computing.

C   and Golang: When Performance is Crucial C and Golang: When Performance is Crucial Apr 13, 2025 am 12:11 AM

C is more suitable for scenarios where direct control of hardware resources and high performance optimization is required, while Golang is more suitable for scenarios where rapid development and high concurrency processing are required. 1.C's advantage lies in its close to hardware characteristics and high optimization capabilities, which are suitable for high-performance needs such as game development. 2.Golang's advantage lies in its concise syntax and natural concurrency support, which is suitable for high concurrency service development.

See all articles