Have you ever needed to modify unstructured JSON data in Go? Maybe you’ve had to delete all blacklisted fields, rename keys from camelCase to snake_case, or convert all number ids to strings because JavaScript does not like int64? If your solution has been to unmarshal everything into a map[string]any using encoding/json and then marshal it back... well, let’s face it, that’s far from efficient!
What if you could loop through the JSON data, grab the path of each item, and decide exactly what to do with it on the fly?
Yes! I have a good news! With the new iterator feature in Go 1.23, there’s a better way to iterate and manipulate JSON. Meet ezpkg.io/iter.json — your powerful and efficient companion for working with JSON in Go.
Given that we have an alice.json file:
{ "name": "Alice", "age": 24, "scores": [9, 10, 8], "address": { "city": "The Sun", "zip": 10101 } }
First, let's use for range Parse() to iterate over the JSON file, then print the path, key, token, and level of each item. See examples/01.iter.
package main import ( "fmt" "ezpkg.io/errorz" iterjson "ezpkg.io/iter.json" ) func main() { data := `{"name": "Alice", "age": 24, "scores": [9, 10, 8], "address": {"city": "The Sun", "zip": 10101}}` // ?Example: iterate over json fmt.Printf("| %12v | %10v | %10v |%v|\n", "PATH", "KEY", "TOKEN", "LVL") fmt.Println("| ------------ | ---------- | ---------- | - |") for item, err := range iterjson.Parse([]byte(data)) { errorz.MustZ(err) fmt.Printf("| %12v | %10v | %10v | %v |\n", item.GetPathString(), item.Key, item.Token, item.Level) } }
The code will output:
| PATH | KEY | TOKEN |LVL| | ------------ | ---------- | ---------- | - | | | | { | 0 | | name | "name" | "Alice" | 1 | | age | "age" | 24 | 1 | | scores | "scores" | [ | 1 | | scores.0 | | 9 | 2 | | scores.1 | | 10 | 2 | | scores.2 | | 8 | 2 | | scores | | ] | 1 | | address | "address" | { | 1 | | address.city | "city" | "The Sun" | 2 | | address.zip | "zip" | 10101 | 2 | | address | | } | 1 | | | | } | 0 |
Use Builder to build a JSON data. It accepts optional arguments for indentation. See examples/02.builder.
b := iterjson.NewBuilder("", " ") // open an object b.Add("", iterjson.TokenObjectOpen) // add a few fields b.Add("name", "Alice") b.Add("age", 22) b.Add("email", "alice@example.com") b.Add("phone", "(+84) 123-456-789") // open an array b.Add("languages", iterjson.TokenArrayOpen) b.Add("", "English") b.Add("", "Vietnamese") b.Add("", iterjson.TokenArrayClose) // close the array // accept any type that can marshal to json b.Add("address", Address{ HouseNumber: 42, Street: "Ly Thuong Kiet", City: "Ha Noi", Country: "Vietnam", }) // accept []byte as raw json b.Add("pets", []byte(`[{"type":"cat","name":"Kitty","age":2},{"type":"dog","name":"Yummy","age":3}]`)) // close the object b.Add("", iterjson.TokenObjectClose) out := errorz.Must(b.Bytes()) fmt.Printf("\n--- build json ---\n%s\n", out)
Which will output the JSON with indentation:
{ "name": "Alice", "age": 22, "email": "alice@example.com", "phone": "(+84) 123-456-789", "languages": [ "English", "Vietnamese" ], "address": {"house_number":42,"street":"Ly Thuong Kiet","city":"Ha Noi","country":"Vietnam"}, "pets": [ { "type": "cat", "name": "Kitty", "age": 2 }, { "type": "dog", "name": "Yummy", "age": 3 } ] }
You can reconstruct or format a JSON data by sending its key and values to a Builder. See examples/03.reformat.
{ // ?Example: minify json b := iterjson.NewBuilder("", "") for item, err := range iterjson.Parse(data) { errorz.MustZ(err) b.AddRaw(item.Key, item.Token) } out := errorz.Must(b.Bytes()) fmt.Printf("\n--- minify ---\n%s\n----------\n", out) } { // ?Example: format json b := iterjson.NewBuilder("? ", "\t") for item, err := range iterjson.Parse(data) { errorz.MustZ(err) b.AddRaw(item.Key, item.Token) } out := errorz.Must(b.Bytes()) fmt.Printf("\n--- reformat ---\n%s\n----------\n", out) }
The first example minifies the JSON while the second example formats it with prefix "?" on each line.
--- minify --- {"name":"Alice","age":24,"scores":[9,10,8],"address":{"city":"The Sun","zip":10101}} ---------- --- reformat --- ? { ? "name": "Alice", ? "age": 24, ? "scores": [ ? 9, ? 10, ? 8 ? ], ? "address": { ? "city": "The Sun", ? "zip": 10101 ? } ? } ----------
In this example, we add line numbers to the JSON output, by adding a b.WriteNewline() before the fmt.Fprintf() call. See examples/04.line_number.
// ?Example: print with line number i := 0 b := iterjson.NewBuilder("", " ") for item, err := range iterjson.Parse(data) { i++ errorz.MustZ(err) b.WriteNewline(item.Token.Type()) // ? add line number fmt.Fprintf(b, "%3d ", i) b.Add(item.Key, item.Token) } out := errorz.Must(b.Bytes()) fmt.Printf("\n--- line number ---\n%s\n----------\n", out)
This will output:
1 { 2 "name": "Alice", 3 "age": 24, 4 "scores": [ 5 9, 6 10, 7 8 8 ], 9 "address": { 10 "city": "The Sun", 11 "zip": 10101 12 } 13 }
By putting a fmt.Fprintf(comment) between b.WriteComma() and b.WriteNewline(), you can add a comment to the end of each line. See examples/05.comment.
i, newlineIdx, maxIdx := 0, 0, 30 b := iterjson.NewBuilder("", " ") for item, err := range iterjson.Parse(data) { errorz.MustZ(err) b.WriteComma(item.Token.Type()) // ? add comment if i > 0 { length := b.Len() - newlineIdx fmt.Fprint(b, strings.Repeat(" ", maxIdx-length)) fmt.Fprintf(b, "// %2d", i) } i++ b.WriteNewline(item.Token.Type()) newlineIdx = b.Len() // save the newline index b.Add(item.Key, item.Token) } length := b.Len() - newlineIdx fmt.Fprint(b, strings.Repeat(" ", maxIdx-length)) fmt.Fprintf(b, "// %2d", i) out := errorz.Must(b.Bytes()) fmt.Printf("\n--- comment ---\n%s\n----------\n", out)
This will output:
{ // 1 "name": "Alice", // 2 "age": 24, // 3 "scores": [ // 4 9, // 5 10, // 6 8 // 7 ], // 8 "address": { // 9 "city": "The Sun", // 10 "zip": 10101 // 11 } // 12 } // 13
There are item.GetPathString() and item.GetRawPath() to get the path of the current item. You can use them to filter the JSON data. See examples/06.filter_print.
Example with item.GetPathString() and regexp:
fmt.Printf("\n--- filter: GetPathString() ---\n") i := 0 for item, err := range iterjson.Parse(data) { i++ errorz.MustZ(err) path := item.GetPathString() switch { case path == "name", strings.Contains(path, "address"): // continue default: continue } // ? print with line number fmt.Printf("%2d %20s . %s\n", i, item.Token, item.GetPath()) }
Example with item.GetRawPath() and path.Match():
fmt.Printf("\n--- filter: GetRawPath() ---\n") i := 0 for item, err := range iterjson.Parse(data) { i++ errorz.MustZ(err) path := item.GetRawPath() switch { case path.Match("name"), path.Contains("address"): // continue default: continue } // ? print with line number fmt.Printf("%2d %20s . %s\n", i, item.Token, item.GetPath()) }
Both examples will output:
{ "name": "Alice", "age": 24, "scores": [9, 10, 8], "address": { "city": "The Sun", "zip": 10101 } }
By combining the Builder with the option SetSkipEmptyStructures(false) and the filtering logic, you can filter the JSON data and return a new JSON. See examples/07.filter_json
package main import ( "fmt" "ezpkg.io/errorz" iterjson "ezpkg.io/iter.json" ) func main() { data := `{"name": "Alice", "age": 24, "scores": [9, 10, 8], "address": {"city": "The Sun", "zip": 10101}}` // ?Example: iterate over json fmt.Printf("| %12v | %10v | %10v |%v|\n", "PATH", "KEY", "TOKEN", "LVL") fmt.Println("| ------------ | ---------- | ---------- | - |") for item, err := range iterjson.Parse([]byte(data)) { errorz.MustZ(err) fmt.Printf("| %12v | %10v | %10v | %v |\n", item.GetPathString(), item.Key, item.Token, item.Level) } }
This example will return a new JSON with only the filtered fields:
| PATH | KEY | TOKEN |LVL| | ------------ | ---------- | ---------- | - | | | | { | 0 | | name | "name" | "Alice" | 1 | | age | "age" | 24 | 1 | | scores | "scores" | [ | 1 | | scores.0 | | 9 | 2 | | scores.1 | | 10 | 2 | | scores.2 | | 8 | 2 | | scores | | ] | 1 | | address | "address" | { | 1 | | address.city | "city" | "The Sun" | 2 | | address.zip | "zip" | 10101 | 2 | | address | | } | 1 | | | | } | 0 |
This is an example for editing values in a JSON data. Assume that we are using number ids for our API. The ids are too big and JavaScript can't handle them. We need to convert them to strings. See examples/08.number_id and order.json.
Iterate over the JSON data, find all _id fields and convert the number ids to strings:
b := iterjson.NewBuilder("", " ") // open an object b.Add("", iterjson.TokenObjectOpen) // add a few fields b.Add("name", "Alice") b.Add("age", 22) b.Add("email", "alice@example.com") b.Add("phone", "(+84) 123-456-789") // open an array b.Add("languages", iterjson.TokenArrayOpen) b.Add("", "English") b.Add("", "Vietnamese") b.Add("", iterjson.TokenArrayClose) // close the array // accept any type that can marshal to json b.Add("address", Address{ HouseNumber: 42, Street: "Ly Thuong Kiet", City: "Ha Noi", Country: "Vietnam", }) // accept []byte as raw json b.Add("pets", []byte(`[{"type":"cat","name":"Kitty","age":2},{"type":"dog","name":"Yummy","age":3}]`)) // close the object b.Add("", iterjson.TokenObjectClose) out := errorz.Must(b.Bytes()) fmt.Printf("\n--- build json ---\n%s\n", out)
This will add quotes to the number ids:
{ "name": "Alice", "age": 22, "email": "alice@example.com", "phone": "(+84) 123-456-789", "languages": [ "English", "Vietnamese" ], "address": {"house_number":42,"street":"Ly Thuong Kiet","city":"Ha Noi","country":"Vietnam"}, "pets": [ { "type": "cat", "name": "Kitty", "age": 2 }, { "type": "dog", "name": "Yummy", "age": 3 } ] }
The ezpkg.io/iter.json package empowers Go developers to handle JSON data with precision and efficiency. Whether you need to iterate through complex JSON structures, build new JSON objects dynamically, format or minify data, filter specific fields, or even transform values, iter.json offers a flexible and powerful solution.
I’m excited to share this package with the community as a tool for effective JSON manipulation without the need for fully parsing the data. While it’s still in early development and there’s room for more features, it already works well for many common use cases.
If you have specific requirements or ideas for improvement, feel free to reach out — I’d love to hear your feedback and help support your use cases! ?
I'm Oliver Nguyen . A software engineer working with Go and JS. I enjoy learning and seeing a better version of myself each day. Occasionally spin off new open source projects. Share knowledge and thoughts during my journey.
The post is also published at olivernguyen.io.
The above is the detailed content of iter.json: A Powerful and Efficient Way to Iterate and Manipulate JSON in Go. For more information, please follow other related articles on the PHP Chinese website!