I have several JSONs with the same structure. Their objects are arrays of objects. How can I append these arrays into an array?

Question content

What I want to do:

I want to send multiple get requests to this url: https://catalog.wb.ru/brands/m/catalog?page=1&limit=300&brand=5786&dest=-1257786&sort=pricedown Then collect all the data inside the "product" object. The value of key "page" is automatically incremented to get data for all pages.

Actually, I'm not really sure if I really need to write a json to send it to the frontend. Maybe it would be better to send a different request when I receive a new response in the for loop?

what did I do:

Made the correct structure. With a single request everything works fine.

Created requestbodybytes []byte and productsbytes []byte so that they can be combined with []bytes## in ioutil.readall # Append together. Printing the length of requestbodybytes I see that it expands with each request, but after I unmarshal it I see empty structures in the output.

I know this is happening because with every request I receive a new json of

type response. But what if I need a slice of product structs consisting of type response 'product' objects in multiple json s?

Note: Need to initialize

requestbodybytes inside the for loop to use it to stop sending requests because the server will give a 200 code and empty json when there is no information on the page.

Thank you in advance!

const URL = "https://catalog.wb.ru/brands/m/catalog?page=%d&limit=300&brand=5786&dest=-1257786&sort=pricedown"

type Response struct {
    Data struct {
        Products []Product `json:"products"`
    } `json:"data"`
}

type Product struct {
    ID     int     `json:"id"`
    Name   string  `json:"name"`
    Price  int     `json:"priceU"`
    Rating float32 `json:"reviewRating"`
    Sale   int     `json:"sale"`
    New    bool    `json:"isNew"`
}

func main() {
    var response Response
    var products Response //Also tried to make it []Response
    var ProductsBytes []byte

    for i := 1; ; i++ {
        resp, err := http.Get(fmt.Sprintf(URL, i))
        if err != nil {
            fmt.Printf("#1 Error: %s", err)
        }
        defer resp.Body.Close()

        bytes, err := ioutil.ReadAll(resp.Body)
        var requestBodyBytes []byte
        requestBodyBytes = append(requestBodyBytes, bytes...)
        ProductsBytes = append(ProductsBytes, bytes...)

        json.Unmarshal(requestBodyBytes, &response)

        fmt.Println(resp.Status)
        fmt.Printf("\nSlice from page #%d\nLength of bytes: %d\n", i, len(bytes))
        fmt.Printf("Length of finalResult: %d\n", len(requestBodyBytes))
        if len(response.Data.Products) == 0 {
            fmt.Println("There's no more data")
            break
        }
    }

    json.Unmarshal(ProductsBytes, &products)

    fmt.Println(response)
    fmt.Println(products)
    fmt.Println(len(products))
}

Correct answer

There is no reason to collect all raw response bytes. Just unmarshal each response individually and append each page's products to some slice that contains all products. Also, calling

defer resp.body.close() inside a loop may not be what you want. Deferred statements are only executed after the loop ends, so the connection cannot be reused for requests. Extracting the loop body into its own function makes this clearer:

package main

import (
    "encoding/json"
    "errors"
    "fmt"
    "log"
    "net/http"
)

const URL = "https://catalog.wb.ru/brands/m/catalog?page=%d&limit=300&brand=5786&dest=-1257786&sort=pricedown"

type Response struct {
    Data struct {
        Products []Product `json:"products"`
    } `json:"data"`
}

type Product struct {
    ID     int     `json:"id"`
    Name   string  `json:"name"`
    Price  int     `json:"priceU"`
    Rating float32 `json:"reviewRating"`
    Sale   int     `json:"sale"`
    New    bool    `json:"isNew"`
}

func main() {
    var allProducts []Product

    for i := 1; ; i++ {
        page, err := fetchPage(i)
        if err != nil {
            log.Fatal(err) // TODO
        }

        allProducts = append(allProducts, page...)

        if len(page) == 0 {
            break
        }
    }

    fmt.Println(allProducts)
    fmt.Println(len(allProducts))
}

func fetchPage(i int) ([]Product, error) {
    resp, err := http.Get(fmt.Sprintf(URL, i))
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200 {
        return nil, errors.New(resp.Status)
    }

    var response Response
    err = json.NewDecoder(resp.Body).Decode(&response)
    if err != nil {
        return nil, err
    }

    return response.Data.Products, nil
}

The above is the detailed content of I have several JSONs with the same structure. Their objects are arrays of objects. How can I append these arrays into an array?. For more information, please follow other related articles on the PHP Chinese website!