html rendering function memory leak
The problem I'm facing is that even trying just 200 requests causes the program to occupy 6gb of the container's memory and ends up being oom kill. My idea is to extract all text nodes present in the html and then process them to extract their name, the html and the text of that tag. So, to generate html for a specific tag, I use the render function from golang.org/x/net/html. Where I provide strings.builder as io.writer to write the generated html. But for some reason the builder takes too much memory.
package main import ( "encoding/csv" "io" "log" "net/http" "strings" "golang.org/x/net/html" ) func main() { mux := http.NewServeMux() mux.HandleFunc("/data", GetData) if err := http.ListenAndServe(":8001", mux); err != nil { log.Println(err) } } type TagInfo struct { Tag string Name string Text string } // http.handler func GetData(w http.ResponseWriter, r *http.Request) { u := r.URL.Query().Get("url") doc, err := GetDoc(u) if err != nil { log.Println(err) w.WriteHeader(500) return } var buf strings.Builder data := Extract(doc, &buf) csvw := csv.NewWriter(io.Discard) for _, d := range data { csvw.Write([]string{d.Name, d.Tag, d.Text}) } } // fires request and get text/html func GetDoc(u string) (*html.Node, error) { res, err := http.Get(u) if err != nil { return nil, err } defer res.Body.Close() return html.Parse(res.Body) } func Extract(doc *html.Node, buf *strings.Builder) []TagInfo { var ( tags = make([]TagInfo, 0, 100) f func(*html.Node) ) f = func(n *html.Node) { if n.Type == html.TextNode { text := strings.TrimSpace(n.Data) if text != "" { parent := n.Parent tag := Render(parent, buf) tagInfo := TagInfo{ Tag: tag, Name: parent.Data, Text: n.Data, } tags = append(tags, tagInfo) } } for child := n.FirstChild; child != nil; child = child.NextSibling { f(child) } } f(doc) return tags } // Render the html around the tag // if node is text then pass the // parent node paramter in function func Render(n *html.Node, buf *strings.Builder) string { defer buf.Reset() if err := html.Render(buf, n); err != nil { log.Println(err) return "" } return buf.String() }
If you want a specific list of URLs, here it is. I made about 60 requests at one time.
I tried using bytes.buffer bytes.buffer
and sync.pool
but both have the same problem. Using pprof
I noticed that the writestring method of
strings.builder caused a lot of memory usage.
Correct answer
So the basic question here is to accept any content-type
, which is not acceptable in terms of crawling, most Websites need to send text/html
.
The problem is that even if the url sends anything that does not represent html data golang.org/x/net/html
it is still accepted without throwing an error.
Let's take an example where application/pdf
is returned, then the body will contain html.Parse
the binary data of the parsed pdf and no error will be returned, this is Weird-behavior library for scraping/crawling accepted binary data.
The solution is: Check the response headers, if only the data is html, then continue, otherwise there will be ambiguity or higher memory usage (maybe lower), but we can't predict what will happen occur.
The above is the detailed content of html rendering function memory leak. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics



OpenSSL, as an open source library widely used in secure communications, provides encryption algorithms, keys and certificate management functions. However, there are some known security vulnerabilities in its historical version, some of which are extremely harmful. This article will focus on common vulnerabilities and response measures for OpenSSL in Debian systems. DebianOpenSSL known vulnerabilities: OpenSSL has experienced several serious vulnerabilities, such as: Heart Bleeding Vulnerability (CVE-2014-0160): This vulnerability affects OpenSSL 1.0.1 to 1.0.1f and 1.0.2 to 1.0.2 beta versions. An attacker can use this vulnerability to unauthorized read sensitive information on the server, including encryption keys, etc.

The article discusses writing unit tests in Go, covering best practices, mocking techniques, and tools for efficient test management.

The article explains how to use the pprof tool for analyzing Go performance, including enabling profiling, collecting data, and identifying common bottlenecks like CPU and memory issues.Character count: 159

The library used for floating-point number operation in Go language introduces how to ensure the accuracy is...

Queue threading problem in Go crawler Colly explores the problem of using the Colly crawler library in Go language, developers often encounter problems with threads and request queues. �...

This article introduces a variety of methods and tools to monitor PostgreSQL databases under the Debian system, helping you to fully grasp database performance monitoring. 1. Use PostgreSQL to build-in monitoring view PostgreSQL itself provides multiple views for monitoring database activities: pg_stat_activity: displays database activities in real time, including connections, queries, transactions and other information. pg_stat_replication: Monitors replication status, especially suitable for stream replication clusters. pg_stat_database: Provides database statistics, such as database size, transaction commit/rollback times and other key indicators. 2. Use log analysis tool pgBadg

Backend learning path: The exploration journey from front-end to back-end As a back-end beginner who transforms from front-end development, you already have the foundation of nodejs,...

The problem of using RedisStream to implement message queues in Go language is using Go language and Redis...
