Go에서 LLM으로 전송된 토큰 수 계산(2부)-Golang-php.cn

Counting the number of Tokens sent to a LLM in Go (part 2)

소개

이것은 선택한 텍스트를 기반으로 사용자가 LLM에 보내는 토큰 수를 결정하기 위한 Go 애플리케이션 작성 작업의 두 번째 부분입니다.

이전 기사에서 Golang으로만 작성된 것을 만들고 싶다고 언급했는데, 제가 살펴본 Github 저장소 중에서 go-hggingface라는 것이 정말 좋은 것 같습니다. 코드는 매우 새로운 것 같지만 저에게는 "일종" 작동합니다.

구현

먼저 코드는 Hugginface에 액세스하여 LLM과 함께 사용되는 모든 "토크나이저" 목록을 가져오므로 사용자는 HF 토큰을 보유해야 합니다. 그래서 표시된 대로 토큰을 .env 파일에 넣었습니다.

HF_TOKEN="your-huggingface-token"

로그인 후 복사

그런 다음 다음 페이지(https://github.com/gomlx/go-huggingface?tab=readme-ov-file)에 제공된 예제를 사용하여 이를 중심으로 나만의 코드를 구축했습니다.

package main

import (
 "bytes"
 "fmt"
 "log"
 "os"
 "os/exec"
 "runtime"

 "github.com/gomlx/go-huggingface/hub"
 "github.com/gomlx/go-huggingface/tokenizers"

 "github.com/joho/godotenv"
 "github.com/sqweek/dialog"

 "fyne.io/fyne/v2"
 "fyne.io/fyne/v2/app"
 "fyne.io/fyne/v2/container"
 "fyne.io/fyne/v2/widget"
 //"github.com/inancgumus/scree"
)

var (
 // Model IDs we use for testing.
 hfModelIDs = []string{
  "ibm-granite/granite-3.1-8b-instruct",
  "meta-llama/Llama-3.3-70B-Instruct",
  "mistralai/Mistral-7B-Instruct-v0.3",
  "google/gemma-2-2b-it",
  "sentence-transformers/all-MiniLM-L6-v2",
  "protectai/deberta-v3-base-zeroshot-v1-onnx",
  "KnightsAnalytics/distilbert-base-uncased-finetuned-sst-2-english",
  "KnightsAnalytics/distilbert-NER",
  "SamLowe/roberta-base-go_emotions-onnx",
 }
)

func runCmd(name string, arg ...string) {
 cmd := exec.Command(name, arg...)
 cmd.Stdout = os.Stdout
 cmd.Run()
}

func ClearTerminal() {
 switch runtime.GOOS {
 case "darwin":
  runCmd("clear")
 case "linux":
  runCmd("clear")
 case "windows":
  runCmd("cmd", "/c", "cls")
 default:
  runCmd("clear")
 }
}

func FileSelectionDialog() string {
 // Open a file dialog box and let the user select a text file
 filePath, err := dialog.File().Filter("Text Files", "txt").Load()
 if err != nil {
  if err.Error() == "Cancelled" {
   fmt.Println("File selection was cancelled.")
  }
  log.Fatalf("Error selecting file: %v", err)
 }

 // Output the selected file name
 fmt.Printf("Selected file: %s\n", filePath)
 return filePath
}

func main() {

 var filePath string

 // read the '.env' file
 err := godotenv.Load()
 if err != nil {
  log.Fatal("Error loading .env file")
 }
 // get the value of the 'HF_TOKEN' environment variable
 hfAuthToken := os.Getenv("HF_TOKEN")
 if hfAuthToken == "" {
  log.Fatal("HF_TOKEN environment variable is not set")
 }

 // to display a list of LLMs to determine the # of tokens later on regarding the given text
 var llm string = ""
 var modelID string = ""
 myApp := app.New()
 myWindow := myApp.NewWindow("Select a LLM in the list")
 items := hfModelIDs
 // Label to display the selected item
 selectedItem := widget.NewLabel("Selected LLM: None")
 // Create a list widget
 list := widget.NewList(
  func() int {
   // Return the number of items in the list
   return len(items)
  },
  func() fyne.CanvasObject {
   // Template for each list item
   return widget.NewLabel("Template")
  },
  func(id widget.ListItemID, obj fyne.CanvasObject) {
   // Update the template with the actual data
   obj.(*widget.Label).SetText(items[id])
  },
 )
 // Handle list item selection
 list.OnSelected = func(id widget.ListItemID) {
  selectedItem.SetText("Selected LLM:" + items[id])
  llm = items[id]
 }

 // Layout with the list and selected item label
 content := container.NewVBox(
  list,
  selectedItem,
 )

 // Set the content of the window
 myWindow.SetContent(content)
 myWindow.Resize(fyne.NewSize(300, 400))
 myWindow.ShowAndRun()
 ClearTerminal()
 fmt.Printf("Selected LLM: %s\n", llm)
 //////

 //List files for the selected model
 for _, modelID := range hfModelIDs {
  if modelID == llm {
   fmt.Printf("\n%s:\n", modelID)
   repo := hub.New(modelID).WithAuth(hfAuthToken)
   for fileName, err := range repo.IterFileNames() {
    if err != nil {
     panic(err)
    }
    fmt.Printf("fileName\t%s\n", fileName)
    fmt.Printf("repo\t%s\n", repo)
    fmt.Printf("modelID\t%s\n", modelID)
   }
  }
 }

 //List tokenizer classes for the selected model
 for _, modelID := range hfModelIDs {
  if modelID == llm {
   fmt.Printf("\n%s:\n", modelID)
   repo := hub.New(modelID).WithAuth(hfAuthToken)
   fmt.Printf("\trepo=%s\n", repo)
   config, err := tokenizers.GetConfig(repo)
   if err != nil {
    panic(err)
   }
   fmt.Printf("\ttokenizer_class=%s\n", config.TokenizerClass)
  }
 }

 // Models URL -> "https://huggingface.co/api/models"
 repo := hub.New(modelID).WithAuth(hfAuthToken)
 tokenizer, err := tokenizers.New(repo)
 if err != nil {
  panic(err)
 }

 // call file selection dialogbox
 filePath = FileSelectionDialog()

 // Open the file
 filerc, err := os.Open(filePath)
 if err != nil {
  fmt.Printf("Error opening file: %v\n", err)
  return
 }
 defer filerc.Close()

 // Put the text file content into a buffer and convert it to a string.
 buf := new(bytes.Buffer)
 buf.ReadFrom(filerc)
 sentence := buf.String()

 tokens := tokenizer.Encode(sentence)
 fmt.Println("Sentence:\n", sentence)

 fmt.Printf("Tokens:  \t%v\n", tokens)
}

로그인 후 복사

"hfModelIDs"의 "var" 섹션에 IBM의 Granite, Meta의 LLama 및 Mistral 모델과 같은 몇 가지 새로운 참조를 추가했습니다.

Huggingface 토큰은 Go 코드 내에서도 직접 소스로 제공되고 읽혀집니다.

LLM 목록(결국 변경 예정)을 표시하는 대화 상자, 파일의 텍스트를 추가하는 대화 상자(그런 종류의 내용이 마음에 드나요?), 지우고 몇 줄의 코드를 추가했습니다. 화면 청소?!

입력 텍스트는 다음과 같습니다

The popularity of the Rust language continues to explode; yet, many critical codebases remain authored in C, and cannot be realistically rewritten by hand. Automatically translating C to Rust is thus an appealing course of action. Several works have gone down this path, handling an ever-increasing subset of C through a variety of Rust features, such as unsafe. While the prospect of automation is appealing, producing code that relies on unsafe negates the memory safety guarantees offered by Rust, and therefore the main advantages of porting existing codebases to memory-safe languages.
We instead explore a different path, and explore what it would take to translate C to safe Rust; that is, to produce code that is trivially memory safe, because it abides by Rust's type system without caveats. Our work sports several original contributions: a type-directed translation from (a subset of) C to safe Rust; a novel static analysis based on "split trees" that allows expressing C's pointer arithmetic using Rust's slices and splitting operations; an analysis that infers exactly which borrows need to be mutable; and a compilation strategy for C's struct types that is compatible with Rust's distinction between non-owned and owned allocations.
We apply our methodology to existing formally verified C codebases: the HACL* cryptographic library, and binary parsers and serializers from EverParse, and show that the subset of C we support is sufficient to translate both applications to safe Rust. Our evaluation shows that for the few places that do violate Rust's aliasing discipline, automated, surgical rewrites suffice; and that the few strategic copies we insert have a negligible performance impact. Of particular note, the application of our approach to HACL* results in a 80,000 line verified cryptographic library, written in pure Rust, that implements all modern algorithms - the first of its kind.

로그인 후 복사

테스트
실행된 코드는 원하는 LLM을 선택할 수 있는 대화상자 bx를 표시합니다.

Counting the number of Tokens sent to a LLM in Go (part 2)

모든 것이 순조롭게 진행되면 다음 단계는 "tokenizer" 파일을 로컬로 다운로드하는 것입니다(Github 저장소의 설명 참조). 그러면 평가할 콘텐츠가 포함된 텍스트 파일을 선택하라는 대화 상자가 표시됩니다. 토큰 개수.

지금까지 Meta LLama 및 Google “google/gemma-2–2b-it” 모델에 대한 액세스 권한을 요청했으며 액세스 권한 부여를 기다리고 있습니다.

google/gemma-2-2b-it:
        repo=google/gemma-2-2b-it
panic: request for metadata from "https://huggingface.co/google/gemma-2-2b-it/resolve/299a8560bedf22ed1c72a8a11e7dce4a7f9f51f8/tokenizer_config.json" failed with the following message: "403 Forbidden"

로그인 후 복사

Counting the number of Tokens sent to a LLM in Go (part 2)