Import cycle in tensorflow protobuf

Question content

I am trying to write client code to communicate with tensorflow server. I need golang protobufs compiled for tensorflow and tensorflow_serving. None of this comes easy, I did it through this. Basically, use buf to generate them. This is buf yaml:

version: v1
managed:
  enabled: true
  optimize_for: code_size

  # go
  go_package_prefix:
    default: "some/path"

plugins:
  - plugin: buf.build/protocolbuffers/go
    out: gen/proto/go

Run successfully, but running application log:

package command-line-arguments
    imports my-package/internal/infer
    imports my-package/internal/infer/tensorflow_serving/apis
    imports my-package/internal/infer/tensorflow/core/protobuf
    imports my-package/internal/infer/tensorflow/compiler/xla/stream_executor
    imports my-package/internal/infer/tensorflow/compiler/xla
    imports my-package/internal/infer/tensorflow/compiler/xla/service
    imports my-package/internal/infer/tensorflow/compiler/xla: import cycle not allowed

Please note that everything under tensorflow and tensorflow_serving is compiled directly from the original repository.

It surprises me that something as widely used as tensorflow should have an import cycle, but maybe it does. How can I solve this problem?

Correct answer

tl;dr

The root cause is that the repository https://www.php.cn/link/1a16abf2a3149fc7cd6083687cce01c2 really didn't organize the original files correctly (or at least didn't make it go-friendly).

The following two files cause an import loop in go (xla->xla/service->xla):

• tensorflow/compiler/xla/xla.proto

  • import "tensorflow/compiler/xla/service/hlo.proto"

• tensorflow/compiler/xla/service/hlo.proto

  • import "tensorflow/compiler/xla/xla_data.proto"

Since xla_data.proto does not import any other files, we can move it into its own package to break the import loop. We can do this using buf's override function. This is the final buf.gen.yaml file:

version: v1
managed:
  enabled: true
  go_package_prefix:
    default: example.com/mymodule/internal
  override:
    go_package:
      # move the generated xla_data.pb.go file into package xla/data to break the import cycle.
      tensorflow/compiler/xla/xla_data.proto: 'example.com/mymodule/internal/tensorflow/compiler/xla/data'
plugins:
  - name: go
    out: internal
    opt:
      - module=example.com/mymodule/internal

  - name: go-grpc
    out: internal
    opt:
      - module=example.com/mymodule/internal

Complete settings for compiling tensorflow proto files using buf

This is the final directory structure:

├── buf.gen.yaml
├── buf.work.yaml
├── buf.yaml
├── go.mod
├── go.sum
├── internal
│   ├── tensorflow
│   └── tensorflow_serving
└── testdata
    ├── serving
    └── tensorflow

buf.gen.yaml: See the "tl;dr" section.

buf.work.yaml：

version: v1
directories:
  - testdata/serving
  - testdata/tensorflow

buf.yaml：

version: v1
breaking:
  use:
    - file
lint:
  use:
    - default

This is my environment:

$ go version
go version go1.20.3 linux/amd64
$ buf --version
1.17.0
$ protoc --version
libprotoc 3.12.4
$ protoc-gen-go --version
protoc-gen-go v1.30.0
$ protoc-gen-go-grpc --version
protoc-gen-go-grpc 1.3.0
$ git version
git version 2.37.2

Now execute the following command in the root directory of this directory:

$ go mod init example.com/mymodule
$ go get google.golang.org/grpc
$ git clone https://www.php.cn/link/1a16abf2a3149fc7cd6083687cce01c2.git testdata/tensorflow
$ git clone https://github.com/tensorflow/serving.git testdata/serving
$ buf generate
$ go build ./...

Comments:

The
1. tensorflow repositories are cloned into the testdata directory so that go build will ignore them.
2. The installation program will generate files in the internal directory. You can modify the buf.gen.yaml files to place them wherever you want.
3. go build ./... will not report any errors. But I'm not sure if the generated file is valid.

The above is the detailed content of Import cycle in tensorflow protobuf. For more information, please follow other related articles on the PHP Chinese website!