FAISS CGO throws 'undefined symbols for architecture arm64' when using libfaiss

使用 libfaiss_c.a 时，FAISS CGO 抛出“架构 arm64 的未定义符号”

php editor Zimo encountered a problem when using libfaiss_c.a, that is, FAISS CGO threw an "undefined symbol for architecture arm64" error. This error may be caused by missing some symbols when using libfaiss_c.a on 64-bit ARM architecture. To solve this problem, you can try to update or recompile libfaiss_c.a to make sure it is the version suitable for the arm64 architecture. Additionally, you can check that the correct compile and link options are used. I hope these solutions can help developers who encounter similar problems.

Question content

I'm trying to write a standalone go-faiss wrapper to call FAISS's KMEANS function. Warehouse: https://www.php.cn/link/4e4faae72b1c3cbd446a70e89e59d8fc

This approach is inspired by https://github.com/andyalm/faissmask/tree/master/FaissMask/runtimes

FAISS static librarylibfaiss_c.aBuild using -DBUILD_SHARED_LIBS=OFF

When I link this library with CGO, I get

error log

# faiss-go/pkg/ivf.test
/usr/local/go/pkg/tool/darwin_arm64/link: running clang failed: exit status 1
Undefined symbols for architecture arm64:
  "faiss::Clustering::Clustering(int, int)", referenced from:
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
  "faiss::Clustering::Clustering(int, int, faiss::ClusteringParameters const&)", referenced from:
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
  "faiss::kmeans_clustering(unsigned long, unsigned long, unsigned long, float const*, float*)", referenced from:
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "faiss::ClusteringParameters::ClusteringParameters()", referenced from:
      _faiss_ClusteringParameters_init in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
  "std::exception_ptr::exception_ptr(std::exception_ptr const&)", referenced from:
      _faiss_get_last_error in libfaiss_c.a(error_impl.cpp.o)
  "std::exception_ptr::~exception_ptr()", referenced from:
      _faiss_get_last_error in libfaiss_c.a(error_impl.cpp.o)
      thread-local wrapper routine for faiss_last_exception in libfaiss_c.a(error_impl.cpp.o)
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering.cold.1 in libfaiss_c.a(Clustering_c.cpp.o)
      ...
  "std::exception_ptr::operator=(std::exception_ptr const&)", referenced from:
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering.cold.1 in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering.cold.2 in libfaiss_c.a(Clustering_c.cpp.o)
  "std::runtime_error::runtime_error(char const*)", referenced from:
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "std::runtime_error::runtime_error(std::runtime_error const&)", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::runtime_error>(std::runtime_error) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "std::runtime_error::~runtime_error()", referenced from:
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::runtime_error>(std::runtime_error) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering.cold.2 in libfaiss_c.a(Clustering_c.cpp.o)
  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      faiss::FaissException::FaissException(faiss::FaissException const&) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "std::exception::~exception()", referenced from:
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::exception>(std::exception) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering.cold.1 in libfaiss_c.a(Clustering_c.cpp.o)
  "std::exception::~exception()", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      faiss::FaissException::FaissException(faiss::FaissException const&) in libfaiss_c.a(Clustering_c.cpp.o)
      faiss::FaissException::~FaissException() in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "std::current_exception()", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::runtime_error>(std::runtime_error) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::exception>(std::exception) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "std::rethrow_exception(std::exception_ptr)", referenced from:
      _faiss_get_last_error in libfaiss_c.a(error_impl.cpp.o)
  "std::terminate()", referenced from:
      ___clang_call_terminate in libfaiss_c.a(Clustering_c.cpp.o)
  "typeinfo for faiss::FaissException", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
      GCC_except_table21 in libfaiss_c.a(Clustering_c.cpp.o)
      GCC_except_table28 in libfaiss_c.a(Clustering_c.cpp.o)
      GCC_except_table29 in libfaiss_c.a(Clustering_c.cpp.o)
      GCC_except_table31 in libfaiss_c.a(Clustering_c.cpp.o)
      ...
  "typeinfo for std::runtime_error", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::runtime_error>(std::runtime_error) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "typeinfo for std::exception", referenced from:
      GCC_except_table0 in libfaiss_c.a(error_impl.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::exception>(std::exception) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
      GCC_except_table21 in libfaiss_c.a(Clustering_c.cpp.o)
      GCC_except_table28 in libfaiss_c.a(Clustering_c.cpp.o)
      GCC_except_table29 in libfaiss_c.a(Clustering_c.cpp.o)
      ...
  "vtable for faiss::FaissException", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      faiss::FaissException::FaissException(faiss::FaissException const&) in libfaiss_c.a(Clustering_c.cpp.o)
      faiss::FaissException::~FaissException() in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for std::exception", referenced from:
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::exception>(std::exception) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "operator delete(void*)", referenced from:
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      faiss::FaissException::~FaissException() in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "operator new(unsigned long)", referenced from:
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
  "___cxa_allocate_exception", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::runtime_error>(std::runtime_error) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::exception>(std::exception) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "___cxa_begin_catch", referenced from:
      _faiss_get_last_error in libfaiss_c.a(error_impl.cpp.o)
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::runtime_error>(std::runtime_error) in libfaiss_c.a(Clustering_c.cpp.o)
      ___clang_call_terminate in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::exception>(std::exception) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      ...
  "___cxa_end_catch", referenced from:
      _faiss_get_last_error in libfaiss_c.a(error_impl.cpp.o)
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::runtime_error>(std::runtime_error) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::exception>(std::exception) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      ...
  "___cxa_free_exception", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "___cxa_throw", referenced from:
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::runtime_error>(std::runtime_error) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::exception>(std::exception) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_train in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_kmeans_clustering in libfaiss_c.a(Clustering_c.cpp.o)
  "___gxx_personality_v0", referenced from:
      _faiss_get_last_error in libfaiss_c.a(error_impl.cpp.o)
      _faiss_Clustering_new in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::runtime_error>(std::runtime_error) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<std::exception>(std::exception) in libfaiss_c.a(Clustering_c.cpp.o)
      std::exception_ptr std::make_exception_ptr[abi:v15006]<faiss::FaissException>(faiss::FaissException) in libfaiss_c.a(Clustering_c.cpp.o)
      faiss::FaissException::FaissException(faiss::FaissException const&) in libfaiss_c.a(Clustering_c.cpp.o)
      _faiss_Clustering_new_with_params in libfaiss_c.a(Clustering_c.cpp.o)
      ...
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Copy after login

I'm using these CGO headers

/*
#cgo darwin LDFLAGS: -L${SRCDIR}/../../cgo/thirdparty/runtimes/osx-arm64/native -lfaiss_c

#include <stdlib.h>
#include <faiss/c_api/Clustering_c.h>
#include <faiss/c_api/impl/AuxIndexStructures_c.h>
#include <faiss/c_api/index_factory_c.h>
#include <faiss/c_api/error_c.h>
*/

Copy after login

However, when we build

libfaiss_c.dylib using -DBUILD_SHARED_LIBS=ON and add libfaiss_c.dylib to /usr/local/lib This CGO code works fine. The CGO header now looks like this

/*
#cgo LDFLAGS: -lfaiss_c

#include <stdlib.h>
#include <faiss/c_api/Clustering_c.h>
#include <faiss/c_api/impl/AuxIndexStructures_c.h>
#include <faiss/c_api/index_factory_c.h>
#include <faiss/c_api/error_c.h>
*/

Copy after login

Please help me link

*.a files with CGO from custom path.

Update 1

When I have these files in

/usr/local/lib

Test run

These files written to

/usr/local/lib are created using the following make file

GIT_COMMAND ?= git
FAISS_COMMIT ?= d87888b13e7eb339bb9c45825e9d20def6665171

all: faiss

.PHONY: faiss
faiss:
    if [ -e faiss ]; then \
        cd faiss && ${GIT_COMMAND} fetch origin && git reset --hard  $(FAISS_COMMIT); \
    else \
        ${GIT_COMMAND} clone https://github.com/facebookresearch/faiss.git && cd faiss && git reset --hard  $(FAISS_COMMIT); \
    fi
    cd faiss && \
        export CMAKE_PREFIX_PATH=/opt/homebrew/opt/openblas:/opt/homebrew/opt/libomp:/opt/homebrew && \
        cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON -DFAISS_ENABLE_PYTHON=OFF . && \
        make -C build && \
        sudo make -C build install && \
        sudo cp build/c_api/libfaiss_c.dylib /usr/local/lib/libfaiss_c.dylib

Copy after login

Update 2

For go-rocks database library, https://github.com/linxGnu/grocksdb#build

They also recommend installing/building the rocksdb library locally and linking it with the cgo interface.

Is this method of independently building the static library "*.a" and providing it as an independent library valid?

Update 3

Go version

$ go version
go version go1.20 darwin/arm64

Copy after login

Operating system and processor architecture?

<p>$ go env
GO111MODULE="on"
GOARCH="arm64"
GOBIN=""
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/me/GolandProjects/0learning/faiss-cgo-kmeans/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"

Copy after login

Update 4

refer to: https://stackoverflow.com/a/63321095/1609570

<code>➜  native git:(master) otool -L libfaiss_c.a
Archive : libfaiss_c.a
libfaiss_c.a(AutoTune_c.cpp.o):
libfaiss_c.a(Clustering_c.cpp.o):
libfaiss_c.a(IndexFlat_c.cpp.o):
libfaiss_c.a(IndexIVFFlat_c.cpp.o):
libfaiss_c.a(IndexIVF_c.cpp.o):
libfaiss_c.a(IndexLSH_c.cpp.o):
libfaiss_c.a(IndexPreTransform_c.cpp.o):
libfaiss_c.a(VectorTransform_c.cpp.o):
libfaiss_c.a(IndexShards_c.cpp.o):
libfaiss_c.a(IndexReplicas_c.cpp.o):
libfaiss_c.a(Index_c.cpp.o):
libfaiss_c.a(IndexBinary_c.cpp.o):
libfaiss_c.a(IndexScalarQuantizer_c.cpp.o):
libfaiss_c.a(MetaIndexes_c.cpp.o):
libfaiss_c.a(clone_index_c.cpp.o):
libfaiss_c.a(error_impl.cpp.o):
libfaiss_c.a(index_factory_c.cpp.o):
libfaiss_c.a(index_io_c.cpp.o):
libfaiss_c.a(AuxIndexStructures_c.cpp.o):
libfaiss_c.a(distances_c.cpp.o):
➜  native git:(master) 


</code>

Copy after login

Solution

Thanks to @HarishGanesan for pointing out that

FAISS c_api depends on FAISS and static linking needs to include it as well.

After studying the

c-rocksdb implementation of CockroachDB, I found some additional flags that can help resolve the architecture arm64 error undefined symbols

/*
#cgo CPPFLAGS: -Ithirdparty/libfaiss-src/c_api
#cgo CFLAGS: -Ithirdparty/libfaiss-src/c_api
#cgo darwin LDFLAGS: -Lthirdparty/runtimes/osx-arm64/native -lfaiss_c -lfaiss -lomp
#cgo darwin LDFLAGS: -Wl,-undefined -Wl,dynamic_lookup
#cgo !darwin LDFLAGS: -Wl,-unresolved-symbols=ignore-all

#include <stdlib.h>
#include <Clustering_c.h>
#include <impl/AuxIndexStructures_c.h>
#include <index_factory_c.h>
#include <error_c.h>
*/

Copy after login


This code currently works on the Macbook M1 chip and M2 chip. 
The complete code can be found here. 
Update 1

https://www.php.cn/link/307eb8ee16198da891c521eca21464c1

https://www.php.cn/link/d585d095b00cd2f5b50acb64add23834
These projects are good reference points for static linking against external C libraries. 
The above is the detailed content of FAISS CGO throws 'undefined symbols for architecture arm64' when using libfaiss_c.a. For more information, please follow other related articles on the PHP Chinese website!