[Enhancement] Update DEVELOPER_GUIDE md file regarding to clang++19 building failure in Mac ARM64 arc. #2272

0ctopus13prime · 2024-11-14T20:29:38Z

Description

When try to build JNI plugin with clang++19, several errors have been occurred.
However, in DEVELOPER_GUIDE we don't have much to say about these failures. It would be good to introduce a new section in there regarding how user can solve the building failures in Mac.

Assumption

User will bump into these issues after they installed llvm via brew. (llvm@19) https://formulae.brew.sh/formula/llvm

⚡  clang++ --version
Homebrew clang version 19.1.3
Target: arm64-apple-darwin23.5.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm/19.1.3/bin
Configuration file: /opt/homebrew/etc/clang/arm64-apple-darwin23.cfg

Failures

Before applying below listed solutions, you have to make sure

No CMakeFiles, CMakeCache.txt and Makefile under jni, external/faiss and external/nmslib.
Ensure cmake is using clang 19.

⚡  cmake . --fresh
Re-run cmake no build system arguments
-- The C compiler identification is Clang 19.1.3
-- The CXX compiler identification is Clang 19.1.3
-- Detecting C compiler ABI info
...

1. std::promise symbol not found

Undefined symbols for architecture arm64:
  "std::exception_ptr::__from_native_exception_pointer(void*)", referenced from:
      std::__1::promise<bool>::~promise() in libfaiss.a[79](WorkerThread.cpp.o)
  "___cxa_init_primary_exception", referenced from:
      std::__1::promise<bool>::~promise() in libfaiss.a[79](WorkerThread.cpp.o)
ld: symbol(s) not found for architecture arm64
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [release/libopensearchknn_faiss.jnilib] Error 1
make[1]: *** [CMakeFiles/opensearchknn_faiss.dir/all] Error 2
make: *** [all] Error 2

Solution:
As the error message suggests, it is failing to get the actual symbol of a std::promise<bool> desctructor during linking time. This can be resolved if we can provide a proper linking path to enable builder to locate the right symbol it wants.

You can use g++ to rebuild the entire project.
Or add this line under https://github.com/opensearch-project/k-NN/blob/main/jni/CMakeLists.txt#L117
1. target_link_options(${TARGET_LIB_FAISS} PRIVATE "-L/opt/homebrew/opt/llvm/lib/c++")
2. /opt/homebrew/opt/llvm/lib/c++ can be acquired -L"$(brew --prefix llvm)"/lib/c++
You will likely hit the same error when building jni_test, and you can address the issue add the same line with a target jni_test.
1. target_link_options(jni_test PRIVATE "-L/opt/homebrew/opt/llvm/lib/c++")

Why it's failing?
Looks like there's a linking issue in clang - Homebrew/homebrew-core#169820

2. Standard library include header not found

[  0%] Building CXX object CMakeFiles/opensearchknn_util.dir/src/jni_util.cpp.o
clang++: warning: -lomp: 'linker' input unused [-Wunused-command-line-argument]
clang++: warning: argument unused during compilation: '-L/opt/homebrew/opt/libomp/lib' [-Wunused-command-line-argument]
In file included from /Users/neetiks/workspace/k-NN/jni/src/jni_util.cpp:12:
/Users/neetiks/workspace/k-NN/jni/include/jni_util.h:16:10: fatal error: 'stdexcept' file not found
   16 | #include <stdexcept>
      |          ^~~~~~~~~~~
1 error generated.

Solution:

Remove CMakeFiles, CMakeCache.txt and Makefile under jni, external/faiss and external/nmslib.
export CC=clang
export CXX=clang++
Ensure cmake is using clang 19.

3. Fortran symbols not found

Undefined symbols for architecture arm64:
  "__gfortran_concat_string", referenced from:
      _sgesvd_ in libopenblas.a[6288](sgesvd.o)
      _sgesvd_ in libopenblas.a[6288](sgesvd.o)
      _sormbr_ in libopenblas.a[6415](sormbr.o)
      _sormbr_ in libopenblas.a[6415](sormbr.o)
      _sormbr_ in libopenblas.a[6415](sormbr.o)
      _sormbr_ in libopenblas.a[6415](sormbr.o)
      _sormlq_ in libopenblas.a[6418](sormlq.o)
      _sormlq_ in libopenblas.a[6418](sormlq.o)
      _sormlq_ in libopenblas.a[6418](sormlq.o)
      ...
ld: symbol(s) not found for architecture arm64

I'm still working on it why this is happening, but suspecting the downloaded libopenblas.a from brew is not compatible with ARM 64 architecture.
Once we remove below line from jni/CMakeCache.txt, it would work.

Remove this vvvvvvvvvvvvvvvvvvvvvvv
BLAS_openblas_LIBRARY:FILEPATH=/opt/homebrew/opt/openblas/lib/libopenblas.a

The text was updated successfully, but these errors were encountered:

navneet1v · 2024-11-14T21:34:08Z

Thanks @0ctopus13prime for starting this issue.

hasnain2808 · 2024-12-07T03:59:41Z

Not sure if it is related but I am getting this while trying to build on my m1 pro

In file included from opensource/opensearch-project-k-NN/jni/src/org_opensearch_knn_jni_FaissService.cpp:18:
In file included from opensource/opensearch-project-k-NN/jni/include/faiss_wrapper.h:16:
In file included from /Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/include/faiss_index_service.h:22:
opensource/opensearch-project-k-NN/jni/include/faiss_stream_support.h:51:24: error: only virtual member functions can be marked 'final'
  int filedescriptor() final {
                       ^~~~~~
opensource/opensearch-project-k-NN/jni/include/faiss_stream_support.h:81:24: error: only virtual member functions can be marked 'final'
  int filedescriptor() final {
                       ^~~~~~
opensource/opensearch-project-k-NN/jni/src/org_opensearch_knn_jni_FaissService.cpp:311:12: warning: implicit conversion of NULL constant to 'jboolean' (aka 'unsigned char') [-Wnull-conversion]
    return NULL;
    ~~~~~~ ^~~~
           '\0'

If I edit the code to remove final I get the error mentioned in the issue description point 3
Fortran symbols not found

But the solution does not work @0ctopus13prime

0ctopus13prime · 2024-12-10T00:20:09Z

@hasnain2808
Hi, final keyword on filedescriptor should work as it is a subclass of faiss' IOReader having the method as a virtual.
Could you check IOReader definition in $ROOT/jni/external/faiss/impl/io.h in your local environment?

0ctopus13prime · 2024-12-10T00:28:56Z

@hasnain2808
Could you share few things from your side?

Result after running cmake . --fresh in $ROOT/jni directory.
Result of clang++ --version
What do you see when building with VERBOSE=1 enabled? Ex: export VERBOSE=1 && make . in $ROOT/jni

hasnain2808 · 2024-12-10T06:41:00Z

@0ctopus13prime

Result after running cmake . --fresh in $ROOT/jni directory.

-- The C compiler identification is AppleClang 15.0.0.15000309
-- The CXX compiler identification is AppleClang 15.0.0.15000309
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using the following git patch command: "am"
CMake Deprecation Warning at external/nmslib/similarity_search/CMakeLists.txt:11 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.


CMake Warning (dev) at external/nmslib/similarity_search/CMakeLists.txt:23 (exec_program):
  Policy CMP0153 is not set: The exec_program command should not be called.
  Run "cmake --help-policy CMP0153" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  Use execute_process() instead.
Call Stack (most recent call first):
  external/nmslib/similarity_search/CMakeLists.txt:34 (CXX_COMPILER_DUMPVERSION)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Build type: Release
-- Header files: /Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/cmd_options.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/distcomp.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/distcomp_edist.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/eval_metrics.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/eval_results.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/experimentconf.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/experiments.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/factory/init_methods.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/factory/init_spaces.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/falconn_heap_mod.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/flexbuff.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/floatdiff.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/global.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/gold_standard.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/hnswquery.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/idtype.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/incremental_quick_select.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/index.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/init.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/inmem_inv_index.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/knnquery.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/knnqueue.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/logging.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/memory.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/meta_analysis.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/dummy.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/hnsw.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/hnsw_distfunc_opt_impl_inline.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/permutation_prefix_index.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/pivot_neighb_common.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/pivot_neighb_invindx.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/seqsearch.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/simple_inverted_index.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/small_world_rand.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/vptree.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/method/vptree_utils.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/methodfactory.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/my_isnan_isinf.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/object.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/params.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/params_cmdline.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/params_def.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/permutation_type.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/permutation_utils.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/portable_align.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/portable_intrinsics.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/portable_popcount.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/portable_prefetch.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/portable_simd.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/ported_boost_progress.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/pow.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/query.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/query_creator.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/rangequery.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/read_data.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/report.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/report_intr_dim.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/searchoracle.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/simddebug.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/sort_arr_bi.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_ab_diverg.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_bit_hamming.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_bit_jaccard.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_bit_vector.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_bregman.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_dummy.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_js.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_l2sqr_sift.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_leven.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_lp.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_rank_correl.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_renyi_diverg.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_scalar.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_bin_common.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_dense_fusion.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_jaccard.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_lp.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_scalar.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_scalar_bin_fast.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_scalar_fast.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_vector.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_vector_inter.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_sqfd.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_string.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_vector.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_vector_gen.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/space/space_word_embed.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/spacefactory.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/thread_pool.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/tune.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/utils.h;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/include/ztimer.h
-- Source files: /Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_bregman.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_diverg.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_edist.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_js.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_l2sqr_sift.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_lp.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_overlap.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_rankcorr.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_scalar.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/distcomp_sparse_scalar_fast.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/experimentconf.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/global.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/hnswquery.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/init.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/knnquery.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/logging.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/memory.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/method/dummy.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/method/hnsw.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/method/hnsw_distfunc_opt.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/method/pivot_neighb_invindx.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/method/seqsearch.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/method/simple_inverted_index.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/method/small_world_rand.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/method/vptree.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/params.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/params_cmdline.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/query.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/rangequery.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/searchoracle.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_ab_diverg.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_bregman.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_dummy.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_js.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_l2sqr_sift.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_lp.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_renyi_diverg.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_scalar.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_dense_fusion.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_jaccard.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_lp.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_scalar_bin_fast.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_scalar_fast.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_vector.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_vector_inter.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_string.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_vector.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/space/space_word_embed.cc;/Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/external/nmslib/similarity_search/src/utils.cc
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Threads.
-- OS:                Darwin
-- Compiler:          Clang 15.0.0
-- CXX flags release: -Wall -Wunreachable-code -Wcast-align -O3 -DNDEBUG -std=c++11 -DHAVE_CXX0X -pthread -fpic  -mcpu=apple-m1
-- CXX flags debug:   -Wall -Wunreachable-code -Wcast-align -ggdb  -DNDEBUG -std=c++11 -DHAVE_CXX0X -pthread -fpic  -mcpu=apple-m1
-- Found ZLIB: /Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/lib/libz.tbd (found version "1.2.12")
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Found BLAS: /opt/homebrew/opt/openblas/lib/libopenblas.a;/opt/homebrew/opt/llvm/lib/libomp.dylib
-- The Fortran compiler identification is GNU 14.2.0
-- Checking whether Fortran compiler has -isysroot
-- Checking whether Fortran compiler has -isysroot - yes
-- Checking whether Fortran compiler supports OSX deployment target flag
-- Checking whether Fortran compiler supports OSX deployment target flag - yes
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/homebrew/bin/gfortran - skipped
-- Looking for Fortran cheev
-- Looking for Fortran cheev - found
-- Found LAPACK: /opt/homebrew/opt/openblas/lib/libopenblas.a;/opt/homebrew/opt/llvm/lib/libomp.dylib;-lm;-ldl
-- Found OpenMP_C: -Xpreprocessor -fopenmp
-- Found OpenMP_CXX: -Xpreprocessor -fopenmp -I/opt/homebrew/opt/llvm/include
-- Found OpenMP_Fortran: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Could NOT find MKL (missing: MKL_LIBRARIES) 
-- Configuring done (0.1s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni/googletest-download
[ 11%] Performing update step for 'googletest'
-- Fetching latest from the remote origin
[ 22%] No patch step for 'googletest'
[ 33%] No configure step for 'googletest'
[ 44%] No build step for 'googletest'
[ 55%] No install step for 'googletest'
[ 66%] No test step for 'googletest'
[ 77%] Completed 'googletest'
[100%] Built target googletest
-- Configuring done (8.3s)
-- Generating done (0.2s)
-- Build files have been written to: /Users/hasnain2808/Projects/opensource/opensearch-project-k-NN/jni

hasnain2808 · 2024-12-10T06:41:44Z

@0ctopus13prime

Result of clang++ --version

Homebrew clang version 19.1.4
Target: arm64-apple-darwin23.5.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm/19.1.4/bin
Configuration file: /opt/homebrew/etc/clang/arm64-apple-darwin23.cfg

hasnain2808 · 2024-12-10T06:51:41Z

@hasnain2808 Hi, final keyword on filedescriptor should work as it is a subclass of faiss' IOReader having the method as a virtual. Could you check IOReader definition in $ROOT/jni/external/faiss/impl/io.h in your local environment?

I do not have the jni/external/faiss/impl/io.h file
I do have jni/external/faiss/faiss/impl/io.h
But it does not have the filedescriptor function

0ctopus13prime · 2024-12-10T19:39:09Z

Hi @hasnain2808
Looks like CMake is using clang++ 15 while clang++ --version gives 19.

-- The C compiler identification is AppleClang 15.0.0.15000309
-- The CXX compiler identification is AppleClang 15.0.0.15000309

Please use JDK 14, 17 or 21 instead.
Could you retry with below?

export CC=clang
export CX=clang++
cd $ROOT/jni
cmake . --fresh
make

Did you run sed in your mac?
Link : https://github.com/opensearch-project/k-NN/blob/main/DEVELOPER_GUIDE.md#additional-setup-for-mac-m-series-machines
IOReader must have filedescriptor
Link : https://github.com/facebookresearch/faiss/blob/main/faiss/impl/io.h#L35
This is somewhat weird to me, could you share the commit?
You can get the commit hash in $ROOT/jni/external/faiss.

Overall, could you retry again with the first remedy above?
Assuming you've had FAISS dependencies already - Link

hasnain2808 · 2024-12-11T14:02:04Z

@0ctopus13prime
doing

export CC=clang
export CX=clang++

made it use clang 19

I also took a fresh clone and the filedescriptor error went, I guess I had an older commit

Yes I run the sed

Now the error I am running into is the 3. Fortran symbols not found

I am new to cmake
if I edit jni/CMakeCache.txt to remove the suggested line
running make or cmake brings it back
Since it is cache it is being rebuilt right

0ctopus13prime added bug Something isn't working untriaged labels Nov 14, 2024

0ctopus13prime changed the title ~~[Enhancement] Update DEVELOPER_GUIDE md file regarding to clang++19 building failure in Mac.~~ [Enhancement] Update DEVELOPER_GUIDE md file regarding to clang++19 building failure in Mac ARM64 arc. Nov 14, 2024

navneet1v added Infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc. and removed untriaged bug Something isn't working labels Nov 14, 2024

This comment was marked as resolved.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Update DEVELOPER_GUIDE md file regarding to clang++19 building failure in Mac ARM64 arc. #2272

[Enhancement] Update DEVELOPER_GUIDE md file regarding to clang++19 building failure in Mac ARM64 arc. #2272

0ctopus13prime commented Nov 14, 2024 •

edited

Loading

navneet1v commented Nov 14, 2024

hasnain2808 commented Dec 7, 2024 •

edited

Loading

0ctopus13prime commented Dec 10, 2024 •

edited

Loading

0ctopus13prime commented Dec 10, 2024

hasnain2808 commented Dec 10, 2024

hasnain2808 commented Dec 10, 2024

hasnain2808 commented Dec 10, 2024

This comment was marked as resolved.

0ctopus13prime commented Dec 10, 2024

hasnain2808 commented Dec 11, 2024

[Enhancement] Update DEVELOPER_GUIDE md file regarding to clang++19 building failure in Mac ARM64 arc. #2272

[Enhancement] Update DEVELOPER_GUIDE md file regarding to clang++19 building failure in Mac ARM64 arc. #2272

Comments

0ctopus13prime commented Nov 14, 2024 • edited Loading

Description

Assumption

Failures

1. std::promise symbol not found

2. Standard library include header not found

3. Fortran symbols not found

navneet1v commented Nov 14, 2024

hasnain2808 commented Dec 7, 2024 • edited Loading

0ctopus13prime commented Dec 10, 2024 • edited Loading

0ctopus13prime commented Dec 10, 2024

hasnain2808 commented Dec 10, 2024

hasnain2808 commented Dec 10, 2024

hasnain2808 commented Dec 10, 2024

This comment was marked as resolved.

0ctopus13prime commented Dec 10, 2024

hasnain2808 commented Dec 11, 2024

0ctopus13prime commented Nov 14, 2024 •

edited

Loading

hasnain2808 commented Dec 7, 2024 •

edited

Loading

0ctopus13prime commented Dec 10, 2024 •

edited

Loading