GH-48926: [C++] Upgrade Abseil/Protobuf/GRPC/Google-Cloud-CPP bundled versions

[raulcd]

TBD

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)

This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)

• GitHub Issue: [C++] Building Flight with bundled gRPC / Abseil fails with gcc-15 and C++20 #48926

[github-actions]

⚠️ GitHub issue #48926 has been automatically assigned in GitHub to PR creator.

[raulcd]

@kou I am submitting this to show some issues I am facing and see if you have any idea on how to solve them. Basically I've tried to bump protobuf/grpc and the related abseil used by default on that version of protobuf. Building flight with bundled dependencies works great but when I want to also build google cloud cpp I start having some problems because newer abseil defines a crc32c target but crc32c also defines the same target name and Google Cloud requires both abseil and crc32c.

-- google-cloud-cpp: CRC32C: Performing Test HAVE_WEAK_GETAUXVAL - Success
CMake Error at /build/cpp/_deps/crc32c-src/CMakeLists.txt:268 (add_library):
  add_library cannot create target "crc32c" because another target with the
  same name already exists.  The existing target is a static library created
  in source directory "/build/cpp/_deps/absl-src/absl/crc".  See
  documentation for policy CMP0002 for more details.


CMake Warning (dev) at /build/cpp/_deps/crc32c-src/CMakeLists.txt:274 (target_sources):
  Policy CMP0076 is not set: target_sources() command converts relative paths
  to absolute.  Run "cmake --help-policy CMP0076" for policy details.  Use
  the cmake_policy command to set the policy and suppress this warning.

  A private source from a directory other than that of target "crc32c" has a
  relative path.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- google-cloud-cpp: Only building the google-cloud-cpp::storage component

I've tried several things locally but to be fair none of them seem to help me move forward. Any idea of how to proceed with those?

[kou]

googleapis/google-cloud-cpp#15402 may solve it.
I'll complete the PR...

[kou]

googleapis/google-cloud-cpp#15402 (comment)

The upcoming v3.0.0 release of the Cloud C++ SDK will use the crc32c provided by Abseil and will no longer require the separate crc32c package.

[raulcd]

@kou should we try patching ourselves with the patch (or similar) you proposed? I haven't been able to get a clear understanding on when are they planning to have v3

[kou]

If we want to try before the v3.0.0 release, we can use GIT_REPOSITORY and GIT_TAG in fetchcontent_declare(): https://cmake.org/cmake/help/latest/module/FetchContent.html#typical-case

We don't need to use my suggested patch.

[raulcd]

I gave that a quick try building from the tag and removing crc32c, see last commit: cdf2f31
I tried disabling opentelemetry and other auto-added features that require grpc on google cloud cpp but it seems that opentelemetry is unconditionally required on v3 (current failures on linux jobs)
Windows failures are related to building abseil time.

Some more work is required to upgrade :(

[raulcd]

@kou I've been able to have a fully bundled dependency build locally with the updated versions! 🥳 we are closer to having this done! Now we have to fix the rest of CI jobs!

The main friction is currently coming from the fact that Google Cloud CPP seems to unconditionally require OpenTelemetry to build and we have several jobs where we are not building with OpenTelemetry enabled.

Nonetheless I think this is a great step, I've been able to build with all our dependencies (--preset ninja-debug-maximal) with the newer versions locally!

[raulcd]

Note to self, I should fix this warning:

 CMake Warning (dev) at C:/hostedtoolcache/windows/Ruby/4.0.1/x64/msys64/ucrt64/share/cmake/Modules/FetchContent.cmake:1963 (message):
  Calling FetchContent_Populate(opentelemetry_cpp) is deprecated, call
  FetchContent_MakeAvailable(opentelemetry_cpp) instead.  Policy CMP0169 can
  be set to OLD to allow FetchContent_Populate(opentelemetry_cpp) to be
  called directly for now, but the ability to call it with declared details
  will be removed completely in a future version.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:3505 (fetchcontent_populate)
  cmake_modules/ThirdpartyToolchain.cmake:193 (build_google_cloud_cpp_storage)
  cmake_modules/ThirdpartyToolchain.cmake:304 (build_dependency)
  cmake_modules/ThirdpartyToolchain.cmake:3577 (resolve_dependency)
  CMakeLists.txt:538 (include)

[kou]

Great! I'll also take a look at the Google Cloud C++ and OpenTelemetry later.

[raulcd]

The MinGW failures seems related to something that happened in the past but I can't really understand what is happening:

FAILED: [code=1] _deps/google_cloud_cpp-build/google/cloud/CMakeFiles/google_cloud_cpp_common.dir/internal/sha256_hash.cc.obj 
D:\a\_temp\msys64\mingw64\bin\ccache.exe D:\a\_temp\msys64\mingw64\bin\c++.exe -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DNOMINMAX -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_WARNINGS -ID:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-src -isystem D:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-build -isystem D:/a/arrow/arrow/build/cpp/_deps/opentelemetry_cpp-src/api/include -Wredundant-move -Wno-noexcept-type -Wno-self-move  -fdiagnostics-color=always  -Wa,-mbig-obj -Wall -Wno-conversion -Wno-sign-conversion -Wdate-time -Wimplicit-fallthrough -Wunused-result -fno-semantic-interposition -mxsave -msse4.2  -O3 -DNDEBUG -O2 -ftree-vectorize  -std=c++20 -Wall -Wextra -Wconversion -Wno-sign-conversion -MD -MT _deps/google_cloud_cpp-build/google/cloud/CMakeFiles/google_cloud_cpp_common.dir/internal/sha256_hash.cc.obj -MF _deps\google_cloud_cpp-build\google\cloud\CMakeFiles\google_cloud_cpp_common.dir\internal\sha256_hash.cc.obj.d -fmodules-ts -fmodule-mapper=_deps\google_cloud_cpp-build\google\cloud\CMakeFiles\google_cloud_cpp_common.dir\internal\sha256_hash.cc.obj.modmap -MD -fdeps-format=p1689r5 -x c++ -o _deps/google_cloud_cpp-build/google/cloud/CMakeFiles/google_cloud_cpp_common.dir/internal/sha256_hash.cc.obj -c D:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-src/google/cloud/internal/sha256_hash.cc
D:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-src/google/cloud/internal/sha256_hash.cc: In function 'google::cloud::v3_2::internal::Sha256Type google::cloud::v3_2::internal::{anonymous}::Sha256Hash(const void*, std::size_t)':
D:/a/arrow/arrow/build/cpp/_deps/google_cloud_cpp-src/google/cloud/internal/sha256_hash.cc:38:3: error: 'BCryptHash' was not declared in this scope; did you mean 'BCryptSignHash'?
   38 |   BCryptHash(BCRYPT_SHA256_ALG_HANDLE, nullptr, 0,
      |   ^~~~~~~~~~
      |   BCryptSignHash

Seems related to old issues raised by us on both MinGW and GCP, @kou you did open the original issues, any idea?

• bcyrpt.h: CNG Algorithm Pseudo-handles are missing mingw-w64/mingw-w64#49
• Can't build with MinGW: BCRYPT_SHA256_ALG_HANDLE isn't defined googleapis/google-cloud-cpp#14436

The only thing I've found is that MinGW's bcrypt.h does declare BCryptHash but it's guarded by #if NTDDI_VERSION >= NTDDI_WINTHRESHOLD (Windows 10), maybe a regression on Google Cloud building on MinGW64?

On another note, I've been able to fix some more CI jobs, I'll continue pushing this.

[kou]

Hmm. I checked Google Cloud C++ changes but I couldn't find any related changes...

How about specifying _WIN32_WINNT=0x0A01 (_WIN32_WINNT_WIN10) in our end explicitly like the following?

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Lines 3878 to 3884 in 560ef02

	if(MINGW)
	# PPROCESSOR_NUMBER requires Windows 7 or later.
	#
	# 0x0601 == _WIN32_WINNT_WIN7
	string(APPEND CMAKE_C_FLAGS " -D_WIN32_WINNT=0x0601")
	string(APPEND CMAKE_CXX_FLAGS " -D_WIN32_WINNT=0x0601")
	endif()

BTW, we can switch to _WIN32_WINNT_WIN10 from _WIN32_WINNT_WIN7.

(Sorry. I haven't taken a look at the Google Cloud C++ and OpenTelemetry...)

[raulcd]

@kou that was a nice one. I also have had to apply a patch to google-cloud-cpp with the guard for some new bcrypt algorithms missing on old MinGW, probably something we can share upstream for a patch (similar to the previous one).
The current failures seem unrelated, I've opened:

• [CI][R] AMD64 Windows R release fails with IOError: Bucket 'ursa-labs-r-test' not found #49609

I'll start exercising extended CI (Linux Packaging, extra C++) next week to continue fixing CI but this is "looking better" than some weeks ago now.

[kou]

we can share upstream for a patch (similar to the previous one).

Let's try it. But they may reject it because the latest MinGW doesn't need the workaround. If it's reject, let's keep the patch in our end until we drop support for old MinGW.

Or we may want to disable GCS on Windows with R build like we did for macOS: #49068

@thisisnic @jonkeane Do you have any opinion for GCS on Windows?

[thisisnic]

we can share upstream for a patch (similar to the previous one).

Let's try it. But they may reject it because the latest MinGW doesn't need the workaround. If it's reject, let's keep the patch in our end until we drop support for old MinGW.

Or we may want to disable GCS on Windows with R build like we did for macOS: #49068

@thisisnic @jonkeane Do you have any opinion for GCS on Windows?

CRAN uses MinGW-w64 version 11.0.1, so we will end up with that issue there. Hard to say as it's unclear how many users use GCS on Windows - what do you think @jonkeane ?

[raulcd]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: 8d90111

Submitted crossbow builds: ursacomputing/crossbow @ actions-2bb6608404

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[raulcd]

@github-actions crossbow submit test-ubuntu-22.04-cpp-emscripten

[github-actions]

Revision: f4bcd48

Submitted crossbow builds: ursacomputing/crossbow @ actions-06a2076e95

Task	Status
test-ubuntu-22.04-cpp-emscripten

[raulcd]

@github-actions crossbow submit test-ubuntu-24.04-cpp-bundled-offline

[github-actions]

Revision: e25649f

Submitted crossbow builds: ursacomputing/crossbow @ actions-9c3dfa52d6

Task	Status
test-ubuntu-24.04-cpp-bundled-offline

[raulcd]

@github-actions crossbow submit test-ubuntu-22.04-cpp-emscripten

[github-actions]

Revision: 77a9fc8

Submitted crossbow builds: ursacomputing/crossbow @ actions-a82cd5c593

Task	Status
test-ubuntu-22.04-cpp-emscripten