Skip to content

GH-49463: [C++][FlightRPC] Add Ubuntu ODBC Support #49564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kou merged 7 commits into from
Mar 28, 2026
Merged

GH-49463: [C++][FlightRPC] Add Ubuntu ODBC Support #49564

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
848fdfe
Enable ODBC Build On Linux
alinaliBQ Mar 19, 2026
9d6728d
Work on kou's comments
alinaliBQ Mar 24, 2026
bec15db
Unify workaround fix for including sql.h
alinaliBQ Mar 25, 2026
07b9ef4
Fix the build by adding back `ARCH` and `UBUNTU`
alinaliBQ Mar 25, 2026
0fa6143
work on co-pilot comments
alinaliBQ Mar 26, 2026
0b5d6f1
Address kou's comments
alinaliBQ Mar 26, 2026
2a49313
Re-use `ubuntu-cpp` as base image
alinaliBQ Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 58 additions & 1 deletion .github/workflows/cpp_extra.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,62 @@ jobs:
cd cpp/examples/minimal_build
../minimal_build.build/arrow-example

odbc-linux:
needs: check-labels
name: ODBC Linux
runs-on: ubuntu-latest
if: >-
needs.check-labels.outputs.force == 'true' ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: C++')
timeout-minutes: 75
strategy:
fail-fast: false
env:
ARCH: amd64
ARCHERY_DEBUG: 1
ARROW_ENABLE_TIMING_TESTS: OFF
DOCKER_VOLUME_PREFIX: ".docker/"
UBUNTU: 24.04
steps:
- name: Checkout Arrow
uses: actions/checkout@v6
with:
fetch-depth: 0
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@v5
with:
path: .docker
key: ubuntu-cpp-odbc-${{ hashFiles('cpp/**') }}
restore-keys: ubuntu-cpp-odbc-
- name: Setup Python on hosted runner
uses: actions/setup-python@v6
with:
python-version: 3
- name: Setup Archery
run: python3 -m pip install -e dev/archery[docker]
- name: Execute Docker Build
env:
ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
run: |
# GH-40558: reduce ASLR to avoid ASAN/LSAN crashes
sudo sysctl -w vm.mmap_rnd_bits=28
source ci/scripts/util_enable_core_dumps.sh
archery docker run ubuntu-cpp-odbc
- name: Docker Push
if: >-
success() &&
github.event_name == 'push' &&
github.repository == 'apache/arrow' &&
github.ref_name == 'main'
env:
ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
continue-on-error: true
run: archery docker push ubuntu-cpp-odbc
Comment on lines +383 to +393
Copy link
Copy Markdown
Member

@kou kou Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this image because we can reuse ubuntu-cpp image.

Suggested change
- name: Docker Push
if: >-
success() &&
github.event_name == 'push' &&
github.repository == 'apache/arrow' &&
github.ref_name == 'main'
env:
ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
continue-on-error: true
run: archery docker push ubuntu-cpp-odbc

Copy link
Copy Markdown
Collaborator Author

@alinaliBQ alinaliBQ Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi kou, I used ubuntu-cpp-odbc for running a custom commands section, since for ODBC we need to run the ODBC registration script before executing tests.
In order to use ubuntu-cpp, I tried code like archery docker run ubuntu-cpp bash -c "<custom_commands>" to replace the commands section and that didn't work, so I created a new image instead. Is there a workaround for this?

Copy link
Copy Markdown
Member

@kou kou Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your approach is correct. I also think that I want to create a new service (ubuntu-cpp-odbc) from an existing service (ubuntu-cpp) with different command: but we can't do it with Docker Compose. So we can create ubuntu-cpp-odbc for this. But the built image by ubuntu-cpp-odbc has the same name (${REPO}:${ARCH}-ubuntu-${UBUNTU}-cpp) and build options. So we don't need to overwrite existing ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cpp image by this.

Copy link
Copy Markdown
Collaborator Author

@alinaliBQ alinaliBQ Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou Sounds good. I have changed ubuntu-cpp-odbc to re-use ubuntu-cpp as a service to reduce duplicate code. I also disabled non-ODBC related libraries in ubuntu-cpp-odbc. Please have another look.


odbc-macos:
needs: check-labels
name: ODBC ${{ matrix.build-type }} ${{ matrix.architecture }} macOS ${{ matrix.macos-version }}
Expand Down Expand Up @@ -435,7 +491,7 @@ jobs:
"$(pwd)/build/cpp/${{ matrix.build-type }}/libarrow_flight_sql_odbc.dylib"
- name: Register Flight SQL ODBC Driver
run: |
sudo cpp/src/arrow/flight/sql/odbc/install/mac/install_odbc.sh $(pwd)/build/cpp/${{ matrix.build-type }}/libarrow_flight_sql_odbc.dylib
sudo cpp/src/arrow/flight/sql/odbc/install/unix/install_odbc.sh $(pwd)/build/cpp/${{ matrix.build-type }}/libarrow_flight_sql_odbc.dylib
- name: Test
shell: bash
run: |
Expand Down Expand Up @@ -698,6 +754,7 @@ jobs:
- jni-linux
- jni-macos
- msvc-arm64
- odbc-linux
- odbc-macos
- odbc-msvc
- odbc-nightly
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,7 @@ repos:
?^cpp/build-support/update-thrift\.sh$|
?^cpp/examples/minimal_build/run\.sh$|
?^cpp/examples/tutorial_examples/run\.sh$|
?^cpp/src/arrow/flight/sql/odbc/install/mac/install_odbc\.sh$|
?^cpp/src/arrow/flight/sql/odbc/install/unix/install_odbc\.sh$|
?^dev/release/05-binary-upload\.sh$|
?^dev/release/08-binary-verify\.sh$|
?^dev/release/binary-recover\.sh$|
Expand Down
1 change: 1 addition & 0 deletions ci/docker/ubuntu-24.04-cpp.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ RUN apt-get update -y -q && \
rsync \
tzdata \
tzdata-legacy \
unixodbc-dev \
uuid-runtime \
unzip \
wget && \
Expand Down
38 changes: 38 additions & 0 deletions compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ x-hierarchy:
- ubuntu-r-only-r
- ubuntu-cpp-bundled
- ubuntu-cpp-bundled-offline
- ubuntu-cpp-odbc
- ubuntu-cpp-minimal
- ubuntu-cuda-cpp:
- ubuntu-cuda-python
Expand Down Expand Up @@ -496,6 +497,43 @@ services:
volumes: *ubuntu-volumes
command: *cpp-command

ubuntu-cpp-odbc:
# Arrow Flight SQL ODBC build with BUNDLED dependencies with downloaded dependencies.
image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cpp
build:
context: .
dockerfile: ci/docker/ubuntu-${UBUNTU}-cpp.dockerfile
cache_from:
- ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cpp
args:
arch: ${ARCH}
base: "${ARCH}/ubuntu:${UBUNTU}"
clang_tools: ${CLANG_TOOLS}
cmake: ${CMAKE}
gcc: ${GCC}
llvm: ${LLVM}
shm_size: *shm-size
cap_add:
- SYS_ADMIN
devices:
- "/dev/fuse:/dev/fuse"
Comment thread
alinaliBQ marked this conversation as resolved.
Outdated
security_opt:
- apparmor:unconfined
ulimits: *ulimits
environment:
<<: [*common, *ccache, *sccache, *cpp]
ARROW_BUILD_TYPE: RELEASE
ARROW_DEPENDENCY_SOURCE: BUNDLED
ARROW_DEPENDENCY_USE_SHARED: OFF
ARROW_FLIGHT_SQL_ODBC: ON
volumes: *ubuntu-volumes
# Register ODBC before running tests
command: >
/bin/bash -c "
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
sudo /arrow/cpp/src/arrow/flight/sql/odbc/install/unix/install_odbc.sh /usr/local/lib/libarrow_flight_sql_odbc.so &&
/arrow/ci/scripts/cpp_test.sh /arrow /build"

ubuntu-cpp-minimal:
# Arrow build with minimal components/dependencies
image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cpp-minimal
Expand Down
4 changes: 0 additions & 4 deletions cpp/cmake_modules/DefineOptions.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,10 +107,6 @@ macro(tsort_bool_option_dependencies)
endmacro()

macro(resolve_option_dependencies)
# Arrow Flight SQL ODBC is available only for Windows and macOS for now.
if(NOT WIN32 AND NOT APPLE)
set(ARROW_FLIGHT_SQL_ODBC OFF)
endif()
if(MSVC_TOOLCHAIN)
set(ARROW_USE_GLOG OFF)
endif()
Expand Down
2 changes: 1 addition & 1 deletion cpp/cmake_modules/ThirdpartyToolchain.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1090,7 +1090,7 @@ function(build_boost)
set(ARROW_BOOST_NEED_MULTIPRECISION FALSE)
endif()
if(ARROW_ENABLE_THREADING)
if(ARROW_WITH_THRIFT OR (ARROW_FLIGHT_SQL_ODBC AND MSVC))
if(ARROW_WITH_THRIFT OR ARROW_FLIGHT_SQL_ODBC)
list(APPEND BOOST_INCLUDE_LIBRARIES locale)
endif()
if(ARROW_BOOST_NEED_MULTIPRECISION)
Expand Down
7 changes: 6 additions & 1 deletion cpp/src/arrow/flight/sql/odbc/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,12 @@ else()
endif()

add_subdirectory(odbc_impl)
add_subdirectory(tests)
if(ARROW_BUILD_TESTS)
if(WIN32 OR APPLE)
# GH-49552 TODO: Enable Linux test build
add_subdirectory(tests)
endif()
endif()

arrow_install_all_headers("arrow/flight/sql/odbc")

Expand Down
11 changes: 10 additions & 1 deletion ...ight/sql/odbc/install/mac/install_odbc.sh → ...ght/sql/odbc/install/unix/install_odbc.sh
View file
Open in desktop
Copy link
Copy Markdown
Member

@kou kou Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update .pre-commit-config.yaml?

Copy link
Copy Markdown
Collaborator Author

@alinaliBQ alinaliBQ Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes good catch, updated

Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,16 @@ if [ ! -f "$ODBC_64BIT" ]; then
exit 1
fi

USER_ODBCINST_FILE="$HOME/Library/ODBC/odbcinst.ini"
case "$(uname)" in
Linux)
USER_ODBCINST_FILE="/etc/odbcinst.ini"
;;
*)
# macOS
USER_ODBCINST_FILE="$HOME/Library/ODBC/odbcinst.ini"
;;
esac

DRIVER_NAME="Apache Arrow Flight SQL ODBC Driver"

mkdir -p "$HOME"/Library/ODBC
Copy link
Copy Markdown
Member

@kou kou Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we update this too?

Copy link
Copy Markdown
Collaborator Author

@alinaliBQ alinaliBQ Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I have moved the command for macOS only

Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/flight/sql/odbc/odbc_api.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@
#endif // defined(_WIN32)

namespace arrow::flight::sql::odbc {
void LoadPropertiesFromDSN(const std::string& dsn, Connection::ConnPropertyMap& props);

SQLRETURN SQLAllocHandle(SQLSMALLINT type, SQLHANDLE parent, SQLHANDLE* result) {
ARROW_LOG(DEBUG) << "SQLAllocHandle called with type: " << type
<< ", parent: " << parent
Expand Down
3 changes: 1 addition & 2 deletions cpp/src/arrow/flight/sql/odbc/odbc_api_internal.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@
#include "arrow/flight/sql/odbc/odbc_impl/platform.h"

#include <sql.h>
#include <sqltypes.h>
#include <sqlucode.h>
#include <sqlext.h>

// \file odbc_api_internal.h
//
Expand Down
20 changes: 13 additions & 7 deletions cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,20 +130,26 @@ if(WIN32)
win_system_dsn.cc)
endif()

if(APPLE)
if(WIN32)
find_package(ODBC REQUIRED)
target_include_directories(arrow_odbc_spi_impl PUBLIC ${ODBC_INCLUDE_DIR})
target_link_libraries(arrow_odbc_spi_impl
PUBLIC arrow_flight_sql_shared arrow_compute_shared Boost::locale
${ODBCINST})
else()
# Unix
target_include_directories(arrow_odbc_spi_impl SYSTEM BEFORE PUBLIC ${ODBC_INCLUDE_DIR})
target_link_libraries(arrow_odbc_spi_impl
PUBLIC arrow_flight_sql_static
arrow_compute_static
Boost::locale
Boost::headers
RapidJSON)
else()
find_package(ODBC REQUIRED)
target_include_directories(arrow_odbc_spi_impl PUBLIC ${ODBC_INCLUDE_DIR})
target_link_libraries(arrow_odbc_spi_impl
PUBLIC arrow_flight_sql_shared arrow_compute_shared Boost::locale
${ODBCINST})

if(NOT APPLE)
# Explicitly link to unix-odbc on Linux
target_link_libraries(arrow_odbc_spi_impl PUBLIC ${ODBCINST})
endif()
Comment thread
alinaliBQ marked this conversation as resolved.
endif()

# CLI
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/flight/sql/odbc/odbc_impl/accessors/binary_array_accessor_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ TEST(BinaryArrayAccessor, Test_CDataType_BINARY_Basic) {
accessor.GetColumnarData(&binding, 0, values.size(), value_offset, false,
diagnostics, nullptr));

for (int i = 0; i < values.size(); ++i) {
for (size_t i = 0; i < values.size(); ++i) {
ASSERT_EQ(values[i].length(), str_len_buffer[i]);
// Beware that CDataType_BINARY values are not null terminated.
// It's safe to create a std::string from this data because we know it's
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/flight/sql/odbc/odbc_impl/accessors/boolean_array_accessor_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ TEST(BooleanArrayFlightSqlAccessor, Test_BooleanArray_CDataType_BIT) {
accessor.GetColumnarData(&binding, 0, values.size(), value_offset, false,
diagnostics, nullptr));

for (int i = 0; i < values.size(); ++i) {
for (size_t i = 0; i < values.size(); ++i) {
ASSERT_EQ(sizeof(unsigned char), str_len_buffer[i]);
ASSERT_EQ(values[i] ? 1 : 0, buffer[i]);
}
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/flight/sql/odbc/odbc_impl/accessors/decimal_array_accessor_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ void AssertNumericOutput(int input_precision, int input_scale,
accessor.GetColumnarData(&binding, 0, values.size(), value_offset, false,
diagnostics, nullptr));

for (int i = 0; i < values.size(); ++i) {
for (size_t i = 0; i < values.size(); ++i) {
ASSERT_EQ(sizeof(NUMERIC_STRUCT), str_len_buffer[i]);

ASSERT_EQ(output_precision, buffer[i].precision);
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/flight/sql/odbc/odbc_impl/accessors/primitive_array_accessor_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ void TestPrimitiveArraySqlAccessor() {
accessor.GetColumnarData(&binding, 0, values.size(), value_offset, false,
diagnostics, nullptr));

for (int i = 0; i < values.size(); ++i) {
for (size_t i = 0; i < values.size(); ++i) {
ASSERT_EQ(sizeof(c_type), str_len_buffer[i]);
ASSERT_EQ(values[i], buffer[i]);
}
Expand Down
4 changes: 2 additions & 2 deletions cpp/src/arrow/flight/sql/odbc/odbc_impl/accessors/string_array_accessor_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ TEST(StringArrayAccessor, Test_CDataType_CHAR_Basic) {
accessor.GetColumnarData(&binding, 0, values.size(), value_offset, false,
diagnostics, nullptr));

for (int i = 0; i < values.size(); ++i) {
for (size_t i = 0; i < values.size(); ++i) {
ASSERT_EQ(values[i].length(), str_len_buffer[i]);
ASSERT_EQ(values[i], std::string(buffer.data() + i * max_str_len));
}
Expand Down Expand Up @@ -102,7 +102,7 @@ TEST(StringArrayAccessor, Test_CDataType_WCHAR_Basic) {
accessor->GetColumnarData(&binding, 0, values.size(), value_offset, false,
diagnostics, nullptr));

for (int i = 0; i < values.size(); ++i) {
for (size_t i = 0; i < values.size(); ++i) {
ASSERT_EQ(values[i].length() * GetSqlWCharSize(), str_len_buffer[i]);
std::vector<uint8_t> expected;
Utf8ToWcs(values[i].c_str(), &expected);
Expand Down
10 changes: 5 additions & 5 deletions cpp/src/arrow/flight/sql/odbc/odbc_impl/accessors/types.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,12 @@ struct ColumnBinding {

ColumnBinding(CDataType target_type, int precision, int scale, void* buffer,
size_t buffer_length, ssize_t* str_len_buffer)
: target_type(target_type),
precision(precision),
scale(scale),
buffer(buffer),
: buffer(buffer),
str_len_buffer(str_len_buffer),
buffer_length(buffer_length),
str_len_buffer(str_len_buffer) {}
target_type(target_type),
precision(precision),
scale(scale) {}
};

/// \brief Accessor interface meant to provide a way of populating data of a
Expand Down
10 changes: 10 additions & 0 deletions cpp/src/arrow/flight/sql/odbc/odbc_impl/attribute_utils.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,11 +158,21 @@ inline void SetAttributeSQLWCHAR(SQLPOINTER new_value, SQLINTEGER input_length_i
thread_local std::vector<uint8_t> utf8_str;
if (input_length_in_bytes == SQL_NTS) {
arrow::flight::sql::odbc::WcsToUtf8(new_value, &utf8_str);
} else if (input_length_in_bytes <= 0) {
// empty string
attribute_to_write.clear();
return;
} else {
arrow::flight::sql::odbc::WcsToUtf8(
new_value, input_length_in_bytes / arrow::flight::sql::odbc::GetSqlWCharSize(),
&utf8_str);
}

// add null-terminator
if (utf8_str.back() != '\0') {
utf8_str.push_back('\0');
}
Comment thread
alinaliBQ marked this conversation as resolved.

attribute_to_write.assign((char*)utf8_str.data());
}

Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/flight/sql/odbc/odbc_impl/blocking_queue.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ class BlockingQueue {

void AddProducer(Supplier supplier) {
active_threads_++;
threads_.emplace_back([=] {
threads_.emplace_back([this, supplier] {
while (!closed_) {
// Block while queue is full
std::unique_lock<std::mutex> unique_lock(mtx_);
Expand Down
Loading
Loading