Enable multiple event listeners to consume for Polaris events by nandorKollar · Pull Request #3973 · apache/polaris

nandorKollar · 2026-03-11T13:43:10Z

This PR enables multiple event listeners watching for Polaris events.

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

nandorKollar · 2026-03-11T13:50:33Z

@jbonofre @adutra @adnanhemani would you mind share your thoughts about this approach?

adutra

@nandorKollar this is look promising thanks!

I think we should avoid calling dispatch() altogether (and avoid materializing the PolarisEvent instance on the heap) if we know the event type is not needed. I would like us to consider that, if a user deactivates all listeners, they would get a performance comparable to what they would get if there were no events in Polaris at all.

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

...rvice/src/main/java/org/apache/polaris/service/events/PolarisEventListenerConfiguration.java

runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventListeners.java

jbonofre · 2026-03-15T10:48:32Z

runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventListeners.java

+  public void onStartup(@Observes StartupEvent event) {
+    String listerTypes = configuration.types().orElse("");
+    if (listerTypes.isBlank()) {
+      return;


If we don't have types here, it means we never reach type() merging bellow (line 51).

If a user only set polaris.event-listener.type (the old config) without setting types, the onStartup() method just returns directly, and no listener is registered.

It means it's a breaking change.

Imho, the type() merge logic should be before the early return, or this check should check both types and type (to be backward compatible).

Good point, thanks! This is indeed broken, fixed in 1a0aaa5

adutra · 2026-03-16T11:13:13Z

...rvice/src/main/java/org/apache/polaris/service/events/PolarisEventListenerConfiguration.java

   * identifier.
   */
-  String type();
+  Optional<String> type();


We don't need this one anymore, right?

I think we need it to read the old config: polaris.event-listener.type. It is optional to allow one to omit. The value of this config (if present), and the new one polaris.event-listener.types should be merged. We should also indicate that polaris.event-listener.type is deprecated, should not be used.

Hmm indeed our evolution guidelines considers changes to the configuration as breaking changes: https://polaris.apache.org/releases/1.3.0/evolution/

So, I think we indeed need to deprecate this config but keep it around for some time.

How can we mark it deprecated?

You should:

Annotate the method with @Deprecated

Add a note to the changelog.

Although RelocateConfigSourceInterceptor sounds really promising, I'm afraid it won't work for us. It relocates the property, but we need to merge the values: if both old and new property is set, the values should be merged. We can alternatively say, that the new property overrides the old one, if both are set:
Option 1:

polaris.event-listener.type=aws-cloudwatch polaris.event-listener.types=persistence-in-memory-buffer

result is: persistence-in-memory-buffer, aws-cloudwatch

Option 2:

polaris.event-listener.type=aws-cloudwatch polaris.event-listener.types=persistence-in-memory-buffer

result is persistence-in-memory-buffer, because the new property overrides the old one

The second approach is super simple with FallbackConfigSourceInterceptor, the first one needs a bit more work. Due to simplicity I'd vote for option 2, if both are set (why would someone set both btw? maybe we can emit a warning if both are set btw, or when the old property is still used) What do you think?

Option 2 LGTM. My rationale: if the admin user sets the new property, we can assume the user is aware of the old config deprecation and took care to set the new value correctly.

Re: warning - IMHO it's not worth the effort (same rationale applies), but if it's simple to implement, why not?

It looks like Option 2 got implemented already 👍

The warning is optional from my POV.

I'm also ok with Option 2. Given that type will be deprecated with this PR, it doesn't make much sense to keep respecting it if types is already defined.

adutra · 2026-03-16T11:20:58Z

...rvice/src/main/java/org/apache/polaris/service/events/PolarisEventListenerConfiguration.java

+   * Comma separated list of event listers, each item must be a registered {@link
+   * PolarisEventListener} identifier.
+   */
+  Optional<String> types();


Suggested change

Optional<String> types();

Optional<List<String>> types();

It's a bit odd to see an Optional<List<...> – but that's how SmallRye Config wants it 😅

Update: Optional<Set<String>> types(); is probably better.

nandorKollar · 2026-03-16T23:05:44Z

runtime/defaults/src/main/resources/application.properties


 polaris.file-io.type=default

-polaris.event-listener.type=no-op


As there's no need to set this property to no-op, should we also delete NoOpPolarisEventListener? It will be a breaking change, because anyone who still refers to no-op with the old polaris.event-listener.type (or even with the new polaris.event-listener.types) will hit the following error: No bean found for required type [interface org.apache.polaris.service.events.listeners.PolarisEventListener] and qualifiers [[@jakarta.enterprise.inject.Any(), @io.smallrye.common.annotation.Identifier(value="no-op")]]

We can mark it deprecated though.

Yes, I think we could delete the no-op listener. Regarding breaking changes, our evolution guidelines accept that: https://polaris.apache.org/releases/1.3.0/evolution/

…tomatically by smallrye

nandorKollar · 2026-03-17T07:22:32Z

@nandorKollar this is look promising thanks!

I think we should avoid calling dispatch() altogether (and avoid materializing the PolarisEvent instance on the heap) if we know the event type is not needed. I would like us to consider that, if a user deactivates all listeners, they would get a performance comparable to what they would get if there were no events in Polaris at all.

Thanks, this makes sense, however I've further questions related to this. Do you happen to know why does AwsCloudWatchEventListener filter only for a single event type? I think event listeners should accept any event type, and allow users to configure a particular event listener to listen for only a certain events, but the listener code itself should not assume that it is working only with event type xyz.

adutra · 2026-03-17T10:09:04Z

Do you happen to know why does AwsCloudWatchEventListener filter only for a single event type? I think event listeners should accept any event type, and allow users to configure a particular event listener to listen for only a certain events, but the listener code itself should not assume that it is working only with event type xyz.

Completely agree here. Listeners should NOT filter events, and especially not in a hard-coded fashion – that's a user decision. @adnanhemani may know why the CoudWatch listener was designed that way. Imho it's not very useful – it can only be useful if you are interested in after-refresh-table events, and hence, in certain table optimization use cases.

nandorKollar · 2026-03-18T08:40:44Z

Oh and we shouldn't forget #2573 either. We need to find a way to trigger event cleanup, or even set retention period for events persisted in the database.

adutra · 2026-03-18T14:40:00Z

@adutra How about introducing the event listener configuration in a follow-up PR, together with the optimization you mentioned earlier (i.e., avoiding materializing the PolarisEvent)?

In that PR, we could address the following points:

Make each event listener configurable. For example, CloudWatch could be configured to receive only a subset of events, while at the same time every event is also persisted in the database.

Ensure the event dispatcher only emits events when there is a registered listener for the given event type. This was partially implemented in Create PolarisEvent only when event type is actually supported by listener #3442 — we could revive that work.

Avoid assumptions in the event listener implementation about handling only a specific event type.

What do you think?

I'm all in for breaking the work in many smaller PRs, but I'm a little confused: you said you would like to introduce the event listener configuration in a follow-up PR – but you also wrote you would like to make each event listener configurable in this PR. Could you clarify this small item?

nandorKollar · 2026-03-18T16:09:40Z

@adutra How about introducing the event listener configuration in a follow-up PR, together with the optimization you mentioned earlier (i.e., avoiding materializing the PolarisEvent)?
In that PR, we could address the following points:

Make each event listener configurable. For example, CloudWatch could be configured to receive only a subset of events, while at the same time every event is also persisted in the database.

Ensure the event dispatcher only emits events when there is a registered listener for the given event type. This was partially implemented in Create PolarisEvent only when event type is actually supported by listener #3442 — we could revive that work.

Avoid assumptions in the event listener implementation about handling only a specific event type.

What do you think?

I'm all in for breaking the work in many smaller PRs, but I'm a little confused: you said you would like to introduce the event listener configuration in a follow-up PR – but you also wrote you would like to make each event listener configurable in this PR. Could you clarify this small item?

I recommend to target only focus on making multiple event listeners work in this PR, and rather implement the rest in a second PR. The only configuration parameter for now is deprecating polaris.event-listener.type and adding polaris.event-listener.types. Of course if you think we should avoid event materialization in this PR, I can continue working on that in this change too.

adutra · 2026-03-18T17:31:26Z

I recommend to target only focus on making multiple event listeners work in this PR, and rather implement the rest in a second PR. The only configuration parameter for now is deprecating polaris.event-listener.type and adding polaris.event-listener.types. Of course if you think we should avoid event materialization in this PR, I can continue working on that in this change too.

That works for me. I think this PR is good to go then, but it would still be nice to add a note to the changelog about the deprecation of polaris.event-listener.type.

nandorKollar · 2026-03-18T18:41:12Z

I recommend to target only focus on making multiple event listeners work in this PR, and rather implement the rest in a second PR. The only configuration parameter for now is deprecating polaris.event-listener.type and adding polaris.event-listener.types. Of course if you think we should avoid event materialization in this PR, I can continue working on that in this change too.

That works for me. I think this PR is good to go then, but it would still be nice to add a note to the changelog about the deprecation of polaris.event-listener.type.

Should we add it only to the changelog?

EDIT: never mind, found the way to deprecate it in the doc too. :)

dimas-b

Sorry about last minute comments. The PR LGTM overall. My only suggestion is to try and add automatic redirects for the old config name.

runtime/service/src/main/java/org/apache/polaris/service/events/EventBusConfigurer.java

dimas-b · 2026-03-19T23:46:26Z

...rvice/src/main/java/org/apache/polaris/service/events/PolarisEventListenerConfiguration.java

   * identifier.
   */
-  String type();
+  Optional<String> type();


We could probably add a config redirect too.

Cf.

polaris/persistence/nosql/persistence/cdi/quarkus/src/main/java/org/apache/polaris/persistence/nosql/quarkus/backend/MongoDBConfigSourceFactory.java

Line 45 in 0872093

static final String ACTIVE_PROPERTY = "quarkus.mongodb.active";

...ervice/src/main/java/org/apache/polaris/service/events/PolarisServiceBusEventDispatcher.java

...ice/src/test/java/org/apache/polaris/service/catalog/iceberg/AbstractIcebergCatalogTest.java

...estFixtures/java/org/apache/polaris/service/events/listeners/TestPolarisEventDispatcher.java

…er_service_bus # Conflicts: # runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

flyrain · 2026-03-21T07:33:51Z

...ervice/src/main/java/org/apache/polaris/service/events/PolarisServiceBusEventDispatcher.java

+
+@ApplicationScoped
+@DefaultBean
+public class PolarisServiceBusEventDispatcher implements PolarisEventDispatcher {


Can we add a Java doc for this class?

flyrain · 2026-03-21T07:41:36Z

CHANGELOG.md


 - The configuration option `polaris.rate-limiter.token-bucket.window` is no longer supported and should be removed.
 - `PolarisConfigurationStore` has been deprecated for removal.
+- The configuration option `polaris.event-listener.type` is no longer supported and should be removed. Please use `polaris.event-listener.types` instead.


~~It mighty be OK to make this breaking change, given event listener are not widely used ATM. Could we share it in the mailing list to gather more feedback to double check? cc @adnanhemani .~~

~~If it turns out to be fine, we should call it out here that it's a breaking change.~~

NVM. It seems to be deprecated. I think the word here isn't accurate.

Suggested change

- The configuration option `polaris.event-listener.type` is no longer supported and should be removed. Please use `polaris.event-listener.types` instead.

- The configuration option `polaris.event-listener.type` is deprecated and will be removed later. Please use `polaris.event-listener.types` instead.

flyrain · 2026-03-21T07:42:15Z

runtime/service/src/main/java/org/apache/polaris/service/events/EventBusConfigurer.java

+import io.vertx.core.eventbus.EventBus;
+import jakarta.enterprise.event.Observes;
+
+public class EventBusConfigurer {


Do we need @ApplicationScoped for this class?

IDK off the top of my head 😅 ... but it does not hurt, I'm pretty sure.

ServiceProducers have the maybeBootstrap() observer without @ApplicationScoped, so I'd guess it is NOT required here.

If we drop @ApplicationScoped it might be worth trying to make the observer method static.

This is not a blocker from my POV

flyrain

Thanks @nandorKollar for working on it. Can we add test to configure polaris.event-listener.types with 2+ listeners (e.g., 'test,in-memory-buffer') and verify all receive events? Can we add test to verify deprecated polaris.event-listener.type still works?

jbonofre · 2026-03-22T05:35:48Z

nit: I updated the PR title (typo on lister instead of listener 😄 ).

… multiple event listener

dimas-b

LGTM, pending resolution of open comment threads 👍 Thanks, @nandorKollar !

nandorKollar · 2026-03-23T19:54:28Z

Great, thanks @dimas-b , @flyrain would you mind also have a look at the latest version? @jbonofre @adutra is there any unresolved comment on your end that I may have forgotten to address?

adnanhemani · 2026-03-23T20:15:55Z

I will look into this later today, please hold merging until then.

adnanhemani

Looks almost there, just a few questions on my end - and then one comment from @flyrain about adding a Javadoc. Will approve it after those are done!

adnanhemani · 2026-03-24T00:51:58Z

runtime/service/src/main/java/org/apache/polaris/service/events/EventBusConfigurer.java

+
+@ApplicationScoped
+public class EventBusConfigurer {
+  private static final String LOCAL_CODEC_NAME = "local";


I'm not an expert but after searching around, it seems like this value is supposed to be unique per codec. So shouldn't we name this something like "polaris-events-local" or something to ensure there is no potential future clash naming-wise?

adnanhemani · 2026-03-24T00:52:22Z

...re/src/main/java/org/apache/polaris/core/connection/OAuthClientCredentialsParametersDpo.java

 */
 public class OAuthClientCredentialsParametersDpo extends AuthenticationParametersDpo {
-
-  private static final Joiner COLON_JOINER = Joiner.on(":");


adnanhemani · 2026-03-24T00:52:33Z

...h/opa/tests/src/intTest/java/org/apache/polaris/extension/auth/opa/test/OpaTestProfiles.java

 package org.apache.polaris.extension.auth.opa.test;

 import io.quarkus.test.junit.QuarkusTestProfile;
-import io.quarkus.test.junit.QuarkusTestProfile.TestResourceEntry;


github-project-automation bot added this to Basic Kanban Board Mar 11, 2026

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Mar 11, 2026

jbonofre self-requested a review March 11, 2026 13:58

nandorKollar force-pushed the multiple_event_listener_service_bus branch 7 times, most recently from d0242ed to 09c5600 Compare March 11, 2026 19:59

Enable multiple event listers to consume for Polaris events

aad3caf

nandorKollar force-pushed the multiple_event_listener_service_bus branch from 09c5600 to aad3caf Compare March 11, 2026 20:13

adutra mentioned this pull request Mar 13, 2026

(Proposal) REST API for Events #3924

Open

6 tasks

adutra reviewed Mar 13, 2026

View reviewed changes

nandorKollar added 2 commits March 14, 2026 20:41

Rewrite listener creation logic + cleanup

9389c01

Backward compatibility: take old listener config into account

1c71676

nandorKollar force-pushed the multiple_event_listener_service_bus branch from 8b05fcf to 1c71676 Compare March 14, 2026 20:20

adjust doc

dd17149

jbonofre reviewed Mar 15, 2026

View reviewed changes

Fix backward compatibility

1a0aaa5

adutra reviewed Mar 16, 2026

View reviewed changes

Fix tests

c9f843b

nandorKollar force-pushed the multiple_event_listener_service_bus branch from 48b5fbd to c9f843b Compare March 16, 2026 22:43

nandorKollar commented Mar 16, 2026

View reviewed changes

Simplify parsing of types config: no need to split, it can be done au…

716f6db

…tomatically by smallrye

nandorKollar force-pushed the multiple_event_listener_service_bus branch from ee21f22 to 716f6db Compare March 16, 2026 23:13

nandorKollar marked this pull request as ready for review March 17, 2026 07:22

adutra previously approved these changes Mar 18, 2026

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Mar 18, 2026

Document deprecation of 'polaris.event-listener.type'

ce9addd

nandorKollar dismissed adutra’s stale review via ce9addd March 18, 2026 19:18

nandorKollar requested review from adutra and jbonofre March 19, 2026 18:28

dimas-b reviewed Mar 19, 2026

View reviewed changes

Address code review comments

b570a2e

nandorKollar force-pushed the multiple_event_listener_service_bus branch from 16170de to b570a2e Compare March 20, 2026 13:16

Merge remote-tracking branch 'origin/main' into multiple_event_listen…

0c39bd6

…er_service_bus # Conflicts: # runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

flyrain requested a review from adnanhemani March 21, 2026 06:34

flyrain reviewed Mar 21, 2026

View reviewed changes

jbonofre changed the title ~~Enable multiple event listers to consume for Polaris events~~ Enable multiple event listeners to consume for Polaris events Mar 22, 2026

nandorKollar added 2 commits March 23, 2026 09:27

fix changelog, add javadoc, add @ApplicationScoped, add test case for…

719823d

… multiple event listener

Fallback to old property, add a test case for verification

12406d9

nandorKollar requested review from dimas-b and flyrain March 23, 2026 09:51

dimas-b approved these changes Mar 23, 2026

View reviewed changes

adnanhemani reviewed Mar 24, 2026

View reviewed changes


		polaris.file-io.type=default

		polaris.event-listener.type=no-op

	- The configuration option `polaris.event-listener.type` is no longer supported and should be removed. Please use `polaris.event-listener.types` instead.
	- The configuration option `polaris.event-listener.type` is deprecated and will be removed later. Please use `polaris.event-listener.types` instead.

Conversation

nandorKollar commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

nandorKollar commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adutra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nandorKollar Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nandorKollar commented Mar 17, 2026

Uh oh!

adutra commented Mar 17, 2026

Uh oh!

nandorKollar commented Mar 18, 2026

Uh oh!

adutra commented Mar 18, 2026

Uh oh!

nandorKollar commented Mar 18, 2026

Uh oh!

adutra commented Mar 18, 2026

Uh oh!

nandorKollar commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nandorKollar commented Mar 11, 2026 •

edited

Loading

nandorKollar commented Mar 11, 2026 •

edited

Loading

dimas-b Mar 23, 2026 •

edited

Loading

nandorKollar Mar 16, 2026 •

edited

Loading

nandorKollar commented Mar 18, 2026 •

edited

Loading

flyrain Mar 21, 2026 •

edited

Loading