Support for Azure Event Hub - Quarkus Dev Service

[JoaoBrandao]

As Developer,
Would be great to be able to have a Dev Service in Quarkus that allows me to use Azure Event Hub without requiring any internet connection for my local environment.

I would suggest having the following functionalities:

• Register topics on the fly
• Use pre-populated topics from the application.properties
• Send events to Azure Event Hub
• Read events from Azure Event Hub

Since Azure Event Hub has compatibility with Apache Kafka, I would recommend using Red panda since is a service already used in Quarkus from another service to support Kafka in Dev Mode.

There are already some functionalities supported by Quarkus community for Kafka https://quarkus.io/guides/kafka#azure-event-hub but this is used only to connect to Azure Event Hub.

Documentation:
https://redpanda.com/

[ppalaga]

First, this would require having an extension for Azure Event Hub under quarkiverse/quarkus-azure-services. We have an extension (including some tests) in Camel Quarkus. The code could be ported from there.

Since Azure Event Hub has compatibility with Apache Kafka, I would recommend using Red panda since is a service already used in Quarkus from another service to support Kafka in Dev Mode.

Could anybody from subject matter experts confirm that this idea could work? Maybe we could use Azurite?

[The-Funk]

Hi all,

I have a relatively rudimentary implementation of this on the go currently for a project.

I use the EventProcessorClientBuilder found here inside of an application scoped bean in a Quarkus 3 application. This works however there are some interesting quirks that I've been hoping someone could help with for a while now.

1. The checkpointstore is not very intuitive and I'd rather use my SQL instance to track checkpoints assuming that the checkpoint is an update operation and not an insert operation (e.g. not going to eat up tons of space just to store checkpoints) The default recommendation seems to be Azure Blob Storage and I'm looking to shy away from that if possible. I'd rather not add another dependency on yet another service.

2. The event processor seemingly conflicts with JBoss logging in some manner such that I occasionally get InterruptedExceptions. My understanding of the inner workings of Netty are tenuous at best, but my guess at what causes this is that since the event processor uses Netty and project Reactor, that the SDK takes over trying to manage threads without consideration for what JBoss is doing. The issue looks to be this. I've tried updating how I subscribe to my updateCheckPointAsync() method to no avail.

3. I cannot figure out how to disable the Azure SDK's logging of events to my console. I'd very much like to disable that logging and provide my own in the processEvent and processError callbacks. They log every message to console by default and this can get a bit out of control and make debugging incredibly difficult.

Update: Sorry for all the edits. Trying to clarify some things. One more piece of information, I've been looking at the camel-quarkus code but I can't seemingly find any runtime code. I see the EventProcessorClient is being used similar to what I'm doing but it all just looks like deployment configuration to me, I can't find any runtime code. Is the camel extension just reading the classes and registering them? Here's what I'm talking about. Maybe someone working on that extension could provide some insight?

[The-Funk]

Suffice it to say I'd very much like to see this feature! Here to help however I may be of use.

[The-Funk]

Update to this:

I found that by making my event processing bean implement Runnable and giving the MS EventHub client its own single thread, this resolved the interrupted exceptions I had been receiving.

I also found that setting the log level for the com.azure package works to silence the logs, so that was pretty cool!

Still don't like the checkpointstore going to blob storage but what can ya do?

These are all quirks of the Azure library but they're useful things to know for anyone implementing it in a Quarkus project currently.