PooledConnectionFactory connected/disconnected events

[johnellinwood]

What is the best way to simply check if a PooledConnectionFactory is healthy or not? As in, if it received connection errors or not on it's last connection attempts, and/or has 0 active and 0 available connections? I just want to notify my application when the remote ldap connection is down, and then back up.

Is there a way to get events from the PooledConnectionFactory when it's underlying connections open, fail, and re-open?

The situation I see is when the remote server goes down, I get these exception in the log at ERROR level.

ERROR org.ldaptive.transport.netty.NettyConnection: Connection open failed for org.ldaptive.transport.netty.NettyConnection 
  org.ldaptive.ConnectException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: ...
  

I want to capture those exceptions, log them internally as DEBUG, and then event to the rest of my app that ldap is down (ui widget, etc). It's all reactive. But I don't see anywhere at the PooledConnectionFactory or DefaultConnectionFactory to capture these exceptions or put in a customized handler of some type.

For healthy I could put a connection activator on the factory or something to just set it to healthy maybe?

I've looked through the user guide, javadocs, and now am digging through the code looking for a good place to put this.

I could periodically poll the pooled connection factory i think, but i wanted to see if you have a hook for this. What would that polling look like? If i set minPoolSize to 1, and activeCount and availableCount are both 0, then that is unhealthy?

Also, do you have a discord or somewhere to chat these kinds of questions instead of opening issues? Sorry for asking this as an issue.

[dfish3r]

Defining what healthy means can be tricky. Are you measuring the health of the LDAP itself, the network path (including load balancers) or just the health of any single connection? If I can connect but searches are failing, is that healthy? I think in general you want to communicate the health of the service as a whole, but it's worth calling out what that means. Connections are closed for various reason that don't necessarily indicate a problem. TCP timeouts may be a normal and expected event. LDAP servers behind load balancers may be restarted regularly, forcing connections to be reopened and may cause intermittent connection failures.

That said, most of the solutions I've seen test whether a connection can be opened and/or a search is successful. They either use a ConnectionValidator with the pool or just perform a search with a DefaultConnectionFactory separate from the pool. That turns "healthy" into a boolean (up or down) decision, which obviously isn't granular enough if you've got a problem where your LDAP is degraded but not down. But it is a simple, fairly easy solution and is generally correct in terms of what you'd communicate to a user.

If you're looking for richer metadata about what is happening in the pool, we'll need to make some API changes to expose those events. I am making some improvements to the ConnectionValidator interface to expose more details, I may be able to tackle these changes as part of that work.

[dfish3r]

Also, do you have a discord or somewhere to chat these kinds of questions instead of opening issues? Sorry for asking this as an issue.

No, posting here is fine. I'm holding my breath that github will provide that sort of functionality (for free).

[johnellinwood]

OK, then I guess I have two different follow-up questions:

1. How can I make that exception go away about the Netty connection? I can't figure out where to capture it and hide it or lower the log level. Eeh, I guess just set its log level in my log configuration to none?

2. Does the connection validator run even when it can't make any connections? I was looking through the code, and it seemed like it would only validate the connections if it was actually able to successfully make a connection. I guess I'll try it.

In my particular use case, I think the majority of the connection issues are when the backend server is just completely shut down, and netty cant connect or reconnect.

[dfish3r]

How can I make that exception go away about the Netty connection? I can't figure out where to capture it and hide it or lower the log level. Eeh, I guess just set its log level in my log configuration to none?

I think the answer to that depends on your logging backend. For instance, if you're using logback I believe you can set the loglevel to OFF. Is it fair to say you don't think the failure to open a connection rises to an error? I agree that may be more appropriate as a warning.

Does the connection validator run even when it can't make any connections? I was looking through the code, and it seemed like it would only validate the connections if it was actually able to successfully make a connection. I guess I'll try it.

The validator in the pool runs on a set period, so it will attempt to validate any connections not in use and bring the pool up to it's configured minSize every time it runs.

[johnellinwood]

Ok, thanks. With your help, I can now make it do what I need. If i set connectOnCreate(false) and minSize(1) and set a connection validator with periodic validation and a validationexceptionhandler... i can wrap up all the events and just set my service to up or down.

I still think... at least the stack traces and error levels on the ldaptive exceptions could be removed or changed to DEBUG. Or have that exception logging be a default handler or something, that could be replaced in configuration.

I don't need the

ERROR org.ldaptive.transport.netty.NettyConnection : Connection open failed
org.ldaptive.transport.netty.NettyConnection : Connection open failed for org.ldaptive.transport.netty.NettyConnection
  org.ldaptive.ConnectException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Conneciton refused
  at ..
  at ..

because I can tell its not up from the ConnectionValidator.

I don't need the

WARN org.ldaptive.PooledConnectionFactory : Connection could not be validated, invoking handler
org.ldaptive.pool.ValidationException: Validation of connection failed for pool
  at ...
  at ..

because the very next thing it does is call my ValidationExceptionHandler with that same exception.

The problem with the logs is

1. the stack traces for things i can already handle
2. ERROR and WARN levels getting into our log handling system
3. I don't want to completely hide all the PooledConnectionFactory and NettyConnection messages... maybe there is some other critical exception i do need to see