[midPoint] midPoint 4.4 clustering issue

[Samuel Harmon]

I have clustering now mostly set up on one of our midPoint instances, but
we're running into a problem with them communicating with each other.

We now have two midPoint 4.4 nodes set up on our dev installation
(midpoint-d-1 and midpoint-d-2, both are Podman containers directly running
HTTPS on port 443 and exposed to their container hosts port 443):
-they have a shared keystore containing both keys (the nodes were both
started standalone and then later clustered, so each server's keys are in
the keystore) & a SAN cert to cover both hostnames for SSL. As far as I can
tell, this part is working correctly- both nodes start on port 443 and
aren't throwing errors about encryption keys.
-they can see each other as nodes *via the database*, but all attempts to
communicate to each other via REST fail with “Authentication Error” and
they see each other in the Nodes view as “Communication Error” while their
own node is seen as “Running”.
-the logs are full of messages on the querying side similar to:

2022-07-14 14:56:49,549 [TASK_MANAGER] [pool-3-thread-2] DEBUG
(com.evolveum.midpoint.task.quartzimpl.execution.remote.RestConnector):
Querying remote scheduler information on midpoint-d-2.case.edu finished
with status 401: Unauthorized

To try to fix this, I have attempted the following:

-I tried changing the instance's nodeId from the container’s generated
internal hostname to the container host’s hostname (which is better for
persistence anyway). That did not fix the communication issue.
-I've tested that calling web services to the other node works from inside
each container using curl.
-I also turned up logging on the receiving end and got the following logs &
stack trace when I refreshed the Nodes list on the querying end:

2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
/ws/cluster/scheduler/information at position 1 of 8 in additional filter
chain; firing Filter: 'HeaderWriterFilter'
2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
/ws/cluster/scheduler/information at position 2 of 8 in additional filter
chain; firing Filter: 'RedirectForLoginPagesWithAuthenticationFilter'
2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
/ws/cluster/scheduler/information at position 3 of 8 in additional filter
chain; firing Filter: 'HttpClusterAuthenticationFilter'
2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter):
Cluster Authentication - Authorization header found for remote address
'129.22.104.212'
2022-07-19 14:09:52,809 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.MidpointProviderManager):
Authentication attempt using
com.evolveum.midpoint.web.security.provider.ClusterProvider
2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] INFO
(com.evolveum.midpoint.web.security.provider.ClusterProvider):
Authentication failed for 129.22.104.212:
web.security.flexAuth.cluster.auth.null
2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] ERROR
(com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider):
Authentication (runtime) error: web.security.flexAuth.cluster.auth.null
org.springframework.security.authentication.AuthenticationServiceException:
web.security.flexAuth.cluster.auth.null
at
com.evolveum.midpoint.web.security.provider.ClusterProvider.internalAuthentication(ClusterProvider.java:59)
at
com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider.authenticate(MidPointAbstractAuthenticationProvider.java:92)
at
com.evolveum.midpoint.web.security.MidpointProviderManager.authenticate(MidpointProviderManager.java:58)
at jdk.internal.reflect.GeneratedMethodAccessor576.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at
org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy181.authenticate(Unknown Source)
at
com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter.doFilterInternal(HttpClusterAuthenticationFilter.java:78)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
at
com.evolveum.midpoint.web.security.filter.RedirectForLoginPagesWithAuthenticationFilter.doFilterInternal(RedirectForLoginPagesWithAuthenticationFilter.java:39)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
at
org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at
org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
at
com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilterInternal(MidpointAuthFilter.java:226)
at
com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilter(MidpointAuthFilter.java:109)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
com.evolveum.midpoint.web.security.filter.TranslateExceptionFilter.doFilterInternal(TranslateExceptionFilter.java:32)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:147)
at
org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
t
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at
org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at
org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at
com.evolveum.midpoint.web.boot.TrailingSlashRedirectingFilter.doFilterInternal(TrailingSlashRedirectingFilter.java:60)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at
com.evolveum.midpoint.web.boot.NodeIdHeaderValve.invoke(NodeIdHeaderValve.java:46)
at
com.evolveum.midpoint.web.boot.TomcatRootValve.invoke(TomcatRootValve.java:62)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter):
Authentication request for failed:
org.springframework.security.authentication.AuthenticationServiceException:
web.security.flexAuth.cluster.auth.null
2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Created
HttpSession as SecurityContext is non-default
2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Stored
com.evolveum.midpoint.web.security.MidpointSecurityContext at 385b4af to
HttpSession [org.apache.catalina.session.StandardSessionFacade at 451674c7]
2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Retrieved
com.evolveum.midpoint.web.security.MidpointSecurityContext at 385b4af
2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.MidPointAuthWebSession): Found locale en
2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
(com.evolveum.midpoint.web.security.MidPointAuthWebSession): Using en as
locale

Any ideas?

Sam
-- 
Sam Harmon
Case Western Reserve University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20220722/12e5d127/attachment-0001.htm>