[midPoint] midPoint 4.4 clustering issue

Fri Jul 22 20:42:56 CEST 2022

Yes, Once I got both keys into the keystore, I copied the .jceks file onto
the other machine.

"-they have a shared keystore containing both keys (the nodes were both
started standalone and then later clustered, so each server's keys are in
the keystore)"

On Fri, Jul 22, 2022 at 2:37 PM Emil Militzer via midPoint <
midpoint at lists.evolveum.com> wrote:

> Hi,
>
> do both nodes use the same keystore?
>
> Kind Regards
> Emil
>
> Am 22.07.2022 um 20:10 schrieb Samuel Harmon via midPoint <
> midpoint at lists.evolveum.com>:
>
> 
> I have clustering now mostly set up on one of our midPoint instances, but
> we're running into a problem with them communicating with each other.
>
> We now have two midPoint 4.4 nodes set up on our dev installation
> (midpoint-d-1 and midpoint-d-2, both are Podman containers directly running
> HTTPS on port 443 and exposed to their container hosts port 443):
> -they have a shared keystore containing both keys (the nodes were both
> started standalone and then later clustered, so each server's keys are in
> the keystore) & a SAN cert to cover both hostnames for SSL. As far as I can
> tell, this part is working correctly- both nodes start on port 443 and
> aren't throwing errors about encryption keys.
> -they can see each other as nodes *via the database*, but all attempts to
> communicate to each other via REST fail with “Authentication Error” and
> they see each other in the Nodes view as “Communication Error” while their
> own node is seen as “Running”.
> -the logs are full of messages on the querying side similar to:
>
> 2022-07-14 14:56:49,549 [TASK_MANAGER] [pool-3-thread-2] DEBUG
> (com.evolveum.midpoint.task.quartzimpl.execution.remote.RestConnector):
> Querying remote scheduler information on midpoint-d-2.case.edu finished
> with status 401: Unauthorized
>
> To try to fix this, I have attempted the following:
>
> -I tried changing the instance's nodeId from the container’s generated
> internal hostname to the container host’s hostname (which is better for
> persistence anyway). That did not fix the communication issue.
> -I've tested that calling web services to the other node works from inside
> each container using curl.
> -I also turned up logging on the receiving end and got the following logs
> & stack trace when I refreshed the Nodes list on the querying end:
>
> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
> /ws/cluster/scheduler/information at position 1 of 8 in additional filter
> chain; firing Filter: 'HeaderWriterFilter'
> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
> /ws/cluster/scheduler/information at position 2 of 8 in additional filter
> chain; firing Filter: 'RedirectForLoginPagesWithAuthenticationFilter'
> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
> /ws/cluster/scheduler/information at position 3 of 8 in additional filter
> chain; firing Filter: 'HttpClusterAuthenticationFilter'
> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter):
> Cluster Authentication - Authorization header found for remote address
> '129.22.104.212'
> 2022-07-19 14:09:52,809 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.MidpointProviderManager):
> Authentication attempt using
> com.evolveum.midpoint.web.security.provider.ClusterProvider
> 2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] INFO
> (com.evolveum.midpoint.web.security.provider.ClusterProvider):
> Authentication failed for 129.22.104.212:
> web.security.flexAuth.cluster.auth.null
> 2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] ERROR
> (com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider):
> Authentication (runtime) error: web.security.flexAuth.cluster.auth.null
> org.springframework.security.authentication.AuthenticationServiceException:
> web.security.flexAuth.cluster.auth.null
> at
> com.evolveum.midpoint.web.security.provider.ClusterProvider.internalAuthentication(ClusterProvider.java:59)
> at
> com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider.authenticate(MidPointAbstractAuthenticationProvider.java:92)
> at
> com.evolveum.midpoint.web.security.MidpointProviderManager.authenticate(MidpointProviderManager.java:58)
> at jdk.internal.reflect.GeneratedMethodAccessor576.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
> at
> org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> at com.sun.proxy.$Proxy181.authenticate(Unknown Source)
> at
> com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter.doFilterInternal(HttpClusterAuthenticationFilter.java:78)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
> at
> com.evolveum.midpoint.web.security.filter.RedirectForLoginPagesWithAuthenticationFilter.doFilterInternal(RedirectForLoginPagesWithAuthenticationFilter.java:39)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
> at
> org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
> at
> org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
> at
> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilterInternal(MidpointAuthFilter.java:226)
> at
> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilter(MidpointAuthFilter.java:109)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
> at
> com.evolveum.midpoint.web.security.filter.TranslateExceptionFilter.doFilterInternal(TranslateExceptionFilter.java:32)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
> at
> org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:147)
> at
> org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
> at
> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
> at
> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
> at
> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
> at
> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
> at
> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
> at
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
> at
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
> t
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
> at
> org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
> at
> org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
> at
> org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
> at
> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
> at
> com.evolveum.midpoint.web.boot.TrailingSlashRedirectingFilter.doFilterInternal(TrailingSlashRedirectingFilter.java:60)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> at
> com.evolveum.midpoint.web.boot.NodeIdHeaderValve.invoke(NodeIdHeaderValve.java:46)
> at
> com.evolveum.midpoint.web.boot.TomcatRootValve.invoke(TomcatRootValve.java:62)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
> at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter):
> Authentication request for failed:
> org.springframework.security.authentication.AuthenticationServiceException:
> web.security.flexAuth.cluster.auth.null
> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Created
> HttpSession as SecurityContext is non-default
> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Stored
> com.evolveum.midpoint.web.security.MidpointSecurityContext at 385b4af to
> HttpSession [org.apache.catalina.session.StandardSessionFacade at 451674c7]
> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Retrieved
> com.evolveum.midpoint.web.security.MidpointSecurityContext at 385b4af
> 2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.MidPointAuthWebSession): Found locale en
> 2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
> (com.evolveum.midpoint.web.security.MidPointAuthWebSession): Using en as
> locale
>
> Any ideas?
>
> Sam
> --
> Sam Harmon
> Case Western Reserve University
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint
>

-- 
Sam Harmon
Case Western Reserve University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20220722/4685798a/attachment-0001.htm>