[midPoint] midPoint 4.4 clustering issue

[Samuel Harmon]

In hopes that this issue was the same as described in MID-7210 (it seems
very similar to the issue described there, especially given that I'm
running my nodes in containers), I upgraded my nodes to midPoint 4.5.
However, this has not resolved the problem- I'm still getting the
Unauthorized errors from the second node when refreshing the list from the
first.

Has anyone else encountered this?

Sam

On Fri, Jul 22, 2022 at 2:42 PM Samuel Harmon <sdh7 at case.edu> wrote:

> Yes, Once I got both keys into the keystore, I copied the .jceks file onto
> the other machine.
>
> "-they have a shared keystore containing both keys (the nodes were both
> started standalone and then later clustered, so each server's keys are in
> the keystore)"
>
> On Fri, Jul 22, 2022 at 2:37 PM Emil Militzer via midPoint <
> midpoint at lists.evolveum.com> wrote:
>
>> Hi,
>>
>> do both nodes use the same keystore?
>>
>> Kind Regards
>> Emil
>>
>> Am 22.07.2022 um 20:10 schrieb Samuel Harmon via midPoint <
>> midpoint at lists.evolveum.com>:
>>
>> 
>> I have clustering now mostly set up on one of our midPoint instances, but
>> we're running into a problem with them communicating with each other.
>>
>> We now have two midPoint 4.4 nodes set up on our dev installation
>> (midpoint-d-1 and midpoint-d-2, both are Podman containers directly running
>> HTTPS on port 443 and exposed to their container hosts port 443):
>> -they have a shared keystore containing both keys (the nodes were both
>> started standalone and then later clustered, so each server's keys are in
>> the keystore) & a SAN cert to cover both hostnames for SSL. As far as I can
>> tell, this part is working correctly- both nodes start on port 443 and
>> aren't throwing errors about encryption keys.
>> -they can see each other as nodes *via the database*, but all attempts to
>> communicate to each other via REST fail with “Authentication Error” and
>> they see each other in the Nodes view as “Communication Error” while their
>> own node is seen as “Running”.
>> -the logs are full of messages on the querying side similar to:
>>
>> 2022-07-14 14:56:49,549 [TASK_MANAGER] [pool-3-thread-2] DEBUG
>> (com.evolveum.midpoint.task.quartzimpl.execution.remote.RestConnector):
>> Querying remote scheduler information on midpoint-d-2.case.edu finished
>> with status 401: Unauthorized
>>
>> To try to fix this, I have attempted the following:
>>
>> -I tried changing the instance's nodeId from the container’s generated
>> internal hostname to the container host’s hostname (which is better for
>> persistence anyway). That did not fix the communication issue.
>> -I've tested that calling web services to the other node works from
>> inside each container using curl.
>> -I also turned up logging on the receiving end and got the following logs
>> & stack trace when I refreshed the Nodes list on the querying end:
>>
>> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
>> /ws/cluster/scheduler/information at position 1 of 8 in additional filter
>> chain; firing Filter: 'HeaderWriterFilter'
>> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
>> /ws/cluster/scheduler/information at position 2 of 8 in additional filter
>> chain; firing Filter: 'RedirectForLoginPagesWithAuthenticationFilter'
>> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
>> /ws/cluster/scheduler/information at position 3 of 8 in additional filter
>> chain; firing Filter: 'HttpClusterAuthenticationFilter'
>> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter):
>> Cluster Authentication - Authorization header found for remote address
>> '129.22.104.212'
>> 2022-07-19 14:09:52,809 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.MidpointProviderManager):
>> Authentication attempt using
>> com.evolveum.midpoint.web.security.provider.ClusterProvider
>> 2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] INFO
>> (com.evolveum.midpoint.web.security.provider.ClusterProvider):
>> Authentication failed for 129.22.104.212:
>> web.security.flexAuth.cluster.auth.null
>> 2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] ERROR
>> (com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider):
>> Authentication (runtime) error: web.security.flexAuth.cluster.auth.null
>> org.springframework.security.authentication.AuthenticationServiceException:
>> web.security.flexAuth.cluster.auth.null
>> at
>> com.evolveum.midpoint.web.security.provider.ClusterProvider.internalAuthentication(ClusterProvider.java:59)
>> at
>> com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider.authenticate(MidPointAbstractAuthenticationProvider.java:92)
>> at
>> com.evolveum.midpoint.web.security.MidpointProviderManager.authenticate(MidpointProviderManager.java:58)
>> at jdk.internal.reflect.GeneratedMethodAccessor576.invoke(Unknown Source)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> at
>> org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
>> at
>> org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>> at com.sun.proxy.$Proxy181.authenticate(Unknown Source)
>> at
>> com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter.doFilterInternal(HttpClusterAuthenticationFilter.java:78)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
>> at
>> com.evolveum.midpoint.web.security.filter.RedirectForLoginPagesWithAuthenticationFilter.doFilterInternal(RedirectForLoginPagesWithAuthenticationFilter.java:39)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
>> at
>> org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
>> at
>> org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
>> at
>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilterInternal(MidpointAuthFilter.java:226)
>> at
>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilter(MidpointAuthFilter.java:109)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>> at
>> com.evolveum.midpoint.web.security.filter.TranslateExceptionFilter.doFilterInternal(TranslateExceptionFilter.java:32)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>> at
>> org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:147)
>> at
>> org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>> at
>> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
>> at
>> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>> at
>> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>> at
>> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
>> at
>> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
>> at
>> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
>> at
>> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
>> t
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>> at
>> org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>> at
>> org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>> at
>> org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>> at
>> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>> at
>> com.evolveum.midpoint.web.boot.TrailingSlashRedirectingFilter.doFilterInternal(TrailingSlashRedirectingFilter.java:60)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
>> at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
>> at
>> com.evolveum.midpoint.web.boot.NodeIdHeaderValve.invoke(NodeIdHeaderValve.java:46)
>> at
>> com.evolveum.midpoint.web.boot.TomcatRootValve.invoke(TomcatRootValve.java:62)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
>> at
>> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
>> at
>> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
>> at
>> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
>> at
>> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at
>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter):
>> Authentication request for failed:
>> org.springframework.security.authentication.AuthenticationServiceException:
>> web.security.flexAuth.cluster.auth.null
>> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Created
>> HttpSession as SecurityContext is non-default
>> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Stored
>> com.evolveum.midpoint.web.security.MidpointSecurityContext at 385b4af to
>> HttpSession [org.apache.catalina.session.StandardSessionFacade at 451674c7]
>> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Retrieved
>> com.evolveum.midpoint.web.security.MidpointSecurityContext at 385b4af
>> 2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.MidPointAuthWebSession): Found locale en
>> 2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>> (com.evolveum.midpoint.web.security.MidPointAuthWebSession): Using en as
>> locale
>>
>> Any ideas?
>>
>> Sam
>> --
>> Sam Harmon
>> Case Western Reserve University
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> https://lists.evolveum.com/mailman/listinfo/midpoint
>>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> https://lists.evolveum.com/mailman/listinfo/midpoint
>>
>
>
> --
> Sam Harmon
> Case Western Reserve University
>


-- 
Sam Harmon
Case Western Reserve University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20220824/bc347930/attachment-0001.htm>