<div dir="ltr">In hopes that this issue was the same as described in MID-7210 (it seems very similar to the issue described there, especially given that I'm running my nodes in containers), I upgraded my nodes to midPoint 4.5.  However, this has not resolved the problem- I'm still getting the Unauthorized errors from the second node when refreshing the list from the first. <br><div><br></div><div>Has anyone else encountered this?</div><div><br></div><div>Sam<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 22, 2022 at 2:42 PM Samuel Harmon <<a href="mailto:sdh7@case.edu">sdh7@case.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Yes, Once I got both keys into the keystore, I copied the .jceks file onto the other machine.<br></div><div><br></div><div>"-they have a shared keystore containing both keys (the nodes were both 
started standalone and then later clustered, so each server's keys are 
in the keystore)" </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 22, 2022 at 2:37 PM Emil Militzer via midPoint <<a href="mailto:midpoint@lists.evolveum.com" target="_blank">midpoint@lists.evolveum.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="ltr"></div><div dir="ltr">Hi,</div><div dir="ltr"><br></div><div dir="ltr">do both nodes use the same keystore? </div><div dir="ltr"><br></div><div dir="ltr">Kind Regards</div><div dir="ltr">Emil</div><div dir="ltr"><br><blockquote type="cite">Am 22.07.2022 um 20:10 schrieb Samuel Harmon via midPoint <<a href="mailto:midpoint@lists.evolveum.com" target="_blank">midpoint@lists.evolveum.com</a>>:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr">I have clustering now mostly set up on one of our midPoint instances, but we're running into a problem with them communicating with each other.<br><br>We now have two midPoint 4.4 nodes set up on our dev installation (midpoint-d-1 and midpoint-d-2, both are Podman containers directly running HTTPS on port 443 and exposed to their container hosts port 443):<br>-they have a shared keystore containing both keys (the nodes were both started standalone and then later clustered, so each server's keys are in the keystore) & a SAN cert to cover both hostnames for SSL. As far as I can tell, this part is working correctly- both nodes start on port 443 and aren't throwing errors about encryption keys.<br>-they can see each other as nodes *via the database*, but all attempts to communicate to each other via REST fail with “Authentication Error” and they see each other in the Nodes view as “Communication Error” while their own node is seen as “Running”.<br>-the logs are full of messages on the querying side similar to:<br><br>2022-07-14 14:56:49,549 [TASK_MANAGER] [pool-3-thread-2] DEBUG (com.evolveum.midpoint.task.quartzimpl.execution.remote.RestConnector): Querying remote scheduler information on <a href="http://midpoint-d-2.case.edu" target="_blank">midpoint-d-2.case.edu</a> finished with status 401: Unauthorized<br><br>To try to fix this, I have attempted the following:<br><br>-I tried changing the instance's nodeId from the container’s generated internal hostname to the container host’s hostname (which is better for persistence anyway). That did not fix the communication issue.<br>-I've tested that calling web services to the other node works from inside each container using curl.<br>-I also turned up logging on the receiving end and got the following logs & stack trace when I refreshed the Nodes list on the querying end:<div><br></div>2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter): /ws/cluster/scheduler/information at position 1 of 8 in additional filter chain; firing Filter: 'HeaderWriterFilter'<br>2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter): /ws/cluster/scheduler/information at position 2 of 8 in additional filter chain; firing Filter: 'RedirectForLoginPagesWithAuthenticationFilter'<br>2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter): /ws/cluster/scheduler/information at position 3 of 8 in additional filter chain; firing Filter: 'HttpClusterAuthenticationFilter'<br>2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter): Cluster Authentication - Authorization header found for remote address '129.22.104.212'<br>2022-07-19 14:09:52,809 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.MidpointProviderManager): Authentication attempt using com.evolveum.midpoint.web.security.provider.ClusterProvider<br>2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] INFO (com.evolveum.midpoint.web.security.provider.ClusterProvider): Authentication failed for <a href="http://129.22.104.212" target="_blank">129.22.104.212</a>: web.security.flexAuth.cluster.auth.null<br>2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] ERROR (com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider): Authentication (runtime) error: web.security.flexAuth.cluster.auth.null<br>org.springframework.security.authentication.AuthenticationServiceException: web.security.flexAuth.cluster.auth.null<br>at com.evolveum.midpoint.web.security.provider.ClusterProvider.internalAuthentication(ClusterProvider.java:59)<br>at com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider.authenticate(MidPointAbstractAuthenticationProvider.java:92)<br>at com.evolveum.midpoint.web.security.MidpointProviderManager.authenticate(MidpointProviderManager.java:58)<br>at jdk.internal.reflect.GeneratedMethodAccessor576.invoke(Unknown Source)<br>at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br>at java.base/java.lang.reflect.Method.invoke(Method.java:566)<br>at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)<br>at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)<br>at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)<br>at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)<br>at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)<br>at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)<br>at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)<br>at com.sun.proxy.$Proxy181.authenticate(Unknown Source)<br>at com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter.doFilterInternal(HttpClusterAuthenticationFilter.java:78)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)<br>at com.evolveum.midpoint.web.security.filter.RedirectForLoginPagesWithAuthenticationFilter.doFilterInternal(RedirectForLoginPagesWithAuthenticationFilter.java:39)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)<br>at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)<br>at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)<br>at com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilterInternal(MidpointAuthFilter.java:226)<br>at com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilter(MidpointAuthFilter.java:109)<br>at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)<br>at com.evolveum.midpoint.web.security.filter.TranslateExceptionFilter.doFilterInternal(TranslateExceptionFilter.java:32)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)<br>at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:147)<br>at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)<br>at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)<br>at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)<br>at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)<br>at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)<br>at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)<br>at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)<br>at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)<br>at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)<br>at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)<br>t org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)<br>at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)<br>at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)<br>at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)<br>at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)<br>at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)<br>at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)<br>at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)<br>at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)<br>at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)<br>at com.evolveum.midpoint.web.boot.TrailingSlashRedirectingFilter.doFilterInternal(TrailingSlashRedirectingFilter.java:60)<br>at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)<br>at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)<br>at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)<br>at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)<br>at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)<br>at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)<br>at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)<br>at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)<br>at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)<br>at com.evolveum.midpoint.web.boot.NodeIdHeaderValve.invoke(NodeIdHeaderValve.java:46)<br>at com.evolveum.midpoint.web.boot.TomcatRootValve.invoke(TomcatRootValve.java:62)<br>at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)<br>at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)<br>at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)<br>at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)<br>at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)<br>at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)<br>at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)<br>at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)<br>at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)<br>at java.base/java.lang.Thread.run(Thread.java:829)<br>2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter): Authentication request for failed: org.springframework.security.authentication.AuthenticationServiceException: web.security.flexAuth.cluster.auth.null<br>2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Created HttpSession as SecurityContext is non-default<br>2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Stored com.evolveum.midpoint.web.security.MidpointSecurityContext@385b4af to HttpSession [org.apache.catalina.session.StandardSessionFacade@451674c7]<br>2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Retrieved com.evolveum.midpoint.web.security.MidpointSecurityContext@385b4af<br>2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.MidPointAuthWebSession): Found locale en<br>2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG (com.evolveum.midpoint.web.security.MidPointAuthWebSession): Using en as locale<br clear="all"><div><br></div><div>Any ideas?</div><div><br></div><div>Sam<br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Sam Harmon<br><div>Case Western Reserve University</div></div></div></div></div></div></div></div>
<span>_______________________________________________</span><br><span>midPoint mailing list</span><br><span><a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a></span><br><span><a href="https://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">https://lists.evolveum.com/mailman/listinfo/midpoint</a></span><br></div></blockquote></div>_______________________________________________<br>
midPoint mailing list<br>
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a><br>
<a href="https://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">https://lists.evolveum.com/mailman/listinfo/midpoint</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Sam Harmon<br><div>Case Western Reserve University</div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Sam Harmon<br><div>Case Western Reserve University</div></div></div></div></div></div></div>