[midPoint] midPoint 4.4 clustering issue

[Paul Spaude]

For posterity, I wanted to update this thread in the hope it may benefit
others out in the community.

We reached out to Evolveum on this issue and they found an issue with
clustering using ip addresses in certain networking scenarios
common with containerization networks. A Jira was created here:
https://jira.evolveum.com/browse/MID-8107 which is related to MID-7210.

Evolveum has identified a work-around:

------ Workaround ------

In case the containers are located all the time on the same host with the
static IPs there is a way how to "inform" the registration process about
the IP. It is inspired by the router configuration approach (networking)
and I have tested it is working on alpine based image (at least for the
propagation part).

------ Idea ------

The propagated IPs are taken from all the presented interfaces including lo
(only the loopback addresses are ignored). We can assign IP "alias" to the
loop interface with "full bit mask" (/32 in case of IPv4 and /128 in case
of IPv6). This way the IP is known to the system but it is causing no
impact on routing information. Next to it default route contain the "src"
attribute set so this IP alias impacting nothing next to the situation that
it is simply exists...

------ Requirements ------

The container by default does not have the permission to set the networking
parameters. To grant this permission it has to be explicitly set :

 --cap-add NET_ADMIN

in the context it would be : "podman run --cap-add NET_ADMIN -p 8080:8080
... evolveum/midpoint:4.5-alpine"

Next to it the IP have to be set before the midPoint start. The most
suitable way is to set it by "command" (in case you are running
evolveum/midpoint:4.5-alpine image where  the "default" commant is
"/opt/midpoint/bin/midpoint.sh container"):

 /bin/bash -c "/sbin/ip a a 129.22.104.212/32 dev lo ;
/opt/midpoint/bin/midpoint.sh container"

------ Example ------

podman run --cap-add NET_ADMIN -p 8080:8080 ...
evolveum/midpoint:4.5-alpine /bin/bash -c "/sbin/ip a a 129.22.104.212/32
dev lo ; /opt/midpoint/bin/midpoint.sh container"

All the rest parameters (...) should be the same as you are using now. The
result would be like :

   midpoint-d-1 : fe80:0:0:0:ac6a:efff:fe03:d82f ; 10.88.0.5 ;
129.22.104.212

With this workaround the cluster would be able to run as expected.

------ Potential issue with the workaround ------

 - can't be more containers on the same node without explicit tuning of the
network (like more IP to NAT the source IP to)
 - the host IP can't change or the update process has to be found and set

--------------------



Thanks,
Paul Spaude


On Wed, Aug 24, 2022 at 7:46 AM Samuel Harmon via midPoint <
midpoint at lists.evolveum.com> wrote:

> In hopes that this issue was the same as described in MID-7210 (it seems
> very similar to the issue described there, especially given that I'm
> running my nodes in containers), I upgraded my nodes to midPoint 4.5.
> However, this has not resolved the problem- I'm still getting the
> Unauthorized errors from the second node when refreshing the list from the
> first.
>
> Has anyone else encountered this?
>
> Sam
>
> On Fri, Jul 22, 2022 at 2:42 PM Samuel Harmon <sdh7 at case.edu> wrote:
>
>> Yes, Once I got both keys into the keystore, I copied the .jceks file
>> onto the other machine.
>>
>> "-they have a shared keystore containing both keys (the nodes were both
>> started standalone and then later clustered, so each server's keys are in
>> the keystore)"
>>
>> On Fri, Jul 22, 2022 at 2:37 PM Emil Militzer via midPoint <
>> midpoint at lists.evolveum.com> wrote:
>>
>>> Hi,
>>>
>>> do both nodes use the same keystore?
>>>
>>> Kind Regards
>>> Emil
>>>
>>> Am 22.07.2022 um 20:10 schrieb Samuel Harmon via midPoint <
>>> midpoint at lists.evolveum.com>:
>>>
>>> 
>>> I have clustering now mostly set up on one of our midPoint instances,
>>> but we're running into a problem with them communicating with each other.
>>>
>>> We now have two midPoint 4.4 nodes set up on our dev installation
>>> (midpoint-d-1 and midpoint-d-2, both are Podman containers directly running
>>> HTTPS on port 443 and exposed to their container hosts port 443):
>>> -they have a shared keystore containing both keys (the nodes were both
>>> started standalone and then later clustered, so each server's keys are in
>>> the keystore) & a SAN cert to cover both hostnames for SSL. As far as I can
>>> tell, this part is working correctly- both nodes start on port 443 and
>>> aren't throwing errors about encryption keys.
>>> -they can see each other as nodes *via the database*, but all attempts
>>> to communicate to each other via REST fail with “Authentication Error” and
>>> they see each other in the Nodes view as “Communication Error” while their
>>> own node is seen as “Running”.
>>> -the logs are full of messages on the querying side similar to:
>>>
>>> 2022-07-14 14:56:49,549 [TASK_MANAGER] [pool-3-thread-2] DEBUG
>>> (com.evolveum.midpoint.task.quartzimpl.execution.remote.RestConnector):
>>> Querying remote scheduler information on midpoint-d-2.case.edu finished
>>> with status 401: Unauthorized
>>>
>>> To try to fix this, I have attempted the following:
>>>
>>> -I tried changing the instance's nodeId from the container’s generated
>>> internal hostname to the container host’s hostname (which is better for
>>> persistence anyway). That did not fix the communication issue.
>>> -I've tested that calling web services to the other node works from
>>> inside each container using curl.
>>> -I also turned up logging on the receiving end and got the following
>>> logs & stack trace when I refreshed the Nodes list on the querying end:
>>>
>>> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
>>> /ws/cluster/scheduler/information at position 1 of 8 in additional filter
>>> chain; firing Filter: 'HeaderWriterFilter'
>>> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
>>> /ws/cluster/scheduler/information at position 2 of 8 in additional filter
>>> chain; firing Filter: 'RedirectForLoginPagesWithAuthenticationFilter'
>>> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.filter.MidpointAuthFilter):
>>> /ws/cluster/scheduler/information at position 3 of 8 in additional filter
>>> chain; firing Filter: 'HttpClusterAuthenticationFilter'
>>> 2022-07-19 14:09:52,808 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter):
>>> Cluster Authentication - Authorization header found for remote address
>>> '129.22.104.212'
>>> 2022-07-19 14:09:52,809 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.MidpointProviderManager):
>>> Authentication attempt using
>>> com.evolveum.midpoint.web.security.provider.ClusterProvider
>>> 2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] INFO
>>> (com.evolveum.midpoint.web.security.provider.ClusterProvider):
>>> Authentication failed for 129.22.104.212:
>>> web.security.flexAuth.cluster.auth.null
>>> 2022-07-19 14:09:52,811 [MODEL] [https-jsse-nio-443-exec-8] ERROR
>>> (com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider):
>>> Authentication (runtime) error: web.security.flexAuth.cluster.auth.null
>>> org.springframework.security.authentication.AuthenticationServiceException:
>>> web.security.flexAuth.cluster.auth.null
>>> at
>>> com.evolveum.midpoint.web.security.provider.ClusterProvider.internalAuthentication(ClusterProvider.java:59)
>>> at
>>> com.evolveum.midpoint.web.security.provider.MidPointAbstractAuthenticationProvider.authenticate(MidPointAbstractAuthenticationProvider.java:92)
>>> at
>>> com.evolveum.midpoint.web.security.MidpointProviderManager.authenticate(MidpointProviderManager.java:58)
>>> at jdk.internal.reflect.GeneratedMethodAccessor576.invoke(Unknown Source)
>>> at
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> at
>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>> at
>>> org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
>>> at
>>> org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>>> at
>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>>> at com.sun.proxy.$Proxy181.authenticate(Unknown Source)
>>> at
>>> com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter.doFilterInternal(HttpClusterAuthenticationFilter.java:78)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
>>> at
>>> com.evolveum.midpoint.web.security.filter.RedirectForLoginPagesWithAuthenticationFilter.doFilterInternal(RedirectForLoginPagesWithAuthenticationFilter.java:39)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
>>> at
>>> org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
>>> at
>>> org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter$VirtualFilterChain.doFilter(MidpointAuthFilter.java:416)
>>> at
>>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilterInternal(MidpointAuthFilter.java:226)
>>> at
>>> com.evolveum.midpoint.web.security.filter.MidpointAuthFilter.doFilter(MidpointAuthFilter.java:109)
>>> at
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>>> at
>>> com.evolveum.midpoint.web.security.filter.TranslateExceptionFilter.doFilterInternal(TranslateExceptionFilter.java:32)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>>> at
>>> org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:147)
>>> at
>>> org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
>>> at
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>>> at
>>> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
>>> at
>>> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
>>> at
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>>> at
>>> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
>>> at
>>> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
>>> at
>>> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
>>> at
>>> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
>>> at
>>> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
>>> t
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>>> at
>>> org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>>> at
>>> org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>>> at
>>> org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>>> at
>>> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>>> at
>>> com.evolveum.midpoint.web.boot.TrailingSlashRedirectingFilter.doFilterInternal(TrailingSlashRedirectingFilter.java:60)
>>> at
>>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
>>> at
>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
>>> at
>>> com.evolveum.midpoint.web.boot.NodeIdHeaderValve.invoke(NodeIdHeaderValve.java:46)
>>> at
>>> com.evolveum.midpoint.web.boot.TomcatRootValve.invoke(TomcatRootValve.java:62)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
>>> at
>>> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
>>> at
>>> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
>>> at
>>> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
>>> at
>>> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> at
>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.filter.HttpClusterAuthenticationFilter):
>>> Authentication request for failed:
>>> org.springframework.security.authentication.AuthenticationServiceException:
>>> web.security.flexAuth.cluster.auth.null
>>> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Created
>>> HttpSession as SecurityContext is non-default
>>> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Stored
>>> com.evolveum.midpoint.web.security.MidpointSecurityContext at 385b4af to
>>> HttpSession [org.apache.catalina.session.StandardSessionFacade at 451674c7]
>>> 2022-07-19 14:09:52,812 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.BasicWebSecurityConfig$1): Retrieved
>>> com.evolveum.midpoint.web.security.MidpointSecurityContext at 385b4af
>>> 2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.MidPointAuthWebSession): Found locale en
>>> 2022-07-19 14:09:52,813 [MODEL] [https-jsse-nio-443-exec-8] DEBUG
>>> (com.evolveum.midpoint.web.security.MidPointAuthWebSession): Using en as
>>> locale
>>>
>>> Any ideas?
>>>
>>> Sam
>>> --
>>> Sam Harmon
>>> Case Western Reserve University
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> https://lists.evolveum.com/mailman/listinfo/midpoint
>>>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> https://lists.evolveum.com/mailman/listinfo/midpoint
>>>
>>
>>
>> --
>> Sam Harmon
>> Case Western Reserve University
>>
>
>
> --
> Sam Harmon
> Case Western Reserve University
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20220929/03c54d57/attachment-0001.htm>