Monday, September 16, 2019

Failed to login a secured Drill on Yarn cluster with MapRSASL authentication

Symptom:

Failed to login a secured Drill on Yarn cluster with MapRSASL authentication.
The sample stacktrace inside drillbit.log when trying to use sqlline to connect is:
2019-09-16 14:23:13,331 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /10.10.72.41:31010 <--> /10.10.72.41:48032 (user server).  Closing connection.
io.netty.handler.codec.DecoderException: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: Bad server key  [Caused by javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:98) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-common-4.0.48.Final.jar:4.0.48.Final]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
Caused by: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: Bad server key  [Caused by javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handleAuthFailure(ServerAuthenticationHandler.java:324) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handle(ServerAuthenticationHandler.java:109) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.BasicServer.handle(BasicServer.java:182) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.BasicServer.handle(BasicServer.java:54) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 ... 31 common frames omitted
Caused by: javax.security.sasl.SaslException: Bad server key
 at com.mapr.security.maprsasl.MaprSaslServer.evaluateResponse(MaprSaslServer.java:190) ~[maprfs-6.1.0-mapr.jar:na]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$1.run(ServerAuthenticationHandler.java:239) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$1.run(ServerAuthenticationHandler.java:236) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_212]
 at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_212]
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669) ~[hadoop-common-2.7.0-mapr-1808.jar:na]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.evaluateResponse(ServerAuthenticationHandler.java:236) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.access$500(ServerAuthenticationHandler.java:53) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$SaslInProgressProcessor.process(ServerAuthenticationHandler.java:176) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$SaslStartProcessor.process(ServerAuthenticationHandler.java:164) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handle(ServerAuthenticationHandler.java:107) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 ... 36 common frames omitted
Caused by: javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2
 at com.mapr.security.maprsasl.MaprSaslServer.evaluateResponse(MaprSaslServer.java:143) ~[maprfs-6.1.0-mapr.jar:na]
 ... 46 common frames omitted
The sample sqlline error message is:
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0]
 at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:174)
 at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
 at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
 at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
 at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
 at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130)
 at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179)
 at sqlline.Commands.connect(Commands.java:1247)
 at sqlline.Commands.connect(Commands.java:1139)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
 at sqlline.SqlLine.dispatch(SqlLine.java:722)
 at sqlline.SqlLine.initArgs(SqlLine.java:416)
 at sqlline.SqlLine.begin(SqlLine.java:514)
 at sqlline.SqlLine.start(SqlLine.java:264)
 at sqlline.SqlLine.main(SqlLine.java:195)
Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0]
 at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:210)
 at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
 at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
 at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:165)
 ... 18 more
Caused by: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0]
 at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$SaslFailedProcessor.process(AuthenticationOutcomeListener.java:230)
 at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.success(AuthenticationOutcomeListener.java:128)
 at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.success(AuthenticationOutcomeListener.java:53)
 at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
 at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:293)
 at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
 at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
 at java.lang.Thread.run(Thread.java:748)

Env:

MapR 6.1
Drill 1.15

Root Cause:

configure.sh does not work for Drill on YARN.
So for a Drill on YARN cluster with MapRSASL authentication, we need to manually configure it in distrib-env.sh from $DRILL_SITE.

Solution:

In $DRILL_SITE, locate the distrib-env.sh to see what is current settings for MapRSASL.
If the current setting is:
export DRILL_JAVA_OPTS="${DRILL_JAVA_OPTS} -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.client=false"
Then it should be changed to:
export DRILL_JAVA_OPTS="${DRILL_JAVA_OPTS} -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dhadoop.login=hybrid_keytab -Dzookeeper.sasl.client=true"

After that, restart the Drill on YARN cluster:
$DRILL_HOME/bin/drill-on-yarn.sh --site $DRILL_SITE start

Posted by at
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts