Spark writing to S3 failed: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument

Symptom:

When using Spark writing to S3, the insert query failed:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V     at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:893)   at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:869)   at org.apache.hadoop.fs.s3a.S3AUtils.getEncryptionAlgorithm(S3AUtils.java:1580)  at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:341)     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)    at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)   at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)    at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:53)     at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:370)   at org.apache.spark.sql.execution.datasources.FindDataSourceTable.$anonfun$readDataSourceTable$1(DataSourceStrategy.scala:252)   at org.sparkproject.guava.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792)     at org.sparkproject.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)    at org.sparkproject.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)    at org.sparkproject.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)     at org.sparkproject.guava.cache.LocalCache$Segment.get(LocalCache.java:2257)

Env:

spark-3.2.1-bin-hadoop3.2
hadoop-aws-3.2.3.jar
aws-java-sdk-bundle-1.11.375.jar
guava-14.0.1.jar

Solution:

Remove guava-14.0.1.jar from Spark and use the Hive's newer guava-27.0-jre.jar.

1 2	$  ls -altr $SPARK_HOME/jars/guava*.jar lrwxrwxrwx 1 xxx xxx 46 Jul 19 09:39 /home/xxx/spark/myspark/jars/guava-27.0-jre.jar -> /home/xxx/hive/myhive/lib/guava-27.0-jre.jar