What Causes the 'NoSuchFieldError: METASTOREURIS' Error in Spark Cluster Mode with Hive Metastore?

I am running Apache Spark on YARN with Hive Metastore and PostgreSQL on an edge node, and HDFS with NodeManagers on five worker nodes.

The setup is as follows:

• One Edge Node (running in containers with --network host):
  • Hive Metastore
  • PostgreSQL (as the Hive Metastore database)
  • Jupyter Notebook
  • YARN ResourceManager in host
• Five Worker Nodes running HDFS and YARN NodeManager.

Issue Summary

When executing a Spark job in client mode, everything works correctly. However, in cluster mode, the job fails with the following error:

PYTHONCopied
Traceback (most recent call last):
  File "/usr/local/hadoop/yarn/nodemanager/data/usercache/spark/appcache/application_1738582462412_0011/container_1738582462412_0011_02_000001/test_query.py", line 8, in <module>
    spark.sql("SHOW DATABASES").show()
java.lang.NoSuchFieldError: METASTOREURIS

The Hive Metastore configuration in Spark (spark-defaults.conf) is:

PROPERTIESCopied
spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.spark_catalog.type=hive
spark.sql.catalog.spark_catalog.uri=thrift://hive-metastore:9083

Networking Details

Since Hive Metastore is running in a container with --network host, I manually added the hostname mappings:

BASHCopied
127.0.0.1       localhost
--add-host hive-metastore:192.168.225.208
--add-host spark-iceberg-notebook:192.168.225.208
--add-host dw2endev01.sme.astro:192.168.225.208
--add-host dw2endev01:192.168.225.208
--add-host dw2nndev01.sme.astro:192.168.225.209
...

And the host machine has the following mappings:

BASHCopied
127.0.0.1 localhost
127.0.0.1 hive-metastore
127.0.0.1 spark-iceberg-notebook

192.168.225.208 dw2endev01.sme.astro dw2endev01 spark-iceberg-notebook hive-metastore
192.168.225.209 dw2nndev01.sme.astro dw2nndev01 sme-hadoop-cluster
192.168.225.210 dw2nndev02.sme.astro dw2nndev02
...

Troubleshooting Efforts

• The Spark job succeeds in client mode but fails in cluster mode.
• The /etc/hosts file on worker nodes correctly resolves hive-metastore.

Question

What could be causing the java.lang.NoSuchFieldError: METASTOREURIS error in cluster mode? Is this a classpath conflict where different versions of Hive-related libraries are used on the cluster, or could it be a networking issue between Spark and Hive Metastore?