Apache spark read file from hadoop file system

Photo by Zoë on Unsplash

The default path for hadoop file system is configured at core-site.xml like

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://host:port</value>
</property>
</configuration>

To get the file from spark, we will need to use SparkContext.

import…

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Donald Le

Donald Le

231 Followers

A passionate automation engineer who strongly believes in “A man can do anything he wants if he puts in the work”.