• Uncategorized

About hadoop : hdfs-dfs–put-with-overwrite

Question Detail

I am using

hdfs dfs -put myfile mypath

and for some files I get

put: 'myfile': File Exists
  • does that mean there is a file with the same name or does that mean the same exact file (size, content) is already there?
  • how can I specify an -overwrite option here?

Thanks!

Question Answer

put: 'myfile': File Exists

Means,the file named “myfile” already exists in hdfs. You cannot have multiple files of the same name in hdfs

You can overwrite it using hadoop fs -put -f /path_to_local /path_to_hdfs

You can overwrite your file in hdfs using -f command.For example

hadoop fs -put -f <localfile> <hdfsDir>

OR

hadoop fs -copyFromLocal -f <localfile> <hdfsDir>

It worked fine for me. However -f command won’t work in case of get or copyToLocal command. check this question

  1. A file with the same name exists at the location you’re trying to write to.
  2. You can overwrite by specifying the -f flag.

Just updates to this answer, in Hadoop 3.X the command a bit different

hdfs dfs -put -f /local/to/path hdfs://localhost:9870/users/XXX/folder/folder2

You may also like...

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.