My FeedDiscussionsHeadless CMS
New
Sign in
Log inSign up
Learn more about Hashnode Headless CMSHashnode Headless CMS
Collaborate seamlessly with Hashnode Headless CMS for Enterprise.
Upgrade ✨Learn more

Hadoop giving error on avro tools concat

Abhishek Pathak's photo
Abhishek Pathak
·Jun 3, 2016

I'm trying to concatenate multiple avro files in an HDFS directory using avro tools. I'm following the approach as this question. However, I'm getting the following error:

Exception in thread "main" java.io.FileNotFoundException: /user/myuser/output/output.avro (No such file or directory)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
    at org.apache.avro.tool.Util.fileOrStdout(Util.java:79)
    at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:58)
    at org.apache.avro.tool.Main.run(Main.java:80)
    at org.apache.avro.tool.Main.main(Main.java:69)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
  • The directory /user/myuser/output/ does exist.
  • The command I'm using is hadoop jar /home/myuser/avro-tools-1.7.4.jar concat ${IN} /user/myuser/output/output.avro
  • $IN has the value /user/hive/warehouse/somedatabase.db/some_table/on_date=2016-05-01/for_date=2016-05-08/confidence=0.8/000000_0
  • The hadoop version is Hadoop 2.0.0-cdh4.7.0.

Theoretically, it should work, however it does not. I feel as if avro-tools is checking the local filesystem for the file(directory?) and that is causing the error. Any pointers?