- First thing i did in HDP 2.1 sandbox is to enable Ambari by clicking on Enable button like this
- Next step was to login into Ambari by going to http://localhost:8080/ and then entering admin/admin as username and password
- In the Ambari Application i went to the Yarn Service screen and i did uncheck Resource Manager -<yarn.log-aggregation-enable checkbox. It disables the log aggregation which copies the log files from file system to HDFS and zips them. Disabling the log aggregation keeps them on the local filesystem.
- Then in the Advanced section i did change value of yarn.nodemanager.debug-delay-sec to 60000 seconds, By default value of this property is 0, which means as soon as the job is done nodemanager deletes all the intermediate files. By setting it to 60000 i am preventing it from getting deleted for 100 seconds.
- THen i did restart all the services using Ambari for my changes to take affect
- After that i did execute couple of hive queries and now i could see my logs in
/hadoop/yarn/logsdirectory like this
- And i could see the job.xml files being conserved in /hadoop/yarn/local/usercache/root/appcache folder for application like this
How to view the log files and job.xml generated by Hive on HortonWorks Sandbox
I was working on building some hive code and i ran into some problems, My query kept failing with message like invalid character in job.xml. I tried to locate the job.xml but could not find it. Same thing i could not find any logs on the local machine. So i wanted to figure out how to debug this problem. I followed these steps.