YARN Timeline Background

Initial support for YARN Timeline was introduced in Apache Hadoop 2.4.0. Support for ACLs in Timeline was introduced in Apache Hadoop 2.6.0. Support for Timeline was introduced in Tez in 0.5.x ( with some experimental support in 0.4.x ). However, Tez ACLs integration with Timeline is only available from Tez 0.6.0 onwards.

How Tez Uses YARN Timeline

Tez uses YARN Timeline as its application history store. Tez stores most of its lifecycle information into this history store such as:

  • DAG information such as:
    • DAG Plan
    • DAG Submission, Start and End times
    • DAG Counters
    • Final status of the DAG and additional diagnostics
  • Vertex, Task and Task Attempt Information
    • Start and End times
    • Counters
    • Diagnostics

Using the above information, a user can analyze a Tez DAG while it is running and after it has completed.

YARN Timeline and Hadoop Versions

Given that the support for YARN Timeline with full security was only realized in Apache Hadoop 2.6.0, some features may or may not be supported depending on which version of Apache Hadoop is used.

Hadoop 2.2.x, 2.3.x Hadoop 2.4.x, 2.5.x Hadoop 2.6.x and higher
Timeline Support No Yes Yes
Timeline with ACLs Support No No Yes

Configuring Tez to use YARN Timeline

By default, Tez writes its history data into a file on HDFS. To use Timeline, add the following property into your tez-site.xml:

<property>
   <name>tez.history.logging.service.class</name>
   <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>

For Tez 0.4.x, the above property is not respected. For 0.4.x, please set the following property:

<property>
   <name>tez.yarn.ats.enabled</name>
   <value>true</value>
</property>

When using Tez with Apache Hadoop 2.4.x or 2.5.x, given that these versions are not fully secure, the following property also needs to be enabled:

<property>
   <name>tez.allow.disabled.timeline-domains</name>
   <value>true</value>
</property>