Chapter 135. DQP operators, query plans and optimisers

135.1. Operators
135.1.1. NIL
135.1.2. SELECT
135.1.3. PROJECT
135.1.4. RENAME
135.1.5. DUPLICATE_ELIMINATION
135.1.6. SORT
135.1.7. TABLE_SCAN
135.1.8. GROUP_BY
135.1.9. SCALAR_GROUP_BY
135.1.10. ONE_ROW_ONLY
135.1.11. PRODUCT
135.1.12. INNER_THETA_JOIN
135.1.13. FULL_OUTER_JOIN
135.1.14. RIGHT_OUTER_JOIN
135.1.15. LEFT_OUTER_JOIN
135.1.16. SEMI_JOIN
135.1.17. ANTI_SEMI_JOIN
135.1.18. UNION
135.1.19. INTERSECTION
135.1.20. DIFFERENCE
135.1.21. APPLY
135.1.22. FILTERED_TABLE_SCAN
135.1.23. QUERY_APPLY
135.1.24. EXCHANGE
135.2. Query plans
135.3. Optimisers
135.3.1. Query Normalise Optimiser
135.3.2. Select Push Down Optimiser
135.3.3. Rename Pull Up Optimiser
135.3.4. Project Pull Up Optimiser
135.3.5. Join Ordering Optimiser
135.3.6. Join Annotation Optimiser
135.3.7. Insert Project After Group By Optimiser
135.3.8. Project Push Down Optimiser
135.3.9. Remove Redundant Project Optimiser
135.3.10. Partitioning Optimiser
135.3.11. Table Scan Implosion Optimiser
135.3.12. Filtered Table Scan Optimiser
135.4. Visualising query plans

[Warning]Warning
This page is very much a work in progress. It has been released early as users were asking questions relating to information on this page. The simplest approach to answering the questions is to release this documentation and update it frequently as more is written.

As described in Chapter 131, How DQP works when DQP receives a query it builds an initial query plan for the query. Query plans are constructed as trees of relational operators. The initial query is optimised by a chain of optimisers that are executed sequentially to produce the final query plan. This final query plan is executed by transforming the plan into multiple OGSA-DAI workflows that are then executed in parallel.

This chapter describes the relational operators and optimisers that are used in DQP

135.1. Operators

The majority of the operators in DQP correspond to relational algebra operations. This section does not describe meanings of these relational algebra operations. For this see a suitable textbook or website, for example Wikipedia. This section lists the operators used by DQP and describes any properties that are specific to this DQP implementation.

135.1.1. NIL

The NIL operator is a unary operator that performs no action. It is simply used to as the root node of query plans.

135.1.2. SELECT

The SELECT operator is a unary operator and is configured with a predicate containing an expression. Example expressions include "age < 20" and "table1.age < table2.maxAge".

135.1.3. PROJECT

The PROJECT operator is unary operator and is configured with ordered list of arithmetic expressions that define the output attributes.

135.1.4. RENAME

The RENAME operator is a unary operator and is configured with a map to convert input attribute names to output attribute names.

135.1.5. DUPLICATE_ELIMINATION

The DUPLICATE_ELIMINATION operator in a unary operator that eliminates duplicate tuples.

135.1.6. SORT

The SORT operator is a unary operator and is configured with a ordered list of attributes on which the sorting is to be performed. Note that the operator does not currently supporting specifying ascending or descending sorting on each attribute. All sorting is currently ascending.

135.1.7. TABLE_SCAN

The TABLE_SCAN operator is a unary operator that represents query that is sent to a data source. Initially all TABLE_SCAN operators are constructed with the name of single table to be scanned. This easily maps into a SQL query to be sent to the database. For example, if the table is myTable then the SQL query is

  SELECT * FROM myTable

An essential optimisation strategy used by DQP is to push as many of the operators as possible into the queries that are sent to the database. For example, if a TABLE_SCAN operator is followed by a SELECT operator with expression "age < 20" then DQP will try to push the functionality of the SELECT operator into the TABLE_SCAN operator and remove the SELECT operator. In our example this would mean that the TABLE_SCAN operator's SQL query would now be

SELECT * FROM myTable WHERE age < 20

Other operators such as PROJECT and INNER_THETA_JOIN can also be pushed into the SQL query where appropriate.

The TABLE_SCAN operator therefore holds a SQL query that is sent to the database rather than just a the name of a single table to be scanned. Many operators can be pushed into a TABLE_SCAN operator. For more information see Section 135.3.11, “Table Scan Implosion Optimiser”.

135.1.8. GROUP_BY

The GROUP_BY operator is a unary operator that groups tuples and possibly also computes aggregation functions on these groups. It is configured with a list of attributes on which to group and a list of aggregation functions to apply. The aggregation functions include the arithmetic expressions that provide the arguments to these functions. For example, if width and height are attributes of the incoming tuples then the following are example of function definitions:

  • MAX( width )

  • AVG( width / height)

135.1.9. SCALAR_GROUP_BY

The SCALAR_GROUP_BY operator is a unary operator that performs aggregations when the query has no group-by clause. It is configured with a list of aggregation functions to apply. The aggregation functions include the arithmetic expressions that provide the arguments to this functions. For example, if width and height are attributes of the incoming tuples then the following are example of function definitions:

  • MAX( width )

  • AVG( width / height)

135.1.10. ONE_ROW_ONLY

The ONE_ROW_ONLY operator is a unary operator that is used detect illegal subqueries which were expected to produce a scalar result. User errors of this kind can only be detected at runtime.

135.1.11. PRODUCT

The PRODUCT operator is a binary operator that performs a product of its two inputs. At execution time the default implementation is to store the data in the second (right hand) child and stream the data in the first (left hand) child.

135.1.12. INNER_THETA_JOIN

The INNER_THETA_JOIN operator is a binary operator that performs an inner theta join of its two inputs. It is configured with a predicate that relates attributes from one side of the join with attributes from the other.

135.1.13. FULL_OUTER_JOIN

The FULL_OUTER_JOIN operator is a binary operator that performs a full outer join of its two inputs. It is configured with a predicate that relates attributes from one side of the join with attributes from the other.

135.1.14. RIGHT_OUTER_JOIN

The RIGHT_OUTER_JOIN operator is a binary operator that performs an right outer join of its two inputs. It is configured with a predicate that relates attributes from one side of the join with attributes from the other.

135.1.15. LEFT_OUTER_JOIN

The LEFT_OUTER_JOIN operator is a binary operator that performs an left outer join of its two inputs. It is configured with a predicate that relates attributes from one side of the join with attributes from the other.

135.1.16. SEMI_JOIN

The SEMI_JOIN operator is a binary operator that performs an semi join of its two inputs. It is configured with a predicate that related attributes from one side of the join with attributes from the other.

135.1.17. ANTI_SEMI_JOIN

The SEMI_JOIN operator is a binary operator that performs an semi join of its two inputs. It is configured with a predicate that related attributes from one side of the join with attributes from the other.

135.1.18. UNION

The UNION operator is a binary operator that performs a union of its two inputs. The operator can be configured to be specify either a set union (where duplicates are eliminated) or a bag union (where duplicates are permitted).

135.1.19. INTERSECTION

The INTERSECTION operator is a binary operator that performs an intersection of its two inputs. The operator can be configured to be specify either a set intersection (where duplicates are eliminated) or a bag intersection (where duplicates are permitted).

135.1.20. DIFFERENCE

The DIFFERENCE operator is a binary operator that performs a difference of its two inputs. The operator can be configured to be specify either a set difference (where duplicates are eliminated) or a bag difference (where duplicates are permitted).

135.1.21. APPLY

[Warning]Warning
This version of DQP is unable to execute queries that require the APPLY operator. DQP will fail to find a builder class that maps the APPLY operator into an implementation built from OGSA-DAI activities.

The APPLY operator is a binary operator used for correlated sub-queries. It is configured with a set of attributes to bind and also a join operator that defines the join type operation that is correlated.

The APPLY operator abstracts "for each" functionality and parameterise relational expression invocation. The following query plan

(APPLY R E(r))

means: for each tuple r from R execute relational expression E on r. The result of the invocation is joined with tuple r using an arbitrary join logic. The join logic is a parameter of the APPLY operator:

(APPLY[INNER_THETA_JOIN] R E(r))

Usually only a subset of attributes from r is needed to execute a parameterised relational expression. A set of attributes to bind is another parameter of the APPLY operator:

(APPLY[INNER_THETA_JOIN; {a.id, a.eid}] R E(a.id, a.eid))

The APPLY operator appears in query plans compiled for queries with correlated subqueries. For example

SELECT e.fname, e.lname
FROM employee e
WHERE EXISTS
     (SELECT * FROM dependent d WHERE e.ssn = d.essn)

will be compiled to

(PROJECT[e.fname, e.lname]
 (APPLY[INNER_THETA_JOIN[e.ssn = d.essn]; {e.ssn}]
   (RENAME[employee->e]
     (TABLE_SCAN employee)
   )
   (RENAME[dependent->d]
     (TABLE_SCAN dependent)
   )
 )
) 

135.1.22. FILTERED_TABLE_SCAN

The FILTERED_TABLE_SCAN operator is a unary operator that is used to represent table scans which are parameterised during run time with attribute values or ranges coming from another branch of the query plan. FILTERED_TABLE_SCAN operators are constructed from a TABLE_SCAN operator and a predicate that restricts the tuples to be produced.

FILTERED_TABLE_SCAN operators are never used in the initial query plan but can be inserted instead to replace a TABLE_SCAN operator by the filtered table scan optimiser (see Section 135.3.12, “Filtered Table Scan Optimiser”) when doing so is likely to make a significant performance improvement.

135.1.23. QUERY_APPLY

The QUERY_APPLY operator binary operator that is a special kind of apply operator which parameterises one query branch with values from the other branch. It is different from the conventional APPLY operator in that the parameter set is extracted from the entire tuple stream and not on a per tuple basis. The parameterised branch is executed only once. The stream which provided parameters and the parameterised stream are joined using the logic of an embedded binary operator. This operator is usually used with a FILTERED_TABLE_SCAN operator.

135.1.24. EXCHANGE

The EXCHANGE operator is a unary operator that is added to the query plan at any point where data must be moved from one evaluation node (OGSA-DAI server) to another.

135.2. Query plans

For each query sent to a DQP resource, DQP generates an initial query plan that can be used to execute the query. These query plans are constructed from the operators described in section Section 135.1, “Operators”. An example initial query plan is shown in Figure 135.1, “DQP initial query plan.”. This query plan is then passed through several optimisers to produce a final query plan that is executed. An example final query plan is shown in Figure 135.2, “DQP query plan.”.

DQP query plan before optimisers are applied.
Initial DQP query plan before any optimisers are applied. For each operator box, the top section shows the schema of the data output by that operator (attribute name, source table name, type and key flag) and the bottom section shows the operator name, estimated cardinality and evaluation node on which the operator will be executed (initially null). The middle section shown any addition properties of the operator, for example for the TABLE_SCAN operator it shows the SQL query to be executed. This has been scaled to fit the page click here to see it at full size.

Figure 135.1. DQP initial query plan.


DQP query plan after optimisers have been applied.
Final DQP query plan after optimisers have been applied. or each operator box, the top section shows the schema of the data output by that operator (attribute name, source table name, type and key flag) and the bottom section shows the operator name, estimated cardinality and evaluation node on which the operator will be executed. The middle section shown any addition properties of the operator, for example for the TABLE_SCAN operator it shows the SQL query to be executed. This has been scaled to fit the page click here to see it at full size.

Figure 135.2. DQP query plan.


All query plans must have the following properties:

  • The root is a NIL operator, this is the only occurrence of the NIL operator.

  • All unary operators must have exactly one child.

  • All binary operators must have exactly two children.

The initial query plan has the following properties:

  • It is valid but may include compilation artifacts like redundant operators which should be removed in the optimisation stage.

The final query plan must have the following properties additional to those required by all query plans:

  • Each operator (except EXCHANGE operator) must be annotated with the data node or evaluation node on which the operator will execute.

  • EXCHANGE operators must be placed between operators that will be executed on different nodes.

135.3. Optimisers

DQP passes query plans through a chain of optimisers that attempt to produce a more efficient query plan. Each optimiser takes a query plan, transforms it and outputs a new query plan. This new query plan is then the input to the next optimiser in the chain. The optimisation chain is configured in the DQPCompilerConfig.xml file for the DQP resource. This file can be found in the following directory (replace RESOURCE_ID with the name of your DQP resource):

[Warning]Warning
Be careful to edit the correct DQPCompilerConfig.xml file as there are multiple versions of this file. Edit the one inside a directory whose name is the ID of the resource you wish to configure.

This file defines the chain of optimisers that are executed. By default the optimisation chain is defined as:

<optimisationChain>
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.QueryNormaliser" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.select.SelectPushDownOptimiser" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.rename.RenamePullUpOptimiser" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.project.ProjectPullUpOptimiser" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.join.JoinOrderingOptimiser" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.join.JoinAnnotation" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.project.groupby.InsertProjectAfterGroupByOptimiser" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.project.pushdown.ProjectPushDownOptimiser" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.project.redundant.RemoveRedundantProjectOptimiser" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.partitioner.PartitioningOptimiser" />
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.implosion.TableScanImplosionOptimiser">
    <property name="selectOnly.resource" value="SomeResource" />
  </optimiser>
  <optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.join.FilteredTableScanOptimiser">
    <property name="bigtable.min.size" value="1000" />
    <property name="table.size.ratio" value="10" />
    <property name="table.to.filter" value="Resource4_faculty" />
  </optimiser>    
</optimisationChain>   

The following sections describe each of these optimisers.

135.3.1. Query Normalise Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.2. Select Push Down Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.3. Rename Pull Up Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.4. Project Pull Up Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.5. Join Ordering Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.6. Join Annotation Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.7. Insert Project After Group By Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.8. Project Push Down Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.9. Remove Redundant Project Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.10. Partitioning Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.11. Table Scan Implosion Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.3.12. Filtered Table Scan Optimiser

This section of the documentation has not yet been written. Please see the online version which may be more up to date.

135.4. Visualising query plans

It is possible to visualise the current query plan at each stage of the optimisation using a visualisation optimiser that does not alter the query plan but instead renders the query plan to a file. To visualise a query plan simply edit the optimiser chain defined by the optimisationChain element of the DQP resource's DQPCompilerConfig.xml file. This file can be found in the following directory (replace RESOURCE_ID with the name of your DQP resource):

[Warning]Warning
Be careful to edit the correct DQPCompilerConfig.xml file as there are multiple versions of this file. Edit the one inside a directory whose name is the ID of the resource you wish to configure.

To add a visualisation optimiser add the following optimiser element at the desired point within the optimisation chain:

<optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.visualise.VisualiseOptimiser">
   <property name="dot.filename" value="PATH/myQueryPlan.dot" />
</optimiser>

where PATH specifies the path to where you wish to query plan rendering to be written. When specifying the filename two keywords can be used which are replaced with appropriate values. These are:

  • REQUESTID: which is replaced with the ID of the user request that contains the DQP query
  • UUID: which is replaced with a string that is unique to this invocation of the optimiser.

For example, if the optimiser is configured as:

<optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.visualise.VisualiseOptimiser">
   <property name="dot.filename" value="PATH/myQueryPlan-REQUESTID.dot" />
</optimiser>

then if the request ID is ogsadai-XYZ then query plan will be written to file

PATH/myQueryPlan-ogsadai-XYZ.dot

Query plans are written out in dot language. The configuration of the visualiser optimiser also supports the execution of a command. This functionality can be used to convert the dot format input into a common graphics format. GraphVis is a useful program to do this. If you have GraphVis then the following optimiser element shows how to specify the execution of a command to generate in image of the query plan in png format:

<optimiser class="uk.org.ogsadai.dqp.lqp.optimiser.visualise.VisualiseOptimiser">
   <property name="dot.filename" value="PATH/myQueryPlan-REQUESTID.dot" />
   <property name="command"
      value="/PATH/TO/GRAPHVIS/dot -Tpng -q PATH/myQueryPlan-REQUESTID.dot -o PATH/myQueryPlan-REQUESTID.png" />
</optimiser>

Note that the REQUESTID and UUID keywords apply to the command value in the same way they do to the dot.filename value.

The visualiser optimisers can be inserted anywhere in the optimiser chain to visualise the query plan at that stage. Mostly commonly it is inserted at the end to visualise the final query plan.

[Important]Important
After editing the DQPCompilerConfig.xml file you must restart the server in order for the change to take effect.

[Warning]Warning
The inclusion of the visualisation optimiser can have an significant impact on query execution time, especially if the command option is used. to invoke a graphics package such as GraphVis.