Merge branch 'devel' of https://github.com/arangodb/arangodb into devel

Documentation/Books/Makefile

@@ -73,13 +73,9 @@ book-check-markdown-leftovers:
 	fi
 
 book-check-dangling-anchors:
-	@grep -R "a href" books/$(NAME) | \
-		grep -v styles/header.js | \
-		grep -v /app.js | \
-		grep -v navigation| \
-		grep -v 'http://' | \
-		grep -v 'https://' | \
-		grep '#'  |sed 's;\(.*\.html\):.*a href="\(.*\)#\(.*\)">.*</a>.*;\1,\2,\3;' > /tmp/anchorlist.txt
+	@grep -R "a href.*#" books/$(NAME) | \
+		egrep -v "(styles/header\.js|/app\.js|class=\"navigation|https*://)" | \
+		sed 's;\(.*\.html\):.*a href="\(.*\)#\(.*\)">.*</a>.*;\1,\2,\3;' > /tmp/anchorlist.txt
 
 	@NO=0; \
 	for i in `cat /tmp/anchorlist.txt`; do \

Documentation/Books/Users/Aql/GraphTraversals.mdpp

@@ -1,13 +1,70 @@
 !CHAPTER Graph traversals in AQL
-Graph traversal are natively executed. They start at a provided origin vertex, and traverse up to a specified depth of edges. You may apply `AQL FILTER` statements that will stop the traversal.
 
-TODO:
-Circle detection not detected! Double traversed edges not detected
-Undirected edegs will traverse back and forth
+!SUBSECTION General query idea
+
+This query is only useful if you use edge collections and/or graphs in your data model.
+It is supposed to walk through your graph.
+Therefore it starts at one specific document (*start-vertex*) and follows all edges connected to this document.
+For all documents (*vertices*) that are targeted by these edges it will again follow all edges connected to them and so on.
+It is possible to define how many of these follow iterations should be executed at least (*min-depth*) and at most (*max-depth*).
+For all vertices that where visited during this process in the range between *min-depth* and *max-depth* iterations you will get a result in form of a set with three items:
+
+1. The visited vertex.
+2. The edge pointing to it
+3. The complete path from start-vertex to the visited vertex as a JSON document with an attribute `edges` and an attribute `vertices`, each a list of the coresponding elements. These lists are sorted, s.t. the first element in `vertices` is the start-vertex and the last is the visited vertex. And that the n-th element in `edges` connects the n-th element with the (n+1)-th element in `vertices`.
+
+Let's take a look at a simple example to explain how it works:
+
+![traversal graph](../Graphs/traversal_graph.png)
+
+At first we have to define a starting point **A**:
+
+![traversal graph step 1](../Graphs/traversal_graph1.png)
+
+Now it walks to one of the direct neighbors of **A**, say **B** (NOTE: ordering is not guaranteed):
+
+![traversal graph step 2](../Graphs/traversal_graph2.png)
+
+The query will remember the state (red circle) and will emit the first result **A** -> **B** (black box).
+Now again it will visit one of the direct neighbors of **B**, say **E**:
+
+![traversal graph step 3](../Graphs/traversal_graph3.png)
+
+Now assume we have limited the query with a *max-depth* of *2* then it will not pick any neighbor of **E** as the path from **A** to **E** already requires *2* steps.
+Instead we will go back one level to **B** and continue with any other direct neighbor there:
+
+![traversal graph step 4](../Graphs/traversal_graph4.png)
+
+Again after we produced this result we will step back to **B**.
+But there is no neighbor of **B** left that we have not yet visited.
+Hence we go another step back to **A** and continue with any other neighbor there.
+
+![traversal graph step 5](../Graphs/traversal_graph5.png)
+
+And identical to the iterations before we will visit **H**:
+
+![traversal graph step 6](../Graphs/traversal_graph6.png)
+
+And **J**:
+
+![traversal graph step 7](../Graphs/traversal_graph7.png)
+
+And after these steps there is no further result left.
+So all together this query has returned the following paths:
+
+1. `A -> B`
+2. `A -> B -> E`
+3. `A -> B -> C`
+4. `A -> G`
+5. `A -> G -> H`
+6. `A -> G -> J`
 
 
 !SUBSECTION Syntax
 
+Now let's see how we can write a query that follows this schema.
+You have to options here, you can either use a managed graph (see [the graphs chaper]() on how to create it)
+
 !SUBSUBSECTION Working on managed graphs:
 
 `FOR ` vertex[, edge[, path]]
@@ -32,36 +89,42 @@ Undirected edegs will traverse back and forth
 
 `FOR ` vertex[, edge[, path]]
  `IN` `MIN`[..`MAX`]
- `OUTBOUND|INBOUND|ANY` startVertex edeCollection1, .., edgeCollectionN
+ `OUTBOUND|INBOUND|ANY` startVertex
+ edgeCollection1, .., edgeCollectionN
 
-Instead of the Graph you may specify a **list of edge collections**. Vertex collections are evaluated from the edges. The rest of the behavior is similar to the managed version.
+Instead of the `GRAPH graphName` you may specify a **list of edge collections**. Vertex collections are evaluated from the edges. The rest of the behavior is similar to the managed version.
 
 !SUBSECTION using filters and the explainer to extrapolate the costs
 
-Depending on the filters you specified some of them may be executed early in the traversal and stop the traversal at that point. You can use this technique to further improve the execution time.
-Certain filter conditions may be executed during the traversal itself; They must only compare one attribute of the path to a value. This may even improve performance in clustered setups.
+All three variables emitted by the traversals might as well be used in filter statements.
+For some of these filter statements the optimizer can detect that it is possible to prune paths of traversals earlier, hence invalid results will not be emitted to the variables in the first place.
+This may significantly improve the performance of your query.
+Whenever a filter is not fulfilled the complete set of **vertex**, **edge** and **path** will be skipped.
+All paths with a length greater than `MAX` will never be computed.
 
-Other filters may be executed for each traversal iteration. For them the result set has to be fetched and prepared.
-
-More complicated filters or filters on vertices or edges will only run to reduce the result set, thus they will not abort the path traversal - up to MAX nodes will be inspected.
-OR'ed filters will fall into that range.
-
-A filter will always discard one set of **vertex**, **edge** and **path**. 
+In the present state OR combined filters cannot be optimized, AND combined filters can.
 
 !SUBSUBSECTION filtering on paths
-Using the path variable you can filter on specific iteration depths. You can filter for absolute positions in the path by **(specifying a positive number)** **(which then qualifies for the optimizations)** or relative positions to the end of the path by specifying a negative number. An asterisk is valid, it will produce an array of all items in the path, and compare that to be exactly similar to what you specify on the right hand side; this may only be executed at the very end and only return paths of exactly that array size.
 
-*Filtering for edge attribute*
+This allows for the most powerful filtering and may have the highest impact on performance.
+Using the path variable you can filter on specific iteration depths.
+You can filter for absolute positions in the path by **(specifying a positive number)** **(which then qualifies for the optimizations)** or relative positions to the end of the path by specifying a negative number.
+**Note**: In the present state there is no way to define **for all elements on the path**. This will be added in the future.
+
+*Filtering edges on the path*
 
 FOR v,e,p IN 0..5 OUTBOUND 'circles/A' GRAPH 'traversalGraph' FILTER p.edges[0].theTruth == true return p
 
-will filter all paths where the start edge (# 0) has the attribute `theTruth` equaling true. The paths will be up to 5 items long.
+will filter all paths where the start edge (# 0) has the attribute `theTruth` equaling true.
+The resulting paths will be up to 5 items long.
+
 
 *combining several filters*
 
-FOR v,e,p IN 0..5 OUTBOUND 'circles/A' GRAPH 'traversalGraph' FILTER p.edges[0].theTruth == true p.edges[1].theFalse == false return p
+FOR v,e,p IN 0..5 OUTBOUND 'circles/A' GRAPH 'traversalGraph' FILTER p.edges[0].theTruth == true AND p.edges[1].theFalse == false RETURN p
 
-will filter all paths where the start edge (# 0) has the attribute `theTruth` equaling true and the second edge (# 1) having the attribute `theFalse` equaling false. The paths will be up to 5 items long.
+will filter all paths where the start edge (# 0) has the attribute `theTruth` equaling true and the second edge (# 1) having the attribute `theFalse` equaling false.
+The resulting paths will be up to 5 items long.
 
 !SUBSUBSECTION Examples
 We will create a simple symetric traversal demonstration graph:
@@ -78,19 +141,19 @@ We will create a simple symetric traversal demonstration graph:
     @END_EXAMPLE_ARANGOSH_OUTPUT
     @endDocuBlock GRAPHTRAV_01_create_graph
 
-To get startet we select the full graph; for better overview we only return the vertex ids:
+To get started we select the full graph; for better overview we only return the vertex ids:
 
     @startDocuBlockInline GRAPHTRAV_02_traverse_all
     @EXAMPLE_ARANGOSH_OUTPUT{GRAPHTRAV_02_traverse_all}
-    db._query("FOR v IN 0..3 OUTBOUND 'circles/A' GRAPH 'traversalGraph' return v._key");
-    db._query("FOR v IN 0..3 OUTBOUND 'circles/A' edges return v._key");
+    db._query("FOR v IN 0..3 OUTBOUND 'circles/A' GRAPH 'traversalGraph' RETURN v._key");
+    db._query("FOR v IN 0..3 OUTBOUND 'circles/A' edges RETURN v._key");
     @END_EXAMPLE_ARANGOSH_OUTPUT
     @endDocuBlock GRAPHTRAV_02_traverse_all
 
 We can nicely see its heading for the first outer vertex, then going back to the branch to descend into the next tree. After that it returns to our start node, to descend again.
-If we don't want to use the graph management facilities we specify a list of edge collections to `OUTBOUND` instead of the graph name as the second query does.
+As we can see both queries return the same result, the first one uses the managed graph, the second directly uses the edge collection.
 
-Now we only want the elements of a specific depth - 2 - th ones that are right behind th fork:
+Now we only want the elements of a specific depth - 2 - the ones that are right behind the fork:
 
     @startDocuBlockInline GRAPHTRAV_03_traverse_3
     @EXAMPLE_ARANGOSH_OUTPUT{GRAPHTRAV_03_traverse_3}
@@ -99,26 +162,27 @@ Now we only want the elements of a specific depth - 2 - th ones that are right b
     @END_EXAMPLE_ARANGOSH_OUTPUT
     @endDocuBlock GRAPHTRAV_03_traverse_3
 
-As you can see, we can express this in two ways, one is to ommit the `MAX` parameter of the expression.
+As you can see, we can express this in two ways, one is to omit the `MAX` parameter of the expression.
 
 !SUBSUBSECTION Filter examples
+
 Now lets start to add some filters. 
 We want to cut of the branch on the right side of th graph, we may filter in two ways:
- - we know the vertex at th depth 1 has `_key` == `G`
+ - we know the vertex at depth 1 has `_key` == `G`
  - we know the `label` attribute of the edge connecting *A* to *G* is `right_foo`
 
     @startDocuBlockInline GRAPHTRAV_04_traverse_4
     @EXAMPLE_ARANGOSH_OUTPUT{GRAPHTRAV_04_traverse_4}
-    db._query("FOR v,e,p IN 1..3 OUTBOUND 'circles/A' GRAPH 'traversalGraph' FILTER p.vertices[1]._key != 'G' return v._key");
-    db._query("FOR v,e,p IN 1..3 OUTBOUND 'circles/A' GRAPH 'traversalGraph' FILTER p.edges[0].label != 'right_foo' return v._key");
+    db._query("FOR v, e, p IN 1..3 OUTBOUND 'circles/A' GRAPH 'traversalGraph' FILTER p.vertices[1]._key != 'G' RETURN v._key");
+    db._query("FOR v, e, p IN 1..3 OUTBOUND 'circles/A' GRAPH 'traversalGraph' FILTER p.edges[0].label != 'right_foo' RETURN v._key");
     @END_EXAMPLE_ARANGOSH_OUTPUT
     @endDocuBlock GRAPHTRAV_04_traverse_4
 
-As we can see all vertices behind *G* are skipped in both queries. The first filters on the vertex `_key`, the second on an edge label.
-Note that only edges will be included in the result that point to a non-filtered vertex, so the first query won't show you `p.edges[0]`
+As we can see all vertices behind *G* are skipped in both queries.
+The first filters on the vertex `_key`, the second on an edge label.
+Note again as soon as a filter is not fulfilled for any of the three elements `v`, `e` or `p` the complete set of these will be excluded from the result.
 
-
-We may combine several filters so we filter out the right (*G*) branch, and the *E* branch:
+We also may combine several filters, for instance to filter out the right branch (*G*), and the *E* branch:
 
     @startDocuBlockInline GRAPHTRAV_05_traverse_5
     @EXAMPLE_ARANGOSH_OUTPUT{GRAPHTRAV_05_traverse_5}
@@ -144,29 +208,27 @@ Since `circles/A` only has outbound edges, we start our queries from `circles/E`
 
 The first traversal will only walk into the forward (`OUTBOUND`) direction. Therefore from *E* we only can see *F*.
 Walking into reverse direction (`INBOUND`) we see the path to *A*: *B*, *A*.
-Walking in forward and reverse direction (`ANY`) we can see a more diverse result. It will also walk back the path it just came.
-We will see all nodes up to 3 hops away from *E*, outmost this is *D* and behind *A* we found *G*. Due to it walking back again
-we also will see duplicate nodes.
+Walking in forward and reverse direction (`ANY`) we can see a more diverse result.
+First of all we see the simple paths to *F* and *A*.
+However these vertices have edges in other directions and they will be traversed.
+**Note here**: The traverser may use identical edges multiple times.
+For instance if it walks from *E* to *F* it will continue to walk from *F* to *E* using the same edge once again.
+Due to this we will see duplicate nodes in the result.
 
-!SUBSUBSECTION Explain whats usefull
+!SUBSUBSECTION Use the AQL explainer for optimizations
 
 Now lets have a look what the optimizer does behind the curtains and inspect traversal queries using [the explainer](Optimizer.md):
 
 
     @startDocuBlockInline GRAPHTRAV_07_traverse_7
     @EXAMPLE_ARANGOSH_OUTPUT{GRAPHTRAV_07_traverse_7}
-    db._explain("FOR v,e,p IN 1..3 OUTBOUND 'circles/' GRAPH 'traversalGraph' LET localScopeVar = RAND() > 0.5 FILTER p.edges[0].theTruth != localScopeVar return v._key", {}, {colors:false});
-    db._explain("FOR v,e,p IN 1..3 OUTBOUND 'circles/A' GRAPH 'traversalGraph' FILTER p.edges[0].label == 'right_foo' return v._key", {}, {colors:false});
+    db._explain("FOR v,e,p IN 1..3 OUTBOUND 'circles/' GRAPH 'traversalGraph' LET localScopeVar = RAND() > 0.5 FILTER p.edges[0].theTruth != localScopeVar RETURN v._key", {}, {colors: false});
+    db._explain("FOR v,e,p IN 1..3 OUTBOUND 'circles/A' GRAPH 'traversalGraph' FILTER p.edges[0].label == 'right_foo' RETURN v._key", {}, {colors: false});
     @END_EXAMPLE_ARANGOSH_OUTPUT
     @endDocuBlock GRAPHTRAV_07_traverse_7
 
 We now see two queries, in one we add a variable `localScopeVar` which is outside the scope of the traversal itself - it is not known inside of the traverser. Therefore this filter can only executed after the traversal, which may be undesired in large graphs.
-The second query on the other hand only operates on the path, and therefore this condition can be used during the execution of the traversal, paths that are filtered out by ths condition won't be processed at all.
-
-
-
-
-
+The second query on the other hand only operates on the path, and therefore this condition can be used during the execution of the traversal, paths that are filtered out by this condition won't be processed at all.
 
 And finally clean it up again:
     @startDocuBlockInline GRAPHTRAV_99_drop_graph
@@ -177,3 +239,6 @@ And finally clean it up again:
     ~removeIgnoreCollection("edges");
     @END_EXAMPLE_ARANGOSH_OUTPUT
     @endDocuBlock GRAPHTRAV_99_drop_graph
+
+
+If this traversal is not powerful enough for your needs, so you cannot describe your conditions as AQL filter statements you might want to look at [manually crafted traverser](../Traversals/README.md)

Documentation/Books/Users/SUMMARY.md

@@ -82,14 +82,14 @@
     * [Object / Document](Aql/DocumentFunctions.md)
     * [Geo](Aql/GeoFunctions.md)
     * [Fulltext](Aql/FulltextFunctions.md)
-    * [Graphs](Aql/Graphs.md)
-      * [Traversion](Aql/GraphTraversals.md)
-      * [Named Operations](Aql/GraphOperations.md)
-      * [Other](Aql/GraphFunctions.md)
     * [Miscellaneous](Aql/MiscellaneousFunctions.md)
   * [Query Results](Aql/QueryResults.md)
   * [Operators](Aql/Operators.md)
   * [High level Operations](Aql/Operations.md)
+  * [Graphs](Aql/Graphs.md)
+    * [Traversal](Aql/GraphTraversals.md)
+    * [Named Operations](Aql/GraphOperations.md)
+    * [Other](Aql/GraphFunctions.md)
   * [Advanced Features](Aql/Advanced.md)
   * [Extending AQL](AqlExtending/README.md)
     * [Conventions](AqlExtending/Conventions.md)

Documentation/Books/Users/Traversals/README.mdpp

@@ -5,6 +5,12 @@ be composed with the low-level edge methods *edges*, *inEdges*, and *outEdges* f
 [edge collections](../Edges/README.md). For more complex operations,
 ArangoDB provides predefined traversal objects.
 
+Also Traversals have been added to AQL.
+Please read the [chapter about AQL traversersals](../Aql/GraphTraversals.md) before you continue reading here.
+Most of the traversal cases are covered by AQL and will be executed in an optimized way.
+Only if the logic for your is too complex to be defined using AQL filters you can use the traversal object defined
+here which gives you complete programmatic access to the data.
+
 For any of the following examples, we'll be using the example collections *v* and *e*,
 populated with continents, countries and capitals data listed below (see [Example Data](../Traversals/ExampleData.md)).
 

arangod/Aql/AggregateNode.cpp

@@ -194,6 +194,7 @@ struct UserVarFinder final : public WalkerWorker<ExecutionNode> {
     else if (en->getType() == ExecutionNode::ENUMERATE_COLLECTION ||
              en->getType() == ExecutionNode::INDEX ||
              en->getType() == ExecutionNode::ENUMERATE_LIST ||
+             en->getType() == ExecutionNode::TRAVERSAL ||
              en->getType() == ExecutionNode::AGGREGATE) {
       depth += 1;
     }

arangod/Aql/ExecutionNode.cpp

@@ -548,6 +548,7 @@ bool ExecutionNode::isInInnerLoop () const {
 
     if (type == ENUMERATE_COLLECTION ||
         type == INDEX ||
+        type == TRAVERSAL ||
         type == ENUMERATE_LIST) {
       // we are contained in an outer loop
       return true;
@@ -1050,8 +1051,7 @@ void ExecutionNode::RegisterPlan::after (ExecutionNode* en) {
       nrRegs.emplace_back(registerId);
 
       for (auto& it : vars) {
-        varInfo.emplace(make_pair(it->id,
-                                 VarInfo(depth, totalNrRegs)));
+        varInfo.emplace(it->id, VarInfo(depth, totalNrRegs));
         totalNrRegs++;
       }
       break;

arangod/Aql/OptimizerRules.cpp

@@ -154,7 +154,8 @@ int triagens::aql::removeRedundantSortsRule (Optimizer* opt,
           }
         }
         else if (current->getType() == EN::ENUMERATE_LIST ||
-                 current->getType() == EN::ENUMERATE_COLLECTION) {
+                 current->getType() == EN::ENUMERATE_COLLECTION ||
+                 current->getType() == EN::TRAVERSAL) {
           // ok, but we cannot remove two different sorts if one of these node types is between them
           // example: in the following query, the one sort will be optimized away:
           //   FOR i IN [ { a: 1 }, { a: 2 } , { a: 3 } ] SORT i.a ASC SORT i.a DESC RETURN i
@@ -848,6 +849,7 @@ int triagens::aql::removeSortRandRule (Optimizer* opt,
         case EN::FILTER: 
         case EN::SUBQUERY:
         case EN::ENUMERATE_LIST:
+        case EN::TRAVERSAL:
         case EN::INDEX: { 
           // if we found another SortNode, an AggregateNode, FilterNode, a SubqueryNode, 
           // an EnumerateListNode or an IndexNode
@@ -1048,6 +1050,7 @@ int triagens::aql::moveCalculationsDownRule (Optimizer* opt,
       else if (currentType == EN::INDEX ||
                currentType == EN::ENUMERATE_COLLECTION ||
                currentType == EN::ENUMERATE_LIST ||
+               currentType == EN::TRAVERSAL ||
                currentType == EN::AGGREGATE ||
                currentType == EN::NORESULTS) {
         // we will not push further down than such nodes
@@ -1749,7 +1752,7 @@ int triagens::aql::useIndexesRule (Optimizer* opt,
                                    ExecutionPlan* plan, 
                                    Optimizer::Rule const* rule) {
   
-  // These are all the FILTER nodes where we start
+  // These are all the nodes where we start traversing (including all subqueries)
   std::vector<ExecutionNode*> nodes(std::move(plan->findEndNodes(true)));
 
   std::unordered_map<size_t, ExecutionNode*> changes;
@@ -1966,6 +1969,7 @@ struct SortToIndexNode final : public WalkerWorker<ExecutionNode> {
 
     bool before (ExecutionNode* en) override final {
       switch (en->getType()) {
+        case EN::TRAVERSAL:
         case EN::ENUMERATE_LIST:
         case EN::SUBQUERY:
         case EN::FILTER:
@@ -1994,7 +1998,6 @@ struct SortToIndexNode final : public WalkerWorker<ExecutionNode> {
         case EN::REMOTE:
         case EN::ILLEGAL:
         case EN::LIMIT:                      // LIMIT is criterion to stop
-        case EN::TRAVERSAL:
           return true;  // abort.
 
         case EN::SORT:     // pulling two sorts together is done elsewhere.