Bug fix/internal issue #316 (#7911)

* allow using scorers outside ArangoSearch view context

Signed-off-by: Andrey Abramov <andrey@arangodb.com>

* ensure query is properly optimized after replacement of scorer functions

* do not apply `handleViewsRule` to queries without views

* simplify optimization rule for ArangoSearch views

* show ArangoSearch view scorers in query explanation

* fix tests

* fix tests

* add stub for scorer related tests

* reformat

* check variable depth in `ViewExpressionContext::getVariableValue`

* add some tests

* address js test failures

* address jslint errors

* ensure `IResearchViewNode` exposes variables used in scorers

* ensure scorers with expressions are deduplicated

* fix deduplication for indexed access

* more tests

* partially address review comments

* address review comments

* simplify code

* remove irrelevant, commented out code

* ensure array comparisons are properly handled

* update changelog & loki

CHANGELOG

@@ -1,6 +1,9 @@
 devel
 -----
 
+* fixed known issue #445: ArangoSearch ignores `_id` attribute even if `includeAllFields`
+  is set to `true`.
+
 * upgraded bundled boost library to version 1.69.0
 
 * upgraded bundled curl library to version 7.63

Documentation/Books/Manual/ReleaseNotes/KnownIssues35.md

@@ -11,13 +11,12 @@ ArangoSearch
 
 | Issue      |
 |------------|
-| **Date Added:** 2018-12-19 <br> **Component:** ArangoSearch <br> **Deployment Mode:** Single-server <br> **Description:** Value of `_id` attribute indexed by ArangoSearch view may become inconsistent after renaming a collection <br> **Affected Versions:** 3.4.2 <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/backlog#514](https://github.com/arangodb/backlog/issues/514) (internal) |
+| **Date Added:** 2018-12-19 <br> **Component:** ArangoSearch <br> **Deployment Mode:** Single-server <br> **Description:** Value of `_id` attribute indexed by ArangoSearch view may become inconsistent after renaming a collection <br> **Affected Versions:** >= 3.5.0 <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/backlog#514](https://github.com/arangodb/backlog/issues/514) (internal) |
 | **Date Added:** 2018-12-03 <br> **Component:** ArangoSearch <br> **Deployment Mode:** Cluster <br> **Description:** Score values evaluated by corresponding score functions (BM25/TFIDF) may differ in single-server and cluster with a collection having more than 1 shard <br> **Affected Versions:** 3.4.0-RC.5 <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/backlog#508](https://github.com/arangodb/backlog/issues/508) (internal) |
 | **Date Added:** 2018-12-03 <br> **Component:** ArangoSearch <br> **Deployment Mode:** Cluster <br> **Description:** ArangoSearch index consolidation does not work during creation of a link on existing collection which may lead to massive file descriptors consumption <br> **Affected Versions:** 3.4.0-RC.5 <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/backlog#509](https://github.com/arangodb/backlog/issues/509) (internal) |
 | **Date Added:** 2018-12-03 <br> **Component:** ArangoSearch <br> **Deployment Mode:** Cluster <br> **Description:** Long-running DML transactions on collections (linked with ArangoSearch view) block "ArangoDB flush thread" making impossible to refresh data "visible" by a view <br> **Affected Versions:** 3.4.0-RC.5 <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/backlog#510](https://github.com/arangodb/backlog/issues/510) (internal) |
 | **Date Added:** 2018-12-03 <br> **Component:** ArangoSearch <br> **Deployment Mode:** All <br> **Description:** ArangoSearch index format included starting from 3.4.0-RC.4 is incompatible to earlier released 3.4.0 release candidates. Dump and restore is needed when upgrading from 3.4.0-RC.4 to a newer 3.4.0.x release <br> **Affected Versions:** 3.4.0-RC.5 <br> **Fixed in Versions:** - <br> **Reference:** N/A |
 | **Date Added:** 2018-12-03 <br> **Component:** ArangoSearch <br> **Deployment Mode:** All <br> **Description:** RocksDB recovery fails sometimes after renaming a view <br> **Affected Versions:** 3.4.0-RC.5 <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/backlog#469](https://github.com/arangodb/backlog/issues/469) (internal) |
-| **Date Added:** 2018-12-03 <br> **Component:** ArangoSearch <br> **Deployment Mode:** All <br> **Description:** ArangoSearch ignores `_id` attribute even if `includeAllFields` is set to `true`  <br> **Affected Versions:** 3.4.0-RC.5 <br> **Fixed in Versions:** 3.4.2 <br> **Reference:** [arangodb/backlog#445](https://github.com/arangodb/backlog/issues/445) (internal) |
 | **Date Added:** 2018-12-03 <br> **Component:** ArangoSearch <br> **Deployment Mode:** All <br> **Description:** Using a loop variable in expressions within a corresponding SEARCH condition is not supported <br> **Affected Versions:** 3.4.0-RC.5 <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/backlog#318](https://github.com/arangodb/backlog/issues/318) (internal) |
 | **Date Added:** 2018-12-03 <br> **Component:** ArangoSearch <br> **Deployment Mode:** All <br> **Description:** Using score functions (BM25/TFIDF) in ArangoDB expression is not supported <br> **Affected Versions:** 3.4.0-RC.5 <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/backlog#316](https://github.com/arangodb/backlog/issues/316) (internal) |
 | **Date Added:** 2018-12-03 <br> **Component:** ArangoSearch <br> **Deployment Mode:** All <br> **Description:** ArangoSearch index format included starting from 3.4.0-RC.3 is incompatible to earlier released 3.4.0 release candidates. Dump and restore is needed when upgrading from 3.4.0-RC.2 to a newer 3.4.0.x release <br> **Affected Versions:** 3.4.0-RC.5 <br> **Fixed in Versions:** - <br> **Reference:** N/A |

arangod/Aql/ClusterBlocks.cpp

@@ -105,12 +105,7 @@ bool OurLessThan::operator()(std::pair<size_t, size_t> const& a,
     int cmp;
 
     if (attributePath.empty()) {
-#if 0  // #ifdef USE_IRESEARCH
-      TRI_ASSERT(reg.comparator);
-      cmp = (*reg.comparator)(reg.scorer.get(), _trx, lhs, rhs);
-#else
       cmp = AqlValue::Compare(_trx, lhs, rhs, true);
-#endif
     } else {
       // Take attributePath into consideration:
       bool mustDestroyA;

arangod/Aql/Expression.h

@@ -82,6 +82,9 @@ class Expression {
     }
   }
 
+  /// @brief get the underlying AST
+  Ast* ast() const noexcept { return _ast; }
+
   /// @brief get the underlying AST node
   inline AstNode const* node() const { return _node; }
 

arangod/Aql/OptimizerRule.h

@@ -166,15 +166,15 @@ struct OptimizerRule {
     // remove redundant filters statements
     removeFiltersCoveredByTraversal,
 
-    // remove calculations that are redundant
-    // needs to run after filter removal
-    removeUnnecessaryCalculationsRule2,
-
 #ifdef USE_IRESEARCH
     // move filters and sort conditions into views and remove them
     handleArangoSearchViewsRule,
 #endif
 
+    // remove calculations that are redundant
+    // needs to run after filter removal
+    removeUnnecessaryCalculationsRule2,
+
     // remove now obsolete path variables
     removeTraversalPathVariable,
     prepareTraversalsRule,

arangod/Aql/Query.h

@@ -296,6 +296,10 @@ class Query {
     _views.emplace(name);
   }
 
+  std::unordered_set<std::string> const& views() const noexcept {
+    return _views;
+  }
+
   /// @brief look up a graph in the _graphs collection
   graph::Graph const* lookupGraphByName(std::string const& name);
 

arangod/Aql/SortBlock.cpp

@@ -44,12 +44,7 @@ class OurLessThan {
       auto const& lhs = _buffer[a.first]->getValueReference(a.second, reg.reg);
       auto const& rhs = _buffer[b.first]->getValueReference(b.second, reg.reg);
 
-#if 0  // #ifdef USE_IRESEARCH
-      TRI_ASSERT(reg.comparator);
-      int const cmp = (*reg.comparator)(reg.scorer.get(), _trx, lhs, rhs);
-#else
       int const cmp = AqlValue::Compare(_trx, lhs, rhs, true);
-#endif
 
       if (cmp < 0) {
         return reg.asc;

arangod/Aql/SortRegister.cpp

@@ -28,45 +28,6 @@
 #include "Aql/ExecutionPlan.h"
 #include "Aql/SortNode.h"
 
-#if 0  // #ifdef USE_IRESEARCH
-#include "IResearch/IResearchOrderFactory.h"
-#include "IResearch/IResearchViewNode.h"
-
-namespace {
-
-int compareIResearchScores(
-    irs::sort::prepared const* comparer,
-    arangodb::transaction::Methods*,
-    arangodb::aql::AqlValue const& lhs,
-    arangodb::aql::AqlValue const& rhs
-) {
-  arangodb::velocypack::ValueLength tmp;
-
-  auto const* lhsScore = reinterpret_cast<irs::byte_type const*>(lhs.slice().getString(tmp));
-  auto const* rhsScore = reinterpret_cast<irs::byte_type const*>(rhs.slice().getString(tmp));
-
-  if (comparer->less(lhsScore, rhsScore)) {
-    return -1;
-  }
-
-  if (comparer->less(rhsScore, lhsScore)) {
-    return 1;
-  }
-
-  return 0;
-}
-
-int compareAqlValues(
-    irs::sort::prepared const*,
-    arangodb::transaction::Methods* trx,
-    arangodb::aql::AqlValue const& lhs,
-    arangodb::aql::AqlValue const& rhs) {
-  return arangodb::aql::AqlValue::Compare(trx, lhs, rhs, true);
-}
-
-}
-#endif
-
 namespace arangodb {
 namespace aql {
 
@@ -77,47 +38,6 @@ namespace aql {
 SortRegister::SortRegister(RegisterId reg, SortElement const& element) noexcept
     : attributePath(element.attributePath), reg(reg), asc(element.ascending) {}
 
-#if 0  // #ifdef USE_IRESEARCH
-
-void SortRegister::fill(
-    ExecutionPlan const& execPlan,
-    ExecutionNode::RegisterPlan const& regPlan,
-    std::vector<SortElement> const& elements,
-    std::vector<SortRegister>& sortRegisters
-) {
-  sortRegisters.reserve(elements.size());
-  std::unordered_map<ExecutionNode const*, size_t> offsets(sortRegisters.capacity());
-
-  irs::sort::ptr comparer;
-
-  auto const& vars = regPlan.varInfo;
-  for (auto const& p : elements) {
-    auto const varId = p.var->id;
-    auto const it = vars.find(varId);
-    TRI_ASSERT(it != vars.end());
-    TRI_ASSERT(it->second.registerId < ExecutionNode::MaxRegisterId);
-    sortRegisters.emplace_back(it->second.registerId, p, &compareAqlValues);
-
-    auto const* setter = execPlan.getVarSetBy(varId);
-
-    if (setter && ExecutionNode::ENUMERATE_IRESEARCH_VIEW == setter->getType()) {
-      // sort condition is covered by `IResearchViewNode`
-
-      auto const* viewNode = ExecutionNode::castTo<iresearch::IResearchViewNode const*>(setter);
-      auto& offset = offsets[viewNode];
-      auto* node = viewNode->sortCondition()[offset++].node;
-
-      if (arangodb::iresearch::OrderFactory::comparer(&comparer, *node)) {
-        auto& reg = sortRegisters.back();
-        reg.scorer = comparer->prepare(); // set score comparer
-        reg.comparator = &compareIResearchScores; // set comparator
-      }
-    }
-  }
-}
-
-#else
-
 void SortRegister::fill(ExecutionPlan const& /*execPlan*/,
                         ExecutionNode::RegisterPlan const& regPlan,
                         std::vector<SortElement> const& elements,
@@ -134,7 +54,5 @@ void SortRegister::fill(ExecutionPlan const& /*execPlan*/,
   }
 }
 
-#endif  // USE_IRESEARCH
-
 }  // namespace aql
 }  // namespace arangodb

arangod/Aql/SortRegister.h

@@ -27,43 +27,18 @@
 #include "Aql/ExecutionNode.h"
 #include "types.h"
 
-#if 0  // #ifdef USE_IRESEARCH
-#include "search/sort.hpp"
-#endif
-
 namespace arangodb {
 namespace aql {
 
 /// @brief sort element for block, consisting of register, sort direction,
 /// and a possible attribute path to dig into the document
 struct SortRegister {
-#if 0  // #ifdef USE_IRESEARCH
-  typedef int(*CompareFunc)(
-    irs::sort::prepared const* scorer,
-    transaction::Methods* trx,
-    AqlValue const& lhs,
-    AqlValue const& rhs
-  );
-
-  irs::sort::prepared::ptr scorer;
-  CompareFunc comparator;
-#endif
   std::vector<std::string> const& attributePath;
   RegisterId reg;
   bool asc;
 
   SortRegister(RegisterId reg, SortElement const& element) noexcept;
 
-#if 0  // #ifdef USE_IRESEARCH
-  SortRegister(
-      RegisterId reg,
-      SortElement const& element,
-      CompareFunc comparator) noexcept
-    : SortRegister(reg, element) {
-    this->comparator = comparator;
-  }
-#endif
-
   static void fill(ExecutionPlan const& /*execPlan*/,
                    ExecutionNode::RegisterPlan const& regPlan,
                    std::vector<SortElement> const& elements,

arangod/IResearch/AqlHelper.cpp

@@ -21,10 +21,13 @@
 ////////////////////////////////////////////////////////////////////////////////
 
 #include "AqlHelper.h"
+#include "Aql/Ast.h"
 #include "Aql/ExecutionPlan.h"
 #include "Aql/Expression.h"
 #include "Aql/ExpressionContext.h"
+#include "Aql/Function.h"
 #include "Aql/Variable.h"
+#include "Basics/fasthash.h"
 #include "IResearchCommon.h"
 #include "IResearchDocument.h"
 #include "Logger/LogMacros.h"
@@ -36,24 +39,298 @@
 namespace {
 
 arangodb::aql::AstNodeType const CmpMap[]{
-    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_EQ,  // NODE_TYPE_OPERATOR_BINARY_EQ:
+    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_EQ,  // NODE_TYPE_OPERATOR_BINARY_EQ: 3 == a <==> a == 3
-                                                  // 3 == a <==> a == 3
+    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_NE,  // NODE_TYPE_OPERATOR_BINARY_NE: 3 != a <==> a != 3
-    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_NE,  // NODE_TYPE_OPERATOR_BINARY_NE:
+    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_GT,  // NODE_TYPE_OPERATOR_BINARY_LT: 3 < a  <==> a > 3
-                                                  // 3 != a <==> a != 3
+    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_GE,  // NODE_TYPE_OPERATOR_BINARY_LE: 3 <= a <==> a >= 3
-    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_GT,  // NODE_TYPE_OPERATOR_BINARY_LT:
+    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_LT,  // NODE_TYPE_OPERATOR_BINARY_GT: 3 > a  <==> a < 3
-                                                  // 3 < a  <==> a > 3
+    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_LE  // NODE_TYPE_OPERATOR_BINARY_GE: 3 >= a <==> a <= 3
-    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_GE,  // NODE_TYPE_OPERATOR_BINARY_LE:
-                                                  // 3 <= a <==> a >= 3
-    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_LT,  // NODE_TYPE_OPERATOR_BINARY_GT:
-                                                  // 3 > a  <==> a < 3
-    arangodb::aql::NODE_TYPE_OPERATOR_BINARY_LE  // NODE_TYPE_OPERATOR_BINARY_GE:
-                                                 // 3 >= a <==> a <= 3
 };
+
 }
 
 namespace arangodb {
 namespace iresearch {
 
+bool equalTo(aql::AstNode const* lhs, aql::AstNode const* rhs) {
+  if (lhs == rhs) {
+    return true;
+  }
+
+  if ((!lhs && rhs) || (lhs && !rhs)) {
+    return false;
+  }
+
+  if (lhs->type != rhs->type) {
+    return false;
+  }
+
+  size_t const n = lhs->numMembers();
+
+  if (n != rhs->numMembers()) {
+    return false;
+  }
+
+  // check members for equality
+  for (size_t i = 0; i < n; ++i) {
+    if (!equalTo(lhs->getMemberUnchecked(i), rhs->getMemberUnchecked(i))) {
+      return false;
+    }
+  }
+
+  switch (lhs->type) {
+    case aql::NODE_TYPE_VARIABLE: {
+      return lhs->getData() == rhs->getData();
+    }
+
+    case aql::NODE_TYPE_OPERATOR_UNARY_PLUS:
+    case aql::NODE_TYPE_OPERATOR_UNARY_MINUS:
+    case aql::NODE_TYPE_OPERATOR_UNARY_NOT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_AND:
+    case aql::NODE_TYPE_OPERATOR_BINARY_OR:
+    case aql::NODE_TYPE_OPERATOR_BINARY_PLUS:
+    case aql::NODE_TYPE_OPERATOR_BINARY_MINUS:
+    case aql::NODE_TYPE_OPERATOR_BINARY_TIMES:
+    case aql::NODE_TYPE_OPERATOR_BINARY_DIV:
+    case aql::NODE_TYPE_OPERATOR_BINARY_MOD:
+    case aql::NODE_TYPE_OPERATOR_BINARY_EQ:
+    case aql::NODE_TYPE_OPERATOR_BINARY_NE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_LT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_LE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_GT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_GE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_IN:
+    case aql::NODE_TYPE_OPERATOR_BINARY_NIN:
+    case aql::NODE_TYPE_OPERATOR_TERNARY:
+    case aql::NODE_TYPE_OBJECT:
+    case aql::NODE_TYPE_CALCULATED_OBJECT_ELEMENT:
+    case aql::NODE_TYPE_ARRAY:
+    case aql::NODE_TYPE_RANGE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_EQ:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_NE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_LT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_LE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_GT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_GE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_IN:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_NIN: {
+      return true;
+    }
+
+    case aql::NODE_TYPE_ATTRIBUTE_ACCESS:
+    case aql::NODE_TYPE_INDEXED_ACCESS:
+    case aql::NODE_TYPE_EXPANSION: {
+      return attributeAccessEqual(lhs, rhs, nullptr);
+    }
+
+    case aql::NODE_TYPE_VALUE: {
+      return 0 == aql::CompareAstNodes(lhs, rhs, true);
+    }
+
+    case aql::NODE_TYPE_OBJECT_ELEMENT: {
+      irs::string_ref lhsValue, rhsValue;
+      iresearch::parseValue(lhsValue, *lhs);
+      iresearch::parseValue(rhsValue, *rhs);
+
+      return lhsValue == rhsValue;
+    }
+
+    case aql::NODE_TYPE_REFERENCE: {
+      return lhs->getData() == rhs->getData();
+    }
+
+    case aql::NODE_TYPE_FCALL: {
+      return lhs->getData() == rhs->getData();
+    }
+
+    case aql::NODE_TYPE_FCALL_USER: {
+      irs::string_ref lhsName, rhsName;
+      iresearch::parseValue(lhsName, *lhs);
+      iresearch::parseValue(rhsName, *rhs);
+
+      return lhsName == rhsName;
+    }
+
+    case aql::NODE_TYPE_QUANTIFIER: {
+      return lhs->value.value._int == rhs->value.value._int;
+    }
+
+    default: {
+      return false;
+    }
+  }
+}
+
+size_t hash(aql::AstNode const* node, size_t hash /*= 0*/) noexcept {
+  if (!node) {
+    return hash;
+  }
+
+  // hash node type
+  auto const& typeString = node->getTypeString();
+
+  hash = fasthash64(
+    static_cast<const void*>(typeString.c_str()),
+    typeString.size(),
+    hash
+  );
+
+  // hash node members
+  for (size_t i = 0, n = node->numMembers(); i < n; ++i) {
+    auto sub = node->getMemberUnchecked(i);
+
+    if (sub) {
+      hash = iresearch::hash(sub, hash);
+    }
+  }
+
+  switch (node->type) {
+    case aql::NODE_TYPE_VARIABLE: {
+      return fasthash64(node->getData(), sizeof(void*), hash);
+    }
+
+    case aql::NODE_TYPE_OPERATOR_UNARY_PLUS:
+    case aql::NODE_TYPE_OPERATOR_UNARY_MINUS:
+    case aql::NODE_TYPE_OPERATOR_UNARY_NOT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_AND:
+    case aql::NODE_TYPE_OPERATOR_BINARY_OR:
+    case aql::NODE_TYPE_OPERATOR_BINARY_PLUS:
+    case aql::NODE_TYPE_OPERATOR_BINARY_MINUS:
+    case aql::NODE_TYPE_OPERATOR_BINARY_TIMES:
+    case aql::NODE_TYPE_OPERATOR_BINARY_DIV:
+    case aql::NODE_TYPE_OPERATOR_BINARY_MOD:
+    case aql::NODE_TYPE_OPERATOR_BINARY_EQ:
+    case aql::NODE_TYPE_OPERATOR_BINARY_NE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_LT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_LE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_GT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_GE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_IN:
+    case aql::NODE_TYPE_OPERATOR_BINARY_NIN:
+    case aql::NODE_TYPE_OPERATOR_TERNARY:
+    case aql::NODE_TYPE_INDEXED_ACCESS:
+    case aql::NODE_TYPE_EXPANSION:
+    case aql::NODE_TYPE_ARRAY:
+    case aql::NODE_TYPE_OBJECT:
+    case aql::NODE_TYPE_CALCULATED_OBJECT_ELEMENT:
+    case aql::NODE_TYPE_RANGE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_EQ:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_NE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_LT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_LE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_GT:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_GE:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_IN:
+    case aql::NODE_TYPE_OPERATOR_BINARY_ARRAY_NIN: {
+      return hash;
+    }
+
+    case aql::NODE_TYPE_ATTRIBUTE_ACCESS: {
+      return aql::AstNode(node->value).hashValue(hash);
+    }
+
+    case aql::NODE_TYPE_VALUE: {
+      switch (node->value.type) {
+        case aql::VALUE_TYPE_NULL:
+          return fasthash64(static_cast<const void*>("null"), 4, hash);
+        case aql::VALUE_TYPE_BOOL:
+          if (node->value.value._bool) {
+            return fasthash64(static_cast<const void*>("true"), 4, hash);
+          }
+          return fasthash64(static_cast<const void*>("false"), 5, hash);
+        case aql::VALUE_TYPE_INT:
+          return fasthash64(static_cast<const void*>(&node->value.value._int),
+                            sizeof(node->value.value._int), hash);
+        case aql::VALUE_TYPE_DOUBLE:
+          return fasthash64(static_cast<const void*>(&node->value.value._double),
+                            sizeof(node->value.value._double), hash);
+        case aql::VALUE_TYPE_STRING:
+          return fasthash64(static_cast<const void*>(node->getStringValue()),
+                            node->getStringLength(), hash);
+      }
+    }
+
+    case aql::NODE_TYPE_OBJECT_ELEMENT: {
+      return fasthash64(static_cast<const void*>(node->getStringValue()),
+                        node->getStringLength(), hash);
+    }
+
+    case aql::NODE_TYPE_REFERENCE: {
+      return fasthash64(node->getData(), sizeof(void*), hash);
+    }
+
+    case aql::NODE_TYPE_FCALL: {
+      auto* fn = static_cast<aql::Function*>(node->getData());
+
+      hash = fasthash64(node->getData(), sizeof(void*), hash);
+      return fasthash64(fn->name.c_str(), fn->name.size(), hash);
+    }
+
+    case aql::NODE_TYPE_FCALL_USER: {
+      return fasthash64(static_cast<const void*>(node->getStringValue()),
+                        node->getStringLength(), hash);
+    }
+
+    case aql::NODE_TYPE_QUANTIFIER: {
+      return fasthash64(static_cast<const void*>(&node->value.value._int),
+                        sizeof(node->value.value._int), hash);
+    }
+
+    default: {
+      return fasthash64(static_cast<void const*>(&node), sizeof(&node), hash);
+    }
+  }
+}
+
+irs::string_ref getFuncName(aql::AstNode const& node) {
+  irs::string_ref fname;
+
+  switch (node.type) {
+    case aql::NODE_TYPE_FCALL:
+      fname = reinterpret_cast<aql::Function const*>(node.getData())->name;
+      break;
+
+    case aql::NODE_TYPE_FCALL_USER:
+      parseValue(fname, node);
+      break;
+
+    default:
+      TRI_ASSERT(false);
+  }
+
+  return fname;
+}
+
+void visitReferencedVariables(
+    aql::AstNode const& root,
+    std::function<void(aql::Variable const&)> const& visitor) {
+  auto preVisitor = [](aql::AstNode const* node) -> bool {
+    return !node->isConstant();
+  };
+
+  auto postVisitor = [&visitor](aql::AstNode const* node) {
+    if (node == nullptr) {
+      return;
+    }
+
+    // reference to a variable
+    if (node->type == aql::NODE_TYPE_REFERENCE) {
+      auto variable = static_cast<aql::Variable const*>(node->getData());
+
+      if (!variable) {
+        THROW_ARANGO_EXCEPTION_MESSAGE(TRI_ERROR_INTERNAL,
+                                       "invalid reference in AST");
+      }
+
+      if (variable->needsRegister()) {
+        visitor(*variable);
+      }
+    }
+  };
+
+  aql::Ast::traverseReadOnly(&root, preVisitor, postVisitor);
+}
+
 // ----------------------------------------------------------------------------
 // --SECTION--                                    AqlValueTraits implementation
 // ----------------------------------------------------------------------------
@@ -199,6 +476,7 @@ bool attributeAccessEqual(arangodb::aql::AstNode const* lhs,
         if (root && offset) {
           aqlValue.reset(*offset);
 
+          if (!aqlValue.isConstant()) {
             if (!ctx) {
               // can't evaluate expression at compile time
               return true;
@@ -208,6 +486,7 @@ bool attributeAccessEqual(arangodb::aql::AstNode const* lhs,
               // failed to execute expression
               return false;
             }
+          }
 
           switch (aqlValue.type()) {
             case arangodb::iresearch::SCOPED_VALUE_TYPE_DOUBLE:

arangod/IResearch/AqlHelper.h

@@ -58,6 +58,16 @@ class Methods;  // forward declaration
 
 namespace iresearch {
 
+//////////////////////////////////////////////////////////////////////////////
+/// @returns true if both nodes are equal, false otherwise
+//////////////////////////////////////////////////////////////////////////////
+bool equalTo(aql::AstNode const* lhs, aql::AstNode const* rhs);
+
+//////////////////////////////////////////////////////////////////////////////
+/// @returns computed hash value for a specified node
+//////////////////////////////////////////////////////////////////////////////
+size_t hash(aql::AstNode const* node, size_t hash = 0) noexcept;
+
 //////////////////////////////////////////////////////////////////////////////
 /// @brief extracts string_ref from an AstNode, note that provided 'node'
 ///        must be an arangodb::aql::VALUE_TYPE_STRING
@@ -69,6 +79,12 @@ inline irs::string_ref getStringRef(aql::AstNode const& node) {
   return irs::string_ref(node.getStringValue(), node.getStringLength());
 }
 
+//////////////////////////////////////////////////////////////////////////////
+/// @returns name of function denoted by a specified AstNode
+/// @note applicable for nodes of type NODE_TYPE_FCALL, NODE_TYPE_FCALL_USER
+//////////////////////////////////////////////////////////////////////////////
+irs::string_ref getFuncName(aql::AstNode const& node);
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief tries to extract 'size_t' value from the specified AstNode 'node'
 /// @returns true on success, false otherwise
@@ -126,6 +142,13 @@ bool visit(aql::SortCondition const& sort, Visitor const& visitor) {
   return true;
 }
 
+////////////////////////////////////////////////////////////////////////////////
+/// @brief visits variables referenced in a specified expression
+////////////////////////////////////////////////////////////////////////////////
+void visitReferencedVariables(
+    aql::AstNode const& root,
+    std::function<void(aql::Variable const&)> const& visitor);
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief visits the specified node using the provided 'visitor' according
 ///        to the specified visiting strategy (preorder/postorder)
@@ -386,62 +409,6 @@ bool visitAttributeAccess(aql::AstNode const*& head, aql::AstNode const* node, T
   }
 }
 
-////////////////////////////////////////////////////////////////////////////////
-/// @brief interprets the specified node as an attribute path description and
-///        visits the members in attribute path order calling the provided
-///        'visitor' on each path sub-index, expecting the following signatures:
-///          bool operator()(irs::string_ref) - string keys
-///          bool operator()(int64_t)         - array offsets
-///          bool operator()()                - any string key or numeric offset
-/// @return success and set head the the starting node of path (reference/value)
-////////////////////////////////////////////////////////////////////////////////
-template <typename T>
-bool visitAttributePath(aql::AstNode const*& head, aql::AstNode const& node, T& visitor) {
-  if (node.numMembers() >= 2 && aql::NODE_TYPE_EXPANSION == node.type) {  // [*]
-    auto* itr = node.getMemberUnchecked(0);
-    auto* ref = node.getMemberUnchecked(1);
-
-    if (itr && itr->numMembers() == 2) {
-      auto* root = itr->getMemberUnchecked(1);
-      auto* var = itr->getMemberUnchecked(0);
-
-      return ref && aql::NODE_TYPE_ITERATOR == itr->type &&
-             aql::NODE_TYPE_REFERENCE == ref->type && root && var &&
-             aql::NODE_TYPE_VARIABLE == var->type &&
-             visitAttributePath(head, *root, visitor)  // 1st visit root
-             && visitor();                             // 2nd visit current node
-    }
-  } else if (node.numMembers() == 2 && aql::NODE_TYPE_INDEXED_ACCESS == node.type) {  // [<something>]
-    auto* root = node.getMemberUnchecked(0);
-    auto* offset = node.getMemberUnchecked(1);
-
-    if (offset && offset->isIntValue()) {
-      return root && offset->getIntValue() >= 0 &&
-             visitAttributePath(head, *root, visitor)  // 1st visit root
-             && visitor(offset->getIntValue());        // 2nd visit current node
-    }
-
-    return root && offset && offset->isStringValue() &&
-           visitAttributePath(head, *root, visitor)       // 1st visit root
-           && visitor(iresearch::getStringRef(*offset));  // 2nd visit current node
-  } else if (node.numMembers() == 1 && aql::NODE_TYPE_ATTRIBUTE_ACCESS == node.type) {
-    auto* root = node.getMemberUnchecked(0);
-
-    return root && aql::VALUE_TYPE_STRING == node.value.type &&
-           visitAttributePath(head, *root, visitor)    // 1st visit root
-           && visitor(iresearch::getStringRef(node));  // 2nd visit current node
-  } else if (!node.numMembers()) {  // end of attribute path (base case)
-    head = &node;
-
-    return aql::NODE_TYPE_REFERENCE == node.type ||
-           (aql::NODE_TYPE_VALUE == node.type &&
-            aql::VALUE_TYPE_STRING == node.value.type &&
-            visitor(iresearch::getStringRef(node)));
-  }
-
-  return false;
-}
-
 struct NormalizedCmpNode {
   aql::AstNode const* attribute;
   aql::AstNode const* value;

arangod/IResearch/IResearchExpressionContext.cpp

@@ -24,19 +24,7 @@
 #include "IResearch/IResearchExpressionContext.h"
 #include "Aql/AqlItemBlock.h"
 #include "IResearch/IResearchViewNode.h"
-
+#include "Basics/StaticStrings.h"
-namespace {
-
-inline arangodb::aql::RegisterId getRegister(arangodb::aql::Variable const& var,
-                                             arangodb::aql::ExecutionNode const& node) noexcept {
-  auto const& vars = node.getRegisterPlan()->varInfo;
-  auto const it = vars.find(var.id);
-
-  return vars.end() == it ? arangodb::aql::ExecutionNode::MaxRegisterId
-                          : it->second.registerId;
-}
-
-}  // namespace
 
 namespace arangodb {
 namespace iresearch {
@@ -82,13 +70,24 @@ AqlValue ViewExpressionContext::getVariableValue(Variable const* var, bool doCop
   }
 
   mustDestroy = false;
-  auto const reg = getRegister(*var, *_node);
 
-  if (reg == arangodb::aql::ExecutionNode::MaxRegisterId) {
+  auto const& vars = _node->getRegisterPlan()->varInfo;
+  auto const it = vars.find(var->id);
+
+  if (vars.end() == it) {
     THROW_ARANGO_EXCEPTION(TRI_ERROR_INTERNAL);
   }
 
-  auto& value = _data->getValueReference(_pos, reg);
+  auto const& varInfo = it->second;
+
+  if (varInfo.depth > decltype(varInfo.depth)(_node->getDepth())) {
+    THROW_ARANGO_EXCEPTION_FORMAT(
+        TRI_ERROR_BAD_PARAMETER,
+        "Variable '%s' is used before being assigned",
+        var->name.c_str());
+  }
+
+  auto& value = _data->getValueReference(_pos, varInfo.registerId);
 
   if (doCopy) {
     mustDestroy = true;

arangod/IResearch/IResearchFeature.cpp

@@ -333,33 +333,6 @@ bool iresearchViewUpgradeVersion0_1(TRI_vocbase_t& vocbase,
   return true;
 }
 
-void registerFunctions(arangodb::aql::AqlFunctionFeature& /*functions*/) {
-#if 0
-  arangodb::iresearch::addFunction(functions, {
-    "__ARANGOSEARCH_SCORE_DEBUG",  // name
-    ".",    // value to convert
-    arangodb::aql::Function::makeFlags(
-      arangodb::aql::Function::Flags::Deterministic,
-      arangodb::aql::Function::Flags::Cacheable,
-      arangodb::aql::Function::Flags::CanRunOnDBServer
-    ),
-    [](arangodb::aql::ExpressionContext*,
-       arangodb::transaction::Methods*,
-       arangodb::SmallVector<arangodb::aql::AqlValue> const& args) {
-      if (args.empty()) {
-        // no such input parameter. return NaN
-        return arangodb::aql::AqlValue(arangodb::aql::AqlValueHintDouble(double_t(std::nan(""))));
-      } else {
-        // unsafe
-        VPackValueLength length;
-        auto const floatValue = *reinterpret_cast<float_t const*>(args[0].slice().getString(length));
-        return arangodb::aql::AqlValue(arangodb::aql::AqlValueHintDouble(double_t(floatValue)));
-      }
-    }
-  });
-#endif
-}
-
 void registerFilters(arangodb::aql::AqlFunctionFeature& functions) {
   using arangodb::iresearch::addFunction;
   auto flags =
@@ -969,7 +942,6 @@ void IResearchFeature::start() {
     if (functions) {
       registerFilters(*functions);
       registerScorers(*functions);
-      registerFunctions(*functions);
     } else {
       LOG_TOPIC(WARN, arangodb::iresearch::TOPIC)
           << "failure to find feature 'AQLFunctions' while registering "

arangod/IResearch/IResearchOrderFactory.cpp

@@ -21,8 +21,7 @@
 /// @author Vasiliy Nabatchikov
 ////////////////////////////////////////////////////////////////////////////////
 
-// otherwise define conflict between 3rdParty\date\include\date\date.h and
+// otherwise define conflict between 3rdParty\date\include\date\date.h and 3rdParty\iresearch\core\shared.hpp
-// 3rdParty\iresearch\core\shared.hpp
 #if defined(_MSC_VER)
 #include "date/date.h"
 #undef NOEXCEPT
@@ -30,10 +29,12 @@
 
 #include "search/scorers.hpp"
 
+#include "Aql/Ast.h"
 #include "Aql/AstNode.h"
+#include "Aql/ExecutionNode.h"
 #include "Aql/Function.h"
 #include "Aql/SortCondition.h"
-#include "AqlHelper.h"
+#include "Basics/fasthash.h"
 #include "IResearchFeature.h"
 #include "IResearchOrderFactory.h"
 #include "VelocyPackHelper.h"
@@ -46,23 +47,24 @@ NS_LOCAL
 
 arangodb::aql::AstNode const EMPTY_ARGS(arangodb::aql::NODE_TYPE_ARRAY);
 
-bool validateFuncArgs(arangodb::aql::AstNode const* args, arangodb::aql::Variable const& ref) {
+// checks a specified args to be deterministic
+// and retuns reference to a loop variable
+arangodb::aql::Variable const* getScorerRef(arangodb::aql::AstNode const* args) noexcept {
   if (!args || arangodb::aql::NODE_TYPE_ARRAY != args->type) {
-    return false;
+    return nullptr;
   }
 
   size_t const size = args->numMembers();
 
   if (size < 1) {
-    return false;  // invalid args
+    return nullptr;  // invalid args
   }
 
   // 1st argument has to be reference to `ref`
   auto const* arg0 = args->getMemberUnchecked(0);
 
-  if (!arg0 || arangodb::aql::NODE_TYPE_REFERENCE != arg0->type ||
+  if (!arg0 || arangodb::aql::NODE_TYPE_REFERENCE != arg0->type) {
-      reinterpret_cast<void const*>(&ref) != arg0->getData()) {
+    return nullptr;
-    return false;
   }
 
   for (size_t i = 1, size = args->numMembers(); i < size; ++i) {
@@ -70,11 +72,11 @@ bool validateFuncArgs(arangodb::aql::AstNode const* args, arangodb::aql::Variabl
 
     if (!arg || !arg->isDeterministic()) {
       // we don't support non-deterministic arguments for scorers
-      return false;
+      return nullptr;
     }
   }
 
-  return true;
+  return reinterpret_cast<arangodb::aql::Variable const*>(arg0->getData());
 }
 
 bool makeScorer(irs::sort::ptr& scorer, irs::string_ref const& name,
@@ -87,15 +89,12 @@ bool makeScorer(irs::sort::ptr& scorer, irs::string_ref const& name,
     case 0:
       break;
     case 1: {
-      // ArangoDB, for API consistency, only supports scorers configurable via
+      // ArangoDB, for API consistency, only supports scorers configurable via jSON
-      // jSON
       scorer = irs::scorers::get(name, irs::text_format::json, irs::string_ref::NIL);
 
       if (!scorer) {
-        // ArangoDB, for API consistency, only supports scorers configurable via
+        // ArangoDB, for API consistency, only supports scorers configurable via jSON
-        // jSON
+        scorer = irs::scorers::get(name, irs::text_format::json, "[]");  // pass arg as json array
-        scorer = irs::scorers::get(name, irs::text_format::json,
-                                   "[]");  // pass arg as json array
       }
     } break;
     default: {  // fall through
@@ -123,10 +122,8 @@ bool makeScorer(irs::sort::ptr& scorer, irs::string_ref const& name,
 
       builder.close();
 
-      // ArangoDB, for API consistency, only supports scorers configurable via
+      // ArangoDB, for API consistency, only supports scorers configurable via jSON
-      // jSON
+      scorer = irs::scorers::get(name, irs::text_format::json, builder.toJson());  // pass arg as json
-      scorer = irs::scorers::get(name, irs::text_format::json,
-                                 builder.toJson());  // pass arg as json
     }
   }
 
@@ -136,15 +133,16 @@ bool makeScorer(irs::sort::ptr& scorer, irs::string_ref const& name,
 bool fromFCall(irs::sort::ptr* scorer, irs::string_ref const& scorerName,
                arangodb::aql::AstNode const* args,
                arangodb::iresearch::QueryContext const& ctx) {
-  if (!validateFuncArgs(args, *ctx.ref)) {
+  auto const* ref = getScorerRef(args);
+
+  if (ref != ctx.ref) {
     // invalid arguments
     return false;
   }
 
   if (!scorer) {
     // cheap shallow check
-    // ArangoDB, for API consistency, only supports scorers configurable via
+    // ArangoDB, for API consistency, only supports scorers configurable via jSON
-    // jSON
     return irs::scorers::exists(scorerName, irs::text_format::json);
   }
 
@@ -207,6 +205,85 @@ NS_END
 NS_BEGIN(arangodb)
 NS_BEGIN(iresearch)
 
+// ----------------------------------------------------------------------------
+// --SECTION--                                    ScorerReplacer implementation
+// ----------------------------------------------------------------------------
+
+void ScorerReplacer::replace(aql::CalculationNode& node) {
+  if (!node.expression()) {
+    return;
+  }
+
+  auto& expr = *node.expression();
+  auto* ast = expr.ast();
+
+  if (!expr.ast()) {
+    // ast is not set
+    return;
+  }
+
+  auto* exprNode = expr.nodeForModification();
+
+  if (!exprNode) {
+    // node is not set
+    return;
+  }
+
+  auto replaceScorers = [this, ast](aql::AstNode* node) {
+    if (aql::NODE_TYPE_FCALL == node->type || aql::NODE_TYPE_FCALL_USER == node->type) {
+      auto* ref = getScorerRef(node->getMember(0));
+
+      if (!ref) {
+        // invalid arguments or reference
+        return node;
+      }
+
+      QueryContext const ctx{nullptr, nullptr, nullptr, nullptr, ref};
+
+      if (!OrderFactory::scorer(nullptr, *node, ctx)) {
+        // not a scorer function
+        return node;
+      }
+
+      HashedScorer const key(ref, node);
+
+      auto it = _dedup.find(key);
+
+      if (it == _dedup.end()) {
+        // create variable
+        auto* var = ast->variables()->createTemporaryVariable();
+
+        it = _dedup.emplace(key, var).first;
+      }
+
+      return ast->createNodeReference(it->second);
+    }
+
+    return node;
+  };
+
+  // Try to modify root node of the expression
+  auto newNode = replaceScorers(exprNode);
+
+  if (exprNode != newNode) {
+    // simple expression, e.g LET x = BM25(d)
+    expr.replaceNode(newNode);
+  } else {
+    aql::Ast::traverseAndModify(exprNode, replaceScorers);
+  }
+}
+
+void ScorerReplacer::extract(aql::Variable const& var, std::vector<Scorer>& scorers) {
+  for (auto it = _dedup.begin(), end = _dedup.end(); it != end;) {
+    if (it->first.var == &var) {
+      scorers.emplace_back(it->second, it->first.node);
+      it = _dedup.erase(it);
+    } else {
+      ++it;
+    }
+  }
+}
+
 // ----------------------------------------------------------------------------
 // --SECTION--                                      OrderFactory implementation
 // ----------------------------------------------------------------------------
@@ -252,8 +329,7 @@ NS_BEGIN(iresearch)
 
   if (!comparer) {
     // cheap shallow check
-    // ArangoDB, for API consistency, only supports scorers configurable via
+    // ArangoDB, for API consistency, only supports scorers configurable via jSON
-    // jSON
     return irs::scorers::exists(scorerName, irs::text_format::json);
   }
 

arangod/IResearch/IResearchOrderFactory.h

@@ -24,6 +24,8 @@
 #ifndef ARANGOD_IRESEARCH__IRESEARCH_ORDER_FACTORY_H
 #define ARANGOD_IRESEARCH__IRESEARCH_ORDER_FACTORY_H 1
 
+#include "AqlHelper.h"
+
 #include "VocBase/voc-types.h"
 
 #include "search/sort.hpp"
@@ -31,7 +33,10 @@
 NS_BEGIN(arangodb)
 NS_BEGIN(aql)
 
+class Ast;
 struct AstNode;
+class CalculationNode;
+struct Expression;
 struct Variable;
 
 NS_END  // aql
@@ -50,18 +55,111 @@ NS_END  // transaction
 
 NS_BEGIN(iresearch)
 
+    ////////////////////////////////////////////////////////////////////////////////
+    /// @struct OrderFactory
+    ////////////////////////////////////////////////////////////////////////////////
     struct OrderFactory {
   ////////////////////////////////////////////////////////////////////////////////
   /// @brief determine if the 'node' can be converted into an iresearch scorer
-  ///        if 'scorer' != nullptr then also append build iresearch scorer
+  ///        if 'scorer' != nullptr then also append build iresearch scorer there
-  ///        there
   ////////////////////////////////////////////////////////////////////////////////
-  static bool scorer(irs::sort::ptr* scorer, arangodb::aql::AstNode const& node,
+  static bool scorer(irs::sort::ptr* scorer, aql::AstNode const& node,
-                     arangodb::iresearch::QueryContext const& ctx);
+                     iresearch::QueryContext const& ctx);
 
-  static bool comparer(irs::sort::ptr* scorer, arangodb::aql::AstNode const& node);
+  ////////////////////////////////////////////////////////////////////////////////
+  /// @brief determine if the 'node' can be converted into an iresearch scorer
+  ///        if 'scorer' != nullptr then also append build iresearch comparer there
+  ////////////////////////////////////////////////////////////////////////////////
+  static bool comparer(irs::sort::ptr* scorer, aql::AstNode const& node);
+
+  OrderFactory() = delete;
 };  // OrderFactory
 
+////////////////////////////////////////////////////////////////////////////////
+/// @struct Scorer
+/// @brief represents IResearch scorer in AQL terms
+////////////////////////////////////////////////////////////////////////////////
+struct Scorer {
+  Scorer() = default;
+
+  constexpr Scorer(aql::Variable const* var, aql::AstNode const* node) noexcept
+      : var(var), node(node) {}
+
+  constexpr bool operator==(Scorer const& rhs) const noexcept {
+    return var == rhs.var && node == rhs.node;
+  }
+
+  constexpr bool operator!=(Scorer const& rhs) const noexcept {
+    return !(*this == rhs);
+  }
+
+  aql::Variable const* var{};  // scorer variable
+  aql::AstNode const* node{};  // scorer node
+};                             // Scorer
+
+////////////////////////////////////////////////////////////////////////////////
+/// @class ScorerReplacer
+/// @brief utility class that replaces scorer function call with corresponding
+///        reference access
+////////////////////////////////////////////////////////////////////////////////
+class ScorerReplacer {
+ public:
+  ScorerReplacer() = default;
+
+  ////////////////////////////////////////////////////////////////////////////////
+  /// @brief replaces all occurences of IResearch scorers in a specified node with
+  ///        corresponding reference access
+  ////////////////////////////////////////////////////////////////////////////////
+  void replace(aql::CalculationNode& node);
+
+  ////////////////////////////////////////////////////////////////////////////////
+  /// @brief extracts replacement results for a given variable
+  ////////////////////////////////////////////////////////////////////////////////
+  void extract(aql::Variable const& var, std::vector<Scorer>& scorers);
+
+  ////////////////////////////////////////////////////////////////////////////////
+  /// @returns true if no scorers were replaced
+  ////////////////////////////////////////////////////////////////////////////////
+  bool empty() const noexcept { return _dedup.empty(); }
+
+  ////////////////////////////////////////////////////////////////////////////////
+  /// @brief visits all replaced scorer entries
+  ////////////////////////////////////////////////////////////////////////////////
+  template <typename Visitor>
+  bool visit(Visitor visitor) const {
+    for (auto& entry : _dedup) {
+      if (!visitor(entry.first)) {
+        return false;
+      }
+    }
+    return true;
+  }
+
+ private:
+  struct HashedScorer : Scorer {
+    HashedScorer(aql::Variable const* var, aql::AstNode const* node)
+        : Scorer(var, node), hash(iresearch::hash(node)) {}
+
+    size_t hash;
+  };  // HashedScorer
+
+  struct ScorerHash {
+    size_t operator()(HashedScorer const& key) const noexcept {
+      return key.hash;
+    }
+  };  // ScorerHash
+
+  struct ScorerEqualTo {
+    bool operator()(HashedScorer const& lhs, HashedScorer const& rhs) const {
+      return iresearch::equalTo(lhs.node, rhs.node);
+    }
+  };  // ScorerEqualTo
+
+  typedef std::unordered_map<HashedScorer, aql::Variable const*, ScorerHash, ScorerEqualTo> DedupScorers;
+
+  DedupScorers _dedup;
+};  // ScorerReplacer
+
 NS_END      // iresearch
     NS_END  // arangodb
 

arangod/IResearch/IResearchViewBlock.cpp

@@ -202,15 +202,16 @@ void IResearchViewBlockBase::reset() {
       irs::order order;
       irs::sort::ptr scorer;
 
-      for (auto const& sort : viewNode.sortCondition()) {
+      for (auto const& scorerNode : viewNode.scorers()) {
-        TRI_ASSERT(sort.node);
+        TRI_ASSERT(scorerNode.node);
 
-        if (!arangodb::iresearch::OrderFactory::scorer(&scorer, *sort.node, queryCtx)) {
+        if (!arangodb::iresearch::OrderFactory::scorer(&scorer, *scorerNode.node, queryCtx)) {
           // failed to append sort
           THROW_ARANGO_EXCEPTION(TRI_ERROR_BAD_PARAMETER);
         }
 
-        order.add(sort.asc, std::move(scorer));
+        // sorting order doesn't matter
+        order.add(true, std::move(scorer));
       }
 
       // compile order
@@ -427,7 +428,7 @@ bool IResearchViewBlock::resetIterator() {
     auto const& viewNode =
         *ExecutionNode::castTo<IResearchViewNode const*>(getPlanNode());
 
-    TRI_ASSERT(numScores == viewNode.sortCondition().size());
+    TRI_ASSERT(numScores == viewNode.scorers().size());
 #endif
   } else {
     _scr = &irs::score::no_score();

arangod/IResearch/IResearchViewNode.cpp

@@ -33,7 +33,6 @@
 #include "Basics/StringUtils.h"
 #include "Cluster/ClusterInfo.h"
 #include "IResearchCommon.h"
-#include "IResearchOrderFactory.h"
 #include "IResearchView.h"
 #include "IResearchViewBlock.h"
 #include "IResearchViewCoordinator.h"
@@ -59,19 +58,18 @@ inline bool filterConditionIsEmpty(aql::AstNode const* filterCondition) {
 // -----------------------------------------------------------------------------
 
 void toVelocyPack(velocypack::Builder& builder,
-                  std::vector<arangodb::iresearch::IResearchSort> const& sorts,
+                  std::vector<arangodb::iresearch::Scorer> const& scorers, bool verbose) {
-                  bool verbose) {
   VPackArrayBuilder arrayScope(&builder);
-  for (auto const sort : sorts) {
+  for (auto const& scorer : scorers) {
     VPackObjectBuilder objectScope(&builder);
-    builder.add("varId", VPackValue(sort.var->id));
+    builder.add("id", VPackValue(scorer.var->id));
-    builder.add("asc", VPackValue(sort.asc));
+    builder.add("name", VPackValue(scorer.var->name));  // for explainer.js
     builder.add(VPackValue("node"));
-    sort.node->toVelocyPack(builder, verbose);
+    scorer.node->toVelocyPack(builder, verbose);
   }
 }
 
-std::vector<arangodb::iresearch::IResearchSort> fromVelocyPack(
+std::vector<arangodb::iresearch::Scorer> fromVelocyPack(
     arangodb::aql::ExecutionPlan& plan, arangodb::velocypack::Slice const& slice) {
   if (!slice.isArray()) {
     LOG_TOPIC(ERR, arangodb::iresearch::TOPIC)
@@ -85,11 +83,11 @@ std::vector<arangodb::iresearch::IResearchSort> fromVelocyPack(
   auto const* vars = plan.getAst()->variables();
   TRI_ASSERT(vars);
 
-  std::vector<arangodb::iresearch::IResearchSort> sorts;
+  std::vector<arangodb::iresearch::Scorer> scorers;
 
   size_t i = 0;
   for (auto const sortSlice : velocypack::ArrayIterator(slice)) {
-    auto const varIdSlice = sortSlice.get("varId");
+    auto const varIdSlice = sortSlice.get("id");
 
     if (!varIdSlice.isNumber()) {
       LOG_TOPIC(ERR, arangodb::iresearch::TOPIC)
@@ -107,24 +105,14 @@ std::vector<arangodb::iresearch::IResearchSort> fromVelocyPack(
       return {};
     }
 
-    auto const ascSlice = sortSlice.get("asc");
-
-    if (!ascSlice.isBoolean()) {
-      LOG_TOPIC(ERR, arangodb::iresearch::TOPIC)
-          << "malformed order mark at line " << i << "', boolean expected";
-      return {};
-    }
-
-    bool const asc = ascSlice.getBoolean();
-
     // will be owned by Ast
     auto* node = new aql::AstNode(ast, sortSlice.get("node"));
 
-    sorts.emplace_back(var, node, asc);
+    scorers.emplace_back(var, node);
     ++i;
   }
 
-  return sorts;
+  return scorers;
 }
 
 // -----------------------------------------------------------------------------
@@ -224,7 +212,7 @@ bool parseOptions(aql::AstNode const* optionsNode,
 // -----------------------------------------------------------------------------
 
 // in loop or non-deterministic
-bool hasDependecies(aql::ExecutionPlan const& plan, aql::AstNode const& node,
+bool hasDependencies(aql::ExecutionPlan const& plan, aql::AstNode const& node,
                     aql::Variable const& ref,
                     std::unordered_set<aql::Variable const*>& vars) {
   if (!node.isDeterministic()) {
@@ -281,16 +269,16 @@ int evaluateVolatility(arangodb::iresearch::IResearchViewNode const& node) {
   // evaluate filter condition volatility
   auto& filterCondition = node.filterCondition();
   if (!::filterConditionIsEmpty(&filterCondition) && inInnerLoop) {
-    irs::set_bit<0>(::hasDependecies(plan, filterCondition, outVariable, vars), mask);
+    irs::set_bit<0>(::hasDependencies(plan, filterCondition, outVariable, vars), mask);
   }
 
   // evaluate sort condition volatility
-  auto& sortCondition = node.sortCondition();
+  auto& scorers = node.scorers();
-  if (!sortCondition.empty() && inInnerLoop) {
+  if (!scorers.empty() && inInnerLoop) {
     vars.clear();
 
-    for (auto const& sort : sortCondition) {
+    for (auto const& scorer : scorers) {
-      if (::hasDependecies(plan, *sort.node, outVariable, vars)) {
+      if (::hasDependencies(plan, *scorer.node, outVariable, vars)) {
         irs::set_bit<1>(mask);
         break;
       }
@@ -315,19 +303,18 @@ namespace iresearch {
 
 IResearchViewNode::IResearchViewNode(aql::ExecutionPlan& plan, size_t id,
                                      TRI_vocbase_t& vocbase,
-                                     std::shared_ptr<const arangodb::LogicalView> const& view,
+                                     std::shared_ptr<const LogicalView> const& view,
-                                     arangodb::aql::Variable const& outVariable,
+                                     aql::Variable const& outVariable,
-                                     arangodb::aql::AstNode* filterCondition,
+                                     aql::AstNode* filterCondition,
-                                     arangodb::aql::AstNode* options,
+                                     aql::AstNode* options, std::vector<Scorer>&& scorers)
-                                     std::vector<IResearchSort>&& sortCondition)
+    : aql::ExecutionNode(&plan, id),
-    : arangodb::aql::ExecutionNode(&plan, id),
       _vocbase(vocbase),
       _view(view),
       _outVariable(&outVariable),
       // in case if filter is not specified
       // set it to surrogate 'RETURN ALL' node
       _filterCondition(filterCondition ? filterCondition : &ALL),
-      _sortCondition(std::move(sortCondition)) {
+      _scorers(std::move(scorers)) {
   TRI_ASSERT(_view);
   TRI_ASSERT(iresearch::DATA_SOURCE_TYPE == _view->type());
   TRI_ASSERT(LogicalView::category() == _view->category());
@@ -348,7 +335,7 @@ IResearchViewNode::IResearchViewNode(aql::ExecutionPlan& plan, velocypack::Slice
       // in case if filter is not specified
       // set it to surrogate 'RETURN ALL' node
       _filterCondition(&ALL),
-      _sortCondition(fromVelocyPack(plan, base.get("sortCondition"))) {
+      _scorers(fromVelocyPack(plan, base.get("scorers"))) {
   // view
   auto const viewIdSlice = base.get("viewId");
 
@@ -442,10 +429,10 @@ void IResearchViewNode::planNodeRegisters(std::vector<aql::RegisterId>& nrRegsHe
   //  }
 
   // plan registers for output scores
-  for (auto const& sort : _sortCondition) {
+  for (auto const& scorer : _scorers) {
     ++nrRegsHere[depth];
     ++nrRegs[depth];
-    varInfo.emplace(sort.var->id, VarInfo(depth, totalNrRegs++));
+    varInfo.emplace(scorer.var->id, VarInfo(depth, totalNrRegs++));
   }
 }
 
@@ -484,8 +471,8 @@ void IResearchViewNode::toVelocyPackHelper(VPackBuilder& nodes, unsigned flags)
   }
 
   // sort condition
-  nodes.add(VPackValue("sortCondition"));
+  nodes.add(VPackValue("scorers"));
-  ::toVelocyPack(nodes, _sortCondition, flags != 0);
+  ::toVelocyPack(nodes, _scorers, flags != 0);
 
   // shards
   {
@@ -544,8 +531,8 @@ aql::ExecutionNode* IResearchViewNode::clone(aql::ExecutionPlan* plan, bool with
 
   auto node =
       std::make_unique<IResearchViewNode>(*plan, _id, _vocbase, _view, *outVariable,
-                                          const_cast<aql::AstNode*>(_filterCondition), nullptr,
+                                          const_cast<aql::AstNode*>(_filterCondition),
-                                          decltype(_sortCondition)(_sortCondition));
+                                          nullptr, decltype(_scorers)(_scorers));
   node->_shards = _shards;
   node->_options = _options;
   node->_volatilityMask = _volatilityMask;
@@ -579,6 +566,10 @@ void IResearchViewNode::getVariablesUsedHere(std::unordered_set<aql::Variable co
   if (!::filterConditionIsEmpty(_filterCondition)) {
     aql::Ast::getReferencedVariables(_filterCondition, vars);
   }
+
+  for (auto& scorer : _scorers) {
+    aql::Ast::getReferencedVariables(scorer.node, vars);
+  }
 }
 
 void IResearchViewNode::filterCondition(aql::AstNode const* node) noexcept {
@@ -665,7 +656,7 @@ std::unique_ptr<aql::ExecutionBlock> IResearchViewNode::createBlock(
   LOG_TOPIC(TRACE, arangodb::iresearch::TOPIC)
       << "Finish getting snapshot for view '" << view.name() << "'";
 
-  if (_sortCondition.empty()) {
+  if (_scorers.empty()) {
     // unordered case
     return std::make_unique<IResearchViewUnorderedBlock>(*reader, engine, *this);
   }

arangod/IResearch/IResearchViewNode.h

@@ -24,6 +24,8 @@
 #ifndef ARANGOD_IRESEARCH__IRESEARCH_VIEW_NODE_H
 #define ARANGOD_IRESEARCH__IRESEARCH_VIEW_NODE_H 1
 
+#include "IResearchOrderFactory.h"
+
 #include "Aql/Collection.h"
 #include "Aql/ExecutionNode.h"
 
@@ -36,25 +38,6 @@ class ExecutionEngine;
 
 namespace iresearch {
 
-struct IResearchSort {
-  IResearchSort() = default;
-
-  IResearchSort(aql::Variable const* var, aql::AstNode const* node, bool asc) noexcept
-      : var(var), node(node), asc(asc) {}
-
-  bool operator==(IResearchSort const& rhs) const noexcept {
-    return var == rhs.var && node == rhs.node && asc == rhs.asc;
-  }
-
-  bool operator!=(IResearchSort const& rhs) const noexcept {
-    return !(*this == rhs);
-  }
-
-  aql::Variable const* var{};
-  aql::AstNode const* node{};
-  bool asc{};
-};  // IResearchSort
-
 /// @brief class EnumerateViewNode
 class IResearchViewNode final : public arangodb::aql::ExecutionNode {
   friend class arangodb::aql::RedundantCalculationsReplacer;
@@ -69,7 +52,7 @@ class IResearchViewNode final : public arangodb::aql::ExecutionNode {
   IResearchViewNode(aql::ExecutionPlan& plan, size_t id, TRI_vocbase_t& vocbase,
                     std::shared_ptr<const arangodb::LogicalView> const& view,
                     aql::Variable const& outVariable, aql::AstNode* filterCondition,
-                    aql::AstNode* options, std::vector<IResearchSort>&& sortCondition);
+                    aql::AstNode* options, std::vector<Scorer>&& scorers);
 
   IResearchViewNode(aql::ExecutionPlan&, velocypack::Slice const& base);
 
@@ -94,10 +77,10 @@ class IResearchViewNode final : public arangodb::aql::ExecutionNode {
 
   /// @brief getVariablesSetHere
   std::vector<arangodb::aql::Variable const*> getVariablesSetHere() const override final {
-    std::vector<arangodb::aql::Variable const*> vars(1 + _sortCondition.size());
+    std::vector<arangodb::aql::Variable const*> vars(1 + _scorers.size());
 
-    *std::transform(_sortCondition.begin(), sortCondition().end(), vars.begin(),
+    *std::transform(_scorers.begin(), _scorers.end(), vars.begin(),
-                    [](IResearchSort const& sort) { return sort.var; }) = _outVariable;
+                    [](auto const& scorer) { return scorer.var; }) = _outVariable;
 
     return vars;
   }
@@ -134,13 +117,11 @@ class IResearchViewNode final : public arangodb::aql::ExecutionNode {
   std::vector<std::string>& shards() noexcept { return _shards; }
 
   /// @brief return the condition to pass to the view
-  std::vector<IResearchSort> const& sortCondition() const noexcept {
+  std::vector<Scorer> const& scorers() const noexcept { return _scorers; }
-    return _sortCondition;
-  }
 
   /// @brief set the sort condition to pass to the view
-  void sortCondition(std::vector<IResearchSort>&& sortCondition) noexcept {
+  void scorers(std::vector<Scorer>&& scorers) noexcept {
-    _sortCondition = std::move(sortCondition);
+    _scorers = std::move(scorers);
   }
 
   /// @brief getVariablesUsedHere, returning a vector
@@ -184,8 +165,8 @@ class IResearchViewNode final : public arangodb::aql::ExecutionNode {
   /// @brief filter node to pass to view
   aql::AstNode const* _filterCondition;
 
-  /// @brief sortCondition to pass to the view
+  /// @brief scorers related to the view
-  std::vector<IResearchSort> _sortCondition;
+  std::vector<Scorer> _scorers;
 
   /// @brief list of shards involved, need this for the cluster
   std::vector<std::string> _shards;

arangod/IResearch/IResearchViewOptimizerRules.cpp

@@ -26,6 +26,7 @@
 #include "Aql/Condition.h"
 #include "Aql/ExecutionNode.h"
 #include "Aql/ExecutionPlan.h"
+#include "Aql/Function.h"
 #include "Aql/Optimizer.h"
 #include "Aql/Query.h"
 #include "Aql/SortNode.h"
@@ -38,6 +39,8 @@
 #include "Utils/CollectionNameResolver.h"
 #include "VocBase/LogicalCollection.h"
 
+#include "utils/misc.hpp"
+
 using namespace arangodb::iresearch;
 using namespace arangodb::aql;
 using EN = arangodb::aql::ExecutionNode;
@@ -58,51 +61,6 @@ size_t numberOfShards(arangodb::CollectionNameResolver const& resolver,
   return numberOfShards;
 }
 
-std::vector<arangodb::iresearch::IResearchSort> buildSort(
-    ExecutionPlan const& plan, arangodb::aql::Variable const& ref,
-    std::vector<std::pair<Variable const*, bool>> const& sorts,
-    std::map<VariableId, AstNode const*> const& vars, bool scorersOnly) {
-  std::vector<IResearchSort> entries;
-
-  QueryContext const ctx{nullptr, nullptr, nullptr, nullptr, &ref};
-
-  for (auto& sort : sorts) {
-    auto const* var = sort.first;
-    auto varId = var->id;
-
-    AstNode const* rootNode = nullptr;
-    auto it = vars.find(varId);
-
-    if (it != vars.end()) {
-      auto const* node = rootNode = it->second;
-
-      while (node && NODE_TYPE_ATTRIBUTE_ACCESS == node->type) {
-        node = node->getMember(0);
-      }
-
-      if (node && NODE_TYPE_REFERENCE == node->type) {
-        var = reinterpret_cast<Variable const*>(node->getData());
-      }
-    } else {
-      auto const* setter = plan.getVarSetBy(varId);
-      if (setter && EN::CALCULATION == setter->getType()) {
-        auto const* expr =
-            ExecutionNode::castTo<CalculationNode const*>(setter)->expression();
-
-        if (expr) {
-          rootNode = expr->node();
-        }
-      }
-    }
-
-    if (var && rootNode && (!scorersOnly || OrderFactory::scorer(nullptr, *rootNode, ctx))) {
-      entries.emplace_back(var, rootNode, sort.second);
-    }
-  }
-
-  return entries;
-}
-
 bool addView(arangodb::LogicalView const& view, arangodb::aql::Query& query) {
   auto* collections = query.collections();
 
@@ -120,167 +78,57 @@ bool addView(arangodb::LogicalView const& view, arangodb::aql::Query& query) {
   return view.visitCollections(visitor);
 }
 
-///////////////////////////////////////////////////////////////////////////////
+bool optimizeSearchCondition(IResearchViewNode& viewNode, Query& query, ExecutionPlan& plan) {
-/// @class IResearchViewConditionFinder
+  auto view = viewNode.view();
-///////////////////////////////////////////////////////////////////////////////
-class IResearchViewConditionHandler final
-    : public arangodb::aql::WalkerWorker<ExecutionNode> {
- public:
-  IResearchViewConditionHandler(ExecutionPlan& plan,
-                                std::set<arangodb::iresearch::IResearchViewNode const*>& processedViewNodes) noexcept
-      : _plan(&plan), _processedViewNodes(&processedViewNodes) {}
-
-  virtual bool before(ExecutionNode*) override;
-
-  virtual bool enterSubquery(ExecutionNode*, ExecutionNode*) override {
-    return false;
-  }
-
- private:
-  bool handleFilterCondition(ExecutionNode* en, Condition& condition);
-
-  ExecutionPlan* _plan;
-  std::vector<std::pair<Variable const*, bool>> _sorts;
-  // map and set are 25-30% faster than corresponding
-  // unordered_set for small number of elements
-  std::map<VariableId, AstNode const*> _variableDefinitions;
-  std::set<arangodb::iresearch::IResearchViewNode const*>* _processedViewNodes;
-};  // IResearchViewConditionFinder
-
-bool IResearchViewConditionHandler::before(ExecutionNode* en) {
-  switch (en->getType()) {
-    case EN::LIMIT:
-      // LIMIT invalidates the sort expression we already found
-      _sorts.clear();
-      break;
-
-    case EN::SINGLETON:
-    case EN::NORESULTS:
-      // in all these cases we better abort
-      return true;
-
-    case EN::SORT: {
-      // register which variables are used in a SORT
-      if (_sorts.empty()) {
-        for (auto& it : EN::castTo<SortNode const*>(en)->elements()) {
-          _sorts.emplace_back(it.var, it.ascending);
-          TRI_IF_FAILURE("IResearchViewConditionFinder::sortNode") {
-            THROW_ARANGO_EXCEPTION(TRI_ERROR_DEBUG);
-          }
-        }
-      }
-      break;
-    }
-
-    case EN::CALCULATION: {
-      auto outvars = en->getVariablesSetHere();
-      TRI_ASSERT(outvars.size() == 1);
-
-      _variableDefinitions.emplace(
-          outvars[0]->id, EN::castTo<CalculationNode const*>(en)->expression()->node());
-      TRI_IF_FAILURE("IResearchViewConditionFinder::variableDefinition") {
-        THROW_ARANGO_EXCEPTION(TRI_ERROR_DEBUG);
-      }
-      break;
-    }
-
-    case EN::ENUMERATE_IRESEARCH_VIEW: {
-      auto node = EN::castTo<IResearchViewNode*>(en);
-      auto& view = *node->view();
 
   // add view and linked collections to the query
-      TRI_ASSERT(_plan && _plan->getAst() && _plan->getAst()->query());
+  if (!addView(*view, query)) {
-      if (!addView(view, *_plan->getAst()->query())) {
     THROW_ARANGO_EXCEPTION_MESSAGE(
         TRI_ERROR_QUERY_PARSE,
         "failed to process all collections linked with the view '" +
-                view.name() + "'");
+            view->name() + "'");
   }
 
-      if (_processedViewNodes->find(node) != _processedViewNodes->end()) {
+  // build search condition
-        // already optimized this node
+  Condition searchCondition(plan.getAst());
-        break;
+
+  if (!viewNode.filterConditionIsEmpty()) {
+    searchCondition.andCombine(&viewNode.filterCondition());
+    searchCondition.normalize(&plan);  // normalize the condition
+
+    if (searchCondition.isEmpty()) {
+      // condition is always false
+      for (auto const& x : viewNode.getParents()) {
+        plan.insertDependency(x, plan.registerNode(
+                                     std::make_unique<NoResultsNode>(&plan, plan.nextId())));
+      }
+      return false;
     }
 
-      Condition filterCondition(_plan->getAst());
+    auto const& varsValid = viewNode.getVarsValid();
 
-      if (!node->filterConditionIsEmpty()) {
+    // remove all invalid variables from the condition
-        filterCondition.andCombine(&node->filterCondition());
+    if (searchCondition.removeInvalidVariables(varsValid)) {
-
+      // removing left a previously non-empty OR block empty...
-        if (!handleFilterCondition(en, filterCondition)) {
+      // this means we can't use the index to restrict the results
-          break;
+      return false;
     }
   }
 
-      auto sortCondition =
+  // check filter condition
-          buildSort(*_plan, node->outVariable(), _sorts, _variableDefinitions,
+  auto const conditionValid =
-                    true  // node->isInInnerLoop() // build scorers only in case
+      !searchCondition.root() ||
-                          // if we're inside a loop
-          );
-
-      if (filterCondition.isEmpty() && sortCondition.empty()) {
-        // no conditions left
-        break;
-      }
-
-      auto const canUseView =
-          !filterCondition.root() ||
       FilterFactory::filter(nullptr,
-                                {nullptr, nullptr, nullptr, nullptr, &node->outVariable()},
+                            {nullptr, nullptr, nullptr, nullptr, &viewNode.outVariable()},
-                                *filterCondition.root());
+                            *searchCondition.root());
 
-      if (!canUseView) {
+  if (!conditionValid) {
     THROW_ARANGO_EXCEPTION_MESSAGE(TRI_ERROR_QUERY_PARSE,
                                    "unsupported SEARCH condition");
   }
 
-      node->filterCondition(filterCondition.root());
+  if (!searchCondition.isEmpty()) {
-      node->sortCondition(std::move(sortCondition));
+    viewNode.filterCondition(searchCondition.root());
-
-      TRI_IF_FAILURE("IResearchViewConditionFinder::insertViewNode") {
-        THROW_ARANGO_EXCEPTION(TRI_ERROR_DEBUG);
-      }
-
-      _processedViewNodes->insert(node);
-
-      break;
-    }
-
-    default:
-      // in these cases we simply ignore the intermediate nodes, note
-      // that we have taken care of nodes that could throw exceptions
-      // above.
-      break;
-  }
-
-  return false;
-}
-
-bool IResearchViewConditionHandler::handleFilterCondition(ExecutionNode* en,
-                                                          Condition& condition) {
-  // normalize the condition
-  condition.normalize(_plan);
-  TRI_IF_FAILURE("ConditionFinder::normalizePlan") {
-    THROW_ARANGO_EXCEPTION(TRI_ERROR_DEBUG);
-  }
-
-  if (condition.isEmpty()) {
-    // condition is always false
-    for (auto const& x : en->getParents()) {
-      auto noRes = new NoResultsNode(_plan, _plan->nextId());
-      _plan->registerNode(noRes);
-      _plan->insertDependency(x, noRes);
-    }
-    return false;
-  }
-
-  auto const& varsValid = en->getVarsValid();
-
-  // remove all invalid variables from the condition
-  if (condition.removeInvalidVariables(varsValid)) {
-    // removing left a previously non-empty OR block empty...
-    // this means we can't use the index to restrict the results
-    return false;
   }
 
   return true;
@@ -288,56 +136,72 @@ bool IResearchViewConditionHandler::handleFilterCondition(ExecutionNode* en,
 
 }  // namespace
 
-NS_BEGIN(arangodb)
+namespace arangodb {
-NS_BEGIN(iresearch)
+namespace iresearch {
 
 /// @brief move filters and sort conditions into views
 void handleViewsRule(arangodb::aql::Optimizer* opt,
                      std::unique_ptr<arangodb::aql::ExecutionPlan> plan,
                      arangodb::aql::OptimizerRule const* rule) {
+  TRI_ASSERT(plan && plan->getAst() && plan->getAst()->query());
+  auto& query = *plan->getAst()->query();
+
+  // ensure 'Optimizer::addPlan' will be called
+  bool modified = false;
+  auto addPlan = irs::make_finally([opt, &plan, rule, &modified]() {
+    opt->addPlan(std::move(plan), rule, modified);
+  });
+
+  if (query.views().empty()) {
+    // nothing to do in absence of views
+    return;
+  }
+
   SmallVector<ExecutionNode*>::allocator_type::arena_type a;
   SmallVector<ExecutionNode*> nodes{a};
 
-  // try to find `EnumerateViewNode`s and process corresponding filters and
+  // replace scorers in all calculation nodes with references
-  // sorts
+  plan->findNodesOfType(nodes, EN::CALCULATION, true);
-  plan->findEndNodes(nodes, true);
 
-  // set of processed view nodes
+  ScorerReplacer scorerReplacer;
-  std::set<IResearchViewNode const*> processedViewNodes;
 
-  IResearchViewConditionHandler handler(*plan, processedViewNodes);
+  for (auto* node : nodes) {
-  for (auto const& n : nodes) {
+    TRI_ASSERT(node && EN::CALCULATION == node->getType());
-    n->walk(handler);
+
+    scorerReplacer.replace(*EN::castTo<CalculationNode*>(node));
   }
 
-  if (!processedViewNodes.empty()) {
+  // register replaced scorers to be evaluated by corresponding view nodes
-    std::unordered_set<ExecutionNode*> toUnlink;
+  nodes.clear();
+  plan->findNodesOfType(nodes, EN::ENUMERATE_IRESEARCH_VIEW, true);
 
-    // remove sort setters covered by a view internally
+  std::vector<Scorer> scorers;
-    for (auto* viewNode : processedViewNodes) {
-      TRI_ASSERT(viewNode);
 
-      for (auto const& sort : viewNode->sortCondition()) {
+  for (auto* node : nodes) {
-        auto const* var = sort.var;
+    TRI_ASSERT(node && EN::ENUMERATE_IRESEARCH_VIEW == node->getType());
+    auto& viewNode = *EN::castTo<IResearchViewNode*>(node);
 
-        if (!var) {
+    if (!optimizeSearchCondition(viewNode, query, *plan)) {
       continue;
     }
 
-        auto* setter = plan->getVarSetBy(var->id);
+    // find scorers that have to be evaluated by a view
+    scorerReplacer.extract(viewNode.outVariable(), scorers);
+    viewNode.scorers(std::move(scorers));
 
-        if (!setter || EN::CALCULATION != setter->getType()) {
+    modified = true;
-          continue;
   }
 
-        toUnlink.emplace(setter);
+  // ensure all replaced scorers are covered by corresponding view nodes
-      }
+  scorerReplacer.visit([](Scorer const& scorer) -> bool {
-    }
+    TRI_ASSERT(scorer.node);
+    auto const funcName = iresearch::getFuncName(*scorer.node);
 
-    plan->unlinkNodes(toUnlink);
+    THROW_ARANGO_EXCEPTION_FORMAT(
-  }
+        TRI_ERROR_QUERY_FUNCTION_ARGUMENT_TYPE_MISMATCH,
-
+        "Non ArangoSearch view variable '%s' is used in scorer function '%s'",
-  opt->addPlan(std::move(plan), rule, !processedViewNodes.empty());
+        scorer.var->name.c_str(), funcName.c_str());
+  });
 }
 
 void scatterViewInClusterRule(arangodb::aql::Optimizer* opt,
@@ -454,8 +318,8 @@ void scatterViewInClusterRule(arangodb::aql::Optimizer* opt,
   opt->addPlan(std::move(plan), rule, wasModified);
 }
 
-NS_END      // iresearch
+}  // namespace iresearch
-    NS_END  // arangodb
+}  // namespace arangodb
 
 // -----------------------------------------------------------------------------
 // --SECTION--                                                       END-OF-FILE

js/common/modules/@arangodb/aql/explainer.js

@@ -1020,7 +1020,15 @@ function processQuery (query, explain, planIndex) {
         if (node.condition && node.condition.hasOwnProperty('type')) {
           condition = ' ' + keyword('SEARCH') + ' ' + buildExpression(node.condition);
         }
-        return keyword('FOR') + ' ' + variableName(node.outVariable) + ' ' + keyword('IN') + ' ' + view(node.view) + condition + '   ' + annotation('/* view query */');
+
+        var scorers = '';
+        if (node.scorers && node.scorers.length > 0) {
+          scorers = keyword(' LET ' ) + node.scorers.map(function(scorer) {
+            return variableName(scorer) + ' = ' + buildExpression(scorer.node);
+          }).join(', ');
+        }
+
+        return keyword('FOR') + ' ' + variableName(node.outVariable) + ' ' + keyword('IN') + ' ' + view(node.view) + condition + scorers + '   ' + annotation('/* view query */');
       case 'IndexNode':
         collectionVariables[node.outVariable.id] = node.collection;
         node.indexes.forEach(function(idx, i) { iterateIndexes(idx, i, node, types, false); });

tests/CMakeLists.txt

@@ -44,6 +44,7 @@ if (USE_IRESEARCH)
     IResearch/IResearchQueryNumericTerm-test.cpp
     IResearch/IResearchQueryOr-test.cpp
     IResearch/IResearchQueryPhrase-test.cpp
+    IResearch/IResearchQueryScorer-test.cpp
     IResearch/IResearchQuerySelectAll-test.cpp
     IResearch/IResearchQueryStartsWith-test.cpp
     IResearch/IResearchQueryStringTerm-test.cpp

tests/IResearch/IResearchQueryJoin-test.cpp

@@ -68,7 +68,6 @@
 #include "IResearch/VelocyPackHelper.h"
 #include "analysis/analyzers.hpp"
 #include "analysis/token_attributes.hpp"
-#include "search/scorers.hpp"
 #include "utils/utf8_path.hpp"
 
 #include <velocypack/Iterator.h>
@@ -77,101 +76,6 @@ extern const char* ARGV0; // defined in main.cpp
 
 NS_LOCAL
 
-struct CustomScorer : public irs::sort {
-  struct Prepared: public irs::sort::prepared_base<float_t> {
-   public:
-    DECLARE_FACTORY(Prepared);
-
-    Prepared(float_t i)
-      : i(i) {
-    }
-
-    virtual void add(irs::byte_type* dst, const irs::byte_type* src) const override {
-      score_cast(dst) += score_cast(src);
-    }
-
-    virtual irs::flags const& features() const override {
-      return irs::flags::empty_instance();
-    }
-
-    virtual bool less(const irs::byte_type* lhs, const irs::byte_type* rhs) const override {
-      return score_cast(lhs) < score_cast(rhs);
-    }
-
-    virtual irs::sort::collector::ptr prepare_collector() const override {
-      return nullptr;
-    }
-
-    virtual void prepare_score(irs::byte_type* score) const override {
-      score_cast(score) = 0.f;
-    }
-
-    virtual irs::sort::scorer::ptr prepare_scorer(
-      irs::sub_reader const&,
-      irs::term_reader const&,
-      irs::attribute_store const&,
-      irs::attribute_view const&
-    ) const override {
-      struct Scorer : public irs::sort::scorer {
-        Scorer(float_t score): i(score) { }
-
-        virtual void score(irs::byte_type* score_buf) override {
-          *reinterpret_cast<score_t*>(score_buf) = i;
-        }
-
-        float_t i;
-      };
-
-      return irs::sort::scorer::make<Scorer>(i);
-    }
-
-    float_t i;
-  };
-
-  static ::iresearch::sort::type_id const& type() {
-    static ::iresearch::sort::type_id TYPE("customscorer");
-    return TYPE;
-  }
-
-  static irs::sort::ptr make(irs::string_ref const& args) {
-    if (args.null()) {
-      return std::make_shared<CustomScorer>(0.f);
-    }
-
-    // velocypack::Parser::fromJson(...) will throw exception on parse error
-    auto json = arangodb::velocypack::Parser::fromJson(args.c_str(), args.size());
-    auto slice = json ? json->slice() : arangodb::velocypack::Slice();
-
-    if (!slice.isArray()) {
-      return nullptr; // incorrect argument format
-    }
-
-    arangodb::velocypack::ArrayIterator itr(slice);
-
-    if (!itr.valid()) {
-      return nullptr;
-    }
-
-    auto const value = itr.value();
-
-    if (!value.isNumber()) {
-      return nullptr;
-    }
-
-    return std::make_shared<CustomScorer>(itr.value().getNumber<size_t>());
-  }
-
-  CustomScorer(size_t i) : irs::sort(CustomScorer::type()), i(i) {}
-
-  virtual irs::sort::prepared::ptr prepare() const override {
-    return irs::memory::make_unique<Prepared>(static_cast<float_t>(i));
-  }
-
-  size_t i;
-}; // CustomScorer
-
-REGISTER_SCORER_JSON(CustomScorer, CustomScorer::make);
-
 // -----------------------------------------------------------------------------
 // --SECTION--                                                 setup / tear-down
 // -----------------------------------------------------------------------------
@@ -1362,12 +1266,6 @@ TEST_CASE("IResearchQueryTestJoin", "[iresearch][iresearch-query]") {
   }
 
   // invalid reference in scorer
-  //
-  // FOR x IN 0..5
-  //   FOR d IN testView
-  //   SEARCH d.seq == x
-  //   SORT customscorer(x,x)
-  // RETURN d;
   {
     std::string const query = "FOR d IN testView FOR i IN 0..5 SORT tfidf(i) DESC RETURN d";
 
@@ -1376,7 +1274,7 @@ TEST_CASE("IResearchQueryTestJoin", "[iresearch][iresearch-query]") {
     ));
 
     auto queryResult = arangodb::tests::executeQuery(vocbase, query);
-    REQUIRE(TRI_ERROR_NOT_IMPLEMENTED == queryResult.code);
+    REQUIRE(TRI_ERROR_QUERY_FUNCTION_ARGUMENT_TYPE_MISMATCH == queryResult.code);
   }
 
   // FOR i IN 1..5
@@ -1457,6 +1355,85 @@ TEST_CASE("IResearchQueryTestJoin", "[iresearch][iresearch-query]") {
     CHECK(expectedDoc == expectedDocs.end());
   }
 
+  {
+    std::string const query = "LET attr = _NONDETERM_('seq') "
+                              "FOR i IN 1..5 "
+                              "  FOR x IN collection_1 FILTER x.seq == i "
+                              "    FOR d IN testView SEARCH d.seq == x.seq AND d.name == x.name "
+                              "      SORT customscorer(d, x[attr]) DESC "
+                              "RETURN d";
+
+    CHECK(arangodb::tests::assertRules(
+      vocbase, query,
+      {
+        arangodb::aql::OptimizerRule::handleArangoSearchViewsRule,
+      }
+    ));
+
+    std::vector<arangodb::velocypack::Slice> expectedDocs {
+      arangodb::velocypack::Slice(insertedDocsView[4].vpack()),
+      arangodb::velocypack::Slice(insertedDocsView[2].vpack()),
+    };
+
+    auto queryResult = arangodb::tests::executeQuery(vocbase, query);
+    REQUIRE(TRI_ERROR_NO_ERROR == queryResult.code);
+
+    auto result = queryResult.result->slice();
+    CHECK(result.isArray());
+
+    arangodb::velocypack::ArrayIterator resultIt(result);
+    REQUIRE(expectedDocs.size() == resultIt.size());
+
+    // Check documents
+    auto expectedDoc = expectedDocs.begin();
+    for (;resultIt.valid(); resultIt.next(), ++expectedDoc) {
+      auto const actualDoc = resultIt.value();
+      auto const resolved = actualDoc.resolveExternals();
+
+      CHECK((0 == arangodb::basics::VelocyPackHelper::compare(arangodb::velocypack::Slice(*expectedDoc), resolved, true)));
+    }
+    CHECK(expectedDoc == expectedDocs.end());
+  }
+
+  // FOR i IN 1..5
+  //  FOR x IN collection_0 SEARCH x.seq == i
+  //    FOR d IN  SEARCH d.seq == x.seq && d.name == x.name
+  // SORT customscorer(d, x.seq)
+  {
+    std::string const query = "FOR i IN 1..5 FOR x IN collection_1 FILTER x.seq == i FOR d IN testView SEARCH d.seq == x.seq AND d.name == x.name SORT customscorer(d, x['seq']) DESC RETURN d";
+
+    CHECK(arangodb::tests::assertRules(
+      vocbase, query,
+      {
+        arangodb::aql::OptimizerRule::handleArangoSearchViewsRule,
+      }
+    ));
+
+    std::vector<arangodb::velocypack::Slice> expectedDocs {
+      arangodb::velocypack::Slice(insertedDocsView[4].vpack()),
+      arangodb::velocypack::Slice(insertedDocsView[2].vpack()),
+    };
+
+    auto queryResult = arangodb::tests::executeQuery(vocbase, query);
+    REQUIRE(TRI_ERROR_NO_ERROR == queryResult.code);
+
+    auto result = queryResult.result->slice();
+    CHECK(result.isArray());
+
+    arangodb::velocypack::ArrayIterator resultIt(result);
+    REQUIRE(expectedDocs.size() == resultIt.size());
+
+    // Check documents
+    auto expectedDoc = expectedDocs.begin();
+    for (;resultIt.valid(); resultIt.next(), ++expectedDoc) {
+      auto const actualDoc = resultIt.value();
+      auto const resolved = actualDoc.resolveExternals();
+
+      CHECK((0 == arangodb::basics::VelocyPackHelper::compare(arangodb::velocypack::Slice(*expectedDoc), resolved, true)));
+    }
+    CHECK(expectedDoc == expectedDocs.end());
+  }
+
   // unable to retrieve `d.seq` from self-referenced variable
   // FOR i IN 1..5
   //  FOR d IN  SEARCH d.seq == i SORT customscorer(d, d.seq)
@@ -1574,8 +1551,6 @@ TEST_CASE("IResearchQueryTestJoin", "[iresearch][iresearch-query]") {
     CHECK(expectedDoc == expectedDocs.end());
   }
 
-  // we don't support scorers as a part of any expression (in sort or filter)
-  //
   // FOR i IN 1..5
   //   FOR d IN testView SEARCH d.seq == i
   //     FOR x IN collection_1 FILTER x.seq == d.seq && x.seq == TFIDF(d)
@@ -1583,7 +1558,7 @@ TEST_CASE("IResearchQueryTestJoin", "[iresearch][iresearch-query]") {
     std::string const query =
       "FOR i IN 1..5 "
       "  FOR d IN testView SEARCH d.seq == i "
-      "    FOR x IN collection_1 FILTER x.seq == d.seq && x.seq == TFIDF(d)"
+      "    FOR x IN collection_1 FILTER x.seq == d.seq && x.seq == customscorer(d, i)"
       "RETURN x";
 
     CHECK(arangodb::tests::assertRules(
@@ -1592,21 +1567,37 @@ TEST_CASE("IResearchQueryTestJoin", "[iresearch][iresearch-query]") {
       }
     ));
 
+    std::vector<arangodb::velocypack::Slice> expectedDocs {
+      arangodb::velocypack::Slice(insertedDocsView[2].vpack()),
+      arangodb::velocypack::Slice(insertedDocsView[4].vpack()),
+    };
+
     auto queryResult = arangodb::tests::executeQuery(vocbase, query);
-    REQUIRE(TRI_ERROR_NOT_IMPLEMENTED == queryResult.code);
+    REQUIRE(TRI_ERROR_NO_ERROR == queryResult.code);
+
+    auto result = queryResult.result->slice();
+    CHECK(result.isArray());
+
+    arangodb::velocypack::ArrayIterator resultIt(result);
+    REQUIRE(expectedDocs.size() == resultIt.size());
+
+    // Check documents
+    auto expectedDoc = expectedDocs.begin();
+    for (;resultIt.valid(); resultIt.next(), ++expectedDoc) {
+      auto const actualDoc = resultIt.value();
+      auto const resolved = actualDoc.resolveExternals();
+
+      CHECK((0 == arangodb::basics::VelocyPackHelper::compare(arangodb::velocypack::Slice(*expectedDoc), resolved, true)));
+    }
+    CHECK(expectedDoc == expectedDocs.end());
   }
 
-  // we don't support scorers as a part of any expression (in sort or filter)
-  //
-  // FOR i IN 1..5
-  //   FOR d IN testView SEARCH d.seq == i
-  //     FOR x IN collection_1 FILTER x.seq == d.seq && x.seq == TFIDF(d)
   {
     std::string const query =
       "FOR i IN 1..5 "
       "  FOR d IN testView SEARCH d.seq == i "
       "    FOR x IN collection_1 FILTER x.seq == d.seq "
-      "SORT 1 + TFIDF(d)"
+      "SORT 1 + customscorer(d, i) DESC "
       "RETURN d";
 
     CHECK(arangodb::tests::assertRules(
@@ -1615,41 +1606,36 @@ TEST_CASE("IResearchQueryTestJoin", "[iresearch][iresearch-query]") {
       }
     ));
 
-    auto queryResult = arangodb::tests::executeQuery(vocbase, query);
+    std::vector<arangodb::velocypack::Slice> expectedDocs {
-    REQUIRE(TRI_ERROR_NOT_IMPLEMENTED == queryResult.code);
+      arangodb::velocypack::Slice(insertedDocsView[4].vpack()),
-  }
+      arangodb::velocypack::Slice(insertedDocsView[2].vpack()),
-
+    };
-  // we don't support scorers outside the view node
-  //
-  // FOR d IN (FOR c IN testView SEARCH c.name >= 'E' && c.seq < 10 SORT customscorer(c) DESC LIMIT 3 RETURN c)
-  //     FOR x IN collection_1 FILTER x.seq == d.seq
-  // SORT customscorer(d, x.seq)
-  {
-    std::string const query =
-     "FOR d IN (FOR c IN testView SEARCH c.name >= 'E' && c.seq < 10 SORT customscorer(c) DESC LIMIT 3 RETURN c) "
-     "  FOR x IN collection_1 FILTER x.seq == d.seq "
-     "    SORT customscorer(d, x.seq) "
-     "RETURN x";
-
-    CHECK(arangodb::tests::assertRules(
-      vocbase, query, {
-        arangodb::aql::OptimizerRule::handleArangoSearchViewsRule,
-      }
-    ));
 
     auto queryResult = arangodb::tests::executeQuery(vocbase, query);
-    REQUIRE(TRI_ERROR_NOT_IMPLEMENTED == queryResult.code);
+    REQUIRE(TRI_ERROR_NO_ERROR == queryResult.code);
+
+    auto result = queryResult.result->slice();
+    CHECK(result.isArray());
+
+    arangodb::velocypack::ArrayIterator resultIt(result);
+    REQUIRE(expectedDocs.size() == resultIt.size());
+
+    // Check documents
+    auto expectedDoc = expectedDocs.begin();
+    for (;resultIt.valid(); resultIt.next(), ++expectedDoc) {
+      auto const actualDoc = resultIt.value();
+      auto const resolved = actualDoc.resolveExternals();
+
+      CHECK((0 == arangodb::basics::VelocyPackHelper::compare(arangodb::velocypack::Slice(*expectedDoc), resolved, true)));
+    }
+    CHECK(expectedDoc == expectedDocs.end());
   }
 
-  // multiple sorts (not supported now)
+  // multiple sorts
-  // FOR i IN 1..5
-  //  FOR d IN  SEARCH d.seq == i SORT customscorer(d, i) ASC
-  //    FOR x IN collection_0 FILTER x.seq == d.seq && x.name == d.name
-  // SORT customscorer(d, i) DESC
   {
     std::string const query =
       "FOR i IN 1..5 "
-      "  FOR d IN testView SEARCH d.seq == i SORT tfidf(d, i) ASC "
+      "  FOR d IN testView SEARCH d.seq == i SORT tfidf(d, i > 0) ASC "
       "    FOR x IN collection_1 FILTER x.seq == d.seq && x.name == d.name "
       "SORT customscorer(d, i) DESC RETURN d";
 
@@ -1665,23 +1651,61 @@ TEST_CASE("IResearchQueryTestJoin", "[iresearch][iresearch-query]") {
     };
 
     auto queryResult = arangodb::tests::executeQuery(vocbase, query);
-    REQUIRE(TRI_ERROR_NOT_IMPLEMENTED == queryResult.code);
+    REQUIRE(TRI_ERROR_NO_ERROR == queryResult.code);
 
-//    auto result = queryResult.result->slice();
+    auto result = queryResult.result->slice();
-//    CHECK(result.isArray());
+    CHECK(result.isArray());
-//
+
-//    arangodb::velocypack::ArrayIterator resultIt(result);
+    arangodb::velocypack::ArrayIterator resultIt(result);
-//    REQUIRE(expectedDocs.size() == resultIt.size());
+    REQUIRE(expectedDocs.size() == resultIt.size());
-//
+
-//    // Check documents
+    // Check documents
-//    auto expectedDoc = expectedDocs.begin();
+    auto expectedDoc = expectedDocs.begin();
-//    for (;resultIt.valid(); resultIt.next(), ++expectedDoc) {
+    for (;resultIt.valid(); resultIt.next(), ++expectedDoc) {
-//      auto const actualDoc = resultIt.value();
+      auto const actualDoc = resultIt.value();
-//      auto const resolved = actualDoc.resolveExternals();
+      auto const resolved = actualDoc.resolveExternals();
-//
+
-//      CHECK((0 == arangodb::basics::VelocyPackHelper::compare(arangodb::velocypack::Slice(*expectedDoc), resolved, true)));
+      CHECK((0 == arangodb::basics::VelocyPackHelper::compare(arangodb::velocypack::Slice(*expectedDoc), resolved, true)));
-//    }
+    }
-//    CHECK(expectedDoc == expectedDocs.end());
+    CHECK(expectedDoc == expectedDocs.end());
+  }
+
+  // x.seq is used before being assigned
+  {
+    std::string const query =
+     "FOR d IN testView SEARCH d.name >= 'E' && d.seq < 10 "
+     "  SORT customscorer(d) DESC "
+     "  LIMIT 3 "
+     "  FOR x IN collection_1 FILTER x.seq == d.seq "
+     "    SORT customscorer(d, x.seq) "
+     "RETURN x";
+
+    CHECK(arangodb::tests::assertRules(
+      vocbase, query, {
+        arangodb::aql::OptimizerRule::handleArangoSearchViewsRule,
+      }
+    ));
+
+    auto queryResult = arangodb::tests::executeQuery(vocbase, query);
+    REQUIRE(TRI_ERROR_BAD_PARAMETER == queryResult.code);
+  }
+
+  // x.seq is used before being assigned
+  {
+    std::string const query =
+     "FOR d IN (FOR c IN testView SEARCH c.name >= 'E' && c.seq < 10 SORT customscorer(c) DESC LIMIT 3 RETURN c) "
+     "  FOR x IN collection_1 FILTER x.seq == d.seq "
+     "    SORT customscorer(d, x.seq) "
+     "RETURN x";
+
+    CHECK(arangodb::tests::assertRules(
+      vocbase, query, {
+        arangodb::aql::OptimizerRule::handleArangoSearchViewsRule,
+      }
+    ));
+
+    auto queryResult = arangodb::tests::executeQuery(vocbase, query);
+    REQUIRE(TRI_ERROR_BAD_PARAMETER == queryResult.code);
   }
 }
 

tests/IResearch/IResearchViewNode-test.cpp

@@ -200,7 +200,7 @@ SECTION("construct") {
     CHECK(query.plan() == node.plan());
     CHECK(42 == node.id());
     CHECK(logicalView == node.view());
-    CHECK(node.sortCondition().empty());
+    CHECK(node.scorers().empty());
     CHECK(!node.volatility().first); // filter volatility
     CHECK(!node.volatility().second); // sort volatility
     CHECK(node.getVariablesUsedHere().empty());
@@ -245,7 +245,7 @@ SECTION("construct") {
     CHECK(query.plan() == node.plan());
     CHECK(42 == node.id());
     CHECK(logicalView == node.view());
-    CHECK(node.sortCondition().empty());
+    CHECK(node.scorers().empty());
     CHECK(!node.volatility().first); // filter volatility
     CHECK(!node.volatility().second); // sort volatility
     CHECK(node.getVariablesUsedHere().empty());
@@ -316,7 +316,7 @@ SECTION("construct") {
     CHECK(query.plan() == node.plan());
     CHECK(42 == node.id());
     CHECK(logicalView == node.view());
-    CHECK(node.sortCondition().empty());
+    CHECK(node.scorers().empty());
     CHECK(!node.volatility().first); // filter volatility
     CHECK(!node.volatility().second); // sort volatility
     CHECK(node.getVariablesUsedHere().empty());
@@ -418,7 +418,7 @@ SECTION("constructFromVPackSingleServer") {
     CHECK(query.plan() == node.plan());
     CHECK(42 == node.id());
     CHECK(logicalView == node.view());
-    CHECK(node.sortCondition().empty());
+    CHECK(node.scorers().empty());
     CHECK(!node.volatility().first); // filter volatility
     CHECK(!node.volatility().second); // sort volatility
     CHECK(node.getVariablesUsedHere().empty());
@@ -453,7 +453,7 @@ SECTION("constructFromVPackSingleServer") {
     CHECK(query.plan() == node.plan());
     CHECK(42 == node.id());
     CHECK(logicalView == node.view());
-    CHECK(node.sortCondition().empty());
+    CHECK(node.scorers().empty());
     CHECK(!node.volatility().first); // filter volatility
     CHECK(!node.volatility().second); // sort volatility
     CHECK(node.getVariablesUsedHere().empty());
@@ -488,7 +488,7 @@ SECTION("constructFromVPackSingleServer") {
     CHECK(query.plan() == node.plan());
     CHECK(42 == node.id());
     CHECK(logicalView == node.view());
-    CHECK(node.sortCondition().empty());
+    CHECK(node.scorers().empty());
     CHECK(!node.volatility().first); // filter volatility
     CHECK(!node.volatility().second); // sort volatility
     CHECK(node.getVariablesUsedHere().empty());
@@ -553,7 +553,7 @@ SECTION("clone") {
       CHECK(&node.vocbase() == &cloned.vocbase());
       CHECK(node.view() == cloned.view());
       CHECK(&node.filterCondition() == &cloned.filterCondition());
-      CHECK(node.sortCondition() == cloned.sortCondition());
+      CHECK(node.scorers() == cloned.scorers());
       CHECK(node.volatility() == cloned.volatility());
 
       CHECK(node.getCost() == cloned.getCost());
@@ -581,7 +581,7 @@ SECTION("clone") {
       CHECK(&node.vocbase() == &cloned.vocbase());
       CHECK(node.view() == cloned.view());
       CHECK(&node.filterCondition() == &cloned.filterCondition());
-      CHECK(node.sortCondition() == cloned.sortCondition());
+      CHECK(node.scorers() == cloned.scorers());
       CHECK(node.volatility() == cloned.volatility());
       CHECK(node.options().forceSync == cloned.options().forceSync);
 
@@ -609,7 +609,7 @@ SECTION("clone") {
       CHECK(&node.vocbase() == &cloned.vocbase());
       CHECK(node.view() == cloned.view());
       CHECK(&node.filterCondition() == &cloned.filterCondition());
-      CHECK(node.sortCondition() == cloned.sortCondition());
+      CHECK(node.scorers() == cloned.scorers());
       CHECK(node.volatility() == cloned.volatility());
       CHECK(node.options().forceSync == cloned.options().forceSync);
 
@@ -658,7 +658,7 @@ SECTION("clone") {
       CHECK(&node.vocbase() == &cloned.vocbase());
       CHECK(node.view() == cloned.view());
       CHECK(&node.filterCondition() == &cloned.filterCondition());
-      CHECK(node.sortCondition() == cloned.sortCondition());
+      CHECK(node.scorers() == cloned.scorers());
       CHECK(node.volatility() == cloned.volatility());
 
       CHECK(node.getCost() == cloned.getCost());
@@ -686,7 +686,7 @@ SECTION("clone") {
       CHECK(&node.vocbase() == &cloned.vocbase());
       CHECK(node.view() == cloned.view());
       CHECK(&node.filterCondition() == &cloned.filterCondition());
-      CHECK(node.sortCondition() == cloned.sortCondition());
+      CHECK(node.scorers() == cloned.scorers());
       CHECK(node.volatility() == cloned.volatility());
       CHECK(node.options().forceSync == cloned.options().forceSync);
 
@@ -714,7 +714,7 @@ SECTION("clone") {
       CHECK(&node.vocbase() == &cloned.vocbase());
       CHECK(node.view() == cloned.view());
       CHECK(&node.filterCondition() == &cloned.filterCondition());
-      CHECK(node.sortCondition() == cloned.sortCondition());
+      CHECK(node.scorers() == cloned.scorers());
       CHECK(node.volatility() == cloned.volatility());
       CHECK(node.options().forceSync == cloned.options().forceSync);
 
@@ -758,7 +758,7 @@ SECTION("clone") {
       CHECK(&node.vocbase() == &cloned.vocbase());
       CHECK(node.view() == cloned.view());
       CHECK(&node.filterCondition() == &cloned.filterCondition());
-      CHECK(node.sortCondition() == cloned.sortCondition());
+      CHECK(node.scorers() == cloned.scorers());
       CHECK(node.volatility() == cloned.volatility());
       CHECK(node.options().forceSync == cloned.options().forceSync);
 
@@ -790,7 +790,7 @@ SECTION("clone") {
       CHECK(&node.vocbase() == &cloned.vocbase());
       CHECK(node.view() == cloned.view());
       CHECK(&node.filterCondition() == &cloned.filterCondition());
-      CHECK(node.sortCondition() == cloned.sortCondition());
+      CHECK(node.scorers() == cloned.scorers());
       CHECK(node.volatility() == cloned.volatility());
       CHECK(node.options().forceSync == cloned.options().forceSync);
 
@@ -821,7 +821,7 @@ SECTION("clone") {
       CHECK(&node.vocbase() == &cloned.vocbase());
       CHECK(node.view() == cloned.view());
       CHECK(&node.filterCondition() == &cloned.filterCondition());
-      CHECK(node.sortCondition() == cloned.sortCondition());
+      CHECK(node.scorers() == cloned.scorers());
       CHECK(node.volatility() == cloned.volatility());
       CHECK(node.options().forceSync == cloned.options().forceSync);
 
@@ -892,7 +892,7 @@ SECTION("serialize") {
       CHECK(&node.vocbase() == &deserialized.vocbase());
       CHECK(node.view() == deserialized.view());
       CHECK(&node.filterCondition() == &deserialized.filterCondition());
-      CHECK(node.sortCondition() == deserialized.sortCondition());
+      CHECK(node.scorers() == deserialized.scorers());
       CHECK(node.volatility() == deserialized.volatility());
       CHECK(node.options().forceSync == deserialized.options().forceSync);
 
@@ -916,7 +916,7 @@ SECTION("serialize") {
       CHECK(&node.vocbase() == &deserialized.vocbase());
       CHECK(node.view() == deserialized.view());
       CHECK(&node.filterCondition() == &deserialized.filterCondition());
-      CHECK(node.sortCondition() == deserialized.sortCondition());
+      CHECK(node.scorers() == deserialized.scorers());
       CHECK(node.volatility() == deserialized.volatility());
       CHECK(node.options().forceSync == deserialized.options().forceSync);
 
@@ -980,7 +980,7 @@ SECTION("serialize") {
       CHECK(&node.vocbase() == &deserialized.vocbase());
       CHECK(node.view() == deserialized.view());
       CHECK(&node.filterCondition() == &deserialized.filterCondition());
-      CHECK(node.sortCondition() == deserialized.sortCondition());
+      CHECK(node.scorers() == deserialized.scorers());
       CHECK(node.volatility() == deserialized.volatility());
       CHECK(node.options().forceSync == deserialized.options().forceSync);
 
@@ -1004,7 +1004,7 @@ SECTION("serialize") {
       CHECK(&node.vocbase() == &deserialized.vocbase());
       CHECK(node.view() == deserialized.view());
       CHECK(&node.filterCondition() == &deserialized.filterCondition());
-      CHECK(node.sortCondition() == deserialized.sortCondition());
+      CHECK(node.scorers() == deserialized.scorers());
       CHECK(node.volatility() == deserialized.volatility());
       CHECK(node.options().forceSync == deserialized.options().forceSync);
 

tests/IResearch/common.cpp

@@ -125,6 +125,101 @@ struct BoostScorer : public irs::sort {
 
 REGISTER_SCORER_JSON(BoostScorer, BoostScorer::make);
 
+struct CustomScorer : public irs::sort {
+  struct Prepared: public irs::sort::prepared_base<float_t> {
+   public:
+    DECLARE_FACTORY(Prepared);
+
+    Prepared(float_t i)
+      : i(i) {
+    }
+
+    virtual void add(irs::byte_type* dst, const irs::byte_type* src) const override {
+      score_cast(dst) += score_cast(src);
+    }
+
+    virtual irs::flags const& features() const override {
+      return irs::flags::empty_instance();
+    }
+
+    virtual bool less(const irs::byte_type* lhs, const irs::byte_type* rhs) const override {
+      return score_cast(lhs) < score_cast(rhs);
+    }
+
+    virtual irs::sort::collector::ptr prepare_collector() const override {
+      return nullptr;
+    }
+
+    virtual void prepare_score(irs::byte_type* score) const override {
+      score_cast(score) = 0.f;
+    }
+
+    virtual irs::sort::scorer::ptr prepare_scorer(
+      irs::sub_reader const&,
+      irs::term_reader const&,
+      irs::attribute_store const&,
+      irs::attribute_view const&
+    ) const override {
+      struct Scorer : public irs::sort::scorer {
+        Scorer(float_t score): i(score) { }
+
+        virtual void score(irs::byte_type* score_buf) override {
+          *reinterpret_cast<score_t*>(score_buf) = i;
+        }
+
+        float_t i;
+      };
+
+      return irs::sort::scorer::make<Scorer>(i);
+    }
+
+    float_t i;
+  };
+
+  static ::iresearch::sort::type_id const& type() {
+    static ::iresearch::sort::type_id TYPE("customscorer");
+    return TYPE;
+  }
+
+  static irs::sort::ptr make(irs::string_ref const& args) {
+    if (args.null()) {
+      return std::make_shared<CustomScorer>(0.f);
+    }
+
+    // velocypack::Parser::fromJson(...) will throw exception on parse error
+    auto json = arangodb::velocypack::Parser::fromJson(args.c_str(), args.size());
+    auto slice = json ? json->slice() : arangodb::velocypack::Slice();
+
+    if (!slice.isArray()) {
+      return nullptr; // incorrect argument format
+    }
+
+    arangodb::velocypack::ArrayIterator itr(slice);
+
+    if (!itr.valid()) {
+      return nullptr;
+    }
+
+    auto const value = itr.value();
+
+    if (!value.isNumber()) {
+      return nullptr;
+    }
+
+    return std::make_shared<CustomScorer>(itr.value().getNumber<size_t>());
+  }
+
+  CustomScorer(size_t i) : irs::sort(CustomScorer::type()), i(i) {}
+
+  virtual irs::sort::prepared::ptr prepare() const override {
+    return irs::memory::make_unique<Prepared>(static_cast<float_t>(i));
+  }
+
+  size_t i;
+}; // CustomScorer
+
+REGISTER_SCORER_JSON(CustomScorer, CustomScorer::make);
+
 }
 
 namespace arangodb {

tests/js/server/aql/aql-optimizer-rule-inline-subqueries.js

@@ -280,7 +280,7 @@ function optimizerRuleViewTestSuite () {
       assertEqual("v", viewNode.condition.subNodes[0].subNodes[0].subNodes[0].subNodes[0].name);
       assertEqual("reference", viewNode.condition.subNodes[0].subNodes[0].subNodes[1].type);
       assertEqual("outer", viewNode.condition.subNodes[0].subNodes[0].subNodes[1].name);
-      assertEqual([], viewNode.sortCondition);
+      assertEqual([], viewNode.scorers);
     },
     
     testNoVariableReplacementInSearchCondition : function () {
@@ -305,7 +305,7 @@ function optimizerRuleViewTestSuite () {
       assertEqual("reference",        viewNode.condition.subNodes[0].subNodes[0].subNodes[0].subNodes[0].type);
       assertEqual("v", viewNode.condition.subNodes[0].subNodes[0].subNodes[0].subNodes[0].name);
       assertEqual("value", viewNode.condition.subNodes[0].subNodes[0].subNodes[1].type);
-      assertEqual([], viewNode.sortCondition);
+      assertEqual([], viewNode.scorers);
     },
 
   };