Merge branch 'devel' of https://github.com/arangodb/arangodb into devel

Documentation/Books/AQL/ExecutionAndPerformance/Optimizer.mdpp

@@ -259,7 +259,6 @@ The following attributes will be returned in the `stats` attribute of an `explai
   indicate a plan was actually modified by a rule)
 - `rulesSkipped`: number of rules skipped by the optimizer
 
-
 !SUBSECTION Warnings
 
 For some queries, the optimizer may produce warnings. These will be returned in
@@ -277,19 +276,26 @@ the `warnings` attribute of the `explain` result:
 There is an upper bound on the number of warning a query may produce. If that
 bound is reached, no further warnings will be returned.
 
+!SUBSECTION Optimization in a cluster
 
-!SUBSECTION Clustering
 When you're running AQL in the cluster, the parsing of the query is done on the
-Coordinator. The coordinator then chops the query into parts, which are to
+coordinator. The coordinator then chops the query into snipets, which are to
 remain on the coordinator, and others that are to be distributed over the network
 to the shards. The cutting sites are interconnected
 via *Scatter-*, *Gather-* and *RemoteNodes*.
-These nodes mark the network borders. The optimizer strives to reduce the amount
+
+These nodes mark the network borders of the snippets. The optimizer strives to reduce the amount
 of data transfered via these network interfaces by pushing `FILTER`s out to the shards,
-as its vital to the query performance to reduce that data amount to transfer over the
+as it is vital to the query performance to reduce that data amount to transfer over the
-network links. As usual the optimizer can only take certain assumptions for granted
+network links.
-when doing so, I.e. [User defined functions have to be executed on the coordinator](../Extending/README.md).
+
-In doubt, you should modify your query to reduce the number of round trips.
+Snippets marked with **DBS** are executed on the shards, **COOR** ones are excuted on the coordinator.
+
+**As usual, the optimizer can only take certain assumptions for granted when doing so,
+i.e. [user-defined functions have to be executed on the coordinator](../Extending/README.md).
+If in doubt, you should modify your query to reduce the number interconnections between your snippets.**
+
+When optimizing your query you may want to look at simpler parts of it first.
 
 !SUBSECTION List of execution nodes
 

Documentation/Books/AQL/Extending/README.mdpp

@@ -1,42 +1,62 @@
 !CHAPTER Extending AQL with User Functions
 
-AQL comes with a built-in set of functions, but it is not a
+AQL comes with a [built-in set of functions](../Functions/README.md), but it is
-fully-featured programming language.
+not a fully-featured programming language.
 
-To add missing functionality or to simplify queries, users
+To add missing functionality or to simplify queries, users may add their own
-may add their own functions to AQL in the selected database. 
+functions to AQL in the selected database. These functions are written in
-These functions can be written in JavaScript, and have to be 
+JavaScript, and are deployed via an API; see [Registering Functions](Functions.md).
-registered via the API; see [Registering Functions](Functions.md).
 
-In order to avoid conflicts with existing or future built-in 
+In order to avoid conflicts with existing or future built-in function names,
-function names, all user functions have to be put into separate
+all user defined functions (**UDF**) have to be put into separate namespaces.
-namespaces. Invoking a user function is then possible by referring
+Invoking a UDF is then possible by referring to the fully-qualified function name,
-to the fully-qualified function name, which includes the namespace,
+which includes the namespace, too; see [Conventions](Conventions.md).
-too; see [Conventions](Conventions.md). 
 
 !SECTION Technical Details
 
-Internally, user-defined functions (UDF) are stored in a system collection named
+!SUBSECTION Known Limitations
-*_aqlfunctions* of the selected database. When an AQL statement refers to such a function,
+
-it is loaded from that collection. The functions will be exclusively
+**UDFs have some implications you should be aware of. Otherwise they can
+introduce serious effects on the performance of your queries and the resource
+usage in ArangoDB.**
+
+Since the optimizer doesn't know anything about the nature of your function,
+**the optimizer can't use indices for UDFs**. So you should never lean on a UDF
+as the primary criterion for a `FILTER` statement to reduce your query result set.
+Instead, put a another `FILTER` statement in front of it. You should make sure
+that this [**`FILTER` statement** is effective](../ExecutionAndPerformance/Optimizer.md)
+to reduce the query result before passing it to your UDF.
+
+Rule of thumb is, the closer the UDF is to your final `RETURN` statement
+(or maybe even inside it), the better. 
+
+When used in clusters, UDFs are always executed on the
+[coordinator](../../Manual/Scalability/Architecture.html).
+
+Using UDFs in clusters may result in a higher resource allocation
+in terms of used V8 contexts and server threads. If you run out 
+of these resources, your query may abort with a
+[**cluster backend unavailable**](../../Manual/Appendix/ErrorCodes.html) error.
+
+To overcome these mentioned limitations, you may want to
+[increase the number of available V8 contexts](../../Manual/Administration/Configuration/Arangod.html#v8-contexts)
+(at the expense of increased memory usage),
+and [the number of available server threads](../../Manual/Administration/Configuration/Arangod.html#server-threads).
+
+!SUBSECTION Deployment Details
+
+Internally, UDFs are stored in a system collection named `_aqlfunctions`
+of the selected database. When an AQL statement refers to such a UDF,
+it is loaded from that collection. The UDFs will be exclusively
 available for queries in that particular database.
 
-When used in clusters, the UDF is executed on the coordinator. Depending on your
+Since the coordinator doesn't have own local collections, the `_aqlfunctions`
-query layout, this may result in many documents having to be passed up from the
-DB-Servers to the Coordinator. To avoid this,
-[you should make sure that the query contains effective `FILTER` statements](../ExecutionAndPerformance/Optimizer.md)
-that can be used on the DB-Server side to reduce the query result
-before passing it up to the Coordinator and your UDF.
-
-Since the Coordinator doesn't have own local collections, the `_aqlfunctions`
 collection is sharded across the cluster. Therefore (as usual), it has to be
-accessed through the coordinator - you mustn't talk to the DB-Servers directly.
+accessed through a coordinator - you mustn't talk to the shards directly.
-Once it is in there, it will be available on all coordinators.
+Once it is in the `_aqlfunctions` collection, it is available on all
-
+coordinators without additional effort.
-Since the optimizer doesn't know anything about this function, it won't be able
-to use indices for user defined functions.
 
 Keep in mind that system collections are excluded from dumps created with
 [arangodump](../../Manual/Administration/Arangodump.html) by default.
-To include AQL user functions in a dump, the dump needs to be started with
+To include AQL UDF in a dump, the dump needs to be started with
 the option *--include-system-collections true*.

Documentation/Books/AQL/Functions/Document.mdpp

@@ -81,6 +81,17 @@ HAS( { name: null }, "name" ) // true
 HAS( { }, "name" )            // false
 ```
 
+Note that `HAS()` can not utilize indexes. If it's not necessary to distinguish
+between explicit and implicit *null* values in your query, you may use an equality
+comparison to test for *null* and create a non-sparse index on the attribute you
+want to test against:
+
+```js
+FILTER !HAS(doc, "name")    // can not use indexes
+FILTER IS_NULL(doc, "name") // can not use indexes
+FILTER doc.name == null     // can utilize non-sparse indexes
+```
+
 !SUBSECTION IS_SAME_COLLECTION()
 
 `IS_SAME_COLLECTION(collectionName, documentHandle) → bool`

Documentation/Books/AQL/Functions/README.mdpp

@@ -28,5 +28,5 @@ i.e. *LENGTH(foo)* and *length(foo)* are equivalent.
 !SUBSECTION Extending AQL
  
 It is possible to extend AQL with user-defined functions. These functions need to
-be written in JavaScript, and have to be registered before usaing them in a query.
+be written in JavaScript, and have to be registered before they can be used in a query.
 Please refer to [Extending AQL](../Extending/index.html) for more details.

Documentation/Books/AQL/Functions/TypeCast.mdpp

@@ -143,7 +143,8 @@ checked for, and false otherwise.
 
 The following type check functions are available:
 
-- `IS_NULL(value) → bool`: Check whether *value* is a *null* value
+- `IS_NULL(value) → bool`: Check whether *value* is a *null* value, also see
+  [HAS()](Document.md#has)
 
 - `IS_BOOL(value) → bool`: Check whether *value* is a *boolean* value
 

Documentation/Books/HTTP/AqlUserFunctions/README.mdpp

@@ -6,8 +6,8 @@ This is an introduction to ArangoDB's HTTP interface for managing AQL
 user functions. AQL user functions are a means to extend the functionality
 of ArangoDB's query language (AQL) with user-defined JavaScript code.
  
-For an overview of how AQL user functions work, please refer to
+For an overview of how AQL user functions and their implications, please refer to
-[Extending AQL](../../AQL/Extending/index.html).
+the [Extending AQL](../../AQL/Extending/index.html) chapter.
 
 The HTTP interface provides an API for adding, deleting, and listing
 previously registered AQL user functions.

Documentation/Books/Manual/ReleaseNotes/UpgradingChanges30.mdpp

@@ -122,12 +122,12 @@ are missing from the replacement document, an `REPLACE` operation will fail.
 
 !SUBSUBSECTION Graph functions
 
-In version 3.0 all former graph related functions have been removed from AQL to be replaced
+In version 3.0 all former graph related functions have been removed from AQL to
-by native AQL constructs.
+be replaced by [native AQL constructs](../../AQL/Graphs/index.html).
 These constructs allow for more fine-grained filtering on several graph levels.
 Also this allows the AQL optimizer to automatically improve these queries by
 enhancing them with appropriate indexes.
-We have create recipes to upgrade from 2.8 to 3.0 when using these functions.
+We have created recipes to upgrade from 2.8 to 3.0 when using these functions.
 
 The functions:
 
@@ -161,7 +161,7 @@ are covered in [Migrating GRAPH_* Measurements from 2.8 or earlier to 3.0](https
 * TRAVERSAL
 * TRAVERSAL_TREE
 
-are covered in  [#Migrating anonymous graph Functions from 2.8 or earlier to 3.0](https://docs.arangodb.com/cookbook/AQL/MigratingEdgeFunctionsTo3.html)
+are covered in [Migrating anonymous graph functions from 2.8 or earlier to 3.0](https://docs.arangodb.com/3/cookbook/AQL/MigratingEdgeFunctionsTo3.html)
 
 !SUBSECTION Typecasting functions
 

lib/V8/v8-utils.cpp

@@ -61,16 +61,18 @@
 #include "V8/v8-globals.h"
 #include "V8/v8-vpack.h"
 
+#include <velocypack/Builder.h>
+#include <velocypack/Slice.h>
+#include <velocypack/Validator.h>
+#include <velocypack/velocypack-aliases.h>
+
 using namespace arangodb;
 using namespace arangodb::application_features;
 using namespace arangodb::basics;
 using namespace arangodb::httpclient;
 using namespace arangodb::rest;
 
-////////////////////////////////////////////////////////////////////////////////
 /// @brief Random string generators
-////////////////////////////////////////////////////////////////////////////////
-
 namespace {
 static UniformCharacter JSAlphaNumGenerator(
     "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789");
@@ -80,10 +82,7 @@ static UniformCharacter JSSaltGenerator(
     "[]:;<>,.?/|");
 }
 
-////////////////////////////////////////////////////////////////////////////////
 /// @brief Converts an object to a UTF-8-encoded and normalized character array.
-////////////////////////////////////////////////////////////////////////////////
-
 TRI_Utf8ValueNFC::TRI_Utf8ValueNFC(TRI_memory_zone_t* memoryZone,
                                    v8::Handle<v8::Value> const obj)
     : _str(nullptr), _length(0), _memoryZone(memoryZone) {
@@ -3667,6 +3666,73 @@ static void JS_IsStopping(v8::FunctionCallbackInfo<v8::Value> const& args) {
   TRI_V8_TRY_CATCH_END
 }
 
+/// @brief convert a V8 value to VPack
+static void JS_V8ToVPack(v8::FunctionCallbackInfo<v8::Value> const& args) {
+  TRI_V8_TRY_CATCH_BEGIN(isolate);
+  v8::HandleScope scope(isolate);
+
+  // extract the argument
+  if (args.Length() != 1) {
+    TRI_V8_THROW_EXCEPTION_USAGE("V8_TO_VPACK(value)");
+  }
+
+  VPackBuilder builder;
+  int res = TRI_V8ToVPack(isolate, builder, args[0], false);
+
+  if (res != TRI_ERROR_NO_ERROR) {
+    TRI_V8_THROW_EXCEPTION(res);
+  }
+
+  VPackSlice slice = builder.slice();
+              
+  V8Buffer* buffer = V8Buffer::New(isolate, slice.startAs<char const>(), slice.byteSize());
+  v8::Local<v8::Object> bufferObject = v8::Local<v8::Object>::New(isolate, buffer->_handle);
+  TRI_V8_RETURN(bufferObject);
+  
+  TRI_V8_RETURN_FALSE();
+  TRI_V8_TRY_CATCH_END
+}
+
+/// @brief convert a VPack value to V8
+static void JS_VPackToV8(v8::FunctionCallbackInfo<v8::Value> const& args) {
+  TRI_V8_TRY_CATCH_BEGIN(isolate);
+  v8::HandleScope scope(isolate);
+
+  // extract the argument
+  if (args.Length() != 1) {
+    TRI_V8_THROW_EXCEPTION_USAGE("VPACK_TO_V8(value)");
+  }
+
+  if (args[0]->IsString() || args[0]->IsStringObject()) {
+    // supplied argument is a string
+    std::string const value = TRI_ObjectToString(args[0]);
+    
+    VPackValidator validator;
+    validator.validate(value.c_str(), value.size(), false); 
+
+    VPackSlice slice(value.c_str());
+    v8::Handle<v8::Value> result = TRI_VPackToV8(isolate, slice);
+    TRI_V8_RETURN(result);
+  } else if (args[0]->IsObject() && V8Buffer::hasInstance(isolate, args[0])) {
+    // argument is a buffer
+    char const* data = V8Buffer::data(args[0].As<v8::Object>());
+    size_t size = V8Buffer::length(args[0].As<v8::Object>());
+
+    VPackValidator validator;
+    validator.validate(data, size, false); 
+
+    VPackSlice slice(data);
+    v8::Handle<v8::Value> result = TRI_VPackToV8(isolate, slice);
+
+    TRI_V8_RETURN(result);
+  } else {
+    TRI_V8_THROW_EXCEPTION_MESSAGE(TRI_ERROR_BAD_PARAMETER, "invalid argument type for VPACK_TO_V8()");
+  }
+  
+  TRI_V8_RETURN_FALSE();
+  TRI_V8_TRY_CATCH_END
+}
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief ArangoError
 ////////////////////////////////////////////////////////////////////////////////
@@ -4434,6 +4500,12 @@ void TRI_InitV8Utils(v8::Isolate* isolate, v8::Handle<v8::Context> context,
   TRI_AddGlobalFunctionVocbase(
       isolate, context, TRI_V8_ASCII_STRING("SYS_IS_STOPPING"), JS_IsStopping);
   
+  TRI_AddGlobalFunctionVocbase(
+      isolate, context, TRI_V8_ASCII_STRING("V8_TO_VPACK"), JS_V8ToVPack);
+  
+  TRI_AddGlobalFunctionVocbase(
+      isolate, context, TRI_V8_ASCII_STRING("VPACK_TO_V8"), JS_VPackToV8);
+
   // .............................................................................
   // create the global variables
   // .............................................................................