Merge branch 'devel' of github.com:arangodb/arangodb into devel

Documentation/DocuBlocks/Rest/Documents/get_read_multiple_documents.md

@@ -0,0 +1,78 @@
+
+@startDocuBlock get_read_document_MULTI
+@brief reads a single document
+
+@RESTHEADER{PUT /_api/document/{collection-name},Read multiple documents, readDocument}
+
+@RESTURLPARAMETERS
+
+@RESTURLPARAM{collection-name,string,required}
+The *collection* from which the documents are to be read.
+
+@RESTQUERYPARAMETERS
+
+@RESTQUERYPARAM{onlyget,boolean,required}
+This parameter is required to be **true**, otherwise a replace
+operation is executed
+
+@RESTQUERYPARAM{ignoreRevs,string,optional}
+Should the value be *true* (the default):
+If a search document contains a value for the *_rev* field,
+then the document is only returned if it has the same revision value. 
+Otherwise a precondition failed error is returned.
+
+@RESTDESCRIPTION
+Returns the documents identified by their *_key* in the body objects. 
+The body of the request _must_ contain a JSON array of either 
+strings (the *_key* values to lookup) or search documents.
+
+A search document _must_ contain at least a value for the *_key* field. 
+A value for `_rev` _may_ be specified to verify whether the document
+has the same revision value, unless _ignoreRevs_ is set to false. 
+
+Cluster only: The search document _may_ contain  
+values for the collection's pre-defined shard keys. Values for the shard keys 
+are treated as hints to improve performance. Should the shard keys
+values be incorrect ArangoDB may answer with a *not found* error.
+
+The returned array of documents contain three special attributes: *_id* containing the document
+handle, *_key* containing key which uniquely identifies a document
+in a given collection and *_rev* containing the revision.
+
+@RESTRETURNCODES
+
+@RESTRETURNCODE{200}
+is returned if no error happened
+
+@RESTRETURNCODE{400}
+is returned if the body does not contain a valid JSON representation
+of an array of documents. The response body contains
+an error document in this case.
+
+@RESTRETURNCODE{404}
+is returned if the collection was not found.
+
+@EXAMPLES
+
+Reading multiple documents handle:
+
+@EXAMPLE_ARANGOSH_RUN{RestDocumentHandlerReadMultiDocument}
+    var cn = "products";
+    db._drop(cn);
+    db._create(cn);
+
+    db.products.save({"_key":"doc1", "hello":"world"});
+    db.products.save({"_key":"doc2", "say":"hi to mom"});
+    var url = "/_api/document/products?onlyget=true";
+    var body = '["doc1", {"_key":"doc2"}]';
+
+    var response = logCurlRequest('PUT', url, body);
+
+    assert(response.code === 200);
+
+    logJsonResponse(response);
+  ~ db._drop(cn);
+@END_EXAMPLE_ARANGOSH_RUN
+
+@endDocuBlock
+

Documentation/DocuBlocks/Rest/Documents/patch_update_document.md

@@ -63,7 +63,10 @@ The body of the request must contain a JSON document with the
 attributes to patch (the patch document). All attributes from the
 patch document will be added to the existing document if they do not
 yet exist, and overwritten in the existing document if they do exist
-there.
+there. 
+
+The value of the `_key` attribute as well as attributes
+used as sharding keys may not be changed.
 
 Setting an attribute value to *null* in the patch document will cause a
 value of *null* to be saved for the attribute by default.
@@ -85,6 +88,11 @@ the *Etag* header field contains the new revision of the document
 (in double quotes) and the *Location* header contains a complete URL
 under which the document can be queried.
 
+Cluster only: The patch document _may_ contain  
+values for the collection's pre-defined shard keys. Values for the shard keys 
+are treated as hints to improve performance. Should the shard keys
+values be incorrect ArangoDB may answer with a *not found* error
+
 Optionally, the query parameter *waitForSync* can be used to force
 synchronization of the updated document operation to disk even in case
 that the *waitForSync* flag had been disabled for the entire collection.

Documentation/DocuBlocks/Rest/Documents/patch_update_multiple_documents.md

@@ -56,6 +56,9 @@ patch documents will be added to the existing documents if they do
 not yet exist, and overwritten in the existing documents if they do
 exist there.
 
+The value of the `_key` attribute as well as attributes
+used as sharding keys may not be changed.
+
 Setting an attribute value to *null* in the patch documents will cause a
 value of *null* to be saved for the attribute by default.
 
@@ -64,8 +67,10 @@ document in the body and its value does not match the revision of
 the corresponding document in the database, the precondition is
 violated.
 
-If the document exists and can be updated, then an *HTTP 201* or
+Cluster only: The patch document _may_ contain  
-an *HTTP 202* is returned (depending on *waitForSync*, see below).
+values for the collection's pre-defined shard keys. Values for the shard keys 
+are treated as hints to improve performance. Should the shard keys
+values be incorrect ArangoDB may answer with a *not found* error
 
 Optionally, the query parameter *waitForSync* can be used to force
 synchronization of the document replacement operation to disk even in case
@@ -107,12 +112,10 @@ cases the error 1200 "revision conflict" and in 10 cases the error
 @RESTRETURNCODES
 
 @RESTRETURNCODE{201}
-is returned if the documents were updated successfully and
+is returned if *waitForSync* was *true* and operations were processed.
-*waitForSync* was *true*.
 
 @RESTRETURNCODE{202}
-is returned if the documents were updated successfully and
+is returned if *waitForSync* was *false* and operations were processed.
-*waitForSync* was *false*.
 
 @RESTRETURNCODE{400}
 is returned if the body does not contain a valid JSON representation

Documentation/DocuBlocks/Rest/Documents/post_create_document.md

@@ -9,7 +9,7 @@
 The *collection* in which the collection is to be created.
 
 @RESTALLBODYPARAM{data,json,required}
-A JSON representation of a single document or of an array of documents.
+A JSON representation of a single document.
 
 @RESTQUERYPARAMETERS
 
@@ -45,14 +45,6 @@ Creates a new document from the document given in the body, unless there
 is already a document with the *_key* given. If no *_key* is given, a new
 unique *_key* is generated automatically.
 
-The body can be an array of documents, in which case all
-documents in the array are inserted with the same semantics as for a
-single document. The result body will contain a JSON array of the
-same length as the input array, and each entry contains the result
-of the operation for the corresponding input. In case of an error
-the entry is a document with attributes *error* set to *true* and
-errorCode set to the error code that has happened.
-
 Possibly given *_id* and *_rev* attributes in the body are always ignored,
 the URL part or the query parameter collection respectively counts.
 
@@ -62,14 +54,12 @@ contains the revision of the document. Both are only set in the single
 document case.
 
 If *silent* is not set to *true*, the body of the response contains a 
-JSON object (single document case) with the following attributes:
+JSON object with the following attributes:
 
   - *_id* contains the document handle of the newly created document
   - *_key* contains the document key
   - *_rev* contains the document revision
 
-In the multi case the body is an array of such objects.
-
 If the collection parameter *waitForSync* is *false*, then the call
 returns as soon as the document has been accepted. It will not wait
 until the documents have been synced to disk.
@@ -101,7 +91,7 @@ is returned if the documents were created successfully and
 
 @RESTRETURNCODE{400}
 is returned if the body does not contain a valid JSON representation
-of one document or an array of documents. The response body contains
+of one document. The response body contains
 an error document in this case.
 
 @RESTRETURNCODE{404}
@@ -112,13 +102,7 @@ The response body contains an error document in this case.
 is returned in the single document case if a document with the
 same qualifiers in an indexed attribute conflicts with an already
 existing document and thus violates that unique constraint. The
-response body contains an error document in this case. In the array
+response body contains an error document in this case.
-case only 201 or 202 is returned, but if an error occurred, the
-additional HTTP header *X-Arango-Error-Codes* is set, which
-contains a map of the error codes that occurred together with their
-multiplicities, as in: *1205:10,1210:17* which means that in 10
-cases the error 1205 "illegal document handle" and in 17 cases the
-error 1210 "unique constraint violated" has happened.
 
 @EXAMPLES
 
@@ -213,27 +197,9 @@ Illegal document
     db._drop(cn);
 @END_EXAMPLE_ARANGOSH_RUN
 
-Insert multiple documents:
-
-@EXAMPLE_ARANGOSH_RUN{RestDocumentHandlerPostMulti1}
-    var cn = "products";
-    db._drop(cn);
-    db._create(cn);
-
-    var url = "/_api/document/" + cn;
-    var body = '[{"Hello":"Earth"}, {"Hello":"Venus"}, {"Hello":"Mars"}]';
-
-    var response = logCurlRequest('POST', url, body);
-
-    assert(response.code === 202);
-
-    logJsonResponse(response);
-    db._drop(cn);
-@END_EXAMPLE_ARANGOSH_RUN
-
 Use of returnNew:
 
-@EXAMPLE_ARANGOSH_RUN{RestDocumentHandlerPostMulti2}
+@EXAMPLE_ARANGOSH_RUN{RestDocumentHandlerPostReturnNew}
     var cn = "products";
     db._drop(cn);
     db._create(cn);

Documentation/DocuBlocks/Rest/Documents/post_create_multiple_document.md

@@ -0,0 +1,165 @@
+@startDocuBlock post_create_document_MULTI
+@brief creates multiple documents
+
+@RESTHEADER{POST /_api/document/{collection-name}, Create document, insertDocument}
+
+@RESTURLPARAMETERS
+
+@RESTURLPARAM{collection-name,string,required}
+The *collection* in which the documents are to be created.
+
+@RESTALLBODYPARAM{data,json,required}
+An array of documents to create.
+
+@RESTQUERYPARAMETERS
+
+@RESTQUERYPARAM{collection,string,optional}
+The name of the collection. This is only for backward compatibility.
+In ArangoDB versions < 3.0, the URL path was */_api/document* and
+this query parameter was required. This combination still works, but
+the recommended way is to specify the collection in the URL path.
+
+@RESTQUERYPARAM{waitForSync,boolean,optional}
+Wait until document has been synced to disk.
+
+@RESTQUERYPARAM{returnNew,boolean,optional}
+Additionally return the complete new document under the attribute *new*
+in the result.
+
+@RESTQUERYPARAM{returnOld,boolean,optional}
+Additionally return the complete old document under the attribute *old*
+in the result. Only available if the overwrite option is used.
+
+@RESTQUERYPARAM{silent,boolean,optional}
+If set to *true*, an empty object will be returned as response. No meta-data 
+will be returned for the created document. This option can be used to
+save some network traffic.
+
+@RESTQUERYPARAM{overwrite,boolean,optional}
+If set to *true*, the insert becomes a replace-insert. If a document with the
+same *_key* already exists the new document is not rejected with unique
+constraint violated but will replace the old document.
+
+@RESTDESCRIPTION
+Creates new documents from the documents given in the body, unless there
+is already a document with the *_key* given. If no *_key* is given, a new
+unique *_key* is generated automatically.
+
+The result body will contain a JSON array of the
+same length as the input array, and each entry contains the result
+of the operation for the corresponding input. In case of an error
+the entry is a document with attributes *error* set to *true* and
+errorCode set to the error code that has happened.
+
+Possibly given *_id* and *_rev* attributes in the body are always ignored,
+the URL part or the query parameter collection respectively counts.
+
+If *silent* is not set to *true*, the body of the response contains an 
+array of JSON objects with the following attributes:
+
+  - *_id* contains the document handle of the newly created document
+  - *_key* contains the document key
+  - *_rev* contains the document revision
+
+If the collection parameter *waitForSync* is *false*, then the call
+returns as soon as the documents have been accepted. It will not wait
+until the documents have been synced to disk.
+
+Optionally, the query parameter *waitForSync* can be used to force
+synchronization of the document creation operation to disk even in
+case that the *waitForSync* flag had been disabled for the entire
+collection. Thus, the *waitForSync* query parameter can be used to
+force synchronization of just this specific operations. To use this,
+set the *waitForSync* parameter to *true*. If the *waitForSync*
+parameter is not specified or set to *false*, then the collection's
+default *waitForSync* behavior is applied. The *waitForSync* query
+parameter cannot be used to disable synchronization for collections
+that have a default *waitForSync* value of *true*.
+
+If the query parameter *returnNew* is *true*, then, for each
+generated document, the complete new document is returned under
+the *new* attribute in the result.
+
+Should an error have occurred with some of the documents
+the additional HTTP header *X-Arango-Error-Codes* is set, which
+contains a map of the error codes that occurred together with their
+multiplicities, as in: *1205:10,1210:17* which means that in 10
+cases the error 1205 "illegal document handle" and in 17 cases the
+error 1210 "unique constraint violated" has happened.
+
+@RESTRETURNCODES
+
+@RESTRETURNCODE{201}
+is returned if *waitForSync* was *true* and operations were processed.
+
+@RESTRETURNCODE{202}
+is returned if *waitForSync* was *false* and operations were processed.
+
+@RESTRETURNCODE{400}
+is returned if the body does not contain a valid JSON representation
+of an array of documents. The response body contains
+an error document in this case.
+
+@RESTRETURNCODE{404}
+is returned if the collection specified by *collection* is unknown.
+The response body contains an error document in this case.
+
+@EXAMPLES
+
+Insert multiple documents:
+
+@EXAMPLE_ARANGOSH_RUN{RestDocumentHandlerPostMulti1}
+    var cn = "products";
+    db._drop(cn);
+    db._create(cn);
+
+    var url = "/_api/document/" + cn;
+    var body = '[{"Hello":"Earth"}, {"Hello":"Venus"}, {"Hello":"Mars"}]';
+
+    var response = logCurlRequest('POST', url, body);
+
+    assert(response.code === 202);
+
+    logJsonResponse(response);
+    db._drop(cn);
+@END_EXAMPLE_ARANGOSH_RUN
+
+Use of returnNew:
+
+@EXAMPLE_ARANGOSH_RUN{RestDocumentHandlerPostMulti2}
+    var cn = "products";
+    db._drop(cn);
+    db._create(cn);
+
+    var url = "/_api/document/" + cn + "?returnNew=true";
+    var body = '[{"Hello":"Earth"}, {"Hello":"Venus"}, {"Hello":"Mars"}]';
+
+    var response = logCurlRequest('POST', url, body);
+
+    assert(response.code === 202);
+
+    logJsonResponse(response);
+    db._drop(cn);
+@END_EXAMPLE_ARANGOSH_RUN
+
+
+Partially illegal documents:
+
+@EXAMPLE_ARANGOSH_RUN{RestDocumentHandlerPostBadJsonMulti}
+    var cn = "products";
+    db._drop(cn);
+    db._create(cn);
+
+    var url = "/_api/document/" + cn;
+    var body = '[{ "_key": 111 }, {"_key":"abc"}]';
+
+    var response = logCurlRequest('POST', url, body);
+
+    assert(response.code === 202);
+
+    logJsonResponse(response);
+    db._drop(cn);
+@END_EXAMPLE_ARANGOSH_RUN
+
+@endDocuBlock
+

Documentation/DocuBlocks/Rest/Documents/put_replace_document.md

@@ -48,6 +48,9 @@ Replaces the document with handle <document-handle> with the one in
 the body, provided there is such a document and no precondition is
 violated.
 
+The value of the `_key` attribute as well as attributes
+used as sharding keys may not be changed.
+
 If the *If-Match* header is specified and the revision of the
 document in the database is unequal to the given revision, the
 precondition is violated.
@@ -65,6 +68,11 @@ the *Etag* header field contains the new revision of the document
 and the *Location* header contains a complete URL under which the
 document can be queried.
 
+Cluster only: The replace documents _may_ contain  
+values for the collection's pre-defined shard keys. Values for the shard keys 
+are treated as hints to improve performance. Should the shard keys
+values be incorrect ArangoDB may answer with a *not found* error.
+
 Optionally, the query parameter *waitForSync* can be used to force
 synchronization of the document replacement operation to disk even in case
 that the *waitForSync* flag had been disabled for the entire collection.

Documentation/DocuBlocks/Rest/Documents/put_replace_multiple_documents.md

@@ -38,13 +38,18 @@ Replaces multiple documents in the specified collection with the
 ones in the body, the replaced documents are specified by the *_key*
 attributes in the body documents.
 
+The value of the `_key` attribute as well as attributes
+used as sharding keys may not be changed.
+
 If *ignoreRevs* is *false* and there is a *_rev* attribute in a
 document in the body and its value does not match the revision of
 the corresponding document in the database, the precondition is
 violated.
 
-If the document exists and can be updated, then an *HTTP 201* or
+Cluster only: The replace documents _may_ contain  
-an *HTTP 202* is returned (depending on *waitForSync*, see below).
+values for the collection's pre-defined shard keys. Values for the shard keys 
+are treated as hints to improve performance. Should the shard keys
+values be incorrect ArangoDB may answer with a *not found* error.
 
 Optionally, the query parameter *waitForSync* can be used to force
 synchronization of the document replacement operation to disk even in case
@@ -86,12 +91,10 @@ cases the error 1200 "revision conflict" and in 10 cases the error
 @RESTRETURNCODES
 
 @RESTRETURNCODE{201}
-is returned if the documents were replaced successfully and
+is returned if *waitForSync* was *true* and operations were processed.
-*waitForSync* was *true*.
 
 @RESTRETURNCODE{202}
-is returned if the documents were replaced successfully and
+is returned if *waitForSync* was *false* and operations were processed.
-*waitForSync* was *false*.
 
 @RESTRETURNCODE{400}
 is returned if the body does not contain a valid JSON representation

Documentation/DocuBlocks/Rest/Simple Queries/put_api_simple_all_keys.md

@@ -1,8 +1,8 @@
 
 @startDocuBlock put_read_all_documents
-@brief reads all documents from collection
+@brief reads all document keys from collection
 
-@RESTHEADER{PUT /_api/simple/all-keys, Read all documents, allDocumentKeys}
+@RESTHEADER{PUT /_api/simple/all-keys, Read all document keys, allDocumentKeys}
 
 @HINTS
 {% hint 'warning' %}

arangod/Cluster/ClusterMethods.cpp

@@ -1232,10 +1232,10 @@ Future<OperationResult> createDocumentOnCoordinator(transaction::Methods const&
   }
   
   std::string const baseUrl =
-      "/_db/" + StringUtils::urlEncode(dbname) + "/_api/document?collection=";
+      "/_db/" + StringUtils::urlEncode(dbname) + "/_api/document/";
 
   std::string const optsUrlPart =
-      std::string("&waitForSync=") + (options.waitForSync ? "true" : "false") +
+      std::string("?waitForSync=") + (options.waitForSync ? "true" : "false") +
       "&returnNew=" + (options.returnNew ? "true" : "false") +
       "&returnOld=" + (options.returnOld ? "true" : "false") +
       "&isRestore=" + (options.isRestore ? "true" : "false") + "&" +
@@ -1682,10 +1682,11 @@ Future<OperationResult> getDocumentOnCoordinator(transaction::Methods& trx,
     for (auto const& it : shardMap) {
       network::Headers headers;
       addTransactionHeaderForShard(trx, *shardIds, /*shard*/ it.first, headers);
-      
+      std::string url;
+      VPackBuffer<uint8_t> buffer;
+
       if (!useMultiple) {
         TRI_ASSERT(it.second.size() == 1);
-
         if (!options.ignoreRevs && slice.hasKey(StaticStrings::RevString)) {
           headers.emplace("if-match", slice.get(StaticStrings::RevString).copyString());
         }
@@ -1695,28 +1696,24 @@ Future<OperationResult> getDocumentOnCoordinator(transaction::Methods& trx,
           keySlice = slice.get(StaticStrings::KeyString);
         }
         VPackStringRef ref = keySlice.stringRef();
-        std::string url = baseUrl + StringUtils::urlEncode(it.first) + "/";
-        url.append(StringUtils::urlEncode(ref.data(), ref.length())).append(optsUrlPart);
-
         // We send to single endpoint
-        futures.emplace_back(network::sendRequestRetry("shard:" + it.first, restVerb,
+        url = baseUrl + StringUtils::urlEncode(it.first) + "/";
-                                                       std::move(url), VPackBuffer<uint8_t>(),
+        url.append(StringUtils::urlEncode(ref.data(), ref.length())).append(optsUrlPart);
-                                                       network::Timeout(CL_DEFAULT_TIMEOUT),
-                                                       headers, /*retryNotFound*/ true));
       } else {
-        VPackBuffer<uint8_t> buffer;
+        // We send to Babies endpoint
+        url = baseUrl + StringUtils::urlEncode(it.first) + optsUrlPart;
         VPackBuilder builder(buffer);
         builder.openArray(/*unindexed*/true);
         for (auto const& value : it.second) {
           builder.add(value);
         }
         builder.close();
-        // We send to Babies endpoint
-        futures.emplace_back(network::sendRequestRetry("shard:" + it.first, restVerb,
-                                                       baseUrl + StringUtils::urlEncode(it.first) + optsUrlPart,
-                                                       std::move(buffer), network::Timeout(CL_DEFAULT_TIMEOUT),
-                                                       headers, /*retryNotFound*/ true));
       }
+
+      futures.emplace_back(network::sendRequestRetry("shard:" + it.first, restVerb,
+                                                     std::move(url), std::move(buffer),
+                                                     network::Timeout(CL_DEFAULT_TIMEOUT),
+                                                     std::move(headers), /*retryNotFound*/ true));
     }
 
     // Now compute the result

arangod/RestHandler/RestDocumentHandler.cpp

@@ -650,8 +650,11 @@ RestStatus RestDocumentHandler::readManyDocuments() {
     return RestStatus::DONE;
   }
 
-  TRI_ASSERT(_request != nullptr);
+  bool success;
-  VPackSlice search = _request->payload(_activeTrx->transactionContextPtr()->getVPackOptions());
+  VPackSlice const search = this->parseVPackBody(success);
+  if (!success) {  // error message generated in parseVPackBody
+    return RestStatus::DONE;
+  }
   
   return waitForFuture(_activeTrx->documentAsync(cname, search, opOptions)
                        .thenValue([=](OperationResult opRes) {

arangod/Transaction/Methods.cpp

@@ -1233,7 +1233,6 @@ Result transaction::Methods::documentFastPath(std::string const& collectionName,
 
   if (_state->isCoordinator()) {
     OperationOptions options;  // use default configuration
-    options.ignoreRevs = true;
 
     OperationResult opRes = documentCoordinator(collectionName, value, options).get();
     if (opRes.fail()) {