Merge branch 'devel' of https://github.com/arangodb/arangodb into devel

Documentation/Books/Manual/DataModeling/Documents/DocumentAddress.mdpp

@@ -78,26 +78,57 @@ As ArangoDB supports MVCC (Multiple Version Concurrency Control),
 documents can exist in more than one
 revision. The document revision is the MVCC token used to specify 
 a particular revision of a document (identified by its `_id`). 
-It is a string value currently
+It is a string value that contained (up to ArangoDB 3.0)
-containing an integer number and is unique within the list of document
+an integer number and is unique within the list of document
-revisions for a single document. Document revisions can be used to
+revisions for a single document. 
-conditionally query, update, replace or delete documents in the database. In
+In ArangoDB >= 3.1 the _rev strings
-order to find a particular revision of a document, you need the document
+are in fact time stamps. They use the local clock of the DBserver that
-handle or key, and the document revision.
+actually writes the document and have millisecond accuracy. 
+Actually, a "Hybrid Logical Clock" is used (for
+this concept see 
+[this paper](http://www.cse.buffalo.edu/tech-reports/2014-04.pdf)).
+
+Within one shard it is guaranteed that two different document revisions
+have a different _rev string, even if they are written in the same
+millisecond, and that these stamps are ascending.
+
+Note however that different servers in your cluster might have a clock
+skew, and therefore between different shards or even between different
+collections the time stamps are not guaranteed to be comparable.
+
+The Hybrid Logical Clock feature does one thing to address this
+issue: Whenever a message is sent from some server A in your cluster to
+another one B, it is ensured that any timestamp taken on B after the
+message has arrived is greater than any timestamp taken on A before the
+message was sent. This ensures that if there is some "causality" between
+events on different servers, time stamps increase from cause to effect.
+A direct consequence of this is that sometimes a server has to take
+timestamps that seem to come from the future of its own clock. It will
+however still produce ever increasing timestamps. If the clock skew is
+small, then your timestamps will relatively accurately describe the time
+when the document revision was actually written.
 
 ArangoDB uses 64bit unsigned integer values to maintain
-document revisions internally. When returning document revisions to
+document revisions internally. At this stage we intentionally do not
+document the exact format of the revision values. When returning 
+document revisions to
 clients, ArangoDB will put them into a string to ensure the revision
 is not clipped by clients that do not support big integers. Clients
 should treat the revision returned by ArangoDB as an opaque string
 when they store or use it locally. This will allow ArangoDB to change
-the format of revisions later if this should be required. Clients can
+the format of revisions later if this should be required (as has happened
+with 3.1 with the Hybrid Logical Clock). Clients can
 use revisions to perform simple equality/non-equality comparisons
 (e.g. to check whether a document has changed or not), but they should
 not use revision ids to perform greater/less than comparisons with them
 to check if a document revision is older than one another, even if this
 might work for some cases.
 
+Document revisions can be used to
+conditionally query, update, replace or delete documents in the database. In
+order to find a particular revision of a document, you need the document
+handle or key, and the document revision.
+
 
 !SUBSECTION Multiple Documents in a single Command
 

Documentation/Books/Manual/ReleaseNotes/NewFeatures31.mdpp

@@ -4,6 +4,21 @@ The following list shows in detail which features have been added or improved in
 ArangoDB 3.1. ArangoDB 3.1 also contains several bugfixes that are not listed
 here.
 
+!SECTION Data format
+
+The format of the revision values stored in the `_rev` attribute of documents
+has been changed in 3.1. Up to 3.0 they were strings containing largish decimal numbers. With 3.1, revision values are still strings, but are actually encoded time stamps of the creation date of the revision of the document. The time stamps are acquired using a hybrid logical clock (HLC) on the DBserver that holds the
+revision (for the concept of a hybrid logical clock see 
+[this paper](http://www.cse.buffalo.edu/tech-reports/2014-04.pdf)).
+See [this manual section](../DataModeling/Documents/DocumentAddress.html#document-revision) for details.
+
+ArangoDB >= 3.1 can ArangoDB 3.0 database directories and will simply continue
+to use the old `_rev` attribute values. New revisions will be written with
+the new time stamps.
+
+It is highly recommended to backup all your data before loading a database
+directory that was written by ArangoDB <= 3.0 into an ArangoDB >= 3.1.
+
 !SECTION Communication Layer
 
 ArangoDB up to 3.0 used [libev](http://software.schmorp.de/pkg/libev.html) for

Documentation/Books/Manual/ReleaseNotes/README.mdpp

@@ -2,6 +2,7 @@
 
 !SECTION Whats New
 
+- [Whats New in 3.1](NewFeatures31.md)
 - [Whats New in 3.0](NewFeatures30.md)
 - [Whats New in 2.8](NewFeatures28.md)
 - [Whats New in 2.7](NewFeatures27.md)
@@ -17,6 +18,7 @@
 Also see [Upgrading](../Administration/Upgrading/README.md) in the 
 Administration chapter.
 
+- [Incompatible changes in 3.1](UpgradingChanges31.md)
 - [Incompatible changes in 3.0](UpgradingChanges30.md)
 - [Incompatible changes in 2.8](UpgradingChanges28.md)
 - [Incompatible changes in 2.7](UpgradingChanges27.md)

Documentation/Books/Manual/ReleaseNotes/UpgradingChanges31.mdpp

@@ -51,7 +51,22 @@ meta-data files ("parameter.json"). Files containing the `maximalSize` attribute
 will still be picked up correctly for not-yet adjusted collections.
 
 The format of the revision values stored in the `_rev` attribute of documents
-has been changed in 3.1. Revision values are stored as hybrid logical clock values.
+has been changed in 3.1. Up to 3.0 they were strings containing largish decimal numbers. With 3.1, revision values are still strings, but are actually encoded time stamps of the creation date of the revision of the document. The time stamps are acquired using a hybrid logical clock (HLC) on the DBserver that holds the
+revision (for the concept of a hybrid logical clock see 
+[this paper](http://www.cse.buffalo.edu/tech-reports/2014-04.pdf)).
+See [this manual section](../DataModeling/Documents/DocumentAddress.html#document-revision) for details.
+
+ArangoDB >= 3.1 can ArangoDB 3.0 database directories and will simply continue
+to use the old `_rev` attribute values. New revisions will be written with
+the new time stamps.
+
+It is highly recommended to backup all your data before loading a database
+directory that was written by ArangoDB <= 3.0 into an ArangoDB >= 3.1.
+
+To change all your old `_rev` attributes into new style time stamps you
+have to use `arangodump` to dump all data out (using ArangoDB 3.0), and 
+use `arangorestore` into the new ArangoDB 3.1, which is the safest
+way to upgrade.
 
 !SECTION HTTP API changes
 

arangod/Aql/ClusterNodes.cpp

@@ -109,7 +109,8 @@ DistributeNode::DistributeNode(ExecutionPlan* plan,
       _varId(base.get("varId").getNumericValue<VariableId>()),
       _alternativeVarId(base.get("alternativeVarId").getNumericValue<VariableId>()),
       _createKeys(base.get("createKeys").getBoolean()),
-      _allowKeyConversionToObject(base.get("allowKeyConversionToObject").getBoolean()) {}
+      _allowKeyConversionToObject(base.get("allowKeyConversionToObject").getBoolean()),
+      _allowSpecifiedKeys(false) {}
 
 /// @brief toVelocyPack, for DistributedNode
 void DistributeNode::toVelocyPackHelper(VPackBuilder& nodes,