Merge pull request #2768 from arangodb/bugfix/secondaries-and-manual-deployments

Bugfix/secondaries and manual deployments

CHANGELOG

@@ -1,6 +1,8 @@
 devel
 -----
 
+* documentation and js fixes for secondaries
+
 * enabled permissions on database and collection level
 
 * added "deduplicate" attribute for array indexes, which controls whether inserting
@@ -13,7 +15,7 @@ devel
       db.test.insert({ tags: ["a", "b"] });
       db.test.insert({ tags: ["c", "d", "c"] }); // will work, because deduplicate = true
       db.test.insert({ tags: ["a"] }); // will fail
-      
+
       // with deduplicate = false
       db._create("test");
       db.test.ensureIndex({ type: "hash", fields: ["tags[*]"], deduplicate: false });
@@ -29,7 +31,7 @@ devel
 * Debian/Ubuntu installer: make messages about future package upgrades more clear
 
 * fix a hangup in VST
-    
+
   The problem happened when the two first chunks of a VST message arrived
   together on a connection that was newly switched to VST.
 
@@ -41,7 +43,7 @@ devel
 * changed VM overcommit recommendation for user-friendliness
 
 * fix a shutdown bug in the cluster: a destroyed query could still be active
- 
+
 * do not terminate the entire server process if a temp file cannot be created 
   (Windows only)
 

Documentation/Books/Manual/Deployment/Distributed.md

@@ -50,16 +50,4 @@ arangod --server.authentication=false --server.endpoint tcp://0.0.0.0:8531 --clu
 
 Obviously, it would no longer be necessary to use different port numbers on different servers. We have chosen to keep all port numbers in comparison to the local setup to minimize the necessary changes.
 
-If you want to setup secondaries, the following commands will do the job:
-
-On 192.168.1.2:
-
-    curl -f -X PUT --data '{"primary": "DBServer001", "oldSecondary": "none", "newSecondary": "Secondary001"}' -H "Content-Type: application/json" http://192.168.1.3:8531/_admin/cluster/replaceSecondary && arangod --server.authentication=false --server.endpoint tcp://0.0.0.0:8629 --cluster.my-id Secondary001 --cluster.my-address tcp://192.168.1.2:8629 --cluster.agency-endpoint tcp://192.168.1.1:5001 --cluster.agency-endpoint tcp://192.168.1.2:5001 --cluster.agency-endpoint tcp://192.168.1.3:5001 --database.directory secondary1 &
-
-On 192.168.1.1:
-
-    curl -f -X PUT --data '{"primary": "DBServer002", "oldSecondary": "none", "newSecondary": "Secondary002"}' -H "Content-Type: application/json" http://localhost:8531/_admin/cluster/replaceSecondary && arangod --server.authentication=false --server.endpoint tcp://0.0.0.0:8630 --cluster.my-id Secondary002 --cluster.my-address tcp://192.168.1.1:8630 --cluster.agency-endpoint tcp://192.168.1.1:5001 --cluster.agency-endpoint tcp://192.168.1.2:5001 --cluster.agency-endpoint tcp://192.168.1.3:5001 --database.directory secondary2 &
-
-Note that we have started the `Secondary002` on the same machine as `DBServer001` and `Secondary001` on the same machine as `DBServer002` to avoid that a complete pair is lost when a machine fails. Furthermore, note that ArangoDB does not yet perform automatic failover to the secondary, if a primary fails. This only works in the Apache Mesos setting. For synchronous replication, automatic failover always works and you do not need to setup secondaries for this.
-
 After having swallowed these longish commands, we hope that you appreciate the simplicity of the setup with Apache Mesos and DC/OS.

Documentation/Books/Manual/Deployment/Local.md

@@ -55,18 +55,3 @@ Upon registering with the agency during startup the cluster will assign an ID to
 Should you ever have to restart a task, simply reuse the same value for `--cluster.my-local-info` and the same ID will be picked.
 
 You have now launched a complete ArangoDB cluster and can contact its coordinator at the endpoint `tcp://127.0.0.1:8531`, which means that you can reach the web UI under `http://127.0.0.1:8531`.
-
-
-### Secondaries
-
-Secondaries need a bit more work. Secondaries need to have some primary assigned. To do that there is a special route. To register a secondary you must first find out the Server-ID of the primary server. Then generate your own ID for the secondary you are about to start and call one of the coordinators like this (replace the value of "newSecondary" in the command):
-
-    curl -f -X PUT --data '{"primary": "DBServer001", "oldSecondary": "none", "newSecondary": "Secondary001"}' -H "Content-Type: application/json" http://127.0.0.1:8531/_admin/cluster/replaceSecondary
-
-If that call was successful you can start the secondary. Instead of providing `--cluster.my-local-info` you should now provide the Id in the curl call above via `--cluster.my-id`. You can omit the `--cluster.my-role` in this case. The secondary will find out from the agency about its role.
-
-To sum it up:
-
-    curl -f -X PUT --data '{"primary": "DBServer001", "oldSecondary": "none", "newSecondary": "Secondary001"}' -H "Content-Type: application/json" http://127.0.0.1:8531/_admin/cluster/replaceSecondary && arangod --server.authentication=false --server.endpoint tcp://0.0.0.0:8629 --cluster.my-id Secondary001 --cluster.my-address tcp://127.0.0.1:8629 --cluster.agency-endpoint tcp://127.0.0.1:5001 --cluster.agency-endpoint tcp://127.0.0.1:5002 --cluster.agency-endpoint tcp://127.0.0.1:5003 secondary1 &
-    curl -f -X PUT --data '{"primary": "DBServer002", "oldSecondary": "none", "newSecondary": "Secondary002"}' -H "Content-Type: application/json" http://127.0.0.1:8531/_admin/cluster/replaceSecondary && arangod --server.authentication=false --server.endpoint tcp://0.0.0.0:8630 --cluster.my-id Secondary002 --cluster.my-address tcp://127.0.0.1:8630 --cluster.agency-endpoint tcp://127.0.0.1:5001 --cluster.agency-endpoint tcp://127.0.0.1:5002 --cluster.agency-endpoint tcp://127.0.0.1:5003 secondary2 &
-

js/actions/api-cluster.js

@@ -753,31 +753,54 @@ actions.defineHttp({
         '"newSecondary" are given in body and are strings');
       return;
     }
-    let agency = ArangoAgency.get('Plan/DBServers/' + body.primary);
+    let dbservers = ArangoAgency.get('Plan/DBServers/' + body.primary).arango.Plan.DBservers;
+    let sID = ArangoAgency.get('Target/MapUniqueToShortID').arango.Target.MapUniqueToShortID;
 
-    if (fetchKey(agency, 'arango', 'Plan', 'DBServers', body.primary) === undefined) {
+    let id = body.primary;
+    let nid = body.newSecondary;
+
+    if (fetchKey(dbservers, id) === undefined) {
+      for (var sid in sID) {
+        if(sID[sid].ShortName === id) {
+          id = sid;
+          break;
+        }
+      }
+      actions.resultError(req, res, actions.HTTP_NOT_FOUND, 0,
+        'Primary with the given ID is not configured in Agency.');
+      return;
+    }
+
+    if (fetchKey(dbservers, nid) === undefined) {
+      for (sid in sID) {
+        if(sID[sid].ShortName === nid) {
+          nid = sid;
+          break;
+        }
+      }
       actions.resultError(req, res, actions.HTTP_NOT_FOUND, 0,
         'Primary with the given ID is not configured in Agency.');
       return;
     }
 
     let operations = {};
-    operations['/arango/Plan/DBServers/' + body.primary] = body.newSecondary;
+    operations['/arango/Plan/DBServers/' + id] = nid;
     operations['/arango/Plan/Version'] = {'op': 'increment'};
 
     let preconditions = {};
-    preconditions['/arango/Plan/DBServers/' + body.primary] = {'old': body.oldSecondary};
+    preconditions['/arango/Plan/DBServers/' + id] = {'old': body.oldSecondary};
 
     try {
+      require('internal').print([[operations, preconditions]]);
       global.ArangoAgency.write([[operations, preconditions]]);
     } catch (e) {
       if (e.code === 412) {
-        let oldValue = ArangoAgency.get('Plan/DBServers/' + body.primary);
+        let oldValue = ArangoAgency.get('Plan/DBServers/' + id);
         actions.resultError(req, res, actions.HTTP_PRECONDITION_FAILED, 0,
           'Primary does not have the given oldSecondary as ' +
           'its secondary, current value: '
           + JSON.stringify(
-            fetchKey(oldValue, 'arango', 'Plan', 'DBServers', body.primary)
+            fetchKey(oldValue, 'arango', 'Plan', 'DBServers', id)
           ));
         return;
       }

js/server/modules/@arangodb/cluster.js

@@ -2144,6 +2144,7 @@ exports.moveShard = moveShard;
 exports.supervisionState = supervisionState;
 exports.waitForSyncRepl = waitForSyncRepl;
 exports.endpoints = endpoints;
+exports.fetchKey = fetchKey;
 
 exports.executePlanForDatabases = executePlanForDatabases;
 exports.executePlanForCollections = executePlanForCollections;

scripts/startLocalCluster.sh

@@ -194,11 +194,12 @@ if [ "$SECONDARIES" == "1" ] ; then
     PORTTOPSE=`expr $SE_BASE + $NRDBSERVERS - 1` 
     for PORT in `seq $SE_BASE $PORTTOPSE` ; do
         mkdir cluster/data$PORT
+        UUID=SCND-$(uuidgen)
+        zfindex=$(printf "%04d" $index)
+        CLUSTER_ID="Secondary$zfindex"
         
-        CLUSTER_ID="Secondary$index"
-        
-        echo Registering secondary $CLUSTER_ID for "DBServer$index"
-        curl -f -X PUT --data "{\"primary\": \"DBServer$index\", \"oldSecondary\": \"none\", \"newSecondary\": \"$CLUSTER_ID\"}" -H "Content-Type: application/json" $ADDRESS:$CO_BASE/_admin/cluster/replaceSecondary
+        echo Registering secondary $CLUSTER_ID for "DBServer$zfindex"
+        curl -f -X PUT --data "{\"primary\": \"DBServer$zfindex\", \"oldSecondary\": \"none\", \"newSecondary\": \"$CLUSTER_ID\"}" -H "Content-Type: application/json" $ADDRESS:$CO_BASE/_admin/cluster/replaceSecondary
         echo Starting Secondary $CLUSTER_ID on port $PORT
         ${BUILD}/bin/arangod \
             -c none \