Merge branch 'devel' of github.com:triAGENS/ArangoDB into devel

CHANGELOG

@@ -1,6 +1,13 @@
 v1.4.0-XXXXX (2013-XX-XX)
 -------------------------
 
+* improved server shutdown time by signalling shutdown to applicationserver,
+  logging, cleanup and compactor threads
+
+* added foxx-manager `replace` command
+
+* added foxx-manager `installed` command (a more intuitive alias for `list`) 
+
 * fixed issue #617: Swagger API is missing '/_api/version'
 
 * fixed issue #615: Swagger API: Some commands have no parameter entry forms

Documentation/UserManual/FoxxManager.md

@@ -42,14 +42,14 @@ The most important commands are
 * `install`: Fetches a Foxx application from the central `foxx-apps` repository, 
   mounts it to a local URL and sets it up
 * `uninstall`: Unmounts a mounted Foxx application and calls its teardown method 
-* `list`: Lists all installed Foxx applications
+* `list`: Lists all installed Foxx applications (alias: `installed`)
 * `config`: Get information about the configuration including the path to the
   app directory.
 
 When dealing with a fresh install of ArangoDB, there should be no installed 
 applications besides the system applications that are shipped with ArangoDB.
 
-    unix> foxx-manager list
+    unix> foxx-manager installed
     Name        Author               Description                                                AppID               Version    Mount               Active    System 
     ---------   ------------------   --------------------------------------------------------   -----------------   --------   -----------------   -------   -------
     aardvark    Michael Hackstein    Foxx application manager for the ArangoDB web interface    app:aardvark:1.0    1.0        /_admin/aardvark    yes       yes    
@@ -70,9 +70,9 @@ should now be able to access the example application under
 
     http://localhost:8529/example
 
-using your favorite browser. It will now also be visible when using the `list` command.
+using your favorite browser. It will now also be visible when using the `installed` command.
 
-    unix> foxx-manager list
+    unix> foxx-manager installed
     Name          Author               Description                                                AppID                   Version    Mount               Active    System 
     -----------   ------------------   --------------------------------------------------------   ---------------------   --------   -----------------   -------   -------
     hello-foxx    Frank Celler         A simple example application.                              app:hello-foxx:1.2.2    1.2.2      /example            yes       no     
@@ -88,7 +88,7 @@ You can install the application again under different mount path.
 You now have two separate instances of the same application. They are 
 completely independent of each other.
 
-    unix> foxx-manager list
+    unix> foxx-manager installed
     Name          Author               Description                                                AppID                   Version    Mount               Active    System 
     -----------   ------------------   --------------------------------------------------------   ---------------------   --------   -----------------   -------   -------
     hello-foxx    Frank Celler         A simple example application.                              app:hello-foxx:1.2.2    1.2.2      /example            yes       no     
@@ -97,7 +97,7 @@ completely independent of each other.
     -----------   ------------------   --------------------------------------------------------   ---------------------   --------   -----------------   -------   -------
     3 application(s) found
 
-The current version of the application is `1.2.2` (check the output of `list` 
+The current version of the application is `1.2.2` (check the output of `installed` 
 for the current version). It is even possible to mount a different version 
 of an application.
 
@@ -178,7 +178,7 @@ We have now two versions of the hello world application. The current version fet
 Let's now mount the application in version 1.2.1 under `/hello`.
 
     unix> foxx-manager mount app:hello-foxx:1.2.1 /hello
-    unix> foxx-manager list
+    unix> foxx-manager installed
     Name          Author               Description                                                AppID                   Version    Mount               Active    System 
     -----------   ------------------   --------------------------------------------------------   ---------------------   --------   -----------------   -------   -------
     hello-foxx    Frank Celler         A simple example application.                              app:hello-foxx:1.2.1    1.2.1      /hello              yes       no     
@@ -216,7 +216,7 @@ Now check the list of collections again.
 
 You can now use the mounted and initialized application.
 
-    unix> foxx-manager list
+    unix> foxx-manager installed
     Name          Author               Description                                                AppID                   Version    Mount               Active    System 
     -----------   ------------------   --------------------------------------------------------   ---------------------   --------   -----------------   -------   -------
     hello-foxx    Frank Celler         A simple example application.                              app:hello-foxx:1.2.2    1.2.2      /example            yes       no     
@@ -242,6 +242,28 @@ however, still reachable. We still need to unmount it.
 
     unix> foxx-manager unmount /hello
 
+Making changes to an existing application
+-----------------------------------------
+
+There are two options for deploying local changes to an existing application:
+
+- the easiest way is to start the server in development mode. This will make 
+  all available foxx applications be available in under the `/dev/` URL prefix.
+  All changes to the application code will become live instantly because all
+  applications are reloaded on each request to a URL starting with `/dev/`.
+  NOte that the constant reloading in the development mode has a performance 
+  impact so it shouldn't be used in product.
+
+- if the development mode is not an option, you can use the `replace` command
+  from foxx-manager. It provides an easy mechanism to re-deploy the code for
+  an already installed application. It can be used as follows:
+
+      unix> foxx-manager replace hello-foxx /hello
+
+  The above will re-deploy the code for the application `hello-foxx` which has
+  to be already installed under the `/hello` mount point. The application's
+  setup function will be called when invoking `replace` but not `teardown`.
+
 Using Multiple Databases {#UserManualFoxxManagerDatabases}
 ==========================================================
 
@@ -277,11 +299,13 @@ Use `help` to see all commands
      mount                mounts a fetched foxx application to a local URL
      setup                setup executes the setup script (app must already be mounted)
      install              fetches a foxx application from the central foxx-apps repository, mounts it to a local URL and sets it up
+     replace              replaces an aleady existing foxx application with the current local version
      teardown             teardown execute the teardown script (app must be still be mounted)
      unmount              unmounts a mounted foxx application
      uninstall            unmounts a mounted foxx application and calls its teardown method
      purge                physically removes a foxx application and all mounts
      list                 lists all installed foxx applications
+     installed            lists all installed foxx applications (alias for list)
      fetched              lists all fetched foxx applications that were fetched into the local repository
      available            lists all foxx applications available in the local repository
      info                 displays information about a foxx application

arangod/VocBase/cleanup.c

@@ -305,10 +305,12 @@ void TRI_CleanupVocBase (void* data) {
       // clean up expired compactor locks
       TRI_CleanupCompactorVocBase(vocbase);
 
+      if (state == 1) {
         TRI_LockCondition(&vocbase->_cleanupCondition);
         TRI_TimedWaitCondition(&vocbase->_cleanupCondition, (uint64_t) CLEANUP_INTERVAL);
         TRI_UnlockCondition(&vocbase->_cleanupCondition);
       }
+    }
 
     if (state == 3) {
       // server shutdown

arangod/VocBase/compactor.c

@@ -1476,9 +1476,11 @@ void TRI_CompactorVocBase (void* data) {
     }
 
 
-    if (vocbase->_state == 1) {
+    if (state != 2 && vocbase->_state == 1) {
       // only sleep while server is still running
-      usleep(COMPACTOR_INTERVAL);
+      TRI_LockCondition(&vocbase->_compactorCondition);
+      TRI_TimedWaitCondition(&vocbase->_compactorCondition, (uint64_t) COMPACTOR_INTERVAL);
+      TRI_UnlockCondition(&vocbase->_compactorCondition);
     }
 
     if (state == 2) {

arangod/VocBase/server.c

@@ -593,6 +593,11 @@ static int OpenDatabases (TRI_server_t* server) {
   files = TRI_FilesDirectory(server->_databasePath);
   n = files._length;
  
+  // open databases in defined order
+  if (n > 1) {
+    qsort(files._buffer, n, sizeof(char**), &NameComparator);
+  }
+
   for (i = 0;  i < n;  ++i) {
     TRI_vocbase_t* vocbase;
     TRI_json_t* json;

arangod/VocBase/vocbase.c

@@ -1340,6 +1340,7 @@ TRI_vocbase_t* TRI_OpenVocBase (TRI_server_t* server,
 
   vocbase->_syncWaiters = 0;
   TRI_InitCondition(&vocbase->_syncWaitersCondition);
+  TRI_InitCondition(&vocbase->_compactorCondition);
   TRI_InitCondition(&vocbase->_cleanupCondition);
 
   // .............................................................................
@@ -1489,10 +1490,18 @@ void TRI_DestroyVocBase (TRI_vocbase_t* vocbase) {
   
   // wait until synchroniser and compactor are finished
   TRI_JoinThread(&vocbase->_synchroniser);
+  
+  TRI_LockCondition(&vocbase->_compactorCondition);
+  TRI_SignalCondition(&vocbase->_compactorCondition);
+  TRI_UnlockCondition(&vocbase->_compactorCondition);
   TRI_JoinThread(&vocbase->_compactor);
 
   // this will signal the cleanup thread to do one last iteration
   vocbase->_state = 3;
+
+  TRI_LockCondition(&vocbase->_cleanupCondition);
+  TRI_SignalCondition(&vocbase->_cleanupCondition);
+  TRI_UnlockCondition(&vocbase->_cleanupCondition);
   TRI_JoinThread(&vocbase->_cleanup);
 
   // free replication  
@@ -1513,9 +1522,7 @@ void TRI_DestroyVocBase (TRI_vocbase_t* vocbase) {
 
   // free collections
   for (i = 0;  i < vocbase->_collections._length;  ++i) {
-    TRI_vocbase_col_t* collection;
+    TRI_vocbase_col_t* collection = (TRI_vocbase_col_t*) vocbase->_collections._buffer[i];
-
-    collection = (TRI_vocbase_col_t*) vocbase->_collections._buffer[i];
 
     TRI_FreeCollectionVocBase(collection);
   }
@@ -1544,6 +1551,7 @@ void TRI_DestroyVocBase (TRI_vocbase_t* vocbase) {
   TRI_DestroyReadWriteLock(&vocbase->_lock);
   TRI_DestroyCondition(&vocbase->_syncWaitersCondition);
   TRI_DestroyCondition(&vocbase->_cleanupCondition);
+  TRI_DestroyCondition(&vocbase->_compactorCondition);
 
   // free name and path
   TRI_Free(TRI_CORE_MEM_ZONE, vocbase->_path);

arangod/VocBase/vocbase.h

@@ -367,6 +367,7 @@ typedef struct TRI_vocbase_s {
   }
   _compactionBlockers;
 
+  TRI_condition_t            _compactorCondition;
   TRI_condition_t            _cleanupCondition;
   TRI_condition_t            _syncWaitersCondition;
   int64_t                    _syncWaiters;

js/apps/system/aardvark/frontend/css/shellView.css

@@ -1,5 +1,4 @@
 /* REPL SHELL */
-/* REPL SHELL */
 
 .shellul {
   border-bottom: 20px solid rgba(0, 0, 0, 0.00);
@@ -13,20 +12,10 @@
   margin-left: 5px;
   margin-right: 5px !important;
   margin-bottom: -25px;
-  padding-bottom: 20px;
   height: 400px;
   background-color: black;
 }
 
-#editor {
-  float:left;
-  width: 150px;
-  height: 400px;
-  background-color:white;
-  overflow: hidden;
-  border: 1px solid #A0A0A0;
-}
-
 .vsplitbar {
   border: 0;
   width: 10px;
@@ -44,15 +33,13 @@
   cursor: col-resize !important;
 }
 
-/* REPL SHELL */
 #replShell {
   float:left;
   min-width: 100px;
-  width: 600px;
+  width: 100%;
   background-color: white;
   position: relative;
-  height: 400px;
+  height: 100%;
-  border-bottom: 2px solid black !important;
 }
 
 #replShell pre {
@@ -128,11 +115,6 @@
   height: 20px !important;
 }
 
-.jqconsole-blurred .jqconsole-header, .jqconsole-blurred .jssuccess, .jqconsole-blurred .jserror,
-.jqconsole-blurred .jqconsole-prompt, .jqconsole-blurred .jqconsole-old-prompt{
-  opacity: 0.7;
-}
-
 #queryOutput .ace_gutter-cell {
   background-color: #F0F0F0 !important;
 }

js/apps/system/aardvark/frontend/js/templates/shellView.ejs

@@ -5,7 +5,6 @@
   <div id="shell_workspace_header"/>
   <div id="shell_workspace">
 
-    <div id="editor"></div>
     <div id="replShell"></div>
 
 </div>

js/apps/system/aardvark/frontend/js/views/shellView.js

@@ -6,8 +6,7 @@ var shellView = Backbone.View.extend({
 
   el: '#content',
   events: {
-    'click #editor-run'     : 'submitEditor',
+    'click #editor-run'     : 'submitEditor'
-    'mouseleave .vsplitbar' : 'renderEditor'
   },
 
   template: new EJS({url: 'js/templates/shellView.ejs'}),
@@ -16,16 +15,9 @@ var shellView = Backbone.View.extend({
     $(this.el).html(this.template.text);
 
     this.replShell();
-    this.editor();
 
-    $("#shell_workspace").splitter({
-      dock: true
-    });
     $("#shell_workspace").trigger("resize", [ 150 ]);
 
-    $('.vsplitbar').append('<div id="editor-run">'+
-      '<button id="submitQueryButton" class="btn btn-success">Submit</button>'+
-    '</div>');
     this.resize();
     $.gritter.removeAll();
 
@@ -41,22 +33,12 @@ var shellView = Backbone.View.extend({
   resize: function () {
     // prevent endless recursion
     if (! this.resizing) {
-      var editorWidth = $('#editor').width();
       this.resizing = true;
       var windowHeight = $(window).height() - 250;
       $('#shell_workspace').height(windowHeight);
-      $("#shell_workspace").trigger("resize", [ editorWidth ]);
       this.resizing = false;
     }
   },
-  renderEditor: function () {
-    var editor = ace.edit("editor");
-    editor.resize();
-  },
-  editor: function () {
-    var editor = ace.edit("editor");
-    editor.getSession().setMode("ace/mode/javascript");
-  },
   executeJs: function (data) {
     try {
       var internal = require("internal");
@@ -73,12 +55,6 @@ var shellView = Backbone.View.extend({
       jqconsole.Write('ReferenceError: ' + e.message + '\n', 'jserror');
     }
   },
-  submitEditor: function () {
-    var editor = ace.edit("editor");
-    var data = editor.getValue();
-    this.executeJs(data);
-    jqconsole.Focus();
-  },
   replShell: function () {
     // Creating the console.
     var internal = require("internal");

js/client/modules/org/arangodb/foxx/manager.js

@@ -705,6 +705,18 @@ exports.run = function (args) {
              res.appId, 
              res.mount);
     }
+    else if (type === 'replace') {
+      if (3 < args.length) {
+        res = exports.replace(args[1], args[2], JSON.parse(args[3]));
+      }
+      else {
+        res = exports.replace(args[1], args[2]);
+      }
+
+      printf("Application %s replaced successfully at mount point %s\n", 
+             res.appId, 
+             res.mount);
+    }
     else if (type === 'uninstall') {
       res = exports.uninstall(args[1]);
 
@@ -722,7 +734,7 @@ exports.run = function (args) {
     else if (type === 'config') {
       exports.config();
     }
-    else if (type === 'list') {
+    else if (type === 'list' || type === 'installed') {
       if (1 < args.length && args[1] === "prefix") {
         exports.list(true);
       }
@@ -901,6 +913,52 @@ exports.unmount = function (mount) {
   return arangosh.checkRequestResult(res);
 };
 
+////////////////////////////////////////////////////////////////////////////////
+/// @brief replaces a FOXX application
+////////////////////////////////////////////////////////////////////////////////
+
+exports.replace = function (name, mount, options) {
+  'use strict';
+
+  checkParameter(
+    "replace(<name>, <mount>, [<options>])",
+    [ [ "Name", "string" ],
+      [ "Mount path", "string" ]],
+    [ name, mount ] );
+
+  validateMount(mount);
+
+  var aal = getStorage();
+  var existing = aal.firstExample({ type: "mount", name: name, mount: mount });
+
+  if (existing === null) {
+    throw new Error("Cannot find application '" + name + "' at mount '" + mount + "'");
+  }
+
+  var appId = existing.app;
+
+  // .............................................................................
+  // install at path
+  // .............................................................................
+
+  if (appId === null) {
+    throw new Error("Cannot extract application id");
+  }
+
+  options = options || {};
+
+  if (typeof options.setup === "undefined") {
+    options.setup = true;
+  }
+
+  options.reload = true;
+
+  exports.unmount(mount);
+  var res = exports.mount(appId, mount, options);
+
+  return res;
+};
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief installs a FOXX application
 ////////////////////////////////////////////////////////////////////////////////
@@ -1359,13 +1417,15 @@ exports.help = function () {
     "fetch"     : "fetches a foxx application from the central foxx-apps repository into the local repository",
     "mount"     : "mounts a fetched foxx application to a local URL",
     "setup"     : "setup executes the setup script (app must already be mounted)",
-    "install"      : "fetches a foxx application from the central foxx-apps repository, mounts it to a local URL "
+    "install"   : "fetches a foxx application from the central foxx-apps repository, mounts it to a local URL " +
-                   + "and sets it up",
+                  "and sets it up",
+    "replace"   : "replaces an aleady existing foxx application with the current local version",
     "teardown"  : "teardown execute the teardown script (app must be still be mounted)",
     "unmount"   : "unmounts a mounted foxx application",
     "uninstall" : "unmounts a mounted foxx application and calls its teardown method",
     "purge"     : "physically removes a foxx application and all mounts",
     "list"      : "lists all installed foxx applications",
+    "installed" : "lists all installed foxx applications (alias for list)",
     "fetched"   : "lists all fetched foxx applications that were fetched into the local repository", 
     "available" : "lists all foxx applications available in the local repository",
     "info"      : "displays information about a foxx application",

lib/ApplicationServer/ApplicationServer.cpp

@@ -39,6 +39,7 @@
 #endif
 
 #include "ApplicationServer/ApplicationFeature.h"
+#include "Basics/ConditionLocker.h"
 #include "Basics/FileUtils.h"
 #include "Basics/RandomGenerator.h"
 #include "Basics/StringUtils.h"
@@ -144,7 +145,8 @@ ApplicationServer::ApplicationServer (std::string const& name, std::string const
     _logLineNumber(false),
     _logSourceFilter(),
     _logContentFilter(),
-    _randomGenerator(5) {
+    _randomGenerator(5),
+    _finishedCondition() {
 }
 
 #else
@@ -185,7 +187,8 @@ ApplicationServer::ApplicationServer (std::string const& name, std::string const
     _logLineNumber(false),
     _logSourceFilter(),
     _logContentFilter(),
-    _randomGenerator(3) {
+    _randomGenerator(3),
+    _finishedCondition() {
   storeRealPrivileges();
 }
 
@@ -623,7 +626,9 @@ void ApplicationServer::wait () {
       running = false;
       break;
     }
-    sleep(1);
+  
+    CONDITION_LOCKER(locker, _finishedCondition);
+    locker.wait((uint64_t) (1000 * 1000));
   }
 }
 
@@ -645,6 +650,8 @@ void ApplicationServer::beginShutdown () {
 void ApplicationServer::stop () {
   beginShutdown();
 
+  CONDITION_LOCKER(locker, _finishedCondition);
+  locker.signal();
 
   // close all features
   for (vector<ApplicationFeature*>::iterator i = _features.begin();  i != _features.end();  ++i) {
@@ -664,8 +671,6 @@ void ApplicationServer::stop () {
     feature->stop();
     LOGGER_TRACE("shut down server feature '" << feature->getName() << "'");
   }
-
-
 }
 
 ////////////////////////////////////////////////////////////////////////////////

lib/ApplicationServer/ApplicationServer.h

@@ -31,6 +31,7 @@
 #include "Basics/Common.h"
 
 #include "Basics/ProgramOptions.h"
+#include "Basics/ConditionVariable.h"
 
 // -----------------------------------------------------------------------------
 // --SECTION--                                              forward declarations
@@ -840,6 +841,12 @@ namespace triagens {
 ////////////////////////////////////////////////////////////////////////////////
 
         uint32_t _randomGenerator;
+
+////////////////////////////////////////////////////////////////////////////////
+/// @brief condition variable for done
+////////////////////////////////////////////////////////////////////////////////
+
+        triagens::basics::ConditionVariable _finishedCondition;
     };
   }
 }

lib/Basics/ConditionLocker.cpp

@@ -111,6 +111,14 @@ void ConditionLocker::broadcast () {
   _conditionVariable->broadcast();
 }
 
+////////////////////////////////////////////////////////////////////////////////
+/// @brief signals an event
+////////////////////////////////////////////////////////////////////////////////
+
+void ConditionLocker::signal () {
+  _conditionVariable->signal();
+}
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @}
 ////////////////////////////////////////////////////////////////////////////////

lib/Basics/ConditionLocker.h

@@ -150,6 +150,12 @@ namespace triagens {
 
         void broadcast ();
 
+////////////////////////////////////////////////////////////////////////////////
+/// @brief signals an event
+////////////////////////////////////////////////////////////////////////////////
+
+        void signal ();
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @}
 ////////////////////////////////////////////////////////////////////////////////

lib/Basics/ConditionVariable.cpp

@@ -109,6 +109,14 @@ void ConditionVariable::broadcast () {
   TRI_BroadcastCondition(&_condition);
 }
 
+////////////////////////////////////////////////////////////////////////////////
+/// @brief signals a waiting thread
+////////////////////////////////////////////////////////////////////////////////
+
+void ConditionVariable::signal () {
+  TRI_SignalCondition(&_condition);
+}
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @}
 ////////////////////////////////////////////////////////////////////////////////

lib/Basics/ConditionVariable.h

@@ -151,6 +151,12 @@ namespace triagens {
 
         void broadcast ();
 
+////////////////////////////////////////////////////////////////////////////////
+/// @brief signals a waiting thread
+////////////////////////////////////////////////////////////////////////////////
+
+        void signal ();
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @}
 ////////////////////////////////////////////////////////////////////////////////

lib/BasicsC/logging.c

@@ -151,6 +151,12 @@ static TRI_log_buffer_t BufferOutput[OUTPUT_LOG_LEVELS][OUTPUT_BUFFER_SIZE];
 
 static TRI_mutex_t BufferLock;
 
+////////////////////////////////////////////////////////////////////////////////
+/// @brief condition variable for the logger
+////////////////////////////////////////////////////////////////////////////////
+
+static TRI_condition_t LogCondition;
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief message queue lock
 ////////////////////////////////////////////////////////////////////////////////
@@ -790,7 +796,9 @@ static void MessageQueueWorker (void* data) {
     }
 
     if (LoggingActive) {
-      usleep(sl);
+      TRI_LockCondition(&LogCondition);
+      TRI_TimedWaitCondition(&LogCondition, (uint64_t) sl);
+      TRI_UnlockCondition(&LogCondition);
     }
     else {
       TRI_LockMutex(&LogMessageQueueLock);
@@ -1905,6 +1913,7 @@ void TRI_InitialiseLogging (bool threaded) {
   ThreadedLogging = threaded;
 
   if (threaded) {
+    TRI_InitCondition(&LogCondition);
     TRI_InitMutex(&LogMessageQueueLock);
 
     TRI_InitVector(&LogMessageQueue, TRI_CORE_MEM_ZONE, sizeof(log_message_t));
@@ -1943,9 +1952,14 @@ bool TRI_ShutdownLogging (bool clearBuffers) {
 
   // join with the logging thread
   if (ThreadedLogging) {
+    TRI_LockCondition(&LogCondition);
+    TRI_SignalCondition(&LogCondition);
+    TRI_UnlockCondition(&LogCondition);
+
     TRI_JoinThread(&LoggingThread);
     TRI_DestroyMutex(&LogMessageQueueLock);
     TRI_DestroyVector(&LogMessageQueue);
+    TRI_DestroyCondition(&LogCondition);
   }
 
   // cleanup appenders

lib/Scheduler/ApplicationScheduler.cpp

@@ -574,7 +574,7 @@ void ApplicationScheduler::stop () {
 
     for (size_t count = 0;  count < MAX_TRIES && _scheduler->isRunning();  ++count) {
       LOGGER_TRACE("waiting for scheduler to stop");
-      usleep(1000000);
+      usleep(100000);
     }
 
     _scheduler->shutdown();