Merge branch 'devel' of ssh://github.com/ArangoDB/ArangoDB into devel

README

@@ -1,84 +0,0 @@
-****** ArangoDB ******
-ArangoDB is a multi-model, open-source database with flexible data models for
-documents, graphs, and key-values. Build high performance applications using a
-convenient SQL-like query language or JavaScript extensions. Use ACID
-transactions if you require them. Scale horizontally with a few mouse clicks.
-The supported data models can be mixed in queries and allow ArangoDB to be the
-aggregation point for your data.
-To get started, try one of our 10 minutes tutorials in your favorite
-programming language or try one of our ArangoDB_Cookbook_recipes.
-For the impatient: download and install ArangoDB. Start the server arangod and
-point your browser to http://127.0.0.1:8529/.
-***** Key Features in ArangoDB *****
-    * Multi-Model: Documents, graphs and key-value pairs — model your data as
-      you see fit for your application.
-    * Joins: Conveniently join what belongs together for flexible ad-hoc
-      querying, less data redundancy.
-    * Transactions: Easy application development keeping your data consistent
-      and safe. No hassle in your client.
-Here is an AQL query that makes use of all those features:
-[AQL Query Example]
-Joins and transactions are key features for flexible, secure data designs,
-widely used in relational databases but lacking in many NoSQL products.
-However, there is no need to forgo them in ArangoDB. You decide how and when to
-use joins and strong consistency guarantees, without sacrificing performance
-and scalability.
-Furthermore, ArangoDB offers a JavaScript framework called Foxx that is
-executed in the database server with direct access to the data. Build your own
-data-centric microservices with a few lines of code:
-Microservice Example
-[Microservice Example]
-By extending the HTTP API with user code written in JavaScript, ArangoDB can be
-turned into a strict schema-enforcing persistence engine.
-Next step, bundle your Foxx application as a docker_container and get it
-running in the cloud.
-Other features of ArangoDB include:
-    * Schema-free schemata let you combine the space efficiency of MySQL with
-      the performance power of NoSQL
-    * Use a data-centric microservices approach with ArangoDB Foxx and fuse
-      your application-logic and database together for maximal throughput
-    * JavaScript for all: no language zoo, you can use one language from your
-      browser to your back-end
-    * ArangoDB is multi-threaded - exploit the power of all your cores
-    * Flexible data modeling: model your data as combination of key-value
-      pairs, documents or graphs - perfect for social relations
-    * Free index choice: use the correct index for your problem, be it a skip
-      list or a fulltext search
-    * Configurable durability: let the application decide if it needs more
-      durability or more performance
-    * Powerful query language (AQL) to retrieve and modify data
-    * Transactions: run queries on multiple documents or collections with
-      optional transactional consistency and isolation
-    * Replication and Sharding: set up the database in a master-slave
-      configuration or spread bigger datasets across multiple servers
-    * It is open source (Apache License 2.0)
-For more in-depth information read the design_goals_of_ArangoDB
-***** Latest Release - ArangoDB 3.0 *****
-The What's_new_in_ArangoDB_3.0 can be found in the documentation.
-Key features of the 3.0 release are:
-    * use of VelocyPack as internal storage format
-    * AQL improvements
-    * much better cluster state management
-    * Synchronous replication (master/master)
-    * unified APIs for CRUD operations
-    * persistent indexes
-    * upgraded version of V8
-    * new web admin interface
-    * Foxx improvements
-    * Logging improvements
-    * improved documentation
-***** More Information *****
-Please check the Installation_Manual for installation and compilation
-instructions.
-The User_Manual has an introductory chapter showing the basic operations of
-ArangoDB.
-***** Stay in Contact *****
-We really appreciate feature requests and bug reports. Please use our Github
-issue tracker for reporting them:
-https://github.com/arangodb/arangodb/issues
-You can use the Google group for improvements, feature requests, comments:
-http://www.arangodb.com/community
-StackOverflow is great for questions about AQL, usage scenarios etc.
-http://stackoverflow.com/questions/tagged/arangodb
-To chat with the community and the developers we offer a Slack chat:
-http://slack.arangodb.com/

README.md

@@ -94,17 +94,17 @@ The [What's new in ArangoDB 3.0](https://docs.arangodb.com/3.0/Manual/ReleaseNot
 
 Key features of the 3.0 release are:
 
-- use of VelocyPack as internal storage format
+- Use of VelocyPack as internal storage format
 - AQL improvements
-- much better cluster state management
+- Much better cluster state management
 - Synchronous replication (master/master)
-- unified APIs for CRUD operations
+- Unified APIs for CRUD operations
-- persistent indexes
+- Persistent indexes
-- upgraded version of V8
+- Upgraded version of V8
-- new web admin interface
+- New web admin interface
 - Foxx improvements
 - Logging improvements
-- improved documentation
+- Improved documentation
 
 More Information
 ----------------

arangod/CMakeLists.txt

@@ -202,6 +202,7 @@ add_executable(${BIN_ARANGOD}
   FulltextIndex/fulltext-query.cpp
   FulltextIndex/fulltext-result.cpp
   GeneralServer/AsyncJobManager.cpp
+  GeneralServer/AuthenticationFeature.cpp
   GeneralServer/GeneralCommTask.cpp
   GeneralServer/GeneralListenTask.cpp
   GeneralServer/GeneralServer.cpp

arangod/Cluster/AgencyComm.cpp

@@ -32,9 +32,10 @@
 #include "Basics/StringUtils.h"
 #include "Basics/VelocyPackHelper.h"
 #include "Basics/WriteLocker.h"
+#include "Cluster/ClusterComm.h"
 #include "Cluster/ServerState.h"
 #include "Endpoint/Endpoint.h"
-#include "GeneralServer/GeneralServerFeature.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "Logger/Logger.h"
 #include "Random/RandomGenerator.h"
 #include "Rest/HttpRequest.h"
@@ -560,7 +561,7 @@ bool AgencyComm::tryInitializeStructure(std::string const& jwtSecret) {
       addEmptyVPackObject("DBServers", builder);
     }
     builder.add("InitDone", VPackValue(true));
-    builder.add("Secret", VPackValue(encodeHex(jwtSecret)));
+    builder.add("Secret", VPackValue(jwtSecret));
   } catch (std::exception const& e) {
     LOG_TOPIC(ERR, Logger::STARTUP) << "Couldn't create initializing structure "
                                     << e.what();
@@ -621,16 +622,22 @@ bool AgencyComm::shouldInitializeStructure() {
 bool AgencyComm::ensureStructureInitialized() {
   LOG_TOPIC(TRACE, Logger::STARTUP) << "Checking if agency is initialized";
 
-  GeneralServerFeature* restServer =
+  AuthenticationFeature* authentication =
-      application_features::ApplicationServer::getFeature<GeneralServerFeature>(
+      application_features::ApplicationServer::getFeature<AuthenticationFeature>(
-          "GeneralServer");
+          "Authentication");
+
+  TRI_ASSERT(authentication != nullptr);
 
   while (true) {
     while (shouldInitializeStructure()) {
       LOG_TOPIC(TRACE, Logger::STARTUP)
           << "Agency is fresh. Needs initial structure.";
       // mop: we initialized it .. great success
-      if (tryInitializeStructure(restServer->jwtSecret())) {
+      std::string secret;
+      if (authentication->isEnabled()) {
+        secret = authentication->jwtSecret();
+      }
+      if (tryInitializeStructure(secret)) {
         LOG_TOPIC(TRACE, Logger::STARTUP) << "Successfully initialized agency";
         break;
       }
@@ -667,8 +674,10 @@ bool AgencyComm::ensureStructureInitialized() {
     LOG(ERR) << "Couldn't find secret in agency!";
     return false;
   }
-
+  std::string const secret = secretValue.copyString();
-  restServer->setJwtSecret(decodeHex(secretValue.copyString()));
+  if (!secret.empty()) {
+    authentication->setJwtSecret(secretValue.copyString());
+  }
   return true;
 }
 
@@ -1768,7 +1777,7 @@ AgencyCommResult AgencyComm::send(
       << "': " << body;
 
   arangodb::httpclient::SimpleHttpClient client(connection, timeout, false);
-
+  client.setJwt(ClusterComm::instance()->jwt());
   client.keepConnectionOnDestruction(true);
 
   // set up headers

arangod/Cluster/ClusterComm.cpp

@@ -30,6 +30,7 @@
 #include "Basics/StringUtils.h"
 #include "Cluster/ClusterInfo.h"
 #include "Cluster/ServerState.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "Logger/Logger.h"
 #include "Scheduler/JobGuard.h"
 #include "Scheduler/SchedulerFeature.h"
@@ -205,7 +206,24 @@ char const* ClusterCommResult::stringifyStatus(ClusterCommOpStatus status) {
 ////////////////////////////////////////////////////////////////////////////////
 
 ClusterComm::ClusterComm()
-    : _backgroundThread(nullptr), _logConnectionErrors(false) {
+    : _backgroundThread(nullptr),
+      _logConnectionErrors(false),
+      _authenticationEnabled(false),
+      _jwt(""),
+      _jwtAuthorization("") {
+  auto authentication = application_features::ApplicationServer::getFeature<AuthenticationFeature>("Authentication");
+  TRI_ASSERT(authentication != nullptr);
+  if (authentication->isEnabled()) {
+    _authenticationEnabled = true;
+    VPackBuilder bodyBuilder;
+    {
+      VPackObjectBuilder p(&bodyBuilder);
+      bodyBuilder.add("server_id", VPackValue(ServerState::instance()->getId()));
+    }
+    _jwt = authentication->authInfo()->generateJwt(bodyBuilder);
+    _jwtAuthorization = "bearer " + _jwt;
+  }
+
   _communicator = std::make_shared<communicator::Communicator>();    
 }
 
@@ -737,7 +755,8 @@ void ClusterComm::asyncAnswer(std::string& coordinatorHeader,
   headers["X-Arango-Coordinator"] = coordinatorHeader;
   headers["X-Arango-Response-Code"] =
       responseToSend->responseString(responseToSend->responseCode());
-  headers["Authorization"] = ServerState::instance()->getAuthentication();
+
+  addAuthorization(&headers);
   TRI_voc_tick_t timeStamp = TRI_HybridLogicalClock();
   headers[StaticStrings::HLCHeader] =
       arangodb::basics::HybridLogicalClock::encodeTimeStamp(timeStamp);
@@ -1222,7 +1241,7 @@ std::pair<ClusterCommResult*, HttpRequest*> ClusterComm::prepareRequest(std::str
       }
     }
   }
-  headersCopy["Authorization"] = ServerState::instance()->getAuthentication();
+  addAuthorization(&headersCopy);
   TRI_voc_tick_t timeStamp = TRI_HybridLogicalClock();
   headersCopy[StaticStrings::HLCHeader] =
     arangodb::basics::HybridLogicalClock::encodeTimeStamp(timeStamp);
@@ -1248,6 +1267,12 @@ std::pair<ClusterCommResult*, HttpRequest*> ClusterComm::prepareRequest(std::str
   return std::make_pair(result, request);
 }
 
+void ClusterComm::addAuthorization(std::unordered_map<std::string, std::string>* headers) {
+  if (_authenticationEnabled) {
+    headers->emplace("Authorization", _jwtAuthorization);
+  }
+}
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief ClusterComm main loop
 ////////////////////////////////////////////////////////////////////////////////

arangod/Cluster/ClusterComm.h

@@ -524,17 +524,14 @@ class ClusterComm {
                          ClusterCommTimeout timeout, size_t& nrDone,
                          arangodb::LogTopic const& logTopic);
 
-  //////////////////////////////////////////////////////////////////////////////
-  /// @brief this is the fast path method for performRequests for the case
-  /// of only a single request in the vector. In this case we can use a single
-  /// syncRequest, which saves a network roundtrip. This is an important
-  /// optimization for the single document operation case.
-  /// Exact same semantics as performRequests.
-  //////////////////////////////////////////////////////////////////////////////
   std::shared_ptr<communicator::Communicator> communicator() {
     return _communicator;
   }
 
+  void addAuthorization(std::unordered_map<std::string, std::string>* headers);
+
+  std::string jwt() { return _jwt; };
+  
  private:
   size_t performSingleRequest(std::vector<ClusterCommRequest>& requests,
                               ClusterCommTimeout timeout, size_t& nrDone,
@@ -635,6 +632,9 @@ class ClusterComm {
   bool _logConnectionErrors;
 
   std::shared_ptr<communicator::Communicator> _communicator;
+  bool _authenticationEnabled;
+  std::string _jwt;
+  std::string _jwtAuthorization;
 };
 
 ////////////////////////////////////////////////////////////////////////////////

arangod/Cluster/ClusterFeature.cpp

@@ -57,6 +57,7 @@ ClusterFeature::ClusterFeature(application_features::ApplicationServer* server)
       _agencyCallbackRegistry(nullptr) {
   setOptional(true);
   requiresElevatedPrivileges(false);
+  startsAfter("Authentication");
   startsAfter("Logger");
   startsAfter("WorkMonitor");
   startsAfter("Database");
@@ -191,7 +192,6 @@ void ClusterFeature::validateOptions(std::shared_ptr<ProgramOptions> options) {
 }
 
 void ClusterFeature::prepare() {
-  ServerState::instance()->setAuthentication(_username, _password);
   ServerState::instance()->setDataPath(_dataPath);
   ServerState::instance()->setLogPath(_logPath);
   ServerState::instance()->setArangodPath(_arangodPath);

arangod/Cluster/HeartbeatThread.cpp

@@ -36,7 +36,7 @@
 #include "Cluster/ClusterMethods.h"
 #include "Cluster/DBServerAgencySync.h"
 #include "Cluster/ServerState.h"
-#include "GeneralServer/GeneralServerFeature.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "GeneralServer/RestHandlerFactory.h"
 #include "Logger/Logger.h"
 #include "RestServer/DatabaseFeature.h"
@@ -306,6 +306,10 @@ void HeartbeatThread::runDBServer() {
 ////////////////////////////////////////////////////////////////////////////////
 
 void HeartbeatThread::runCoordinator() {
+  AuthenticationFeature* authentication =
+    application_features::ApplicationServer::getFeature<AuthenticationFeature>(
+        "Authentication");
+  TRI_ASSERT(authentication != nullptr);
   LOG_TOPIC(TRACE, Logger::HEARTBEAT)
       << "starting heartbeat thread (coordinator version)";
 
@@ -424,7 +428,9 @@ void HeartbeatThread::runCoordinator() {
 
           if (userVersion > 0 && userVersion != oldUserVersion) {
             oldUserVersion = userVersion;
-            GeneralServerFeature::AUTH_INFO.outdate();
+            if (authentication->isEnabled()) {
+              authentication->authInfo()->outdate();
+            }
           }
         }
 

arangod/Cluster/ServerState.cpp

@@ -50,7 +50,6 @@ ServerState::ServerState()
       _dbserverConfig(),
       _coordinatorConfig(),
       _address(),
-      _authentication(),
       _lock(),
       _role(),
       _idOfPrimary(""),
@@ -160,22 +159,6 @@ std::string ServerState::stateToString(StateEnum state) {
   return "";
 }
 
-////////////////////////////////////////////////////////////////////////////////
-/// @brief set the authentication data for cluster-internal communication
-////////////////////////////////////////////////////////////////////////////////
-
-void ServerState::setAuthentication(std::string const& username,
-                                    std::string const& password) {
-  _authentication =
-      "Basic " + basics::StringUtils::encodeBase64(username + ":" + password);
-}
-
-////////////////////////////////////////////////////////////////////////////////
-/// @brief get the authentication data for cluster-internal communication
-////////////////////////////////////////////////////////////////////////////////
-
-std::string ServerState::getAuthentication() { return _authentication; }
-
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief find and set our role
 ////////////////////////////////////////////////////////////////////////////////

arangod/Cluster/ServerState.h

@@ -88,12 +88,6 @@ class ServerState {
   /// @brief sets the initialized flag
   void setClusterEnabled() { _clusterEnabled = true; }
 
-  /// @brief set the authentication data for cluster-internal communication
-  void setAuthentication(std::string const&, std::string const&);
-
-  /// @brief get the authentication data for cluster-internal communication
-  std::string getAuthentication();
-
   /// @brief flush the server state (used for testing)
   void flush();
 
@@ -312,9 +306,6 @@ class ServerState {
   /// @brief the server's own address, can be set just once
   std::string _address;
 
-  /// @brief the authentication data used for cluster-internal communication
-  std::string _authentication;
-
   /// @brief r/w lock for state
   arangodb::basics::ReadWriteLock _lock;
 

arangod/Cluster/v8-cluster.cpp

@@ -26,6 +26,7 @@
 #include "Cluster/ClusterInfo.h"
 #include "Cluster/ServerState.h"
 #include "Cluster/ClusterComm.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "V8/v8-buffer.h"
 #include "V8/v8-conv.h"
 #include "V8/v8-globals.h"
@@ -2015,6 +2016,34 @@ static void JS_GetId(v8::FunctionCallbackInfo<v8::Value> const& args) {
   TRI_V8_TRY_CATCH_END
 }
 
+static void JS_ClusterDownload(v8::FunctionCallbackInfo<v8::Value> const& args) {
+  TRI_V8_TRY_CATCH_BEGIN(isolate);
+  AuthenticationFeature* authentication =
+      application_features::ApplicationServer::getFeature<AuthenticationFeature>("Authentication");
+  
+  if (authentication->isEnabled()) {
+    // mop: really quick and dirty
+    v8::Handle<v8::Object> options = v8::Object::New(isolate);
+    v8::Handle<v8::Object> headers = v8::Object::New(isolate);
+    if (args.Length() > 2) {
+      if (args[2]->IsObject()) {
+        options = v8::Handle<v8::Object>::Cast(args[2]);
+        if (options->Has(TRI_V8_ASCII_STRING("headers"))) {
+          headers = v8::Handle<v8::Object>::Cast(options->Get(TRI_V8_ASCII_STRING("headers")));
+        }
+      }
+    }
+    options->Set(TRI_V8_ASCII_STRING("headers"), headers);
+    
+    std::string const authorization = "bearer " + ClusterComm::instance()->jwt();
+    v8::Handle<v8::String> v8Authorization = TRI_V8_STD_STRING(authorization);
+    headers->Set(TRI_V8_ASCII_STRING("Authorization"), v8Authorization);
+    args[2] = options;
+  }
+  TRI_V8_TRY_CATCH_END
+  return JS_Download(args);
+}
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief creates a global cluster context
 ////////////////////////////////////////////////////////////////////////////////
@@ -2227,4 +2256,7 @@ void TRI_InitV8Cluster(v8::Isolate* isolate, v8::Handle<v8::Context> context) {
     TRI_AddGlobalVariableVocbase(isolate, context,
                                  TRI_V8_ASCII_STRING("ArangoClusterComm"), ss);
   }
+  TRI_AddGlobalFunctionVocbase(
+      isolate, context, TRI_V8_ASCII_STRING("SYS_CLUSTER_DOWNLOAD"),
+      JS_ClusterDownload);
 }

arangod/GeneralServer/AuthenticationFeature.cpp

@@ -0,0 +1,150 @@
+////////////////////////////////////////////////////////////////////////////////
+/// DISCLAIMER
+///
+/// Copyright 2016 ArangoDB GmbH, Cologne, Germany
+///
+/// Licensed under the Apache License, Version 2.0 (the "License");
+/// you may not use this file except in compliance with the License.
+/// You may obtain a copy of the License at
+///
+///     http://www.apache.org/licenses/LICENSE-2.0
+///
+/// Unless required by applicable law or agreed to in writing, software
+/// distributed under the License is distributed on an "AS IS" BASIS,
+/// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+/// See the License for the specific language governing permissions and
+/// limitations under the License.
+///
+/// Copyright holder is ArangoDB GmbH, Cologne, Germany
+///
+/// @author Andreas Streichardt <andreas@arangodb.com>
+////////////////////////////////////////////////////////////////////////////////
+
+#include "AuthenticationFeature.h"
+
+#include "ProgramOptions/ProgramOptions.h"
+#include "RestServer/QueryRegistryFeature.h"
+#include "Random/RandomGenerator.h"
+
+using namespace arangodb;
+using namespace arangodb::options;
+
+AuthenticationFeature::AuthenticationFeature(
+    application_features::ApplicationServer* server)
+    : ApplicationFeature(server, "Authentication"),
+      _authenticationUnixSockets(true),
+      _authenticationSystemOnly(true),
+      _jwtSecretProgramOption(""),
+      _active(true) {
+
+  setOptional(true);
+  requiresElevatedPrivileges(false);
+  startsAfter("Random");
+}
+
+void AuthenticationFeature::collectOptions(
+    std::shared_ptr<ProgramOptions> options) {
+  options->addSection("server", "Server features");
+
+  options->addOldOption("server.disable-authentication",
+                        "server.authentication");
+  options->addOldOption("server.disable-authentication-unix-sockets",
+                        "server.authentication-unix-sockets");
+  options->addOldOption("server.authenticate-system-only",
+                        "server.authentication-system-only");
+  options->addOldOption("server.allow-method-override",
+                        "http.allow-method-override");
+  options->addOldOption("server.hide-product-header",
+                        "http.hide-product-header");
+  options->addOldOption("server.keep-alive-timeout", "http.keep-alive-timeout");
+  options->addOldOption("server.default-api-compatibility", "");
+  options->addOldOption("no-server", "server.rest-server");
+
+  options->addOption("--server.authentication",
+                     "enable or disable authentication for ALL client requests",
+                     new BooleanParameter(&_active));
+
+  options->addOption(
+      "--server.authentication-system-only",
+      "use HTTP authentication only for requests to /_api and /_admin",
+      new BooleanParameter(&_authenticationSystemOnly));
+
+#ifdef ARANGODB_HAVE_DOMAIN_SOCKETS
+  options->addOption("--server.authentication-unix-sockets",
+                     "authentication for requests via UNIX domain sockets",
+                     new BooleanParameter(&_authenticationUnixSockets));
+#endif
+
+  options->addOption("--server.jwt-secret",
+                     "secret to use when doing jwt authentication",
+                     new StringParameter(&_jwtSecretProgramOption));
+
+}
+
+void AuthenticationFeature::validateOptions(std::shared_ptr<ProgramOptions>) {
+  if (!_active) {
+    forceDisable();
+    return;
+  }
+  if (!_jwtSecretProgramOption.empty()) {
+    if (_jwtSecretProgramOption.length() > _maxSecretLength) {
+      LOG(ERR) << "Given JWT secret too long. Max length is "
+               << _maxSecretLength;
+      FATAL_ERROR_EXIT();
+    }
+  }
+}
+
+std::string AuthenticationFeature::generateNewJwtSecret() {
+  std::string jwtSecret = "";
+  uint16_t m = 254;
+
+  for (size_t i = 0; i < _maxSecretLength; i++) {
+    jwtSecret += (1 + RandomGenerator::interval(m));
+  }
+  return jwtSecret;
+}
+
+void AuthenticationFeature::start() {
+  LOG(INFO) << "Authentication is turned " << (_active ? "on" : "off");
+
+  if (!isEnabled()) {
+    return;
+  }
+  auto queryRegistryFeature =
+    application_features::ApplicationServer::getFeature<QueryRegistryFeature>("QueryRegistry");
+  authInfo()->setQueryRegistry(queryRegistryFeature->queryRegistry());
+
+  if (_authenticationSystemOnly) {
+    LOG(INFO) << "Authentication system only";
+  }
+
+#ifdef ARANGODB_HAVE_DOMAIN_SOCKETS
+  LOG(INFO) << "Authentication for unix sockets is turned "
+    << (_authenticationUnixSockets ? "on" : "off");
+#endif
+}
+
+AuthInfo* AuthenticationFeature::authInfo() {
+  // mop: catch misused stuff..authentication is disabled...why would you
+  // need any authentication info?
+  TRI_ASSERT(isEnabled());
+  return &_authInfo;
+}
+
+void AuthenticationFeature::unprepare() {
+}
+
+void AuthenticationFeature::prepare() {
+  if (!isEnabled()) {
+    return;
+  }
+  std::string jwtSecret = _jwtSecretProgramOption;
+  if (jwtSecret.empty()) {
+    jwtSecret = generateNewJwtSecret();
+  }
+  authInfo()->setJwtSecret(jwtSecret);
+}
+
+void AuthenticationFeature::stop() {
+}

arangod/GeneralServer/AuthenticationFeature.h

@@ -0,0 +1,64 @@
+////////////////////////////////////////////////////////////////////////////////
+/// DISCLAIMER
+///
+/// Copyright 2016 ArangoDB GmbH, Cologne, Germany
+///
+/// Licensed under the Apache License, Version 2.0 (the "License");
+/// you may not use this file except in compliance with the License.
+/// You may obtain a copy of the License at
+///
+///     http://www.apache.org/licenses/LICENSE-2.0
+///
+/// Unless required by applicable law or agreed to in writing, software
+/// distributed under the License is distributed on an "AS IS" BASIS,
+/// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+/// See the License for the specific language governing permissions and
+/// limitations under the License.
+///
+/// Copyright holder is ArangoDB GmbH, Cologne, Germany
+///
+/// @author Andreas Streichardt <andreas@arangodb.com>
+////////////////////////////////////////////////////////////////////////////////
+
+#ifndef APPLICATION_FEATURES_AUTHENTICATION_FEATURE_H
+#define APPLICATION_FEATURES_AUTHENTICATION_FEATURE_H 1
+
+#include "ApplicationFeatures/ApplicationFeature.h"
+#include "VocBase/AuthInfo.h"
+
+namespace arangodb {
+class AuthenticationFeature final
+    : public application_features::ApplicationFeature {
+ private:
+  const size_t _maxSecretLength = 64;
+
+ public:
+  explicit AuthenticationFeature(application_features::ApplicationServer*);
+
+ public:
+  void collectOptions(std::shared_ptr<options::ProgramOptions>) override final;
+  void validateOptions(std::shared_ptr<options::ProgramOptions>) override final;
+  void prepare() override final;
+  void start() override final;
+  void stop() override final;
+  void unprepare() override final;
+
+ private:
+  AuthInfo _authInfo;
+  bool _authenticationUnixSockets;
+  bool _authenticationSystemOnly;
+
+  std::string _jwtSecretProgramOption;
+  bool _active;
+
+ public:
+  bool authenticationUnixSockets() const { return _authenticationUnixSockets; }
+  bool authenticationSystemOnly() const { return _authenticationSystemOnly; }
+  std::string jwtSecret() { return authInfo()->jwtSecret(); }
+  std::string generateNewJwtSecret();
+  void setJwtSecret(std::string const& jwtSecret) { authInfo()->setJwtSecret(jwtSecret); }
+  AuthInfo* authInfo();
+};
+};
+
+#endif

arangod/GeneralServer/GeneralServerFeature.cpp

@@ -33,13 +33,13 @@
 #include "Cluster/RestAgencyCallbacksHandler.h"
 #include "Cluster/RestShardHandler.h"
 #include "Cluster/TraverserEngineRegistry.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "GeneralServer/GeneralServer.h"
 #include "GeneralServer/RestHandlerFactory.h"
 #include "InternalRestHandler/InternalRestTraverserHandler.h"
 #include "ProgramOptions/Parameters.h"
 #include "ProgramOptions/ProgramOptions.h"
 #include "ProgramOptions/Section.h"
-#include "Random/RandomGenerator.h"
 #include "Rest/Version.h"
 #include "RestHandler/RestAdminLogHandler.h"
 #include "RestHandler/RestAqlFunctionsHandler.h"
@@ -85,17 +85,12 @@ using namespace arangodb::options;
 rest::RestHandlerFactory* GeneralServerFeature::HANDLER_FACTORY = nullptr;
 rest::AsyncJobManager* GeneralServerFeature::JOB_MANAGER = nullptr;
 GeneralServerFeature* GeneralServerFeature::GENERAL_SERVER = nullptr;
-AuthInfo GeneralServerFeature::AUTH_INFO;
 
 GeneralServerFeature::GeneralServerFeature(
     application_features::ApplicationServer* server)
     : ApplicationFeature(server, "GeneralServer"),
       _allowMethodOverride(false),
-      _authentication(true),
-      _authenticationUnixSockets(true),
-      _authenticationSystemOnly(true),
       _proxyCheck(true),
-      _jwtSecret(""),
       _verificationMode(SSL_VERIFY_NONE),
       _verificationCallback(nullptr),
       _handlerFactory(nullptr),
@@ -103,6 +98,7 @@ GeneralServerFeature::GeneralServerFeature(
   setOptional(true);
   requiresElevatedPrivileges(false);
   startsAfter("Agency");
+  startsAfter("Authentication");
   startsAfter("CheckVersion");
   startsAfter("Database");
   startsAfter("Endpoint");
@@ -118,12 +114,6 @@ void GeneralServerFeature::collectOptions(
     std::shared_ptr<ProgramOptions> options) {
   options->addSection("server", "Server features");
 
-  options->addOldOption("server.disable-authentication",
-                        "server.authentication");
-  options->addOldOption("server.disable-authentication-unix-sockets",
-                        "server.authentication-unix-sockets");
-  options->addOldOption("server.authenticate-system-only",
-                        "server.authentication-system-only");
   options->addOldOption("server.allow-method-override",
                         "http.allow-method-override");
   options->addOldOption("server.hide-product-header",
@@ -132,25 +122,6 @@ void GeneralServerFeature::collectOptions(
   options->addOldOption("server.default-api-compatibility", "");
   options->addOldOption("no-server", "server.rest-server");
 
-  options->addOption("--server.authentication",
-                     "enable or disable authentication for ALL client requests",
-                     new BooleanParameter(&_authentication));
-
-  options->addOption(
-      "--server.authentication-system-only",
-      "use HTTP authentication only for requests to /_api and /_admin",
-      new BooleanParameter(&_authenticationSystemOnly));
-
-#ifdef ARANGODB_HAVE_DOMAIN_SOCKETS
-  options->addOption("--server.authentication-unix-sockets",
-                     "authentication for requests via UNIX domain sockets",
-                     new BooleanParameter(&_authenticationUnixSockets));
-#endif
-
-  options->addOption("--server.jwt-secret",
-                     "secret to use when doing jwt authentication",
-                     new StringParameter(&_jwtSecret));
-
   options->addSection("http", "HttpServer features");
 
   options->addHiddenOption("--http.allow-method-override",
@@ -211,14 +182,6 @@ void GeneralServerFeature::validateOptions(std::shared_ptr<ProgramOptions>) {
                        }),
         _accessControlAllowOrigins.end());
   }
-
-  if (!_jwtSecret.empty()) {
-    if (_jwtSecret.length() > GeneralServerFeature::_maxSecretLength) {
-      LOG(ERR) << "Given JWT secret too long. Max length is "
-               << GeneralServerFeature::_maxSecretLength;
-      FATAL_ERROR_EXIT();
-    }
-  }
 }
 
 static TRI_vocbase_t* LookupDatabaseFromRequest(GeneralRequest* request) {
@@ -247,6 +210,8 @@ static TRI_vocbase_t* LookupDatabaseFromRequest(GeneralRequest* request) {
 }
 
 static bool SetRequestContext(GeneralRequest* request, void* data) {
+  auto authentication = application_features::ApplicationServer::getFeature<AuthenticationFeature>("Authentication");
+  TRI_ASSERT(authentication != nullptr);
   TRI_vocbase_t* vocbase = LookupDatabaseFromRequest(request);
 
   // invalid database name specified, database not found etc.
@@ -261,27 +226,14 @@ static bool SetRequestContext(GeneralRequest* request, void* data) {
   }
 
   VocbaseContext* ctx = new arangodb::VocbaseContext(
-      request, vocbase, GeneralServerFeature::getJwtSecret());
+      request, vocbase);
   request->setRequestContext(ctx, true);
 
   // the "true" means the request is the owner of the context
   return true;
 }
 
-void GeneralServerFeature::generateNewJwtSecret() {
-  _jwtSecret = "";
-  uint16_t m = 254;
-
-  for (size_t i = 0; i < GeneralServerFeature::_maxSecretLength; i++) {
-    _jwtSecret += (1 + RandomGenerator::interval(m));
-  }
-}
-
 void GeneralServerFeature::prepare() {
-  if (_jwtSecret.empty()) {
-    generateNewJwtSecret();
-  }
-
   RestHandlerFactory::setMaintenance(true);
   GENERAL_SERVER = this;
 }
@@ -302,22 +254,13 @@ void GeneralServerFeature::start() {
     server->startListening();
   }
 
-  LOG(INFO) << "Authentication is turned " << (_authentication ? "on" : "off");
-
-  if (_authentication) {
-    if (_authenticationSystemOnly) {
-      LOG(INFO) << "Authentication system only";
-    }
-
-#ifdef ARANGODB_HAVE_DOMAIN_SOCKETS
-    LOG(INFO) << "Authentication for unix sockets is turned "
-              << (_authenticationUnixSockets ? "on" : "off");
-#endif
-  }
-
   // populate the authentication cache. otherwise no one can access the new
   // database
-  GeneralServerFeature::AUTH_INFO.outdate();
+  auto authentication = application_features::ApplicationServer::getFeature<AuthenticationFeature>("Authentication");
+  TRI_ASSERT(authentication != nullptr);
+  if (authentication->isEnabled()) {
+    authentication->authInfo()->outdate();
+  }
 }
 
 void GeneralServerFeature::stop() {
@@ -383,6 +326,11 @@ void GeneralServerFeature::defineHandlers() {
           "Cluster");
   TRI_ASSERT(cluster != nullptr);
   
+  AuthenticationFeature* authentication =
+      application_features::ApplicationServer::getFeature<AuthenticationFeature>(
+          "Authentication");
+  TRI_ASSERT(authentication != nullptr);
+
   auto queryRegistry = QueryRegistryFeature::QUERY_REGISTRY;
   auto traverserEngineRegistry =
       TraverserEngineRegistryFeature::TRAVERSER_ENGINE_REGISTRY;
@@ -552,10 +500,11 @@ void GeneralServerFeature::defineHandlers() {
       "/_admin/shutdown",
       RestHandlerCreator<arangodb::RestShutdownHandler>::createNoData);
   
+  if (authentication->isEnabled()) {
     _handlerFactory->addPrefixHandler(
-      "/_open/auth", RestHandlerCreator<arangodb::RestAuthHandler>::createData<
+        "/_open/auth",
-                         std::string const*>,
+        RestHandlerCreator<arangodb::RestAuthHandler>::createNoData);
-      &_jwtSecret);
+  }
 
   // ...........................................................................
   // /_admin

arangod/GeneralServer/GeneralServerFeature.h

@@ -30,7 +30,6 @@
 
 #include "Basics/asio-helper.h"
 #include "Actions/RestActionHandler.h"
-#include "VocBase/AuthInfo.h"
 
 namespace arangodb {
 namespace rest {
@@ -51,7 +50,6 @@ class GeneralServerFeature final
  public:
   static rest::RestHandlerFactory* HANDLER_FACTORY;
   static rest::AsyncJobManager* JOB_MANAGER;
-  static AuthInfo AUTH_INFO;
 
  public:
   static double keepAliveTimeout() {
@@ -72,9 +70,6 @@ class GeneralServerFeature final
   static verification_callback_asio verificationCallbackAsio() {
     return  GENERAL_SERVER->_verificationCallbackAsio;
   };
-  static bool authenticationEnabled() {
-    return GENERAL_SERVER != nullptr && GENERAL_SERVER->_authentication;
-  }
   
   static bool hasProxyCheck() {
     return GENERAL_SERVER != nullptr && GENERAL_SERVER->proxyCheck();
@@ -88,14 +83,6 @@ class GeneralServerFeature final
     return GENERAL_SERVER->trustedProxies();
   }
 
-  static std::string getJwtSecret() {
-    if (GENERAL_SERVER == nullptr) {
-      return std::string();
-    }
-
-    return GENERAL_SERVER->jwtSecret();
-  }
-
   static bool allowMethodOverride() {
     if (GENERAL_SERVER == nullptr) {
       return false;
@@ -116,7 +103,6 @@ class GeneralServerFeature final
 
  private:
   static GeneralServerFeature* GENERAL_SERVER;
-  static const size_t _maxSecretLength = 64;
 
  public:
   explicit GeneralServerFeature(application_features::ApplicationServer*);
@@ -139,28 +125,18 @@ class GeneralServerFeature final
  private:
   double _keepAliveTimeout = 300.0;
   bool _allowMethodOverride;
-  bool _authentication;
-  bool _authenticationUnixSockets;
-  bool _authenticationSystemOnly;
 
   bool _proxyCheck;
   std::vector<std::string> _trustedProxies;
   std::vector<std::string> _accessControlAllowOrigins;
 
-  std::string _jwtSecret;
   int _verificationMode;
   verification_callback_fptr _verificationCallback;
   verification_callback_asio _verificationCallbackAsio;
 
  public:
-  bool authentication() const { return _authentication; }
-  bool authenticationUnixSockets() const { return _authenticationUnixSockets; }
-  bool authenticationSystemOnly() const { return _authenticationSystemOnly; }
   bool proxyCheck() const { return _proxyCheck; }
   std::vector<std::string> trustedProxies() const { return _trustedProxies; }
-  std::string jwtSecret() const { return _jwtSecret; }
-  void generateNewJwtSecret();
-  void setJwtSecret(std::string const& jwtSecret) { _jwtSecret = jwtSecret; }
 
  private:
   void buildServers();

arangod/GeneralServer/HttpCommTask.cpp

@@ -152,6 +152,7 @@ void HttpCommTask::addResponse(HttpResponse* response) {
   if (!buffer->empty()) {
     LOG_TOPIC(TRACE, Logger::REQUESTS)
         << "\"http-request-response\",\"" << (void*)this << "\",\""
+        << _fullUrl << "\",\""
         << StringUtils::escapeUnicode(
                std::string(buffer->c_str(), buffer->length()))
         << "\"";

arangod/GeneralServer/VppCommTask.cpp

@@ -35,6 +35,7 @@
 #include "Basics/HybridLogicalClock.h"
 #include "Basics/StringBuffer.h"
 #include "Basics/VelocyPackHelper.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "GeneralServer/GeneralServer.h"
 #include "GeneralServer/GeneralServerFeature.h"
 #include "GeneralServer/RestHandler.h"
@@ -62,10 +63,11 @@ VppCommTask::VppCommTask(EventLoop loop, GeneralServer* server,
       GeneralCommTask(loop, server, std::move(socket), std::move(info),
                       timeout),
       _authenticatedUser(),
-      _authenticationEnabled(
+      _authentication(nullptr) {
-          application_features::ApplicationServer::getFeature<
+  _authentication = application_features::ApplicationServer::getFeature<AuthenticationFeature>(
-              GeneralServerFeature>("GeneralServer")
+    "Authentication");
-              ->authenticationEnabled()) {
+  TRI_ASSERT(_authentication != nullptr);
+
   _protocol = "vpp";
   _readBuffer.reserve(
       _bufferLength);  // ATTENTION <- this is required so we do not
@@ -180,6 +182,35 @@ bool VppCommTask::isChunkComplete(char* start) {
   return true;
 }
 
+void VppCommTask::handleAuthentication(VPackSlice const& header, uint64_t messageId) {
+  // std::string encryption = header.at(2).copyString();
+  std::string user = header.at(3).copyString();
+  std::string pass = header.at(4).copyString();
+
+  bool authOk = false;
+  if (!_authentication->isEnabled()) {
+    authOk = true;
+  } else {
+    auto auth = basics::StringUtils::encodeBase64(user + ":" + pass);
+    AuthResult result = _authentication->authInfo()->checkAuthentication(
+        AuthInfo::AuthType::BASIC, auth);
+    
+    authOk = result._authorized;
+  }
+
+  if (authOk) {
+    // mop: hmmm...user should be completely ignored if there is no auth IMHO
+    _authenticatedUser = std::move(user);
+    handleSimpleError(rest::ResponseCode::OK, TRI_ERROR_NO_ERROR,
+        "authentication successful", messageId);
+  } else {
+    _authenticatedUser.clear();
+    handleSimpleError(rest::ResponseCode::UNAUTHORIZED,
+        TRI_ERROR_HTTP_UNAUTHORIZED, "authentication failed",
+        messageId);
+  }
+}
+
 // reads data from the socket
 bool VppCommTask::processRead() {
   RequestStatisticsAgent agent(true);
@@ -248,24 +279,7 @@ bool VppCommTask::processRead() {
 
     // handle request types
     if (type == 1000) {
-      // do authentication
+      handleAuthentication(header, chunkHeader._messageID);
-      // std::string encryption = header.at(2).copyString();
-      std::string user = header.at(3).copyString();
-      std::string pass = header.at(4).copyString();
-      auto auth = basics::StringUtils::encodeBase64(user + ":" + pass);
-      AuthResult result = GeneralServerFeature::AUTH_INFO.checkAuthentication(
-          AuthInfo::AuthType::BASIC, auth);
-
-      if (!_authenticationEnabled || result._authorized) {
-        _authenticatedUser = std::move(user);
-        handleSimpleError(rest::ResponseCode::OK, TRI_ERROR_NO_ERROR,
-                          "authentication successful", chunkHeader._messageID);
-      } else {
-        _authenticatedUser.clear();
-        handleSimpleError(rest::ResponseCode::UNAUTHORIZED,
-                          TRI_ERROR_HTTP_UNAUTHORIZED, "authentication failed",
-                          chunkHeader._messageID);
-      }
     } else {
       // the handler will take ownersip of this pointer
       std::unique_ptr<VppRequest> request(new VppRequest(
@@ -276,9 +290,9 @@ bool VppCommTask::processRead() {
       // check authentication
       std::string const& dbname = request->databaseName();
       AuthLevel level = AuthLevel::RW;
-      if (_authenticationEnabled &&
+      if (_authentication->isEnabled() &&
           (!_authenticatedUser.empty() || !dbname.empty())) {
-        level = GeneralServerFeature::AUTH_INFO.canUseDatabase(
+        level = _authentication->authInfo()->canUseDatabase(
             _authenticatedUser, dbname);
       }
 

arangod/GeneralServer/VppCommTask.h

@@ -34,6 +34,9 @@
 #include "lib/Rest/VppResponse.h"
 
 namespace arangodb {
+
+class AuthenticationFeature;
+
 namespace rest {
 
 class VppCommTask : public GeneralCommTask {
@@ -63,6 +66,7 @@ class VppCommTask : public GeneralCommTask {
   std::unique_ptr<GeneralResponse> createResponse(
       rest::ResponseCode, uint64_t messageId) override final;
   
+  void handleAuthentication(VPackSlice const& header, uint64_t messageId);
   void handleSimpleError(rest::ResponseCode code, uint64_t id) override {
     VppResponse response(code, id);
     addResponse(&response);
@@ -132,6 +136,10 @@ class VppCommTask : public GeneralCommTask {
       char const* vpackBegin, char const* chunkEnd);
 
   std::string _authenticatedUser;
+  AuthenticationFeature* _authentication;
+  // user
+  // authenticated or not
+  // database aus url
   bool _authenticationEnabled;
 };
 }

arangod/Replication/Syncer.cpp

@@ -97,8 +97,13 @@ Syncer::Syncer(TRI_vocbase_t* vocbase,
 
       std::string username = _configuration._username;
       std::string password = _configuration._password;
+      std::string jwt = _configuration._jwt;
       
+      if (!username.empty()) {
         _client->setUserNamePassword("/", username, password);
+      } else {
+        _client->setJwt(_configuration._jwt);
+      }
       _client->setLocationRewriter(this, &rewriteLocation);
 
       _client->_maxRetries = 2;

arangod/RestHandler/RestAuthHandler.cpp

@@ -27,7 +27,7 @@
 #include <velocypack/velocypack-aliases.h>
 
 #include "Basics/StringUtils.h"
-#include "GeneralServer/GeneralServerFeature.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "Logger/Logger.h"
 #include "Rest/HttpRequest.h"
 #include "Ssl/SslInterface.h"
@@ -38,22 +38,16 @@ using namespace arangodb::basics;
 using namespace arangodb::rest;
 
 RestAuthHandler::RestAuthHandler(GeneralRequest* request,
-                                 GeneralResponse* response,
+                                 GeneralResponse* response)
-                                 std::string const* jwtSecret)
     : RestVocbaseBaseHandler(request, response),
-      _jwtSecret(*jwtSecret),
       _validFor(60 * 60 * 24 * 30) {}
 
 bool RestAuthHandler::isDirect() const { return false; }
 
 std::string RestAuthHandler::generateJwt(std::string const& username,
                                          std::string const& password) {
-  VPackBuilder headerBuilder;
+  auto authentication = application_features::ApplicationServer::getFeature<AuthenticationFeature>("Authentication");
-  {
+  TRI_ASSERT(authentication != nullptr);
-    VPackObjectBuilder h(&headerBuilder);
-    headerBuilder.add("alg", VPackValue("HS256"));
-    headerBuilder.add("typ", VPackValue("JWT"));
-  }
 
   std::chrono::seconds exp =
       std::chrono::duration_cast<std::chrono::seconds>(
@@ -66,15 +60,7 @@ std::string RestAuthHandler::generateJwt(std::string const& username,
     bodyBuilder.add("iss", VPackValue("arangodb"));
     bodyBuilder.add("exp", VPackValue(exp.count()));
   }
-
+  return authentication->authInfo()->generateJwt(bodyBuilder);
-  std::string fullMessage(StringUtils::encodeBase64(headerBuilder.toJson()) +
-                          "." +
-                          StringUtils::encodeBase64(bodyBuilder.toJson()));
-  std::string signature =
-      sslHMAC(_jwtSecret.c_str(), _jwtSecret.length(), fullMessage.c_str(),
-              fullMessage.length(), SslInterface::Algorithm::ALGORITHM_SHA256);
-
-  return fullMessage + "." + StringUtils::encodeBase64U(signature);
 }
 
 RestStatus RestAuthHandler::execute() {
@@ -110,8 +96,11 @@ RestStatus RestAuthHandler::execute() {
   _username = usernameSlice.copyString();
   std::string const password = passwordSlice.copyString();
   
+  auto authentication =
+      application_features::ApplicationServer::getFeature<AuthenticationFeature>(
+          "Authentication");
   AuthResult auth =
-      GeneralServerFeature::AUTH_INFO.checkPassword(_username, password);
+    authentication->authInfo()->checkPassword(_username, password);
 
   if (auth._authorized) {
     VPackBuilder resultBuilder;

arangod/RestHandler/RestAuthHandler.h

@@ -32,8 +32,7 @@
 namespace arangodb {
 class RestAuthHandler : public RestVocbaseBaseHandler {
  public:
-  RestAuthHandler(GeneralRequest*, GeneralResponse*,
+  RestAuthHandler(GeneralRequest*, GeneralResponse*);
-                  std::string const* jwtSecret);
 
   std::string generateJwt(std::string const&, std::string const&);
 

arangod/RestHandler/RestReplicationHandler.cpp

@@ -3149,6 +3149,8 @@ void RestReplicationHandler::handleCommandMakeSlave() {
       VelocyPackHelper::getStringValue(body, "username", "");
   std::string const password =
       VelocyPackHelper::getStringValue(body, "password", "");
+  std::string const jwt =
+      VelocyPackHelper::getStringValue(body, "jwt", "");
   std::string const restrictType =
       VelocyPackHelper::getStringValue(body, "restrictType", "");
 
@@ -3162,6 +3164,7 @@ void RestReplicationHandler::handleCommandMakeSlave() {
   config._database = database;
   config._username = username;
   config._password = password;
+  config._jwt = jwt;
   config._includeSystem =
       VelocyPackHelper::getBooleanValue(body, "includeSystem", true);
   config._requestTimeout = VelocyPackHelper::getNumericValue<double>(
@@ -3325,6 +3328,8 @@ void RestReplicationHandler::handleCommandSync() {
       VelocyPackHelper::getStringValue(body, "username", "");
   std::string const password =
       VelocyPackHelper::getStringValue(body, "password", "");
+  std::string const jwt =
+      VelocyPackHelper::getStringValue(body, "jwt", "");
   bool const verbose =
       VelocyPackHelper::getBooleanValue(body, "verbose", false);
   bool const includeSystem =
@@ -3365,6 +3370,7 @@ void RestReplicationHandler::handleCommandSync() {
   config._database = database;
   config._username = username;
   config._password = password;
+  config._jwt = jwt;
   config._includeSystem = includeSystem;
   config._verbose = verbose;
   config._useCollectionId = useCollectionId;
@@ -3481,6 +3487,11 @@ void RestReplicationHandler::handleCommandApplierSetConfig() {
     config._password = password.copyString();
   }
   
+  VPackSlice const jwt = body.get("jwt");
+  if (jwt.isString()) {
+    config._jwt = jwt.copyString();
+  }
+
   config._requestTimeout = VelocyPackHelper::getNumericValue<double>(
       body, "requestTimeout", config._requestTimeout);
   config._connectTimeout = VelocyPackHelper::getNumericValue<double>(

arangod/RestHandler/RestUploadHandler.h

@@ -37,7 +37,7 @@ class RestUploadHandler : public RestVocbaseBaseHandler {
   ~RestUploadHandler();
 
  public:
-  RestStatus execute();
+  RestStatus execute() override;
   char const* name() const override final { return "RestUploadHandler"; }
 
   //////////////////////////////////////////////////////////////////////////////

arangod/RestServer/DatabaseFeature.cpp

@@ -34,7 +34,7 @@
 #include "Basics/files.h"
 #include "Cluster/ServerState.h"
 #include "Cluster/v8-cluster.h"
-#include "GeneralServer/GeneralServerFeature.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "Logger/Logger.h"
 #include "ProgramOptions/ProgramOptions.h"
 #include "ProgramOptions/Section.h"
@@ -230,6 +230,7 @@ DatabaseFeature::DatabaseFeature(ApplicationServer* server)
       _upgrade(false) {
   setOptional(false);
   requiresElevatedPrivileges(false);
+  startsAfter("Authentication");
   startsAfter("DatabasePath");
   startsAfter("EngineSelector");
   startsAfter("LogfileManager");
@@ -823,7 +824,9 @@ std::vector<std::string> DatabaseFeature::getDatabaseNamesForUser(
       TRI_vocbase_t* vocbase = p.second;
       TRI_ASSERT(vocbase != nullptr);
 
-      auto level = GeneralServerFeature::AUTH_INFO.canUseDatabase(
+      auto authentication = application_features::ApplicationServer::getFeature<AuthenticationFeature>(
+          "Authentication");
+      auto level = authentication->authInfo()->canUseDatabase(
           username, vocbase->name());
 
       if (level == AuthLevel::NONE) {

arangod/RestServer/VocbaseContext.cpp

@@ -32,7 +32,7 @@
 #include "Basics/tri-strings.h"
 #include "Cluster/ServerState.h"
 #include "Endpoint/ConnectionInfo.h"
-#include "GeneralServer/GeneralServerFeature.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "Logger/Logger.h"
 #include "Ssl/SslInterface.h"
 #include "Utils/Events.h"
@@ -46,36 +46,19 @@ using namespace arangodb::rest;
 double VocbaseContext::ServerSessionTtl =
     60.0 * 60.0 * 24 * 60;  // 2 month session timeout
 
-VocbaseContext::VocbaseContext(GeneralRequest* request, TRI_vocbase_t* vocbase,
+VocbaseContext::VocbaseContext(GeneralRequest* request, TRI_vocbase_t* vocbase)
-                               std::string const& jwtSecret)
+    : RequestContext(request),
-    : RequestContext(request), _vocbase(vocbase), _jwtSecret(jwtSecret) {
+    _vocbase(vocbase),
+    _authentication(nullptr) {
   TRI_ASSERT(_vocbase != nullptr);
+  _authentication =
+      application_features::ApplicationServer::getFeature<AuthenticationFeature>(
+          "Authentication");
+  TRI_ASSERT(_authentication != nullptr);
 }
 
 VocbaseContext::~VocbaseContext() { _vocbase->release(); }
 
-////////////////////////////////////////////////////////////////////////////////
-/// @brief whether or not to use special cluster authentication
-////////////////////////////////////////////////////////////////////////////////
-
-bool VocbaseContext::useClusterAuthentication() const {
-  auto role = ServerState::instance()->getRole();
-
-  if (ServerState::instance()->isDBServer(role)) {
-    return true;
-  }
-
-  if (ServerState::instance()->isCoordinator(role)) {
-    std::string const& s = _request->requestPath();
-
-    if (s == "/_api/shard-comm" || s == "/_admin/shutdown") {
-      return true;
-    }
-  }
-
-  return false;
-}
-
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief checks the authentication
 ////////////////////////////////////////////////////////////////////////////////
@@ -83,11 +66,7 @@ bool VocbaseContext::useClusterAuthentication() const {
 rest::ResponseCode VocbaseContext::authenticate() {
   TRI_ASSERT(_vocbase != nullptr);
 
-  auto restServer =
+  if (!_authentication->isEnabled()) {
-      application_features::ApplicationServer::getFeature<GeneralServerFeature>(
-          "GeneralServer");
-
-  if (!restServer->authentication()) {
     // no authentication required at all
     return rest::ResponseCode::OK;
   }
@@ -109,15 +88,24 @@ rest::ResponseCode VocbaseContext::authenticate() {
     }
   }
   
-  // check that we are allowed to see the database
+  if (result != rest::ResponseCode::OK) {
-  if (result == rest::ResponseCode::OK && !forceOpen) {
+    return result;
-    if (!StringUtils::isPrefix(path, "/_api/user/")) {
+  }
+  
   std::string const& username = _request->user();
+  // mop: internal request => no username present
+  if (username.empty()) {
+    return rest::ResponseCode::OK;
+  }
+
+  // check that we are allowed to see the database
+  if (!forceOpen) {
+    if (!StringUtils::isPrefix(path, "/_api/user/")) {
       std::string const& dbname = _request->databaseName();
       
       if (!username.empty() || !dbname.empty()) {
         AuthLevel level =
-            GeneralServerFeature::AUTH_INFO.canUseDatabase(username, dbname);
+            _authentication->authInfo()->canUseDatabase(username, dbname);
 
         if (level != AuthLevel::RW) {
           events::NotAuthorized(_request);
@@ -131,16 +119,13 @@ rest::ResponseCode VocbaseContext::authenticate() {
 }
 
 rest::ResponseCode VocbaseContext::authenticateRequest(bool* forceOpen) {
-  auto restServer =
-      application_features::ApplicationServer::getFeature<GeneralServerFeature>(
-          "GeneralServer");
 #ifdef ARANGODB_HAVE_DOMAIN_SOCKETS
   // check if we need to run authentication for this type of
   // endpoint
   ConnectionInfo const& ci = _request->connectionInfo();
 
   if (ci.endpointType == Endpoint::DomainType::UNIX &&
-      !restServer->authenticationUnixSockets()) {
+      !_authentication->authenticationUnixSockets()) {
     // no authentication required for unix socket domain connections
     return rest::ResponseCode::OK;
   }
@@ -148,7 +133,7 @@ rest::ResponseCode VocbaseContext::authenticateRequest(bool* forceOpen) {
 
   std::string const& path = _request->requestPath();
 
-  if (restServer->authenticationSystemOnly()) {
+  if (_authentication->authenticationSystemOnly()) {
     // authentication required, but only for /_api, /_admin etc.
 
     if (!path.empty()) {
@@ -209,32 +194,7 @@ rest::ResponseCode VocbaseContext::authenticateRequest(bool* forceOpen) {
 ////////////////////////////////////////////////////////////////////////////////
 
 rest::ResponseCode VocbaseContext::basicAuthentication(const char* auth) {
-  if (useClusterAuthentication()) {
+  AuthResult result = _authentication->authInfo()->checkAuthentication(
-    std::string const expected = ServerState::instance()->getAuthentication();
-
-    if (expected.substr(6) != std::string(auth)) {
-      events::UnknownAuthenticationMethod(_request);
-      return rest::ResponseCode::UNAUTHORIZED;
-    }
-
-    std::string const up = StringUtils::decodeBase64(auth);
-    std::string::size_type n = up.find(':', 0);
-
-    if (n == std::string::npos || n == 0 || n + 1 > up.size()) {
-      LOG(TRACE) << "invalid authentication data found, cannot extract "
-                    "username/password";
-
-      events::UnknownAuthenticationMethod(_request);
-      return rest::ResponseCode::BAD;
-    }
-
-    _request->setUser(up.substr(0, n));
-
-    events::Authenticated(_request, rest::AuthenticationMethod::BASIC);
-    return rest::ResponseCode::OK;
-  }
-
-  AuthResult result = GeneralServerFeature::AUTH_INFO.checkAuthentication(
       AuthInfo::AuthType::BASIC, auth);
 
   _request->setUser(std::move(result._username));
@@ -265,7 +225,7 @@ rest::ResponseCode VocbaseContext::basicAuthentication(const char* auth) {
 ////////////////////////////////////////////////////////////////////////////////
 
 rest::ResponseCode VocbaseContext::jwtAuthentication(std::string const& auth) {
-  AuthResult result = GeneralServerFeature::AUTH_INFO.checkAuthentication(
+  AuthResult result = _authentication->authInfo()->checkAuthentication(
       AuthInfo::AuthType::JWT, auth);
 
   if (!result._authorized) {

arangod/RestServer/VocbaseContext.h

@@ -32,6 +32,7 @@
 #include "Rest/HttpRequest.h"
 #include "Rest/HttpResponse.h"
 #include "Rest/RequestContext.h"
+#include "GeneralServer/AuthenticationFeature.h"
 
 #include "Rest/GeneralRequest.h"
 #include "Rest/GeneralResponse.h"
@@ -44,7 +45,7 @@ class VocbaseContext : public arangodb::RequestContext {
   static double ServerSessionTtl;
 
  public:
-  VocbaseContext(GeneralRequest*, TRI_vocbase_t*, std::string const&);
+  VocbaseContext(GeneralRequest*, TRI_vocbase_t*);
   ~VocbaseContext();
 
  public:
@@ -53,9 +54,6 @@ class VocbaseContext : public arangodb::RequestContext {
  public:
   rest::ResponseCode authenticate() override final;
 
- private:
-  bool useClusterAuthentication() const;
-
  private:
   //////////////////////////////////////////////////////////////////////////////
   /// @brief checks the authentication (basic)
@@ -78,7 +76,8 @@ class VocbaseContext : public arangodb::RequestContext {
 
  private:
   TRI_vocbase_t* _vocbase;
-  std::string const _jwtSecret;
+  AuthenticationFeature* _authentication;
+
 };
 }
 

arangod/RestServer/arangod.cpp

@@ -42,6 +42,7 @@
 #include "ApplicationFeatures/VersionFeature.h"
 #include "Basics/ArangoGlobalContext.h"
 #include "Cluster/ClusterFeature.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "GeneralServer/GeneralServerFeature.h"
 #include "Logger/LoggerBufferFeature.h"
 #include "Logger/LoggerFeature.h"
@@ -107,15 +108,16 @@ static int runServer(int argc, char** argv) {
 
   std::vector<std::string> nonServerFeatures = {
       "Action",                 "Affinity",     "Agency",
-      "Cluster",    "Daemon",        "Dispatcher",
+      "Authentication",         "Cluster",      "Daemon",
-      "FoxxQueues", "GeneralServer", "LoggerBufferFeature",
+      "Dispatcher",             "FoxxQueues",   "GeneralServer",
-      "Server",     "Scheduler",     "SslServer",
+      "LoggerBufferFeature",    "Server",       "Scheduler",
-      "Statistics", "Supervisor"};
+      "SslServer",              "Statistics",   "Supervisor"};
 
   int ret = EXIT_FAILURE;
 
   server.addFeature(new ActionFeature(&server));
   server.addFeature(new AgencyFeature(&server));
+  server.addFeature(new AuthenticationFeature(&server));
   server.addFeature(new BootstrapFeature(&server));
   server.addFeature(new CheckVersionFeature(&server, &ret, nonServerFeatures));
   server.addFeature(new ClusterFeature(&server));

arangod/Scheduler/EventLoop.h

@@ -27,7 +27,7 @@
 
 #include "Basics/Common.h"
 
-#include <boost/asio.hpp>
+#include "Basics/asio-helper.h"
 
 namespace arangodb {
 namespace rest {

arangod/Scheduler/Scheduler.h

@@ -27,9 +27,9 @@
 
 #include "Basics/Common.h"
 
-#include <boost/asio.hpp>
 #include <boost/asio/steady_timer.hpp>
 
+#include "Basics/asio-helper.h"
 #include "Basics/Mutex.h"
 #include "Basics/MutexLocker.h"
 #include "Basics/socket-utils.h"

arangod/Scheduler/SchedulerFeature.h

@@ -26,7 +26,7 @@
 
 #include "ApplicationFeatures/ApplicationFeature.h"
 
-#include <boost/asio.hpp>
+#include "Basics/asio-helper.h"
 
 namespace arangodb {
 namespace rest {

arangod/Utils/Transaction.cpp

@@ -2635,12 +2635,14 @@ OperationResult Transaction::truncate(std::string const& collectionName,
 /// @brief remove all documents in a collection, coordinator
 ////////////////////////////////////////////////////////////////////////////////
 
+#ifndef USE_ENTERPRISE
 OperationResult Transaction::truncateCoordinator(std::string const& collectionName,
                                                  OperationOptions& options) {
   return OperationResult(
       arangodb::truncateCollectionOnCoordinator(_vocbase->name(),
                                                 collectionName));
 }
+#endif
 
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief remove all documents in a collection, local

arangod/V8Server/v8-replication.cpp

@@ -23,6 +23,8 @@
 
 #include "v8-replication.h"
 #include "Basics/ReadLocker.h"
+#include "Cluster/ClusterComm.h"
+#include "Cluster/ClusterFeature.h"
 #include "Replication/InitialSyncer.h"
 #include "Rest/Version.h"
 #include "RestServer/ServerIdFeature.h"
@@ -185,6 +187,33 @@ static void JS_LastLoggerReplication(
   TRI_V8_TRY_CATCH_END
 }
 
+void addReplicationAuthentication(v8::Isolate* isolate,
+    v8::Handle<v8::Object> object,
+    TRI_replication_applier_configuration_t &config) {
+  bool hasUsernamePassword = false;
+  if (object->Has(TRI_V8_ASCII_STRING("username"))) {
+    if (object->Get(TRI_V8_ASCII_STRING("username"))->IsString()) {
+      hasUsernamePassword = true;
+      config._username =
+        TRI_ObjectToString(object->Get(TRI_V8_ASCII_STRING("username")));
+    }
+  }
+
+  if (object->Has(TRI_V8_ASCII_STRING("password"))) {
+    if (object->Get(TRI_V8_ASCII_STRING("password"))->IsString()) {
+      hasUsernamePassword = true;
+      config._password =
+        TRI_ObjectToString(object->Get(TRI_V8_ASCII_STRING("password")));
+    }
+  }
+  if (!hasUsernamePassword) {
+    auto cluster = application_features::ApplicationServer::getFeature<ClusterFeature>("Cluster");
+    if (cluster->isEnabled()) {
+      config._jwt = ClusterComm::instance()->jwt();
+    }
+  }
+}
+
 ////////////////////////////////////////////////////////////////////////////////
 /// @brief sync data from a remote master
 ////////////////////////////////////////////////////////////////////////////////
@@ -219,15 +248,6 @@ static void JS_SynchronizeReplication(
     database = vocbase->name();
   }
 
-  std::string username;
-  if (object->Has(TRI_V8_ASCII_STRING("username"))) {
-    username = TRI_ObjectToString(object->Get(TRI_V8_ASCII_STRING("username")));
-  }
-
-  std::string password;
-  if (object->Has(TRI_V8_ASCII_STRING("password"))) {
-    password = TRI_ObjectToString(object->Get(TRI_V8_ASCII_STRING("password")));
-  }
 
   std::unordered_map<std::string, bool> restrictCollections;
   if (object->Has(TRI_V8_ASCII_STRING("restrictCollections")) &&
@@ -273,8 +293,8 @@ static void JS_SynchronizeReplication(
   TRI_replication_applier_configuration_t config;
   config._endpoint = endpoint;
   config._database = database;
-  config._username = username;
+
-  config._password = password;
+  addReplicationAuthentication(isolate, object, config);
 
   if (object->Has(TRI_V8_ASCII_STRING("chunkSize"))) {
     if (object->Get(TRI_V8_ASCII_STRING("chunkSize"))->IsNumber()) {
@@ -457,20 +477,7 @@ static void JS_ConfigureApplierReplication(
         config._database = vocbase->name();
       }
     }
-
+    addReplicationAuthentication(isolate, object, config);
-    if (object->Has(TRI_V8_ASCII_STRING("username"))) {
-      if (object->Get(TRI_V8_ASCII_STRING("username"))->IsString()) {
-        config._username =
-            TRI_ObjectToString(object->Get(TRI_V8_ASCII_STRING("username")));
-      }
-    }
-
-    if (object->Has(TRI_V8_ASCII_STRING("password"))) {
-      if (object->Get(TRI_V8_ASCII_STRING("password"))->IsString()) {
-        config._password =
-            TRI_ObjectToString(object->Get(TRI_V8_ASCII_STRING("password")));
-      }
-    }
 
     if (object->Has(TRI_V8_ASCII_STRING("requestTimeout"))) {
       if (object->Get(TRI_V8_ASCII_STRING("requestTimeout"))->IsNumber()) {

arangod/V8Server/v8-vocbase.cpp

@@ -52,6 +52,7 @@
 #include "Cluster/ClusterInfo.h"
 #include "Cluster/ClusterMethods.h"
 #include "Cluster/ServerState.h"
+#include "GeneralServer/AuthenticationFeature.h"
 #include "GeneralServer/GeneralServerFeature.h"
 #include "ReadCache/GlobalRevisionCache.h"
 #include "Rest/Version.h"
@@ -940,7 +941,11 @@ static void JS_ReloadAuth(v8::FunctionCallbackInfo<v8::Value> const& args) {
     TRI_V8_THROW_EXCEPTION_USAGE("RELOAD_AUTH()");
   }
   
-  GeneralServerFeature::AUTH_INFO.outdate();
+  auto authentication = application_features::ApplicationServer::getFeature<AuthenticationFeature>(
+    "Authentication");
+  if (authentication->isEnabled()) {
+    authentication->authInfo()->outdate();
+  }
 
   TRI_V8_RETURN_TRUE();
   TRI_V8_TRY_CATCH_END
@@ -2639,6 +2644,11 @@ static void JS_TrustedProxies(v8::FunctionCallbackInfo<v8::Value> const& args) {
 
 static void JS_AuthenticationEnabled(
     v8::FunctionCallbackInfo<v8::Value> const& args) {
+  auto authentication = application_features::ApplicationServer::getFeature<AuthenticationFeature>(
+    "Authentication");
+
+  TRI_ASSERT(authentication != nullptr);
+
   // mop: one could argue that this is a function because this might be
   // changable on the fly at some time but the sad truth is server startup
   // order
@@ -2647,7 +2657,7 @@ static void JS_AuthenticationEnabled(
   v8::HandleScope scope(isolate);
 
   v8::Handle<v8::Boolean> result =
-      v8::Boolean::New(isolate, GeneralServerFeature::authenticationEnabled());
+      v8::Boolean::New(isolate, authentication->isEnabled());
 
   TRI_V8_RETURN(result);
   TRI_V8_TRY_CATCH_END

arangod/VocBase/AuthInfo.cpp

@@ -23,12 +23,11 @@
 
 #include "AuthInfo.h"
 
-#include <chrono>
-
 #include <velocypack/Builder.h>
 #include <velocypack/Iterator.h>
 #include <velocypack/velocypack-aliases.h>
 
+#include "Aql/Query.h"
 #include "Basics/ReadLocker.h"
 #include "Basics/VelocyPackHelper.h"
 #include "Basics/WriteLocker.h"
@@ -142,6 +141,16 @@ AuthLevel AuthEntry::canUseDatabase(std::string const& dbname) const {
   return it->second;
 }
 
+void AuthInfo::setJwtSecret(std::string const& jwtSecret) {
+  WRITE_LOCKER(writeLocker, _authJwtLock);
+  _jwtSecret = jwtSecret;
+  _authJwtCache.clear();
+}
+
+std::string AuthInfo::jwtSecret() {
+  return _jwtSecret;
+}
+
 void AuthInfo::clear() {
   _authInfo.clear();
   _authBasicCache.clear();
@@ -222,30 +231,40 @@ void AuthInfo::reload() {
     return;
   }
   
+  MUTEX_LOCKER(locker, _queryLock);
+  if (!_outdated) {
+    return;
+  }
+  std::string queryStr("FOR user IN _users RETURN user");
+  auto nullBuilder = std::make_shared<VPackBuilder>();
+  VPackBuilder options;
+  {
+    VPackObjectBuilder b(&options);
+  }
+  auto objectBuilder = std::make_shared<VPackBuilder>(options);
+  
+  arangodb::aql::Query query(false, vocbase, queryStr.c_str(),
+                             queryStr.length(), nullBuilder, objectBuilder,
+                             arangodb::aql::PART_MAIN);
+
   LOG(DEBUG) << "starting to load authentication and authorization information";
+  TRI_ASSERT(_queryRegistry != nullptr); 
+  auto queryResult = query.execute(_queryRegistry);
   
-  SingleCollectionTransaction trx(StandaloneTransactionContext::Create(vocbase),
+  if (queryResult.code != TRI_ERROR_NO_ERROR) {
-                                  TRI_COL_NAME_USERS, TRI_TRANSACTION_READ);
+    if (queryResult.code == TRI_ERROR_REQUEST_CANCELED ||
-
+        (queryResult.code == TRI_ERROR_QUERY_KILLED)) {
-  int res = trx.begin();
+      THROW_ARANGO_EXCEPTION(TRI_ERROR_REQUEST_CANCELED);
-
+    }
-  if (res != TRI_ERROR_NO_ERROR) {
+    _outdated = false;
-    LOG(ERR) << "cannot start transaction to load authentication";
     return;
   }
   
-  OperationResult users =
+  VPackSlice usersSlice = queryResult.result->slice();
-      trx.all(TRI_COL_NAME_USERS, 0, UINT64_MAX, OperationOptions());
 
-  trx.finish(users.code);
+  if (usersSlice.isNone()) {
-
+    THROW_ARANGO_EXCEPTION(TRI_ERROR_OUT_OF_MEMORY);
-  if (users.failed()) {
-    LOG(ERR) << "cannot read users from _users collection";
-    return;
   }
-
-  auto usersSlice = users.slice();
-
   if (!usersSlice.isArray()) {
     LOG(ERR) << "cannot read users from _users collection";
     return;
@@ -401,11 +420,36 @@ AuthResult AuthInfo::checkAuthenticationBasic(std::string const& secret) {
   return result;
 }
 
-AuthResult AuthInfo::checkAuthenticationJWT(std::string const& secret) {
+AuthResult AuthInfo::checkAuthenticationJWT(std::string const& jwt) {
-  std::vector<std::string> const parts = StringUtils::split(secret, '.');
+  try {
+    READ_LOCKER(readLocker, _authJwtLock);
+    auto result = _authJwtCache.get(jwt);
+    if (result._expires) {
+      std::chrono::system_clock::time_point now =
+        std::chrono::system_clock::now();
+
+      if (now >= result._expireTime) {
+        readLocker.unlock();
+        WRITE_LOCKER(writeLocker, _authJwtLock);
+        result = _authJwtCache.get(jwt);
+        if (result._expires && now >= result._expireTime) {
+          try {
+            _authJwtCache.remove(jwt);
+          } catch (std::range_error const& e) {
+          }
+        }
+        return AuthResult();
+      }
+    }
+    return (AuthResult) result;
+  } catch (std::range_error const& e) {
+    // mop: not found
+  }
+
+  std::vector<std::string> const parts = StringUtils::split(jwt, '.');
 
   if (parts.size() != 3) {
-    LOG(DEBUG) << "Secret contains " << parts.size() << " parts";
+    LOG(TRACE) << "Secret contains " << parts.size() << " parts";
     return AuthResult();
   }
   
@@ -413,29 +457,26 @@ AuthResult AuthInfo::checkAuthenticationJWT(std::string const& secret) {
   std::string const& body = parts[1];
   std::string const& signature = parts[2];
 
+  if (!validateJwtHeader(header)) {
+    LOG(TRACE) << "Couldn't validate jwt header " << header;
+    return AuthResult();
+  }
+
+  AuthJwtResult result = validateJwtBody(body);
+  if (!result._authorized) {
+    LOG(TRACE) << "Couldn't validate jwt body " << body;
+    return AuthResult();
+  }
+
   std::string const message = header + "." + body;
 
-  if (!validateJwtHeader(header)) {
-    LOG(DEBUG) << "Couldn't validate jwt header " << header;
-    return AuthResult();
-  }
-
-  std::string username;
-  if (!validateJwtBody(body, &username)) {
-    LOG(DEBUG) << "Couldn't validate jwt body " << body;
-    return AuthResult();
-  }
-
   if (!validateJwtHMAC256Signature(message, signature)) {
-    LOG(DEBUG) << "Couldn't validate jwt signature " << signature;
+    LOG(TRACE) << "Couldn't validate jwt signature " << signature << " " << _jwtSecret;
     return AuthResult();
   }
-
+  WRITE_LOCKER(writeLocker, _authJwtLock);
-  AuthResult result;
+  _authJwtCache.put(jwt, result);
-  result._username = username;
+  return (AuthResult) result;
-  result._authorized = true;
-
-  return result;
 }
 
 std::shared_ptr<VPackBuilder> AuthInfo::parseJson(std::string const& str,
@@ -491,40 +532,47 @@ bool AuthInfo::validateJwtHeader(std::string const& header) {
   return true;
 }
 
-bool AuthInfo::validateJwtBody(std::string const& body, std::string* username) {
+AuthJwtResult AuthInfo::validateJwtBody(std::string const& body) {
   std::shared_ptr<VPackBuilder> bodyBuilder =
       parseJson(StringUtils::decodeBase64(body), "jwt body");
+  AuthJwtResult authResult;
   if (bodyBuilder.get() == nullptr) {
-    return false;
+    return authResult;
   }
 
   VPackSlice const bodySlice = bodyBuilder->slice();
   if (!bodySlice.isObject()) {
-    return false;
+    return authResult;
   }
 
   VPackSlice const issSlice = bodySlice.get("iss");
   if (!issSlice.isString()) {
-    return false;
+    return authResult;
   }
 
   if (issSlice.copyString() != "arangodb") {
-    return false;
+    return authResult;
   }
   
+  if (bodySlice.hasKey("preferred_username")) {
     VPackSlice const usernameSlice = bodySlice.get("preferred_username");
     if (!usernameSlice.isString()) {
-    return false;
+      return authResult;
+    }
+    authResult._username = usernameSlice.copyString();
+  } else if (bodySlice.hasKey("server_id")) {
+    // mop: hmm...nothing to do here :D
+    // authResult._username = "root";
+  } else {
+    return authResult;
   }
-
-  *username = usernameSlice.copyString();
 
   // mop: optional exp (cluster currently uses non expiring jwts)
   if (bodySlice.hasKey("exp")) {
     VPackSlice const expSlice = bodySlice.get("exp");
 
     if (!expSlice.isNumber()) {
-      return false;
+      return authResult;
     }
 
     std::chrono::system_clock::time_point expires(
@@ -533,19 +581,67 @@ bool AuthInfo::validateJwtBody(std::string const& body, std::string* username) {
         std::chrono::system_clock::now();
 
     if (now >= expires) {
-      return false;
+      return authResult;
     }
+    authResult._expires = true;
+    authResult._expireTime = expires;
   }
-  return true;
+
+  authResult._authorized = true;
+  return authResult;
 }
 
 bool AuthInfo::validateJwtHMAC256Signature(std::string const& message,
                                            std::string const& signature) {
   std::string decodedSignature = StringUtils::decodeBase64U(signature);
   
-  std::string const& jwtSecret = GeneralServerFeature::getJwtSecret();
+  return verifyHMAC(_jwtSecret.c_str(), _jwtSecret.length(), message.c_str(),
-  return verifyHMAC(jwtSecret.c_str(), jwtSecret.length(), message.c_str(),
                     message.length(), decodedSignature.c_str(),
                     decodedSignature.length(),
                     SslInterface::Algorithm::ALGORITHM_SHA256);
 }
+
+std::string AuthInfo::generateRawJwt(VPackBuilder const& bodyBuilder) {
+  VPackBuilder headerBuilder;
+  {
+    VPackObjectBuilder h(&headerBuilder);
+    headerBuilder.add("alg", VPackValue("HS256"));
+    headerBuilder.add("typ", VPackValue("JWT"));
+  }
+
+  std::string fullMessage(StringUtils::encodeBase64(headerBuilder.toJson()) +
+                          "." +
+                          StringUtils::encodeBase64(bodyBuilder.toJson()));
+
+  std::string signature =
+      sslHMAC(_jwtSecret.c_str(), _jwtSecret.length(), fullMessage.c_str(),
+              fullMessage.length(), SslInterface::Algorithm::ALGORITHM_SHA256);
+
+  return fullMessage + "." + StringUtils::encodeBase64U(signature);
+}
+
+std::string AuthInfo::generateJwt(VPackBuilder const& payload) {
+  if (!payload.slice().isObject()) {
+    std::string error = "Need an object to generate a JWT. Got: ";
+    error += payload.slice().typeName();
+    throw std::runtime_error(error);
+  }
+  bool hasIss = payload.slice().hasKey("iss");
+  bool hasIat = payload.slice().hasKey("iat");
+  VPackBuilder bodyBuilder;
+  if (hasIss && hasIat) {
+    bodyBuilder = payload;
+  } else {
+    VPackObjectBuilder p(&bodyBuilder);
+    if (!hasIss) {
+      bodyBuilder.add("iss", VPackValue("arangodb"));
+    }
+    if (!hasIat) {
+      bodyBuilder.add("iat", VPackValue(TRI_microtime() / 1000));
+    }
+    for (auto const& obj : VPackObjectIterator(payload.slice())) {
+      bodyBuilder.add(obj.key.copyString(), obj.value);
+    }
+  }
+  return generateRawJwt(bodyBuilder);
+}

arangod/VocBase/AuthInfo.h

@@ -26,10 +26,14 @@
 
 #include "Basics/Common.h"
 
+#include <chrono>
+
 #include <velocypack/Builder.h>
 #include <velocypack/velocypack-aliases.h>
 
+#include "Aql/QueryRegistry.h"
 #include "Basics/ReadWriteLock.h"
+#include "Basics/LruCache.h"
 
 namespace arangodb {
 namespace velocypack {
@@ -90,6 +94,13 @@ class AuthResult {
   bool _mustChange;
 };
 
+class AuthJwtResult: public AuthResult {
+ public:
+  AuthJwtResult() : AuthResult(), _expires(false) {}
+  bool _expires;
+  std::chrono::system_clock::time_point _expireTime;
+};
+
 class AuthInfo {
  public:
   enum class AuthType {
@@ -97,9 +108,18 @@ class AuthInfo {
   };
 
  public:
-  AuthInfo() : _outdated(true) {}
+  AuthInfo()
+    : _outdated(true),
+    _authJwtCache(16384),
+    _jwtSecret(""),
+    _queryRegistry(nullptr) {
+  }
   
  public:
+  void setQueryRegistry(aql::QueryRegistry* registry) {
+    TRI_ASSERT(registry != nullptr);
+    _queryRegistry = registry;
+  };
   void outdate() { _outdated = true; }
 
   AuthResult checkPassword(std::string const& username,
@@ -111,6 +131,11 @@ class AuthInfo {
   AuthLevel canUseDatabase(std::string const& username,
                            std::string const& dbname);
   
+  void setJwtSecret(std::string const&);
+  std::string jwtSecret();
+  std::string generateJwt(VPackBuilder const&);
+  std::string generateRawJwt(VPackBuilder const&);
+ 
  private:
   void reload();
   void clear();
@@ -120,16 +145,21 @@ class AuthInfo {
   AuthResult checkAuthenticationBasic(std::string const& secret);
   AuthResult checkAuthenticationJWT(std::string const& secret);
   bool validateJwtHeader(std::string const&);
-  bool validateJwtBody(std::string const&, std::string*);
+  AuthJwtResult validateJwtBody(std::string const&);
   bool validateJwtHMAC256Signature(std::string const&, std::string const&);
   std::shared_ptr<VPackBuilder> parseJson(std::string const&, std::string const&);
 
  private:
   basics::ReadWriteLock _authInfoLock;
+  basics::ReadWriteLock _authJwtLock;
+  Mutex _queryLock;
   std::atomic<bool> _outdated;
 
   std::unordered_map<std::string, arangodb::AuthEntry> _authInfo;
   std::unordered_map<std::string, arangodb::AuthResult> _authBasicCache;
+  arangodb::basics::LruCache<std::string, arangodb::AuthJwtResult> _authJwtCache;
+  std::string _jwtSecret;
+  aql::QueryRegistry* _queryRegistry;
 };
 }
 

arangod/VocBase/replication-applier.cpp

@@ -128,17 +128,26 @@ static int LoadConfiguration(TRI_vocbase_t* vocbase,
 
   // read username / password
   value = slice.get("username");
-
+  bool hasUsernamePassword = false;
   if (value.isString()) {
+    hasUsernamePassword = true;
     config->_username = value.copyString();
   }
 
   value = slice.get("password");
 
   if (value.isString()) {
+    hasUsernamePassword = true;
     config->_password = value.copyString();
   }
 
+  if (!hasUsernamePassword) {
+    value = slice.get("jwt");
+    if (value.isString()) {
+      config->_jwt = value.copyString();
+    }
+  }
+
   value = slice.get("requestTimeout");
 
   if (value.isNumber()) {
@@ -445,6 +454,7 @@ TRI_replication_applier_configuration_t::
       _database(),
       _username(),
       _password(),
+      _jwt(),
       _requestTimeout(600.0),
       _connectTimeout(10.0),
       _ignoreErrors(0),
@@ -489,12 +499,19 @@ void TRI_replication_applier_configuration_t::toVelocyPack(
   if (!_database.empty()) {
     builder.add("database", VPackValue(_database));
   }
+  
+  bool hasUsernamePassword = false;
   if (!_username.empty()) {
+    hasUsernamePassword = true;
     builder.add("username", VPackValue(_username));
   }
   if (includePassword) {
+    hasUsernamePassword = true;
     builder.add("password", VPackValue(_password));
   }
+  if (!hasUsernamePassword && !_jwt.empty()) {
+    builder.add("jwt", VPackValue(_jwt));
+  }
 
   builder.add("requestTimeout", VPackValue(_requestTimeout));
   builder.add("connectTimeout", VPackValue(_connectTimeout));
@@ -789,6 +806,7 @@ void TRI_replication_applier_configuration_t::update(
   _database = src->_database;
   _username = src->_username;
   _password = src->_password;
+  _jwt = src->_jwt;
   _requestTimeout = src->_requestTimeout;
   _connectTimeout = src->_connectTimeout;
   _ignoreErrors = src->_ignoreErrors;

arangod/VocBase/replication-applier.h

@@ -46,6 +46,7 @@ class TRI_replication_applier_configuration_t {
   std::string _database;
   std::string _username;
   std::string _password;
+  std::string _jwt;
   double _requestTimeout;
   double _connectTimeout;
   uint64_t _ignoreErrors;

js/apps/system/_admin/aardvark/APP/aardvark.js

@@ -48,10 +48,12 @@ module.exports = router;
 router.get('/config.js', function (req, res) {
   const scriptName = req.get('x-script-name');
   const basePath = req.trustProxy && scriptName || '';
+  const isEnterprise = internal.isEnterprise();
   res.send(
     `var frontendConfig = ${JSON.stringify({
       basePath: basePath,
       db: req.database,
+      isEnterprise: isEnterprise,
       authenticationEnabled: internal.authenticationEnabled(),
       isCluster: cluster.isCluster()
     })}`

js/apps/system/_admin/aardvark/APP/frontend/build/index-min.html

@@ -1005,6 +1005,7 @@ if (list.length > 0) {
         </div>
       </div> <% graphs.forEach(function(graph) {
         var graphName = graph.get("_key");
+        var isSmart = graph.get("isSmart");
       %> <div class="tile tile-graph pure-u-1-1 pure-u-sm-1-2 pure-u-md-1-3 pure-u-lg-1-4 pure-u-xl-1-6" id="<%=graphName %>_tile">
         <div class="paddingBox">
           <div class="borderBox"></div>
@@ -1013,7 +1014,7 @@ if (list.length > 0) {
           </div>
           <span class="icon_arangodb_edge5 tile-icon"></span>
           <span class="icon_arangodb_edge5 icon_arangodb_edge5-2 tile-icon"></span>
-          <div class="tileBadge"></div>
+          <div class="tileBadge"> <% if (isSmart === true) { %> <span><div class="corneredBadge inProgress">Smart</div></span> <% } %> </div>
           <h5 class="collectionName"><%=graphName %></h5>
         </div>
       </div> <%});%> </div>
@@ -2740,4 +2741,4 @@ var cutByResolution = function (str) {
   </div>
 
   <div id="workMonitorContent" class="innerContent">
-  </div></script></head><body><nav class="navbar" style="display: none"><div class="primary"><div class="navlogo"><a class="logo big" href="#"><img id="ArangoDBLogo" class="arangodbLogo" src="img/arangodb_logo_big.svg"></a><a class="logo small" href="#"><img class="arangodbLogo" src="img/arangodb_logo_small.png"></a><a class="version"><span>VERSION: </span><span id="currentVersion"></span></a></div><div class="statmenu" id="statisticBar"></div><div class="navmenu" id="navigationBar"></div></div></nav><div id="modalPlaceholder"></div><div class="bodyWrapper" style="display: none"><div class="centralRow"><div id="navbar2" class="navbarWrapper secondary"><div class="subnavmenu" id="subNavigationBar"></div></div><div class="resizecontainer contentWrapper"><div id="loadingScreen" class="loadingScreen" style="display: none"><i class="fa fa-circle-o-notch fa-spin fa-3x fa-fw margin-bottom"></i> <span class="sr-only">Loading...</span></div><div id="content" class="centralContent"></div><footer class="footer"><div id="footerBar"></div></footer></div></div></div><div id="progressPlaceholder" style="display:none"></div><div id="spotlightPlaceholder" style="display:none"></div><div id="graphSettingsContent" style="display: none"></div><div id="offlinePlaceholder" style="display:none"><div class="offline-div"><div class="pure-u"><div class="pure-u-1-4"></div><div class="pure-u-1-2 offline-window"><div class="offline-header"><h3>You have been disconnected from the server</h3></div><div class="offline-body"><p>The connection to the server has been lost. The server may be under heavy load.</p><p>Trying to reconnect in <span id="offlineSeconds">10</span> seconds.</p><p class="animation_state"><span><button class="button-success">Reconnect now</button></span></p></div></div><div class="pure-u-1-4"></div></div></div></div><div class="arangoFrame" style=""><div class="outerDiv"><div class="innerDiv"></div></div></div><script src="libs.js?version=1476457280679"></script><script src="app.js?version=1476457280679"></script></body></html>
+  </div></script></head><body><nav class="navbar" style="display: none"><div class="primary"><div class="navlogo"><a class="logo big" href="#"><img id="ArangoDBLogo" class="arangodbLogo" src="img/arangodb_logo_big.svg"></a><a class="logo small" href="#"><img class="arangodbLogo" src="img/arangodb_logo_small.png"></a><a class="version"><span>VERSION: </span><span id="currentVersion"></span></a></div><div class="statmenu" id="statisticBar"></div><div class="navmenu" id="navigationBar"></div></div></nav><div id="modalPlaceholder"></div><div class="bodyWrapper" style="display: none"><div class="centralRow"><div id="navbar2" class="navbarWrapper secondary"><div class="subnavmenu" id="subNavigationBar"></div></div><div class="resizecontainer contentWrapper"><div id="loadingScreen" class="loadingScreen" style="display: none"><i class="fa fa-circle-o-notch fa-spin fa-3x fa-fw margin-bottom"></i> <span class="sr-only">Loading...</span></div><div id="content" class="centralContent"></div><footer class="footer"><div id="footerBar"></div></footer></div></div></div><div id="progressPlaceholder" style="display:none"></div><div id="spotlightPlaceholder" style="display:none"></div><div id="graphSettingsContent" style="display: none"></div><div id="offlinePlaceholder" style="display:none"><div class="offline-div"><div class="pure-u"><div class="pure-u-1-4"></div><div class="pure-u-1-2 offline-window"><div class="offline-header"><h3>You have been disconnected from the server</h3></div><div class="offline-body"><p>The connection to the server has been lost. The server may be under heavy load.</p><p>Trying to reconnect in <span id="offlineSeconds">10</span> seconds.</p><p class="animation_state"><span><button class="button-success">Reconnect now</button></span></p></div></div><div class="pure-u-1-4"></div></div></div></div><div class="arangoFrame" style=""><div class="outerDiv"><div class="innerDiv"></div></div></div><script src="libs.js?version=1476701130414"></script><script src="app.js?version=1476701130414"></script></body></html>

js/apps/system/_admin/aardvark/APP/frontend/js/routers/versionCheck.js

@@ -138,15 +138,6 @@
         var currentVersion =
         window.versionHelper.fromString(data.version);
 
-        if (data.license) {
-          window.frontendConfig.license = data.license;
-          if (data.license !== 'community') {
-            $('#ArangoDBLogo').attr('src', 'img/arangodb_logo_alt.svg');
-          } else {
-            $('.enterprise-menu').show();
-          }
-        }
-
         $('.navbar #currentVersion').html(
           ' ' + data.version.substr(0, 5) + '<i class="fa fa-exclamation-circle"></i>'
         );

js/apps/system/_admin/aardvark/APP/frontend/js/views/graphManagementView.js

@@ -201,6 +201,82 @@
       });
     },
 
+    toggleSmartGraph: function () {
+      var i;
+      var self = this;
+
+      if ($('#new-is_smart').is(':checked') === true) {
+        for (i = 0; i < this.counter; i++) {
+          $('#newEdgeDefinitions' + i).select2({
+            tags: []
+          });
+          self.cachedNewEdgeDefinitions = $('#newEdgeDefinitions' + i).select2('data');
+          self.cachedNewEdgeDefinitionsState = $('#newEdgeDefinitions' + i).attr('disabled');
+          $('#newEdgeDefinitions' + i).select2('data', '');
+          $('#newEdgeDefinitions' + i).attr('disabled', false);
+
+          $('#fromCollections' + i).select2({
+            tags: []
+          });
+          self.cachedFromCollections = $('#fromCollections' + i).select2('data');
+          self.cachedFromCollectionsState = $('#fromCollections' + i).attr('disabled');
+          $('#fromCollections' + i).select2('data', '');
+          $('#fromCollections' + i).attr('disabled', false);
+
+          $('#toCollections' + i).select2({
+            tags: []
+          });
+          self.cachedToCollections = $('#toCollections' + i).select2('data');
+          self.cachedToCollectionsState = $('#toCollections' + i).attr('disabled');
+          $('#toCollections' + i).select2('data', '');
+          $('#toCollections' + i).attr('disabled', false);
+
+          $('#newVertexCollections' + i).select2({
+            tags: []
+          });
+          self.cachedNewVertexCollections = $('#newVertexCollections' + i).select2('data');
+          self.cachedNewVertexCollectionsState = $('#newVertexCollections' + i).attr('disabled');
+          $('#newVertexCollections' + i).select2('data', '');
+          $('#newVertexCollections' + i).attr('disabled', false);
+        }
+      } else {
+        var collList = []; var collections = this.options.collectionCollection.models;
+
+        collections.forEach(function (c) {
+          if (c.get('isSystem')) {
+            return;
+          }
+          collList.push(c.id);
+        });
+
+        for (i = 0; i < this.counter; i++) {
+          $('#newEdgeDefinitions' + i).select2({
+            tags: this.eCollList
+          });
+          $('#newEdgeDefinitions' + i).select2('data', self.cachedNewEdgeDefinitions);
+          $('#newEdgeDefinitions' + i).attr('disabled', self.cachedNewEdgeDefinitionsState);
+
+          $('#fromCollections' + i).select2({
+            tags: collList
+          });
+          $('#fromCollections' + i).select2('data', self.cachedFromCollections);
+          $('#fromCollections' + i).attr('disabled', self.cachedFromCollectionsState);
+
+          $('#toCollections' + i).select2({
+            tags: collList
+          });
+          $('#toCollections' + i).select2('data', self.cachedToCollections);
+          $('#toCollections' + i).attr('disabled', self.cachedToCollectionsState);
+
+          $('#newVertexCollections' + i).select2({
+            tags: collList
+          });
+          $('#newVertexCollections' + i).select2('data', self.cachedNewVertexCollections);
+          $('#newVertexCollections' + i).attr('disabled', self.cachedNewVertexCollectionsState);
+        }
+      }
+    },
+
     render: function (name, refetch) {
       var self = this;
       this.collection.fetch({
@@ -226,6 +302,7 @@
           self.events['click .graphViewer-icon-button'] = self.addRemoveDefinition.bind(self);
           self.events['click #graphTab a'] = self.toggleTab.bind(self);
           self.events['click .createExampleGraphs'] = self.createExampleGraphs.bind(self);
+          self.events['click #new-is_smart'] = self.toggleSmartGraph.bind(self);
           self.events['focusout .select2-search-field input'] = function (e) {
             if ($('.select2-drop').is(':visible')) {
               if (!$('#select2-search-field input').is(':focus')) {
@@ -733,7 +810,7 @@
         )
       );
 
-      if (window.frontendConfig.license === 'enterprise') {
+      if (window.frontendConfig.isEnterprise === false) {
         var advanced = {};
         var advancedTableContent = [];
 
@@ -742,7 +819,7 @@
             'new-is_smart',
             'Smart Graph',
             true,
-            'Do you want to create a smart graph?',
+            'Create a Smart Graph? Edge and vertex collections will be automatically generated. They are not allowed to be present before graph creation.',
             false
           )
         );

js/apps/system/_admin/aardvark/APP/frontend/js/views/navigationView.js

@@ -107,6 +107,13 @@
 
       self.resize();
 
+      console.log(window.frontendConfig);
+      if (window.frontendConfig.isEnterprise === true) {
+        $('#ArangoDBLogo').attr('src', 'img/arangodb_logo_alt.svg');
+      } else {
+        $('.enterprise-menu').show();
+      }
+
       return this;
     },
 

js/common/bootstrap/modules/internal.js

@@ -304,6 +304,11 @@ global.DEFINE_MODULE('internal', (function () {
     delete global.SYS_DOWNLOAD;
   }
   
+  if (global.SYS_CLUSTER_DOWNLOAD) {
+    exports.clusterDownload = global.SYS_CLUSTER_DOWNLOAD;
+    delete global.SYS_CLUSTER_DOWNLOAD;
+  }
+
   // //////////////////////////////////////////////////////////////////////////////
   // / @brief whether or not Statistics are enabled
   // //////////////////////////////////////////////////////////////////////////////

js/server/modules/@arangodb/arango-collection.js

@@ -124,30 +124,11 @@ ArangoCollection.prototype.toArray = function () {
 
 ArangoCollection.prototype.truncate = function () {
   var cluster = require('@arangodb/cluster');
-
   if (cluster.isCoordinator()) {
     if (this.status() === ArangoCollection.STATUS_UNLOADED) {
       this.load();
     }
-    var dbName = require('internal').db._name();
-    var shards = cluster.shardList(dbName, this.name());
-    var coord = { coordTransactionID: ArangoClusterComm.getId() };
-    var options = { coordTransactionID: coord.coordTransactionID, timeout: 360 };
-
-    shards.forEach(function (shard) {
-      ArangoClusterComm.asyncRequest('put',
-        'shard:' + shard,
-        dbName,
-        '/_api/collection/' + encodeURIComponent(shard) + '/truncate',
-        '',
-        { },
-        options);
-    });
-
-    cluster.wait(coord, shards.length);
-    return;
   }
-
   return this.TRUNCATE();
 };
 

js/server/modules/@arangodb/cluster.js

@@ -33,7 +33,7 @@ var arangodb = require('@arangodb');
 var ArangoCollection = arangodb.ArangoCollection;
 var ArangoError = arangodb.ArangoError;
 var errors = require("internal").errors;
-var request = require('@arangodb/request').request;
+var request = require('@arangodb/request').clusterRequest;
 var wait = require('internal').wait;
 var _ = require('lodash');
 
@@ -1119,7 +1119,7 @@ function synchronizeOneShard (database, shard, planId, leader) {
             shard, 300);
           console.debug('lockJobId:', lockJobId);
         } catch (err1) {
-          console.error('synchronizeOneShard: exception in startReadLockOnLeader:', err1);
+          console.error('synchronizeOneShard: exception in startReadLockOnLeader:', err1, err1.stack);
         }
         finally {
           cancelBarrier(ep, database, sy.barrierId);

js/server/modules/@arangodb/replication.js

@@ -29,7 +29,12 @@
 
 var internal = require('internal');
 var endpointToURL = require('@arangodb/cluster').endpointToURL;
-var request = require('@arangodb/request').request;
+var request;
+if (ArangoServerState.role() == 'PRIMARY') {
+  request = require('@arangodb/request').clusterRequest;
+} else {
+  request = require('@arangodb/request').request;
+}
 
 var logger = { };
 var applier = { };

js/server/modules/@arangodb/request.js

@@ -0,0 +1,155 @@
+/* jshint sub: true */
+/* global exports: true */
+'use strict';
+
+// //////////////////////////////////////////////////////////////////////////////
+// / @brief node-request-style HTTP requests
+// /
+// / @file
+// /
+// / DISCLAIMER
+// /
+// / Copyright 2015 triAGENS GmbH, Cologne, Germany
+// /
+// / Licensed under the Apache License, Version 2.0 (the "License")
+// / you may not use this file except in compliance with the License.
+// / You may obtain a copy of the License at
+// /
+// /     http://www.apache.org/licenses/LICENSE-2.0
+// /
+// / Unless required by applicable law or agreed to in writing, software
+// / distributed under the License is distributed on an "AS IS" BASIS,
+// / WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// / See the License for the specific language governing permissions and
+// / limitations under the License.
+// /
+// / Copyright holder is triAGENS GmbH, Cologne, Germany
+// /
+// / @author Andreas Streichardt <andreas@arangodb.com>
+// / @author Copyright 2016, ArangoDB GmbH, Cologne, Germany
+// //////////////////////////////////////////////////////////////////////////////
+
+const internal = require('internal');
+const Buffer = require('buffer').Buffer;
+const extend = require('lodash').extend;
+const url = require('url');
+const is = require('@arangodb/is');
+const httperr = require('http-errors');
+const querystring = require('querystring');
+const qs = require('qs');
+
+let request = require('../../../common/modules/@arangodb/request.js');
+let Response = request.Response;
+
+exports = request;
+
+function querystringify (query, useQuerystring) {
+  if (!query) {
+    return '';
+  }
+  if (typeof query === 'string') {
+    return query.charAt(0) === '?' ? query.slice(1) : query;
+  }
+  return (useQuerystring ? querystring : qs).stringify(query)
+    .replace(/[!'()*]/g, function (c) {
+      // Stricter RFC 3986 compliance
+      return '%' + c.charCodeAt(0).toString(16);
+    });
+}
+
+function clusterRequest(req) {
+  if (typeof req === 'string') {
+    req = {url: req, method: 'GET'};
+  }
+
+  let path = req.url || req.uri;
+  if (!path) {
+    throw new Error('Request URL must not be empty.');
+  }
+
+  let pathObj = typeof path === 'string' ? url.parse(path) : path;
+  if (pathObj.auth) {
+    let auth = pathObj.auth.split(':');
+    req = extend({
+      auth: {
+        username: decodeURIComponent(auth[0]),
+        password: decodeURIComponent(auth[1])
+      }
+    }, req);
+    delete pathObj.auth;
+  }
+  let query = typeof req.qs === 'string' ? req.qs : querystringify(req.qs, req.useQuerystring);
+  if (query) {
+    pathObj.search = query;
+  }
+  path = url.format(pathObj);
+
+  let contentType;
+  let body = req.body;
+  if (req.json) {
+    body = JSON.stringify(body);
+    contentType = 'application/json';
+  } else if (typeof body === 'string') {
+    contentType = 'text/plain; charset=utf-8';
+  } else if (typeof body === 'object' && body instanceof Buffer) {
+    contentType = 'application/octet-stream';
+  } else if (!body) {
+    if (req.form) {
+      contentType = 'application/x-www-form-urlencoded';
+      body = typeof req.form === 'string' ? req.form : querystringify(req.form, req.useQuerystring);
+    } else if (req.formData) {
+      // contentType = 'multipart/form-data'
+      // body = formData(req.formData)
+      throw new Error('Multipart form encoding is currently not supported.');
+    } else if (req.multipart) {
+      // contentType = 'multipart/related'
+      // body = multipart(req.multipart)
+      throw new Error('Multipart encoding is currently not supported.');
+    }
+  }
+
+  const headers = {};
+
+  if (contentType) {
+    headers['content-type'] = contentType;
+  }
+
+  if (req.headers) {
+    Object.keys(req.headers).forEach(function (name) {
+      headers[name.toLowerCase()] = req.headers[name];
+    });
+  }
+
+  if (req.auth) {
+    headers['authorization'] = ( // eslint-disable-line dot-notation
+      req.auth.bearer ?
+        'Bearer ' + req.auth.bearer :
+        'Basic ' + new Buffer(
+          req.auth.username + ':' +
+          req.auth.password
+        ).toString('base64')
+    );
+  }
+
+  let options = {
+    method: (req.method || 'get').toUpperCase(),
+    headers: headers,
+    returnBodyAsBuffer: true,
+    returnBodyOnError: req.returnBodyOnError !== false
+  };
+  if (is.existy(req.timeout)) {
+    options.timeout = req.timeout;
+  }
+  if (is.existy(req.followRedirect)) {
+    options.followRedirects = req.followRedirect; // [sic] node-request compatibility
+  }
+  if (is.existy(req.maxRedirects)) {
+    options.maxRedirects = req.maxRedirects;
+  } else {
+    options.maxRedirects = 10;
+  }
+  let result = internal.clusterDownload(path, body, options);
+  return new Response(result, req.encoding, req.json);
+}
+exports.clusterRequest = clusterRequest;
+module.exports = exports;

lib/Basics/LruCache.h

@@ -0,0 +1,133 @@
+////////////////////////////////////////////////////////////////////////////////
+/// DISCLAIMER
+///
+/// Copyright 2014-2016 ArangoDB GmbH, Cologne, Germany
+/// Copyright 2004-2014 triAGENS GmbH, Cologne, Germany
+///
+/// Licensed under the Apache License, Version 2.0 (the "License");
+/// you may not use this file except in compliance with the License.
+/// You may obtain a copy of the License at
+///
+///     http://www.apache.org/licenses/LICENSE-2.0
+///
+/// Unless required by applicable law or agreed to in writing, software
+/// distributed under the License is distributed on an "AS IS" BASIS,
+/// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+/// See the License for the specific language governing permissions and
+/// limitations under the License.
+///
+/// Copyright holder is ArangoDB GmbH, Cologne, Germany
+///
+/// Portions of the code are:
+///
+/// Copyright (c) 2014, lamerman
+/// All rights reserved.
+/// 
+/// Redistribution and use in source and binary forms, with or without
+/// modification, are permitted provided that the following conditions are met:
+/// 
+/// * Redistributions of source code must retain the above copyright notice, this
+///   list of conditions and the following disclaimer.
+/// 
+/// * Redistributions in binary form must reproduce the above copyright notice,
+///   this list of conditions and the following disclaimer in the documentation
+///   and/or other materials provided with the distribution.
+/// 
+/// * Neither the name of lamerman nor the names of its
+///   contributors may be used to endorse or promote products derived from
+///   this software without specific prior written permission.
+/// 
+/// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+/// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+/// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+/// DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+/// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+/// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+/// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+/// CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+/// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+/// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+/// 
+//////////////////////////////////////////////////////////////////////////////////
+
+#ifndef ARANGODB_LRUCACHE_H
+#define	ARANGODB_LRUCACHE_H 1
+
+#include <unordered_map>
+#include <list>
+#include <cstddef>
+#include <stdexcept>
+
+namespace arangodb {
+namespace basics {
+
+template<typename key_t, typename value_t>
+class LruCache {
+  public:
+    typedef typename std::pair<key_t, value_t> key_value_pair_t;
+    typedef typename std::list<key_value_pair_t>::iterator list_iterator_t;
+
+    LruCache(size_t max_size) :
+      _max_size(max_size) {
+      }
+
+    void put(const key_t& key, const value_t& value) {
+      auto it = _cache_items_map.find(key);
+      _cache_items_list.push_front(key_value_pair_t(key, value));
+      if (it != _cache_items_map.end()) {
+        _cache_items_list.erase(it->second);
+        _cache_items_map.erase(it);
+      }
+      _cache_items_map[key] = _cache_items_list.begin();
+
+      if (_cache_items_map.size() > _max_size) {
+        auto last = _cache_items_list.end();
+        last--;
+        _cache_items_map.erase(last->first);
+        _cache_items_list.pop_back();
+      }
+    }
+
+    const value_t& get(const key_t& key) {
+      auto it = _cache_items_map.find(key);
+      if (it == _cache_items_map.end()) {
+        throw std::range_error("There is no such key in cache");
+      } else {
+        _cache_items_list.splice(_cache_items_list.begin(), _cache_items_list, it->second);
+        return it->second->second;
+      }
+    }
+
+    void remove(key_t const& key) {
+      auto it = _cache_items_map.find(key);
+      if (it == _cache_items_map.end()) {
+        throw std::range_error("There is no such key in cache");
+      } else {
+        _cache_items_list.erase(it->second);
+        _cache_items_map.erase(it);
+      }
+    }
+
+    void clear() {
+      _cache_items_map.clear();
+      _cache_items_list.clear();
+    }
+
+    bool exists(const key_t& key) const {
+      return _cache_items_map.find(key) != _cache_items_map.end();
+    }
+
+    size_t size() const {
+      return _cache_items_map.size();
+    }
+
+  private:
+    std::list<key_value_pair_t> _cache_items_list;
+    std::unordered_map<key_t, list_iterator_t> _cache_items_map;
+    size_t _max_size;
+};
+
+}
+}
+
+#endif

lib/SimpleHttpClient/SimpleHttpClient.cpp

@@ -58,6 +58,7 @@ SimpleHttpClient::SimpleHttpClient(GeneralClientConnection* connection,
       _locationRewriter({nullptr, nullptr}),
       _nextChunkedSize(0),
       _result(nullptr),
+      _jwt(""),
       _maxPacketSize(MaxPacketSize),
       _maxRetries(3),
       _retryWaitTime(1 * 1000 * 1000),
@@ -443,6 +444,10 @@ void SimpleHttpClient::clearReadBuffer() {
 /// @brief sets username and password
 ////////////////////////////////////////////////////////////////////////////////
 
+void SimpleHttpClient::setJwt(std::string const& jwt) {
+  _jwt = jwt;
+}
+
 void SimpleHttpClient::setUserNamePassword(std::string const& prefix,
                                            std::string const& username,
                                            std::string const& password) {
@@ -572,6 +577,11 @@ void SimpleHttpClient::setRequest(
       _writeBuffer.appendText(TRI_CHAR_LENGTH_PAIR("\r\n"));
     }
   }
+  if (!_jwt.empty()) {
+    _writeBuffer.appendText(TRI_CHAR_LENGTH_PAIR("Authorization: bearer "));
+    _writeBuffer.appendText(_jwt);
+    _writeBuffer.appendText(TRI_CHAR_LENGTH_PAIR("\r\n"));
+  }
 
   for (auto const& header : headers) {
     _writeBuffer.appendText(header.first);

lib/SimpleHttpClient/SimpleHttpClient.h

@@ -163,6 +163,8 @@ class SimpleHttpClient {
   /// @param password                       password
   //////////////////////////////////////////////////////////////////////////////
 
+  void setJwt(std::string const& jwt);
+  
   void setUserNamePassword(std::string const& prefix,
                            std::string const& username,
                            std::string const& password);
@@ -415,6 +417,7 @@ class SimpleHttpClient {
   SimpleHttpResult* _result;
 
   std::vector<std::pair<std::string, std::string>> _pathToBasicAuth;
+  std::string _jwt;
 
   size_t _maxPacketSize;
 

lib/V8/v8-utils.cpp

@@ -539,7 +539,7 @@ static std::string GetEndpointFromUrl(std::string const& url) {
 /// @LIT{body} attribute of the result object.
 ////////////////////////////////////////////////////////////////////////////////
 
-static void JS_Download(v8::FunctionCallbackInfo<v8::Value> const& args) {
+void JS_Download(v8::FunctionCallbackInfo<v8::Value> const& args) {
   TRI_V8_TRY_CATCH_BEGIN(isolate);
   v8::HandleScope scope(isolate);
 

lib/V8/v8-utils.h

@@ -191,4 +191,6 @@ void TRI_InitV8Utils(v8::Isolate* isolate, v8::Handle<v8::Context>,
                      std::string const& startupPath,
                      std::string const& modules);
 
+void JS_Download(v8::FunctionCallbackInfo<v8::Value> const& args);
+
 #endif

scripts/startArangoDBClusterLocal.sh

@@ -89,7 +89,7 @@ ${ARANGOD} \
     --server.authentication false \
     --server.endpoint tcp://127.0.0.1:$(( $BASE + $aid )) \
     --server.statistics false \
-    --server.threads $NATH \
+    --server.threads 4 \
     --log.force-direct true \
     > /tmp/cluster/$(( $BASE + $aid )).stdout 2>&1 &
 

scripts/startLocalCluster.sh

@@ -9,6 +9,7 @@ function help() {
   echo "  -d/--ndbservers    # db servers        (odd integer      default: 2))"
   echo "  -s/--secondaries   Start secondaries   (0|1              default: 0)"
   echo "  -t/--transport     Protocol            (ssl|tcp          default: tcp)"
+  echo "  -j/--jwt-secret    JWT-Secret          (string           default: )"
   echo "     --log-level-a   Log level (agency)  (INFO|DEBUG|TRACE default: INFO)"
   echo "     --log-level-c   Log level (cluster) (INFO|DEBUG|TRACE default: INFO)"
   echo "  -i/--interactive   Interactive mode    (C|D|R            default: '')"
@@ -34,6 +35,7 @@ XTERM="x-terminal-emulator"
 XTERMOPTIONS="--geometry=80x43"
 SECONDARIES=0
 BUILD="build"
+JWT_SECRET=""
 
 while [[ ${1} ]]; do
     case "${1}" in
@@ -69,6 +71,10 @@ while [[ ${1} ]]; do
       INTERACTIVE_MODE=${2}
       shift
       ;;
+    -j|--jwt-secret)
+      JWT_SECRET=${2}
+      shift
+      ;;
     -x|--xterm)
       XTERM=${2}
       shift
@@ -103,14 +109,6 @@ if [ "$POOLSZ" == "" ] ; then
 fi
 
 
-if [ "$TRANSPORT" == "ssl" ]; then
-  SSLKEYFILE="--ssl.keyfile UnitTests/server.pem"
-  CURL="curl --insecure -s -f -X GET https:"
-else
-  SSLKEYFILE=""
-  CURL="curl -s -f -X GET http:"
-fi
-
 printf "Starting agency ... \n"
 printf "  # agents: %s," "$NRAGENTS"
 printf " # db servers: %s," "$NRDBSERVERS"
@@ -158,6 +156,22 @@ if [ -d cluster-init ];then
 fi
 mkdir -p cluster
 
+if [ -z "$JWT_SECRET" ];then
+  AUTHENTICATION="--server.authentication false"
+  AUTHORIZATION_HEADER=""
+else
+  AUTHENTICATION="--server.jwt-secret $JWT_SECRET"
+  AUTHORIZATION_HEADER="Authorization: bearer $(jwtgen -a HS256 -s $JWT_SECRET -c 'iss=arangodb' -c 'preferred_username=root')"
+fi
+
+if [ "$TRANSPORT" == "ssl" ]; then
+  SSLKEYFILE="--ssl.keyfile UnitTests/server.pem"
+  CURL="curl --insecure $CURL_AUTHENTICATION -s -f -X GET https:"
+else
+  SSLKEYFILE=""
+  CURL="curl -s -f $CURL_AUTHENTICATION -X GET http:"
+fi
+
 echo Starting agency ... 
 for aid in `seq 0 $(( $NRAGENTS - 1 ))`; do
     port=$(( $BASE + $aid ))
@@ -180,12 +194,12 @@ for aid in `seq 0 $(( $NRAGENTS - 1 ))`; do
         --javascript.startup-directory ./js \
         --javascript.module-directory ./enterprise/js \
         --javascript.v8-contexts 1 \
-        --server.authentication false \
         --server.endpoint $TRANSPORT://0.0.0.0:$port \
         --server.statistics false \
         --server.threads 16 \
         --log.file cluster/$port.log \
         --log.force-direct true \
+        $AUTHENTICATION \
         $SSLKEYFILE \
         > cluster/$port.stdout 2>&1 &
 done
@@ -213,11 +227,11 @@ start() {
        --log.level info \
        --server.statistics true \
        --server.threads 5 \
-       --server.authentication false \
        --javascript.startup-directory ./js \
        --javascript.module-directory ./enterprise/js \
        --javascript.app-path cluster/apps$PORT \
        --log.force-direct true \
+        $AUTHENTICATION \
         $SSLKEYFILE \
        > cluster/$PORT.stdout 2>&1 &
 }
@@ -247,7 +261,7 @@ startTerminal() {
         --javascript.startup-directory ./js \
         --javascript.module-directory ./enterprise/js \
         --javascript.app-path ./js/apps \
-        --server.authentication false \
+        $AUTHENTICATION \
         $SSLKEYFILE \
         --console &
 }
@@ -278,7 +292,7 @@ startDebugger() {
       --javascript.module-directory ./enterprise/js \
       --javascript.app-path ./js/apps \
       $SSLKEYFILE \
-      --server.authentication false &
+      $AUTHENTICATION &
       $XTERM $XTERMOPTIONS -e gdb ${BUILD}/bin/arangod -p $! &
 }
 
@@ -307,7 +321,7 @@ startRR() {
         --javascript.startup-directory ./js \
         --javascript.module-directory ./enterprise/js \
         --javascript.app-path ./js/apps \
-        --server.authentication false \
+        $AUTHENTICATION \
         $SSLKEYFILE \
         --console &
 }
@@ -345,7 +359,11 @@ echo Waiting for cluster to come up...
 testServer() {
     PORT=$1
     while true ; do
+        if [ -z "$AUTHORIZATION_HEADER" ]; then
           ${CURL}//127.0.0.1:$PORT/_api/version > /dev/null 2>&1
+        else
+          ${CURL}//127.0.0.1:$PORT/_api/version -H "$AUTHORIZATION_HEADER" > /dev/null 2>&1
+        fi
         if [ "$?" != "0" ] ; then
             echo Server on port $PORT does not answer yet.
         else
@@ -385,7 +403,7 @@ if [ "$SECONDARIES" == "1" ] ; then
             --server.statistics true \
             --javascript.startup-directory ./js \
             --javascript.module-directory ./enterprise/js \
-            --server.authentication false \
+            $AUTHENTICATION \
             $SSLKEYFILE \
             --javascript.app-path ./js/apps \
             > cluster/$PORT.stdout 2>&1 &