1
0
Fork 0
Commit Graph

75 Commits

Author SHA1 Message Date
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00
Simon 17b1a2aafb Rest middleware refactoring (#5332) 2018-05-14 17:43:10 +02:00
Tobias Gödderz 8c87f51429 Feature/fix inconsistent distribute shards like job (#4743) 2018-05-07 16:53:08 +02:00
Simon a1416e1067 Make v8 optional on startup (#5220) 2018-04-30 12:48:57 +02:00
Jan 2b84348b77
remove call to requiresElevatedPrivileges with default value (#5172) 2018-04-23 11:28:24 +02:00
Simon 8be273efb8 Replication cleanup (#5105) 2018-04-17 08:17:42 +02:00
Frank Celler 59953d641c
added /_admin/status (#5055) 2018-04-08 20:43:18 +02:00
Vasiliy 635db3b409 issue 355.4: remove redundant methods and code, use 'cp' instead of 'cmake copy_directory' where possible, use vocbase reference instead of pointer 2018-04-04 10:53:48 +03:00
Michael Hackstein c1650702bf
Feature/aql server based locking (#4783)
* Started Implementing the ServerBasedlocking. There now is a container that can contain multiple query snippets. It now has to setup the necessary calls to the Servers

* Added backwards linking of QueryEngines, sth. DBServers can contact their Coordinators.

* Added LogTopic AQL

* Made AccessMode::Type Hashable

* Created a Mapping Server => LockLevel => Shard and createad a JSON object containing the Lock information for a complete AQL query per server

* Added code to build coordinator engines

* Finished with first draft of Coordinator-side of new DBServer based locking.

* Added a _api/aql/setup route that creates and locks all snippets/collections for one DBServer in a single go

* Fixed some Coordinator parts

* Index node now gracefully reports if it could not find it's collection when created from vpack. Otherwise it just hardly crashed...

* Modified the Coordinator Snippet collector to be able to handle subqueries properly.

* Started adding GraphNode handling. WIP. Need to deploy engines properly. Coordinator crashes on Graph tests

* Fixed compiler errors

* WIP: EngineInfoContainer

* Separated the EngineInfoContainers for Coordinator and DBServer into different files. They diverged more than anticipated

* Added forgotten files. THe DBServer container now creates the TraverserEngine Mapping and moves it into the Infos. They are not keeping it yet and need to add it to the message as well.

* The DBServer engine infos now persist the TraverserEngine infos. Need to add them to messages though.

* The new aql exec-engine now sends out traverserEngines as well

* Formatting and adding DEBUG level output

* Made the RestAQLHandler aware of the TraverserEngineRegistry. Also created the engines now. Return format changed server-side coordinator side needs fix.

* Adapted the Coordinator side for the DBServer based Shard Locking

* The DBServer based Locking now honors restrictions to certain shards

* Fixed a strange double lock bug in the new AQL Server based locking technique. Add some DEBUG output

* Fixed usage of MAINTAINERMODE macro. The assertion was never active

* Added TestCase for ContainerCoordinatorTest to cmake

* Added -DTEST_VIRTUAL to CMAKE. This is used to define virtual functions for mocking ONLY on test-builds.

* Fixed usage of ENABLE_MAINTAINER_MODE ifdef. CLANG format

* On non-enterprise builds ENTERPRISE_VERT defaults to TEST_VIRTUAL => virtual in test else non-virtual

* Added TEST_VIRTUAL to ExecutionEngine, Query and QueryRegistry

* Added first testcase for EngineInfoContainerCoordinator not yet ready.

* Mode CreateBlock a member function of engine, we have the engine in our hands anyways no need to make it static. Included some more TEST_VIRTUAL functions.

* Fixed clang/MacOs compile error. Added some more TEST_VIRTUAL declarations

* Finally fixed the first buildEngines UnitTest \o/

* Added a unit-test for backward linking of dependencies in CoordinatorPlanner

* Added multi-snippet test for EngineInfoContainerCoordinator

* Removed QueryRegistry.h from central header files and replaced by a forward declartion.

* Added a createBlocks method on the ExecutionEngine. It should be responsible to create all those blocks at once. Adapted the UnitTests as well. Not included Tests for the new createBlocks functionality. Need to mock the options feature first

* Added another test that Coordinator Snippets of queries can be created correctly

* Fixed Coordinator-site cleanup of QueryRegistry, if any of the query creations fails with error, incl UnitTest.

* Added first test for RestAqlHandler::setup. It does only test the setup and gives prepartion for real testing.

* Added a assertion of http return code. Still no creation of queries is tested. Requires a huge amount of mocking.

* fix some deadlocks found by evil lock manager (tm)

* fix duplicate lock

* fix indentation

* ensure proper lock dependencies

* fix lock acquisition

* removed useless comment

* do not lock twice

* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not

* AQL micro optimizations

* use explicit constructor

* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context

entering and exiting the selected context is then done later on without having to hold the ConditionLocker

* remove some recursive locks

* Disable custom deadlock detection when Thread Sanitizer is enabled

* Changing ifdef's

* grr

* broke gcc

* Using atomic for ApplicationServer::_server

* fix premature unlock

* add some asserts

* honor collection locking in cluster

* yet one more lock fix

* removed assertion

* Allow the clustercomm to send nolock headers on count. This is used form within AQL

* IsLocked on transactions will now always yield true IF LOCK_NEVER is set. We simply assume someone else holds the lock for us. Also LOCK_NEVER is now set on collection/count if noLock header is send.

* Moved the flag if collections need to be locked into the TraverserEngines.

* Added enterprise-satellite hooks in EngineInfoContainerDBServer

* Removed now obsolete code

* Replaced throwing of Exception by an ResultObject

* Added some more tests and moved adding snippet to query engine more to the outside.

* Added the AQL result type

* Make the branch compile again

* Register WITH collections for Graphs in the new Collector.

* Fixed test code for failing query clone. Idea was to once clone successfully and second time to fail, we verify that first clone is cleaned up properly. However test failed on first clone...

* Removed a double builderClose

* Added Changelog entry

* Removed empty if

* Removed obsolete todo

* Properly initialize the AqlResult with nullptr on error case

* Updated comment

* Simplified Assertion

* Removed debug output object...

* Added additional catch case for std::exception to get some more error info

* Clarified evaluation order for move case

* Added Explicit

* Fixed cleanup of Coordinator if Registry fails to insert query.

* Allow to use other locks than Read/Write for AQL collections. Not yet in API.

* Updated Comments for other Locks on DBSide. Adapted Destruction CatchTests

* Fixed double builderClose and removed unnecessary double commits

* Added a comment to clarify the state

* Moved error output to trace. Leftover from debugging

* Added some tests for complex subquery patterns

* Added a 'fireAndForgetRequests' methods to cluster comm which allows to send out a bunch of messages but does not wait for their results

* Properly cleanup leftovers of queries if the instantiation step already failed

* Added code comment for fireAndForget

* Added indexes to subquery test to make the plan a bit easier

* The cleanup on DBServerEngines in error case now also cleans up traverser engines.

* Removed unnecessary includes

* Removed debug logging

* Fixed hidden merge conflict
2018-03-20 16:52:19 +01:00
Simon fe9768dc44 Replace /_api/simple/by-example (#4779)
* Replacing /_api/simple/by-example

* Requested changes
2018-03-09 09:35:48 +01:00
Simon 345fc3c0b7 Refactor Authentication Layer (devel) (#4592)
* Cherry Picking LDAP changes

* Adding missing merges

* Fixing remaining mentions of FeatureCacheFeature

* Fix jslint

* Fixing some failed tests

* Fixing cluster authentication issue, red tests

* Fixing ldap testsuite, adding trace logging

* Fixint ldap tesuite setup and LDAP recognition

(cherry picked from commit 686d28a779)

* Fixing wrong assert

* Adding changelog entry, making requested changes from code review

* Fixing dump_authentication, fix typos

* improvements found during code review

* oops

* more use of sessionstorage

* fix tests

* Fixing broken handling, disallowing adding of local users when disabled

* Fixing testInvalidGrants

* Removing undefined auth level externally

* Fixing previous commit

* added tests for ldap search mode

* intentionally removed `after` methods from tests

because they are executed before the tests start
no cleanup is performed right now after the authentication tests
however, a cleanup is done at start of every test

* ldap tests all modes

* forward port changes from 3.3

* added generated files

* forward port missing changes for web UI

* added generated files

* added generated files
2018-02-28 13:24:28 +01:00
Simon 11a7bbf321 Replacing statistics handler with c++ (#4653) 2018-02-26 15:33:43 +01:00
Wilfried Goesgens 619155a62d Feature/aqlfunction to cpp (#4573) 2018-02-13 11:50:49 +01:00
Jan b2b6c06cbf
Feature/efficiency (#3736) 2018-01-05 16:51:31 +01:00
Jan 9c76613e63
fix premature unlock (#3802)
* fix some deadlocks found by evil lock manager (tm)

* fix duplicate lock

* fix indentation

* ensure proper lock dependencies

* fix lock acquisition

* removed useless comment

* do not lock twice

* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not

* AQL micro optimizations

* use explicit constructor

* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context

entering and exiting the selected context is then done later on without having to hold the ConditionLocker

* remove some recursive locks

* Disable custom deadlock detection when Thread Sanitizer is enabled

* Changing ifdef's

* grr

* broke gcc

* Using atomic for ApplicationServer::_server

* fix premature unlock

* add some asserts

* honor collection locking in cluster

* yet one more lock fix

* removed assertion

* some more bugfixes

* Fixing assert

(cherry picked from commit 1155df173bfb67303077fbe04ee8d909517bfd21)
2017-12-13 13:27:42 +01:00
m0ppers 6cbf7159be Feature/server mode (#3590)
* Switching from ttl to supervision based failover mechanism

* Allowing canceling of ongoing actions

* refactored asyncjobmanager

* refactoring some code

* adding read-only flag

* Fixing some bugs

* Fixing tests

* Canceling ongoing operations

* removing some unused code + some asserts

* catching some exceptions to reduce log pollution, removing unnecessary code, removing tests for _changeMode

* fixing "createsANewDatabaseWithAnInvalidUser"

* Current work

* proper ifs

* Migrate resthandler to c++ and implement mode

* readonly server mode spec test

* Add changelog

* change code so it expects a full object and not just a string

* Fix jslint
2017-11-10 17:56:21 +01:00
Simon Grätzer ee8209943f Missing things for active / passive (#3578)
* Switching from ttl to supervision based failover mechanism

* Allowing canceling of ongoing actions

* refactored asyncjobmanager

* refactoring some code

* adding read-only flag

* catching some exceptions to reduce log pollution, removing unnecessary code, removing tests for _changeMode

* fixing "createsANewDatabaseWithAnInvalidUser"

* auth = off does not longer make everyone superuser

* Fixing cluster_sync and maybe resilience
2017-11-04 20:30:23 +01:00
Simon Grätzer 64e9377c05 Replacing /_api/collection with RestHandler (#3543) 2017-11-02 14:57:17 +01:00
Simon Grätzer 7c31960cf2 Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
Jan Christoph Uhde 0eaf4cabd9 Feature/js to cpp transaction handler (#2985)
* implement transaction handler in c++

* add fake request object

* move general code into extra file

* improve error messages for RestJobHandler

* support cancel operation for async jobs with RestTransactionHandler

* fix integration test and add some optional debugging output

* fix returning of context

* fix error type

* fix code in cluster test

* remove unused variable
2017-08-11 09:29:34 +02:00
Jan ed9d15156e remove dependency on MMFiles features from non-MMFiles files (#2925) 2017-08-01 22:16:43 +02:00
Dan Larkin 5f772b09a7 Converted rest handler for explain from JS to C++. (#2907) 2017-08-01 22:10:39 +02:00
Dan Larkin 4cc46f9b96 Converted a portion of the admin routing API from JS to C++ (#2919)
* Added a partial C++ implementation for admin routing API.

* Finished C++ implementation for admin routing API.
2017-08-01 17:14:08 +02:00
Dan Larkin 750fc96ab4 Converted endpoint handler from JS to C++ (#2905)
* Added native C++ rest handler for endpoints.

* Added test of rest endpoint interface and removed JS handler.
2017-08-01 09:53:04 +02:00
Simon Grätzer 7fd47f7232 Fixing enumerate db's in cluster (#2844)
* Fixing enumerate db's in cluster

* update coordinators
2017-07-19 20:34:15 +02:00
Frank Celler a692577f5b Feature/auth context (#2815) 2017-07-16 09:39:03 +02:00
Frank Celler bbe7484521 Feature/auth context (#2704)
* added read-only users
2017-07-02 23:15:57 +02:00
Simon Grätzer 492d832695 Refactored /_api/index and /_api/database (#2582)
* working documents rest handler

* fixed cluster tests

* Consolidating database APIs

* clang-format

* Fixing issue with user creation through db._createDatabase

* replaced and refactored api-index and index api

* fixed cluster

* renaming some files

* added user methods

* removed files intended for later

* Fixed CC build

* Fixed method signature

* Fixing shell_server, shell_client tests
2017-06-19 23:47:40 +02:00
Dan Larkin a0c4e3d9df Improved RocksDB engine statistics with docs and rest endpoint (#2598)
* Improved statistics reporting for RocksDB engine (including cache info).

* Added documentation about db._engineStats().

* Added rest endpoint for engine stats.
2017-06-18 22:55:25 +02:00
Dan Larkin af8e1889a1 Moved MMFiles replication handler. 2017-04-18 15:35:45 -04:00
jsteemann fbb14ceffc added db._engine() 2017-03-22 11:19:31 +01:00
jsteemann b8e011eee5 move attributes into engine 2017-03-21 10:31:50 +01:00
jsteemann 25a380f77c Merge branch 'devel' of https://github.com/arangodb/arangodb into views 2017-03-16 12:50:11 +01:00
Dan Larkin 966e05b596 Added very basic rest handler for views. 2017-03-14 19:05:14 -04:00
Simon Grätzer 7b35f75b23 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel
# Conflicts:
#	arangod/CMakeLists.txt
#	arangod/GeneralServer/GeneralServerFeature.cpp
2017-03-13 12:25:28 +01:00
jsteemann 45a68f54f8 Merge branch 'devel' of https://github.com/arangodb/arangodb into engine-api 2017-03-13 12:10:54 +01:00
Simon Grätzer 871272d7c1 Merge from devel 2017-03-12 12:32:22 +01:00
jsteemann b217a9d6af remove unused feature 2017-03-10 23:59:09 +01:00
jsteemann 0a07bb3222 make RestWalHandler a handler of the MMFiles engine 2017-03-10 09:00:37 +01:00
Simon Grätzer 27c617fe10 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel
# Conflicts:
#	3rdParty/V8/v8
#	arangod/Transaction/Methods.h
#	arangod/Utils/UserTransaction.h
#	arangod/V8Server/v8-collection.cpp
2017-03-01 14:52:35 +01:00
jsteemann dc72ec3ee7 fixed a race when dropping databases 2017-02-23 15:22:50 +01:00
Simon Grätzer edab268572 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel
# Conflicts:
#	arangod/Aql/FunctionDefinitions.cpp
#	arangod/Aql/Functions.h
#	arangod/Utils/ExplicitTransaction.h
2017-02-10 15:21:24 +01:00
jsteemann d024a6d00a remove logging for non-topics 2017-02-10 09:32:50 +01:00
jsteemann d503a4d354 added reactor type 2017-02-07 12:41:41 +01:00
Simon Grätzer 13f19fe51b Merge branch 'devel' of https://github.com/arangodb/arangodb into devel
# Conflicts:
#	arangod/CMakeLists.txt
#	arangod/RestServer/arangod.cpp
#	arangod/Utils/ExplicitTransaction.h
#	arangod/V8Server/v8-collection.cpp
2017-02-02 11:25:23 +01:00
jsteemann f9fb701505 moved LogfileManager into engine 2017-01-23 10:30:55 +01:00
Simon Grätzer d71ebf7c1d Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-01-20 14:42:19 +01:00
Andreas Streichardt 191f399ce2 Move AgencyCallback stuff to cluster so it is (hopefully) clear that this
is being used within the cluster and not within the agency.
2017-01-13 18:08:27 +01:00
Simon Grätzer 7afebc2223 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2016-12-17 14:10:03 +01:00
jsteemann 4f10a6fb9c experimental micro optimizations 2016-12-16 15:52:16 +01:00