1
0
Fork 0
Commit Graph

229 Commits

Author SHA1 Message Date
Michael Hackstein c1650702bf
Feature/aql server based locking (#4783)
* Started Implementing the ServerBasedlocking. There now is a container that can contain multiple query snippets. It now has to setup the necessary calls to the Servers

* Added backwards linking of QueryEngines, sth. DBServers can contact their Coordinators.

* Added LogTopic AQL

* Made AccessMode::Type Hashable

* Created a Mapping Server => LockLevel => Shard and createad a JSON object containing the Lock information for a complete AQL query per server

* Added code to build coordinator engines

* Finished with first draft of Coordinator-side of new DBServer based locking.

* Added a _api/aql/setup route that creates and locks all snippets/collections for one DBServer in a single go

* Fixed some Coordinator parts

* Index node now gracefully reports if it could not find it's collection when created from vpack. Otherwise it just hardly crashed...

* Modified the Coordinator Snippet collector to be able to handle subqueries properly.

* Started adding GraphNode handling. WIP. Need to deploy engines properly. Coordinator crashes on Graph tests

* Fixed compiler errors

* WIP: EngineInfoContainer

* Separated the EngineInfoContainers for Coordinator and DBServer into different files. They diverged more than anticipated

* Added forgotten files. THe DBServer container now creates the TraverserEngine Mapping and moves it into the Infos. They are not keeping it yet and need to add it to the message as well.

* The DBServer engine infos now persist the TraverserEngine infos. Need to add them to messages though.

* The new aql exec-engine now sends out traverserEngines as well

* Formatting and adding DEBUG level output

* Made the RestAQLHandler aware of the TraverserEngineRegistry. Also created the engines now. Return format changed server-side coordinator side needs fix.

* Adapted the Coordinator side for the DBServer based Shard Locking

* The DBServer based Locking now honors restrictions to certain shards

* Fixed a strange double lock bug in the new AQL Server based locking technique. Add some DEBUG output

* Fixed usage of MAINTAINERMODE macro. The assertion was never active

* Added TestCase for ContainerCoordinatorTest to cmake

* Added -DTEST_VIRTUAL to CMAKE. This is used to define virtual functions for mocking ONLY on test-builds.

* Fixed usage of ENABLE_MAINTAINER_MODE ifdef. CLANG format

* On non-enterprise builds ENTERPRISE_VERT defaults to TEST_VIRTUAL => virtual in test else non-virtual

* Added TEST_VIRTUAL to ExecutionEngine, Query and QueryRegistry

* Added first testcase for EngineInfoContainerCoordinator not yet ready.

* Mode CreateBlock a member function of engine, we have the engine in our hands anyways no need to make it static. Included some more TEST_VIRTUAL functions.

* Fixed clang/MacOs compile error. Added some more TEST_VIRTUAL declarations

* Finally fixed the first buildEngines UnitTest \o/

* Added a unit-test for backward linking of dependencies in CoordinatorPlanner

* Added multi-snippet test for EngineInfoContainerCoordinator

* Removed QueryRegistry.h from central header files and replaced by a forward declartion.

* Added a createBlocks method on the ExecutionEngine. It should be responsible to create all those blocks at once. Adapted the UnitTests as well. Not included Tests for the new createBlocks functionality. Need to mock the options feature first

* Added another test that Coordinator Snippets of queries can be created correctly

* Fixed Coordinator-site cleanup of QueryRegistry, if any of the query creations fails with error, incl UnitTest.

* Added first test for RestAqlHandler::setup. It does only test the setup and gives prepartion for real testing.

* Added a assertion of http return code. Still no creation of queries is tested. Requires a huge amount of mocking.

* fix some deadlocks found by evil lock manager (tm)

* fix duplicate lock

* fix indentation

* ensure proper lock dependencies

* fix lock acquisition

* removed useless comment

* do not lock twice

* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not

* AQL micro optimizations

* use explicit constructor

* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context

entering and exiting the selected context is then done later on without having to hold the ConditionLocker

* remove some recursive locks

* Disable custom deadlock detection when Thread Sanitizer is enabled

* Changing ifdef's

* grr

* broke gcc

* Using atomic for ApplicationServer::_server

* fix premature unlock

* add some asserts

* honor collection locking in cluster

* yet one more lock fix

* removed assertion

* Allow the clustercomm to send nolock headers on count. This is used form within AQL

* IsLocked on transactions will now always yield true IF LOCK_NEVER is set. We simply assume someone else holds the lock for us. Also LOCK_NEVER is now set on collection/count if noLock header is send.

* Moved the flag if collections need to be locked into the TraverserEngines.

* Added enterprise-satellite hooks in EngineInfoContainerDBServer

* Removed now obsolete code

* Replaced throwing of Exception by an ResultObject

* Added some more tests and moved adding snippet to query engine more to the outside.

* Added the AQL result type

* Make the branch compile again

* Register WITH collections for Graphs in the new Collector.

* Fixed test code for failing query clone. Idea was to once clone successfully and second time to fail, we verify that first clone is cleaned up properly. However test failed on first clone...

* Removed a double builderClose

* Added Changelog entry

* Removed empty if

* Removed obsolete todo

* Properly initialize the AqlResult with nullptr on error case

* Updated comment

* Simplified Assertion

* Removed debug output object...

* Added additional catch case for std::exception to get some more error info

* Clarified evaluation order for move case

* Added Explicit

* Fixed cleanup of Coordinator if Registry fails to insert query.

* Allow to use other locks than Read/Write for AQL collections. Not yet in API.

* Updated Comments for other Locks on DBSide. Adapted Destruction CatchTests

* Fixed double builderClose and removed unnecessary double commits

* Added a comment to clarify the state

* Moved error output to trace. Leftover from debugging

* Added some tests for complex subquery patterns

* Added a 'fireAndForgetRequests' methods to cluster comm which allows to send out a bunch of messages but does not wait for their results

* Properly cleanup leftovers of queries if the instantiation step already failed

* Added code comment for fireAndForget

* Added indexes to subquery test to make the plan a bit easier

* The cleanup on DBServerEngines in error case now also cleans up traverser engines.

* Removed unnecessary includes

* Removed debug logging

* Fixed hidden merge conflict
2018-03-20 16:52:19 +01:00
Jan 67e7e4181e
Bug fix/remove v8 executor (#4756) 2018-03-13 10:41:02 +01:00
Simon 102998dd65 Streaming Query Cursor (#4769) 2018-03-08 09:18:41 +01:00
Jan 48c42bf135
slightly simplify query API (#4676) 2018-02-26 23:08:46 +01:00
Michael Hackstein 2b0aa318ec Bug fix/sparse graph register warning (#3569) 2017-11-10 14:33:15 +01:00
Jan 5165155ed1 Bug fix/fixes 0609 (#3227)
* do not use V8 variant of AQL functions in early optimization stage when a C++ variant is available

* additionally, simplify AQL function definitions and aliases

* warn when more than 90% of max mappings are in use

* added C++ variant of replication catchup

* added `--log.role` option

* updated CHANGELOG

* removed non-existing scheduler.threads option from config

* removed useless __FILE__, __LINE__ invocations

* updated CHANGELOG

* allow a priority V8 context

* remove TRI_CORE_MEM_ZONE

* try to fix Windows errors & warnings

* cleanup

* removed memory zones altogether

* exclude system collections from collection tests
2017-09-13 16:28:21 +02:00
Frank Celler 6d08d4f4aa Bug fix/scheduler delete (#3077)
* removed delete call

* cleanup

* lower cpu activity of log thread too

* fix log messages

* do not enter threads into unordered_set, as it is unneeded

* do not compile in calls to disabled plan cache

* moved AQL regex cache from thread local variables to a class of its own

* more sensible thread creation and destruction
2017-08-25 12:00:17 +02:00
Jan e6c6c5acda Bug fix/pre release woes (#2988)
* fixed issue #2982

* updated CHANGELOG
2017-08-10 09:39:28 +02:00
Jan 9396d4d952 fix query execution times that may have different between batches (#2606)
* fix query execution times that may have different between batches

* applied query statistic change to ui

* do not expose state "finished", as it is useless
2017-06-19 17:47:32 +02:00
jsteemann bf209397cd fixed shutdown issues 2017-06-08 12:30:43 +02:00
jsteemann a35f85bcc7 added proper docs for --query.fail-on-warning 2017-05-29 15:46:48 +02:00
jsteemann a4fde59fd2 some refactoring 2017-05-23 13:18:51 +02:00
jsteemann 53fa66e60d move parts from ProgramOptions from headers into cpp files 2017-05-23 10:30:36 +02:00
jsteemann d398b5e4b7 added option `--query.fail-on-warning` 2017-05-22 10:51:53 +02:00
jsteemann 8a69f5716f show collection load time in slow queries 2017-03-17 09:28:53 +01:00
jsteemann 3ed19e291d added "shardIds" option for AQL queries to restrict queries to specific shards 2017-03-08 15:07:10 +01:00
jsteemann 644ef08ce0 added "bindVars" to attributes of currently running and slow queries 2017-03-07 00:19:01 +01:00
Michael Hackstein 7afccbd6e9 Fixed forward declarition. Struct vs. class 2017-03-06 10:48:37 +01:00
jsteemann 1b7b668066 simplify query list a great deal and use less dynamic memory for its entries 2017-03-02 13:20:37 +01:00
jsteemann 62f894c2b8 split big file 2017-03-01 15:49:57 +01:00
jsteemann 666b2f8da9 renaming 2017-02-27 14:38:27 +01:00
jsteemann f5c8932744 initial version of query execution plan cache, still very rough 2017-02-21 11:58:38 +01:00
jsteemann be8d6b0a9e simplifications 2017-02-17 20:55:52 +01:00
jsteemann ea87cc52b7 simplifications 2017-02-17 17:10:13 +01:00
jsteemann 10ddb52964 remove unused property 2017-02-17 15:41:45 +01:00
jsteemann 7e690a0c61 renaming 2017-02-13 15:18:40 +01:00
jsteemann eb73dbf960 renaming 2017-02-13 13:30:57 +01:00
jsteemann f09ec85127 renaming 2017-02-13 12:27:26 +01:00
jsteemann a7bc1ad213 use std::unique_ptr 2017-02-06 23:24:00 +01:00
jsteemann 3c5edeb9b2 fix cluster AQL statistics 2017-02-03 11:47:36 +01:00
jsteemann b8e1eb8d47 fix compile warnings in MSVC 2017-01-31 16:47:22 +01:00
jsteemann 044499e454 added startup option --query.memory-limit 2017-01-27 17:33:57 +01:00
jsteemann 369b2c7bc6 added optional memory limit for AQL queries 2016-12-06 13:23:41 +01:00
Max Neunhoeffer 14d8d35fd9 Put tracing of AQL queries in.
Use by setting { tracing: 1} (only calls) or { tracing: 2} (with
result) into 4th argument of db._query
2016-10-19 11:36:29 +02:00
Jan Steemann e88460efcb add state "loading collections" to query profile results 2016-10-13 10:31:16 +02:00
Michael Hackstein e22014a272 Merge branch 'MIT' of github.com:arangodb/arangodb into generic-col-types 2016-09-06 09:33:23 +02:00
Michael Hackstein 81a21aaea0 Now initializing the Query when injecting the transaction. 2016-08-02 12:43:19 +02:00
Michael Hackstein 64d7f690da Removed AqlTransaction wherever possible. Added an InjectTransaction for the aql query, used by TraverserEngine. Also fixed TraverserEngine from/to velocypack functions. 2016-07-28 17:05:00 +02:00
jsteemann 5d6cd9ef71 moved query id generation 2016-07-27 17:44:31 +02:00
Jan Steemann 9626d33960 re-added --database.slow-query-threshold 2016-06-09 11:07:51 +02:00
jsteemann 7d9e93e61e removed unused includes of JsonHelper.h 2016-05-13 19:00:23 +02:00
Jan Steemann 6489bd1364 fixed crashes 2016-05-04 12:38:44 +02:00
jsteemann 9d865c7047 optimizations, bugfixes 2016-05-04 01:50:28 +02:00
Frank Celler dd44191a91 WIP 2016-04-15 21:03:53 -07:00
Frank Celler cd91d3654e Merge remote-tracking branch 'origin' into FMH 2016-04-15 19:01:38 -07:00
jsteemann 4a7c9f0f68 simplified bloated comments 2016-04-09 11:27:30 +02:00
Frank Celler e83c1f09b4 WIP 2016-04-02 23:41:21 +02:00
Frank Celler af00417cf4 skeleton arangod 2016-03-25 15:14:43 +01:00
jsteemann 0720ccdecb harmonized 2016-03-16 18:49:42 +01:00
jsteemann a61c6b8ffa fixed query options 2016-03-16 17:53:39 +01:00
jsteemann 4f8265bfc8 temporary builder leaser 2016-03-15 14:08:03 +01:00
Michael Hackstein 42e3d94bf6 Removed unused obsolete Code and Imports 2016-03-08 13:29:09 +01:00
Michael Hackstein e60e7a3652 Aql/QueryResult now only uses VelocyPack instead of TRI_json_t. Also adapted all calling places 2016-03-08 13:08:05 +01:00
Michael Hackstein 7c3690c4db AqlQuery now is contructed from VelocyPack builders only. Also variables and Execution Plan is constructed form VelocyPack only. 2016-03-07 19:04:37 +01:00
Michael Hackstein a01a1605b1 Merge branch 'spdvpk' of github.com:arangodb/arangodb into spdvpk 2016-03-07 12:46:27 +01:00
Michael Hackstein 038573b0db Removed large parts of JSON out of RestAqlHandler. There is still some JSON contained because auf Query still using JSON. Included several warnings for the specfic code pieces 2016-03-07 12:45:34 +01:00
Jan Steemann 6f6f7e23ba executionstats use vpack 2016-03-07 11:20:04 +01:00
Jan Steemann def9d86868 Merge branch 'devel' of https://github.com/arangodb/arangodb into spdvpk 2016-03-07 10:51:24 +01:00
Jan Steemann c6fa638b67 fixed issue #1765, issue #1770 2016-03-07 10:49:18 +01:00
Jan Steemann 06da362f93 some json removal 2016-03-02 17:51:05 +01:00
Jan Steemann 8f2e2de451 Merge branch 'devel' of https://github.com/arangodb/arangodb into spdvpk 2016-02-25 10:46:13 +01:00
jsteemann 1823ae4fd2 removed unused forwards and functions 2016-02-25 00:48:06 +01:00
Jan Steemann f468cd00ad create transaction context as shared ptr 2016-02-23 12:18:39 +01:00
Michael Hackstein a3cd1aa54e Merge branch 'devel' of github.com:arangodb/arangodb into vpack 2016-02-15 10:25:31 +01:00
Jan Steemann e6ff2a5100 disable copy construction 2016-02-08 16:14:22 +01:00
Michael Hackstein 6eeb686287 Added dummy to collect a shared VPackBuilder from a query object. Has to be properly implemented as soon as VelocyPack is the default AqlValue format 2016-02-03 10:40:51 +01:00
jsteemann 349a6d9754 more logging 2016-01-31 21:48:59 +01:00
jsteemann b635983d9c removed unused functions and headers 2016-01-31 19:08:39 +01:00
Max Neunhoeffer 238937128a Merge branch 'devel' into ClustUp2 2016-01-29 13:59:11 +01:00
Jan Steemann 9046e1831b clang-format 2016-01-27 13:43:46 +01:00
Max Neunhoeffer 887e571a26 Merge branch 'devel' into ClustUp2 2016-01-26 18:01:42 +01:00
Michael Hackstein a6f58ce14d Use forward declarations for VelocyPack instead of all includes everywhere 2016-01-24 14:29:31 +01:00
jsteemann 842384016d namespace cleanup 2016-01-21 00:20:22 +01:00
jsteemann 431900f17a changed namespace from triagens to arangodb 2016-01-17 00:44:53 +01:00
Max Neunhoeffer b0283077db Merge branch 'devel' into ClustUp2 2016-01-11 11:41:36 +01:00
Max Neunhoeffer dd364a9d20 Fix merge. 2016-01-11 11:38:10 +01:00
Max Neunhoeffer e7eae3a73a Merge branch 'devel' into ClustUp2 2016-01-11 11:01:45 +01:00
Jan Steemann 687d6133f0 comments reformatting 2016-01-11 09:52:39 +01:00
Jan Steemann 3ad20c0cae auto-generated headers 2016-01-06 18:41:51 +01:00
Jan Steemann a3b026d9d1 removed useless sections from code, unified include guards 2016-01-06 14:15:22 +01:00
Jan Steemann f422576ee0 clang-format 2016-01-05 17:43:24 +01:00
jsteemann a6cb463e06 Merge branch 'devel' of https://github.com/arangodb/arangodb into vpack 2016-01-05 00:25:10 +01:00
Max Neunhoeffer fa17c1ac22 Add reporting about query state. Needed for synchronous replication. 2015-12-30 16:52:47 +01:00
Jan Steemann 223c44d572 Merge branch 'devel' of https://github.com/arangodb/arangodb into vpack 2015-12-15 12:02:47 +01:00
Wilfried Goesgens f51e5efb34 Implement tests for de/serializing of TraversalNodes. 2015-12-14 18:07:04 +01:00
Jan Steemann 5c033f566b Merge branch 'devel' of https://github.com/arangodb/arangodb into vpack 2015-12-04 13:47:08 +01:00
Jan Steemann 89e0f887b9 velocypack update 2015-11-19 13:28:40 +01:00
Michael Hackstein fbee0b0859 Started to transform Parts of AQL Query to VPack. Especially the QueryResult is now using VPack internally. 2015-11-12 11:09:35 +01:00
Wilfried Goesgens 92ad802bd6 Cache Graphs inside of the Query object
- split Graph clas into Aql/Graphs
  - cleanup includes
2015-11-11 13:52:33 +01:00
Jan Steemann 291891036f internal refactoring of query API 2015-09-18 17:33:39 +02:00
CoDEmanX 3a9648d78a More Americanization 2015-09-03 22:25:43 +02:00
Jan Steemann 97d67e4399 use binary-safe string representations 2015-08-20 01:46:20 +02:00
Jan Steemann 9791e467d4 fixed results for query profiling 2015-07-27 18:33:58 +02:00
Jan Steemann 124fe7a34c use new/delete for TRI_vocbase_t 2015-07-20 12:08:13 +02:00
Jan Steemann 91621685fb cache invalidation 2015-06-24 23:02:19 +02:00
Jan Steemann e3a15227ce query cache for AQL queries. no invalidation yet, so results will be wrong when underlying collections are changed 2015-06-23 18:42:58 +02:00
Jan Steemann 3d5ec4d906 speedup for V8-based AQL functions with huge literal array/object parameters 2015-06-12 15:40:08 +02:00
Jan Steemann 03461903fe fixed over-eager V8 context entering and exiting in non-cluster mode 2015-05-27 14:02:13 +02:00
Jan Steemann c88b731acf save some mallocs for strings in AQL 2015-05-18 23:40:28 +02:00
Jan Steemann 80a99ab23e refactored cursor API 2015-03-30 14:58:51 +02:00