1
0
Fork 0
Commit Graph

107 Commits

Author SHA1 Message Date
Andrey Abramov e117a97fb4
Feature/arangosearch scatter node (#5536)
* remove collection dependency from scatter

* simplify `createBlock` API

* remove useless IResearchViewScatter node

* some fixes

* code cleanup

* scatter arangosearch view

* cleanup

* fix compilation errors

* fix test issues

* store shared_ptr rather than raw pointer in IResearchViewNode

* refactor ScatterBlock

* refactor DistributeBlock

* ensure no leaks during execution node cloning

* some fixes

* code cleanup

* remove vocbase from `DistributeNode`

* add missing ifdef

* fixes after review
2018-06-06 20:42:53 +03:00
Andrey Abramov 1a83809b84
split gather block (#5506)
* initial split into sorted and unsorted blocks

* some fixes

* move gather blocks out of header

* cleanup

* cleanup

* ensure ExecutionBlock constructor is noexcept

* extract sorting logic into separate SortingStrategy class

* ensure we not rebuild heap each getSome/skipSome call

* cleanup API attempt

* Revert "cleanup API attempt"

This reverts commit c824de1a1d29e2cb7f405138f830cc0e02e7667b.

* Revert "Revert "cleanup API attempt""

This reverts commit 65a2c95f6fdbbcbd00d6582af4d354fe339d4738.

* fix comments

* fixes after review

* more fixes after review
2018-06-01 22:26:51 +03:00
Andrey Abramov f708e46973
Feature/arangosearch gather node (#5487)
* integrate ArangoSearch into GatherNode/GatherBlock logic

* fix inconsistency after merge

* add serialization/deserealization logic for IResearchViewNode sorting

* fix optimizer rule order

* fix copy/past typo

* fix tests

* ensure `GatherNode` is properly set up for ArangoSearch views

* adjust optimizer rule order
2018-05-30 02:53:21 +03:00
Andrey Abramov 0f0235e8a9
Remove collection dependency from gather node (#5457)
* remove collection dependency

* some fixes

* fix test issues

* add tests for newly created functionality

* fix enterprise test fails

* add comment
2018-05-25 17:26:33 +03:00
Jan f55df6786a
blind commit of work, as I managed to break my local git repository (#5456) 2018-05-25 13:07:28 +02:00
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00
Jan 72ec463a8d
Bug fix/cleanup 2005 (#5404) 2018-05-20 20:57:43 +02:00
Vasiliy 843e584746 issue 389.5: refactor StandaloneContext to be constructed with a TRI_vocbase_t& (#5370)
* issue 389.5: refactor StandaloneContext to be constructed with a TRI_vocbase_t&

* backport: address build issues
2018-05-17 01:15:50 +03:00
jsteemann 27e985d5d0 safer casts for ExecutionNode types 2018-05-10 20:08:37 +02:00
Simon fdee0544b7 Using asio::io_context::strands instead of locks (#5266)
* initial try adding strands

* working, stable amount of threads

* improve shell_client cluster

* Fixing some accounting for the scheduler

* Fix accounting

* Fixing wrong strand usage

* add missing return

* Fixing thread accounting

* More scheduler accounting issues

* Fixing various things

* Fixing some stuff

* Fixing some stuff

* Some more subtle bugfixes

* Some cleanup code

* fixing some stuff

* adding some more fixes

* Fixing possible issues

* Fixing missing _storeResult

* Fixing some stuff

* Reducing lambda stack, perhaps fixing hangups

* Fix writeunlocker

* Fixing possible issues

* adding some debugging stuff

* refactor sockets

* possible fixes

* Adding more job guards

* Fixin possible bug

* cleaning up some stuff

* working impl

* Remove debugging output

* Fixing build

* fixing import

* Fixing another bug

* removing debug log

* Removing examples

* Reverting scheduler working code

* Cleanup

* Addressing review comments
2018-05-07 15:58:19 +02:00
Jan 973220e1ab
only clone complex AQL values in SortBlock and GatherBlock (#5253) 2018-05-07 10:53:07 +02:00
Jan 9c0460acff
Feature/aql optimizations (#5237) 2018-05-03 16:55:58 +02:00
Jan 30b12e311b
Bug fix/remove most of aql js (#5223) 2018-04-30 11:17:11 +02:00
Jan 5105e2b3c0
Remove "atLeast" from the internal query APIs (#4848) 2018-03-21 09:10:28 +01:00
Michael Hackstein c1650702bf
Feature/aql server based locking (#4783)
* Started Implementing the ServerBasedlocking. There now is a container that can contain multiple query snippets. It now has to setup the necessary calls to the Servers

* Added backwards linking of QueryEngines, sth. DBServers can contact their Coordinators.

* Added LogTopic AQL

* Made AccessMode::Type Hashable

* Created a Mapping Server => LockLevel => Shard and createad a JSON object containing the Lock information for a complete AQL query per server

* Added code to build coordinator engines

* Finished with first draft of Coordinator-side of new DBServer based locking.

* Added a _api/aql/setup route that creates and locks all snippets/collections for one DBServer in a single go

* Fixed some Coordinator parts

* Index node now gracefully reports if it could not find it's collection when created from vpack. Otherwise it just hardly crashed...

* Modified the Coordinator Snippet collector to be able to handle subqueries properly.

* Started adding GraphNode handling. WIP. Need to deploy engines properly. Coordinator crashes on Graph tests

* Fixed compiler errors

* WIP: EngineInfoContainer

* Separated the EngineInfoContainers for Coordinator and DBServer into different files. They diverged more than anticipated

* Added forgotten files. THe DBServer container now creates the TraverserEngine Mapping and moves it into the Infos. They are not keeping it yet and need to add it to the message as well.

* The DBServer engine infos now persist the TraverserEngine infos. Need to add them to messages though.

* The new aql exec-engine now sends out traverserEngines as well

* Formatting and adding DEBUG level output

* Made the RestAQLHandler aware of the TraverserEngineRegistry. Also created the engines now. Return format changed server-side coordinator side needs fix.

* Adapted the Coordinator side for the DBServer based Shard Locking

* The DBServer based Locking now honors restrictions to certain shards

* Fixed a strange double lock bug in the new AQL Server based locking technique. Add some DEBUG output

* Fixed usage of MAINTAINERMODE macro. The assertion was never active

* Added TestCase for ContainerCoordinatorTest to cmake

* Added -DTEST_VIRTUAL to CMAKE. This is used to define virtual functions for mocking ONLY on test-builds.

* Fixed usage of ENABLE_MAINTAINER_MODE ifdef. CLANG format

* On non-enterprise builds ENTERPRISE_VERT defaults to TEST_VIRTUAL => virtual in test else non-virtual

* Added TEST_VIRTUAL to ExecutionEngine, Query and QueryRegistry

* Added first testcase for EngineInfoContainerCoordinator not yet ready.

* Mode CreateBlock a member function of engine, we have the engine in our hands anyways no need to make it static. Included some more TEST_VIRTUAL functions.

* Fixed clang/MacOs compile error. Added some more TEST_VIRTUAL declarations

* Finally fixed the first buildEngines UnitTest \o/

* Added a unit-test for backward linking of dependencies in CoordinatorPlanner

* Added multi-snippet test for EngineInfoContainerCoordinator

* Removed QueryRegistry.h from central header files and replaced by a forward declartion.

* Added a createBlocks method on the ExecutionEngine. It should be responsible to create all those blocks at once. Adapted the UnitTests as well. Not included Tests for the new createBlocks functionality. Need to mock the options feature first

* Added another test that Coordinator Snippets of queries can be created correctly

* Fixed Coordinator-site cleanup of QueryRegistry, if any of the query creations fails with error, incl UnitTest.

* Added first test for RestAqlHandler::setup. It does only test the setup and gives prepartion for real testing.

* Added a assertion of http return code. Still no creation of queries is tested. Requires a huge amount of mocking.

* fix some deadlocks found by evil lock manager (tm)

* fix duplicate lock

* fix indentation

* ensure proper lock dependencies

* fix lock acquisition

* removed useless comment

* do not lock twice

* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not

* AQL micro optimizations

* use explicit constructor

* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context

entering and exiting the selected context is then done later on without having to hold the ConditionLocker

* remove some recursive locks

* Disable custom deadlock detection when Thread Sanitizer is enabled

* Changing ifdef's

* grr

* broke gcc

* Using atomic for ApplicationServer::_server

* fix premature unlock

* add some asserts

* honor collection locking in cluster

* yet one more lock fix

* removed assertion

* Allow the clustercomm to send nolock headers on count. This is used form within AQL

* IsLocked on transactions will now always yield true IF LOCK_NEVER is set. We simply assume someone else holds the lock for us. Also LOCK_NEVER is now set on collection/count if noLock header is send.

* Moved the flag if collections need to be locked into the TraverserEngines.

* Added enterprise-satellite hooks in EngineInfoContainerDBServer

* Removed now obsolete code

* Replaced throwing of Exception by an ResultObject

* Added some more tests and moved adding snippet to query engine more to the outside.

* Added the AQL result type

* Make the branch compile again

* Register WITH collections for Graphs in the new Collector.

* Fixed test code for failing query clone. Idea was to once clone successfully and second time to fail, we verify that first clone is cleaned up properly. However test failed on first clone...

* Removed a double builderClose

* Added Changelog entry

* Removed empty if

* Removed obsolete todo

* Properly initialize the AqlResult with nullptr on error case

* Updated comment

* Simplified Assertion

* Removed debug output object...

* Added additional catch case for std::exception to get some more error info

* Clarified evaluation order for move case

* Added Explicit

* Fixed cleanup of Coordinator if Registry fails to insert query.

* Allow to use other locks than Read/Write for AQL collections. Not yet in API.

* Updated Comments for other Locks on DBSide. Adapted Destruction CatchTests

* Fixed double builderClose and removed unnecessary double commits

* Added a comment to clarify the state

* Moved error output to trace. Leftover from debugging

* Added some tests for complex subquery patterns

* Added a 'fireAndForgetRequests' methods to cluster comm which allows to send out a bunch of messages but does not wait for their results

* Properly cleanup leftovers of queries if the instantiation step already failed

* Added code comment for fireAndForget

* Added indexes to subquery test to make the plan a bit easier

* The cleanup on DBServerEngines in error case now also cleans up traverser engines.

* Removed unnecessary includes

* Removed debug logging

* Fixed hidden merge conflict
2018-03-20 16:52:19 +01:00
Simon 35136a89c0 Fix some problems with active failover (#4540) 2018-02-09 15:11:53 +01:00
Jan 140452c80c
fix memory hunger of cache (#4270) 2018-01-12 12:14:11 +01:00
Jan d113b755a0
fixed issue #4199: Internal failure: JavaScript exception in file 'arangosh.js' at 98,7: ArangoError 4: Expecting type String (#4202) 2018-01-05 14:51:26 +01:00
Jan Christoph Uhde c8300db459 Improve error handling and tracing of getSome (#4123) 2018-01-03 16:01:36 +01:00
Jan 1ef0ec08c9
fix heap sort in GatherBlock (#3900) 2017-12-06 10:38:12 +01:00
Jan Christoph Uhde 987aea2af3 implement heap merge for cluster blocks (#3617) 2017-11-15 18:05:43 +01:00
Jan e13a611047 fix cache for gather block (#3656) 2017-11-11 22:30:38 +01:00
Jan bef52d7dc3
Bug fix/cleanup after cppcheck (#3639) 2017-11-10 13:53:28 +01:00
Jan 360184be87 fix misleading cluster AQL OOM messages (#3359) 2017-10-13 12:16:18 +02:00
Jan b9d48a4304 use forwarding to build AqlValues in place (#3238) 2017-09-15 14:30:40 +02:00
Simon Grätzer f3eb0c2ac0 No access collections (#3088)
* Added virtual attributes for enterprise on Methods.cpp

* Working no access collections

* align a comment

* Documentation and test fix

* fixing community build
2017-08-25 13:59:03 +02:00
jsteemann 8f92dde3c3 remove only-assigned string variable 2017-06-08 22:20:48 +02:00
Andreas Streichardt 06e32f37a0 aql now properly propagates the error again 2017-06-08 15:56:01 +02:00
Andreas Streichardt 749f4b980b Revert "Revert "Response can always be a nullptr""
This reverts commit c3c76e9a8f.
2017-06-08 15:56:01 +02:00
Andreas Streichardt bf2efb387e Fix unlocking of AQL queries in the cluster 2017-04-10 17:01:24 +02:00
jsteemann 3ed19e291d added "shardIds" option for AQL queries to restrict queries to specific shards 2017-03-08 15:07:10 +01:00
jsteemann f5c8932744 initial version of query execution plan cache, still very rough 2017-02-21 11:58:38 +01:00
jsteemann d49c5c3fc0 simplified internal APIs a bit 2017-02-15 12:42:07 +01:00
jsteemann df3a3eff9f refactoring 2017-02-15 10:26:05 +01:00
jsteemann 01d3ad67b1 Merge branch 'devel' of https://github.com/arangodb/arangodb into engine-api 2017-02-08 00:59:16 +01:00
jsteemann b76eeee92e reuse more AqlItemBlocks 2017-02-08 00:59:06 +01:00
Max Neunhoeffer 883c11ea45 Handle the case that ClusterComm is already shut down gracefully.
This touches every single place where ClusterComm is being used.
2017-02-07 15:31:40 +01:00
jsteemann a3509aa041 centralize access to AqlItemBlocks memory management 2017-02-06 23:33:43 +01:00
Max Neunhoeffer 71dae630c3 Store ClusterComm singleton in shared_ptr. 2017-02-06 15:25:18 +01:00
jsteemann 93426ca720 fix sorting issue in cluster 2017-02-06 09:44:33 +01:00
jsteemann 3c5edeb9b2 fix cluster AQL statistics 2017-02-03 11:47:36 +01:00
jsteemann 88518f6bf8 added "httpRequests" execution stats figure 2017-02-01 18:03:26 +01:00
jsteemann fc5d7311b5 use general BlockCollector in all cases in which it is appropriate 2017-01-31 15:03:24 +01:00
jsteemann 3c2a466c48 use block collector for IndexBlock 2017-01-31 12:16:53 +01:00
Michael Hackstein 36e398f3f4 Fixed a bug with sort optimization in indexes. Also adapted tests. 2017-01-23 09:45:22 +01:00
Max Neunhoeffer d038ba61e7 Implement attributePath access for SortElementVector and in GatherBlock.
Note: This is not yet used anywhere, so the changes should be entirely
neutral. We are going to use this in the optimizer rules
  scatterInClusterRule
and
  distributeInClusterRule
to make the GatherNode do a merge sort, if an index delivers sorted
streams of documents.
2017-01-18 15:32:59 +01:00
jsteemann ac48dbec14 optimization for ItemBlock serialization 2016-12-13 10:00:28 +01:00
jsteemann 369b2c7bc6 added optional memory limit for AQL queries 2016-12-06 13:23:41 +01:00
Max Neunhoeffer 4bf4c7f11a Fixed UPSERT problems in cluster. 2016-10-20 00:58:01 +02:00
Michael Hackstein 59bac5e631 Added some more AQL trace output. 2016-10-19 15:17:44 +02:00