1
0
Fork 0
Commit Graph

156 Commits

Author SHA1 Message Date
Tobias Gödderz fc3e11dbbc Async AQL (#5806)
* Modified header to new initializeCursor API

* Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet

* Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later

* First attempt to simulate thread waiting over information within the query

* Small fix to allow for isDirect handlers to go to sleep.

* Waiting in the necessary places now for the async request to be send.

* Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here?

* Include cond-var header

* Fixed mutex/cond_var usage

* Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested

* Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries.

* Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts

* Fixed endless virtual recursion

* Implemented new getOrSkip API in SortBlock

* Implemented new getOrSkip API in LimitBlock

* Initilaize all variables

* Fixed logic bug in SubqueryBlock

* getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly!

* Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime.

* Added IndexBlock::skipSome and IndexBlock::getSome

* getBlock now returns its old return value along with the state

* Switch from getBlockOld to getBlock in IndexBlock::skipSome

* Switch from getBlockOld to getBlock in IndexBlock::getSome

* ShortestPathBlock::skipSome is not implemented! Added a regression test

* Attempt to fix SubQueryResult memory management

* Fixed LIMIT Block

* Moved from ShortestPathBlock::getSomeOld to ::getSome

* Implemented ASYNC api on SingletonBlock

* Adapted EnumerateCollectionBlock to new async API

* Fixed FilterBlock and adapted return block to async API

* Adapted NORESULTS block to async AQL api.

* Adapted Modification Blocks to async API

* Fixed some initialize cursor functions to reset values required during get/skipSome

* First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still

* Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place.

* Adapted CalculationBlock to async API

* Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN

* Adapted EnumerateListBlock to Async AQL api

* Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now

* Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing

* Adapted IReasearchViewBlock to ASYNC AQL API

* Fixed SortingGatherBlock in WAITING state.

* Adapted IResearch ExecutionBlockMock to Async API

* Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍

* Implemented tonly heoretically reachable function of non void function.

* Fixed last commit

* Added inline TODO comments

* fix warning

* Fixed a clearing logic bug in RemoveNodes

* Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter.

* remove unused methods

* Fixed failure test

* implement skipping

* Moved the Query Waiting out of the ExecutionEngine.

* changed one of the collect blocks

* Removed _upstreamState from ExecutionBlockMock, that is in the base-class now

* Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api.

* do not check "hasMore" if not necessary

* Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type

(still uses getBlockOld)

* Save state to resume in DistinctCollectBlock::getOrSkipSome

* Extracted redundant code

* fixed some ops

* added one more test

* fix endless blocking

* fix compile error

* fix test

* Refactored HashedCollectBlock::getOrSkipSome

* Return blocks to the manager

* Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome

* remove unused shutdown calls, simplify ownership for expressions

* Removed superfluous variable

* Capture const variable by value

* Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome

* Added a working version of SortedCollectBlock::getOrSkipSome

Has yet to be cleaned up

* Removed isTotalAggregation special treatment

* On no input, return a group of nulls (instead of no group at all)

* Bugfixes

* Simplified code

* Move return to the end, eliminate duplicate code

* Corrected skipped count in HashedCollectBlock

* Aligned getNextRow() implementations

* Added comments

* some cleanup

* fix potential memleak

* Bugfix

* Fixed failure tests

* Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome

* Replaced hasMore with an async implementation (mostly)

* Removed getBlockOld()

* Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState)

* RemoteBlock now uses the async hasMoreState route

* remove job queue

* options

* Bugfixes in the async implementation of LimitBlock

* LimitBlock::getOrSkipSome now always skips when calculating the fullcount

* fix compile warnings

* restrict threads

* Fixed api of Waiting ExecBlockMock. Unused yet

* Made SortedGatherBlock async-capable

* Removed nonEmptyIndex hack

* Removed duplicate traceGetSome~ calls, moved all to getSome

* Added asserts before replacing getNr*Registers

* Added a TODO note and a comment

* Removed getSomeWithoutRegisterClearoutOld()

* Removed skip()

* Removed common code by using getNr*Registers()

* Use getNr*Registers() in the TraversalBlock as well

* started to add lane

* started to add lane

* added lane

* completed lane

* removed debug output

* fixed merge

* Began working on a test suite for AQL tracing/profiling

* Added more tests and asserts in aql-profiler

* Made some ExecutionBlocks final

* Added a type enum to all blocks and the per-block stats

* Add block type to stats nodes when tracing AQL on block level

* Removed initializeCursor call from instantiateFromPlan

* Avoided additional getSome calls after DONE

* Added more profiler tests

* Refactored ExecutionBlock::getOrSkipSome and fixed two bugs

- set _upstreamState also when skipping
- explicitly use xecutionBlock::getHasMoreState()

* Bugfix: update state

* Reuse parent _skipped wherever possible; rename where not (LimitBlock)

* Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code

* Implemented missing virtual function (with USE_FAILURE)

* Reset neccessary values during initializeCursor

* Simplified code in EnumerateListBlock a little

* Added a test for DistinctCollectBlock in aql-profiler

* Avoid redundant getSome calls in DistinctCollectBlock

* fix compilation

* Fixed DistinctCollectBlock profiler test

* Added a second profiler test for the DistinctCollectBlock

* Added a profiler test for EnumerateCollectionBlock

* Bugfix in EnumerateListBlock

* added --server.fifoN-size

* Simplified EnumerateCollectionBlock::getSome

* Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE

* Fix testEnumerateCollectionBlock1 for mmfiles

* do not pass by reference

* Fixed compile error

* fixed merge conflicts

* Added profiler tests for EnumerateCollectionBlock

* Test fix for mmfiles

* Fixed IResearch tests

* Bugfix in DistinctCollectBlock and a regression test

* Updated comment

* Bugfix for query statistics in cluster

* Check plan in distinct test

* Fix aql-profiler tests in cluster

* Removed unused line / bugfix for single server test runs

* This commit implements waking up of AQL queries. (#5651)

* Non-compiling intermediate commit for handover.

* Make branch compile again

* Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested!

* Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine.

* First successful query wakeup.

* The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler

* renamed finalizeExecute to shutdownExecute

* Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly

* fix audit log

* Removed callback from deleteQueryCursor. This cannot be waiting

* use CONDITION_LOCKER

* removed yet another thread-local variable

* Fixed forward declaration

* Made RestAqlHandler repeatable

* Use defer to close the query in RestAqlHandler. Now waiting will close the query as well.

* Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler

* Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case

* Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded.

* Removed reaction to existing nolockheaders in Coordinator Query Planning Phase

* Removed incorrect assertion.

* Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly.

* Fixed UserManager, now deletes nolock headers properly

* Swing to the Symphony of Destruction

* Forgot about community build...

* Fixed compiling of Catch tests

* Fixed community build

* need thread for size

* Made the restSimpleHndler repeatable

* Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor

* Reactivated StreamingCursors

* Removed debug output.

* Fixed false query continuation

* Reset thread output to non-debug

* Added missing return statements

* Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers.

* Fixed hidden merge conflict

* Bugfix in aql-profiler.js: use plan.nodes order, not stats

* Added two profiler tests for filter

* Avoid too many getBlock calls in the FilterBlock

* Removed debug output

* RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility.

* Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete

* Added another test for FILTER, and one test for the HashedCollectBlock

* Added more tests for HashedCollectBlock; avoid unneccessary getSome calls

* Added an profiler IndexBlock test

* IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls

* Added a second test profiling IndexBlock

* Added a third test for IndexBlock

* Moved general code to module

* Moved noncluster tests into a separate file

* Split aql-profiler testsuite into three files

* Added profiler tests for LimitBlock

* Added a test for NoResultsBlock

* Added profiler tests for TraversalBlock

* Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still

* Optimized TraversalBlock getSome calls due to new (nightly) test results

* Fixed std::min calls I broke

* Let shutdown calls in AQL wait, if the query is executed successfully.

* Fixed queryResult going out of scope

* fix compile error through merge conflict with devel

* Fixed compiler warning "mismatching tags"

* Removed debug log output

* Added TODO notes

* Fixed test fail due to devel merge

* Fixed some invalid sync waiting implementations

* Added a profiler test for SortBlock

* Added profiler tests for SortedCollectBlock

* Fixed bug introduced by devel merge

* Fixed Remoteblocks ignoring errors!

* Added some more continue Callbacks in used places. And removed debug log

* Removed debug log output

* Suppress clang warnings

* Bugfix: use of invalid stack pointer

* Bugfix: RemoteBlock::shutdown now sends code as int, not string

* Revert "Suppress clang warnings"

This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0.

* Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome

* Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed

* Allow copy elision
2018-07-09 14:24:10 +02:00
Jan Christoph Uhde a2dcb6cc5d WIP - start adding optional overwrite to insert operation (RepSert) (#5268) 2018-05-24 19:47:15 +02:00
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00
Vasiliy 635db3b409 issue 355.4: remove redundant methods and code, use 'cp' instead of 'cmake copy_directory' where possible, use vocbase reference instead of pointer 2018-04-04 10:53:48 +03:00
Simon fe9768dc44 Replace /_api/simple/by-example (#4779)
* Replacing /_api/simple/by-example

* Requested changes
2018-03-09 09:35:48 +01:00
Jan b4f6ee9273 Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
Simon Grätzer ee8209943f Missing things for active / passive (#3578)
* Switching from ttl to supervision based failover mechanism

* Allowing canceling of ongoing actions

* refactored asyncjobmanager

* refactoring some code

* adding read-only flag

* catching some exceptions to reduce log pollution, removing unnecessary code, removing tests for _changeMode

* fixing "createsANewDatabaseWithAnInvalidUser"

* auth = off does not longer make everyone superuser

* Fixing cluster_sync and maybe resilience
2017-11-04 20:30:23 +01:00
Simon Grätzer 64e9377c05 Replacing /_api/collection with RestHandler (#3543) 2017-11-02 14:57:17 +01:00
Simon Grätzer 7c31960cf2 Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
Dan Larkin 750fc96ab4 Converted endpoint handler from JS to C++ (#2905)
* Added native C++ rest handler for endpoints.

* Added test of rest endpoint interface and removed JS handler.
2017-08-01 09:53:04 +02:00
Frank Celler bbe7484521 Feature/auth context (#2704)
* added read-only users
2017-07-02 23:15:57 +02:00
Frank Celler 2807ef559c Feature/move shard fix (#2626)
Major overhaul of handling of synchronous replication.
2017-06-26 16:55:01 +02:00
Simon Grätzer 492d832695 Refactored /_api/index and /_api/database (#2582)
* working documents rest handler

* fixed cluster tests

* Consolidating database APIs

* clang-format

* Fixing issue with user creation through db._createDatabase

* replaced and refactored api-index and index api

* fixed cluster

* renaming some files

* added user methods

* removed files intended for later

* Fixed CC build

* Fixed method signature

* Fixing shell_server, shell_client tests
2017-06-19 23:47:40 +02:00
Jan Christoph Uhde b83ae2ab82 refactor some code to make use of arangodb::Result 2017-03-30 09:39:21 +02:00
jsteemann b8e011eee5 move attributes into engine 2017-03-21 10:31:50 +01:00
Dan Larkin 966e05b596 Added very basic rest handler for views. 2017-03-14 19:05:14 -04:00
jsteemann 0a07bb3222 make RestWalHandler a handler of the MMFiles engine 2017-03-10 09:00:37 +01:00
jsteemann 6f9927fea0 avoid constructing temporary lookup strings 2017-01-19 14:27:04 +01:00
jsteemann a655a0e5c8 removed unused parameter 2016-12-19 09:56:37 +01:00
jsteemann 504b102268 added HTTP REST APIs for online loglevel adjustments:
- GET `/_admin/log/level` returns the current loglevel settings
- PUT `/_admin/log/level` modifies the current loglevel settings
2016-09-08 11:15:49 +02:00
jsteemann 7ae36773ec moved implementation of `/_admin/wal` REST handlers to C++ 2016-09-07 13:48:52 +02:00
Michael Hackstein e22014a272 Merge branch 'MIT' of github.com:arangodb/arangodb into generic-col-types 2016-09-06 09:33:23 +02:00
jsteemann 2134afaa83 renamed method 2016-08-30 10:36:34 +02:00
Jan Christoph Uhde 1febeadfd7 finish ResponseCode enum 2016-08-23 15:46:42 +02:00
Michael Hackstein 3dac8c27ed Implemented infrastructure for the TraversalEngines. We can now create and delete these engines accross several requests/threads. Right now the engines are empty. 2016-07-27 10:08:45 +02:00
Jan Christoph Uhde b2e29c4353 fix: createResponse -> setResponseCode; remove delete
remove delete _response in RestEdgesHandler
2016-07-07 08:54:41 +00:00
Frank Celler 38a55ed13c Merge branch 'devel' of github.com:arangodb/arangodb into FMH 2016-07-04 17:09:40 +02:00
Frank Celler af0b0900af WIP 2016-05-22 21:08:37 +02:00
Jan Steemann 8e090e642e removed "createCollection" attribute from APIs 2016-05-12 15:17:45 +02:00
Michael Hackstein 7b8d3d6df7 Insertion of documents in Cluster (babies case) now passes through the X-Arango-Error-Codes. 2016-05-09 12:45:31 +02:00
Michael Hackstein 0e2d31f633 Moved GET _api/document?collection=xxx to PUT _api/simple/all-keys {collection: xxx}. Adapted all tests. Also removed Transaction::allKeys which got obsolete by this commit. 2016-04-15 11:23:25 +02:00
Jan Christoph Uhde c3c403532b client is now able to request velocypack 2016-04-08 13:29:27 +02:00
Jan Steemann e37029aec6 removed unused private variable 2016-04-07 18:52:32 +02:00
Jan Steemann 9f8c24b1cd Merge branch 'devel' of https://github.com/arangodb/arangodb into spdvpk 2016-04-06 15:43:27 +02:00
Frank Celler 068f9defd7 splitted HttpResponse.h 2016-04-01 10:39:38 +02:00
Frank Celler 486011d399 Merge branch 'devel' of github.com:arangodb/arangodb into spdvpk 2016-03-31 13:41:09 +02:00
Frank Celler 494bdca847 splitted HttpRequest into GeneralRequest/HttpRequest 2016-03-31 11:45:21 +02:00
Jan Steemann 9ac39b7ad6 Merge branch 'devel' of https://github.com/arangodb/arangodb into spdvpk 2016-03-23 16:36:30 +01:00
jsteemann fb50c5c0bb removed wrong location header 2016-03-15 10:59:28 +01:00
Max Neunhoeffer 627ef39b81 Finish returnNew and returnOld for RestDocumentHandler. 2016-03-11 23:16:45 +01:00
Max Neunhoeffer e77159b008 New returnNew option for insert. 2016-03-11 12:44:36 +01:00
Max Neunhoeffer 9074f99f09 Get rid of update policy in RestHandlers. 2016-03-10 00:02:47 +01:00
Jan Steemann 6eea757c01 simplifications 2016-03-04 19:40:24 +01:00
Jan Steemann ad4a9542ea removed file 2016-03-04 18:32:54 +01:00
Jan Steemann 4de5e41044 simplifications 2016-03-04 17:48:27 +01:00
Jan Steemann eb82d96836 constexpr 2016-03-04 12:33:59 +01:00
Jan Steemann eae225f4c8 changed marker storage 2016-02-29 18:33:16 +01:00
jsteemann f0aeab0d12 cleanup 2016-02-27 01:10:42 +01:00
Jan Steemann ebf2d1520b removed TRI_doc_mptr_copy_t 2016-02-26 11:46:33 +01:00
Kaveh Vahedipour 34976d462f agency on 2016-02-26 09:31:05 +01:00