mirror of https://gitee.com/bigwinds/arangodb
7 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Frank Celler
|
ac9f375fb5 | big reformat | |
Simon
|
a79c8af310 | fix non-blocking stream cursors (#6064) | |
Tobias Gödderz
|
fc3e11dbbc |
Async AQL (#5806)
* Modified header to new initializeCursor API * Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet * Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later * First attempt to simulate thread waiting over information within the query * Small fix to allow for isDirect handlers to go to sleep. * Waiting in the necessary places now for the async request to be send. * Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here? * Include cond-var header * Fixed mutex/cond_var usage * Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested * Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries. * Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts * Fixed endless virtual recursion * Implemented new getOrSkip API in SortBlock * Implemented new getOrSkip API in LimitBlock * Initilaize all variables * Fixed logic bug in SubqueryBlock * getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly! * Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime. * Added IndexBlock::skipSome and IndexBlock::getSome * getBlock now returns its old return value along with the state * Switch from getBlockOld to getBlock in IndexBlock::skipSome * Switch from getBlockOld to getBlock in IndexBlock::getSome * ShortestPathBlock::skipSome is not implemented! Added a regression test * Attempt to fix SubQueryResult memory management * Fixed LIMIT Block * Moved from ShortestPathBlock::getSomeOld to ::getSome * Implemented ASYNC api on SingletonBlock * Adapted EnumerateCollectionBlock to new async API * Fixed FilterBlock and adapted return block to async API * Adapted NORESULTS block to async AQL api. * Adapted Modification Blocks to async API * Fixed some initialize cursor functions to reset values required during get/skipSome * First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still * Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place. * Adapted CalculationBlock to async API * Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN * Adapted EnumerateListBlock to Async AQL api * Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now * Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing * Adapted IReasearchViewBlock to ASYNC AQL API * Fixed SortingGatherBlock in WAITING state. * Adapted IResearch ExecutionBlockMock to Async API * Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍 * Implemented tonly heoretically reachable function of non void function. * Fixed last commit * Added inline TODO comments * fix warning * Fixed a clearing logic bug in RemoveNodes * Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter. * remove unused methods * Fixed failure test * implement skipping * Moved the Query Waiting out of the ExecutionEngine. * changed one of the collect blocks * Removed _upstreamState from ExecutionBlockMock, that is in the base-class now * Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api. * do not check "hasMore" if not necessary * Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type (still uses getBlockOld) * Save state to resume in DistinctCollectBlock::getOrSkipSome * Extracted redundant code * fixed some ops * added one more test * fix endless blocking * fix compile error * fix test * Refactored HashedCollectBlock::getOrSkipSome * Return blocks to the manager * Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome * remove unused shutdown calls, simplify ownership for expressions * Removed superfluous variable * Capture const variable by value * Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome * Added a working version of SortedCollectBlock::getOrSkipSome Has yet to be cleaned up * Removed isTotalAggregation special treatment * On no input, return a group of nulls (instead of no group at all) * Bugfixes * Simplified code * Move return to the end, eliminate duplicate code * Corrected skipped count in HashedCollectBlock * Aligned getNextRow() implementations * Added comments * some cleanup * fix potential memleak * Bugfix * Fixed failure tests * Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome * Replaced hasMore with an async implementation (mostly) * Removed getBlockOld() * Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState) * RemoteBlock now uses the async hasMoreState route * remove job queue * options * Bugfixes in the async implementation of LimitBlock * LimitBlock::getOrSkipSome now always skips when calculating the fullcount * fix compile warnings * restrict threads * Fixed api of Waiting ExecBlockMock. Unused yet * Made SortedGatherBlock async-capable * Removed nonEmptyIndex hack * Removed duplicate traceGetSome~ calls, moved all to getSome * Added asserts before replacing getNr*Registers * Added a TODO note and a comment * Removed getSomeWithoutRegisterClearoutOld() * Removed skip() * Removed common code by using getNr*Registers() * Use getNr*Registers() in the TraversalBlock as well * started to add lane * started to add lane * added lane * completed lane * removed debug output * fixed merge * Began working on a test suite for AQL tracing/profiling * Added more tests and asserts in aql-profiler * Made some ExecutionBlocks final * Added a type enum to all blocks and the per-block stats * Add block type to stats nodes when tracing AQL on block level * Removed initializeCursor call from instantiateFromPlan * Avoided additional getSome calls after DONE * Added more profiler tests * Refactored ExecutionBlock::getOrSkipSome and fixed two bugs - set _upstreamState also when skipping - explicitly use xecutionBlock::getHasMoreState() * Bugfix: update state * Reuse parent _skipped wherever possible; rename where not (LimitBlock) * Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code * Implemented missing virtual function (with USE_FAILURE) * Reset neccessary values during initializeCursor * Simplified code in EnumerateListBlock a little * Added a test for DistinctCollectBlock in aql-profiler * Avoid redundant getSome calls in DistinctCollectBlock * fix compilation * Fixed DistinctCollectBlock profiler test * Added a second profiler test for the DistinctCollectBlock * Added a profiler test for EnumerateCollectionBlock * Bugfix in EnumerateListBlock * added --server.fifoN-size * Simplified EnumerateCollectionBlock::getSome * Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE * Fix testEnumerateCollectionBlock1 for mmfiles * do not pass by reference * Fixed compile error * fixed merge conflicts * Added profiler tests for EnumerateCollectionBlock * Test fix for mmfiles * Fixed IResearch tests * Bugfix in DistinctCollectBlock and a regression test * Updated comment * Bugfix for query statistics in cluster * Check plan in distinct test * Fix aql-profiler tests in cluster * Removed unused line / bugfix for single server test runs * This commit implements waking up of AQL queries. (#5651) * Non-compiling intermediate commit for handover. * Make branch compile again * Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested! * Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine. * First successful query wakeup. * The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler * renamed finalizeExecute to shutdownExecute * Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly * fix audit log * Removed callback from deleteQueryCursor. This cannot be waiting * use CONDITION_LOCKER * removed yet another thread-local variable * Fixed forward declaration * Made RestAqlHandler repeatable * Use defer to close the query in RestAqlHandler. Now waiting will close the query as well. * Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler * Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case * Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded. * Removed reaction to existing nolockheaders in Coordinator Query Planning Phase * Removed incorrect assertion. * Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly. * Fixed UserManager, now deletes nolock headers properly * Swing to the Symphony of Destruction * Forgot about community build... * Fixed compiling of Catch tests * Fixed community build * need thread for size * Made the restSimpleHndler repeatable * Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor * Reactivated StreamingCursors * Removed debug output. * Fixed false query continuation * Reset thread output to non-debug * Added missing return statements * Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers. * Fixed hidden merge conflict * Bugfix in aql-profiler.js: use plan.nodes order, not stats * Added two profiler tests for filter * Avoid too many getBlock calls in the FilterBlock * Removed debug output * RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility. * Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete * Added another test for FILTER, and one test for the HashedCollectBlock * Added more tests for HashedCollectBlock; avoid unneccessary getSome calls * Added an profiler IndexBlock test * IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls * Added a second test profiling IndexBlock * Added a third test for IndexBlock * Moved general code to module * Moved noncluster tests into a separate file * Split aql-profiler testsuite into three files * Added profiler tests for LimitBlock * Added a test for NoResultsBlock * Added profiler tests for TraversalBlock * Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still * Optimized TraversalBlock getSome calls due to new (nightly) test results * Fixed std::min calls I broke * Let shutdown calls in AQL wait, if the query is executed successfully. * Fixed queryResult going out of scope * fix compile error through merge conflict with devel * Fixed compiler warning "mismatching tags" * Removed debug log output * Added TODO notes * Fixed test fail due to devel merge * Fixed some invalid sync waiting implementations * Added a profiler test for SortBlock * Added profiler tests for SortedCollectBlock * Fixed bug introduced by devel merge * Fixed Remoteblocks ignoring errors! * Added some more continue Callbacks in used places. And removed debug log * Removed debug log output * Suppress clang warnings * Bugfix: use of invalid stack pointer * Bugfix: RemoteBlock::shutdown now sends code as int, not string * Revert "Suppress clang warnings" This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0. * Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome * Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed * Allow copy elision |
|
|
Simon
|
8c48fdf4ab | Replace rocksdb export cursor with query (#5657) | |
Dan Larkin
|
6c9f12ea85 | Fixed RocksDB export API. | |
jsteemann
|
79cda92a2a | move common code into its own file | |
Dan Larkin
|
e303e2a51f | Added RocksDB export functionality and some bug fixes. |