mirror of https://gitee.com/bigwinds/arangodb
10 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Jan
|
2c5f79c9fb |
Make scheduler enforce queue limits (#10026)
* initial commit * fix typo * honor @mpoeter 's comments. Thanks! * honor @mpoeter 's comment * adjust scheduler queue sizes * apply suggestion * adjust the PR for 3.5: do not use bounded_push |
|
Michael Hackstein
|
d5840c125a |
Bug fix 3.5/min replication factor (#9524)
* Cherry-pick minReplicationFactor * Bug fix/failover with min replication factor (#9486) * Improve collection time of IResearchQueryOptimizationTest * Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it * Added some assertion son minReplicationFactor * Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled * added minReplicationFactor to the user interface, preparation for the collection api changes * added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo * added minReplicationFactor usage to tests * TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME * minReplicationFactor now able to change via collection properties route * fixed wrongly assert * added minReplicationFactor to the graph management ui * added minReplicationFactor to the gharial api * Fixed off-by-one error in minReplicationFactor. We actually enforced one more. * adjusted description of minReplicationFactor * FollowerInfo Refactoring * added gharial api graph creation tests with minimal replication factor * proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests * added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor * Debug logging * MORE Debug logging * Included replication fast lane * Use correct minreplicationfactor * modified debug logging * Fixed compileissues * MORE Debug logging * MORE Debug logging * MORE Debug logging * MORE Debug logging * MORE Debug logging * MORE Debug logging * MORE Debug logging * Revert "MORE Debug logging" This reverts commit |
|
jsteemann
|
0e24c18253 | re-add optimization for single servers | |
|
Jan
|
207c978d1e
|
Bug fix/move a few actions from js to cpp (#8549) | |
Lars Maier
|
12eebb15fe |
Feature/new server infra (#7733)
* Decoupled IO handling from Scheduler. * Fixed SSL start up bug. * Replaced Scheduler with new worker farm implementation. * Added minimal statistics and info string for Scheduler. * Added support for timed submissions. * Updated delayed submission api. Updated code that used timers. * Extracted new Scheduler into a virtual parent class. The implementation can now depend on the usecase. * Signal handler now working. * Changed threads names, `_stop` is atomic, check for failure during thread start + exception handling like old scheduler did. * Commented on source code and added TODOs. * Played around with start-stop-conditions * Play around with start stop condition. * start stop cond * Sart Stop Conditions * Removed bad cv_status check. * Bug fix: now compare the actual objects instead of pointer values. Setup t1 and t2 depending on the thread id. * Moved most of the stuff now unrelated to the Scheduler to GeneralServer. Got rid of JobGuard. * Instead of waiting for a thread to terminate, put it on a clean up list and check for its termination in each supervisor run. * Allow detaching long running threads. * Fixed test mock. * Updated the WorkHandle logic. Removed post functions. * Fixed crash when obtaining shared_ptr from this in destructor. * Added lost mutex. * Fixed memory leak. * Fixed merge bug. * Changed a lot of code to optimize the scheduler. * Fixed bug of invalidated iterator. Dont remove task on shutdown at different places. Let scheduler threads run until queue is empty. * Only by value calls to queue. * Added options again. * Clean up of code. * UI Request Lane added. * Bug fixes in Scheduler. * Applied reformat. * Use sigaction. |
|
Frank Celler
|
ac9f375fb5 | big reformat | |
Matthew Von-Maszewski
|
97ba8ca2be | Bugfix: More 3.4 scheduler changes backported (#7091) | |
|
Jan
|
43b53bea57
|
document the RequestLanes, and "fix" the lanes of a few handlers (#6152) | |
Frank Celler
|
a688dc0962
|
Feature/remove job queue thread (#5986)
limiting V8 calls in flight |
|
Tobias Gödderz
|
fc3e11dbbc |
Async AQL (#5806)
* Modified header to new initializeCursor API * Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet * Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later * First attempt to simulate thread waiting over information within the query * Small fix to allow for isDirect handlers to go to sleep. * Waiting in the necessary places now for the async request to be send. * Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here? * Include cond-var header * Fixed mutex/cond_var usage * Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested * Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries. * Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts * Fixed endless virtual recursion * Implemented new getOrSkip API in SortBlock * Implemented new getOrSkip API in LimitBlock * Initilaize all variables * Fixed logic bug in SubqueryBlock * getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly! * Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime. * Added IndexBlock::skipSome and IndexBlock::getSome * getBlock now returns its old return value along with the state * Switch from getBlockOld to getBlock in IndexBlock::skipSome * Switch from getBlockOld to getBlock in IndexBlock::getSome * ShortestPathBlock::skipSome is not implemented! Added a regression test * Attempt to fix SubQueryResult memory management * Fixed LIMIT Block * Moved from ShortestPathBlock::getSomeOld to ::getSome * Implemented ASYNC api on SingletonBlock * Adapted EnumerateCollectionBlock to new async API * Fixed FilterBlock and adapted return block to async API * Adapted NORESULTS block to async AQL api. * Adapted Modification Blocks to async API * Fixed some initialize cursor functions to reset values required during get/skipSome * First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still * Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place. * Adapted CalculationBlock to async API * Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN * Adapted EnumerateListBlock to Async AQL api * Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now * Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing * Adapted IReasearchViewBlock to ASYNC AQL API * Fixed SortingGatherBlock in WAITING state. * Adapted IResearch ExecutionBlockMock to Async API * Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍 * Implemented tonly heoretically reachable function of non void function. * Fixed last commit * Added inline TODO comments * fix warning * Fixed a clearing logic bug in RemoveNodes * Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter. * remove unused methods * Fixed failure test * implement skipping * Moved the Query Waiting out of the ExecutionEngine. * changed one of the collect blocks * Removed _upstreamState from ExecutionBlockMock, that is in the base-class now * Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api. * do not check "hasMore" if not necessary * Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type (still uses getBlockOld) * Save state to resume in DistinctCollectBlock::getOrSkipSome * Extracted redundant code * fixed some ops * added one more test * fix endless blocking * fix compile error * fix test * Refactored HashedCollectBlock::getOrSkipSome * Return blocks to the manager * Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome * remove unused shutdown calls, simplify ownership for expressions * Removed superfluous variable * Capture const variable by value * Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome * Added a working version of SortedCollectBlock::getOrSkipSome Has yet to be cleaned up * Removed isTotalAggregation special treatment * On no input, return a group of nulls (instead of no group at all) * Bugfixes * Simplified code * Move return to the end, eliminate duplicate code * Corrected skipped count in HashedCollectBlock * Aligned getNextRow() implementations * Added comments * some cleanup * fix potential memleak * Bugfix * Fixed failure tests * Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome * Replaced hasMore with an async implementation (mostly) * Removed getBlockOld() * Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState) * RemoteBlock now uses the async hasMoreState route * remove job queue * options * Bugfixes in the async implementation of LimitBlock * LimitBlock::getOrSkipSome now always skips when calculating the fullcount * fix compile warnings * restrict threads * Fixed api of Waiting ExecBlockMock. Unused yet * Made SortedGatherBlock async-capable * Removed nonEmptyIndex hack * Removed duplicate traceGetSome~ calls, moved all to getSome * Added asserts before replacing getNr*Registers * Added a TODO note and a comment * Removed getSomeWithoutRegisterClearoutOld() * Removed skip() * Removed common code by using getNr*Registers() * Use getNr*Registers() in the TraversalBlock as well * started to add lane * started to add lane * added lane * completed lane * removed debug output * fixed merge * Began working on a test suite for AQL tracing/profiling * Added more tests and asserts in aql-profiler * Made some ExecutionBlocks final * Added a type enum to all blocks and the per-block stats * Add block type to stats nodes when tracing AQL on block level * Removed initializeCursor call from instantiateFromPlan * Avoided additional getSome calls after DONE * Added more profiler tests * Refactored ExecutionBlock::getOrSkipSome and fixed two bugs - set _upstreamState also when skipping - explicitly use xecutionBlock::getHasMoreState() * Bugfix: update state * Reuse parent _skipped wherever possible; rename where not (LimitBlock) * Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code * Implemented missing virtual function (with USE_FAILURE) * Reset neccessary values during initializeCursor * Simplified code in EnumerateListBlock a little * Added a test for DistinctCollectBlock in aql-profiler * Avoid redundant getSome calls in DistinctCollectBlock * fix compilation * Fixed DistinctCollectBlock profiler test * Added a second profiler test for the DistinctCollectBlock * Added a profiler test for EnumerateCollectionBlock * Bugfix in EnumerateListBlock * added --server.fifoN-size * Simplified EnumerateCollectionBlock::getSome * Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE * Fix testEnumerateCollectionBlock1 for mmfiles * do not pass by reference * Fixed compile error * fixed merge conflicts * Added profiler tests for EnumerateCollectionBlock * Test fix for mmfiles * Fixed IResearch tests * Bugfix in DistinctCollectBlock and a regression test * Updated comment * Bugfix for query statistics in cluster * Check plan in distinct test * Fix aql-profiler tests in cluster * Removed unused line / bugfix for single server test runs * This commit implements waking up of AQL queries. (#5651) * Non-compiling intermediate commit for handover. * Make branch compile again * Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested! * Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine. * First successful query wakeup. * The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler * renamed finalizeExecute to shutdownExecute * Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly * fix audit log * Removed callback from deleteQueryCursor. This cannot be waiting * use CONDITION_LOCKER * removed yet another thread-local variable * Fixed forward declaration * Made RestAqlHandler repeatable * Use defer to close the query in RestAqlHandler. Now waiting will close the query as well. * Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler * Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case * Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded. * Removed reaction to existing nolockheaders in Coordinator Query Planning Phase * Removed incorrect assertion. * Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly. * Fixed UserManager, now deletes nolock headers properly * Swing to the Symphony of Destruction * Forgot about community build... * Fixed compiling of Catch tests * Fixed community build * need thread for size * Made the restSimpleHndler repeatable * Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor * Reactivated StreamingCursors * Removed debug output. * Fixed false query continuation * Reset thread output to non-debug * Added missing return statements * Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers. * Fixed hidden merge conflict * Bugfix in aql-profiler.js: use plan.nodes order, not stats * Added two profiler tests for filter * Avoid too many getBlock calls in the FilterBlock * Removed debug output * RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility. * Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete * Added another test for FILTER, and one test for the HashedCollectBlock * Added more tests for HashedCollectBlock; avoid unneccessary getSome calls * Added an profiler IndexBlock test * IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls * Added a second test profiling IndexBlock * Added a third test for IndexBlock * Moved general code to module * Moved noncluster tests into a separate file * Split aql-profiler testsuite into three files * Added profiler tests for LimitBlock * Added a test for NoResultsBlock * Added profiler tests for TraversalBlock * Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still * Optimized TraversalBlock getSome calls due to new (nightly) test results * Fixed std::min calls I broke * Let shutdown calls in AQL wait, if the query is executed successfully. * Fixed queryResult going out of scope * fix compile error through merge conflict with devel * Fixed compiler warning "mismatching tags" * Removed debug log output * Added TODO notes * Fixed test fail due to devel merge * Fixed some invalid sync waiting implementations * Added a profiler test for SortBlock * Added profiler tests for SortedCollectBlock * Fixed bug introduced by devel merge * Fixed Remoteblocks ignoring errors! * Added some more continue Callbacks in used places. And removed debug log * Removed debug log output * Suppress clang warnings * Bugfix: use of invalid stack pointer * Bugfix: RemoteBlock::shutdown now sends code as int, not string * Revert "Suppress clang warnings" This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0. * Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome * Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed * Allow copy elision |