mirror of https://gitee.com/bigwinds/arangodb
66 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Jan
|
b124113190
|
abort write transactions (#10248) | |
Dan Larkin-York
|
a83c2323c9 | Refactor ApplicationServer stack (#9965) | |
Tobias Gödderz
|
82b0a5e745 |
De-inline in AQL (#10025)
* Enable IPO (LTO) via CMake * Use separate IPO_ENABLED variable * Enabled IPO for more binaries * Enabled IPO for arangobackup * Suppress an MSVC warning (though it is not unjustified...) * Fix static builds, broken due to Policy CMP0060 introduced in cmake 3.3 * Removed policies superseded by minimum cmake version * Disable IPO with google tests due to a g++ bug * Disable IPO with google tests only on AUTO * Disable warning for correct line * Begin de-inlining Aql * de-inlining Aql: 2nd batch * de-inlining Aql: 3rd batch * de-inlining Aql: 4th batch * Moved code out of ifdef * Enabled IPO for additional libs * Fixed some problems found with IPO enabled - Fixed ODR violation in Pregel - Removed unused code in ClusterMethods - Avoid possible uninitialized variable usage * Set IPO globally, except for 3rdParty libs * De-inlined AstHelper again (was undone due to a previous merge conflict) * Added missing .cpp file * Fixed compilation with MSVC * Removed superfluous includes from deinlined AQL header files * Added includes for size_t * Deleted outdated files * De-inlined files that were undone during the merge * Removed includes from files that were undone during merge * Fixed merge conflicts |
|
Michael Hackstein
|
265eb1549d
|
Feature/aql subquery opt shadow row interface (#9987)
* Added first draft of ShadowRow Interface * Refactored RegisterPlan and pulled it out of the ExecutionNode * AqlItemBlock now has an additional hidden register to store subquery depth infromation. Right now no shadow rows are created, however thy could now make use of this situation * Added API and test to insert new shadow rows into an OutputRow. * Extrated test helper function * Added API and tests to consume ShadowRows. Interface is there and compiles with templates, we might need to implement further underlying functions later on as we make progress on ShadowRows, it is only implemented in the minimal way (intentionally) * Added a test for nested ShadowRows and adapted OutputRow accordingly. * Added additional memory include, seems to be required under GCC not udner CLANG * it actually helps to save fies before commiting them... * Fixed serialization/Deserialization of AqlItemBlock to contain the hidden subquery register now * Added a c++ test for AqlItemBlock. It just covers the basics thus far and needs to be improved. * Fixed toVPack of AqlInputRow for subqueries. Also added serialization / deserialization tests for AqlItemRow * Added a c++ test for inputAqlItem row serialization => AqlItemBlock deserialization * Attempt to fix :windows: compile warning * Added a serialization format to be backwards compatible to 3.5 whenever AQL item blocks need to be send over to remote nodes. * Added more tests for serialization of shadow rows, and fixed a bug revealed by this * Apply suggestions from code review Thank you for review! Co-Authored-By: Tobias Gödderz <tobias@arangodb.com> * Review comments. Thanks for spotting * Fixed hidden merge conflicts * Attempt to explain windows on how one can actually cast numbers into other number types... * Update arangod/Aql/AqlItemBlockSerializationFormat.h Co-Authored-By: Tobias Gödderz <tobias@arangodb.com> |
|
Jan Christoph Uhde
|
0b8c75c7b7 | one shard db - devel (#9395) | |
Tobias Gödderz
|
08afad7296 |
Remove AqlItemBlockShell (#8681)
* Added RemoteExecutor skeleton * Moved RemoteBlock implementations to ExecutionBlockImpl<RemoteExecutor> * Remove unnecessary include to avoid unused function warnings * Fixed gcc compile error * Moved Scatter/Distribute block implementations to their new Executor versions * Applied clang-format * Added factory, infos and a skeleton for the unordered view executor * Removed assert based on wrong assumption * Added members from IResearchViewBlockBase to IResearchViewExecutor * Moved more code into the ViewExecutor, hopefully enough to produce a working version now * Added missing reset code, made produceRow work mostly correct * Removed superfluous parentheses to get more useful output from Catch * Ported fix 923b6e81ac723d1fe37f8e7bf1ab81149f3a08ef Original commit message was: Fixed a race condition in RemoteBlock which was triggered during shutdown overtaking getSome. * Applied review comments * Inject input row instead of an item block + pos into the expression context, plus fixed some tests * Adapted test. Search tests are now green. * Do not ask upstream when already DONE * Removed `limit` from next() * Simplified code that could handle producing more than one document * Minor readability change * Solved two TODOs noted in the review * Removed leftover references to DistributeNode members in the DistributeBlock * Reverted removal of "exhausted" * WIP: Implemented variant with scorers * Fixed compile errors of the last commit * Fixed some asserts and calculations * Fixed violated assertions * Moved files from IResearch/ to Aql/ * Replaced recursive call with a loop * Worked on a few TODOs * Removed IResearchViewBlock * Set input registers correctly * Eliminated dependency to the Node in the Executor * Don't misuse the volatility variables for initialization * Extended a TODO note * Removed obsolete includes * Removed an obsolete include from the tests * Added missing include * Read PKs in batches * Fixed merge conflict * Fixed merge conflict * Restrict prefetching of PKs to the number of rows in the current output block * Fixed merge * Fix IResearch ASan errors * Revert "Restrict prefetching of PKs to the number of rows in the current output block" This reverts commit |
|
Heiko
|
8e308680d8 |
Bug fix/exec block cleanup (#8710)
* remove unused functions * more cleanup, virtual functions * rm old functions * more cleanup * more cleanup, rm unused code * rm of inheritRegisters and cleaerRegisters * more cleanup, move functions into IMPL and cluster blocks * used wrong shutdown function * rm not reachable code * moved lots of ExecutionBlock stuff to Impl and ClusterBlocks * trace functions back to executionblock * rm trace * rm empty protected * this getBlock addition, might be useless * more fixes * fixes for the distribute executor, hopefully almost done now * removed obsolete todos * shutdown var order * applied requested changes * re added const * suppress a warning * added forgotten test changes * default destructor, removed not needed function * refactor name * Update tests/CMakeLists.txt Co-Authored-By: hkernbach <hkernbach@users.noreply.github.com> |
|
|
Michael Hackstein
|
48b709ae16
|
Feature/subquery by line (#8553)
Subquery Executor by line |
|
Luschgy
|
b7b84beebf |
Feature/aql blocks by line (#6553)
* Fixed hidden merge conflict * Added BlockFetcher, added a minimal test for SingleRowFetcher * move RegInfo in types.h * fix getSomeLogic * Added the AqlItemMatrix implementation. Not yet tested! * Fixed nr called => Waiting ratio in AllBlocksFetcherMock * fix compiling * add RegInfo to produceRow * BlockFetcherHelper now returns a matrix * pass RegInfo as requested into AqlItemRow ctor I would still keep it out as the ItemRow should be a thin view. * allow to change the AqlItemRows cheaper / remove _sourceRow member * implement copyRow for AqlItemRow * organize cleanup for outputblock used in getsome / shrink if necessary * Added tests for SingleRowFetcher, cleanup in AqlItemRow * start mocking * Added test helpers, began work on a more complex test for SingleRowFetcher * Re-added RegInfo again to AqlItemRow * helper, added test case for FilterExecutor * merge fixed compile * Removed trx name where it is actually not used * Made some functions TEST_VIRTUAL for mocking. * implemented Sort executor * Fixed the BlockFetcherHelper API, to always return a matrix if it is not waiting * Implemented a SortExecutor test with data * disable borken test * Fixed test on SortExecutor. Green now, needs some more testing * change a bit how the AqlItemRow is used in a test * Added another test for the SingleRowFetcher * Added more tests for SingleRowFetcher * fix getRegisterInfo * Moved RegInfo into ExecutorInfos. Also now use the FilterExecutor implementation in production code instead of FilterBlock. Starting integration testing. * add more asserts * Fixed undefined behaviour in TestCode. * Do not shrink an empty output block * switch to block pointers in AqlItemRow * return blocks in the row fetcher * Updated grammar.cpp * Updated (fixed) tests to verify returnBlock * Improved const-ness * Changed BlockFetcher constructor in order to make it mockable * Implemented BlockFetcherMock and rewrote some tests with it * Updated a comment * Updated comments * init commit - EnumerateListExecutor, including EnumerateListExecutorInfos * add potential fix for ItemMatrix * Replaced fakeit with BlockFetcherMock in another GIVEN block * Replaced the last fakeit mocks with BlockFetcherMock and removed fakeit * Updated function definition * included missing arary included * Renamed AqlItemRow to InputAqlItemRow * Split AqlItemRow into Input~ and Output~ classes * EnumerateListExecutor + Tests * Added tests for AqlItemRows * Added test file for AqlItemRows... * Made the ostream operator for AqlItemBlocks available to Logger and Catch info * build error * Added a second test for AqlItemRows. Need to make input space invalid now, sth. acidential referencing will show up * Introduce a block ID in the InputAqlItemRow * Copy output values correctly, avoid InputAqlItemRow pointers * Added a test for registerClearout * Fixed tests * Added a test for adding new values * Added a test for an incomplete row write to not count as produced. * Fixed getSome and a test * Added a test for larger AQL values in AqlItemRows * Fixed compile error in hidden merge conflict * Added tests for Filter counter in FilterExecutor * OutputAqlItemRow now takes responsibility of the block * Handed responsibility of blocks in ExecutionBlockImpl to OutputAqlItemRow * EnumerateListExecutor should be close to its finish. Tests need to be fixed, not ready yet. * better swap * OutputAqlItemRow now takes ExecutorInfos - added several assert - implemented a more strict and safe produced() logic * enum list test * Added a more specific error message * Removed OutputAqlItemRow.changeRow() * fix invalid block freeing * changed order of input/output registers * Fixed a test * Fixed matrix width in test * fix failure tests * finished enumlist tests * Implemented AllRowsFetcher and tests for it * Bugfix in skipSome stub * insert sort and enumerate list executors into exectuion nodes' create functions * fixed singlerowfetcherhelper, added tests * small fixes * add missing Infos * restore accidentally deleted scorer * make it compile * fix executor info creation / usage * make use of output var in EnumarateListNode * Added Executor::Stats and FilterStats * Extended FilterExecutor tests with stats, added node tracing to getSome * Fixed Sort-&EnumerateList- Executor to use the new return type with stats * Fixed compile error in tests * Changed *ExecutorInfo constructors and classes to contain the necessary register informations * Made an ExecutorInfo construction more readable * Updated comments * Fixed checks in OutputAqlItemRow::setValue() * Add comparison method to SortRegister. Forbid construction without one. * Implemented ExecutionBlockImpl::initializeCursor * more mocking inside ExecutionBlockImplTest * Minor changes * Avoid AqlValueGuard destroying the value copied into the output block * Disabled failure tests that do not apply to the new Sort implementation * further mocking of test modules * Removed constexpr for compatibility with outdated gcc versions * add some failure tests - getsome works differently therefor we do not test the old behavior * fix sort node * clean up alq item matrix * Removed the transaction methods from read-only access to AqlValues (get and at) and repliced by CollectionNameResolver where necessary. Still left: some calls extract VPackOptions from the Transaction. Compare needs a BuilderLeaser only. * textexecutionhelper, execimpl test * Eliminated FilterBlock * Eliminated the EnumerateListBlock * first successful running test of executorblockimpl * Eliminated SortBlock * fixed testexecutor helper * Made the ExecutorInfos private to the executors again. * Implement and use AqlItemBlockShell (#6562) * Added AqlItemBlockShell * Minor changes * Mostly replaced AqlItemBlock with ~Shell. Current state: - arangod compiles - OutputAqlItemRow has yet to be changed - catch tests have yet to be fixed * Use AqlItemBlockShell in OutputAqlItemRow * Added registersToKeep to AqlItemBlockShell * Removed ExecutorInfos from OutputAqlItemRow * Split AqlItemBlockShell into Input~ and Output~ * Changed tests so they compile again - they're still not yet working. * Fixed tests * fixed building, modified impl test * Removed now superfluous BlockFetcherMock functions * more execimpl testing * Removed block IDs * more impl testing * Reduced dependencies of BlockFetcher construction * Bugfix, cleared documentation * Removed debug-output * Adressed minor code review remarks * Moved method definitions into .cpp files to avoid includes * Added assertions * Added comments * Addressed several more review comments * Added missing includes * more testing, not finalized yet * added missing header file * added a test for execution block impl * removed old comment * Improved lookup algorithm to identify correct block/row combination in AqlItemMatrix * added test for ExecutionBlockImpl * move utility to the right position * Bugfix and a TODO note * Bugfix: use a uniform ResourceMonitor in tests * Made OutputAqlItemRow::setValue clone by default and copy on move * Replaced the getRow lookup with a binary search * enumerate tests * Disable emplacing of AqlValues in ItemBlock * Reenabled std::enable_if will warn until oskar is using c++17 as standard. * Add helper function to create a shared pointer of an unordered_map<RegisterId> * CalculationExecutor (#6570) * add CalculationExecutor skeleton * first shot of CalculationExecutor * fix minor bugs * handle references * use new shared pointers * commit to share work * fix failure test * reformat with clang-format * add test for CalculationExecutor * fix CalcualtionExecutor test * clean up * Removes unused code * clean up includes * fix/remove comments * add changes requrested by hacki * ReturnExecutor (#7932) * add skeleton to CMakeLists.txt * build most parts of the ReturnExecutor * This commit removes ReturnBlock and adds some debug messages removed inheritRegisters functionality * finish ReturnExecutor * add test for return executor * remove unused code * add stats and clean up includes * verify result in ReturnExecutorTest and move *Stats to "Aql/Stats.h" * Fixed algorithm to find correct row in a MultiRow Input * fix full count and failure tests (#7969) * fix full count and failure tests * fix filter test * WIP: fixes * fixed * NoResultsExecutor (#7953) * add NoResultsExecutor * clean up * some review changes * fix test * fix * fix WAIT -> DONE * remove unused code (#7995) * create skeleton for SingletonExecutor (#7957) * create skeleton for SingletonExecutor * WIP: continue implementation * WIP: add fetcher skel * WIP: add inject function to ConstFetcher * rename SingletonExecutor to IdExecutor * WIP: replace singleton node * WIP: More Debugging on IdExecutor * WIP: IdExecutor * WIP: add failure tests * make iresarch test work but not pass * WIP: add ConstFetcherHelper * fix test * fix tests * fix iresearch tests * fix wording * add review changes * add review change to SingleRowFetcher as well * fix test * fix error when increasing index * temporarily remove fold-expression * clean up * Fix gcc compile error * Feature/aql limit executor (#7982) * limit executor * removed not needed tri assert * rm _invariable in limitnode * find out if we are in a subquery * queryDepth in limit infos * added missing queryDepth * finalizing limit executor * added more tests, fixed a condition * add assert to test * added missing fullcount check * typo * minor changes, fixed test * test * wrong check * fixed tests * false merge * remove basic blocks * remove basic block includes * include sort order * removed not needed basic blocks * removed not needed basic blocks * fix test * Feature/remove input output shells (#8090) * Added template parameters to allow for block pass-through to be chosen at compile time * Adapted the tests to match the new template parameters * Adapted ConstFetcher and IdExecutor to match the new templated fetcher * Implemented pass-through, fixed SingleRowFetcherTest * Fixed implicit merge conflicts * Fix assumptions that are violated by block pass-through * Added 2 regression tests * Fixed test expected data * Fix nullptr dereference * Fix handling of non-invalidated input variables * Fix nullptr dereference * Fix handling of non-invalidated input variables * Fixed compile error * Fixed a useless and leaking allocation * Try a less implicit fix * Allow IdExecutor and ConstFetcher to be used with pass-through * Performance patches from @mchacki * Fixed merge conflicts, and bugs in ConstFetcher * Fixed atMost pass-through, and blocks that call their parents too often * Removed InputAqlItemBlockShell (except from tests) * Removed OutputAqlItemBlockShell (except for tests) * Moved implementations of OutputAqlItemRow methods in the header * Moved SingleRowFetcher::fetchRow to the header to allow inlining * Reenabled returnInheritedResults optimization * Enable passThrough for ReturnExecutor with inherited results * Specialized CalculationExecutor to avoid unneccessary conditional jumps * Made tests compile again (but are currently broken) * Fixed ReturnExecutor test * Move created InputRows out of AqlItemMatrix * Limit- and EnumerateList Executors no longer fetch from upstream after DONE * Added comments * Added experimental AqlItemBlock optimizations * Removed _numEntries, added comments * Allow executors to pass an "atMost"-limit to the SingleRowFetcher (for LIMIT) * Let the LimitExecutor pass atMost upstream * Bugfix for FullCount * Feature/aql enumeratecollection executor (#8028) * initial commit * enum coll executor * buildable version of enumeratecollectionexecutor * wrong usage of outVariable * Added template parameters to allow for block pass-through to be chosen at compile time * Adapted the tests to match the new template parameters * Adapted ConstFetcher and IdExecutor to match the new templated fetcher * more fixes, still some tests failing * Implemented pass-through, fixed SingleRowFetcherTest * Fixed implicit merge conflicts * adding missing set value (remember optimize todo), optimized constructor * removed old enumerate collection blocks * Fix assumptions that are violated by block pass-through * Added 2 regression tests * Fixed test expected data * Fix nullptr dereference * Fix handling of non-invalidated input variables * Fix nullptr dereference * Fix handling of non-invalidated input variables * Fixed compile error * Fixed a useless and leaking allocation * added broken uncompilable executor test * Try a less implicit fix * fix mocking * enumerate collection executor test * remove class, use only static function. * clean up test after merging with main branch * Allow IdExecutor and ConstFetcher to be used with pass-through * Performance patches from @mchacki * Fixed merge conflicts, and bugs in ConstFetcher * Fixed atMost pass-through, and blocks that call their parents too often * Removed InputAqlItemBlockShell (except from tests) * Removed OutputAqlItemBlockShell (except for tests) * Moved implementations of OutputAqlItemRow methods in the header * Moved SingleRowFetcher::fetchRow to the header to allow inlining * Reenabled returnInheritedResults optimization * Enable passThrough for ReturnExecutor with inherited results * Specialized CalculationExecutor to avoid unneccessary conditional jumps * Made tests compile again (but are currently broken) * Fixed ReturnExecutor test * Limit- and EnumerateList Executors no longer fetch from upstream after DONE * Removed an assertion which seems to be incorrect during bootstrap. * Inlined AQL ItemMatrix and removed unused elements in other Executors * Now inline + Move in AqlItemMatrix, seems to be overall best performing * Bugfix: Do not returnInheritedResults on DBServers * Fixed LIMIT test, and made it more thorough. * Fixed createion of NoResultBlock creation in IResearch Nodes. Also relaxed an assertion * Reactivated failure point, now identical with anotherone in OutputRow, need to unified * Bug fix: do not mutate the offset in the LimitExecutor (#8109) * Introduced test * Fixed LimitExecutor to not mutate its infos, so it works correctly in subqueries * Fixed crashes in CatchTests * Fixed non-maintainer build * try to fix subqueries * Fix isInSubQuery for non-singleton nodes. * remove useless while true * Feature/aql blocks by line dan sort (#8143) * Add SortNode::SorterType to SortExecutorInfoss. * prepare imple of sort executor * WIP: add some code for dan sort * WIP: working at pushRows * WIP - try to add Dan's heap-sort * re add missing test * 2 fixes * fix * Dan's tests are working * remove unused variable * remove SortBlock.cpp from CMakeLists.txt * remove hack and provide an extra method to set the _baseIndex * Simplified ContrainedSortExecutor. * fix forgotten variable * update doc * Assert that we do not use functions that might return wrong results. Preconditions or invariants the functions rely on are broken. Therefore the functions can not be used. This is asserted in maintainer mode. * add function to restore row invariant * do not emplace and pop_back if not necessary * Use clear registers in case we can move the rows. * WIndows opst zirkus * Actually destroy AqlItemValue * Feature/traversal executor by line (#8000) * Added stubs for TraversalExecutor, no implementation, not compiled in yet * Included production code of TraversalExecutor and made it compile. Now implement * Implemented production code for trivial tests, if no row is reaching the traverser. * Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next * Intermediate blind commit might not work * Fixed compile issue in test code * Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work. * Fixed travesalExecutor tests. Now able to create tests with actual input. * Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister) * Further test cases * Added stubs for TraversalExecutor, no implementation, not compiled in yet * Included production code of TraversalExecutor and made it compile. Now implement * Implemented production code for trivial tests, if no row is reaching the traverser. * Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next * Intermediate blind commit might not work * Fixed compile issue in test code * Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work. * Fixed travesalExecutor tests. Now able to create tests with actual input. * Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister) * Fixed leftovers from rebase * Refixed mocks in different folder * Fixed Mock import * Fixed Calculators Exector Test to use the unified mocked server. * Implemented logic for the TraversalExecutor. Also added tests for it. Not fully complete though * Added validation of result values * Added usage of constant register in traversal executor * Included filter variables in traersal executor. Added test for const input. And moved MMDR into base traverser class for simplicity * Added ostream operators on AQLValue and AqlInputRow. Not 100% done yet but good enough * Added a shutdownable ExecutionBlock. Some blocks need to implement a specific shutdown still. I added this class to make it easier to eliminate them when the time is due. * Removed ostream operator for INPUT row again, does for some reason not compile on Linux0o * Fixed initialization order * Removed classwrapper for shutdown. * Applied review changes * Refixed failing tests, API is not subject to change * Fixed CatchTest (wrong usage of string literal in test case) * Feature/shortest path executor (#8042) * Added stubs for TraversalExecutor, no implementation, not compiled in yet * Included production code of TraversalExecutor and made it compile. Now implement * Implemented production code for trivial tests, if no row is reaching the traverser. * Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next * Intermediate blind commit might not work * Fixed compile issue in test code * Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work. * Fixed travesalExecutor tests. Now able to create tests with actual input. * Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister) * Further test cases * Added stubs for TraversalExecutor, no implementation, not compiled in yet * Included production code of TraversalExecutor and made it compile. Now implement * Implemented production code for trivial tests, if no row is reaching the traverser. * Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next * Intermediate blind commit might not work * Fixed compile issue in test code * Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work. * Fixed travesalExecutor tests. Now able to create tests with actual input. * Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister) * Fixed leftovers from rebase * Refixed mocks in different folder * Fixed Mock import * Fixed Calculators Exector Test to use the unified mocked server. * Implemented logic for the TraversalExecutor. Also added tests for it. Not fully complete though * Added validation of result values * Added usage of constant register in traversal executor * Included filter variables in traersal executor. Added test for const input. And moved MMDR into base traverser class for simplicity * Added ostream operators on AQLValue and AqlInputRow. Not 100% done yet but good enough * Added a shutdownable ExecutionBlock. Some blocks need to implement a specific shutdown still. I added this class to make it easier to eliminate them when the time is due. * Removed ostream operator for INPUT row again, does for some reason not compile on Linux0o * Fixed initialization order * First draft of shortest path executor * Fixed logic bugs, Local AQL tests succeed * Added first version of ShortestPathExecutor tests. Edge output tests are missing yet. * Added EDGE output tests * Replaced TraversalStats with NoStats, they have never been tracked... * Removed obsolete ShortestPathBlock. * Removed classwrapper for shutdown. * Applied review changes * Refixed failing tests, API is not subject to change * windows macro opst circus * Clear registers per block, not per row (#8164) * Fixed compilation under linux again... |
|
Frank Celler
|
ac9f375fb5 | big reformat | |
|
Tobias Gödderz
|
1800a26297 |
Remove hasMore (#5810)
* Fixed hasMoreState remote communication; cleanup in RemoteBlock::getSome and ::skipSome * Removed /_api/aql/hasMore and all ExecutionBlock::hasMoreState() methods * Removed now useless method RestAqlHandler::getInfoQuery |
|
|
Tobias Gödderz
|
fc3e11dbbc |
Async AQL (#5806)
* Modified header to new initializeCursor API * Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet * Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later * First attempt to simulate thread waiting over information within the query * Small fix to allow for isDirect handlers to go to sleep. * Waiting in the necessary places now for the async request to be send. * Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here? * Include cond-var header * Fixed mutex/cond_var usage * Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested * Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries. * Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts * Fixed endless virtual recursion * Implemented new getOrSkip API in SortBlock * Implemented new getOrSkip API in LimitBlock * Initilaize all variables * Fixed logic bug in SubqueryBlock * getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly! * Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime. * Added IndexBlock::skipSome and IndexBlock::getSome * getBlock now returns its old return value along with the state * Switch from getBlockOld to getBlock in IndexBlock::skipSome * Switch from getBlockOld to getBlock in IndexBlock::getSome * ShortestPathBlock::skipSome is not implemented! Added a regression test * Attempt to fix SubQueryResult memory management * Fixed LIMIT Block * Moved from ShortestPathBlock::getSomeOld to ::getSome * Implemented ASYNC api on SingletonBlock * Adapted EnumerateCollectionBlock to new async API * Fixed FilterBlock and adapted return block to async API * Adapted NORESULTS block to async AQL api. * Adapted Modification Blocks to async API * Fixed some initialize cursor functions to reset values required during get/skipSome * First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still * Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place. * Adapted CalculationBlock to async API * Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN * Adapted EnumerateListBlock to Async AQL api * Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now * Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing * Adapted IReasearchViewBlock to ASYNC AQL API * Fixed SortingGatherBlock in WAITING state. * Adapted IResearch ExecutionBlockMock to Async API * Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍 * Implemented tonly heoretically reachable function of non void function. * Fixed last commit * Added inline TODO comments * fix warning * Fixed a clearing logic bug in RemoveNodes * Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter. * remove unused methods * Fixed failure test * implement skipping * Moved the Query Waiting out of the ExecutionEngine. * changed one of the collect blocks * Removed _upstreamState from ExecutionBlockMock, that is in the base-class now * Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api. * do not check "hasMore" if not necessary * Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type (still uses getBlockOld) * Save state to resume in DistinctCollectBlock::getOrSkipSome * Extracted redundant code * fixed some ops * added one more test * fix endless blocking * fix compile error * fix test * Refactored HashedCollectBlock::getOrSkipSome * Return blocks to the manager * Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome * remove unused shutdown calls, simplify ownership for expressions * Removed superfluous variable * Capture const variable by value * Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome * Added a working version of SortedCollectBlock::getOrSkipSome Has yet to be cleaned up * Removed isTotalAggregation special treatment * On no input, return a group of nulls (instead of no group at all) * Bugfixes * Simplified code * Move return to the end, eliminate duplicate code * Corrected skipped count in HashedCollectBlock * Aligned getNextRow() implementations * Added comments * some cleanup * fix potential memleak * Bugfix * Fixed failure tests * Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome * Replaced hasMore with an async implementation (mostly) * Removed getBlockOld() * Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState) * RemoteBlock now uses the async hasMoreState route * remove job queue * options * Bugfixes in the async implementation of LimitBlock * LimitBlock::getOrSkipSome now always skips when calculating the fullcount * fix compile warnings * restrict threads * Fixed api of Waiting ExecBlockMock. Unused yet * Made SortedGatherBlock async-capable * Removed nonEmptyIndex hack * Removed duplicate traceGetSome~ calls, moved all to getSome * Added asserts before replacing getNr*Registers * Added a TODO note and a comment * Removed getSomeWithoutRegisterClearoutOld() * Removed skip() * Removed common code by using getNr*Registers() * Use getNr*Registers() in the TraversalBlock as well * started to add lane * started to add lane * added lane * completed lane * removed debug output * fixed merge * Began working on a test suite for AQL tracing/profiling * Added more tests and asserts in aql-profiler * Made some ExecutionBlocks final * Added a type enum to all blocks and the per-block stats * Add block type to stats nodes when tracing AQL on block level * Removed initializeCursor call from instantiateFromPlan * Avoided additional getSome calls after DONE * Added more profiler tests * Refactored ExecutionBlock::getOrSkipSome and fixed two bugs - set _upstreamState also when skipping - explicitly use xecutionBlock::getHasMoreState() * Bugfix: update state * Reuse parent _skipped wherever possible; rename where not (LimitBlock) * Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code * Implemented missing virtual function (with USE_FAILURE) * Reset neccessary values during initializeCursor * Simplified code in EnumerateListBlock a little * Added a test for DistinctCollectBlock in aql-profiler * Avoid redundant getSome calls in DistinctCollectBlock * fix compilation * Fixed DistinctCollectBlock profiler test * Added a second profiler test for the DistinctCollectBlock * Added a profiler test for EnumerateCollectionBlock * Bugfix in EnumerateListBlock * added --server.fifoN-size * Simplified EnumerateCollectionBlock::getSome * Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE * Fix testEnumerateCollectionBlock1 for mmfiles * do not pass by reference * Fixed compile error * fixed merge conflicts * Added profiler tests for EnumerateCollectionBlock * Test fix for mmfiles * Fixed IResearch tests * Bugfix in DistinctCollectBlock and a regression test * Updated comment * Bugfix for query statistics in cluster * Check plan in distinct test * Fix aql-profiler tests in cluster * Removed unused line / bugfix for single server test runs * This commit implements waking up of AQL queries. (#5651) * Non-compiling intermediate commit for handover. * Make branch compile again * Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested! * Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine. * First successful query wakeup. * The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler * renamed finalizeExecute to shutdownExecute * Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly * fix audit log * Removed callback from deleteQueryCursor. This cannot be waiting * use CONDITION_LOCKER * removed yet another thread-local variable * Fixed forward declaration * Made RestAqlHandler repeatable * Use defer to close the query in RestAqlHandler. Now waiting will close the query as well. * Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler * Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case * Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded. * Removed reaction to existing nolockheaders in Coordinator Query Planning Phase * Removed incorrect assertion. * Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly. * Fixed UserManager, now deletes nolock headers properly * Swing to the Symphony of Destruction * Forgot about community build... * Fixed compiling of Catch tests * Fixed community build * need thread for size * Made the restSimpleHndler repeatable * Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor * Reactivated StreamingCursors * Removed debug output. * Fixed false query continuation * Reset thread output to non-debug * Added missing return statements * Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers. * Fixed hidden merge conflict * Bugfix in aql-profiler.js: use plan.nodes order, not stats * Added two profiler tests for filter * Avoid too many getBlock calls in the FilterBlock * Removed debug output * RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility. * Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete * Added another test for FILTER, and one test for the HashedCollectBlock * Added more tests for HashedCollectBlock; avoid unneccessary getSome calls * Added an profiler IndexBlock test * IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls * Added a second test profiling IndexBlock * Added a third test for IndexBlock * Moved general code to module * Moved noncluster tests into a separate file * Split aql-profiler testsuite into three files * Added profiler tests for LimitBlock * Added a test for NoResultsBlock * Added profiler tests for TraversalBlock * Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still * Optimized TraversalBlock getSome calls due to new (nightly) test results * Fixed std::min calls I broke * Let shutdown calls in AQL wait, if the query is executed successfully. * Fixed queryResult going out of scope * fix compile error through merge conflict with devel * Fixed compiler warning "mismatching tags" * Removed debug log output * Added TODO notes * Fixed test fail due to devel merge * Fixed some invalid sync waiting implementations * Added a profiler test for SortBlock * Added profiler tests for SortedCollectBlock * Fixed bug introduced by devel merge * Fixed Remoteblocks ignoring errors! * Added some more continue Callbacks in used places. And removed debug log * Removed debug log output * Suppress clang warnings * Bugfix: use of invalid stack pointer * Bugfix: RemoteBlock::shutdown now sends code as int, not string * Revert "Suppress clang warnings" This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0. * Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome * Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed * Allow copy elision |
|
|
Jan
|
edfdd49a3f
|
Bug fix/simplify skip hasmore (#5466)
* simplify skip and hasMore * include less dependencies in header files |
|
|
Jan
|
f55df6786a
|
blind commit of work, as I managed to break my local git repository (#5456) | |
|
Jan
|
72ec463a8d
|
Bug fix/cleanup 2005 (#5404) | |
|
Jan
|
349e2c1681 |
forward-port AQL optimizations from 3.3 (#5020)
* forward-port AQL optimizations from 3.3 * fix restrict-to-single-shard rule * added tests for specific cases * ModificationNodes now can know if they are restricted or not * OptimizerRule now updates the ModificationNode instead of setting ownName on remote node. * The DBServerEngine container now only sends a subset of snippets (those which are relevant) It is able to restrict any part of the query to a certain shard or even a list of shards * ExecutionEngine CreateBlocks in ClusterMode now returns Result instead of throwing. * Adapted Coordinator side to a bitter mapping of RemoteNode => QuerySnippets. This is still ongoing work. * Fixed Coordinator enginecontainer test and finisehd implementation of EngineInfoContainerDBServer. Seems to work so far, need to adapt test to prove it. * Adapted test for modification nodes and restrict to shard. * restrictTo can now be de-/serialized on Modification Nodes * Fixed Community Compile Error * Added a test for Indexes/EnumerateCollection nodes * EnumerateCollection and IndexNode now have options to be restricted to a single shard, opt rule not yet updated * DBServer part of planning now checks Enumerate and IndexNodes for single shard optimization * Added tests for simple single shard optimization on IndexNodes * Adapted optimzierrule for SingleShard optimierung on IndexNodes * Fixed an issue with externally restricted shard ids in combination with shard_id based optimization * Added a test for multiple single shard optimizations on the same collection * Removed collection/Shard duplicate check of optimize to single shard rules * Updated Explainer to handle optimize to single shard * Let more tests pass an optimized AQL variant * Temporarily disabled a tests that can only be solved after some work in transaction |
|
|
Jan
|
5105e2b3c0
|
Remove "atLeast" from the internal query APIs (#4848) | |
|
Michael Hackstein
|
c1650702bf
|
Feature/aql server based locking (#4783)
* Started Implementing the ServerBasedlocking. There now is a container that can contain multiple query snippets. It now has to setup the necessary calls to the Servers * Added backwards linking of QueryEngines, sth. DBServers can contact their Coordinators. * Added LogTopic AQL * Made AccessMode::Type Hashable * Created a Mapping Server => LockLevel => Shard and createad a JSON object containing the Lock information for a complete AQL query per server * Added code to build coordinator engines * Finished with first draft of Coordinator-side of new DBServer based locking. * Added a _api/aql/setup route that creates and locks all snippets/collections for one DBServer in a single go * Fixed some Coordinator parts * Index node now gracefully reports if it could not find it's collection when created from vpack. Otherwise it just hardly crashed... * Modified the Coordinator Snippet collector to be able to handle subqueries properly. * Started adding GraphNode handling. WIP. Need to deploy engines properly. Coordinator crashes on Graph tests * Fixed compiler errors * WIP: EngineInfoContainer * Separated the EngineInfoContainers for Coordinator and DBServer into different files. They diverged more than anticipated * Added forgotten files. THe DBServer container now creates the TraverserEngine Mapping and moves it into the Infos. They are not keeping it yet and need to add it to the message as well. * The DBServer engine infos now persist the TraverserEngine infos. Need to add them to messages though. * The new aql exec-engine now sends out traverserEngines as well * Formatting and adding DEBUG level output * Made the RestAQLHandler aware of the TraverserEngineRegistry. Also created the engines now. Return format changed server-side coordinator side needs fix. * Adapted the Coordinator side for the DBServer based Shard Locking * The DBServer based Locking now honors restrictions to certain shards * Fixed a strange double lock bug in the new AQL Server based locking technique. Add some DEBUG output * Fixed usage of MAINTAINERMODE macro. The assertion was never active * Added TestCase for ContainerCoordinatorTest to cmake * Added -DTEST_VIRTUAL to CMAKE. This is used to define virtual functions for mocking ONLY on test-builds. * Fixed usage of ENABLE_MAINTAINER_MODE ifdef. CLANG format * On non-enterprise builds ENTERPRISE_VERT defaults to TEST_VIRTUAL => virtual in test else non-virtual * Added TEST_VIRTUAL to ExecutionEngine, Query and QueryRegistry * Added first testcase for EngineInfoContainerCoordinator not yet ready. * Mode CreateBlock a member function of engine, we have the engine in our hands anyways no need to make it static. Included some more TEST_VIRTUAL functions. * Fixed clang/MacOs compile error. Added some more TEST_VIRTUAL declarations * Finally fixed the first buildEngines UnitTest \o/ * Added a unit-test for backward linking of dependencies in CoordinatorPlanner * Added multi-snippet test for EngineInfoContainerCoordinator * Removed QueryRegistry.h from central header files and replaced by a forward declartion. * Added a createBlocks method on the ExecutionEngine. It should be responsible to create all those blocks at once. Adapted the UnitTests as well. Not included Tests for the new createBlocks functionality. Need to mock the options feature first * Added another test that Coordinator Snippets of queries can be created correctly * Fixed Coordinator-site cleanup of QueryRegistry, if any of the query creations fails with error, incl UnitTest. * Added first test for RestAqlHandler::setup. It does only test the setup and gives prepartion for real testing. * Added a assertion of http return code. Still no creation of queries is tested. Requires a huge amount of mocking. * fix some deadlocks found by evil lock manager (tm) * fix duplicate lock * fix indentation * ensure proper lock dependencies * fix lock acquisition * removed useless comment * do not lock twice * create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not * AQL micro optimizations * use explicit constructor * only use V8DealerFeature's ConditionLocker for acquiring a free V8 context entering and exiting the selected context is then done later on without having to hold the ConditionLocker * remove some recursive locks * Disable custom deadlock detection when Thread Sanitizer is enabled * Changing ifdef's * grr * broke gcc * Using atomic for ApplicationServer::_server * fix premature unlock * add some asserts * honor collection locking in cluster * yet one more lock fix * removed assertion * Allow the clustercomm to send nolock headers on count. This is used form within AQL * IsLocked on transactions will now always yield true IF LOCK_NEVER is set. We simply assume someone else holds the lock for us. Also LOCK_NEVER is now set on collection/count if noLock header is send. * Moved the flag if collections need to be locked into the TraverserEngines. * Added enterprise-satellite hooks in EngineInfoContainerDBServer * Removed now obsolete code * Replaced throwing of Exception by an ResultObject * Added some more tests and moved adding snippet to query engine more to the outside. * Added the AQL result type * Make the branch compile again * Register WITH collections for Graphs in the new Collector. * Fixed test code for failing query clone. Idea was to once clone successfully and second time to fail, we verify that first clone is cleaned up properly. However test failed on first clone... * Removed a double builderClose * Added Changelog entry * Removed empty if * Removed obsolete todo * Properly initialize the AqlResult with nullptr on error case * Updated comment * Simplified Assertion * Removed debug output object... * Added additional catch case for std::exception to get some more error info * Clarified evaluation order for move case * Added Explicit * Fixed cleanup of Coordinator if Registry fails to insert query. * Allow to use other locks than Read/Write for AQL collections. Not yet in API. * Updated Comments for other Locks on DBSide. Adapted Destruction CatchTests * Fixed double builderClose and removed unnecessary double commits * Added a comment to clarify the state * Moved error output to trace. Leftover from debugging * Added some tests for complex subquery patterns * Added a 'fireAndForgetRequests' methods to cluster comm which allows to send out a bunch of messages but does not wait for their results * Properly cleanup leftovers of queries if the instantiation step already failed * Added code comment for fireAndForget * Added indexes to subquery test to make the plan a bit easier * The cleanup on DBServerEngines in error case now also cleans up traverser engines. * Removed unnecessary includes * Removed debug logging * Fixed hidden merge conflict |
|
Andrey Abramov
|
f6c184df33 | Factory method for execution block (#4729) | |
jsteemann
|
d49c5c3fc0 | simplified internal APIs a bit | |
|
jsteemann
|
7e690a0c61 | renaming | |
|
jsteemann
|
eb73dbf960 | renaming | |
|
jsteemann
|
f09ec85127 | renaming | |
|
Jan Steemann
|
a1c2b7cc63 | fixed issue #1941 | |
|
Michael Hackstein
|
832d2a94f3 |
Squashed commit of the following:
commit 6b5b1fc821f304ce0445220bbf781060bd2995af
Merge: c987b9f 1754e78
Author: Michael Hackstein <michael@arangodb.com>
Date: Tue Jun 7 12:55:24 2016 +0200
Merge branch 'AQL_GRAPHICATION' of github.com:arangodb/arangodb into AQL_GRAPHICATION
commit c987b9f71f75a95a45a32d5cb15ed5570b2ff938
Merge: c4437e2
|
|
|
jsteemann
|
4a7c9f0f68 | simplified bloated comments | |
|
jsteemann
|
279fb105e2 | cppcheck | |
|
Jan Steemann
|
9046e1831b | clang-format | |
|
jsteemann
|
842384016d | namespace cleanup | |
|
jsteemann
|
431900f17a | changed namespace from triagens to arangodb | |
|
Jan Steemann
|
687d6133f0 | comments reformatting | |
|
Jan Steemann
|
3ad20c0cae | auto-generated headers | |
|
Jan Steemann
|
a3b026d9d1 | removed useless sections from code, unified include guards | |
|
Jan Steemann
|
f422576ee0 | clang-format | |
CoDEmanX
|
3a9648d78a | More Americanization | |
|
Jan Steemann
|
0099fe260a | return value optimization for AQL | |
|
Jan Steemann
|
bf3a1daaa9 | recycle AqlItemBlocks | |
|
Jan Steemann
|
7c5cc16a8a | less locking for cluster status checks | |
|
Jan Steemann
|
a327016312 | added method for cluster test | |
Max Neunhoeffer
|
05c069f361 | Another fix for the distributed locking problem in cluster AQL. | |
|
Max Neunhoeffer
|
e50a705d9a |
Squashed commit of the following: organise locking in distributed AQL
Locking is now done in an extra round after the query is fully instanciated in the cluster. All participating shards are locked in alphabetical order of their shard ID (local collection name). For this to work there is a new action in the RestAqlHandler plus a mechanism to prevent the usual locking from happening: Each thread has a thread local static class variable of triagens::arango::Transaction::_makeNolockHeaders which is of type std::unordered_set<std::string>*. Whenever this is not equal to nullptr and a local collection name is stored in there, no locking or unlocking takes place. This information is forwarded by the X-Arango-Nolock HTTP header, whenever an HTTP request is sent via ClusterComm to a shard. |
|
Willi Goesgens
|
5ed8816788 | Remove Explain of cluster snippets | |
|
Willi Goesgens
|
91e04e3230 | Implement outputting of splitted plans which are to be distributed to the db servers | |
|
Jan Steemann
|
54a0f182e2 | call shutdown using an errorCode | |
|
Jan Steemann
|
e2ef21c33a | fixed non-working DistributeBlock | |
|
Willi Goesgens
|
b3beff3053 | Call shutdown() on execution abortion, so we don't leave trash behind on the cluster nodes. | |
|
Jan Steemann
|
c93c952c05 | fixed from/to-handling in cluster | |
|
Jan Steemann
|
4ff58364f1 | fixed cloning of plans | |
|
Willi Goesgens
|
4b399236f1 | Make the Transaction a smart pointer living inside of Query. | |
|
Jan Steemann
|
643ffa104d | moved isCoordinator to function of its own |