* Bug fix 3.4/collection babies (#9033)
* Prepare API to create multiple collections in a single request to ClusterMethods to improve speedup
* Added counter on how many collections are successfully created
* Allow multi collection creation one level higher
* CollectionMethods now allow batch createion of Collections
* Improved array size assertions
* Now a graph is createad within a single roundtrip in the agency.
* Added new header files
* Insert collections in the AGENCY with TTL and a isBuilding flag, collections with this flag should not be visisible in the coordinator
* Added forgotten C++ file
* Fixed a rare race condition, and the failing IResearch Tests
* readded callback on DONE, otherwise lists are out of sync
* Fixed assertions to let mocked tests pass...
* Fixed community cluster
* Started fixing IResearch analyzer test, catch-tests are failing ;(
* Solved missed merge-conflict
* Added helper functions in AnalyzerFeature-test
* Refactoring AnalyzerTest Section-Auth
* Refactoring AnalyzerTest Section-Emplace-Duplicates
* Refactoring AnalyzerTest Section-Emplace-Error-Cases. Recovery-Test is now red, it seemed to be green because of invalid test case before.
* Refactoring AnalyzerTest, split GET test into multiple parts, still left 'cluster simulation'.
* Attempt to extract Coordinator / DBServer tests a little bit. This commit starts to break all Coordinator tests. However i am convinced that earlier version did NOT test a cluster situation at all, but some hybrid of SingleServer with full local storage that got told to be a Coordinator from now on, but without any Coordinator setup...
* Temporarly disabled some tests in AnalyzerFeature, as discussed with @gnusi.
* Fixed include guard.
* Temporarily deactivated failing tests
* You shall save your files before you commit...
* Fixed test asserting on plan version, which is now higher than before
* do not persist legacy analyzers into _analyzers table
* fix arangosearch upgrade in cluster
* get rid of Vasiliy's shit
* address review comments
* ensure link is synchronized after creation in upgrade
* fix compilation error
* minor cleanup
* fix tests
* distribute '_analyzers' collection as '_graphs'
* comment out Vasiliy's shit part 2
* Added RemoteExecutor skeleton
* Moved RemoteBlock implementations to ExecutionBlockImpl<RemoteExecutor>
* Remove unnecessary include to avoid unused function warnings
* Fixed gcc compile error
* Moved Scatter/Distribute block implementations to their new Executor versions
* Applied clang-format
* Added factory, infos and a skeleton for the unordered view executor
* Removed assert based on wrong assumption
* Added members from IResearchViewBlockBase to IResearchViewExecutor
* Moved more code into the ViewExecutor, hopefully enough to produce a working version now
* Added missing reset code, made produceRow work mostly correct
* Removed superfluous parentheses to get more useful output from Catch
* Ported fix 923b6e81ac723d1fe37f8e7bf1ab81149f3a08ef
Original commit message was:
Fixed a race condition in RemoteBlock which was triggered during
shutdown overtaking getSome.
* Applied review comments
* Inject input row instead of an item block + pos into the expression context, plus fixed some tests
* Adapted test. Search tests are now green.
* Do not ask upstream when already DONE
* Removed `limit` from next()
* Simplified code that could handle producing more than one document
* Minor readability change
* Solved two TODOs noted in the review
* Removed leftover references to DistributeNode members in the DistributeBlock
* Reverted removal of "exhausted"
* WIP: Implemented variant with scorers
* Fixed compile errors of the last commit
* Fixed some asserts and calculations
* Fixed violated assertions
* Moved files from IResearch/ to Aql/
* Replaced recursive call with a loop
* Worked on a few TODOs
* Removed IResearchViewBlock
* Set input registers correctly
* Eliminated dependency to the Node in the Executor
* Don't misuse the volatility variables for initialization
* Extended a TODO note
* Removed obsolete includes
* Removed an obsolete include from the tests
* Added missing include
* Read PKs in batches
* Fixed merge conflict
* Fixed merge conflict
* Restrict prefetching of PKs to the number of rows in the current output block
* Fixed merge
* Fix IResearch ASan errors
* Revert "Restrict prefetching of PKs to the number of rows in the current output block"
This reverts commit e0fd8698a3.
* Revert "Read PKs in batches"
This reverts commit c06c4d7a36.
* Began some small step refactoring to introduce batch-reading correctly
* Extracted method fillBuffer
* Extracted method evaluateScores
* Minor changes
* Read data from iresearch index in batches
* Replaced std::deque<IndexResult> buffer by a new class
* Solved minor TODOs
* Fixed last commit
* Fixed merge conflict
* Removed accidentally re-added view blocks
* Implemented SharedAqlItemBlockPtr
* Replaced all AqlItemBlockShell, shared_ptr<AqlItemBlock>, unique_ptr<AqlItemBlock> with SharedAqlItemBlockPtr
* Removed AqlItemBlockShell
* Bugfixes
* Added missing noexcept (used in returnBlock())
* Added nullptr constructor/operator= and noexcept specs
* Removed references to the shell
* Implemented review comments
* Fixed a compile error clang somehow ignored
* Two bugfixes and additional asserts
* Fixed ASan error
* protected AqlItemBlock destructor, and some cleanup
* added SharedAqlItemBlockPtr include in QueryCursor.h
* Added some asserts
* Fixed merge conflicts
* protected returnBlock in AqlItemBlockManager
* issue 496.5: minor API cleanup and error reportin enhancements, update iresearch to commit d69f7bd184e009da7bf0a478efd34a0c85b74291
* add workaround for shell-collection-rocksdb-noncluster.js::testSystemSpecial test failure
* fix typo
* Modified header to new initializeCursor API
* Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet
* Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later
* First attempt to simulate thread waiting over information within the query
* Small fix to allow for isDirect handlers to go to sleep.
* Waiting in the necessary places now for the async request to be send.
* Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here?
* Include cond-var header
* Fixed mutex/cond_var usage
* Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested
* Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries.
* Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts
* Fixed endless virtual recursion
* Implemented new getOrSkip API in SortBlock
* Implemented new getOrSkip API in LimitBlock
* Initilaize all variables
* Fixed logic bug in SubqueryBlock
* getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly!
* Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime.
* Added IndexBlock::skipSome and IndexBlock::getSome
* getBlock now returns its old return value along with the state
* Switch from getBlockOld to getBlock in IndexBlock::skipSome
* Switch from getBlockOld to getBlock in IndexBlock::getSome
* ShortestPathBlock::skipSome is not implemented! Added a regression test
* Attempt to fix SubQueryResult memory management
* Fixed LIMIT Block
* Moved from ShortestPathBlock::getSomeOld to ::getSome
* Implemented ASYNC api on SingletonBlock
* Adapted EnumerateCollectionBlock to new async API
* Fixed FilterBlock and adapted return block to async API
* Adapted NORESULTS block to async AQL api.
* Adapted Modification Blocks to async API
* Fixed some initialize cursor functions to reset values required during get/skipSome
* First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still
* Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place.
* Adapted CalculationBlock to async API
* Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN
* Adapted EnumerateListBlock to Async AQL api
* Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now
* Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing
* Adapted IReasearchViewBlock to ASYNC AQL API
* Fixed SortingGatherBlock in WAITING state.
* Adapted IResearch ExecutionBlockMock to Async API
* Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍
* Implemented tonly heoretically reachable function of non void function.
* Fixed last commit
* Added inline TODO comments
* fix warning
* Fixed a clearing logic bug in RemoveNodes
* Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter.
* remove unused methods
* Fixed failure test
* implement skipping
* Moved the Query Waiting out of the ExecutionEngine.
* changed one of the collect blocks
* Removed _upstreamState from ExecutionBlockMock, that is in the base-class now
* Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api.
* do not check "hasMore" if not necessary
* Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type
(still uses getBlockOld)
* Save state to resume in DistinctCollectBlock::getOrSkipSome
* Extracted redundant code
* fixed some ops
* added one more test
* fix endless blocking
* fix compile error
* fix test
* Refactored HashedCollectBlock::getOrSkipSome
* Return blocks to the manager
* Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome
* remove unused shutdown calls, simplify ownership for expressions
* Removed superfluous variable
* Capture const variable by value
* Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome
* Added a working version of SortedCollectBlock::getOrSkipSome
Has yet to be cleaned up
* Removed isTotalAggregation special treatment
* On no input, return a group of nulls (instead of no group at all)
* Bugfixes
* Simplified code
* Move return to the end, eliminate duplicate code
* Corrected skipped count in HashedCollectBlock
* Aligned getNextRow() implementations
* Added comments
* some cleanup
* fix potential memleak
* Bugfix
* Fixed failure tests
* Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome
* Replaced hasMore with an async implementation (mostly)
* Removed getBlockOld()
* Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState)
* RemoteBlock now uses the async hasMoreState route
* remove job queue
* options
* Bugfixes in the async implementation of LimitBlock
* LimitBlock::getOrSkipSome now always skips when calculating the fullcount
* fix compile warnings
* restrict threads
* Fixed api of Waiting ExecBlockMock. Unused yet
* Made SortedGatherBlock async-capable
* Removed nonEmptyIndex hack
* Removed duplicate traceGetSome~ calls, moved all to getSome
* Added asserts before replacing getNr*Registers
* Added a TODO note and a comment
* Removed getSomeWithoutRegisterClearoutOld()
* Removed skip()
* Removed common code by using getNr*Registers()
* Use getNr*Registers() in the TraversalBlock as well
* started to add lane
* started to add lane
* added lane
* completed lane
* removed debug output
* fixed merge
* Began working on a test suite for AQL tracing/profiling
* Added more tests and asserts in aql-profiler
* Made some ExecutionBlocks final
* Added a type enum to all blocks and the per-block stats
* Add block type to stats nodes when tracing AQL on block level
* Removed initializeCursor call from instantiateFromPlan
* Avoided additional getSome calls after DONE
* Added more profiler tests
* Refactored ExecutionBlock::getOrSkipSome and fixed two bugs
- set _upstreamState also when skipping
- explicitly use xecutionBlock::getHasMoreState()
* Bugfix: update state
* Reuse parent _skipped wherever possible; rename where not (LimitBlock)
* Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code
* Implemented missing virtual function (with USE_FAILURE)
* Reset neccessary values during initializeCursor
* Simplified code in EnumerateListBlock a little
* Added a test for DistinctCollectBlock in aql-profiler
* Avoid redundant getSome calls in DistinctCollectBlock
* fix compilation
* Fixed DistinctCollectBlock profiler test
* Added a second profiler test for the DistinctCollectBlock
* Added a profiler test for EnumerateCollectionBlock
* Bugfix in EnumerateListBlock
* added --server.fifoN-size
* Simplified EnumerateCollectionBlock::getSome
* Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE
* Fix testEnumerateCollectionBlock1 for mmfiles
* do not pass by reference
* Fixed compile error
* fixed merge conflicts
* Added profiler tests for EnumerateCollectionBlock
* Test fix for mmfiles
* Fixed IResearch tests
* Bugfix in DistinctCollectBlock and a regression test
* Updated comment
* Bugfix for query statistics in cluster
* Check plan in distinct test
* Fix aql-profiler tests in cluster
* Removed unused line / bugfix for single server test runs
* This commit implements waking up of AQL queries. (#5651)
* Non-compiling intermediate commit for handover.
* Make branch compile again
* Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested!
* Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine.
* First successful query wakeup.
* The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler
* renamed finalizeExecute to shutdownExecute
* Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly
* fix audit log
* Removed callback from deleteQueryCursor. This cannot be waiting
* use CONDITION_LOCKER
* removed yet another thread-local variable
* Fixed forward declaration
* Made RestAqlHandler repeatable
* Use defer to close the query in RestAqlHandler. Now waiting will close the query as well.
* Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler
* Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case
* Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded.
* Removed reaction to existing nolockheaders in Coordinator Query Planning Phase
* Removed incorrect assertion.
* Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly.
* Fixed UserManager, now deletes nolock headers properly
* Swing to the Symphony of Destruction
* Forgot about community build...
* Fixed compiling of Catch tests
* Fixed community build
* need thread for size
* Made the restSimpleHndler repeatable
* Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor
* Reactivated StreamingCursors
* Removed debug output.
* Fixed false query continuation
* Reset thread output to non-debug
* Added missing return statements
* Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers.
* Fixed hidden merge conflict
* Bugfix in aql-profiler.js: use plan.nodes order, not stats
* Added two profiler tests for filter
* Avoid too many getBlock calls in the FilterBlock
* Removed debug output
* RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility.
* Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete
* Added another test for FILTER, and one test for the HashedCollectBlock
* Added more tests for HashedCollectBlock; avoid unneccessary getSome calls
* Added an profiler IndexBlock test
* IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls
* Added a second test profiling IndexBlock
* Added a third test for IndexBlock
* Moved general code to module
* Moved noncluster tests into a separate file
* Split aql-profiler testsuite into three files
* Added profiler tests for LimitBlock
* Added a test for NoResultsBlock
* Added profiler tests for TraversalBlock
* Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still
* Optimized TraversalBlock getSome calls due to new (nightly) test results
* Fixed std::min calls I broke
* Let shutdown calls in AQL wait, if the query is executed successfully.
* Fixed queryResult going out of scope
* fix compile error through merge conflict with devel
* Fixed compiler warning "mismatching tags"
* Removed debug log output
* Added TODO notes
* Fixed test fail due to devel merge
* Fixed some invalid sync waiting implementations
* Added a profiler test for SortBlock
* Added profiler tests for SortedCollectBlock
* Fixed bug introduced by devel merge
* Fixed Remoteblocks ignoring errors!
* Added some more continue Callbacks in used places. And removed debug log
* Removed debug log output
* Suppress clang warnings
* Bugfix: use of invalid stack pointer
* Bugfix: RemoteBlock::shutdown now sends code as int, not string
* Revert "Suppress clang warnings"
This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0.
* Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome
* Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed
* Allow copy elision
Jan
Simon
Dan Larkin-York
Jan Christoph Uhde
Markus Pfeiffer
Michael Hackstein
Andrey Abramov
Tobias Gödderz
Vasiliy
Frank Celler
m0ppers