* Applied review cAdded sceleton optimizer functionomments
* Added mask for rule
* Applied review cAdded sceleton optimizer functionomments
* Added mask for rule
* adding plan traversal for rule
* Rule execution block developing
* Experiment with additional block
* LIMIT node modifications
* implemented late materialization base version.
* Code cleanup
* Added check for variable usage
* Implemented serialization and cloning
* Fixed build errors
* Implemented late materialization for LimitNode
* Moved materialization to Sort Node
* Removed LimitNode modiications
* Implemented skipRows to speedup single server
* Added tests for optimizer rule
* fixed build
* Fix mac build
* Fixed mac build
* Fixed PR test run errors
* Added tmp fix for cluster test runs
* removed debug print
* Make materialization stick to last sort node. Add late materialization to explainer
* Updated optimization rule to search optimal SortNode to do materialization
* Refined optimization rules for cluster
* Added tests for SortExecutor and IResearchViewNode
* Fixed mac buid. Removed misplaced assertion.
* Fixed build
* Updated tests. Coded cleanup
* Code cleanup
* made function name for SortedExecutor match name in ConstraintSortedExecutor
* cleanup
* Fixed test run
* made scoring results stable across cluster
* Make optimize rule stop if sort node for limit node is not suitable
* Disabled constrained sort for materializing SortNode
* reverted SortNode modifications
* Implemented separate materializer node
* Fixed tests to account new materialize node
* Code cleanup
* reverted debugging change
* Fixed test jslint error and comments
* MaterializeNode renamed
* enum value renamed for materialize node
* Code cleanup
* code cleanup and optimization
* Fixed clang warning
* Applied review comments
* Out variable planning moved to materialize node
* inlined getters in ReadContex
* Replace template <bool blockPassthrough>
by template <BlockPassthrough blockPassthrough> where BlockPassthrough is an
enum class.
This is mainly for better readability and for some type safety.
* Fixup some implicit conversions to bool
* Add Start/End subquery node skeleton
* Added a test query that triggers complex interna in the future and needs to proof that our work is side-effect free.
* Added first draft of ShadowRow Interface
* Refactored RegisterPlan and pulled it out of the ExecutionNode
* AqlItemBlock now has an additional hidden register to store subquery depth infromation. Right now no shadow rows are created, however thy could now make use of this situation
* Added API and test to insert new shadow rows into an OutputRow.
* Extrated test helper function
* Add template function to statically assert size of type
* Add Start/End subquery node skeleton
* Add Start/End subquery node creation to ExecutionNode
* Cleanup SubqueryEndNode
* Add test file
* Add isEqualTo to ExecutionNode
* Subquery nodes cleanup
* Update ExecutionNode tests
* Added API and tests to consume ShadowRows. Interface is there and compiles with templates, we might need to implement further underlying functions later on as we make progress on ShadowRows, it is only implemented in the minimal way (intentionally)
* Updates to testing code
* Added a test for nested ShadowRows and adapted OutputRow accordingly.
* More updates
* Add optimizer rule
* Fix optimizer rule
* Added additional memory include, seems to be required under GCC not udner CLANG
* it actually helps to save fies before commiting them...
* Fix optimizer rule (again)
* Fixed serialization/Deserialization of AqlItemBlock to contain the hidden subquery register now
* Added a c++ test for AqlItemBlock. It just covers the basics thus far and needs to be improved.
* Fixed toVPack of AqlInputRow for subqueries. Also added serialization / deserialization tests for AqlItemRow
* Added a c++ test for inputAqlItem row serialization => AqlItemBlock deserialization
* Add a public function to get plan from query
* Fix register planning
* Add test for splicing subquery nodes
* add static asserts for sizes of SubqueryStartNode and SubqueryEndNode
* Add comment about statically asserted size
* Fix isEqualTo virtual and override dance.
* Attempt to fix :windows: compile warning
* Fix isEqualTo, remove static_assert_size
* Remove a TODO; the plan owns the ExecutionNodes
* Only add stealPlan to Query when testing
* Remove another static_assert_size
* Fixed mac compile issues
* Added a serialization format to be backwards compatible to 3.5 whenever AQL item blocks need to be send over to remote nodes.
* Handle ShadowAqlItemRow in SingleRowFetcher
* Move some code from header to cpp
* Added more tests for serialization of shadow rows, and fixed a bug revealed by this
* Apply suggestions from code review
Thank you for review!
Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
* Review comments. Thanks for spotting
* Fixed hidden merge conflicts
* Removed unused variable
* Added first test on ShadowRows in SingleRowFetcher. Found inconvenience in original code with it.
* Added a test for multi level shadow rows
* Added a test and fixed production for multiple consecutirve shadow rows.
* Added tests for pass through blocks and shadow rows
* Attempt to explain windows on how one can actually cast numbers into other number types...
* Add SubqueryEndExecutor
* Adapted RowFetcherHelper to be able to work on ShadowRows
* Fixed minor error in RowFetcherHelper that causes LIMIT tests to error
* Temporarily disabled test suites (sorry there is only one 'test case' in this suites that does all or nothing tests ...) as it tests a feature we cannot support in this intermediate state, we need to default disable our new rule first.
* IMplemented missing function to copy over SubqueryDepth between blocks
* Seperated assert, to see faster which part is violated
* Update arangod/Aql/AqlItemBlockSerializationFormat.h
Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
* Disabled subquery splice rule by default
* Forgotten merge marker
* temporarily modified test
* Some Subquery End Executor work
* Fix RowFetcherHelper to work with ShadowRows
* Add tests for SubqueryEndExecutor
* Rewrite SubqueryEndExecutor
* Update test
* Add createBlock code for SubqueryEndExecutionNode
* Some fixups adding ExecutionBlockImpl for SubqueryEndExecutor
* Fix SubqueryEndExecutor
* Update SubqueryEndExecutorTest
* Update endexecutor test
* Fix the endnode executor once more
* Fix some merge fallout
* Update test
* Add missing advanceRow to SubqueryEndExecutor
* SubqueryEndExecutor Tests
* Activate a death test
* Better handling of death test
* RowFetcherTest refactoring Part 1, still ongoing. We use this for AqlItemMatrix and SingleRowFetcher
* Update and cleanup the SubqueryEndExecutor tests
* Check that expected shadow row depths are met
* Remove commented out code in AqlItemBlock.cpp
* Remove a misplaced comment
* Sort out SubqueryEndExecutor includes
* Remove superflous include
* Consistently use NoStats{} in SubqueryEndExecutor
* Make SubqueryEndExecutor properties constexpr
* Reactivate IResearchTests
* Do not copy the accumulator in SubqueryEndNode
* Return a more sensible value for expectedNumberOfRows of SubqueryEndNodeExecutor
* Constify an argument
* Fix ownership problems with the buffer for _accumulator
* Cleanup SubqueryEndExecutor tests
Also add a test that writes to a register other than 0
* Patch RowFetcherHelper to count the number of Items returned instead of number
of calls
* Make MSVC happy
* Enable IPO (LTO) via CMake
* Use separate IPO_ENABLED variable
* Enabled IPO for more binaries
* Enabled IPO for arangobackup
* Suppress an MSVC warning (though it is not unjustified...)
* Fix static builds, broken due to Policy CMP0060 introduced in cmake 3.3
* Removed policies superseded by minimum cmake version
* Disable IPO with google tests due to a g++ bug
* Disable IPO with google tests only on AUTO
* Disable warning for correct line
* Begin de-inlining Aql
* de-inlining Aql: 2nd batch
* de-inlining Aql: 3rd batch
* de-inlining Aql: 4th batch
* Moved code out of ifdef
* Enabled IPO for additional libs
* Fixed some problems found with IPO enabled
- Fixed ODR violation in Pregel
- Removed unused code in ClusterMethods
- Avoid possible uninitialized variable usage
* Set IPO globally, except for 3rdParty libs
* De-inlined AstHelper again (was undone due to a previous merge conflict)
* Added missing .cpp file
* Fixed compilation with MSVC
* Removed superfluous includes from deinlined AQL header files
* Added includes for size_t
* Deleted outdated files
* De-inlined files that were undone during the merge
* Removed includes from files that were undone during merge
* Fixed merge conflicts
* Added first draft of ShadowRow Interface
* Refactored RegisterPlan and pulled it out of the ExecutionNode
* AqlItemBlock now has an additional hidden register to store subquery depth infromation. Right now no shadow rows are created, however thy could now make use of this situation
* Added API and test to insert new shadow rows into an OutputRow.
* Extrated test helper function
* Added API and tests to consume ShadowRows. Interface is there and compiles with templates, we might need to implement further underlying functions later on as we make progress on ShadowRows, it is only implemented in the minimal way (intentionally)
* Added a test for nested ShadowRows and adapted OutputRow accordingly.
* Added additional memory include, seems to be required under GCC not udner CLANG
* it actually helps to save fies before commiting them...
* Fixed serialization/Deserialization of AqlItemBlock to contain the hidden subquery register now
* Added a c++ test for AqlItemBlock. It just covers the basics thus far and needs to be improved.
* Fixed toVPack of AqlInputRow for subqueries. Also added serialization / deserialization tests for AqlItemRow
* Added a c++ test for inputAqlItem row serialization => AqlItemBlock deserialization
* Attempt to fix :windows: compile warning
* Added a serialization format to be backwards compatible to 3.5 whenever AQL item blocks need to be send over to remote nodes.
* Added more tests for serialization of shadow rows, and fixed a bug revealed by this
* Apply suggestions from code review
Thank you for review!
Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
* Review comments. Thanks for spotting
* Fixed hidden merge conflicts
* Attempt to explain windows on how one can actually cast numbers into other number types...
* Update arangod/Aql/AqlItemBlockSerializationFormat.h
Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
* Some refactoring to implement helper methods iff necessary
* Updated comments
* Added static assertions
* Re-enabled check for maintainer mode
* Allow pass-through for LimitExecutor. This is not yet working with fullCount correctly!
* Get fullCount before returning the last row
* Adapted fullCount tests: LimitExecutor doesn't lie about the state of the last row anymore
* Use correct row
* Handle LimitExecutor-stats during fetchBlockForPassthrough
* Fixed LimitExecutorTest
* Removed _stats member
* Added more catch tests
* Bugfix for fullStats
* Removed an erroneous assertion
* Implemented LimitExecutor::skipRows
* Avoid name clash of class enum member to please MSVC
* Test MSVC
* Revert "Test MSVC"
This reverts commit d8325d95318bb83b79c8c4cc9f886d36e484c870.
* CREATE_NEW seems to be taken, too. Next try for MSVC
* Built VPack to ItemBlock helpers, and made SingleRowFetcherHelper more general
* Removed ostream operator because MSVC doesn't like it...
* Implemented SingleRowFetcherHelper::fetchBlockForPassthrough, plus a small framework to test executors
* Removed erroneous include
* Moved code in separate files
* Began writing the extended schema of runExecutor
* Added output operator for ExecutorCall
* Fixed removeWaiting
* Fixed expected output
* Added skipRows to runExecutor
* Added two tests
* Allow empty blocks
* Added two more tests
* Added another test
* Built two different parametrized classes for LimitExecutor
* Some cleanup
* Even more tests
* Fixed a bug found by the new tests
* Fix compile error on windows
* Use native matrix representation in tests
* Updated some comments
* update iresearch from upstream
* adjust tfidf/bm25 to original behaviour
* update iresearch from upstream
* fix compilation issues
* reduce number of virtual calls
* upgrade iresearch from upstream
* add js test
* move IResearchViewSort into separate unit
* ensure sort is propagated to IResearchLink
* update iresearch and extend some tests
* implement insertion of sorted values, store value comparator in link
* add tests for VPackComparer
* minor cleanup
* fix issue with sort definiton propagation
* propagate error message from index factory
* add integration tests for sorted indexes, single server
* initial implementation of IResearchViewMergeExecutor
* move iresearch::VPackComparer to separate translation unit
* store collection in IndexReadBuffer for IResearchViewMergeExecutor
* set primary sort from optimizer rule
* add some tests for sorted view case
* extend IResearchViewNode tests
* simplify primary sort definition
* pretty print order condition covered by arangosearch in query explanation
* fix compilation with g++-8
* address catch test failures
* fix tests
* adjust arangosearch optimizer rule to optimize sort in case of cluster
* make tests for sorted indexes work in cluster too
* fix tests
* extend js tests for sorted indices
* fix keyword in query explainer
* ensure sort-limit rule works with views
* backport ngram analyzer fix from upstream
* address jslint errors
* modify IResearchViewSort to support boolean direction specification format
* add some tests
* add tests
* add skip functionality
* extend tests
* add more catch tests
* inline some methods
* address jslint errors
* extend catch tests to reproduce the failure
* add some tests for check fullcount
* add more tests
* extend tests
* temporarily comment out broken tests
* extend js tests
* address review comments
* address jslint errors
* update iresearch
* testing skipSome implementation, default and passThrough
* added simple implementation for default and passthrough skipsome methods
* added skipRows function to enumerate collection executor
* added tests for the enumerate collection skipRows function
* implemented and tested enumeratecollection skipSome
* skipSome tests
* fixed a test, prepared iresearch view exec for skipping
* rm logs
* gcc bug workaround
* changed to original test code as it has been before
* iresearch skipping, added ires skip test
* added index executor skipRows
* added skip blockfetcher, quick exit index
* test
* const dummy and singlerowfetcher skip
* forgot return value
* input wrong initialized
* trying to remove dynamic cast and solve with a different approach
* cleanup
* const skip
* jslint
* Handle skipSome of subqueries correctly
* Removed unused code
* Removed unused member
* Simplified skip variants and enabled IndexExecutor skipping
* A little cleanup, fixed DependencyProxy::skipSome
* Reverted test change
* Tried to make testLimitBlock3 clearer
* Extended test suite
* Bugfix
* Added stats when skipping and fixed a few other things
* Bugfixes
* Moar bugfixes
* Update arangod/Aql/IResearchViewExecutor.cpp
Co-Authored-By: hkernbach <hkernbach@users.noreply.github.com>
* Update arangod/Aql/IndexExecutor.cpp
Co-Authored-By: hkernbach <hkernbach@users.noreply.github.com>
* Update arangod/Aql/IndexExecutor.cpp
Co-Authored-By: hkernbach <hkernbach@users.noreply.github.com>
* applied requested changes
* Fixed a bug in LimitExecutor::expectedNumberOfRows()
* Fix skipSome in catch test RowFetcherHelper
* Fixed a bug in the tests
* Two bugfixes in LimitExecutor::expectedNumberOfRows
* Avoid passing large batch sizes from skipSome to getSome
* Fixed Windows compile errors
* Fixed a skip bug with WAITING in unsorting gather blocks
* Make aql-skipping find some cluster bugs, like the last commit
* Bugfix and additional asserts
* Fixed skipSome counting in IndexExecutor
* Resolved merge conflicts
* Removed useless parameters of buildCallback
* Renamed produceRow to produceRows and adapted a comment
* Renamed BlockFetcher to DependencyProxy
* Applied git-clang-format
* remove unused functions
* more cleanup, virtual functions
* rm old functions
* more cleanup
* more cleanup, rm unused code
* rm of inheritRegisters and cleaerRegisters
* more cleanup, move functions into IMPL and cluster blocks
* used wrong shutdown function
* rm not reachable code
* moved lots of ExecutionBlock stuff to Impl and ClusterBlocks
* trace functions back to executionblock
* rm trace
* rm empty protected
* this getBlock addition, might be useless
* more fixes
* fixes for the distribute executor, hopefully almost done now
* removed obsolete todos
* shutdown var order
* applied requested changes
* re added const
* suppress a warning
* added forgotten test changes
* default destructor, removed not needed function
* refactor name
* Update tests/CMakeLists.txt
Co-Authored-By: hkernbach <hkernbach@users.noreply.github.com>
* Added RemoteExecutor skeleton
* Moved RemoteBlock implementations to ExecutionBlockImpl<RemoteExecutor>
* Remove unnecessary include to avoid unused function warnings
* Fixed gcc compile error
* Moved Scatter/Distribute block implementations to their new Executor versions
* Applied clang-format
* Added factory, infos and a skeleton for the unordered view executor
* Removed assert based on wrong assumption
* Added members from IResearchViewBlockBase to IResearchViewExecutor
* Moved more code into the ViewExecutor, hopefully enough to produce a working version now
* Added missing reset code, made produceRow work mostly correct
* Removed superfluous parentheses to get more useful output from Catch
* Ported fix 923b6e81ac723d1fe37f8e7bf1ab81149f3a08ef
Original commit message was:
Fixed a race condition in RemoteBlock which was triggered during
shutdown overtaking getSome.
* Applied review comments
* Inject input row instead of an item block + pos into the expression context, plus fixed some tests
* Adapted test. Search tests are now green.
* Do not ask upstream when already DONE
* Removed `limit` from next()
* Simplified code that could handle producing more than one document
* Minor readability change
* Solved two TODOs noted in the review
* Removed leftover references to DistributeNode members in the DistributeBlock
* Reverted removal of "exhausted"
* WIP: Implemented variant with scorers
* Fixed compile errors of the last commit
* Fixed some asserts and calculations
* Fixed violated assertions
* Moved files from IResearch/ to Aql/
* Replaced recursive call with a loop
* Worked on a few TODOs
* Removed IResearchViewBlock
* Set input registers correctly
* Eliminated dependency to the Node in the Executor
* Don't misuse the volatility variables for initialization
* Extended a TODO note
* Removed obsolete includes
* Removed an obsolete include from the tests
* Added missing include
* Read PKs in batches
* Fixed merge conflict
* Fixed merge conflict
* Restrict prefetching of PKs to the number of rows in the current output block
* Fixed merge
* Fix IResearch ASan errors
* Revert "Restrict prefetching of PKs to the number of rows in the current output block"
This reverts commit e0fd8698a3.
* Revert "Read PKs in batches"
This reverts commit c06c4d7a36.
* Began some small step refactoring to introduce batch-reading correctly
* Extracted method fillBuffer
* Extracted method evaluateScores
* Minor changes
* Read data from iresearch index in batches
* Replaced std::deque<IndexResult> buffer by a new class
* Solved minor TODOs
* Fixed last commit
* Fixed merge conflict
* Removed accidentally re-added view blocks
* Implemented review comments
Dronplane
Markus Pfeiffer
Tobias Gödderz
Michael Hackstein
Frank Celler
Andrey Abramov
Heiko
Tobias Gödderz