1
0
Fork 0
Commit Graph

375 Commits

Author SHA1 Message Date
Tobias Gödderz 168103af88 Splicing gather cleanup (#10479)
Some cleanup and deprecation information
2019-11-21 13:00:31 +01:00
Iurii Popov 26309c5c96 Feature/issue #642 (#10352)
* Index late materialization draft

* Index late materialization. Refactoring.

* Index late materialization. Expansion draft.

* HashIndexMock

* Refactoring.

* Hash index mock expansion fixes

* Fix compare of attributes.

* Hash index mock. Last expansion support.

* Hash index mock. Take value for array from document, not equal key.

* Optimizations.

* Query tests

* lateMaterialized improvements

* velocypack fixes and tests for index node

* Fix after merge.

* Const expression in a condition test

* Clone tests

* More tests

* Fix for materialized view (nullptr var)

* Fixed index vars registers count planning

* javascript tests

* Fixed cluster tests

* Cluster support

* MaterializeExecutor and MaterializerExecutorInfos refactoring

* More tests

* js tests rocksdb only

* js test. Removed print()

* js testd added skiplist

* C++ unit tests for invalid json

* js test var i

* js tests == -> ===

* iFix js tests ===

* jslint ;

* int unsigned int fix

* double deletion fix

* refactoring

* Added const for collection source

* override final, added materialize namespace, fixed femove in hash index mock

* Removed expansion support. Index can contain null value in case of several object with same fields

* isNumber<>

* Optimization for single collection case

* Renaming

* refactoring

* Unit tests. Added checking documents

* !sliceIt.valid() instead of !(!=)

* Fix primary index (object instead of array)

* Edge collection test

* Windows compilation fix

* IndexExecutor getCallback optimization

* ADB_UNLIKELY

* explainer.js

* removed unsused variable in js test

* fix explainer.js

* do not use several indexes with js test

* Removed several indexes implementation

* space

* fix explainer.js

* fix explainer.js

* code alignment
2019-11-07 19:16:33 +03:00
Simon Grätzer b0a2e207c1
Merge branch 'devel' of github.com:arangodb/arangodb into feature/parallel-aql-phase-one 2019-10-31 14:40:28 +01:00
Tobias Gödderz 6a3f50fb88 Enable AQL subquery optimization (only on getSome) (#10267)
Activate the Subquery Optimization rule in getSome only cases
2019-10-30 17:10:01 +01:00
jsteemann 791c926327 make "parallelize-gather" an optimizer rule, so it can be turned off 2019-10-30 11:54:28 +01:00
Jan 0e97f322e5
activate early pruning (#10308) 2019-10-28 14:22:33 +01:00
Dan Larkin-York 6fd419714f Move containers into their own folder and namespace. (#10244) 2019-10-14 10:55:55 +02:00
Dronplane a1014ac701 Bug fix/internal issue #600 - Late document materialization (#10006)
* Applied review cAdded sceleton optimizer functionomments

* Added mask for rule

* Applied review cAdded sceleton optimizer functionomments

* Added mask for rule

* adding plan traversal for rule

* Rule execution block developing

* Experiment with additional block

* LIMIT node modifications

* implemented late materialization base version.

* Code cleanup

* Added check for variable usage

* Implemented serialization and cloning

* Fixed build errors

* Implemented late materialization for  LimitNode

* Moved materialization to Sort Node

* Removed LimitNode modiications

* Implemented skipRows to speedup single server

* Added tests for optimizer rule

* fixed build

* Fix mac build

* Fixed mac build

* Fixed PR test run errors

* Added tmp fix for cluster test runs

* removed debug print

* Make materialization stick to last sort node. Add late materialization to explainer

* Updated optimization rule to search optimal SortNode to do materialization

* Refined optimization rules for cluster

* Added tests for SortExecutor and IResearchViewNode

* Fixed  mac buid. Removed misplaced assertion.

* Fixed build

* Updated tests. Coded cleanup

* Code cleanup

* made function name for SortedExecutor match name in ConstraintSortedExecutor

* cleanup

* Fixed test run

* made scoring results stable across cluster

* Make optimize rule stop if  sort node for limit node is not suitable

* Disabled constrained sort for materializing SortNode

* reverted SortNode modifications

* Implemented separate materializer node

* Fixed tests  to account new materialize node

* Code cleanup

* reverted debugging change

* Fixed test jslint error  and comments

* MaterializeNode renamed

* enum value renamed for materialize node

* Code cleanup

* code cleanup and optimization

* Fixed clang warning

* Applied review comments

* Out variable planning moved to materialize node

* inlined getters in ReadContex
2019-10-08 18:38:47 +03:00
Jan 6f14048bbe
implement early pruning of documents in EnumerateCollectionExecutor (#10159) 2019-10-05 13:38:23 +02:00
Markus Pfeiffer 32884568cb Feature/subquery end node executor (#10087)
* Add Start/End subquery node skeleton

* Added a test query that triggers complex interna in the future and needs to proof that our work is side-effect free.

* Added first draft of ShadowRow Interface

* Refactored RegisterPlan and pulled it out of the ExecutionNode

* AqlItemBlock now has an additional hidden register to store subquery depth infromation. Right now no shadow rows are created, however thy could now make use of this situation

* Added API and test to insert new shadow rows into an OutputRow.

* Extrated test helper function

* Add template function to statically assert size of type

* Add Start/End subquery node skeleton

* Add Start/End subquery node creation to ExecutionNode

* Cleanup SubqueryEndNode

* Add test file

* Add isEqualTo to ExecutionNode

* Subquery nodes cleanup

* Update ExecutionNode tests

* Added API and tests to consume ShadowRows. Interface is there and compiles with templates, we might need to implement further underlying functions later on as we make progress on ShadowRows, it is only implemented in the minimal way (intentionally)

* Updates to testing code

* Added a test for nested ShadowRows and adapted OutputRow accordingly.

* More updates

* Add optimizer rule

* Fix optimizer rule

* Added additional memory include, seems to be required under GCC not udner CLANG

* it actually helps to save fies before commiting them...

* Fix optimizer rule (again)

* Fixed serialization/Deserialization of AqlItemBlock to contain the hidden subquery register now

* Added a c++ test for AqlItemBlock. It just covers the basics thus far and needs to be improved.

* Fixed toVPack of AqlInputRow for subqueries. Also added serialization / deserialization tests for AqlItemRow

* Added a c++ test for inputAqlItem row serialization => AqlItemBlock deserialization

* Add a public function to get plan from query

* Fix register planning

* Add test for splicing subquery nodes

* add static asserts for sizes of SubqueryStartNode and SubqueryEndNode

* Add comment about statically asserted size

* Fix isEqualTo virtual and override dance.

* Attempt to fix :windows: compile warning

* Fix isEqualTo, remove static_assert_size

* Remove a TODO; the plan owns the ExecutionNodes

* Only add stealPlan to Query when testing

* Remove another static_assert_size

* Fixed mac compile issues

* Added a serialization format to be backwards compatible to 3.5 whenever AQL item blocks need to be send over to remote nodes.

* Handle ShadowAqlItemRow in SingleRowFetcher

* Move some code from header to cpp

* Added more tests for serialization of shadow rows, and fixed a bug revealed by this

* Apply suggestions from code review

Thank you for review!

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review comments. Thanks for spotting

* Fixed hidden merge conflicts

* Removed unused variable

* Added first test on ShadowRows in SingleRowFetcher. Found inconvenience in original code with it.

* Added a test for multi level shadow rows

* Added a test and fixed production for multiple consecutirve shadow rows.

* Added tests for pass through blocks and shadow rows

* Attempt to explain windows on how one can actually cast numbers into other number types...

* Add SubqueryEndExecutor

* Adapted RowFetcherHelper to be able to work on ShadowRows

* Fixed minor error in RowFetcherHelper that causes LIMIT tests to error

* Temporarily disabled test suites (sorry there is only one 'test case' in this suites that does all or nothing tests ...) as it tests a feature we cannot support in this intermediate state, we need to default disable our new rule first.

* IMplemented missing function to copy over SubqueryDepth between blocks

* Seperated assert, to see faster which part is violated

* Update arangod/Aql/AqlItemBlockSerializationFormat.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Disabled subquery splice rule by default

* Forgotten merge marker

* temporarily modified test

* Some Subquery End Executor work

* Fix RowFetcherHelper to work with ShadowRows

* Add tests for SubqueryEndExecutor

* Rewrite SubqueryEndExecutor

* Update test

* Add createBlock code for SubqueryEndExecutionNode

* Some fixups adding ExecutionBlockImpl for SubqueryEndExecutor

* Fix SubqueryEndExecutor

* Update SubqueryEndExecutorTest

* Update endexecutor test

* Fix the endnode executor once more

* Fix some merge fallout

* Update test

* Add missing advanceRow to SubqueryEndExecutor

* SubqueryEndExecutor Tests

* Activate a death test

* Better handling of death test

* RowFetcherTest refactoring Part 1, still ongoing. We use this for AqlItemMatrix and SingleRowFetcher

* Update and cleanup the SubqueryEndExecutor tests

* Check that expected shadow row depths are met

* Remove commented out code in AqlItemBlock.cpp

* Remove a misplaced comment

* Sort out SubqueryEndExecutor includes

* Remove superflous include

* Consistently use NoStats{} in SubqueryEndExecutor

* Make SubqueryEndExecutor properties constexpr

* Reactivate IResearchTests

* Do not copy the accumulator in SubqueryEndNode

* Return a more sensible value for expectedNumberOfRows of SubqueryEndNodeExecutor

* Constify an argument

* Fix ownership problems with the buffer for _accumulator

* Cleanup SubqueryEndExecutor tests

Also add a test that writes to a register other than 0

* Patch RowFetcherHelper to count the number of Items returned instead of number
of calls

* Make MSVC happy
2019-09-30 15:24:13 +02:00
Markus Pfeiffer 0b9dfb4b9b Subquery Start/End Nodes (#9983)
* Add Start/End subquery node skeleton

* Added a test query that triggers complex interna in the future and needs to proof that our work is side-effect free.

* Add template function to statically assert size of type

* Add Start/End subquery node skeleton

* Add Start/End subquery node creation to ExecutionNode

* Cleanup SubqueryEndNode

* Add test file

* Add isEqualTo to ExecutionNode

* Subquery nodes cleanup

* Update ExecutionNode tests

* Updates to testing code

* More updates

* Add optimizer rule

* Fix optimizer rule

* Fix optimizer rule (again)

* Add a public function to get plan from query

* Fix register planning

* Add test for splicing subquery nodes

* add static asserts for sizes of SubqueryStartNode and SubqueryEndNode

* Add comment about statically asserted size

* Fix isEqualTo virtual and override dance.

* Fix isEqualTo, remove static_assert_size

* Remove a TODO; the plan owns the ExecutionNodes

* Only add stealPlan to Query when testing

* Remove another static_assert_size

* Disabled subquery splice rule by default

* temporarily modified test

* Fixed windows compile

* Fixed unused variable warning
2019-09-25 12:35:18 +02:00
Tobias Gödderz 82b0a5e745 De-inline in AQL (#10025)
* Enable IPO (LTO) via CMake

* Use separate IPO_ENABLED variable

* Enabled IPO for more binaries

* Enabled IPO for arangobackup

* Suppress an MSVC warning (though it is not unjustified...)

* Fix static builds, broken due to Policy CMP0060 introduced in cmake 3.3

* Removed policies superseded by minimum cmake version

* Disable IPO with google tests due to a g++ bug

* Disable IPO with google tests only on AUTO

* Disable warning for correct line

* Begin de-inlining Aql

* de-inlining Aql: 2nd batch

* de-inlining Aql: 3rd batch

* de-inlining Aql: 4th batch

* Moved code out of ifdef

* Enabled IPO for additional libs

* Fixed some problems found with IPO enabled

- Fixed ODR violation in Pregel
- Removed unused code in ClusterMethods
- Avoid possible uninitialized variable usage

* Set IPO globally, except for 3rdParty libs

* De-inlined AstHelper again (was undone due to a previous merge conflict)

* Added missing .cpp file

* Fixed compilation with MSVC

* Removed superfluous includes from deinlined AQL header files

* Added includes for size_t

* Deleted outdated files

* De-inlined files that were undone during the merge

* Removed includes from files that were undone during merge

* Fixed merge conflicts
2019-09-25 11:20:03 +02:00
Michael Hackstein 265eb1549d
Feature/aql subquery opt shadow row interface (#9987)
* Added first draft of ShadowRow Interface

* Refactored RegisterPlan and pulled it out of the ExecutionNode

* AqlItemBlock now has an additional hidden register to store subquery depth infromation. Right now no shadow rows are created, however thy could now make use of this situation

* Added API and test to insert new shadow rows into an OutputRow.

* Extrated test helper function

* Added API and tests to consume ShadowRows. Interface is there and compiles with templates, we might need to implement further underlying functions later on as we make progress on ShadowRows, it is only implemented in the minimal way (intentionally)

* Added a test for nested ShadowRows and adapted OutputRow accordingly.

* Added additional memory include, seems to be required under GCC not udner CLANG

* it actually helps to save fies before commiting them...

* Fixed serialization/Deserialization of AqlItemBlock to contain the hidden subquery register now

* Added a c++ test for AqlItemBlock. It just covers the basics thus far and needs to be improved.

* Fixed toVPack of AqlInputRow for subqueries. Also added serialization / deserialization tests for AqlItemRow

* Added a c++ test for inputAqlItem row serialization => AqlItemBlock deserialization

* Attempt to fix :windows: compile warning

* Added a serialization format to be backwards compatible to 3.5 whenever AQL item blocks need to be send over to remote nodes.

* Added more tests for serialization of shadow rows, and fixed a bug revealed by this

* Apply suggestions from code review

Thank you for review!

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review comments. Thanks for spotting

* Fixed hidden merge conflicts

* Attempt to explain windows on how one can actually cast numbers into other number types...

* Update arangod/Aql/AqlItemBlockSerializationFormat.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
2019-09-24 09:09:52 +02:00
Jan Christoph Uhde 0b8c75c7b7 one shard db - devel (#9395) 2019-09-23 15:48:37 +02:00
Tobias Gödderz fbcb4b7152 Avoid overfetch in sorting gather executor (#10047)
* Avoid overfetch in sorting gather executor

* Updated CHANGELOG

* Temporarily disabled sort-limit in cluster when fullCount is enabled, until we can fix SortingGather

* Added a cluster test for sort-limit

* Fixed non-maintainer compile

* Fixed jslint errors

* Disabled sort profiler tests until SortingGather is fixed
2019-09-20 17:00:18 +02:00
Jan Christoph Uhde fb839d1c40 stop optimization for nested collects (#9484) 2019-07-29 09:00:03 +02:00
Markus Pfeiffer a14386267a k Shortest Paths (#8715)
* Add k-shortest-paths finder
2019-04-12 21:58:18 +02:00
Tobias Gödderz d1416bed74 AQL by line: IResearch blocks as executors (#8387)
* Added RemoteExecutor skeleton

* Moved RemoteBlock implementations to ExecutionBlockImpl<RemoteExecutor>

* Remove unnecessary include to avoid unused function warnings

* Fixed gcc compile error

* Moved Scatter/Distribute block implementations to their new Executor versions

* Applied clang-format

* Added factory, infos and a skeleton for the unordered view executor

* Removed assert based on wrong assumption

* Added members from IResearchViewBlockBase to IResearchViewExecutor

* Moved more code into the ViewExecutor, hopefully enough to produce a working version now

* Added missing reset code, made produceRow work mostly correct

* Removed superfluous parentheses to get more useful output from Catch

* Ported fix 923b6e81ac723d1fe37f8e7bf1ab81149f3a08ef

Original commit message was:
Fixed a race condition in RemoteBlock which was triggered during
shutdown overtaking getSome.

* Applied review comments

* Inject input row instead of an item block + pos into the expression context, plus fixed some tests

* Adapted test. Search tests are now green.

* Do not ask upstream when already DONE

* Removed `limit` from next()

* Simplified code that could handle producing more than one document

* Minor readability change

* Solved two TODOs noted in the review

* Removed leftover references to DistributeNode members in the DistributeBlock

* Reverted removal of "exhausted"

* WIP: Implemented variant with scorers

* Fixed compile errors of the last commit

* Fixed some asserts and calculations

* Fixed violated assertions

* Moved files from IResearch/ to Aql/

* Replaced recursive call with a loop

* Worked on a few TODOs

* Removed IResearchViewBlock

* Set input registers correctly

* Eliminated dependency to the Node in the Executor

* Don't misuse the volatility variables for initialization

* Extended a TODO note

* Removed obsolete includes

* Removed an obsolete include from the tests

* Added missing include

* Read PKs in batches

* Fixed merge conflict

* Fixed merge conflict

* Restrict prefetching of PKs to the number of rows in the current output block

* Fixed merge

* Fix IResearch ASan errors

* Revert "Restrict prefetching of PKs to the number of rows in the current output block"

This reverts commit e0fd8698a3.

* Revert "Read PKs in batches"

This reverts commit c06c4d7a36.

* Began some small step refactoring to introduce batch-reading correctly

* Extracted method fillBuffer

* Extracted method evaluateScores

* Minor changes

* Read data from iresearch index in batches

* Replaced std::deque<IndexResult> buffer by a new class

* Solved minor TODOs

* Fixed last commit

* Fixed merge conflict

* Removed accidentally re-added view blocks

* Implemented review comments
2019-04-04 23:40:10 +03:00
KVS85 2d5bf58c94 Remove USE_IRESEARCH definition and usage (#8613) 2019-03-28 17:23:51 +01:00
Michael Hackstein a12a28b094
Bug fix/fixes asan uninitialized by line (#8588)
* Fixed maybe-uninitialized warnings by removing unnecessary boost:optionals

* Fixed use after free

* Update arangod/Aql/SingleRemoteModificationExecutor.cpp

* Fixed another autocast number=>bool thanks c !

* Fixed wrong usage of almost identically named variables
2019-03-27 20:54:36 +01:00
Michael Hackstein 48b709ae16
Feature/subquery by line (#8553)
Subquery Executor by line
2019-03-25 17:02:07 +01:00
Jan Christoph Uhde 5711854531 add SingleRemoteModificationExecutor (#8245) 2019-03-25 07:07:57 +01:00
Jan Christoph Uhde 5d5b77abb8 add ModificationExecutor (#8179)
All modification operator as Executors
2019-03-24 20:48:38 +01:00
Jan 39a3f5bc4e
reintroduce smart joins after temporarily reverting them in devel (#8543) 2019-03-23 20:36:02 +01:00
jsteemann dc381a99df Revert "Feature/ncc1701 (#8440)"
This reverts commit 59ad583796.
2019-03-21 19:18:46 +01:00
Jan 59ad583796
Feature/ncc1701 (#8440) 2019-03-21 15:05:36 +01:00
Heiko ce51797609 Feature/aql hashed collect (#8337)
Implementation of Hashed Collect Executor
2019-03-21 07:19:26 +01:00
Dan Larkin-York 2eadab33e7 Index hints (#8431) 2019-03-19 09:14:18 +01:00
Jan 44c6a2d732
Feature/ttl index (#8169) 2019-02-19 14:12:21 +01:00
Tobias Gödderz 1adb438412 Calculate registersToKeep correctly to avoid redundant copying of empty registers (#8168) 2019-02-15 12:55:18 +01:00
Luschgy b7b84beebf Feature/aql blocks by line (#6553)
* Fixed hidden merge conflict

* Added BlockFetcher, added a minimal test for SingleRowFetcher

* move RegInfo in types.h

* fix getSomeLogic

* Added the AqlItemMatrix implementation. Not yet tested!

* Fixed nr called => Waiting ratio in AllBlocksFetcherMock

* fix compiling

* add RegInfo to produceRow

* BlockFetcherHelper now returns a matrix

* pass RegInfo as requested into AqlItemRow ctor

I would still keep it out as the ItemRow should be a thin view.

* allow to change the AqlItemRows cheaper / remove _sourceRow member

* implement copyRow for AqlItemRow

* organize cleanup for outputblock used in getsome / shrink if necessary

* Added tests for SingleRowFetcher, cleanup in AqlItemRow

* start mocking

* Added test helpers, began work on a more complex test for SingleRowFetcher

* Re-added RegInfo again to AqlItemRow

* helper, added test case for FilterExecutor

* merge fixed compile

* Removed trx name where it is actually not used

* Made some functions TEST_VIRTUAL for mocking.

* implemented Sort executor

* Fixed the BlockFetcherHelper API, to always return a matrix if it is not waiting

* Implemented a SortExecutor test with data

* disable borken test

* Fixed test on SortExecutor. Green now, needs some more testing

* change a bit how the AqlItemRow is used in a test

* Added another test for the SingleRowFetcher

* Added more tests for SingleRowFetcher

* fix getRegisterInfo

* Moved RegInfo into ExecutorInfos. Also now use the FilterExecutor implementation in production code instead of FilterBlock. Starting integration testing.

* add more asserts

* Fixed undefined behaviour in TestCode.

* Do not shrink an empty output block

* switch to block pointers in AqlItemRow

* return blocks in the row fetcher

* Updated grammar.cpp

* Updated (fixed) tests to verify returnBlock

* Improved const-ness

* Changed BlockFetcher constructor in order to make it mockable

* Implemented BlockFetcherMock and rewrote some tests with it

* Updated a comment

* Updated comments

* init commit - EnumerateListExecutor, including EnumerateListExecutorInfos

* add potential fix for ItemMatrix

* Replaced fakeit with BlockFetcherMock in another GIVEN block

* Replaced the last fakeit mocks with BlockFetcherMock and removed fakeit

* Updated function definition

* included missing arary included

* Renamed AqlItemRow to InputAqlItemRow

* Split AqlItemRow into Input~ and Output~ classes

* EnumerateListExecutor + Tests

* Added tests for AqlItemRows

* Added test file for AqlItemRows...

* Made the ostream operator for AqlItemBlocks available to Logger and Catch info

* build error

* Added a second test for AqlItemRows. Need to make input space invalid now, sth. acidential referencing will show up

* Introduce a block ID in the InputAqlItemRow

* Copy output values correctly, avoid InputAqlItemRow pointers

* Added a test for registerClearout

* Fixed tests

* Added a test for adding new values

* Added a test for an incomplete row write to not count as produced.

* Fixed getSome and a test

* Added a test for larger AQL values in AqlItemRows

* Fixed compile error in hidden merge conflict

* Added tests for Filter counter in FilterExecutor

* OutputAqlItemRow now takes responsibility of the block

* Handed responsibility of blocks in ExecutionBlockImpl to OutputAqlItemRow

* EnumerateListExecutor should be close to its finish. Tests need to be fixed, not ready yet.

* better swap

* OutputAqlItemRow now takes ExecutorInfos

- added several assert
- implemented a more strict and safe produced() logic

* enum list test

* Added a more specific error message

* Removed OutputAqlItemRow.changeRow()

* fix invalid block freeing

* changed order of input/output registers

* Fixed a test

* Fixed matrix width in test

* fix failure tests

* finished enumlist tests

* Implemented AllRowsFetcher and tests for it

* Bugfix in skipSome stub

* insert sort and enumerate list executors

into exectuion nodes' create functions

* fixed singlerowfetcherhelper, added tests

* small fixes

* add missing Infos

* restore accidentally deleted scorer

* make it compile

* fix executor info creation / usage

* make use of output var in EnumarateListNode

* Added Executor::Stats and FilterStats

* Extended FilterExecutor tests with stats, added node tracing to getSome

* Fixed Sort-&EnumerateList- Executor to use the new return type with stats

* Fixed compile error in tests

* Changed *ExecutorInfo constructors and classes to contain the necessary register informations

* Made an ExecutorInfo construction more readable

* Updated comments

* Fixed checks in OutputAqlItemRow::setValue()

* Add comparison method to SortRegister. Forbid construction without one.

* Implemented ExecutionBlockImpl::initializeCursor

* more mocking inside ExecutionBlockImplTest

* Minor changes

* Avoid AqlValueGuard destroying the value copied into the output block

* Disabled failure tests that do not apply to the new Sort implementation

* further mocking of test modules

* Removed constexpr for compatibility with outdated gcc versions

* add some failure tests - getsome works differently

therefor we do not test the old behavior

* fix sort node

* clean up alq item matrix

* Removed the transaction methods from read-only access to AqlValues (get and at) and repliced by CollectionNameResolver where necessary. Still left: some calls extract VPackOptions from the Transaction. Compare needs a BuilderLeaser only.

* textexecutionhelper, execimpl test

* Eliminated FilterBlock

* Eliminated the EnumerateListBlock

* first successful running test of executorblockimpl

* Eliminated SortBlock

* fixed testexecutor helper

* Made the ExecutorInfos private to the executors again.

* Implement and use AqlItemBlockShell (#6562)

* Added AqlItemBlockShell

* Minor changes

* Mostly replaced AqlItemBlock with ~Shell.

Current state:
- arangod compiles
- OutputAqlItemRow has yet to be changed
- catch tests have yet to be fixed

* Use AqlItemBlockShell in OutputAqlItemRow

* Added registersToKeep to AqlItemBlockShell

* Removed ExecutorInfos from OutputAqlItemRow

* Split AqlItemBlockShell into Input~ and Output~

* Changed tests so they compile again - they're still not yet working.

* Fixed tests

* fixed building, modified impl test

* Removed now superfluous BlockFetcherMock functions

* more execimpl testing

* Removed block IDs

* more impl testing

* Reduced dependencies of BlockFetcher construction

* Bugfix, cleared documentation

* Removed debug-output

* Adressed minor code review remarks

* Moved method definitions into .cpp files to avoid includes

* Added assertions

* Added comments

* Addressed several more review comments

* Added missing includes

* more testing, not finalized yet

* added missing header file

* added a test for execution block impl

* removed old comment

* Improved lookup algorithm to identify correct block/row combination in AqlItemMatrix

* added test for ExecutionBlockImpl

* move utility to the right position

* Bugfix and a TODO note

* Bugfix: use a uniform ResourceMonitor in tests

* Made OutputAqlItemRow::setValue clone by default and copy on move

* Replaced the getRow lookup with a binary search

*  enumerate tests

* Disable emplacing of AqlValues in ItemBlock

* Reenabled std::enable_if will warn until oskar is using c++17 as standard.

* Add helper function to create a shared pointer of an unordered_map<RegisterId>

* CalculationExecutor (#6570)

* add CalculationExecutor skeleton

* first shot of CalculationExecutor

* fix minor bugs

* handle references

* use new shared pointers

* commit to share work

* fix failure test

* reformat with clang-format

* add test for CalculationExecutor

* fix CalcualtionExecutor test

* clean up

* Removes unused code

* clean up includes

* fix/remove comments

* add changes requrested by hacki

* ReturnExecutor (#7932)

* add skeleton to CMakeLists.txt

* build most parts of the ReturnExecutor

* This commit removes ReturnBlock and adds some debug messages

removed inheritRegisters functionality

* finish ReturnExecutor

* add test for return executor

* remove unused code

* add stats and clean up includes

* verify result in ReturnExecutorTest and move *Stats to "Aql/Stats.h"

* Fixed algorithm to find correct row in a MultiRow Input

* fix full count and failure tests (#7969)

* fix full count and failure tests

* fix filter test

* WIP: fixes

* fixed

* NoResultsExecutor (#7953)

* add NoResultsExecutor

* clean up

* some review changes

* fix test

* fix

* fix WAIT -> DONE

* remove unused code (#7995)

* create skeleton for SingletonExecutor (#7957)

* create skeleton for SingletonExecutor

* WIP: continue implementation

* WIP: add fetcher skel

* WIP: add inject function to ConstFetcher

* rename SingletonExecutor to IdExecutor

* WIP: replace singleton node

* WIP: More Debugging on IdExecutor

* WIP: IdExecutor

* WIP: add failure tests

* make iresarch test work but not pass

* WIP: add ConstFetcherHelper

* fix test

* fix tests

* fix iresearch tests

* fix wording

* add review changes

* add review change to SingleRowFetcher as well

* fix test

* fix error when increasing index

* temporarily remove fold-expression

* clean up

* Fix gcc compile error

* Feature/aql limit executor (#7982)

* limit executor

* removed not needed tri assert

* rm _invariable in limitnode

* find out if we are in a subquery

* queryDepth in limit infos

* added missing queryDepth

* finalizing limit executor

* added more tests, fixed a condition

* add assert to test

* added missing fullcount check

* typo

* minor changes, fixed test

* test

* wrong check

* fixed tests

* false merge

* remove basic blocks

* remove basic block includes

* include sort order

* removed not needed basic blocks

* removed not needed basic blocks

* fix test

* Feature/remove input output shells (#8090)

* Added template parameters to allow for block pass-through to be chosen at compile time

* Adapted the tests to match the new template parameters

* Adapted ConstFetcher and IdExecutor to match the new templated fetcher

* Implemented pass-through, fixed SingleRowFetcherTest

* Fixed implicit merge conflicts

* Fix assumptions that are violated by block pass-through

* Added 2 regression tests

* Fixed test expected data

* Fix nullptr dereference

* Fix handling of non-invalidated input variables

* Fix nullptr dereference

* Fix handling of non-invalidated input variables

* Fixed compile error

* Fixed a useless and leaking allocation

* Try a less implicit fix

* Allow IdExecutor and ConstFetcher to be used with pass-through

* Performance patches from @mchacki

* Fixed merge conflicts, and bugs in ConstFetcher

* Fixed atMost pass-through, and blocks that call their parents too often

* Removed InputAqlItemBlockShell (except from tests)

* Removed OutputAqlItemBlockShell (except for tests)

* Moved implementations of OutputAqlItemRow methods in the header

* Moved SingleRowFetcher::fetchRow to the header to allow inlining

* Reenabled returnInheritedResults optimization

* Enable passThrough for ReturnExecutor with inherited results

* Specialized CalculationExecutor to avoid unneccessary conditional jumps

* Made tests compile again (but are currently broken)

* Fixed ReturnExecutor test

* Move created InputRows out of AqlItemMatrix

* Limit- and EnumerateList Executors no longer fetch from upstream after DONE

* Added comments

* Added experimental AqlItemBlock optimizations

* Removed _numEntries, added comments

* Allow executors to pass an "atMost"-limit to the SingleRowFetcher (for LIMIT)

* Let the LimitExecutor pass atMost upstream

* Bugfix for FullCount

* Feature/aql enumeratecollection executor (#8028)

* initial commit

* enum coll executor

* buildable version of enumeratecollectionexecutor

* wrong usage of outVariable

* Added template parameters to allow for block pass-through to be chosen at compile time

* Adapted the tests to match the new template parameters

* Adapted ConstFetcher and IdExecutor to match the new templated fetcher

* more fixes, still some tests failing

* Implemented pass-through, fixed SingleRowFetcherTest

* Fixed implicit merge conflicts

* adding missing set value (remember optimize todo), optimized constructor

* removed old enumerate collection blocks

* Fix assumptions that are violated by block pass-through

* Added 2 regression tests

* Fixed test expected data

* Fix nullptr dereference

* Fix handling of non-invalidated input variables

* Fix nullptr dereference

* Fix handling of non-invalidated input variables

* Fixed compile error

* Fixed a useless and leaking allocation

* added broken uncompilable executor test

* Try a less implicit fix

* fix mocking

* enumerate collection executor test

* remove class, use only static function.

* clean up test after merging with main branch

* Allow IdExecutor and ConstFetcher to be used with pass-through

* Performance patches from @mchacki

* Fixed merge conflicts, and bugs in ConstFetcher

* Fixed atMost pass-through, and blocks that call their parents too often

* Removed InputAqlItemBlockShell (except from tests)

* Removed OutputAqlItemBlockShell (except for tests)

* Moved implementations of OutputAqlItemRow methods in the header

* Moved SingleRowFetcher::fetchRow to the header to allow inlining

* Reenabled returnInheritedResults optimization

* Enable passThrough for ReturnExecutor with inherited results

* Specialized CalculationExecutor to avoid unneccessary conditional jumps

* Made tests compile again (but are currently broken)

* Fixed ReturnExecutor test

* Limit- and EnumerateList Executors no longer fetch from upstream after DONE

* Removed an assertion which seems to be incorrect during bootstrap.

* Inlined AQL ItemMatrix and removed unused elements in other Executors

* Now inline + Move in AqlItemMatrix, seems to be overall best performing

* Bugfix: Do not returnInheritedResults on DBServers

* Fixed LIMIT test, and made it more thorough.

* Fixed createion of NoResultBlock creation in IResearch Nodes. Also relaxed an assertion

* Reactivated failure point, now identical with anotherone in OutputRow, need to unified

* Bug fix: do not mutate the offset in the LimitExecutor (#8109)

* Introduced test

* Fixed LimitExecutor to not mutate its infos, so it works correctly in subqueries

* Fixed crashes in CatchTests

* Fixed non-maintainer build

* try to fix subqueries

* Fix isInSubQuery for non-singleton nodes.

* remove useless while true

* Feature/aql blocks by line dan sort (#8143)

* Add SortNode::SorterType to SortExecutorInfoss.

* prepare imple of sort executor

* WIP: add some code for dan sort

* WIP: working at pushRows

* WIP - try to add Dan's heap-sort

* re add missing test

* 2 fixes

* fix

* Dan's tests are working

* remove unused variable

* remove SortBlock.cpp from CMakeLists.txt

* remove hack and provide an extra method to set the _baseIndex

* Simplified ContrainedSortExecutor.

* fix forgotten variable

* update doc

* Assert that we do not use functions that might return wrong results.

Preconditions or invariants the functions rely on are broken. Therefore
the functions can not be used. This is asserted in maintainer mode.

* add function to restore row invariant

* do not emplace and pop_back if not necessary

* Use clear registers in case we can move the rows.

* WIndows opst zirkus

* Actually destroy AqlItemValue

* Feature/traversal executor by line (#8000)

* Added stubs for TraversalExecutor, no implementation, not compiled in yet

* Included production code of TraversalExecutor and made it compile. Now implement

* Implemented production code for trivial tests, if no row is reaching the traverser.

* Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next

* Intermediate blind commit might not work

* Fixed compile issue in test code

* Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work.

* Fixed travesalExecutor tests. Now able to create tests with actual input.

* Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister)

* Further test cases

* Added stubs for TraversalExecutor, no implementation, not compiled in yet

* Included production code of TraversalExecutor and made it compile. Now implement

* Implemented production code for trivial tests, if no row is reaching the traverser.

* Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next

* Intermediate blind commit might not work

* Fixed compile issue in test code

* Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work.

* Fixed travesalExecutor tests. Now able to create tests with actual input.

* Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister)

* Fixed leftovers from rebase

* Refixed mocks in different folder

* Fixed Mock import

* Fixed Calculators Exector Test to use the unified mocked server.

* Implemented logic for the TraversalExecutor. Also added tests for it. Not fully complete though

* Added validation of result values

* Added usage of constant register in traversal executor

* Included filter variables in traersal executor. Added test for const input. And moved MMDR into base traverser class for simplicity

* Added ostream operators on AQLValue and AqlInputRow. Not 100% done yet but good enough

* Added a shutdownable ExecutionBlock. Some blocks need to implement a specific shutdown still. I added this class to make it easier to eliminate them when the time is due.

* Removed ostream operator for INPUT row again, does for some reason not compile on Linux0o

* Fixed initialization order

* Removed classwrapper for shutdown.

* Applied review changes

* Refixed failing tests, API is not subject to change

* Fixed CatchTest (wrong usage of string literal in test case)

* Feature/shortest path executor (#8042)

* Added stubs for TraversalExecutor, no implementation, not compiled in yet

* Included production code of TraversalExecutor and made it compile. Now implement

* Implemented production code for trivial tests, if no row is reaching the traverser.

* Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next

* Intermediate blind commit might not work

* Fixed compile issue in test code

* Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work.

* Fixed travesalExecutor tests. Now able to create tests with actual input.

* Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister)

* Further test cases

* Added stubs for TraversalExecutor, no implementation, not compiled in yet

* Included production code of TraversalExecutor and made it compile. Now implement

* Implemented production code for trivial tests, if no row is reaching the traverser.

* Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next

* Intermediate blind commit might not work

* Fixed compile issue in test code

* Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work.

* Fixed travesalExecutor tests. Now able to create tests with actual input.

* Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister)

* Fixed leftovers from rebase

* Refixed mocks in different folder

* Fixed Mock import

* Fixed Calculators Exector Test to use the unified mocked server.

* Implemented logic for the TraversalExecutor. Also added tests for it. Not fully complete though

* Added validation of result values

* Added usage of constant register in traversal executor

* Included filter variables in traersal executor. Added test for const input. And moved MMDR into base traverser class for simplicity

* Added ostream operators on AQLValue and AqlInputRow. Not 100% done yet but good enough

* Added a shutdownable ExecutionBlock. Some blocks need to implement a specific shutdown still. I added this class to make it easier to eliminate them when the time is due.

* Removed ostream operator for INPUT row again, does for some reason not compile on Linux0o

* Fixed initialization order

* First draft of shortest path executor

* Fixed logic bugs, Local AQL tests succeed

* Added first version of ShortestPathExecutor tests. Edge output tests are missing yet.

* Added EDGE output tests

* Replaced TraversalStats with NoStats, they have never been tracked...

* Removed obsolete ShortestPathBlock.

* Removed classwrapper for shutdown.

* Applied review changes

* Refixed failing tests, API is not subject to change

* windows macro opst circus

* Clear registers per block, not per row (#8164)

* Fixed compilation under linux again...
2019-02-14 19:13:08 +01:00
Jan 9ce9312239
speed up containers that contain ExecutionNodes' variables (#7937) 2019-01-14 13:38:56 +01:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Jan f5cf42e4f2
Bug fix/refactor query optimizer part2 (#7337) 2018-11-15 18:19:07 +01:00
Jan 6d54518145
forward port changes from 3.4 (#6849) 2018-10-12 17:49:17 +02:00
Jan 46efcff7d7
micro improvements (#6674) 2018-10-05 10:25:13 +02:00
Jan 5022ccc24d
Bug fix/fixes 2508 (#6254) 2018-08-27 21:36:39 +02:00
Jan 0d48677619
Bug fix/aql optimizer improvements 2308 (#6241) 2018-08-24 13:42:22 +02:00
Jan 99f176bed5 Feature/add search keyword (#6166)
* added SEARCH keyword

* fixes jslint errors

* add incompatibility notice

* remove VIEW keyword (does not work yet!!)

* add initial support for SEARCH keyword to optimizer rules

* replace FILTER with SEARCH keyword in tests

* removed VIEW keyword, make shell_server_aql tests pass again in single server

* cleanup

* handle SEARCH clause for views

* make SEARCH a non-keyword

* fixed an issue with duplicate variables

* fixed 3 tests

* fix SEARCH statement support for views

* minor refactoring

* fix integration tests

* optimize SEARCH conditions a bit more

* fix jslint error

* fixed wrong comments and typo in class name

* fix documentation

* adjust recovery tests to use SEARCH instead of FILTER

* fix another failing test
2018-08-18 20:22:58 +03:00
Jan 95d1892f18
remove canThrow attribute from AQL functions (#6123)
Its was somehow overlapping with the isDeterministic attribute (but with the opposite meaning)
2018-08-09 16:45:54 +02:00
Jan bf3ce0b704
fixed issue #6106: Arithmetic operator type casting documentation inc… (#6113) 2018-08-09 11:30:54 +02:00
Jan Christoph Uhde bcf9e30a4d Feature/cluster single operations (#5585) 2018-07-12 20:18:36 +02:00
Jan Christoph Uhde f81e42af36 Add optimizer rule that replaces js function calls (NEAR/WITHIN/FULLTEXT) with pure AQL (#5529) 2018-06-12 13:48:31 +02:00
Jan af9d2c05bc port COLLECT bugfix from 3.3 (#5519) 2018-06-08 15:45:41 +02:00
Jan edb24deafe
cleanup collection/vocbase interface of ExecutionNodes (#5557) 2018-06-08 15:28:48 +02:00
Andrey Abramov e117a97fb4
Feature/arangosearch scatter node (#5536)
* remove collection dependency from scatter

* simplify `createBlock` API

* remove useless IResearchViewScatter node

* some fixes

* code cleanup

* scatter arangosearch view

* cleanup

* fix compilation errors

* fix test issues

* store shared_ptr rather than raw pointer in IResearchViewNode

* refactor ScatterBlock

* refactor DistributeBlock

* ensure no leaks during execution node cloning

* some fixes

* code cleanup

* remove vocbase from `DistributeNode`

* add missing ifdef

* fixes after review
2018-06-06 20:42:53 +03:00
Jan dc97eb0111
Subquery optimizations (#5398) 2018-05-28 13:56:03 +02:00
Jan f55df6786a
blind commit of work, as I managed to break my local git repository (#5456) 2018-05-25 13:07:28 +02:00
Jan a295eaa120
show AQL functions used in query in explain output (#5451) 2018-05-24 21:17:51 +02:00
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00