* added highly experimental startup option `--query.parallelize-gather-writes`
Turning this option on will enable the `parallelize-gather` AQL
optimizer rule for certain write queries. This option is `false`
by default.
The feature is highly experimental and should not be used on a
production system. The option is therefore also hidden.
* parallelize certain write operations in AQL
* remove unneeded _vocbase
* remove unneeded _vocbase
* added startup option and tests
* Index late materialization draft
* Index late materialization. Refactoring.
* Index late materialization. Expansion draft.
* HashIndexMock
* Refactoring.
* Hash index mock expansion fixes
* Fix compare of attributes.
* Hash index mock. Last expansion support.
* Hash index mock. Take value for array from document, not equal key.
* Optimizations.
* Query tests
* lateMaterialized improvements
* velocypack fixes and tests for index node
* Fix after merge.
* Const expression in a condition test
* Clone tests
* More tests
* Fix for materialized view (nullptr var)
* Fixed index vars registers count planning
* javascript tests
* Fixed cluster tests
* Cluster support
* MaterializeExecutor and MaterializerExecutorInfos refactoring
* More tests
* js tests rocksdb only
* js test. Removed print()
* js testd added skiplist
* C++ unit tests for invalid json
* js test var i
* js tests == -> ===
* iFix js tests ===
* jslint ;
* int unsigned int fix
* double deletion fix
* refactoring
* Added const for collection source
* override final, added materialize namespace, fixed femove in hash index mock
* Removed expansion support. Index can contain null value in case of several object with same fields
* isNumber<>
* Optimization for single collection case
* Renaming
* refactoring
* Unit tests. Added checking documents
* !sliceIt.valid() instead of !(!=)
* Fix primary index (object instead of array)
* Edge collection test
* Windows compilation fix
* IndexExecutor getCallback optimization
* ADB_UNLIKELY
* explainer.js
* removed unsused variable in js test
* fix explainer.js
* do not use several indexes with js test
* Removed several indexes implementation
* space
* fix explainer.js
* fix explainer.js
* code alignment
* Applied review cAdded sceleton optimizer functionomments
* Added mask for rule
* Applied review cAdded sceleton optimizer functionomments
* Added mask for rule
* adding plan traversal for rule
* Rule execution block developing
* Experiment with additional block
* LIMIT node modifications
* implemented late materialization base version.
* Code cleanup
* Added check for variable usage
* Implemented serialization and cloning
* Fixed build errors
* Implemented late materialization for LimitNode
* Moved materialization to Sort Node
* Removed LimitNode modiications
* Implemented skipRows to speedup single server
* Added tests for optimizer rule
* fixed build
* Fix mac build
* Fixed mac build
* Fixed PR test run errors
* Added tmp fix for cluster test runs
* removed debug print
* Make materialization stick to last sort node. Add late materialization to explainer
* Updated optimization rule to search optimal SortNode to do materialization
* Refined optimization rules for cluster
* Added tests for SortExecutor and IResearchViewNode
* Fixed mac buid. Removed misplaced assertion.
* Fixed build
* Updated tests. Coded cleanup
* Code cleanup
* made function name for SortedExecutor match name in ConstraintSortedExecutor
* cleanup
* Fixed test run
* made scoring results stable across cluster
* Make optimize rule stop if sort node for limit node is not suitable
* Disabled constrained sort for materializing SortNode
* reverted SortNode modifications
* Implemented separate materializer node
* Fixed tests to account new materialize node
* Code cleanup
* reverted debugging change
* Fixed test jslint error and comments
* MaterializeNode renamed
* enum value renamed for materialize node
* Code cleanup
* code cleanup and optimization
* Fixed clang warning
* Applied review comments
* Out variable planning moved to materialize node
* inlined getters in ReadContex
* Add Start/End subquery node skeleton
* Added a test query that triggers complex interna in the future and needs to proof that our work is side-effect free.
* Add template function to statically assert size of type
* Add Start/End subquery node skeleton
* Add Start/End subquery node creation to ExecutionNode
* Cleanup SubqueryEndNode
* Add test file
* Add isEqualTo to ExecutionNode
* Subquery nodes cleanup
* Update ExecutionNode tests
* Updates to testing code
* More updates
* Add optimizer rule
* Fix optimizer rule
* Fix optimizer rule (again)
* Add a public function to get plan from query
* Fix register planning
* Add test for splicing subquery nodes
* add static asserts for sizes of SubqueryStartNode and SubqueryEndNode
* Add comment about statically asserted size
* Fix isEqualTo virtual and override dance.
* Fix isEqualTo, remove static_assert_size
* Remove a TODO; the plan owns the ExecutionNodes
* Only add stealPlan to Query when testing
* Remove another static_assert_size
* Disabled subquery splice rule by default
* temporarily modified test
* Fixed windows compile
* Fixed unused variable warning
* Added RemoteExecutor skeleton
* Moved RemoteBlock implementations to ExecutionBlockImpl<RemoteExecutor>
* Remove unnecessary include to avoid unused function warnings
* Fixed gcc compile error
* Moved Scatter/Distribute block implementations to their new Executor versions
* Applied clang-format
* Added factory, infos and a skeleton for the unordered view executor
* Removed assert based on wrong assumption
* Added members from IResearchViewBlockBase to IResearchViewExecutor
* Moved more code into the ViewExecutor, hopefully enough to produce a working version now
* Added missing reset code, made produceRow work mostly correct
* Removed superfluous parentheses to get more useful output from Catch
* Ported fix 923b6e81ac723d1fe37f8e7bf1ab81149f3a08ef
Original commit message was:
Fixed a race condition in RemoteBlock which was triggered during
shutdown overtaking getSome.
* Applied review comments
* Inject input row instead of an item block + pos into the expression context, plus fixed some tests
* Adapted test. Search tests are now green.
* Do not ask upstream when already DONE
* Removed `limit` from next()
* Simplified code that could handle producing more than one document
* Minor readability change
* Solved two TODOs noted in the review
* Removed leftover references to DistributeNode members in the DistributeBlock
* Reverted removal of "exhausted"
* WIP: Implemented variant with scorers
* Fixed compile errors of the last commit
* Fixed some asserts and calculations
* Fixed violated assertions
* Moved files from IResearch/ to Aql/
* Replaced recursive call with a loop
* Worked on a few TODOs
* Removed IResearchViewBlock
* Set input registers correctly
* Eliminated dependency to the Node in the Executor
* Don't misuse the volatility variables for initialization
* Extended a TODO note
* Removed obsolete includes
* Removed an obsolete include from the tests
* Added missing include
* Read PKs in batches
* Fixed merge conflict
* Fixed merge conflict
* Restrict prefetching of PKs to the number of rows in the current output block
* Fixed merge
* Fix IResearch ASan errors
* Revert "Restrict prefetching of PKs to the number of rows in the current output block"
This reverts commit e0fd8698a3.
* Revert "Read PKs in batches"
This reverts commit c06c4d7a36.
* Began some small step refactoring to introduce batch-reading correctly
* Extracted method fillBuffer
* Extracted method evaluateScores
* Minor changes
* Read data from iresearch index in batches
* Replaced std::deque<IndexResult> buffer by a new class
* Solved minor TODOs
* Fixed last commit
* Fixed merge conflict
* Removed accidentally re-added view blocks
* Implemented review comments
* added SEARCH keyword
* fixes jslint errors
* add incompatibility notice
* remove VIEW keyword (does not work yet!!)
* add initial support for SEARCH keyword to optimizer rules
* replace FILTER with SEARCH keyword in tests
* removed VIEW keyword, make shell_server_aql tests pass again in single server
* cleanup
* handle SEARCH clause for views
* make SEARCH a non-keyword
* fixed an issue with duplicate variables
* fixed 3 tests
* fix SEARCH statement support for views
* minor refactoring
* fix integration tests
* optimize SEARCH conditions a bit more
* fix jslint error
* fixed wrong comments and typo in class name
* fix documentation
* adjust recovery tests to use SEARCH instead of FILTER
* fix another failing test
* Added feature phases
* BasicsPhase and DatabasePhase to the required files. Server now has Feature circles and does not boot. Will be sorted out later on.
* Added ClusterPhase to features
* Added V8Phase to the required features
* Added AQLPhase to the affected features
* Added ServerPhase to Features
* Added FoxxPhase to the relevant features
* Added AgencyPhase to the relevant features
* Moved registration from local variable SYS_SYSTEM_REPLICATION_FACTOR from cluster to V8 as their ordering is now vice versa
* Moved Bootstrap feature into FoxxPhase. It could be moved to ServerPhase easily if the FoxxQueue dependency would be removed
* Final movement of Startup Phases. Now solved all circles.
* Removed merge conflict
* Moved ReplicationTimeout into cluster phase and fixed cross-phase requirements
* Added greetings phase. This phase separates the Basics Phase and is the first to be run. Includes Logger and Hello/Goodbye
* Added the GreetingsPhase in the corresponding features. Now all BasicsPhase features start after greetings Phase. There is some issue in this branch which prevents the Agency from Gossipping right now. Will be fixed next
* Moved creation of the Agent into the prepare phase of the feature. THereby it is guaranteed that agents at least exists before the GeneralServer is activating endpoints
* Recovery needs to be started after the ServerID
* Moved log output of FeaturePhases to DEBUG instead of ERROR.
* Added feature phases for clients
* ClusterFeature now does not directly require AgencyFeature any more
* Added requirement of TravEngineRegistryFeature in AQL feature. Otherwise shutdown may be undefined
* The ApplicationServer can now handout the list of ordered features. Used for testing purposes
* Fixed IResearchVew Tests Setup to honor new feature ordering
* Fixed IResearchViewDBServer Tests Setup to honor new feature ordering
* Started fixing IResearchView Coordinator tests with startup ordering. Not finished yet
* Added startup phases to ViewCoordinator test
* Disabled expected logoutput in ClusterRepairsTest
* Fixed indention in test code
* LinkCoordinator now honors startup ordering
* Link meta now honors startup rdering
* Supress expected cluster logs in ViewTest
* Removed '#' accidentially added.
* add initial implementation of scatter view rule and node
* add tests for `IResearchViewNode` and `IResearchViewScatterNode`
* add missing check
* modify IResearch execution nodes to use references instead of pointers
* use view id in searialized `ExecutionNode` representation instead of the name
* add cluster mode stubs and checks
* very first attempt to distribute IResearchViewNode
* further implementation of cluster-wide arangosearch views
* fix invalid json format
* add tests for coordinator iresearch view
* allow to retrieve a list of existing views on a coordinator
* more tests for coordinator iresearch view
* some fixes to enable query explanation
* remove Collection dependency from RemoteNode
* remove unnecessary remote ArangoSearch view scatter
* fix explanation appearance
* add some assertions
* minor fixes
* implement IResearchViewCoordinator::updateProperties
* fix view DDL issues
* handle link modifications in DDL operations
* add coordinator implementation of iresearch view links
* fix tests
* further coordinator based view DDL implementation
* further IResearchViewCoordinator implementation
* add initial implementation of AgencyMock
* fix some tests
* code cleanup
* extend test + some fixes
* more tests for IResearchViewCoordinator
* fix tests for IResearchLinkCoordinator
* some fixes after merge
* fix tests
* remove declaration of nonexistent (previously removed) method
* some fixes after review
* remove string duplication
* more tests and fixes
* more fixes and tests
* more tests
* one more test
* fix 'use-after-free' asan error
* fix non-enterprise tests issues
* forward-port AQL optimizations from 3.3
* fix restrict-to-single-shard rule
* added tests for specific cases
* ModificationNodes now can know if they are restricted or not
* OptimizerRule now updates the ModificationNode instead of setting ownName on remote node.
* The DBServerEngine container now only sends a subset of snippets (those which are relevant) It is able to restrict any part of the query to a certain shard or even a list of shards
* ExecutionEngine CreateBlocks in ClusterMode now returns Result instead of throwing.
* Adapted Coordinator side to a bitter mapping of RemoteNode => QuerySnippets. This is still ongoing work.
* Fixed Coordinator enginecontainer test and finisehd implementation of EngineInfoContainerDBServer. Seems to work so far, need to adapt test to prove it.
* Adapted test for modification nodes and restrict to shard.
* restrictTo can now be de-/serialized on Modification Nodes
* Fixed Community Compile Error
* Added a test for Indexes/EnumerateCollection nodes
* EnumerateCollection and IndexNode now have options to be restricted to a single shard, opt rule not yet updated
* DBServer part of planning now checks Enumerate and IndexNodes for single shard optimization
* Added tests for simple single shard optimization on IndexNodes
* Adapted optimzierrule for SingleShard optimierung on IndexNodes
* Fixed an issue with externally restricted shard ids in combination with shard_id based optimization
* Added a test for multiple single shard optimizations on the same collection
* Removed collection/Shard duplicate check of optimize to single shard rules
* Updated Explainer to handle optimize to single shard
* Let more tests pass an optimized AQL variant
* Temporarily disabled a tests that can only be solved after some work in transaction
* do not use V8 variant of AQL functions in early optimization stage when a C++ variant is available
* additionally, simplify AQL function definitions and aliases
* warn when more than 90% of max mappings are in use
* added C++ variant of replication catchup
* added `--log.role` option
* updated CHANGELOG
* removed non-existing scheduler.threads option from config
* removed useless __FILE__, __LINE__ invocations
* updated CHANGELOG
* allow a priority V8 context
* remove TRI_CORE_MEM_ZONE
* try to fix Windows errors & warnings
* cleanup
* removed memory zones altogether
* exclude system collections from collection tests
* reduce extractions to projections
* recycle string buffers in SocketTask
* micro optimizations for mmfiles indexes
* added special lookup function for _key
* moved function into the correct file
* speed up key buffer allocations a bit
* added noexcept specifier
* correctly name variable
* explicitly move bounds
* fix and speedup from/toPersistent functions
* reuse string from ManagedDocumentResult for multiple lookups
* use move-assign
* a bit less work for single server
* speedup AQL function HASH
* single fetch optimization
* performance optimization for the case when no documents need to be returned
* make reduce-extraction-to-projection a RocksDB-only optimizer rule
* cppcheck
* try to fix compile error on MacOS
* bug fix for MacOSX
* missing namespace (in Windows compile)
* reduce extractions to projections
* recycle string buffers in SocketTask
* micro optimizations for mmfiles indexes
* added special lookup function for _key
* moved function into the correct file
* speed up key buffer allocations a bit
* added noexcept specifier
* correctly name variable
* explicitly move bounds
* fix and speedup from/toPersistent functions
* reuse string from ManagedDocumentResult for multiple lookups
* use move-assign
* a bit less work for single server
* speedup AQL function HASH
* single fetch optimization
* performance optimization for the case when no documents need to be returned
* make reduce-extraction-to-projection a RocksDB-only optimizer rule
* Added a first test for remove-filter-covered-by-traverser rule.
* initial try at adding the functions
* Initial try at fixing all cases
* Fixing optimizer rule order
* Changing rule
* excluding quantifier ANY
* Fixing assertion error
* quick fix for some failed traverser cases
* Fixing quantifier related tests
* Adding upper limit for scannedIndex in cluster
* Fixing the mmfiles asserts
* Allow the coordinator to do condition checking on edges again. Used in enterprise.
* Fix naming of optimizer rules (#2739)
Jan
Jan Christoph Uhde
Iurii Popov
Simon Grätzer
Tobias Gödderz
jsteemann
Dronplane
Markus Pfeiffer
Dan Larkin-York
Tobias Gödderz
KVS85
Frank Celler
Vasiliy
Michael Hackstein
Andrey Abramov