1
0
Fork 0
Commit Graph

43 Commits

Author SHA1 Message Date
Iurii Popov 26309c5c96 Feature/issue #642 (#10352)
* Index late materialization draft

* Index late materialization. Refactoring.

* Index late materialization. Expansion draft.

* HashIndexMock

* Refactoring.

* Hash index mock expansion fixes

* Fix compare of attributes.

* Hash index mock. Last expansion support.

* Hash index mock. Take value for array from document, not equal key.

* Optimizations.

* Query tests

* lateMaterialized improvements

* velocypack fixes and tests for index node

* Fix after merge.

* Const expression in a condition test

* Clone tests

* More tests

* Fix for materialized view (nullptr var)

* Fixed index vars registers count planning

* javascript tests

* Fixed cluster tests

* Cluster support

* MaterializeExecutor and MaterializerExecutorInfos refactoring

* More tests

* js tests rocksdb only

* js test. Removed print()

* js testd added skiplist

* C++ unit tests for invalid json

* js test var i

* js tests == -> ===

* iFix js tests ===

* jslint ;

* int unsigned int fix

* double deletion fix

* refactoring

* Added const for collection source

* override final, added materialize namespace, fixed femove in hash index mock

* Removed expansion support. Index can contain null value in case of several object with same fields

* isNumber<>

* Optimization for single collection case

* Renaming

* refactoring

* Unit tests. Added checking documents

* !sliceIt.valid() instead of !(!=)

* Fix primary index (object instead of array)

* Edge collection test

* Windows compilation fix

* IndexExecutor getCallback optimization

* ADB_UNLIKELY

* explainer.js

* removed unsused variable in js test

* fix explainer.js

* do not use several indexes with js test

* Removed several indexes implementation

* space

* fix explainer.js

* fix explainer.js

* code alignment
2019-11-07 19:16:33 +03:00
jsteemann 9f9b35d685 added assertions 2019-10-30 13:51:26 +01:00
jsteemann 791c926327 make "parallelize-gather" an optimizer rule, so it can be turned off 2019-10-30 11:54:28 +01:00
Simon d526805e81 Bug fix/fix suspicous stuff (#10273) 2019-10-17 15:34:22 +02:00
Dronplane a1014ac701 Bug fix/internal issue #600 - Late document materialization (#10006)
* Applied review cAdded sceleton optimizer functionomments

* Added mask for rule

* Applied review cAdded sceleton optimizer functionomments

* Added mask for rule

* adding plan traversal for rule

* Rule execution block developing

* Experiment with additional block

* LIMIT node modifications

* implemented late materialization base version.

* Code cleanup

* Added check for variable usage

* Implemented serialization and cloning

* Fixed build errors

* Implemented late materialization for  LimitNode

* Moved materialization to Sort Node

* Removed LimitNode modiications

* Implemented skipRows to speedup single server

* Added tests for optimizer rule

* fixed build

* Fix mac build

* Fixed mac build

* Fixed PR test run errors

* Added tmp fix for cluster test runs

* removed debug print

* Make materialization stick to last sort node. Add late materialization to explainer

* Updated optimization rule to search optimal SortNode to do materialization

* Refined optimization rules for cluster

* Added tests for SortExecutor and IResearchViewNode

* Fixed  mac buid. Removed misplaced assertion.

* Fixed build

* Updated tests. Coded cleanup

* Code cleanup

* made function name for SortedExecutor match name in ConstraintSortedExecutor

* cleanup

* Fixed test run

* made scoring results stable across cluster

* Make optimize rule stop if  sort node for limit node is not suitable

* Disabled constrained sort for materializing SortNode

* reverted SortNode modifications

* Implemented separate materializer node

* Fixed tests  to account new materialize node

* Code cleanup

* reverted debugging change

* Fixed test jslint error  and comments

* MaterializeNode renamed

* enum value renamed for materialize node

* Code cleanup

* code cleanup and optimization

* Fixed clang warning

* Applied review comments

* Out variable planning moved to materialize node

* inlined getters in ReadContex
2019-10-08 18:38:47 +03:00
Jan 6f14048bbe
implement early pruning of documents in EnumerateCollectionExecutor (#10159) 2019-10-05 13:38:23 +02:00
Tobias Gödderz f120919dfe Implement a constrained sorting gather variety (#10057)
* Added TODO notes

* Pass limit of constrained sort to sorting gather

* Count output rows, and added assertions

* Extracted row-producing code in a reusable way

* Implemented SortingGatherExecutor::skipRows

* Fixed signature

* Fixed an assertion

* Added limit to (de)serialization

* Fixed an assertion

* Fixed an atMost calculation

* In the sort-limit optimization, apply limit to sorting gather as well

* Fixed optimizer rule reporting modification

* Added missing initialization

* Fix: Do not produce more rows than the limit when skipping first

* Fix: pass limit when cloning

* Updated CHANGELOG

* Fixed atMost intricacies
2019-09-27 10:50:06 +02:00
Markus Pfeiffer 0b9dfb4b9b Subquery Start/End Nodes (#9983)
* Add Start/End subquery node skeleton

* Added a test query that triggers complex interna in the future and needs to proof that our work is side-effect free.

* Add template function to statically assert size of type

* Add Start/End subquery node skeleton

* Add Start/End subquery node creation to ExecutionNode

* Cleanup SubqueryEndNode

* Add test file

* Add isEqualTo to ExecutionNode

* Subquery nodes cleanup

* Update ExecutionNode tests

* Updates to testing code

* More updates

* Add optimizer rule

* Fix optimizer rule

* Fix optimizer rule (again)

* Add a public function to get plan from query

* Fix register planning

* Add test for splicing subquery nodes

* add static asserts for sizes of SubqueryStartNode and SubqueryEndNode

* Add comment about statically asserted size

* Fix isEqualTo virtual and override dance.

* Fix isEqualTo, remove static_assert_size

* Remove a TODO; the plan owns the ExecutionNodes

* Only add stealPlan to Query when testing

* Remove another static_assert_size

* Disabled subquery splice rule by default

* temporarily modified test

* Fixed windows compile

* Fixed unused variable warning
2019-09-25 12:35:18 +02:00
Jan Christoph Uhde 0b8c75c7b7 one shard db - devel (#9395) 2019-09-23 15:48:37 +02:00
Tobias Gödderz 2f59a79435 Bug fix/sort limit rule too lax (#10014)
* Added regression test for consecutive constrained sorts

* Bugfix: sort-limit rule skipped too many node types

* Added CHANGELOG entry

* Apply sort-limit rule before single document operations
2019-09-18 09:04:43 +02:00
Frank Celler aa3d3f8e40
Feature/cleanup ccpcheck (#9665) 2019-08-12 11:11:49 +02:00
Jan c52f2a8315
refactoring (#9411) 2019-07-09 11:15:52 +02:00
KVS85 2d5bf58c94 Remove USE_IRESEARCH definition and usage (#8613) 2019-03-28 17:23:51 +01:00
Jan 39a3f5bc4e
reintroduce smart joins after temporarily reverting them in devel (#8543) 2019-03-23 20:36:02 +01:00
jsteemann dc381a99df Revert "Feature/ncc1701 (#8440)"
This reverts commit 59ad583796.
2019-03-21 19:18:46 +01:00
Jan 59ad583796
Feature/ncc1701 (#8440) 2019-03-21 15:05:36 +01:00
Dan Larkin-York efb8364d99 Optimize memory usage in AQL by using constrained heap sort when reasonable (#7990) 2019-01-31 22:46:57 +01:00
Andrey Abramov d4a010baf9
Bug fix/internal issue #316 (#7911)
* allow using scorers outside ArangoSearch view context

Signed-off-by: Andrey Abramov <andrey@arangodb.com>

* ensure query is properly optimized after replacement of scorer functions

* do not apply `handleViewsRule` to queries without views

* simplify optimization rule for ArangoSearch views

* show ArangoSearch view scorers in query explanation

* fix tests

* fix tests

* add stub for scorer related tests

* reformat

* check variable depth in `ViewExpressionContext::getVariableValue`

* add some tests

* address js test failures

* address jslint errors

* ensure `IResearchViewNode` exposes variables used in scorers

* ensure scorers with expressions are deduplicated

* fix deduplication for indexed access

* more tests

* partially address review comments

* address review comments

* simplify code

* remove irrelevant, commented out code

* ensure array comparisons are properly handled

* update changelog & loki
2019-01-10 21:04:19 +03:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Jan 6d54518145
forward port changes from 3.4 (#6849) 2018-10-12 17:49:17 +02:00
Jan 0d48677619
Bug fix/aql optimizer improvements 2308 (#6241) 2018-08-24 13:42:22 +02:00
Jan 99f176bed5 Feature/add search keyword (#6166)
* added SEARCH keyword

* fixes jslint errors

* add incompatibility notice

* remove VIEW keyword (does not work yet!!)

* add initial support for SEARCH keyword to optimizer rules

* replace FILTER with SEARCH keyword in tests

* removed VIEW keyword, make shell_server_aql tests pass again in single server

* cleanup

* handle SEARCH clause for views

* make SEARCH a non-keyword

* fixed an issue with duplicate variables

* fixed 3 tests

* fix SEARCH statement support for views

* minor refactoring

* fix integration tests

* optimize SEARCH conditions a bit more

* fix jslint error

* fixed wrong comments and typo in class name

* fix documentation

* adjust recovery tests to use SEARCH instead of FILTER

* fix another failing test
2018-08-18 20:22:58 +03:00
Jan Christoph Uhde bcf9e30a4d Feature/cluster single operations (#5585) 2018-07-12 20:18:36 +02:00
Jan Christoph Uhde f81e42af36 Add optimizer rule that replaces js function calls (NEAR/WITHIN/FULLTEXT) with pure AQL (#5529) 2018-06-12 13:48:31 +02:00
Andrey Abramov f708e46973
Feature/arangosearch gather node (#5487)
* integrate ArangoSearch into GatherNode/GatherBlock logic

* fix inconsistency after merge

* add serialization/deserealization logic for IResearchViewNode sorting

* fix optimizer rule order

* fix copy/past typo

* fix tests

* ensure `GatherNode` is properly set up for ArangoSearch views

* adjust optimizer rule order
2018-05-30 02:53:21 +03:00
Jan dc97eb0111
Subquery optimizations (#5398) 2018-05-28 13:56:03 +02:00
Jan 9c0460acff
Feature/aql optimizations (#5237) 2018-05-03 16:55:58 +02:00
Simon 828f1d423c S2 based Geo-Spatial index (#5249) 2018-05-02 23:54:41 +02:00
Andrey Abramov 4649b40b96
Coordinator ArangoSearch view + Execution nodes + AgencyMock (#5160)
* add initial implementation of scatter view rule and node

* add tests for `IResearchViewNode` and `IResearchViewScatterNode`

* add missing check

* modify IResearch execution nodes to use references instead of pointers

* use view id in searialized `ExecutionNode` representation instead of the name

* add cluster mode stubs and checks

* very first attempt to distribute IResearchViewNode

* further implementation of cluster-wide arangosearch views

* fix invalid json format

* add tests for coordinator iresearch view

* allow to retrieve a list of existing views on a coordinator

* more tests for coordinator iresearch view

* some fixes to enable query explanation

* remove Collection dependency from RemoteNode

* remove unnecessary remote ArangoSearch view scatter

* fix explanation appearance

* add some assertions

* minor fixes

* implement IResearchViewCoordinator::updateProperties

* fix view DDL issues

* handle link modifications in DDL operations

* add coordinator implementation of iresearch view links

* fix tests

* further coordinator based view DDL implementation

* further IResearchViewCoordinator implementation

* add initial implementation of AgencyMock

* fix some tests

* code cleanup

* extend test + some fixes

* more tests for IResearchViewCoordinator

* fix tests for IResearchLinkCoordinator

* some fixes after merge

* fix tests

* remove declaration of nonexistent (previously removed) method

* some fixes after review

* remove string duplication

* more tests and fixes

* more fixes and tests

* more tests

* one more test

* fix 'use-after-free' asan error

* fix non-enterprise tests issues
2018-05-02 00:15:11 +03:00
Jan 349e2c1681 forward-port AQL optimizations from 3.3 (#5020)
* forward-port AQL optimizations from 3.3

* fix restrict-to-single-shard rule

* added tests for specific cases

* ModificationNodes now can know if they are restricted or not

* OptimizerRule now updates the ModificationNode instead of setting ownName on remote node.

* The DBServerEngine container now only sends a subset of snippets (those which are relevant) It is  able to restrict any part of the query to a certain shard or even a list of shards

* ExecutionEngine CreateBlocks in ClusterMode now returns Result instead of throwing.

* Adapted Coordinator side to a bitter mapping of RemoteNode => QuerySnippets. This is still ongoing work.

* Fixed Coordinator enginecontainer test and finisehd implementation of EngineInfoContainerDBServer. Seems to work so far, need to adapt test to prove it.

* Adapted test for modification nodes and restrict to shard.

* restrictTo can now be de-/serialized on Modification Nodes

* Fixed Community Compile Error

* Added a test for Indexes/EnumerateCollection nodes

* EnumerateCollection and IndexNode now have options to be restricted to a single shard, opt rule not yet updated

* DBServer part of planning now checks Enumerate and IndexNodes for single shard optimization

* Added tests for simple single shard optimization on IndexNodes

* Adapted optimzierrule for SingleShard optimierung on IndexNodes

* Fixed an issue with externally restricted shard ids in combination with shard_id based optimization

* Added a test for multiple single shard optimizations on the same collection

* Removed collection/Shard duplicate check of optimize to single shard rules

* Updated Explainer to handle optimize to single shard

* Let more tests pass an optimized AQL variant

* Temporarily disabled a tests that can only be solved after some work in transaction
2018-04-30 11:25:09 +02:00
Jan 65bfeb7054
Bug fix/forward port 3.3.5 changes (#4956) 2018-04-03 17:31:34 +02:00
Jan 11cfbf5039
Feature/optimize cluster collect (#4897) 2018-04-03 17:28:07 +02:00
Jan 2a96df5ca5
Feature iresearch (#4071) 2017-12-18 15:04:59 +01:00
Simon Grätzer ec80ee39fd Fulltext Index Cursor (#3807) 2017-12-06 14:29:57 +01:00
Jan 5165155ed1 Bug fix/fixes 0609 (#3227)
* do not use V8 variant of AQL functions in early optimization stage when a C++ variant is available

* additionally, simplify AQL function definitions and aliases

* warn when more than 90% of max mappings are in use

* added C++ variant of replication catchup

* added `--log.role` option

* updated CHANGELOG

* removed non-existing scheduler.threads option from config

* removed useless __FILE__, __LINE__ invocations

* updated CHANGELOG

* allow a priority V8 context

* remove TRI_CORE_MEM_ZONE

* try to fix Windows errors & warnings

* cleanup

* removed memory zones altogether

* exclude system collections from collection tests
2017-09-13 16:28:21 +02:00
Frank Celler a5a25754ed Feature/reduce extraction to projection (#2792)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule

* cppcheck

* try to fix compile error on MacOS

* bug fix for MacOSX

* missing namespace (in Windows compile)
2017-07-14 08:40:29 +02:00
Frank Celler 40d73d5a8b Revert "[WIP] Feature/reduce extraction to projection (#2735)"
This reverts commit 5bfcff30cd.
2017-07-12 12:52:14 +02:00
Jan 5bfcff30cd [WIP] Feature/reduce extraction to projection (#2735)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule
2017-07-12 11:22:29 +02:00
Michael Hackstein 2cd48e614b Feature/optimizer rule remove filter covered by traversal (#2731)
* Added a first test for remove-filter-covered-by-traverser rule.

* initial try at adding the functions

* Initial try at fixing all cases

* Fixing optimizer rule order

* Changing rule

* excluding quantifier ANY

* Fixing assertion error

* quick fix for some failed traverser cases

* Fixing quantifier related tests

* Adding upper limit for scannedIndex in cluster

* Fixing the mmfiles asserts

* Allow the coordinator to do condition checking on edges again. Used in enterprise.

* Fix naming of optimizer rules (#2739)
2017-07-10 10:15:17 +02:00
Frank Celler 6b47544611 Revert "Feature/optimizer rule remove filter covered by traversal" (#2724)
* Revert "Feature/authorization query cache (#2720)"

This reverts commit 83712b7b4a.

* Revert "properly return index in case of unique constraint violation (#2716)"

This reverts commit c3f346e0a5.

* Revert "fix https://github.com/arangodb/planning/issues/388 (#2714)"

This reverts commit 1d944b97a4.

* Revert "fix typo (#2718)"

This reverts commit 61a80ed697.

* Revert "Feature/optimizer rule remove filter covered by traversal (#2676)"

This reverts commit c54b81fb69.
2017-07-04 11:13:36 +02:00
Simon Grätzer c54b81fb69 Feature/optimizer rule remove filter covered by traversal (#2676) 2017-07-03 18:39:32 +02:00
jsteemann 010478ccd2 removed occurrences of mmfiles 2017-03-09 16:40:42 +01:00
jsteemann 57dd92902c move engine-specific optimizer rules into engine 2017-03-07 13:10:58 +01:00