1
0
Fork 0
Commit Graph

157 Commits

Author SHA1 Message Date
Jan b610d99d9f Parallelize certain AQL write operations (#10503)
* added highly experimental startup option `--query.parallelize-gather-writes`

Turning this option on will enable the `parallelize-gather` AQL
optimizer rule for certain write queries. This option is `false`
by default.

The feature is highly experimental and should not be used on a
production system. The option is therefore also hidden.

* parallelize certain write operations in AQL

* remove unneeded _vocbase

* remove unneeded _vocbase

* added startup option and tests
2019-11-27 19:05:04 +01:00
Jan 4213654624 Allow the optimizer to use indexes (#10541)
when a collection attribute is compared to anexpansion followed by an attribute name, e.g. `doc.value IN something[*].name`.
2019-11-27 16:46:31 +01:00
Jan Christoph Uhde 1cd71a8d11 rename timeout to maxRuntime (#10552)
* rename timeout out maxRuntime

* Update CHANGELOG

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
2019-11-27 15:08:50 +01:00
Iurii Popov 9f4926bfec Feature/issue #653 (#10427)
* late document materialization for view draft

* MaterializeType

* asserts and removed std::function

* std::map -> std::array

* Common search ast nodes code to a separate file

* Unit tests for create and clone a node

* js tests

* tmp unfinished

* Replace variables for late materialization

* Move removing

* Removed wrong clears

* Fixed variable replacement

* removed old tests (for separate materialization)

* check if late materialization arangosearch rule is enabled

* js test. added space

* moved a function to anonym namespace

* Subquery assert

* Review improvements

* Review improvements 2

* View node state to a separate var (microoptimization)

* Fixed index assert

* alignment

* C++ unit tests
2019-11-20 22:11:54 +03:00
Jan 932f8b5874
undistribute a few more queries (#10459) 2019-11-20 08:21:44 +01:00
Dronplane af1a10f913 Made test tolerant to leftover links from previous test-runs (#10480)
* Made test tolerant to leftover links from previous test-runs

* Added semicolon
2019-11-19 21:52:39 +03:00
Tobias Gödderz 7a57a72620 Feature/aql subquery splicing with gather (#10341)
Allow Splicing with GATHER nodes
2019-11-18 19:05:44 +01:00
Jan c142e8bc21
fixed issue #10440: Incorrect sorting with sort criteria partially covered by index (#10442) 2019-11-18 15:43:34 +01:00
Iurii Popov caffa6a22e Bug fix/fix late materialization used after sort (#10447)
* Fixed applying late materialization with calc node between sort an limit

* fixed typo

* move micro optimization

* Fixed subqueries error
2019-11-15 17:06:50 +03:00
Andrey Abramov c6fc77858f
Revert "Fixed applying late materialization with calc node between sort an limit (#10439)" (#10446)
This reverts commit e765137c3e.
2019-11-15 12:56:05 +03:00
Dronplane e765137c3e Fixed applying late materialization with calc node between sort an limit (#10439)
* Fixed applying late materialization with calc node between sort an limit

* fixed typo

* move micro optimization
2019-11-15 12:48:55 +03:00
Dronplane a4f2653f4e Feature/internal issue #509 (#10407)
* Added Hint for index creation

* Added tests for proper hint passing

* Added tests

* Added failed link deletion test

* Fixed mmfiles not deleting  failed index

* Fixed cluster run

* Added sync
2019-11-14 18:56:18 +03:00
Jan fd87abc522
allow using `RANDOM_TOKEN` AQL function with an argument value of `0`. (#10414) 2019-11-13 22:20:00 +01:00
Simon 5f7803e705 Parallel SortingGatherExecutor (#10410) 2019-11-13 10:39:47 +01:00
jsteemann 03d02bab49 attempt to make test more deterministic 2019-11-12 17:10:55 +01:00
Jan Christoph Uhde 7b221e94aa kill AQL query after timeout (#10363) 2019-11-12 14:34:52 +01:00
Jan Christoph Uhde 2365ea419a potential fix for DEVSUP-492 (#10379) 2019-11-12 10:57:00 +01:00
Jan 90a4db9440
do not materialize huge ranges (#10376) 2019-11-08 19:54:02 +01:00
Jan 62e39e80b3
yet another micro-optimization (#10383) 2019-11-08 16:09:52 +01:00
Jan f3736b482d
make test work with more than 2 db servers (#10384) 2019-11-07 21:23:38 +01:00
Iurii Popov 26309c5c96 Feature/issue #642 (#10352)
* Index late materialization draft

* Index late materialization. Refactoring.

* Index late materialization. Expansion draft.

* HashIndexMock

* Refactoring.

* Hash index mock expansion fixes

* Fix compare of attributes.

* Hash index mock. Last expansion support.

* Hash index mock. Take value for array from document, not equal key.

* Optimizations.

* Query tests

* lateMaterialized improvements

* velocypack fixes and tests for index node

* Fix after merge.

* Const expression in a condition test

* Clone tests

* More tests

* Fix for materialized view (nullptr var)

* Fixed index vars registers count planning

* javascript tests

* Fixed cluster tests

* Cluster support

* MaterializeExecutor and MaterializerExecutorInfos refactoring

* More tests

* js tests rocksdb only

* js test. Removed print()

* js testd added skiplist

* C++ unit tests for invalid json

* js test var i

* js tests == -> ===

* iFix js tests ===

* jslint ;

* int unsigned int fix

* double deletion fix

* refactoring

* Added const for collection source

* override final, added materialize namespace, fixed femove in hash index mock

* Removed expansion support. Index can contain null value in case of several object with same fields

* isNumber<>

* Optimization for single collection case

* Renaming

* refactoring

* Unit tests. Added checking documents

* !sliceIt.valid() instead of !(!=)

* Fix primary index (object instead of array)

* Edge collection test

* Windows compilation fix

* IndexExecutor getCallback optimization

* ADB_UNLIKELY

* explainer.js

* removed unsused variable in js test

* fix explainer.js

* do not use several indexes with js test

* Removed several indexes implementation

* space

* fix explainer.js

* fix explainer.js

* code alignment
2019-11-07 19:16:33 +03:00
Markus Pfeiffer 972e0b2ce0 Rewrite ModificationExecutor (#10221) 2019-11-07 14:48:26 +01:00
jsteemann c75ad7c1ee fix test 2019-11-07 01:18:54 +01:00
jsteemann 2fbeba9191 fix tests 2019-11-06 19:03:42 +01:00
Simon Grätzer 359723add4
return has more 2019-11-06 18:26:02 +01:00
Simon Grätzer ce539002e1
Merge branch 'devel' of github.com:arangodb/arangodb into feature/parallel-aql-phase-one 2019-11-06 12:09:45 +01:00
Jan 3bafcf2556
try to improve query shutdown in case DB server(s) can't be reached (#10359) 2019-11-05 20:41:51 +01:00
Simon Grätzer 867233b293
remove debug logs 2019-11-05 13:39:36 +01:00
Simon Grätzer b0a2e207c1
Merge branch 'devel' of github.com:arangodb/arangodb into feature/parallel-aql-phase-one 2019-10-31 14:40:28 +01:00
jsteemann 2afe8910bf un-greylist traverser test 2019-10-30 18:16:17 +01:00
Tobias Gödderz 6a3f50fb88 Enable AQL subquery optimization (only on getSome) (#10267)
Activate the Subquery Optimization rule in getSome only cases
2019-10-30 17:10:01 +01:00
Jan f949b175b2
vale, vamos a ver! (#10337) 2019-10-30 16:04:55 +01:00
jsteemann a9a078a22f fix more tests 2019-10-30 13:02:00 +01:00
jsteemann 407d9dcd49 fix more tests 2019-10-30 11:54:19 +01:00
Jan 0e97f322e5
activate early pruning (#10308) 2019-10-28 14:22:33 +01:00
Jan ae818e07d5
Feature/add cluster force one shard option (#10300) 2019-10-28 13:23:33 +01:00
Jan cab9ea9132
fixed issue #10270 (#10271) 2019-10-17 12:00:09 +02:00
KVS85 a33d943684
Temp greylist aql-graph-traverser.js 2019-10-15 17:40:06 +03:00
Jan a3da5cec88
disallow subqueries in AQL traversal PRUNE conditions (#10232) 2019-10-14 16:01:55 +02:00
Tobias Gödderz 9c92287707 Bug fix/fix simple example dep proxy skip some regression test (#10213)
* Added regression test for #10190
2019-10-10 14:30:07 +02:00
Tobias Gödderz 4e86de00d2 AQL Subquery: MultiDependencyRowFetcher (#10101)
Add ShadowRow support for MultiDependencyRowFetcher
2019-10-10 13:08:02 +02:00
Dronplane a1014ac701 Bug fix/internal issue #600 - Late document materialization (#10006)
* Applied review cAdded sceleton optimizer functionomments

* Added mask for rule

* Applied review cAdded sceleton optimizer functionomments

* Added mask for rule

* adding plan traversal for rule

* Rule execution block developing

* Experiment with additional block

* LIMIT node modifications

* implemented late materialization base version.

* Code cleanup

* Added check for variable usage

* Implemented serialization and cloning

* Fixed build errors

* Implemented late materialization for  LimitNode

* Moved materialization to Sort Node

* Removed LimitNode modiications

* Implemented skipRows to speedup single server

* Added tests for optimizer rule

* fixed build

* Fix mac build

* Fixed mac build

* Fixed PR test run errors

* Added tmp fix for cluster test runs

* removed debug print

* Make materialization stick to last sort node. Add late materialization to explainer

* Updated optimization rule to search optimal SortNode to do materialization

* Refined optimization rules for cluster

* Added tests for SortExecutor and IResearchViewNode

* Fixed  mac buid. Removed misplaced assertion.

* Fixed build

* Updated tests. Coded cleanup

* Code cleanup

* made function name for SortedExecutor match name in ConstraintSortedExecutor

* cleanup

* Fixed test run

* made scoring results stable across cluster

* Make optimize rule stop if  sort node for limit node is not suitable

* Disabled constrained sort for materializing SortNode

* reverted SortNode modifications

* Implemented separate materializer node

* Fixed tests  to account new materialize node

* Code cleanup

* reverted debugging change

* Fixed test jslint error  and comments

* MaterializeNode renamed

* enum value renamed for materialize node

* Code cleanup

* code cleanup and optimization

* Fixed clang warning

* Applied review comments

* Out variable planning moved to materialize node

* inlined getters in ReadContex
2019-10-08 18:38:47 +03:00
Jan 0da109bf82
fixed issue #10158 (#10162) 2019-10-04 13:53:31 +02:00
Jan ae05fafa4e
Bug fix/fix parsing unterminated strings (#10155) 2019-10-03 13:38:05 +02:00
Wilfried Goesgens 2b743e0e61 Bug fix/speedup tests (#10109) 2019-10-01 18:23:00 +02:00
Jan 1d84dd861d
use setUpAll and tearDownAll to reduce test durations (#10107) 2019-09-30 17:05:20 +02:00
Tobias Gödderz f120919dfe Implement a constrained sorting gather variety (#10057)
* Added TODO notes

* Pass limit of constrained sort to sorting gather

* Count output rows, and added assertions

* Extracted row-producing code in a reusable way

* Implemented SortingGatherExecutor::skipRows

* Fixed signature

* Fixed an assertion

* Added limit to (de)serialization

* Fixed an assertion

* Fixed an atMost calculation

* In the sort-limit optimization, apply limit to sorting gather as well

* Fixed optimizer rule reporting modification

* Added missing initialization

* Fix: Do not produce more rows than the limit when skipping first

* Fix: pass limit when cloning

* Updated CHANGELOG

* Fixed atMost intricacies
2019-09-27 10:50:06 +02:00
Jan 9da2cadf83
Bug fix/issue 10078 (#10081) 2019-09-26 12:08:48 +02:00
Markus Pfeiffer 0b9dfb4b9b Subquery Start/End Nodes (#9983)
* Add Start/End subquery node skeleton

* Added a test query that triggers complex interna in the future and needs to proof that our work is side-effect free.

* Add template function to statically assert size of type

* Add Start/End subquery node skeleton

* Add Start/End subquery node creation to ExecutionNode

* Cleanup SubqueryEndNode

* Add test file

* Add isEqualTo to ExecutionNode

* Subquery nodes cleanup

* Update ExecutionNode tests

* Updates to testing code

* More updates

* Add optimizer rule

* Fix optimizer rule

* Fix optimizer rule (again)

* Add a public function to get plan from query

* Fix register planning

* Add test for splicing subquery nodes

* add static asserts for sizes of SubqueryStartNode and SubqueryEndNode

* Add comment about statically asserted size

* Fix isEqualTo virtual and override dance.

* Fix isEqualTo, remove static_assert_size

* Remove a TODO; the plan owns the ExecutionNodes

* Only add stealPlan to Query when testing

* Remove another static_assert_size

* Disabled subquery splice rule by default

* temporarily modified test

* Fixed windows compile

* Fixed unused variable warning
2019-09-25 12:35:18 +02:00
Jan 4f2e1d4054
fixed issue #10062: AQL: Could not extract custom attribute (#10068) 2019-09-24 14:36:53 +02:00