1
0
Fork 0
Commit Graph

74 Commits

Author SHA1 Message Date
Tobias Gödderz 168103af88 Splicing gather cleanup (#10479)
Some cleanup and deprecation information
2019-11-21 13:00:31 +01:00
Jan 932f8b5874
undistribute a few more queries (#10459) 2019-11-20 08:21:44 +01:00
Tobias Gödderz 7a57a72620 Feature/aql subquery splicing with gather (#10341)
Allow Splicing with GATHER nodes
2019-11-18 19:05:44 +01:00
Simon 5f7803e705 Parallel SortingGatherExecutor (#10410) 2019-11-13 10:39:47 +01:00
Jan d400830814
Feature/parallel aql phase one 2 (#10408) 2019-11-12 15:47:31 +01:00
Jan 98880f3937
clean up a bit (#10391) 2019-11-11 09:28:18 +01:00
Markus Pfeiffer 972e0b2ce0 Rewrite ModificationExecutor (#10221) 2019-11-07 14:48:26 +01:00
jsteemann 791c926327 make "parallelize-gather" an optimizer rule, so it can be turned off 2019-10-30 11:54:28 +01:00
jsteemann d6d8f321e7 move static variable from function into TU scope 2019-10-22 16:04:55 +02:00
Simon Grätzer 02760bff67
ffffff 2019-10-22 12:48:18 +02:00
Simon Grätzer 5ce1b8bcc1
ffffff 2019-10-22 12:47:44 +02:00
Simon Grätzer b575c3a971
Support for fetching in parallel & skipping 2019-10-15 16:38:08 +02:00
Dan Larkin-York 6fd419714f Move containers into their own folder and namespace. (#10244) 2019-10-14 10:55:55 +02:00
Dan Larkin-York 13e24b2db9 Convert many uses of ClusterComm to Fuerte (#10154) 2019-10-10 14:03:33 +02:00
Jan 6f14048bbe
implement early pruning of documents in EnumerateCollectionExecutor (#10159) 2019-10-05 13:38:23 +02:00
Markus Pfeiffer fc634a2fa0 Replace template <bool blockPassthrough> (#10127)
* Replace template <bool blockPassthrough>

by template <BlockPassthrough blockPassthrough> where BlockPassthrough is an
enum class.

This is mainly for better readability and for some type safety.

* Fixup some implicit conversions to bool
2019-10-02 09:47:09 +02:00
Tobias Gödderz f120919dfe Implement a constrained sorting gather variety (#10057)
* Added TODO notes

* Pass limit of constrained sort to sorting gather

* Count output rows, and added assertions

* Extracted row-producing code in a reusable way

* Implemented SortingGatherExecutor::skipRows

* Fixed signature

* Fixed an assertion

* Added limit to (de)serialization

* Fixed an assertion

* Fixed an atMost calculation

* In the sort-limit optimization, apply limit to sorting gather as well

* Fixed optimizer rule reporting modification

* Added missing initialization

* Fix: Do not produce more rows than the limit when skipping first

* Fix: pass limit when cloning

* Updated CHANGELOG

* Fixed atMost intricacies
2019-09-27 10:50:06 +02:00
Tobias Gödderz 82b0a5e745 De-inline in AQL (#10025)
* Enable IPO (LTO) via CMake

* Use separate IPO_ENABLED variable

* Enabled IPO for more binaries

* Enabled IPO for arangobackup

* Suppress an MSVC warning (though it is not unjustified...)

* Fix static builds, broken due to Policy CMP0060 introduced in cmake 3.3

* Removed policies superseded by minimum cmake version

* Disable IPO with google tests due to a g++ bug

* Disable IPO with google tests only on AUTO

* Disable warning for correct line

* Begin de-inlining Aql

* de-inlining Aql: 2nd batch

* de-inlining Aql: 3rd batch

* de-inlining Aql: 4th batch

* Moved code out of ifdef

* Enabled IPO for additional libs

* Fixed some problems found with IPO enabled

- Fixed ODR violation in Pregel
- Removed unused code in ClusterMethods
- Avoid possible uninitialized variable usage

* Set IPO globally, except for 3rdParty libs

* De-inlined AstHelper again (was undone due to a previous merge conflict)

* Added missing .cpp file

* Fixed compilation with MSVC

* Removed superfluous includes from deinlined AQL header files

* Added includes for size_t

* Deleted outdated files

* De-inlined files that were undone during the merge

* Removed includes from files that were undone during merge

* Fixed merge conflicts
2019-09-25 11:20:03 +02:00
Michael Hackstein 265eb1549d
Feature/aql subquery opt shadow row interface (#9987)
* Added first draft of ShadowRow Interface

* Refactored RegisterPlan and pulled it out of the ExecutionNode

* AqlItemBlock now has an additional hidden register to store subquery depth infromation. Right now no shadow rows are created, however thy could now make use of this situation

* Added API and test to insert new shadow rows into an OutputRow.

* Extrated test helper function

* Added API and tests to consume ShadowRows. Interface is there and compiles with templates, we might need to implement further underlying functions later on as we make progress on ShadowRows, it is only implemented in the minimal way (intentionally)

* Added a test for nested ShadowRows and adapted OutputRow accordingly.

* Added additional memory include, seems to be required under GCC not udner CLANG

* it actually helps to save fies before commiting them...

* Fixed serialization/Deserialization of AqlItemBlock to contain the hidden subquery register now

* Added a c++ test for AqlItemBlock. It just covers the basics thus far and needs to be improved.

* Fixed toVPack of AqlInputRow for subqueries. Also added serialization / deserialization tests for AqlItemRow

* Added a c++ test for inputAqlItem row serialization => AqlItemBlock deserialization

* Attempt to fix :windows: compile warning

* Added a serialization format to be backwards compatible to 3.5 whenever AQL item blocks need to be send over to remote nodes.

* Added more tests for serialization of shadow rows, and fixed a bug revealed by this

* Apply suggestions from code review

Thank you for review!

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review comments. Thanks for spotting

* Fixed hidden merge conflicts

* Attempt to explain windows on how one can actually cast numbers into other number types...

* Update arangod/Aql/AqlItemBlockSerializationFormat.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>
2019-09-24 09:09:52 +02:00
Jan Christoph Uhde 0b8c75c7b7 one shard db - devel (#9395) 2019-09-23 15:48:37 +02:00
Jan 98fea326ea
handle K_SHORTEST_PATHS where it was forgotten (#9954) 2019-09-09 14:54:31 +02:00
Tobias Gödderz 62ed576c8e Resolved or removed some TODOs (#8756) 2019-04-15 14:49:12 +02:00
Heiko 8e308680d8 Bug fix/exec block cleanup (#8710)
* remove unused functions

* more cleanup, virtual functions

* rm old functions

* more cleanup

* more cleanup, rm unused code

* rm of inheritRegisters and cleaerRegisters

* more cleanup, move functions into IMPL and cluster blocks

* used wrong shutdown function

* rm not reachable code

* moved lots of ExecutionBlock stuff to Impl and ClusterBlocks

* trace functions back to executionblock

* rm trace

* rm empty protected

* this getBlock addition, might be useless

* more fixes

* fixes for the distribute executor, hopefully almost done now

* removed obsolete todos

* shutdown var order

* applied requested changes

* re added const

* suppress a warning

* added forgotten test changes

* default destructor, removed not needed function

* refactor name

* Update tests/CMakeLists.txt

Co-Authored-By: hkernbach <hkernbach@users.noreply.github.com>
2019-04-12 16:11:03 +02:00
Michael Hackstein a12a28b094
Bug fix/fixes asan uninitialized by line (#8588)
* Fixed maybe-uninitialized warnings by removing unnecessary boost:optionals

* Fixed use after free

* Update arangod/Aql/SingleRemoteModificationExecutor.cpp

* Fixed another autocast number=>bool thanks c !

* Fixed wrong usage of almost identically named variables
2019-03-27 20:54:36 +01:00
Jan Christoph Uhde c3f7961b88 apply unique log ids (#8561) 2019-03-25 20:26:51 +01:00
Jan Christoph Uhde 5711854531 add SingleRemoteModificationExecutor (#8245) 2019-03-25 07:07:57 +01:00
Jan Christoph Uhde 5d5b77abb8 add ModificationExecutor (#8179)
All modification operator as Executors
2019-03-24 20:48:38 +01:00
Michael Hackstein 20d3141486
Feature/aql gather executors (#8514)
Implemented AQL Gather Executors by line
2019-03-24 17:09:07 +01:00
Tobias Gödderz 9203579218 Initialize alternativeRegId (#8480) 2019-03-20 15:00:08 +01:00
Tobias Gödderz f97d0a211f Implement ScatterExecutor and DistributeExecutor (#8292) 2019-03-20 13:40:20 +01:00
Tobias Gödderz fc5f75a452 Implement RemoteExecutor (#8280) 2019-03-11 10:16:55 +01:00
Andrey Abramov 36cfec1cce
evaluate sort mode of view gather block more precisely (#8008)
* evaluate sort mode of view gather block more precisely

* minor cleanup

* add missing line
2019-01-22 12:22:36 +03:00
Jan 9ce9312239
speed up containers that contain ExecutionNodes' variables (#7937) 2019-01-14 13:38:56 +01:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Jan 762c23bf77
add more tests and improve queries a bit for single remote AQL operations (#7658) 2018-12-05 17:37:25 +01:00
Jan 6fc8bb4a45
speed up remove ops for RocksDB engine (#7639) 2018-12-04 19:23:49 +01:00
Jan fbf2904ab0
fixed issue #6601 (#6608) 2018-09-26 13:42:50 +02:00
Jan 0d48677619
Bug fix/aql optimizer improvements 2308 (#6241) 2018-08-24 13:42:22 +02:00
jsteemann 2d9e5288ab try to fix windows compile warnings and errors 2018-07-13 11:48:05 +02:00
Jan Christoph Uhde bcf9e30a4d Feature/cluster single operations (#5585) 2018-07-12 20:18:36 +02:00
Jan edb24deafe
cleanup collection/vocbase interface of ExecutionNodes (#5557) 2018-06-08 15:28:48 +02:00
Andrey Abramov e117a97fb4
Feature/arangosearch scatter node (#5536)
* remove collection dependency from scatter

* simplify `createBlock` API

* remove useless IResearchViewScatter node

* some fixes

* code cleanup

* scatter arangosearch view

* cleanup

* fix compilation errors

* fix test issues

* store shared_ptr rather than raw pointer in IResearchViewNode

* refactor ScatterBlock

* refactor DistributeBlock

* ensure no leaks during execution node cloning

* some fixes

* code cleanup

* remove vocbase from `DistributeNode`

* add missing ifdef

* fixes after review
2018-06-06 20:42:53 +03:00
Andrey Abramov 1a83809b84
split gather block (#5506)
* initial split into sorted and unsorted blocks

* some fixes

* move gather blocks out of header

* cleanup

* cleanup

* ensure ExecutionBlock constructor is noexcept

* extract sorting logic into separate SortingStrategy class

* ensure we not rebuild heap each getSome/skipSome call

* cleanup API attempt

* Revert "cleanup API attempt"

This reverts commit c824de1a1d29e2cb7f405138f830cc0e02e7667b.

* Revert "Revert "cleanup API attempt""

This reverts commit 65a2c95f6fdbbcbd00d6582af4d354fe339d4738.

* fix comments

* fixes after review

* more fixes after review
2018-06-01 22:26:51 +03:00
Andrey Abramov c41b056209
add missing ifdefs (#5499) 2018-05-30 17:22:27 +03:00
Andrey Abramov f708e46973
Feature/arangosearch gather node (#5487)
* integrate ArangoSearch into GatherNode/GatherBlock logic

* fix inconsistency after merge

* add serialization/deserealization logic for IResearchViewNode sorting

* fix optimizer rule order

* fix copy/past typo

* fix tests

* ensure `GatherNode` is properly set up for ArangoSearch views

* adjust optimizer rule order
2018-05-30 02:53:21 +03:00
Andrey Abramov 0f0235e8a9
Remove collection dependency from gather node (#5457)
* remove collection dependency

* some fixes

* fix test issues

* add tests for newly created functionality

* fix enterprise test fails

* add comment
2018-05-25 17:26:33 +03:00
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00
Vasiliy 27f7f4a0aa issue 389.9: return TRI_vocbase_t& from Query::vocbase() (#5420)
* issue 389.9: return TRI_vocbase_t& from Query::vocbase()

* backport: use view name instead of view id in error messages
2018-05-22 19:03:14 +03:00
Jan 9c0460acff
Feature/aql optimizations (#5237) 2018-05-03 16:55:58 +02:00
Andrey Abramov 4649b40b96
Coordinator ArangoSearch view + Execution nodes + AgencyMock (#5160)
* add initial implementation of scatter view rule and node

* add tests for `IResearchViewNode` and `IResearchViewScatterNode`

* add missing check

* modify IResearch execution nodes to use references instead of pointers

* use view id in searialized `ExecutionNode` representation instead of the name

* add cluster mode stubs and checks

* very first attempt to distribute IResearchViewNode

* further implementation of cluster-wide arangosearch views

* fix invalid json format

* add tests for coordinator iresearch view

* allow to retrieve a list of existing views on a coordinator

* more tests for coordinator iresearch view

* some fixes to enable query explanation

* remove Collection dependency from RemoteNode

* remove unnecessary remote ArangoSearch view scatter

* fix explanation appearance

* add some assertions

* minor fixes

* implement IResearchViewCoordinator::updateProperties

* fix view DDL issues

* handle link modifications in DDL operations

* add coordinator implementation of iresearch view links

* fix tests

* further coordinator based view DDL implementation

* further IResearchViewCoordinator implementation

* add initial implementation of AgencyMock

* fix some tests

* code cleanup

* extend test + some fixes

* more tests for IResearchViewCoordinator

* fix tests for IResearchLinkCoordinator

* some fixes after merge

* fix tests

* remove declaration of nonexistent (previously removed) method

* some fixes after review

* remove string duplication

* more tests and fixes

* more fixes and tests

* more tests

* one more test

* fix 'use-after-free' asan error

* fix non-enterprise tests issues
2018-05-02 00:15:11 +03:00