1
0
Fork 0
Commit Graph

36 Commits

Author SHA1 Message Date
Jan c52f2a8315
refactoring (#9411) 2019-07-09 11:15:52 +02:00
Tobias Gödderz d1416bed74 AQL by line: IResearch blocks as executors (#8387)
* Added RemoteExecutor skeleton

* Moved RemoteBlock implementations to ExecutionBlockImpl<RemoteExecutor>

* Remove unnecessary include to avoid unused function warnings

* Fixed gcc compile error

* Moved Scatter/Distribute block implementations to their new Executor versions

* Applied clang-format

* Added factory, infos and a skeleton for the unordered view executor

* Removed assert based on wrong assumption

* Added members from IResearchViewBlockBase to IResearchViewExecutor

* Moved more code into the ViewExecutor, hopefully enough to produce a working version now

* Added missing reset code, made produceRow work mostly correct

* Removed superfluous parentheses to get more useful output from Catch

* Ported fix 923b6e81ac723d1fe37f8e7bf1ab81149f3a08ef

Original commit message was:
Fixed a race condition in RemoteBlock which was triggered during
shutdown overtaking getSome.

* Applied review comments

* Inject input row instead of an item block + pos into the expression context, plus fixed some tests

* Adapted test. Search tests are now green.

* Do not ask upstream when already DONE

* Removed `limit` from next()

* Simplified code that could handle producing more than one document

* Minor readability change

* Solved two TODOs noted in the review

* Removed leftover references to DistributeNode members in the DistributeBlock

* Reverted removal of "exhausted"

* WIP: Implemented variant with scorers

* Fixed compile errors of the last commit

* Fixed some asserts and calculations

* Fixed violated assertions

* Moved files from IResearch/ to Aql/

* Replaced recursive call with a loop

* Worked on a few TODOs

* Removed IResearchViewBlock

* Set input registers correctly

* Eliminated dependency to the Node in the Executor

* Don't misuse the volatility variables for initialization

* Extended a TODO note

* Removed obsolete includes

* Removed an obsolete include from the tests

* Added missing include

* Read PKs in batches

* Fixed merge conflict

* Fixed merge conflict

* Restrict prefetching of PKs to the number of rows in the current output block

* Fixed merge

* Fix IResearch ASan errors

* Revert "Restrict prefetching of PKs to the number of rows in the current output block"

This reverts commit e0fd8698a3.

* Revert "Read PKs in batches"

This reverts commit c06c4d7a36.

* Began some small step refactoring to introduce batch-reading correctly

* Extracted method fillBuffer

* Extracted method evaluateScores

* Minor changes

* Read data from iresearch index in batches

* Replaced std::deque<IndexResult> buffer by a new class

* Solved minor TODOs

* Fixed last commit

* Fixed merge conflict

* Removed accidentally re-added view blocks

* Implemented review comments
2019-04-04 23:40:10 +03:00
KVS85 2d5bf58c94 Remove USE_IRESEARCH definition and usage (#8613) 2019-03-28 17:23:51 +01:00
Jan 39a3f5bc4e
reintroduce smart joins after temporarily reverting them in devel (#8543) 2019-03-23 20:36:02 +01:00
jsteemann dc381a99df Revert "Feature/ncc1701 (#8440)"
This reverts commit 59ad583796.
2019-03-21 19:18:46 +01:00
Jan 59ad583796
Feature/ncc1701 (#8440) 2019-03-21 15:05:36 +01:00
Jan 1798036ea0
Bug fix/optimizations 18022019 (#8180) 2019-02-19 19:24:04 +01:00
Dan Larkin-York efb8364d99 Optimize memory usage in AQL by using constrained heap sort when reasonable (#7990) 2019-01-31 22:46:57 +01:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Jan 6d54518145
forward port changes from 3.4 (#6849) 2018-10-12 17:49:17 +02:00
Jan d3c446156c
slightly refactor the handling of optimizer rules (#6855) 2018-10-12 13:49:46 +02:00
Jan f59761dcc2
small issues (#6675) 2018-10-02 17:47:22 +02:00
Jan 0d48677619
Bug fix/aql optimizer improvements 2308 (#6241) 2018-08-24 13:42:22 +02:00
Jan 99f176bed5 Feature/add search keyword (#6166)
* added SEARCH keyword

* fixes jslint errors

* add incompatibility notice

* remove VIEW keyword (does not work yet!!)

* add initial support for SEARCH keyword to optimizer rules

* replace FILTER with SEARCH keyword in tests

* removed VIEW keyword, make shell_server_aql tests pass again in single server

* cleanup

* handle SEARCH clause for views

* make SEARCH a non-keyword

* fixed an issue with duplicate variables

* fixed 3 tests

* fix SEARCH statement support for views

* minor refactoring

* fix integration tests

* optimize SEARCH conditions a bit more

* fix jslint error

* fixed wrong comments and typo in class name

* fix documentation

* adjust recovery tests to use SEARCH instead of FILTER

* fix another failing test
2018-08-18 20:22:58 +03:00
Vasiliy 6fd541d110 issue 427.5: use ApplicationServer reference instead of pointer (#6145)
* issue 427.5: use ApplicationServer reference instead of pointer

* address MSVC build failure
2018-08-15 12:16:02 +03:00
Michael Hackstein 7a95c5e675
Feature/feature phases (#5272)
* Added feature phases

* BasicsPhase and DatabasePhase to the required files. Server now has Feature circles and does not boot. Will be sorted out later on.

* Added ClusterPhase to features

* Added V8Phase to the required features

* Added AQLPhase to the affected features

* Added ServerPhase to Features

* Added FoxxPhase to the relevant features

* Added AgencyPhase to the relevant features

* Moved registration from local variable SYS_SYSTEM_REPLICATION_FACTOR from cluster to V8 as their ordering is now vice versa

* Moved Bootstrap feature into FoxxPhase. It could be moved to ServerPhase easily if the FoxxQueue dependency would be removed

* Final movement of Startup Phases. Now solved all circles.

* Removed merge conflict

* Moved ReplicationTimeout into cluster phase and fixed cross-phase requirements

* Added greetings phase. This phase separates the Basics Phase and is the first to be run. Includes Logger and Hello/Goodbye

* Added the GreetingsPhase in the corresponding features. Now all BasicsPhase features start after greetings Phase. There is some issue in this branch which prevents the Agency from Gossipping right now. Will be fixed next

* Moved creation of the Agent into the prepare phase of the feature. THereby it is guaranteed that agents at least exists before the GeneralServer is activating endpoints

* Recovery needs to be started after the ServerID

* Moved log output of FeaturePhases to DEBUG instead of ERROR.

* Added feature phases for clients

* ClusterFeature now does not directly require AgencyFeature any more

* Added requirement of TravEngineRegistryFeature in AQL feature. Otherwise shutdown may be undefined

* The ApplicationServer can now handout the list of ordered features. Used for testing purposes

* Fixed IResearchVew Tests Setup to honor new feature ordering

* Fixed IResearchViewDBServer Tests Setup to honor new feature ordering

* Started fixing IResearchView Coordinator tests with startup ordering. Not finished yet

* Added startup phases to ViewCoordinator test

* Disabled expected logoutput in ClusterRepairsTest

* Fixed indention in test code

* LinkCoordinator now honors startup ordering

* Link meta now honors startup rdering

* Supress expected cluster logs in ViewTest

* Removed '#' accidentially added.
2018-07-16 14:09:36 +02:00
Jan Christoph Uhde bcf9e30a4d Feature/cluster single operations (#5585) 2018-07-12 20:18:36 +02:00
Jan Christoph Uhde f81e42af36 Add optimizer rule that replaces js function calls (NEAR/WITHIN/FULLTEXT) with pure AQL (#5529) 2018-06-12 13:48:31 +02:00
Jan dc97eb0111
Subquery optimizations (#5398) 2018-05-28 13:56:03 +02:00
Simon 828f1d423c S2 based Geo-Spatial index (#5249) 2018-05-02 23:54:41 +02:00
Andrey Abramov 4649b40b96
Coordinator ArangoSearch view + Execution nodes + AgencyMock (#5160)
* add initial implementation of scatter view rule and node

* add tests for `IResearchViewNode` and `IResearchViewScatterNode`

* add missing check

* modify IResearch execution nodes to use references instead of pointers

* use view id in searialized `ExecutionNode` representation instead of the name

* add cluster mode stubs and checks

* very first attempt to distribute IResearchViewNode

* further implementation of cluster-wide arangosearch views

* fix invalid json format

* add tests for coordinator iresearch view

* allow to retrieve a list of existing views on a coordinator

* more tests for coordinator iresearch view

* some fixes to enable query explanation

* remove Collection dependency from RemoteNode

* remove unnecessary remote ArangoSearch view scatter

* fix explanation appearance

* add some assertions

* minor fixes

* implement IResearchViewCoordinator::updateProperties

* fix view DDL issues

* handle link modifications in DDL operations

* add coordinator implementation of iresearch view links

* fix tests

* further coordinator based view DDL implementation

* further IResearchViewCoordinator implementation

* add initial implementation of AgencyMock

* fix some tests

* code cleanup

* extend test + some fixes

* more tests for IResearchViewCoordinator

* fix tests for IResearchLinkCoordinator

* some fixes after merge

* fix tests

* remove declaration of nonexistent (previously removed) method

* some fixes after review

* remove string duplication

* more tests and fixes

* more fixes and tests

* more tests

* one more test

* fix 'use-after-free' asan error

* fix non-enterprise tests issues
2018-05-02 00:15:11 +03:00
Jan 349e2c1681 forward-port AQL optimizations from 3.3 (#5020)
* forward-port AQL optimizations from 3.3

* fix restrict-to-single-shard rule

* added tests for specific cases

* ModificationNodes now can know if they are restricted or not

* OptimizerRule now updates the ModificationNode instead of setting ownName on remote node.

* The DBServerEngine container now only sends a subset of snippets (those which are relevant) It is  able to restrict any part of the query to a certain shard or even a list of shards

* ExecutionEngine CreateBlocks in ClusterMode now returns Result instead of throwing.

* Adapted Coordinator side to a bitter mapping of RemoteNode => QuerySnippets. This is still ongoing work.

* Fixed Coordinator enginecontainer test and finisehd implementation of EngineInfoContainerDBServer. Seems to work so far, need to adapt test to prove it.

* Adapted test for modification nodes and restrict to shard.

* restrictTo can now be de-/serialized on Modification Nodes

* Fixed Community Compile Error

* Added a test for Indexes/EnumerateCollection nodes

* EnumerateCollection and IndexNode now have options to be restricted to a single shard, opt rule not yet updated

* DBServer part of planning now checks Enumerate and IndexNodes for single shard optimization

* Added tests for simple single shard optimization on IndexNodes

* Adapted optimzierrule for SingleShard optimierung on IndexNodes

* Fixed an issue with externally restricted shard ids in combination with shard_id based optimization

* Added a test for multiple single shard optimizations on the same collection

* Removed collection/Shard duplicate check of optimize to single shard rules

* Updated Explainer to handle optimize to single shard

* Let more tests pass an optimized AQL variant

* Temporarily disabled a tests that can only be solved after some work in transaction
2018-04-30 11:25:09 +02:00
Jan 11cfbf5039
Feature/optimize cluster collect (#4897) 2018-04-03 17:28:07 +02:00
Jan 2a96df5ca5
Feature iresearch (#4071) 2017-12-18 15:04:59 +01:00
Simon Grätzer ec80ee39fd Fulltext Index Cursor (#3807) 2017-12-06 14:29:57 +01:00
Jan 5165155ed1 Bug fix/fixes 0609 (#3227)
* do not use V8 variant of AQL functions in early optimization stage when a C++ variant is available

* additionally, simplify AQL function definitions and aliases

* warn when more than 90% of max mappings are in use

* added C++ variant of replication catchup

* added `--log.role` option

* updated CHANGELOG

* removed non-existing scheduler.threads option from config

* removed useless __FILE__, __LINE__ invocations

* updated CHANGELOG

* allow a priority V8 context

* remove TRI_CORE_MEM_ZONE

* try to fix Windows errors & warnings

* cleanup

* removed memory zones altogether

* exclude system collections from collection tests
2017-09-13 16:28:21 +02:00
Frank Celler a5a25754ed Feature/reduce extraction to projection (#2792)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule

* cppcheck

* try to fix compile error on MacOS

* bug fix for MacOSX

* missing namespace (in Windows compile)
2017-07-14 08:40:29 +02:00
Frank Celler 40d73d5a8b Revert "[WIP] Feature/reduce extraction to projection (#2735)"
This reverts commit 5bfcff30cd.
2017-07-12 12:52:14 +02:00
Jan 5bfcff30cd [WIP] Feature/reduce extraction to projection (#2735)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule
2017-07-12 11:22:29 +02:00
Michael Hackstein 2cd48e614b Feature/optimizer rule remove filter covered by traversal (#2731)
* Added a first test for remove-filter-covered-by-traverser rule.

* initial try at adding the functions

* Initial try at fixing all cases

* Fixing optimizer rule order

* Changing rule

* excluding quantifier ANY

* Fixing assertion error

* quick fix for some failed traverser cases

* Fixing quantifier related tests

* Adding upper limit for scannedIndex in cluster

* Fixing the mmfiles asserts

* Allow the coordinator to do condition checking on edges again. Used in enterprise.

* Fix naming of optimizer rules (#2739)
2017-07-10 10:15:17 +02:00
Frank Celler 6b47544611 Revert "Feature/optimizer rule remove filter covered by traversal" (#2724)
* Revert "Feature/authorization query cache (#2720)"

This reverts commit 83712b7b4a.

* Revert "properly return index in case of unique constraint violation (#2716)"

This reverts commit c3f346e0a5.

* Revert "fix https://github.com/arangodb/planning/issues/388 (#2714)"

This reverts commit 1d944b97a4.

* Revert "fix typo (#2718)"

This reverts commit 61a80ed697.

* Revert "Feature/optimizer rule remove filter covered by traversal (#2676)"

This reverts commit c54b81fb69.
2017-07-04 11:13:36 +02:00
Simon Grätzer c54b81fb69 Feature/optimizer rule remove filter covered by traversal (#2676) 2017-07-03 18:39:32 +02:00
Simon Grätzer df7e12d79b Working geo index + tests 2017-05-09 15:04:51 +02:00
jsteemann ef3322b81f make SORT RAND() removal a feature of the old engine 2017-04-04 08:55:08 +02:00
jsteemann 645889949c move engine-specific V8 functions into storage engine 2017-03-09 12:59:44 +01:00
jsteemann 57dd92902c move engine-specific optimizer rules into engine 2017-03-07 13:10:58 +01:00