1
0
Fork 0
Commit Graph

1457 Commits

Author SHA1 Message Date
Iurii Popov 9f4926bfec Feature/issue #653 (#10427)
* late document materialization for view draft

* MaterializeType

* asserts and removed std::function

* std::map -> std::array

* Common search ast nodes code to a separate file

* Unit tests for create and clone a node

* js tests

* tmp unfinished

* Replace variables for late materialization

* Move removing

* Removed wrong clears

* Fixed variable replacement

* removed old tests (for separate materialization)

* check if late materialization arangosearch rule is enabled

* js test. added space

* moved a function to anonym namespace

* Subquery assert

* Review improvements

* Review improvements 2

* View node state to a separate var (microoptimization)

* Fixed index assert

* alignment

* C++ unit tests
2019-11-20 22:11:54 +03:00
Jan 932f8b5874
undistribute a few more queries (#10459) 2019-11-20 08:21:44 +01:00
Iurii Popov 26309c5c96 Feature/issue #642 (#10352)
* Index late materialization draft

* Index late materialization. Refactoring.

* Index late materialization. Expansion draft.

* HashIndexMock

* Refactoring.

* Hash index mock expansion fixes

* Fix compare of attributes.

* Hash index mock. Last expansion support.

* Hash index mock. Take value for array from document, not equal key.

* Optimizations.

* Query tests

* lateMaterialized improvements

* velocypack fixes and tests for index node

* Fix after merge.

* Const expression in a condition test

* Clone tests

* More tests

* Fix for materialized view (nullptr var)

* Fixed index vars registers count planning

* javascript tests

* Fixed cluster tests

* Cluster support

* MaterializeExecutor and MaterializerExecutorInfos refactoring

* More tests

* js tests rocksdb only

* js test. Removed print()

* js testd added skiplist

* C++ unit tests for invalid json

* js test var i

* js tests == -> ===

* iFix js tests ===

* jslint ;

* int unsigned int fix

* double deletion fix

* refactoring

* Added const for collection source

* override final, added materialize namespace, fixed femove in hash index mock

* Removed expansion support. Index can contain null value in case of several object with same fields

* isNumber<>

* Optimization for single collection case

* Renaming

* refactoring

* Unit tests. Added checking documents

* !sliceIt.valid() instead of !(!=)

* Fix primary index (object instead of array)

* Edge collection test

* Windows compilation fix

* IndexExecutor getCallback optimization

* ADB_UNLIKELY

* explainer.js

* removed unsused variable in js test

* fix explainer.js

* do not use several indexes with js test

* Removed several indexes implementation

* space

* fix explainer.js

* fix explainer.js

* code alignment
2019-11-07 19:16:33 +03:00
jsteemann 791c926327 make "parallelize-gather" an optimizer rule, so it can be turned off 2019-10-30 11:54:28 +01:00
Simon Grätzer 5e0321900e
Merge branch 'feature/parallel-aql-phase-one' of github.com:arangodb/arangodb into feature/parallel-aql-phase-one 2019-10-29 11:47:13 +01:00
jsteemann b2f6fb105c added "parallel" output to explainer 2019-10-22 16:05:12 +02:00
Michael Hackstein bf2423d024
Feature/explain spliced subqueries (#10298)
* Implemented explain output for Subquery start and End nodes. Unfortunately i had to inject a temporary output variable for the Subquery start.
2019-10-22 15:38:33 +02:00
Wilfried Goesgens 9883518621 when changing between leader and follower re-attempt connecting to circumvent OS resource shortages (#10243) 2019-10-14 11:54:37 +03:00
Dronplane a1014ac701 Bug fix/internal issue #600 - Late document materialization (#10006)
* Applied review cAdded sceleton optimizer functionomments

* Added mask for rule

* Applied review cAdded sceleton optimizer functionomments

* Added mask for rule

* adding plan traversal for rule

* Rule execution block developing

* Experiment with additional block

* LIMIT node modifications

* implemented late materialization base version.

* Code cleanup

* Added check for variable usage

* Implemented serialization and cloning

* Fixed build errors

* Implemented late materialization for  LimitNode

* Moved materialization to Sort Node

* Removed LimitNode modiications

* Implemented skipRows to speedup single server

* Added tests for optimizer rule

* fixed build

* Fix mac build

* Fixed mac build

* Fixed PR test run errors

* Added tmp fix for cluster test runs

* removed debug print

* Make materialization stick to last sort node. Add late materialization to explainer

* Updated optimization rule to search optimal SortNode to do materialization

* Refined optimization rules for cluster

* Added tests for SortExecutor and IResearchViewNode

* Fixed  mac buid. Removed misplaced assertion.

* Fixed build

* Updated tests. Coded cleanup

* Code cleanup

* made function name for SortedExecutor match name in ConstraintSortedExecutor

* cleanup

* Fixed test run

* made scoring results stable across cluster

* Make optimize rule stop if  sort node for limit node is not suitable

* Disabled constrained sort for materializing SortNode

* reverted SortNode modifications

* Implemented separate materializer node

* Fixed tests  to account new materialize node

* Code cleanup

* reverted debugging change

* Fixed test jslint error  and comments

* MaterializeNode renamed

* enum value renamed for materialize node

* Code cleanup

* code cleanup and optimization

* Fixed clang warning

* Applied review comments

* Out variable planning moved to materialize node

* inlined getters in ReadContex
2019-10-08 18:38:47 +03:00
Jan 6f14048bbe
implement early pruning of documents in EnumerateCollectionExecutor (#10159) 2019-10-05 13:38:23 +02:00
Wilfried Goesgens 823640e8f7 Feature/testing result evaluation (#10124) 2019-10-02 17:49:32 +02:00
Wilfried Goesgens 636b2e5d5f fixed version of arangosh result evaluation refactoring - properly count objects (#10080) 2019-09-26 12:07:52 +02:00
Jan 4f2e1d4054
fixed issue #10062: AQL: Could not extract custom attribute (#10068) 2019-09-24 14:36:53 +02:00
Heiko b9dd097f8d Bug fix/arangosh gharial vertex deletion (#9921)
* typo in route

* unused function removed

* added general graph standalone and cluster tests, formatting

* added changelog entry

* Update CHANGELOG

Co-Authored-By: Michael Hackstein <michael@arangodb.com>

* Update CHANGELOG

Co-Authored-By: Michael Hackstein <michael@arangodb.com>

* removed fail, removed print
2019-09-06 12:26:10 +02:00
Markus Pfeiffer 753ff4aa67 Feature/atomic database creation 2 (#9826) 2019-09-05 12:38:07 +02:00
Jan 1e9ad7fbb9
show query string length and cacheability in explain output (#9767) 2019-08-21 12:06:32 +02:00
KVS85 0c6d3b7ddc
Set jsunity output to gtest style (#9646)
* Set jsunity output to gtest style

* More pretty printing

* Fix jslint

* Remove unnecessary spaces
2019-08-07 22:37:00 +03:00
Simran 71d2651f2c
Aardvark: Add k Shortest Paths example graph to UI (#9491)
* Add example graph to UI

* Add kShortestPathsGraph to examples.js

* Update example-graph.js

* Update aardvark.js
2019-08-07 16:41:38 +02:00
Wilfried Goesgens c76d3a0323 fix test result handling of tests running in own spawned arangosh (#9576) 2019-07-26 15:03:24 +02:00
jsteemann f9cb4d6590 fix explainer for compatibility with 3.4 2019-07-24 17:55:50 +02:00
Jan 973f00a997
Added startup error for bad temporary directory setting. (#9552)
* Added startup error for bad temporary directory setting.

  If the temporary directory (--temp.path) setting is identical to the database
  directory (`--database.directory`) this can eventually lead to data loss, as
  temporary files may be created inside the temporary directory, causing overwrites of
  existing database files/directories with the same names.
  Additionally the temporary directory may be cleaned at some point, and this would lead
  to an unintended cleanup of the database files/directories as well.
  Now, if the database directory and temporary directory are set to the same path, there
  will be a startup warning about potential data loss (though in ArangoDB 3.4 allowing to
  continue the startup - in 3.5 and higher we will abort the startup).
2019-07-24 12:47:42 +02:00
Michael Hackstein cbcf561450
Feature/min replication factor (#9433)
* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it

* Added some assertion son minReplicationFactor

* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled

* added minReplicationFactor to the user interface, preparation for the collection api changes

* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo

* added minReplicationFactor usage to tests

* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME

* minReplicationFactor now able to change via collection  properties route

* fixed wrongly assert

* added minReplicationFactor to the graph management ui

* added minReplicationFactor to the gharial api

* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.

* adjusted description of minReplicationFactor

* FollowerInfo Refactoring

* added gharial api graph creation tests with minimal replication factor

* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests

* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor

* Debug logging

* MORE Debug logging

* Included replication fast lane

* Use correct minreplicationfactor

* modified debug logging

* Fixed compileissues

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* Revert "MORE Debug logging"

This reverts commit dab5af28c0.

* Revert "MORE Debug logging"

This reverts commit 6134b664bd.

* Revert "MORE Debug logging"

This reverts commit 80160bdf3b.

* Revert "MORE Debug logging"

This reverts commit 06aabcdfe1.

* Removed debug output

* Added replication fast lane. Also refactored the commands as i cannot take it any more...

* Put some requests of RocksDBReplication onto CATCHUP Lane.

* Put some requests of MMFilesReplication onto CATCHUP Lane.

* Adjusted Fast and MED lane usage in Supervised scheduler

* Added changelog entry

* Added new features entry

* A new leader will now keep old followers in case of failover

* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Fixed JSLINT

* Unified lane handling of replication handlers

* Sorry forgotten in last commit

* replaced strings with static strings

* more use of static strings

* optimized min repl description in the ui

* decr initial loop variable

* clean up of the createWithId test

* more use of static strings

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Added some comments on condition, renamed variable as suggested in review

* Added check for min replicationFactor to be non-zero

* Added assertion

* Added function to modify min and max replication factor in one go

* added missing semicolon

* rm log devel

* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place

* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers

* check replFactor against nr dbservers

* Add lie reporting in CURRENT

* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out

* move replication checks from logical collection to rest collection handler

* added more replication tests

* Include assert only if we are not in gtest

* jslint

* set min repl factor to zero if satellite collection

* check replication attributes in v8 collection

* fixed ires tests

* fixed wrong assert

* properly check uint

* repl factor attr check

* adjusted test to be more preciese now

* Fixed race on atomics comparison

* Fixed invalid number type

* Update tests/js/common/shell/shell-cluster-collection.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review fixes

* More review fixes
2019-07-19 13:02:28 +02:00
Wilfried Goesgens 1c408dc2fb connect the mocha testresults if we run it in an extra arangosh instance (#9485)
* connect the mocha testresults if we run it in an extra arangosh instance

* fix status handling, lint
2019-07-18 16:41:11 +03:00
Wilfried Goesgens 9449f6a85a migrate mochaGrep into --testCase parametrizing (#9438)
* migrate mochaGrep into --testCase parametrizing

* fix filter forwarding for spawned arangoshs - add mocha version

* sometimes we have the string undefined to ignore
2019-07-18 16:35:44 +03:00
Andrey Abramov cf7d5bfa95 fix arangosearch explanation (#9365) 2019-06-28 19:51:34 +02:00
Jan 074397d367
allow "stream" attribute on top level (#9195) 2019-06-05 15:48:58 +02:00
Markus Pfeiffer 73ac4f194d (K_)SHORTEST PATHS: Display correct directions in explainer/profiler (#9080) 2019-05-29 15:31:09 +02:00
Michael Hackstein 1a6719e144
Revert "(K_)SHORTEST PATHS: Display correct directions in explainer/profiler (#9049)" (#9074)
This reverts commit 44d15db812.
2019-05-22 16:46:45 +02:00
Markus Pfeiffer 44d15db812 (K_)SHORTEST PATHS: Display correct directions in explainer/profiler (#9049) 2019-05-22 16:06:46 +02:00
Jan 1ddb8351b9
properly display the selectivity estimates of unique indexes in the explainer (#8993) 2019-05-14 16:15:49 +02:00
Andrey Abramov 4e12fdb30b
ArangoSearch sorted indexes (#8888)
* move IResearchViewSort into separate unit

* ensure sort is propagated to IResearchLink

* update iresearch and extend some tests

* implement insertion of sorted values, store value comparator in link

* add tests for VPackComparer

* minor cleanup

* fix issue with sort definiton propagation

* propagate error message from index factory

* add integration tests for sorted indexes, single server

* initial implementation of IResearchViewMergeExecutor

* move iresearch::VPackComparer to separate translation unit

* store collection in IndexReadBuffer for IResearchViewMergeExecutor

* set primary sort from optimizer rule

* add some tests for sorted view case

* extend IResearchViewNode tests

* simplify primary sort definition

* pretty print order condition covered by arangosearch in query explanation

* fix compilation with g++-8

* address catch test failures

* fix tests

* adjust arangosearch optimizer rule to optimize sort in case of cluster

* make tests for sorted indexes work in cluster too

* fix tests

* extend js tests for sorted indices

* fix keyword in query explainer

* ensure sort-limit rule works with views

* backport ngram analyzer fix from upstream

* address jslint errors

* modify IResearchViewSort to support boolean direction specification format

* add some tests

* add tests

* add skip functionality

* extend tests

* add more catch tests

* inline some methods

* address jslint errors

* extend catch tests to reproduce the failure

* add some tests for check fullcount

* add more tests

* extend tests

* temporarily comment out broken tests

* extend js tests

* address review comments

* address jslint errors

* update iresearch
2019-05-11 17:58:56 +03:00
KVS85 014490e459
Fix example for kShortestPathsGraph 2019-05-08 17:23:23 +03:00
Jan Christoph Uhde f3a2eaf6d9 Bug fix/permission tests (#8890) 2019-05-07 15:07:59 +02:00
Heiko 8f5ea9b771 jslint (#8915) 2019-05-06 15:52:01 +02:00
Markus Pfeiffer 3109dd6747 Doc - Add documentation for k Shortest Paths (#8836) 2019-05-06 12:56:03 +02:00
Jan 7acb9a9092
make creating debug packages smart graph-aware (#8886) 2019-05-06 09:18:10 +02:00
Jan Christoph Uhde 677a79026c Foxx Security (#8845) 2019-04-25 09:56:29 +02:00
Wilfried Goesgens 6984daf67b record the times used for the setup/teardown methods of jsunity tests; exposed to the json report file to be enabled along with the junit xml report files. (#8838) 2019-04-24 17:00:38 +03:00
Alan Plum 40aff3fdcd Upgrade JS deps (#8773)
* Make jsunity hash function avoid recursion

This guards against traversals of recursive data structures causing a stack overflow. This is crucial when using assertEqual with unknown inputs.

* Update deps

accepts: 1.3.4 -> 1.3.5
ansi_up: 2.0.2 -> 4.0.3
content-disposition: 0.5.2 -> 0.5.3
dedent: 0.6.0 -> 0.7.0
error-stack-parser: 1.3.6 -> 2.0.2
eslint: 2.13.1 -> 5.16.0
eslint-config-semistandard: 6.0.2 -> removed
eslint-config-standard: 5.3.1 -> removed
eslint-plugin-promise: 1.3.2 -> removed
eslint-plugin-standard: 1.3.2 -> removed
highlight.js: 9.12.0 -> 9.15.6
http-errors: 1.6.2 -> 1.7.2
iconv-lite: 0.4.19 -> 0.4.24
joi: 9.2.0 -> 14.3.1
joi-to-json-schema: 2.3.0 -> 4.0.1
js-yaml: 3.10.0 -> 3.13.1
marked: 0.3.9 -> 0.6.2
mime-types: 2.1.12 -> 2.1.22
mocha: 2.5.3 -> 6.1.3
qs: 6.5.1 -> 6.7.0
semver: 5.4.1 -> 6.0.0
statuses: 1.4.0 -> 1.5.0
timezone: 1.0.13 -> 1.0.22
type-is: 1.6.15 -> 1.6.16
underscore: 1.8.3 -> 1.9.1

* Inline eslint semistandard config

The eslint standard config has become extremely opinionated and incompatible with the existing coding style. This subset matches the existing coding style most closely. In the future we should migrate to an autoformatter like prettier and avoid this problem altogether.

* Linting

* Fix mocha runner

* New joi has additional property
2019-04-17 13:36:26 +02:00
Markus Pfeiffer a14386267a k Shortest Paths (#8715)
* Add k-shortest-paths finder
2019-04-12 21:58:18 +02:00
Alan Plum 3007c928aa Add collection.documentId method (#8749) 2019-04-12 19:53:01 +02:00
Jan 207c978d1e
Bug fix/move a few actions from js to cpp (#8549) 2019-03-26 10:02:21 +01:00
Jan 39a3f5bc4e
reintroduce smart joins after temporarily reverting them in devel (#8543) 2019-03-23 20:36:02 +01:00
jsteemann dc381a99df Revert "Feature/ncc1701 (#8440)"
This reverts commit 59ad583796.
2019-03-21 19:18:46 +01:00
Jan 59ad583796
Feature/ncc1701 (#8440) 2019-03-21 15:05:36 +01:00
Jan ad50a51efa
don't check for version upgrades in enterprise edition when firing up the arangosh (#8495) 2019-03-21 13:40:13 +01:00
Dan Larkin-York 2eadab33e7 Index hints (#8431) 2019-03-19 09:14:18 +01:00
Wilfried Goesgens 93792a3d85 when filtering for tests and matching nothing output the non matching tests (#8420) 2019-03-14 13:33:14 +01:00
Michael Hackstein 0403d6cb6f
Bug fix/prune neighbors and explain (#8304)
* Fixed Neighbors prune on depth0

* Fixed the Explainer output

* Rmeoved debug output
2019-03-11 10:13:57 +01:00
Michael Hackstein 6e8d43b2da
Feature/aql traversal prune (#8255)
Allow PRUNE in the AQL Traverser.
2019-02-27 11:43:06 +01:00