1
0
Fork 0
Commit Graph

3357 Commits

Author SHA1 Message Date
Alan Plum a57b6b86ef Fix Foxx security (#9020)
* Unbreak aardvark when --foxx.api false

The easiest way to disable the aardvark foxx routes is to just skip the
import and mount altogether. The previous approach caused the service to
throw an error because the module doesn't actually export anything.
There's no need to prevent the export if we just don't import the module
to begin with.

* Add an error code for service api disabled

* Throw an actual error when accessing Foxx API routes

* Throw an error when using legacy Foxx API

There's no easy way to use a catch-all here so disable them one by one. No new routes will be added to this API and it'll go away in 4.0 so there's no risk of forgetting to add the throw-check in new routes.

* Hide Foxx services in frontend if Foxx API is disabled

* Render subnav without delay

The subnav doesn't need the perm check results but we need the subnav to exist so we can populate it in other parts of the app. This fixes the subnav not showing up when reloading certain views (e.g. install service via upload).

* Hide ONLY the Foxx store when it is disabled

We still want to allow installing and upgrading/replacing services. We just don't want to show the store.

* Build aardvark
2019-05-17 16:41:34 +02:00
Jan 1ddb8351b9
properly display the selectivity estimates of unique indexes in the explainer (#8993) 2019-05-14 16:15:49 +02:00
Andrey Abramov 4e12fdb30b
ArangoSearch sorted indexes (#8888)
* move IResearchViewSort into separate unit

* ensure sort is propagated to IResearchLink

* update iresearch and extend some tests

* implement insertion of sorted values, store value comparator in link

* add tests for VPackComparer

* minor cleanup

* fix issue with sort definiton propagation

* propagate error message from index factory

* add integration tests for sorted indexes, single server

* initial implementation of IResearchViewMergeExecutor

* move iresearch::VPackComparer to separate translation unit

* store collection in IndexReadBuffer for IResearchViewMergeExecutor

* set primary sort from optimizer rule

* add some tests for sorted view case

* extend IResearchViewNode tests

* simplify primary sort definition

* pretty print order condition covered by arangosearch in query explanation

* fix compilation with g++-8

* address catch test failures

* fix tests

* adjust arangosearch optimizer rule to optimize sort in case of cluster

* make tests for sorted indexes work in cluster too

* fix tests

* extend js tests for sorted indices

* fix keyword in query explainer

* ensure sort-limit rule works with views

* backport ngram analyzer fix from upstream

* address jslint errors

* modify IResearchViewSort to support boolean direction specification format

* add some tests

* add tests

* add skip functionality

* extend tests

* add more catch tests

* inline some methods

* address jslint errors

* extend catch tests to reproduce the failure

* add some tests for check fullcount

* add more tests

* extend tests

* temporarily comment out broken tests

* extend js tests

* address review comments

* address jslint errors

* update iresearch
2019-05-11 17:58:56 +03:00
KVS85 014490e459
Fix example for kShortestPathsGraph 2019-05-08 17:23:23 +03:00
Jan Christoph Uhde f3a2eaf6d9 Bug fix/permission tests (#8890) 2019-05-07 15:07:59 +02:00
Heiko 8f5ea9b771 jslint (#8915) 2019-05-06 15:52:01 +02:00
Markus Pfeiffer 3109dd6747 Doc - Add documentation for k Shortest Paths (#8836) 2019-05-06 12:56:03 +02:00
Jan 7acb9a9092
make creating debug packages smart graph-aware (#8886) 2019-05-06 09:18:10 +02:00
Jan Christoph Uhde 677a79026c Foxx Security (#8845) 2019-04-25 09:56:29 +02:00
Wilfried Goesgens 6984daf67b record the times used for the setup/teardown methods of jsunity tests; exposed to the json report file to be enabled along with the junit xml report files. (#8838) 2019-04-24 17:00:38 +03:00
Dan Larkin-York 48fb15e94f Auth-related tests for audit logging (#8790) 2019-04-18 10:45:41 +02:00
Alan Plum 40aff3fdcd Upgrade JS deps (#8773)
* Make jsunity hash function avoid recursion

This guards against traversals of recursive data structures causing a stack overflow. This is crucial when using assertEqual with unknown inputs.

* Update deps

accepts: 1.3.4 -> 1.3.5
ansi_up: 2.0.2 -> 4.0.3
content-disposition: 0.5.2 -> 0.5.3
dedent: 0.6.0 -> 0.7.0
error-stack-parser: 1.3.6 -> 2.0.2
eslint: 2.13.1 -> 5.16.0
eslint-config-semistandard: 6.0.2 -> removed
eslint-config-standard: 5.3.1 -> removed
eslint-plugin-promise: 1.3.2 -> removed
eslint-plugin-standard: 1.3.2 -> removed
highlight.js: 9.12.0 -> 9.15.6
http-errors: 1.6.2 -> 1.7.2
iconv-lite: 0.4.19 -> 0.4.24
joi: 9.2.0 -> 14.3.1
joi-to-json-schema: 2.3.0 -> 4.0.1
js-yaml: 3.10.0 -> 3.13.1
marked: 0.3.9 -> 0.6.2
mime-types: 2.1.12 -> 2.1.22
mocha: 2.5.3 -> 6.1.3
qs: 6.5.1 -> 6.7.0
semver: 5.4.1 -> 6.0.0
statuses: 1.4.0 -> 1.5.0
timezone: 1.0.13 -> 1.0.22
type-is: 1.6.15 -> 1.6.16
underscore: 1.8.3 -> 1.9.1

* Inline eslint semistandard config

The eslint standard config has become extremely opinionated and incompatible with the existing coding style. This subset matches the existing coding style most closely. In the future we should migrate to an autoformatter like prettier and avoid this problem altogether.

* Linting

* Fix mocha runner

* New joi has additional property
2019-04-17 13:36:26 +02:00
Markus Pfeiffer a14386267a k Shortest Paths (#8715)
* Add k-shortest-paths finder
2019-04-12 21:58:18 +02:00
Alan Plum 3007c928aa Add collection.documentId method (#8749) 2019-04-12 19:53:01 +02:00
jsteemann 16a1fabf20 remove dead reference to jqconsole 2019-04-01 09:42:48 +02:00
Jan 207c978d1e
Bug fix/move a few actions from js to cpp (#8549) 2019-03-26 10:02:21 +01:00
Jan 39a3f5bc4e
reintroduce smart joins after temporarily reverting them in devel (#8543) 2019-03-23 20:36:02 +01:00
Simon 3ada15fc35 The Legendary El Cheapo (#8485) 2019-03-22 11:38:33 +01:00
jsteemann dc381a99df Revert "Feature/ncc1701 (#8440)"
This reverts commit 59ad583796.
2019-03-21 19:18:46 +01:00
Jan 59ad583796
Feature/ncc1701 (#8440) 2019-03-21 15:05:36 +01:00
Jan ad50a51efa
don't check for version upgrades in enterprise edition when firing up the arangosh (#8495) 2019-03-21 13:40:13 +01:00
Dan Larkin-York 2eadab33e7 Index hints (#8431) 2019-03-19 09:14:18 +01:00
Wilfried Goesgens 93792a3d85 when filtering for tests and matching nothing output the non matching tests (#8420) 2019-03-14 13:33:14 +01:00
Michael Hackstein 0403d6cb6f
Bug fix/prune neighbors and explain (#8304)
* Fixed Neighbors prune on depth0

* Fixed the Explainer output

* Rmeoved debug output
2019-03-11 10:13:57 +01:00
Michael Hackstein 6e8d43b2da
Feature/aql traversal prune (#8255)
Allow PRUNE in the AQL Traverser.
2019-02-27 11:43:06 +01:00
Wilfried Goesgens 492d05c1f1 Feature/upgrade v8 7.1.302.28 (#8088) 2019-02-19 11:15:34 +01:00
Luschgy b7b84beebf Feature/aql blocks by line (#6553)
* Fixed hidden merge conflict

* Added BlockFetcher, added a minimal test for SingleRowFetcher

* move RegInfo in types.h

* fix getSomeLogic

* Added the AqlItemMatrix implementation. Not yet tested!

* Fixed nr called => Waiting ratio in AllBlocksFetcherMock

* fix compiling

* add RegInfo to produceRow

* BlockFetcherHelper now returns a matrix

* pass RegInfo as requested into AqlItemRow ctor

I would still keep it out as the ItemRow should be a thin view.

* allow to change the AqlItemRows cheaper / remove _sourceRow member

* implement copyRow for AqlItemRow

* organize cleanup for outputblock used in getsome / shrink if necessary

* Added tests for SingleRowFetcher, cleanup in AqlItemRow

* start mocking

* Added test helpers, began work on a more complex test for SingleRowFetcher

* Re-added RegInfo again to AqlItemRow

* helper, added test case for FilterExecutor

* merge fixed compile

* Removed trx name where it is actually not used

* Made some functions TEST_VIRTUAL for mocking.

* implemented Sort executor

* Fixed the BlockFetcherHelper API, to always return a matrix if it is not waiting

* Implemented a SortExecutor test with data

* disable borken test

* Fixed test on SortExecutor. Green now, needs some more testing

* change a bit how the AqlItemRow is used in a test

* Added another test for the SingleRowFetcher

* Added more tests for SingleRowFetcher

* fix getRegisterInfo

* Moved RegInfo into ExecutorInfos. Also now use the FilterExecutor implementation in production code instead of FilterBlock. Starting integration testing.

* add more asserts

* Fixed undefined behaviour in TestCode.

* Do not shrink an empty output block

* switch to block pointers in AqlItemRow

* return blocks in the row fetcher

* Updated grammar.cpp

* Updated (fixed) tests to verify returnBlock

* Improved const-ness

* Changed BlockFetcher constructor in order to make it mockable

* Implemented BlockFetcherMock and rewrote some tests with it

* Updated a comment

* Updated comments

* init commit - EnumerateListExecutor, including EnumerateListExecutorInfos

* add potential fix for ItemMatrix

* Replaced fakeit with BlockFetcherMock in another GIVEN block

* Replaced the last fakeit mocks with BlockFetcherMock and removed fakeit

* Updated function definition

* included missing arary included

* Renamed AqlItemRow to InputAqlItemRow

* Split AqlItemRow into Input~ and Output~ classes

* EnumerateListExecutor + Tests

* Added tests for AqlItemRows

* Added test file for AqlItemRows...

* Made the ostream operator for AqlItemBlocks available to Logger and Catch info

* build error

* Added a second test for AqlItemRows. Need to make input space invalid now, sth. acidential referencing will show up

* Introduce a block ID in the InputAqlItemRow

* Copy output values correctly, avoid InputAqlItemRow pointers

* Added a test for registerClearout

* Fixed tests

* Added a test for adding new values

* Added a test for an incomplete row write to not count as produced.

* Fixed getSome and a test

* Added a test for larger AQL values in AqlItemRows

* Fixed compile error in hidden merge conflict

* Added tests for Filter counter in FilterExecutor

* OutputAqlItemRow now takes responsibility of the block

* Handed responsibility of blocks in ExecutionBlockImpl to OutputAqlItemRow

* EnumerateListExecutor should be close to its finish. Tests need to be fixed, not ready yet.

* better swap

* OutputAqlItemRow now takes ExecutorInfos

- added several assert
- implemented a more strict and safe produced() logic

* enum list test

* Added a more specific error message

* Removed OutputAqlItemRow.changeRow()

* fix invalid block freeing

* changed order of input/output registers

* Fixed a test

* Fixed matrix width in test

* fix failure tests

* finished enumlist tests

* Implemented AllRowsFetcher and tests for it

* Bugfix in skipSome stub

* insert sort and enumerate list executors

into exectuion nodes' create functions

* fixed singlerowfetcherhelper, added tests

* small fixes

* add missing Infos

* restore accidentally deleted scorer

* make it compile

* fix executor info creation / usage

* make use of output var in EnumarateListNode

* Added Executor::Stats and FilterStats

* Extended FilterExecutor tests with stats, added node tracing to getSome

* Fixed Sort-&EnumerateList- Executor to use the new return type with stats

* Fixed compile error in tests

* Changed *ExecutorInfo constructors and classes to contain the necessary register informations

* Made an ExecutorInfo construction more readable

* Updated comments

* Fixed checks in OutputAqlItemRow::setValue()

* Add comparison method to SortRegister. Forbid construction without one.

* Implemented ExecutionBlockImpl::initializeCursor

* more mocking inside ExecutionBlockImplTest

* Minor changes

* Avoid AqlValueGuard destroying the value copied into the output block

* Disabled failure tests that do not apply to the new Sort implementation

* further mocking of test modules

* Removed constexpr for compatibility with outdated gcc versions

* add some failure tests - getsome works differently

therefor we do not test the old behavior

* fix sort node

* clean up alq item matrix

* Removed the transaction methods from read-only access to AqlValues (get and at) and repliced by CollectionNameResolver where necessary. Still left: some calls extract VPackOptions from the Transaction. Compare needs a BuilderLeaser only.

* textexecutionhelper, execimpl test

* Eliminated FilterBlock

* Eliminated the EnumerateListBlock

* first successful running test of executorblockimpl

* Eliminated SortBlock

* fixed testexecutor helper

* Made the ExecutorInfos private to the executors again.

* Implement and use AqlItemBlockShell (#6562)

* Added AqlItemBlockShell

* Minor changes

* Mostly replaced AqlItemBlock with ~Shell.

Current state:
- arangod compiles
- OutputAqlItemRow has yet to be changed
- catch tests have yet to be fixed

* Use AqlItemBlockShell in OutputAqlItemRow

* Added registersToKeep to AqlItemBlockShell

* Removed ExecutorInfos from OutputAqlItemRow

* Split AqlItemBlockShell into Input~ and Output~

* Changed tests so they compile again - they're still not yet working.

* Fixed tests

* fixed building, modified impl test

* Removed now superfluous BlockFetcherMock functions

* more execimpl testing

* Removed block IDs

* more impl testing

* Reduced dependencies of BlockFetcher construction

* Bugfix, cleared documentation

* Removed debug-output

* Adressed minor code review remarks

* Moved method definitions into .cpp files to avoid includes

* Added assertions

* Added comments

* Addressed several more review comments

* Added missing includes

* more testing, not finalized yet

* added missing header file

* added a test for execution block impl

* removed old comment

* Improved lookup algorithm to identify correct block/row combination in AqlItemMatrix

* added test for ExecutionBlockImpl

* move utility to the right position

* Bugfix and a TODO note

* Bugfix: use a uniform ResourceMonitor in tests

* Made OutputAqlItemRow::setValue clone by default and copy on move

* Replaced the getRow lookup with a binary search

*  enumerate tests

* Disable emplacing of AqlValues in ItemBlock

* Reenabled std::enable_if will warn until oskar is using c++17 as standard.

* Add helper function to create a shared pointer of an unordered_map<RegisterId>

* CalculationExecutor (#6570)

* add CalculationExecutor skeleton

* first shot of CalculationExecutor

* fix minor bugs

* handle references

* use new shared pointers

* commit to share work

* fix failure test

* reformat with clang-format

* add test for CalculationExecutor

* fix CalcualtionExecutor test

* clean up

* Removes unused code

* clean up includes

* fix/remove comments

* add changes requrested by hacki

* ReturnExecutor (#7932)

* add skeleton to CMakeLists.txt

* build most parts of the ReturnExecutor

* This commit removes ReturnBlock and adds some debug messages

removed inheritRegisters functionality

* finish ReturnExecutor

* add test for return executor

* remove unused code

* add stats and clean up includes

* verify result in ReturnExecutorTest and move *Stats to "Aql/Stats.h"

* Fixed algorithm to find correct row in a MultiRow Input

* fix full count and failure tests (#7969)

* fix full count and failure tests

* fix filter test

* WIP: fixes

* fixed

* NoResultsExecutor (#7953)

* add NoResultsExecutor

* clean up

* some review changes

* fix test

* fix

* fix WAIT -> DONE

* remove unused code (#7995)

* create skeleton for SingletonExecutor (#7957)

* create skeleton for SingletonExecutor

* WIP: continue implementation

* WIP: add fetcher skel

* WIP: add inject function to ConstFetcher

* rename SingletonExecutor to IdExecutor

* WIP: replace singleton node

* WIP: More Debugging on IdExecutor

* WIP: IdExecutor

* WIP: add failure tests

* make iresarch test work but not pass

* WIP: add ConstFetcherHelper

* fix test

* fix tests

* fix iresearch tests

* fix wording

* add review changes

* add review change to SingleRowFetcher as well

* fix test

* fix error when increasing index

* temporarily remove fold-expression

* clean up

* Fix gcc compile error

* Feature/aql limit executor (#7982)

* limit executor

* removed not needed tri assert

* rm _invariable in limitnode

* find out if we are in a subquery

* queryDepth in limit infos

* added missing queryDepth

* finalizing limit executor

* added more tests, fixed a condition

* add assert to test

* added missing fullcount check

* typo

* minor changes, fixed test

* test

* wrong check

* fixed tests

* false merge

* remove basic blocks

* remove basic block includes

* include sort order

* removed not needed basic blocks

* removed not needed basic blocks

* fix test

* Feature/remove input output shells (#8090)

* Added template parameters to allow for block pass-through to be chosen at compile time

* Adapted the tests to match the new template parameters

* Adapted ConstFetcher and IdExecutor to match the new templated fetcher

* Implemented pass-through, fixed SingleRowFetcherTest

* Fixed implicit merge conflicts

* Fix assumptions that are violated by block pass-through

* Added 2 regression tests

* Fixed test expected data

* Fix nullptr dereference

* Fix handling of non-invalidated input variables

* Fix nullptr dereference

* Fix handling of non-invalidated input variables

* Fixed compile error

* Fixed a useless and leaking allocation

* Try a less implicit fix

* Allow IdExecutor and ConstFetcher to be used with pass-through

* Performance patches from @mchacki

* Fixed merge conflicts, and bugs in ConstFetcher

* Fixed atMost pass-through, and blocks that call their parents too often

* Removed InputAqlItemBlockShell (except from tests)

* Removed OutputAqlItemBlockShell (except for tests)

* Moved implementations of OutputAqlItemRow methods in the header

* Moved SingleRowFetcher::fetchRow to the header to allow inlining

* Reenabled returnInheritedResults optimization

* Enable passThrough for ReturnExecutor with inherited results

* Specialized CalculationExecutor to avoid unneccessary conditional jumps

* Made tests compile again (but are currently broken)

* Fixed ReturnExecutor test

* Move created InputRows out of AqlItemMatrix

* Limit- and EnumerateList Executors no longer fetch from upstream after DONE

* Added comments

* Added experimental AqlItemBlock optimizations

* Removed _numEntries, added comments

* Allow executors to pass an "atMost"-limit to the SingleRowFetcher (for LIMIT)

* Let the LimitExecutor pass atMost upstream

* Bugfix for FullCount

* Feature/aql enumeratecollection executor (#8028)

* initial commit

* enum coll executor

* buildable version of enumeratecollectionexecutor

* wrong usage of outVariable

* Added template parameters to allow for block pass-through to be chosen at compile time

* Adapted the tests to match the new template parameters

* Adapted ConstFetcher and IdExecutor to match the new templated fetcher

* more fixes, still some tests failing

* Implemented pass-through, fixed SingleRowFetcherTest

* Fixed implicit merge conflicts

* adding missing set value (remember optimize todo), optimized constructor

* removed old enumerate collection blocks

* Fix assumptions that are violated by block pass-through

* Added 2 regression tests

* Fixed test expected data

* Fix nullptr dereference

* Fix handling of non-invalidated input variables

* Fix nullptr dereference

* Fix handling of non-invalidated input variables

* Fixed compile error

* Fixed a useless and leaking allocation

* added broken uncompilable executor test

* Try a less implicit fix

* fix mocking

* enumerate collection executor test

* remove class, use only static function.

* clean up test after merging with main branch

* Allow IdExecutor and ConstFetcher to be used with pass-through

* Performance patches from @mchacki

* Fixed merge conflicts, and bugs in ConstFetcher

* Fixed atMost pass-through, and blocks that call their parents too often

* Removed InputAqlItemBlockShell (except from tests)

* Removed OutputAqlItemBlockShell (except for tests)

* Moved implementations of OutputAqlItemRow methods in the header

* Moved SingleRowFetcher::fetchRow to the header to allow inlining

* Reenabled returnInheritedResults optimization

* Enable passThrough for ReturnExecutor with inherited results

* Specialized CalculationExecutor to avoid unneccessary conditional jumps

* Made tests compile again (but are currently broken)

* Fixed ReturnExecutor test

* Limit- and EnumerateList Executors no longer fetch from upstream after DONE

* Removed an assertion which seems to be incorrect during bootstrap.

* Inlined AQL ItemMatrix and removed unused elements in other Executors

* Now inline + Move in AqlItemMatrix, seems to be overall best performing

* Bugfix: Do not returnInheritedResults on DBServers

* Fixed LIMIT test, and made it more thorough.

* Fixed createion of NoResultBlock creation in IResearch Nodes. Also relaxed an assertion

* Reactivated failure point, now identical with anotherone in OutputRow, need to unified

* Bug fix: do not mutate the offset in the LimitExecutor (#8109)

* Introduced test

* Fixed LimitExecutor to not mutate its infos, so it works correctly in subqueries

* Fixed crashes in CatchTests

* Fixed non-maintainer build

* try to fix subqueries

* Fix isInSubQuery for non-singleton nodes.

* remove useless while true

* Feature/aql blocks by line dan sort (#8143)

* Add SortNode::SorterType to SortExecutorInfoss.

* prepare imple of sort executor

* WIP: add some code for dan sort

* WIP: working at pushRows

* WIP - try to add Dan's heap-sort

* re add missing test

* 2 fixes

* fix

* Dan's tests are working

* remove unused variable

* remove SortBlock.cpp from CMakeLists.txt

* remove hack and provide an extra method to set the _baseIndex

* Simplified ContrainedSortExecutor.

* fix forgotten variable

* update doc

* Assert that we do not use functions that might return wrong results.

Preconditions or invariants the functions rely on are broken. Therefore
the functions can not be used. This is asserted in maintainer mode.

* add function to restore row invariant

* do not emplace and pop_back if not necessary

* Use clear registers in case we can move the rows.

* WIndows opst zirkus

* Actually destroy AqlItemValue

* Feature/traversal executor by line (#8000)

* Added stubs for TraversalExecutor, no implementation, not compiled in yet

* Included production code of TraversalExecutor and made it compile. Now implement

* Implemented production code for trivial tests, if no row is reaching the traverser.

* Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next

* Intermediate blind commit might not work

* Fixed compile issue in test code

* Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work.

* Fixed travesalExecutor tests. Now able to create tests with actual input.

* Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister)

* Further test cases

* Added stubs for TraversalExecutor, no implementation, not compiled in yet

* Included production code of TraversalExecutor and made it compile. Now implement

* Implemented production code for trivial tests, if no row is reaching the traverser.

* Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next

* Intermediate blind commit might not work

* Fixed compile issue in test code

* Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work.

* Fixed travesalExecutor tests. Now able to create tests with actual input.

* Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister)

* Fixed leftovers from rebase

* Refixed mocks in different folder

* Fixed Mock import

* Fixed Calculators Exector Test to use the unified mocked server.

* Implemented logic for the TraversalExecutor. Also added tests for it. Not fully complete though

* Added validation of result values

* Added usage of constant register in traversal executor

* Included filter variables in traersal executor. Added test for const input. And moved MMDR into base traverser class for simplicity

* Added ostream operators on AQLValue and AqlInputRow. Not 100% done yet but good enough

* Added a shutdownable ExecutionBlock. Some blocks need to implement a specific shutdown still. I added this class to make it easier to eliminate them when the time is due.

* Removed ostream operator for INPUT row again, does for some reason not compile on Linux0o

* Fixed initialization order

* Removed classwrapper for shutdown.

* Applied review changes

* Refixed failing tests, API is not subject to change

* Fixed CatchTest (wrong usage of string literal in test case)

* Feature/shortest path executor (#8042)

* Added stubs for TraversalExecutor, no implementation, not compiled in yet

* Included production code of TraversalExecutor and made it compile. Now implement

* Implemented production code for trivial tests, if no row is reaching the traverser.

* Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next

* Intermediate blind commit might not work

* Fixed compile issue in test code

* Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work.

* Fixed travesalExecutor tests. Now able to create tests with actual input.

* Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister)

* Further test cases

* Added stubs for TraversalExecutor, no implementation, not compiled in yet

* Included production code of TraversalExecutor and made it compile. Now implement

* Implemented production code for trivial tests, if no row is reaching the traverser.

* Fixed TraversalExecutor code to pull all rows from above and added tests. No traversal yet, this is next

* Intermediate blind commit might not work

* Fixed compile issue in test code

* Created a shared place for Mocks in tests and moved the StorageEngineMock there, also started a shared place for mock servers, most test will need the same server in order to work.

* Fixed travesalExecutor tests. Now able to create tests with actual input.

* Added successful test in TraversalExecutor, that Traverser as started with correct start vertices (by inputRegister)

* Fixed leftovers from rebase

* Refixed mocks in different folder

* Fixed Mock import

* Fixed Calculators Exector Test to use the unified mocked server.

* Implemented logic for the TraversalExecutor. Also added tests for it. Not fully complete though

* Added validation of result values

* Added usage of constant register in traversal executor

* Included filter variables in traersal executor. Added test for const input. And moved MMDR into base traverser class for simplicity

* Added ostream operators on AQLValue and AqlInputRow. Not 100% done yet but good enough

* Added a shutdownable ExecutionBlock. Some blocks need to implement a specific shutdown still. I added this class to make it easier to eliminate them when the time is due.

* Removed ostream operator for INPUT row again, does for some reason not compile on Linux0o

* Fixed initialization order

* First draft of shortest path executor

* Fixed logic bugs, Local AQL tests succeed

* Added first version of ShortestPathExecutor tests. Edge output tests are missing yet.

* Added EDGE output tests

* Replaced TraversalStats with NoStats, they have never been tracked...

* Removed obsolete ShortestPathBlock.

* Removed classwrapper for shutdown.

* Applied review changes

* Refixed failing tests, API is not subject to change

* windows macro opst circus

* Clear registers per block, not per row (#8164)

* Fixed compilation under linux again...
2019-02-14 19:13:08 +01:00
Alan Plum f6467126a3 Fix isAqlQuery with empty query (#8141) 2019-02-13 08:48:35 +01:00
Dan Larkin-York efb8364d99 Optimize memory usage in AQL by using constrained heap sort when reasonable (#7990) 2019-01-31 22:46:57 +01:00
Jan 6944115de6
various optimizations (#7994) 2019-01-21 13:40:39 +01:00
Tobias Gödderz a1d3bc3e94 Foxx queue jobs hanging after Foxxmaster crash (#7922)
* Fixed bug where the Foxxmaster doesn't reset jobs after a crash when it should, or a non-master coordinator removes jobs in progress during startup

* Added a regression test

* Updated CHANGELOG

* Fixed non-maintainer compile
2019-01-14 16:08:08 +01:00
Andrey Abramov d4a010baf9
Bug fix/internal issue #316 (#7911)
* allow using scorers outside ArangoSearch view context

Signed-off-by: Andrey Abramov <andrey@arangodb.com>

* ensure query is properly optimized after replacement of scorer functions

* do not apply `handleViewsRule` to queries without views

* simplify optimization rule for ArangoSearch views

* show ArangoSearch view scorers in query explanation

* fix tests

* fix tests

* add stub for scorer related tests

* reformat

* check variable depth in `ViewExpressionContext::getVariableValue`

* add some tests

* address js test failures

* address jslint errors

* ensure `IResearchViewNode` exposes variables used in scorers

* ensure scorers with expressions are deduplicated

* fix deduplication for indexed access

* more tests

* partially address review comments

* address review comments

* simplify code

* remove irrelevant, commented out code

* ensure array comparisons are properly handled

* update changelog & loki
2019-01-10 21:04:19 +03:00
Jan fe81007438
support db._explain with all plans (#7895) 2019-01-08 18:06:46 +01:00
Wilfried Goesgens 879c7bb479 slash all left over processes (if) and set status to bad. (#7557) 2018-11-30 16:24:07 +01:00
Alan Plum c36c8c48ee Teach Foxx req.makeAbsolute about unix sockets (#7327)
* Expose rawReq.server.endpoint in JS

* Make rawReq.portType enumerable

This is not strictly necessary but we don't gain much from hiding its existence.

* Allow using http: protocol in local requests

This helps distinguishing local request URLs from file paths in ambiguous contexts.
The alternative would require the user to specify the local endpoint but this doesn't work with
sockets (and for non-sockets would incur the penalty of a real network roundtrip).

* Teach Foxx req.makeAbsolute about unix sockets

* Add Foxx socket tests
2018-11-20 14:20:56 +01:00
Jan 0dd1776467
Make recovery more reliable (#7297) 2018-11-19 13:59:18 +01:00
Markus Pfeiffer 39bdebf851 Port bug-fix-3.4/timeout-create-coll to devel (#7307)
* Fix loophole in error handling.
* Fix inquiry case of id not found: 404.
* Also handle correctly in AgencyComm.
* Fix agency tests.
* Fix error handling in dropCollectionOnCoordinator.
2018-11-14 10:03:55 +01:00
Jan 4631cfb5e8
do not create _modules collection for new databases/installations (#7291) 2018-11-13 11:40:15 +01:00
Dan Larkin-York 0bc6a4d7a2 Fix query profiling discrepancy in console. (#7241) 2018-11-06 14:50:32 +01:00
Jan 4bc646f207
increase the maximum allowed number of collections in an AQL query (#7154) 2018-10-31 17:20:30 +01:00
Jan 9bb460c3d5
do not rely on _modules collection being present at startup (#7118) 2018-10-29 13:54:34 +01:00
Wilfried Goesgens 15a366b17f Bug fix/fix duplicate testcasenames (#7003) 2018-10-23 11:18:22 +02:00
KVS85 3b6191305c Added timestamps to each test for jsUnity (#6926) 2018-10-17 15:32:16 +02:00
Wilfried Goesgens 239771b5f3 Better result handling for recovery tests (#6826) 2018-10-15 13:17:45 +02:00
Jan 6d54518145
forward port changes from 3.4 (#6849) 2018-10-12 17:49:17 +02:00
Jan a0980ed2f2
Bug fix/fixes 111018 (#6843) 2018-10-12 12:51:11 +02:00
Wilfried Goesgens 056ee7064c Feature/log timestamp on test start (#6759) 2018-10-08 16:18:04 +02:00
Jan 46efcff7d7
micro improvements (#6674) 2018-10-05 10:25:13 +02:00
Jan 4b2588d74f
Bug fix/remove valgrind (#6644) 2018-09-28 17:51:53 +02:00
Alan Plum 78425e510c Add support for nested aql queries in JS (#6482)
* Add support for nested aql queries in JS

* updated documentation
2018-09-26 14:21:24 +02:00
Jan ae85fbc4ea
much better diagnostics for AQL query results cache (#6580) 2018-09-25 09:49:11 +02:00
Simon 0fa7f01c66 Resilience test failure points (#6539) 2018-09-20 01:05:10 +02:00
Jan dbe4559ed1
fix some issues with sorted variant of COLLECT (#6433) 2018-09-10 17:01:51 +02:00
Jan 3752c7deab
fix profile output for nodes without calls to getSome (#6397) 2018-09-10 09:19:01 +02:00
Jan 07abfca588
Bug fix/cleanup 020918 (#6338) 2018-09-03 12:56:41 +02:00
Wilfried Goesgens a30a93aa3f fix example, so geo indices are actually used (#6305) 2018-08-30 19:22:13 +02:00
Wilfried Goesgens 72b2909dc9 Doc - use standard geo-json in this example (#6291) 2018-08-30 04:41:57 +02:00
Alan Plum ecfdd8e698 Fix getReadableName (#6272)
* Move getReadableName to new test folder

* More tests

* Fix getReadableName

* Linting
2018-08-29 17:20:52 +02:00
jsteemann 05cbc5a9ec jslint 2018-08-28 15:12:28 +02:00
Jan 494764b95c
Bug fix/fiddle version string into test paths (#6267) 2018-08-28 14:58:35 +02:00
Alan Plum cfa0e0db55 Better foxx errors (#6264)
* Fix codeFrame formatting

Pushing the codeFrame first results in ugly line breaks or extra spaces.

* Include cause in MODULE_FAILURE message

The error is always a wrapper for an underlying cause so it should be easier to determine what caused it.

* Add a codeFrame for failed _install

We normally only want this in dev mode but service CRUD should always provide detailed info.

* Expose codeFrame in Foxx CRUD

The codeFrame can only come from a failed _install, so let's make it visible to Foxx CLI and friends.

* Add codeFrame to runScript

Scripts can also only be executed with access to the Foxx API so we can provide the same info as during CRUD.

* Trim codeFrame fileName to basePath

If we have a basePath, we don't need to expose the full path to the user for the codeFrame to be meaningful.

* Fix getReadableName

Previously 'catch-fire' would become 'Catch  Ire'.

* Add to changelog
2018-08-28 14:36:33 +02:00
Jan 0a73e64b0b Bug fix/fiddle version string into test paths (#6266)
* fiddle version string into test paths
2018-08-28 09:27:32 +02:00
Jan 5022ccc24d
Bug fix/fixes 2508 (#6254) 2018-08-27 21:36:39 +02:00
KVS85 e41a854f59 Move testInTokensFilterSortTFIDF test to nondetermenistic module. (#6246)
* Move testInTokensFilterSortTFIDF test to nondetermenistic module.

* Lost file.

* Rename to conform with other named test modules.

* Resolve conflicts
2018-08-27 15:50:28 +03:00
Andrey Abramov 615e1cf7de
Feature/add options to view (#6255)
* added OPTIONS for views

* pass options too

* validation for OPTIONS

* fix OPTIONS validation

* removed debug log message

* add forceSync options

* add forceSync view option

* fix comments

* fix some memleaks

* fix typo

* fix integration tests

* fix integration tests

* fix another integration test

* rename 'forceSync' option to 'waitForSync'

* another attempt to fix test

* ensure query caches are properly invalidated

* address integration test failures

* another attempt to fix test
2018-08-26 22:30:23 +03:00
Jan 0d48677619
Bug fix/aql optimizer improvements 2308 (#6241) 2018-08-24 13:42:22 +02:00
Kaveh Vahedipour 28754cbf15 Feature/schmutz plus plus (#5972)
- Schmutz now called "Maintenance" and completely implemented in C++
 - Fix index locking bug in mmfiles
 - Fix a bug in mmfiles with silent option and repsert
 - Slightly increase supervision okperiod and graceperiod
2018-08-24 12:15:35 +02:00
Vasiliy e04b5318b4 issue 450.5: rename consolidation-related configuration properties and remove locale (#6232) 2018-08-23 16:04:25 +03:00
Vasiliy febd445410 issue 451.1: ensure errors are properly propagated during LogicalView creation (#6218)
* issue 451.1: ensure errors are properly propagated during LogicalView creation

* address Ruby test failures

* ensure invalid view type returns TRI_ERROR_BAD_PARAMETER

* address jslint errors
2018-08-22 16:11:24 +03:00
jsteemann 9959218d1d revert unification of collection name validation 2018-08-21 18:46:13 +02:00
Vasiliy 325e61b4ed issue 450.3: use a single configuration policy differentiated by 'type' with arbitrary number of configurable parameters (#6205)
* issue 450.3: use a single configuration policy differentiated by 'type' with arbitrary number of configurable parameters

* ensure test data is persisted to the test directory

* add missed declaration

* fix typo

* add full namespace and includes
2018-08-21 14:46:28 +03:00
Jan 2aa879a53b
unify validation of collection names in AQL (#6203) 2018-08-21 11:11:15 +02:00
Simon fc204a2b3e fix profiling code (#6190) 2018-08-20 12:59:07 +02:00
Jan 7cb1da12cf
make AQL functions construction easier (#6195) 2018-08-20 11:56:38 +02:00
Vasiliy c08ab01f67 issue 450.1: remove the "commit" configuration layer from ArangoSearch properties (#6185)
* issue 450.1: remove the "commit" configuration layer from ArangoSearch properties

* fix broken tests
2018-08-20 02:18:02 +03:00
Jan 99f176bed5 Feature/add search keyword (#6166)
* added SEARCH keyword

* fixes jslint errors

* add incompatibility notice

* remove VIEW keyword (does not work yet!!)

* add initial support for SEARCH keyword to optimizer rules

* replace FILTER with SEARCH keyword in tests

* removed VIEW keyword, make shell_server_aql tests pass again in single server

* cleanup

* handle SEARCH clause for views

* make SEARCH a non-keyword

* fixed an issue with duplicate variables

* fixed 3 tests

* fix SEARCH statement support for views

* minor refactoring

* fix integration tests

* optimize SEARCH conditions a bit more

* fix jslint error

* fixed wrong comments and typo in class name

* fix documentation

* adjust recovery tests to use SEARCH instead of FILTER

* fix another failing test
2018-08-18 20:22:58 +03:00
jsteemann d019989d7c fixed jslint warning 2018-08-17 15:09:23 +02:00
jsteemann 6cbe24f7c5 added generated file 2018-08-17 13:14:19 +02:00
jsteemann aa68414207 fix jslint warnings 2018-08-17 11:05:58 +02:00
Alan Plum 1698687f4f Add uuidv4 & genRandomBytes methods (#5732) 2018-08-17 11:04:43 +02:00
Alan Plum 1f69cb6d9d Fix Foxx CRUD errors (#5876)
* Fix off-by-2 error for SyntaxError line nums

* Actually log Foxx HTTP API errors

* Fix line wrap when logging error stacks
2018-08-17 10:09:12 +02:00
Jan d7e631eddf
Bug fix/more sparsung (#6177) 2018-08-17 08:47:45 +02:00
Jan d78adb7ae1
Bug fix/turn off range deletes in cluster (#6178) 2018-08-17 08:41:00 +02:00
Heiko a4683fa118 added lost gharial tests (#6173) 2018-08-16 18:40:03 +02:00
jsteemann c6de98882c jslint 2018-08-16 09:26:44 +02:00
Jan 2bc672cebd
centralized more of the sharding strategies code (#6140) 2018-08-15 14:37:01 +02:00
Alan Plum dc2bca23cf Fix #5736 (#5844)
* Fix issue 5736

* Add #5736 to CHANGELOG
2018-08-15 14:11:20 +02:00
KVS85 c5cb2fa10a Bug fix/fix arangosearch authentication tests for cluster (#6161)
* Initial changes in assertions

* Fix typo

* Fix for issues #452, #453

* Fix drop task test

* Deal with fuetre possible bug on connection

* More fixes

* Enable shell_*_aql tests for RF > 1
2018-08-15 12:13:15 +03:00
Dan Larkin-York 5f87f57cd0 Improved sharding algorithms (#6089) 2018-08-09 19:03:32 +02:00
jsteemann 1e466f6545 fix jslint warnings 2018-08-09 11:39:21 +02:00
Jan 9abf1b1562
make tests non-timecritical (#6129) 2018-08-09 11:28:29 +02:00
Tobias Gödderz de4f5587ae Gharial rewrite in C++ (#5631)
* Built a C++ skeleton REST handler for gharial, with fallback to the JS handler

* Moved aql::Graph to graph::Graph

* Added complete edge definitions to Graph

Also:
- some cleanup
- used forward-declarations in headers
- use Graph in graph rest handler

* Handle graph lookup failures according to the test suite

* Added GET vertex

* Bugfixes in ResultT

- Added missing #include
- Fixed move semantics

* Move central code of readVertex to GraphOperations

* ResultT fixes and complements

* Implemented a graph cache

* Added and used graph cache to the rest handler

* Added GET edge

* Added DELETE edge

* Extracted some code

* Added PATCH and PUT for both edge and vertex

* Moved update/replace transaction code to GraphOperations

* Added stub routes for POST and a TODO note

* Added a test checking that deleting a vertex removes all incident edges as well

* Added a test checking that deleting a vertex does not remove edges in non-graph collections

* fixed compiler warnings and errors

* Began work on DELETE vertex

For this, added a V8Context to allow for AQL queries to use subtransactions

* Continued work on DELETE vertex (still WIP)

* prep for graph post routes

* fixed removeVertex operation (aql)

* added post vertex and post edge gharial routes

* wasSynchronoues flag changed

* gharial post c++ handler, naming conventions

* added gharial tests

* temporary disabled cache (because not completed), added graph property read functions

* added c++ gharial list vertex collections

* added c++ gharial graph config

* added c++ gharial list graphs

* added graph manager class

* first implementation of create graph in c++, WIP

* changed error messages

* added etag to create graph api, still multiple edge definition check missing

* finished POST /_api/gharial/<graph>

* WIP - DELETE /_api/gharial/<graphName>

* added DELETE /_api/gharial/<graphName> validation, still missing correct response

* gharial delete

* fixed delete gharial lock

* finished DELETE /_api/gharial/<graphName>

* added routes for graph based vertices and edge definitions

* improved delete route

* added add new edge definition to existing graph

* patch edge definition in a graph, still <WIP>

* finished edit edge definition route

* code changes due to devel code changes

* added remove edge definition route

* added vertex delete function

* added todo note regarding one drop collection issue

* add oprhan collection to graph route implemented

* Added a test

* Updated a comment

* Several minor changes

* Minor changes during review

* Changes during review

* Changes during review

* Bugfix: orphans may be null or omitted

* Bugfix: resolve externals

* minor code changes

* seperated graph class to independent component classes

* seperated graph class to independent component classes

* removed log output

* fixed create collection behaviour in a cluster environment

* fixed enterprise graphs behaviour in c++ gharial api

* removed log output

* formatting

* improved error handling, fixed a linux compile bug

* more result refactoring

* more result type cleanup

* fixed wrongly defined test

* result handling

* error handling

* more refactoring

* Bugfix: avoid race condition in cluster when creating collections

* updated graph documentation

* added graph related static strings

* static strings, new method to create options for gharial created collections

* Some minor cleanup

* more use of static strings

* minor code changes, review

* added missing parseint

* removed gharial foxx, added js common module, added v8 general graph module

* correct use of virtual method

* more v8, js general graph, broken state

* more v8 graph functions

* fixed editEdgeDefinition, added drop function

* fixed drop behaviour

* added _list, _exists

* added c++ rename graph collections, added v8 + graph module function

* Added a regression test

* added graph._deleteEdgeDefinition, v8, server

* more v8g

* added _removeVertexCollection

* added _extendEditDefinitions

* todo, need to add a helper sort method for a local defined relation

* fixed test

* fixed lots of tests, added more client functions, _addVertexCollection on client module is still broken

* added more client graph functions, all tests green

* more client functions

* add del edge def route

* Fix use after move

* Minor changes in client general-graph.js module

* Make a copy before sorting (don't touch the argument)

* Minor changes and some additional asserts in graph tests

* Consistently set parameter defaults

* Renamed static strings

* Remove superfluous function

* Made comment more verbose

* Minor changes in general-graph-common.js

* Added missing template arguments

* Fixed community build

* Cleanup in editEdgeDefinition

* Regression test & bugfix: comparison of edge definitions didn' order from and to

* Fixed errors introduced by merge

* Minor changes in v8-general-graph.cpp

* Fixed test failure due to wrong error code in CE

* added missing id field

* Added permission checks for graph._create

* Removed assertion that is no longer valid

* Moved removeGraph from GraphOperations to GraphManager

* Allow C++ implementation of graph._drop to handle smart graphs

* Flush js client db cache after creating/dropping collections via the general graph module

* Added _deleteEdgeDefinition to the general graph client module

* WIP: Added permission checks for drop graph

* Fixed permission checks for drop graph

* Added permission checks for other graph operations

* Bugfix: assert edge definitions are returned in order

* Some cleanup

* Removed unused method

* Minor improvements in GraphManager

* Fixed a type in general-graph common module

* Most useful fix of all times ever: Do not auto cast from bool to int and alternate error/noerror by this

* Added the initial keyword to StaticStrings

* Added a new error code, used whenever a user tries to inject a documentcollection as a relation into the graph, which is invalid

* Some GraphManager/Ops/Graph cleanup. Less Slice parsing, more usage of GraphObjects

* Test edgeDefinitions in graphs with a defined ordering

* GraphClass Layout cleanup

* Do not test error messages, use codes instead

* Recreated backwards compatibility of Graph Creation Permission errors

* Changed error-code if edgedefinition is used twice

* Added a StaticString for the GraphName

* Renamed graphToVpack => graphForClient

* Partly fixed graph-api test to work with better error messages. Still red: The edgeDefinitions are now sorted, the test is supposed to sort his own list, but appearently does not do so. Under investigation

* Added a new error code that rejects injection of differently sharded smart collection into smartgraph. Should be more helpful to our users

* graph createCollectionOptions now require an open object to be cross-called from enterprise. Made enterprise switch for creation of graph more elegant.

* Updated graphs.cpp

* Massive refactoring. Made Factories for graphs to make SmartGraph much more transparent. Also reduced amount of multiple implementations of the same stuff. Killed vocbase/graphs use GraphManager instead. Removed usage of GraphCache, was not completely implemented anyway and only partially used, which is bad at the moment. Option for later improvement never the less

* Adapted JS code to now really use c++ variants. ALso included 3 Classes: Graph, SmartGraph and GraphModule.

* Fixed undefined behaviour in Remove Vertex. Fixed smartgraph sharding if one collection already exists.

* Removed DEBUG output

* Removed DEBUG logs

* Removed dead code

* Fixed Graph EdgeDefinition test, they now have a different ordering.

* Added a test when adding a vertexCollection that it is actually valid in the graph

* Client Graph API now correctly sends `orphanCollections` and not `orphans`

* Let GraphOperations modify the graph in-place. It should now properly handle edgeDefinitions.

* Added initial cid StaticString

* Included the vocbase in fromPersistence creation of Graphs. Only required to enhance 3.3 SmartGraphs on the fly.

* Fixed internal error message

* Fixed compiler isses originiated from merging

* Removed unused imports

* Regenerated generated file
2018-08-09 09:30:04 +02:00
Jan 93222b15d4
track last used keys in cluster key generators, track key on cluster document insert (#6101) 2018-08-08 14:32:16 +02:00
jsteemann d448b4367b adjusted test 2018-08-07 11:53:29 +02:00
Kaveh Vahedipour fd60b359b6 fixed parallel creation of indexes in cluster (#6088)
* fixed parallel creation of indexes in cluster

* added tests
2018-08-07 10:00:15 +02:00
Jan 42fdef45d4
Bug fix/fix profile query (#6073) 2018-08-06 17:02:27 +02:00
Simon 4238cff3ff Adding cacheUsage property and tests (#6075)
* Adding cacheUsage property and tests

* Make cacheUsage more permissible
2018-08-06 14:55:20 +02:00
Jan e3ce2e8cfb
fix anonymization with unique indexes (#6068) 2018-08-03 12:17:05 +02:00
Heiko 434cfce699 Feature/query profiling ui (#6027) 2018-08-03 10:58:22 +02:00
jsteemann 7d83f20d59 extended tests 2018-08-02 09:56:19 +02:00