1
0
Fork 0
Commit Graph

5808 Commits

Author SHA1 Message Date
Simon 392118bd62 Use RangeDelete where possible (#6121) 2018-08-15 18:52:09 +02:00
Shivam Dave 9b926dc2fd Resolved regex matches integration tests 2018-08-09 16:33:19 -07:00
Shivam Dave 1360cfc3c3 Resolved regex matches integration tests and documentation 2018-08-09 15:09:47 -07:00
Jan 8267c61ec2
Merge branch 'devel' into feature/add-aql-function-regexmatches 2018-08-09 20:28:06 +02:00
Shivam Dave 47bea99bbe Regex matches implementation integration tests and documentation 2018-08-09 10:17:32 -07:00
Dan Larkin-York 5f87f57cd0 Improved sharding algorithms (#6089) 2018-08-09 19:03:32 +02:00
Jan 95d1892f18
remove canThrow attribute from AQL functions (#6123)
Its was somehow overlapping with the isDeterministic attribute (but with the opposite meaning)
2018-08-09 16:45:54 +02:00
Jan bf3ce0b704
fixed issue #6106: Arithmetic operator type casting documentation inc… (#6113) 2018-08-09 11:30:54 +02:00
Tobias Gödderz de4f5587ae Gharial rewrite in C++ (#5631)
* Built a C++ skeleton REST handler for gharial, with fallback to the JS handler

* Moved aql::Graph to graph::Graph

* Added complete edge definitions to Graph

Also:
- some cleanup
- used forward-declarations in headers
- use Graph in graph rest handler

* Handle graph lookup failures according to the test suite

* Added GET vertex

* Bugfixes in ResultT

- Added missing #include
- Fixed move semantics

* Move central code of readVertex to GraphOperations

* ResultT fixes and complements

* Implemented a graph cache

* Added and used graph cache to the rest handler

* Added GET edge

* Added DELETE edge

* Extracted some code

* Added PATCH and PUT for both edge and vertex

* Moved update/replace transaction code to GraphOperations

* Added stub routes for POST and a TODO note

* Added a test checking that deleting a vertex removes all incident edges as well

* Added a test checking that deleting a vertex does not remove edges in non-graph collections

* fixed compiler warnings and errors

* Began work on DELETE vertex

For this, added a V8Context to allow for AQL queries to use subtransactions

* Continued work on DELETE vertex (still WIP)

* prep for graph post routes

* fixed removeVertex operation (aql)

* added post vertex and post edge gharial routes

* wasSynchronoues flag changed

* gharial post c++ handler, naming conventions

* added gharial tests

* temporary disabled cache (because not completed), added graph property read functions

* added c++ gharial list vertex collections

* added c++ gharial graph config

* added c++ gharial list graphs

* added graph manager class

* first implementation of create graph in c++, WIP

* changed error messages

* added etag to create graph api, still multiple edge definition check missing

* finished POST /_api/gharial/<graph>

* WIP - DELETE /_api/gharial/<graphName>

* added DELETE /_api/gharial/<graphName> validation, still missing correct response

* gharial delete

* fixed delete gharial lock

* finished DELETE /_api/gharial/<graphName>

* added routes for graph based vertices and edge definitions

* improved delete route

* added add new edge definition to existing graph

* patch edge definition in a graph, still <WIP>

* finished edit edge definition route

* code changes due to devel code changes

* added remove edge definition route

* added vertex delete function

* added todo note regarding one drop collection issue

* add oprhan collection to graph route implemented

* Added a test

* Updated a comment

* Several minor changes

* Minor changes during review

* Changes during review

* Changes during review

* Bugfix: orphans may be null or omitted

* Bugfix: resolve externals

* minor code changes

* seperated graph class to independent component classes

* seperated graph class to independent component classes

* removed log output

* fixed create collection behaviour in a cluster environment

* fixed enterprise graphs behaviour in c++ gharial api

* removed log output

* formatting

* improved error handling, fixed a linux compile bug

* more result refactoring

* more result type cleanup

* fixed wrongly defined test

* result handling

* error handling

* more refactoring

* Bugfix: avoid race condition in cluster when creating collections

* updated graph documentation

* added graph related static strings

* static strings, new method to create options for gharial created collections

* Some minor cleanup

* more use of static strings

* minor code changes, review

* added missing parseint

* removed gharial foxx, added js common module, added v8 general graph module

* correct use of virtual method

* more v8, js general graph, broken state

* more v8 graph functions

* fixed editEdgeDefinition, added drop function

* fixed drop behaviour

* added _list, _exists

* added c++ rename graph collections, added v8 + graph module function

* Added a regression test

* added graph._deleteEdgeDefinition, v8, server

* more v8g

* added _removeVertexCollection

* added _extendEditDefinitions

* todo, need to add a helper sort method for a local defined relation

* fixed test

* fixed lots of tests, added more client functions, _addVertexCollection on client module is still broken

* added more client graph functions, all tests green

* more client functions

* add del edge def route

* Fix use after move

* Minor changes in client general-graph.js module

* Make a copy before sorting (don't touch the argument)

* Minor changes and some additional asserts in graph tests

* Consistently set parameter defaults

* Renamed static strings

* Remove superfluous function

* Made comment more verbose

* Minor changes in general-graph-common.js

* Added missing template arguments

* Fixed community build

* Cleanup in editEdgeDefinition

* Regression test & bugfix: comparison of edge definitions didn' order from and to

* Fixed errors introduced by merge

* Minor changes in v8-general-graph.cpp

* Fixed test failure due to wrong error code in CE

* added missing id field

* Added permission checks for graph._create

* Removed assertion that is no longer valid

* Moved removeGraph from GraphOperations to GraphManager

* Allow C++ implementation of graph._drop to handle smart graphs

* Flush js client db cache after creating/dropping collections via the general graph module

* Added _deleteEdgeDefinition to the general graph client module

* WIP: Added permission checks for drop graph

* Fixed permission checks for drop graph

* Added permission checks for other graph operations

* Bugfix: assert edge definitions are returned in order

* Some cleanup

* Removed unused method

* Minor improvements in GraphManager

* Fixed a type in general-graph common module

* Most useful fix of all times ever: Do not auto cast from bool to int and alternate error/noerror by this

* Added the initial keyword to StaticStrings

* Added a new error code, used whenever a user tries to inject a documentcollection as a relation into the graph, which is invalid

* Some GraphManager/Ops/Graph cleanup. Less Slice parsing, more usage of GraphObjects

* Test edgeDefinitions in graphs with a defined ordering

* GraphClass Layout cleanup

* Do not test error messages, use codes instead

* Recreated backwards compatibility of Graph Creation Permission errors

* Changed error-code if edgedefinition is used twice

* Added a StaticString for the GraphName

* Renamed graphToVpack => graphForClient

* Partly fixed graph-api test to work with better error messages. Still red: The edgeDefinitions are now sorted, the test is supposed to sort his own list, but appearently does not do so. Under investigation

* Added a new error code that rejects injection of differently sharded smart collection into smartgraph. Should be more helpful to our users

* graph createCollectionOptions now require an open object to be cross-called from enterprise. Made enterprise switch for creation of graph more elegant.

* Updated graphs.cpp

* Massive refactoring. Made Factories for graphs to make SmartGraph much more transparent. Also reduced amount of multiple implementations of the same stuff. Killed vocbase/graphs use GraphManager instead. Removed usage of GraphCache, was not completely implemented anyway and only partially used, which is bad at the moment. Option for later improvement never the less

* Adapted JS code to now really use c++ variants. ALso included 3 Classes: Graph, SmartGraph and GraphModule.

* Fixed undefined behaviour in Remove Vertex. Fixed smartgraph sharding if one collection already exists.

* Removed DEBUG output

* Removed DEBUG logs

* Removed dead code

* Fixed Graph EdgeDefinition test, they now have a different ordering.

* Added a test when adding a vertexCollection that it is actually valid in the graph

* Client Graph API now correctly sends `orphanCollections` and not `orphans`

* Let GraphOperations modify the graph in-place. It should now properly handle edgeDefinitions.

* Added initial cid StaticString

* Included the vocbase in fromPersistence creation of Graphs. Only required to enhance 3.3 SmartGraphs on the fly.

* Fixed internal error message

* Fixed compiler isses originiated from merging

* Removed unused imports

* Regenerated generated file
2018-08-09 09:30:04 +02:00
jsteemann 2172eaf419 fix jslint warnings 2018-08-06 10:32:24 +02:00
Andrey Abramov b8e7752787 Feature/arangosearch stabilization (#6069)
* Added authentication tests for ArangoSearch views creation (client and task)

* Added authentication tests for ArangoSearch views querying (client and task)

* Added authentication tests for ArangoSearch views update (client and task) and correct other tests

* Added authentication tests for ArangoSearch views drop (client and task) and correct other tests

* Fix tests to support additional 'properties' layer of a view

* Add data to dumped/restored collection with a link from a view to single server test and view dump/restore to cluster tests

* backport from iresearch upstream: use a bitvector instead of an acceptor lambda for consolidation policies

* ensure view access permissions are checked for REST calls

* ensure view access permissions are checked for V8 calls

* do not allow LogicalView property modification if reading the properties fails

* Fixed authentication tests for ArangoSearch views to pass regardless of errorNum and possible multiple REST codes

* add missing ldap feature for enterprise version

* do not store useless data in ArangoSearch index (see #289, #441)
2018-08-05 20:47:22 +02:00
Jan b278d6874a
allow master & slave to work in parallel for RocksDB WAL tailing (#6059) 2018-08-03 13:37:53 +02:00
Jan Christoph Uhde a60e04b251 fix timeout issue next try: no subquery nodes on dbservers (#6037) 2018-08-03 09:43:11 +02:00
Simon 42cabb858a Fix dumping of views in the cluster (#6024) 2018-08-02 17:25:49 +02:00
jsteemann 7d83f20d59 extended tests 2018-08-02 09:56:19 +02:00
jsteemann 44d7a11bd9 Merge branch 'bug-fix/improve-intermediate-commits' of https://github.com/arangodb/arangodb into devel 2018-08-01 17:49:17 +02:00
shivamdave24 d4c6431ee5 Regex Split Implementation (unfinished) (#5991) 2018-08-01 12:09:36 +02:00
Jan 7f01b7f728
fixed issue #6031: Broken LIMIT in nested list iterations (#6033) 2018-08-01 08:32:00 +02:00
Simon Grätzer 8eeff44f30 adjusting documentation 2018-07-31 15:33:28 +02:00
Simon Grätzer a7c78a3f37 Merge branch 'devel' of github.com:arangodb/arangodb into bug-fix/improve-intermediate-commits
# Conflicts:
#	arangod/RocksDBEngine/RocksDBIncrementalSync.cpp
#	arangod/RocksDBEngine/RocksDBIterators.cpp
2018-07-31 15:19:57 +02:00
jsteemann 0e9e700656 fix test 2018-07-30 19:27:00 +02:00
Simon Grätzer 6b776e7e9e Merge branch 'devel' of github.com:arangodb/arangodb into bug-fix/improve-intermediate-commits 2018-07-30 14:53:39 +02:00
jsteemann cda104475f fix issue in community build 2018-07-30 14:15:51 +02:00
KVS85 0693df6e32 Fix replicationFuzzGlobal tests (#6021) 2018-07-30 12:11:33 +02:00
Simon Grätzer 20d7e8b2af Merge branch 'devel' of github.com:arangodb/arangodb into bug-fix/improve-intermediate-commits 2018-07-30 11:43:48 +02:00
jsteemann cfab312d81 remove explain output from test 2018-07-29 20:21:43 +02:00
Simon Grätzer d96b25580f fix tests 2018-07-28 17:26:28 +02:00
Simon Grätzer 6abb62165b fix more tests 2018-07-28 17:25:11 +02:00
Simon Grätzer 03f8ec790b fixes some tests 2018-07-28 17:04:07 +02:00
Simon Grätzer c7aacea583 fix test 2018-07-28 15:56:06 +02:00
Simon Grätzer 683dd0a4c6 fixing tests 2018-07-27 21:34:12 +02:00
Vasiliy 11ba71ec27 issue 432.1: do not modify IResearchView 'locale' encoding and make it non-updatable (#6014)
* issue 432.1: do not modify IResearchView 'locale' encoding and make it non-updatable

* add a temporary workaround for MacOS build issue
2018-07-27 15:08:23 +03:00
Alan Plum 046a90e8a3 Add pbkdf2 to Foxx auth module (#5667)
* Add pbkdf2 to Foxx auth module

* Add to CHANGELOG
2018-07-27 11:34:52 +02:00
jsteemann eba7c7d360 fix test 2018-07-26 13:39:26 +02:00
jsteemann b75d4cc1f3 fix function name 2018-07-26 11:52:52 +02:00
Jan Christoph Uhde 6210b9be34 bug-fix/double-modification-bug (#5979) 2018-07-26 10:35:42 +02:00
Jan 562372ee8c
fix single-shard optimization checks (only affects devel) (#5985) 2018-07-26 10:32:10 +02:00
Simon 2dd8593609 View Replication (#5915) 2018-07-26 10:28:46 +02:00
Jan 213c42d2f6
fixed issue #5977: Unexpected execution plan when subquery contains COLLECT (#5984) 2018-07-25 17:33:55 +02:00
shivamdave24 7db28da251 Levenshtein Distance function implementation with documentation and i… (#5922) 2018-07-25 12:48:58 +02:00
Tobias Gödderz aa31cc3f1e Bugfix: Fix `_rev` handling in `UPDATE`/`REPLACE` `WITH` clauses (#5966) 2018-07-25 12:13:09 +02:00
Michael Hackstein 2d5e8b5dcd Bug fix/non deterministic test (#5968) 2018-07-25 09:41:41 +02:00
Jan Christoph Uhde 4c0f881764 compatibilty 3.3 <-> 3.4 (#5963) 2018-07-25 09:05:34 +02:00
Vasiliy fae47715eb issue 410.3: remove the extra neting of ArangoSearch view definition into 'properties' (#5956)
* issue 410.3: remove the extra neting of ArangoSearch view definition into 'properties'

* address merge issue

* backport: address test failures

* backport: address dump test failure
2018-07-24 16:27:28 +03:00
Tobias Gödderz 2c77483654 Fix test: increase plan version after changing the plan (#5951) 2018-07-24 09:14:54 +02:00
Vasiliy 6edebf0384 issue 410.2: ensure documentation matches implementation (#5939)
* issue 410.2: ensure documentation matches implementation

* minor documentation update
2018-07-21 01:22:05 +03:00
jsteemann 188d635954 remove useless printing in tests 2018-07-19 17:41:24 +02:00
jsteemann 2138dc0479 Merge branch 'bug-fix/fixes-1707' of https://github.com/arangodb/arangodb into devel 2018-07-19 16:02:30 +02:00
Vasiliy 3c384bf3e8 issue 410.1: ensure both JavaScript and Rest API for views matches API for collections (#5903)
* issue 410.1: ensure both JavaScript and Rest API for views matches API for collections
* backport: address some FIXMEs
* backport: address test failures
* backport: address more test failures
* backport: fix some Ruby tests
* temporarily comment out broken Ruby tests
* backport: fix more broken Ruby tests
* temporarily comment out failing Ruby tests
* backport: match result structure for View instances with Collection instances
* backport: return different result structure for view listing and view definition
* backport: fix even more broken Ruby tests
* backport: do not depend on unordered_set stable order
* update documentation
2018-07-19 15:07:17 +02:00
jsteemann 21980cfc52 Merge branch 'devel' of https://github.com/arangodb/arangodb into bug-fix/fixes-1707 2018-07-17 22:13:08 +02:00
jsteemann 8a6650f1f6 updated documentation 2018-07-17 22:03:13 +02:00
Vasiliy a401934d62 issue 399.3: remove internal persisted proerty 'collections' from externally generated IResearchView jSON (#5874)
* issue 399.3: remove internal persisted proerty 'collections' from externally generated IResearchView jSON

* backport: address test failure

* backport: address another test failure
2018-07-17 15:21:07 +03:00
Jan 2710b19487
properly report out-of-range coordinates in geo index queries (#5880) 2018-07-13 21:26:42 +02:00
jsteemann ff626314df turn off profiling in non-profiling test 2018-07-13 17:56:57 +02:00
shivamdave24 c45a1b6ac1 Soundex function implementation with integration tests and minor bug fixes to feature/add-aql-tobase64-tohex-encodeuricomponent-uuid branch (#5851) 2018-07-13 17:47:25 +02:00
Simran 34ec56d421 Feature/misc spelling corrections (#5164) 2018-07-13 13:06:20 +02:00
jsteemann 2d795c1f8a fix some undefined behavior 2018-07-12 21:54:51 +02:00
Jan Christoph Uhde bcf9e30a4d Feature/cluster single operations (#5585) 2018-07-12 20:18:36 +02:00
Wilfried Goesgens 5e296cb0c9 add ability to filter jsunity testsuites by testcase (#5807) 2018-07-12 13:35:29 +02:00
Vasiliy 64666d0edd issue 400.2.4: register IResearchView sync tasks with IResearchFeature, remove unused configuration parameter (#5813)
* issue 400.2.4: register IResearchView sync tasks with IResearchFeature, remove unused configuration parameter

* set flag before notify, fix failures in new tests from 'devel'

* revert set flag before notify, fix more failures in new tests from 'devel'

* fix even more failures in new tests from 'devel', backport: do not reset notification flag if pending tasks were seen

* fix still more failures in new tests from 'devel'

* try to work around buggy test failures
2018-07-10 18:15:01 +03:00
jsteemann 2aa17551f3 fix non-deterministic "excludesNull" flags 2018-07-10 13:38:07 +02:00
Jan Christoph Uhde 1e3e3623da Bug fix/always distribute restrict to single shard later (#5791) 2018-07-10 11:11:26 +02:00
Tobias Gödderz fc3e11dbbc Async AQL (#5806)
* Modified header to new initializeCursor API

* Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet

* Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later

* First attempt to simulate thread waiting over information within the query

* Small fix to allow for isDirect handlers to go to sleep.

* Waiting in the necessary places now for the async request to be send.

* Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here?

* Include cond-var header

* Fixed mutex/cond_var usage

* Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested

* Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries.

* Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts

* Fixed endless virtual recursion

* Implemented new getOrSkip API in SortBlock

* Implemented new getOrSkip API in LimitBlock

* Initilaize all variables

* Fixed logic bug in SubqueryBlock

* getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly!

* Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime.

* Added IndexBlock::skipSome and IndexBlock::getSome

* getBlock now returns its old return value along with the state

* Switch from getBlockOld to getBlock in IndexBlock::skipSome

* Switch from getBlockOld to getBlock in IndexBlock::getSome

* ShortestPathBlock::skipSome is not implemented! Added a regression test

* Attempt to fix SubQueryResult memory management

* Fixed LIMIT Block

* Moved from ShortestPathBlock::getSomeOld to ::getSome

* Implemented ASYNC api on SingletonBlock

* Adapted EnumerateCollectionBlock to new async API

* Fixed FilterBlock and adapted return block to async API

* Adapted NORESULTS block to async AQL api.

* Adapted Modification Blocks to async API

* Fixed some initialize cursor functions to reset values required during get/skipSome

* First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still

* Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place.

* Adapted CalculationBlock to async API

* Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN

* Adapted EnumerateListBlock to Async AQL api

* Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now

* Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing

* Adapted IReasearchViewBlock to ASYNC AQL API

* Fixed SortingGatherBlock in WAITING state.

* Adapted IResearch ExecutionBlockMock to Async API

* Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍

* Implemented tonly heoretically reachable function of non void function.

* Fixed last commit

* Added inline TODO comments

* fix warning

* Fixed a clearing logic bug in RemoveNodes

* Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter.

* remove unused methods

* Fixed failure test

* implement skipping

* Moved the Query Waiting out of the ExecutionEngine.

* changed one of the collect blocks

* Removed _upstreamState from ExecutionBlockMock, that is in the base-class now

* Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api.

* do not check "hasMore" if not necessary

* Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type

(still uses getBlockOld)

* Save state to resume in DistinctCollectBlock::getOrSkipSome

* Extracted redundant code

* fixed some ops

* added one more test

* fix endless blocking

* fix compile error

* fix test

* Refactored HashedCollectBlock::getOrSkipSome

* Return blocks to the manager

* Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome

* remove unused shutdown calls, simplify ownership for expressions

* Removed superfluous variable

* Capture const variable by value

* Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome

* Added a working version of SortedCollectBlock::getOrSkipSome

Has yet to be cleaned up

* Removed isTotalAggregation special treatment

* On no input, return a group of nulls (instead of no group at all)

* Bugfixes

* Simplified code

* Move return to the end, eliminate duplicate code

* Corrected skipped count in HashedCollectBlock

* Aligned getNextRow() implementations

* Added comments

* some cleanup

* fix potential memleak

* Bugfix

* Fixed failure tests

* Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome

* Replaced hasMore with an async implementation (mostly)

* Removed getBlockOld()

* Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState)

* RemoteBlock now uses the async hasMoreState route

* remove job queue

* options

* Bugfixes in the async implementation of LimitBlock

* LimitBlock::getOrSkipSome now always skips when calculating the fullcount

* fix compile warnings

* restrict threads

* Fixed api of Waiting ExecBlockMock. Unused yet

* Made SortedGatherBlock async-capable

* Removed nonEmptyIndex hack

* Removed duplicate traceGetSome~ calls, moved all to getSome

* Added asserts before replacing getNr*Registers

* Added a TODO note and a comment

* Removed getSomeWithoutRegisterClearoutOld()

* Removed skip()

* Removed common code by using getNr*Registers()

* Use getNr*Registers() in the TraversalBlock as well

* started to add lane

* started to add lane

* added lane

* completed lane

* removed debug output

* fixed merge

* Began working on a test suite for AQL tracing/profiling

* Added more tests and asserts in aql-profiler

* Made some ExecutionBlocks final

* Added a type enum to all blocks and the per-block stats

* Add block type to stats nodes when tracing AQL on block level

* Removed initializeCursor call from instantiateFromPlan

* Avoided additional getSome calls after DONE

* Added more profiler tests

* Refactored ExecutionBlock::getOrSkipSome and fixed two bugs

- set _upstreamState also when skipping
- explicitly use xecutionBlock::getHasMoreState()

* Bugfix: update state

* Reuse parent _skipped wherever possible; rename where not (LimitBlock)

* Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code

* Implemented missing virtual function (with USE_FAILURE)

* Reset neccessary values during initializeCursor

* Simplified code in EnumerateListBlock a little

* Added a test for DistinctCollectBlock in aql-profiler

* Avoid redundant getSome calls in DistinctCollectBlock

* fix compilation

* Fixed DistinctCollectBlock profiler test

* Added a second profiler test for the DistinctCollectBlock

* Added a profiler test for EnumerateCollectionBlock

* Bugfix in EnumerateListBlock

* added --server.fifoN-size

* Simplified EnumerateCollectionBlock::getSome

* Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE

* Fix testEnumerateCollectionBlock1 for mmfiles

* do not pass by reference

* Fixed compile error

* fixed merge conflicts

* Added profiler tests for EnumerateCollectionBlock

* Test fix for mmfiles

* Fixed IResearch tests

* Bugfix in DistinctCollectBlock and a regression test

* Updated comment

* Bugfix for query statistics in cluster

* Check plan in distinct test

* Fix aql-profiler tests in cluster

* Removed unused line / bugfix for single server test runs

* This commit implements waking up of AQL queries. (#5651)

* Non-compiling intermediate commit for handover.

* Make branch compile again

* Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested!

* Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine.

* First successful query wakeup.

* The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler

* renamed finalizeExecute to shutdownExecute

* Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly

* fix audit log

* Removed callback from deleteQueryCursor. This cannot be waiting

* use CONDITION_LOCKER

* removed yet another thread-local variable

* Fixed forward declaration

* Made RestAqlHandler repeatable

* Use defer to close the query in RestAqlHandler. Now waiting will close the query as well.

* Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler

* Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case

* Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded.

* Removed reaction to existing nolockheaders in Coordinator Query Planning Phase

* Removed incorrect assertion.

* Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly.

* Fixed UserManager, now deletes nolock headers properly

* Swing to the Symphony of Destruction

* Forgot about community build...

* Fixed compiling of Catch tests

* Fixed community build

* need thread for size

* Made the restSimpleHndler repeatable

* Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor

* Reactivated StreamingCursors

* Removed debug output.

* Fixed false query continuation

* Reset thread output to non-debug

* Added missing return statements

* Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers.

* Fixed hidden merge conflict

* Bugfix in aql-profiler.js: use plan.nodes order, not stats

* Added two profiler tests for filter

* Avoid too many getBlock calls in the FilterBlock

* Removed debug output

* RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility.

* Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete

* Added another test for FILTER, and one test for the HashedCollectBlock

* Added more tests for HashedCollectBlock; avoid unneccessary getSome calls

* Added an profiler IndexBlock test

* IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls

* Added a second test profiling IndexBlock

* Added a third test for IndexBlock

* Moved general code to module

* Moved noncluster tests into a separate file

* Split aql-profiler testsuite into three files

* Added profiler tests for LimitBlock

* Added a test for NoResultsBlock

* Added profiler tests for TraversalBlock

* Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still

* Optimized TraversalBlock getSome calls due to new (nightly) test results

* Fixed std::min calls I broke

* Let shutdown calls in AQL wait, if the query is executed successfully.

* Fixed queryResult going out of scope

* fix compile error through merge conflict with devel

* Fixed compiler warning "mismatching tags"

* Removed debug log output

* Added TODO notes

* Fixed test fail due to devel merge

* Fixed some invalid sync waiting implementations

* Added a profiler test for SortBlock

* Added profiler tests for SortedCollectBlock

* Fixed bug introduced by devel merge

* Fixed Remoteblocks ignoring errors!

* Added some more continue Callbacks in used places. And removed debug log

* Removed debug log output

* Suppress clang warnings

* Bugfix: use of invalid stack pointer

* Bugfix: RemoteBlock::shutdown now sends code as int, not string

* Revert "Suppress clang warnings"

This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0.

* Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome

* Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed

* Allow copy elision
2018-07-09 14:24:10 +02:00
Jan 1c6ded6724
use sparse indexes in more cases (#5790) 2018-07-09 09:01:10 +02:00
Simon f47367a495 Allow dumping / restoring of views (#5774)
* allow dumping and restoring of views

* fix restore, adding some tests

* remove debug print

* fix global replication
2018-07-05 17:55:28 +02:00
Jan 3bcaa9e95c
simplify edge function JS calls (#5773) 2018-07-05 14:23:13 +02:00
Jan f9dc610c1e
Bug fix/internal issue 2658 (#5760) (#5778) 2018-07-05 10:19:28 +02:00
Simon b8e140493a Remove redundant geo code, remove WITHIN_RECTANGLE (#5757) 2018-07-04 20:21:00 +02:00
KVS85 e2d1d1e898 Bug fix/convert arangosearch tests to common (#5772)
* Create shell_client_aql test alias and move aql-view-arangosearch-noncluster to it

*  Changes to be committed:
	modified:   js/client/modules/@arangodb/testsuites/aql.js

* Move remained AS AQL tests (aql-view-arangosearch-cluster, aql-view-arangosearch-ddl-noncluster, aql-view-arangosearch-ddl-cluster, aql-view-arangosearch-feature) to common and fix aql.js to really use arangosh in client execution

* Update aql.js of client modules

* 	modified:   aql-view-arangosearch-noncluster.js

* Remove toString() from assertions

* Minor typo fixes and changes to aql-view-arangosearch-noncluster tests from devel

* Added lost assertion for testInTokensFilterSortTFIDF

* Fix typos.

* Restored modified in devel tests to be common.

* Fix db._query() typo.

* Remove unsupported AQL query and shell_*_aql from shell_* groups of
tests.

* Fix conflict.

* Fix conflict.

* Remove moved tests.

* Revert "Remove moved tests."

This reverts commit bcdbde4baed610df2ae2ab4768a8c17b79ed3f64.

* test
2018-07-04 21:16:00 +03:00
jsteemann 889f2db6fb fix jslint warning 2018-07-04 18:16:18 +02:00
Andrey Abramov 17e955d837
ArangoSearch advanced AQL functions (#5756)
* extend filter checking

* extend IResearchViewNode tests

* distinguish scorer and filter arangosearch functions

* initial implementation

* add some tests for the recently added aql functions

* more tests

* force phrase function to use context-wise analyzer

* more tests

* ensure exists function uses analyzer from context

* fix query execution tests

* allow to override context analyzer in phrase function

* modify EXISTS signature to accept optional analyzer, fix tests

* more tests

* even moar tests

* more tests

* update documentation

* add testing scorer

* fixes after review

* attempt to fix msvc compilation issues

* add extra check

* add another check
2018-07-04 02:59:31 +03:00
Jan b88fde9659
added key generators "uuid" and "padded" (#5740) 2018-07-03 15:23:47 +02:00
Jan 46e36d55f1
replace busy-waiting with just waiting (#5722) 2018-07-03 09:00:54 +02:00
Simon 5ee96fe208 fix multipolygon parser (#5713) 2018-06-28 20:26:30 +02:00
Jan 96cbe699b9
honor "restrictType" and "restrictCollections" when syncing (#5689) 2018-06-28 19:14:14 +02:00
jsteemann b1002075b3 Merge branch 'feature/add-distinct-aggregator' of https://github.com/arangodb/arangodb into devel 2018-06-28 19:03:16 +02:00
Simon 4387fabacf Converting Pregel AQL function to c++ and fixing a bug (#5620) 2018-06-28 10:46:16 +02:00
Vasiliy bf1f2eab29 issue 401.2: split IResearchViewMeta into the user configurable part and internal state (#5683)
* issue 401.2: split IResearchViewMeta into the user configurable part and internal state

* backport: update documentation and integration tests to reflect the IResearchViewMeta changes

* backport: addresss typo

* backport: address another typo
2018-06-27 16:40:51 +03:00
jsteemann 7ff481ff66 fix single server tests 2018-06-26 23:38:26 +02:00
jsteemann d71f8181ca added tests 2018-06-26 23:33:32 +02:00
jsteemann 2610af55f8 Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/add-distinct-aggregator 2018-06-26 20:16:46 +02:00
jsteemann 10d3609e15 fixed jslint warnings 2018-06-22 13:07:23 +02:00
Simon 52f44596ba Adding MultiPolygon support, fixing some outstanding bugs (#5613)
* Adding MultiPolygon support, fixing some outstanding bugs

* removing dead code

* Adding MultiPolygon tests

* Added multi-polygon docs

* adjust the docs

* fixing remaining occurrences of geo1 / geo2, added note into UpgradingChanges34.md

* some clarification

* Improve Geo docs, reformatting

* Release notes: responses are JSON, reformatting

* Update release notes: /_api/aqlfunction has an isNewlyCreated attribute now

* Update Geo.md
2018-06-22 10:03:27 +02:00
Andrey Abramov 5eef6cd618
Feature/test iresearch (#5610)
* start implementing arangosearch cluster tests.

* backport: ensure view lookup is done via collectionNameResover, ensure updateProperties returns current view properties

* first attempt to fix failing tests

* refactor cluster wide view creation logic

* if view is not found in the new plan then check the old plan too

* ensure the cluster-wide view is looked up in vocbase as well on startup/recovery

* do not store cluster-wide IResearchView in vocbase

* move stale view cleanup to the shared pointer deleter, address test failures

* do not print warning

* enable arangosearch tests by default

* fix catch tests

* address icorrect return value for cluster-wide links

* address some issues with test failures due to cluster-view allocated within TRI_vocbase_t

* simplify per-cid view name, address 'catch' test failures

* ensure IResearchViewNode volatility is properly calculated in cluster

* invoke callbacks directly in AgencyMock instead of waiting for timeout

* ensure view updates via JavaScript always use the latest view definition

* pass a list of shards to `IResearchViewDBServer::snapshot`

* extend cluster aql tests

* fixes after merge

* fix class/struct inconsistencies

* comment failing tests

* remove debug logging

* add debug function

* tests cleanup

* simplify upcoming merge: pass resolver from a side

* backport: move all transaction status callback logic to Methods

* add changes missed from previous commit

* fix js and ruby tests

* more tests for IResearchViewNode

* pass transaction to IResearchViewDBServer::snapshot, address IResearchViewDBServer tests segfault

* pass transaction to IResearchView::snapshot instead of transaction state

* temporarily add trace log output to tests to try to find the cause of the core dump on Jenkins

* add more temporary debug output to trace down the segfault on Jenkins

* add even more temporary debug output to trace down the segfault on Jenkins

* ensure Vieew related maps are cleared during shutdown

* reset ClusterInfo::instance() before DatabaseFeature::unprepare()

* remove extraneous debug output

* missed line from previous commit

* uncomment required line

* add nullptr checks to RocksDBIndexFactory::prepareIndexes(...) similar to the ones in MMFilesIndexFactory::prepareIndexes(...)

* attempt to fix deadlock in tests

* add comment as per reviewer request

* fix aql test suite name

* add some debug logging

* address deadlock between ClusterInfo::loadPlan() and CollectionNameResolver::localNameLookup(...)

* eplicitly state which index definition failed in the log message

* use vocbase from shard-view isntead just in case

* explicitly state which index definition failed in the log message

* do not create shard-view instances from cluster-link instances (only register existing ones)

* add some tests
2018-06-21 20:35:04 +03:00
jsteemann 164ec9ed03 fix jslint 2018-06-21 11:48:18 +02:00
Wilfried Goesgens b1686cb902 fix broken test rule (#5616) 2018-06-21 11:42:38 +02:00
Jan 36e8feb681
fixed internal issue #2559 (#5635) 2018-06-20 11:45:57 +02:00
jsteemann 8af5e06223 refactoring of aggregators 2018-06-20 02:33:20 +02:00
jsteemann 69dec8ba17 Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/add-distinct-aggregator 2018-06-19 21:53:19 +02:00
Max Neunhöffer a84d9f7335
Add an API to query for status of moveShard and cleanOutServer jobs. (#5593)
This is so far intentionally undocumented, since we want to collect
experience with it first.
2018-06-15 16:27:47 +02:00
Jan 6754aca831
remove unused JS functions (#5619) 2018-06-15 15:53:43 +02:00
Jan 5df12917d6
prevent an optimization from destroying variables (#5604) 2018-06-14 18:09:10 +02:00
Jan 08b12942bd
fix AQL DOCUMENT lookup in case a collection has multiple shards and custom shard keys (#5602) 2018-06-14 12:20:06 +02:00
jsteemann 2712f9ce14 Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/add-distinct-aggregator 2018-06-14 00:46:22 +02:00
Jan Christoph Uhde f81e42af36 Add optimizer rule that replaces js function calls (NEAR/WITHIN/FULLTEXT) with pure AQL (#5529) 2018-06-12 13:48:31 +02:00
Jan 7351423ad5
cleanup import messages, reuse StringBuffer for JSON import (#5569) 2018-06-11 09:47:26 +02:00
Simon 352fedd29c Seeded pagerank (#5491)
* Seeded pagerank

* Fix the docs
2018-06-08 16:44:23 +02:00
Jan af9d2c05bc port COLLECT bugfix from 3.3 (#5519) 2018-06-08 15:45:41 +02:00
Heiko 743d4430bf added replication ui (#5555) 2018-06-08 11:19:05 +02:00
Jan 80f6574c6a
remove unused files (#5520) 2018-06-04 11:26:38 +02:00
Andrey Abramov f708e46973
Feature/arangosearch gather node (#5487)
* integrate ArangoSearch into GatherNode/GatherBlock logic

* fix inconsistency after merge

* add serialization/deserealization logic for IResearchViewNode sorting

* fix optimizer rule order

* fix copy/past typo

* fix tests

* ensure `GatherNode` is properly set up for ArangoSearch views

* adjust optimizer rule order
2018-05-30 02:53:21 +03:00
Jan dc97eb0111
Subquery optimizations (#5398) 2018-05-28 13:56:03 +02:00
jsteemann e5ee8e0573 added new aggregators 2018-05-28 08:48:01 +02:00
Jan edfdd49a3f
Bug fix/simplify skip hasmore (#5466)
* simplify skip and hasMore

* include less dependencies in header files
2018-05-26 09:27:28 +02:00
Jan a295eaa120
show AQL functions used in query in explain output (#5451) 2018-05-24 21:17:51 +02:00
Jan Christoph Uhde a2dcb6cc5d WIP - start adding optional overwrite to insert operation (RepSert) (#5268) 2018-05-24 19:47:15 +02:00
Jan 1eea2df7d0
fixed issue #5252 (#5427) 2018-05-23 08:30:27 +02:00
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00
Jan 245c759523
fix incompatibility after fullCount change (#5399) 2018-05-22 11:28:17 +02:00
Wilfried Goesgens 7b8fce07a4 aql current_user in cpp, test (#5302) 2018-05-19 21:26:53 +02:00
Jan df23ed092e
Bug fix/cleanup 1705 (#5376) 2018-05-18 09:21:50 +02:00
Wilfried Goesgens eed1235893 Feature/aql date trunc 2 (#5295) 2018-05-16 17:33:27 +02:00
Mark 435523167f Bug-fix Foxx empty json request body (#5304) 2018-05-16 16:54:38 +02:00
Mark 3f557d64cc Fix Foxx queues (#5352) 2018-05-16 16:54:03 +02:00
Wilfried Goesgens e1a8db0eb6 Feature/aql format date (#5294) 2018-05-16 14:05:55 +02:00
Jan 07e5750737
Feature/change fullcount behavior (#5285) 2018-05-14 15:46:07 +02:00
Jan 1c1d8f2528
fix AQL ZIP function with duplicate attribute names (#5336) 2018-05-14 12:59:18 +02:00
Tobias Gödderz afe13fba57 Fix race condition between foxx install and selfHeal (#5273) 2018-05-11 12:27:56 +02:00
Jan e1da502c67
remove now-superfluous tests for V8 versions of AQL functions (#5319) 2018-05-11 08:31:14 +02:00
Tobias Gödderz 8c87f51429 Feature/fix inconsistent distribute shards like job (#4743) 2018-05-07 16:53:08 +02:00
Jan 7247816422
Feature/aql cleanup (#5261) 2018-05-04 11:20:47 +02:00
Jan 9c0460acff
Feature/aql optimizations (#5237) 2018-05-03 16:55:58 +02:00
Alan Plum d5a93f5926 Teach executeTransaction about ArangoCollection objects (#5183)
* Teach executeTransaction about ArangoCollection objects

* Simplify logic

* Don't test with allowImplicit

* Update arango-database.js
2018-05-03 14:17:31 +02:00
Jan Christoph Uhde 1c88902962 Fix AQL LIKE function - escapes * when building pattern. (#5239) 2018-05-03 13:14:58 +02:00
Simon 828f1d423c S2 based Geo-Spatial index (#5249) 2018-05-02 23:54:41 +02:00
jsteemann 996cba393f potential test fix 2018-05-02 10:37:19 +02:00
Dan Larkin-York b9d05351fb Parallelize arangorestore using the tools developed for arangodump (#5206) 2018-05-01 10:56:11 +02:00
jsteemann 3fee25dd9d remove references to V8() function from test file 2018-04-30 23:35:26 +02:00
jsteemann b1c48f60cd jslint 2018-04-30 18:38:06 +02:00
Wilfried Goesgens ba278a64c9 add native implementation of the FAIL aql function (#5232) 2018-04-30 18:35:14 +02:00
Jan 349e2c1681 forward-port AQL optimizations from 3.3 (#5020)
* forward-port AQL optimizations from 3.3

* fix restrict-to-single-shard rule

* added tests for specific cases

* ModificationNodes now can know if they are restricted or not

* OptimizerRule now updates the ModificationNode instead of setting ownName on remote node.

* The DBServerEngine container now only sends a subset of snippets (those which are relevant) It is  able to restrict any part of the query to a certain shard or even a list of shards

* ExecutionEngine CreateBlocks in ClusterMode now returns Result instead of throwing.

* Adapted Coordinator side to a bitter mapping of RemoteNode => QuerySnippets. This is still ongoing work.

* Fixed Coordinator enginecontainer test and finisehd implementation of EngineInfoContainerDBServer. Seems to work so far, need to adapt test to prove it.

* Adapted test for modification nodes and restrict to shard.

* restrictTo can now be de-/serialized on Modification Nodes

* Fixed Community Compile Error

* Added a test for Indexes/EnumerateCollection nodes

* EnumerateCollection and IndexNode now have options to be restricted to a single shard, opt rule not yet updated

* DBServer part of planning now checks Enumerate and IndexNodes for single shard optimization

* Added tests for simple single shard optimization on IndexNodes

* Adapted optimzierrule for SingleShard optimierung on IndexNodes

* Fixed an issue with externally restricted shard ids in combination with shard_id based optimization

* Added a test for multiple single shard optimizations on the same collection

* Removed collection/Shard duplicate check of optimize to single shard rules

* Updated Explainer to handle optimize to single shard

* Let more tests pass an optimized AQL variant

* Temporarily disabled a tests that can only be solved after some work in transaction
2018-04-30 11:25:09 +02:00
Jan 30b12e311b
Bug fix/remove most of aql js (#5223) 2018-04-30 11:17:11 +02:00
Simon 468231efc5 AQL Profiling code (#5165)
* initial start of profiling

* adding profiling code

* Fixing remote block tracing, fixing width and units

* Fixing some tests

* Various fixes

* adressing review comments
2018-04-24 16:17:30 +02:00
Simon 45fbed497b Supervision Job for Active Failover (#5066) 2018-04-23 12:49:41 +02:00
Kaveh Vahedipour 3d043b35a3 Feature/supervsion maintenance mode (#5108)
* Supervision goes to Maintenance mode, when /arango/Supervision/Maintenance exists
* coordinator route stands
* stop updates in transient, when supervision off
2018-04-20 13:23:22 +02:00
Alan Plum 2a6f8a21aa Allow specifying Foxx manifest.json value (#5157) 2018-04-20 12:55:18 +02:00
Simon fad4c0f9f4 Moving out more from #5066 (#5124) 2018-04-17 16:55:19 +02:00
Simon 8be273efb8 Replication cleanup (#5105) 2018-04-17 08:17:42 +02:00
Tobias Gödderz 089b21e3de fix dump&restore of smart graphs (#5093)
* fix dump&restore of smart graphs (#5093)

* Added tests for dump-restore of SmartGraphs

* Arangosh will now expose isSmart and the smartGraphAttribute on properties

* RestReplicationHandler will now ignore smart-graph collections unless you execute it with force

* Added changelog

* Reactivated original mmfiles dump/restore test

* Skip hidden smart graph collections in arangodump

* Do not dump shadowCollections metadata of smart edge collections

* Cluster optimizer rules for soerted gather nodes now handle virtual edge collections correctly

* Added a dump/restore tests for smartgraphs in rocksdb as well

* Deactivated checks for writesExecuted statistics in dump/restore tests for smartgraphs mmfiles

* Really exclude shadowCollections

* Reduced loglevel

* Added tests

* Don't change single-server behaviour

* Fix tests for omitted shadowCollections and hidden collections

* Activated statistics in MMFIles dump test again and included isEnterprise in rocksdb dump test

* A modification node can now disableStatistics, which means it does not contribute to query->extras() this is only relevant in SmartGraph case so far.

* Added a test to dump&restore satellite collections

* Bugfix: restore satellite collections properly

* Added regression test for internal issue #2237

* Fix bug #2237

* Updated CHANGELOG

* Copied dump/restore tests to rockdsb

* Removed enterprise test

* Added inline comment for smart-edge collections in optimizer rules

* Removed duplicate CHANGELOG entry

* Simplified removal of shadowCollections

* fix jslint

* Made internal.isEnterprise() available in js client context. Taken from 3.3/59f655fa

* fix broken dump_encrypted test

* Removed accidentally copied CHANGELOG entry
2018-04-16 12:28:13 +02:00
Wilfried Goesgens ac2a8721e6 Feature/aql native call apply (#5100) 2018-04-13 16:07:06 +02:00
Jan 79eef177ce
remove sorts from GatherNode if not required (#5019) 2018-04-13 16:02:00 +02:00
Jan 167dfcdf7f
Feature/index only scans (#5005) 2018-04-03 17:34:28 +02:00
Jan 65bfeb7054
Bug fix/forward port 3.3.5 changes (#4956) 2018-04-03 17:31:34 +02:00
Simon f6d3032f59 Enabling cache (#4880) 2018-04-03 14:18:57 +02:00
jsteemann eb1af0e14f attempt to fix non-deterministic test 2018-03-28 14:46:26 +02:00
Max Neunhöffer 8168cc976a
Fix replication test w.r.t. indexes comparison. (#4978) 2018-03-28 09:42:47 +02:00
Jan 2c3632118c
fixed issue #4934: Wrong used GeoIndex depending on FILTER order (#4948) 2018-03-26 09:42:49 +02:00
Wilfried Goesgens a16d4b0109 Feature/cpp aql reverse (#4911) 2018-03-23 09:54:37 +01:00
Jan 69ba136a6b
fixed issue #4900 (#4917) 2018-03-22 18:26:24 +01:00
Manuel B 215777736e Move AQL Date Functions => CPP 2018-03-22 15:22:24 +01:00