1
0
Fork 0
Commit Graph

3508 Commits

Author SHA1 Message Date
Jan c8cad8a4f8
try again in case of a conflict when updating users (#7614) 2018-12-04 16:51:21 +01:00
jsteemann 1dbaa9d5ab fix segfault 2018-12-04 08:57:11 +01:00
Jan adc651e338
Bug fix/cxxcheck 28112018 (#7518) 2018-11-29 13:39:04 +01:00
Jan b2924057e7
cleanup (#7507) 2018-11-28 19:42:37 +01:00
Jan 1bef7e842c
do not allow MMFiles single document operations overtaking each other… (#7469) 2018-11-28 13:03:47 +01:00
Simon d0efd95a37 Bug fix/restore index refactor (#7470) 2018-11-27 20:22:36 +01:00
Simon c584527d79 Fix restore of views, add --view option (#7425) 2018-11-22 19:24:24 +01:00
Simon d5cb94d2d0 Minor refactoring (#7408) 2018-11-22 16:16:05 +01:00
Vasiliy 1a0b9b9261 issue 153: ensure views are dropped in Agency when database is dropped in cluster, minor fixes (#7370)
* issue 153: ensure views are dropped in Agency when database is dropped in cluster, minor fixes

* backport: add test to ensure views are dropped when database is dropped from plan, fix some issues in ClusterInfo

* optimize primary key lookups in ArangoSearch

* fix test

* Add JS tests

* temporary comment optimizations
2018-11-21 19:18:34 +03:00
Wilfried Goesgens 05a7d4e96e add alternative to ClusterInfo::getCollection() that doesn't throw (#7339) 2018-11-20 16:05:57 +01:00
Tobias Gödderz c61ed1d77a MMFiles-replication-get-followers-under-lock (forward-port) (#7343)
* Forward-port of bug-fix-3.4/mmfiles-replication-get-followers-under-lock

Fix resign order

Fixed a typo

Get followers later, add TODOs

Added a callback parameter to collection insert methods

Get followers under the lock if necessary

Extracted the replication of inserts into a separate method

Move shortcut into replicate method

Added callbacks for remove, replace and update

Added missing overrides

Extracted replication code from modifyLocal and removeLocal

Update followers under lock also during replace, update, remove

Fix changes from the last commit for update/replace

Update comments, add asserts

Remove changes for document-level locks that will be done in another PR

Unify replication

Adapt log messages to the devel ones

Move common methods from its descendants to TransactionCollection, fix Mock on the way

More IResearch test / mock fixes

Relax asserts for nested transactions

Reformat

Fix non-babies remove and modify replication

* Remove some changes introduced by the merge

* Fixed compile errors introduced by merge
2018-11-20 09:43:26 +01:00
Dan Larkin-York 32db9157bf Updates to collection versioning. (#7261) 2018-11-19 09:45:52 +01:00
Jan 21979e6906
Bug fix/do not create routing collection (#7290) 2018-11-13 16:54:06 +01:00
Jan 4631cfb5e8
do not create _modules collection for new databases/installations (#7291) 2018-11-13 11:40:15 +01:00
Dan Larkin-York b447f98a7a Upgrade old MMFiles collections for compatibility with ArangoSearch. (#7248) 2018-11-07 15:52:37 +01:00
Vasiliy facb4043a1 issue 496.5: minor API cleanup and error reportin enhancements, update iresearch to commit d69f7bd184e009da7bf0a478efd34a0c85b74291 (#7216)
* issue 496.5: minor API cleanup and error reportin enhancements, update iresearch to commit d69f7bd184e009da7bf0a478efd34a0c85b74291

* add workaround for shell-collection-rocksdb-noncluster.js::testSystemSpecial test failure

* fix typo
2018-11-06 13:59:18 +03:00
Tobias Gödderz b096d3b6e4 Restore /_api/view to return internal view names again (#7222) 2018-11-06 11:40:55 +01:00
Simon 0eb5142df8 Use shared_ptr for LogicalCollection (#7220) 2018-11-05 18:31:57 +01:00
Vasiliy 68953ae33a issue 496.4.1: move StorageEngine-specific flag out of the genric API and closer to the storage engine (#7212) 2018-11-04 16:52:28 +03:00
Simon b883026f45 Server stream cursor (#7186) 2018-11-02 16:19:48 +01:00
Vasiliy f088733420 issue 496.3: move more coordinator-related logic out of TRI_vocbase_t, rename some arangosearch view configuration parameters, remove some consolidation policies, update iresearch to revision 6fd9760d81b136f769e277ea5b8f53996ed7a1ca (#7166)
* issue 496.3: move more coordinator-related logic out of TRI_vocbase_t, rename some arangosearch view configuration parameters, remove some consolidation policies, update iresearch to revision 6fd9760d81b136f769e277ea5b8f53996ed7a1ca

* address potential deadlock between link creation and FlushThread

* remove code causing nullptr access

* add back lock around reader reopen

* revert: address potential deadlock between link creation and FlushThread

* invalidate payload for each field in FieldIterator before setting a value
2018-11-01 23:10:01 +03:00
Matthew Von-Maszewski a054e31f73 port 3.4 changes that give libcurl time to retry a failed DNS query. Also add changes to vocbase.cpp that were missed in previous PR. (#7132) 2018-10-30 21:00:13 +01:00
Vasiliy 8f44afb6cf issue 496.1: switch scope of responsibility between a TRI_vocbase_t and a LogicalView in respect to view creation/deletion (#7101)
* issue 496.1: switch scope of responsibility between a TRI_vocbase_t and a LogicalView in respect to view creation/deletion

* backport: address test failures

* backport: ensure arangosearch links get exported in the dump

* backport: ensure view is created during restore on the coordinator

* Updates for ArangoSearch DDL tests, IResearchView unregistration and known issues

* Add fix for internal issue 483
2018-10-30 12:50:35 +03:00
Simon c72818a9dc Make ensureIndexOnCoordinator more robust (#7110) 2018-10-29 17:45:46 +01:00
Matthew Von-Maszewski 97ba8ca2be Bugfix: More 3.4 scheduler changes backported (#7091) 2018-10-26 17:09:20 +02:00
Simon 10dc287eb3 Silence Tsan warnings (#7075) 2018-10-25 15:50:39 +02:00
Jan 221d036d5d
Bug fix/fix catch test issues (#7044) 2018-10-25 11:39:55 +02:00
Simon d23aaa2198 Better agency pool update (#7040) 2018-10-24 16:23:21 +02:00
Jan 1924f74171
fix https://github.com/arangodb/release-3.4/issues/99 (#6952) 2018-10-19 13:04:24 +02:00
Vasiliy 18b2eb6d98 issue 485: ensure LogicalDataSource::drop() is called on vocbase drop (#6895)
* issue 485: ensure LogicalDataSource::drop() is called on vocbase drop

* add missed change

* backport: address race between make(...) and async job

* add another missed change

* backport: ensure recursive lock reports itself as locked correctly

* backport: address test failure on mmfiles

* backport: remove redundant lock already held by async task

* backport: reset reader before unlinking directory
2018-10-16 17:03:51 +03:00
Jan 8f44d6b658
Bug fix 3.4/views in aql collections (#6714) (#6785) 2018-10-11 10:52:38 +02:00
Lars Maier fac7b48c74 [3.5] Feature/decoupled io (#6281)
* Decoupled IO from Scheduler.
* Fixed SSL start up bug.
* Updated messages and thread names. Fixed missing code from cherry-pick.
* Reintroduced checks for executing thread to be correct. Modifed default value for io-context depending on cores.
* Fixed memory leak caused by cyclic references.
* Actually distribute endpoints. Move handlers into function and do not copy them for each encapsulation.
* Inserted debug output.
* BUG FIXED! One has to call drain() on every queue as temporary work around.
* Added some flags and output for testing.
* More debug output!!!
* Manuel is right.
* Removed debug output.
2018-10-08 13:05:12 +02:00
Jan e78d1aa541
Bug fix/even more ldap debugging (#6736) 2018-10-08 09:42:11 +02:00
jsteemann 56147843c2 Merge branch 'feature/additional-logging' of https://github.com/arangodb/arangodb into devel 2018-10-02 17:53:31 +02:00
Jan c06f2d77da
Feature/velocypack update (#6678) 2018-10-02 14:04:14 +02:00
Dan Larkin-York b922d260bc More cleanup and additional logging. 2018-10-02 07:50:26 -04:00
Simon fcd7f49293 Fix error in dump test, create views before syncing data (#6636) 2018-09-28 16:59:38 +02:00
Jan 44448321e1
Bug fix/optimizations 210918 (#6573) 2018-09-24 20:16:16 +02:00
jsteemann f63fedf139 increase timeout for dropping collections 2018-09-24 18:38:20 +02:00
Jan 024a9d7ac1
less RocksDB background thread activity in case there is nothing to do (#6577) 2018-09-24 13:45:13 +02:00
Simon 0fa7f01c66 Resilience test failure points (#6539) 2018-09-20 01:05:10 +02:00
Jan c38051519e
Bug fix/simplify things (#6516) 2018-09-18 17:47:01 +02:00
Lars Maier f93b6fd7b8 Bugfix/early out invalid links in view creation (#6502) 2018-09-17 09:50:28 +02:00
Simon 623fbdf4e9 Do not hide indexes (#6507) 2018-09-14 17:38:04 +02:00
Dan Larkin-York 0dfabd8f04 Fix several TSan warnings (#6473) 2018-09-14 11:16:45 +02:00
Jan e18e9158d8
fix cluster selectivity estimates (#6488) 2018-09-14 00:22:01 +02:00
Lars Maier e2930ccba6 Completely export views dump restore (#6466) 2018-09-12 17:21:22 +02:00
Jan 3b16913b1b
fix cluster index selectivity (#6467) 2018-09-12 14:35:39 +02:00
Vasiliy b509e2e70a issue 465.5: when renaming a datasource use the object from within vocbase (#6413) 2018-09-07 16:01:04 +03:00
Vasiliy 0635adec56 issue 465.3: remove unused functions and variables, ensure the proper object is tracked during renaming, remove duplicate feature instantiation (#6382) 2018-09-05 18:26:32 +03:00
Vasiliy 5329f34771 issue 465.2.2: remove redudnant heap allocations and simplify API (#6349)
* issue 465.2.2: remove redudnant heap allocations and simplify API

* address merge issue

* address more merge issues

* address more merge issues

* address review comments

* do not deallocate non-allocated instances
2018-09-05 13:37:37 +03:00
Simon 0661a4c1fe Hide Links from getIndexes() (#6325) 2018-09-03 15:17:24 +02:00
Vasiliy e862efdc3b issue 458.4: retrieve the system database via the SystemDatabaseFeature (#6299) 2018-08-31 19:45:10 +02:00
Jan 191ae2dc43
do not report figures for an index by default (#6296)
the change in behavior crept in unintentionally due to my last
change of the index API. this PR reverts the unintended change
here so figures are only returned when explicitly requested
2018-08-31 17:22:00 +02:00
jsteemann 9650911d53 improve wording here and there 2018-08-29 12:34:29 +02:00
Jan 5022ccc24d
Bug fix/fixes 2508 (#6254) 2018-08-27 21:36:39 +02:00
Jan 73e44bc605
invalidate query cache after modification of views (#6198) 2018-08-24 19:54:10 +02:00
Jan 5f0403ed1c
Bug fix/add aql collection count cache (#6227) 2018-08-23 16:05:51 +02:00
Vasiliy febd445410 issue 451.1: ensure errors are properly propagated during LogicalView creation (#6218)
* issue 451.1: ensure errors are properly propagated during LogicalView creation

* address Ruby test failures

* ensure invalid view type returns TRI_ERROR_BAD_PARAMETER

* address jslint errors
2018-08-22 16:11:24 +03:00
jsteemann 9959218d1d revert unification of collection name validation 2018-08-21 18:46:13 +02:00
Jan 2aa879a53b
unify validation of collection names in AQL (#6203) 2018-08-21 11:11:15 +02:00
Simon 229c09d434 Allow dirty-reads from passive (#6136) 2018-08-20 16:26:14 +02:00
Simon 2d29232c59 fix restoring iresearch view in cluster (#6159) 2018-08-17 09:28:20 +02:00
Frank Celler a688dc0962
Feature/remove job queue thread (#5986)
limiting V8 calls in flight
2018-08-10 12:17:43 +02:00
Dan Larkin-York 5f87f57cd0 Improved sharding algorithms (#6089) 2018-08-09 19:03:32 +02:00
Tobias Gödderz de4f5587ae Gharial rewrite in C++ (#5631)
* Built a C++ skeleton REST handler for gharial, with fallback to the JS handler

* Moved aql::Graph to graph::Graph

* Added complete edge definitions to Graph

Also:
- some cleanup
- used forward-declarations in headers
- use Graph in graph rest handler

* Handle graph lookup failures according to the test suite

* Added GET vertex

* Bugfixes in ResultT

- Added missing #include
- Fixed move semantics

* Move central code of readVertex to GraphOperations

* ResultT fixes and complements

* Implemented a graph cache

* Added and used graph cache to the rest handler

* Added GET edge

* Added DELETE edge

* Extracted some code

* Added PATCH and PUT for both edge and vertex

* Moved update/replace transaction code to GraphOperations

* Added stub routes for POST and a TODO note

* Added a test checking that deleting a vertex removes all incident edges as well

* Added a test checking that deleting a vertex does not remove edges in non-graph collections

* fixed compiler warnings and errors

* Began work on DELETE vertex

For this, added a V8Context to allow for AQL queries to use subtransactions

* Continued work on DELETE vertex (still WIP)

* prep for graph post routes

* fixed removeVertex operation (aql)

* added post vertex and post edge gharial routes

* wasSynchronoues flag changed

* gharial post c++ handler, naming conventions

* added gharial tests

* temporary disabled cache (because not completed), added graph property read functions

* added c++ gharial list vertex collections

* added c++ gharial graph config

* added c++ gharial list graphs

* added graph manager class

* first implementation of create graph in c++, WIP

* changed error messages

* added etag to create graph api, still multiple edge definition check missing

* finished POST /_api/gharial/<graph>

* WIP - DELETE /_api/gharial/<graphName>

* added DELETE /_api/gharial/<graphName> validation, still missing correct response

* gharial delete

* fixed delete gharial lock

* finished DELETE /_api/gharial/<graphName>

* added routes for graph based vertices and edge definitions

* improved delete route

* added add new edge definition to existing graph

* patch edge definition in a graph, still <WIP>

* finished edit edge definition route

* code changes due to devel code changes

* added remove edge definition route

* added vertex delete function

* added todo note regarding one drop collection issue

* add oprhan collection to graph route implemented

* Added a test

* Updated a comment

* Several minor changes

* Minor changes during review

* Changes during review

* Changes during review

* Bugfix: orphans may be null or omitted

* Bugfix: resolve externals

* minor code changes

* seperated graph class to independent component classes

* seperated graph class to independent component classes

* removed log output

* fixed create collection behaviour in a cluster environment

* fixed enterprise graphs behaviour in c++ gharial api

* removed log output

* formatting

* improved error handling, fixed a linux compile bug

* more result refactoring

* more result type cleanup

* fixed wrongly defined test

* result handling

* error handling

* more refactoring

* Bugfix: avoid race condition in cluster when creating collections

* updated graph documentation

* added graph related static strings

* static strings, new method to create options for gharial created collections

* Some minor cleanup

* more use of static strings

* minor code changes, review

* added missing parseint

* removed gharial foxx, added js common module, added v8 general graph module

* correct use of virtual method

* more v8, js general graph, broken state

* more v8 graph functions

* fixed editEdgeDefinition, added drop function

* fixed drop behaviour

* added _list, _exists

* added c++ rename graph collections, added v8 + graph module function

* Added a regression test

* added graph._deleteEdgeDefinition, v8, server

* more v8g

* added _removeVertexCollection

* added _extendEditDefinitions

* todo, need to add a helper sort method for a local defined relation

* fixed test

* fixed lots of tests, added more client functions, _addVertexCollection on client module is still broken

* added more client graph functions, all tests green

* more client functions

* add del edge def route

* Fix use after move

* Minor changes in client general-graph.js module

* Make a copy before sorting (don't touch the argument)

* Minor changes and some additional asserts in graph tests

* Consistently set parameter defaults

* Renamed static strings

* Remove superfluous function

* Made comment more verbose

* Minor changes in general-graph-common.js

* Added missing template arguments

* Fixed community build

* Cleanup in editEdgeDefinition

* Regression test & bugfix: comparison of edge definitions didn' order from and to

* Fixed errors introduced by merge

* Minor changes in v8-general-graph.cpp

* Fixed test failure due to wrong error code in CE

* added missing id field

* Added permission checks for graph._create

* Removed assertion that is no longer valid

* Moved removeGraph from GraphOperations to GraphManager

* Allow C++ implementation of graph._drop to handle smart graphs

* Flush js client db cache after creating/dropping collections via the general graph module

* Added _deleteEdgeDefinition to the general graph client module

* WIP: Added permission checks for drop graph

* Fixed permission checks for drop graph

* Added permission checks for other graph operations

* Bugfix: assert edge definitions are returned in order

* Some cleanup

* Removed unused method

* Minor improvements in GraphManager

* Fixed a type in general-graph common module

* Most useful fix of all times ever: Do not auto cast from bool to int and alternate error/noerror by this

* Added the initial keyword to StaticStrings

* Added a new error code, used whenever a user tries to inject a documentcollection as a relation into the graph, which is invalid

* Some GraphManager/Ops/Graph cleanup. Less Slice parsing, more usage of GraphObjects

* Test edgeDefinitions in graphs with a defined ordering

* GraphClass Layout cleanup

* Do not test error messages, use codes instead

* Recreated backwards compatibility of Graph Creation Permission errors

* Changed error-code if edgedefinition is used twice

* Added a StaticString for the GraphName

* Renamed graphToVpack => graphForClient

* Partly fixed graph-api test to work with better error messages. Still red: The edgeDefinitions are now sorted, the test is supposed to sort his own list, but appearently does not do so. Under investigation

* Added a new error code that rejects injection of differently sharded smart collection into smartgraph. Should be more helpful to our users

* graph createCollectionOptions now require an open object to be cross-called from enterprise. Made enterprise switch for creation of graph more elegant.

* Updated graphs.cpp

* Massive refactoring. Made Factories for graphs to make SmartGraph much more transparent. Also reduced amount of multiple implementations of the same stuff. Killed vocbase/graphs use GraphManager instead. Removed usage of GraphCache, was not completely implemented anyway and only partially used, which is bad at the moment. Option for later improvement never the less

* Adapted JS code to now really use c++ variants. ALso included 3 Classes: Graph, SmartGraph and GraphModule.

* Fixed undefined behaviour in Remove Vertex. Fixed smartgraph sharding if one collection already exists.

* Removed DEBUG output

* Removed DEBUG logs

* Removed dead code

* Fixed Graph EdgeDefinition test, they now have a different ordering.

* Added a test when adding a vertexCollection that it is actually valid in the graph

* Client Graph API now correctly sends `orphanCollections` and not `orphans`

* Let GraphOperations modify the graph in-place. It should now properly handle edgeDefinitions.

* Added initial cid StaticString

* Included the vocbase in fromPersistence creation of Graphs. Only required to enhance 3.3 SmartGraphs on the fly.

* Fixed internal error message

* Fixed compiler isses originiated from merging

* Removed unused imports

* Regenerated generated file
2018-08-09 09:30:04 +02:00
jsteemann 987c33dfc3 fix compilation with g++5.5 2018-08-08 16:00:31 +02:00
Jan 93222b15d4
track last used keys in cluster key generators, track key on cluster document insert (#6101) 2018-08-08 14:32:16 +02:00
Vasiliy 7518d0267d issue 434.1: remove redundant checks (#6090) 2018-08-07 14:44:19 +02:00
Simon 4238cff3ff Adding cacheUsage property and tests (#6075)
* Adding cacheUsage property and tests

* Make cacheUsage more permissible
2018-08-06 14:55:20 +02:00
Andrey Abramov b8e7752787 Feature/arangosearch stabilization (#6069)
* Added authentication tests for ArangoSearch views creation (client and task)

* Added authentication tests for ArangoSearch views querying (client and task)

* Added authentication tests for ArangoSearch views update (client and task) and correct other tests

* Added authentication tests for ArangoSearch views drop (client and task) and correct other tests

* Fix tests to support additional 'properties' layer of a view

* Add data to dumped/restored collection with a link from a view to single server test and view dump/restore to cluster tests

* backport from iresearch upstream: use a bitvector instead of an acceptor lambda for consolidation policies

* ensure view access permissions are checked for REST calls

* ensure view access permissions are checked for V8 calls

* do not allow LogicalView property modification if reading the properties fails

* Fixed authentication tests for ArangoSearch views to pass regardless of errorNum and possible multiple REST codes

* add missing ldap feature for enterprise version

* do not store useless data in ArangoSearch index (see #289, #441)
2018-08-05 20:47:22 +02:00
Jan b278d6874a
allow master & slave to work in parallel for RocksDB WAL tailing (#6059) 2018-08-03 13:37:53 +02:00
Simon a79c8af310 fix non-blocking stream cursors (#6064) 2018-08-03 12:41:52 +02:00
Simon 42cabb858a Fix dumping of views in the cluster (#6024) 2018-08-02 17:25:49 +02:00
Simon c26ab64b8e fixing exception in ClusterIndexFactory (#6053) 2018-08-02 08:59:20 +02:00
Vasiliy 62ca1235ac issue 430.3: remove redundant constructor from SingleCollectionTransaction (#5996) 2018-07-26 16:54:53 +03:00
Jan e4d7f1c5f0
Bug fix/wenn der shard mann 2mal klingelt (#5890) 2018-07-26 15:37:40 +02:00
Simon 2dd8593609 View Replication (#5915) 2018-07-26 10:28:46 +02:00
Dan Larkin-York 83f393500a Add C++ implementation, load-balancing support, and user restriction for tasks API (#5973) 2018-07-25 09:17:43 +02:00
Jan 21064144c8
Bug fix/replication improvements (#5962) 2018-07-25 09:04:50 +02:00
Jan a5bb50b0bf
remove methods from VelocyPackHelper that are also in VPackSlice (#5946) 2018-07-25 09:01:29 +02:00
Vasiliy 08d3cb3802 issue 399.4: use LogicalCollection& instead of LogicalCollection* in Index (#5929)
* issue 399.4: use LogicalCollection& instead of LogicalCollection* in Index

* backport: address enterprise build issues
2018-07-20 14:55:58 +03:00
Vasiliy a401934d62 issue 399.3: remove internal persisted proerty 'collections' from externally generated IResearchView jSON (#5874)
* issue 399.3: remove internal persisted proerty 'collections' from externally generated IResearchView jSON

* backport: address test failure

* backport: address another test failure
2018-07-17 15:21:07 +03:00
Jan f4b99bb5ed
Bug fix/meierloch (#5895) 2018-07-17 11:39:50 +02:00
Vasiliy 1240090cd8 issue 399.2: use a LogicalCollection& instead of LogicalCollection* in PhysicalCollection (#5861) 2018-07-13 15:02:43 +03:00
Simran 34ec56d421 Feature/misc spelling corrections (#5164) 2018-07-13 13:06:20 +02:00
Tobias Gödderz fc3e11dbbc Async AQL (#5806)
* Modified header to new initializeCursor API

* Adapted initializeCursor to DONE/WAITING API. Compiles but not tested and no one reacts to WAITING state, it is not returned anywhere yet

* Subqueries now expect a WAITING return from initilize cursor. However they will just return a nullptr and pretend the query is empty, this will be fixed later

* First attempt to simulate thread waiting over information within the query

* Small fix to allow for isDirect handlers to go to sleep.

* Waiting in the necessary places now for the async request to be send.

* Thank you auto-casting compiler, you are totally right i absolutely wanted to use this bool value as an index in may Array. How could i possibly not want to use it here?

* Include cond-var header

* Fixed mutex/cond_var usage

* Added oldAPI wrappers in AQL Blocks for get/skip some variants. This Commit compiles but is NOT tested

* Let getSome now return unique_ptr of AqlItemsBlocks. Also implemented the async variant of getSome in subqueries.

* Removed all references to OLD implementations in AQL. only the base wrappers are allowed to call OLD functions from now on. Now the testing part starts

* Fixed endless virtual recursion

* Implemented new getOrSkip API in SortBlock

* Implemented new getOrSkip API in LimitBlock

* Initilaize all variables

* Fixed logic bug in SubqueryBlock

* getBlock in ExecutionBlock now returns a state. All blocks need to handle this properly!

* Createad a wrapper getBlockOld that servers the old sync api and is used now in AQL. To be replaced overtime.

* Added IndexBlock::skipSome and IndexBlock::getSome

* getBlock now returns its old return value along with the state

* Switch from getBlockOld to getBlock in IndexBlock::skipSome

* Switch from getBlockOld to getBlock in IndexBlock::getSome

* ShortestPathBlock::skipSome is not implemented! Added a regression test

* Attempt to fix SubQueryResult memory management

* Fixed LIMIT Block

* Moved from ShortestPathBlock::getSomeOld to ::getSome

* Implemented ASYNC api on SingletonBlock

* Adapted EnumerateCollectionBlock to new async API

* Fixed FilterBlock and adapted return block to async API

* Adapted NORESULTS block to async AQL api.

* Adapted Modification Blocks to async API

* Fixed some initialize cursor functions to reset values required during get/skipSome

* First steps to adapt ClusterNodes to Async AQL api. Not there yet, need to implement the core still

* Added asnyc implementation for xxxForShard in ClusterBlocks. This commit changes internal logic of _doneForShard. Needs additional testing as soon as everything is in place.

* Adapted CalculationBlock to async API

* Adapted TraversalBlocks to ASYNC Aql. This is not optimal yet, we need a better decission if we are DONE or not on RETURN

* Adapted EnumerateListBlock to Async AQL api

* Adapted RemoteBlock to ASYNC API in getSome/skipSome. The whole thing is now LIVE in the cluster. Exetensive testing to be started now

* Fixed IndexBlock WAITING behaviour if Waiting occurs during a index processing

* Adapted IReasearchViewBlock to ASYNC AQL API

* Fixed SortingGatherBlock in WAITING state.

* Adapted IResearch ExecutionBlockMock to Async API

* Unified the HASMORE/DONE distinction. Code is much more readable now and harder to get incorrect 👍

* Implemented tonly heoretically reachable function of non void function.

* Fixed last commit

* Added inline TODO comments

* fix warning

* Fixed a clearing logic bug in RemoveNodes

* Fixed Error Handling in RemoteBlocks. Also fixed a logic bug (true/false simply has a 50% chance of getting it wrong) in Distribute and Scatter.

* remove unused methods

* Fixed failure test

* implement skipping

* Moved the Query Waiting out of the ExecutionEngine.

* changed one of the collect blocks

* Removed _upstreamState from ExecutionBlockMock, that is in the base-class now

* Added a Test Mock for a an ExecutionBlock that simulates the WAITING/HASMORE/DONE api.

* do not check "hasMore" if not necessary

* Added DistinctCollectBlock::getOrSkipSome from ~Old and changed its return type

(still uses getBlockOld)

* Save state to resume in DistinctCollectBlock::getOrSkipSome

* Extracted redundant code

* fixed some ops

* added one more test

* fix endless blocking

* fix compile error

* fix test

* Refactored HashedCollectBlock::getOrSkipSome

* Return blocks to the manager

* Replaced usage of getBlockOld in HashedCollectBlock::getOrSkipSome

* remove unused shutdown calls, simplify ownership for expressions

* Removed superfluous variable

* Capture const variable by value

* Removed SortedCollectBlock::getOrSkipSomeOld in favour of getOrSkipSome

* Added a working version of SortedCollectBlock::getOrSkipSome

Has yet to be cleaned up

* Removed isTotalAggregation special treatment

* On no input, return a group of nulls (instead of no group at all)

* Bugfixes

* Simplified code

* Move return to the end, eliminate duplicate code

* Corrected skipped count in HashedCollectBlock

* Aligned getNextRow() implementations

* Added comments

* some cleanup

* fix potential memleak

* Bugfix

* Fixed failure tests

* Removed usage of getBlockOld in ExecutionBlock::getOrSkipSome

* Replaced hasMore with an async implementation (mostly)

* Removed getBlockOld()

* Added hasMoreState to the AQL API (and renamed hasMore methods to hasMoreState)

* RemoteBlock now uses the async hasMoreState route

* remove job queue

* options

* Bugfixes in the async implementation of LimitBlock

* LimitBlock::getOrSkipSome now always skips when calculating the fullcount

* fix compile warnings

* restrict threads

* Fixed api of Waiting ExecBlockMock. Unused yet

* Made SortedGatherBlock async-capable

* Removed nonEmptyIndex hack

* Removed duplicate traceGetSome~ calls, moved all to getSome

* Added asserts before replacing getNr*Registers

* Added a TODO note and a comment

* Removed getSomeWithoutRegisterClearoutOld()

* Removed skip()

* Removed common code by using getNr*Registers()

* Use getNr*Registers() in the TraversalBlock as well

* started to add lane

* started to add lane

* added lane

* completed lane

* removed debug output

* fixed merge

* Began working on a test suite for AQL tracing/profiling

* Added more tests and asserts in aql-profiler

* Made some ExecutionBlocks final

* Added a type enum to all blocks and the per-block stats

* Add block type to stats nodes when tracing AQL on block level

* Removed initializeCursor call from instantiateFromPlan

* Avoided additional getSome calls after DONE

* Added more profiler tests

* Refactored ExecutionBlock::getOrSkipSome and fixed two bugs

- set _upstreamState also when skipping
- explicitly use xecutionBlock::getHasMoreState()

* Bugfix: update state

* Reuse parent _skipped wherever possible; rename where not (LimitBlock)

* Simplified SortedCollectBlock::getOrSkipSome and reused general pattern & code

* Implemented missing virtual function (with USE_FAILURE)

* Reset neccessary values during initializeCursor

* Simplified code in EnumerateListBlock a little

* Added a test for DistinctCollectBlock in aql-profiler

* Avoid redundant getSome calls in DistinctCollectBlock

* fix compilation

* Fixed DistinctCollectBlock profiler test

* Added a second profiler test for the DistinctCollectBlock

* Added a profiler test for EnumerateCollectionBlock

* Bugfix in EnumerateListBlock

* added --server.fifoN-size

* Simplified EnumerateCollectionBlock::getSome

* Simplified EnumerateCollectionBlock::getSome, and return HASMORE less often when DONE

* Fix testEnumerateCollectionBlock1 for mmfiles

* do not pass by reference

* Fixed compile error

* fixed merge conflicts

* Added profiler tests for EnumerateCollectionBlock

* Test fix for mmfiles

* Fixed IResearch tests

* Bugfix in DistinctCollectBlock and a regression test

* Updated comment

* Bugfix for query statistics in cluster

* Check plan in distinct test

* Fix aql-profiler tests in cluster

* Removed unused line / bugfix for single server test runs

* This commit implements waking up of AQL queries. (#5651)

* Non-compiling intermediate commit for handover.

* Make branch compile again

* Started implementation of continueable rest cursor handler by moving the callbacks to the outer part. This is not yet fully tested!

* Made finalizeExecute noexcept. We cannot react to this errors as the response was potentially written before. Also introduced continueExecution in the RestHandler engine.

* First successful query wakeup.

* The wakeup callback now posts on the scheduler directly. A resthandler only needs to provide a callback that encapsulates the continueExecution call on this handler

* renamed finalizeExecute to shutdownExecute

* Added a differentiation between Handler and Callback in Query continuation. Handler will be posted in IO service. Callback will be executed directly

* fix audit log

* Removed callback from deleteQueryCursor. This cannot be waiting

* use CONDITION_LOCKER

* removed yet another thread-local variable

* Fixed forward declaration

* Made RestAqlHandler repeatable

* Use defer to close the query in RestAqlHandler. Now waiting will close the query as well.

* Added a mutex in the RestHandlers to make sure if the callback over network is too fast that there is only one Thread running in the RestHandler

* Captured the GeneralCommTask if it is posted to a RestHandler. This is necessary in the PAUSED case

* Refactoring of _noLockHeader responsibilities. Now the BaseHandler selects them and resets them after it is done. Only Coordinators are allowed to define them if a query is loaded.

* Removed reaction to existing nolockheaders in Coordinator Query Planning Phase

* Removed incorrect assertion.

* Further refactoring of NoLockHeaders. Now there is a wrapper class around it which allows for debugging and logging. The state now seems to be better. Also all non-rest-handler triggered queries clean up the NoLockHeaders properly.

* Fixed UserManager, now deletes nolock headers properly

* Swing to the Symphony of Destruction

* Forgot about community build...

* Fixed compiling of Catch tests

* Fixed community build

* need thread for size

* Made the restSimpleHndler repeatable

* Implemented dump and dumpSync in Cursors, Sync will block a thread, dump allows to wait, only relevant for Streaming cursor

* Reactivated StreamingCursors

* Removed debug output.

* Fixed false query continuation

* Reset thread output to non-debug

* Added missing return statements

* Allow some CollectionMethods to hand-in a context that may contain a transaction. This is meant to honor nolock headers.

* Fixed hidden merge conflict

* Bugfix in aql-profiler.js: use plan.nodes order, not stats

* Added two profiler tests for filter

* Avoid too many getBlock calls in the FilterBlock

* Removed debug output

* RemoteBlock API will now send a done(bool) flag whenever we request documents from remote Servers. It is possible that we are DONE and have a result. The pre 3.4.0 API uses exhausted which is exclusive to a result. This API is still implemented for beckwards compatibility.

* Implemented an executeSync function in AqlQuery. This will block the thread until query execution is complete

* Added another test for FILTER, and one test for the HashedCollectBlock

* Added more tests for HashedCollectBlock; avoid unneccessary getSome calls

* Added an profiler IndexBlock test

* IndexBlock: avoid redundant getSome calls, added missing traceGetSomeEnd calls

* Added a second test profiling IndexBlock

* Added a third test for IndexBlock

* Moved general code to module

* Moved noncluster tests into a separate file

* Split aql-profiler testsuite into three files

* Added profiler tests for LimitBlock

* Added a test for NoResultsBlock

* Added profiler tests for TraversalBlock

* Shutdown of an AQL query is now asynchronous. However in Error-Cases it will be executed in a blocking way still

* Optimized TraversalBlock getSome calls due to new (nightly) test results

* Fixed std::min calls I broke

* Let shutdown calls in AQL wait, if the query is executed successfully.

* Fixed queryResult going out of scope

* fix compile error through merge conflict with devel

* Fixed compiler warning "mismatching tags"

* Removed debug log output

* Added TODO notes

* Fixed test fail due to devel merge

* Fixed some invalid sync waiting implementations

* Added a profiler test for SortBlock

* Added profiler tests for SortedCollectBlock

* Fixed bug introduced by devel merge

* Fixed Remoteblocks ignoring errors!

* Added some more continue Callbacks in used places. And removed debug log

* Removed debug log output

* Suppress clang warnings

* Bugfix: use of invalid stack pointer

* Bugfix: RemoteBlock::shutdown now sends code as int, not string

* Revert "Suppress clang warnings"

This reverts commit 05591649c59743c992edd5e78814edc8ca2a83e0.

* Bugfix: cleanup state in RemoteBlock ::shutdown, ::getSome and ::skipSome

* Bugfix in Subquery shutdown: don't skip subquery shutdown when main query shutdown failed

* Allow copy elision
2018-07-09 14:24:10 +02:00
Simon f47367a495 Allow dumping / restoring of views (#5774)
* allow dumping and restoring of views

* fix restore, adding some tests

* remove debug print

* fix global replication
2018-07-05 17:55:28 +02:00
jsteemann fe9b2d5bb5 honor cppcheck comments 2018-07-04 22:25:33 +02:00
Dan Larkin-York 820bfee329 Refactor syncer state and make notes for future parallelization (#5742) 2018-07-03 21:32:16 +02:00
Jan b88fde9659
added key generators "uuid" and "padded" (#5740) 2018-07-03 15:23:47 +02:00
Dan Larkin-York 21e16a8a24 Add load balancer awareness for cursor API (#5682) 2018-07-03 14:29:09 +02:00
Simon 545561e9a9 Read only server (#5652) 2018-07-03 09:58:16 +02:00
Vasiliy 7aaeab50fb issue 402.1: share sync thread between IResearchView and IResearchViewDBServer (#5733) 2018-07-02 15:03:00 +03:00
Simon ec0d2a1b7b Remove Coordinator DBs (#5661) 2018-06-25 19:18:11 +02:00
Andrey Abramov 5eef6cd618
Feature/test iresearch (#5610)
* start implementing arangosearch cluster tests.

* backport: ensure view lookup is done via collectionNameResover, ensure updateProperties returns current view properties

* first attempt to fix failing tests

* refactor cluster wide view creation logic

* if view is not found in the new plan then check the old plan too

* ensure the cluster-wide view is looked up in vocbase as well on startup/recovery

* do not store cluster-wide IResearchView in vocbase

* move stale view cleanup to the shared pointer deleter, address test failures

* do not print warning

* enable arangosearch tests by default

* fix catch tests

* address icorrect return value for cluster-wide links

* address some issues with test failures due to cluster-view allocated within TRI_vocbase_t

* simplify per-cid view name, address 'catch' test failures

* ensure IResearchViewNode volatility is properly calculated in cluster

* invoke callbacks directly in AgencyMock instead of waiting for timeout

* ensure view updates via JavaScript always use the latest view definition

* pass a list of shards to `IResearchViewDBServer::snapshot`

* extend cluster aql tests

* fixes after merge

* fix class/struct inconsistencies

* comment failing tests

* remove debug logging

* add debug function

* tests cleanup

* simplify upcoming merge: pass resolver from a side

* backport: move all transaction status callback logic to Methods

* add changes missed from previous commit

* fix js and ruby tests

* more tests for IResearchViewNode

* pass transaction to IResearchViewDBServer::snapshot, address IResearchViewDBServer tests segfault

* pass transaction to IResearchView::snapshot instead of transaction state

* temporarily add trace log output to tests to try to find the cause of the core dump on Jenkins

* add more temporary debug output to trace down the segfault on Jenkins

* add even more temporary debug output to trace down the segfault on Jenkins

* ensure Vieew related maps are cleared during shutdown

* reset ClusterInfo::instance() before DatabaseFeature::unprepare()

* remove extraneous debug output

* missed line from previous commit

* uncomment required line

* add nullptr checks to RocksDBIndexFactory::prepareIndexes(...) similar to the ones in MMFilesIndexFactory::prepareIndexes(...)

* attempt to fix deadlock in tests

* add comment as per reviewer request

* fix aql test suite name

* add some debug logging

* address deadlock between ClusterInfo::loadPlan() and CollectionNameResolver::localNameLookup(...)

* eplicitly state which index definition failed in the log message

* use vocbase from shard-view isntead just in case

* explicitly state which index definition failed in the log message

* do not create shard-view instances from cluster-link instances (only register existing ones)

* add some tests
2018-06-21 20:35:04 +03:00
Jan 5ce22a3bc9
make API for POST /_api/aqlfunction downwards-compatible again (#5636) 2018-06-20 12:52:54 +02:00
Simon 3bec336aff TransactionState::addCollection refactoring (#5606) 2018-06-14 15:34:58 +02:00
Vasiliy a5f274234a issue 380.2.2: remove extraneous pointers from StorageEngine API (#5589) 2018-06-13 15:22:51 +03:00
Jan 448a435713
clean up key generators a bit (#5573) 2018-06-12 11:28:38 +02:00
Heiko 743d4430bf added replication ui (#5555) 2018-06-08 11:19:05 +02:00
Vasiliy 4253dca6aa issue 381.5: ensure the LogicalView definition that is persisted to the Agency matches the definition that gets created (#5518)
* issue 381.5: ensure the LogicalView definition that is persisted to the Agency matches the definition that gets created

* backport: correct comment
2018-06-02 17:21:55 +03:00
Andrey Abramov f708e46973
Feature/arangosearch gather node (#5487)
* integrate ArangoSearch into GatherNode/GatherBlock logic

* fix inconsistency after merge

* add serialization/deserealization logic for IResearchViewNode sorting

* fix optimizer rule order

* fix copy/past typo

* fix tests

* ensure `GatherNode` is properly set up for ArangoSearch views

* adjust optimizer rule order
2018-05-30 02:53:21 +03:00
Jan e6dcff8360
Bug fix/cleanup 2805 (#5477) 2018-05-29 10:16:48 +02:00
Jan cee529727c
Squashed commit of the following: (#4092) 2018-05-26 10:56:26 +02:00
Jan Christoph Uhde a2dcb6cc5d WIP - start adding optional overwrite to insert operation (RepSert) (#5268) 2018-05-24 19:47:15 +02:00
Andrey Abramov 0489322cd1
Distributed ArangoSearch query execution (part 2) (#5397)
* further implementation of arangosearch query distribution

* do not store `aql::Collection` in `IResearchViewNode`, use `Query::collections()` instead

* some fixes

* cleanup

* some fixes after review

* more fixes after review

* use assertion instead of condition
2018-05-23 19:29:18 +03:00
Simon 332a7958f5 Cleanup cluster selectivity (#5440) 2018-05-23 18:00:14 +02:00
Jan 8e6d5df129
fixed minor several compiler complaints (#5406) 2018-05-23 11:50:00 +02:00
Jan 30d0d840c2
Bug fix/validate from to consistently (#5434) 2018-05-23 11:41:30 +02:00
Vasiliy 94ddd7803d issue 389.10: refactor CollectionNameResolver to use TRI_vocbase_t& (#5424) 2018-05-23 00:59:08 +03:00
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00
Jan df23ed092e
Bug fix/cleanup 1705 (#5376) 2018-05-18 09:21:50 +02:00
Vasiliy 843e584746 issue 389.5: refactor StandaloneContext to be constructed with a TRI_vocbase_t& (#5370)
* issue 389.5: refactor StandaloneContext to be constructed with a TRI_vocbase_t&

* backport: address build issues
2018-05-17 01:15:50 +03:00
Jan 0d98c0f3ba
Bug fix/fix codescan issues (#5371) 2018-05-16 20:40:37 +02:00
Vasiliy 6a53154160 issue 389.2: use static strings for Index definition json attributes, use TRI_vocbase_t references instead of pointers in V8Context, use TRI_vocbase_t references instead of pointers in DatabaseInitialSyncer (#5344) 2018-05-14 19:06:24 +03:00
Simon 17b1a2aafb Rest middleware refactoring (#5332) 2018-05-14 17:43:10 +02:00
Andrey Abramov 671734b32f
do not scatter empty view (#5318)
* do not distribute scatter views without links

* simplify view downcasting

* refactor & add some tests

* adjust comments
2018-05-11 11:45:22 +03:00
Vasiliy 0d1cf45097 issue 373.3: use TRI_vocbase_t& for Upgrade tasks, remove redundant checks for null TRI_vocbase_t (#5301) 2018-05-09 15:54:07 +03:00
Simon 08e355aed8 Simple dump speedup (#5298) 2018-05-09 12:51:04 +02:00
Vasiliy 5d8abce7bb issue 373.2: move toVelocyPack into LogicalDataSource (#5288)
* issue 373.2: move toVelocyPack into LogicalDataSource

* backport: move static DataSource related strings into StaticStrings, add support for registering snapshots on DBServer views
2018-05-08 17:28:39 +03:00
jsteemann 7f8a1cc614 Merge branch 'bug-fix/add-missing-overrides-and-final' of https://github.com/arangodb/arangodb into devel 2018-05-07 23:02:46 +02:00
Vasiliy 2ce20e86d7 issue 373.1: move globally-unique id generation from collection into data-source (#5182) 2018-05-07 22:14:40 +03:00
Andrey Abramov b69b5bdfdf
Bug fix/issue #5186 (#5269)
* do not persist view on startup

* small refactoring

* ensure view is being opened after creation
2018-05-06 20:38:32 +03:00
jsteemann 52de92d334 add missing override specifiers, add final specifiers 2018-05-04 09:01:50 +02:00
Simon 828f1d423c S2 based Geo-Spatial index (#5249) 2018-05-02 23:54:41 +02:00
Andrey Abramov 4649b40b96
Coordinator ArangoSearch view + Execution nodes + AgencyMock (#5160)
* add initial implementation of scatter view rule and node

* add tests for `IResearchViewNode` and `IResearchViewScatterNode`

* add missing check

* modify IResearch execution nodes to use references instead of pointers

* use view id in searialized `ExecutionNode` representation instead of the name

* add cluster mode stubs and checks

* very first attempt to distribute IResearchViewNode

* further implementation of cluster-wide arangosearch views

* fix invalid json format

* add tests for coordinator iresearch view

* allow to retrieve a list of existing views on a coordinator

* more tests for coordinator iresearch view

* some fixes to enable query explanation

* remove Collection dependency from RemoteNode

* remove unnecessary remote ArangoSearch view scatter

* fix explanation appearance

* add some assertions

* minor fixes

* implement IResearchViewCoordinator::updateProperties

* fix view DDL issues

* handle link modifications in DDL operations

* add coordinator implementation of iresearch view links

* fix tests

* further coordinator based view DDL implementation

* further IResearchViewCoordinator implementation

* add initial implementation of AgencyMock

* fix some tests

* code cleanup

* extend test + some fixes

* more tests for IResearchViewCoordinator

* fix tests for IResearchLinkCoordinator

* some fixes after merge

* fix tests

* remove declaration of nonexistent (previously removed) method

* some fixes after review

* remove string duplication

* more tests and fixes

* more fixes and tests

* more tests

* one more test

* fix 'use-after-free' asan error

* fix non-enterprise tests issues
2018-05-02 00:15:11 +03:00
Simon a1416e1067 Make v8 optional on startup (#5220) 2018-04-30 12:48:57 +02:00
Jan 30b12e311b
Bug fix/remove most of aql js (#5223) 2018-04-30 11:17:11 +02:00
Vasiliy 012aaa9469 issue 383.4: push vocbase validity check up from Query constructor out into arangodb::consensus::State, StatisticsWorker and AQLUserFunctions calls (#5177) 2018-04-23 14:52:42 +03:00
Simon 45fbed497b Supervision Job for Active Failover (#5066) 2018-04-23 12:49:41 +02:00
Vasiliy 9062c41592 issue 383.3: implement remainder of IResearchViewDBServer tests, use the data-source id (primary key) instead of an arbitrary instance for dropCollection()/dropView(), backport from iresearch upstream: ensure block is flushed if key index is full (#5176) 2018-04-23 00:33:46 +03:00
Vasiliy 6053056b6a issue 383.1: initial stub implementation of IResearchViewDBServer (#5132) 2018-04-18 13:51:21 +03:00
Vasiliy 4c566a6889 backport: explicitly document implicit behaviour 2018-04-13 15:22:18 +03:00
Vasiliy f392925903 issue 374.3: use a reference to vocbase instead of a pointer in DatabaseGuard 2018-04-13 09:56:49 +03:00
Vasiliy d470371f7c issue 374.2: use a reference to vocbase instead of a pointer in StorageEngime LogicalDatasource related DDL operations, use vocbase from the found collection 2018-04-11 18:49:46 +03:00
Vasiliy d1ce3a97ef issue 355.7: ensure LogicalDataSource::vocbase() returns a reference 2018-04-09 15:38:24 +03:00
Vasiliy e4368b0991 issue 355.6: remove create() from LogicalView, remove IResearch dependency from IndexFactory, store vocbase reference in LogicalDataSource 2018-04-06 16:38:34 +03:00
Vasiliy 99b83ba8c8 issue 355.5: remove more unused methods, move view-related storage engine functionality from vocbase into DBServerLogicalView, address MSVC cmake dependency issue 2018-04-05 16:17:07 +03:00
Andrey Abramov f6c27ce073
Merge pull request #5017 from arangodb/bug-fix/internal-issue-#355.4
issue 355.4: remove redundant methods and code, use 'cp' instead of 'cmake copy_directory' where possible, use vocbase reference instead of pointer
2018-04-05 14:27:44 +03:00
jsteemann c5660ac54e remove useless warnings 2018-04-04 17:16:15 +02:00
Vasiliy 635db3b409 issue 355.4: remove redundant methods and code, use 'cp' instead of 'cmake copy_directory' where possible, use vocbase reference instead of pointer 2018-04-04 10:53:48 +03:00
Jan e567392e49
improve startup resilience in case there are datafile errors (#4975) 2018-04-03 17:32:30 +02:00
Vasiliy 7c25902b27 issue 355.3: allow IResearchLink creation, as opposed to IResearch View update, to direct which CIDs get included in an IResearchView snapshot 2018-04-02 15:27:48 +03:00
Andrey Abramov 2cca7a926b remove code duplication 2018-03-30 16:01:00 +03:00
Andrey Abramov 57b5823fcf some fixes after review 2018-03-30 14:34:15 +03:00
Andrey Abramov 3eadf5aaf9 add convenient factory method for views 2018-03-29 20:19:03 +03:00
Max Neunhoeffer 5d5a44b01b
Do not store name in properties. 2018-03-28 23:29:43 +02:00
Max Neunhoeffer c7b77076a9
Merge remote-tracking branch 'origin/devel' into feature/arangosearch-cluster-views 2018-03-28 23:26:48 +02:00
Andrey Abramov e2a6e751ed do not use dynamic_pointer_cast in release 2018-03-27 21:26:21 +03:00