1
0
Fork 0
Commit Graph

5558 Commits

Author SHA1 Message Date
Andrey Abramov 04bb3da337
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug-fix/internal-issue-#345 2018-03-20 19:04:54 +03:00
Michael Hackstein c1650702bf
Feature/aql server based locking (#4783)
* Started Implementing the ServerBasedlocking. There now is a container that can contain multiple query snippets. It now has to setup the necessary calls to the Servers

* Added backwards linking of QueryEngines, sth. DBServers can contact their Coordinators.

* Added LogTopic AQL

* Made AccessMode::Type Hashable

* Created a Mapping Server => LockLevel => Shard and createad a JSON object containing the Lock information for a complete AQL query per server

* Added code to build coordinator engines

* Finished with first draft of Coordinator-side of new DBServer based locking.

* Added a _api/aql/setup route that creates and locks all snippets/collections for one DBServer in a single go

* Fixed some Coordinator parts

* Index node now gracefully reports if it could not find it's collection when created from vpack. Otherwise it just hardly crashed...

* Modified the Coordinator Snippet collector to be able to handle subqueries properly.

* Started adding GraphNode handling. WIP. Need to deploy engines properly. Coordinator crashes on Graph tests

* Fixed compiler errors

* WIP: EngineInfoContainer

* Separated the EngineInfoContainers for Coordinator and DBServer into different files. They diverged more than anticipated

* Added forgotten files. THe DBServer container now creates the TraverserEngine Mapping and moves it into the Infos. They are not keeping it yet and need to add it to the message as well.

* The DBServer engine infos now persist the TraverserEngine infos. Need to add them to messages though.

* The new aql exec-engine now sends out traverserEngines as well

* Formatting and adding DEBUG level output

* Made the RestAQLHandler aware of the TraverserEngineRegistry. Also created the engines now. Return format changed server-side coordinator side needs fix.

* Adapted the Coordinator side for the DBServer based Shard Locking

* The DBServer based Locking now honors restrictions to certain shards

* Fixed a strange double lock bug in the new AQL Server based locking technique. Add some DEBUG output

* Fixed usage of MAINTAINERMODE macro. The assertion was never active

* Added TestCase for ContainerCoordinatorTest to cmake

* Added -DTEST_VIRTUAL to CMAKE. This is used to define virtual functions for mocking ONLY on test-builds.

* Fixed usage of ENABLE_MAINTAINER_MODE ifdef. CLANG format

* On non-enterprise builds ENTERPRISE_VERT defaults to TEST_VIRTUAL => virtual in test else non-virtual

* Added TEST_VIRTUAL to ExecutionEngine, Query and QueryRegistry

* Added first testcase for EngineInfoContainerCoordinator not yet ready.

* Mode CreateBlock a member function of engine, we have the engine in our hands anyways no need to make it static. Included some more TEST_VIRTUAL functions.

* Fixed clang/MacOs compile error. Added some more TEST_VIRTUAL declarations

* Finally fixed the first buildEngines UnitTest \o/

* Added a unit-test for backward linking of dependencies in CoordinatorPlanner

* Added multi-snippet test for EngineInfoContainerCoordinator

* Removed QueryRegistry.h from central header files and replaced by a forward declartion.

* Added a createBlocks method on the ExecutionEngine. It should be responsible to create all those blocks at once. Adapted the UnitTests as well. Not included Tests for the new createBlocks functionality. Need to mock the options feature first

* Added another test that Coordinator Snippets of queries can be created correctly

* Fixed Coordinator-site cleanup of QueryRegistry, if any of the query creations fails with error, incl UnitTest.

* Added first test for RestAqlHandler::setup. It does only test the setup and gives prepartion for real testing.

* Added a assertion of http return code. Still no creation of queries is tested. Requires a huge amount of mocking.

* fix some deadlocks found by evil lock manager (tm)

* fix duplicate lock

* fix indentation

* ensure proper lock dependencies

* fix lock acquisition

* removed useless comment

* do not lock twice

* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not

* AQL micro optimizations

* use explicit constructor

* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context

entering and exiting the selected context is then done later on without having to hold the ConditionLocker

* remove some recursive locks

* Disable custom deadlock detection when Thread Sanitizer is enabled

* Changing ifdef's

* grr

* broke gcc

* Using atomic for ApplicationServer::_server

* fix premature unlock

* add some asserts

* honor collection locking in cluster

* yet one more lock fix

* removed assertion

* Allow the clustercomm to send nolock headers on count. This is used form within AQL

* IsLocked on transactions will now always yield true IF LOCK_NEVER is set. We simply assume someone else holds the lock for us. Also LOCK_NEVER is now set on collection/count if noLock header is send.

* Moved the flag if collections need to be locked into the TraverserEngines.

* Added enterprise-satellite hooks in EngineInfoContainerDBServer

* Removed now obsolete code

* Replaced throwing of Exception by an ResultObject

* Added some more tests and moved adding snippet to query engine more to the outside.

* Added the AQL result type

* Make the branch compile again

* Register WITH collections for Graphs in the new Collector.

* Fixed test code for failing query clone. Idea was to once clone successfully and second time to fail, we verify that first clone is cleaned up properly. However test failed on first clone...

* Removed a double builderClose

* Added Changelog entry

* Removed empty if

* Removed obsolete todo

* Properly initialize the AqlResult with nullptr on error case

* Updated comment

* Simplified Assertion

* Removed debug output object...

* Added additional catch case for std::exception to get some more error info

* Clarified evaluation order for move case

* Added Explicit

* Fixed cleanup of Coordinator if Registry fails to insert query.

* Allow to use other locks than Read/Write for AQL collections. Not yet in API.

* Updated Comments for other Locks on DBSide. Adapted Destruction CatchTests

* Fixed double builderClose and removed unnecessary double commits

* Added a comment to clarify the state

* Moved error output to trace. Leftover from debugging

* Added some tests for complex subquery patterns

* Added a 'fireAndForgetRequests' methods to cluster comm which allows to send out a bunch of messages but does not wait for their results

* Properly cleanup leftovers of queries if the instantiation step already failed

* Added code comment for fireAndForget

* Added indexes to subquery test to make the plan a bit easier

* The cleanup on DBServerEngines in error case now also cleans up traverser engines.

* Removed unnecessary includes

* Removed debug logging

* Fixed hidden merge conflict
2018-03-20 16:52:19 +01:00
Andrey Abramov 294c36f778 use ERROR_ARANGO_DATA_SOURCE_NOT_FOUND instead of ERROR_ARANGO_COLLECTION_NOT_FOUND/ERROR_ARANGO_VIEW_NOT_FOUND in js part too 2018-03-19 13:55:25 +03:00
Max Neunhöffer 793101528f Repair resilience-sharddist test by increasing timeout. (#4857) 2018-03-15 13:29:55 +01:00
Jan aad64bfdd7
fixed issue #4843: Query-Result has more Docs than the Collection itself (#4855) 2018-03-15 10:52:38 +01:00
Jan b325d05357
Bug fix/aql auto keep on collect (#4795)
* automatically detect which variables to keep in AQL COLLECT

If a COLLECT INTO is used, then it is detected which sub-attributes
of the into variables are used later in the query, and automatic
KEEP instructions are added to the COLLECT if possible

Example query:

    FOR doc1 IN collection1
      FOR doc2 IN collection2
        COLLECT x = doc1.x INTO g
        RETURN { x, all: g[*].doc1.y }

would be turned into

    FOR doc1 IN collection1
      FOR doc2 IN collection2
        COLLECT x = doc1.x INTO g KEEP doc1
        RETURN { x, all: g[*].doc1.y }

and prevent `doc2` from being temporarily stored in the variable `g`

* fix runtime warnings

* increase default flush timeout

* added tests
2018-03-14 13:59:46 +01:00
Jan 2f47cdcaf9
fixed issue #4827: COLLECT on edge _to field doesn't group distinct values as expected (#4830) 2018-03-13 13:40:26 +01:00
Jan 67e7e4181e
Bug fix/remove v8 executor (#4756) 2018-03-13 10:41:02 +01:00
Wilfried Goesgens 87edf275e3 implement SPLIT Aql-Function in C++ (#4786) 2018-03-09 09:12:30 +01:00
Simon 272859c5fd Replacing js upgrade logic (#4061) 2018-03-08 13:57:30 +01:00
Andrey Abramov 29b7a7ea44 remove logger view (#4764) 2018-03-08 09:18:21 +01:00
Mark 292117e3cf Bug fix devel/bfs filter vertices (#4752) 2018-03-08 09:06:15 +01:00
Jan 46a56792ed
fix `fullCount` value in some cases in cluster mode (#4723) 2018-03-01 18:33:10 +01:00
Simon 345fc3c0b7 Refactor Authentication Layer (devel) (#4592)
* Cherry Picking LDAP changes

* Adding missing merges

* Fixing remaining mentions of FeatureCacheFeature

* Fix jslint

* Fixing some failed tests

* Fixing cluster authentication issue, red tests

* Fixing ldap testsuite, adding trace logging

* Fixint ldap tesuite setup and LDAP recognition

(cherry picked from commit 686d28a779)

* Fixing wrong assert

* Adding changelog entry, making requested changes from code review

* Fixing dump_authentication, fix typos

* improvements found during code review

* oops

* more use of sessionstorage

* fix tests

* Fixing broken handling, disallowing adding of local users when disabled

* Fixing testInvalidGrants

* Removing undefined auth level externally

* Fixing previous commit

* added tests for ldap search mode

* intentionally removed `after` methods from tests

because they are executed before the tests start
no cleanup is performed right now after the authentication tests
however, a cleanup is done at start of every test

* ldap tests all modes

* forward port changes from 3.3

* added generated files

* forward port missing changes for web UI

* added generated files

* added generated files
2018-02-28 13:24:28 +01:00
Jan 9ca52c0cf7
fixed issue #4651 (#4659) 2018-02-28 12:56:07 +01:00
Vasiliy 423ce33ed1 Changes since last PR: remove 'dataPath' IResearchView configuration property and use a hardcoded path similar to a LogicalCollection (#4683) 2018-02-27 18:10:32 +01:00
Jan da1dd2b063
fix issue #4677 (#4679) 2018-02-27 11:29:09 +01:00
Jan Christoph Uhde 586a66ebbf Fix: #4583 - adds AQL ASSERT and WARN (#4584) 2018-02-26 14:25:24 +01:00
Mark cd97787adf Fix Foxx API (#4614) 2018-02-20 10:22:27 +01:00
jsteemann aa5e02c82d make syncer more compatible and ignore revs if not present 2018-02-19 18:44:08 +01:00
Jan 63c50c3993
Bug fix/handle single delete (#4618) 2018-02-19 14:49:58 +01:00
Wilfried Goesgens bda5c79bce Implement FIND_FIRST and FIND_LAST in c++ (#4608) 2018-02-19 10:41:21 +01:00
Wilfried Goesgens 101a9893a4 migrate r/l-trim, split trim so we can reuse that code. (#4578) 2018-02-16 14:10:00 +01:00
Vasiliy 828f8fb48c locally tested only, Changes since last PR: ensure parser.cc dependency is tracked in the build directory to avoid bison invocation on MSVC, aquire IResearchView snapshot at transaction begin() and reuse it for all queries, remove unused callback registration from Methods (#4564) 2018-02-14 12:32:29 +01:00
jsteemann a0ace6ef73 fix jslint 2018-02-13 12:21:20 +01:00
Wilfried Goesgens 619155a62d Feature/aqlfunction to cpp (#4573) 2018-02-13 11:50:49 +01:00
Wilfried Goesgens 16648fc628 fix windows compile (#4557) 2018-02-11 18:56:13 +01:00
Dan Larkin 934e936bae Fix index selectivity estimate issues with RocksDB engine (#4429) 2018-02-09 16:05:33 +01:00
Simon 35136a89c0 Fix some problems with active failover (#4540) 2018-02-09 15:11:53 +01:00
Jan 8da61a9875
Bug fix/more replication tests (#4500) 2018-02-09 13:06:47 +01:00
Jan 4f872b62ee
add missing edge filter NeighborsEnumerator (#4547) 2018-02-08 15:15:29 +01:00
Jan 0eae1d225f
fix internal issue #1848 (#4521)
AQL optimizer was trying to resolve attribute accesses
to attributes of constant object values at query compile time, but only did so far
the very first attribute in each object

this fixes https://stackoverflow.com/questions/48648737/beginner-bug-in-for-loops-from-objects
2018-02-08 09:23:15 +01:00
jsteemann 24da298f0a removed useless explain 2018-02-08 09:01:03 +01:00
Alan Plum 232abd4bd0 Fix legacy Foxx HTTP API compat layer (#4265) (#4449) 2018-02-05 12:35:51 +01:00
Mark 599da158b5 Bugfix Foxx API tests (#4446) 2018-02-05 11:31:27 +01:00
Alan Plum 58cfd340cc Don't greedily delete legit app folders (#4437)
See #4395
2018-02-05 11:30:05 +01:00
Vasiliy aa041ddfb9 locally tested only (#4476) 2018-02-05 11:24:34 +01:00
Dan Larkin bf606bdf89 Rename more '-iresearch' tests to '-arangosearch'. (#4477) 2018-01-31 18:01:04 +01:00
Dan Larkin a25384fe6c Rename -iresearch files to -arangosearch for proper test selection. (#4386) 2018-01-31 17:25:59 +01:00
Jan Christoph Uhde fd166d07f4 try to fix windows build (#4468) 2018-01-31 10:09:39 +01:00
Alan Plum 05d0c81d7d More consistent config/deps output in Foxx API (devel) (#3792)
* Correctly expose parsed/raw config

This makes accessing the configuration more consistent by distinguishing between the raw (assigned) and parsed (validated/default) values.
Previously there was no way to recognise default values for unset config options and the "current" value did not actually reflect what the service sees.

* Add minimal option to config/deps routes

The default values are backwards compatible with the existing behaviour but can be overridden to consistently get more/less output.

* Actually merge warnings into non-mininal response
2018-01-30 16:50:36 +01:00
Michael Hackstein 7a5a9a620c
Bug fix/distribute shards like (#4415) 2018-01-29 13:07:06 +01:00
jsteemann a73408182c fix invalid filename (my bad) 2018-01-25 16:46:15 +01:00
Jan caf409b638
fix issue #4272 (#4419) 2018-01-25 15:57:13 +01:00
Wilfried Goesgens 14c5c3aa35 fix restoring a dump created from a cluster to single server (#4350) 2018-01-24 13:50:27 +01:00
Jan abd3355784
renamed files (#4405) 2018-01-24 09:54:24 +01:00
Jan 1a7177bc3d
added traversal test (#4376) 2018-01-23 15:18:09 +01:00
Simon 3de120d915 Doc - Move cursor documentation out of deprecation folder (#4353) 2018-01-23 11:01:29 +00:00
Andrey Abramov ffbb00bbe2 standardize on the 'analyzer' and 'trackListPositions' naming (#4384) 2018-01-22 16:39:53 +01:00
Max Neunhöffer c221dddad4 Wait for 5 dbservers to be there and GOOD before test starts. (#4364) 2018-01-19 18:37:46 +01:00