* Squashed commit of feature-3.5/hotbackup_devel.
This puts hotbackup into 3.5.
* Port atomic-database-creation-2 to 3.5.
* Remove some wrongly ported code.
* Fix compilation.
* Fix a manual merge error.
* Remove a feature from the mocks which does not exist in 3.5.
* Add some code which was forgotten in manual merge.
* Fix a problem introduced in a manual merge.
* reuse function
* Address some whitespace issues that came up in review
* aardvark should not create the frontend collection
* create _frontend collection from c++
* recheckAndUpdate Callback in CollectionWatcher
* Wrong author ;)
* rm outdated todo
* Update lib/Basics/VelocyPackHelper.h
Co-Authored-By: Michael Hackstein <michael@arangodb.com>
* use logger unique id, use startup logger
* not needed
* optimized vector shardid method
* do not create _modules collection lazy anymre
* Formatting.
* Assert instead of if/TRI_ASSERT(false)
* Don't use exceptions as control structure
* Re-add READ_LOCKER that got lost in translation
* Fix audit log in case database creation fails early.
* legacy sharding
* Add CHANGELOG entry.
* Retry database cancellation indefinitely
* Do not use exceptions in UpgradeTask
* DropCollection is a FAST_LANE action and should not need much time or else retry.
* Remove superflous addition of LdapFeature
Proudly brought to you by ASAN tests
* Fixed check for distributShardsLike sharding on _system database
* Fixed compile issue on tests
* Removed assertion that seems to be not correct yet on devel.
* Sort out google cloud storage as remote. (#9918)
* Add successful method to ClusterCommResult.
* Improve error forwarding for cluster internal communication.
* Feature/hotbackup list retries (#9924)
* retry hot backup listing for 2 minutes in cluster before giving up
* Enable api by default.
* fix broken list of non existing id (#9957)
* Fix compilation after manual merge.
* Fix another compilation problem.
* Yet more fixes for compilation.
* More compilation fixes.
- Schmutz now called "Maintenance" and completely implemented in C++
- Fix index locking bug in mmfiles
- Fix a bug in mmfiles with silent option and repsert
- Slightly increase supervision okperiod and graceperiod
* Started Implementing the ServerBasedlocking. There now is a container that can contain multiple query snippets. It now has to setup the necessary calls to the Servers
* Added backwards linking of QueryEngines, sth. DBServers can contact their Coordinators.
* Added LogTopic AQL
* Made AccessMode::Type Hashable
* Created a Mapping Server => LockLevel => Shard and createad a JSON object containing the Lock information for a complete AQL query per server
* Added code to build coordinator engines
* Finished with first draft of Coordinator-side of new DBServer based locking.
* Added a _api/aql/setup route that creates and locks all snippets/collections for one DBServer in a single go
* Fixed some Coordinator parts
* Index node now gracefully reports if it could not find it's collection when created from vpack. Otherwise it just hardly crashed...
* Modified the Coordinator Snippet collector to be able to handle subqueries properly.
* Started adding GraphNode handling. WIP. Need to deploy engines properly. Coordinator crashes on Graph tests
* Fixed compiler errors
* WIP: EngineInfoContainer
* Separated the EngineInfoContainers for Coordinator and DBServer into different files. They diverged more than anticipated
* Added forgotten files. THe DBServer container now creates the TraverserEngine Mapping and moves it into the Infos. They are not keeping it yet and need to add it to the message as well.
* The DBServer engine infos now persist the TraverserEngine infos. Need to add them to messages though.
* The new aql exec-engine now sends out traverserEngines as well
* Formatting and adding DEBUG level output
* Made the RestAQLHandler aware of the TraverserEngineRegistry. Also created the engines now. Return format changed server-side coordinator side needs fix.
* Adapted the Coordinator side for the DBServer based Shard Locking
* The DBServer based Locking now honors restrictions to certain shards
* Fixed a strange double lock bug in the new AQL Server based locking technique. Add some DEBUG output
* Fixed usage of MAINTAINERMODE macro. The assertion was never active
* Added TestCase for ContainerCoordinatorTest to cmake
* Added -DTEST_VIRTUAL to CMAKE. This is used to define virtual functions for mocking ONLY on test-builds.
* Fixed usage of ENABLE_MAINTAINER_MODE ifdef. CLANG format
* On non-enterprise builds ENTERPRISE_VERT defaults to TEST_VIRTUAL => virtual in test else non-virtual
* Added TEST_VIRTUAL to ExecutionEngine, Query and QueryRegistry
* Added first testcase for EngineInfoContainerCoordinator not yet ready.
* Mode CreateBlock a member function of engine, we have the engine in our hands anyways no need to make it static. Included some more TEST_VIRTUAL functions.
* Fixed clang/MacOs compile error. Added some more TEST_VIRTUAL declarations
* Finally fixed the first buildEngines UnitTest \o/
* Added a unit-test for backward linking of dependencies in CoordinatorPlanner
* Added multi-snippet test for EngineInfoContainerCoordinator
* Removed QueryRegistry.h from central header files and replaced by a forward declartion.
* Added a createBlocks method on the ExecutionEngine. It should be responsible to create all those blocks at once. Adapted the UnitTests as well. Not included Tests for the new createBlocks functionality. Need to mock the options feature first
* Added another test that Coordinator Snippets of queries can be created correctly
* Fixed Coordinator-site cleanup of QueryRegistry, if any of the query creations fails with error, incl UnitTest.
* Added first test for RestAqlHandler::setup. It does only test the setup and gives prepartion for real testing.
* Added a assertion of http return code. Still no creation of queries is tested. Requires a huge amount of mocking.
* fix some deadlocks found by evil lock manager (tm)
* fix duplicate lock
* fix indentation
* ensure proper lock dependencies
* fix lock acquisition
* removed useless comment
* do not lock twice
* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not
* AQL micro optimizations
* use explicit constructor
* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context
entering and exiting the selected context is then done later on without having to hold the ConditionLocker
* remove some recursive locks
* Disable custom deadlock detection when Thread Sanitizer is enabled
* Changing ifdef's
* grr
* broke gcc
* Using atomic for ApplicationServer::_server
* fix premature unlock
* add some asserts
* honor collection locking in cluster
* yet one more lock fix
* removed assertion
* Allow the clustercomm to send nolock headers on count. This is used form within AQL
* IsLocked on transactions will now always yield true IF LOCK_NEVER is set. We simply assume someone else holds the lock for us. Also LOCK_NEVER is now set on collection/count if noLock header is send.
* Moved the flag if collections need to be locked into the TraverserEngines.
* Added enterprise-satellite hooks in EngineInfoContainerDBServer
* Removed now obsolete code
* Replaced throwing of Exception by an ResultObject
* Added some more tests and moved adding snippet to query engine more to the outside.
* Added the AQL result type
* Make the branch compile again
* Register WITH collections for Graphs in the new Collector.
* Fixed test code for failing query clone. Idea was to once clone successfully and second time to fail, we verify that first clone is cleaned up properly. However test failed on first clone...
* Removed a double builderClose
* Added Changelog entry
* Removed empty if
* Removed obsolete todo
* Properly initialize the AqlResult with nullptr on error case
* Updated comment
* Simplified Assertion
* Removed debug output object...
* Added additional catch case for std::exception to get some more error info
* Clarified evaluation order for move case
* Added Explicit
* Fixed cleanup of Coordinator if Registry fails to insert query.
* Allow to use other locks than Read/Write for AQL collections. Not yet in API.
* Updated Comments for other Locks on DBSide. Adapted Destruction CatchTests
* Fixed double builderClose and removed unnecessary double commits
* Added a comment to clarify the state
* Moved error output to trace. Leftover from debugging
* Added some tests for complex subquery patterns
* Added a 'fireAndForgetRequests' methods to cluster comm which allows to send out a bunch of messages but does not wait for their results
* Properly cleanup leftovers of queries if the instantiation step already failed
* Added code comment for fireAndForget
* Added indexes to subquery test to make the plan a bit easier
* The cleanup on DBServerEngines in error case now also cleans up traverser engines.
* Removed unnecessary includes
* Removed debug logging
* Fixed hidden merge conflict
Squashed commit of the following:
commit bc0472d212b2efef0d7b1b8a36f2b43e8432ba86
Merge: eb47631 5f87674
Author: Michael Hackstein <michael@arangodb.com>
Date: Mon May 29 09:39:31 2017 +0200
Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup
commit eb476310b0ca6165be10b37b960933886c2757f5
Merge: d725f21 32149d2
Author: Michael Hackstein <michael@arangodb.com>
Date: Mon May 29 09:00:41 2017 +0200
Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup
commit d725f21f7d61f7c79ba31fc0338881e35f4d8f48
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 15:53:23 2017 +0200
Activated collection warmup in Cluster Mode. Added a test for collection warmup (SingleServerOnly)
commit 132bf4f9565b0dcf4ec9f84e93897b482a67ce7f
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 12:46:20 2017 +0200
Implemented client-side warmup. It can now be triggered via http and arangosh
commit 78ea449dff86118814a2f87bdb59dc16544d92b6
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 11:25:40 2017 +0200
Fixed assertion.
commit dae80f6277dde1a52eadda506858cc36e235bd55
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 11:08:46 2017 +0200
Improved the CuckooIndexEstimator computeEstimate function. It is now much better on collections with many different values
commit 7abf57876511ba369d7a577e1995d4575e98c7c8
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 11:06:43 2017 +0200
Edge index warmup will now first check if a document is stored and if not will insert it. Furthermore it resizes the cache with an estimate of how many documents will be most likely inserted
commit 890d8ad4cdfd155249f060fedd5c798b9531d556
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 11:04:47 2017 +0200
Adjusted thresholds in transactional cache. Wastly increased the amount of tries to get the lock. Lowered the fill-grade boundaries
commit 60972ab7151a3acb78e1aae4149de11d0da7aceb
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 10:45:38 2017 +0200
Added new Logtpopic CACHE that should be used to log important information regarding caches, like debug ouput on evictions / resizing, or memory-pressure resizing going on
commit 2dfacb9aef6e3fde169032514baca386786d059c
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 10:44:21 2017 +0200
Fixed an assertion in index iterator. It expected _from/_to to be cached but we modified to cache entire document
commit f05beccef65853c30eda1859a39c28ea2295bc71
Merge: 13c6abf 5c46430
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 07:56:58 2017 +0200
Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup
commit 13c6abfbc2ddb451a689011110411df68247435e
Author: Michael Hackstein <michael@arangodb.com>
Date: Wed May 24 09:52:01 2017 +0200
Added a getExtra feature for the edgeIndex. It now returns the complete edge as extra (temporary). Modified the EdgeCache to cache token and edge document (temporary). Added a warmup Function to collections that is used to warmup the index caches on demand.
commit 3952bf2009
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 19 11:00:52 2016 +0100
remove debug code
commit 004da0c614
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 19 10:52:42 2016 +0100
only do not use geoindex in inner loop if we want to sort
commit 32e1c7eac4
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 19 09:16:09 2016 +0100
manually cloning fixes the issue
commit 75823fe879
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Fri Dec 16 15:19:10 2016 +0100
another try to fix the plan
commit 2a404cfabe
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Wed Dec 14 12:21:21 2016 +0100
add dump of ExectionPlan.
Index Node seems to be created!
commit 0f8b496c57
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Wed Dec 14 10:46:24 2016 +0100
add debugging info for geoindex
commit 3d17672feb
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 13 17:03:01 2016 +0100
better tests
commit 615b85e5f5
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 13 16:09:00 2016 +0100
add tests - sort && filter and add check for IndexNodes
commit e5fcdb7386
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 13 15:53:44 2016 +0100
partial rewrite - now we start at endnodes
and work upward towards singleton node
commit 7d93fe0bdd
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 13 11:48:40 2016 +0100
add link to geo-index doc to sourcefile to clarify goals
commit aac38c8a00
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 13 11:42:49 2016 +0100
move collection acccesspath check to identify section
so we can compare nodes
commit b8cc674219
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 13 10:59:02 2016 +0100
prepare to fix filter-sort
commit 783a9aff89
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 12 16:24:52 2016 +0100
check if nodes are really sorted
commit 1d5e28d969
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 12 15:36:43 2016 +0100
fix geo-index for cluster case
commit 634eeabc0b
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 12 13:46:20 2016 +0100
add cluster to .gitignore
commit ae33a790fb
Merge: 1acfd6527099a1
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 12 13:31:54 2016 +0100
Merge branch 'devel' into obi-geo-index
* devel:
Clarify the default value of req.body
fix VS warning
moveShard jobs running
cppcheck
cppcheck
commit 1acfd65467
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 12 13:30:38 2016 +0100
work on geo-index in cluster
commit 32d0bdc846
Merge: b660672ad4ba24
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 12 09:15:36 2016 +0100
Merge branch 'devel' of github.com:arangodb/arangodb into obi-geo-index
* 'devel' of github.com:arangodb/arangodb: (132 commits)
try to fix travis build
Generate unique Swagger operationIds
Add support for Swagger tags
Add swagger route to Foxx API
Require at least one arg in route def
ported velocypack compatibility fix from arangodb/velocypack
use bulk allocator for index elements
fix memleak
fix test in cluster
remove unused code
fixed resilience
Update zwagger
Readd satellite node
Fix link
Document OAuth1
Nicer OAuth2 docs
properly start the tarball
fix VS compile errors
clones method in Jobs more useful
we don't need the dpkg-shlibdeps workaround anymore.
...
commit b6606727ee
Author: jsteemann <jan@arangodb.com>
Date: Thu Dec 8 11:13:10 2016 +0100
fixes
commit 5416755049
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Wed Dec 7 15:06:44 2016 +0100
avoid use of geo-index-rule in cases where it could yield an invalid result
commit 3e24624c6e
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 6 15:56:22 2016 +0100
back to performance
commit a941808ea8
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 6 10:29:38 2016 +0100
remove debug code for performance testing
commit 2bae135b84
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 6 10:12:12 2016 +0100
add test case
commit 71be8c215a
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Dec 6 10:00:11 2016 +0100
node replacement is now working
commit d1a3e4482d
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 5 20:18:31 2016 +0100
condition replace should be almost working
commit 275e36b603
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 5 19:49:10 2016 +0100
add pointer that needs to be replaced with true
in sort or filter condition
commit 6b93b9d2eb
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 5 19:38:01 2016 +0100
bring functions in correct order
commit 5aabbb0ac0
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 5 16:21:21 2016 +0100
fix const-ness so the condition can be modiefied
commit c84223d598
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 5 15:31:50 2016 +0100
Add information if AstNode has been found in subexpression
this information can be used in rewriting the condition and deciding
if a sort node can be deleted or not
commit b0b660eb85
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 5 15:19:06 2016 +0100
add iterative preorder walk for and conditions
commit 5dcf61c9ba
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 5 14:00:14 2016 +0100
now work to a certain degree with binaray/naray-and in FILTER/SORT
condition
commit 9b4e01eb3b
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 5 13:17:01 2016 +0100
move geoIndexRule to a positon before inexes have been touched by
optimizer
commit 9d04b37e58
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Mon Dec 5 12:46:23 2016 +0100
within queries are now working
commit d858b9aa71
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Fri Dec 2 16:15:23 2016 +0100
further implement within
commit 72d4790c68
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Fri Dec 2 12:22:07 2016 +0100
remove boost::optional and make use of nodetype
commit e0220be12d
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Fri Dec 2 11:53:31 2016 +0100
stronger split between identification of candidates and application of geo index rule
commit f27a22db06
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Fri Dec 2 11:20:09 2016 +0100
fix debug code
commit fab9af483e
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Fri Dec 2 11:00:39 2016 +0100
add extra log topic for development
commit 56b6be851c
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Fri Dec 2 10:44:42 2016 +0100
add functions providing capability to check ast for parts of geoindex
rules
commit 08ef943c83
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Thu Dec 1 23:01:50 2016 +0100
fix tests
commit 56614ac8c8
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Thu Dec 1 16:30:09 2016 +0100
switch unittests to chai and add failing test for FILTER condition
commit 8bb719c615
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Thu Dec 1 15:59:10 2016 +0100
add first tests for geoindex
commit e619ef3e4e
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Thu Dec 1 12:21:51 2016 +0100
now inspect sort and filter nodes
commit 5dbf5e14e3
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Thu Dec 1 10:03:24 2016 +0100
refactor geoOptimization into smaller fucntions
this prepares the creation of the within rules
commit 2110736d36
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Wed Nov 30 16:50:02 2016 +0100
fix logical error in nextBabies
commit 972af3af4c
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Wed Nov 30 15:44:46 2016 +0100
add within to geoindex iterator
commit 80c89d5f97
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Wed Nov 30 15:25:11 2016 +0100
geo condition is now build with a extra parameter for within
commit 8bafcdfe92
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Wed Nov 30 14:10:16 2016 +0100
GeoIndexIterator now takes parameters via ConditionNode
commit 86c21eb733
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Nov 29 14:13:35 2016 +0100
make use of geo index cursor api
commit 5b3be69e10
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date: Tue Nov 29 11:48:32 2016 +0100
WIP first working version of geoindex via aql
added more memory diagnostics for memory usage:
- collection.figures() now returns a "readCache" attribute which contains the
number of and memory usage of entries in the document revisions cache for the
collection, and a "revisions" attribute with the number and memory usage
of entries in the storage engine's revision lookup table for the collection
- the default value for --database.revision-cache-target-size was changed from
75% of RAM to 30% of RAM
Max Neunhöffer
Jan Christoph Uhde
Dan Larkin-York
Jan
Frank Celler
Dan Larkin-York
Kaveh Vahedipour
Michael Hackstein
Andrey Abramov
Manuel B
Simon Grätzer
jsteemann
hkernbach
Jan Christoph Uhde
Kaveh Vahedipour