* Changes since last PR: use LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups
* backport: address cluster LogicalDataSource resolution failure
* Cherry Picking LDAP changes
* Adding missing merges
* Fixing remaining mentions of FeatureCacheFeature
* Fix jslint
* Fixing some failed tests
* Fixing cluster authentication issue, red tests
* Fixing ldap testsuite, adding trace logging
* Fixint ldap tesuite setup and LDAP recognition
(cherry picked from commit 686d28a779)
* Fixing wrong assert
* Adding changelog entry, making requested changes from code review
* Fixing dump_authentication, fix typos
* improvements found during code review
* oops
* more use of sessionstorage
* fix tests
* Fixing broken handling, disallowing adding of local users when disabled
* Fixing testInvalidGrants
* Removing undefined auth level externally
* Fixing previous commit
* added tests for ldap search mode
* intentionally removed `after` methods from tests
because they are executed before the tests start
no cleanup is performed right now after the authentication tests
however, a cleanup is done at start of every test
* ldap tests all modes
* forward port changes from 3.3
* added generated files
* forward port missing changes for web UI
* added generated files
* added generated files
* Added some currently unnecessary locking as preventative measure against potential future changes.
* return by move will disable RVO, fix compiler warning
* add parameter to increase timeouts per 4096 size of total request package
* remove log output
* change initial timeout math from integer to double ... avoids reviewer confusion.
* fix some deadlocks found by evil lock manager (tm)
* fix duplicate lock
* fix indentation
* ensure proper lock dependencies
* fix lock acquisition
* removed useless comment
* do not lock twice
* create either a V8 transaction context or a standalone transaction context, depending on if we are called from within V8 or not
* AQL micro optimizations
* use explicit constructor
* only use V8DealerFeature's ConditionLocker for acquiring a free V8 context
entering and exiting the selected context is then done later on without having to hold the ConditionLocker
* remove some recursive locks
* Disable custom deadlock detection when Thread Sanitizer is enabled
* Changing ifdef's
* grr
* broke gcc
* Using atomic for ApplicationServer::_server
* fix premature unlock
* add some asserts
* honor collection locking in cluster
* yet one more lock fix
* removed assertion
* some more bugfixes
* Fixing assert
(cherry picked from commit 1155df173bfb67303077fbe04ee8d909517bfd21)
if yes, the server may dynamically adapt the size of the response, in order to ensure that
HTTP responses do not get out of hand size-wise. This is a new feature in devel, and this
commit now makes it optional so that older clients do not need to be changed.
* remove some non-unused V8 persistents
* do not throw that many bogus assertions
* do not rely on server role being defined
* slightly better debug output for V8 context debugging
* fix collection ids in inventory response
* simplify bootstrap a bit
* slightly better error handling
* make elapsed time a queryable value
* use less memory for stub collections
* added assertions that will always make sense
* added assertions
* do not garbage-collect while waiting
* less copying of parameters
* do not show "load indexes into memory" buttons for mmfiles engine
as all indexes are in memory anyway
* when a collection is truncated via the web interface, flush the WAL and rotate all active journals
this will make close all open journals on leader and followers and make them subject to compaction opportunities
* fix invalid server id values being passed from web interface to backend
* introduce afterTruncate method for indexes
* added test case for issue #3447
* updated CHANGELOG
* don't warn about replicationFactor for system collections
* check that the queries actually use the geo index and not some other index
* properly report error in web interface
* fix some internals checks that made truncate fail for bigger collections in maintainer mode
* also run a compact() operation after a serious truncate
in order to make iteration over the truncated range much faster
when the collection is next accessed
* increase default maximum number of V8 contexts to at least 16
* Switching from ttl to supervision based failover mechanism
* Allowing canceling of ongoing actions
* refactored asyncjobmanager
* refactoring some code
* adding read-only flag
* catching some exceptions to reduce log pollution, removing unnecessary code, removing tests for _changeMode
* fixing "createsANewDatabaseWithAnInvalidUser"
* auth = off does not longer make everyone superuser
* Fixing cluster_sync and maybe resilience
* properly initialize all properties
* use faster comparison
* properly detect and handle "method not allowed"
* code-style
* remove unused variable
* narrow variable scope
* handle non-existance of AuthenticationFeature
* remove dead code
* replace some C string handling with std::strings
* moved assertion to the correct place
* honor number of array members for IN operator
* slightly adjust error messages
* slighty adjust some error messages
* try to fix issue with lingering replication contexts on shutdown
* clean up heartbeat thread a little bit
* small fixes
* remove memory zone remainders
* remove useless calls
* attempt to fix non-deterministic CRC test
* attempt to fix non-deterministic replication tests
* fix document removal with multiple documents in cluster
* attempt to fix non-determinism of tests
* fix privilege checks for accessing collections by collection id, not by name
* Make isRestore work in the cluster.
This covers sharded collections with default sharding and non-default
sharding.
* always use locally generate revision ids for storing and looking up documents
* do not use V8 variant of AQL functions in early optimization stage when a C++ variant is available
* additionally, simplify AQL function definitions and aliases
* warn when more than 90% of max mappings are in use
* added C++ variant of replication catchup
* added `--log.role` option
* updated CHANGELOG
* removed non-existing scheduler.threads option from config
* removed useless __FILE__, __LINE__ invocations
* updated CHANGELOG
* allow a priority V8 context
* remove TRI_CORE_MEM_ZONE
* try to fix Windows errors & warnings
* cleanup
* removed memory zones altogether
* exclude system collections from collection tests
* fixed issue #3075
* update the known issues document, as some issues have already been fixed
* speed up equality lookups in unique vpack indexes
* Allow for re-use of RocksDBKey objects through transaction leaser.
* Converted all RocksDBKey variants to non-static constructors.
* Added checks everywhere for CachedValue::construct.
* Enforce 16MiB limit on keys and 4GiB limit on values.
* Protected members behind function calls in anticipation of format change.
* Shrunk CachedValue struct.
* Fixed alignment of CachedValues.
* Fixed improper comparison.
* Added virtual attributes for enterprise on Methods.cpp
* Working no access collections
* align a comment
* Documentation and test fix
* fixing community build
* removed delete call
* cleanup
* lower cpu activity of log thread too
* fix log messages
* do not enter threads into unordered_set, as it is unneeded
* do not compile in calls to disabled plan cache
* moved AQL regex cache from thread local variables to a class of its own
* more sensible thread creation and destruction
* remove API docs for endpoints that were deleted in 3.0
* remove unused variable
* set all scalar values in the ctor, so check tools do not complain
* move Result objects
* prevent potential nullptr access
* remove ambiguous boolean parameter with no clear meaning
* make code check tools happy
* don't allow copying cache Findings, allow moving Results
* the debug functionality for counting RocksDB key ranges was broken
this produced sporadic test failures
* remove superfluous iostream include
* fix range checks
* don't leak if an error happens during database removal
* try to simplify Result a bit
* try to deletion checks a bit more robust
* fixed issue #3075
* update the known issues document, as some issues have already been fixed
* speed up equality lookups in unique vpack indexes
* fixed wording
* add "cluster selectivity estimates" to CHANGELOG
* add some documentation to RocksDBRestReplicationHandler
* fix building with relative paths
* add some more doc
* add some tests for the replication api
* fix RocksDBRestReplicationHandler and add tests
* update documentation
* remove obsolete parameter
* fix error message
* Implementing logger-first-tick, logger-tick-ranges. Fixing dump `chunkSize` documentation
* we must now ignore that datafiles are not sealed
this is because an unsealed datafile may have been produced by
renaming multiple journals to datafiles at server start
* acquire collection count after we have acquired the lock
* count the null byte as well
* fix count value acquisition
* send query fragments to the correct servers, even after failover or when a follower drops
the problem with using the previous shard-based approach is that responsibilities for shards may change at runtime
however, an AQL query must send all requests for the query to the initially used servers.
if there is a failover while the query is executing, we must still send all following requests to the same servers, and not the newly responsible servers
otherwise we potentially would try to get data from a query from server B while the query was only
instanciated on server A.
* Count as checksum
* Make readLockId optional as well so upgrades are possible
* fix option name in startup script
* fix some replication issues with RocksDB engine
* simplify index API a bit
* fix fulltext index removal performance, simplified code
* updated CHANGELOG
* fix hanging test
* try to fix shutdown problem
* improve fulltext query performance
* fixed duplicate var
* removed obsolete code
* fix some shutdown races
* do not call ensureIndex that often
* Added a backup test suite. This suite is supposed to entirely drop an ArangoDB _system and restore it into a fresh one. This also includes system collections.
* Added more test cases for backup suite. Now tests several authorization/rights cenarios
* Fixed RestReplication Handlers to restore _user collections Properly.
* Updated Changelog
* Added special handling of _users in Restore for MMFiles as well.
* Added JWT secret for cluster execution of this test, also added JWT secret to shutdown call
* cleaning up several duplicate versions of Index::memory, Index::cleanup, Index::drop, integrating branch issue-383-ensure-delete
* fixing wrong column family
* reduce extractions to projections
* recycle string buffers in SocketTask
* micro optimizations for mmfiles indexes
* added special lookup function for _key
* moved function into the correct file
* speed up key buffer allocations a bit
* added noexcept specifier
* correctly name variable
* explicitly move bounds
* fix and speedup from/toPersistent functions
* reuse string from ManagedDocumentResult for multiple lookups
* use move-assign
* a bit less work for single server
* speedup AQL function HASH
* single fetch optimization
* performance optimization for the case when no documents need to be returned
* make reduce-extraction-to-projection a RocksDB-only optimizer rule
* cppcheck
* try to fix compile error on MacOS
* bug fix for MacOSX
* missing namespace (in Windows compile)
* reduce extractions to projections
* recycle string buffers in SocketTask
* micro optimizations for mmfiles indexes
* added special lookup function for _key
* moved function into the correct file
* speed up key buffer allocations a bit
* added noexcept specifier
* correctly name variable
* explicitly move bounds
* fix and speedup from/toPersistent functions
* reuse string from ManagedDocumentResult for multiple lookups
* use move-assign
* a bit less work for single server
* speedup AQL function HASH
* single fetch optimization
* performance optimization for the case when no documents need to be returned
* make reduce-extraction-to-projection a RocksDB-only optimizer rule
* Testing hash for documents
* fixing logspam error
* added base support for kHash in the edge index
* Current version of hash enabled edge index
* removed rocksdb key type byte, dynamic prefixes for hash-index
* Fixing issues with key classes
* Fixing merge conflict
* Fixing version with semi-working hash index
* fixing various issues
* Fixing hanger in EdgeIndexIterator nextExta, when _cache is disabled. Reenable cache
* Fixing geo index bounds, and fillIndexes error handling
* Adding back failure point
* Fixing RocksDBKey::vertexId
* Added version check on startup
* Fixing version check
* added one more check
* properly detect column families at startup
* Adding additional sanity checking
* Adding missing column family
* some fixes
* Turned off geoindex sync
* fixed error
* added asserts (untested)
* Adding more asserts
* remove unused code
* print values of all options
* don't print "warning(s):" if there are none
* remove remainders of old file deletion functionality
* remove unused function
* Added recovery tests for views and fixed a few related bugs.
* Added more view recovery tests.
* Modified view recovery tests to add a waitForSync operation after.
* fixed usage of wrong view type
* fixed recovery of view change markers
* Augmented cache API and fixed a potential deadlock scenario.
* The RocksDBEdgeIndex now tries to read up to 10 times form the cache on lock_timeout. Instead of rewriting everything after first attempt
Furthermore fixed an issue with Column Family usage in drop.
commit 708cd81362b56213dd4d7a69b1e4eea120de0db4
Merge: b8896ea 924ef44
Author: Michael Hackstein <michael@arangodb.com>
Date: Thu Jun 8 14:47:56 2017 +0200
Merge branch 'devel' of github.com:arangodb/arangodb into feature/drop-collection-batches
commit b8896ea5479d56f3e6d55d6094ec2658b058dc04
Author: Michael Hackstein <michael@arangodb.com>
Date: Thu Jun 8 14:47:37 2017 +0200
Fixed drop collection in rocksdb for all entries that are not compacted in a complete file. it was using the wrong column family.
commit 59f6a9d51b1c12d665eba683d3b09f1b01ce77fc
Author: Michael Hackstein <michael@arangodb.com>
Date: Thu Jun 8 13:00:18 2017 +0200
Remove large range now uses ColumnFamily comparator instead of default one. Also it writes batches of 1000 documents to RocksDB instead of full deletion at once.
Squashed commit of the following:
commit 860ddfc7a74866919ce5139ac3d874f09501ec97
Merge: d9ec0e3 bf20939
Author: Michael Hackstein <michael@arangodb.com>
Date: Thu Jun 8 12:45:14 2017 +0200
Merge branch 'devel' of github.com:arangodb/arangodb into feature/views-column-family
commit d9ec0e3c88122c6b9ee8acdbc6e69ca6af62e19f
Author: Michael Hackstein <michael@arangodb.com>
Date: Thu Jun 8 12:44:43 2017 +0200
Added a column family for views. It is no yet used anywhere. Just preparation.
* Import RocksDB encryption support
* Adding enterprise hooks
* Allocate scratch space in BlockCipher, used for all subsequent calls to EncryptBlock
* Adding more enterprise hooks
* Added encryption test script
* Added (commented) entry for encryption-keyfile
* Use correct env
* Updated ARANGODB value
* Log startup at debug level
* Removed TTY flag
* Fixed build
* Added prefix block support with unique initial counter & IV for CTR implementation
Squashed commit of the following:
commit bc0472d212b2efef0d7b1b8a36f2b43e8432ba86
Merge: eb47631 5f87674
Author: Michael Hackstein <michael@arangodb.com>
Date: Mon May 29 09:39:31 2017 +0200
Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup
commit eb476310b0ca6165be10b37b960933886c2757f5
Merge: d725f21 32149d2
Author: Michael Hackstein <michael@arangodb.com>
Date: Mon May 29 09:00:41 2017 +0200
Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup
commit d725f21f7d61f7c79ba31fc0338881e35f4d8f48
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 15:53:23 2017 +0200
Activated collection warmup in Cluster Mode. Added a test for collection warmup (SingleServerOnly)
commit 132bf4f9565b0dcf4ec9f84e93897b482a67ce7f
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 12:46:20 2017 +0200
Implemented client-side warmup. It can now be triggered via http and arangosh
commit 78ea449dff86118814a2f87bdb59dc16544d92b6
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 11:25:40 2017 +0200
Fixed assertion.
commit dae80f6277dde1a52eadda506858cc36e235bd55
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 11:08:46 2017 +0200
Improved the CuckooIndexEstimator computeEstimate function. It is now much better on collections with many different values
commit 7abf57876511ba369d7a577e1995d4575e98c7c8
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 11:06:43 2017 +0200
Edge index warmup will now first check if a document is stored and if not will insert it. Furthermore it resizes the cache with an estimate of how many documents will be most likely inserted
commit 890d8ad4cdfd155249f060fedd5c798b9531d556
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 11:04:47 2017 +0200
Adjusted thresholds in transactional cache. Wastly increased the amount of tries to get the lock. Lowered the fill-grade boundaries
commit 60972ab7151a3acb78e1aae4149de11d0da7aceb
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 10:45:38 2017 +0200
Added new Logtpopic CACHE that should be used to log important information regarding caches, like debug ouput on evictions / resizing, or memory-pressure resizing going on
commit 2dfacb9aef6e3fde169032514baca386786d059c
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 10:44:21 2017 +0200
Fixed an assertion in index iterator. It expected _from/_to to be cached but we modified to cache entire document
commit f05beccef65853c30eda1859a39c28ea2295bc71
Merge: 13c6abf 5c46430
Author: Michael Hackstein <michael@arangodb.com>
Date: Fri May 26 07:56:58 2017 +0200
Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup
commit 13c6abfbc2ddb451a689011110411df68247435e
Author: Michael Hackstein <michael@arangodb.com>
Date: Wed May 24 09:52:01 2017 +0200
Added a getExtra feature for the edgeIndex. It now returns the complete edge as extra (temporary). Modified the EdgeCache to cache token and edge document (temporary). Added a warmup Function to collections that is used to warmup the index caches on demand.
commit 19a7210119cc284af64251b202f690ab62bf615c
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 23:20:10 2017 +0200
Allow access to /_api/user/ without database check in VST as in HTTP.
This is a try to fix#2465
commit f973c1335652540174bf6b78df42290bded357da
Merge: 30bdc4c 365de45
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 21:32:15 2017 +0200
Merge remote-tracking branch 'origin/devel' into vst-cleanup
commit 30bdc4ca38e028d58cebea8268683d9f94e87ad7
Merge: 14acaea 086f6a2
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 14:52:23 2017 +0200
Merge remote-tracking branch 'origin/devel' into vst-cleanup
commit 14acaea8cb5fff2c1c8c27eee32294bfd612f629
Merge: 5bd0e79 fb088d3
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 12:40:29 2017 +0200
Merge remote-tracking branch 'origin/devel' into vst-cleanup
commit 5bd0e79ab053447f94f2fd959a33b1e15302b6a8
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 12:36:23 2017 +0200
Fix bug in authentication in VST
commit 9a999ce5e629aee5816a4f61b194b086d11fd169
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 12:24:35 2017 +0200
Open up endpoints /_open/* in VST without being authenticated
commit 32f62db42af6d5bfc24214ebb4ca7c1fb473b55e
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 12:11:31 2017 +0200
Simplify logic by not using boost::optional.
commit 925ce2f7b890c6f14205fd3e98553939d89bec61
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 12:11:20 2017 +0200
Add JWT case for authentication for VST
commit f1d7f67a9fc20c91ca000e9a4dad91e0f50f1652
Merge: 040ace3 7a9ccc9
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 08:58:51 2017 +0200
Merge remote-tracking branch 'origin/devel' into vst-cleanup
commit 040ace3e9dce47ddea5f51d29f0153a70d257c9e
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 08:23:16 2017 +0200
Write out chunk buffer using little endian uints
commit 66ad4c0e8d3bc94091664505986b00e7cff39f2f
Author: Max Neunhoeffer <max@arangodb.com>
Date: Fri May 12 08:12:38 2017 +0200
Move maxChunkSize query to constructor, remove dead code
commit f7b4c26cc09c6d512362340c6bf6061bbaae61a9
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 17:57:14 2017 +0200
Do not take protocolVersion from request for VST.
commit c76d6685f9507701715abf882f0ac0e192ead59f
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 13:20:58 2017 +0200
Remove dead code for compression which did never work well.
commit b823765b7c041dc5d143f7ab2e13d92f5f42d3eb
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 13:09:53 2017 +0200
Fix renaming to make compiler happy.
commit b327830012d146f4f4e2d0f0467dab351fbd3bd0
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 13:07:36 2017 +0200
Rename _defaultPortVpp to _defaultPortVst.
commit 5eb98a9e64d702476e8ec098b7506b1a63145d64
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 13:06:28 2017 +0200
Rename header guards VPP -> VST.
commit a3e96be26d49d6151898d8d757ede6225ac6abd2
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 13:05:55 2017 +0200
Rename VppRequest and VppResponse files to Vst.
commit 1d5f6f196490c3cf9312be5927b967b4abc03b67
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 13:04:19 2017 +0200
Change names of header guards VPP -> VST.
commit f004b25fbef75050b34bcd4eae4eb17b8b467230
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 13:03:19 2017 +0200
Rename files VppNetwork.h and VstMessage.h
commit c04d3aec19e8f0a8a3b90cd48e7736bff037359d
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 12:59:15 2017 +0200
Implement correct chunk header for sending out VelocyStream 1.1 messages.
commit 73aeedfbbef24d5929c44bd26c16343480514270
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 01:11:23 2017 +0200
A lot more renaming VPP -> VST.
commit 2acedc77f16a82251bc4f9e39526184e2a6f0dec
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 00:49:20 2017 +0200
Rename header guard.
commit 2ade43946aa5ea0048dab81eed5ffd931ce35ea8
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 00:48:01 2017 +0200
Rename files VppCommTask.* to VstCommTask.*
commit 0ffcda02a51cbad35e5ec06117d397aac37323cd
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 00:46:31 2017 +0200
Rename class VppCommTask to VstCommTask.
commit 9478c6c56c995b014b7a14408544b76fa676a317
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 00:30:47 2017 +0200
Rename protocol versions to VST in ProtocolVersion enum.
commit 0b69b7add6d7b5155ac71b8257def282ab4a1637
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 00:29:29 2017 +0200
Recognize incoming chunk headers in VST/1.1 format.
commit f40173c4fc6dc48613ae67ac3fd810d002c9b6cd
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 00:11:54 2017 +0200
Recognise and remember VST/1.0 and VST/1.1 protocol versions.
commit 29c798dc1af2d650c41eb9ed06f635786faf10f6
Author: Max Neunhoeffer <max@arangodb.com>
Date: Thu May 11 00:11:23 2017 +0200
Create protocol version VST/1.1 and disable vst+ prefix for endpoints.
* 'devel' of https://github.com/arangodb/arangodb:
Add bye bye messages
update rocksdb logging to error level
protect vector of isolate data with a mutex
avoid nullptr in Inception
remove satelites
Wilfried Goesgens
Jan
Vasiliy
Simon
Andrey Abramov
Dan Larkin-York
jsteemann
Andrey Abramov
Jan Christoph Uhde
Michael Hackstein
Matthew Von-Maszewski
Max Neunhoeffer
Frank Celler
m0ppers
Jan Christoph Uhde
Ewout Prangsma
Dan Larkin
Andreas Streichardt