* remove recent _activeThreadCondition. it made things worse. moved all ClusterCommThread methods to end of file to ease review.
* attempt at avoiding Scheduler io_context being nullptr in late shutdown steps
* manually revert last change since bug is realy about devel branch not 3.4 branch
* issue 496.5: backport 3.4: minor API cleanup and error reportin enhancements, update iresearch to commit d69f7bd184e009da7bf0a478efd34a0c85b74291
* add workaround for shell-collection-rocksdb-noncluster.js::testSystemSpecial test failure
* fix typo
* issue 496.3: backport 3.4: move more coordinator-related logic out of TRI_vocbase_t, rename some arangosearch view configuration parameters, remove some consolidation policies, update iresearch to revision 6fd9760d81b136f769e277ea5b8f53996ed7a1ca
* address merge issue
* backport: remove code causing nullptr access
* invalidate payload for each field in FieldIterator before setting a value
* address compilation issues
* Improve logging on coordinator when doing `arangorestore`.
* Return more error information in `mergeResults`.
* Longer timeout for communication coordinator -> leader for writes.
This is taking into account possible write stops from followers needed
to get in sync.
* Fix compilation.
* Get rid of numbers in exception log messages.
* Fix compilation.
* Fix indentation.
* Feature/arangosearch speedup removals (#7134)
* speedup document removals and optimize data model
* fix invalid constexpr
* reduce number of heap allocations for removals (#7157)
* open up connect limit to allow one DNS retry
* add CHANGELOG entry for dns retry fix
* update warning message wording
* Add lock line that was missed in recent scheduler merge
* backport: switch scope of responsibility between a TRI_vocbase_t and a LogicalView in respect to view creation/deletion
* backport: ensure arangosearch links get exported in the dump
* backport: ensure view is created during restore on the coordinator
* Updates for ArangoSearch DDL tests, IResearchView unregistration and known issues
* Add fix for internal issue 483
* Stop libcurl from trying to POST stdin
* Stop relocking every iteration in wait
* Remove unimplemented function
* Restrict setting of empty POSTFIELDS to POST requests
* Revert locking change
* Implement `syncCollectionCatchup` in DatabaseTailingSyncer.
First stab, might not even compile.
* Fixed a typo.
* Fix a typo and a compilation problem.
* Further compilation fix.
* Implement two stage catchup.
* Two small corrections.
* Unified error messages in Synchronize shard job.
* Improved a code comment.
* Fixed autocasting bool->double and double->bool issue. That is truely one of the best features ever invented... </irony>
* Renamed doHardLock => toSoftLockOnly and inverted default value
* Merged soft/hard foot logic with Transaction splits
* Use scopeguards to cancel readlocks
* Removed incorrect skipping of Batches in RocksDB Tailing syncer. This caused issues, whenever one transaction was spiltted.
* Added a test for Splitting a large transaction in RocksDB
* Reactivated skipping in RocksDB Wal Tailing (reverts initial fix)
* Actually include lastScannedTick in CollectionFinalize. Proper fix, kudos to @jsteemann.
* Fixed healFollower task in split-large-transaction test
* First attempt to not block the thread that requires the EXCLUSIVE sync-up lock
* Fixed insertion of query into registry in rest replication handler.
* Removed unnecessary / false asserts as suggested in review. Fixed code comments.
* Replaced auto with a correct type as suggested in review
* Added a helper function to validate if a query is in use in the registry
* Fixed logic bug in usage of query registry
* Fixed compile issue
* Implemented optional 'doHardLock' parameter in the replication acquire read-lock call. A hard-lock guarntees to stop all writes, a soft-lock may not.
* Fixed compile issue
* Automaticly transfrom int -> bool in initializerlist sucks...
* Inverted boolen logic bug hidden due to int->bool beeing logically inverted.
* Today it seems that bools are too complicated for my brain.
* Removed failure point, didn't write a test for it, and it is hard to write it in the current test environment. Need to find a better solution in future
* Applied chenges required by @goedderz in review
* Renamed doHardLock => toSoftLockOnly and inverted default value
* First attempt to not block the thread that requires the EXCLUSIVE sync-up lock
* Fixed insertion of query into registry in rest replication handler.
* Removed unnecessary / false asserts as suggested in review. Fixed code comments.
* Replaced auto with a correct type as suggested in review
* Added a helper function to validate if a query is in use in the registry
* Fixed logic bug in usage of query registry
* Fixed compile issue
* Automaticly transfrom int -> bool in initializerlist sucks...
* Inverted boolen logic bug hidden due to int->bool beeing logically inverted.
* Today it seems that bools are too complicated for my brain.
* Removed failure point, didn't write a test for it, and it is hard to write it in the current test environment. Need to find a better solution in future
* Applied chenges required by @goedderz in review
* Start ClusterComm threads in `ClusterFeature::start`. Stop ClusterComm threads in `ClusterFeature::stop`.
* Do not free objects in `Scheduler::shutdown`. Let the `unique_ptr` do their job. Stop ClusterComm threads in `ClusterFeature::stop`, but free instance in `ClusterFeature::unprepare`.
* `io_context` may contains lambdas that hold `shared_ptr`s to `Tasks` the required a functional `VocBase` in their destructor.
* Clean up.
* issue 485: ensure LogicalDataSource::drop() is called on vocbase drop
* add missed change
* backport: address race between make(...) and async job
* add another missed change
* backport: ensure recursive lock reports itself as locked correctly
* backport: address test failure on mmfiles
* backport: remove redundant lock already held by async task
* backport: reset reader before unlinking directory
* enable the ability to push results processing to threads
* have ClusterComm push libcurl response processing to Scheduler threads
* tuning changes from Matthew and Michael
* give new defaults to minimum thread count
* create multiple ClusterCommThreads, each with own Communicator object
* put PR notes in change log
* correct speling
* Also drain V8 queue.
* Add prio V8 to switch in canPostDirectly.
* Accept --server.minimal-threads even if maximal threads is not set.
* Reactivate stopping of threads.
* untested connection count management code.
* add digit to identify location of identical error messages.
* unit tests created, code working and integrated
* subtle changes to smoothing algorithm, close ... but still not perfect
* use scheduler queue() instead of post() to avoid occassional extreme latencies. throughput improved!
* temporary hack to hardcode queue size much lower
* force direct calls to post() to instead route through queue()
* correct issue with previous check-in, but performance has dropped with this on
* revert hardcoded value in SchedulerFeature.cpp
* queue to high priority to be consistent with other Aql actions
* backout all edits to Scheduler.cpp and Scheduler.h
* oops, forgot to remove some testing code.
* 3.4 merge/update
* add changelog entry
* Properly remove unplanned followers in leader and Current.
* Add a catch test.
* Fix tests.
* Fix a bug with a temporary object.
* Protect against exception from getCollection not found.
* AgentConfiguration cleanup
* static strings in maintenance / agency
* fix windows build
* test bogus
* got rid of old inefficient create method
* completed with NonAction
* this works with osx / windows
* map creation can be outside function
* string init order fiasco
* startup init fiasco
* fix init-order fiasco with static strings (#6475)
* try to work around compile errors
* Removed broken and unused strings.
* Fixed issue with condition finder pulling filter past modification node into index.
* Fixed issue with traversal condition finder pulling filter past modification node into index.
* fixed jslint
the change in behavior crept in unintentionally due to my last
change of the index API. this PR reverts the unintended change
here so figures are only returned when explicitly requested
* issue 458.2.1: ensure LogicalCollection presence is checked before granting/revoking permissions
* try to address test failures
* backport: support wildcard for database too
* create collection before granting
* adjust ruby tests to expect behaviour as defined by issue #458
* adjust expected Ruby test result
* create required collection in Ruby test
* revert back to previous test code since Ruby refuses to create required collection
* missed revert
* potential bugfix for planning/#2865
* speed up dump tests setup
* enable authentication for backup tests
* make arangodump provide a "serverId" to the server
this allows the server to track arangodump as an active dump client
so any data required for the dumping may be retained while the
dump is ongoing
* don't log binary stuff into the logfile
* issue 459.3: ensure collection permissions are checked before updating/dropping an IResearch view
* backport: ensure collection permissions are checked before updating/dropping an IResearch view on cluster
* backport: address test failures
* backport: address more test failures
* reuse existing classes for scoping ExecContext
- Schmutz now called "Maintenance" and completely implemented in C++
- Fix index locking bug in mmfiles
- Fix a bug in mmfiles with silent option and repsert
- Slightly increase supervision okperiod and graceperiod
* added test for recursive edge deletion, still missing c++ implementation
* test
* added deletion of connected edges when removing an edge or a vertex
* Added a helper function to iterate over all graphs.
* applied requested test changes
* edge definition helper function
* moved removeVertex and removeEdge inside a helper function
* more optimizations
* compile error
* typo
* requested changes
* const func
* issue 450.3: use a single configuration policy differentiated by 'type' with arbitrary number of configurable parameters
* ensure test data is persisted to the test directory
* add missed declaration
* fix typo
* add full namespace and includes
this prevents stacking many useless SavePoints in transactions with
many operations.
The PopSavePoint method in RocksDB is forward-ported from upstream
RocksDB commit 90f744941d48f15d5d070dfcab5d51b77abac8ad.
This is likely to be released with a future release of RocksDB.
* added SEARCH keyword
* fixes jslint errors
* add incompatibility notice
* remove VIEW keyword (does not work yet!!)
* add initial support for SEARCH keyword to optimizer rules
* replace FILTER with SEARCH keyword in tests
* removed VIEW keyword, make shell_server_aql tests pass again in single server
* cleanup
* handle SEARCH clause for views
* make SEARCH a non-keyword
* fixed an issue with duplicate variables
* fixed 3 tests
* fix SEARCH statement support for views
* minor refactoring
* fix integration tests
* optimize SEARCH conditions a bit more
* fix jslint error
* fixed wrong comments and typo in class name
* fix documentation
* adjust recovery tests to use SEARCH instead of FILTER
* fix another failing test
* initial checkin of isRetryOK(). Includes fixes to known code that has previously hung shutdowns by performing infinite retries.
* slight help on getting out of a loop faster during shutdown. not essential.
non-sealing of datafiles could actually happen when more than a single journal
was found at startup, and all but the last journals were converted into datafiles
automatically. these datafiles were not sealed, triggering the assertion failures
later in other places
* Built a C++ skeleton REST handler for gharial, with fallback to the JS handler
* Moved aql::Graph to graph::Graph
* Added complete edge definitions to Graph
Also:
- some cleanup
- used forward-declarations in headers
- use Graph in graph rest handler
* Handle graph lookup failures according to the test suite
* Added GET vertex
* Bugfixes in ResultT
- Added missing #include
- Fixed move semantics
* Move central code of readVertex to GraphOperations
* ResultT fixes and complements
* Implemented a graph cache
* Added and used graph cache to the rest handler
* Added GET edge
* Added DELETE edge
* Extracted some code
* Added PATCH and PUT for both edge and vertex
* Moved update/replace transaction code to GraphOperations
* Added stub routes for POST and a TODO note
* Added a test checking that deleting a vertex removes all incident edges as well
* Added a test checking that deleting a vertex does not remove edges in non-graph collections
* fixed compiler warnings and errors
* Began work on DELETE vertex
For this, added a V8Context to allow for AQL queries to use subtransactions
* Continued work on DELETE vertex (still WIP)
* prep for graph post routes
* fixed removeVertex operation (aql)
* added post vertex and post edge gharial routes
* wasSynchronoues flag changed
* gharial post c++ handler, naming conventions
* added gharial tests
* temporary disabled cache (because not completed), added graph property read functions
* added c++ gharial list vertex collections
* added c++ gharial graph config
* added c++ gharial list graphs
* added graph manager class
* first implementation of create graph in c++, WIP
* changed error messages
* added etag to create graph api, still multiple edge definition check missing
* finished POST /_api/gharial/<graph>
* WIP - DELETE /_api/gharial/<graphName>
* added DELETE /_api/gharial/<graphName> validation, still missing correct response
* gharial delete
* fixed delete gharial lock
* finished DELETE /_api/gharial/<graphName>
* added routes for graph based vertices and edge definitions
* improved delete route
* added add new edge definition to existing graph
* patch edge definition in a graph, still <WIP>
* finished edit edge definition route
* code changes due to devel code changes
* added remove edge definition route
* added vertex delete function
* added todo note regarding one drop collection issue
* add oprhan collection to graph route implemented
* Added a test
* Updated a comment
* Several minor changes
* Minor changes during review
* Changes during review
* Changes during review
* Bugfix: orphans may be null or omitted
* Bugfix: resolve externals
* minor code changes
* seperated graph class to independent component classes
* seperated graph class to independent component classes
* removed log output
* fixed create collection behaviour in a cluster environment
* fixed enterprise graphs behaviour in c++ gharial api
* removed log output
* formatting
* improved error handling, fixed a linux compile bug
* more result refactoring
* more result type cleanup
* fixed wrongly defined test
* result handling
* error handling
* more refactoring
* Bugfix: avoid race condition in cluster when creating collections
* updated graph documentation
* added graph related static strings
* static strings, new method to create options for gharial created collections
* Some minor cleanup
* more use of static strings
* minor code changes, review
* added missing parseint
* removed gharial foxx, added js common module, added v8 general graph module
* correct use of virtual method
* more v8, js general graph, broken state
* more v8 graph functions
* fixed editEdgeDefinition, added drop function
* fixed drop behaviour
* added _list, _exists
* added c++ rename graph collections, added v8 + graph module function
* Added a regression test
* added graph._deleteEdgeDefinition, v8, server
* more v8g
* added _removeVertexCollection
* added _extendEditDefinitions
* todo, need to add a helper sort method for a local defined relation
* fixed test
* fixed lots of tests, added more client functions, _addVertexCollection on client module is still broken
* added more client graph functions, all tests green
* more client functions
* add del edge def route
* Fix use after move
* Minor changes in client general-graph.js module
* Make a copy before sorting (don't touch the argument)
* Minor changes and some additional asserts in graph tests
* Consistently set parameter defaults
* Renamed static strings
* Remove superfluous function
* Made comment more verbose
* Minor changes in general-graph-common.js
* Added missing template arguments
* Fixed community build
* Cleanup in editEdgeDefinition
* Regression test & bugfix: comparison of edge definitions didn' order from and to
* Fixed errors introduced by merge
* Minor changes in v8-general-graph.cpp
* Fixed test failure due to wrong error code in CE
* added missing id field
* Added permission checks for graph._create
* Removed assertion that is no longer valid
* Moved removeGraph from GraphOperations to GraphManager
* Allow C++ implementation of graph._drop to handle smart graphs
* Flush js client db cache after creating/dropping collections via the general graph module
* Added _deleteEdgeDefinition to the general graph client module
* WIP: Added permission checks for drop graph
* Fixed permission checks for drop graph
* Added permission checks for other graph operations
* Bugfix: assert edge definitions are returned in order
* Some cleanup
* Removed unused method
* Minor improvements in GraphManager
* Fixed a type in general-graph common module
* Most useful fix of all times ever: Do not auto cast from bool to int and alternate error/noerror by this
* Added the initial keyword to StaticStrings
* Added a new error code, used whenever a user tries to inject a documentcollection as a relation into the graph, which is invalid
* Some GraphManager/Ops/Graph cleanup. Less Slice parsing, more usage of GraphObjects
* Test edgeDefinitions in graphs with a defined ordering
* GraphClass Layout cleanup
* Do not test error messages, use codes instead
* Recreated backwards compatibility of Graph Creation Permission errors
* Changed error-code if edgedefinition is used twice
* Added a StaticString for the GraphName
* Renamed graphToVpack => graphForClient
* Partly fixed graph-api test to work with better error messages. Still red: The edgeDefinitions are now sorted, the test is supposed to sort his own list, but appearently does not do so. Under investigation
* Added a new error code that rejects injection of differently sharded smart collection into smartgraph. Should be more helpful to our users
* graph createCollectionOptions now require an open object to be cross-called from enterprise. Made enterprise switch for creation of graph more elegant.
* Updated graphs.cpp
* Massive refactoring. Made Factories for graphs to make SmartGraph much more transparent. Also reduced amount of multiple implementations of the same stuff. Killed vocbase/graphs use GraphManager instead. Removed usage of GraphCache, was not completely implemented anyway and only partially used, which is bad at the moment. Option for later improvement never the less
* Adapted JS code to now really use c++ variants. ALso included 3 Classes: Graph, SmartGraph and GraphModule.
* Fixed undefined behaviour in Remove Vertex. Fixed smartgraph sharding if one collection already exists.
* Removed DEBUG output
* Removed DEBUG logs
* Removed dead code
* Fixed Graph EdgeDefinition test, they now have a different ordering.
* Added a test when adding a vertexCollection that it is actually valid in the graph
* Client Graph API now correctly sends `orphanCollections` and not `orphans`
* Let GraphOperations modify the graph in-place. It should now properly handle edgeDefinitions.
* Added initial cid StaticString
* Included the vocbase in fromPersistence creation of Graphs. Only required to enhance 3.3 SmartGraphs on the fly.
* Fixed internal error message
* Fixed compiler isses originiated from merging
* Removed unused imports
* Regenerated generated file
* add "DBSERVER" as an alias for "PRIMARY"
This allows specifying the value "DBSERVER" for `--cluster.my-role`.
"DBSERVER" is only treated as an alias for "PRIMARY", because several
other parts of the code and APIs use the string "PRIMARY".
Changing these from "PRIMARY" to "DBSERVER" would make the change
downwards-incompatible, which we do not want.
The downside of this alias-only solution is that even when specifying
a role value of "DBSERVER", the server will still report its role as
"PRIMARY", which may be a bit confusing. The server will also generate
its id as "PRMR-XXXX" as before:
2018-08-03T15:23:09Z [9584] INFO {cluster} Starting up with role PRIMARY
2018-08-03T15:23:09Z [9584] INFO {cluster} Cluster feature is turned on. Agency version: {"server":"arango","version":"3.4.devel","license":"enterprise"}, Agency endpoints: http+tcp://[::]:4001, server id: 'PRMR-f655b728-4cea-44ac-88e9-8b34baa80958', internal address: tcp://[::1]:8629, role: PRIMARY
* adjusted documentation to use "DBSERVER" instead of "PRIMARY"
* api doc
- secondary role not used anymore. stated.
- primary database is not clear. replaced with dbserver
- brief referenced only dbserver and coordinator - better to provide wider description, in line with what is described below, as other roles can be returned
* typo
* typo
* added starting from 3.4
* additional warning
* cited in the release note
* Added authentication tests for ArangoSearch views creation (client and task)
* Added authentication tests for ArangoSearch views querying (client and task)
* Added authentication tests for ArangoSearch views update (client and task) and correct other tests
* Added authentication tests for ArangoSearch views drop (client and task) and correct other tests
* Fix tests to support additional 'properties' layer of a view
* Add data to dumped/restored collection with a link from a view to single server test and view dump/restore to cluster tests
* backport from iresearch upstream: use a bitvector instead of an acceptor lambda for consolidation policies
* ensure view access permissions are checked for REST calls
* ensure view access permissions are checked for V8 calls
* do not allow LogicalView property modification if reading the properties fails
* Fixed authentication tests for ArangoSearch views to pass regardless of errorNum and possible multiple REST codes
* add missing ldap feature for enterprise version
* do not store useless data in ArangoSearch index (see #289, #441)
to successful initial creation of these collections
Instead, try to create the indexes on every instance start
That way even if the index creation failed initially, the indexes
will be created on the next startup, and no user intervention
is required but restarting
The previous implementation did not create the indexes at all
then, and left the user with a message saying "please create
the index manually"
* issue 410.3: remove the extra neting of ArangoSearch view definition into 'properties'
* address merge issue
* backport: address test failures
* backport: address dump test failure
* Added cleanup functions class that allows to execute functions on unexpected shutdown (like FATAL_EXIT). First use: delete the LOCK file. (#5218)
* Removed superfluous log message
Dan Larkin-York
Andrey Abramov
jsteemann
Simon
Matthew Von-Maszewski
Jan
Tobias Gödderz
Vasiliy
Max Neunhöffer
Michael Hackstein
Heiko
Lars Maier
Kaveh Vahedipour
Wilfried Goesgens
Frank Celler
Dan Larkin-York
Andrey Abramov
Simran
Frank Celler
Vadim Kondratyev
Shivam Dave
Jan Christoph Uhde
shivamdave24