1
0
Fork 0
Commit Graph

527 Commits

Author SHA1 Message Date
Max Neunhöffer ac689fc725 Allow to ignore superuser traffic with an option. (#10289)
* Allow to ignore superuser traffic with an option.

Also: Fix some accounting bugs.

* CHANGELOG.

* Implement separate accounting for superuser and user traffic.

* Fix VST case with no authentication.

* Add description of new user-only traffic statistics.

* Add figures for user traffic.

* Simplify forwarding accounting.

* Remove brackets in metrics names.

* Finalize naming of metrics.
2019-10-24 13:11:51 +03:00
Jan 2c5f79c9fb Make scheduler enforce queue limits (#10026)
* initial commit

* fix typo

* honor @mpoeter 's comments. Thanks!

* honor @mpoeter 's comment

* adjust scheduler queue sizes

* apply suggestion

* adjust the PR for 3.5: do not use bounded_push
2019-10-16 17:43:04 +03:00
Jan 345a06ade4 remove debug logging (#10192)
* remove debug logging

* removed yet another log message
2019-10-09 14:58:15 +03:00
Max Neunhöffer b3a1b028d8 Disable hotbackup api on Windows. (#10131) 2019-10-01 23:47:19 +03:00
Max Neunhöffer 328f46e3d6 This merges hotbackup and atomic-db-creation into 3.5. (#9968)
* Squashed commit of feature-3.5/hotbackup_devel.

This puts hotbackup into 3.5.

* Port atomic-database-creation-2 to 3.5.

* Remove some wrongly ported code.

* Fix compilation.

* Fix a manual merge error.

* Remove a feature from the mocks which does not exist in 3.5.

* Add some code which was forgotten in manual merge.

* Fix a problem introduced in a manual merge.

* reuse function

* Address some whitespace issues that came up in review

* aardvark should not create the frontend collection

* create _frontend collection from c++

* recheckAndUpdate Callback in CollectionWatcher

* Wrong author ;)

* rm outdated todo

* Update lib/Basics/VelocyPackHelper.h

Co-Authored-By: Michael Hackstein <michael@arangodb.com>

* use logger unique id, use startup logger

* not needed

* optimized vector shardid method

* do not create _modules collection lazy anymre

* Formatting.

* Assert instead of if/TRI_ASSERT(false)

* Don't use exceptions as control structure

* Re-add READ_LOCKER that got lost in translation

* Fix audit log in case database creation fails early.

* legacy sharding

* Add CHANGELOG entry.

* Retry database cancellation indefinitely

* Do not use exceptions in UpgradeTask

* DropCollection is a FAST_LANE action and should not need much time or else retry.

* Remove superflous addition of LdapFeature

Proudly brought to you by ASAN tests

* Fixed check for distributShardsLike sharding on _system database

* Fixed compile issue on tests

* Removed assertion that seems to be not correct yet on devel.

* Sort out google cloud storage as remote. (#9918)

* Add successful method to ClusterCommResult.
* Improve error forwarding for cluster internal communication.

* Feature/hotbackup list retries (#9924)

* retry hot backup listing for 2 minutes in cluster before giving up

* Enable api by default.

* fix broken list of non existing id (#9957)

* Fix compilation after manual merge.

* Fix another compilation problem.

* Yet more fixes for compilation.

* More compilation fixes.
2019-09-11 13:13:54 +03:00
Jan bc1df0e78d adjust log level for "closing stream" message (#9815) 2019-08-27 16:51:25 +03:00
Michael Hackstein d5840c125a Bug fix 3.5/min replication factor (#9524)
* Cherry-pick minReplicationFactor

* Bug fix/failover with min replication factor (#9486)

* Improve collection time of IResearchQueryOptimizationTest

* Added a minReplicationFactor field in Collections. It is not possible to modify it yet and noone cares for it

* Added some assertion son minReplicationFactor

* Transaction API will now reject writes as soon as minimal replication factor is NOT fulfilled

* added minReplicationFactor to the user interface, preparation for the collection api changes

* added minReplicationFactor to VocBaseCollection, RestReplicationHandler, RestCollectionHandler, ClusterMethods, ClusterInfo and ClusterCollectionCreationInfo

* added minReplicationFactor usage to tests

* TODO TEMOPORARY COMMIT FOR TESTING PLEASE REVERT ME

* minReplicationFactor now able to change via collection  properties route

* fixed wrongly assert

* added minReplicationFactor to the graph management ui

* added minReplicationFactor to the gharial api

* Fixed off-by-one error in minReplicationFactor. We actually enforced one more.

* adjusted description of minReplicationFactor

* FollowerInfo Refactoring

* added gharial api graph creation tests with minimal replication factor

* proper cleanup of shell collection tests, removed lots of duplicate code, preparation for some new tests

* added collection create tests using invalid/valid names, replicationFactor and minReplicationFactor

* Debug logging

* MORE Debug logging

* Included replication fast lane

* Use correct minreplicationfactor

* modified debug logging

* Fixed compileissues

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* MORE Debug logging

* Revert "MORE Debug logging"

This reverts commit dab5af28c0.

* Revert "MORE Debug logging"

This reverts commit 6134b664bd.

* Revert "MORE Debug logging"

This reverts commit 80160bdf3b.

* Revert "MORE Debug logging"

This reverts commit 06aabcdfe1.

* Removed debug output

* Added replication fast lane. Also refactored the commands as i cannot take it any more...

* Put some requests of RocksDBReplication onto CATCHUP Lane.

* Put some requests of MMFilesReplication onto CATCHUP Lane.

* Adjusted Fast and MED lane usage in Supervised scheduler

* Added changelog entry

* Added new features entry

* A new leader will now keep old followers in case of failover

* Update arangod/Cluster/ClusterCollectionCreationInfo.cpp

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Fixed JSLINT

* Unified lane handling of replication handlers

* Sorry forgotten in last commit

* replaced strings with static strings

* more use of static strings

* optimized min repl description in the ui

* decr initial loop variable

* clean up of the createWithId test

* more use of static strings

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Added some comments on condition, renamed variable as suggested in review

* Added check for min replicationFactor to be non-zero

* Added assertion

* Added function to modify min and max replication factor in one go

* added missing semicolon

* rm log devel

* Added a second information to follower info that can keep track of followers that have been in sync before a failover has taken place

* Maintenance reports previous version now to follower info. instead of lying by itself. The Follower Info now gets a failover save mode to report insync followers

* check replFactor against nr dbservers

* Add lie reporting in CURRENT

* Reverted most of my recent commits about Failover situation. The intended plan simply does not work out

* move replication checks from logical collection to rest collection handler

* added more replication tests

* Include assert only if we are not in gtest

* jslint

* set min repl factor to zero if satellite collection

* check replication attributes in v8 collection

* Initial commit, old plan, does not yet work

* fixed ires tests

* Included FailoverCandidates key. Not fully implemented

* fixed wrong assert

* unified in sync follower reporting

* fixed compiler errors

* Cleanup locking, and fixed potential deadlocks

* Comments about locking order in FollowerInfo.

* properly check uint

* Keep old leader as potential failover candidate

* Transaction methods now use followerInfo to check if the leader can write, this might have the sideeffect that 'failoverCandidates' are updated

* Let agency check failoverCandidates if possible

* Initialize member variables

* Use unified follower reporting in DBServerAgencySync

* Removed obsolete variable, collecting it somewhere else

* repl factor attr check

* Reimplemented previous followers, second attempt now. PhaseOne and PhaseTwo can now synchronize on current.

* Fixed assertion, forgot an off-by-one

* adjusted test to be more preciese now

* Fixed failove candidates list

* Disable write on dropping too many followers

* Allow to run updateFailoerCandidates multiple times with same leader.

* Final fixes, resilience tests now green, crossing fingers for jenkins

* Fixed race on atomics comparison

* Fixed invalid number type

* added nullptr handling

* added nullptr handling

* Removed invalid assert

* Make takeover of leadership an atomic operation

* Update tests/js/common/shell/shell-cluster-collection.js

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Review fixes

* Fixed creation code to use takeoverLeadership

* Update arangod/Cluster/FollowerInfo.h

Co-Authored-By: Tobias Gödderz <tobias@arangodb.com>

* Applied review fixes

* There is no timeout

* Moved AQL + Pregel to INTERNAL_AQL lane, which is medium priority, to avoid deadlocks with Sync replication

* More review fixes

* Use difference if you want to compare two vectors...

* Use std::string ...

* Now check if we are in recovery mode

* Added documentation for minReplicationFactor

* Added readme update as well in documenation

* Removed merge conflict leftovers 0o, i should not trust the IDE

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/collectionsView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/Books/Manual/Architecture/Replication/README.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update CHANGELOG

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/Books/Manual/DataModeling/Collections/DatabaseMethods.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/Books/Manual/ReleaseNotes/NewFeatures35.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/DocuBlocks/Rest/Collections/1_structs.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/graphManagementView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update js/apps/system/_admin/aardvark/APP/frontend/js/views/graphManagementView.js

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Update Documentation/DocuBlocks/Rest/Graph/1_structs.md

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Apply suggestions from code review

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Adepted review requests, thanks for finding!

* Removed unnecessary const

* Apply suggestions from code review

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>

* Moved initilization of variable more downwards

* Apply lock before notify_all()

* Remove documentation except DocuBlocks, covered by PR in docs repo

* Remove accidental indent
2019-07-22 17:48:34 +03:00
Jan 47d7f9e3e2 Bug fix 3.5/fix uninitialized value (#9515)
* fix uninitialized time point value, which led to potential bogus error
messages

* fix typo in info message
2019-07-18 23:26:51 +03:00
Simon 1432156e91 Port changes to fix the keep alive timeout (#9483) 2019-07-16 19:39:40 +03:00
Lars Maier eb1aa6e024 Response compression (#9300)
* First draft of response compression.

* Cleanup.

* Removed compression from /_api/version.

* Added ruby test for response compression.
2019-06-28 10:02:48 +02:00
Tobias Gödderz da6e9da820 Measure time a request takes to get executed by the scheduler (#9224) 2019-06-18 17:34:33 +02:00
Frank Celler 2d732be590
wait until task is shut down (#9239) 2019-06-10 10:07:31 +02:00
Simon 1402395f1c Port fuerte changes from network pool branch (#9125) 2019-06-04 16:35:15 +02:00
jsteemann a0676dcf20 add commtask debugging again 2019-06-03 11:10:31 +02:00
Jan 208f22c842
only execute requests directly if there is one client on the IoContext (#9090) 2019-05-24 14:29:58 +02:00
Jan 3d79491f36
exclude VST requests from direct execution (#9075) 2019-05-24 09:53:15 +02:00
Jan 79258e072a
Bug fix/remove io task (#9056) 2019-05-22 14:34:49 +02:00
jsteemann 0e24c18253 re-add optimization for single servers 2019-05-22 14:29:12 +02:00
Lars Maier 4fc2790863 [devel] Direct Exec Scheduler (#9004) 2019-05-20 11:38:57 +02:00
jsteemann 0f6af587ad remove debug output 2019-05-16 19:49:22 +02:00
jsteemann 01dad862a4 move stopping more down 2019-05-16 13:52:46 +02:00
jsteemann 73a7990a31 add debugging for CommTasks creation & destruction 2019-05-16 10:37:12 +02:00
Jan 30e9fbafe0
Bug fix/cleanup commtasks (#8893) 2019-05-08 16:33:19 +02:00
Jan Christoph Uhde e0b073543b add symlinks for windows (#8846) 2019-05-02 17:32:29 +02:00
Jan 0cbdfe9289
Bug fix/vpack update (#8875) 2019-04-30 12:33:26 +02:00
Jan 1408654d2c
fix a few issues in BackupNoAuthSysTest (#8868) 2019-04-29 19:13:39 +02:00
Jan Christoph Uhde 677a79026c Foxx Security (#8845) 2019-04-25 09:56:29 +02:00
Dan Larkin-York 777ba1a364 Improved audit logging (#8740) 2019-04-15 14:51:32 +02:00
Jan 13b2d40aef
fix compile warnings (#8743) 2019-04-15 11:22:24 +02:00
Vasiliy 0410588723 issue 526.7.1: remove deprecated functions, update tests to set up required analyzers (#8688)
* issue 526.7.1: remove deprecated functions, update tests to set up required analyzers

* address merge issues

* address more merge issues
2019-04-05 18:10:42 +03:00
Lars Maier 06526b0a32 Feature devel/allowed paths jwt (#8678) 2019-04-04 13:55:02 +02:00
Simon 7cd84a785a Remove Obsolete code (#8657) 2019-04-03 13:40:44 +02:00
Jan 207c978d1e
Bug fix/move a few actions from js to cpp (#8549) 2019-03-26 10:02:21 +01:00
Jan Christoph Uhde c3f7961b88 apply unique log ids (#8561) 2019-03-25 20:26:51 +01:00
Jan e078f35285
fixed typos, removed unneeded includes (#8547) 2019-03-25 12:09:37 +01:00
Jan f2ab6da5f8
dont whine about --server.jwt-secret if it isn't used (#8546) 2019-03-25 12:08:43 +01:00
Simon 3ada15fc35 The Legendary El Cheapo (#8485) 2019-03-22 11:38:33 +01:00
Lars Maier e598148707 Bug fix/jwt secret file logging (#7976)
* Print with correct log topic.

* Moved message to a place where logger is enabled.
2019-03-20 14:14:13 +01:00
Kaveh Vahedipour ee751e8ba3 [devel] clear compilation warnings (#8345) 2019-03-08 10:35:09 +01:00
Jan 3755b4ab7d
remove unused includes (#8269) 2019-02-26 18:59:01 +01:00
Jan 44c6a2d732
Feature/ttl index (#8169) 2019-02-19 14:12:21 +01:00
Jan 0053a72a6a update velocypack (#8076) 2019-01-31 17:31:14 +01:00
Simran 33220fb589 Doc - Deprecate --server.jwt-secret startup option (#7947) 2019-01-28 09:12:18 +01:00
Jan e42befdc52
do not simplify non-deterministic conditions (#7926) 2019-01-11 14:48:34 +01:00
Jan fa7de56cf8
upgrade to boost 1.69.0 (#7910) 2019-01-09 17:17:33 +01:00
Lars Maier 12eebb15fe Feature/new server infra (#7733)
* Decoupled IO handling from Scheduler.

* Fixed SSL start up bug.

* Replaced Scheduler with new worker farm implementation.

* Added minimal statistics and info string for Scheduler.

* Added support for timed submissions.

* Updated delayed submission api. Updated code that used timers.

* Extracted new Scheduler into a virtual parent class. The implementation can now depend on the usecase.

* Signal handler now working.

* Changed threads names, `_stop` is atomic, check for failure during thread start + exception handling like old scheduler did.

* Commented on source code and added TODOs.

* Played around with start-stop-conditions

* Play around with start stop condition.

* start stop cond

* Sart Stop Conditions

* Removed bad cv_status check.

* Bug fix: now compare the actual objects instead of pointer values. Setup t1 and t2 depending on the thread id.

* Moved most of the stuff now unrelated to the Scheduler to GeneralServer. Got rid of JobGuard.

* Instead of waiting for a thread to terminate, put it on a clean up list and check for its termination in each supervisor run.

* Allow detaching long running threads.

* Fixed test mock.

* Updated the WorkHandle logic. Removed post functions.

* Fixed crash when obtaining shared_ptr from this in destructor.

* Added lost mutex.

* Fixed memory leak.

* Fixed merge bug.

* Changed a lot of code to optimize the scheduler.

* Fixed bug of invalidated iterator. Dont remove task on shutdown at different places. Let scheduler threads run until queue is empty.

* Only by value calls to queue.

* Added options again.

* Clean up of code.

* UI Request Lane added.

* Bug fixes in Scheduler.

* Applied reformat.

* Use sigaction.
2019-01-08 10:12:02 +01:00
Lars Maier 5c53ebceb0 Feature/jwt keyfile (#7863)
* Added jwt-keyfile option and warning for old option.
* CHANGELOG
* Add trimming to --auth.jwt-secret-keyfile
* Adjust some comments.
2019-01-02 14:38:38 +01:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Jan 99809a445e
do not use an internal error for JSON parse errors (#7799) 2018-12-19 12:54:47 +01:00
Max Neunhöffer c78e3c89e0
Call license key check. (#7594) (#7704)
* Call license key check. (#7594)

* Add CHANGELOG entry.

* Fix super user JWT token behaviour with non-ex. db. (#7656)
2018-12-10 10:33:00 +01:00