1
0
Fork 0
Commit Graph

87 Commits

Author SHA1 Message Date
Dan Larkin-York 8f9ebce0e0 [3.4] Fix issue with cache allocation statistics. (#10030)
* Fix issue with cache allocation statistics.

* updated CHANGELOG
2019-09-20 16:20:20 +03:00
Simon 185c8724a3 Pregel various fixes (#8879) 2019-05-07 09:53:18 +02:00
Matthew Von-Maszewski 474f0cde31 Bug fix 3.4/scheduler empty reformat (#7872)
* added check for empty scheduler

* removed log, old is 1 not 0

* require running in this thread

* test

* added isDirect to callback

* signature fixed

* added drain

* added allowDirectHandling

* disabled for testing

* Add ExecContextScope object to direct call.

* try alternate initialization of ExecContextScope

* remove ExecContextScope, no help.  try _fifoSize as part of direct decision.

* strand management to minimize reuse of same strand per listen socket

* blind attempt to address Jenkins shutdown lock up.  may remove quickly.

* add filename and line to existing error log message

* Adjust queueOperation() to stop accepting items once isStopping() becomes true.

* revert previous check-in to MMFilesCollectorThread.cpp

* big reformat

* fixed merge conflicts

* Add CHANGELOG entry.
2019-01-08 20:39:42 +01:00
Frank Celler 9477af198b big reformat 2018-12-26 00:57:05 +01:00
Jan 1a4a6e7d2b
minor fixes (#7691) 2018-12-07 16:22:32 +01:00
Jan f1086bac4f
added option `--rocksdb.enforce-block-cache-size-limit` (#7508) 2018-11-28 20:40:20 +01:00
Matthew Von-Maszewski a9ce39f85c Bugfix 3.4: Merge scheduler changes by Michael & Frank into recent overlapping code changes (#6928)
* manual recreation of bug-fix-3.4/scheduler-high-low within recent Scheduler changes.

* restore Documentation that was unintentionally deleted
2018-10-16 22:51:00 +02:00
Vasiliy 6fd541d110 issue 427.5: use ApplicationServer reference instead of pointer (#6145)
* issue 427.5: use ApplicationServer reference instead of pointer

* address MSVC build failure
2018-08-15 12:16:02 +03:00
Frank Celler a688dc0962
Feature/remove job queue thread (#5986)
limiting V8 calls in flight
2018-08-10 12:17:43 +02:00
Michael Hackstein 7a95c5e675
Feature/feature phases (#5272)
* Added feature phases

* BasicsPhase and DatabasePhase to the required files. Server now has Feature circles and does not boot. Will be sorted out later on.

* Added ClusterPhase to features

* Added V8Phase to the required features

* Added AQLPhase to the affected features

* Added ServerPhase to Features

* Added FoxxPhase to the relevant features

* Added AgencyPhase to the relevant features

* Moved registration from local variable SYS_SYSTEM_REPLICATION_FACTOR from cluster to V8 as their ordering is now vice versa

* Moved Bootstrap feature into FoxxPhase. It could be moved to ServerPhase easily if the FoxxQueue dependency would be removed

* Final movement of Startup Phases. Now solved all circles.

* Removed merge conflict

* Moved ReplicationTimeout into cluster phase and fixed cross-phase requirements

* Added greetings phase. This phase separates the Basics Phase and is the first to be run. Includes Logger and Hello/Goodbye

* Added the GreetingsPhase in the corresponding features. Now all BasicsPhase features start after greetings Phase. There is some issue in this branch which prevents the Agency from Gossipping right now. Will be fixed next

* Moved creation of the Agent into the prepare phase of the feature. THereby it is guaranteed that agents at least exists before the GeneralServer is activating endpoints

* Recovery needs to be started after the ServerID

* Moved log output of FeaturePhases to DEBUG instead of ERROR.

* Added feature phases for clients

* ClusterFeature now does not directly require AgencyFeature any more

* Added requirement of TravEngineRegistryFeature in AQL feature. Otherwise shutdown may be undefined

* The ApplicationServer can now handout the list of ordered features. Used for testing purposes

* Fixed IResearchVew Tests Setup to honor new feature ordering

* Fixed IResearchViewDBServer Tests Setup to honor new feature ordering

* Started fixing IResearchView Coordinator tests with startup ordering. Not finished yet

* Added startup phases to ViewCoordinator test

* Disabled expected logoutput in ClusterRepairsTest

* Fixed indention in test code

* LinkCoordinator now honors startup ordering

* Link meta now honors startup rdering

* Supress expected cluster logs in ViewTest

* Removed '#' accidentially added.
2018-07-16 14:09:36 +02:00
Dan Larkin-York 66e9fb1dbd Avoid memset on non-POD types for portability. (#5558) 2018-06-12 08:49:37 +02:00
Wilfried Goesgens ee9352c0ce Bug fix/refactor read write locks (#5215) 2018-05-24 21:18:00 +02:00
Jan 72ec463a8d
Bug fix/cleanup 2005 (#5404) 2018-05-20 20:57:43 +02:00
Simon 17b1a2aafb Rest middleware refactoring (#5332) 2018-05-14 17:43:10 +02:00
jsteemann 7f8a1cc614 Merge branch 'bug-fix/add-missing-overrides-and-final' of https://github.com/arangodb/arangodb into devel 2018-05-07 23:02:46 +02:00
Simon fdee0544b7 Using asio::io_context::strands instead of locks (#5266)
* initial try adding strands

* working, stable amount of threads

* improve shell_client cluster

* Fixing some accounting for the scheduler

* Fix accounting

* Fixing wrong strand usage

* add missing return

* Fixing thread accounting

* More scheduler accounting issues

* Fixing various things

* Fixing some stuff

* Fixing some stuff

* Some more subtle bugfixes

* Some cleanup code

* fixing some stuff

* adding some more fixes

* Fixing possible issues

* Fixing missing _storeResult

* Fixing some stuff

* Reducing lambda stack, perhaps fixing hangups

* Fix writeunlocker

* Fixing possible issues

* adding some debugging stuff

* refactor sockets

* possible fixes

* Adding more job guards

* Fixin possible bug

* cleaning up some stuff

* working impl

* Remove debugging output

* Fixing build

* fixing import

* Fixing another bug

* removing debug log

* Removing examples

* Reverting scheduler working code

* Cleanup

* Addressing review comments
2018-05-07 15:58:19 +02:00
jsteemann 52de92d334 add missing override specifiers, add final specifiers 2018-05-04 09:01:50 +02:00
Jan 2b84348b77
remove call to requiresElevatedPrivileges with default value (#5172) 2018-04-23 11:28:24 +02:00
Dan Larkin-York abe3f9134e Fix a small logic issue in cache resize metadata method. (#4631) 2018-02-20 18:46:13 +01:00
Dan Larkin fcc5db8876 Downgraded log level for debug statements in caching system. (#4448) 2018-01-26 17:07:29 +01:00
Simon Grätzer dce677720d Fixing an issue with intermediate commits (#3975) 2017-12-12 09:17:18 +01:00
Dan Larkin 5e3b5526d0 Fixed return value of cache rebalancer in a few spots. (#3967) 2017-12-08 16:01:54 +01:00
Jan 833f026160
fix issues found by codescan (#3870) 2017-12-06 10:44:19 +01:00
Jan e8ef1a21b6 fix some issues found by coverity code scan (#3256) 2017-09-21 10:26:59 +02:00
Jan 5165155ed1 Bug fix/fixes 0609 (#3227)
* do not use V8 variant of AQL functions in early optimization stage when a C++ variant is available

* additionally, simplify AQL function definitions and aliases

* warn when more than 90% of max mappings are in use

* added C++ variant of replication catchup

* added `--log.role` option

* updated CHANGELOG

* removed non-existing scheduler.threads option from config

* removed useless __FILE__, __LINE__ invocations

* updated CHANGELOG

* allow a priority V8 context

* remove TRI_CORE_MEM_ZONE

* try to fix Windows errors & warnings

* cleanup

* removed memory zones altogether

* exclude system collections from collection tests
2017-09-13 16:28:21 +02:00
Simon Grätzer 88d01b89b5 Optimizations for Caches and Graph Traversals (#3169) 2017-08-31 18:33:10 +02:00
Dan Larkin e2fdab431c Fix compiler warnings and needlessly failing catch tests. (#3171) 2017-08-31 17:27:25 +02:00
jsteemann 5677baf561 fixed some MSVC warnings 2017-08-30 13:11:55 +02:00
Dan Larkin 6a068d062e Changed some tasks to post through Scheduler instead of directly. (#3159) 2017-08-30 10:41:41 +02:00
Dan Larkin 18e8c19271 Improvements to CachedValue internals and handling. (#3126)
* Added checks everywhere for CachedValue::construct.

* Enforce 16MiB limit on keys and 4GiB limit on values.

* Protected members behind function calls in anticipation of format change.

* Shrunk CachedValue struct.

* Fixed alignment of CachedValues.

* Fixed improper comparison.
2017-08-28 15:07:16 +02:00
Frank Celler 6d08d4f4aa Bug fix/scheduler delete (#3077)
* removed delete call

* cleanup

* lower cpu activity of log thread too

* fix log messages

* do not enter threads into unordered_set, as it is unneeded

* do not compile in calls to disabled plan cache

* moved AQL regex cache from thread local variables to a class of its own

* more sensible thread creation and destruction
2017-08-25 12:00:17 +02:00
Jan a212f45a86 Bug fix/issues 2308 (#3117)
* remove API docs for endpoints that were deleted in 3.0

* remove unused variable

* set all scalar values in the ctor, so check tools do not complain

* move Result objects

* prevent potential nullptr access

* remove ambiguous boolean parameter with no clear meaning

* make code check tools happy

* don't allow copying cache Findings, allow moving Results

* the debug functionality for counting RocksDB key ranges was broken

this produced sporadic test failures

* remove superfluous iostream include

* fix range checks

* don't leak if an error happens during database removal

* try to simplify Result a bit

* try to deletion checks a bit more robust
2017-08-24 14:47:22 +02:00
Simon Grätzer b76b53d87a Fix edge cache behaviour (#3079)
* fixing cache resizing bug

* reenabling test
2017-08-22 10:40:52 +02:00
Dan Larkin 77c63bdc9f Fixed a deadlock issue in cache shutdown. (#2661) 2017-06-26 16:30:32 +02:00
Dan Larkin a3ae2b7191 Augmented cache API and fixed a potential deadlock scenario. (#2579)
* Augmented cache API and fixed a potential deadlock scenario.

* The RocksDBEdgeIndex now tries to read up to 10 times form the cache on lock_timeout. Instead of rewriting everything after first attempt
2017-06-19 17:28:04 +02:00
Dan Larkin a0c4e3d9df Improved RocksDB engine statistics with docs and rest endpoint (#2598)
* Improved statistics reporting for RocksDB engine (including cache info).

* Added documentation about db._engineStats().

* Added rest endpoint for engine stats.
2017-06-18 22:55:25 +02:00
jsteemann f40a6b622f removed unused includes 2017-06-08 13:37:30 +02:00
Michael Hackstein 52406567cd Replaced EdgeCursors callback to return cid,rev instead of _id. Does less costly copying and comparision internally 2017-05-31 17:49:46 +02:00
Dan Larkin a4aa1eedd2 Updated plain cache to properly count evictions. 2017-05-30 13:04:30 -04:00
Michael Hackstein b9d2faa7cc Added warmup function for indexes.
Squashed commit of the following:

commit bc0472d212b2efef0d7b1b8a36f2b43e8432ba86
Merge: eb47631 5f87674
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:39:31 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit eb476310b0ca6165be10b37b960933886c2757f5
Merge: d725f21 32149d2
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:00:41 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit d725f21f7d61f7c79ba31fc0338881e35f4d8f48
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 15:53:23 2017 +0200

    Activated collection warmup in Cluster Mode. Added a test for collection warmup (SingleServerOnly)

commit 132bf4f9565b0dcf4ec9f84e93897b482a67ce7f
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 12:46:20 2017 +0200

    Implemented client-side warmup. It can now be triggered via http and arangosh

commit 78ea449dff86118814a2f87bdb59dc16544d92b6
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:25:40 2017 +0200

    Fixed assertion.

commit dae80f6277dde1a52eadda506858cc36e235bd55
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:08:46 2017 +0200

    Improved the CuckooIndexEstimator computeEstimate function. It is now much better on collections with many different values

commit 7abf57876511ba369d7a577e1995d4575e98c7c8
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:06:43 2017 +0200

    Edge index warmup will now first check if a document is stored and if not will insert it. Furthermore it resizes the cache with an estimate of how many documents will be most likely inserted

commit 890d8ad4cdfd155249f060fedd5c798b9531d556
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:04:47 2017 +0200

    Adjusted thresholds in transactional cache. Wastly increased the amount of tries to get the lock. Lowered the fill-grade boundaries

commit 60972ab7151a3acb78e1aae4149de11d0da7aceb
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:45:38 2017 +0200

    Added new Logtpopic CACHE that should be used to log important information regarding caches, like debug ouput on evictions / resizing, or memory-pressure resizing going on

commit 2dfacb9aef6e3fde169032514baca386786d059c
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:44:21 2017 +0200

    Fixed an assertion in index iterator. It expected _from/_to to be cached but we modified to cache entire document

commit f05beccef65853c30eda1859a39c28ea2295bc71
Merge: 13c6abf 5c46430
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 07:56:58 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit 13c6abfbc2ddb451a689011110411df68247435e
Author: Michael Hackstein <michael@arangodb.com>
Date:   Wed May 24 09:52:01 2017 +0200

    Added a getExtra feature for the edgeIndex. It now returns the complete edge as extra (temporary). Modified the EdgeCache to cache token and edge document (temporary). Added a warmup Function to collections that is used to warmup the index caches on demand.
2017-05-29 09:40:58 +02:00
Dan Larkin 5ec026b659 Added sizeHint method for caches to improve warmup procedures. 2017-05-24 13:45:53 -04:00
Dan Larkin e935c83762 Exposed isMigrating() method on caches. 2017-05-24 10:44:34 -04:00
Dan Larkin 71846d0ad7 Added hinting so cache table is aware of eviction rate problems. 2017-05-23 19:05:50 -04:00
Dan Larkin 78c80c3a3d Doubled size of cache buckets to increase associativity. 2017-05-23 17:51:47 -04:00
Dan Larkin 8decc3c8b8 Fixed some uninitialized values. 2017-05-17 09:38:31 -04:00
Andreas Streichardt af4da5796f More windows warnings 2017-05-16 17:00:31 +02:00
Dan Larkin 5817de528f Adjusted some cache constants and added eviction rate tracking. 2017-05-15 13:24:30 -04:00
jsteemann 11cf6ed5c6 re-set fillgrades back to original values 2017-05-15 16:47:41 +02:00
jsteemann fc44711531 adjust fillgrades as suggested by @neunhoef 2017-05-15 15:03:29 +02:00
jsteemann f49a32dff9 added missing casts 2017-05-15 14:39:27 +02:00