1
0
Fork 0
Commit Graph

332 Commits

Author SHA1 Message Date
Jan ea915d3d69
turn off the in-memory cache on agency nodes, as it is not needed there (#6742) 2018-10-08 09:39:45 +02:00
Simon 877c9d9eb3 Do not load counters until recovey is done (#6719) 2018-10-05 12:52:00 +02:00
Jan 3414825696
Feature/improvements 031018 (#6712) 2018-10-04 15:10:54 +02:00
Dan Larkin-York 302f6d1852 Fix compile mistakes. 2018-10-02 08:55:49 -04:00
Dan Larkin-York 44600636e7 Cleanup logging in RocksDBEngine. 2018-10-02 07:50:02 -04:00
Jan 55cd434aa2
fix handling of range deletions (#6639) 2018-09-28 11:09:41 +02:00
Jan d1241b1deb
Bug fix/aql modify cleanup (#6380) 2018-09-11 08:54:45 +02:00
Jan 840ad8c20d
added option `--rocksdb.use-file-logging` (#6308) 2018-08-31 17:23:00 +02:00
Jan 5022ccc24d
Bug fix/fixes 2508 (#6254) 2018-08-27 21:36:39 +02:00
Jan d78adb7ae1
Bug fix/turn off range deletes in cluster (#6178) 2018-08-17 08:41:00 +02:00
Simon 392118bd62 Use RangeDelete where possible (#6121) 2018-08-15 18:52:09 +02:00
Vasiliy 6fd541d110 issue 427.5: use ApplicationServer reference instead of pointer (#6145)
* issue 427.5: use ApplicationServer reference instead of pointer

* address MSVC build failure
2018-08-15 12:16:02 +03:00
Jan b278d6874a
allow master & slave to work in parallel for RocksDB WAL tailing (#6059) 2018-08-03 13:37:53 +02:00
Matthew Von-Maszewski 9f880dd529 (devel) add bool option rocksdb.debug-logging to easily enable rocksdb logging (#6016) 2018-07-30 14:30:20 +02:00
Jan 1f16898027
added more RocksDB options and stats (#6006) 2018-07-26 16:48:25 +02:00
Jan 21023945ba
disable RocksDB background syncing on Windows (#6004) 2018-07-26 16:48:11 +02:00
Simon 2dd8593609 View Replication (#5915) 2018-07-26 10:28:46 +02:00
Jan 21064144c8
Bug fix/replication improvements (#5962) 2018-07-25 09:04:50 +02:00
Jan 1a128799f2
finalize `--rocksdb.sync-interval` feature (#5856) 2018-07-17 11:09:57 +02:00
Michael Hackstein 7a95c5e675
Feature/feature phases (#5272)
* Added feature phases

* BasicsPhase and DatabasePhase to the required files. Server now has Feature circles and does not boot. Will be sorted out later on.

* Added ClusterPhase to features

* Added V8Phase to the required features

* Added AQLPhase to the affected features

* Added ServerPhase to Features

* Added FoxxPhase to the relevant features

* Added AgencyPhase to the relevant features

* Moved registration from local variable SYS_SYSTEM_REPLICATION_FACTOR from cluster to V8 as their ordering is now vice versa

* Moved Bootstrap feature into FoxxPhase. It could be moved to ServerPhase easily if the FoxxQueue dependency would be removed

* Final movement of Startup Phases. Now solved all circles.

* Removed merge conflict

* Moved ReplicationTimeout into cluster phase and fixed cross-phase requirements

* Added greetings phase. This phase separates the Basics Phase and is the first to be run. Includes Logger and Hello/Goodbye

* Added the GreetingsPhase in the corresponding features. Now all BasicsPhase features start after greetings Phase. There is some issue in this branch which prevents the Agency from Gossipping right now. Will be fixed next

* Moved creation of the Agent into the prepare phase of the feature. THereby it is guaranteed that agents at least exists before the GeneralServer is activating endpoints

* Recovery needs to be started after the ServerID

* Moved log output of FeaturePhases to DEBUG instead of ERROR.

* Added feature phases for clients

* ClusterFeature now does not directly require AgencyFeature any more

* Added requirement of TravEngineRegistryFeature in AQL feature. Otherwise shutdown may be undefined

* The ApplicationServer can now handout the list of ordered features. Used for testing purposes

* Fixed IResearchVew Tests Setup to honor new feature ordering

* Fixed IResearchViewDBServer Tests Setup to honor new feature ordering

* Started fixing IResearchView Coordinator tests with startup ordering. Not finished yet

* Added startup phases to ViewCoordinator test

* Disabled expected logoutput in ClusterRepairsTest

* Fixed indention in test code

* LinkCoordinator now honors startup ordering

* Link meta now honors startup rdering

* Supress expected cluster logs in ViewTest

* Removed '#' accidentially added.
2018-07-16 14:09:36 +02:00
Vasiliy 1240090cd8 issue 399.2: use a LogicalCollection& instead of LogicalCollection* in PhysicalCollection (#5861) 2018-07-13 15:02:43 +03:00
Jan b88fde9659
added key generators "uuid" and "padded" (#5740) 2018-07-03 15:23:47 +02:00
Jan 297279f4f3
turn off MMFiles collection and compaction (#5734) 2018-07-03 15:23:27 +02:00
Simon 439b12331f Big endian document keys for rocksdb (#5712) 2018-07-03 14:11:22 +02:00
Simon 3bec336aff TransactionState::addCollection refactoring (#5606) 2018-06-14 15:34:58 +02:00
Vasiliy a5f274234a issue 380.2.2: remove extraneous pointers from StorageEngine API (#5589) 2018-06-13 15:22:51 +03:00
Jan Christoph Uhde f81e42af36 Add optimizer rule that replaces js function calls (NEAR/WITHIN/FULLTEXT) with pure AQL (#5529) 2018-06-12 13:48:31 +02:00
Heiko 743d4430bf added replication ui (#5555) 2018-06-08 11:19:05 +02:00
Vasiliy e021085aac issue 381.3.1: remove redundant copy of CollectionNameResolver from TransactionState (#5459) 2018-05-25 16:48:43 +03:00
Simon 35992ad67b Coordinator storage engine (#5405) 2018-05-22 19:30:27 +02:00
Vasiliy 843e584746 issue 389.5: refactor StandaloneContext to be constructed with a TRI_vocbase_t& (#5370)
* issue 389.5: refactor StandaloneContext to be constructed with a TRI_vocbase_t&

* backport: address build issues
2018-05-17 01:15:50 +03:00
Vasiliy 6a53154160 issue 389.2: use static strings for Index definition json attributes, use TRI_vocbase_t references instead of pointers in V8Context, use TRI_vocbase_t references instead of pointers in DatabaseInitialSyncer (#5344) 2018-05-14 19:06:24 +03:00
jsteemann 39da191b11 report missing cache hit information as 0 values, not as non-representable types 2018-05-11 14:49:42 +02:00
Vasiliy 2ce20e86d7 issue 373.1: move globally-unique id generation from collection into data-source (#5182) 2018-05-07 22:14:40 +03:00
Andrey Abramov b69b5bdfdf
Bug fix/issue #5186 (#5269)
* do not persist view on startup

* small refactoring

* ensure view is being opened after creation
2018-05-06 20:38:32 +03:00
Simon 828f1d423c S2 based Geo-Spatial index (#5249) 2018-05-02 23:54:41 +02:00
Vasiliy d470371f7c issue 374.2: use a reference to vocbase instead of a pointer in StorageEngime LogicalDatasource related DDL operations, use vocbase from the found collection 2018-04-11 18:49:46 +03:00
Vasiliy e4368b0991 issue 355.6: remove create() from LogicalView, remove IResearch dependency from IndexFactory, store vocbase reference in LogicalDataSource 2018-04-06 16:38:34 +03:00
Vasiliy 99b83ba8c8 issue 355.5: remove more unused methods, move view-related storage engine functionality from vocbase into DBServerLogicalView, address MSVC cmake dependency issue 2018-04-05 16:17:07 +03:00
Vasiliy 635db3b409 issue 355.4: remove redundant methods and code, use 'cp' instead of 'cmake copy_directory' where possible, use vocbase reference instead of pointer 2018-04-04 10:53:48 +03:00
Andrey Abramov 2cca7a926b remove code duplication 2018-03-30 16:01:00 +03:00
Andrey Abramov b641cf9b13
remove view implementation part 1 2018-03-26 23:18:08 +03:00
Andrey Abramov 6b9bffa343 remove PhysicalView part 3 2018-03-23 15:17:05 +03:00
Andrey Abramov 47fd5808c3 remove PhysicalView part 2 2018-03-22 22:41:28 +03:00
Andrey Abramov a874fda823 remove PhysicalView part 1 2018-03-22 20:42:42 +03:00
Vasiliy 06eb8ade01 issue 344.7: remove more redundant functions (#4863)
* issue 344.7: remove more redundant functions

* backport: fix missed functions under USE_ENTERPRISE
2018-03-15 17:10:28 +01:00
Vasiliy bf0068e7f5 Changes since last PR: use a single map in vocbase for storing LogicalDataSource instances 2018-03-08 17:28:08 +03:00
Vasiliy daa1db659c manually-merge: differentiate data-source types by category e.g. LogicalCollection/LogicalView 2018-03-08 13:40:25 +03:00
Vasiliy f9a45032cd LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups (#4732)
* Changes since last PR: use LogicalDataSource for Methods::StateRegistrationCallback instead of TRI_voc_cid_t to avoid unnecessary lookups
* backport: address cluster LogicalDataSource resolution failure
2018-03-06 11:41:41 +01:00
Simon cde96d97aa RocksDB replication hardening & cleanup (#4639) 2018-02-23 18:15:04 +01:00
Dan Larkin 934e936bae Fix index selectivity estimate issues with RocksDB engine (#4429) 2018-02-09 16:05:33 +01:00
Jan 8da61a9875
Bug fix/more replication tests (#4500) 2018-02-09 13:06:47 +01:00
Jan fe0fca9029
Bug fix/restore unlock (#4387) 2018-01-25 15:56:27 +01:00
Jan b32cf3822a
explicitly control activation of RocksDB write-throttling feature (#4360)
also consistently use -DNDEBUG to avoid different class layouts when
the same class are used by different translation units
2018-01-19 09:33:01 +01:00
Jan af87e115c1
next attempt (#4254) 2018-01-10 00:07:51 +01:00
Jan 7f860153ba
Bug fix/msvc fixes (#4243) 2018-01-08 11:20:53 +01:00
Jan e6572e6804
ensure proper shutdown of throttle (#4234) 2018-01-05 15:09:22 +01:00
Jan 52d0a4593e
Bug fix/add recovery tests for intermediate commits (#4108)
* added several recovery tests that check intermediate commits

* small fixes

* fix tests
2017-12-20 09:46:06 +01:00
Jan 2a96df5ca5
Feature iresearch (#4071) 2017-12-18 15:04:59 +01:00
Simon Grätzer 19f68df395 Use uuid in Initial sync to ensure proper collection mapping (#3965)
* Use uuid in initalsync to ensure proper collection mapping

* fix compile error after merge
2017-12-13 22:12:47 +01:00
Jan eb0190066b
allow passing a serverId value of "none" to replication APIs (#3909) 2017-12-08 11:35:53 +01:00
Jan 282be208cc
remove TRI_usleep and TRI_sleep, and use std::this_thread::sleep_for … (#3817) 2017-12-06 18:43:49 +01:00
Matthew Von-Maszewski 97944f012e Fix issues due to throttle started with too small a flush (#3707) 2017-11-16 09:13:08 +01:00
Jan 3b0a8a9cdf make the replication applier auto-start for the RocksDB engine if con… (#3647) 2017-11-15 12:02:37 +01:00
Max Neunhoeffer bc1891dfd0
Comment out RocksDBThrottle. 2017-11-14 16:53:42 +01:00
Matthew Von-Maszewski bbce4caefa Feature/mv basho smoothing1 (#3384) 2017-11-13 15:43:44 +01:00
Simon Grätzer 87f441753b RocksDB WAL tailing fixes (#3595) 2017-11-10 09:31:53 +01:00
Jan 7613bc4314 Bug fix/fixes 0211 (#3568)
* remove some non-unused V8 persistents

* do not throw that many bogus assertions

* do not rely on server role being defined

* slightly better debug output for V8 context debugging

* fix collection ids in inventory response

* simplify bootstrap a bit

* slightly better error handling

* make elapsed time a queryable value

* use less memory for stub collections

* added assertions that will always make sense

* added assertions

* do not garbage-collect while waiting

* less copying of parameters

* do not show "load indexes into memory" buttons for mmfiles engine

  as all indexes are in memory anyway

* when a collection is truncated via the web interface, flush the WAL and rotate all active journals

this will make close all open journals on leader and followers and make them subject to compaction opportunities

* fix invalid server id values being passed from web interface to backend

* introduce afterTruncate method for indexes

* added test case for issue #3447

* updated CHANGELOG

* don't warn about replicationFactor for system collections

* check that the queries actually use the geo index and not some other index

* properly report error in web interface

* fix some internals checks that made truncate fail for bigger collections in maintainer mode

* also run a compact() operation after a serious truncate

in order to make iteration over the truncated range much faster
when the collection is next accessed

* increase default maximum number of V8 contexts to at least 16
2017-11-09 12:48:15 +01:00
Jan 0195ede1d8 potential fix for issue #3581: Unexpected "rocksdb unique constraint violated" with unique hash index (#3596) 2017-11-07 10:42:41 +01:00
Simon Grätzer 64e9377c05 Replacing /_api/collection with RestHandler (#3543) 2017-11-02 14:57:17 +01:00
Jan 720e6df82e Bug fix/fixes 1910 (#3471)
* properly initialize all properties

* use faster comparison

* properly detect and handle "method not allowed"

* code-style

* remove unused variable

* narrow variable scope

* handle non-existance of AuthenticationFeature

* remove dead code

* replace some C string handling with std::strings

* moved assertion to the correct place

* honor number of array members for IN operator

* slightly adjust error messages

* slighty adjust some error messages

* try to fix issue with lingering replication contexts on shutdown

* clean up heartbeat thread a little bit

* small fixes
2017-10-23 09:17:36 +02:00
Simon Grätzer 7c31960cf2 Feature/async failover (#3451) 2017-10-18 23:59:29 +02:00
Wilfried Goesgens a551879158 give usefull errors in case of rocksdb init aborting with 'IOError' w… (#3178)
* give usefull errors in case of rocksdb init aborting with 'IOError' which happens i.e. on cifs mounts

* fix typo
2017-09-04 18:02:38 +02:00
Dan Larkin a775e37c76 Feature/unique vpack index lookup performance improvements with rocksdbkey optimizations (#3123)
* fixed issue #3075

* update the known issues document, as some issues have already been fixed

* speed up equality lookups in unique vpack indexes

* Allow for re-use of RocksDBKey objects through transaction leaser.

* Converted all RocksDBKey variants to non-static constructors.
2017-08-31 20:00:05 +02:00
Simon Grätzer 88d01b89b5 Optimizations for Caches and Graph Traversals (#3169) 2017-08-31 18:33:10 +02:00
jsteemann 86db523e02 fix compile warning in non-maintainer mode 2017-08-30 15:14:16 +02:00
Jan c29a2745d2 Bug fix/fixes 2808 (#3156)
* fixed jslint warnings

* fix undefined behavior

* move check into if, so check tools do not complain
2017-08-28 15:16:41 +02:00
Jan a212f45a86 Bug fix/issues 2308 (#3117)
* remove API docs for endpoints that were deleted in 3.0

* remove unused variable

* set all scalar values in the ctor, so check tools do not complain

* move Result objects

* prevent potential nullptr access

* remove ambiguous boolean parameter with no clear meaning

* make code check tools happy

* don't allow copying cache Findings, allow moving Results

* the debug functionality for counting RocksDB key ranges was broken

this produced sporadic test failures

* remove superfluous iostream include

* fix range checks

* don't leak if an error happens during database removal

* try to simplify Result a bit

* try to deletion checks a bit more robust
2017-08-24 14:47:22 +02:00
Jan 3350164ec6 a few simplifications and extensions (#3115) 2017-08-24 14:46:16 +02:00
Dan Larkin 84f11c223d Fixed issue with periodic syncing and WAL file retention. (#3098) 2017-08-22 15:51:12 +02:00
Jan Christoph Uhde df76bbc690 Bug fix/planning issue 514 replication api (#3021)
* add "cluster selectivity estimates" to CHANGELOG

* add some documentation to RocksDBRestReplicationHandler

* fix building with relative paths

* add some more doc

* add some tests for the replication api

* fix RocksDBRestReplicationHandler and add tests

* update documentation

* remove obsolete parameter

* fix error message

* Implementing logger-first-tick, logger-tick-ranges. Fixing dump `chunkSize` documentation
2017-08-22 10:46:06 +02:00
Dan Larkin 5708fc8fab Added test to ensure RocksDB recovers keygen data outside WAL. (#3072) 2017-08-22 10:42:46 +02:00
Simon Grätzer 4cc8686e82 RocksDB: Optimizing dropCollection / dropIndex / fillIndex (#2927)
* cleaning up several duplicate versions of Index::memory, Index::cleanup, Index::drop, integrating branch issue-383-ensure-delete

* fixing wrong column family
2017-08-05 21:09:41 +02:00
Jan 4f5d5ff54d slightly move responsibility for recovery (#2922) 2017-08-01 14:25:13 +02:00
Simon Grätzer 59dc8d0135 Update rocksdb options (#2880)
* changing some defaults

* Maybe fixing performance

* Modified changelog

* Updated documentation to reflect options
2017-07-27 16:05:31 +02:00
Frank Celler a5a25754ed Feature/reduce extraction to projection (#2792)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule

* cppcheck

* try to fix compile error on MacOS

* bug fix for MacOSX

* missing namespace (in Windows compile)
2017-07-14 08:40:29 +02:00
Jan 722870b70c fix access to cache object if cache is turned off (#2782) 2017-07-12 22:11:50 +02:00
Frank Celler 40d73d5a8b Revert "[WIP] Feature/reduce extraction to projection (#2735)"
This reverts commit 5bfcff30cd.
2017-07-12 12:52:14 +02:00
Jan 5bfcff30cd [WIP] Feature/reduce extraction to projection (#2735)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule
2017-07-12 11:22:29 +02:00
Jan bee32b6325 properly compact ranges so data actually gets deleted after drop etc. (#2771) 2017-07-11 21:47:16 +02:00
Simon Grätzer d8c57f2cc8 Feature/rocksdb hashsearch (#2713)
* Testing hash for documents

* fixing logspam error

* added base support for kHash in the edge index

* Current version of hash enabled edge index

* removed rocksdb key type byte, dynamic prefixes for hash-index

* Fixing issues with key classes

* Fixing merge conflict

* Fixing version with semi-working hash index

* fixing various issues

* Fixing hanger in EdgeIndexIterator nextExta, when _cache is disabled. Reenable cache

* Fixing geo index bounds, and fillIndexes error handling

* Adding back failure point

* Fixing RocksDBKey::vertexId

* Added version check on startup

* Fixing version check

* added one more check

* properly detect column families at startup

* Adding additional sanity checking

* Adding missing column family

* some fixes

* Turned off geoindex sync

* fixed error

* added asserts (untested)

* Adding more asserts
2017-07-05 10:18:52 +02:00
Jan ccbe1ba792 Bug fix/fix rocksdb options (#2702)
* cleanup rocksdb options
2017-07-02 23:13:42 +02:00
Dan Larkin 05c8d3ae71 Converted Get calls to use PinnableSlice where possible. (#2683) 2017-07-01 00:27:27 +02:00
Dan Larkin ee8cb749bb Added recovery tests for views and fixed a few related bugs. (#2682)
* Added recovery tests for views and fixed a few related bugs.

* Added more view recovery tests.

* Modified view recovery tests to add a waitForSync operation after.

* fixed usage of wrong view type

* fixed recovery of view change markers
2017-06-29 17:09:43 +02:00
Jan cfbd8ed93c Bug fix/rocksdb autoincrement (#2648)
* Added autoincrement keygen support to RocksDB engine with test.

* fixed key generator state keeping for mmfiles engine
2017-06-23 15:27:09 +02:00
Frank Celler ce8b1fb885 Feature/rocksdb update 5.6 (#2621) 2017-06-20 18:12:43 +02:00
Dan Larkin a0c4e3d9df Improved RocksDB engine statistics with docs and rest endpoint (#2598)
* Improved statistics reporting for RocksDB engine (including cache info).

* Added documentation about db._engineStats().

* Added rest endpoint for engine stats.
2017-06-18 22:55:25 +02:00
Michael Hackstein f21962dd03 Merged in Column Family for Views
Squashed commit of the following:

commit 860ddfc7a74866919ce5139ac3d874f09501ec97
Merge: d9ec0e3 bf20939
Author: Michael Hackstein <michael@arangodb.com>
Date:   Thu Jun 8 12:45:14 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/views-column-family

commit d9ec0e3c88122c6b9ee8acdbc6e69ca6af62e19f
Author: Michael Hackstein <michael@arangodb.com>
Date:   Thu Jun 8 12:44:43 2017 +0200

    Added a column family for views. It is no yet used anywhere. Just preparation.
2017-06-08 12:46:01 +02:00
jsteemann 427fc35f35 fix MSVC compile warnings 2017-06-08 10:48:39 +02:00
Ewout Prangsma 4d6fa5bb87 Import RocksDB encryption support (#2491)
* Import RocksDB encryption support

* Adding enterprise hooks

* Allocate scratch space in BlockCipher, used for all subsequent calls to EncryptBlock

* Adding more enterprise hooks

* Added encryption test script

* Added (commented) entry for encryption-keyfile

* Use correct env

* Updated ARANGODB value

* Log startup at debug level

* Removed TTY flag

* Fixed build

* Added prefix block support with unique initial counter & IV for CTR implementation
2017-06-08 08:51:02 +02:00