1
0
Fork 0
Commit Graph

333 Commits

Author SHA1 Message Date
Ewout Prangsma 4d6fa5bb87 Import RocksDB encryption support (#2491)
* Import RocksDB encryption support

* Adding enterprise hooks

* Allocate scratch space in BlockCipher, used for all subsequent calls to EncryptBlock

* Adding more enterprise hooks

* Added encryption test script

* Added (commented) entry for encryption-keyfile

* Use correct env

* Updated ARANGODB value

* Log startup at debug level

* Removed TTY flag

* Fixed build

* Added prefix block support with unique initial counter & IV for CTR implementation
2017-06-08 08:51:02 +02:00
Dan Larkin c48eb92c1e Added some more pass-through options for RocksDB and optimized a couple defaults. 2017-06-06 10:51:01 -04:00
jsteemann 6b05b39825 expose currently used block cache size 2017-06-06 11:19:11 +02:00
Simon Grätzer 66f10870dd Renaming files 2017-05-31 18:12:47 +02:00
Frank Celler f45325fe3f comment out new options. Needs further investigation into performance impact 2017-05-31 09:15:45 +02:00
jsteemann 3886aece76 updated manual 2017-05-30 13:09:06 +02:00
jsteemann 53b150b206 added tests for intermediate commits 2017-05-30 12:03:58 +02:00
jsteemann c7e1877fd6 workaround for compile issue 2017-05-29 13:02:22 +02:00
Frank Celler 4b61f5b8bb added --rocksdb.num-uncompressed-levels 2017-05-29 11:56:48 +02:00
jsteemann 76f2ce5b2c nicer error message on startup when RocksDB data version does not fit 2017-05-29 11:36:05 +02:00
Frank Celler b7961ef3dd Reenabled snappy (#2504)
* fixed snappy config

* added numactl and GLIBCXX_FORCE_NEW

* enabled snappy

* fix snappy include path

* windows paths
2017-05-26 20:53:23 +02:00
Simon Grätzer 21dd147ad6 Fixing issues 2017-05-26 14:03:12 +02:00
Simon Grätzer d5f85d5bbb Adding more colum families 2017-05-26 10:43:17 +02:00
Simon Grätzer 555a7fcecd Merge branch 'devel' of https://github.com/arangodb/arangodb into column-family 2017-05-24 19:04:20 +02:00
jsteemann d6bb581b18 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-05-24 11:33:53 +02:00
jsteemann 81786c0aa6 print compression support 2017-05-24 11:33:46 +02:00
jsteemann 631453a1f3 more options 2017-05-24 11:15:36 +02:00
jsteemann b23285df81 added more options, but not effective yet 2017-05-24 11:02:24 +02:00
Simon Grätzer 9679aecf53 Adding bounds back 2017-05-22 17:39:19 +02:00
Simon Grätzer ad3da410b2 Pregel improvements 2017-05-22 11:10:28 +02:00
Simon Grätzer 45d7c5c715 Various debugging changes 2017-05-20 17:44:28 +02:00
Simon Grätzer e73f446c5b Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-05-18 22:08:39 +02:00
Simon Grätzer 2bd2a15cb3 Column family support 2017-05-18 22:08:34 +02:00
Michael Hackstein e4050f9804 Temporary Commit, modifying some RocksDB Options and the EdgeIndex Cache size. This commit is used for NoSQL Performance Banchmark validation. 2017-05-18 15:45:49 +02:00
jsteemann a8320e99dc don't force compaction after every collection drop operation 2017-05-16 12:22:30 +02:00
Michael Hackstein d7d650e27d Implemented index estimates for RocksDB. Now the following indexes have
a proper estimate:
* HashIndex
* SkiplistIndex / Persistent
* EdgeIndex
2017-05-15 17:17:58 +02:00
jsteemann 257e3a05e3 update rocksdb logging to error level 2017-05-12 17:27:22 +02:00
jsteemann b1aae2efdd remove unused options 2017-05-12 12:12:09 +02:00
jsteemann b848030818 enable rocksdb logger 2017-05-12 12:03:54 +02:00
Simon Grätzer e11ef2bf54 Addding locking, fixing conversion warning 2017-05-11 17:38:50 +02:00
Simon Grätzer 9950fa7245 removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
Simon Grätzer 5fa85761a2 adding transactions to the geo index 2017-05-10 14:54:39 +02:00
Dan Larkin 9662b26002 Merge branch 'devel' of github.com:arangodb/arangodb into devel 2017-05-09 18:44:15 -04:00
Dan Larkin ca91c0f7c9 Added some optimizations enabled by prefix extraction. 2017-05-09 18:36:12 -04:00
Simon Grätzer 3b7ef439c8 Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/rocksdb-geoindex 2017-05-09 17:42:27 +02:00
jsteemann 488e4be485 preparation for RocksDB logger 2017-05-09 12:42:25 +02:00
jsteemann 587418fd1b constified 2017-05-09 02:13:51 +02:00
Dan Larkin e601e1efd9 Added prefix extractor. 2017-05-08 16:47:36 -04:00
jsteemann 12b46d2086 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-05-08 18:27:07 +02:00
jsteemann 7577fed6fb added logger 2017-05-08 18:24:40 +02:00
Dan Larkin f6a20e1f41 Merge branch 'devel' of github.com:arangodb/arangodb into devel 2017-05-08 12:09:10 -04:00
Dan Larkin 284bb6fb51 Fixed some issues with RocksDB passthrough options. 2017-05-08 12:08:41 -04:00
Jan Christoph Uhde a4a04350a3 fix replication 2017-05-08 17:43:36 +02:00
Jan Christoph Uhde acee270075 fix small bugs revealed by tests 2017-05-08 15:43:09 +02:00
Jan Christoph Uhde 8ee0406c64 move last logger creation to engines 2017-05-08 15:18:19 +02:00
Simon Grätzer a610d81e3d fixing compiler warnings 2017-05-08 14:14:23 +02:00
Jan Christoph Uhde cf333f3b0a move creation of tickranges and first ticks into storage engines 2017-05-08 14:11:41 +02:00
Dan Larkin 9924fa1480 Added more RocksDB passthrough options. 2017-05-07 11:08:25 -04:00
Simon Grätzer dd0b0c6b28 RocksDB Fülltext index 2017-05-06 01:14:49 +02:00
jsteemann 8c2be23d73 do not call SyncWAL() on Windows, as it does not work 2017-05-05 15:37:38 +02:00
jsteemann 6a30df3363 add some safety padding for WAL file deletion 2017-05-05 15:31:57 +02:00
jsteemann d352794360 optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
Dan Larkin 037f3f38b9 Removed accidental log spam. 2017-05-04 15:30:09 -04:00
Dan Larkin d77efe38dc Added explicit WAL file lifecycle management to reduce space overhead from replication. 2017-05-04 15:16:24 -04:00
Simon Grätzer 40b8eb2824 Trigger compaction on drop and truncate 2017-05-04 13:45:30 +02:00
Simon Grätzer 1c82d4448c Adding compaction and estimateSize method for rocksdb 2017-05-04 13:08:02 +02:00
jsteemann 1e72f8ccdc allow configuration of use_fsync from the outside 2017-05-04 01:15:09 +02:00
jsteemann f1763e491d fix db._engineStats() 2017-05-04 00:33:36 +02:00
jsteemann 2e3d704d10 fix MSVC compile warnings 2017-05-03 15:47:45 +02:00
jsteemann 9a48bd3411 fix unintended change of option value 2017-05-03 15:18:44 +02:00
jsteemann c716fcacd3 write settings on shutdown 2017-05-03 15:18:17 +02:00
Simon Grätzer e62d82f1d0 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel
# Conflicts:
#	arangod/RocksDBEngine/RocksDBEngine.cpp
#	arangod/RocksDBEngine/RocksDBV8Functions.cpp
2017-05-03 12:27:58 +02:00
Simon Grätzer b5df720870 Added rocksdb properties 2017-05-03 12:23:19 +02:00
Jan Christoph Uhde b7f42f638e add empty impl for recalculating counts in rocksdb engine 2017-05-02 17:54:36 +02:00
Jan Christoph Uhde da02fd36c6 move engine specific syncer code into engines 2017-05-02 16:36:27 +02:00
Dan Larkin e6611ddc4f Fixed ongoing replication bugs and split the test. 2017-04-28 16:01:29 -04:00
Simon Grätzer 84ed7ec100 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel
# Conflicts:
#	arangod/RocksDBEngine/RocksDBRestReplicationHandler.cpp
2017-04-28 14:40:17 +02:00
Simon Grätzer 0a00a7eb67 Fixed broken JSON output 2017-04-28 14:28:34 +02:00
Michael Hackstein 360d2b3b99 Fixed restoreIndox for RocksDB. Also removed Primary and EdgeIndexes from Inventory, which is identical to MMFiles 2017-04-28 11:09:53 +02:00
Dan Larkin e2853e4b52 Enabled saving applier config and fixed some replication tests/behavior. 2017-04-27 22:39:48 -04:00
Simon Grätzer 9e498851a9 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-04-27 18:23:14 +02:00
Michael Hackstein 70f5413fe8 RocksDB inventory now includes indexees as well 2017-04-27 18:11:06 +02:00
Simon Grätzer 88c62c05a9 Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-04-27 17:38:06 +02:00
Simon Grätzer ddf7ae60cf WAL tailing 2017-04-27 17:38:02 +02:00
Michael Hackstein e0d9fd3cc3 Redesigned Collection.drop() in RocksDB. Does not write LogData and has a different workflow to create a consistent user-view 2017-04-27 12:58:51 +02:00
Dan Larkin f8efe6e627 Added create index and change collection marker creation. 2017-04-26 15:23:25 -04:00
Michael Hackstein e423a5fde1 Added WAL MARKERS for CreateCollection, RenameCollection and DropIndex, also adjested WalMarkerType constructors. 2017-04-26 18:32:28 +02:00
Simon Grätzer 6472c4928d Various warnings 2017-04-26 11:26:08 +02:00
jsteemann f36a889709 try to fix shutdown issues 2017-04-26 09:58:19 +02:00
Jan Christoph Uhde fca8f01df6 create logger state just in one place 2017-04-25 14:58:32 +02:00
jsteemann 0434d88ab4 rocksdb options 2017-04-25 02:22:00 +02:00
Simon Grätzer 88dc999001 Counter manager fixes 2017-04-24 12:43:44 +02:00
jsteemann 361d513087 re-enable user settings for RocksDB 2017-04-21 15:28:29 +02:00
jsteemann 0444a5a068 disable usage of (bad) default values 2017-04-21 14:02:50 +02:00
jsteemann 862ec7bae3 remove log garbage 2017-04-21 12:57:20 +02:00
Simon Grätzer 98df9c21a2 Fixing warnings 2017-04-20 19:07:30 +02:00
Simon Grätzer b304d35ac2 Added rocksdb background thread 2017-04-20 18:50:52 +02:00
jsteemann 569450321d cppcheck 2017-04-20 16:47:10 +02:00
Simon Grätzer 6dd02bab0d Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 2017-04-20 12:47:28 +02:00
Simon Grätzer 3319a5d607 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
# Conflicts:
#	arangod/RocksDBEngine/RocksDBRestReplicationHandler.cpp
2017-04-20 12:47:22 +02:00
jsteemann 213387e64f fix shutdown issue 2017-04-20 12:36:54 +02:00
jsteemann 87477d20a9 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 2017-04-20 12:26:08 +02:00
jsteemann 3727e7fb7a first stab at exclusive locks 2017-04-20 12:25:53 +02:00
Simon Grätzer 388121158e Started inventory rest handler 2017-04-20 12:16:40 +02:00
Jan Christoph Uhde 325c64773d make use of ReplicationManager and ReplicationContext 2017-04-20 12:05:40 +02:00
Dan Larkin dfd1e78f35 Added objectId-to-database+collection mapping. 2017-04-19 13:56:17 -04:00
Jan Christoph Uhde f529e58b71 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
* 'engine-api' of https://github.com/arangodb/arangodb:
  cppcheck
  jslint
  renamed engine-specific test files
  fix query cache invalidation
  fix compile warning
  fix README_maintainers, provide debug output for invalid invocations
  slightly improve error message
  ui event bugfix
2017-04-19 14:08:17 +02:00
jsteemann 8e779aab24 cppcheck 2017-04-19 13:20:46 +02:00
Jan Christoph Uhde 507b870ff1 add RocksDBOptionFeature to unify configuration of RocksDB
instances for storage-engine and persistent-index that are used mutual
exclusive.
2017-04-19 12:20:41 +02:00
Dan Larkin 7ad0c75745 Fixed bug in index info persistence/recovery. 2017-04-18 14:55:58 -04:00
Jan Christoph Uhde c12bd6f2ed remove code that tracks size of index operations
in RocksDBTransactions
2017-04-18 16:58:44 +02:00
Jan Christoph Uhde 4336d31e7b createIndex of RocksDBEngine must not throw! 2017-04-18 13:32:57 +02:00
Jan Christoph Uhde 9e366d2193 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
* 'engine-api' of https://github.com/arangodb/arangodb:
  slightly adjust error messages
  Fixing jslint errors
  Add changelog from 3.1
  Properly generate an error message during import
  Fixed scannedIndex values in traverser tests. It did not report the correct number of lookups required after modification for new Storage-Engine
  fixed issue #2429
  issue #2427: change while into an if
  don't fail when non-array, but simply ignore it
2017-04-18 13:21:54 +02:00
Jan Christoph Uhde a02e816598 add reset function to RocksDBTransactionState 2017-04-18 13:14:15 +02:00
jsteemann 031e9adb3c slightly adjust error messages 2017-04-18 12:43:21 +02:00
Dan Larkin 2e0fdbf666 Fixed some bugs. 2017-04-13 20:06:25 -04:00
Simon Grätzer 3b3ee5f996 Counter fixes 2017-04-12 17:10:35 +02:00
Jan Christoph Uhde d67aa33fe3 limit transaction size and work on intermediate commits 2017-04-12 09:01:13 +02:00
Simon Grätzer 901ffd5e2f Fixed rocksdb options 2017-04-11 09:31:39 +02:00
Jan Christoph Uhde 8df9fe1d21 fix error in insert 2017-04-10 21:27:52 +02:00
Jan Christoph Uhde c82fadc863 prepare limitation of transaction size 2017-04-10 14:00:55 +02:00
Jan Christoph Uhde b4b2ac0dbd work on counting bug 2017-04-10 09:23:35 +02:00
jsteemann 940209001c Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 2017-04-07 12:06:04 +02:00
jsteemann 2b0673c3fe removed type "Index" 2017-04-07 12:05:56 +02:00
Simon Grätzer 42a79ed1ac Fixing counter WAL parsing 2017-04-07 11:26:52 +02:00
Simon Grätzer 4ec65589f5 Added counter remove 2017-04-06 19:00:01 +02:00
Simon Grätzer 08167c2614 Conversion warnings 2017-04-06 10:56:50 +02:00
Simon Grätzer 19f220726e Fixed dump setup 2017-04-05 17:43:38 +02:00
Simon Grätzer 060ddcea59 Fixed database drop 2017-04-05 15:17:54 +02:00
Simon Grätzer 342613bd59 Fixing export 2017-04-05 14:03:58 +02:00
Simon Grätzer 5822e77843 invokeOnAllElements 2017-04-05 13:38:43 +02:00
jsteemann 9f330881a2 constify method 2017-04-05 09:00:06 +02:00
jsteemann b338abba5d Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 2017-04-05 08:31:20 +02:00
Dan Larkin 872dd95ca8 Merge branch 'engine-api' of github.com:arangodb/arangodb into engine-api 2017-04-04 17:35:30 -04:00
Dan Larkin e303e2a51f Added RocksDB export functionality and some bug fixes. 2017-04-04 17:33:14 -04:00
Simon Grätzer 01ea3556b8 Added REST Wal handler 2017-04-04 18:27:19 +02:00
jsteemann 78302d1766 make versionFilename() a property of the engine 2017-04-04 17:23:02 +02:00
jsteemann fd258acc1d fix index removal 2017-04-04 16:45:19 +02:00
Simon Grätzer 96dc714139 Fixed some tests in shell_server 2017-04-04 15:10:17 +02:00
Simon Grätzer 244b034aeb Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 2017-04-04 13:38:56 +02:00
Jan Christoph Uhde dfeadeabe1 begin to split http tests for mmfiles and rocksdb 2017-04-04 13:22:05 +02:00
jsteemann 949985dff1 report a path, make it restartable 2017-04-04 12:03:29 +02:00
Simon Grätzer 580bb19bd7 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 2017-04-04 11:38:57 +02:00
Simon Grätzer 3066a67005 Collection rename 2017-04-04 11:38:52 +02:00
Jan Christoph Uhde b3526d2626 add error handling 2017-04-04 11:35:55 +02:00
Jan Christoph Uhde cea5fa8e4f delete views when dropping database 2017-04-04 11:22:22 +02:00
Jan Christoph Uhde 11a048c420 replace prepare drop with empty stub 2017-04-04 10:29:04 +02:00
Dan Larkin 9ec54dc23b Some bug fixes. 2017-04-03 21:01:16 -04:00
Jan Christoph Uhde 5773a028c8 add more code for dropping databases 2017-04-03 21:29:25 +02:00
Jan Christoph Uhde b48975fc37 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
* 'engine-api' of https://github.com/arangodb/arangodb:
  Fixed allocation bug for many-cache situation.
  Fixed comparator
  added internal.wal.flush()
  added v8 functions
  RocksDB Indexes do now destroy their cache as soon as they are dropped, instead of during deletion
2017-04-03 17:51:01 +02:00
Jan Christoph Uhde 256b2b2d6a construct rocksdbkey from slice 2017-04-03 17:50:36 +02:00
jsteemann b9bf7d298b Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 2017-04-03 17:14:19 +02:00
jsteemann 8e3d83aa6e added v8 functions 2017-04-03 17:14:09 +02:00
Jan Christoph Uhde 1806624e41 fix compile error 2017-04-03 17:09:11 +02:00
Jan Christoph Uhde ae75917640 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
* 'engine-api' of https://github.com/arangodb/arangodb:
  added assertion to Comparator
  fix RemoveLargeRange
  fix ub
  fix assertion failure at shutdown
  Remove provisional code
  Fixed counter
  remove usleeps
  Fixed deadlock
  Fixed broken merge
  Fixing some issues
  added p/ to gitignore
  Integrated counters with collections
  properly include the javascript enterprise folders into the test tarballs
  fix parsing for URLs starting with ldap://
  fix filter for replication testcases
  Added counter manager
  updated licenses, added generator
  removed jshint completely
2017-04-03 16:54:09 +02:00
Jan Christoph Uhde 234180c08b wip - drop database 2017-04-03 16:52:21 +02:00
Simon Grätzer 664f45d029 Fixed counter 2017-04-03 15:16:14 +02:00
Simon Grätzer bc36c56716 Fixed broken merge 2017-04-03 14:52:12 +02:00
Jan Christoph Uhde ec1fbb29fd now drop elements for indexes as well 2017-04-03 13:32:17 +02:00
Jan Christoph Uhde e49024f16e add drop collection to rocksdb engine 2017-04-03 10:41:02 +02:00
Dan Larkin 23038fdc3b Fixed uniqueness checking in primary index. 2017-03-31 18:03:50 -04:00
Dan Larkin f30f4bdb47 Implemented views methods. 2017-03-31 17:09:27 -04:00
Jan Christoph Uhde 46cfae8a7e Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
* 'engine-api' of https://github.com/arangodb/arangodb:
  comment out log spam
  added comments
  fix cross book links
2017-03-31 15:45:43 +02:00
Jan Christoph Uhde 772a7a32da add buffer() to RocksDBValue for data retrieval 2017-03-31 15:41:27 +02:00
jsteemann 6f3e37ac36 comment out log spam 2017-03-31 15:37:33 +02:00
jsteemann ff0e1b1a54 added comments 2017-03-31 15:25:57 +02:00
Simon Grätzer 0fd98ed0bb Fixed HashIndex matchesDefinition method 2017-03-31 12:22:58 +02:00
Simon Grätzer 807d38784c Added some small fixes 2017-03-31 11:29:27 +02:00
jsteemann c1b6c815bc fix dropping of collections 2017-03-30 11:20:23 +02:00
jsteemann e73417aef1 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 2017-03-29 17:50:49 +02:00
Simon Grätzer cd9e9a13cf Path based enumerator 2017-03-29 17:33:02 +02:00
jsteemann d75a72eff2 honor feature optionality 2017-03-29 12:53:05 +02:00
Jan Christoph Uhde 796c32a341 improve error handling in some places 2017-03-29 11:04:04 +02:00
Dan Larkin 9b259d91fa Added comparator to engine/index. 2017-03-28 11:58:04 -04:00
Dan Larkin 3ec6934eb5 Changed RocksDBValue API slightly. 2017-03-28 11:42:52 -04:00
Jan Christoph Uhde 3c31ec2314 add function for looking up revisions to collection 2017-03-28 14:49:19 +02:00
Dan Larkin cbafcadccd Finished conversion of RocksDBEntry to RocksDBKey and RocksDBValue. 2017-03-27 16:51:30 -04:00
jsteemann 6ca2b237d1 add method to augment index creation data 2017-03-27 11:01:09 +02:00
jsteemann 2c54a6ac1e added some basic transaction handling 2017-03-24 17:23:32 +01:00
jsteemann 2aa2fd65a4 more startup 2017-03-24 14:20:51 +01:00
jsteemann 02c0dcc689 properly report error locations 2017-03-23 20:18:33 +01:00
jsteemann f17ff29518 added stubs for indexes 2017-03-23 15:24:00 +01:00
jsteemann 45da5ffe9c more diagnostic output for "not implemented" 2017-03-23 14:10:00 +01:00
jsteemann 55c38befbf enum databases, views and collections at startup 2017-03-23 11:52:22 +01:00
jsteemann 806c33eb1c cleanup 2017-03-23 09:01:21 +01:00
Jan Christoph Uhde 59eafed99d Add code that checks during startup if _system database exists 2017-03-22 18:28:35 +01:00
Jan Christoph Uhde 7c7268b269 Try to load existing databases 2017-03-22 17:58:38 +01:00
Jan Christoph Uhde 0deac5fe62 Add functon to create slices from RocksDBEntryTypes 2017-03-22 14:45:54 +01:00
Jan Christoph Uhde 66423f97dc delete on shutdown 2017-03-22 13:07:27 +01:00
Jan Christoph Uhde e7202945dc Merge branch 'devel' of https://github.com/arangodb/arangodb into engine-api 2017-03-22 12:26:10 +01:00
Jan Christoph Uhde d34f75548f enable engine selection 2017-03-22 12:14:39 +01:00
Jan Christoph Uhde 58c1f3f70f Work on RocksDBEngine and enable engine selection 2017-03-21 16:02:40 +01:00
Jan Christoph Uhde 2795eedabd add empty implementation of RocksDBEngine 2017-03-20 14:24:42 +01:00