Hawkin
/
arangodb
mirror of https://gitee.com/bigwinds/arangodb
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
44,497 Commits 358 Branches 495 Tags 994 MiB
Commit Graph

182 Commits

Author SHA1 Message Date
Simon Grätzer 93aa6011c3 Feature/warmup speed (#2971) 2017-08-15 23:05:20 +02:00
Simon Grätzer 4cc8686e82 RocksDB: Optimizing dropCollection / dropIndex / fillIndex (#2927) 
* cleaning up several duplicate versions of Index::memory, Index::cleanup, Index::drop, integrating branch issue-383-ensure-delete

* fixing wrong column family
 2017-08-05 21:09:41 +02:00
Jan Christoph Uhde ed8efe3566 Feature/issue 387 cluster index estimates (#2866) 2017-08-01 09:53:58 +02:00
Jan bee32b6325 properly compact ranges so data actually gets deleted after drop etc. (#2771) 2017-07-11 21:47:16 +02:00
Simon Grätzer d8c57f2cc8 Feature/rocksdb hashsearch (#2713) 
* Testing hash for documents

* fixing logspam error

* added base support for kHash in the edge index

* Current version of hash enabled edge index

* removed rocksdb key type byte, dynamic prefixes for hash-index

* Fixing issues with key classes

* Fixing merge conflict

* Fixing version with semi-working hash index

* fixing various issues

* Fixing hanger in EdgeIndexIterator nextExta, when _cache is disabled. Reenable cache

* Fixing geo index bounds, and fillIndexes error handling

* Adding back failure point

* Fixing RocksDBKey::vertexId

* Added version check on startup

* Fixing version check

* added one more check

* properly detect column families at startup

* Adding additional sanity checking

* Adding missing column family

* some fixes

* Turned off geoindex sync

* fixed error

* added asserts (untested)

* Adding more asserts
 2017-07-05 10:18:52 +02:00
Jan Christoph Uhde 903bbbaf9a add missing try loop to nextExtra 2017-06-28 13:45:17 +02:00
Jan 6db52fe25f fix compile warnings once again (#2673) 2017-06-26 20:23:15 +02:00
Jan cb8014dbdd use correct column families for size estimates (#2624) 2017-06-22 12:32:08 +02:00
Dan Larkin a3ae2b7191 Augmented cache API and fixed a potential deadlock scenario. (#2579) 
* Augmented cache API and fixed a potential deadlock scenario.

* The RocksDBEdgeIndex now tries to read up to 10 times form the cache on lock_timeout. Instead of rewriting everything after first attempt
 2017-06-19 17:28:04 +02:00
Jan 329cb53952 enable RVO (#2566) 2017-06-16 09:42:10 +02:00
Jan c176a11930 control RocksDB block cache filling for the primary and edge indexes (#2532) 2017-06-09 08:32:46 +02:00
Simon Grätzer 5fc966e267 fixing warmup mode 2017-06-07 15:47:08 +02:00
jsteemann 2d96b9ab4c fix issues found by coverity scan tool 2017-06-07 13:22:18 +02:00
Michael Hackstein 7aa2a9d4c7 Merge branch 'devel' of github.com:arangodb/arangodb into devel 2017-06-07 09:27:49 +02:00
Michael Hackstein 0a98ff9b16 EdgeIndexCache now stores unindexed VPack Arrays. Saves some memory without loosing performance. 2017-06-07 09:27:44 +02:00
Dan Larkin 8c7ba2aaa3 Changed several index/collection APIs to provide better context messages. 
Addresses Issue #342.
 2017-06-06 12:27:08 -04:00
Michael Hackstein 6ac0c0e673 Removed dead code 2017-06-06 13:13:50 +02:00
Simon Grätzer 634254b80f adding load and unload 2017-06-02 14:44:38 +02:00
Simon Grätzer 10f6242932 removing dead code 2017-06-02 12:22:39 +02:00
Simon Grätzer 87de810566 Edge index refactoring 2017-06-02 11:20:15 +02:00
Simon Grätzer 3e8c16d1a7 Adding edge key into edge index 2017-06-01 16:07:39 +02:00
Michael Hackstein 52406567cd Replaced EdgeCursors callback to return cid,rev instead of _id. Does less costly copying and comparision internally 2017-05-31 17:49:46 +02:00
Simon Grätzer 5509776503 Fixing various issues 2017-05-30 13:44:26 +02:00
Simon Grätzer ab292b2717 Fixing edge bounds 2017-05-30 11:30:48 +02:00
Simon Grätzer 3ad939ea41 changed storage format 2017-05-30 10:12:39 +02:00
Michael Hackstein b9d2faa7cc Added warmup function for indexes. 
Squashed commit of the following:

commit bc0472d212b2efef0d7b1b8a36f2b43e8432ba86
Merge: eb47631 5f87674
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:39:31 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit eb476310b0ca6165be10b37b960933886c2757f5
Merge: d725f21 32149d2
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:00:41 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit d725f21f7d61f7c79ba31fc0338881e35f4d8f48
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 15:53:23 2017 +0200

    Activated collection warmup in Cluster Mode. Added a test for collection warmup (SingleServerOnly)

commit 132bf4f9565b0dcf4ec9f84e93897b482a67ce7f
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 12:46:20 2017 +0200

    Implemented client-side warmup. It can now be triggered via http and arangosh

commit 78ea449dff86118814a2f87bdb59dc16544d92b6
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:25:40 2017 +0200

    Fixed assertion.

commit dae80f6277dde1a52eadda506858cc36e235bd55
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:08:46 2017 +0200

    Improved the CuckooIndexEstimator computeEstimate function. It is now much better on collections with many different values

commit 7abf57876511ba369d7a577e1995d4575e98c7c8
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:06:43 2017 +0200

    Edge index warmup will now first check if a document is stored and if not will insert it. Furthermore it resizes the cache with an estimate of how many documents will be most likely inserted

commit 890d8ad4cdfd155249f060fedd5c798b9531d556
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:04:47 2017 +0200

    Adjusted thresholds in transactional cache. Wastly increased the amount of tries to get the lock. Lowered the fill-grade boundaries

commit 60972ab7151a3acb78e1aae4149de11d0da7aceb
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:45:38 2017 +0200

    Added new Logtpopic CACHE that should be used to log important information regarding caches, like debug ouput on evictions / resizing, or memory-pressure resizing going on

commit 2dfacb9aef6e3fde169032514baca386786d059c
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:44:21 2017 +0200

    Fixed an assertion in index iterator. It expected _from/_to to be cached but we modified to cache entire document

commit f05beccef65853c30eda1859a39c28ea2295bc71
Merge: 13c6abf 5c46430
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 07:56:58 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit 13c6abfbc2ddb451a689011110411df68247435e
Author: Michael Hackstein <michael@arangodb.com>
Date:   Wed May 24 09:52:01 2017 +0200

    Added a getExtra feature for the edgeIndex. It now returns the complete edge as extra (temporary). Modified the EdgeCache to cache token and edge document (temporary). Added a warmup Function to collections that is used to warmup the index caches on demand.
 2017-05-29 09:40:58 +02:00
Simon Grätzer d5f85d5bbb Adding more colum families 2017-05-26 10:43:17 +02:00
Simon Grätzer 67b9f75fec Merge branch 'devel' of https://github.com/arangodb/arangodb into column-family 
# Conflicts:
#	arangod/RocksDBEngine/RocksDBEdgeIndex.cpp
 2017-05-22 11:16:26 +02:00
Michael Hackstein 71241e6048 Fixed usage of std::min on apple clang. It complains that size_t != uint64_t. Now both sides are uint64_t 2017-05-22 08:03:18 +02:00
Michael Hackstein c9206ab218 The RocksDBEdgeIndexCache now can increase the values size it wants to cache, there is no hardcoded upperbound anymore. Also moved a lot of state out of the iterator. 2017-05-21 23:30:33 +02:00
Simon Grätzer a99611e7c0 Various changes 2017-05-19 16:43:00 +02:00
Simon Grätzer e73f446c5b Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-05-18 22:08:39 +02:00
Simon Grätzer 2bd2a15cb3 Column family support 2017-05-18 22:08:34 +02:00
Michael Hackstein e4050f9804 Temporary Commit, modifying some RocksDB Options and the EdgeIndex Cache size. This commit is used for NoSQL Performance Banchmark validation. 2017-05-18 15:45:49 +02:00
jsteemann 226920d7fa optimizations 2017-05-17 23:41:16 +02:00
Simon Grätzer 4b6035ff64 Fixed merge 2017-05-16 17:07:15 +02:00
Simon Grätzer e25f88ef4d Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 
# Conflicts:
#	arangod/RocksDBEngine/RocksDBCuckooIndexEstimator.h
#	arangod/RocksDBEngine/RocksDBEdgeIndex.cpp
#	arangod/RocksDBEngine/RocksDBIndex.cpp
 2017-05-16 16:59:50 +02:00
Simon Grätzer 5265d2a063 intermediate commits 2017-05-16 16:21:39 +02:00
Jan Christoph Uhde 9aab78a6e9 remove unused code 2017-05-16 13:06:33 +02:00
Simon Grätzer df3eed2289 Various changes 2017-05-16 10:58:15 +02:00
Michael Hackstein d7d650e27d Implemented index estimates for RocksDB. Now the following indexes have 
a proper estimate:
* HashIndex
* SkiplistIndex / Persistent
* EdgeIndex
 2017-05-15 17:17:58 +02:00
jsteemann c423cf05a7 storage revisions in edge cache 2017-05-15 15:05:46 +02:00
jsteemann 52ba50e281 micro optimizations for edge index 2017-05-15 12:06:27 +02:00
jsteemann c6d87e0060 ctor, then dtor, then others 2017-05-15 09:31:40 +02:00
jsteemann 51e8ee50c6 fix indentation 2017-05-15 09:31:08 +02:00
jsteemann 7c959cf81f fix indentation 2017-05-15 09:30:38 +02:00
Jan Christoph Uhde 4a85f93814 avoid search in cache when iterator was resumed 2017-05-12 21:38:29 +02:00
Jan Christoph Uhde b7ca835671 release findings sooner if they contain many entries 2017-05-12 16:09:12 +02:00
Jan Christoph Uhde 227d6648a2 fix not advancing iterator in rocksdb edge index 2017-05-12 14:47:42 +02:00
Jan Christoph Uhde 07bb07de39 Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/edge-index-cache 
* 'devel' of https://github.com/arangodb/arangodb:
  add debug output
  more unique ptrs
  fixed comment
  moved description into correct section
  remove unused options
  enable rocksdb logger
 2017-05-12 13:56:57 +02:00
Jan Christoph Uhde db38ef3522 more clean up for edge index 2017-05-12 13:51:05 +02:00
Jan Christoph Uhde a1454ce87c clean up RocksDBEdgeIndex code 2017-05-12 13:27:25 +02:00
Jan Christoph Uhde dd00e03010 fix2 2017-05-12 13:18:13 +02:00
Jan Christoph Uhde 3afdb2a666 fix 2017-05-12 12:35:35 +02:00
Jan Christoph Uhde 7902d0e44e Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/edge-index-cache 
* 'devel' of https://github.com/arangodb/arangodb: (25 commits)
  Add footer to docs, unify slogan
  Docs: make graph traversal images hard copies (fixes Cygwin builds)
  Added documentation for RocksDB and hash cache options to manual.
  Changed default cache size to something reasonable.
  cppcheck
  added comments
  fix shutdown order
  Added document cache.
  fix some replication issues
  Invalidate current coordinators on every 2nd heartbeatrun
  Show timeout when failing
  Addding locking, fixing conversion warning
  Fixing locking error
  resync changelog to 3.1
  Fix jslint
  CHANGELOG: SIGHUP forwarding
  forward SIG_HUP from supervisor
  Allow removing dbnodes
  Rocksb: index locking issue
  Fix serverAlias map...this was ridiculous
  ...
 2017-05-12 10:37:24 +02:00
jsteemann abeae0a30d cppcheck 2017-05-11 21:21:58 +02:00
Jan Christoph Uhde 9e9faf7bbd hopefully fix the cache 2017-05-11 18:07:30 +02:00
Jan Christoph Uhde dce3250862 add more logging to edge index 2017-05-11 16:42:49 +02:00
Simon Grätzer 310166d3cf Fixing locking error 2017-05-11 15:55:06 +02:00
Jan Christoph Uhde ba3c797704 fix bug in edgeIndex 2017-05-11 15:20:39 +02:00
Jan Christoph Uhde c6fa6bb1ae add empty array slices in cache if no primary key for fromTo can be found 2017-05-11 13:13:15 +02:00
Jan Christoph Uhde 36d8b31335 decrease log level for edge index cache 2017-05-11 12:56:12 +02:00
Jan Christoph Uhde ee6e28a33d first tests pass for rocksdb index with cache 2017-05-11 12:11:00 +02:00
Jan Christoph Uhde e36d795dad crashing during lookup 2017-05-11 11:18:44 +02:00
Jan Christoph Uhde db7fdd2b7f add postprocessRemove for RocksDBEdgeIndex 2017-05-11 09:50:23 +02:00
Jan Christoph Uhde 847dd3ad5d Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/edge-index-cache 
* 'devel' of https://github.com/arangodb/arangodb:
  geo index fixes
  adding transactions to the geo index
 2017-05-10 18:43:20 +02:00
Jan Christoph Uhde 3e25aec2f3 fix enabling of caches 2017-05-10 15:58:28 +02:00
Jan Christoph Uhde e5db579b25 Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/edge-index-cache 
* 'devel' of https://github.com/arangodb/arangodb: (54 commits)
  attempt to fix races in replication
  grunt build
  use exclusive transaction in syncer
  change feature order around
  dashboard
  added api cluster api routes
  add selection of storage engine
  use some parallelism
  micro optimizations
  Fix truncate for geo
  added assertions
  remove unused include
  try to fix initial sync problem
  remove unnecessary includes
  Finally fix the error where suddenly an array of dbservers are being called
  Update queryView.js
  initialize bool variable
  remove debug output
  removed unused private field
  Wrong index number
  ...
 2017-05-10 15:14:08 +02:00
Simon Grätzer 5fa85761a2 adding transactions to the geo index 2017-05-10 14:54:39 +02:00
Jan Christoph Uhde 4cfbe85e4a honor batch size && length limit for cache values 2017-05-10 14:01:10 +02:00
Jan Christoph Uhde 7ea4eaf774 fix index figures 2017-05-10 14:00:40 +02:00
Jan Christoph Uhde e6d41cac9a clean up edge-index code - handling of invalidated caches still missing 2017-05-10 09:03:21 +02:00
Dan Larkin ca91c0f7c9 Added some optimizations enabled by prefix extraction. 2017-05-09 18:36:12 -04:00
Jan Christoph Uhde 738989f9a3 add blacklisting for keys in rocksdb-edge-index on insert/remove 2017-05-09 18:33:05 +02:00
Jan Christoph Uhde e29db44af6 begin work on cache for edge index 2017-05-09 17:22:27 +02:00
Simon Grätzer d9bff18814 Fulltext index 2017-05-05 13:58:11 +02:00
Simon Grätzer 804676ae65 Moving estimatedSize to memory() 2017-05-04 18:57:22 +02:00
Simon Grätzer 94e4437c3a Added some non transactional truncate logic 2017-05-04 18:07:47 +02:00
Simon Grätzer 1c82d4448c Adding compaction and estimateSize method for rocksdb 2017-05-04 13:08:02 +02:00
Simon Grätzer 0faa1adac1 fixed unique contraints 2017-05-03 17:45:21 +02:00
Simon Grätzer a2e2c00d62 fillIndex non transactional + tests 2017-05-03 16:07:29 +02:00
Jan Christoph Uhde b7f42f638e add empty impl for recalculating counts in rocksdb engine 2017-05-02 17:54:36 +02:00
jsteemann fef62ecda6 added more assertions 2017-04-28 14:32:52 +02:00
jsteemann 7344a03623 fix crashes 2017-04-25 15:47:11 +02:00
jsteemann 4300c77d3e fixes for non-array IN lookups, added tests 2017-04-25 01:47:07 +02:00
jsteemann d3f32799c6 fixed assertion failure 2017-04-18 14:54:29 +02:00
Dan Larkin 841ef8b1a0 Changed index VPack building to include forPersistence flag. 2017-04-17 17:04:50 -04:00
Simon Grätzer e0e00b0b6b Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api 
# Conflicts:
#	arangod/RocksDBEngine/RocksDBEdgeIndex.cpp
 2017-04-13 17:48:52 +02:00
Simon Grätzer 866d8b7451 Fixing tests 2017-04-13 17:45:16 +02:00
Michael Hackstein 1087065ea5 Merge branch 'devel' of github.com:arangodb/arangodb into engine-api 2017-04-13 17:34:25 +02:00
Jan Christoph Uhde e690e7fc80 Merge remote-tracking branch 'origin/feature/engine-api/edgeindex-report-as-single-index' into engine-api 
* origin/feature/engine-api/edgeindex-report-as-single-index:
  change mmfiles indexes to work with slightly changed API
  fix rocksdb engine index reporting but break report for mmfiles engine
  update error messages
 2017-04-13 08:46:54 +02:00
Dan Larkin d435d274ea Merge branch 'engine-api' of github.com:arangodb/arangodb into engine-api 2017-04-12 09:35:19 -04:00
Jan Christoph Uhde 2700a6d79d fix rocksdb engine index reporting but break report for mmfiles engine 2017-04-12 14:50:55 +02:00
Simon Grätzer fac53e5a06 Fixing edge iterator 2017-04-11 10:01:56 +02:00
Dan Larkin b030cdb347 Fixed an edge index iterator bug. 2017-04-10 20:12:15 -04:00
jsteemann 0e98a32455 fix objectId handling 2017-04-06 11:19:27 +02:00
jsteemann e5d733867e make more tests pass 2017-04-05 14:57:18 +02:00
Michael Hackstein 069394634c RocksDB Indexes do now destroy their cache as soon as they are dropped, instead of during deletion 2017-04-03 17:10:10 +02:00
Simon Grätzer 7afcac9ed7 Fixing some issues 2017-04-03 14:24:58 +02:00
Jan Christoph Uhde e49024f16e add drop collection to rocksdb engine 2017-04-03 10:41:02 +02:00