1
0
Fork 0
Commit Graph

202 Commits

Author SHA1 Message Date
Jan fe0fca9029
Bug fix/restore unlock (#4387) 2018-01-25 15:56:27 +01:00
Michael Hackstein 40fa74f626
Bug fix/selectivity estimates on abort (#4337) 2018-01-16 18:21:55 +01:00
Michael Hackstein 802fda8e10 Revert "Bug fix/selectivity estimates on abort (#4314)"
This reverts commit d32f3b5fe9.
2018-01-16 14:39:02 +01:00
Michael Hackstein d32f3b5fe9
Bug fix/selectivity estimates on abort (#4314) 2018-01-16 12:39:03 +01:00
Michael Hackstein d32ac3b9c8
Bug fix/rocksdb truncate (#4060) 2018-01-16 09:01:04 +01:00
Jan b2b6c06cbf
Feature/efficiency (#3736) 2018-01-05 16:51:31 +01:00
Jan 2a96df5ca5
Feature iresearch (#4071) 2017-12-18 15:04:59 +01:00
Jan 282be208cc
remove TRI_usleep and TRI_sleep, and use std::this_thread::sleep_for … (#3817) 2017-12-06 18:43:49 +01:00
Jan 4eab985e90 fix wrong ifdefs (#3803) 2017-11-27 14:20:04 +01:00
Jan b4f6ee9273 Feature/improved index api for unique constraints and replication (#3715) 2017-11-16 21:02:01 +01:00
jsteemann 571d4587ca Revert "Return offending key for unique constraint violations (#3624)"
This reverts commit 68bd31ac99.
2017-11-10 16:40:56 +01:00
Dan Larkin 68bd31ac99 Return offending key for unique constraint violations (#3624) 2017-11-10 16:03:10 +01:00
Jan 7613bc4314 Bug fix/fixes 0211 (#3568)
* remove some non-unused V8 persistents

* do not throw that many bogus assertions

* do not rely on server role being defined

* slightly better debug output for V8 context debugging

* fix collection ids in inventory response

* simplify bootstrap a bit

* slightly better error handling

* make elapsed time a queryable value

* use less memory for stub collections

* added assertions that will always make sense

* added assertions

* do not garbage-collect while waiting

* less copying of parameters

* do not show "load indexes into memory" buttons for mmfiles engine

  as all indexes are in memory anyway

* when a collection is truncated via the web interface, flush the WAL and rotate all active journals

this will make close all open journals on leader and followers and make them subject to compaction opportunities

* fix invalid server id values being passed from web interface to backend

* introduce afterTruncate method for indexes

* added test case for issue #3447

* updated CHANGELOG

* don't warn about replicationFactor for system collections

* check that the queries actually use the geo index and not some other index

* properly report error in web interface

* fix some internals checks that made truncate fail for bigger collections in maintainer mode

* also run a compact() operation after a serious truncate

in order to make iteration over the truncated range much faster
when the collection is next accessed

* increase default maximum number of V8 contexts to at least 16
2017-11-09 12:48:15 +01:00
Jan 9eb5e1545f Bug fix/fix cluster foxx queue startup (#3507)
* slightly clean up indexes

* proper stringification of error code

* do not run AQL queries during Foxx queue startup

* always create keyspace
2017-10-25 18:07:27 +02:00
Jan 0561bf45ce Bug fix/isrestore (#3283)
* Make isRestore work in the cluster.

This covers sharded collections with default sharding and non-default
sharding.

* always use locally generate revision ids for storing and looking up documents
2017-10-03 11:53:49 +02:00
Jan 2525a3a1bd Feature/mmfiles hash lookup performance (#3265)
* speed up full collection scans in mmfiles engine

* some API cleanup
2017-09-26 14:48:06 +02:00
Dan Larkin a775e37c76 Feature/unique vpack index lookup performance improvements with rocksdbkey optimizations (#3123)
* fixed issue #3075

* update the known issues document, as some issues have already been fixed

* speed up equality lookups in unique vpack indexes

* Allow for re-use of RocksDBKey objects through transaction leaser.

* Converted all RocksDBKey variants to non-static constructors.
2017-08-31 20:00:05 +02:00
Simon Grätzer 88d01b89b5 Optimizations for Caches and Graph Traversals (#3169) 2017-08-31 18:33:10 +02:00
Dan Larkin 18e8c19271 Improvements to CachedValue internals and handling. (#3126)
* Added checks everywhere for CachedValue::construct.

* Enforce 16MiB limit on keys and 4GiB limit on values.

* Protected members behind function calls in anticipation of format change.

* Shrunk CachedValue struct.

* Fixed alignment of CachedValues.

* Fixed improper comparison.
2017-08-28 15:07:16 +02:00
Jan 4f11bb2a03 reduce log spam (#3051) 2017-08-17 21:24:45 +02:00
Simon Grätzer 93aa6011c3 Feature/warmup speed (#2971) 2017-08-15 23:05:20 +02:00
Simon Grätzer 4cc8686e82 RocksDB: Optimizing dropCollection / dropIndex / fillIndex (#2927)
* cleaning up several duplicate versions of Index::memory, Index::cleanup, Index::drop, integrating branch issue-383-ensure-delete

* fixing wrong column family
2017-08-05 21:09:41 +02:00
Jan Christoph Uhde ed8efe3566 Feature/issue 387 cluster index estimates (#2866) 2017-08-01 09:53:58 +02:00
Jan bee32b6325 properly compact ranges so data actually gets deleted after drop etc. (#2771) 2017-07-11 21:47:16 +02:00
Simon Grätzer d8c57f2cc8 Feature/rocksdb hashsearch (#2713)
* Testing hash for documents

* fixing logspam error

* added base support for kHash in the edge index

* Current version of hash enabled edge index

* removed rocksdb key type byte, dynamic prefixes for hash-index

* Fixing issues with key classes

* Fixing merge conflict

* Fixing version with semi-working hash index

* fixing various issues

* Fixing hanger in EdgeIndexIterator nextExta, when _cache is disabled. Reenable cache

* Fixing geo index bounds, and fillIndexes error handling

* Adding back failure point

* Fixing RocksDBKey::vertexId

* Added version check on startup

* Fixing version check

* added one more check

* properly detect column families at startup

* Adding additional sanity checking

* Adding missing column family

* some fixes

* Turned off geoindex sync

* fixed error

* added asserts (untested)

* Adding more asserts
2017-07-05 10:18:52 +02:00
Jan Christoph Uhde 903bbbaf9a add missing try loop to nextExtra 2017-06-28 13:45:17 +02:00
Jan 6db52fe25f fix compile warnings once again (#2673) 2017-06-26 20:23:15 +02:00
Jan cb8014dbdd use correct column families for size estimates (#2624) 2017-06-22 12:32:08 +02:00
Dan Larkin a3ae2b7191 Augmented cache API and fixed a potential deadlock scenario. (#2579)
* Augmented cache API and fixed a potential deadlock scenario.

* The RocksDBEdgeIndex now tries to read up to 10 times form the cache on lock_timeout. Instead of rewriting everything after first attempt
2017-06-19 17:28:04 +02:00
Jan 329cb53952 enable RVO (#2566) 2017-06-16 09:42:10 +02:00
Jan c176a11930 control RocksDB block cache filling for the primary and edge indexes (#2532) 2017-06-09 08:32:46 +02:00
Simon Grätzer 5fc966e267 fixing warmup mode 2017-06-07 15:47:08 +02:00
jsteemann 2d96b9ab4c fix issues found by coverity scan tool 2017-06-07 13:22:18 +02:00
Michael Hackstein 7aa2a9d4c7 Merge branch 'devel' of github.com:arangodb/arangodb into devel 2017-06-07 09:27:49 +02:00
Michael Hackstein 0a98ff9b16 EdgeIndexCache now stores unindexed VPack Arrays. Saves some memory without loosing performance. 2017-06-07 09:27:44 +02:00
Dan Larkin 8c7ba2aaa3 Changed several index/collection APIs to provide better context messages.
Addresses Issue #342.
2017-06-06 12:27:08 -04:00
Michael Hackstein 6ac0c0e673 Removed dead code 2017-06-06 13:13:50 +02:00
Simon Grätzer 634254b80f adding load and unload 2017-06-02 14:44:38 +02:00
Simon Grätzer 10f6242932 removing dead code 2017-06-02 12:22:39 +02:00
Simon Grätzer 87de810566 Edge index refactoring 2017-06-02 11:20:15 +02:00
Simon Grätzer 3e8c16d1a7 Adding edge key into edge index 2017-06-01 16:07:39 +02:00
Michael Hackstein 52406567cd Replaced EdgeCursors callback to return cid,rev instead of _id. Does less costly copying and comparision internally 2017-05-31 17:49:46 +02:00
Simon Grätzer 5509776503 Fixing various issues 2017-05-30 13:44:26 +02:00
Simon Grätzer ab292b2717 Fixing edge bounds 2017-05-30 11:30:48 +02:00
Simon Grätzer 3ad939ea41 changed storage format 2017-05-30 10:12:39 +02:00
Michael Hackstein b9d2faa7cc Added warmup function for indexes.
Squashed commit of the following:

commit bc0472d212b2efef0d7b1b8a36f2b43e8432ba86
Merge: eb47631 5f87674
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:39:31 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit eb476310b0ca6165be10b37b960933886c2757f5
Merge: d725f21 32149d2
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:00:41 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit d725f21f7d61f7c79ba31fc0338881e35f4d8f48
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 15:53:23 2017 +0200

    Activated collection warmup in Cluster Mode. Added a test for collection warmup (SingleServerOnly)

commit 132bf4f9565b0dcf4ec9f84e93897b482a67ce7f
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 12:46:20 2017 +0200

    Implemented client-side warmup. It can now be triggered via http and arangosh

commit 78ea449dff86118814a2f87bdb59dc16544d92b6
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:25:40 2017 +0200

    Fixed assertion.

commit dae80f6277dde1a52eadda506858cc36e235bd55
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:08:46 2017 +0200

    Improved the CuckooIndexEstimator computeEstimate function. It is now much better on collections with many different values

commit 7abf57876511ba369d7a577e1995d4575e98c7c8
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:06:43 2017 +0200

    Edge index warmup will now first check if a document is stored and if not will insert it. Furthermore it resizes the cache with an estimate of how many documents will be most likely inserted

commit 890d8ad4cdfd155249f060fedd5c798b9531d556
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:04:47 2017 +0200

    Adjusted thresholds in transactional cache. Wastly increased the amount of tries to get the lock. Lowered the fill-grade boundaries

commit 60972ab7151a3acb78e1aae4149de11d0da7aceb
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:45:38 2017 +0200

    Added new Logtpopic CACHE that should be used to log important information regarding caches, like debug ouput on evictions / resizing, or memory-pressure resizing going on

commit 2dfacb9aef6e3fde169032514baca386786d059c
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:44:21 2017 +0200

    Fixed an assertion in index iterator. It expected _from/_to to be cached but we modified to cache entire document

commit f05beccef65853c30eda1859a39c28ea2295bc71
Merge: 13c6abf 5c46430
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 07:56:58 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit 13c6abfbc2ddb451a689011110411df68247435e
Author: Michael Hackstein <michael@arangodb.com>
Date:   Wed May 24 09:52:01 2017 +0200

    Added a getExtra feature for the edgeIndex. It now returns the complete edge as extra (temporary). Modified the EdgeCache to cache token and edge document (temporary). Added a warmup Function to collections that is used to warmup the index caches on demand.
2017-05-29 09:40:58 +02:00
Simon Grätzer d5f85d5bbb Adding more colum families 2017-05-26 10:43:17 +02:00
Simon Grätzer 67b9f75fec Merge branch 'devel' of https://github.com/arangodb/arangodb into column-family
# Conflicts:
#	arangod/RocksDBEngine/RocksDBEdgeIndex.cpp
2017-05-22 11:16:26 +02:00
Michael Hackstein 71241e6048 Fixed usage of std::min on apple clang. It complains that size_t != uint64_t. Now both sides are uint64_t 2017-05-22 08:03:18 +02:00
Michael Hackstein c9206ab218 The RocksDBEdgeIndexCache now can increase the values size it wants to cache, there is no hardcoded upperbound anymore. Also moved a lot of state out of the iterator. 2017-05-21 23:30:33 +02:00
Simon Grätzer a99611e7c0 Various changes 2017-05-19 16:43:00 +02:00
Simon Grätzer e73f446c5b Merge branch 'devel' of https://github.com/arangodb/arangodb into devel 2017-05-18 22:08:39 +02:00
Simon Grätzer 2bd2a15cb3 Column family support 2017-05-18 22:08:34 +02:00
Michael Hackstein e4050f9804 Temporary Commit, modifying some RocksDB Options and the EdgeIndex Cache size. This commit is used for NoSQL Performance Banchmark validation. 2017-05-18 15:45:49 +02:00
jsteemann 226920d7fa optimizations 2017-05-17 23:41:16 +02:00
Simon Grätzer 4b6035ff64 Fixed merge 2017-05-16 17:07:15 +02:00
Simon Grätzer e25f88ef4d Merge branch 'devel' of https://github.com/arangodb/arangodb into devel
# Conflicts:
#	arangod/RocksDBEngine/RocksDBCuckooIndexEstimator.h
#	arangod/RocksDBEngine/RocksDBEdgeIndex.cpp
#	arangod/RocksDBEngine/RocksDBIndex.cpp
2017-05-16 16:59:50 +02:00
Simon Grätzer 5265d2a063 intermediate commits 2017-05-16 16:21:39 +02:00
Jan Christoph Uhde 9aab78a6e9 remove unused code 2017-05-16 13:06:33 +02:00
Simon Grätzer df3eed2289 Various changes 2017-05-16 10:58:15 +02:00
Michael Hackstein d7d650e27d Implemented index estimates for RocksDB. Now the following indexes have
a proper estimate:
* HashIndex
* SkiplistIndex / Persistent
* EdgeIndex
2017-05-15 17:17:58 +02:00
jsteemann c423cf05a7 storage revisions in edge cache 2017-05-15 15:05:46 +02:00
jsteemann 52ba50e281 micro optimizations for edge index 2017-05-15 12:06:27 +02:00
jsteemann c6d87e0060 ctor, then dtor, then others 2017-05-15 09:31:40 +02:00
jsteemann 51e8ee50c6 fix indentation 2017-05-15 09:31:08 +02:00
jsteemann 7c959cf81f fix indentation 2017-05-15 09:30:38 +02:00
Jan Christoph Uhde 4a85f93814 avoid search in cache when iterator was resumed 2017-05-12 21:38:29 +02:00
Jan Christoph Uhde b7ca835671 release findings sooner if they contain many entries 2017-05-12 16:09:12 +02:00
Jan Christoph Uhde 227d6648a2 fix not advancing iterator in rocksdb edge index 2017-05-12 14:47:42 +02:00
Jan Christoph Uhde 07bb07de39 Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/edge-index-cache
* 'devel' of https://github.com/arangodb/arangodb:
  add debug output
  more unique ptrs
  fixed comment
  moved description into correct section
  remove unused options
  enable rocksdb logger
2017-05-12 13:56:57 +02:00
Jan Christoph Uhde db38ef3522 more clean up for edge index 2017-05-12 13:51:05 +02:00
Jan Christoph Uhde a1454ce87c clean up RocksDBEdgeIndex code 2017-05-12 13:27:25 +02:00
Jan Christoph Uhde dd00e03010 fix2 2017-05-12 13:18:13 +02:00
Jan Christoph Uhde 3afdb2a666 fix 2017-05-12 12:35:35 +02:00
Jan Christoph Uhde 7902d0e44e Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/edge-index-cache
* 'devel' of https://github.com/arangodb/arangodb: (25 commits)
  Add footer to docs, unify slogan
  Docs: make graph traversal images hard copies (fixes Cygwin builds)
  Added documentation for RocksDB and hash cache options to manual.
  Changed default cache size to something reasonable.
  cppcheck
  added comments
  fix shutdown order
  Added document cache.
  fix some replication issues
  Invalidate current coordinators on every 2nd heartbeatrun
  Show timeout when failing
  Addding locking, fixing conversion warning
  Fixing locking error
  resync changelog to 3.1
  Fix jslint
  CHANGELOG: SIGHUP forwarding
  forward SIG_HUP from supervisor
  Allow removing dbnodes
  Rocksb: index locking issue
  Fix serverAlias map...this was ridiculous
  ...
2017-05-12 10:37:24 +02:00
jsteemann abeae0a30d cppcheck 2017-05-11 21:21:58 +02:00
Jan Christoph Uhde 9e9faf7bbd hopefully fix the cache 2017-05-11 18:07:30 +02:00
Jan Christoph Uhde dce3250862 add more logging to edge index 2017-05-11 16:42:49 +02:00
Simon Grätzer 310166d3cf Fixing locking error 2017-05-11 15:55:06 +02:00
Jan Christoph Uhde ba3c797704 fix bug in edgeIndex 2017-05-11 15:20:39 +02:00
Jan Christoph Uhde c6fa6bb1ae add empty array slices in cache if no primary key for fromTo can be found 2017-05-11 13:13:15 +02:00
Jan Christoph Uhde 36d8b31335 decrease log level for edge index cache 2017-05-11 12:56:12 +02:00
Jan Christoph Uhde ee6e28a33d first tests pass for rocksdb index with cache 2017-05-11 12:11:00 +02:00
Jan Christoph Uhde e36d795dad crashing during lookup 2017-05-11 11:18:44 +02:00
Jan Christoph Uhde db7fdd2b7f add postprocessRemove for RocksDBEdgeIndex 2017-05-11 09:50:23 +02:00
Jan Christoph Uhde 847dd3ad5d Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/edge-index-cache
* 'devel' of https://github.com/arangodb/arangodb:
  geo index fixes
  adding transactions to the geo index
2017-05-10 18:43:20 +02:00
Jan Christoph Uhde 3e25aec2f3 fix enabling of caches 2017-05-10 15:58:28 +02:00
Jan Christoph Uhde e5db579b25 Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/edge-index-cache
* 'devel' of https://github.com/arangodb/arangodb: (54 commits)
  attempt to fix races in replication
  grunt build
  use exclusive transaction in syncer
  change feature order around
  dashboard
  added api cluster api routes
  add selection of storage engine
  use some parallelism
  micro optimizations
  Fix truncate for geo
  added assertions
  remove unused include
  try to fix initial sync problem
  remove unnecessary includes
  Finally fix the error where suddenly an array of dbservers are being called
  Update queryView.js
  initialize bool variable
  remove debug output
  removed unused private field
  Wrong index number
  ...
2017-05-10 15:14:08 +02:00
Simon Grätzer 5fa85761a2 adding transactions to the geo index 2017-05-10 14:54:39 +02:00
Jan Christoph Uhde 4cfbe85e4a honor batch size && length limit for cache values 2017-05-10 14:01:10 +02:00
Jan Christoph Uhde 7ea4eaf774 fix index figures 2017-05-10 14:00:40 +02:00
Jan Christoph Uhde e6d41cac9a clean up edge-index code - handling of invalidated caches still missing 2017-05-10 09:03:21 +02:00
Dan Larkin ca91c0f7c9 Added some optimizations enabled by prefix extraction. 2017-05-09 18:36:12 -04:00
Jan Christoph Uhde 738989f9a3 add blacklisting for keys in rocksdb-edge-index on insert/remove 2017-05-09 18:33:05 +02:00
Jan Christoph Uhde e29db44af6 begin work on cache for edge index 2017-05-09 17:22:27 +02:00
Simon Grätzer d9bff18814 Fulltext index 2017-05-05 13:58:11 +02:00
Simon Grätzer 804676ae65 Moving estimatedSize to memory() 2017-05-04 18:57:22 +02:00
Simon Grätzer 94e4437c3a Added some non transactional truncate logic 2017-05-04 18:07:47 +02:00
Simon Grätzer 1c82d4448c Adding compaction and estimateSize method for rocksdb 2017-05-04 13:08:02 +02:00
Simon Grätzer 0faa1adac1 fixed unique contraints 2017-05-03 17:45:21 +02:00