1
0
Fork 0
Commit Graph

54 Commits

Author SHA1 Message Date
Jan 9ffbc908f3
replication bugfixes and improvements ported from 3.2 (#4497) 2018-02-09 13:39:31 +01:00
Jan 4c6337611b
properly unlock after restore-indexes (#4369) 2018-01-25 15:56:13 +01:00
Jan a212f45a86 Bug fix/issues 2308 (#3117)
* remove API docs for endpoints that were deleted in 3.0

* remove unused variable

* set all scalar values in the ctor, so check tools do not complain

* move Result objects

* prevent potential nullptr access

* remove ambiguous boolean parameter with no clear meaning

* make code check tools happy

* don't allow copying cache Findings, allow moving Results

* the debug functionality for counting RocksDB key ranges was broken

this produced sporadic test failures

* remove superfluous iostream include

* fix range checks

* don't leak if an error happens during database removal

* try to simplify Result a bit

* try to deletion checks a bit more robust
2017-08-24 14:47:22 +02:00
Simon Grätzer 4cc8686e82 RocksDB: Optimizing dropCollection / dropIndex / fillIndex (#2927)
* cleaning up several duplicate versions of Index::memory, Index::cleanup, Index::drop, integrating branch issue-383-ensure-delete

* fixing wrong column family
2017-08-05 21:09:41 +02:00
Frank Celler a5a25754ed Feature/reduce extraction to projection (#2792)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule

* cppcheck

* try to fix compile error on MacOS

* bug fix for MacOSX

* missing namespace (in Windows compile)
2017-07-14 08:40:29 +02:00
Frank Celler 40d73d5a8b Revert "[WIP] Feature/reduce extraction to projection (#2735)"
This reverts commit 5bfcff30cd.
2017-07-12 12:52:14 +02:00
Jan 5bfcff30cd [WIP] Feature/reduce extraction to projection (#2735)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule
2017-07-12 11:22:29 +02:00
Jan bee32b6325 properly compact ranges so data actually gets deleted after drop etc. (#2771) 2017-07-11 21:47:16 +02:00
Simon Grätzer d8c57f2cc8 Feature/rocksdb hashsearch (#2713)
* Testing hash for documents

* fixing logspam error

* added base support for kHash in the edge index

* Current version of hash enabled edge index

* removed rocksdb key type byte, dynamic prefixes for hash-index

* Fixing issues with key classes

* Fixing merge conflict

* Fixing version with semi-working hash index

* fixing various issues

* Fixing hanger in EdgeIndexIterator nextExta, when _cache is disabled. Reenable cache

* Fixing geo index bounds, and fillIndexes error handling

* Adding back failure point

* Fixing RocksDBKey::vertexId

* Added version check on startup

* Fixing version check

* added one more check

* properly detect column families at startup

* Adding additional sanity checking

* Adding missing column family

* some fixes

* Turned off geoindex sync

* fixed error

* added asserts (untested)

* Adding more asserts
2017-07-05 10:18:52 +02:00
Michael Hackstein 2e1d7b0fe1 RocksDB drop collection now does intermedieate writes of batches.
Furthermore fixed an issue with Column Family usage in drop.

commit 708cd81362b56213dd4d7a69b1e4eea120de0db4
Merge: b8896ea 924ef44
Author: Michael Hackstein <michael@arangodb.com>
Date:   Thu Jun 8 14:47:56 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/drop-collection-batches

commit b8896ea5479d56f3e6d55d6094ec2658b058dc04
Author: Michael Hackstein <michael@arangodb.com>
Date:   Thu Jun 8 14:47:37 2017 +0200

    Fixed drop collection in rocksdb for all entries that are not compacted in a complete file. it was using the wrong column family.

commit 59f6a9d51b1c12d665eba683d3b09f1b01ce77fc
Author: Michael Hackstein <michael@arangodb.com>
Date:   Thu Jun 8 13:00:18 2017 +0200

    Remove large range now uses ColumnFamily comparator instead of default one. Also it writes batches of 1000 documents to RocksDB instead of full deletion at once.
2017-06-08 14:48:33 +02:00
Michael Hackstein f7e40a14ca RocksDBBounds now give info about their columnFamily. Adapted convenience methods that used those bounds to use correct column family now 2017-06-08 11:47:02 +02:00
Michael Hackstein b9d2faa7cc Added warmup function for indexes.
Squashed commit of the following:

commit bc0472d212b2efef0d7b1b8a36f2b43e8432ba86
Merge: eb47631 5f87674
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:39:31 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit eb476310b0ca6165be10b37b960933886c2757f5
Merge: d725f21 32149d2
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:00:41 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit d725f21f7d61f7c79ba31fc0338881e35f4d8f48
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 15:53:23 2017 +0200

    Activated collection warmup in Cluster Mode. Added a test for collection warmup (SingleServerOnly)

commit 132bf4f9565b0dcf4ec9f84e93897b482a67ce7f
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 12:46:20 2017 +0200

    Implemented client-side warmup. It can now be triggered via http and arangosh

commit 78ea449dff86118814a2f87bdb59dc16544d92b6
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:25:40 2017 +0200

    Fixed assertion.

commit dae80f6277dde1a52eadda506858cc36e235bd55
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:08:46 2017 +0200

    Improved the CuckooIndexEstimator computeEstimate function. It is now much better on collections with many different values

commit 7abf57876511ba369d7a577e1995d4575e98c7c8
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:06:43 2017 +0200

    Edge index warmup will now first check if a document is stored and if not will insert it. Furthermore it resizes the cache with an estimate of how many documents will be most likely inserted

commit 890d8ad4cdfd155249f060fedd5c798b9531d556
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:04:47 2017 +0200

    Adjusted thresholds in transactional cache. Wastly increased the amount of tries to get the lock. Lowered the fill-grade boundaries

commit 60972ab7151a3acb78e1aae4149de11d0da7aceb
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:45:38 2017 +0200

    Added new Logtpopic CACHE that should be used to log important information regarding caches, like debug ouput on evictions / resizing, or memory-pressure resizing going on

commit 2dfacb9aef6e3fde169032514baca386786d059c
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:44:21 2017 +0200

    Fixed an assertion in index iterator. It expected _from/_to to be cached but we modified to cache entire document

commit f05beccef65853c30eda1859a39c28ea2295bc71
Merge: 13c6abf 5c46430
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 07:56:58 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit 13c6abfbc2ddb451a689011110411df68247435e
Author: Michael Hackstein <michael@arangodb.com>
Date:   Wed May 24 09:52:01 2017 +0200

    Added a getExtra feature for the edgeIndex. It now returns the complete edge as extra (temporary). Modified the EdgeCache to cache token and edge document (temporary). Added a warmup Function to collections that is used to warmup the index caches on demand.
2017-05-29 09:40:58 +02:00
Simon Grätzer 21dd147ad6 Fixing issues 2017-05-26 14:03:12 +02:00
Michael Hackstein 91e19071a9 The Cuckoo index estimator now uses 32bit counters instead of 16bit. It should be more relyable now. 2017-05-23 12:56:24 +02:00
jsteemann b3c87b1f29 fix dropping indexes 2017-05-18 14:29:54 +02:00
jsteemann 226920d7fa optimizations 2017-05-17 23:41:16 +02:00
Simon Grätzer df3eed2289 Various changes 2017-05-16 10:58:15 +02:00
Michael Hackstein d7d650e27d Implemented index estimates for RocksDB. Now the following indexes have
a proper estimate:
* HashIndex
* SkiplistIndex / Persistent
* EdgeIndex
2017-05-15 17:17:58 +02:00
jsteemann 422b06211f fix some replication issues 2017-05-11 19:00:32 +02:00
Jan Christoph Uhde 26112be584 do checking before creating copy of slice 2017-04-26 08:33:23 +02:00
Jan Christoph Uhde 7b2acaebfb use stringref instead of allocation 2017-04-26 07:33:14 +02:00
Dan Larkin 9dd275b497 Fixed dump/restore tests. 2017-04-25 13:54:56 -04:00
Dan Larkin 57701ff766 Added (rough) logger-follow handler. 2017-04-20 14:59:49 -04:00
Dan Larkin dfd1e78f35 Added objectId-to-database+collection mapping. 2017-04-19 13:56:17 -04:00
Dan Larkin eaac6c013c Fixed issue with cache transactions. 2017-04-07 16:58:22 -04:00
jsteemann 4cf5fedca9 honor subcodes in rocksdb statuses 2017-04-07 14:55:55 +02:00
jsteemann 2b0673c3fe removed type "Index" 2017-04-07 12:05:56 +02:00
Dan Larkin e303e2a51f Added RocksDB export functionality and some bug fixes. 2017-04-04 17:33:14 -04:00
Simon Grätzer 96dc714139 Fixed some tests in shell_server 2017-04-04 15:10:17 +02:00
Simon Grätzer 245b4b87ea Fixing comparator use 2017-04-04 14:11:14 +02:00
Jan Christoph Uhde 34aa22948d add helper function for non-transactional iteration of bounds 2017-04-04 09:54:58 +02:00
Jan Christoph Uhde 5773a028c8 add more code for dropping databases 2017-04-03 21:29:25 +02:00
Jan Christoph Uhde 256b2b2d6a construct rocksdbkey from slice 2017-04-03 17:50:36 +02:00
Jan Christoph Uhde ae75917640 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
* 'engine-api' of https://github.com/arangodb/arangodb:
  added assertion to Comparator
  fix RemoveLargeRange
  fix ub
  fix assertion failure at shutdown
  Remove provisional code
  Fixed counter
  remove usleeps
  Fixed deadlock
  Fixed broken merge
  Fixing some issues
  added p/ to gitignore
  Integrated counters with collections
  properly include the javascript enterprise folders into the test tarballs
  fix parsing for URLs starting with ldap://
  fix filter for replication testcases
  Added counter manager
  updated licenses, added generator
  removed jshint completely
2017-04-03 16:54:09 +02:00
Jan Christoph Uhde 234180c08b wip - drop database 2017-04-03 16:52:21 +02:00
jsteemann c4ecfbf089 fix RemoveLargeRange 2017-04-03 15:44:30 +02:00
Simon Grätzer bc36c56716 Fixed broken merge 2017-04-03 14:52:12 +02:00
Simon Grätzer c46db78904 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
# Conflicts:
#	arangod/RocksDBEngine/RocksDBCommon.h
#	arangod/RocksDBEngine/RocksDBEngine.cpp
2017-04-03 13:27:04 +02:00
Jan Christoph Uhde e49024f16e add drop collection to rocksdb engine 2017-04-03 10:41:02 +02:00
Simon Grätzer 01036b65d9 Added counter manager 2017-04-03 09:23:35 +02:00
Jan Christoph Uhde c038815941 Merge branch 'engine-api' of https://github.com/arangodb/arangodb into engine-api
* 'engine-api' of https://github.com/arangodb/arangodb:
  transaction-specific count tracking
  initial implementation of count()
  Fixed memory leaks in traverser.
  Fix links in docs
  Fix links in docs
  pass trx into numberDocuments
  Fixed undefined behaviour in TraverserCache.
  improve error message in case wrong JSON is sent to the server
2017-03-31 15:06:21 +02:00
Jan Christoph Uhde 0140822dc4 implement persist index 2017-03-31 15:06:06 +02:00
jsteemann 07cdc49504 transaction-specific count tracking 2017-03-31 14:57:21 +02:00
Simon Grätzer fd52c1a7f4 Added count function 2017-03-30 19:20:42 +02:00
Simon Grätzer e246a8d8cb Moved „removeLargeRange“ method to commons 2017-03-30 14:46:21 +02:00
jsteemann 38ac3c23c8 fixed serialization bug 2017-03-29 09:17:23 +02:00
Simon Grätzer 7a26fb22c1 Pushing commons method 2017-03-28 14:07:01 +02:00
Simon Grätzer 9e1a181393 Edge index tax changes 2017-03-28 14:02:29 +02:00
Dan Larkin 61037c3078 Added some documentation to key/value/comparator classes. 2017-03-27 21:07:52 -04:00
Dan Larkin cbafcadccd Finished conversion of RocksDBEntry to RocksDBKey and RocksDBValue. 2017-03-27 16:51:30 -04:00