1
0
Fork 0
Commit Graph

72 Commits

Author SHA1 Message Date
Jan 49babbc66b
handle incomplete read errors (#9021) 2019-05-20 14:13:41 +02:00
Simon 4e80e1b351 Simplify recovery code (#8699) 2019-04-09 21:48:04 +02:00
Simon 7cd84a785a Remove Obsolete code (#8657) 2019-04-03 13:40:44 +02:00
Jan Christoph Uhde c3f7961b88 apply unique log ids (#8561) 2019-03-25 20:26:51 +01:00
Jan 1798036ea0
Bug fix/optimizations 18022019 (#8180) 2019-02-19 19:24:04 +01:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Simon cc55ef9f82 Faster index creation (#7348) (#7383) 2018-11-21 09:53:14 +01:00
Jan 0dd1776467
Make recovery more reliable (#7297) 2018-11-19 13:59:18 +01:00
Dan Larkin-York 44600636e7 Cleanup logging in RocksDBEngine. 2018-10-02 07:50:02 -04:00
Jan d78adb7ae1
Bug fix/turn off range deletes in cluster (#6178) 2018-08-17 08:41:00 +02:00
Simon 392118bd62 Use RangeDelete where possible (#6121) 2018-08-15 18:52:09 +02:00
Simon Grätzer 1cfb328bcc fix intermediate commits 2018-07-27 17:40:38 +02:00
Simon 439b12331f Big endian document keys for rocksdb (#5712) 2018-07-03 14:11:22 +02:00
Simon Grätzer 15000c5989 revert change 2018-05-12 02:07:00 +02:00
Simon Grätzer 5e4d133324 Slightly increase batch size 2018-05-12 01:27:58 +02:00
Simon Grätzer 40b45bf4d1 Make huge deletes go in smaller batches 2018-05-12 01:25:01 +02:00
Simon cde96d97aa RocksDB replication hardening & cleanup (#4639) 2018-02-23 18:15:04 +01:00
Jan 8da61a9875
Bug fix/more replication tests (#4500) 2018-02-09 13:06:47 +01:00
Jan fe0fca9029
Bug fix/restore unlock (#4387) 2018-01-25 15:56:27 +01:00
Jan 2a96df5ca5
Feature iresearch (#4071) 2017-12-18 15:04:59 +01:00
Jan a212f45a86 Bug fix/issues 2308 (#3117)
* remove API docs for endpoints that were deleted in 3.0

* remove unused variable

* set all scalar values in the ctor, so check tools do not complain

* move Result objects

* prevent potential nullptr access

* remove ambiguous boolean parameter with no clear meaning

* make code check tools happy

* don't allow copying cache Findings, allow moving Results

* the debug functionality for counting RocksDB key ranges was broken

this produced sporadic test failures

* remove superfluous iostream include

* fix range checks

* don't leak if an error happens during database removal

* try to simplify Result a bit

* try to deletion checks a bit more robust
2017-08-24 14:47:22 +02:00
Simon Grätzer 4cc8686e82 RocksDB: Optimizing dropCollection / dropIndex / fillIndex (#2927)
* cleaning up several duplicate versions of Index::memory, Index::cleanup, Index::drop, integrating branch issue-383-ensure-delete

* fixing wrong column family
2017-08-05 21:09:41 +02:00
Frank Celler a5a25754ed Feature/reduce extraction to projection (#2792)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule

* cppcheck

* try to fix compile error on MacOS

* bug fix for MacOSX

* missing namespace (in Windows compile)
2017-07-14 08:40:29 +02:00
Frank Celler 40d73d5a8b Revert "[WIP] Feature/reduce extraction to projection (#2735)"
This reverts commit 5bfcff30cd.
2017-07-12 12:52:14 +02:00
Jan 5bfcff30cd [WIP] Feature/reduce extraction to projection (#2735)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule
2017-07-12 11:22:29 +02:00
Jan bee32b6325 properly compact ranges so data actually gets deleted after drop etc. (#2771) 2017-07-11 21:47:16 +02:00
Simon Grätzer d8c57f2cc8 Feature/rocksdb hashsearch (#2713)
* Testing hash for documents

* fixing logspam error

* added base support for kHash in the edge index

* Current version of hash enabled edge index

* removed rocksdb key type byte, dynamic prefixes for hash-index

* Fixing issues with key classes

* Fixing merge conflict

* Fixing version with semi-working hash index

* fixing various issues

* Fixing hanger in EdgeIndexIterator nextExta, when _cache is disabled. Reenable cache

* Fixing geo index bounds, and fillIndexes error handling

* Adding back failure point

* Fixing RocksDBKey::vertexId

* Added version check on startup

* Fixing version check

* added one more check

* properly detect column families at startup

* Adding additional sanity checking

* Adding missing column family

* some fixes

* Turned off geoindex sync

* fixed error

* added asserts (untested)

* Adding more asserts
2017-07-05 10:18:52 +02:00
Michael Hackstein 2e1d7b0fe1 RocksDB drop collection now does intermedieate writes of batches.
Furthermore fixed an issue with Column Family usage in drop.

commit 708cd81362b56213dd4d7a69b1e4eea120de0db4
Merge: b8896ea 924ef44
Author: Michael Hackstein <michael@arangodb.com>
Date:   Thu Jun 8 14:47:56 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/drop-collection-batches

commit b8896ea5479d56f3e6d55d6094ec2658b058dc04
Author: Michael Hackstein <michael@arangodb.com>
Date:   Thu Jun 8 14:47:37 2017 +0200

    Fixed drop collection in rocksdb for all entries that are not compacted in a complete file. it was using the wrong column family.

commit 59f6a9d51b1c12d665eba683d3b09f1b01ce77fc
Author: Michael Hackstein <michael@arangodb.com>
Date:   Thu Jun 8 13:00:18 2017 +0200

    Remove large range now uses ColumnFamily comparator instead of default one. Also it writes batches of 1000 documents to RocksDB instead of full deletion at once.
2017-06-08 14:48:33 +02:00
Michael Hackstein f7e40a14ca RocksDBBounds now give info about their columnFamily. Adapted convenience methods that used those bounds to use correct column family now 2017-06-08 11:47:02 +02:00
Michael Hackstein b9d2faa7cc Added warmup function for indexes.
Squashed commit of the following:

commit bc0472d212b2efef0d7b1b8a36f2b43e8432ba86
Merge: eb47631 5f87674
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:39:31 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit eb476310b0ca6165be10b37b960933886c2757f5
Merge: d725f21 32149d2
Author: Michael Hackstein <michael@arangodb.com>
Date:   Mon May 29 09:00:41 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit d725f21f7d61f7c79ba31fc0338881e35f4d8f48
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 15:53:23 2017 +0200

    Activated collection warmup in Cluster Mode. Added a test for collection warmup (SingleServerOnly)

commit 132bf4f9565b0dcf4ec9f84e93897b482a67ce7f
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 12:46:20 2017 +0200

    Implemented client-side warmup. It can now be triggered via http and arangosh

commit 78ea449dff86118814a2f87bdb59dc16544d92b6
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:25:40 2017 +0200

    Fixed assertion.

commit dae80f6277dde1a52eadda506858cc36e235bd55
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:08:46 2017 +0200

    Improved the CuckooIndexEstimator computeEstimate function. It is now much better on collections with many different values

commit 7abf57876511ba369d7a577e1995d4575e98c7c8
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:06:43 2017 +0200

    Edge index warmup will now first check if a document is stored and if not will insert it. Furthermore it resizes the cache with an estimate of how many documents will be most likely inserted

commit 890d8ad4cdfd155249f060fedd5c798b9531d556
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 11:04:47 2017 +0200

    Adjusted thresholds in transactional cache. Wastly increased the amount of tries to get the lock. Lowered the fill-grade boundaries

commit 60972ab7151a3acb78e1aae4149de11d0da7aceb
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:45:38 2017 +0200

    Added new Logtpopic CACHE that should be used to log important information regarding caches, like debug ouput on evictions / resizing, or memory-pressure resizing going on

commit 2dfacb9aef6e3fde169032514baca386786d059c
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 10:44:21 2017 +0200

    Fixed an assertion in index iterator. It expected _from/_to to be cached but we modified to cache entire document

commit f05beccef65853c30eda1859a39c28ea2295bc71
Merge: 13c6abf 5c46430
Author: Michael Hackstein <michael@arangodb.com>
Date:   Fri May 26 07:56:58 2017 +0200

    Merge branch 'devel' of github.com:arangodb/arangodb into feature/rocks-index-warmup

commit 13c6abfbc2ddb451a689011110411df68247435e
Author: Michael Hackstein <michael@arangodb.com>
Date:   Wed May 24 09:52:01 2017 +0200

    Added a getExtra feature for the edgeIndex. It now returns the complete edge as extra (temporary). Modified the EdgeCache to cache token and edge document (temporary). Added a warmup Function to collections that is used to warmup the index caches on demand.
2017-05-29 09:40:58 +02:00
Simon Grätzer 21dd147ad6 Fixing issues 2017-05-26 14:03:12 +02:00
Michael Hackstein 91e19071a9 The Cuckoo index estimator now uses 32bit counters instead of 16bit. It should be more relyable now. 2017-05-23 12:56:24 +02:00
jsteemann b3c87b1f29 fix dropping indexes 2017-05-18 14:29:54 +02:00
jsteemann 226920d7fa optimizations 2017-05-17 23:41:16 +02:00
Simon Grätzer df3eed2289 Various changes 2017-05-16 10:58:15 +02:00
Michael Hackstein d7d650e27d Implemented index estimates for RocksDB. Now the following indexes have
a proper estimate:
* HashIndex
* SkiplistIndex / Persistent
* EdgeIndex
2017-05-15 17:17:58 +02:00
jsteemann 422b06211f fix some replication issues 2017-05-11 19:00:32 +02:00
Jan Christoph Uhde 26112be584 do checking before creating copy of slice 2017-04-26 08:33:23 +02:00
Jan Christoph Uhde 7b2acaebfb use stringref instead of allocation 2017-04-26 07:33:14 +02:00
Dan Larkin 9dd275b497 Fixed dump/restore tests. 2017-04-25 13:54:56 -04:00
Dan Larkin 57701ff766 Added (rough) logger-follow handler. 2017-04-20 14:59:49 -04:00
Dan Larkin dfd1e78f35 Added objectId-to-database+collection mapping. 2017-04-19 13:56:17 -04:00
Dan Larkin eaac6c013c Fixed issue with cache transactions. 2017-04-07 16:58:22 -04:00
jsteemann 4cf5fedca9 honor subcodes in rocksdb statuses 2017-04-07 14:55:55 +02:00
jsteemann 2b0673c3fe removed type "Index" 2017-04-07 12:05:56 +02:00
Dan Larkin e303e2a51f Added RocksDB export functionality and some bug fixes. 2017-04-04 17:33:14 -04:00
Simon Grätzer 96dc714139 Fixed some tests in shell_server 2017-04-04 15:10:17 +02:00
Simon Grätzer 245b4b87ea Fixing comparator use 2017-04-04 14:11:14 +02:00
Jan Christoph Uhde 34aa22948d add helper function for non-transactional iteration of bounds 2017-04-04 09:54:58 +02:00
Jan Christoph Uhde 5773a028c8 add more code for dropping databases 2017-04-03 21:29:25 +02:00