1
0
Fork 0
arangodb/arangod/MMFiles
Frank Celler a5a25754ed Feature/reduce extraction to projection (#2792)
* reduce extractions to projections

* recycle string buffers in SocketTask

* micro optimizations for mmfiles indexes

* added special lookup function for _key

* moved function into the correct file

* speed up key buffer allocations a bit

* added noexcept specifier

* correctly name variable

* explicitly move bounds

* fix and speedup from/toPersistent functions

* reuse string from ManagedDocumentResult for multiple lookups

* use move-assign

* a bit less work for single server

* speedup AQL function HASH

* single fetch optimization

* performance optimization for the case when no documents need to be returned

* make reduce-extraction-to-projection a RocksDB-only optimizer rule

* cppcheck

* try to fix compile error on MacOS

* bug fix for MacOSX

* missing namespace (in Windows compile)
2017-07-14 08:40:29 +02:00
..
CMakeLists.txt optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
MMFilesAllocatorThread.cpp remove logging for non-topics 2017-02-10 09:32:50 +01:00
MMFilesAllocatorThread.h
MMFilesAqlFunctions.cpp fix handling of externals 2017-03-30 17:37:09 +02:00
MMFilesAqlFunctions.h functionality for plugging REST handlers 2017-03-09 17:12:00 +01:00
MMFilesCleanupThread.cpp removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesCleanupThread.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesCollection.cpp Feature/reduce extraction to projection (#2792) 2017-07-14 08:40:29 +02:00
MMFilesCollection.h Feature/reduce extraction to projection (#2792) 2017-07-14 08:40:29 +02:00
MMFilesCollectionExport.cpp move common code into its own file 2017-05-22 17:46:12 +02:00
MMFilesCollectionExport.h move common code into its own file 2017-05-22 17:46:12 +02:00
MMFilesCollectionKeys.cpp removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesCollectionKeys.h move collection keys into engine 2017-03-21 14:23:29 +01:00
MMFilesCollectionReadLocker.h honor exclusive locks 2017-04-25 12:56:56 +02:00
MMFilesCollectionWriteLocker.h honor exclusive locks 2017-04-25 12:56:56 +02:00
MMFilesCollectorCache.h renaming 2017-03-22 13:01:40 +01:00
MMFilesCollectorThread.cpp Bug fix/small issues (#2696) 2017-07-01 00:30:45 +02:00
MMFilesCollectorThread.h Moved ditches into MMFiles. 2017-02-23 09:02:34 +01:00
MMFilesCompactionLocker.h renamed classes 2017-02-28 17:59:48 +01:00
MMFilesCompactorThread.cpp removing unused and unimplemented methods 2017-05-10 18:25:41 +02:00
MMFilesCompactorThread.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesDatafile.cpp do not include our datafiles in core files (#2630) 2017-06-22 12:34:27 +02:00
MMFilesDatafile.h fix WAL file tick updating (happened too late before) (#2762) 2017-07-10 17:41:08 +02:00
MMFilesDatafileHelper.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesDatafileStatistics.cpp moved files 2017-03-16 16:45:02 +01:00
MMFilesDatafileStatistics.h moved files 2017-03-16 16:45:02 +01:00
MMFilesDatafileStatisticsContainer.cpp moved files 2017-03-16 16:45:02 +01:00
MMFilesDatafileStatisticsContainer.h moved files 2017-03-16 16:45:02 +01:00
MMFilesDitch.cpp renamed classes 2017-02-27 11:40:15 +01:00
MMFilesDitch.h renamed classes 2017-02-27 11:40:15 +01:00
MMFilesDocumentOperation.cpp Moved primaryIndex knowledge out of Logical Collection. Also moved logic for close into MMFiles 2017-02-21 14:12:07 +01:00
MMFilesDocumentOperation.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesDocumentPosition.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesEdgeIndex.cpp cleanup error codes a bit 2017-06-09 14:50:05 +02:00
MMFilesEdgeIndex.h Changed several index/collection APIs to provide better context messages. 2017-06-06 12:27:08 -04:00
MMFilesEngine.cpp change init order of mmfiles logfile manager and database feature (#2697) 2017-07-01 08:47:57 +02:00
MMFilesEngine.h Feature/reduce extraction to projection (#2792) 2017-07-14 08:40:29 +02:00
MMFilesExportCursor.cpp move common code into its own file 2017-05-22 17:46:12 +02:00
MMFilesExportCursor.h move attributes into engine 2017-03-21 10:31:50 +01:00
MMFilesFulltextIndex.cpp unify index struct return values in cluster and non-cluster for fulltext and geo indexes (#2534) 2017-06-08 13:35:00 +02:00
MMFilesFulltextIndex.h Changed several index/collection APIs to provide better context messages. 2017-06-06 12:27:08 -04:00
MMFilesGeoIndex.cpp Changed several index/collection APIs to provide better context messages. 2017-06-06 12:27:08 -04:00
MMFilesGeoIndex.h Changed several index/collection APIs to provide better context messages. 2017-06-06 12:27:08 -04:00
MMFilesHashIndex.cpp Feature/reduce extraction to projection (#2792) 2017-07-14 08:40:29 +02:00
MMFilesHashIndex.h Changed several index/collection APIs to provide better context messages. 2017-06-06 12:27:08 -04:00
MMFilesIncrementalSync.h Bug fix/adjust agency comm timeouts (#2765) 2017-07-13 00:44:28 +02:00
MMFilesIndexElement.cpp
MMFilesIndexElement.h
MMFilesIndexFactory.cpp Feature/reduce extraction to projection (#2792) 2017-07-14 08:40:29 +02:00
MMFilesIndexFactory.h Removed code paths that wrote objectIds into the Agency. This did break replication. 2017-05-02 14:59:19 +02:00
MMFilesLogfileManager.cpp Feature/reduce extraction to projection (#2792) 2017-07-14 08:40:29 +02:00
MMFilesLogfileManager.h change init order of mmfiles logfile manager and database feature (#2697) 2017-07-01 08:47:57 +02:00
MMFilesOptimizerRules.cpp Working geo index + tests 2017-05-09 15:04:51 +02:00
MMFilesOptimizerRules.h Working geo index + tests 2017-05-09 15:04:51 +02:00
MMFilesPathBasedIndex.cpp added "deduplicate" attribute for array indexes (#2644) 2017-06-30 17:53:50 +02:00
MMFilesPathBasedIndex.h added "deduplicate" attribute for array indexes (#2644) 2017-06-30 17:53:50 +02:00
MMFilesPersistentIndex.cpp added "deduplicate" attribute for array indexes (#2644) 2017-06-30 17:53:50 +02:00
MMFilesPersistentIndex.h Changed several index/collection APIs to provide better context messages. 2017-06-06 12:27:08 -04:00
MMFilesPersistentIndexFeature.cpp Bug fix/fix rocksdb options (#2702) 2017-07-02 23:13:42 +02:00
MMFilesPersistentIndexFeature.h change --rocksdb.enabled to be a obosolete option 2017-04-19 14:54:10 +02:00
MMFilesPersistentIndexKeyComparator.cpp class renaming 2017-03-08 17:54:00 +01:00
MMFilesPersistentIndexKeyComparator.h class renaming 2017-03-08 17:54:00 +01:00
MMFilesPrimaryIndex.cpp Changed several index/collection APIs to provide better context messages. 2017-06-06 12:27:08 -04:00
MMFilesPrimaryIndex.h Changed several index/collection APIs to provide better context messages. 2017-06-06 12:27:08 -04:00
MMFilesRemoverThread.cpp remove logging for non-topics 2017-02-10 09:32:50 +01:00
MMFilesRemoverThread.h
MMFilesRestExportHandler.cpp move common code into its own file 2017-05-22 17:46:12 +02:00
MMFilesRestExportHandler.h move common code into its own file 2017-05-22 17:46:12 +02:00
MMFilesRestHandlers.cpp Moved MMFiles replication handler. 2017-04-18 15:35:45 -04:00
MMFilesRestHandlers.h make RestWalHandler a handler of the MMFiles engine 2017-03-10 09:00:37 +01:00
MMFilesRestReplicationHandler.cpp Bug fix/adjust agency comm timeouts (#2765) 2017-07-13 00:44:28 +02:00
MMFilesRestReplicationHandler.h more dictributeShardsLike code mergedfrom 3.1 2017-04-24 15:13:40 +02:00
MMFilesRestWalHandler.cpp issue #2505 2017-05-29 10:29:47 +02:00
MMFilesRestWalHandler.h make RestWalHandler a handler of the MMFiles engine 2017-03-10 09:00:37 +01:00
MMFilesRevisionsCache.cpp renaming 2017-03-22 13:01:40 +01:00
MMFilesRevisionsCache.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesSkiplist.h
MMFilesSkiplistIndex.cpp Feature/reduce extraction to projection (#2792) 2017-07-14 08:40:29 +02:00
MMFilesSkiplistIndex.h Changed several index/collection APIs to provide better context messages. 2017-06-06 12:27:08 -04:00
MMFilesSynchronizerThread.cpp remove logging for non-topics 2017-02-10 09:32:50 +01:00
MMFilesSynchronizerThread.h
MMFilesToken.h
MMFilesTransactionCollection.cpp slightly less allocations for MMFiles transaction ops 2017-06-07 00:30:18 +02:00
MMFilesTransactionCollection.h slightly less allocations for MMFiles transaction ops 2017-06-07 00:30:18 +02:00
MMFilesTransactionContextData.cpp removed unused includes 2017-02-27 17:29:28 +01:00
MMFilesTransactionContextData.h removed unused includes 2017-02-27 17:29:28 +01:00
MMFilesTransactionManager.cpp optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
MMFilesTransactionManager.h optimize away own transaction management for RocksDB engine 2017-05-04 23:16:53 +02:00
MMFilesTransactionState.cpp improve waitForSync performance for mmfiles 2017-06-06 12:44:39 +02:00
MMFilesTransactionState.h added tests for intermediate commits 2017-05-30 12:03:58 +02:00
MMFilesV8Functions.cpp fix db._engineStats() 2017-05-04 00:33:36 +02:00
MMFilesV8Functions.h fix db._engineStats() 2017-05-04 00:33:36 +02:00
MMFilesView.cpp some fixes for view handling 2017-03-17 11:18:46 +01:00
MMFilesView.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesWalLogfile.cpp move encoding functions into their own file 2017-02-14 12:57:15 +01:00
MMFilesWalLogfile.h renaming 2017-03-22 13:01:40 +01:00
MMFilesWalMarker.h added stubs for indexes 2017-03-23 15:24:00 +01:00
MMFilesWalRecoverState.cpp Added recovery tests for views and fixed a few related bugs. (#2682) 2017-06-29 17:09:43 +02:00
MMFilesWalRecoverState.h renaming 2017-03-22 13:01:40 +01:00
MMFilesWalRecoveryFeature.cpp try to fix startup order 2017-04-25 11:14:10 +02:00
MMFilesWalRecoveryFeature.h
MMFilesWalSlot.cpp fix WAL file tick updating (happened too late before) (#2762) 2017-07-10 17:41:08 +02:00
MMFilesWalSlot.h fix WAL file tick updating (happened too late before) (#2762) 2017-07-10 17:41:08 +02:00
MMFilesWalSlots.cpp fix WAL file tick updating (happened too late before) (#2762) 2017-07-10 17:41:08 +02:00
MMFilesWalSlots.h fix WAL file tick updating (happened too late before) (#2762) 2017-07-10 17:41:08 +02:00
MMFilesWalSyncRegion.h
README.md remove transferMarkers from StorageEngine.h 2017-02-21 16:00:37 +01:00
mmfiles-fulltext-common.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-handles.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-handles.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-index.cpp fix invalid new/free pairings 2017-06-07 13:21:41 +02:00
mmfiles-fulltext-index.h cppcheck 2017-05-10 18:24:34 +02:00
mmfiles-fulltext-list.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-list.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-query.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-query.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-result.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-fulltext-result.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-geo-index.cpp move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-geo-index.h move replication-dump into engine 2017-03-21 13:55:57 +01:00
mmfiles-replication-dump.cpp Squashed commit of the following: 2017-05-12 23:22:22 +02:00
mmfiles-replication-dump.h Bug fix/cleanup stuff (#2734) 2017-07-05 19:34:16 +02:00

README.md

MMFilesEngine

How operations are stored - Overview

All operations like insert or remove are written as markers to a write ahead log (WAL). This WAL consists of files of a certain size and if such a file is full (or is manually flushed), all relevant markers are transferred (transferMarkers()) to the journals of the respective collections. During the transfer any obsolete markers will be thrown away: a sequence of insert, remove, insert on the same document will result in the last insert discarding the previous operations. When a journal file of size (journalSize()) is full, it will be sealed and renamed. By applying these operations it will become a datafile that is read-only. Datafiles will eventually be merged by a compactor thread that does about the same work as the transferMarkers function, reducing the size of the stored data.

Ditches

Ditches are used to pin objects in WAL or journal as long as they are used by other operations.