1
0
Fork 0
Commit Graph

51 Commits

Author SHA1 Message Date
Jan 46674831c9
Bug fix/improve stringutils performance (#10208) 2019-10-10 17:08:03 +02:00
Jan 032dba152f
fixed issue #10183: Arangoimport imports on _system when you try to create a new database (#10185) 2019-10-08 20:30:47 +02:00
Dan Larkin-York a83c2323c9 Refactor ApplicationServer stack (#9965) 2019-09-25 17:31:59 +02:00
Dan Larkin-York 3d0246cb18 Decentralize includes (#9623) 2019-08-06 15:32:09 +02:00
Matthew Von-Maszewski 91b56a50a3
Feature: Add gzip and encryption to import/export (#9560)
* port of feature-3.4/mv-gzip-export to devel branch

* add explicit namespaces so gcc 6.3.0 would successfully compile

* add conditional cleanup of ENCRYPTION file from another branch

* use _lseek() for Windows build to avoid deprecated warning.

* change from ifdef for lseek variants to TRI_LSEEK.

* force Windows lseek to return Linux expected type.
2019-07-26 07:53:39 -04:00
Matthew Von-Maszewski 40a5c4f8b8
Bugfix: Import would not report communication errors (#9494)
* correct handleResult() to report transmission errors and stop.  Add reporting of line numbers impacted if CSV

* always generate import summary, even if import stops due to error

* pass line numbers to senderThread for potential error reporting.

* move line count from zero based to one based in error report.

* forgot to update CHANGELOG
2019-07-17 18:29:07 +02:00
Jan Christoph Uhde 677a79026c Foxx Security (#8845) 2019-04-25 09:56:29 +02:00
Jan Christoph Uhde c3f7961b88 apply unique log ids (#8561) 2019-03-25 20:26:51 +01:00
Jan a841d472be
emit a warning when connecting with the client tools to a DBSERVER (#8397) 2019-03-13 13:08:41 +01:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Vasiliy 6fd541d110 issue 427.5: use ApplicationServer reference instead of pointer (#6145)
* issue 427.5: use ApplicationServer reference instead of pointer

* address MSVC build failure
2018-08-15 12:16:02 +03:00
Jan b278d6874a
allow master & slave to work in parallel for RocksDB WAL tailing (#6059) 2018-08-03 13:37:53 +02:00
Michael Hackstein 7a95c5e675
Feature/feature phases (#5272)
* Added feature phases

* BasicsPhase and DatabasePhase to the required files. Server now has Feature circles and does not boot. Will be sorted out later on.

* Added ClusterPhase to features

* Added V8Phase to the required features

* Added AQLPhase to the affected features

* Added ServerPhase to Features

* Added FoxxPhase to the relevant features

* Added AgencyPhase to the relevant features

* Moved registration from local variable SYS_SYSTEM_REPLICATION_FACTOR from cluster to V8 as their ordering is now vice versa

* Moved Bootstrap feature into FoxxPhase. It could be moved to ServerPhase easily if the FoxxQueue dependency would be removed

* Final movement of Startup Phases. Now solved all circles.

* Removed merge conflict

* Moved ReplicationTimeout into cluster phase and fixed cross-phase requirements

* Added greetings phase. This phase separates the Basics Phase and is the first to be run. Includes Logger and Hello/Goodbye

* Added the GreetingsPhase in the corresponding features. Now all BasicsPhase features start after greetings Phase. There is some issue in this branch which prevents the Agency from Gossipping right now. Will be fixed next

* Moved creation of the Agent into the prepare phase of the feature. THereby it is guaranteed that agents at least exists before the GeneralServer is activating endpoints

* Recovery needs to be started after the ServerID

* Moved log output of FeaturePhases to DEBUG instead of ERROR.

* Added feature phases for clients

* ClusterFeature now does not directly require AgencyFeature any more

* Added requirement of TravEngineRegistryFeature in AQL feature. Otherwise shutdown may be undefined

* The ApplicationServer can now handout the list of ordered features. Used for testing purposes

* Fixed IResearchVew Tests Setup to honor new feature ordering

* Fixed IResearchViewDBServer Tests Setup to honor new feature ordering

* Started fixing IResearchView Coordinator tests with startup ordering. Not finished yet

* Added startup phases to ViewCoordinator test

* Disabled expected logoutput in ClusterRepairsTest

* Fixed indention in test code

* LinkCoordinator now honors startup ordering

* Link meta now honors startup rdering

* Supress expected cluster logs in ViewTest

* Removed '#' accidentially added.
2018-07-16 14:09:36 +02:00
Jan 7351423ad5
cleanup import messages, reuse StringBuffer for JSON import (#5569) 2018-06-11 09:47:26 +02:00
Matthew Von-Maszewski 3678dc7b68 latency histogram option for arangoimport (#5552)
* basic histogram thread using non-arangod threading ... soon to change.
* Add --latency flag.  start histogram thread if set.
* convert quick_hist to use arangodb thread and condition objects
* move histogram object to be within ImportStatistics structure.
* update the thread shutdown to 1. work and 2. post last data
* add text that values are in microseconds to command line help.
* CHANGELOG note for new import latency option
* quick_hist.h rename to QuickHistogram.h
* update include file name to QuickHistogram.h
* update variable names to match coding standard
* a couple more code standard changes.  try/catch around push_back
2018-06-08 11:17:25 +02:00
Matthew Von-Maszewski 9ff6a41236 Add pacing logic to arangoimp (#5238)
* initial check-in of working auto block size tuning for import.  Needs clean-up.
* partial fix in this branch to enable testing, awaiting better fix in different branch.
* move pacing code from findIdleSender to AutoTuneThread object.
* move pacing code from findIdleSender to AutoTuneThread object.  clean up dead code.  add comments about algorithm.
* edits to algorithm discussion
* correct logging level.  add CHANGELOG entry.
* update comment concerning proper usage of shutdown()
* initialize new static member MaxBatchSize.  Correct sendCsvBuffer() call to addPeriodByteCount() since _outputBuffer drained by sendData().
* remove redundant semicolons.  add new static member MaxBatchSize.
* move MaxBatchSize to ImportHelper so both ImportFeature object and AutoTuneThread object can access it.
* minor updates to comments and statics
* add pacing discussion to import manual page
* minor edit
2018-05-02 22:34:04 +02:00
Dan Larkin-York 38f162e344 Parallelize arangodump (#4356) 2018-03-26 15:55:14 +02:00
Jan Christoph Uhde bddadda0ee add 'create database' to 'import feature' (#4278) 2018-01-15 15:15:31 +01:00
Jan 6040b4ae9f
add ignore-missing option to arangoimport (#4129) 2017-12-22 11:29:10 +01:00
Simon Grätzer 885ea3dd5f Remove attribute option for arangoimp (#3141)
* Adding a remove attribute option to arangoimp (only for csv)

* Adding documentation

* Fixed log statement

* Update CHANGELOG
2017-08-30 10:45:02 +02:00
Simon Grätzer 2bc6a71d48 Fixing error reporting in arangoimp (#3153) 2017-08-28 15:04:45 +02:00
Simon Grätzer f50a36181d Various authentication related issues (#2824) 2017-07-18 22:22:09 +02:00
jsteemann 2275fc8174 better automatic file-type detection for arangoimp 2017-06-08 14:55:21 +02:00
jsteemann 2cfd7460d7 fix output 2017-05-31 11:38:20 +02:00
Simon Grätzer 0bf4467457 small fixes 2017-05-24 19:21:23 +02:00
Simon Grätzer 2f2d07ab9a Multihreaded import 2017-05-24 18:37:45 +02:00
jsteemann 53fa66e60d move parts from ProgramOptions from headers into cpp files 2017-05-23 10:30:36 +02:00
jsteemann aab138a4e6 added input type jsonl 2017-03-06 17:39:44 +01:00
jsteemann 6478c6d043 updated CHANGELOG 2017-03-03 15:03:52 +01:00
jsteemann db9099f0f9 Merge branch 'devel' of https://github.com/arangodb/arangodb into engine-api 2017-03-03 12:07:50 +01:00
jsteemann 7c0c84c21d added `--translate` option for arangoimp to translate attribute names from the input files to attriubte names expected by ArangoDB
The `--translate` option can be specified multiple times (once per translation
to be executed). The following example renames the "id" column from the input
file to "_key", and the "from" column to "_from", and the "to" column to "_to":

    arangoimp --type csv --file data.csv --translate "id=_key" --translate "from=_from" --translate "to=_to"

`--translate` works for CSV and TSV inputs only.
2017-03-03 11:35:16 +01:00
jsteemann 6e4e29943d added 'auto' file type 2017-03-02 23:35:52 +01:00
jsteemann d024a6d00a remove logging for non-topics 2017-02-10 09:32:50 +01:00
Jan Steemann 4b5249dc5e issue #2022: double max allowed request body size, cap --batch-size value in arangoimp 2016-08-29 09:34:23 +02:00
jsteemann 85cb227a4f fixed wrong default values for _sslOptions 2016-08-23 14:34:05 +02:00
Jan Steemann 7b62d71c63 added --skip-lines for import 2016-07-04 11:18:15 +02:00
Jan Steemann 33b40e3bf6 fix default separator 2016-06-15 18:24:10 +02:00
Jan Steemann 1726c53805 fixed some import edge cases 2016-06-15 18:16:49 +02:00
Jan Steemann 924bc0143d properly handle \t 2016-06-15 17:45:27 +02:00
Frank Celler 54300ed30c moved Console/ClientFeature to arangosh 2016-06-01 09:27:29 +02:00
jsteemann d62143ce83 make boolean command-line parameters optional 2016-04-27 11:37:42 +02:00
jsteemann 348e1e3397 client tools simplifications and fixes 2016-04-25 22:28:43 +02:00
jsteemann dcc5825f93 cleaned up features a bit 2016-04-24 14:29:29 +02:00
Frank Celler d6e8046992 updated cleanupCFiles 2016-04-23 16:34:57 -07:00
Frank Celler 3bb24146a6 Merge branch 'devel' of github.com:arangodb/arangodb into FMH 2016-04-19 19:18:02 -07:00
Jan Steemann b4e876e282 prefer explicit options over positional argument 2016-04-19 16:57:26 +02:00
Frank Celler e298ac9881 Merge branch 'devel' of github.com:arangodb/arangodb into FMH 2016-04-15 21:05:52 -07:00
Jan Steemann bbfaee3bfe allow specifying collection name prefixes for `_from` and `_to` in arangoimp: 2016-04-15 16:18:23 +02:00
Frank Celler 1b7c47c6bc added --console 2016-04-10 13:47:53 +02:00
Frank Celler af00417cf4 skeleton arangod 2016-03-25 15:14:43 +01:00