1
0
Fork 0
Commit Graph

48 Commits

Author SHA1 Message Date
Jan c1a0274ad0 fixed issue #10183: Arangoimport imports on _system when you try to create a new database. (#10187) 2019-10-09 14:54:46 +03:00
Matthew Von-Maszewski 0832c6d546 Feature 3.5: add gzip and encryption support to arangoexport and arangoimport (#9579)
* port gzip/encryption for arangoexport/import from devel branch

* revert Dump default use of gzip from true to false.

* Stylefix in changelog.
2019-07-27 00:06:33 +03:00
Matthew Von-Maszewski 4441a806de Bugfix 3.5: Import would not report communication errors (#9500)
* arangoimport was neither reporting nor stopping on communications errors.  Add reporting of impacted lines for CSV imports.

* correct warning from Windows build about size_t to int conversion
2019-07-18 16:40:01 +03:00
Jan Christoph Uhde 677a79026c Foxx Security (#8845) 2019-04-25 09:56:29 +02:00
Jan Christoph Uhde c3f7961b88 apply unique log ids (#8561) 2019-03-25 20:26:51 +01:00
Jan a841d472be
emit a warning when connecting with the client tools to a DBSERVER (#8397) 2019-03-13 13:08:41 +01:00
Frank Celler ac9f375fb5 big reformat 2018-12-26 00:54:03 +01:00
Vasiliy 6fd541d110 issue 427.5: use ApplicationServer reference instead of pointer (#6145)
* issue 427.5: use ApplicationServer reference instead of pointer

* address MSVC build failure
2018-08-15 12:16:02 +03:00
Jan b278d6874a
allow master & slave to work in parallel for RocksDB WAL tailing (#6059) 2018-08-03 13:37:53 +02:00
Michael Hackstein 7a95c5e675
Feature/feature phases (#5272)
* Added feature phases

* BasicsPhase and DatabasePhase to the required files. Server now has Feature circles and does not boot. Will be sorted out later on.

* Added ClusterPhase to features

* Added V8Phase to the required features

* Added AQLPhase to the affected features

* Added ServerPhase to Features

* Added FoxxPhase to the relevant features

* Added AgencyPhase to the relevant features

* Moved registration from local variable SYS_SYSTEM_REPLICATION_FACTOR from cluster to V8 as their ordering is now vice versa

* Moved Bootstrap feature into FoxxPhase. It could be moved to ServerPhase easily if the FoxxQueue dependency would be removed

* Final movement of Startup Phases. Now solved all circles.

* Removed merge conflict

* Moved ReplicationTimeout into cluster phase and fixed cross-phase requirements

* Added greetings phase. This phase separates the Basics Phase and is the first to be run. Includes Logger and Hello/Goodbye

* Added the GreetingsPhase in the corresponding features. Now all BasicsPhase features start after greetings Phase. There is some issue in this branch which prevents the Agency from Gossipping right now. Will be fixed next

* Moved creation of the Agent into the prepare phase of the feature. THereby it is guaranteed that agents at least exists before the GeneralServer is activating endpoints

* Recovery needs to be started after the ServerID

* Moved log output of FeaturePhases to DEBUG instead of ERROR.

* Added feature phases for clients

* ClusterFeature now does not directly require AgencyFeature any more

* Added requirement of TravEngineRegistryFeature in AQL feature. Otherwise shutdown may be undefined

* The ApplicationServer can now handout the list of ordered features. Used for testing purposes

* Fixed IResearchVew Tests Setup to honor new feature ordering

* Fixed IResearchViewDBServer Tests Setup to honor new feature ordering

* Started fixing IResearchView Coordinator tests with startup ordering. Not finished yet

* Added startup phases to ViewCoordinator test

* Disabled expected logoutput in ClusterRepairsTest

* Fixed indention in test code

* LinkCoordinator now honors startup ordering

* Link meta now honors startup rdering

* Supress expected cluster logs in ViewTest

* Removed '#' accidentially added.
2018-07-16 14:09:36 +02:00
Jan 7351423ad5
cleanup import messages, reuse StringBuffer for JSON import (#5569) 2018-06-11 09:47:26 +02:00
Matthew Von-Maszewski 3678dc7b68 latency histogram option for arangoimport (#5552)
* basic histogram thread using non-arangod threading ... soon to change.
* Add --latency flag.  start histogram thread if set.
* convert quick_hist to use arangodb thread and condition objects
* move histogram object to be within ImportStatistics structure.
* update the thread shutdown to 1. work and 2. post last data
* add text that values are in microseconds to command line help.
* CHANGELOG note for new import latency option
* quick_hist.h rename to QuickHistogram.h
* update include file name to QuickHistogram.h
* update variable names to match coding standard
* a couple more code standard changes.  try/catch around push_back
2018-06-08 11:17:25 +02:00
Matthew Von-Maszewski 9ff6a41236 Add pacing logic to arangoimp (#5238)
* initial check-in of working auto block size tuning for import.  Needs clean-up.
* partial fix in this branch to enable testing, awaiting better fix in different branch.
* move pacing code from findIdleSender to AutoTuneThread object.
* move pacing code from findIdleSender to AutoTuneThread object.  clean up dead code.  add comments about algorithm.
* edits to algorithm discussion
* correct logging level.  add CHANGELOG entry.
* update comment concerning proper usage of shutdown()
* initialize new static member MaxBatchSize.  Correct sendCsvBuffer() call to addPeriodByteCount() since _outputBuffer drained by sendData().
* remove redundant semicolons.  add new static member MaxBatchSize.
* move MaxBatchSize to ImportHelper so both ImportFeature object and AutoTuneThread object can access it.
* minor updates to comments and statics
* add pacing discussion to import manual page
* minor edit
2018-05-02 22:34:04 +02:00
Dan Larkin-York 38f162e344 Parallelize arangodump (#4356) 2018-03-26 15:55:14 +02:00
Jan Christoph Uhde bddadda0ee add 'create database' to 'import feature' (#4278) 2018-01-15 15:15:31 +01:00
Jan 6040b4ae9f
add ignore-missing option to arangoimport (#4129) 2017-12-22 11:29:10 +01:00
Simon Grätzer 885ea3dd5f Remove attribute option for arangoimp (#3141)
* Adding a remove attribute option to arangoimp (only for csv)

* Adding documentation

* Fixed log statement

* Update CHANGELOG
2017-08-30 10:45:02 +02:00
Simon Grätzer 2bc6a71d48 Fixing error reporting in arangoimp (#3153) 2017-08-28 15:04:45 +02:00
Simon Grätzer f50a36181d Various authentication related issues (#2824) 2017-07-18 22:22:09 +02:00
jsteemann 2275fc8174 better automatic file-type detection for arangoimp 2017-06-08 14:55:21 +02:00
jsteemann 2cfd7460d7 fix output 2017-05-31 11:38:20 +02:00
Simon Grätzer 0bf4467457 small fixes 2017-05-24 19:21:23 +02:00
Simon Grätzer 2f2d07ab9a Multihreaded import 2017-05-24 18:37:45 +02:00
jsteemann 53fa66e60d move parts from ProgramOptions from headers into cpp files 2017-05-23 10:30:36 +02:00
jsteemann aab138a4e6 added input type jsonl 2017-03-06 17:39:44 +01:00
jsteemann 6478c6d043 updated CHANGELOG 2017-03-03 15:03:52 +01:00
jsteemann db9099f0f9 Merge branch 'devel' of https://github.com/arangodb/arangodb into engine-api 2017-03-03 12:07:50 +01:00
jsteemann 7c0c84c21d added `--translate` option for arangoimp to translate attribute names from the input files to attriubte names expected by ArangoDB
The `--translate` option can be specified multiple times (once per translation
to be executed). The following example renames the "id" column from the input
file to "_key", and the "from" column to "_from", and the "to" column to "_to":

    arangoimp --type csv --file data.csv --translate "id=_key" --translate "from=_from" --translate "to=_to"

`--translate` works for CSV and TSV inputs only.
2017-03-03 11:35:16 +01:00
jsteemann 6e4e29943d added 'auto' file type 2017-03-02 23:35:52 +01:00
jsteemann d024a6d00a remove logging for non-topics 2017-02-10 09:32:50 +01:00
Jan Steemann 4b5249dc5e issue #2022: double max allowed request body size, cap --batch-size value in arangoimp 2016-08-29 09:34:23 +02:00
jsteemann 85cb227a4f fixed wrong default values for _sslOptions 2016-08-23 14:34:05 +02:00
Jan Steemann 7b62d71c63 added --skip-lines for import 2016-07-04 11:18:15 +02:00
Jan Steemann 33b40e3bf6 fix default separator 2016-06-15 18:24:10 +02:00
Jan Steemann 1726c53805 fixed some import edge cases 2016-06-15 18:16:49 +02:00
Jan Steemann 924bc0143d properly handle \t 2016-06-15 17:45:27 +02:00
Frank Celler 54300ed30c moved Console/ClientFeature to arangosh 2016-06-01 09:27:29 +02:00
jsteemann d62143ce83 make boolean command-line parameters optional 2016-04-27 11:37:42 +02:00
jsteemann 348e1e3397 client tools simplifications and fixes 2016-04-25 22:28:43 +02:00
jsteemann dcc5825f93 cleaned up features a bit 2016-04-24 14:29:29 +02:00
Frank Celler d6e8046992 updated cleanupCFiles 2016-04-23 16:34:57 -07:00
Frank Celler 3bb24146a6 Merge branch 'devel' of github.com:arangodb/arangodb into FMH 2016-04-19 19:18:02 -07:00
Jan Steemann b4e876e282 prefer explicit options over positional argument 2016-04-19 16:57:26 +02:00
Frank Celler e298ac9881 Merge branch 'devel' of github.com:arangodb/arangodb into FMH 2016-04-15 21:05:52 -07:00
Jan Steemann bbfaee3bfe allow specifying collection name prefixes for `_from` and `_to` in arangoimp: 2016-04-15 16:18:23 +02:00
Frank Celler 1b7c47c6bc added --console 2016-04-10 13:47:53 +02:00
Frank Celler af00417cf4 skeleton arangod 2016-03-25 15:14:43 +01:00
Frank Celler 53749a8e6d renamed feature 2016-03-15 22:15:25 +01:00