ElasticDump

elasticsearch javascript node.js taskrabbit 
Fri Apr 22 2016 - Originally posted at tech.taskrabbit.com



Intro

At TaskRabbit, we use ElasticSearch for a number of things (which include search of course). In our development, we follow the normal pattern of having a few distinct environments which we use to build and test our code. The ‘acceptance’ environment is supposed to be a mirror of production, including having a copy of its data. However, we could not find a good tool to help us copy our Elastic Search indices… so we made one!

taskrabbit/elasticsearch-dump

Use

elasticdump works by sending an input to an output. Both can be either an elasticsearch URL or a File.

  • Elasticsearch:
    • format: {proticol}://{host}:{port}/{index}
    • example: http://127.0.0.1:9200/my_index
  • File:
    • format: {FilePath}
    • example: /Users/evantahler/Desktop/dump.json

You can then do things like:

Copy an index from production to staging:

1elasticdump --input=http://production.es.com:9200/my_index --output=http://staging.es.com:9200/my_index

Backup an index to a file:

1elasticdump --input=http://production.es.com:9200/my_index --output=/var/dat/es.json

Options

  • — input (required) (see above)
  • — output (required) (see above)
  • — limit how many objects to move in bulk per operation (default: 100)
  • — debug display the elasticsearch commands being used (default: false)
  • — delete delete documents one-by-one from the input as they are moved (default: false)

Notes

  • elasticdump (and elasticsearch in general) will create indices if they don’t exist upon import
  • we are using the put method to write objects. This means new objects will be created and old objects with the same ID will be updated
  • the file transport will overwrite any existing files
  • If you need basic http auth, you can use it like this: — input=http://name:password@production.es.com:9200/my_index

Inspired by https://github.com/crate/elasticsearch-inout-plugin and https://github.com/jprante/elasticsearch-knapsack

You can download elasticdump from NPM or GitHub

Originally published at 06 Jan 2014

Hi, I'm Evan

I write about Technology, Software, and Startups. I use my Product Management, Software Engineering, and Leadership skills to build teams that create world-class digital products.

Get in touch