Shyft Migrator

Petabytes of data has been ingested in vendor specific format across many Enterprise Security Organizations to their Security Data Lake. Data stored in some of these Data Lakes, are in proprietary format which can only be consumed by vendors, like Splunk, ElasticSearch etc.

In general, compared to open text compressed format, Stored Data in proprietary format are:

• 3x more expensive to store.
• 3x more expensive to perform searches.
• 2x – 3x slower to perform searches.

Organizations need a solution to address the portability of this historically ingested data. They require a robust solution that scales and provides proper flexibility to allow formatting, transiting, and seamless use of data exactly as it is required by security and compliance.

Open text format data lakes and vendor agnostic data lakes are very important for any public or private organizations. It gives flexibility for various groups/teams in the organization to be able to access the data and perform required analytics with their respective domain knowledge. This in turn helps drive business initiatives.

Shyft is the product that can help organizations with large size data exports from “Splunk SmartStore”, “Splunk Cold/Frozen Buckets” or other Security Data Products like ElasticSearch. Exporting data would reduce cost of storage by at least up to 70 percent on an average. There is no reason for any organization to lock petabytes of useful data in vendor proprietary format.

Shyft is a unique solution built with an embedded python distributed framework to scale and perform the tasks. Architecture diagram illustrates the workflow process as shown below

In essence, Shyft allows you to unlock the value of your historical data by freeing it from stringent and heavy proprietary formats. This helps organizations slash costs and utilize data efficiently in any tools they desire.