ABSTRACT
This chapter outlines the challenges and tools for handling big data. It discusses the security challenges that arise in the process of using those tools as well as some of security applications that are enabled with their use. Big data is characterized by the three Vs: volume, velocity and variety. Volume refers to the size of the data. Velocity refers to the rate at which it is being generated and stored. Variety refers to the heterogeneity of the data. Data ingestion is the process of adding raw data to the system, which was always present, but which is far more complex with big data. Visualizing data is one of the useful ways to spot trends and make sense of a large number of data points. Composed of Logstash for data collection, Elasticsearch for indexing data, and Kibana for visualization, the Elastic Stack can be used with big data systems to visually interface with the results of calculations or raw metrics.
