Big Data Intelligence for Security

What is big data? And how can my organization use it with Security?

There is an idea  within Theoretical Computer Science that involves when working with algorithms to classify down into two categories. The first being complexity theory and the later computability.  Complexity theory looks at several factors when calculating the total completion time of a given algorithm.  It takes into account, how many cycles that algorithm uses then looks at it from a hardware stance.  How much ram, processing power and system resources is available to compute the given problem.  More often than not, you will be limited by the system resources you have available to you, as you are limited to run your algorithm on one workstation. For this reason Big Data has awaken a new sleeping giant “Big Data”.

Big data changes the way we think about system resources.  Within a Hadoop environment it will allow for these resources intensive algorithms to be run in a mater of minutes / hours as compared to days and weeks on a single system. Depending on the build type of the hadoop system it could even reduce your query time down to seconds. It is this revolution of data processing that has captured the attention of the professional world at large. It is this attention and the ingenuity of individuals who are shaping the future of “Big Data”.

How can Security play a role within the Big Data space? 

Big data can offer an organization insight into problems through the use of a data science engineer.  There has always been a need to processes data, but it has never existed as it does today. As we produce more and more data (logs, netFlow data (network traffic), emails, IT assets, Vulnerability assessments, Threat Intelligence reports, application and system behaviors, structured data and un-structured data), we need a way to efficiently carve that data up.  You can see that for big data to be successful you will need to have a variety of data available within the big data container.

In this way it will be highly automated, use advance data aggregation, event correlation, statistical and heuristical analysis and monitoring in real time. The currently limitations of computing power is also limited by the skilled professionals.  As you anticipate a need for big data within your organization, you may want to research the cost of hiring a data scientist. Find a good data scientist that can be security conscience can increase the power of your deliver-ables.

What type of information an you expect to get from your Big Data? 

Again, this depends on the data sources available to you and keep in mind that this is from a security standpoint.  You should be aiming your deliver-ables from the data collected and also from company objectives, large risk factors and even audit findings. Examples of data you can find hidden in your terabytes of data can include: persistence threats, insider threats, ad-aware and Trojan activity, data leakage, phis-hing attacks, .  It is also important to make sure that your data streams live into your hadoop environment.  This will insure that you can create real time alerts.

What skills do you need in house to accomplish Big Data for Security? 

Here is a list of in-house skills you will need to be successful in your big data mining.

  • Multivariate statistical analysis
  • Data mining
  • Predictive modeling
  • Natural language modeling
  • Content analysis
  • Text analysis
  • Social networking analysis

This list doesn’t include the technical skills needed to support the hadoop environment.


It is important not to introduce a more complex and unyielding system into your security environment, but create your big data with several layers that will offset the complexity of hadoop and map reduce.  That being said, an investment into the big data for security is a huge task, which few are up to the task to accomplish.

I wait with anticipation for the future of Big Data for Security to unfold within the next few years. Now stating an opinion, I think that once the total cost of ownership comes down, this big data will help shape a more secure future from hackers and insider threats.  If with this next version of Big Data 2.0 we will see a level of automation once thought impossible.