What's the solution? Cue the rise of artificial intelligence, or at least AI that can work in tandem with human analysts to spot digital clues that could be signs of trouble.
A research team from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and machine-learning startup PatternEx have developed anartificial intelligence platform called AI2 -- or AI "squared" -- that can predict cyber attacks 85 percent of the time, working together with input from human analysts. This is about three times better than benchmarks set by past systems, reducing the number of false positive results by a factor of five, the group said in a press release.
This system was tested on 3.6 billion pieces of data, or "log lines," that were produced by millions of users over a three-month period. AI2 sifts through all the data and then clusters them into patterns through unsupervised, machine-learning. Suspicious patterns of activity are sent over to human analysts who confirm whether or not these are actual attacks or false-positives. The AI system then takes this information and includes it in models to retrieve even more accurate results for the next data set -- so it gets better and better as time goes on.
Development of the system began two years ago, when PatternEx was founded. CSAIL research scientist Kalyan Veeramachaneni developed AI2 with Ignacio Arnaldo, a chief data scientist at PatternEx and a former CSAIL postdoc.
The goal was to figure out how to bring artificial intelligence technology to the infotech space, Veeramachaneni told CBS News.
"We looked at a couple of machine-learning solutions, and basically would go to the data and tried to identify some structure in that data. You are trying to find outliers and the problem was there were number of outliers that we were trying to show the analysts -- there were just too many of them," Veeramachaneni said. "Even if they are outliers, you know, they aren't necessarily attacks. We realized, finding the actual attacks involved a mix of supervised and unsupervised machine-learning. We saw that's what worked, and that's what was missing in the industry. We decided that we should start building such a system -- machine-learning that also involved human input."
If this collaboration between man and machine is so much effective at defending against cyber attacks, why was it missing from the industry? Veeramachaneni said that until very recently, artificial intelligence systems were just not advanced enough for this kind of prediction accuracy.
"One of the primary reasons this wasn't around was that now we have the storage and the infrastructure processing technologies. We have all of this big data processing now. The second thing was that we also now have the ability to get human input at the scale that was never imaginable before," he stressed. "You have seamless productivity with human input now. I mean the feedback you get from cellphones now, you didn't have five years ago. The third biggest piece is that machine-learning has really come to the forefront. That innovation has really jumped. All of those things came together in the last five years."
Veeramachaneni presented a paper about AI2 during the Institute of Electrical and Electronics Engineers (IEEE) International Conference on Big Data Security last week in New York.
He said that, so far, the response from companies has been positive.
"Security has become a very very important issue for everyone. For every company. More of our data -- everything, really -- is online. The need for this kind of system has become even greater."
Where does he see artificial intelligence going five years from now?
"I think, like our system in that it is augmenting a process, AI will be about augmentation. It will be about making processes more efficient. They are going to make things more efficient so people can move on to more interesting things," he said. "If you ask me, I would love to see AI being used to tackle problems that have more of a survival impact. How can AI move in directions that address problems that have direct impacts on society."