Protect your enterprise with AI-powered anomaly detection

Isma Rizvi

Product Manager

March 30, 2023

Reading Time: 3 minutes

Reading Time: 3 minutes

Spot by NetApp is excited to announce our latest innovation for Spot Security to protect your cloud environment: a machine learning model that detects anomalous events.

Event logs: Both a blessing and a curse

Event Logs, such as CloudTrail or Activity Logs, are services from cloud providers that record the activity of your cloud accounts via the Web Console or via scripts leveraging APIs. These logs monitor the actions taken by users, roles, and cloud services within your accounts and are useful for identifying potential security threats. However, despite their benefits, event logs alone may not be adequate to track anomalous events.

One major challenge with CloudTrail and Activity Logs is that they produce an extremely large volume of data. With thousands of API calls being made every minute from both users and applications, it is very time-consuming and resource-intensive (if not totally impossible) to manually review these logs and identify any suspicious activity. It’s like trying to find a needle in a haystack.

Additionally, event logs can be complex and difficult to understand. Atomic/standalone events are hardly useful to understand the scope of a user action, especially for those who are not familiar with the technical details of cloud providers. This can make it burdensome for non-technical users to spot potential threat indicators, even if they are able to inspect and analyze the contents of these logs.

Moreover, event logs contain only the information related to the calls made to the cloud services. They might not provide a complete picture of the infrastructure and resources. For example, event logs don’t record the internal communications between different resources or the changes made to the resources. Nor do they record the actions performed by users who have access to the resources but did not make the calls through the services or Programmatic API access (e.g., Boto3).

Due to the scale, it is not a problem that can be realistically fixed by adding more security analysts. This is where automation and tooling come into play. Automation can help cut through the noise and assist security analysts in identifying anomalies and strengthening their organization’s security posture.

AI: Making sense of event log complexity

Deep learning techniques such as neural networks are particularly well suited for processing large volumes of data. These algorithms can identify patterns in the data without the need for time-consuming manual feature engineering. They can also handle complex and unstructured data, such as Activity Logs or CloudTrail events.

Our machine learning model for detecting anomalous events is based on the autoencoder-decoder architecture. In essence, this model is trained to reconstruct the input data. The encoder compresses the input into a lower-dimensional representation, also known as the latent space, and then the decoder uses this representation to reconstruct the original input.

In Spot Security, we have trained our anomaly detection model on a large dataset of more than 50 million normal events. The encoder component is trained to learn a compressed representation of the normal event sequences (i.e., dubbed context), while the decoder component is trained to reconstruct the next event (given this context) from the compressed representation.

This deep learning model analyzes new events and identifies any patterns that deviate from the normal behavior that learned during the training phase. The input events are compressed into the latent space and then reconstructed by the decoder component. If the reconstructed event deviates from the actual event, it is flagged as anomalous and reported to the user as a potential security threat, thus reducing the time for an analyst to identify and process this event.

We also implemented a continuous learning mechanism in our model, which allows the model to adapt to new patterns and threats as they arise and prevents model decay. As the model is continuously exposed to new CloudTrail events or Activity Logs, it updates its understanding of normal behaviors and can detect new anomalies.

The biggest challenge in security, not just for AI models, is false positives rates. When security analysts’ time is scarce, it is important that they focus on “real” alerts. We tested our model’s performance, and we can fine-tune our prediction of the next event with 99.561% accuracy.