Storage admins should continuously monitor systems to identify and act upon unusual behavior. Anomaly detection can proactively address issues before they become serious problems.
Anomaly detection plays an increasingly important role in data and storage management, as admins seek to improve security of systems.
Managing data and storage is more complex because of distributed and multiplatform workloads. At the same time, data volumes are growing at staggering rates. Much of that data is unstructured. On top of it all, cyber attacks are more aggressive, sophisticated and targeted.
In response to these developments, more vendors incorporate storage anomaly detection capabilities into their products, often including them as part of a larger management platform. By using these products, IT teams can take a more proactive approach to managing their storage-related hardware and software, and ensure that their data remains viable and secure.
What is anomaly detection and how does it work?
Anomaly detection refers to the process of identifying items, events, patterns, data points, observations or changes that differ significantly from the expected behavior. It works under the assumption that anomalies are rare events that operate outside what is considered common.
Storage anomaly detection can help organizations identify and react to unusual behavior much faster than with traditional monitoring alone.
Anomalies often indicate some type of problem, such as malfunctioning equipment, faulty software or compromised data. For example, unusual withdrawals from a bank account might point to the hack of a supporting storage system. That said, an anomaly does not always mean there's a problem. It might be an indicator of a positive trend, such as an unexpected surge in online sales. In such cases, an anomaly could represent a business opportunity rather than a potential problem.
Anomalies are often categorized as one of three types:
Point or global anomaly. An anomaly that stands out in some significant way from the expected pattern or behavior, such as a brief spike in I/O activity on a disk array with no discernable cause.
Contextual anomaly. An anomaly that has meaning only within the context of its environment or circumstances, such as a sudden demand on SAN at a time of day when usage should be at its lowest.
Collective anomaly. An anomaly whose meaning is derived from multiple data points that collectively indicate an unusual pattern. For example, multiple drives that fail in a discernable pattern could represent a collective anomaly.
How anomaly detection applies to storage
IT teams often track information such as static alert thresholds or key performance indicators. This approach is often not enough, however, because admins can miss unusual events or patterns due to the overwhelming amount of information they need to process. As a result, they might fail to act quickly enough to address software or hardware issues or to fend off a cyber attack.
Storage anomaly detection can help organizations identify and react to unusual behavior much faster than with traditional monitoring alone. This practice is necessary to ensure optimal data and storage operations, and to address potential security threats as quickly as possible.
By employing real-time anomaly detection, IT teams can strengthen their security posture and minimize operational and business risks. This approach can even lead to better customer service or help organizations identify patterns and trends that could represent potential business opportunities.
Anomaly detection can play a key role in reducing the disruptive effects of storage-related hardware and software issues. It can help mitigate the impact of cyberattacks or prevent them altogether. In this way, the data is more secure and reliably accessible, and the storage systems can operate at peak efficiency.
Storage anomaly detection makes it possible for IT teams to identify unusual events and circumstances that represent a departure from normal storage and data operations. It might be easy to identify a failed disk, for instance, but it's not nearly as straightforward to detect subtle changes in performance over time. With anomaly detection, IT can discover these changes before full disk failure occurs.
Anomaly detection can help evaluate system logs to better understand service disruptions. It can play an important role in storage and data security by monitoring traffic, evaluating access patterns and looking for other types of abnormal behavior, whether related to fraud or a potential cyber attack.
Storage and data security go hand in hand with network security, particularly as it applies to NAS or a SAN. For example, a team might deploy an intrusion detection system that monitors incoming and outgoing network traffic in real time to identify anomalies that represent potential security risks.
Products, features on the market
Vendors have added anomaly detection features to their platforms as the technology continues to grow more important to storage and data management. Because of the size and diversity of the data, most storage anomaly detection approaches incorporate machine learning (ML) algorithms that can handle various types and amounts of data. They're also typically flexible enough to accommodate shifting data patterns and workloads.
Here is a sampling of vendors that provide anomaly detection:
Dell CloudIQ. CloudIQ is a cloud-based AIOps service that combines monitoring, machine learning and predictive analytics into a single application for tracking Dell storage systems and other products. CloudIQ uses ML and predictive analytics to identify anomalies in its monitored systems. It compares performance metrics with historical values to find deviations outside of normal ranges.
Hitachi Ops Center Analyzer. Ops Center Analyzer is an AI-driven management suite for Hitachi storage and other infrastructure. It uses ML to correlate and analyze telemetry and operational data, providing insights into a wide range of metrics. The product also identifies at-risk resources, diagnoses problems and helps to resolve unmet service levels.
HPE InfoSight. InfoSight is a cloud-based service for managing HPE storage systems and other HPE infrastructure. It uses AI, ML and predictive analytics to continuously analyze the data it collects from millions of sensors. The service can proactively detect anomalous behavior such as high latency.
NetApp OnCommand Insight. OnCommand Insight is a management software platform that can track metrics about an organization's entire infrastructure. The product's anomaly detection features use ML to uncover performance changes in processing patterns and behaviors in areas such as latency, utilization and IOPS.
Microsoft Windows Server 2019 and 2022. Both server products now include System Insights, a predictive analytics feature that analyzes Windows system data to provide understanding of the server's function. System Insights includes disk anomaly detection, which identifies when the server's disks are behaving unusually.
Other vendors that include anomaly detection include AWS, with services such as SageMaker, Kinesis and Quick Start, and Nutanix Prism, with its ML predictive monitoring features.
Robert Sheldon is a technical consultant and freelance technology writer. He has written numerous books, articles and training materials related to Windows, databases, business intelligence and other areas of technology.
