Apache Spot is a community-driven cybersecurity project, built from the ground up, to bring advanced analytics to all IT Telemetry data on an open, scalable platform. It is an open source software for leveraging insights from flow and packet analysis. Spot expedites threat detection, investigation, and remediation via machine learning and consolidates all enterprise security data into a comprehensive IT telemetry hub based on open data models. Spot’s scalability and machine learning capabilities support an ecosystem of ML-based applications that can run simultaneously on a single, shared, enriched data set to provide organizations with maximum analytic flexibility.
Apache Spot helps enterprises and service providers gain insight on their computed environments through transparency of service delivery and identification of potential security threats or attacks happening among resources operating at cloud scale. While current threat intelligence tools help, identifying unknown threats and attacks remains a challenge. Spot harnesses a diverse community of expertise from Centrify, Cloudera, Cybraics, Endgame, Intel, Jask, Streamsets, and Webroot. Apache Spot provides tools to accelerate companies’ ability to expose suspicious connections and previously unseen attacks using flow and packet analysis technologies.
Apache Spot uses the best tools just to provide the best functionality preventing attacks, identifying unknown threats, and providing cybersecurity. Perimeter Flows, DNS, Proxy and Internal Flows provide remarkable advantages creating fast and efficient data analysis.
Apache Spot is functional after just one day and just keeps improving through feedback and machine learning.
Apache Spot uses machine learning as a filter for separating bad traffic from benign and to characterize the unique behavior of network traffic. A proven process, of context enrichment, noise filtering, whitelisting and heuristics, is also applied to network data to produce a shortlist of most likely security threats.
The systems uses opens source’s optimized decoders for binary flow and pack data, to load into HDFS and data structures inside Hadoop. The decoded data is stored in multiple formats so it is available for searching, used by machine learning, transfer to law enforcement, or inputs to other systems.
The system uses a combination of Apache Spark to run scalable machine learning algorithms. The machine learning component works not only as a filter for separating bad traffic from benign, but also as a way to characterize the unique behavior of network traffic in an organization.
In addition to machine learning, a proven process of context enrichment, noise filtering, whitelisting, and heuristics are applied to network data to produce a short list of the most likely patterns, which may be security threats.
Apache Spot is capable of performing deep-packet inspection of DNS traffic to build a profile of probable and improbable DNS payloads. After visualizing, normalizing, and conducting pattern searches, the analyst has a shortlist of the most likely threats present in DNS traffic.
Given an IP address, Apache Spot gathers all the characteristics about the communication associated with it – the “social network” of that IP address. Then Apache Spot builds a timeline of the conversations that originated with that IP.
Apache Spot uses advanced machine learning to build a model of the machines on the network and their communication patterns. The connections between the machines that are the lowest probability are then visualized, filtered for noise, and searched for known patterns. The result is the most likely threat patterns in the data, a few hundred flows picked from billions.
After an analyst has investigated a threat, the need still exists to communicate the event up and across the organization. A “dashboard” gives quick answers to the questions you already know to ask. What the analyst requires is a “storyboard,” something that tells who, what, where, and how of the story in words and interactive visualizations.
Apache Spot delivers common Open Data Models for network, endpoint and user, providing a standard format of enriched event data that makes it easier to integrate cross application data to gain complete enterprise visibility and develop net new analytic functionality. Spot’s Open Data Models helps organizations quickly share new analytics with one another as new threats are discovered.
Apache Spot's Open Data Models help organizations to quickly share new analytics among them as new threats are discovered. And, with Hadoop, organizations able to run these analytics against comprehensive historic data sets, helping organizations identify past threats that have slipped through the cracks. With this capability, Spot aims to give security professionals the ability to collaborate like cybercriminals do.
The primary use case initially supported by Spot includes Network Traffic Analysis for network flows (Netflow, sflow, etc.), DNS and Proxy. The Spot open data model strategy aims to extend Spot capabilities to support a broader set of cybersecurity use cases.
Spot accelerates the development of cybersecurity applications by providing a cybersecurity analytics framework. This means more solutions can be created faster. This is because Spot allows organizations to focus developing the analytics and visualizations for applications that discover cybercrime rather than spending time building systems to ingest, integrate, store, and process myriad volumes or varieties of security data.
Join the Apache Spot community and collaborate with us using a common framework.
Spot provides the capability for a central data store that houses ALL the data needed to facilitate an investigation, returning investigative query results in seconds and minutes (vs. hours and days), which effectively reduces incident MTTR and minimizes the adverse impacts of a breach.
Spot improves the efficacy of threat hunting by providing the analytic flexibility to perform ad-hoc searches and queries over vast amounts of data, as well as applying ad-hoc algorithms to detect the needle in the haystack.
Offloading data from legacy cybersecurity systems (e.g., SIEMs) to Spot delivers immediate economic value, because of the cost of data storage and processing with Hadoop. It also opens up future value as organizations deploy one of the many analytics use cases on their newly formed security data hub.
Identify the needle in the haystack with patterns that provide insight into potential threats.
Apache Spot is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
The contents of this website are © 2020 Apache Software Foundation under the terms of the Apache License v2. Apache Spot and its logo are trademarks of the Apache Software Foundation.
