Observability Logging is a logging platform that is broken up into two key functional areas:
- Elasticsearch Log Storage and Tools
- Ingest Pipeline
Observability Logging was designed to help technology operations teams focus on leveraging their operational data rather than spending time supporting additional applications to capture and store it at scale.
We provide two versions of Elasticsearch based on a team’s logging requirements. In addition, we strongly recommend teams use Vector for their log shipping needs.
While we would like to support every feature that is provided by Elasticsearch, some features require testing in our environment before they are approved for use in our community.
For more information on the platform that supports these features, check out our page on Elasticsearch logging powered by DaaS
Some Elasticsearch functionality is available out of the box while other features are enabled by client request. Please see the out of the box column to understand what is provided immediately after deployment.
How to use it?
The DevX-Accelerate program strongly believes that Vector, the open source log shipping service, is the tool Comcast should be using in this space. Vector supports parsing logs at the edge which will reduce overall Elasticsearch cost.
Observability clients are responsible for running their own Vector instances and making the necessary configuration changes to ship and parse logs to the ingest platform. We provide documentation for Ansible playbooks and a number of deployment patterns for our customers. In addition, we have training and knowledgeable individuals ready to help.
| Feature | Description |
|---|---|
| Vector for log Shipping and Parsing | Vector is an open source observability pipeline tool that specializes in accepting observability data sources, transforming those sources, and sending the data on to a storage solution. |
| Elastic Common Schema (ECS) / Comcast Schema Extensions (CSE) | In addition to parsing logs, the DevX-Accelerate program supports an initiative to help standardize Observability data. We require that new teams use these schemas prior to onboarding. |
Standard Logging
Our most used Elasticsearch logging solution that has no licensing cost. This logging solution is based on the Elasticsearch Free and Open License.
| Feature | Description | Out of the Box |
|---|---|---|
| Logging Storage | A standalone Elasticsearch cluster will be provided for customers for ingestion exceeding 200 GB of logs a day. Shared clusters are used for clients that don’t require as much data. Data can be tuned to Hot, Warm, and Cold storage nodes based on usage. | Yes |
| Logging Data Retention | We provide 14 days of logging data retention. We may provide up to 30 days with the appropriate business case. | Yes |
| Kibana UI | Kibana is the Elasticsearch’s frontend that is deployed along with your data storage cluster. | Yes |
| Elasticsearch Discover | Discover is a Kibana UI component that is use for searching and triage with log data. | Yes |
| Elasticsearch Visualize and Dashboards | Visualize is a Kibana UI component that can be used to build graphs and pictures from your logging data. Visualizations can then be added to dashboards. | Yes |
| Canvas | Canvas is a Kibana UI component for executive presentations. | Post Deployment Request |
| Elasticsearch Reporting | There are may types of reports that can be generated from Kibana in Elasticsearch. Click the link to see what is available. | Yes |
| Rollup Jobs and Transforms | After data is ingested into Elasticsearch, it can be rolled/aggregated into a new index or transformed into a new view. Note: This functionality is experimental in elasticsearch and may be removed in the future. In addition we strongly believe that teams should use Metrics for these functions. If you are reliant on this feature we suggest you prioritize using our metrics product. | Yes |
Raise an onboarding request.
Premium Logging
Everything available in our standard logging solution is also provided with our Premium offering. The Premium offering has Elasticsearch Platinum license applied.
If you do required the below features, please let us know during onboarding. We require business justification to apply a license to your cluster. During onboarding, we will evaluate your use case. If it’s covered we may initiate financial discussions depending on which organization your team is in.
| Feature | Description | Out of the Box |
|---|---|---|
| Alerting: Alerts and Actions | Alerting from log data is not ideal. We only provide alerting for customers migrating from Splunk. We strongly suggest that customers use our metrics solution for their alerting needs. | Post Deployment Request |
Alerting: Watchers Deprecated | Watchers is elasticsearch legacy alerting feature. Elastic is no longer investing in watcher development. We will allow the use of watchers if your request for alerting meets the following criteria. First, your team’s log data has previously been on Splunk. Second, Elastic’s alerts and actions does not provide the functionality that you need at this time. In the future, we will roll off of watchers completely. | Post Deployment Request |
| SSO and RBAC | SSO and RBAC are provided for licensed customers only. With SSO and RBAC, we integrate Elastic with Comcast Azure AD and help setup the necessary groups to support your users. | Post Deployment Request |
| Cross Cluster Search | Cross cluster search enables our clients to search any approved Elasticsearch cluster in our ecosystem. When you request logging service with us, please let us know if there are other teams in the company that rely on your data, or if you rely on other teams data. If that is the case, we will flag your deployment for our federated logging environment | Post Deployment Request |
| Machine Learning | Machine Learning helps you identify insights and anomalies from your Elasticsearch data. Machine learning is an Observability + functionality that requires additional licenses to be applied on your Elasticsearch deployment. | Post Deployment Request |
Raise an onboarding request.
Observability Ingest Platform
The Observability Ingest Platform is a key component of the DevX-Accelerate program that enables the collection and processing of observability data from various sources. It acts as the foundation for our logging, metrics, and distributed tracing solutions. The platform is designed to handle large volumes of data, ensure data integrity, and provide scalable and reliable data ingestion.
Ingest Key Features
| Feature | Description |
|---|---|
| Data Ingestion | The Observability Ingest Platform supports the ingestion of logs, metrics, and distributed tracing data from a wide range of sources, including applications, infrastructure components, and external systems. It provides flexible and customizable data ingestion pipelines to handle different data formats and protocols. |
| Data Processing | Once data is ingested, the platform performs various processing tasks such as parsing, filtering, transformation, and enrichment to make the data usable for analysis and visualization. It supports the application of data schemas and standardization techniques, like the Elastic Common Schema (ECS), to ensure consistency across different data types. |
| Scalability and Resilience | The platform is designed to handle high volumes of data and is horizontally scalable to accommodate increasing data loads. It leverages distributed computing technologies and resilient architectures to ensure data ingestion and processing capabilities even under heavy workloads or failure scenarios. |
| Data Quality and Validation | The platform facilitates data conformance to a common schema, such as ECS for logging. By enforcing a standardized schema, it helps ensure consistency and coherence in the data. However, it does not handle the detection of data anomalies or missing data points. Clients are responsible for validating the quality and completeness of the data they send to the platform. |
| Metadata Management | The platform provides robust metadata management capabilities to organize and categorize observability data. It allows for the efficient searching, filtering, and tagging of data, enabling users to easily navigate and locate specific data sets based on different criteria. |
| Integration with Observability Tools | The Observability Ingest Platform seamlessly integrates with various observability tools, including logging solutions (such as Elasticsearch and Kibana) and metrics platforms (such as Prometheus and Grafana). It enables data to be ingested into these tools for further analysis, visualization, and monitoring. |
The Observability Ingest Platform forms the backbone of our observability solutions, providing a reliable and scalable infrastructure for handling and processing observability data. It ensures that data from different sources can be efficiently collected, transformed, and made available for analysis, enabling teams to gain valuable insights into the behavior and performance of their applications and infrastructure.
Please note that the Distributed Tracing solution is currently under development and will be added to the product offerings in the near future.
in Philadelphia
