INGEST: Publish data to the dx Data Foundry leveraging: Open Ingest: a self-service tool to create Data Pipelines from multiple sources, including files, relational databases, object storage, or streaming data (via Headwaters).
Additionally, data can be added to the dx Data Foundry as a Custom Data Producer . This method allows processed data to be loaded directly to the platform in its final format.
For more information, view the Open Ingest Overview and the Open Ingest User Guide.
STREAM: Headwaters, the data streaming platform, is API-driven and supports real-time applications data streams for both Producers and Consumers.
For more information, view the Headwaters User Guide.
COMPUTE: Leverage Spark-based processing with our On-Prem Compute platform (powered by Kubernetes) or Cloud Compute platform (powered by Databricks). These notebook environments encourage collaboration with the ability to access enterprise compute resources at-scale.
STORE: The dx Data Lake is a centralized and governed environment that consists of two components: our Cloud Data Lake (powered by AWS S3) and On-Prem Data Lake (powered by MinIO).
Additionally, the dx Data Foundry includes an On-Prem Data Warehouse ( the National Data Warehouse (NDW) ) with a standardized data architecture to support data consumers.
SEARCH: Search the Metadata Catalog for information about all data in the dx Data Foundry. Standard metadata attributes included are object names, object location, column names, and data lineage.
For more information and tips and tricks, check out the Get Started with Metadata Search Guide.
DISCOVER: Data Products are fully supported, source-of-truth datasets developed using the Enterprise Data Product Model. The Data Products are designed to support multiple analytical use cases across data domains (e.g., Customer, Equipment, Business Services, Product Usage, and more).
ANALYZE: Query Fabric allows users to access all data across the dx Data Foundry, regardless of its physical location and/or technology. Consumers typically analyze the data through SQL clients (e.g., DBeaver ) or visualization software ( Tableau ).
For more information, view the Query Fabric Overview and the Query Fabric User Guide .
in Philadelphia
