The vision of Open Ingest is a unified self-service UI for users to create and manage data pipelines. The data sources for these data pipelines are from various sources; both internal and external to dx and Comcast.
Examples of the benefits and value that teams and users can receive using Open Ingest include:
- Improved speed to on-board and reliability of data used for the business. After the one-time onboarding and access are completed, dx Data Pipelines can be set up in a matter of minutes.
- Simplified and improved operational efficiencies using self-service and easy-to-manage dx Data Pipelines and workflows. Teams no longer have the overhead of setting up data pipeline infrastructure, which increases the speed to market for data producers.
- Integration with other components in the Data Foundry ecosystem, such as dx Storage and Compute frameworks. Also supports the integration through the dx Data Lake platform services that persist the Landing Zone data delivered by Open Ingest.
It consists of multiple data pipeline engines that can run the appropriate use case of data ingest through frameworks in the Cloud or On-Premise, depending on the source and location of the data. The intent is to support repeatable, configurable, template-driven jobs.
Who is it for?
Open Ingest is a valuable tool for teams and individuals who need to:
- Ingest data from various sources: Open Ingest supports both internal and external data sources, making it a versatile solution for a wide range of data pipelines.
- Accelerate data onboarding: With a unified self-service UI and pre-configured templates, Open Ingest can significantly reduce the time required to onboard new data sources.
- Improve operational efficiency: By automating data pipeline creation and management, Open Ingest helps teams streamline their workflows and reduce operational overhead.
- Leverage the Data Foundry ecosystem: Open Ingest integrates seamlessly with other components of the Data Foundry, such as dx Storage and Compute frameworks, providing a comprehensive solution for data management.
Whether you’re a data engineer, data analyst, or business user, Open Ingest can help you streamline your data pipeline processes and deliver valuable insights faster.
How to use it?
To get started, go to the Open Ingest Dashboard site on the dx Portal. Users can also access it directly from the dx Portal by clicking the PUBLISH DATA button in the Data Pipelines Tile.
in Philadelphia
