Headwaters is a Streaming Data Platform that fosters real-time data availability and facilitates data exchanges between Comcast software systems. Powered by AWS Kinesis and Apache Kafka, Headwaters provides a secure, scalable and federated platform backed by a fair cost model. Additionally, it decouples the real-time data producers from their consumers, while adding features to enhance communication, monitoring, and administration of the data streams.
Headwaters provides the following benefits for users:
- Assist data producers with AWS Kinesis data stream management and scaling
- Set up a communication channel to connect data stream stakeholders
- Hosts infrastructure within the Headwaters Community AWS account, adhering to the limits and with the approval of dx Finance.
- Fair cost model: teams that benefit from the data finance the infrastructure cost
- Bring your own AWS account and custom configurable namespaces for Kinesis and Kafka
- 100% encrypted data streams
- Support dedicated consumption with provisioned throughput using Kinesis fan-out/http2
- Light-weight governance process: producers control who can consume from their data streams
- All data streams in Headwaters are inventoried in dx Meta-Data Management System (IDDL) and visible in the dx Portal
Who is it for?
Ut enim ligula, scelerisque sit amet porttitor nec, condimentum sit amet libero. Etiam blandit viverra ipsum ac tincidunt. In pulvinar at risus vel porttitor. Suspendisse dictum dapibus nisi, id lobortis sapien. Suspendisse dapibus ultrices dapibus.
How to use it?
Headwaters is a Comcast streaming data platform based on AWS Kinesis and Apache Kafka. It provides several services on top of Kinesis and Kafka to truly decouple the real-time data producers from their consumers, while adding features to enhance the communication, monitoring, and administration of the Kinesis data streams.
The diagram below depicts the architecture of Headwaters 3:
The Headwaters platform is centered around AWS Kinesis data streams. Read more about Kinesis data streams here.
Headwaters manages the data streams using a single API, which manages users/Accounts, data streams, consumers, and namespaces (via BYO AWS accounts), and can be accessed via the dx Portal. To become a producer and/or consumer, you will need to register as a Headwaters user. See the User Creation section for instructions. Once the Headwaters user is created, you can assume the role of a data stream producer and consumer, which allows respective write and read access to Kinesis data streams.
Additionally, technical users can access the Management API directly. See documentation for Setting Up and Using the Headwaters API Using Postman and Authenticating with Headwaters3
To get started with Kinesis, view these code samples in GitHub.
Support & Feedback
Please summit a ticket for support.
in Philadelphia
