Please NoteIf you are new to Singer, you may want to check out my last post Creating Singer Tap to Capture Ebay Completed Items. It provides a high level background of the specification and how taps & targets work together.
The Challenge
Last week I created a walk through of Creating Singer Tap to Capture Ebay Completed Items. While it's great to capture data, it's not overly useful without persisting the data to a target destination.
There are some useful targets p...
[More]
What is Singer
The Singer specification bills itself as "the open-source standard for writing scripts that move data". PipelineWise and Meltano are popular open source platforms that use the Singer specification to accommodate ingest and replication of data from various sources to various destinations.
Singer describes how data extraction scripts—called "taps" —and data loading scripts—called "targets"— should communicate, allowing them to be used in any combinatio...
[More]
I have been a fan of Dagster for data orchestration for a little while and wanted to share some of the basics. There are a lot of cool things I like about it (compared to airflow and other schedulers):
It is Declarative (via their Software-Defined Asset object). I personally like tools and frameworks that allow me to declare a desired end state (Terraform, dbt, Puppet, Ansible, etc.) vs. frameworks that have me build a bunch of imperative tasks that get daisy chained together.
I...
[More]