Creating Singer Target to Send Data to Web Endpoint

Please NoteIf you are new to Singer, you may want to check out my last post Creating Singer Tap to Capture Ebay Completed Items. It provides a high level background of the specification and how taps & targets work together.   The Challenge Last week I created a walk through of Creating Singer Tap to Capture Ebay Completed Items. While it's great to capture data, it's not overly useful without persisting the data to a target destination. There are some useful targets p... [More]

Creating Singer Tap to Capture Ebay Completed Items

What is Singer The Singer specification bills itself as "the open-source standard for writing scripts that move data". PipelineWise and Meltano are popular open source platforms that use the Singer specification to accommodate ingest and replication of data from various sources to various destinations. Singer describes how data extraction scripts—called "taps" —and data loading scripts—called "targets"— should communicate, allowing them to be used in any combinatio... [More]

Modern Data Stack in a box (all Open Source)

This week I participated in a hackathon for fun with a couple of friends. There were 3 rules for each of us: Upgrade something you are unhappy with in a side project Try some new piece of tech Document what you did My project Over the past 10 or so years, I have had a lot of fun running CollectZ - Research and Arbitrage Platform for collectibles categories.  TL;DR - The platform informs what undervalued collectibles I should consider buying and when + where + at what p... [More]

The 4 Analytics Questions of Subscription Ecommerce

I have spent over 20 years building my own subscription based service (SportsCollectors.Net) and working for companies with subscription offerings (Dell, Jupiterimages, Weight Watchers, Rent the Runway, ElysiumHelath). While the business models, value propositions and customer segments of these companies may be very different, there are similarities I recognized as these companies (all with product market fit) looked to accelerate their growth. From this, there are 4 question... [More]

Optimizing spend in Snowflake

Since about 2016, I have introduced Snowflake at two of my day jobs (Rent the Runway and Luma Financial Technologies) and have witnessed the platform's considerable evolution.  As with any hosted SaaS platform, it is very important to understand both how the solution is architected, how users interact with it, and how it is priced.  A very common concern among dats folks is that their Snowflake bills grown considerably as their use cases for it expand. A few months ago I saw this post... [More]

Dagster with Python, Singer, and Meltano

I have been a fan of Dagster for data orchestration for a little while and wanted to share some of the basics. There are a lot of cool things I like about it (compared to airflow and other schedulers): It is Declarative (via their Software-Defined Asset object). I personally like tools and frameworks that allow me to declare a desired end state (Terraform, dbt, Puppet, Ansible, etc.) vs. frameworks that have me build a bunch of imperative tasks that get daisy chained together.    I... [More]

My Snowflake Set up with Terraform

I have been working with Snowflake since 2016 when I proposed and chose to bring it into Rent the Runway to replace our very painful on premise Vertica implementation (yes we had it in a data center in NJ). Since then, Snowflake has grown considerably and is now one of the leading Data Warehouse offerings.After implementing data warehouses at several companies and having lots of conversations with some very smart folks, I've learned some things along the way. I've found it is a really good idea ... [More]

Fun First Project with my 3D Printer

I have long been curious about 3D printers and my daughter has been playing with CAD programs for her middle school STEM Club.  So I decided to do some research and buy the Creality Ender 3 3D Printer Fully Open Source with Resume Printing DIY FDM Printer 220x220x250mm off Amazon for $165. I also bought 2 1KG spools of PLA Filament so I could print things. I am completely new to 3D printers, but i have put together my share of projects and I do engineering for my day job. I decided that 10... [More]

Automating Image clean up with Python

If you have been reading this blog, you'll know I collect sports cards. It's fun to share what I have with other collectors (by posting scans on sportscollectors.net, facebook, etc.). A few years ago I bought a Brother MFC-9130CW all in one printer/scanner that I use to do my scanning. I usually set it to scan documents in legal format so I can fit 9 cards on a scan (3 rows by 3 columns). And since I want to do this most efficiently, I generally save them as a 200dpi pdf file with multiple page... [More]

Open Source Data Tools I like

At a recent meetup for the NYC Snowflake user group, the moderator asked the panelists what open source data tools do they use. I thought it would be nice to share my answers here. [More]