Dr. Claude: Or, How I Learned to Stop Worrying and Love Whatever It Is This Is

I’ve been an analyst for more than 15 years. Throughout that time, the standard of analysis has evolved slowly, from Excel models to dbt models, briefly into Data Science (“the sexiest job of the 21st century!”), then ML, and then back to statistical process control, metrics, and reporting (thanks, Amazon).

Throughout this change, for me and the other people who were doing it right, SQL was the one constant. It has always been the best way to declare data transformations, and I hope it always will be. I have written a couple of love letters to SQL - first with sqlfmt and then with Harlequin. I created Harlequin because more and more of my work was being done in WSL, and I needed a better SQL client in my Linux shell, where it could easily access the same data as my Python programs.

For a couple of years there, it felt like we (Data teams) had it figured out. Push all your data into a warehouse or data lake, run a SQL engine on top, write declarative data models, and point BI and ML tools at those models. I thought we could keep doing this forever, and I would be maintaining and improving Harlequin until I died.

At first slowly, and then this spring very suddenly, AI dramatically changed the way that I work. It hasn’t all been for the better. My day job is as a Fractional Chief Data Officer for start-ups and other fast-growing companies. Over the past 15 years I have built my reputation and earned my seat at the table by aggregating and making sense of an organization’s knowledge - both from their data and from talking to other humans. I have been a trusted advisor in all sorts of high stakes decisions. I am often the first person other executives turn to when they are kept up all night by a tricky issue or suddenly need to know the reality of what is going on in their function.

And suddenly, I wasn’t. Something happened around Opus 4.7 where I began to feel like I had been disintermediated. I started to hear, after the fact, that instead of coming to me, clients were going to Claude with their burning questions. They started showing me beautiful decks that Claude made with data queried from Metabase, then they demoed new custom reporting packages in a web app behind SSO, then a complex analysis of labor dynamics and fuel prices driven by the war with Iran.

The artifacts Claude produced weren’t perfect, but it was very clear to me that they didn’t have to be, and that I was in trouble. For my entire career as an analyst, correctness was the foundational pillar that my work stood on (I learned this at Bain, where the learned standard is called “Zero Defect”). Since GPT-3, my line when asked about AI was, “as long as being right is more important than being fast, then I will still have a job.” But this year, the asymmetrical advantage of Claude’s speed and the perceived quality of its output has me completely overmatched. I work part-time at several companies concurrently, which only exacerbates this asymmetry: I no longer have time to even read, let alone review, check, or think critically about, most of the analyses being performed by and shared at my clients. People have always wanted dashboards, but now they don’t want to look at dashboards, they want Claude to do that for them. I typically work with smaller organizations, where it is possible to keep tabs on and have a point of view on everything. But I can feel and see these organizations scaling out their work in scope and ambition faster than I (or frankly, they) can keep up.

As an executive and decision maker, I’m not quite sure where we go from here. What does it mean for an organization to know something? It used to be that you got a bunch of leaders in a room together, you reviewed a document, you discussed its implications, and you all internalized some of what you learned. You took that with you and it informed your decisions. That was how organizations learned, grew, and changed.

But next year will that be the case? Will Amazon continue to run a WBR? Or will Claude do it for them, and then write some bullets to a file called EXECUTIVE SKILL.md on some shared drive that all other corporate agents must read before taking any actions?

I have no clue. But I do know that I need to move faster. I still write a lot of SQL by hand, and spend a lot of time looking at data, but for the first time I’m seeing a future where that just won’t be the case. I recently had Claude run a lagged conversion rate analysis for me (using my Convoys fork) and he nailed it, in 1/10th the time it would have taken me 6 months ago. I didn’t write a single line of Python or SQL, but Claude did, and watching him run that SQL was fascinating. Claude’s favorite SQL client was a Python library, and he would ‘python -c import redshift_connector…’, writing a 20-line bash and Python script to execute a query every time he wanted to check his model. Agents are really good at these scripts and it seemed to work every time, but it was total lunacy.

If only Claude could use Harlequin, I thought. We could share one profile with my credentials, we could look at output together, and we could both be much more efficient.

So we went to work. I asked Claude to design the best SQL client for an agent, building on Harlequin’s foundation. We both agreed psql provided a familiar starting point, and the duckdb CLI added important capabilities. We could borrow that headless UI and use Harlequin’s adapter ecosystem to provide one consistent and familiar interface to any database. But there were other features missing in these tools that were built for humans and scripting but not agents. Harlequin could simplify catalog discovery and add safety features (low default limits, enforceable read-only modes). And finally, it can provide a “multiplayer” experience, where workflows can be handed back and forth between humans and agents, with result handoffs, shared query history, and more.

Yesterday I released Harlequin 2.9.0, which includes the first version of hsql, the headless CLI for agents that shares the config and query engine with Harlequin. There are many more features to come, but hsql is already extremely capable: it works with any Harlequin adapter (dozens of databases), executes SQL from files or the command line, and can output results in 10 different formats.

Until this release, I wrote all of Harlequin’s code by hand. On the Python Show last year I said that Harlequin was my art project, and I didn’t want any AI writing Harlequin code. I still feel the same sense of craft and love for Harlequin, but letting Claude contribute has unblocked me and revitalized this project and will hopefully bring it forward into the next era. I am still reviewing every line of docs and code, still steering the decisions, and still view Harlequin as my creation. But maybe one day that won’t be the case anymore, and agents will just read a TED.md file. And when that time comes, I’ll ride off into the sunset, but maybe at least the project will live on forever.