Booknotes 3.

I’m writing a guide to building your own database server. Here’s what I’ve been doing:

Progress

I’ve written more on the chapter on joining data and made improvements to the test runner.

Talks

I’ve submitted a lighting talk to HYTRADBOI. The slightly-silly talk goes some of the shade the PostgreSQL documentation throws on the SQL standard.

I also have an idea for a talk about implementing part of a database server that I might submit to LRUG for February.

Restarting my implementation

I’ve restarted my sample implementation so I can go through the test-suite as someone following the guide for the first time would. I haven’t started completely from scratch since I’m re-using chunks from my previous implementation.

This has led me to add better error handling so even unexpected errors are formatted sensibly and a feature that prints out tests that have passed for the first time. I’ve also added some automated tests for the test suite itself that run as a Github Action against different Python versions.

Since I’ve rearranged a few chapters, this process has also let me build a git history that better matches the order of the guide.

Iterators

I’m deciding when I should introduce iterators in the guide. Many real databases use the iterator pattern to process different parts of the SELECT queries. Iterators let you pull in data as it’s needed. This avoids redundant processing and has lower memory requirements compared to running each part of the query in sequence.

In the first part of the guide I want to discourage people thinking from about performance of queries and working on optimizations that might make their implementation harder to change. In some ways iterators are an optimization but I’m trying to decide if they are also a more helpful abstraction for processing data that could lead to more understandable solutions.

Goal for February

By the end of February I’d like to have the guide at the point where I can share it with a few people that can try it out.

The main things I want to do before then are:

• Write last two chapters on aggregate functions and grouping
• Complete sample implementation for all chapters

The things I’d like to do:

• Have a chapter/section about using iterators to process queries
• Write implementation hints for every chapter

And the things I think I can leave to later:

• Write an introduction about SQL
• Complete all diagrams
• Write an appendixes about working on the test suite and ideas for bigger extensions