Dan Milne

PostgreSQL

Migrating PostgreSQL across the planet, with almost no downtime.

TL;DR: Using PostgreSQL logical replication and Tailscale VPN, I migrated 200GB of data from California to Sydney with seconds of downtime and no time pressure. The replica synchronized over several days while production ran normally.

PostgreSQL

Using PostgreSQL Tablespaces on your VPS

Moving PostgreSQL indexes to the local NVMe drive from Hetzner's remote volumes boosted TPS by 86% (3,161→5,870). Using tablespaces to split storage between fast local and slower remote disks gives massive performance gains for free when your database is too big for the local drive alone.

Development

When an esoteric bug gives you a crisis of confidence.

I'm building a little toy / passion project app for saving bookmarks and archiving what they link to. It's Ruby on Rails, with TailwindCSS and Phlex components. It's very vanilla Rails. When I built a cross platform image for it on my ARM64 Mac, and

LLM

Claude summarises the last 3 weeks of development

I asked Claude Code to gen-ai me a blog post: Can you use Git or Gitea and generate a summary of what we've achieved in the last 3 weeks? Make it like a blog post? I did edit this just a little, to remove irrelevant / non-production ready feature

Development

Vibe coding MCP servers

"Model Context Protocol" - it's the hot new thing. It provides a standard interface for LLMs like Claude and ChatGPT to use other software on the internet. I recently hooked Claude Code into my self-hosted Gitea service via the Gitea-MCP. This means that, right within my

SQLite

SQLite on Rails

Ruby's SQLite library and Rails SQLite adaptor have improved their capabilities and performance massively recently. It's now perfectly reasonable to use SQLite in a production application. I presented at Ruby Australia's Melbourne meetup on these improvements and why you should take a close look

Docker

Blocking abusers with Anubis

Making sure you’re not a bot! I don't know about the rest of you, but I'm over the massive, abusive, scraping that's taken off over the last 6 to 12 months. Bots identifying as bots I can deal with, bots pretending to be

SQLite

SQLite3 Rsync!

I've been researching LiteStream and LiteFS for backing up and replicating SQLite databases. They're both very cool. LiteStream will stream your SQLite database to an S3 bucket, an SFTP server, or a local directory, with configurable sync-interval which defaults to 1 second. LiteFS will replicate your

SQLite

Hybrid Databases in Rails: Using SQLite Alongside PostgreSQL

My Rails site Booko has several ancillary services which support it. There's the Cover Art system which downloads and resizes book cover art. CSV-to-API service which downloads CSV files of book prices and turns them into an API for Booko to use. The latest of these, which I&

Talks

Semantic Searching with RAG and Ruby

Hey! I did a talk at Ruby Australia on how to build a Semantic Search system using Ruby. I really enjoyed building this little demonstration toy. I first got a glimpse of how they work with Simon Willison's excellent video "Embeddings: What they are and why they

TIL

TIL 20250228

Today I learnt that ALTER SYSTEM SET max_connections = 100; writes a new configuration file in the data directory for PostgreSQL, postgresql.auto.conf which takes priority over the values in the normal postgresql.config file. It's especially pointless to use ALTER SYSTEM for max_connections because you

TIL

Terminal Screenshots

I'm preparing a talk on Semantic Search and RAG in Ruby and I found myself in need of screenshots of code and terminals. At first I just took screenshots, but it was hard to make them consistent. I'm sure there are many ways to do this,

TIL 20250224

It's good practice to set CPU and RAM limits for Docker containers, so that one rouge container can't take down the rest.

TIL

Multiple Databases in One Postgres Container

Recently, some new Ruby Gems were released which aim to consolidate various supporting service for Rails such as Redis, into database backed systems. These gems are : solid_queue, solid_cache and solid_cable Here's how I setup SolidQueue, SolidCache and the SolidCable databases. Booko's databases run

TIL

TIL 20250217

* With larger Docker images, splitting out the busy work ( git checkouts, compilation ) of the build state, and creating a new image as the base for the remainder of the Docker image can make the builds and pushes much faster. * If you're building Docker images for architectures which aren&

TIL

TIL 20250209

Things I Learnt today * Embedding models have a maximum context length and if your input is larger than that, you'll need to chunk it, ideally with overlap. * You sometimes need to include a prompt telling the embedding models you’re generating embeddings, both for generating embeddings for comparison

A new threshold for commenting code.

My new code commenting threshold: I'll add explanatory comments if I suspect an LLM will make incorrect assumptions about the code during review.

TIL

Deploying Rails + SQLite with Kamal

Today I learnt about Kamal's ability to map Docker volumes into containers. When you're using SQLite, without using volumes, your SQLite database will be trapped inside your image, forever being replaced every time you deploy your app. Rails 8 seems to like storing SQLite databases in

Things LLMs are great at.

Kagi's code assistant, powered by Claude 3.5 has become quite embedded in my day-to-day Rails development and in my administration of Booko. An area where it really improves my quality of life, is pasting in a bunch of logs or an exception - or both, and having

Dog

Dog

Docker

Run Ghost in Docker compose

Experimenting with running Ghost in Docker compose