Matcha Byron Bay started as a stall at the Byron Bay Farmers Market in July 2022. The business is run by Japanese locals who source their matcha from Uji (near Kyoto), and these days they sell online Australia-wide and supply matcha and hojicha to cafes, retailers and other businesses.

This blog post gives an overview of the new website, some of the technology behind it (kept at a high level), the design choices we made, and where things are at with wholesale, SEO and sales so far.

Where we started

The previous website was built on WordPress with WooCommerce. It did the job, but it had some issues that were costing the business:

1. Scroll-jacking on the home page that would stop you scrolling all of a sudden.
2. Adding to cart was slow, and some layouts broke at certain screen sizes.
3. The blog only ever showed 9 articles, even though there were roughly 37 of them.

The guiding principle for the rebuild was that the website is the means, not the end. It exists to sell products, generate wholesale enquiries, build trust in the brand and reduce the effort it takes for someone to buy. Every decision below was weighed against those outcomes.

The new website launched in July 2026. As part of that, we migrated the products, collections, customers, historical orders, articles and media across to Shopify, and mapped the legacy URLs over with redirects so that existing search traffic still landed somewhere sensible.

Website at a high-level

Shopify is the commerce backbone. It owns the catalogue, pricing, checkout, customer accounts, orders, payments, subscriptions and (more recently) wholesale companies.

The storefront itself is headless. It is a TanStack Start React application that talks to Shopify through the Storefront API, using Shopify's new framework-agnostic Hydrogen developer preview (what I have been referring to as "Hydrogen 2.0").

This version of Hydrogen describes itself as a toolkit rather than a framework. The original Hydrogen was tied to Remix (and later React Router), whereas the new core is plain JavaScript with thin bindings on top. That gives you a typed Storefront API client, cart handlers that work before any client JavaScript has loaded, and consent-aware analytics, without being locked into a particular framework. TanStack Start is not one of the officially listed frameworks in the preview, but because the core is framework-agnostic, wiring it in was straightforward.

The storefront is server-rendered on Cloudflare Workers, so pages render close to the customer and static assets are served from Cloudflare's edge. Checkout is still Shopify's hosted checkout, so payments stay with Shopify.

The following is a simplified look at the architecture:

The Worker renders pages using data from the Storefront API, and Shopify's checkout takes over for payment. Paid orders and other events then flow back into our background jobs, which write to storage. How the code gets deployed is covered further down.

A few reasons for going headless over a Shopify theme:

1. We have full control over the design, the markup and the performance budget.
2. Features like the wholesale portal (more on that below) need real application logic, background jobs and storage, which would have been awkward to bolt onto a theme.
3. Search engine work such as structured data, canonical URLs, redirects and sitemaps all live in code that we can review and test.

The trade-off is that we own more. Things Shopify themes give you for free (like predictive search and account pages) needed to be built or handed off to Shopify-hosted pages.

Under the hood

Similar to Pocket Rice, I wanted the infrastructure to be reviewable and repeatable rather than clicked together in dashboards.

At a high level:

1. Pulumi manages the long-lived resources: key-value storage, queues, a database, private object storage, and the GitHub deployment environments and their secrets.
2. Wrangler owns the Worker itself: the build, versions and bindings.
3. GitHub Actions orchestrates releases. A merge to main runs the tests, applies any infrastructure changes, builds the app, runs database migrations and then deploys the Worker.

Feature branches each get their own disposable preview Worker with a stable URL. That makes it easy to share changes with the business before anything goes live, and previews are never indexed by search engines.

Anything that does not need to happen during a page request is pushed into the background. Form submissions and Shopify webhooks (for events like paid orders) are recorded and handed to Cloudflare Queues, where consumers handle things like enquiry emails, wholesale invoices and purchase attribution. For the work that really matters (like invoices), a scheduled job picks up anything that gets stuck, so a dropped message does not mean dropped work.

For observability, errors go to Sentry, each request emits a single structured "wide event" to Cloudflare's logs, and product analytics go through PostHog once a visitor has given consent.

On the code side, the server is written with Effect so that failures (Shopify being slow, a bad response, a missing permission) are typed and handled explicitly rather than silently turning into empty pages.

Design choices

The design leans on the brand's colours (deep green, olive and cream), with photos from the market stall and the tea fields in Uji doing a lot of the storytelling.

One small trade-off worth calling out: some of the light yellow text on green does not meet colour contrast requirements, so the site will not score a perfect accessibility audit. We decided to keep the brand colours rather than water them down.

The home page puts proof up front. The hero carries a Google rating badge, and directly underneath it is a "Found at" strip with logos of the cafes and markets where you can find the matcha, followed by a carousel of real Google reviews.

From there, the focus is on helping someone pick the right tea quickly. We added a comparison between the three main teas, badges for what each one is best for, and size selection with add-to-cart directly on the card.

The shop is broken up into collections such as ceremonial matcha, culinary matcha, hojicha, tea tools and ritual sets.

The product pages went through a conversion review after launch. The goal was to answer four questions as quickly as possible: which tea suits me, what will it taste like, how much tea am I getting, and what will delivery cost? That led to things like an approximate number of servings and cost per serving next to the price, delivery information next to the add to cart button, and a subscription option at 10% off.

Since launch we have also added moderated product reviews with verified-purchase badges, plus a free delivery tracker and a complementary add-on in the cart.

The flow for a retail customer looks roughly like this:

The market QR codes are worth a mention. The stalls are still a big part of how people find Matcha Byron Bay, so for physical placements like the stall we generate QR codes with campaign tags baked in. That way offline traffic can be attributed like any other campaign.

A Japanese-owned business

The owner, Ayumu, is Japanese, and the business began as a dream between Japanese friends living in Byron Bay. The brand story is very much about the connection between Uji and Byron Bay, and the about page tells that story directly, including the team's names in katakana.

The about page walks through the journey from that first market tent to where they are today.

This had some practical implications for the build as well:

1. The wholesale buyer guide and staff guide are published in both English and Japanese.
2. The wholesale walkthrough video is narrated in both languages with captions, and you can switch between them on the support page.
3. Japanese copy needs a review from a native speaker before it goes out, including Shopify's menu names, which differ between languages.
4. Some customers write in Japanese (a few of the Google reviews on the home page are in Japanese), so anything that renders customer-provided text has to handle it. The one place that does not yet is the PDF invoice renderer, which uses standard Latin fonts. Rather than silently mangling a name, anything it cannot render is flagged for review.

Wholesale

Commercially, the wholesale page is one of the most important pages on the site. Cafes, retailers and workplaces buy in much larger quantities than retail customers, so every new wholesale partner matters.

The page went through a dedicated SEO and messaging review. The first version had a lovely brand story, but its main heading ("Bring beautiful matcha to your business") did not say what the page was actually for. The heading was changed to say exactly what the page is for, and the page was restructured to answer the questions a cafe owner actually has before they enquire.

Social proof comes first, with logos from the wholesale community.

The business started in Byron Bay, but it supplies cafes Australia-wide. Rather than create a thin landing page for every city (which tends to backfire), the page talks about where we supply in one place and answers the city-specific questions in the FAQ.

One of the bigger decisions was to publish guide pricing. It is common for wholesale pages to hide pricing behind a form, but cafe owners want to compare cost per gram and cost per cup before talking to anyone. Publishing it builds trust, and it also gives search engines something concrete to rank for. The prices a buyer actually pays come from their own catalogue in Shopify.

Further down, the page explains exactly how ordering works, from first enquiry to paid invoice.

The enquiry form asks for enough to qualify the lead (business type, location and what they are interested in). Submitting it emails the business our wholesale pricing and partnership PDF and notifies the team, so nobody is left waiting for a reply to find out the basics.

Finally, the FAQ covers the long-tail questions cafe owners ask (and search for), such as whether we supply Sydney and Melbourne, which matcha suits lattes, and how many drinks you get out of 1kg.

The wholesale journey end-to-end looks like this:

The wholesale portal

The newest piece of the website is a wholesale portal where approved cafe and retail buyers can place orders themselves.

Without a portal, every wholesale order and invoice is a manual back-and-forth. That works with a handful of partners, but it does not scale.

The split of responsibilities is intentional:

1. Shopify owns identity, company permissions, wholesale pricing, orders and payments. Buyers sign in with Shopify's hosted email sign-in and receive a one-time code, so there is no new password system to maintain.
2. Our website owns the portal experience: choosing the venue you are ordering for, the order form, order history, reordering and the invoices.

Each venue is a Shopify company location with its own pricing, delivery address and order history. Buyers can pay by card through Shopify's checkout, or choose bank transfer. For bank transfers, the portal shows Shopify's calculated delivery and GST for review, places an unpaid order and emails an invoice with the payment details. Staff mark it as paid in Shopify once the money arrives.

Invoices are generated in the background from Shopify's order events. They are stored privately, versioned when an order changes, and only served after checking that the buyer still has access to that order.

Onboarding was treated as part of the feature rather than an afterthought. The support page works before you sign in and has the walkthrough video with chapters for each step, plus the printable buyer guides in English and Japanese.

Other key pages

The find us page lists every market the stall attends with the day, hours, location and a link to a map. It is also marked up with local business structured data for search engines.

The blog holds the preparation guides, tea explainers and local guides. Like Pocket Rice, the posts now live as MDX in the repository rather than in a CMS.

SEO and backlinks

SEO for a small local business has been the hardest part of the project, and it is very much still in progress.

The first challenge was the migration itself. After launch, a number of old WordPress URLs were still indexed, and some of them led to pages that no longer existed. We audited the legacy URLs, added redirects for anything with value and made sure the sitemap only contains the pages we want indexed. Indexing is also locked to the production domain, so preview deployments never end up in search results.

Locally, things are in good shape. The Google Business Profile ranks first in the map pack for searches like "byron bay matcha". One lesson here: rank trackers can be misleading for local queries. Our tracker reported a big drop for "best matcha near me", but it turned out it was being served Sydney results rather than what someone in Byron Bay would see.

Backlinks are where it gets hard. Third-party tools show hundreds of referring domains for the site, which looks great until you look at them. Nearly all of them are automated spam and "checker" sites that started linking in May 2026 and carry no value. The genuine links are few and far between (and one of them is Pocket Rice, which I will admit is not exactly an independent vote).

What we really want are links from the cafes and businesses we supply, as well as the markets and local guides that already mention the brand. That is harder than it sounds:

1. Most of the cafes we supply are small, busy hospitality businesses. Adding a link to their supplier is understandably not high on their list.
2. Some are more active on Instagram than on their own website, so there is not always an obvious place for a link to go.
3. Market organisers' stallholder pages and local guides often already mention Matcha Byron Bay by name, but do not link to the website.

So the approach is to make linking worthwhile and easy. We write content that features partners (such as where to find our hojicha at Old Maids in Brunswick Heads and the best matcha around Byron Bay and the Northern Rivers), give partners something they would want to share, and ask market organisers and local guides to turn existing mentions into links. A handful of real, local links would outweigh the entire spam profile.

The blog has been the weakest area so far. It gets plenty of impressions but very few clicks, and some of the older articles overlap and compete with each other. The plan is to consolidate those into fewer, stronger guides with named authors and proper updated dates.

Results so far

It is still early, and the numbers are small enough that I want to be careful with them, but there are some encouraging signs.

At launch, Google's PageSpeed lab tests across 11 equivalent pages had the new site scoring 96 to 100 on mobile and 100 on desktop. For comparison, the old home page scored 83 on mobile.

From the September report (compared to August):

1. Paid online order value was up roughly 79%, with average order value holding steady.
2. Google organic clicks were up about 22% and impressions up about 8%, with the average position improving from 11 to 7.8.
3. The wholesale page roughly quadrupled its organic clicks, and its average position improved from around 19 to around 10.

Experimenting with Google Ads

In September we started running Google Shopping campaigns for the ceremonial matcha and hojicha. Getting there took more work than expected: the Merchant Center feed and the website needed clear shipping, returns and business information before Google was happy to show the products.

The campaigns are delivering clicks, but it is too early to say whether they are worth it. Attribution is the hard part:

1. Analytics only run once a visitor has consented, which is the right thing to do but leaves gaps.
2. Checkout happens on Shopify, so the click and the purchase live on different systems.
3. Google's own conversion tracking has only recorded a small number of purchases so far, and some ad-assisted sales are likely going unattributed.

To close that gap, we built our own purchase attribution. With consent, the campaign tags from the landing page are carried through to the cart, and when Shopify tells us an order has been paid, a background job records the purchase against the original campaign. Until that data is reliable, we are holding off on increasing spend.

What is next

1. Onboarding the existing wholesale partners onto the portal and measuring how many orders move across.
2. Getting purchase tracking to a point where we can make real decisions about Google Ads.
3. Consolidating blog content and earning genuine local backlinks.
4. Asking customers for reviews once they have received their order.

I may write a follow-up once we have a few more months of data, particularly on the wholesale side.

Links and further reading

• Matcha Byron Bay homepage
• Matcha Byron Bay collections
• Nami ceremonial matcha
• Matcha Byron Bay wholesale
• Matcha Byron Bay wholesale portal support
• About Matcha Byron Bay
• Find Matcha Byron Bay at the markets
• Matcha Byron Bay blog
• Hojicha in Brunswick Heads at Old Maids
• Best matcha in Byron Bay and the Northern Rivers
• Building the Pocket Rice marketing website
• Hydrogen developer preview
• Shopify headless storefronts
• Shopify Storefront API
• Shopify Customer Account API
• Shopify B2B
• TanStack Start
• Cloudflare Workers
• Cloudflare Queues
• Wrangler
• Pulumi
• Effect
• Sentry
• PostHog
• Google Search Console
• Google Merchant Center
• Google Business Profile

As part of her recent push for the business, we set up the Pocket Rice marketing website as means of generating reach within the local region and having a more professional presence.

This blog post gives an overview of the marketing site build and the backend with a core pillar of the build being to keep build costs low.

Website at a high-level

I added the marketing website and backend to a monorepo that I have already been maintaining for other apps that manage our ordering system.

The website itself actually started as a Next.js website that my partner "vibe coded" which I had to take over and optimise for certain desired outcomes (asset optimisation etc.) as well as figure out the infrastructure for.

Initially, the website handed over used the Next.js app router for the backend which I migrated over to a Go HTTP backend with their standard library router. After the migration, I converted the Next.js website to be a static website only.

For hosting, the code is (currently) hosted on GitHub and using GitHub Actions to deploy to AWS via the AWS CDK infrastructure. The repo already had this configured from the order management system application I had already configured, so all I needed to do was add a new stack.

This stack itself hosts everything behind AWS CloudFront which has a number of benefits, but one of those was the ability to alter behaviour based on the route.

All key Next.js static pages are redirected to a private S3 origin to serve static assets and pages, while anything under the /api/* path will be directed to an AWS API Gateway which has a proxy integration to an AWS Lambda running the Golang server.

The following is a simplified look at the architecture:

There are a number of benefits to this:

1. The lambda at the expected volume will remain in free tier.
2. There are no cross-origin resource sharing setups that I need to configure.
3. We get the benefit of all the security rules and DDOS protections that can be applied directly with a CloudFront origin.

These benefits alone have me incredibly bullish. Because we are using the Golang standard library HTTP router, if we ever outgrow lambda or costs become non-insignificant, it is easy enough to migrate over to an alternative like Elastic Container Service (or anywhere I decide to change hosting to, but ECS can also be an API Gateway target and is easy to migrate to).

Once deployed via CI/CD on a merge to main, our marketing website becomes reachable.

Blog pages

We host a blog on the marketing website as a means for helping to generate organic traffic.

Our goal is to have a funnel that can be used to drive awareness of our brand within Byron Bay.

Each blog post itself has a simple MDX configuration that allows us to reuse defined React.js components through the posts themselves.

Our goal here will be to identify some key blog posts and drive content around them. Some of our current "anchor" blog posts include:

• Education on the difference between onigiri and sushi, a question we get all the time at markets.
• Market guides such as the Byron Bay Community Market guide in an attempt to win a share of market traffic.
• Guides to Japanese good around Byron Bay which we hope to begin improving our ranking for.

It is still early days here, so we will be looking to drive traffic and partnerships going forward for this.

Online store

As part of our expansion online, my partner also wanted to introduce a store to sell Japanese home goods.

Australia is currently going through it's "Japan phase" where everyone is holidaying in Japan and being introduced to Japanese culture. The hope is that this can be a way to connect those positive feelings that Australians have towards Japan and to bring them into homes here.

The website itself has a landing page for the shop.

As you would expect, each item and variant has its own page for adding to a cart.

We are using Stripe checkout to manage our payments and manually doing shipments ourselves.

Other key pages

In addition to this, we also have a few other key pages to help with organic traffic or things that we wanted to highlight.

Our markets page highlights where people can find us.

We list our suppliers that we use for our onigiri as well to also highlight our local community members and importers.

Finally we have a contact page to have a way for others to contact us.

SEO and indexing

At the moment, there is a very annoying bug where you cannot add newer Google Service accounts for the indexing API, meaning that right now I am stuck doing a manual process with the Google Seach console in order to update sitemaps and request page indexing.

Things that I am also doing at the moment include indexing for other search engines as well as working through some SEO books. I've given myself 12 weeks from the first release of the website in order to see if I can start ranking highly across the web for our Pocket Rice brand in key categories such as markets to see and food you must eat. I may even write a follow-up post on how success is going in that area.

Links and further reading

• Pocket Rice homepage
• Pocket Rice blog
• Onigiri vs sushi
• Byron Bay Community Market guide
• Japanese food around Byron Bay
• Pocket Rice markets
• Pocket Rice suppliers
• Pocket Rice contact page
• AWS CloudFront
• AWS API Gateway
• API Gateway proxy integration
• AWS Lambda
• Amazon Elastic Container Service
• Stripe Checkout
• Google Search Console service account issue
• Google Search Console

The main reason is not that the implementation is complicated. It usually is not. The value is in learning how to recognize when the problem is giving you permission to do less work.

If an array is sorted, or if a string can be scanned from both ends, or if you are really maintaining a shrinking window, there is a good chance a pointer-based approach is available.

This post starts with a walkthrough of one of the more common patterns you begin with online and then adds some other examples and visuals for other solutions.

const nums = [1, 2, 3, 4, 6, 8, 11];
const target = 10;

export function pairSumSortedBruteForce(nums, target) {
  for (let i = 0; i < nums.length; i++) {
    for (let j = i + 1; j < nums.length; j++) {
      if (nums[i] + nums[j] === target) {
        return [i, j];
      }
    }
  }
  return [];
}

export function pairSumSorted(nums, target) {
  let left = 0;
  let right = nums.length - 1;

  while (left < right) {
    const sum = nums[left] + nums[right];

    if (sum === target) {
      return [left, right];
    }

    if (sum < target) {
      left += 1;
    } else {
      right -= 1;
    }
  }

  return [];
}

export function pairSumWithSort(nums, target) {
  const pairs = nums.map((value, index) => ({ value, index }));
  pairs.sort((a, b) => a.value - b.value);

  let left = 0;
  let right = pairs.length - 1;

  while (left < right) {
    const sum = pairs[left].value + pairs[right].value;

    if (sum === target) {
      return [pairs[left].index, pairs[right].index];
    }

    if (sum < target) {
      left += 1;
    } else {
      right -= 1;
    }
  }

  return [];
}

export function pairSumHashMap(nums, target) {
  const seen = new Map();

  for (let i = 0; i < nums.length; i++) {
    const complement = target - nums[i];

    if (seen.has(complement)) {
      return [seen.get(complement), i];
    }

    seen.set(nums[i], i);
  }

  return [];
}

function isSubsequence(s: string, t: string): boolean {
  let i = 0,
    j = 0;

  while (i < s.length && j < t.length) {
    if (s[i] === t[j]) {
      i++;
    }

    j++;
  }

  return i === s.length;
}

function twoSum(numbers: number[], target: number): number[] {
  let left = 0;
  let right = numbers.length - 1;

  while (left < right) {
    const sum = numbers[left] + numbers[right];

    if (sum === target) {
      return [left + 1, right + 1];
    }

    if (sum < target) {
      left++;
    } else {
      right--;
    }
  }

  return [];
}

function threeSum(nums: number[]): number[][] {
  nums.sort((a, b) => a - b);
  const results: number[][] = [];
  const n = nums.length;

  for (let i = 0; i < n - 2; i++) {
    if (i > 0 && nums[i] === nums[i - 1]) continue;

    let left = i + 1;
    let right = n - 1;

    while (left < right) {
      const sum = nums[i] + nums[left] + nums[right];

      if (sum === 0) {
        results.push([nums[i], nums[left], nums[right]]);

        while (left < right && nums[left] === nums[left + 1]) left++;
        while (left < right && nums[right] === nums[right - 1]) right--;

        left++;
        right--;
      } else if (sum < 0) {
        left++;
      } else {
        right--;
      }
    }
  }

  return results;
}

My last interview was back in November 2023, which in this day and age can be appropriately measured by available LLM models: Claude 2.1, GPT 4-Turbo and Gemini 1.0 Pro. Yes, that long ago.

As Seneca says, "Luck is where opportunity meets preparation". This post covers my attempt to reconcile previous approaches to software interviews alongside any tidbits of information I can find from my network and beyond.

I will be covering resources around the following topics:

1. The job market.
2. What job interviews look like.
3. How to prepare for interviews.
4. Advice on your applications.

Given the scope of software engineering, there is no point in pretending there is a one-size-fits-all scope.

Instead, this post is more of an anchor that links out where appropriate and gives my two cents (it is my blog after all).

A guiding principle to abide by in the age of agents

Before going any further, there is a quote from a tweet that I have been thinking about a lot:

You can outsource thinking but you cannot outsource your understanding.

In this day and age, this hits hard.

AI is tool to be used, but it can be a slippery slope. You need to best be able to determine when the use and costs are a net win and have the discipline of when to say no.

My current mindset: there is a place for delegating, vibe coding and multitasking, but it is your responsibility to be aware of the where and when. When it comes to guiding AI for anything mission critical, you need to be the driver. Don't slip into the habit of spinning the poker machine.

We will touch on AI a bit more later, but I think addressing the elephant in the room early is an appropriate starting point.

The job market

TrueUp is a great companion on our search to understand the job market. It scans the tech job market in realtime and provides a trends section.

Some of the positive notes (from the time of writing):

1. The number of tech jobs has been steadily increasing since the low of 2023.
2. The remote job options have remained fairly steady, with a recent increase happening.
3. (For more own sanity) Australia sits at #10 for top countries for tech.

Although it's from 2025, The Pragmatic Engineer's blog post State of the software engineering job market in 2025, part 1, what the data says provides some insightful bonus graphs on job available by senority and the type of company.

Without spoiling too much, it looks positive for the entry-level and mid-tier job market.

More specifically within my market (the Australian market), a useful pulse check is the Government Jobs and Skills data on software engineers website.

The latest data here from April 6th, 2026 indicates that in Australia there are 55,200 employed software engineers.

Based on TrueUp's data, there looks to be around 663 software engineer jobs advertised, which would indicate ~1% of the total employed work force.

As a not-very-science-y pulse check on LinkedIn for software engineering jobs in Australia, I paginated through ~34 pages of 20 jobs each (680) and they all still had "engineer" in the title. Make of that what you will (I will come back with a better approach when I get a free moment in future).

All-in-all, if you have come here fearing doom-and-gloom then fear not my friends. Our time is coming again!

As for finding available jobs, here is a list of potential job search websites (in no particular order):

• TrueUp provides access to their list of jobs
• remote.ok for remote-friendly jobs
• LinkedIn Jobs

Some colleagues also shared some sites:

• hatch
• Airtree
• Blackbird
• Squarepeg

A list of Australian-specific options:

• seek
• careerone
• indeed

If you are interested in specific companies, then I also recommend going straight to their own advertised jobs board.

If you are open to any jobs and don't mind someone coming to you with options, chat to a recruiter.

What do the job interviews look like?

In my general experience, the bulk of the interviews you will do are this:

1. First screening.
2. Technical interview(s).
3. System design interview.
4. Behavioral interview.

Depending on the your field of engineering (e.g. if you are a solution's engineer), you may find some variance but these have been at the core of my experience.

This looks to be aligned with the Tech Interview Handbook, a guide that I came across which has a walkthrough on interview preparation.

The link also contains some really good links to courses and preparation tools, but my own personal comments on that link from my own experience:

• The site puts "Take home assignment" as rare but my experience has been to get one ~40% of the time.
• A "quiz" is said to be occasional for a quick filter, but admittedly I've never really done one.

It also recommends a coding interview study plan of 11 hours a week for three months. In my honest opinion, that feels overkill (assuming this is not your first time with preparation).

What I would do before committing to a timeline is look for feedback on the existing interview processes of the jobs you are interested in and then narrow down what you need to be prepped for.

An example: In Australia, one job link shared with me was a role as a Software engineer at Macquarie Group. I'm not personally interested in this role, but assuming I am then I generally do a few things:

1. Visit the company on Glassdoor and check things like the interview feedback.
2. Check their reported salaries (for my own sake on expectations).
3. I also review their online recruitment/interview process if they have one.

As an example (from the above three links), I can grok the follow for mid-level software engineers:

• They earn AUD$90K-130K per year.
• Their interview process has a minimum of three rounds.
• Previous interviewees indicate a technical interview with a live coding session over (at minimum) a four week process.

Understanding this process goes a long way to getting a better idea about the process prior to your initial contact.

Technical interviews

My honest experience with technical interviews is this: if I grok the question and can quickly ascertain the correct path to go down, it's normally pretty straight forward. If I don't, it's a massive slog and sometimes a miserable failure.

Few love the idea of writing code under interview conditions, but it is still something best to prepare for.

For Leetcode-style interviews, Algomonster have a "return on investment" style layout for tech interview preparation that reflects a lot of my own experience.

It outlines problem-solving approaches and your likelihood to run into those style questions:

If you are short on time, I recommend at least working through the high ROI problem solving approaches.

Some websites with an array of problems on those topics:

1. LeetCode
2. HackerRank
3. CodeSignal

These same websites can also provide varying levels of online content for learning as well.

In addition to the above websites, I've also found ByteByteGo to be a great resource for books that comes with great visual guidelines.

Each of these are paid solutions which come with their own cost. Over the coming weeks I will also likely introduce my own online blog resources to help with some interactive learning.

Most of my personal experience with technical interviews have been fairly straightforward, but you need to be able to ask the right questions and quickly identify which approach to take. There have still been times when I've flunked straightforward questions.

Admittedly, I've never had to do any interviews when I "invert a binary tree" or anything of the sort. Seems impractical to test on but there are horror stories out there.

System design interview

For me, system design interviews feel like my bread and butter. I find them much more practical in my day-to-day work and I find them more intuitive to reason through.

My experience is normally to follow my own acronym CASMHDR:

The acronym itself is not something that I've taken from a course. It's just my lightweight go-to approach for something short like system design interviews.

I've personally committed the analogy of "crossing the chasm in high definition" to memory in order to remember the acronym, and so it's something that is always in my back pocket for working through problems.

System design interviews are generally open-ended and are an open discussion with the interviews. Working through clarifying the problem with clear communication (to me) feels like half the battle.

Some resources:

• ByteByteGo has a very detailed system design course and book.
• Educative also has a similar course.
• As always, there is a awesome repo for System Design Resources.
• System design nuggets is a nice subscription to link to small subsets of info.

I don't have a table to indicate any return on investment for resources here, but some key topics it is worth getting a high-level overview of at a minimum (based on my experience):

• Caching
• Consistency
• Content Delivery networks (CDNs)
• Reliability
• Fault tolerance
• Idempotency
• Rate limiting
• Message queues
• I/O bound vs compute bound

Most of my system design interviews have focused on implementation of high-level systems. There requirements have never been as extreme as what is covered in the system design books and courses I have taken.

My general guideline would be this:

1. Clarify requirements and scale estimation. Questions like "how many users?", "how many requests do we expect?", "are loads consistent or spiky?", "what are the payloads and how large are we expecting?" and "are there times with lower and higher traffic?" are some of my gotos.
2. Jot down the key data models that the system will be working with based on the information. Clarify their properties and be prepared to re-model during review time.
3. Start as simple as possible for your first high-level overview. My first pass is almost always a standard three-layer system. On your second pass, introduce systems alongside reasoning as you go through the "review" phase and then repeat the process to add those elements in.
4. For databases, start relational until you can introduce reasoning for not doing so.
5. Look for opportunities to scale and/or reduce the load for key servers. Introduce things like rate limits, queues as needed.
6. Also spend time reviewing the solution and offering suggestions on improvements or when changes would be needed in future. Systems are fluid and different solutions optimize for different scenarios.

My experience is that there is never really a "right answer". It more of a conversation with the interviewers around what solutions would you suggest to look into based on X requirements.

Behavioral interviews

This one for me has always felt more vague than the others. When most companies ask "why do you want to work for X company?", my honest answer has been "to keep a roof over my head" but that doesn't fly so well.

I find the list of questions on the tech interview handbook website covers basically every question I've ever received.

Some other tidbits from my experiences here:

• Interviewers seem more interested in my business pursuits than my side projects.
• Prepares your stories/approaches on conflict resolution.
• Be ready to explain why are you looking for your next role and why this role specifically (again, steer away from the bottom of Maslow's Hierarchy of Needs).
• Prepare some notes on some of your career work that you are most proud of. If possible, pick a few moments that can speak to from complementary domains (technical, communication, management etc.).

What about AI interviews?

This is where my experience screeches to a halt. From conversations with peers, some key points that they have mentioned:

1. Some live coding exams do not allow AI.
2. There are general conversations around how you use AI in your workflow.

Another peer shared through a LinkedIn post on 20 AI fluency interview questions. Some of the slides to me feel a little off but I think understanding some of ubiquitous language used around AI is important for expressing your own experiences with it.

Concepts around agents, memory, harnesses, skills, tools are all key concepts to know, and safely executing agents is another hot topic right now.

If you haven't spent time working with AI, dip your toes in and form your own opinion.

Assuming I am to talk about my use of AI tooling in upcoming interviews, I would probably be honest and just refer to the challenges and benefits I have personally seen.

Some key experiences off the top of my head:

1. It has been a force multiplier for tedious tasks. Examples like linking JIRA tickets and merge requests for security patches are now delegated and save me an hour or so each time.
2. Code outputs have been a mixed bag. I find that a "clean" codebase with strict standards and guidelines have been easier to work it than others and tooling I hadn't used before like open telemetry on localhost have become indispensable tools for agent verification.
3. Multi-tasking across multiple agents introduces significant brain strain for me. This leads to sleepiness and a tendency to slide in standards. I personally opt to spend a few hours a day working without AI when I need a flow state and I also look for tasks that it can significantly outpace me without burdening me with unexpected work.
4. It's very easy to "do everything" now that clicking "continue" is an option. My experience with AI is that you need to put on your product hat and understand when to say no.
5. AI is incredible when it comes to collating debugging information that you collect and actioning it.

I will update this section in future as I begin the interview process and give some updates on when AI comes into the process.

Advice on your applications

For this section, I would take anything I say with a grain of salt. I am constantly hounded in my reviews about how my resume looks so you are better off deferring to others on this boat.

Online guides that I will be referencing:

• ByteByteGo has another guide on writing a good resume
• Harvard have a list of tips and action verbs
• My alma mater provides some resume tips if you prefer an Australian lens as well as tips for your cover letters
• Reddit can also provide a nice ear-to-the-ground with tips

Conclusion

This has been my own guidance to software engineering interviews in 2026 built upon previous experience and my own expectations of what to expect in the upcoming process.

My hope is that there are some nuggets of wisdom embedded within this that can assist in your own search for work.

I will continue to update this page as the process goes on or as I get more feedback from peers. If you have any recommendations that you aren't pushing for your own agenda, send them through and I will update the page.

Although BuildKite offers some recommended templates for AWS that can scale CI alongside your team, I have been keen to first check out a more controlled approach with AWS CodeBuild which I believe can enable some cost-savings early in the project.

This blog post is a recap on my own internal investigation into configuring this setup.

Overview

The blog post covers my proof-of-concept. In it, we still host code on a private GitHub repository but creating a pull request should trigger CI to run with BuildKite + CodeBuild.

We will be using the AWS CDK to configure as much as possible for this, but there will still be some requirements where you will need to jump onto the AWS dashboard.

The desired outcome here is to run a simple build which makes use of AWS CodeBuild. Do not expect anything more advanced in this tutorial. I will cover more advanced use cases in future blog posts.

The following sequence diagram is an overview of our desired workflow for this post:

Prerequisites

You need the following already configured if you are following along with this tutorial:

• A Buildkite account.
• A GitHub account.
• A prepared, private GitHub repository.
• An AWS account.

The following is also recommended:

• Experience with the AWS CDK for declaring infrastructure-as-code.
• Experience with managing GitHub repositories.

Configuring a deploy key for your private GitHub repository

Assuming that you have already prepared a private Github repository for this tutorial, you will need to configure a deploy key so that Buildkite can access this private repository.

From the command line, you can run the following:

# Generate the key at ~/.ssh/buildkite-codebuild-poc
$ ssh-keygen -t ed25519 -C "buildkite-codebuild-poc" -f ~/.ssh/buildkite-codebuild-poc

When generating a key, do not add a passphrase as this key will be used by the Buildkite agent.

If created successfully, it should output both the public key ~/.ssh/buildkite-codebuild-poc.pub and the private key ~/.ssh/buildkite-codebuild-poc.pub.

You first need to add the public key to your private GitHub repository as a deploy key. You can find this under your private repository settings.

The private key itself will need to be added as a secret for your Buildkite agent.

Setting the private deploy key as a Buildkite agent secret

Within Buildkite, there are domain-specific terms for clusters, pipelines, agents and queues.

While I won't be deep diving into these terms, please note that will be touching on all of these at some point.

To add your private key as a secret, head to the Buildkite dashboard and then go to your selected agent cluster. If you do not have one configured, select your default cluster.

From within your cluster, there will be a setting option for Secrets. Add your secret under our very apt key name SUPER_SECRET_SSH_KEY. This will map across to our Buildkite pipeline configuration we add later.

Getting an agent token for your Buildkite cluster

Next, you will need to configure an agent token. This token while be used by AWS components in order for communication between CodeBuild and Buildkite.

Within your same selected agent cluster, there will be a setting option for Agent Tokens. Create an agent token from within this UI.

For agent tokens, you can also lock down which inbound IP addresses are allowed. For this post, we will just allow all IP addresses for now.

Creating a Buildkite pipeline

Follow the Buildkite Pipelines UI and create a new pipeline. As part of this process, you will need to select your private GitHub repository to use.

Be sure that you select the pipeline to run from the agent that you have already configured with the secrets.

When creating, be sure to leave "auto-create webhooks" on so that Buildkite will create the webhooks for you within the Github repository.

As for the section about the "script" to run, add the following YAML:

steps:
  - label: ":aws: hello from codebuild"
    command: ./scripts/hello-world.sh
    plugins:
      - git-ssh-checkout#v0.4.1:
          ssh-secret-key-name: SUPER_SECRET_SSH_KEY
          repository-url: git@github.com:okeeffed/blog-codebuild-buildkite.git
    agents:
      queue: default-queue
      project: codebuild-buildkite-codebuild-poc-runner

Albeit we haven't wrote any code yet, the pipeline code about does the following:

• Runs a script from ./scripts/hello-world.sh.
• Makes use of a plugin git-ssh-checkout which makes use of our deploy key to authorize the pipeline to pull down the codebase.
• Sets the agent to run on the default-queue where we expect the project name to be codebuild-buildkite-codebuild-poc-runner.

At this point, our Buildkite configuration is ready to go, but we need to configure AWS to run.

Writing the CodeBuild Foundation CDK stack

For our configuration with the AWS CDK, we will be deploying two stacks:

1. A "foundation" stack which sets up our AWS SecretsManager secret as well as an AWS CodeConnection which authorizes CodeBuild with GitHub.
2. A "runner" stack which will create our AWS CodeBuild build service.

First, we need to configure the foundation stack. I will be doing so in TypeScript.

If you have not yet configured it, please follow the AWS CDK guide to initializing a stack.

Within lib/buildkite-foundation-stack.ts, add the following code:

import * as cdk from "aws-cdk-lib";
import * as codeconnections from "aws-cdk-lib/aws-codeconnections";
import * as secretsmanager from "aws-cdk-lib/aws-secretsmanager";
import * as ssm from "aws-cdk-lib/aws-ssm";
import { Construct } from "constructs";

const buildkiteAgentTokenSecretName =
  "buildkite-codebuild-poc/buildkite-agent-token";
export const codeConnectionsArnParameterName =
  "/buildkite-codebuild-poc/codeconnections-arn";

export class BuildkiteFoundationStack extends cdk.Stack {
  public readonly buildkiteAgentTokenSecretName = buildkiteAgentTokenSecretName;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const buildkiteAgentToken = new cdk.CfnParameter(
      this,
      "BuildkiteAgentToken",
      {
        type: "String",
        noEcho: true,
        description: "Buildkite agent token from your Buildkite cluster.",
      }
    );

    const existingCodeConnectionsArn = new cdk.CfnParameter(
      this,
      "ExistingCodeConnectionsArn",
      {
        type: "String",
        default: "",
        description:
          "Optional existing authorized AWS CodeConnections ARN. Leave empty to let this stack create the connection resource.",
      }
    );

    const createCodeConnection = new cdk.CfnParameter(
      this,
      "CreateCodeConnection",
      {
        type: "String",
        default: "true",
        allowedValues: ["true", "false"],
        description:
          "Set to true to create the CodeConnections resource in this stack. You must still authorize it in the AWS console.",
      }
    );

    const codeConnectionName = new cdk.CfnParameter(
      this,
      "CodeConnectionName",
      {
        type: "String",
        default: "buildkite-github",
        description: "Name to use when CreateCodeConnection=true.",
      }
    );

    const useManagedCodeConnection = new cdk.CfnCondition(
      this,
      "UseManagedCodeConnection",
      {
        expression: cdk.Fn.conditionEquals(
          createCodeConnection.valueAsString,
          "true"
        ),
      }
    );

    const agentTokenSecret = new secretsmanager.CfnSecret(
      this,
      "BuildkiteAgentTokenSecret",
      {
        name: buildkiteAgentTokenSecretName,
        description:
          "Buildkite agent token for the CodeBuild-hosted Buildkite runner proof of concept.",
        secretString: buildkiteAgentToken.valueAsString,
      }
    );

    const managedCodeConnection = new codeconnections.CfnConnection(
      this,
      "ManagedCodeConnection",
      {
        connectionName: codeConnectionName.valueAsString,
        providerType: "GitHub",
      }
    );
    managedCodeConnection.cfnOptions.condition = useManagedCodeConnection;

    // A CodeConnections resource can be provisioned by CloudFormation, but the
    // GitHub authorization handshake still needs to be completed in the console.
    const effectiveCodeConnectionsArn = cdk.Fn.conditionIf(
      useManagedCodeConnection.logicalId,
      managedCodeConnection.attrConnectionArn,
      existingCodeConnectionsArn.valueAsString
    ).toString();
    const buildkiteAgentTokenSecretArn =
      secretsmanager.CfnSecret.arnForSecret(agentTokenSecret);

    new ssm.StringParameter(this, "CodeConnectionsArnParameter", {
      parameterName: codeConnectionsArnParameterName,
      stringValue: effectiveCodeConnectionsArn,
      description:
        "CodeConnections ARN used by the Buildkite CodeBuild runner stack.",
    });

    new cdk.CfnOutput(this, "BuildkiteAgentTokenSecretName", {
      value: this.buildkiteAgentTokenSecretName,
      description:
        "Secrets Manager secret that stores the Buildkite agent token.",
    });

    new cdk.CfnOutput(this, "BuildkiteAgentTokenSecretArn", {
      value: buildkiteAgentTokenSecretArn,
      description: "Secrets Manager ARN for the Buildkite agent token secret.",
    });

    new cdk.CfnOutput(this, "CodeConnectionsArnOutput", {
      value: effectiveCodeConnectionsArn,
      description:
        "CodeConnections ARN for the repository provider used by the runner project.",
    });

    new cdk.CfnOutput(this, "CodeConnectionsArnParameterName", {
      value: codeConnectionsArnParameterName,
      description:
        "SSM parameter name that stores the CodeConnections ARN for the runner stack.",
    });

    new cdk.CfnOutput(this, "ManagedCodeConnectionArn", {
      value: managedCodeConnection.attrConnectionArn,
      description:
        "Authorize this connection in the AWS console before deploying the runner stack.",
      condition: useManagedCodeConnection,
    });

    new cdk.CfnOutput(this, "ManagedCodeConnectionStatus", {
      value: managedCodeConnection.attrConnectionStatus,
      description:
        "The managed CodeConnections status. It remains pending until the GitHub authorization is completed.",
      condition: useManagedCodeConnection,
    });
  }
}

As mentioned previously, this code will set up our Buildkite agent secret within AWS SecretsManager and also prepare our CodeConnection to GitHub.

The latter requires a manual step, so we will need to deploy this stack and then jump into the AWS dashboard to finish off the authorization.

Before doing that, we will finish off our "runner" stack as well.

Writing the CodeBuild Runner CDK stack

Within lib/codebuild-runner-stack.ts, add the following code:

import * as cdk from "aws-cdk-lib";
import * as codebuild from "aws-cdk-lib/aws-codebuild";
import * as codeconnections from "aws-cdk-lib/aws-codeconnections";
import * as cr from "aws-cdk-lib/custom-resources";
import * as iam from "aws-cdk-lib/aws-iam";
import * as secretsmanager from "aws-cdk-lib/aws-secretsmanager";
import { Construct } from "constructs";

const projectName = "buildkite-codebuild-poc-runner";
const buildkiteAgentTag = `codebuild-${projectName}`;
const buildkiteAgentTokenSecretName =
  "buildkite-codebuild-poc/buildkite-agent-token";

export class BuildkiteCodebuildPocStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const buildkiteAgentToken = new cdk.CfnParameter(
      this,
      "BuildkiteAgentToken",
      {
        type: "String",
        noEcho: true,
        description: "Buildkite agent token from your Buildkite cluster.",
      }
    );

    const codeConnectionsArn = new cdk.CfnParameter(
      this,
      "CodeConnectionsArn",
      {
        type: "String",
        default: "",
        description:
          "AWS CodeConnections ARN for the repository provider used by the Buildkite runner project.",
      }
    );

    const createCodeConnection = new cdk.CfnParameter(
      this,
      "CreateCodeConnection",
      {
        type: "String",
        default: "false",
        allowedValues: ["true", "false"],
        description:
          "Set to true to create the CodeConnections resource in this stack. You must still authorize it in the AWS console.",
      }
    );

    const codeConnectionName = new cdk.CfnParameter(
      this,
      "CodeConnectionName",
      {
        type: "String",
        default: "buildkite-github",
        description: "Name to use when CreateCodeConnection=true.",
      }
    );

    const useManagedCodeConnection = new cdk.CfnCondition(
      this,
      "UseManagedCodeConnection",
      {
        expression: cdk.Fn.conditionEquals(
          createCodeConnection.valueAsString,
          "true"
        ),
      }
    );

    const managedCodeConnection = new codeconnections.CfnConnection(
      this,
      "ManagedCodeConnection",
      {
        connectionName: codeConnectionName.valueAsString,
        providerType: "GitHub",
      }
    );
    managedCodeConnection.cfnOptions.condition = useManagedCodeConnection;

    // A CodeConnections resource can be provisioned by CloudFormation, but the
    // GitHub authorization handshake still needs to be completed in the console.
    const effectiveCodeConnectionsArn = cdk.Fn.conditionIf(
      useManagedCodeConnection.logicalId,
      managedCodeConnection.attrConnectionArn,
      codeConnectionsArn.valueAsString
    ).toString();

    // The console-created runner project grants permissions against both the
    // newer codeconnections ARN form and the legacy codestar-connections ARN form.
    const connectionId = cdk.Fn.select(
      1,
      cdk.Fn.split("connection/", effectiveCodeConnectionsArn)
    );
    const codestarConnectionsArn = cdk.Stack.of(this).formatArn({
      service: "codestar-connections",
      resource: "connection",
      resourceName: connectionId,
    });

    const agentTokenSecret = new secretsmanager.CfnSecret(
      this,
      "BuildkiteAgentTokenSecret",
      {
        name: buildkiteAgentTokenSecretName,
        description:
          "Buildkite agent token for the CodeBuild-hosted Buildkite runner proof of concept.",
        secretString: buildkiteAgentToken.valueAsString,
      }
    );

    const serviceRole = new iam.Role(this, "CodeBuildServiceRole", {
      assumedBy: new iam.ServicePrincipal("codebuild.amazonaws.com"),
      description:
        "Service role for the Buildkite runner proof of concept CodeBuild project.",
    });

    // The runner project needs the usual CodeBuild log permissions, read access
    // to the Buildkite agent token secret, and CodeConnections access so CodeBuild
    // can validate and use the linked repository connection.
    const runnerAccessPolicy = new iam.Policy(
      this,
      "CodeBuildRunnerAccessPolicy",
      {
        statements: [
          new iam.PolicyStatement({
            actions: [
              "logs:CreateLogGroup",
              "logs:CreateLogStream",
              "logs:PutLogEvents",
            ],
            resources: ["*"],
          }),
          new iam.PolicyStatement({
            actions: ["secretsmanager:GetSecretValue"],
            resources: [agentTokenSecret.attrId],
          }),
          new iam.PolicyStatement({
            // CodeBuild validates the connection during project updates and uses
            // the connection at job runtime, so it needs more than just UseConnection.
            actions: [
              "codeconnections:GetConnection",
              "codeconnections:GetConnectionToken",
              "codeconnections:UseConnection",
              "codestar-connections:GetConnection",
              "codestar-connections:GetConnectionToken",
              "codestar-connections:UseConnection",
            ],
            resources: [effectiveCodeConnectionsArn, codestarConnectionsArn],
          }),
        ],
      }
    );
    serviceRole.attachInlinePolicy(runnerAccessPolicy);

    const runnerProject = new codebuild.Project(
      this,
      "BuildkiteRunnerProject",
      {
        projectName,
        description:
          "Proof-of-concept CodeBuild project that runs Buildkite jobs in an ephemeral CodeBuild runner.",
        role: serviceRole,
        // Runner builds ignore the project buildspec unless explicitly overridden,
        // but NO_SOURCE projects still require an inline buildspec in the API model.
        buildSpec: codebuild.BuildSpec.fromObject({
          version: "0.2",
          phases: {
            build: {
              commands: [
                'echo "Buildkite runner project uses the managed runner build flow"',
              ],
            },
          },
        }),
        environment: {
          computeType: codebuild.ComputeType.SMALL,
          buildImage: codebuild.LinuxBuildImage.STANDARD_6_0,
        },
        environmentVariables: {
          CODEBUILD_CONFIG_BUILDKITE_AGENT_TOKEN: {
            type: codebuild.BuildEnvironmentVariableType.SECRETS_MANAGER,
            value: buildkiteAgentTokenSecretName,
          },
        },
        queuedTimeout: cdk.Duration.minutes(30),
        timeout: cdk.Duration.minutes(30),
        visibility: codebuild.ProjectVisibility.PRIVATE,
      }
    );

    const runnerProjectResource = runnerProject.node
      .defaultChild as codebuild.CfnProject;
    // Buildkite runner projects created in the console still use NO_SOURCE, but
    // they also carry CodeConnections-backed source auth and clone depth settings.
    // Without these source properties, CreateWebhook(RUNNER_BUILDKITE_BUILD) fails.
    runnerProjectResource.addPropertyOverride("Source.Auth", {
      Type: "CODECONNECTIONS",
      Resource: effectiveCodeConnectionsArn,
    });
    runnerProjectResource.addPropertyOverride("Source.GitCloneDepth", 1);
    runnerProjectResource.addPropertyOverride("Source.BuildSpec", "");

    runnerProject.node.addDependency(agentTokenSecret);
    // Updating the project to use CodeConnections can fail if CloudFormation
    // applies the project update before the runner access policy exists.
    runnerProject.node.addDependency(runnerAccessPolicy);

    const runnerWebhook = new cr.AwsCustomResource(
      this,
      "BuildkiteRunnerWebhook",
      {
        installLatestAwsSdk: false,
        onCreate: {
          service: "CodeBuild",
          action: "createWebhook",
          parameters: {
            projectName,
            buildType: "RUNNER_BUILDKITE_BUILD",
            filterGroups: [[{ type: "EVENT", pattern: "WORKFLOW_JOB_QUEUED" }]],
            // The console includes a default PR approval policy for GitHub-backed
            // Buildkite runner webhooks. Without this field, CreateWebhook returns
            // a generic InvalidInputException for an otherwise valid project.
            pullRequestBuildPolicy: {
              requiresCommentApproval: "ALL_PULL_REQUESTS",
              approverRoles: [
                "GITHUB_WRITE",
                "GITHUB_MAINTAIN",
                "GITHUB_ADMIN",
              ],
            },
          },
          physicalResourceId: cr.PhysicalResourceId.of(
            `${projectName}-buildkite-webhook`
          ),
        },
        onUpdate: {
          service: "CodeBuild",
          action: "updateWebhook",
          parameters: {
            projectName,
            buildType: "RUNNER_BUILDKITE_BUILD",
            filterGroups: [[{ type: "EVENT", pattern: "WORKFLOW_JOB_QUEUED" }]],
            pullRequestBuildPolicy: {
              requiresCommentApproval: "ALL_PULL_REQUESTS",
              approverRoles: [
                "GITHUB_WRITE",
                "GITHUB_MAINTAIN",
                "GITHUB_ADMIN",
              ],
            },
          },
          physicalResourceId: cr.PhysicalResourceId.of(
            `${projectName}-buildkite-webhook`
          ),
        },
        onDelete: {
          service: "CodeBuild",
          action: "deleteWebhook",
          parameters: {
            projectName,
          },
          ignoreErrorCodesMatching: "ResourceNotFoundException",
        },
        policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
          resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
        }),
      }
    );

    runnerWebhook.node.addDependency(runnerProject);

    new cdk.CfnOutput(this, "CodeBuildProjectName", {
      value: projectName,
      description:
        "The CodeBuild project name referenced from Buildkite agent tags.",
    });

    new cdk.CfnOutput(this, "BuildkiteAgentTagProjectValue", {
      value: buildkiteAgentTag,
      description: "Use this value in Buildkite YAML as agents.project.",
    });

    new cdk.CfnOutput(this, "BuildkiteWebhookPayloadUrl", {
      value: runnerWebhook.getResponseField("webhook.payloadUrl"),
      description:
        "Copy this into the Buildkite organization webhook URL field.",
    });

    new cdk.CfnOutput(this, "BuildkiteWebhookSecret", {
      value: runnerWebhook.getResponseField("webhook.secret"),
      description:
        "Copy this into the Buildkite organization webhook token field.",
    });

    new cdk.CfnOutput(this, "BuildkiteAgentTokenSecretName", {
      value: buildkiteAgentTokenSecretName,
      description:
        "Secrets Manager secret that stores the Buildkite agent token.",
    });

    new cdk.CfnOutput(this, "CodeConnectionsArnOutput", {
      value: effectiveCodeConnectionsArn,
      description:
        "CodeConnections ARN attached to the Buildkite runner project source auth.",
    });

    new cdk.CfnOutput(this, "ManagedCodeConnectionArn", {
      value: managedCodeConnection.attrConnectionArn,
      description:
        "Authorize this connection in the AWS console before using it for the runner project.",
      condition: useManagedCodeConnection,
    });

    new cdk.CfnOutput(this, "ManagedCodeConnectionStatus", {
      value: managedCodeConnection.attrConnectionStatus,
      description:
        "The managed CodeConnections status. It will remain pending until the GitHub authorization is completed.",
      condition: useManagedCodeConnection,
    });
  }
}

There is a quite a lot going on here, so I will keep what it does as concise as possible:

• Provisions a CodeBuild project that runs ephemeral Buildkite agents to execute pipeline jobs
• Stores the Buildkite agent token securely in AWS Secrets Manager and injects it at runtime
• Supports GitHub integration via AWS CodeConnections (existing or optionally created)
• Creates an IAM role with permissions for logs, secrets access, and CodeConnections usage
• Configures CodeBuild to act as a Buildkite runner using CodeConnections-backed source auth
• Sets up a custom webhook (RUNNER_BUILDKITE_BUILD) to trigger builds from Buildkite events
• Ensures correct resource dependencies and deployment order for reliable setup
• Outputs key values (webhook URL/secret, project name, agent tag) for Buildkite configuration

Finally, we will need to update our bin/buildkite-codebuild-poc.ts file to have the following:

#!/usr/bin/env node
import * as cdk from "aws-cdk-lib";
import { BuildkiteFoundationStack } from "../lib/buildkite-foundation-stack";
import { BuildkiteRunnerStack } from "../lib/buildkite-runner-stack";

const app = new cdk.App();

const env = {
  account: process.env.CDK_DEFAULT_ACCOUNT,
  region: process.env.CDK_DEFAULT_REGION,
};

new BuildkiteFoundationStack(app, "BuildkiteFoundationStack", {
  env,
});

new BuildkiteRunnerStack(app, "BuildkiteRunnerStack", {
  env,
});

At this point, we are ready to deploy our AWS stacks and do our final configuration touches.

Deploying our CodeBuild Foundation CDK stack

To deploy our foundation stack, first be sure to obtain your Buildkite agent token that we created earlier.

Next, run the following:

$ npx cdk deploy BuildkiteFoundationStack --parameters BuildkiteAgentToken=<replace-with-your-agent-token>

If successful, this will give you outputs like the following:

Outputs:
BuildkiteFoundationStack.BuildkiteAgentTokenSecretArn = REDACTED
BuildkiteFoundationStack.BuildkiteAgentTokenSecretName = buildkite-codebuild-poc/buildkite-agent-token
BuildkiteFoundationStack.CodeConnectionsArnOutput = REDACTED
BuildkiteFoundationStack.CodeConnectionsArnParameterName = REDACTED
BuildkiteFoundationStack.ManagedCodeConnectionArn = REDACTED
BuildkiteFoundationStack.ManagedCodeConnectionStatus = PENDING
Stack ARN:
arn:aws:cloudformation:REDACTED:REDACTED:stack/BuildkiteFoundationStack/REDACTED

You will notice the ManagedCodeConnectionStatus is PENDING. In this state, you need to head to your AWS Dashboard and find the correlating code connection to authorize access.

I find this easiest when heading to the CloudFormation dashboard, finding the foundation stack we deployed and clicking through the resource link directly to the code connection page.

From there, you can find and click through to the pending code connection.

Select to complete the process and you will get a popup:

If successful, the connection should move to an available status:

Deploying our CodeBuild Runner CDK Stack

With this configured, we can next deploy our CodeBuild Runner stack with npx cdk deploy BuildkiteRunnerStack.

Once successful, you will get outputs like the following:

Outputs:
BuildkiteRunnerStack.BuildkiteAgentTagProjectValue = codebuild-buildkite-codebuild-poc-runner
BuildkiteRunnerStack.BuildkiteWebhookPayloadUrl = REDACTED
BuildkiteRunnerStack.BuildkiteWebhookSecret = REDACTED
BuildkiteRunnerStack.CodeBuildProjectName = buildkite-codebuild-poc-runner
BuildkiteRunnerStack.CodeConnectionsArnOutput = REDACTED
Stack ARN:
arn:aws:cloudformation:REDACTED:REDACTED:stack/BuildkiteRunnerStack/REDACTED

You will notice that the BuildkiteAgentTagProjectValue of codebuild-buildkite-codebuild-poc-runner matches what we configured earlier in our Buildkite pipeline YAML.

Add the CodeBuild webhook into Buildkite

Now that we have deployed the resources, you will need to copy the outputs BuildkiteWebhookSecret and BuildkiteWebhookPayloadUrl from the runner stack and add them into your Buildkite Notification Services configuration.

This is found from within your Buildkite organization settings.

Once there, select to add a webhook and set the payload URL as the webhook URL and set the token as the webhook secret. Make sure that it selected for sending the token as X-Buildkite-Token.

For the webhook, you only need to configure notifications for job.scheduled.

Finally, make sure that you set it to only send for our created pipeline.

Pushing our code

Everything should now be configured to run for our Buildkite + CodeBuild set up.

Push your code up to the repository and then check both your Buildkite dashboard as well as your CodeBuild dashboard to confirm everything is now working.

On Buildkite:

On CodeBuild:

Your GitHub status checks should now also be passing:

Success!

Conclusion

In this post, I recapped my proof-of-concept for getting Buildkite and AWS CodeBuild working together for a private GitHub repository.

We also used the AWS CDK to configure the AWS portion of the infrastructure.

For further ideas on this project:

• Clean up the CDK code further.
• Attempt a more complicated pipeline.
• Make use of different Docker containers for running the code.
• Use another IaC provider to configure the BuildKite portion of the code.
• Make use of the Buildkite SDK/CLI for management of the pipeline.

In future posts, I may recap some of these ideas in my own implementation.

Links and Further Reading

• Buildkite
• AWS CodeBuild
• Tutorial: Configure a CodeBuild-hosted Buildkite runner
• Buildkite Git SSH Checkout plugin

Photo credit: johnnyb803

]]>

It's extensible nature means that we can configure our own models. This is where AWS Bedrock, an AWS offering for foundational models and agent tools, comes in.

Using AWS Bedrock with opencode

opencode (v1.0.68) will automatically enable AWS Bedrock model selection within the CLI if your terminal environment exports the expected enviornment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION). Personally, I make use of aws-vault for managing this.

AWS Bedrock Foundational models

For some foundational models (such as Claude models), you need to head to Bedrock on the AWS console and submit the use cases for Anthropic.

This is a requirement for usage of those models in opencode.

Links and Further Reading

• opencode
• AWS Bedrock
• aws-vault

Photo credit: johnnyb803

The Functional Optionals Pattern

According to the book, an example of the pattern is as follows:

type options struct {
    port *int
}

type Option func(options *options) error

func WithPort(port int) Option {
    return func(options *options) error {
        if port < 0 {
            return errors.New("port should be positive")
        }
        options.port = &port
        return nil
    }
}

// Usage example:
server, err := httplib.NewServer(
    "localhost",
    httplib.WithPort(8080),
    httplib.WithTimeout(time.Second),
)

An overview for the main idea behind the Functional Options Pattern is this (taken directly from the source):

1. An unexported struct holds the configuration: options.
2. Each option is a function that returns the same type: type Option func(options *options) error. For example, WithPort accepts an int argument that represents the port and returns an Option type that represents how to update the options struct.

This enables us to construct a NewServer while maintaining some flexibility and being wary of Golang Zero values:

func NewServer(addr string, opts ...Option) (*http.Server, error) {
    var options options

    for _, opt := range opts {
        err := opt(&options)
        if err != nil {
            return nil, err
        }
    }

    // At this stage, the options struct is built and contains the config.
    // Therefore, we can implement our logic related to port configuration.
    var port int

    if options.port == nil {
        port = defaultHTTPPort
    } else {
        if *options.port == 0 {
            port = randomPort()
        } else {
            port = *options.port
        }
    }

    // ...
}

The first part of the "constructor" function applies the options, the second part handles an nil values to set defaults.

An interface-based option type

Wanting to understand more about "flexibility" with constructors in Golang, I looked to some popular open source projects and saw an approach with gRPC-Go and Uber's Zap logger that look to make use of an interface that has an apply function defined within it.

Interestingly enough, they both had an approach that made use of an interface with an apply method defined.

We'll show both case studies before refactoring the httplib example that we've defined so far.

type DialOption interface {
    apply(*dialOptions)
}
// and most options are implemented as functions that modify *dialOptions

conn, err := grpc.NewClient(
    "localhost:50051",
    grpc.WithTransportCredentials(creds),
    grpc.WithBlock(),
    grpc.WithUnaryInterceptor(myInterceptor),
)

func NewClient(target string, opts ...DialOption) (conn *ClientConn, err error) {
	cc := &ClientConn{
		target: target,
		conns:  make(map[*addrConn]struct{}),
		dopts:  defaultDialOptions(),
	}

	cc.retryThrottler.Store((*retryThrottler)(nil))
	cc.safeConfigSelector.UpdateConfigSelector(&defaultConfigSelector{nil})
	cc.ctx, cc.cancel = context.WithCancel(context.Background())

	// Apply dial options.
	disableGlobalOpts := false
	for _, opt := range opts {
		if _, ok := opt.(*disableGlobalDialOptions); ok {
			disableGlobalOpts = true
			break
		}
	}

	if !disableGlobalOpts {
		for _, opt := range globalDialOptions {
			opt.apply(&cc.dopts)
		}
	}

    // NOTE: We iterate through the range of opts and call their
    // `apply` method.
	for _, opt := range opts {
		opt.apply(&cc.dopts)
	}

	// ... code implementation omitted

	return cc, nil
}

// funcDialOption wraps a function that modifies dialOptions into an
// implementation of the DialOption interface.
type funcDialOption struct {
	f func(*dialOptions)
}

func (fdo *funcDialOption) apply(do *dialOptions) {
	fdo.f(do)
}

func newFuncDialOption(f func(*dialOptions)) *funcDialOption {
	return &funcDialOption{
		f: f,
	}
}

func WithBlock() DialOption {
	return newFuncDialOption(func(o *dialOptions) {
		o.block = true
	})
}

// An Option configures a Logger.
type Option interface {
	apply(*Logger)
}

// optionFunc wraps a func so it satisfies the Option interface.
type optionFunc func(*Logger)

func (f optionFunc) apply(log *Logger) {
	f(log)
}

func New(core zapcore.Core, options ...Option) *Logger {
	if core == nil {
		return NewNop()
	}
	log := &Logger{
		core:        core,
		errorOutput: zapcore.Lock(os.Stderr),
		addStack:    zapcore.FatalLevel + 1,
		clock:       zapcore.DefaultClock,
	}
	return log.WithOptions(options...)
}


// WithOptions clones the current Logger, applies the supplied Options, and
// returns the resulting Logger. It's safe to use concurrently.
func (log *Logger) WithOptions(opts ...Option) *Logger {
	c := log.clone()
	for _, opt := range opts {
		opt.apply(c)
	}
	return c
}

// optionFunc wraps a func so it satisfies the Option interface.
type optionFunc func(*Logger)

func (f optionFunc) apply(log *Logger) {
	f(log)
}

// ... an example option
// WithClock specifies the clock used by the logger to determine the current
// time for logged entries. Defaults to the system clock with time.Now.
func WithClock(clock zapcore.Clock) Option {
	return optionFunc(func(log *Logger) {
		log.clock = clock
	})
}

type options struct {
    port *int
    // timeout *time.Duration, etc...
}

// Option configures how we set up the server.
type Option interface {
    apply(*options) error
}

// optionFunc wraps a func so it satisfies the Option interface.
type optionFunc func(*options) error

func (f optionFunc) apply(o *options) error {
    return f(o)
}

func WithPort(port int) Option {
    return optionFunc(func(o *options) error {
        if port < 0 {
            return errors.New("port should be positive")
        }
        o.port = &port
        return nil
    })
}

func NewServer(addr string, opts ...Option) (*http.Server, error) {
    var o options

    for _, opt := range opts {
        if err := opt.apply(&o); err != nil {
            return nil, err
        }
    }

    // existing port resolution logic...
}

// Usage example:
server, err := httplib.NewServer(
    "localhost",
    httplib.WithPort(8080),
    // ... other supported options
    // httplib.WithTimeout(time.Second),
)

package main

import (
	"errors"
	"fmt"
)

type options struct {
	port *int
	// timeout *time.Duration, etc...
}

// Option configures how we set up the server.
type Option interface {
	apply(*options) error
}

// optionFunc wraps a func so it satisfies the Option interface.
type optionFunc func(*options) error

func (f optionFunc) apply(o *options) error {
	return f(o)
}

func WithPort(port int) Option {
	return optionFunc(func(o *options) error {
		if port < 0 {
			return errors.New("port should be positive")
		}
		o.port = &port
		return nil
	})
}

type Server struct {
	addr string
	port int
}

func NewServer(addr string, opts ...Option) (*Server, error) {
	var o options

	for _, opt := range opts {
		if err := opt.apply(&o); err != nil {
			return nil, err
		}
	}

	srv := &Server{
		addr: addr,
		port: *o.port,
	}

	return srv, nil
}

func main() {
	fmt.Println("=== Example 1: Valid usage ===")

	srv, err := NewServer("localhost",
		WithPort(8080),
	)
	if err != nil {
		fmt.Println("unexpected error:", err)
		return
	}
	fmt.Printf("Server struct: %+v\n\n", srv)

	fmt.Println("=== Example 2: Invalid usage (error) ===")

	srv2, err := NewServer("localhost",
		WithPort(-5), // invalid: negative port
	)
	if err != nil {
		fmt.Println("expected error:", err)
		return
	}
	fmt.Printf("Server struct: %+v\n", srv2)
}

$ go run main.go
=== Example 1: Valid usage ===
Server struct: &{addr:localhost port:8080}

=== Example 2: Invalid usage (error) ===
expected error: port should be positive

Prerequisites

• Swift (this post uses Swift 6.1).

Initializing a Swift executable package

You can create a new Swift executable package with the following (from the command line):

$ mkdir hello-swift
$ cd hello-swift
$ swift package init --type executable

You can run swift package init --help to get more information around arguments like --type. For example:

swift package init --help
OVERVIEW: Initialize a new package

USAGE: swift package init [--type <type>] [--enable-xctest] [--disable-xctest] [--enable-swift-testing] [--disable-swift-testing] [--name <name>]

OPTIONS:
  --type <type>           Package type: (default: library)
        library           - A package with a library.
        executable        - A package with an executable.
        tool              - A package with an executable that uses
                            Swift Argument Parser. Use this template if you
                            plan to have a rich set of command-line arguments.
        build-tool-plugin - A package that vends a build tool plugin.
        command-plugin    - A package that vends a command plugin.
        macro             - A package that vends a macro.
        empty             - An empty package with a Package.swift manifest.
  --enable-xctest/--disable-xctest
                          Enable support for XCTest
  --enable-swift-testing/--disable-swift-testing
                          Enable support for Swift Testing
  --name <name>           Provide custom package name
  --version               Show the version.
  -h, -help, --help       Show help information.

With this new package initialized as an executable, you can navigate to Sources/main.swift to see the generated entry file:

/ The Swift Programming Language
// https://docs.swift.org/swift-book

print("Hello, world!")

Running our executable

You can make use of swift run to run our executable:

$ swift run
[1/1] Planning build
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'hello-swift' complete! (0.19s)
Hello, world!

We can re-use this to run after any file changes:

$ swift run
Building for debugging...
[7/7] Applying hello-swift
Build of product 'hello-swift' complete! (1.01s)
Hello again, world!

If you have a program installed such as watchexec, you can set it to watch swift extension files with watchexec --exts swift -- swift run :

$ watchexec --exts swift -- swift run
[Running: swift run]
[1/1] Planning build
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'hello-swift' complete! (0.20s)
Hello again, world!
[Command was successful]
[Running: swift run]
[1/1] Planning build
Building for debugging...
[7/7] Applying hello-swift
Build of product 'hello-swift' complete! (0.73s)
Hello again and again, world!
[Command was successful]

Building files

You can run swift build from the project root to build out the executable.

$ swift build
[1/1] Planning build
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build complete! (0.23s)

By default, swift build will build with debug configuration.

Outputs are placed in .build:

$ tree .build -L 3
.build
├── arm64-apple-macosx
│   └── debug
│       ├── description.json
│       ├── hello_swift.build
│       ├── hello-swift
│       ├── hello-swift-entitlement.plist
│       ├── hello-swift.dSYM
│       ├── hello-swift.product
│       ├── index
│       ├── ModuleCache
│       ├── Modules
│       ├── plugin-tools-description.json
│       └── swift-version--58304C5D6DBC2206.txt
├── artifacts
├── build.db
├── checkouts
├── debug -> arm64-apple-macosx/debug
├── debug.yaml
├── plugin-tools.yaml
├── repositories
└── workspace-state.json

For an optimized build, pass --configuration release:

$ swift build --configuration release
Building for production...
[5/5] Linking hello-swift
Build complete! (1.21s)
$ tree .build -L 3
.build
├── arm64-apple-macosx
│   ├── debug
│   │   ├── description.json
│   │   ├── hello_swift.build
│   │   ├── hello-swift
│   │   ├── hello-swift-entitlement.plist
│   │   ├── hello-swift.dSYM
│   │   ├── hello-swift.product
│   │   ├── index
│   │   ├── ModuleCache
│   │   ├── Modules
│   │   ├── plugin-tools-description.json
│   │   └── swift-version--58304C5D6DBC2206.txt
│   └── release
│       ├── description.json
│       ├── hello_swift.build
│       ├── hello-swift
│       ├── hello-swift.dSYM
│       ├── hello-swift.product
│       ├── ModuleCache
│       ├── Modules
│       ├── plugin-tools-description.json
│       └── swift-version--58304C5D6DBC2206.txt
├── artifacts
├── build.db
├── checkouts
├── debug -> arm64-apple-macosx/debug
├── debug.yaml
├── plugin-tools.yaml
├── release -> arm64-apple-macosx/release
├── release.yaml
├── repositories
└── workspace-state.json

20 directories, 14 files

Once compiled, you'll have an optimized binary you can run directly (if your platform is the correct target):

$ ./.build/release/hello-swift
Hello again and again, world!

Other useful flags include --triple to cross-compile for another platform.

Makefile

You can make use of a Makefile for common operations:

.PHONY: build run clean release test

build:
	swift build

release:
	swift build -c release

run:
	swift run

test:
	swift test

clean:
	swift package clean

In use:

$ make clean
swift package clean

$ make build
swift build
Building for debugging...
[8/8] Applying hello-swift
Build complete! (1.06s)

$ make run
swift run
[1/1] Planning build
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'hello-swift' complete! (0.21s)
Hello again and again, world!

Managing packages

You can manage packages in Package.swift or via the command line.

The default file that was created:

// Package.swift
// swift-tools-version: 6.1
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "hello-swift",
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "hello-swift"),
    ]
)

From the command line, you can install a package using swift package add-dependency <URL>:

$ swift package add-dependency https://github.com/Alamofire/Alamofire --from 5.10.2
Updating package manifest at Package.swift... done.

In the above case, the package is installed via the GitHub repository for the popular networking package Alamofire.

You must also provide a version.

$ swift package add-dependency https://github.com/Alamofire/Alamofire
error: must specify one of --exact, --branch, --revision, --from, or --up-to-next-minor-from

Once installed, your Package.swift file will be updated:

// swift-tools-version: 6.1
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "hello-swift",
    dependencies: [
        .package(url: "https://github.com/Alamofire/Alamofire", from: "5.10.2"),
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "hello-swift"),
    ]
)

In order to use it, you must update the relevant targets:

// swift-tools-version: 6.1
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "hello-swift",
    dependencies: [
        .package(url: "https://github.com/Alamofire/Alamofire", from: "5.10.2"),
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "hello-swift",
            dependencies: [
                "Alamofire"   // 👈 add this line
            ]),
    ]
)

We can see this in action by updating our main.swift file:

// Sources/main.swift
// The Swift Programming Language
// https://docs.swift.org/swift-book

import Alamofire
import Foundation

@main
struct Hello {
    static func main() {
        AF.request("https://jsonplaceholder.typicode.com/todos/1")
            .responseString { response in
                print(response.value ?? "no data")
                CFRunLoopStop(CFRunLoopGetMain()) // stop loop when done
            }

        // keep the program alive until CFRunLoopStop is called
        CFRunLoopRun()
    }
}

Let's see it in action:

$ swift run
Fetching https://github.com/Alamofire/Alamofire
Fetched https://github.com/Alamofire/Alamofire from cache (22.64s)
Computing version for https://github.com/Alamofire/Alamofire
Computed https://github.com/Alamofire/Alamofire at 5.10.2 (23.82s)
Creating working copy for https://github.com/Alamofire/Alamofire
Working copy of https://github.com/Alamofire/Alamofire resolved at 5.10.2
[1/1] Planning build
Building for debugging...
[52/52] Applying hello-swift
Build of product 'hello-swift' complete! (6.15s)
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}

Perfect, we got our response from an installed package.

Conclusion

This post was recounting some baby steps with initializing Swift packages. I will likely be writing a few tools for MacOS with this approach for learning more about the language while diving into iOS.

Links and Further Reading

• Alamofire
• watchexec
• swiftlang/swift-package-manager
• Swift Package Manager

Photo credit: johnnyb803

It explores initializing an executable, working through some simple Swift syntax and managing packages with SwiftPM (Swift Package Manager).

Prerequisites

• Swift (this post uses Swift 6.1).

Initializing a Swift executable package

You can create a new Swift executable package with the following (from the command line):

$ mkdir hello-swift
$ cd hello-swift
$ swift package init --type executable

You can run swift package init --help to get more information around arguments like --type. For example:

swift package init --help
OVERVIEW: Initialize a new package

USAGE: swift package init [--type <type>] [--enable-xctest] [--disable-xctest] [--enable-swift-testing] [--disable-swift-testing] [--name <name>]

OPTIONS:
  --type <type>           Package type: (default: library)
        library           - A package with a library.
        executable        - A package with an executable.
        tool              - A package with an executable that uses
                            Swift Argument Parser. Use this template if you
                            plan to have a rich set of command-line arguments.
        build-tool-plugin - A package that vends a build tool plugin.
        command-plugin    - A package that vends a command plugin.
        macro             - A package that vends a macro.
        empty             - An empty package with a Package.swift manifest.
  --enable-xctest/--disable-xctest
                          Enable support for XCTest
  --enable-swift-testing/--disable-swift-testing
                          Enable support for Swift Testing
  --name <name>           Provide custom package name
  --version               Show the version.
  -h, -help, --help       Show help information.

With this new package initialized as an executable, you can navigate to Sources/main.swift to see the generated entry file:

/ The Swift Programming Language
// https://docs.swift.org/swift-book

print("Hello, world!")

Running our executable

You can make use of swift run to run our executable:

$ swift run
[1/1] Planning build
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'hello-swift' complete! (0.19s)
Hello, world!

We can re-use this to run after any file changes:

$ swift run
Building for debugging...
[7/7] Applying hello-swift
Build of product 'hello-swift' complete! (1.01s)
Hello again, world!

If you have a program installed such as watchexec, you can set it to watch swift extension files with watchexec --exts swift -- swift run :

$ watchexec --exts swift -- swift run
[Running: swift run]
[1/1] Planning build
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'hello-swift' complete! (0.20s)
Hello again, world!
[Command was successful]
[Running: swift run]
[1/1] Planning build
Building for debugging...
[7/7] Applying hello-swift
Build of product 'hello-swift' complete! (0.73s)
Hello again and again, world!
[Command was successful]

Building files

You can run swift build from the project root to build out the executable.

$ swift build
[1/1] Planning build
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build complete! (0.23s)

By default, swift build will build with debug configuration.

Outputs are placed in .build:

$ tree .build -L 3
.build
├── arm64-apple-macosx
│   └── debug
│       ├── description.json
│       ├── hello_swift.build
│       ├── hello-swift
│       ├── hello-swift-entitlement.plist
│       ├── hello-swift.dSYM
│       ├── hello-swift.product
│       ├── index
│       ├── ModuleCache
│       ├── Modules
│       ├── plugin-tools-description.json
│       └── swift-version--58304C5D6DBC2206.txt
├── artifacts
├── build.db
├── checkouts
├── debug -> arm64-apple-macosx/debug
├── debug.yaml
├── plugin-tools.yaml
├── repositories
└── workspace-state.json

For an optimized build, pass --configuration release:

$ swift build --configuration release
Building for production...
[5/5] Linking hello-swift
Build complete! (1.21s)
$ tree .build -L 3
.build
├── arm64-apple-macosx
│   ├── debug
│   │   ├── description.json
│   │   ├── hello_swift.build
│   │   ├── hello-swift
│   │   ├── hello-swift-entitlement.plist
│   │   ├── hello-swift.dSYM
│   │   ├── hello-swift.product
│   │   ├── index
│   │   ├── ModuleCache
│   │   ├── Modules
│   │   ├── plugin-tools-description.json
│   │   └── swift-version--58304C5D6DBC2206.txt
│   └── release
│       ├── description.json
│       ├── hello_swift.build
│       ├── hello-swift
│       ├── hello-swift.dSYM
│       ├── hello-swift.product
│       ├── ModuleCache
│       ├── Modules
│       ├── plugin-tools-description.json
│       └── swift-version--58304C5D6DBC2206.txt
├── artifacts
├── build.db
├── checkouts
├── debug -> arm64-apple-macosx/debug
├── debug.yaml
├── plugin-tools.yaml
├── release -> arm64-apple-macosx/release
├── release.yaml
├── repositories
└── workspace-state.json

20 directories, 14 files

Once compiled, you'll have an optimized binary you can run directly (if your platform is the correct target):

$ ./.build/release/hello-swift
Hello again and again, world!

Other useful flags include --triple to cross-compile for another platform.

Makefile

You can make use of a Makefile for common operations:

.PHONY: build run clean release test

build:
	swift build

release:
	swift build -c release

run:
	swift run

test:
	swift test

clean:
	swift package clean

In use:

$ make clean
swift package clean

$ make build
swift build
Building for debugging...
[8/8] Applying hello-swift
Build complete! (1.06s)

$ make run
swift run
[1/1] Planning build
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'hello-swift' complete! (0.21s)
Hello again and again, world!

Managing packages

You can manage packages in Package.swift or via the command line.

The default file that was created:

// Package.swift
// swift-tools-version: 6.1
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "hello-swift",
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "hello-swift"),
    ]
)

From the command line, you can install a package using swift package add-dependency <URL>:

$ swift package add-dependency https://github.com/Alamofire/Alamofire --from 5.10.2
Updating package manifest at Package.swift... done.

In the above case, the package is installed via the GitHub repository for the popular networking package Alamofire.

You must also provide a version.

$ swift package add-dependency https://github.com/Alamofire/Alamofire
error: must specify one of --exact, --branch, --revision, --from, or --up-to-next-minor-from

Once installed, your Package.swift file will be updated:

// swift-tools-version: 6.1
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "hello-swift",
    dependencies: [
        .package(url: "https://github.com/Alamofire/Alamofire", from: "5.10.2"),
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "hello-swift"),
    ]
)

In order to use it, you must update the relevant targets:

// swift-tools-version: 6.1
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "hello-swift",
    dependencies: [
        .package(url: "https://github.com/Alamofire/Alamofire", from: "5.10.2"),
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "hello-swift",
            dependencies: [
                "Alamofire"   // 👈 add this line
            ]),
    ]
)

We can see this in action by updating our main.swift file:

// Sources/main.swift
// The Swift Programming Language
// https://docs.swift.org/swift-book

import Alamofire
import Foundation

@main
struct Hello {
    static func main() {
        AF.request("https://jsonplaceholder.typicode.com/todos/1")
            .responseString { response in
                print(response.value ?? "no data")
                CFRunLoopStop(CFRunLoopGetMain()) // stop loop when done
            }

        // keep the program alive until CFRunLoopStop is called
        CFRunLoopRun()
    }
}

Let's see it in action:

$ swift run
Fetching https://github.com/Alamofire/Alamofire
Fetched https://github.com/Alamofire/Alamofire from cache (22.64s)
Computing version for https://github.com/Alamofire/Alamofire
Computed https://github.com/Alamofire/Alamofire at 5.10.2 (23.82s)
Creating working copy for https://github.com/Alamofire/Alamofire
Working copy of https://github.com/Alamofire/Alamofire resolved at 5.10.2
[1/1] Planning build
Building for debugging...
[52/52] Applying hello-swift
Build of product 'hello-swift' complete! (6.15s)
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}

Perfect, we got our response from an installed package.

Conclusion

This post was recounting some baby steps with initializing Swift packages. I will likely be writing a few tools for MacOS with this approach for learning more about the language while diving into iOS.

Links and Further Reading

• Alamofire
• watchexec
• swiftlang/swift-package-manager
• Swift Package Manager

Photo credit: johnnyb803

Given the amount of time I spend working with TUIs like k9s, lazygit, yazi and dive, I decided the time is now ripe to start learning the ins and outs of TUI development.

This post will be a walkthrough of me testing out the waters and learning more about tview, a Golang package for building TUIs.

Prerequisites

This post expects that you've worked with Golang before.

I will be using go.1.24.5.

Getting started

Start by initialising a new Go module and fetching the required packages.

# Create a project directory
mkdir blog-tview-introduction
cd blog-tview-introduction

# Initialise the module
go mod init tview.dennisokeeffe.com

# Fetch the tview package
go get github.com/rivo/tview

# Create a main.go file
touch main.go

At this point, we are ready to get started.

Exploring tview

A great place to start getting to understand the package is the tview wiki and the pkg.go documentation page.

The wiki outlines a number of basic components that can be used for drawing out our TUI and we can reinforce some of what we learn by checking out the docs.

We can update main.go to start with the example Grid component:

package main

import (
 "github.com/rivo/tview"
)

func main() {
 newPrimitive := func(text string) tview.Primitive {
  return tview.NewTextView().
   SetTextAlign(tview.AlignCenter).
   SetText(text)
 }
 menu := newPrimitive("Menu")
 main := newPrimitive("Main content")
 sideBar := newPrimitive("Side Bar")

 grid := tview.NewGrid().
  SetRows(3, 0, 3).
  SetColumns(30, 0, 30).
  SetBorders(true).
  AddItem(newPrimitive("Header"), 0, 0, 1, 3, 0, 0, false).
  AddItem(newPrimitive("Footer"), 2, 0, 1, 3, 0, 0, false)

 // Layout for screens narrower than 100 cells (menu and side bar are hidden).
 grid.AddItem(menu, 0, 0, 0, 0, 0, 0, false).
  AddItem(main, 1, 0, 1, 3, 0, 0, false).
  AddItem(sideBar, 0, 0, 0, 0, 0, 0, false)

 // Layout for screens wider than 100 cells.
 grid.AddItem(menu, 1, 0, 1, 1, 0, 100, false).
  AddItem(main, 1, 1, 1, 1, 0, 100, false).
  AddItem(sideBar, 1, 2, 1, 1, 0, 100, false)

 if err := tview.NewApplication().SetRoot(grid, true).SetFocus(grid).Run(); err != nil {
  panic(err)
 }
}

Most of the above code should make sense just from a read through:

1. We built out a few primitives (more on this later) for the "Menu", "Main content" and "Side bar".
2. We built out a grid that adds a new "Header" and "Footer" primitive.
3. We add two different layouts based on available cell size.
4. We start the application with the grid as our root and focus.

We can visualize this TUI by running go run main.go:

As you may already have expected from the comments above, a smaller viewport for our terminal will change what is rendered:

The part of the code which might not make so much sense right now are the arguments for methods such as AddItem.

Let's walk through a few of these to start getting a sense for the package.

AddItem is a Grid method with the following definition:

func (g *tview.Grid) AddItem(p tview.Primitive, row int, column int, rowSpan int, colSpan int, minGridHeight int, minGridWidth int, focus bool) *tview.Grid

So reading an example like AddItem(newPrimitive("Header"), 0, 0, 1, 3, 0, 0, false) from left to right, we can understand the numerical arguments as:

1. Row 0.
2. Column 0.
3. Row span 1.
4. Column span 3.
5. Minimum height 0.
6. Minimum width 0.

We can assume that row span and column span relate to our default values, so exploring the SetRows and SetColumns method definitions can help us work backwards to understand the relationship:

func (g *tview.Grid) SetColumns(columns ...int) *tview.Grid
func (g *tview.Grid) SetRows(rows ...int) *tview.Grid

Both functions follow the same principle, so it's best to read an excerpt from the documentation:

SetColumns defines how the columns of the grid are distributed. Each value defines the size of one column, starting with the leftmost column. Values greater than 0 represent absolute column widths (gaps not included). Values less than or equal to 0 represent proportional column widths or fractions of the remaining free space, where 0 is treated the same as -1. That is, a column with a value of -3 will have three times the width of a column with a value of -1 (or 0). The minimum width set with SetMinSize() is always observed.

So in our definition of SetColumns(30, 0, 30) and SetRows(3, 0, 3), we are basically setting the values for 3 rows and 3 columns, with the 0 value being the remaining cell space after the first and last of each row/column are set.

We can understand this visually from the screenshots above. Basically, we can expect our footer to double in size if we bump the row size for the footer to 6.

Nice.

Making things interactive

From little things, big things grow. With that pearl of wisdom out of the way, let's add in a few more elements including a InputField to start to understand a bit more about interactivity.

package main

import (
 "github.com/gdamore/tcell/v2"
 "github.com/rivo/tview"
)

func main() {
 app := tview.NewApplication()
 newPrimitive := func(text string) tview.Primitive {
  return tview.NewTextView().
   SetTextAlign(tview.AlignCenter).
   SetText(text)
 }

 // NEW: Adding the inputField
 inputField := tview.NewInputField().
  SetLabel("Enter value: ").
  SetFieldWidth(0).
  SetFieldTextColor(tcell.ColorBlack).
  SetFieldBackgroundColor(tcell.ColorWhite).
  SetDoneFunc(func(key tcell.Key) {
   app.Stop()
  })

  // NEW: Adding the flex container
 header := tview.NewFlex().
  SetDirection(tview.FlexRow).
  AddItem(newPrimitive("Header"), 2, 0, false).
  AddItem(inputField, 0, 1, true)

 menu := newPrimitive("Menu")
 main := newPrimitive("Main content")
 sideBar := newPrimitive("Side Bar")

 grid := tview.NewGrid().
  SetRows(3, 0, 6).
  SetColumns(30, 0, 30).
  SetBorders(true).
  AddItem(header, 0, 0, 1, 3, 0, 0, false).
  AddItem(newPrimitive("Footer"), 2, 0, 1, 3, 0, 0, false)

 // Layout for screens narrower than 100 cells (menu and side bar are hidden).
 grid.AddItem(menu, 0, 0, 0, 0, 0, 0, false).
  AddItem(main, 1, 0, 1, 3, 0, 0, false).
  AddItem(sideBar, 0, 0, 0, 0, 0, 0, false)

 // Layout for screens wider than 100 cells.
 grid.AddItem(menu, 1, 0, 1, 1, 0, 100, false).
  AddItem(main, 1, 1, 1, 1, 0, 100, false).
  AddItem(sideBar, 1, 2, 1, 1, 0, 100, false)

 if err := app.SetRoot(grid, true).SetFocus(inputField).Run(); err != nil {
  panic(err)
 }
}

In our code above, we've updated the grid header code we had before to add in a new header variable that we've declared.

That header in itself is adding the primitive text and also the inputField variable which we've declared above.

You can start to see a pattern occurring now with the AddItem method also appearing with the tview.NewFlex component. The method in this case has a different method signature, but I'll omit going over it as you should begin to understand that you can check the docs/intellisense and get a feel for what numeric values change.

The current state of the TUI looks like so:

With the inputField declaration, we can see the SetDoneFunc method that we invoke which takes an anonymous function.

In our case, we've added app.Stop() as part of the anonymous function, so when we hit enter it will result in closing our application.

(From the docs) The callback function is provided with the key that was pressed, which is one of the following:

• KeyEnter: Done entering text.
• KeyEscape: Abort text input.
• KeyTab: Move to the next field.
• KeyBacktab: Move to the previous field.

We can make use of this to make our interactivity a little more interesting.

Let's abstract the main content into a new text view and update our SetDoneFunc anonymous function to update the text with the value added.

package main

import (
 "github.com/gdamore/tcell/v2"
 "github.com/rivo/tview"
)

func main() {
 app := tview.NewApplication()
 newPrimitive := func(text string) tview.Primitive {
  return tview.NewTextView().
   SetTextAlign(tview.AlignCenter).
   SetText(text)
 }

 // NEW: Adding the main content to be updated
 mainContent := tview.NewTextView().
  SetTextAlign(tview.AlignCenter).
  SetText("Main content - waiting for input...")

 inputField := tview.NewInputField().
  SetLabel("Enter value: ").
  SetFieldWidth(0).
  SetFieldTextColor(tcell.ColorBlack).
  SetFieldBackgroundColor(tcell.ColorWhite)

 inputField.SetDoneFunc(func(key tcell.Key) {
  if key == tcell.KeyEnter {
   inputValue := inputField.GetText()
   mainContent.SetText("You entered: " + inputValue)
   inputField.SetText("")
   app.SetFocus(inputField)
  }
 })

 header := tview.NewFlex().
  SetDirection(tview.FlexRow).
  AddItem(newPrimitive("Header"), 2, 0, false).
  AddItem(inputField, 0, 1, true)

 menu := newPrimitive("Menu")
 sideBar := newPrimitive("Side Bar")

 grid := tview.NewGrid().
  SetRows(3, 0, 6).
  SetColumns(30, 0, 30).
  SetBorders(true).
  AddItem(header, 0, 0, 1, 3, 0, 0, false).
  AddItem(newPrimitive("Footer"), 2, 0, 1, 3, 0, 0, false)

 // Layout for screens narrower than 100 cells (menu and side bar are hidden).
 grid.AddItem(menu, 0, 0, 0, 0, 0, 0, false).
  AddItem(mainContent, 1, 0, 1, 3, 0, 0, false).
  AddItem(sideBar, 0, 0, 0, 0, 0, 0, false)

 // Layout for screens wider than 100 cells.
 grid.AddItem(menu, 1, 0, 1, 1, 0, 100, false).
  AddItem(mainContent, 1, 1, 1, 1, 0, 100, false).
  AddItem(sideBar, 1, 2, 1, 1, 0, 100, false)

 if err := app.SetRoot(grid, true).SetFocus(inputField).Run(); err != nil {
  panic(err)
 }
}

Restarting our TUI, we can see what our UI looks like with text in the input:

And then after hitting the enter button:

Great! Our TUI is looking a little boring, but we are making some progress.

Changing view focus

At this point, our supremely basic TUI can now dynamically update text based on it's input.

The InputField is currently our initial focus, but how can we swap between different grid items and change focus?

We can introduce the List TUI component and make some changes based on things we've already worked with in order to get a simple list we can navigate up and down, while pressing : to focus on the filter bar and enter to re-enter the main list content.

We will also update the footer now to dynamically show different text help messages based on which component we are focused on.

package main

import (
 "strings"

 "github.com/gdamore/tcell/v2"
 "github.com/rivo/tview"
)

func main() {
 app := tview.NewApplication()
 newPrimitive := func(text string) tview.Primitive {
  return tview.NewTextView().
   SetTextAlign(tview.AlignCenter).
   SetText(text)
 }

 // Sample data to filter
 items := []string{"Apple", "Banana", "Cherry", "Date", "Elderberry", "Fig", "Grape", "Honeydew", "Kiwi", "Lemon", "Mango", "Orange"}

 // Create the filterable list
 list := tview.NewList().
  ShowSecondaryText(false).
  SetHighlightFullLine(true)

 // Function to populate/filter the list
 updateList := func(filter string) {
  list.Clear()
  for _, item := range items {
   if strings.Contains(strings.ToLower(item), strings.ToLower(filter)) {
    list.AddItem(item, "", 0, nil)
   }
  }
 }

 // Initially populate with all items
 updateList("")

 // Create dynamic footer for guidance text
 footer := tview.NewTextView().
  SetTextAlign(tview.AlignCenter).
  SetText("↑/↓ Navigate list • : Filter")

 // Function to update guidance text based on focus
 updateGuidance := func(focusedOnList bool) {
  if focusedOnList {
   footer.SetText("↑/↓ Navigate list • : Filter")
  } else {
   footer.SetText("Type to filter • Enter to focus list")
  }
 }

 inputField := tview.NewInputField().
  SetLabel("Filter: ").
  SetFieldWidth(0).
  SetFieldTextColor(tcell.ColorBlack).
  SetFieldBackgroundColor(tcell.ColorWhite)

 // Add real-time filtering as user types
 inputField.SetChangedFunc(func(text string) {
  updateList(text)
 })

 // On Enter, focus the list for navigation
 inputField.SetDoneFunc(func(key tcell.Key) {
  if key == tcell.KeyEnter {
   app.SetFocus(list)
   updateGuidance(true)
  }
 })

 // Add list navigation - return to input on selection
 list.SetDoneFunc(func() {
  app.SetFocus(inputField)
  updateGuidance(false)
 })

 // Handle keys for list: escape to return to input, ':' to focus filter
 list.SetInputCapture(func(event *tcell.EventKey) *tcell.EventKey {
  if event.Key() == tcell.KeyEscape {
   app.SetFocus(inputField)
   updateGuidance(false)
   return nil
  } else if event.Rune() == ':' {
   app.SetFocus(inputField)
   updateGuidance(false)
   return nil
  }
  return event
 })

 header := tview.NewFlex().
  SetDirection(tview.FlexRow).
  AddItem(newPrimitive("Header"), 2, 0, false).
  AddItem(inputField, 0, 1, true)

 menu := newPrimitive("Menu")
 sideBar := newPrimitive("Side Bar")

 grid := tview.NewGrid().
  SetRows(3, 0, 6).
  SetColumns(30, 0, 30).
  SetBorders(true).
  AddItem(header, 0, 0, 1, 3, 0, 0, false).
  AddItem(footer, 2, 0, 1, 3, 0, 0, false)

 // Layout for screens narrower than 100 cells (menu and side bar are hidden).
 grid.AddItem(menu, 0, 0, 0, 0, 0, 0, false).
  AddItem(list, 1, 0, 1, 3, 0, 0, false).
  AddItem(sideBar, 0, 0, 0, 0, 0, 0, false)

 // Layout for screens wider than 100 cells.
 grid.AddItem(menu, 1, 0, 1, 1, 0, 100, false).
  AddItem(list, 1, 1, 1, 1, 0, 100, false).
  AddItem(sideBar, 1, 2, 1, 1, 0, 100, false)

 if err := app.SetRoot(grid, true).SetFocus(list).Run(); err != nil {
  panic(err)
 }
}

Even though we've added quite a bit more code, the only real changes that we've made that look new are:

1. The creation of the list.
2. Abstracting the anonymous function and assigning it to updateList where the logic filters a hardcoded list of strings.
3. Doing the same with an anonymous function updateGuidance for changing the TextView footer component text.

At this point, when we run go run main.go we start focused now on the main content list.

We can toggle up and down to move through the list.

In order to focus on the InputField, we can press :. This focuses on the InputField and updates the footer helper text.

Typing in letters for a basic filter will narrow down the list, and we can hit enter/return in order to focus back on the list and navigate the filtered values.

Navigation

Finally, the last thing we will walk through is this idea of "simple navigation" where selecting a value will navigate into a "list detail" page from the "list" view and then enable a way to navigate back.

Again, we'll hardcode the detail strings for now, but the idea will be that we augment our "listeners" to jump back and forward from the "list" and "detail" view with the enter/escape button respectively.

The code now looks like the following:

package main

import (
 "strings"

 "github.com/gdamore/tcell/v2"
 "github.com/rivo/tview"
)

func main() {
 app := tview.NewApplication()
 newPrimitive := func(text string) tview.Primitive {
  return tview.NewTextView().
   SetTextAlign(tview.AlignCenter).
   SetText(text)
 }

 // Sample data to filter with details
 items := []string{"Apple", "Banana", "Cherry", "Date", "Elderberry", "Fig", "Grape", "Honeydew", "Kiwi", "Lemon", "Mango", "Orange"}

 // Sample detail data for each fruit
 details := map[string]string{
  "Apple":      "🍎 A crisp, sweet fruit that's perfect for snacking.\n\nNutritional Facts:\n• High in fiber\n• Rich in vitamin C\n• Contains antioxidants\n\nOrigin: Central Asia",
  "Banana":     "🍌 A tropical fruit with potassium and natural sugars.\n\nNutritional Facts:\n• High in potassium\n• Good source of vitamin B6\n• Natural sugars for energy\n\nOrigin: Southeast Asia",
  "Cherry":     "🍒 Small, sweet-tart stone fruits perfect for desserts.\n\nNutritional Facts:\n• Rich in antioxidants\n• Contains melatonin\n• Anti-inflammatory properties\n\nOrigin: Asia Minor",
  "Date":       "🫒 Sweet, chewy fruits from date palm trees.\n\nNutritional Facts:\n• High in fiber\n• Rich in potassium\n• Natural sweeteners\n\nOrigin: Middle East",
  "Elderberry": "🫐 Dark purple berries known for immune support.\n\nNutritional Facts:\n• High in vitamin C\n• Rich in antioxidants\n• Immune system support\n\nOrigin: Europe",
  "Fig":        "🟤 Sweet, soft fruits with edible seeds inside.\n\nNutritional Facts:\n• High in fiber\n• Good source of calcium\n• Contains potassium\n\nOrigin: Western Asia",
  "Grape":      "🍇 Juicy fruits that can be eaten fresh or made into wine.\n\nNutritional Facts:\n• Contains resveratrol\n• Rich in vitamin K\n• Natural sugars\n\nOrigin: Middle East",
  "Honeydew":   "🍈 Sweet, pale green melon with refreshing taste.\n\nNutritional Facts:\n• High water content\n• Rich in vitamin C\n• Low in calories\n\nOrigin: France",
  "Kiwi":       "🥝 Fuzzy brown fruit with bright green flesh inside.\n\nNutritional Facts:\n• Extremely high in vitamin C\n• Good source of fiber\n• Contains vitamin K\n\nOrigin: China",
  "Lemon":      "🍋 Sour citrus fruit used for cooking and drinks.\n\nNutritional Facts:\n• Very high in vitamin C\n• Contains citric acid\n• Low in calories\n\nOrigin: Asia",
  "Mango":      "🥭 Tropical stone fruit with sweet, juicy flesh.\n\nNutritional Facts:\n• High in vitamin A\n• Rich in vitamin C\n• Contains folate\n\nOrigin: India",
  "Orange":     "🍊 Citrus fruit packed with vitamin C and fiber.\n\nNutritional Facts:\n• Excellent source of vitamin C\n• High in fiber\n• Contains folate\n\nOrigin: China",
 }

 // Create the filterable list
 list := tview.NewList().
  ShowSecondaryText(false).
  SetHighlightFullLine(true)

 // Function to populate/filter the list
 updateList := func(filter string) {
  list.Clear()
  for _, item := range items {
   if strings.Contains(strings.ToLower(item), strings.ToLower(filter)) {
    list.AddItem(item, "", 0, nil)
   }
  }
 }

 // Initially populate with all items
 updateList("")

 // Create dynamic footer for guidance text
 footer := tview.NewTextView().
  SetTextAlign(tview.AlignCenter).
  SetText("↑/↓ Navigate list • : Filter")

 // Create detail view
 detailView := tview.NewTextView().
  SetDynamicColors(true).
  SetWordWrap(true).
  SetText("Select an item to view details")

 // Create input field and header components
 inputField := tview.NewInputField().
  SetLabel("Filter: ").
  SetFieldWidth(0).
  SetFieldTextColor(tcell.ColorBlack).
  SetFieldBackgroundColor(tcell.ColorWhite)

 header := tview.NewFlex().
  SetDirection(tview.FlexRow).
  AddItem(newPrimitive("Header"), 2, 0, false).
  AddItem(inputField, 0, 1, true)

 menu := newPrimitive("Menu")
 sideBar := newPrimitive("Side Bar")

 // Create the grid early so it can be referenced by functions
 grid := tview.NewGrid().
  SetRows(3, 0, 6).
  SetColumns(30, 0, 30).
  SetBorders(true).
  AddItem(header, 0, 0, 1, 3, 0, 0, false).
  AddItem(footer, 2, 0, 1, 3, 0, 0, false)

 // State management for current view
 var currentView string = "list" // "list" or "detail"

 // Function to update guidance text based on current state
 updateGuidance := func() {
  switch currentView {
  case "list":
   if app.GetFocus() == list {
    footer.SetText("↑/↓ Navigate • Enter Select • : Filter")
   } else {
    footer.SetText(": for focus on filter • Enter to focus list")
   }
  case "detail":
   footer.SetText("Escape to return to list")
  }
 }

 // Function to show detail view
 showDetail := func(itemName string) {
  if detail, exists := details[itemName]; exists {
   detailView.SetText(detail)
  } else {
   detailView.SetText("No details available for " + itemName)
  }
  currentView = "detail"
  updateGuidance()

  // Update grid to show detail view
  grid.RemoveItem(list)
  grid.AddItem(detailView, 1, 0, 1, 3, 0, 0, false)   // narrow layout
  grid.AddItem(detailView, 1, 1, 1, 1, 0, 100, false) // wide layout
  app.SetFocus(detailView)
 }

 // Function to show list view
 showList := func() {
  currentView = "list"
  updateGuidance()

  // Update grid to show list view
  grid.RemoveItem(detailView)
  grid.AddItem(list, 1, 0, 1, 3, 0, 0, false)   // narrow layout
  grid.AddItem(list, 1, 1, 1, 1, 0, 100, false) // wide layout
  app.SetFocus(list)
 }

 // Add real-time filtering as user types
 inputField.SetChangedFunc(func(text string) {
  updateList(text)
 })

 // On Enter, focus the list for navigation (ignore Escape)
 inputField.SetDoneFunc(func(key tcell.Key) {
  if key == tcell.KeyEnter {
   app.SetFocus(list)
   updateGuidance()
  }
  // Explicitly ignore Escape key - do nothing
 })

 // Add list navigation - show detail view on selection
 list.SetDoneFunc(func() {
  currentIndex := list.GetCurrentItem()
  itemText, _ := list.GetItemText(currentIndex)
  showDetail(itemText)
 })

 // Handle keys for list: ':' focuses filter, Enter shows detail when on main list screen
 list.SetInputCapture(func(event *tcell.EventKey) *tcell.EventKey {
  // Only handle keys when we're in list view (not detail view)
  switch currentView {
  case "list":
   if event.Rune() == ':' {
    app.SetFocus(inputField)
    updateGuidance()
    return nil
   } else if event.Key() == tcell.KeyEnter {
    // Explicitly handle Enter to show detail view
    currentIndex := list.GetCurrentItem()
    itemText, _ := list.GetItemText(currentIndex)
    showDetail(itemText)
    return nil
   } else if event.Key() == tcell.KeyEscape {
    // Do nothing
    return nil
   }
  case "detail":
   if event.Key() == tcell.KeyEscape {
    showList()
    return nil
   }
  }

  // Let all other keys pass through
  return event
 })

 // Layout for screens narrower than 100 cells (menu and side bar are hidden).
 grid.AddItem(menu, 0, 0, 0, 0, 0, 0, false).
  AddItem(list, 1, 0, 1, 3, 0, 0, false).
  AddItem(sideBar, 0, 0, 0, 0, 0, 0, false)

 // Layout for screens wider than 100 cells.
 grid.AddItem(menu, 1, 0, 1, 1, 0, 100, false).
  AddItem(list, 1, 1, 1, 1, 0, 100, false).
  AddItem(sideBar, 1, 2, 1, 1, 0, 100, false)

 // Initialize guidance text
 updateGuidance()

 app.SetInputCapture(func(event *tcell.EventKey) *tcell.EventKey {
  if currentView == "detail" && event.Key() == tcell.KeyEscape {
   showList()
   return nil
  }

  // Let all other keys pass through
  return event
 })

 // Set up global input capture to handle Escape only in detail view
 if err := app.SetRoot(grid, true).SetFocus(list).Run(); err != nil {
  panic(err)
 }
}

It's fairly rough code, but the list.SetInputCapture anonymous function here handles the filter focus and navigation to the "detail" view.

For navigating back, we use app.SetInputCapture to handle the escape button to invoke showList().

Running go run main.go shows a familiar landing view for our TUI.

Selecting enter onto any of our list items now will shift us into the "detail view" to see our hardcoded detail text that maps to the selected value:

Cool beans. At this point, we can visually see how we can set different views for our TUI.

Where to next?

At this point, I'll call it on the blog post. This should be enough to help you on your way to playing around with components and UIs and shows some of the more common workflows you might expect.

As for next steps, you could either augment this simple example to refactor and improve the handlers, or add more handlers for toggling between a number of different views.

Afterwards, instead of having hardcoded values, you could try implementing a TUI that uses dynamic data coming from a database somewhere.

Referring to the tview GitHub wiki and online documentation will go a long way from here.

Conclusion

Today's post builds up a simple TUI with Golang and tview from first steps until we have a simple application with navigation and filtering.

Although trivial in nature, my hope is that this guide is enough to get you comfortable with where to go from here and how to reference the documentation.

Links and Further Reading

• k9s - Kubernetes CLI for managing clusters
• lazygit - Simple terminal UI for git commands
• yazi - Blazing fast terminal file manager
• dive - Tool for exploring Docker image layers
• tview - Terminal UI library for Go
• tview wiki - Official tview documentation and examples
• tview pkg.go documentation - Complete API reference

Photo credit: pcbulai

This has led to experimentation with both Node.js stream classes and generators.

Although each have their pros and cons, in retrospect I've decided that class streams are the preference for the work that I am doing.

Some of my personal reasons:

1. They can be easier to reason about.
2. They can easily have dependency injected with inversion-of-control libraries like Awilix and Inversify.
3. They don't require currying or partial application for aforementioned dependency injection.

Streams that I work with

Normally you would expect this to be Strings or Buffers, but in scenarios where I am processing job records.

This has led me to make use of object mode and Transform streams more often than not in day-to-day work.

Because of this, I've also opted to build out an abstract class for me to implement in TypeScript. The purpose of it is to contain the objectMode requirement and require one method that can be used for processing each chunk of a stream.

An abstract class for object-mode transforms

The following is a general look at the abstracted stream.

abstract class ChunkStream<
  TInput = unknown,
  TResult = unknown,
> extends Transform {
  constructor() {
    super({ objectMode: true });
  }

  abstract processChunk(chunk: TInput): TResult;

  async _transform(
    chunk: TInput,
    _encoding: BufferEncoding,
    callback: (error?: Error | null, data?: TResult) => void
  ): Promise<void> {
    try {
      const result = await this.processChunk(chunk);
      if (result !== undefined && result !== null) {
        // Handle empty arrays - only push if there's actual data
        if (Array.isArray(result) && result.length === 0) {
          callback(); // Don't push empty arrays
        } else {
          callback(null, result);
        }
      } else {
        callback();
      }
    } catch (error) {
      // Transform callbacks only expect error of type `Error | null`.
      // Handling of this is entirely up to you if you wish to alter
      // this approach.
      if (error instanceof Error) {
        callback(error);
      } else {
        console.warn("Unhandled error which is not an instanceof Error", error);
        throw error;
      }
    }
  }
}

As mentioned earlier, the processChunk method is abstract and requires a concrete implementation of that behaviour.

The typing probably leaves more to be desired (it could do with some constraints), but my current implementation expects to take the incoming object chunk and then return a value that will be used as the callback for the transform stream.

Implementing a concrete transform stream

A contrived implementation may look like this:

import { err, ok } from "neverthrow";

type User = {
  name: string;
  age: number;
  color: string;
};

class UserData {
  readonly _tag = "UserData";

  name: string;
  age: number;

  constructor(name: string, age: number) {
    this.name = name;
    this.age = age;
  }
}

// Making use of the `ChunkStream`
class SetUserDataStream extends ChunkStream {
  #logger: Logger;

  constructor(logger: Logger) {
    super();
    this.#logger = logger;
  }

  // Generator method for processing chunks
  async processChunk(chunk: User) {
    const transformed = new UserData(chunk.name, chunk.age);
    return ok(transformed);
  }

  // OPTIONAL: Override a Transform method
  _flush(callback) {
    this.#logger.info("Stream completed");
    callback();
  }
}

The above stream expects an object that structurally matches User as the incoming chunk input and returns that value serialized into a UserData instance and wrapped inside of a Ok result.

There is also an optional _flush method that I have overridden. You can override any Transform stream method. _flush has been useful for logging checkpoints at the end of a stream.

In use, it could look something like a long-running script where you fetch batches of data remotely, run a process and then finalize it somewhere:

import { pipeline } from "stream/promises";

const logger = new Logger();

// Assume this is a script that you're running to bulk run a long extract-transform-load job
async function main() {
  // Assume all streams in the middle are duplex or transform streams.
  // They could even inherit from our `ChunkStream` class.
  await pipeline(
    // Fetch data from remote
    Readable.from(fetchDataFromRemote()),
    // Serialize data
    new SetUserDataStream(logger),
    // Split into batches of 20 records to process
    new BatchStream(20),
    // Run the upload process
    new UploadToNewDataSource(),
    // Final sink to log results
    new Writable({
      write(
        chunk: User,
        _encoding: BufferEncoding,
        callback: (error?: Error | null) => void
      ) {
        // Log the output from `UploadToNewDataSource`
        logger.info("Output:", chunk);
        callback();
      },
      objectMode: true,
    })
  );
}

main();

Testing with this stream

For unit tests on a stream, I do the following:

1. Create a Readable stream for test data used as input.
2. Create an instance of my Transform stream to test.
3. Create a Writable "sink" where I push chunk results into an array I can assert against.

An example of this:

it("should batch items into groups of 2", async () => {
  const results: number[][] = [];
  const inputStream = [1, 2, 3, 4];
  const sink = new Writable({
    write(
      chunk: User,
      _encoding: BufferEncoding,
      callback: (error?: Error | null) => void
    ) {
      results.push(chunk);
      callback();
    },
    objectMode: true,
  });

  expect(results).toBe([
    [1, 2],
    [3, 4],
  ]);
});

Caveats to this approach

Node.js Streams at the time of writing have limited type support from what I could see.

Take my approach with a grain of salt and know that it is there to be improved upon.

Links and Further Reading

• Node.js Stream Object Mode
• Transform Streams in Node.js
• Neverthrow Library

Photo credit: pcbulai

I loaded the ISO onto a USB using my Mac, so this is a specific recount with these items in case it clarifies anything for others configuring their setup.

I will reference any point where it is best to refer back to DHH's video or the archlinux docs.

Preparing the ISO on a bootable USB drive

The following is best done alongside the archlinux installation documentation.

First, download the Arch Linux ISO from the official website. The official docs recommend using a BitTorrent download, however (due to work restrictions) it was required for me to make a HTTP direct download.

For myself in Australia, I found a mirror website to download the ISO directly, and then made use of the b2sum CLI utility in order to verify the BLAKE2b checksums.

To do this, download the b2sum.txt file from the downloads section into the same folder that you downloaded the ISO and run b2sum -c b2sums.txt in order to validate the ISO checksum.

• b2sum is a command-line utility for computing and verifying BLAKE2b checksums.
• BLAKE2b is a cryptographic hash function that's faster than MD5, SHA-1, and SHA-2, while providing high security. It's used to verify file integrity by comparing computed hashes against known good values.

On macOS, you can use tools like dd or graphical utilities like Balena Etcher to create a bootable USB drive.

# Example using dd (replace /dev/diskX with your USB device)
sudo dd if=archlinux.iso of=/dev/diskX bs=4M status=progress

This is also listed on the archlinux USB flash installation guide under 1.3 In macOS:

dd if=path/to/archlinux-version-x86_64.iso of=/dev/rdiskX bs=1m

dd (data duplicator) is a low-level copying utility that reads from an input file and writes to an output file. The options used are:

• if= (input file): specifies the source file to read from
• of= (output file): specifies the destination to write to
• bs=1m (block size): sets the read/write block size to 1 megabyte for faster copying
• rdisk prefix: on macOS, this accesses the raw disk device for faster performance

The command provided by archlinux will run silently. You can press Ctrl + t to get updates.

Once the USB is loaded, you can dismount the disk and we can plug it into our Beelink SER 5.

Entering UEFI on the Beelink SER 5

Since I pulled out my Beelink SER 5 fresh from the box, so it came preloaded with Windows 11.

We don't care about that preloaded OS, so we need to boot into UEFI.

You may be more familiar with BIOS. They are different technologies that do the same job.

• UEFI (Unified Extensible Firmware Interface) is the modern successor to BIOS. It provides a more advanced interface with better security features (like Secure Boot), support for larger hard drives (over 2TB), faster boot times, and a graphical interface instead of text-only menus.
• BIOS (Basic Input/Output System) is firmware that initializes and tests hardware components during the boot process. It provides the interface between the operating system and hardware, allowing you to configure boot order, hardware settings, and system parameters before the OS loads.

To enter UEFI on the Beelink SER 5:

1. Power on the device
2. Immediately press and hold the Delete key (or supposedly F2, but I didn't try) key repeatedly during startup
3. This will bring you into the UEFI setup menu
4. Navigate to the boot options and set your USB drive as the first boot priority
5. Save and exit to boot from the Arch Linux USB

Running archinstall

At this point, the system will boot into a very basic terminal UI. For most of this section, I followed along with DHH's setup tutorial.

We want to run archinstall to open the Arch installer and setup our configurations.

I won't go too deep into this section since you can follow along with the video, but the general overview is this:

1. Run the Arch bootscreen
2. (optional for Wifi) Configure WiFi with iwctl
3. Open the archinstall TUI with archinstall
4. Go down top-to-bottom on the left-hand sidebar
5. (optional) Configure your locales
6. Select your mirror regions to install from somewhere close to you
7. Disk configuration with the default layout and using the recommended btrfs file system
8. Turn on disk encryption (a must for Omarchy) -- select type LUKS, set the password and apply the encrytion
9. Setup hostname
10. Set root password
11. Add user account
12. For Audio, use pipewire
13. For network configuration, copy over what you used for install
14. For "Additional packages", ensure wget is installed as it will be used for Omarchy
15. Save and run the installation

At this point, the installer will take over and it can take a few minutes for everything to e configured.

Once you've configured archinstall and run the installer, it will prompt you to restart the system.

The following sub-headings are some details that I thought I would callout which goes a little bit deeper than the video:

archinstall

archinstall is Arch Linux's official guided installer that provides a text-based interface for installing Arch Linux. It simplifies the traditionally manual installation process by offering:

• Automated partitioning and filesystem setup
• Package selection and installation
• Bootloader configuration
• User account creation
• Network configuration

bootloader

A bootloader is a small program that loads the operating system kernel into memory during startup. Common Linux bootloaders include GRUB and systemd-boot. The bootloader handles the transition from firmware (BIOS/UEFI) to the operating system, and can manage multiple OS installations on the same machine.

btrfs

Btrfs (B-tree file system) is a modern copy-on-write filesystem for Linux that supports advanced features like snapshots, checksums, and built-in RAID. It's designed for fault tolerance, repair, and easy administration.

zstd

Zstd (Zstandard) is a fast compression algorithm developed by Facebook that provides excellent compression ratios with high speed. It's commonly used for filesystem compression and package compression in Linux distributions.

wget

wget (web get) is a command-line utility for downloading files from web servers using HTTP, HTTPS, and FTP protocols. It supports recursive downloads, resuming interrupted transfers, and can follow redirects. The -qO- options make it run quietly (-q) and output to stdout (-O-) instead of saving to a file.

Installing Omarchy

Once the system reboots, you'll be prompted to unlock your encrypted disk and then log in with the configurations that you setup before.

In the previous section, we preemptively configured wget. We can now use it to install Omarchy.

wget -qO- https://omarchy.org/install | bash

At my time of installation, this will prompt you for your password a few times and request some configurations for Git.

Afterwards, it will just do it's thing and install DHH's opinionated setup.

Once completed, you will be prompted to restart the system to apply the changes.

Congratulations, you have now configured Omarchy!

I use Arch btw

Conclusion

In this recount, we walked through how I configured my new Beelink SER5 for Omarchy with boot loading. This made use of my Mac for downloading, validating and configuring my USB drive for the Arch Linux ISO.

Hopefully this comes in handy for anyone else who may be keen to make the jump but wants to understand a little bit more about the configuration.

At this point, it's worth restarting DHH's video for the basics on how to work with Omarchy.

Links and Further Reading

• Omarchy Official Website
• Arch Linux Installation Guide
• Btrfs Wiki
• Zstandard Compression
• Hyprland Window Manager
• DHH's 37signals
• DHH's Omarchy introduction video

Photo credit: johnnyb803

Private access modifiers in TypeScript

If you've been working with TypeScript for a while, you may be familiar with the public, private and protected access modifiers for class properties and methods:

class Modifiers {
  public a: string;
  private b: string;
  protected c: string;

  // ... omitted
}

These access modifiers enforce the member visibility you may be familiar with if you've worked in other object-oriented languages before.

The public modifier makes members accessible from anywhere, private restricts access to only within the same class, and protected allows access from the class and its subclasses. By default, all class members are public in TypeScript, so you typically only need to explicitly declare private and protected members.

The problem with this approach is that these modifiers only apply in TypeScript. Once the code is transpiled, you can still access the elements on the class as if they were always public.

For an example of this, take the following code:

class Target {
  private name: string;

  constructor(name: string) {
    this.name = name;
  }

  call() {
    this.log();
  }

  private log() {
    console.log(this.name);
  }
}

class A {
  private target: Target;

  constructor(target: Target) {
    this.target = target;
  }
}

const target = new Target("name");
export const a = new A(target);

In this code, we've set target as a private property on our contrived class A.

Once we transpile the code, we can see the following out:

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.b = exports.a = void 0;
class Target {
  name;
  constructor(name) {
    this.name = name;
  }
  call() {
    this.log();
  }
  log() {
    console.log(this.name);
  }
}
class A {
  target;
  constructor(target) {
    this.target = target;
  }
}
const target = new Target("name");
exports.a = new A(target);

As we can see, the transpiled code strips the modifiers that were enforced only for type checking purposes.

We can confirm this by making use of this code in the Node REPL:

$ node
Welcome to Node.js v24.3.0.
Type ".help" for more information.
> const {a} = require('./index.js')
undefined
> a.target.name
'name'
>

As you can see, we were able to access both our "private" properties target, as well as that variable's private property name.

So what can we do about this?

class Target {
  #name: string;

  constructor(name: string) {
    this.#name = name;
  }

  call() {
    this.log();
  }

  private log() {
    console.log(this.#name);
  }
}

class A {
  #target: Target;

  constructor(target: Target) {
    this.#target = target;
  }

  call() {
    this.#target.call();
  }
}

const target = new Target("name");
export const a = new A(target);

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.a = void 0;
class Target {
  #name;
  constructor(name) {
    this.#name = name;
  }
  call() {
    this.log();
  }
  log() {
    console.log(this.#name);
  }
}
class A {
  #target;
  constructor(target) {
    this.#target = target;
  }
  call() {
    this.#target.call();
  }
}
const target = new Target("name");
exports.a = new A(target);

$ node
Welcome to Node.js v24.3.0.
Type ".help" for more information.
> const {a} = require('./index.js')
undefined
> a.#target.#name
a.#target.#name
 ^

Uncaught SyntaxError: Private field '#target' must be declared in an enclosing class
> a.call()
name
undefined

Tail recursion is a memory-optimization technique (using tail call optimization) for the call stack.

Instead of piling on more data onto the stack data structure for recursive calls, tail recursion enables the optimization to deallocate the previous function call's memory prior to making the recursive call.

Not all languages support tail call optimization. Almost all functional languages support it, some systems languages support it, but a number of popular languages such as Python and Ruby do not support it. It's still worth knowing about.

How Can I Visualize Tail Recursion?

Think of a stack of plates.

Each plate in the stack represents in a recursive call that is not tail call optimized.

Given enough recursive calls, the stack will build up enough, run out of space and cause a stack overflow that will crash the system.

In the case of tail recursion, you can think of this as one plate being added to the stack, but once it finishes it's run and makes a recursive call, the current "plate" is removed from the stack before the next "plate" is added.

You can think of this as a plate stack with one plate being added and removed for each recursive iteration.

This analogy helps to visualize the lifecycle of a call stack and how that data structure works. Regardless of tail recursion, each time you are adding and removing from the stack it is like a stack of plates where you add and remove from the top.

An Example of Tail Recursion

I will make use of Zig to compare an approach with and without tail call optimization being used.

const std = @import("std");
const print = std.debug.print;

// Get approximate stack pointer to measure stack usage
fn getStackPointer() usize {
    var dummy: u8 = 0;
    return @intFromPtr(&dummy);
}

fn measureMaxStackUsage() void {
    var max_stack_used: i64 = 0;
    const initial_sp = getStackPointer();

    const result = tailRecursiveWithMeasurement(1000, 1, initial_sp, &max_stack_used);

    print("=== Maximum Stack Usage Test ===\n", .{});
    print("Result: {}\n", .{result});
    print("Maximum stack used during recursion: {} bytes\n", .{max_stack_used});
}

// Separate function that tracks max usage
fn tailRecursiveWithMeasurement(n: u64, acc: u64, initial: usize, max_used: *i64) u64 {
    const current_sp = getStackPointer();
    const used = @as(i64, @intCast(initial)) - @as(i64, @intCast(current_sp));
    if (used > max_used.*) {
        max_used.* = used;
    }

    // Print occasionally to show progress
    if (n % 100 == 0) {
        print("  n={}, current stack: {} bytes, max so far: {} bytes\n", .{ n, used, max_used.* });
    }

    if (n <= 1) return acc;
    return @call(.always_tail, tailRecursiveWithMeasurement, .{ n - 1, acc + n, initial, max_used });
}

fn compareStackUsage() void {
    var tail_max: i64 = 0;
    var regular_max: i64 = 0;
    const initial_sp = getStackPointer();

    print("Testing tail recursion (1000 calls)...\n", .{});
    _ = tailRecursiveWithMeasurement(1000, 0, initial_sp, &tail_max);

    print("\nTesting regular recursion (20 calls)...\n", .{});
    _ = regularRecursiveWithMeasurement(20, initial_sp, &regular_max);

    print("\n📊 RESULTS:\n", .{});
    print("Tail recursion (1000 calls):    {} bytes max\n", .{tail_max});
    print("Regular recursion (20 calls):   {} bytes max\n", .{regular_max});
    print("Bytes per call - Tail:          {:.1} bytes\n", .{@as(f64, @floatFromInt(tail_max)) / 1000.0});
    print("Bytes per call - Regular:       {:.1} bytes\n", .{@as(f64, @floatFromInt(regular_max)) / 20.0});
}

fn regularRecursiveWithMeasurement(n: u64, initial: usize, max_used: *i64) u64 {
    const current_sp = getStackPointer();
    const used = @as(i64, @intCast(initial)) - @as(i64, @intCast(current_sp));
    if (used > max_used.*) {
        max_used.* = used;
    }

    // Print occasionally to show usage
    if (n % 10 == 0) {
        print("  n={}, current stack: {} bytes, max so far: {} bytes\n", .{ n, used, max_used.* });
    }

    if (n <= 1) return 1;
    return 1 + regularRecursiveWithMeasurement(n - 1, initial, max_used);
}

pub fn main() !void {
    print("\n=== Stack Usage Comparison ===\n", .{});
    compareStackUsage();
}

Running this code results in the following:

$ zig run -O ReleaseFast tail_recursion.zig

=== Stack Usage Comparison ===
Testing tail recursion (1000 calls)...
  n=1000, current stack: -72 bytes, max so far: 0 bytes
  n=900, current stack: -72 bytes, max so far: 0 bytes
  n=800, current stack: -72 bytes, max so far: 0 bytes
  n=700, current stack: -72 bytes, max so far: 0 bytes
  n=600, current stack: -72 bytes, max so far: 0 bytes
  n=500, current stack: -72 bytes, max so far: 0 bytes
  n=400, current stack: -72 bytes, max so far: 0 bytes
  n=300, current stack: -72 bytes, max so far: 0 bytes
  n=200, current stack: -72 bytes, max so far: 0 bytes
  n=100, current stack: -72 bytes, max so far: 0 bytes

Testing regular recursion (20 calls)...
  n=20, current stack: 57 bytes, max so far: 57 bytes
  n=10, current stack: 857 bytes, max so far: 857 bytes

📊 RESULTS:
Tail recursion (1000 calls):    0 bytes max
Regular recursion (20 calls):   1577 bytes max
Bytes per call - Tail:          0.0e0 bytes
Bytes per call - Regular:       7.9e1 bytes

The -72 bytes for the tail recursion is a bit of noise, but you can see in the difference here that 20 calls with regular recursion used up around 8 bytes for invocation, while the tail recursion resulted in no memory buildup.

Links and Further Reading

• Ziggit | Recursive calling strategy
• Wikipedia | Tail call

Photo credit: nervum

Please note that this is a very loose use of the term and it certainly does not plan on focuses on the methods that come with data classes in Kotlin.

What are data classes?

As defined in the Kotlin documentation, "Data classes ... are primarily used to hold data. For each data class, the compiler automatically generates additional member functions ... Data classes are marked with data".

So for an example definition in Kotlin, you could define a data class with data class User(val name: String, val age: Int).

As mentioned, I'm not too focused on the additional methods that come with data classes. There already is another library alexeyraspopov/dataclass that enables something like this.

What I am interested in is the type properties as it relates to TypeScript that come with the idea of storing data in classes that are, by designed, not meant to add any additional functionality.

I want to explore the idea of using these data containers as classes.

interface User {
  name: string;
}

function uppercaseUserName(user: User) {
  return user.name.toUpperCase();
}

const obj = {
  name: "something",
};

console.log(uppercaseUserName(obj));

interface User {
  name: string;
}

function uppercaseUserName(user: User) {
  return user.name.toUpperCase();
}

const obj = {
  name: "something",
  age: 22,
  address: "123 fake street",
};

console.log(uppercaseUserName(obj));

class DataUser {
  public name: string;

  constructor(name: string) {
    this.name = name;
  }
}

function uppercaseUserName(user: DataUser) {
  return user.name.toUpperCase();
}

const obj = {
  name: "something",
  age: 22,
  address: "123 fake street",
};

const user = new DataUser(obj.name);
console.log(uppercaseUserName(user));

val jane = User("Jane", 35)
val (name, age) = jane
println("$name, $age years of age")
// Jane, 35 years of age

class DataUser {
  readonly _tag = "DataUser";

  public name: string;

  constructor(name: string) {
    this.name = name;
  }
}

interface User {
  name: string;
}

interface Address {
  address: string;
}

function getUserOrAddress({
  name,
  address,
}: {
  name: string;
  address: string;
}): User | Address {
  if (Math.random() > 0.5) {
    return {
      name,
    };
  }

  return {
    address,
  };
}

const obj = {
  name: "something",
  age: 22,
  address: "123 fake",
};

const result = getUserOrAddress(obj);

if ("name" in result) {
  result.name;
} else if ("address" in result) {
  result.address;
}

interface User {
  _tag: "User";
  name: string;
}

interface Address {
  _tag: "Address";
  address: string;
}

function getUserOrAddress({
  name,
  address,
}: {
  name: string;
  address: string;
}): User | Address {
  if (Math.random() > 0.5) {
    return {
      _tag: "User",
      name,
    };
  }

  return {
    _tag: "Address",
    address,
  };
}

const obj = {
  name: "something",
  age: 22,
  address: "123 fake",
};

const result = getUserOrAddress(obj);

switch (result._tag) {
  case "User":
    console.log("is user");
    break;
  case "Address":
    console.log("is address");
}

class DataUser {
  readonly _tag = "DataUser";
  public name: string;

  constructor(name: string) {
    this.name = name;
  }
}

class DataAddress {
  readonly _tag = "DataAddress";
  public name: string;

  constructor(name: string) {
    this.name = name;
  }
}

function getUserOrAddress({
  name,
  address,
}: {
  name: string;
  address: string;
}): DataUser | DataAddress {
  if (Math.random() > 0.5) {
    return new DataUser(name);
  }

  return new DataAddress(address);
}

const obj = {
  name: "something",
  age: 22,
  address: "123 fake",
};

const result = getUserOrAddress(obj);

switch (result._tag) {
  case "DataUser":
    console.log("is user");
    break;
  case "DataAddress":
    console.log("is address");
}

type SchemeType = "Visa" | "MasterCard";
type SchemeTypeDataUser = `${SchemeType}DataUser`;

class DataUser<T extends SchemeTypeDataUser = SchemeTypeDataUser> {
  readonly _tag;
  name: string;
  age: number;

  constructor(tag: T, name: string, age: number) {
    this._tag = tag;
    this.name = name;
    this.age = age;
  }
}

const mc = new DataUser("MasterCardDataUser", "Dennis", 33);
const visa = new DataUser("VisaDataUser", "Dan", 21);

mc._tag; // typed as "MasterCardDataUser"
visa._tag; // typed as "VisaDataUser"

type SchemeType = "Visa" | "MasterCard";
type SchemeTypeDataUser = `${SchemeType}DataUser`;

class DataUser<T extends SchemeTypeDataUser = SchemeTypeDataUser> {
  readonly _tag;
  name: string;
  age: number;

  private constructor(tag: T, name: string, age: number) {
    this._tag = tag;
    this.name = name;
    this.age = age;
  }

  static mastercard(name: string, age: number) {
    return new DataUser("MasterCardDataUser", name, age);
  }

  static visa(name: string, age: number) {
    return new DataUser("VisaDataUser", name, age);
  }
}

const mc = DataUser.mastercard("Dennis", 33);
const visa = DataUser.visa("Dan", 21);

mc._tag; // typed as "MasterCardDataUser"
visa._tag; // typed as "VisaDataUser"

// !!! TypeScript error due to the private constructor
// const cantDo = new DataUser('MasterCardDataUser', 'test', 25);

We will be using Prisma and Vitest to demonstrate this, but please note that this approach is just as applicable to using alternative test runners and PostgresQL-compatible libraries such as Jest and Drizzle.

Example code used for this blog post can be found on my GitHub.

Testing purgatory

There are four things that lead to testing purgatory:

1. Boundaries
2. Isolation
3. Mocking
4. Asserting side-effects

If your test boundaries are too narrow, you lose test confidence.

If you fail to properly isolate your tests, it tends towards flakiness.

If you over-mock or mock the wrong things, you introduce unnecessary code volume to your application and present an inaccurate reflection of your application.

Finally, if you over-assert side-effects within a module that are separate to the desired functionality of the program, you run the risk of false negatives once refactoring occurs.

// vitest.config.ts
import { defineConfig } from "vitest/config";
import { cpus } from "node:os";

const maxThreads = Math.max(1, cpus().length - 1);

export default defineConfig({
  test: {
    pool: "threads",
    poolOptions: {
      threads: {
        maxThreads,
        minThreads: 1,
        isolate: true,
      },
    },

    setupFiles: ["./src/test/setup.ts"],
    globalSetup: "./src/test/global-setup.ts",

    // Add the types file to be loaded automatically
    typecheck: {
      include: ["**/*.{test,spec}.{js,ts,jsx,tsx}"],
    },

    testTimeout: 15000,
    hookTimeout: 10000,

    include: ["**/*.{test,spec}.{js,ts,jsx,tsx}"],
    exclude: ["node_modules", "dist", "build"],

    environment: "node",
  },
});

import { generateDbInitSql } from "./db-setup";

export async function setup() {
  console.log("🔧 Setting up test environment...");

  try {
    // Generate the schema SQL
    await generateDbInitSql();
    console.log("✅ Database schema generated and cached");
  } catch (error) {
    console.error("❌ Failed to generate database schema:", error);
    throw error;
  }
}

export async function teardown() {
  console.log("🧹 Global teardown complete");
}

import { readFileSync } from "node:fs";

import type { PrismaClient } from "../../generated/prisma";
import type { PGlite } from "@electric-sql/pglite";
import { exec } from "node:child_process";
import { promisify } from "node:util";
import { resolve } from "node:path";
import { cwd } from "node:process";
const execAsync = promisify(exec);

declare global {
  var testPrisma: PrismaClient;
  var testDb: PGlite;
}

const migrationOutputPath = resolve(cwd(), "tmp/migration.sql");

/**
 * The Prisma CLI does not (yet) support initializing a PGLite database. Instead, we get Prisma to
 * dump a single migration file which can then be loaded by a PGLite instance to setup the
 * database.
 *
 * @see https://github.com/lucasthevenet/pglite-utils/issues/8#issuecomment-2147944548
 */
export async function generateDbInitSql() {
  const prismaSchemaPath = resolve(cwd(), "prisma/schema.prisma");
  await execAsync(
    `prisma migrate diff --from-empty --to-schema-datamodel ${prismaSchemaPath} --script > ${migrationOutputPath}`
  );
}

export function getSchemaSql() {
  return readFileSync(migrationOutputPath, "utf-8");
}

import { beforeAll, afterAll, beforeEach } from "vitest";
import { PGlite } from "@electric-sql/pglite";
import { PrismaPGlite } from "pglite-prisma-adapter";
import { PrismaClient } from "../../generated/prisma";
import { getSchemaSql } from "./db-setup";

let db: PGlite;
let prisma: PrismaClient;

beforeAll(async () => {
  console.log("🚀 Setting up database for test file...");

  db = new PGlite({
    dataDir: "memory://",
  });

  await db.exec(getSchemaSql());

  prisma = new PrismaClient({
    adapter: new PrismaPGlite(db),
  });

  // Make available globally
  global.testDb = db;
  global.testPrisma = prisma;

  console.log("✅ Database ready!");
});

afterAll(async () => {
  console.log("🧹 Cleaning up database...");

  try {
    await prisma.$disconnect();
    await db.close();
  } catch (error) {
    console.warn("Cleanup warning:", error);
  }

  console.log("✅ Database cleaned up!");
});

beforeEach(async () => {
  // Get all table names excluding Prisma migration tables
  const result = await db.query<{ tablename: string }>(`
      SELECT tablename 
      FROM pg_tables 
      WHERE schemaname = 'public' 
      AND tablename NOT LIKE '_prisma%'
      ORDER BY tablename;
    `);

  const tableNames = result.rows.map((row) => row.tablename);

  if (tableNames.length > 0) {
    // Disable foreign key checks, truncate all tables, re-enable
    await db.exec(`
        SET session_replication_role = replica;
        TRUNCATE TABLE ${tableNames
          .map((name) => `"${name}"`)
          .join(", ")} RESTART IDENTITY CASCADE;
        SET session_replication_role = DEFAULT;
      `);
  }
});

import type { PrismaClient, Role } from "../generated/prisma";

export interface CreateUserData {
  email: string;
  name?: string;
  role?: Role;
}

export interface UpdateUserData {
  email?: string;
  name?: string;
  role?: Role;
}

export class UserRepository {
  private prisma: PrismaClient;

  constructor(prisma: PrismaClient) {
    this.prisma = prisma;
  }

  /**
   * Naively return all users with no limit
   */
  async getAllUsers() {
    return this.prisma.user.findMany();
  }

  /**
   * Get user by ID
   */
  async getUser(id: number) {
    return this.prisma.user.findUnique({
      where: { id },
    });
  }

  /**
   * Create a new user
   */
  async createUser(data: CreateUserData) {
    return this.prisma.user.create({
      data: {
        email: data.email,
        name: data.name,
        role: data.role || "USER",
      },
    });
  }

  /**
   * Update user
   */
  async updateUser(id: number, data: UpdateUserData) {
    return this.prisma.user.update({
      where: { id },
      data,
    });
  }

  /**
   * Delete user
   */
  async deleteUser(id: number) {
    try {
      await this.prisma.user.delete({
        where: { id },
      });
      return true;
    } catch (error) {
      // Handle user not found
      return false;
    }
  }
}

import { beforeEach, describe, expect, it } from "vitest";
import { UserRepository } from "./user-repository";
import { faker } from "@faker-js/faker";
import { fail } from "node:assert";

describe("UserRepository", () => {
  let userRepo: UserRepository;

  beforeEach(() => {
    userRepo = new UserRepository(global.testPrisma);
  });

  describe("getUser", () => {
    it("should return user by ID", async () => {
      const user = await global.testPrisma.user.create({
        data: {
          email: faker.internet.email(),
          name: faker.person.fullName(),
        },
      });

      const result = await userRepo.getUser(user.id);
      if (!result) {
        fail("Result was null");
      }

      expect(result).not.toBeNull();
      expect(result.id).toBe(user.id);
      expect(result.email).toBe(user.email);
    });

    it("should return null for non-existent user", async () => {
      const result = await userRepo.getUser(99999);
      expect(result).toBeNull();
    });
  });

  describe("createUser", () => {
    it("should create user with all fields", async () => {
      const userData = {
        email: faker.internet.email(),
        name: faker.person.fullName(),
        role: "ADMIN" as const,
      };

      const result = await userRepo.createUser(userData);

      expect(result.email).toBe(userData.email);
      expect(result.name).toBe(userData.name);
      expect(result.role).toBe(userData.role);
      expect(result.id).toBeDefined();
    });

    it("should create user with default role when not specified", async () => {
      const userData = {
        email: faker.internet.email(),
        name: faker.person.fullName(),
      };

      const result = await userRepo.createUser(userData);

      expect(result.email).toBe(userData.email);
      expect(result.name).toBe(userData.name);
      expect(result.role).toBe("USER");
    });

    it("should create user without name (optional field)", async () => {
      const userData = {
        email: faker.internet.email(),
      };

      const result = await userRepo.createUser(userData);

      expect(result.email).toBe(userData.email);
      expect(result.name).toBeNull();
      expect(result.role).toBe("USER");
    });

    it("should throw error for duplicate email", async () => {
      const email = faker.internet.email();

      await userRepo.createUser({
        email,
        name: faker.person.fullName(),
      });

      await expect(
        userRepo.createUser({
          email, // Same email
          name: faker.person.fullName(),
        })
      ).rejects.toThrow();
    });
  });

  describe("updateUser", () => {
    it("should update user successfully", async () => {
      const user = await global.testPrisma.user.create({
        data: {
          email: faker.internet.email(),
          name: faker.person.fullName(),
          role: "USER",
        },
      });

      const updateData = {
        name: faker.person.fullName(),
        role: "ADMIN" as const,
      };

      const result = await userRepo.updateUser(user.id, updateData);
      if (!result) {
        fail("Update result was null");
      }

      expect(result.id).toBe(user.id);
      expect(result.name).toBe(updateData.name);
      expect(result.role).toBe(updateData.role);
      expect(result.email).toBe(user.email); // Should remain unchanged
    });

    it("should update only specified fields", async () => {
      const user = await global.testPrisma.user.create({
        data: {
          email: faker.internet.email(),
          name: faker.person.fullName(),
          role: "USER",
        },
      });

      const newName = faker.person.fullName();
      const result = await userRepo.updateUser(user.id, { name: newName });
      if (!result) {
        fail("Update result was null");
      }

      expect(result.name).toBe(newName);
      expect(result.email).toBe(user.email); // Unchanged
      expect(result.role).toBe(user.role); // Unchanged
    });

    it("should return null for non-existent user", async () => {
      await expect(
        userRepo.updateUser(99999, {
          name: faker.person.fullName(),
        })
      ).rejects.toThrow();
    });

    it("should throw error when updating to duplicate email", async () => {
      const existingUser = await global.testPrisma.user.create({
        data: {
          email: faker.internet.email(),
          name: faker.person.fullName(),
        },
      });

      const userToUpdate = await global.testPrisma.user.create({
        data: {
          email: faker.internet.email(),
          name: faker.person.fullName(),
        },
      });

      await expect(
        userRepo.updateUser(userToUpdate.id, {
          email: existingUser.email, // Duplicate email
        })
      ).rejects.toThrow();
    });
  });

  describe("deleteUser", () => {
    it("should delete user successfully", async () => {
      const user = await global.testPrisma.user.create({
        data: {
          email: faker.internet.email(),
          name: faker.person.fullName(),
        },
      });

      const result = await userRepo.deleteUser(user.id);
      expect(result).toBe(true);

      // Verify user was actually deleted
      const deletedUser = await global.testPrisma.user.findUnique({
        where: { id: user.id },
      });
      expect(deletedUser).toBeNull();
    });

    it("should return false for non-existent user", async () => {
      const result = await userRepo.deleteUser(99999);
      expect(result).toBe(false);
    });
  });

  describe("getUsers", () => {
    it("should return all users", async () => {
      const data = [
        {
          email: faker.internet.email(),
          name: faker.person.fullName(),
        },
        {
          email: faker.internet.email(),
          name: faker.person.fullName(),
        },
        {
          email: faker.internet.email(),
          name: faker.person.fullName(),
        },
        {
          email: faker.internet.email(),
          name: faker.person.fullName(),
        },
        {
          email: faker.internet.email(),
          name: faker.person.fullName(),
        },
      ];
      await global.testPrisma.user.createMany({
        data,
      });

      const users = await userRepo.getAllUsers();

      // Naive test just checking database length
      expect(users.length).toBe(5);
    });
  });
});

Test Files  100 passed (100)
    Tests  1300 passed (1300)
  Start at  07:35:40
  Duration  25.00s (transform 982ms, setup 5.07s, collect 14.88s, tests 206.29s, environment 12ms, prepare 7.53s)

vi.mock("./path/to/your/module", async (importOriginal) => {
  const { PGlite } = await vi.importActual("@electric-sql/pglite");
  const { PrismaPGlite } = await vi.importActual("@prisma/adapter-pglite");
  const { PrismaClient } = await vi.importActual("@prisma/client");

  const db = new PGlite();
  const adapter = new PrismaPGlite(db);
  const prisma = new PrismaClient({ adapter });

  // Apply your generated schema
  const schemaSQL = getGeneratedSchema();
  await db.exec(schemaSQL);

  return { prisma, db };
});

The target audience is for TypeScript/JavaScript developers who understand the basics of interfacing with PostgreSQL, but do not yet feel confident in working directly with SQL or the impacts of certain table design decisions. For those more confident with their PostgreSQL capabilities, this post will not be anymore than a reference for you.

We will be using Prisma to abstract some of the table schema design work. This is purely to lower the barrier to entry for developers so we can spend more take talking about and showing improvements.

I've chosen Prisma primarily as it helps for me to share this piece with work colleagues at my workplace (at the time of writing), but note that what is shown focuses more on PostgreSQL and is not ORM-bound.

Getting started

Conclusion

Links and Further Reading

Photo credit: aleksdahlberg

The approach sounded like an interesting alternative to managing environments with microservices, something that I've been pondering resolutions to myself at work to cutdown lead time to feature deployment.

This blog post looks to built out a minimal approach to using multitenancy within TypeScript microservices, but also goes a step further to see how a unified approach to multitenancy can also be grateful support by network mocking solutions for solving those difficult-to-emulate response scenarios.

Getting started

For this project, I've built out a minimal Turborepo starting place to illustrate some of the different approaches and trade-offs with minimal Hono servers.

For the testing and servers, they are not a true reflection of my approaches to these things, but are there to focus on some key points.

For a better reflection on some of my thoughts behind testing in TypeScript for projects, please read my blog post The Boundaries of TypeScript Testing.

For the companion code, please see here.

Our scenario

For this particular scenario, we are going to work with a chain of requests across three Hono servers:

1. service-a will be our core server.
2. downstream-b will be our stand-in for a service we need to call. This could be a placeholder for another micro-service within our system.
3. downstream-c will be our stand-in for a server that downstream-b needs to call. This could be a placeholder for another micro-service or a third-party API we need to interact with.

The scenario is straight forward: A calls B, B calls C.

Let's see this in action.

Some of the code from Service A which runs on localhost:3000:

app.post("/users", async (c) => {
  try {
    // ResultAsync.fromPromise comes from neverthrow
    const body = await ResultAsync.fromPromise(c.req.json(), (err) => err);
    if (body.isErr()) {
      c.status(400);
      return c.json(new NoJsonBodyError());
    }

    const userBodyResult = UsersPostBody.safeParse(body);
    if (!userBodyResult.success) {
      c.status(400);
      return c.json(new ValidationError(userBodyResult.error));
    }

    // We aren't storing this. Just illustrating some adjustments to the value
    const newUser = {
      id: v4(),
      name: userBodyResult.data.name,
    };

    const res = await axios.post(
      "http://localhost:3001/notifications",
      new NewUserNotification(newUser)
    );

    switch (res.data._tag) {
      case "ItemQueued":
        return c.json(new UserCreatedResponse(newUser));
      default:
        return c.json(new UnhandledResponseTagError());
    }
  } catch (err) {
    console.error("Something went wrong", err);
    return c.json(new InternalServerError());
  }
});

The "creates" a new user, then forwards the request onto another service Downstream B on localhost:3001:

app.post("/notifications", async (c) => {
  try {
    const body = await c.req.json();
    const notificationBodyResult = NotificationPostBody.safeParse(body);
    if (!notificationBodyResult.success) {
      c.status(400);
      return c.json(new ValidationError(notificationBodyResult.error));
    }

    const res = await addToQueue(
      new NewUserNotification(notificationBodyResult.data.value)
    );

    switch (res.data._tag) {
      case "SuccessfullyQueued":
        return c.json(new ItemQueuedResponse());
      // Propagate the following errors back under one
      // unified error type for the sake of it.
      case "ValidationError":
      case "TimeoutError":
        return c.json(new QueueError(res.data));
      default:
        return c.json(new InternalServerError());
    }
  } catch (error) {
    console.error("Error processing notification", error);
    return c.json(new InternalServerError());
  }
});

That code attempts to add it to a "queue" (we are not really doing this) on another endpoint localhost:3002 for our service Downstream C.

If it is successfully queued, then it will return the instance of ItemQueuedResponse, but our logic is handling scenarios for ValidationError and TimeoutError error as well where is returns an instance of QueueError.

For example's sake, Downstream C does not implement handling for a timeout error, however we can make the assumption that this has been implemented and that Downstream B can possibly return errors based off that.

From the companion code, we can run pnpm dev from the root directory to start all three servers and test our code is working:

# Start the servers
$ pnpm dev

# In another terminal window, testing the endpoint with HTTPie
$ http POST http://localhost:3000/users name="Dennis"                                          HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 92
Date: Sat, 24 May 2025 05:06:39 GMT
Keep-Alive: timeout=5
content-type: application/json

{
    "_tag": "UserCreated",
    "value": {
        "id": "37ba97de-df86-488f-9258-4867a655fec0",
        "name": "Dennis"
    }
}

At this point, we can tell that our configuration returns a response along our successful path.

We can also confirm that our NoJsonBodyError and ValidationError works by dropping the name from the request or setting it to be an invalid type:

# NoJsonBody Error
$ http POST http://localhost:3000/users
HTTP/1.1 400 Bad Request
Connection: keep-alive
Content-Length: 68
Date: Sat, 24 May 2025 05:31:17 GMT
Keep-Alive: timeout=5
content-type: application/json

{
    "_tag": "NoJsonBody",
    "value": {
        "message": "Request body is required"
    }
}

# ValidationError
$ http POST http://localhost:3000/users name:=1
HTTP/1.1 400 Bad Request
Connection: keep-alive
Content-Length: 139
Date: Sat, 24 May 2025 05:33:30 GMT
Keep-Alive: timeout=5
content-type: application/json

{
    "_tag": "ValidationError",
    "value": {
        "details": "✖ Invalid input: expected string, received number\n  → at name",
        "message": "Bad Request"
    }
}

But what happens for error scenarios that are not so easy to represent from our remote services?

In our code, there are potential errors to bubble up like the downstream services like TimeoutError which are not so easy to replicate reliably. Let's take a look at a few of our options.

Options for testing

Our setup is non-trivial. Although my test setup demonstrates multiple micro-services within the same repository, we may not always have that luxury.

At your own workplace, you may have services distributed across multiple teams and repositories, and co-ordination for this level of testing is of utmost importance across the board.

Let's walkthrough some possible solutions for us now.

An approach using testing library mocks

The most basic possibility for emulating certain scenarios is to make use of library mocks.

Popular TypeScript testing frameworks such as Jest and Vitest have mocking capabilities out of the box.

Depending on you configure this, you can stub a return value for a request library (like our Axios library from the examples) or even a module that interacts with the service in order to test certain scenarios.

The benefits of this approach is that it is a quick-and-easy approach that comes built into our testing tools.

The major downside is that your test boundary ends where the mock starts. Your test can no longer give certain guarantees and confidence from the tools that are now mocked out. If we mock out our request library, then we have no confidence in the request library returning the response from the network request itself.

This can be a major loss from the point-of-view of testing that aims to, at a minimum, test all modules within a service until (at least) the network boundary.

An approach using mock server infrastructure

On the other end of the scale is to stand up infrastructure to emulate network traffic. This could be a scenario of running different environments for your service, but it can also move to more extreme cases of standing up infrastructure in an attempt to emulate third-party services and APIs that you interface with in production.

This approach certainly has a spectrum of complexity.

On the side of standing up internal service infrastructure environments, it can give you a chance to run test boundaries that give you test confidence as far as deeper data layers like your databases.

The trade-off with the internal services is that there is always a cost to infrastructure which is not just monetary. There is time-to-production costs, infrastructure management, complexity increases and more. In scenarios where you have multiple environments and service with multiple internal down stream services, you can also face challenges around all environments being at parity for running tests.

It is still standardized to run different environments for your services, but it's worth noting that the approach to this varies from company-to-company and there is still such a thing as over-kill or misprovisioning of resources.

In the case of third-party mock service infrastructure, you face the cost of re-implementing and maintaining an emulation of their third party service's logic and run the risk of implementation logic mismatches.

An approach using Mock Service Worker

With JavaScript/TypeScript, there is the delightful Mock Service Worker (MSW) library which enables you to stub out responses at the network level.

This differs from the module-mocking approach we spoke about earlier in that you service still operates end-to-end from within it's service boundary. As long as your mocked response matches that of the expected success/error responses, then MSW is an unbelievably useful option for emulating response types that are hard to emulate.

A basic example of MSW from their documentation:

// src/mocks/handlers.js
import { http, HttpResponse } from "msw";

export const handlers = [
  // Intercept "GET https://example.com/user" requests...
  http.get("https://example.com/user", () => {
    // ...and respond to them using this JSON response.
    return HttpResponse.json({
      id: "c7b3d8e0-5e0b-4b0f-8b3a-3b9f4b3d3b3d",
      firstName: "John",
      lastName: "Maverick",
    });
  }),
];

When MSW is intercepting network calls, a call to https://example.com/user from your request library will respond with a valid HTTP response with a JSON body.

An approach using multi-tenancy

Uber takes an approach using multi-tenancy.

By enabling multiple tenants (such as test environments, canary releases, and product lines) to coexist within the same infrastructure, Uber achieves code isolation, flexible traffic routing, and improved integration testing, ultimately supporting rapid feature deployment and reliable service performance.

More relevant around the integration testing, Uber uses routing to test services without the constraints of static environment configuration.

Consider a scenario where Uber needs to test a new version of Service B, which interacts with Services A, C, and D. Instead of deploying this new version in a separate staging environment, Uber deploys it as a distinct tenant within the production environment, referred to as Service B' (Service B Prime). Test traffic is routed specifically to Service B', while production traffic continues to flow through the original Service B. This setup ensures that the new version can be tested in a live environment without impacting the production services.

Uber uses HTTP headers to implement multi-tenancy for integration testing in their microservice architecture.

Here's how it works:

• Tenancy Context Propagation: When a request is made (for example, from a test client), Uber adds a special tenancy header to the HTTP request (e.g., X-Tenant-ID: integration-test-123).
• Traffic Routing: Uber's internal routing and service discovery systems use this header to direct the request to the correct tenant-specific instance of the service (e.g., a test version of Service B).
• End-to-End Isolation: This header is propagated across all service calls, ensuring that the entire chain of microservices (B → C → D) remains within the same test tenant. This allows for full end-to-end testing without touching production instances.

While this approach puts a lot of emphasis on a consistent system, it also enables us to reduce some of the complexities previously spoken about around managing environments for microservices and we can leverage an approach similar using HTTP headers (or a consistent equivalent) in order to make a combination of previously explained processes in an attempt to simplify testing new versions or routing to specific HTTP mock responses.

Rethinking our approach to integration testing

Keeping in mind the different tooling and options that we currently have, we can start to pick-and-choose a combination of options to best reflect the trade-offs we want to make.

In our current setup, I'm thinking this will be what works best:

1. Making use of headers to conditionally intercept requests when MSW is enabled.
2. Configuring middleware to conditionally return certain HTTP responses from the service itself.
3. Providing route management using the headers to conditionally alter where requests are sent.

With that in mind, let's work through what some of this can look like.

Taking another look at our Mock Service Worker approach

With the idea in mind around using HTTP headers, we can make use of MSW's passthrough helper in order to conditionally fire off the original request without mocking.

When running in test modes, we can enable MSW and then using those testing headers in order to run our tests. Let's look at how that could look.

import { http, passthrough, HttpResponse } from "msw";

class QueueError {
  readonly _tag = "QueueError";
  readonly value: unknown;

  constructor(value: unknown) {
    this.value = value;
  }
}

export const handlers = [
  http.post("http://localhost:3001/notifications", ({ request }) => {
    switch (request.headers.get("X-Tenancy-ID")) {
      case "DownstreamB/ValidationError":
        return HttpResponse.json(
          new QueueError({
            _tag: "ValidationError",
          })
        );
      case "DownstreamB/TimeoutError":
        return HttpResponse.json(
          new QueueError({
            _tag: "TimeoutError",
          })
        );
      default:
        return passthrough();
    }
  }),
];

The above code will conditionally switch on a X-Tenancy-ID value if provided and only when MSW is enabled and running. In scenarios where MSW is not enabled, it will not have an impact.

In the case of the development server and testing, we can conditionally enable MSW for the duration of the tests.

On the assumption of a Vitest test setup that has MSW configured to run using the following hooks:

// vitest.setup.ts
import { beforeAll, afterEach, afterAll } from "vitest";
import { server } from "../src/mocks/node";

// Start server before all tests
beforeAll(() => {
  server.listen({
    // If you wanted to, you could set warn here in dev mode
    onUnhandledRequest: "bypass",
  });
});

// Reset handlers after each test (important for test isolation)
afterEach(() => {
  server.resetHandlers();
});

// Close server after all tests
afterAll(() => {
  server.close();
});

Then we can write a test to confirm that our configuration works with the X-Tenancy-ID configuration we've made with the following test:

import axios, { isAxiosError } from "axios";
import { ResultAsync } from "neverthrow";
import { describe, expect, it } from "vitest";

/**
 * These tests expect all the servers to be up and running.
 */
describe("app", () => {
  describe("creating a user", () => {
    describe("success states", () => {
      it("successfully creates a user", async () => {
        const response = await axios.post(
          "http://localhost:3000/users",
          {
            name: "Dennis",
          },
          {
            headers: {
              "Content-Type": "application/json",
            },
          }
        );

        expect(response.status).toBe(200);
        expect(response.data._tag).toBe("UserCreated");
      });
    });

    describe("error states", () => {
      it("requires a name property within the json body", async () => {
        const response = await ResultAsync.fromPromise(
          axios.post(
            "http://localhost:3000/users",
            {},
            {
              headers: {
                "Content-Type": "application/json",
              },
            }
          ),
          (err) => err
        );

        // There is likely a better way to handle this, but will
        // leave it in as it's the assertions that I care most
        // about using for illustration purposes.
        if (
          !response.isErr() ||
          !isAxiosError(response.error) ||
          !response.error.response
        ) {
          throw new Error("Unexpected request success");
        }

        expect(response.error.response.status).toBe(400);
        expect(response.error.response.data._tag).toBe("ValidationError");
      });

      it("returns information on the queue service TimeoutError", async () => {
        const response = await ResultAsync.fromPromise(
          axios.post(
            "http://localhost:3000/users",
            {
              name: "Dennis",
            },
            {
              headers: {
                "Content-Type": "application/json",
                "X-Tenancy-ID": "DownstreamB/TimeoutError",
              },
            }
          ),
          (err) => err
        );

        // There is likely a better way to handle this, but will
        // leave it in as it's the assertions that I care most
        // about using for illustration purposes.
        if (
          !response.isErr() ||
          !isAxiosError(response.error) ||
          !response.error.response
        ) {
          throw new Error("Unexpected request success");
        }

        expect(response.error.response.status).toBe(500);
        expect(response.error.response.data._tag).toBe("QueueError");
        expect(response.error.response.data.value._tag).toBe("TimeoutError");
      });

      it("returns information on the queue service ValidationError", async () => {
        const response = await ResultAsync.fromPromise(
          axios.post(
            "http://localhost:3000/users",
            {
              name: "Dennis",
            },
            {
              headers: {
                "Content-Type": "application/json",
                "X-Tenancy-ID": "DownstreamB/ValidationError",
              },
            }
          ),
          (err) => err
        );

        // There is likely a better way to handle this, but will
        // leave it in as it's the assertions that I care most
        // about using for illustration purposes.
        if (
          !response.isErr() ||
          !isAxiosError(response.error) ||
          !response.error.response
        ) {
          throw new Error("Unexpected request success");
        }

        expect(response.error.response.status).toBe(500);
        expect(response.error.response.data._tag).toBe("QueueError");
        expect(response.error.response.data.value._tag).toBe("ValidationError");
      });
    });
  });
});

Our tests are checking the bare minimum, and I've written them in a way that favors illustrating the MSW mock conditionally in action (it could certainly do with some TLC).

In our test scenarios, MSW mocks are configured to run for each test.

• Our successful scenario bypasses the MSW mock and calls into our microservices Downstream B and Downstream C.
• Our failure scenarios that pass the X-Tenancy-ID values that match our MSW mock conditions will be intercepted at the network level and never make it to Downstream B. That being said, the mocked JSON responses are a one-to-one representation of responses from Downstream B.

Of course, we can also make use of this if MSW is enabled on our development servers to check that we are getting the expected responses.

# An example of an MSW intercepted endpoint
$ http POST http://localhost:3000/users "X-Tenancy-ID:DownstreamB/TimeoutError" name="Dennis"
HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 53
Date: Sun, 25 May 2025 11:42:24 GMT
Keep-Alive: timeout=5
content-type: application/json

{
    "_tag": "QueueError",
    "value": {
        "_tag": "TimeoutError"
    }
}

Great! So we've managed to use the X-Tenancy-ID header to incept requests for difficult-to-emulate downstream error responses in our tests to ensure that Service A can handle those as expected on its infrastructure.

The next step for us is make use of that header for redirecting requests.

Implementing multi-tenancy for services

On real network infrastructure, there are a number of ways to manage routing. Examples of this could be re-routing at API Gateway, load balancers, reverse proxies or within services themselves (along with more alternatives!). Each come with their own value propositions and trade-offs.

In our example, we're going to simplify things by conditionally routing requests based on the X-Tenancy-ID from within a service application. Again, it's best to do a technical spike and review your own systems in order to understand what may be most beneficial for your needs.

For the example, I'm going to create a duplicate of the code in Downstream C. For the demonstration today, we'll name it Downstream C Prime, but you can think it of as emulating a separate, updated deployment of Downstream C that we want to test out.

For some contrived code in Service B that conditionally re-routes traffic based on the X-Tenancy-ID header, I have the following:

app.post("/notifications", async (c) => {
  try {
    const body = await c.req.json();
    const notificationBodyResult = NotificationPostBody.safeParse(body);
    if (!notificationBodyResult.success) {
      c.status(400);
      return c.json(new ValidationError(notificationBodyResult.error));
    }

    // This is shoe-horned in for demonstration. In reality, you should come up
    // with a better approach to resolving this value.
    let baseUrl;
    switch (c.req.header("X-Tenancy-ID")) {
      case "Feature/QueueUpdate":
        baseUrl = "http://localhost:3003";
        break;
      default:
        baseUrl = "http://localhost:3002";
    }

    const res = await addToQueue(
      new NewUserNotification(notificationBodyResult.data.value),
      baseUrl
    );

    switch (res.data._tag) {
      case "SuccessfullyQueued":
        return c.json(new ItemQueuedResponse());
      case "SuccessfullyQueuedAlt":
        return c.json(new ItemQueuedNewVersionResponse());
      // Propagate the following errors back under one
      // unified error type for the sake of it.
      case "ValidationError":
      case "TimeoutError":
        return c.json(new QueueError(res.data));
      default:
        return c.json(new InternalServerError());
    }
  } catch (error) {
    console.error("Error processing notification", error);
    return c.json(new InternalServerError());
  }
});

In the above code, we have code-dependent conditional logic to resolve the base URL to use for the request with a fallback to the default service.

The updated Downstream C Prime service also has an different response to emulate an "updated" version of the service.

We can test this works against our running servers by making a few calls:

# Without the tenancy ID
$ http POST http://localhost:3000/users name="Dennis"
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 92
Date: Sun, 25 May 2025 09:59:40 GMT
Keep-Alive: timeout=5
content-type: application/json

{
    "_tag": "UserCreated",
    "value": {
        "id": "e0371a7e-56d4-4c4e-a3f3-8942caca0003",
        "name": "Dennis"
    }
}

# With the tenancy ID
$ http POST http://localhost:3000/users X-Tenancy-ID:Feature/QueueUpdate name="Dennis"
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 68
Date: Sun, 25 May 2025 09:58:48 GMT
Keep-Alive: timeout=5
content-type: application/json

{
    "_message": "This uses the alternative tenant",
    "_tag": "UserCreated"
}

As we can see, our servers are now responding with different values for our service.

We can also configure our tests for us to test both these alternatives by adding the alternative route as another success route:

describe("success states", () => {
  it("successfully creates a user", async () => {
    const response = await axios.post(
      "http://localhost:3000/users",
      {
        name: "Dennis",
      },
      {
        headers: {
          "Content-Type": "application/json",
        },
      }
    );

    expect(response.status).toBe(200);
    expect(response.data._tag).toBe("UserCreated");
  });

  it("successfully creates a user taking the alternative path", async () => {
    // This test will run against Downstream C Prime thanks
    // to the X-Tenancy-ID
    const response = await axios.post(
      "http://localhost:3000/users",
      {
        name: "Dennis",
      },
      {
        headers: {
          "Content-Type": "application/json",
          "X-Tenancy-ID": "Feature/QueueUpdate",
        },
      }
    );

    expect(response.status).toBe(200);
    expect(response.data._tag).toBe("UserCreated");
    expect(response.data._message).toBe("This uses the alternative tenant");
  });
});

When we run our tests with pnpm test, we can see that all tests pass.

At this point, we now have a solution to optionally intercept or re-route traffic for our services based on the X-Tenancy-ID provided.

Conclusion

This post explored a proof-of-concept for multi-tenant testing, as well as using networking mocking solutions to also reduce the amount of infrastructure management required.

These approaches adequately showed the potential that this approach has with enough buy-in and unification. I believe that refining these ideas with a technical design relevant to your situation has the potential to greatly simplify testing complicated microservice architecture systems.

Links and Further Reading

• Uber Blog | Why We Leverage Multi-tenancy in Uber’s Microservice Architecture
• Uber Blog | Shifting E2E Left At Uber
• Mock Service Worker (MSW)

Photo credit: aleksdahlberg

If you work extensively with Zod and tools like Tanstack Query (if you are using Vue/React at this point in time), it can be a natural fit and time saver to generate and distribute your clients this way.

In recent work, I've been using TS Rest to pair up my smaller Hono projects with clients, but it becomes immediately apparent to me just how good Tanstack Query can be out of the box with it's out-of-the-box capabilities around retries and caching.

This short blog post aims to look at ways to make use of TS Rest customization in order to have some control over how we can introduce this to our other clients.

Custom TS Rest clients

TS Rest gives you the capability to create your own custom client which can be used for controlling how your HTTP calls are made.

As a simple example:

import { initClient, tsRestFetchApi } from "@ts-rest/core";
import { contract } from "./contract";

const client = initClient(contract, {
  baseUrl: "http://localhost:5003",
  baseHeaders: {},
  api: async (args) => {
    // Add anything you want here!

    return tsRestFetchApi(args);
  },
});

In their example code here, they expose their default tsRestFetchApi and allow you to control anything you want based off the args prior to making any calls.

There are also guides on how to create your own custom client with Axios:

import axios, { Method, AxiosError, AxiosResponse, isAxiosError } from "axios";
import { initClient } from "@ts-rest/core";
import { contract } from "./contract";

const client = initClient(contract, {
  baseUrl: "http://localhost:3333/api",
  baseHeaders: {
    "Content-Type": "application/json",
  },
  api: async ({ path, method, headers, body }) => {
    const baseUrl = "http://localhost:3333/api"; //baseUrl is not available as a param, yet
    try {
      const result = await axios.request({
        method: method as Method,
        url: `${this.baseUrl}/${path}`,
        headers,
        data: body,
      });
      return {
        status: result.status,
        body: result.data,
        headers: result.headers,
      };
    } catch (e: Error | AxiosError | any) {
      if (isAxiosError(e)) {
        const error = e as AxiosError;
        const response = error.response as AxiosResponse;
        return {
          status: response.status,
          body: response.data,
          headers: response.headers,
        };
      }
      throw e;
    }
  },
});

In our example code today, we will be creating a custom handler for our api property with Axios.

What can we extend?

We will be doing more "showing" than "telling" in this short blog post.

In short, the standard ApiFetcherArgs which are passed to the api handler extending the exported InitClientArgs. Both ApiFetcherArgs and InitClientArgs are available type exports from ts-rest.

Let's show some type definitions from my current version of TS Rest to show what we have to work with.

For ApiFetcher with is the type of api:

export type ApiFetcher = (args: ApiFetcherArgs) => Promise<{
  status: number;
  body: unknown;
  headers: Headers;
}>;

So we know from this function signature that whatever we pass for our custom client must return a promise with status, body and headers.

As for the args property of type ApiFetcherArgs, we can actually extend this ourselves. TS Rest docs also have an entire section demonstrating this:

const client = initClient(contract, {
  baseUrl: "http://localhost:5003",
  baseHeaders: {},
  api: async (args: ApiFetcherArgs & { myCustomArg?: string }) => {
    if (args.myCustomArg) {
      // do something with myCustomArg ✨
    }

    return tsRestFetchApi(args);
  },
});

As long as we follow that contractual obligation, we can customize things however we want.

Enabling retries, timeouts and more

At this point, we can show some code.

It's worth highlighting: this code is a rough proof-of-concept and nothing special. It will illustrate enough to get the point across, but be sure to make adjustments as your needs suit.

The code I will show you is a class that has a handler method which meets the contractual obligation to be used.

import type { ApiFetcherArgs } from "@ts-rest/core";
import axios, { isAxiosError } from "axios";
import type { AxiosError, AxiosRequestConfig } from "axios";
import { merge } from "es-toolkit";

const BASE_MS = 100;
const MAX_MS = 10000;
const JITTER_FACTOR = 0.5;

interface TsRestClientConfig {
  span?: { name: string; options?: Record<string, unknown> };
  shouldRetry?: (error: AxiosError) => boolean; // Custom function to determine retry
  retries?: number;
  timeout?: number; // in milliseconds
  retryableStatuses?: number[]; // Add specific status codes to retry
  retryDelay?: {
    baseMs?: number; // Base delay in milliseconds (default: 100)
    maxMs?: number; // Maximum delay cap in milliseconds (default: 10000)
    jitterFactor?: number; // How much jitter to add (0-1, default: 0.5)
  };
}

type ExtendedApiFetcherArgs = TsRestClientConfig & ApiFetcherArgs;

const configDefaults = {
  retries: 0,
  retryDelay: {
    baseMs: BASE_MS,
    maxMs: MAX_MS,
    jitterFactor: JITTER_FACTOR,
  },
  retryableStatuses: [408, 429, 500, 502, 503, 504],
};

export class TsRestClient {
  private config: TsRestClientConfig;
  private axiosRequestConfig: AxiosRequestConfig;

  constructor(
    config: TsRestClientConfig = {},
    axiosRequestConfig: AxiosRequestConfig = {}
  ) {
    this.config = config;
    this.axiosRequestConfig = axiosRequestConfig;
  }

  public handler = async (extendedApiFetcherArgs: ExtendedApiFetcherArgs) => {
    return this.tryExecute({
      extendedApiFetcherArgs,
      attempt: 0,
    });
  };

  private async tryExecute({
    extendedApiFetcherArgs,
    attempt,
  }: {
    extendedApiFetcherArgs: ExtendedApiFetcherArgs;
    attempt: number;
  }) {
    const args = merge(this.config, extendedApiFetcherArgs);
    const {
      span,
      timeout,
      retries = configDefaults.retries,
      retryDelay = configDefaults.retryDelay,
    } = args;

    try {
      const axiosConfig: AxiosRequestConfig = {
        ...this.axiosRequestConfig,
        url: args.path,
        method: args.method,
        headers: args.headers,
        data: args.body,
        timeout,
      };

      const response = await axios(axiosConfig);

      if (span) {
        // TODO: End span
      }

      return {
        headers: response.headers,
        status: response.status,
        body: response.data,
      };
    } catch (error) {
      if (attempt < retries && this.isRetryable(error)) {
        const baseDelay = 2 ** (attempt + 1) * (retryDelay.baseMs ?? BASE_MS);
        const jitter =
          1 -
          (retryDelay.jitterFactor ?? JITTER_FACTOR) +
          Math.random() * 2 * (retryDelay.jitterFactor ?? JITTER_FACTOR);
        const delay = Math.min(
          Math.floor(baseDelay * jitter),
          retryDelay.maxMs ?? MAX_MS
        );

        console.log(
          `Retrying in ${delay}ms (attempt ${attempt + 1} of ${retries})...`
        );
        await new Promise((resolve) => setTimeout(resolve, delay));

        return this.tryExecute({
          extendedApiFetcherArgs,
          attempt: attempt + 1,
        });
      }

      if (span) {
        // TODO: End span
      }

      if (isAxiosError(error) && error.response) {
        return {
          headers: error.response.headers,
          status: error.response.status,
          body: error.response.data,
        };
      }

      throw error;
    }
  }

  private isRetryable(error: unknown): boolean {
    if (this.config.shouldRetry && isAxiosError(error))
      return this.config.shouldRetry(error);
    if (isAxiosError(error)) {
      return (
        !error.response ||
        (
          this.config.retryableStatuses ?? configDefaults.retryableStatuses
        ).includes(error.response.status)
      );
    }
    return (
      error instanceof Error &&
      (error.message.includes("timeout") ||
        error.message.includes("network") ||
        error.message.includes("connection"))
    );
  }
}

In the above code, we have some helpers to determine whether a code is "retry-able", we also pass a timeout argument property to the Axios client where Axios can control the abort signals for our calls and we leave some placeholder space for starting and stopping "spans".

An example of implementing our client could now look like this:

import { initClient } from "@ts-rest/core";
import { contract } from "./contract";
import { TsRestClient } from "./rpc-class";

async function main() {
  // Initialize the client with the custom API
  const client = initClient(contract, {
    baseUrl: "http://localhost:3000",
    baseHeaders: {},
    api: new TsRestClient().handler,
  });

  // Example usage with helper options
  const result = await client.authors.getAuthors({
    retries: 3,
    span: { name: "fetch-authors", options: { operation: "read" } },
    timeout: 10000,
  });
  console.log(result.status);

  switch (result.status) {
    case 200:
      console.log(result.body);
      break;
    case 404:
      console.log(result.body);
      break;
    case 500:
      console.log(result.body);
      break;
  }
}

main();

In the above code, I pass the api code the handler method property of a new TsRestClient instance.

More notably, it is type safe for my to pass custom arguments for retries, span and timeout.

In the case of my contract, I expect I can return 200, 404 or 500s. If I run the code, I get the following for each scenario:

# 200
npx tsx src/client.ts
Starting span: fetch-authors
attempt 0
Ending span: fetch-authors
200
[ { id: '123', email: 'example@example.com', name: 'Test' } ]

# 404
npx tsx src/client.ts
Starting span: fetch-authors
attempt 0
Ending span: fetch-authors
404
{ message: 'Not found' }

# 500
$ npx tsx src/client.ts
Starting span: fetch-authors
attempt 0
Retrying in 245ms (attempt 1 of 3)...
attempt 1
Retrying in 312ms (attempt 2 of 3)...
attempt 2
Retrying in 1007ms (attempt 3 of 3)...
attempt 3
Ending span: fetch-authors
500
{ message: 'Internal server error' }

Other than the fact that my logging is zero-indexed (oops), you can now see that each code has different behaviour based on the responses.

In the case of my 200, I have a span start, span end and log the contract result from my test server.

The 404 behaves in a similar manner, but returns with not found after just one attempt. This follows my default configuration behind what are the retryable statuses.

Finally, the 500 demonstrates the retries. I makes four attempts in total, each with some exponential back-off based on my settings.

Our current API allows for custom overrides per invocation. So if we wanted to, we could enable 404s to be retryable.

An example:

const result = await client.authors.getAuthors({
  retries: 3,
  retryableStatuses: [404],
  span: { name: "fetch-authors", options: { operation: "read" } },
  timeout: 10000,
});

Running this, we can see that I ended up making one retry attempt before getting the 200:

$ npx tsx src/client.ts
Retrying in 120ms (attempt 1 of 3)...
200
[ { id: '123', email: 'example@example.com', name: 'Test' } ]

Improvements

As mentioned, we have placeholders for spans there and I haven't touched on caching.

It would be trivial to augment this to use an in-memory cache that you can opt into and extend our API options to include cache options.

As for the spans, you're more likely to be using something like Otel. In that case, you might have a configuration like the following:

import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
import { AWSXRayIdGenerator } from "@opentelemetry/id-generator-aws-xray";
import { AWSXRayPropagator } from "@opentelemetry/propagator-aws-xray";
import { AWSXRayExporter } from "@opentelemetry/exporter-aws-xray";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";

import { trace, context, propagation } from "@opentelemetry/api";

// Create the tracer provider with AWS X-Ray ID generator
const provider = new NodeTracerProvider({
  idGenerator: new AWSXRayIdGenerator(),
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "my-node-service",
  }),
});

// Set the X-Ray exporter
provider.addSpanProcessor(new SimpleSpanProcessor(new AWSXRayExporter()));

// Register the provider
provider.register();

// Set the global propagator to X-Ray format
propagation.setGlobalPropagator(new AWSXRayPropagator());

// Optional: register all known instrumentations
registerInstrumentations({
  instrumentations: [getNodeAutoInstrumentations()],
});

// Now you can use OpenTelemetry as usual
const tracer = trace.getTracer("my-node-service");

Once you have a tracer available, you can start and stop spans like so:

const tracer = trace.getTracer("my-node-service");

// Somewhere in your business logic
const span = tracer.startSpan("fraudCheck");

context.with(trace.setSpan(context.active(), span), () => {
  try {
    // do fraud check
    if (someRiskyCondition) {
      span.setAttribute("fraud.score", 95);
    }
    span.addEvent("fraud check complete");
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: SpanStatusCode.ERROR });
  } finally {
    span.end();
  }
});

Instead of passing in the span arguments we made, you might pass in the span or context itself and have your client manage its lifecycle or attribute properties.

Conclusion

Short post today. I wanted to just show off a proof-of-concept that I had in order to illustrate that you can encapsulate some of that timeout and retry logic into your custom client logic so that others can make use it in their projects when calling your servers.

Links and Further Reading

• TS Rest
• Zod
• Tanstack Query
• Custom Clients in TS Rest
• Extra Query Arguments in TS Rest
• Axios Introduction
• InversifyJS

Photo credit: mickdepaola

In this blog post, we'll be working through the process of distributing JWTs and verifying them in another service by making use of a "well-known" public endpoint.

You can find the demo code examples on my GitHub.

Well-known endpoints

Well-known URIs are public endpoints prefixed with /.well-known/.

Defined in RFC 8615, the purpose of well known endpoints is to add a consistent resolution to web applications that require the discovery of information about an origin (also known as "site-wide metadata").

| Well-Known Endpoint | Purpose | Example URL | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | --- | | openid-configuration | Provides OpenID Connect provider configuration details, including endpoints and supported capabilities. | https://example.com/.well-known/openid-configuration | | jwks.json | Supplies the JSON Web Key Set (JWKS), containing public keys used to verify JSON Web Tokens (JWTs) issued by the authorization server. | https://example.com/.well-known/jwks.json | | webfinger | Enables discovery of information about entities (e.g., users) identified by a URI, often used in federated systems. | https://example.com/.well-known/webfinger | | oauth-authorization-server | Provides metadata about the OAuth 2.0 authorization server, detailing its capabilities and endpoints. | https://example.com/.well-known/oauth-authorization-server | | security.txt | Specifies security policies and contact information for reporting vulnerabilities. | https://example.com/.well-known/security.txt |  |

Well-known endpoints standardize the process of discovering essential service information and ease integration across various services.

As noted in the above table, we can use this supply the JSON Web Key Set that contains the public keys that help us with the JWT verification process.

Verifying JWTS without well-known endpoints

Before demonstrating an example of a well-known endpoint assisting with JWT verification, we can take a look at how verification works.

In order to do this, we need to generate a key-pair to use. For the sake of mixing things up compared to my previous posts, I'll generate this key pair using node-forge:

import forge from "node-forge";
import fs from "node:fs";

const { pki } = forge;

// 1. Generate RSA keypair
const keypair = pki.rsa.generateKeyPair(2048);

// 2. Convert keys to PEM format
const privateKeyPem = pki.privateKeyToPem(keypair.privateKey);
const publicKeyPem = pki.publicKeyToPem(keypair.publicKey);

// 3. Print or save them
console.log("🔐 Private Key:\n", privateKeyPem);
console.log("🔓 Public Key:\n", publicKeyPem);

// Optional: save to files
fs.writeFileSync("keys/private.pem", privateKeyPem);
fs.writeFileSync("keys/public.pem", publicKeyPem);

Running this script will generate my key-pair for me in a keys folder.

Once completed, we can use another script to generate a JSON Web Token (JWT) and then run verification across it.

Here is a simple script doing just that:

import jwt from "jsonwebtoken";
import fs from "node:fs";

const privateKey = fs.readFileSync("./keys/private.pem");

const issuer = "your-issuer";
const audience = "your-audience";
const algorithm = "RS256";

const token = jwt.sign({ sub: "user_123", role: "admin" }, privateKey, {
  algorithm,
  keyid: "test-key",
  expiresIn: "1h",
  issuer,
  audience,
});

const publicKey = fs.readFileSync("./keys/public.pem");

try {
  const decoded = jwt.verify(token, publicKey, {
    algorithms: [algorithm],
    issuer, // optional, but recommended
    audience, // optional
  });

  console.log("✅ Token is valid!");
  console.log("Decoded payload:", decoded);
} catch (err) {
  console.error("❌ Token verification failed:", (err as Error).message);
}

If we run this script, we can validate that everything is verified as expected:

npx tsx ./simple-jwt-verification.ts
✅ Token is valid!
Decoded payload: {
  sub: 'user_123',
  role: 'admin',
  iat: 1744184754,
  exp: 1744188354,
  aud: 'your-audience',
  iss: 'your-issuer'
}

As you might expect, if you update any of the verification values to expect something different i.e. a different issuer or audience, then re-running the script will fail the validation.

Verification with a well-known JWKS endpoint

Now that we understand how verification can work for JWTs, let's use a local server with a well-known endpoint to perform the verification this time.

For our server file, we can add the following:

import { Hono } from "hono";
import { exportJWK, importSPKI } from "jose";
import fs from "node:fs";
import { serve } from "@hono/node-server";

const publicKey = fs.readFileSync("./keys/public.pem", "utf8");

// Convert PEM to KeyLike (SPKI = public key in PEM format)
const key = await importSPKI(publicKey, "RS256");

// Convert the public key to JWK format
const jwk = await exportJWK(key);
jwk.kid = "test-key"; // must match JWT `kid`

const app = new Hono();

// Serve JWKS
app.get("/.well-known/jwks.json", (c) => {
  return c.json({ keys: [jwk] });
});

serve(app, (info) => {
  console.log(`Listening on http://localhost:${info.port}`);
});

We import helper functions from the widely used jose package for converting our public key string into a format that can be exported as a JSON Web Key.

The SPKI format is used for representing public keys, and it's this format that we need to pass as an argument for exportJWK.

We start up a simple Hono web server and serve our JSON Web Key as part of an array of keys from the endpoint /.well-known/jwks.json.

As for the verification script, we can follow a similar script to before with some helper functions to derive our public key used for the verification:

import jwt from "jsonwebtoken";
import axios from "axios";
import jwkToPem from "jwk-to-pem";
import fs from "node:fs";

const issuer = "http://localhost:3000"; // your issuer that serves the .well-known/jwks.json
const audience = "your-audience";
const algorithm = "RS256";
const kid = "test-key";
const jwksUrl = `${issuer}/.well-known/jwks.json`;

const privateKey = fs.readFileSync("./keys/private.pem");

const token = jwt.sign({ sub: "user_123", role: "admin" }, privateKey, {
  algorithm,
  keyid: kid,
  expiresIn: "1h",
  issuer,
  audience,
});

async function getPublicKeyFromJwks(kid: string): Promise<string> {
  const { data } = await axios.get(jwksUrl);
  const jwk = data.keys.find((key: any) => key.kid === kid);

  if (!jwk) {
    throw new Error(`Key with kid "${kid}" not found in JWKS`);
  }

  return jwkToPem(jwk);
}

async function verifyToken(token: string) {
  const decodedHeader = jwt.decode(token, { complete: true })?.header;

  if (!decodedHeader || !decodedHeader.kid) {
    throw new Error("Missing or invalid token header");
  }

  const publicKey = await getPublicKeyFromJwks(decodedHeader.kid);

  const payload = jwt.verify(token, publicKey, {
    algorithms: [algorithm],
    issuer,
    audience,
  });

  return payload;
}

verifyToken(token)
  .then((decoded) => {
    console.log("✅ Token is valid!");
    console.log("Decoded payload:", decoded);
  })
  .catch((err) => {
    console.error("❌ Token verification failed:", (err as Error).message);
  });

In this script, we make use of the jwk-to-pem library to convert the JWK that we fetch into the PEM format. We also make use of an HTTP client Axios in order to make our GET request.

It's worth also calling out here that we could have used the jose library as well. It has some helper functions for verifying JWTs and importing JWKs. I opted not to use it in order to keep our script that previously verified with the jsonwebtoken library aligned to make it easier to see the differences.

Another potentially confusing point to clarify: I am creating the JWT within the script with the private key that we generated previously, but using the well-known endpoint to verify it with the public key information that we also generated earlier in this blog post.

If we run our updated script, it will give us our desired verification:

$ npx tsx server-jwt-verification.ts
✅ Token is valid!
Decoded payload: {
  sub: 'user_123',
  role: 'admin',
  iat: 1744191716,
  exp: 1744195316,
  aud: 'your-audience',
  iss: 'http://localhost:3000'
}

Great! We successfully verified our JSON Web Token using the well-known endpoint that shared our JSON Web Keys.

Conclusion

We explored the use of Well-Known JSON Web Key Sets (JWKS) for dynamically verifying JSON Web Tokens (JWTs) across distributed services.

This becomes useful in your own services if you have a centralized service delegating JWTs that you need extra verification of that are durable against key-rotations.

This was done using well-known endpoints, which are worth knowing as these endpoints have many use cases for distributing service metadata. You are likely to require doing this yourself with other 3rd parties at some stage.

Links and Further Reading

• node-forge (npm)
• jsonwebtoken (npm)
• jose (npm)
• Hono web framework
• jwk-to-pem (npm)
• Axios (npm)
• RFC 8615 - Well-Known URIs
• Wikipedia - Well-known URI
• GitHub demo code | okeeffed/demo-well-known-jwks

Photo credit: alisonstardust

It's an extra layer of security that can be applied at the networking layer of the OSI model (layer 4) and is used extensively within architectures such as financial systems and service meshes.

This post will work through defining what mTLS is, work through some key components such as Certificate Authorities (CAs), Certificate Signing Requests (CSRs) and x.509 certificates, work through building an example application with your own chain of trust that includes a root CA, intermediary CA, client and server and round off with some real-world case studies.

Mutual Transport Layer Security

To understand how mTLS extends TLS, let's recap how TLS 1.3 works at a high level.

The verification process of TLS 1.3 introduces a more streamlined handshake compared to earlier versions.

It reduces the number of round trips and encrypts more of the handshake itself while maintaining strong security guarantees.

In standard (one-way) TLS, only the server is authenticated.

A high-level summary of the one-way TLS 1.3 handshake:

1. Client sends a ClientHello, which includes supported cipher suites, extensions (like ALPN: Application-Layer Protocol Negotiation), and its key share (used for Diffie-Hellman key exchange).
2. Server responds with a ServerHello, its own key share, and its certificate (usually signed by a trusted Certificate Authority).
3. The client verifies the server's certificate using its trust store.
4. Both sides use their key shares to derive a shared secret (ECDHE: Elliptic Curve Diffie-Hellman Ephemeral is common).
5. A secure, encrypted channel is established using symmetric encryption based on the shared secret.

The ClientHello and ServerHello messages are described in detail in the TLS 1.3 specification.

In this process, the server proves its identity via the certificate, but the client remains anonymous.

In mutual TLS (mTLS), the handshake includes an optional client authentication step, allowing both parties to verify each other's identities.

A high-level summary of the mutual TLS 1.3 handshake:

1. Client initiates a connection with a ClientHello and its key share.
2. Server replies with ServerHello, its own key share, and its certificate.
3. Client verifies the server's certificate.
4. Server sends a CertificateRequest message, asking the client to authenticate.
5. Client responds with its own certificate and a CertificateVerify message (proving it holds the private key).
6. Server verifies the client's certificate and signature.
7. Both parties complete the handshake using the agreed key derived from their shared key shares.
8. A secure, mutually authenticated channel is established.

In mTLS, the connection is only considered secure after both certificates have been verified and the shared key is established.

TLS 1.3's enhancements make the handshake more efficient while preserving the flexibility to support mTLS when mutual identity verification is required.

In order to understand more on verification with a trusted root Certificate Authority, we need to better understand how certificates work in general and their relation to this process.

Certificates, Authorities, Signing Requests

Let's start by defining some of the key terms:

Most of these terms require some form of memorization. To help ease that process, here is an analogy with airport security that can help you better visualize how each of these concepts play a role in mTLS.

For the sake of clarification in the analogy, the airport represents the server while you are the client. Where you need to get your create juices out a bit is understanding how the certificates (passports) are shown.

• In regular TLS, only the server shows its passport (this is where the creative thinking needs to come in, since airports obviously don't have passports). In this scenario, the client (you) checks if it is valid, but you don't need to show your ID.
• In mutual TLS, both the server and client show their passports. The airport checks your passport and you verify theirs. No one gets in without being trusted on both sides.

To best solidify these elements and the roles that they play in mTLS, we can build out our own local representation of these.

Demo: Generating certificates

Based on our key terms, we know there are a few things that we would like to emulate locally:

1. The Root Certificate Authority.
2. An Intermediate Certificate Authority that acts on behalf of the Root Certificate Authority.
3. A valid Server certificate.
4. A valid Client certificate.

We can make use of the openssl tool in order to create these certificates.

First, we need to create the private key that represents our Root CA's private key, then use that key to generate a x.509 certificate. Our Root CA in our case will be self-signed.

# Root private key
openssl genrsa -out certs/rootCA.key 4096

# Root self-signed certificate
openssl req -x509 -new -nodes -key certs/rootCA.key -sha256 -days 3650 \
  -subj "/CN=MyRootCA" -out certs/rootCA.crt

We can the use it to create our Intermediary Certificate Authority. This process requires us to create a Certificate Signing Request (CSR) file.

The CSR file that create is then used alongside our root CA certificate and private key in order to create a our intermediate CA certificate.

To create the Intermediary CA private key, CSR and finally x.509 certificate, we can do the following:

# Intermediate CA config
cat > intermediate-ext.cnf <<EOF
[ext]
basicConstraints = critical, CA:TRUE, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
EOF

# Intermediate CA private key
openssl genrsa -out certs/intermediateCA.key 4096

# Intermediate CSR
openssl req -new -key certs/intermediateCA.key -out certs/intermediateCA.csr \
  -subj "/CN=MyIntermediateCA"

# Use Root CA to sign the intermediate's CSR
openssl x509 -req -in certs/intermediateCA.csr -CA certs/rootCA.crt -CAkey certs/rootCA.key \
  -CAcreateserial -out certs/intermediateCA.crt -days 1825 -sha256 \
  -extfile intermediate-ext.cnf

We first setup a configuration file that is used during the signing process, then create the intermediary private key and certificate signing request before doing the signing to generate the certificate.

You might also be wondering at this point: Why use an intermediary certificate authority?

Here's the key idea:

• The root certificate (public key, usually self-signed) is shared and trusted.
• The root private key is the secret signing key and can be kept offline for safety.
• The root only needs to be used to sign intermediates, which can then go on to sign leaf certs (like for clients or servers).

So in practice:

• The root CA (offline) signs the intermediate CA's cert once.
• The intermediate CA's cert (now signed by the root) is publicly distributed.
• Clients will trust the root certificate (already in their trust store).

In our case, we obviously have the representation of our root CA's certificate stored on our local system, but in practice this would be stored away securely.

Understanding why we don't need the root certificate private key should become clearer as we sign the client and server certificates and show them in use.

As a next step, we can follow a similar process for signing and creating the server certificate:

# Server config
cat > server-ext.cnf <<EOF
[ext]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:localhost
EOF

# Server private key
openssl genrsa -out certs/server.key 2048

# Server CSR
openssl req -new -key certs/server.key -out certs/server.csr \
  -subj "/CN=localhost"

# Sign with intermediate CA
openssl x509 -req -in certs/server.csr -CA certs/intermediateCA.crt -CAkey certs/intermediateCA.key \
  -CAcreateserial -out certs/server.crt -days 825 -sha256 \
  -extfile server-ext.cnf -extensions ext

Note that we use the Intermediate Certificate Authority's certificate and private key for signing this certificate.

Now, let's do the same for the client:

# Client config
cat > client-ext.cnf <<EOF
[ext]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
EOF

# Client private key
openssl genrsa -out certs/client.key 2048

# Client CSR
openssl req -new -key certs/client.key -out certs/client.csr \
  -subj "/CN=client"

# Sign with intermediate CA
openssl x509 -req -in certs/client.csr -CA certs/intermediateCA.crt -CAkey certs/intermediateCA.key \
  -CAcreateserial -out certs/client.crt -days 825 -sha256 \
  -extfile client-ext.cnf -extensions ext

Crucially, the server and client certs were signed by the intermediate, but clients (and the server) only trust the root.

We need to send the full chain when we present a cert. In order to do so, we need to combine our output certificate files:

# Concatenation of certificates
cat certs/server.crt certs/intermediateCA.crt > certs/server-chain.crt
cat certs/client.crt certs/intermediateCA.crt > certs/client-chain.crt

Doing this will create two new certificate files that are inclusive of our intermediary certificate. This way, when the certificate requestor receives the cert, it can verify:

1. The "leaf" certificate (client/server) was signed by intermediate.
2. The intermediate was signed by root.
3. The root is in the trust store.

The concatenated files will be used as our certificates, while the public rootCA.crt file will be used as our CA cert in our mTLS configuration that you will soon see, however it's worth pointing out that you could also concatenate the intermediate and root CA certificats to be used as the CA certificate and then just use the server certificate. The verification process will have the same result.

# Create a full CA chain file for verification as an alternative
cat certs/rootCA.crt certs/intermediateCA.crt > certs/ca-chain.crt

To see how all of this ties in together, let's see it at work with a simple Node.js application that can show when and how our certificates are used.

// server.ts
import https from "node:https";
import fs from "node:fs";
import type { TLSSocket } from "node:tls";

try {
  // Read the files
  const serverKey = fs.readFileSync("certs/server.key");
  const serverCert = fs.readFileSync("certs/server-chain.crt"); // Using individual cert
  const caCert = fs.readFileSync("certs/rootCA.crt"); // Using full CA chain

  function isTlsSocket(socket: unknown): socket is TLSSocket {
    return !!socket && typeof socket === "object" && "authorized" in socket;
  }

  const server = https.createServer(
    {
      key: serverKey,
      cert: serverCert,
      ca: caCert,
      requestCert: true,
      rejectUnauthorized: true,
    },
    (req, res) => {
      const tlsSocket = req.socket;

      if (!isTlsSocket(tlsSocket)) {
        console.error("Invalid socket properties");
        res.writeHead(500);
        res.end("Server error");
        return;
      }

      console.log("TLS Connection Details:");
      console.log(`- Authorized: ${tlsSocket.authorized}`);
      console.log(`- Protocol: ${tlsSocket.getProtocol()}`);
      console.log(`- Cipher: ${tlsSocket.getCipher().name}`);

      if (tlsSocket.getPeerCertificate) {
        const cert = tlsSocket.getPeerCertificate(true);
        console.log(`- Client Subject: ${cert.subject?.CN || "Unknown"}`);
        console.log(`- Client Issuer: ${cert.issuer?.CN || "Unknown"}`);
      }

      if (!tlsSocket.authorized) {
        console.error(`Auth Error: ${tlsSocket.authorizationError}`);
        res.writeHead(401);
        res.end(
          `Client certificate not authorized: ${tlsSocket.authorizationError}`
        );
        return;
      }

      res.writeHead(200);
      res.end("Hello, secure world with intermediate CA!");
    }
  );

  server.on("tlsClientError", (err) => {
    console.error("TLS Client Error:", err);
  });

  server.listen(3000, () => {
    console.log("✅ HTTPS server running at https://localhost:3000");
  });
} catch (err) {
  console.error("Failed to start server:", err);
}

// client.ts
import https from "node:https";
import fs from "node:fs";
import axios from "axios";

// Read the files with better error handling
try {
  const clientKey = fs.readFileSync("certs/client.key");
  const clientCert = fs.readFileSync("certs/client-chain.crt"); // Using individual cert, not chain
  const caCert = fs.readFileSync("certs/rootCA.crt"); // Using full CA chain

  const agent = new https.Agent({
    key: clientKey,
    cert: clientCert,
    ca: caCert,
    rejectUnauthorized: true,
  });

  console.log("Starting mTLS client request...");

  axios
    .get("https://localhost:3000", {
      httpsAgent: agent,
      timeout: 10000, // Longer timeout for debugging
    })
    .then((res) => {
      console.log("✅ Server response status:", res.status);
      console.log("✅ Server response data:", res.data);
    })
    .catch((err) => {
      console.error("❌ Request failed:");
      if (err.response) {
        console.error(`Status: ${err.response.status}`);
        console.error(`Response: ${JSON.stringify(err.response.data)}`);
      } else if (err.request) {
        console.error("No response received from server");
      } else {
        console.error(`Error: ${err.message}`);
      }

      if (err.code) {
        console.error(`Error code: ${err.code}`);
      }

      // More detailed error info for TLS issues
      if (err.cause) {
        console.error("Error cause:", err.cause);
      }
    });
} catch (err) {
  console.error("Failed to read certificate files:", err);
}

$ npx tsx server.ts

> demo-node-mtls@1.0.0 server
> npx tsx server.ts

✅ HTTPS server running at https://localhost:3000
TLS Connection Details:
- Authorized: true
- Protocol: TLSv1.3
- Cipher: TLS_AES_256_GCM_SHA384
- Client Subject: client
- Client Issuer: MyIntermediateCA

$ npx tsx client.ts

> demo-node-mtls@1.0.0 client
> npx tsx client.ts

Starting mTLS client request...
✅ Server response status: 200
✅ Server response data: Hello, secure world with intermediate CA!

This post will cover some of the fundamentals of rolling your own API keys in TypeScript and Node.js 22, but the knowledge is language-agnostic (and runtime agnostic for that matter). I will also close with some remarks on why TypeScript might not be the language of choice for your API keys if you expect any sort of scale.

Two strategies around API Keys

This post will focus on two strategies around rolling your own API keys:

1. Static API keys
2. HMAC-signed API keys

Both strategies have their trade-offs, but if you are time-poor, then I would focus on the keys with request signatures.

We will work through some simple implementations for both that focuses on authentication and, in the case of signed keys, tamper-proofing.

Static keys

Static API keys are simply tokens that a client can include with each request.

Server-side, we verify and ensure the token is valid in order to authenticate the request against our token data store.

You can consider this analogous in ways to providing a password with a request. The key must be sent with every request, kept secret and only transmitted over TLS to prevent eavesdropping.

An example implementation of static keys with a Hono router:

import { OpenAPIHono, createRoute, z } from "@hono/zod-openapi";
import * as crypto from "node:crypto";
import Keyv from "keyv";

// Headers
const apiKeyHeader = z.object({
  Authorization: z.string().openapi({ example: "Bearer pk_live_abc123" }),
});

// In-memory key store
const keyStore = new Keyv({ namespace: "api-key-store" });

function generateApiKey(): string {
  return `pk_live_${crypto.randomBytes(16).toString("hex")}`;
}

function hashApiKey(key: string): string {
  return crypto.createHash("sha256").update(key).digest("hex");
}

export class ApiKeysController extends OpenAPIHono {
  constructor() {
    super();
    this.create();
    this.verify();
  }

  public create() {
    return this.openapi(
      createRoute({
        method: "post",
        path: "/keys/create",
        description: "Create an API key and store its SHA-256 hash",
        responses: {
          200: {
            description: "API key created",
            content: {
              "application/json": {
                schema: z.object({
                  apiKey: z.string(),
                }),
              },
            },
          },
        },
      }),
      async (c) => {
        const plainKey = generateApiKey();
        const hashedKey = hashApiKey(plainKey);

        await keyStore.set(`hash:${plainKey}`, hashedKey);

        return c.json({ apiKey: plainKey });
      }
    );
  }

  public verify() {
    return this.openapi(
      createRoute({
        method: "post",
        path: "/keys/verify",
        description: "Verify an API key using SHA-256 lookup",
        request: {
          headers: apiKeyHeader,
          body: {
            content: {
              "application/json": {
                schema: z.object({}),
              },
            },
          },
        },
        responses: {
          200: {
            description: "Verification result",
            content: {
              "application/json": {
                schema: z.object({
                  valid: z.boolean(),
                }),
              },
            },
          },
        },
      }),
      async (c) => {
        const authHeader = c.req.header("Authorization") ?? "";
        const token = authHeader.replace("Bearer ", "").trim();
        const expectedHash = await keyStore.get(`hash:${token}`);

        if (!expectedHash) return c.json({ valid: false });

        const isValid = hashApiKey(token) === expectedHash;
        return c.json({ valid: isValid });
      }
    );
  }
}

For the sake of understanding our example, I am using the in-memory store from keyv as a stand-in for our data store, @hono/zod-openapi as our augmented Hono server and some helpers from the Node.js crypto module for generating random bytes for our API key and hashing that key to be stored in the database.

Our server has two endpoints:

1. One for creating API keys at /keys/create.
2. A second for verifying those API keys at /keys/verify.

In our /keys/create endpoint, we generate an API key, hash it and then store the hash as a key-value in the database under hash:${plainkey}.

We can call our endpoint like so:

curl http://localhost:3000/v1/api-keys/keys/create \
  --request POST

If we call our endpoint, we end up with a result like the following:

{
  "apiKey": "pk_live_6f6bd5c1b85c5040f4d269ae8c84ce3d"
}

If we then provide that API key under the Authorization header at /keys/verify such as the following:

curl http://localhost:3000/v1/api-keys/keys/verify \
  --request POST \
  --header 'Authorization: Bearer pk_live_6f6bd5c1b85c5040f4d269ae8c84ce3d' \
  --header 'Content-Type: application/json' \
  --data '{}'

...then we can see that our API key is valid.

{
  "valid": true
}

Although static keys can be adequate for many cases, they fall down when extra security is needed. Although that can be rolled or revoked easily, simple API keys cannot verify the identity of the caller or prevent tampering of request parameters in transit, so there are weaker integrity guarantees.

We can work around these problems by introducing request signatures.

Keys with request signatures

Request signatures introduce two values to be shared with the client at the time of API key generation:

1. The access key ID.
2. The secret access key.

Typically, the secret access key is only shared once for the user, so if they lose the secret access key then they will need to re-generate keys.

The client never shares the secret key back to the server. Instead, the access key ID is sent alongside a signature of the request data which is signed using the secret key.

Once the data is sent, the server looks up the token secret and recomputes the signature. If they match, the request is authenticated and proven integral (which is spiel for not being tampered with).

Let's take a look at how an implementation of this works:

import { OpenAPIHono, createRoute, z } from "@hono/zod-openapi";
import * as crypto from "node:crypto";
import Keyv from "keyv";

// Headers
const apiKeyHeader = z.object({
  Authorization: z.string().openapi({ example: "Bearer pk_live_abc123" }),
});

const signedHeaders = apiKeyHeader.extend({
  "X-Signature": z.string().openapi({ example: "a3d4c9..." }),
  "X-Timestamp": z.string().openapi({ example: "1712333445" }),
});

// In-memory key store
const keyStore = new Keyv({ namespace: "api-key-store" });

function generateApiKey(): string {
  return `pk_live_${crypto.randomBytes(16).toString("hex")}`;
}

function hashApiKey(key: string): string {
  return crypto.createHash("sha256").update(key).digest("hex");
}

function generateHmac(key: string, secret: string): string {
  return crypto.createHmac("sha256", secret).update(key).digest("hex");
}

export class ApiKeysController extends OpenAPIHono {
  constructor() {
    super();

    this.verify();
    this.secureVerify();

  public secureCreate() {
    return this.openapi(
      createRoute({
        method: "post",
        path: "/keys/secure-create",
        description: "Create an HMAC-style API key",
        responses: {
          200: {
            description: "Key pair created",
            content: {
              "application/json": {
                schema: z.object({
                  accessKeyId: z.string(),
                  secretAccessKey: z.string(),
                }),
              },
            },
          },
        },
      }),
      async (c) => {
        const accessKeyId = generateApiKey();
        const secretAccessKey = crypto.randomBytes(32).toString("hex");

        // WARNING: In real-world applications, encrypt this value.
        await keyStore.set(`hmac:${accessKeyId}`, { secret: secretAccessKey });

        return c.json({
          accessKeyId,
          secretAccessKey, // return only once!
        });
      },
    );
  }

  public secureVerify() {
    return this.openapi(
      createRoute({
        method: "post",
        path: "/keys/secure-verify",
        description: "Verify HMAC-signed API request using access/secret key",
        request: {
          headers: signedHeaders,
          body: {
            content: {
              "application/json": {
                schema: z.object({
                  message: z.string(),
                }),
              },
            },
          },
        },
        responses: {
          200: {
            description: "Verification result",
            content: {
              "application/json": {
                schema: z.object({
                  valid: z.boolean(),
                }),
              },
            },
          },
        },
      }),
      async (c) => {
        const authHeader = c.req.header("Authorization") ?? "";
        const signature = c.req.header("X-Signature") ?? "";
        const timestamp = c.req.header("X-Timestamp") ?? "";
        const token = authHeader.replace("Bearer ", "").trim();

        const stored = await keyStore.get(`hmac:${token}`);
        if (!stored?.secret) return c.json({ valid: false });

        const body = await c.req.json();
        const digest = generateHmac(
          `${timestamp}:${JSON.stringify(body)}`,
          // WARNING: In real-world applications, this would be an encrypted value
          // that we should decrypt first.
          stored.secret,
        );

        const valid = crypto.timingSafeEqual(
          Buffer.from(signature),
          Buffer.from(digest),
        );

        return c.json({ valid });
      },
    );
  }
}

In our modified code, we again have two endpoints:

1. /keys/secure-create
2. /keys/secure-verify

In addition to this, our /keys/secure-verify endpoint now requires three headers:

1. Authorization: Our same authorization header.
2. X-Timestamp: A timestamp to denote when the signing occurred.
3. X-Signature: Our signed payload.

We will see those headers in action in a moment, but let's park that there for now.

In our /keys/secure-create endpoint, we now return two key pieces of information. Let's see this in action.

curl http://localhost:3000/v1/api-keys/keys/secure-create \
  --request POST

This returns the following JSON:

{
  "accessKeyId": "pk_live_f5c4672457c78c8e019df66eeabbe33e",
  "secretAccessKey": "2a39d2316ef07ebb7c763f3eb06b0ef3aafbd7f0b3d0f2b3bd5acfa6e0b57fd5"
}

On the server, we store the secret for us to fetch later (please see the callout below on this) in order to be used for validating the signature. We then return both values once. Warn the user to securely store the secret, as we won't be returning that value again.

In order to see how signing works, here is a helper script that I used to test against our verification endpoint /keys/secure-verify:

import crypto from "node:crypto";
import axios from "axios";

const { accessKeyId, secretAccessKey } = {
  accessKeyId: "pk_live_f5c4672457c78c8e019df66eeabbe33e",
  secretAccessKey:
    "2a39d2316ef07ebb7c763f3eb06b0ef3aafbd7f0b3d0f2b3bd5acfa6e0b57fd5",
};
const timestamp = Math.floor(Date.now() / 1000).toString();

const body = {
  message: "Hello, HMAC-secured world!",
};

// Compute HMAC using timestamp and JSON body
const digest = crypto
  .createHmac("sha256", secretAccessKey)
  .update(`${timestamp}:${JSON.stringify(body, null, 0)}`)
  .digest("hex");

const start = performance.now();
const res = await axios.post(
  "http://localhost:3000/v1/api-keys/keys/secure-verify",
  body,
  {
    headers: {
      Authorization: `Bearer ${accessKeyId}`,
      "X-Timestamp": timestamp,
      "X-Signature": digest,
      "Content-Type": "application/json",
    },
  }
);
const end = performance.now();
console.log("Time:", end - start);

console.log("Response:", res.data);

In the above code we use the crypto.createHmac function in order to sign the request, and then we pass it along as our X-Signature header. The header itself uses update to incorporate our concatenated timestamp and body payload string into the signature.

The data used to verify the signature can include whatever we want, so if there are any extra headers or pieces of data that are important to validating integrity, then be sure to reflect that in the signature.

Once signed, we send a request that includes the required headers. Values such a X-Timestamp are required in order to verify the signature server-side and can also be used to reject signatures older than a certain time period.

In order to compare the signatures on the server-side implementation, we have the following:

const valid = crypto.timingSafeEqual(
  Buffer.from(signature),
  Buffer.from(digest)
);

crypto.timingSafeEqual is a function that compares the underlying bytes using a constant-time algorithm. It does not leak timing information which is an attack surface for malicious actors and is suitable for comparing HMAC digests.

If you are interested in reading up more about authentication practices like timing attacks, then see the OWASP authentication cheat sheet.

If we run the Node.js script, we will get back the expect Response: { valid: true } payload.

Play around and attempt to tamper some of the values and see what happens. Unsurprisingly, valid is false.

Authorization: AWS4-HMAC-SHA256
Credential=AKIAIOSFODNN7EXAMPLE/20220830/us-east-1/ec2/aws4_request,
SignedHeaders=host;x-amz-date,
Signature=calculated-signature

https://ec2.amazonaws.com/?
Action=DescribeInstances&
Version=2016-11-15&
X-Amz-Algorithm=AWS4-HMAC-SHA256&
X-Amz-Credential=AKIAIOSFODNN7EXAMPLE/20220830/us-east-1/ec2/aws4_request&
X-Amz-Date=20220830T123600Z&
X-Amz-SignedHeaders=host;x-amz-date&
X-Amz-Signature=calculated-signature

Increasing your test confidence vector is a hot topic. Some of my greatest mistakes and, as far as I am concerned, most useless code has written done around testing.

In this post, we'll focus on principles for static analysis, unit testing and integration testing that can increase your confidence vector as much as possible while limiting some of the pain points. Some of these points are certain to be contentious and are unashamedly biased from the parts of testing that have consumed way-too-many-hours-to-admit from my short tenure on this planet.

Companion code

Unlike my other blog posts, this one won't be building anything from scratch. There is a companion project to this blog post that I will often link to and share code from throughout this post.

The principles I will talk through today are mostly agnostic to language, tool and runtime, although whether it's idiomatic or frowned up depends on your community and how well-equipped your chosen ecosystem is.

We'll be walking through these principles in TypeScript to satiate some of my recent thoughts.

Issues to acknowledge with test coverage

Software teams use a minimum coverage percentage as a quality gate, but what that number is varies from source-to-source. Google's testing blog has testing coverage values where 60% is acceptable, 75% is commendable and 90% is exemplary. Atlassian's page on code coverage mentions that 80% coverage is good to aim for, but higher coverage might yield dimishing returns.

I personally have two main gripes around test coverage:

1. The reported coverage can be misleading.
2. Strict quality gates can lead to writing tests for the sake of writing tests, which leads to bad tests.

To illustrate some of the problems with code coverage, let's start with the basics. Take the following pure function add:

export function add(a: number, b: number) {
  return a + b;
}

We can write a simple test to validate this:

import { expect, test } from "vitest";
import { add } from "./math";

test("should add 1 + 2", () => {
  expect(add(1, 2)).toEqual(3);
});

With vitest, if I test with coverage e.g. vitest --coverage, I get the following output:

--------------------------|---------|----------|---------|---------|-------------------
File                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
--------------------------|---------|----------|---------|---------|-------------------
All files                 |     100 |      100 |     100 |     100 |
 utils                    |     100 |      100 |     100 |     100 |
  math.ts                 |     100 |      100 |     100 |     100 |
--------------------------|---------|----------|---------|---------|-------------------

Great! We have 100% coverage! No need for tests to try to trip up the boundaries, right? Well, writing a test to check 1 + 2 isn't exactly adventurous. We hardly pushed our function and it looks as if we're at the finish line.

Other than pushing the limits of our function, what happens if someone has come in and adds some sweet nothings to our function?

interface Loggable {
  log: (msg: string) => void;
}

function sideEffect(logger: Loggable) {
  logger.log("yo");
}

export function add(a: number, b: number, logger: Loggable) {
  sideEffect(logger);
  return a + b;
}

We have a new side effect introduced, but what happens if we update our test to pass through a console argument?

import { expect, test, vi } from "vitest";
import { add } from "./math";

test("should add 1 + 2", () => {
  const result = add(1, 2, console);
  expect(result).toEqual(3);
});

--------------------------|---------|----------|---------|---------|-------------------
File                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
--------------------------|---------|----------|---------|---------|-------------------
All files                 |     100 |      100 |     100 |     100 |
 utils                    |     100 |      100 |     100 |     100 |
  math.ts                 |     100 |      100 |     100 |     100 |
--------------------------|---------|----------|---------|---------|-------------------

Still 100%, what gives!? In a world where console.log is injected a logger object that throws an error e.g. add(1, 2, { log: () => { throw new Error('KABOOM') }}), we're still at 100% coverage.

Worst of all, what happens if they changed our test to validate only the logging?

import { expect, test, vi } from "vitest";
import { add } from "./math";

test("should add 1 + 2", () => {
  const spy = vi.spyOn(console, "log");
  add(1, 2, console);
  expect(spy).toHaveBeenCalledOnce();
});

If we check out test coverage again...

--------------------------|---------|----------|---------|---------|-------------------
File                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
--------------------------|---------|----------|---------|---------|-------------------
All files                 |     100 |      100 |     100 |     100 |
 utils                    |     100 |      100 |     100 |     100 |
  math.ts                 |     100 |      100 |     100 |     100 |
--------------------------|---------|----------|---------|---------|-------------------

We didn't even test our return value!

Code coverage, at least in the forms that I've seen it, is a guideline. In reality, it's unlikely for me to come across tests that don't verify something as important as the return value. What has been problematic for me though have been high test coverage thresholds that force your hand.

This has happened to me plenty of times where I have ended up writing tests that do nothing other than get me passed the numbers required to pass CI:

// code
function someService(id: string): { id: string; title: string } {
  return someRepo(id);
}

// test
test("someService", () => {
  const spy = vi.spyOn(someRepo);
  spy.mockImplementationOnce((id: string) => ({ id: "1", title: "title" }));
  const res = someService("123");
  expect(res).toBe({ id: "1", title: "title" });
});

In testing environments that require you to mock out responses with hazy boundaries, you can end up writing tests like this simply to make coverage requirements.

To summarize: committing to code coverage can be hazardous. If you're going to do it, pick a safe threshold and embed it into the engineering culture at your company to prioritize high-quality tests that are not easily fallible.

As we continue into the age of AI assistance, configure your review agents to help scrutinize your choice of edge cases and make sure that any iffy boundaries are covered by tests.

Issues to acknowledge with mocking, spies and test boundaries

Mocks are commonly used within testing to help with isolation. This can be helpful, but can become a scapegoat with setups that feel non-trivial to configure correctly to better represent your system.

In the most recent code example, we hinted towards test boundaries that can negatively impact our test confidence.

But what is a "test boundary"? My definition of the test boundary is the scope of what we can accurately test against. Accuracy being the crucial point here.

Let's look out test again, make some adjustments and add some more context on what the someRepo function does:

// code
async function someRepo(id: string): { id: string; title: string } {
  console.log("Attempting fetch from cache");
  const cacheResult = await checkCache(id);
  if (cacheResult) {
    return cacheResult;
  }

  console.log("Attemping fetch from database");
  const dbResult = await fetchFromDb(id);
  return dbResult;
}

async function someService(id: string): { id: string; title: string } {
  return someRepo(id);
}

// test
test("someService", async () => {
  const spy = vi.spyOn(someRepo);
  spy.mockImplementationOnce((id: string) => ({ id: "1", title: "title" }));
  const res = await someService("123");
  expect(res).toBe({ id: "1", title: "title" });
});

In our code now, the someRepo function checks a remote cache for a result, returns it if there is a hit or returns the result from the database. The test hasn't changed from what it was previously.

Ask yourself the question: what boundaries of this functionality are tested with the current implementation? The entire raison d'être for our someService test is to evaluate it's functionality, but here we do not step in into our code beyond the current someService context due to our mock.

In practice, increasing our test confidence requires us to increase how far we can "step in" our code in order to reliably test as close to our system as possible.

As a first pass at our test refactoring, what happens if we update our mocks to move to the boundary where they mock our checkCache and fetchFromDatabase instead?

describe("someService", () => {
  test("returns expected value from database", async () => {
    const consoleSpy = vi.spyOn(console);
    const checkCacheSpy = vi.spyOn(checkCache);
    const fetchFromDbSpy = vi.spyOn(fetchFromDb);
    checkCacheSpy.mockImplementationOnce((id: string) => null);
    fetchFromDbSpy.mockImplementationOnce((id: string) => ({
      id: "1",
      title: "title",
    }));

    const res = await someService("123");
    expect(res).toBe({ id: "1", title: "title" });
    expect(checkCacheSpy).toHaveBeenCalledOnce();
    expect(fetchFromDb).toHaveBeenCalledOnce();
    expect(console).toHaveBeenCalledTimes(2);
  });

  test("returns expected value from cache", async () => {
    const consoleSpy = vi.spyOn(console);
    const checkCacheSpy = vi.spyOn(checkCache);
    const fetchFromDbSpy = vi.spyOn(fetchFromDb);
    checkCacheSpy.mockImplementationOnce((id: string) => ({
      id: "1",
      title: "title",
    }));

    const res = await someService("123");
    expect(res).toBe({ id: "1", title: "title" });
    expect(checkCacheSpy).toHaveBeenCalledOnce();
    expect(fetchFromDb).not.toHaveBeenCalledOnce();
    expect(console).toHaveBeenCalledOnce();
  });
});

In our tests now, we have shifted the test boundaries a little deeper to where checkCache and fetchFromDb begin. We can improve on this more, but first let's take a moment to consider our spies and .toHaveBeenCalledOnce calls.

Consider the following refactor:

async function someRepo(id: string): { id: string; title: string } {
  console.log("Running someRepo");
  const cacheResult = await checkCache(id);
  if (cacheResult) {
    return cacheResult;
  }

  // CONSOLE LOG REMOVED
  const dbResult = await fetchFromDb(id);
  return dbResult;
}

In our someRepo code, we've refactored the logs but the core functionality has not changed. If we run our tests though... they fail.

You can probably know why: one of our tests expect console to have been called twice. This is where we fall into "false negative" territory. As far as desired output behavior goes for our someRepo function, it still operates the same as before, but our tests were written as a false representation of that expected behavior.

Unless it is absolutely necessary to ensure a function is called as part of another, my ethos is that you should omit it entirely. We'll justify the reasons in an upcoming section, but for now my starting guideline would be to test the inputs and outputs.

For now, let's refactor the tests back to the following:

describe("someService", () => {
  test("returns expected value from database", async () => {
    const checkCacheSpy = vi.spyOn(checkCache);
    const fetchFromDbSpy = vi.spyOn(fetchFromDb);
    checkCacheSpy.mockImplementationOnce((id: string) => null);
    fetchFromDbSpy.mockImplementationOnce((id: string) => ({
      id: "1",
      title: "title",
    }));

    const res = await someService("123");
    expect(res).toBe({ id: "1", title: "title" });
  });

  test("returns expected value from cache", async () => {
    const checkCacheSpy = vi.spyOn(checkCache);
    checkCacheSpy.mockImplementationOnce((id: string) => ({
      id: "1",
      title: "title",
    }));

    const res = await someService("123");
    expect(res).toBe({ id: "1", title: "title" });
  });
});

Now we can freely refactor non-critical side-effects like console.log without returning false negatives about the desired function's behavior.

An example where validating calls made would be for functions where their entire purpose is to emit expected, measurable side-effects that the system should prevent from unintended changes.

In the accompanying repository, see these tests for a Proxy where the entire purpose is to emit logs at the beginning and end of a function call.

import { beforeEach, describe, expect, test, vi } from "vitest";
import { addTrace, addTraceCurried } from "./add-trace";
import { container } from "../config/ioc-test";
import { ILoggerService } from "@/services/logger-service";
import { IocKeys } from "@/config/ioc-keys";

class Target {
  needle() {
    return null;
  }

  async needleAsync() {
    return Promise.resolve(null);
  }
}

describe("trace proxies", () => {
  let logger: ILoggerService;

  beforeEach(() => {
    logger = container.get<ILoggerService>(IocKeys.LoggerService);
  });

  describe("addTrace", () => {
    test("invokes two trace logs at the beginning and end of a function", () => {
      const spy = vi.spyOn(logger, "trace");
      const target = addTrace(new Target(), logger);

      expect(target.needle()).toBeNull();
      expect(spy).toHaveBeenCalledTimes(2);
      expect(spy.mock.calls).toEqual([
        [{ args: [], className: "Target" }, `Target.needle() called`],
        [
          { className: "Target", duration: expect.any(String) },
          `Target.needle() returned`,
        ],
      ]);
    });

    test("invokes two trace logs at the beginning and end of an async function", async () => {
      const spy = vi.spyOn(logger, "trace");
      const target = addTrace(new Target(), logger);

      expect(await target.needleAsync()).toBeNull();
      expect(spy).toHaveBeenCalledTimes(2);
      expect(spy.mock.calls).toEqual([
        [{ args: [], className: "Target" }, `Target.needleAsync() called`],
        [
          { className: "Target", duration: expect.any(String) },
          `Target.needleAsync() returned`,
        ],
      ]);
    });
  });

  describe("addTraceCurried", () => {
    test("full application invokes two trace logs at the beginning and end of a function", () => {
      const spy = vi.spyOn(logger, "trace");
      const partiallyAppliedFn = addTraceCurried(logger);
      const target = partiallyAppliedFn(new Target());

      expect(target.needle()).toBeNull();
      expect(spy).toHaveBeenCalledTimes(2);
      expect(spy.mock.calls).toEqual([
        [{ args: [], className: "Target" }, `Target.needle() called`],
        [
          { className: "Target", duration: expect.any(String) },
          `Target.needle() returned`,
        ],
      ]);
    });

    test("invokes two trace logs at the beginning and end of an async function", async () => {
      const spy = vi.spyOn(logger, "trace");
      const partiallyAppliedFn = addTraceCurried(logger);
      const target = partiallyAppliedFn(new Target());

      expect(await target.needleAsync()).toBeNull();
      expect(spy).toHaveBeenCalledTimes(2);
      expect(spy.mock.calls).toEqual([
        [{ args: [], className: "Target" }, `Target.needleAsync() called`],
        [
          { className: "Target", duration: expect.any(String) },
          `Target.needleAsync() returned`,
        ],
      ]);
    });
  });
});

In these tests, I explicitly spy on the instance of the ILoggerService to ensure that logger.trace is called with the expected arguments as that is the purpose of the proxy.

For this particular test, asserting that the dependency calls is as much confidence as I want from these tests.

For the astute, you will have notice that spies don't inherently mock out functionality unless explicitly stated. This means that I can assert the desired functionality while testing against a real implementation.

Just to reiterate: I do not use spies like this for every function call.

Remember what we covered about false negatives. I will only ever do this in scenarios where a side effect is the sole desired outcome for the function existing.

In these scenarios, there is normally no response or the side effect has higher priority. In my example, the proxy return type plays second fiddle to the trace logging.

For impure functions that return values that you care about (from databases etc), assert against output and do not get caught up on the implementation details.

In the accompanying codebase, you can see some examples of testing a repository layer that cares more about the output than the implementation details:

describe("getBlog", () => {
  let post: Post;

  beforeEach(async () => {
    post = postFactory.build();

    await prisma.post.create({
      data: post,
    });
  });

  test("can get a blog", async () => {
    const result = await blogRepository.getBlog({ param: { blogId: post.id } });
    expect(result.isOk()).toBe(true);
    result.map((value) => {
      expect(value.data?.title).toBe(post.title);
      expect(value.data?.content).toBe(post.content);
    });
  });

  test("can retrieve a blog from cache", async () => {
    const noCacheResult = await blogRepository.getBlog({
      param: { blogId: post.id },
    });
    expect(noCacheResult.isOk()).toBe(true);
    noCacheResult.map((value) => {
      expect(value._cacheHit).toBe(false);
    });

    const result = await blogRepository.getBlog({ param: { blogId: post.id } });
    expect(result.isOk()).toBe(true);
    result.map((value) => {
      expect(value._cacheHit).toBe(true);
    });
  });

  test("returns BlogNotFoundError when there is no matching blog found", async () => {
    const result = await blogRepository.getBlog({
      param: { blogId: faker.string.uuid() },
    });
    expect(result.isErr()).toBe(true);
    result.mapErr((e) => {
      expect(e._tag).toBe("BlogNotFoundError");
    });
  });
});

Notice that it does not check that certain wrapper library functionality is called. Is purely cares on whether or not the functionality returns the expected data and information about hitting the cache. If I swapped out cache implementation or ORM but kept the contract, then this test will not need to change and it will be the best indicator that I've maintained the expected functionality.

Finally, I wanted to finish by acknowledging an area where I find mocking appropriate at the cost of shortening the test boundary: error cases that are not easily replicated and are explicitly handled for unhappy paths.

Applications are fallible to the most bizarre things, so if you're working with a system that handles edge cases that are not easily replicated but are actively handled within your application, then mocking will come to the rescue.

For example (albeit contrived), what happens if you're writing code that specifically handles cases where a disk is full?

// file-service.ts
import { promises as fs } from "fs";
import path from "path";
import { err, ok } from "neverthrow";

/**
 * Saves data to a file, with retry logic for edge cases
 * @param filePath Path to save the file
 * @param data Content to write
 */
async function saveFile(filePath: string, data: string): Promise<void> {
  try {
    // Create directory if it doesn't exist
    const directory = path.dirname(filePath);
    await fs.mkdir(directory, { recursive: true });

    // Write the file
    await fs.writeFile(filePath, data, { encoding: "utf8" });
    return ok(true);
  } catch (error) {
    if (err.code === "ENOSPC") {
      // No space left on device
      // Very difficult to emulate without actually filling disk
      return err({
        _tag: "DiskFull",
        somethingImportant: true,
      });
    }

    return err({
      _tag: "ContrivedCatchAll",
    });
  }
}

In this example, replicating the disk full scenario is not easily done. Mocking comes to our rescue here:

// file-service.test.ts
import { describe, beforeEach, it, expect, vi } from "vitest";
import { promises as fs } from "fs";
import path from "path";
import { saveFile } from "./file-service";

// Mock the fs module
vi.mock("fs", () => ({
  promises: {
    writeFile: vi.fn(),
    mkdir: vi.fn().mockResolvedValue(undefined),
  },
}));

describe("saveFile", () => {
  beforeEach(() => {
    vi.resetAllMocks();
    vi.mocked(fs.mkdir).mockResolvedValue(undefined);
  });

  it("should handle disk full error correctly", async () => {
    // Mock writeFile to throw an ENOSPC error
    const mockError = new Error("No space left on device");
    // Add the code property to the error
    (mockError as any).code = "ENOSPC";

    // Setup the mock to reject with our custom error
    vi.mocked(fs.writeFile).mockRejectedValue(mockError);

    const result = await saveFile("/path/to/file.txt", "test data");

    // Verify the error was handled correctly
    expect(result.isErr()).toBeTruthy();
    result.mapErr((e) => {
      expect(e._tag).toBe("DiskFull");
      expect(e.somethingImportant).toBe(true);
    });
  });
});

Guidelines for test boundaries

With our test boundaries, how far can and should we push? I believe this strongly depends on preferences and configuration, but my guidelines:

1. For tests that compose of functionality that makes network requests, test out until the network boundary with request interceptors like MSW where possible.
2. For tests that compose of functionality that interface directly with databases, attempt to run tests against a local copy of the expected database using tools like testcontainers.
3. For tests that interface with third party library utilities, test without mocking out the utilities.
4. For other side effects types like interfacing with file systems, if it's undesirable to write to the file-system and clean up, then attempt to mock in-memory. For something like Node.js, there exists utilities like mock-fs.

These guidelines begin to blur the definitions between unit testing and integration testing. If you're familiar with the testing pyramid, then you may favor unit tests as your baseline. Over time, I've personally grown to become more in favor of model closer to the testing trophy which places emphasis on static analysis as your baseline and integration testing as the testing layer to focus on most.

Most of the time, blurring these lines is inconsequential for projects. For projects that require more setup and most importantly are actively becoming an acknowledged problem and time consuming to run, then consider updating your testing configuration to separate these.

Finally, one common bottleneck that I haven't called out here is that to prevent state management race conditions for dependencies like databases, you generally need to forego any parallelization for tests running on the same machine. I should reiterate that until this becomes a problem, I wouldn't be too concerned about this. Locally, tests runners allow running a subset of tests. On CI, there are also strategies to split tests across test runners with isolated database that might be worth considering first. Seek out alternatives first and proactively evaluate any changes to your approach.

An example configuration for enforcing single-threaded, sequential testing with Vitest:

import { defineConfig } from "vite";

export default defineConfig({
  test: {
    // ... some data omitted
    pool: "forks",
    poolOptions: {
      forks: {
        singleFork: true,
      },
    },
    globalSetup: ["./src/tests/vitest-global-setup.ts"],
    setupFiles: ["./src/tests/vitest-setup.ts", "./src/tests/matchers.ts"],
  },
});

In my example code, I'm spinning up both a Valkey and Postgres test container, along with using MSW to intercept my HTTP requests to a make-believe service, so I use the global setup and setup files to prepare this.

My global setup will stand up the databases and tear them down at the end of the tests:

import { execSync } from "child_process";
import { psqlContainer, valkeyContainer } from "../lib/testcontainers";

export async function setup() {
  // Start the PostgreSQL container
  const [psql, valkey] = await Promise.all([psqlContainer, valkeyContainer]);
  const databaseUrl = psql.getConnectionUri();
  process.env.DATABASE_URL = databaseUrl;

  // Start the Valkey container
  const valkeyUrl = valkey.getConnectionUrl();
  process.env.VALKEY_URL = valkeyUrl;

  // Run migrations and seed your database
  execSync("npx prisma migrate reset --force --skip-seed", {
    stdio: "inherit",
    env: { ...process.env, DATABASE_URL: databaseUrl },
  });
}

export async function teardown() {
  const psql = await psqlContainer;
  const valkey = await valkeyContainer;

  await Promise.all([psql.stop(), valkey.stop()]);
}

My setup files apply Vitest hooks in order to ensure that my databases and MSW mocks are cleared between runs to prevent race conditions:

import { server } from "../mocks/server";
import { PrismaClient } from "@prisma/client";
import { container } from "../config/ioc-test";
import { beforeAll, afterEach, afterAll, beforeEach } from "vitest";
import { IocKeys } from "../config/ioc-keys";

const keyv = container.get<any>(IocKeys.KeyvClient);
const prisma = container.get<PrismaClient>(IocKeys.PrismaClient);

/**
 * Clean all tables in the database
 */
async function cleanDatabase() {
  // Get all tables
  const tables = await prisma.$queryRaw<Array<{ tablename: string }>>`
    SELECT tablename FROM pg_tables 
    WHERE schemaname='public' 
    AND tablename NOT IN ('_prisma_migrations')
  `;

  // Truncate each table
  for (const { tablename } of tables) {
    try {
      await prisma.$executeRawUnsafe(`TRUNCATE "${tablename}" CASCADE;`);
    } catch (error) {
      console.log(`Error truncating ${tablename}`);
    }
  }
}

beforeAll(async () => {
  server.listen({ onUnhandledRequest: "bypass" });
}, 30000);

beforeEach(async () => {
  // Clean the database before each test to start fresh
  await cleanDatabase();
});

afterEach(async () => {
  await keyv.clear();
  server.resetHandlers();
});

afterAll(async () => {
  server.close();

  await prisma.$disconnect();
  await keyv.disconnect();
});

In the example code, I've chosen to use table truncation for clearing the relational database between runs as it feels the simplest for my use case.

The time is takes for me to run the tests from a fresh start with cached containers is about 7 seconds for the full suite, which includes starting up the Docker containers (albeit there are only ~30 tests).

Remotely, the CI job takes ~16 seconds.

Test data guidelines

The previous section covered database integration into our testing, so let's recap some guidelines that around this:

1. Serialize tests if tests share the same database.
2. Write global setup and teardown functionality to ensure safe setup and teardown of your database(s).
3. Enforce a clean slate before each test.
4. Colocate the test data within the test file. It should be easy to track down where data was initialized from.

There may be instances where some of these guidelines don't apply. For example. some possible options for database isolation that have different trade-offs for cost:

1. Database isolation per test.
2. Transactions per test that are rolled back.
3. Truncating tables after each tests.

In addition to that, testing in CI may open up avenues to splits tests up across shards that may allow you to configure different databases per shard.

The problems with emulation

I'll keep this section short, but as a general rule of thumb: if you are going to test your application beyond the application boundary, then beware emulation. If you can't use the actual technology locally, then question it's use at all.

This is more a warning than a "never do it" scenario, because there are times where things like drop-in replacements for providing in-memory APIs can be incredibly useful.

In general though, if you're using something 3rd party to emulate something in your environment that belongs to closed-source work, know that any breaking changes or upgrades to the original product will cause you grief (on top of any bugs or lack-of-support from the 3rd party software). Weigh up the trade-offs before implementing something like this.

Using cloud technologies is an example of something I will never bother to emulate for testing or development in my personal work. If something is not officially provided to run locally, I don't bother. I learned this the hard way and spent way too many hours debugging 3rd party emulators when running on the actual platform worked as expected.

You're better off testing on cloud hardware if you ever end up in a scenario like this. Once again, evaluate whether or not the cost and complexity of doing so is worth it.

Pure functions and bounded domains and ranges

Pure functions have two properties:

1. No side-effects: Things like logging, interfacing with networks or file systems constitutes a side-effect.
2. Same input == same output: When passing the exact same arguments in, we expect the same result every time.

Let's go back to a previous example of implementing add:

interface Loggable {
  log: (msg: string) => void;
}

function sideEffect(logger: Loggable) {
  logger.log("yo");
}

export function add(a: number, b: number, logger: Loggable) {
  sideEffect(logger);
  return a + b;
}

We know this violates rule (1) of pure functions due to it's side-effect, so let's refactor a little more and add some constraints around our add function:

function constrainedAdd(a: number, b: number) {
  if (isNaN(a) || isNaN(b)) {
    return 0;
  }

  a = a < 0 ? 0 : a > 2 ? 2 : a;
  b = b < 0 ? 0 : b > 2 ? 2 : b;

  return Math.floor(a) + Math.floor(b);
}

This completely useless function will add two integers together, with any argument smaller than 0 being set as 0 and any argument larger than 2 being set to 2. It also enforces the numbers added to be integers.

What happens if we were to put it to use?

constrainedAdd(1, 2); // 3
constrainedAdd(-1, -1); // 0
constrainedAdd(-1, 3); // 2
constrainedAdd(3, 3); // 4
constrainedAdd(1.9, 2.1); // 3
constrainedAdd(NaN, 2); // 0

This function may appear useless, but it is a pure function.

1. There are no side effects.
2. Each valid input will return the same output.

Here is another useful tidbit about pure functions: functions that are only composed of other pure functions are themselves pure. Assuming Math.floor and isNaN themselves are pure functions, our constrainedAdd does not violate any rules on pure functions. This has some useful properties that make our lives with testing much easier.

The next important lesson we can take away from constrainedAdd is how we define the domain and range of the function.

The domain describes the bounds of valid input, and any given domain input itself for a pure function can map directly to the range which is a set of all possible outputs.

For our function, our domain for the arguments can be described as any valid number, but with this implementation it's our range that is is most interesting. Our output is locked to be any valid integer within the set {0,1,2,3,4}.

The beauty of pure functions it makes life easier to test against edge cases if you can understand the range.

it.each([
  // [a, b, expected]
  [1, 2, 3], // Normal case within constraints
  [-1, -1, 0], // Both negative values constrained to 0
  [-1, 3, 2], // Negative and above constraint combination
  [3, 3, 4], // Both values above constraints
  [1.9, 2.1, 3], // Decimal values with flooring and constraints
  [NaN, 2, 0], // NaN constrains the output to 0
])("adds constrained %f and %f to equal %i", (a, b, expected) => {
  expect(constrainedAdd(a, b)).toBe(expected);
});

In a test like the one above, we increase our test confidence vector by increasing our assertion coverage for as many edge cases as we can infer from the domain and range.

Although math and numbers are the most relatable when it comes to domain and range (for anyone stuck doing Calculus anyway), this also applies to any argument where the output is consistent for the same input.

The following is also a pure function:

function getSecret(code: "alpha" | "bravo" | "delta") {
  switch (code) {
    case "alpha":
      return "hello";
    case "bravo":
      return "from";
    case "delta":
      return "planet earth";
  }
}

In the above, our domain is one of three values alpha, bravo or delta while the range is hello, from and planet earth.

In hindsight after writing this, explaining domain and range with getSecret seems easier to understand than my constrainedAdd function...

Functional core, imperative shell

This pattern is something I first came across at Ruby conferences five or so years ago and it is a pattern that I've written about before.

The heart of this concepts lies around what we've learned so far related to pure functions and composition. Pure functions are easier to reason about, and so if you start with your "deepest" code as being composed of pure functions (a.k.a. a functional core), then testing those functions can yield more consistency.

destroyallsoftware has a great screencast outlining this pattern.

In practical terms, if we consider our controller-service-repository layer application, we already know that we can't avoid side-effects with our repository layers. That's okay, the idea is to invert as much of our impure logic upward until we hit our "imperative shell". The boundary of the shell and core can be thought of as fluid.

For example, take the following:

function functionB() {
  console.log("core started");
  somePureFunction();
  console.log("core ended");
}

function functionA() {
  functionB();
}

functionA();

At our current state, assuming that somePureFunction is in fact pure, then our functional core boundary starts at the somePureFunction invocation (and our imperative shell boundary ends), because functionB is impure thanks to the logs.

Let's refactor again:

function functionB() {
  somePureFunction();
}

function functionA() {
  console.log("core started");
  functionB();
  console.log("core ended");
}

functionA();

If we "push up" the logs to functionA, then functionB through composition has become a pure function itself, so now the functional core starts at the invocation of functionB.

In my mind, the ideology of "functional core, imperative shell" isn't explicit that certain layers are entirely pure or impure. I think of it as "dipping into" functional cores at those boundaries. You can had methods within your service layer that are entire pure, or they're impure because they call into functionality that interfaces with something like our databases.

Something I think is worthwhile highlighting is that my "production" Inversify container injects a proxy for logging into the layers with addTrace, whereas my Inversify test container does not. This means that, when testing, none of the composed classes are automatically made impure from the proxy. This is a purposeful decision.

In practice, I find that service layers methods are the point where you normally "dip in and out" of the line between the functional core and imperative shell, but that's not a hard rule. Business layer logic that operates on concrete instances can normally be implemented as functional while calls out to the database can only ever be impure.

Dependency injection

Dependency injection has many benefits, but it also enables a level of control that can be second-to-none around testing. Inverting control of what is constructed for a class instance can be a powerful tool.

Take the following code:

interface IUser {
  id: string;
  name: string;
}

interface IUserRepository {
  findById(id: string) => IUser;
}

class UserRepository extends IUserRepository {
  async findById(id: string) {
    return someWayToGetUser(id);
  }
}

class Service {
  private userRepo: IUserRepository

  constructor(userRepo: IUserRepository) {
    this.userRepo = userRepo;
  }

  async findUser(id: string) {
    const user = await findUser(id);
    return user;
  }
}

In our contrived code, we've inverted control over the userRepo to be anything that adheres to the IUserRepository.

When it comes to testing, we can inject anything to adheres to the IUserRepository contract, be that an instance of UserRepository or any mock that adheres to the contract.

In cases where you use tools like Effect, tsyringe or Inversify, this can make it trivial to provide test-specific dependencies where required.

Within the accompanying repository, I've configured Inversify with a test-specific container. This testing container can be used to supply the mock implementations or remove unnecessary behaviour (like I have with removing higher-order proxies).

Conclusion

Today's post covered a lot of testing methodology and guidelines that enable you to reason more about what effective testing can look like for you.

A recap of the most important talking points:

• The problem with code coverage
• The impacts mocking can have on your test confidence vector

Photo credit: zhangkaiyv

"First, your refactoring was not part of our negotiations nor our agreement so I must do nothing. And secondly, you must be a lead developer for changes to the developer's code to apply and you're not. And thirdly, the code is more what you'd call 'rules' than actual guidelines. Welcome aboard Corporate Development, Miss Turner" -- Barbossa, Project Lead, 2003.

We will be covering a few topics where each is ramping up into the next:

1. The prerequisites
   • Code volume
   • Control flow
   • State management
2. The principles
   • Composition over inheritance
   • Parse, don't validate
   • Never throw errors
   • Metadata
   • Define your source of truth
   • Let controllers tell you everything
   • Don't emulate network infrastructure
   • Don't let AI take the driver's seat
   • Generate as much code as possible
   • Write to refactor programmatically
   • Don't go overkill on abstraction layers

I won't be covering teamwork and only briefly touching upon inter-service communication at points.

The prerequisites

This section will briefly cover three elements of the development process that come from an older Enterprise Architecture Patterns course that I took back in 2021, but have found to been of constant importance.

These cornerstones have a direct correlation to complexity and brain overload. They are:

1. Code volume.
2. Control flow.
3. State management.

Any mishandling of the following topics will lead development towards "developer purgatory" and, as a bonus side-effect, make future devs die inside.

Code volume

Code volume relates to the amount of code that you have to manage. In layperson spiel: less lines of code to manage is better.

I always find this concept to be at odds with Clean Code and Gang of Four design patterns. My reason for this is that Clean Code and GO4 patterns become the scapegoat to justify over-bloat and unnecessary abstractions, or premature abstractions that doesn't keep the necessary separation of concerns for the domain.

Over-bloat and unnecessary abstractions may be easier to imagine. If you're working in the codebase where you need to make jumps to ten different definitions in order to understand the inheritance chain or follow the path of the business logic, then you have probably over-engineered the shit out of it. Principles like "composition over inheritance" and "parse, don't validate" can help mitigate volume creep (which I touch on in their own section), but there are some general guiding principles that I recommend to get around this:

1. Service layers that handle the business logic should be as "shallow" as possible. What I mean by this is that service layers should not compose other service layers, and layers that are required here should themselves aim to stay as shallow as possible.
2. Functional core, imperative shell. The idea here is that the "deepest" layers invoked should be as pure as possible. Side-effects like logging should be the responsibility the the imperative shell (in our case, we aim for that to be the service layer or injected at the service layer). This also becomes a God-send for a controlled domain and range of tests for pure functions.

As for the premature abstractions, this one might be harder to understand at first. Dan Abramov's "Goodbye Clean Code" does a good job of outlining the core idea that combining similar interfaces to reduce the code volume is not always the best idea.

But what gives? Isn't the idea that less code is better? You need to understand when contradictions to this guideline are necessary. I'll touch later on approaches to mitigate this problem with code generation.

The important thing to remember here is that the code volume cornerstone applies to code that you have to manage as a developer, not the size of the codebase itself.

Control flow

Control flow speaks to the branches in logic. Choices in how you apply control flow, how much of it there is and where logic "branches" can make or break your day.

We'll be touching on this topic with the following sections:

• Let your controllers tell your everything.
• Don't go overkill on abstraction layers.
• Never throw errors.
• Metadata.

State management

State management refers to how applications track and maintain data that changes over time. This "state" includes things like user inputs, API responses, UI configurations, and application settings.

Mismanagement here can create some deeply-knotted problems:

1. Unpredictable Behavior: When state can be modified from multiple places without clear patterns, applications become unpredictable. Developers can't easily reason about what will happen when code executes.
2. Debugging Nightmares: Without clear state flows, finding the root cause of bugs becomes extremely difficult. A bug might manifest in one component but originate from state modifications elsewhere.
3. Technical Debt Accumulation: Poor state management compounds over time through things like duplicated state, stale state and side-effects.
4. Readability and Maintainability Issues: New developers struggle to understand applications.
5. Performance Problems: Unnecessary re-renders, Memory leaks, Network request redundancy.

Although this post won't spend too much time on state management, it is also partly related to these topics:

• Parse, don't validate.
• Never throw errors.
• Metadata.
• Define your source of truth.

The principles

With the foundation prerequisites out of the way, the rest of the section will speak to some examples scenarios. You should keep those three cornerstones in mind while reading through each guideline.

Composition over inheritance

Composition over inheritance is a design principle that suggests you should prefer building complex functionality by combining simpler objects (composition) rather than through class inheritance hierarchies. Key Benefits of Composition

Some of the benefits:

• Easier to change behavior at runtime
• Looser coupling between components
• Simpler to test isolated components

Don't:

abstract class Pizza {
  prepare() {
    console.log("Preparing something hidden away from concrete classes");
  }

  // This method needs to be implemented by subclasses
  abstract addToppings(): void;
}

// Concrete subclasses
class PineapplePizza extends Pizza {
  addToppings(): void {
    console.log("Adding pineapple chunks");
  }
}

const p = new PineapplePizza();
p.prepare();
p.addToppings();

Do:

interface Pizza {
  prepare(): void;
  addToppings(): void;
}

class PineapplePizza implements Pizza {
  prepare() {
    console.log("Preparing pineable pizza");
  }

  addToppings(): void {
    console.log("Adding pineapple chunks");
  }
}

const p = new PineapplePizza();
p.prepare();
p.addToppings();

One thing about this approach: use composition over inheritance for enforcing behaviors.

When it comes to properties on the class, don't rely on implementation an interface to enforce that property. Implementing interfaces comes with the caveat that you cannot implement non-public properties. In those scenarios, use dependency injection.

Don't:

interface Pizza {
  id: string;
  name: string;
  prepare(): void;
  addToppings(): void;
}

class PineapplePizza implements Pizza {
  // Can't be private
  public id: string;
  public name: string;

  constructor(id: string, name: string) {
    this.id = id;
    this.name = name;
  }

  prepare() {
    console.log("Preparing pineable pizza");
  }

  addToppings(): void {
    console.log("Adding pineapple chunks");
  }
}

const p = new PineapplePizza("1", "pineapple");

Do:

interface Pizza {
  prepare(): void;
  addToppings(): void;
}

interface PizzaProps {
  id: string;
  name: string;
}

class PineapplePizza implements Pizza {
  private id: string;
  private name: string;

  constructor(props: PizzaProps) {
    this.id = props.id;
    this.name = props.name;
  }

  prepare() {
    console.log("Preparing pineable pizza");
  }

  addToppings(): void {
    console.log("Adding pineapple chunks");
  }
}

const p = new PineapplePizza({ id: "1", name: "pineapple" });

As a natural extension to this when it come to interfaces, avoid extending interfaces with other interfaces.

interface Request {
  body: unknown;
}

class Service {
  action(body: unknown) {
    // First check if body is a non-null object
    if (body !== null && typeof body === "object") {
      // Now TypeScript knows body is an object, so we can use 'in' operator
      if ("name" in body) {
        // Do something with body.name
        // Note: At this point, TypeScript knows body has a 'name' property,
        // but doesn't know its type. If needed, we could add further type narrowing:
        const typedBody = body as { name: string };
        // Now we can use typedBody.name as a string
      }
    }
  }
}

class Controller {
  private service: Service;

  constructor(service: Service) {
    this.service = service;
  }

  get(req: Request) {
    this.service.action(req.body);

    // ... omitted
  }
}

import { z } from "zod";

interface Request {
  body: unknown;
}

// Define the schema for our request body
const UserSchema = z.object({
  name: z.string(),
  email: z.string().email(),
  age: z.number().int().positive().optional(),
});

// Derive TypeScript type from the schema
type User = z.infer<typeof UserSchema>;

class Service {
  action(user: User) {
    // No need to validate
  }
}

class Controller {
  private service: Service;

  constructor(service: Service) {
    this.service = service;
  }

  get(req: Request) {
    // Parse and validate at the boundary
    const userResult = UserSchema.safeParse(req.body);

    if (!userResult.success) {
      // Handle error case
      return;
    }

    // Pass the parsed and typed data to the service
    this.service.action(userResult.data);

    // ... omitted
  }
}

class Data {
  readonly _tag = "Data";
  private myData: string;

  constructor(myData: string) {
    this.myData = myData;
  }
}

class Service {
  create() {
    // stand-in
    const isErr = false;

    if (isErr) {
      throw new Error("ErrorOne");
    }

    if (isErr) {
      throw new Error("ErrorTwo");
    }

    // Stand-in for applying actual work
    return new Data("Yay");
  }
}

class Controller {
  private service: Service;

  constructor(service: Service) {
    this.service = service;
  }

  create() {
    try {
      const createResult = this.service.create();

      return {
        status: 200,
        message: "Success",
      };
    } catch (err) {
      // Some catch error handling
      // ... omitted

      // If we can't match the expected cases here, throw to the catch-all handler.
      throw err;
    }
  }
}

import { ok, err } from "neverthrow";

class ExpectedErrorOne {
  readonly _tag = "ExpectedErrorOne";
}

class ExpectedErrorTwo {
  readonly _tag = "ExpectedErrorTwo";
}

class Data {
  readonly _tag = "Data";
  private myData: string;

  constructor(myData: string) {
    this.myData = myData;
  }
}

class Service {
  create() {
    // Stand-in
    const isErr = false;

    if (isErr) {
      return err(new ExpectedErrorOne());
    }

    if (isErr) {
      return err(new ExpectedErrorTwo());
    }

    // Stand-in for applying actual work
    return ok(new Data("Yay"));
  }
}

class Controller {
  private service: Service;

  constructor(service: Service) {
    this.service = service;
  }

  create() {
    const createResult = this.service.create();

    return createResult.match(
      // Success case
      (data) => ({
        status: 200,
        message: "Success",
      }),

      // Error case - pattern match on _tag
      (error) => {
        switch (error._tag) {
          case "ExpectedErrorOne":
            return {
              status: 400,
              message: "Bad request",
            };
          case "ExpectedErrorTwo":
            return {
              status: 422,
              message: "Unprocessable entity",
            };
          default:
            // Optional Exhaustiveness check - will error if we add a new response type
            // and forget to handle it.
            const _exhaustiveCheck: never = error;
        }
      }
    );
  }
}

class InvalidDatabaseObject {
  readonly tag = "InvalidDatabaseObject";
  readonly errorId = 123;
  readonly interalMessage = "To resolve, check a...b...c..."
  readonly statusCode = 422;
  readonly message = "We are unable to process your request at this time";
}

class Controller() {
  create() {
    // Assume this created object was the response
    const result = new InvalidDatabaseObject();

    if (result.tag === "InvalidDatabaseObject") {
      // Implement any observability if we need based on error properties
      logger.error("Failed to do something", {
        tag: result.tag,
        errorId: result.errorId,
        internalMessage: result.interalMessage,
      })
    }

    return {
      tag: result.tag,
      status: result.statusCode,
      message: result.message
    }
  }
}

class InvalidDatabaseObject {
  readonly _tag = "InvalidDatabaseObject";
  readonly _errorId = 123;
  readonly _interalMessage = "To resolve, check a...b...c..."
  readonly statusCode = 422;
  readonly message = "We are unable to process your request at this time";
}

class Controller() {
  create() {
    // Assume this created object was the response
    const result = new InvalidDatabaseObject();

    if (result._tag === "InvalidDatabaseObject") {
      // Implement any observability if we need based on error properties
      logger.error("Failed to do something", {
        _tag: result._tag,
        _errorId: result._errorId,
        _internalMessage: result._interalMessage,
      })
    }

    return {
      _tag: result._tag,
      status: result.statusCode,
      message: result.message
    }
  }
}

interface User {
  name: string;
}

function getUser(): User {
  const response: { data: unknown } = await getUserFromRemote();

  return response.data as unknown as User;
}

import { z } from "zod";

const UserSchema = z.object({
  name: z.string(),
});

type User = z.infer<typeof UserSchema>;

function getUser(): User {
  const response: { data: unknown } = await getUserFromRemote();
  const userResult = UserSchema.parse(response.data);

  return userResult;
}

const getUsersQueryParams = z
  .object({
    name: z.string(),
    age: z.coerce.number(),
  })
  .partial();

const getUserQueryParams = z
  .object({
    name: z.string(),
    age: z.string(),
  })
  .partial();

const getUserCoercedQueryParams = z
  .object({
    name: z.string(),
    age: z.coerce.number(),
  })
  .partial();

// In use
const result = getUserQueryParams
  .pipe(getUserCoercedQueryParams)
  .parse({ name: "Daniel", age: "22" });

class Data {
  readonly _tag = "Data";
  private myData: string;

  constructor(myData: string) {
    this.myData = myData;
  }
}

class Service {
  create() {
    // stand-in
    const isErr = false;

    if (isErr) {
      throw new Error("ErrorOne");
    }

    if (isErr) {
      throw new Error("ErrorTwo");
    }

    // Stand-in for applying actual work
    return new Data("Yay");
  }
}

class Controller {
  private service: Service;

  constructor(service: Service) {
    this.service = service;
  }

  create() {
    const createResult = this.service.create();

    return {
      status: 200,
      message: "Success",
    };

    // NO ERROR HANDLING, all sent to the catch-all. BAD.
  }
}

import { ok, err } from "neverthrow";

class ExpectedErrorOne {
  readonly _tag = "ExpectedErrorOne";
}

class ExpectedErrorTwo {
  readonly _tag = "ExpectedErrorTwo";
}

class Data {
  readonly _tag = "Data";
  private myData: string;

  constructor(myData: string) {
    this.myData = myData;
  }
}

class Service {
  create() {
    // Stand-in
    const isErr = false;

    if (isErr) {
      return err(new ExpectedErrorOne());
    }

    if (isErr) {
      return err(new ExpectedErrorTwo());
    }

    // Stand-in for applying actual work
    return ok(new Data("Yay"));
  }
}

class Controller {
  private service: Service;

  constructor(service: Service) {
    this.service = service;
  }

  create() {
    const createResult = this.service.create();

    return createResult.match(
      // Success case
      (data) => ({
        status: 200,
        message: "Success",
      }),

      // Error case - pattern match on _tag
      (error) => {
        switch (error._tag) {
          case "ExpectedErrorOne":
            return {
              status: 400,
              message: "Bad request",
            };
          case "ExpectedErrorTwo":
            return {
              status: 422,
              message: "Unprocessable entity",
            };
          default:
            // Optional Exhaustiveness check - will error if we add a new response type
            // and forget to handle it.
            const _exhaustiveCheck: never = error;
        }
      }
    );
  }
}

import { Hono } from "hono";
import { HTTPException } from "hono/http-exception";

const app = new Hono();

// Error handling middleware
app.use("*", async (c, next) => {
  try {
    await next();
  } catch (error) {
    console.error("Error caught:", error);

    // Handle Hono's built-in HTTPException
    if (error instanceof HTTPException) {
      return error.getResponse();
    }

    // Custom error with status code
    if ("status" in error && typeof error.status === "number") {
      return c.json(
        {
          message: error.message || "An error occurred",
          code: error.code || "UNKNOWN_ERROR",
        },
        error.status
      );
    }

    // Authentication errors
    if (
      error.name === "AuthenticationError" ||
      error.code === "UNAUTHENTICATED"
    ) {
      return c.json(
        {
          message: "Authentication failed",
          details: error.message,
        },
        401
      );
    }

    // Validation errors with details
    if (error.name === "ValidationError" || "validationErrors" in error) {
      return c.json(
        {
          message: "Validation failed",
          errors: error.validationErrors || error.details || [error.message],
        },
        400
      );
    }

    // Database errors
    if ("code" in error && error.code?.startsWith("DB_")) {
      // Log database errors but don't expose details to client
      console.error("Database error:", error);
      return c.json({ message: "Database operation failed" }, 500);
    }

    // Default case - generic 500 error
    return c.json({ message: "Internal server error" }, 500);
  }
});

export default app;

import { Hono } from "hono";
import { HTTPException } from "hono/http-exception";

const app = new Hono();

// Error handling middleware
app.use("*", async (c, next) => {
  try {
    await next();
  } catch (error) {
    // Handle observability here. Log errors, capture with third-party tools like Sentry etc.

    // Default case - generic 500 error
    return c.json({ message: "Internal server error" }, 500);
  }
});

export default app;

interface Retryable {
  redrive: () => void;
}

interface Queue extends Retryable {
  process: () => Promise<void>;
}

class DataQueue implements Queue {
  private dlq: string[] = [];
  private q: string[] = [];

  redrive() {
    this.q = [...this.q, ...this.dlq];
    this.dlq = [];
  }

  async process() {
    await Promise.all(this.q.map((el) => console.log));
  }

  // Rest omitted
}

interface Retryable {
  redrive: () => void;
}

interface Queue {
  process: () => Promise<void>;
}

class DataQueue implements Retryable, Queue {
  private dlq: string[] = [];
  private q: string[] = [];

  redrive() {
    this.q = [...this.q, ...this.dlq];
    this.dlq = [];
  }

  async process() {
    await Promise.all(this.q.map((el) => console.log));
  }

  // Rest omitted
}

Context setting

For anyone that follows my content, you will know that I am not a C# nor .NET developer. My experience with the language itself is limited to time spent working with C# at university or through hobby projects (game development, Arduino).

That being said, work has recently asked me to make some changes to our old .NET 6 server. Some of the challenges with this legacy code base has been tracking error management and understanding potential return types where the current style of relying on raising exceptions for control flow lacks intellisense for developers working on the codebase.

It should be disclosed that as an unseasoned C# developer, this can make adjudicating good practice and idiomatic C# a little more difficult. On the flip-side of this, one could see it as me looking at C# with a fresh set of eyes.

Reviewing the .NET error handling page

One of the challenges I was facing while reading through the legacy codebase was understanding how our non-2xx error code responses were happening. A lot of responses were returning as 2xx series responses with null values, even in scenarios where the request body itself was invalid.

I opted to first read through the recommendations for error handling that the .NET documentation presents (here I am sharing the latest version of the docs, although I was looking through .NET 6 documentation).

Something that stood out to me was the lambda exception handler where we could pass a C# handler to the app.UseExceptionHandler method:

var app = builder.Build();

if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler(exceptionHandlerApp =>
    {
        exceptionHandlerApp.Run(async context =>
        {
            context.Response.StatusCode = StatusCodes.Status500InternalServerError;

            // using static System.Net.Mime.MediaTypeNames;
            context.Response.ContentType = Text.Plain;

            await context.Response.WriteAsync("An exception was thrown.");

            var exceptionHandlerPathFeature =
                context.Features.Get<IExceptionHandlerPathFeature>();

            if (exceptionHandlerPathFeature?.Error is FileNotFoundException)
            {
                await context.Response.WriteAsync(" The file was not found.");
            }

            if (exceptionHandlerPathFeature?.Path == "/")
            {
                await context.Response.WriteAsync(" Page: Home.");
            }
        });
    });

    app.UseHsts();
}

In the example above, the code could be configured within a .NET application to capture raised exceptions and use control flow to act upon them.

In particular, the example code sets the 500 status code and conditionally sends responses from the server based on business logic.

The challenges that raising exceptions presents

The current .NET configuration for the application that I was working with had a version of this configured, but this is where abstract boolean logic itself can take a nasty turn. The file itself is a minefield of nested if/else statements where the final response is "funneled" through that logic based on the type of exception it is (similar to exceptionHandlerPathFeature?.Error is FileNotFoundException above), as well as some business-specific control flow qualifiers (statements not based on the error type).

Contributing to the file itself would be adding to the debt that had built up, but in some cases for legacy code that may be considered fine (and may actually end up being what I am requested to do), but there are two reasons why I struggle with this style of controller management:

1. Type signatures do not inform the developer of potential raised exceptions.
2. Controllers should be the entry point and exit point for all expected HTTP responses.

It is of my personal belief that controllers should be the "air traffic controllers" - they should handle the delegating of business logic to the services, while be in control of returning all expected responses.

What I mean by this is that, as a developer, if I am wanting to look at how an endpoint is handled, I should be able to start at the controller in order to grok how data is handled coming in while also understanding all expected responses going out -- not just the "ok" response.

When I say "expected responses", that includes management for all non-200 responses for percievable issues such as request validation failure, unprocessible issues due to dependencies, etc. There is always the possibility for unexpected errors (which I normally refer to as defects), and in my opinion only these errors should fall outside of the domain of controller management.

The code shared above for the recommended app.UseExceptionHandler I find to be appropriate for defects. I consider defects to be the unexpected errors of the application.

I consider the app.UseExceptionHandler to be appropriate for these unforeseen circumstances. Within the lambda, I would configure the 500 internal error response and have any 3rd party error tracking occur in this lambda only. The UseExceptionHandler for me should only contain the business logic around handling these unexpected scenarios. Everything else should be managed by the controller as far as I am concerned.

Return types in .NET

The first of the two reasons that I presented for why raising exceptions is challenging is that the type signatures do not reflect the return type.

Take the following code example:

public async Task<string> DecryptCipherText(string cipherText)
{
    if (string.IsNullOrEmpty(cipherText))
    {
        throw new ArgumentNullException(nameof(cipherText), "Cipher text cannot be null or empty");
    }

    // ... pretend to process things
    return "Processed cipherText";
}

In this code block, we have a contrived example of a function that expects to decrypt some cipher text. In the scenario that the cipherText passed in is null or empty, we raise an ArgumentNullException.

However, if we look at the type signature, all we see is (awaitable) Task<string> CardDataEncryption.DecryptCipherText(string cipherText). There is no indication to anyone from the call site of the function (be it another service or whatever) that an ArgumentNullException can be thrown.

While this is not unique to C# itself, there are alternatives that offer up solutions for us. This is where I want to introduce OneOf.

As mentioned as the beginning of the post, OneOf is a library that provides F# style unions for C#.

If we look at our function again with the return signature Task<string>, then we note that out-of-the-box, there is no "union" style returns where we could indicate different return types. This is where OneOf steps in.

It enables us to be able to alter signatures to indicate multiple return types and provides tools to help us match on those return types.

For example, we could alter our code to the following:

public async Task<OneOf<string, ArgumentNullException>> DecryptCipherText(string cipherText)
{
    if (string.IsNullOrEmpty(cipherText))
    {
        // Note that we are now returning this value
        return new ArgumentNullException(nameof(cipherText), "Cipher text cannot be null or empty");
    }

    // ... pretend to process things
    return "Processed cipherText";
}

In the above, instead of raising an exception, we can now return the exception.

Looking at the type signature now, we see it's adjusted to (awaitable) Task<OneOf<string, ArgumentNullException>> CardDataEncryption.DecryptCipherText(string cipherText).

From a developer point-of-view, Intellisense providing more information on potential errors that can go wrong is a powerful to invert control of how to manage results from a function from the caller. This is particularly powerful for developers like myself who stumble into a codebase for the first time.

The library itself provides a number of ways to fetch values from this response type, but the two that I use the most are Match, TryPickT... or directly accessing the value itself with the value property AsT... where the ... in both cases represents a number for the argument order passed to the type signature (i.e. AsT0 for our example would be string while AsT1 would be ArgumentNullException).

For example, with our function defined above we could access the results like so:

const result = await DecryptCipherText('someCipherTextValue');

// Using match to determine to send a 200 or 400
return result.Match(
    result => Ok(result),
    ArgumentNullException => new BadRequest("...")
);

// Using TryPick - where you might want to pick out the error to return early and bubble up the error
if (thingOrNotFoundOrError.TryPickT1(out ArgumentNullException error, out var remainder)) {
  return error;
}

// Accessing the property directly
Console.WriteLine(result.AsT0); // Possibly the string!

The above demonstrates the few ways that I've used the library.

1. The first example with Match is how I managed the "air traffic controller" for responding from the controller. The Match type is exhaustive and requires you to handle all expected responses.
2. The second example allows you to "extract" a value to use as required. Normally for myself, this might be early propagation of the error up to where it can be handled (normally at the controller).
3. Demonstrates access the value directly. I normally use this in tandem with (2) to ensure that the value exists before applying more logic to it (or you could do the opposite and pick out the valid response and return all others!).

The library itself extends possible return values as you have more applied. For a very contrived example of this, the following would have AsTN where N could be 0 to 4:

(awaitable) Task<OneOf<string, ArgumentNullException, ArithmeticException, ArgumentException, PaymentException>> CardDataEncryption.DecryptCipherText(string cipherText)

To nail the point home - the value of having a separation of "expected" and "unexpected" errors is that the former allows you to self-document and manage all expected responses in a matter-of-fact way.

Keeping that separation can even enable things such as more effective testing given that the domain and range of a function makes the testing strongly in favour of "input-output testing". It means functions will no longer contain the side-effects of thrown functions that do not return, which is heavensent when it comes to testing functions orientated around business logic.

Conclusion

Today's blog post introduces the idea of using the library OneOf in order to better manage union response types, enabling the control flow and development of C# to focus on handling expected outcomes.

This was demonstrated by following along an example that I had personally come across where reliance on control flow with throwing exceptions leads to a more difficult developer experience and merging of error state management across different concerns.

Further links and resources

• Handling errors in ASP.NET Core
• mcintyre321/OneOf

Photo credit: wolfgang_hasselmann

From all the sources I have read so far, it looks to have been something that the team has had in the works for a long time. It comes in a few different flavours, but this blog post will focus more so on playing around with DSQL and seeing it in action.

To keep things simple, we will be using the AWS console to setup a new cluster and then connecting to it with Bun, TypeScript and Drizzle.

Getting started

First, let's configure the project with Bun and Drizzle.

# Configure git, initialise the project
$ mkdir demo-dsql-drizzle
$ cd demo-dsql-drizzle
$ bun init
$ bun add drizzle-orm pg dotenv
$ bun add -D drizzle-kit @types/pg
$ mkdir -p src/db drizzle
$ touch src/index.ts src/db/schema.ts drizzle.config.ts

We're going to follow a slightly modified version of Drizzle's getting started guide for Postgres. Please note that DSQL is only Postgres-compatible at the time of the announcement.

First, we need to configure our basic schema. Add the following to src/db/schema.ts:

import { integer, pgTable, varchar } from "drizzle-orm/pg-core";

export const usersTable = pgTable("users", {
  id: integer().primaryKey().unique(),
  name: varchar({ length: 255 }).notNull(),
  age: integer().notNull(),
  email: varchar({ length: 255 }).notNull().unique(),
});

DSQL has limitations around constraints that I won't dive too deep into here, just note that you cannot use the generatedAlwaysAsIdentity() constraint as expected. AWS will respond with an error during the migration. You can read more about the unsupported features here.

At this point, we can pivot to the AWS console and create a new Aurora DSQL cluster. This will be required for filling out the remaining files as we will need some configuration setup.

Creating a new Aurora DSQL cluster

Through the AWS console, you can search and navigate to Aurora DSQL.

If you are not currently in a supported region, AWS will prompt you to change regions.

Once you are in a supported region, you can select to create a cluster to move forward. Once it's created, go into the details for the cluster and you will see a "connect" button that will give you the endpoint and other details required to connect to the cluster.

At this point, we'll have a cluster running with some details about the endpoint and cluster ID, etc. Collect the details for the endpoint and we can move forward.

Connecting to the Aurora DSQL cluster

Back in our project, we can configure out connection. AWS has some more details about using the PostgreSQL protocol that you can read, but in our case I will walk use through what we need.

We'll need to configure our drizzle.config.ts file in the root of the project in order to run our migration. Given that this is just a walkthrough, I won't be using any .env configuration files, but the security is back on you to manage. We'll be using the admin configuration in our case, so the default password limit is 15 minutes to add some sense of security. In practice, you'll either need to manage IAM or make use of APIs to generate the configuration required to connect.

import "dotenv/config";
import { defineConfig } from "drizzle-kit";

export default defineConfig({
  out: "./drizzle",
  schema: "./src/db/schema.ts",
  dialect: "postgresql",
  dbCredentials: {
    host: "<add in the endpoint from the console here>",
    port: 5432,
    user: "admin",
    password: "<add in the entire password string from the console here>",
    database: "postgres",
    ssl: true,
  },
});

By default, the user will be admin and the database will be postgres. The password will be the one you set when creating the cluster.

The password itself includes the host as well. It's a very long string but just note that you need the entire thing in the password field.

Please note that ssl is required for the connection to work.

Running a simple migration

Once this is all configured, we can run the migration:

$ bunx drizzle-kit push

You can read up more on Drizzle kit migrations on the Drizzle migrations documentation.

If everything is successful, you should the following:

$ bunx drizzle-kit push
No config path provided, using default 'drizzle.config.ts'
Reading config file '/Users/dennis.okeeffe/code/okeeffed/demo-dsql-drizzle/drizzle.config.ts'
Using 'pg' driver for database querying
[✓] Pulling schema from database...
[✓] Changes applied

Let's finish by seeding some data and applying some basics calls against the database.

Running some code against the database

We'll modify an example script from Drizzle to run against the database slightly as we will need to provide the integer IDs ourselves. Not having certain key constraints means that we will need to enforce it at the application layer.

In src/index.ts, add the following:

import "dotenv/config";
import { drizzle } from "drizzle-orm/node-postgres";
import { eq } from "drizzle-orm";
import { usersTable } from "./db/schema";

const db = drizzle({
  connection: {
    host: "<add in the endpoint from the console here>",
    port: 5432,
    user: "admin",
    password: "<add in the entire password string from the console here>",
    database: "postgres",
    ssl: true,
  },,
});

async function main() {
  const user: typeof usersTable.$inferInsert = {
    id: 1234,
    name: "John",
    age: 30,
    email: "john@example.com",
  };
  const user2: typeof usersTable.$inferInsert = {
    id: 5678,
    name: "Tracy",
    age: 48,
    email: "tracy@example.com",
  };

  await db.insert(usersTable).values([user, user2]);
  console.log("New user created!");

  const users = await db.select().from(usersTable);
  console.log("Getting all users from the database: ", users);
  /*
  const users: {
    id: number;
    name: string;
    age: number;
    email: string;
  }[]
  */

  await db
    .update(usersTable)
    .set({
      age: 31,
    })
    .where(eq(usersTable.email, user.email));
  console.log("User info updated!");

  await db.delete(usersTable).where(eq(usersTable.email, user.email));
  console.log("User deleted!");
}

main();

Similar to before, I'm just dumping in the credentials here for the sake of the walkthrough, but the connection object will look just like what we configured previously in the drizzle.config.ts file.

Running bun src/index.ts, we can finally test our code against the DSQL cluster:

$ bun src/index.ts
New user created!
Getting all users from the database:  [
  {
    id: 1234,
    name: "John",
    age: 30,
    email: "john@example.com",
  }, {
    id: 5678,
    name: "Tracy",
    age: 48,
    email: "tracy@example.com",
  }
]
User info updated!
User deleted!

Conclusion

This blog post was more "show" than "tell". I have plenty more about Aurora DSQL that I need to dive into and understand, but hopefully this will give you some insight into how you can setup and connect to your own DSQL cluster.

Don't forget to tear down your cluster once you are done to avoid any unnecessary costs. As far as I can tell, costs are free during this preview period, but it's always good to be cautious.

Further links and resources

• Drizzle's getting started guide for Postgres
• Amazon announced the preview of Amazon Aurora DSQL
• Drizzle migrations documentation
• Aurora DSQL details about using the PostgreSQL protocol
• Aurora DSQL user guide

Photo credit: joelfilip

This demo walks through using Git worktree with an example using different npm dependencies to demonstrate a use case for multiple worktrees.

Our demonstration will emulate (albeit contrived and small) a "major" change within the repository. We're going to dumb it down to something contrived like migrating from Lodash to es-toolkit, but you'll need to use your imagination and pretend like our work may take something like a week or so to complete, and that you may need to ship other features while doing the migration.

It really doesn't matter which language you're working with, but I'm using JavaScript to demonstrate here because you have likely found yourself jump between trees and needing to re-install different dependencies at some stage as you switch back-and-forth.

The process will look like this:

1. Have a base worktree with Lodash.
2. Create a separate worktree for the migration.
3. Re-enact the process where we need to need to add a new feature on the base tree.
4. Continue on with the migration.

Getting started

# Configure git, initialise the project
$ mkdir demo-git-worktree
$ cd demo-git-worktree
$ npm init -y
$ npm install lodash
$ touch index.js README.md
$ git init
$ echo node_modules > .gitignore

In index.js, add the following to represent our large repository code:

const _ = require("lodash");

function main() {
  console.log(_.camelCase("Our crazy big home project"));
}

main();

You can now run this code to see what the project is doing:

$ node index.js
ourCrazyBigHomeProject

Let's commit what's we've got:

# Add our commits
$ git add --all
$ git commit -m "feat: first commit"
[main (root-commit) 3fd5292] feat: first commit
 5 files changed, 86 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 README.md
 create mode 100644 index.js
 create mode 100644 package-lock.json
 create mode 100644 package.json

Perfect! At this stage, you git log should show something like this:

# Show git log
$ git log
commit 3fd5292b0a49ed8a5a6c4a5e2b9ee7394453e290 (HEAD -> main)
Author: Dennis O'Keeffe <hello@dennisokeeffe.com>
Date:   Wed Aug 14 08:20:12 2024 +1000

    feat: first commit

Now let's emulate our project to migrate over.

Using Git worktree

We're creating a new worktree for this.

Run the following:

# List our worktree
$ git worktree list
/path/to/okeeffed/demo-git-worktree  3fd5292 [main]

# Create a new branch for our migration
$ git branch feat/estoolkit-migration

# Create a new worktree
$ git worktree add ../demo-git-worktree-estoolkit-migration feat/estoolkit-migration
Preparing worktree (checking out 'feat/estoolkit-migration')
HEAD is now at 3fd5292 feat: first commit


# List our trees again
$ git worktree list
git worktree list
/path/to/okeeffed/demo-git-worktree                      3fd5292 [main]
/path/to/okeeffed/demo-git-worktree-estoolkit-migration  3fd5292 [feat/estoolkit-migration]

Great! At this point, we now have another directory at ../demo-git-worktree-estoolkit-migration which represents our estoolkit-migration worktree. It is checked out onto feat/estoolkit-migration.

Git operates differently between the main worktree and a linked worktree. It's best to reference the docs, but here are some quick pointers.

The main differences in created linked work trees are:

1. Separate working directories: Each worktree has its own working directory, allowing you to work on different branches simultaneously without switching.
2. Shared repository: All worktrees share the same Git repository (objects and refs).
3. Independent HEAD: Each worktree has its own HEAD, index, and config file.
4. Branch isolation: You can't check out the same branch in multiple worktrees simultaneously (except for bare repositories).

At this point, we can work separately between the two folders for our changes.

Making our updates for the toolkit migration

Heading into ../demo-git-worktree-estoolkit-migration and make the following changes:

# Change into our linked worktree directory
$ cd ../demo-git-worktree-estoolkit-migration

# Add es-toolkit and remove Lodash
$ npm install es-toolkit

# Remove Lodash
$ npm uninstall lodash

Now let's update index.js:

const { camelCase } = require("es-toolkit");

function main() {
  console.log(camelCase("Our crazy big home project"));
}

main();

If we run node index.js again, we'll notice that nothing has changed:

$ node index.js
ourCrazyBigHomeProject

Great! Let's commit.

# Our commit
$ git add --all
$ git commit -m "feat: migrated lodash camelCase"

Going back for our "new feature"

Let's say now that we need to pivot midway through our migration for something else.

Back in the main worktree, let's go through a flow of adding another feature. Again, this will be contrived, but work with me.

# Back to our main worktree
$ cd ../demo-git-worktree

Even though Lodash is used in the repo, notice that if we run index.js, everything still works even though we remove that dependency in the linked worktree:

# Running on our main worktree without re-installations
$ node index.js
ourCrazyBigHomeProject

Great! Let's branch off, create a new feature and then merge back in.

# Create a new branch
$ git branch feat/snakecase
$ git checkout feat/snakecase

Let's update our index.js file:

const _ = require("lodash");

function main() {
  console.log(_.camelCase("Our crazy big home project"));
  console.log(_.snakeCase("Our crazy big feature update"));
}

main();

Running this will log out the following:

$ node index.js
ourCrazyBigHomeProject
our_crazy_big_feature_update

Let's commit and merge to main:

# Commit
$ git add index.js
$ git commit -m "feat: added snakecase"
[feat/snakecase ab1e6d5] feat: added snakecase
 2 files changed, 173 insertions(+), 1 deletion(-)

# Head back to the target branch
$ git checkout main
# Merge into main
$ git merge feat/snakecase
Updating 3fd5292..37ed0e0
Fast-forward
 index.js  |   1 +
 2 files changed, 174 insertions(+), 1 deletion(-)

Normally in the work setting we would have our pull request made for the merge.

At this point, our target main branch has also differed from our worktree. Let's fix that.

Rebasing from another worktree

We can rebase from our worktree just like we normally would.

Let's head to our worktree and rebase:

# Go back to the linked tree
$ cd ../demo-git-worktree-estoolkit-migration

# Run the rebase
$ git rebase main
Auto-merging index.js
CONFLICT (content): Merge conflict in index.js
error: could not apply 8325647... feat: migrated lodash camelCase
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Could not apply 8325647... feat: migrated lodash camelCase

At this point, we have our conflict that we expected from the change.

Let's fix our conflict by using snakeCase from es-toolkit as well. Our code should look like this:

const { camelCase, snakeCase } = require("es-toolkit");

function main() {
  console.log(camelCase("Our crazy big home project"));
  console.log(snakeCase("Our crazy big feature update"));
}

main();

Again, we can check it works without the need to re-install anything:

# Running the changes
$ node index.js
ourCrazyBigHomeProject
our_crazy_big_feature_update

Perfect! Let's finish the rebase by continuing.

# Continue the rebase
$ git add index.js
$ git rebase --continue

Let's bring this project to a close after merging a our linked tree code to our main tree.

Merging back to main

Head back to the main tree and merge in the branch.

# Make sure we commit from within our merge tree
$ cd ../demo-git-worktree
$ git merge feat/estoolkit-migration
Merge made by the 'ort' strategy.
 index.js          |  6 +++---
 package-lock.json | 10 +++++-----
 package.json      |  2 +-
 3 files changed, 9 insertions(+), 9 deletions(-)

Again, we merged locally but you'd normally have a pull request for this.

At this point, you would notice that if you tried to run index.js, then finally we would run into an issue:

# Demoing the issue
$ node index.js
node:internal/modules/cjs/loader:1147
  throw err;
  ^

Error: Cannot find module 'es-toolkit'
Require stack:
- /path/to/okeeffed/okeeffed/demo-git-worktree/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1144:15)
    at Module._load (node:internal/modules/cjs/loader:985:27)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.<anonymous> (/path/to/okeeffed/okeeffed/demo-git-worktree/index.js:1:34)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Module._load (node:internal/modules/cjs/loader:1023:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/path/to/okeeffed/demo-git-worktree/index.js' ]
}

Node.js v20.10.0

Naturally, both of our worktrees maintained a different node_modules folder. That's how we were able to jump between both seamlessly.

All we need to do is update our main worktree to install the latest package.json configuration.

# Install current configuration
$ npm install

added 1 package, removed 1 package, and audited 2 packages in 1s

found 0 vulnerabilities

# Attempt run again
$ node index.js
ourCrazyBigHomeProject
our_crazy_big_feature_update

Happy days! All we have left to do is cleanup.

Cleaning up our linked tree

We can do so with the following:

# Remove the worktree
$ git worktree remove ../demo-git-worktree-estoolkit-migration
$ git worktree list
/Users/dennis.okeeffe/code/okeeffed/demo-git-worktree  8a9f3a1 [main]

At this point, our linked tree has been cleaned up.

If you run git branch --list, you'll notice that although the worktree is gone, our branch remains.

# List our branches
$ git branch --list
  feat/estoolkit-migration
  feat/snakecase
* main

Given the rules of worktrees, now that our other worktree is gone, we're free to checkout feat/estoolkit-migration from our main worktree now.

Further links and resources

• Git worktree documentation
• GitKraken | worktree

Photo credit: roadtripwithraj

I personally waiver back and forward between how confident I feel about my ability to express my thoughts. It's something that I've been thinking a lot about more recently as some long-term projects come to a close at my company. Today's post aims to share some of thoughts.

We'll cover why I believe expression to be important, why it feels so hard a software engineer and a strategy that I've come up with to help me improve.

The importance of expression

How often in your daily role in software have you come across a situation where you want to make your opinion known but felt unable to find the words to do so? Worry not. You're not alone.

The ability to either express yourself well or not to have a wide-ranging set of implications.

Some of the positives:

• Feeling heard.
• Constructive conversations and feedback.
• An exponential speedup in learning.

In contrast, the inability to express yourself does not leave one in a neutral state. I find it likely to have the opposite effect:

• Feeling like you have no say.
• Internal conflict.
• A undesirable state of dogmatism.

This could be the point to slam you, my revered reader, with some statistics to back up my claims. However, given that I've never actually looked into this, we're going off a data point of one: me.

And for me, our honorable data point, I find how well I express myself at any given point over my career directly impacts employee happiness and satisfaction.

My expression couldn't lead to constructive conversation? I feel my opinion is not valued and I find myself devaluing theirs. I couldn't adequately disagree with our approach? It festers inside and it can lead to self-imposed stress or irritation. My approach was not understood well and dismissed? Sometimes I could lean harder into my idea without taking a step back to think more holistically about alternatives.

Why is it that we find ourselves going through these motions? It's human nature. Not everyone will have the same responses I feel, but I believe the majority of us to share some common cornerstones:

1. We are social beings.
2. We value achievement.
3. We value contribution.

These three points are a natural extension of communicating effectively. If you ever end up in a situation where you fundamentally disagree with a concept or approach, express it well and followup with an engaging, constructive conversation, the outcome is likely to be positive irrespective of the approach taken. If you've expressed your idea clearly followed up by a well-thought discussion, you are likely to come to an understanding of the other party's perspective while still having the satisfaction of feeling you have made your contribution towards the final result.

All of this is a key part of evaluating our own self-worth. But why does it feel so hard for us to do as software engineers?

Why is expression hard for software engineers?

The obvious answer to my last dramatic question: it is not hard for all software engineers. The key to great communication is the adage of practice makes perfect.

Software engineers who are following their career path often end up in a situation where they often choose between a line towards management/technical lead, or a line towards honing their expertise as an individual contributor.

In my opinion, the former requires you to express yourself more often. The nature of the role is to engage with others and make decisions. The latter, however, naturally tends towards more hours working diligently and less time expressing yourself to others.

Working as an individual contributor in software can have some fundamentally unhealthy side effects:

1. We spend a lot time at the desk.
2. We spend a lot of time in our heads.

Unless we find ways to counter this, you'll may come to find (as our have) that if you are not "practicing" good communication, then it becomes a use-it-or-lose-it situation.

So how do we counter this? Without going into a specific approach just yet, I want to share some of the ways that I feel great individual contributors keep up their communication skills: the teach and/or share often.

The method of teaching is also important. For any Duolingo fanatics out there, you may have found yourself surprised to learn that even after 1000 consecutive days of learning Spanish, you still can't speak it. You might be able to read and write incredibly well if you were diligent, but speaking is another dimension to it.

What I'm trying to illustrate here is that although outlets like writing documentation and verbally shouting opinion pieces (like this one) into the void are in themselves very constructive for certain forms of communication, they are not going to result in the same level of improvement for your capability to convey opinions with speech.

This is a fundamental theme behind this post: it's not that as individuals we don't have opinions or thoughts, it's that we fall out of practice with expressing them.

If you've ever watched developers on YouTube or on stream that you feel do an incredibly job of adhoc-explaining their thoughts or processes off-script, then you'll be pleased to know that is a result of practice.

Some things that great individual contributors do that helps them become better at expressing themselves:

1. Pair programming.
2. Streaming.
3. Public speaking.
4. Video content.

If the chasm towards this still feels untenable or too far a chasm to cross, then even begin to express your thoughts to yourself in the mirror. Refine your thoughts and opinions. Practice makes perfect.

An approach to clear expression

The approach I have working on recent is what I call the CLEAR framework (ba dum tss). It's a simple, easily remember acronym that touches on the key points:

• C: Clarity
• L: Lens
• E: Ease
• A: Align
• R: Rehearse

How to visualize the CLEAR framework

Imagine you have a glass prism. Light enters the prism and is refracted into a spectrum of colors. For a more vivid version of this, think of The Great Pyramids of Gaza as our glass prism, with the sunlight coming through being refracted out the other side of the pyramid as a gigantic rainbow.

• The light coming into the prism as the input we want to comment on.
• The glass prism itself is representational of our opinion forming process.
• The spectrum of colors represents our expression of that opinion.

The glass prism is where our CLEAR process comes in. It guides our approach to taking that input, making sense of where we stand on it and how we want to convey it.

Let's walk through each step.

Clarity

Seek clarify on your opinion by asking yourself questions. If open questions are too broad to answer, try to ask yourself closed questions (yes/no). This will help you to refine your stance on the matter.

Lens

You can think about this point in a number of ways:

1. What is the lens that you are looking through?
2. What is the lens that your audience is looking through?
3. What is a "lens" that you could use to simplify what you want to say?

Ease

This in a natural extension of the "lens" property. Simplify the lens that you and others can look through.

I think of this point as being the most integral. For difficult concepts, close that chasm that others need to cross in order to understand your thought process.

In regards to technical terms, concepts like Domain Name Server (DNS) can be simplified to "the phone book of the internet". Examples like this can go a long way to making your opinion more accessible.

This is also true for non-technical terms. If you wish to convey why you dislike certain processes or approaches around project management or team dynamics, seek a way to simplify your opinion to a more relatable viewpoint.

Align

For this point, you can ask yourself these questions:

1. How will your opinion align with the desired outcome?
2. How does it align with your values or the values or the greater team?

Keeping the end in mind for your opinion will guide how you speak.

Rehearse

This point speaks to the practice that we mentioned. Of course, this is not always possible in the moment, but you'll often find that the values that drive your opinion are often the same values that drive your current state as a software engineer.

If you have common gripes or event some post-trauma about something that went wrong, then you likely already have strong opinions on those things. Rehearse those opinions. Refine them. Make them more accessible. That preparation will help you in the moment.

Some final points on CLEAR

Here is the reality of forming an opinion: you won't always nail the delivery.

Don't sweat it. Even the best communicators have moments where their delivery doesn't nail the landing.

It's all a process. The more you practice, the better your average delivery will become.

Finally, if CLEAR doesn't work for you or the visual I provided doesn't resonate, then this is a great chance to put this into practice for yourself. Make your own approach that helps you to express your thoughts better.

Conclusion

In conclusion, expressing ideas in software development can indeed be challenging, but it's an essential skill that significantly impacts both personal satisfaction and professional growth. As software engineers, the balance between technical expertise and effective communication is crucial. By recognizing the importance of expression and actively working to improve it, we can enhance our contributions, foster constructive conversations, and ultimately achieve better outcomes in our projects.

The CLEAR framework provides a structured approach to refining our communication skills. By focusing on CLEAR, we can systematically enhance how we articulate our thoughts and opinions. Remember, practice is key. Whether through pair programming, public speaking, or simply rehearsing your ideas, the more you practice, the more proficient you will become at expressing yourself.

Effective communication is not just about being heard but about creating a dialogue that leads to understanding and collaboration. By continuously working on our ability to express our ideas, we not only improve our own satisfaction but also contribute to a more inclusive and innovative working environment. So, take the time to practice, refine your approach, and don't be afraid to express your thoughts. It's a journey worth taking.

Disclaimer: This blog post used AI to generate the images used for the analogy. You may have figured that out when you saw the light not entering the prism as a straight line but shining at the tip. I left it in because it felt cool.

Photo credit: acharki95

We've come towards the end of the Node.js Streams blog series. Everything we've covered so far will take you a long way in understanding how to work with streams in Node.js.

In this post, we're going to introduce the PassThrough stream type and then take everything that we've learned so far and apply it through a practical (but somewhat contrived) example using the AWS CDK, AWS S3 and AWS Transfer Family.

If you're here because you are most interested in configuring AWS Transfer Family, then jump down to the project implementation.

What are PassThrough streams?

Previously, we covered the four fundamental stream types:

1. Readable streams
2. Writable streams
3. Duplex streams
4. Transform streams

So how does stream.PassThrough fit into the picture?

A PassThrough stream in Node.js is a special type of Transform stream that simply passes data through from its writable side to its readable side without modifying it. It's essentially a "do-nothing" transform stream that can be useful in various scenarios.

In general, Node.js docs describe the PassThrough stream's primary purpose as being for examples and testing, but there are also some novel use cases for PassThrough streams.

Some of these novel uses of a PassThrough stream are:

1. To provide a placeholder for a transform operation that may be added later.
2. To delay the piping of data from one stream to another.
3. To observe data passing through a pipeline without altering it.

(3) is particularly useful when you are tying together third-party stream types that you want to pipe through pipelines or when demultiplexing streams. We will be using the PassThrough stream later in our example to do just that.

For clarity, it's not necessarily a tool that you'll use everyday, but I have found some use cases.

How to think about PassThrough streams

With our four fundamental data types, we've used water-related analogies to help understand how streams work in Node.js.

1. Readable streams: Think of a water tap.
2. Writable streams: Think of a water bucket.
3. Duplex streams: A two-way water pipe (with valves to represent the read/write elements).
4. Transform streams: A pipe going through a water treatment plant.

So how do we think about PassThrough streams?

We can think of them as a clear, transparent water pipe. The water flows from one end to another without any modification.

To push forward the idea of this analogy:

• Like a PassThrough stream, a transparent pipe section allows water to flow through without changing it.
• You can observe the water (data) as it passes through.
• It can be easily inserted into an existing pipeline without affecting the flow.
• It provides a point where you could potentially add a "filter" or other "treatment" later if needed.

With that out of the way, let's move on onto our project.

Uploading and downloading a CSV file to AWS S3 using the AWS CDK

Our project today is to create an "upload" and "download" script to exemplify some of the concepts we've learned throughout the series.

Here is the overview of what we are building:

• We will be using the AWS CDK to create an S3 bucket and an AWS Transfer Family server.
• We will create a special Duplex stream that will act as a demultiplexer.
• Our upload script will have three pipelines: one to generate a CSV stream from an array of objects which is compressed and then demultiplexed, while the other two will be writable from our demux. One will upload the compressed CSV file to S3, and the other will SFTP the CSV file to our Transfer Family Server (and end up in the same bucket).
• Our download script will have four pipelines: one to download our S3 upload file, decompress it and then demultiplex it. The two receivers from the demux will write our the CSV file and and YAML file to disk (just for fun). The fourth pipeline will download the compressed CSV file via SFTP from our Transfer Family Server, decompress it then write it to disk.

You can see the full project code on my GitHub repository.

Prerequisites

It is expected that you are familiar with the following (at least to some extent):

1. AWS CDK
2. AWS S3
3. AWS Transfer Family
4. Have read the Node.js stream series

Setting up the project

First, we need to initialize our CDK TypeScript project. We can do that by running the following commands:

$ mkdir demo-stream-to-s3-and-back
$ cd demo-stream-to-s3-and-back
$ npx cdk init app --language typescript

Once the project is initialized, we can install the necessary dependencies for our other scripts:

$ npm i @aws-sdk/client-s3 @aws-sdk/lib-storage fast-csv ssh2-sftp-client yaml
$ npm i --save-dev tsx

Finally, let's configure some files for our project:

$ mkdir src
$ touch src/upload.ts src/download.ts src/

Creating an asymmetric SSH key pair

First step is for us to create an asymmetric SSH key pair. We will use the public key for our SFTP server.

From the root directory of our project, run the following:

$ ssh-keygen -t rsa -b 2048 -f sftp_key

Running this code will generate two files: sftp_key and sftp_key.pub. We will use the public key in our CDK code, while the private key will need to be referenced in our local code.

Creating the S3 bucket and Transfer Family server

Next, let's write our AWS CDK infrastructure code.

If you followed my naming convention above and called the project directory demo-stream-to-s3-and-back, then you will have a file lib/demo-stream-to-s3-and-back-stack.ts already created for you.

Update that file with the following:

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as s3 from "aws-cdk-lib/aws-s3";
import * as transfer from "aws-cdk-lib/aws-transfer";
import * as iam from "aws-cdk-lib/aws-iam";
import { readFileSync } from "fs";
import * as path from "path";

const SFTP_PUBLIC_KEY = readFileSync(path.join(__dirname, "../sftp_key.pub"));

export class DemoStreamToS3AndBackStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const bucket = new s3.Bucket(this, "StreamToS3AndBackBucket", {
      bucketName: "stream-to-s3-and-back",
      versioned: true,
      encryption: s3.BucketEncryption.S3_MANAGED,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

    // Create an IAM role for SFTP users
    const sftpUserRole = new iam.Role(this, "SftpUserRole", {
      assumedBy: new iam.ServicePrincipal("transfer.amazonaws.com"),
    });

    // Grant read/write permissions to the S3 bucket
    bucket.grantReadWrite(sftpUserRole);

    // Create the SFTP server
    const server = new transfer.CfnServer(this, "SftpServer", {
      protocols: ["SFTP"],
      identityProviderType: "SERVICE_MANAGED",
      loggingRole: sftpUserRole.roleArn,
    });

    // Create an SFTP user
    const user = new transfer.CfnUser(this, "SftpUser", {
      userName: "testuser",
      serverId: server.attrServerId,
      role: sftpUserRole.roleArn,
      homeDirectory: `/${bucket.bucketName}`,
      sshPublicKeys: [SFTP_PUBLIC_KEY.toString()],
    });

    // Output the SFTP server endpoint
    new cdk.CfnOutput(this, "SftpEndpoint", {
      value: `${server.attrServerId}.server.transfer.${this.region}.amazonaws.com`,
      description: "SFTP Server Endpoint",
    });
  }
}

The above code does the following:

• Import Necessary Modules: Import AWS CDK libraries, Node.js modules for file handling, and necessary AWS services.
• Read SFTP Public Key: Read the SFTP public key from a file.
• Define the Stack Class: Define a class that extends the CDK Stack to encapsulate all resources.
• Create an S3 Bucket: Create an S3 bucket with versioning, encryption, and a removal policy.
• Create an IAM Role for SFTP Users: Create an IAM role to be assumed by the SFTP service.
• Grant Permissions to the S3 Bucket: Grant read and write permissions to the IAM role for the S3 bucket.
• Create the SFTP Server: Create an SFTP server using AWS Transfer Family with the IAM role for logging.
• Create an SFTP User: Create an SFTP user associated with the server and provide the SSH public key.
• Output the SFTP Server Endpoint: Output the SFTP server endpoint URL for connecting to the server.

We are referencing the same S3 bucket we will use in uploads for our SFTP server to simplify things.

Once this is done, we can deploy our stack with the following command:

$ npx cdk deploy

The above step expects that your AWS credentials are set up correctly. If you have not done so, please follow a guide like this one for the AWS CLI.

Make sure to take note of the output for the SFTP server endpoint.

Once that is done, let's move onto our code to interface with our infrastructure.

Writing our demultiplexer

We covered a version of multiplexing in our Duplex streams post. In this case, we will be writing a demultiplexer.

Our demultiplexer will take in PassThrough stream types in the constructor and ensure that they are written to when we are processing data.

Let's see how that looks. Add the following to src/demultiplexer.ts:

import { Duplex, PassThrough } from "node:stream";

/**
 * We create a demultiplexer to write out to two different
 * streams.
 *
 * 1. To handle writing the CSV file.
 * 2. To handle writing the JSON file.
 *
 * Reading is handled by the pipeline implementation.
 */
export class Demultiplexer extends Duplex {
  private outputStreams: PassThrough[];

  constructor(outputStreams: PassThrough[]) {
    super({
      objectMode: true,
    });
    this.outputStreams = outputStreams;
  }

  _write(
    chunk: any,
    _encoding: BufferEncoding,
    callback: (error?: Error | null) => void
  ): void {
    for (const outputStream of this.outputStreams) {
      outputStream.write(chunk);
    }
    callback();
  }

  _final(callback: (error?: Error | null) => void): void {
    for (const outputStream of this.outputStreams) {
      outputStream.end();
    }
    callback();
  }
}

Our demultiplexer has implementations for both _write and _final. The _write method writes the chunk to all the output streams, while the _final method ends all the output streams.

Although this is not use case, a simple example of this in use:

import { PassThrough } from "node:stream";

const outputStream1 = new PassThrough();
const outputStream2 = new PassThrough();
const demux = new Demultiplexer([outputStream1, outputStream2]);

This would configure our multiplex to write messages to both outputStream1 and outputStream2.

The implementation itself isn't super flexible. In hindsight, you would ideally allow anything that is a Writable stream to be passed in. But it will suffice for our example (and refactoring this could be some nice homework).

Writing our upload script

Now that we have our infrastructure set up and our demultiplexer ready, let's write our upload script.

Add the following to src/upload.ts:

import { readFileSync } from "fs";
import { pipeline } from "node:stream/promises";
import { format } from "fast-csv";
import { createGzip } from "node:zlib";
import { S3Client } from "@aws-sdk/client-s3";
import { PassThrough, Readable } from "node:stream";
import { Upload } from "@aws-sdk/lib-storage";
import { Demultiplexer } from "./demultiplexer";
import * as path from "node:path";
import Client = require("ssh2-sftp-client");

const BUCKET_NAME = "stream-to-s3-and-back";
const sftp = new Client();
const s3Client = new S3Client();

const config = {
  host: "cdk-output.server.transfer.ap-southeast-2.amazonaws.com",
  port: 22,
  username: "testuser",
  privateKey: readFileSync(path.join(__dirname, "../sftp_key")),
};

async function uploadToS3() {
  try {
    const data = [
      {
        flavor: "vanilla",
        topping: "sprinkles",
      },
      {
        flavor: "chocolate",
        topping: "fudge",
      },
      {
        flavor: "strawberry",
        topping: "whipped cream",
      },
    ];

    await sftp.connect(config);
    console.log("SFTP connected successfully");

    // We use a PassThrough to support uploading
    // via the pipeline API.
    const uploadToS3PassThrough = new PassThrough();
    const upload = new Upload({
      client: s3Client,
      params: {
        Bucket: BUCKET_NAME,
        Key: "output.csv.gz",
        Body: uploadToS3PassThrough,
      },
    });

    // PassThroughs for Demux
    const s3UploadPassThrough = new PassThrough({
      objectMode: true,
    });

    const sftpUploadPassThrough = new PassThrough({
      objectMode: true,
    });
    const demux = new Demultiplexer([
      s3UploadPassThrough,
      sftpUploadPassThrough,
    ]);

    await Promise.all([
      pipeline(
        Readable.from(data),
        format({ quoteColumns: true, headers: ["flavor", "topping"] }),
        createGzip(),
        demux
      ),
      pipeline(s3UploadPassThrough, uploadToS3PassThrough),
      pipeline(
        sftpUploadPassThrough,
        sftp.createWriteStream(`/${BUCKET_NAME}/sftp-upload.csv.gz`)
      ),
    ]);

    await sftp.end();
    await upload.done();

    console.log("File processed and uploaded successfully");
  } catch (error) {
    console.error("Error:", error);
  }
}

async function main() {
  await uploadToS3();
}

main();

This script is more complex than the others we've written so far. Although it is a little contrived. Here's a breakdown of what it does:

• Import Necessary Modules: Import various Node.js and AWS SDK modules for file handling, streaming, CSV formatting, compression, and S3 operations.
• Define Constants: Define the S3 bucket name, SFTP client instance, and S3 client instance.
• Read SFTP Private Key: Read the SFTP private key from a file.
• Set SFTP Configuration: Set up the SFTP connection configuration.
• Define uploadToS3 Function: Define an asynchronous function to handle the data processing and uploading.
• Prepare Data: Create an array of data to be uploaded.
• Connect to SFTP: Connect to the SFTP server using the configuration.
• Create PassThrough Streams: Create PassThrough streams to facilitate the pipeline process.
• Create S3 Upload Object: Create an upload object for S3 using the AWS SDK.
• Create Demultiplexer: Create a Demultiplexer instance to split the data stream.
• Setup Pipelines: Set up parallel pipelines for processing data, compressing it, and sending it to both S3 and SFTP.
• End SFTP Connection: End the SFTP connection after the upload is complete.
• Log Success or Error: Log the success message or handle errors if they occur.
• Define main Function: Define a main function to call uploadToS3.
• Execute main Function: Execute the main function to start the process.

You can visualize the process a bit better with the following flow chart:

The significant part is our pipelines:

await Promise.all([
  pipeline(
    Readable.from(data),
    format({ quoteColumns: true, headers: ["flavor", "topping"] }),
    createGzip(),
    demux
  ),
  pipeline(s3UploadPassThrough, uploadToS3PassThrough),
  pipeline(
    sftpUploadPassThrough,
    sftp.createWriteStream(`/${BUCKET_NAME}/sftp-upload.csv.gz`)
  ),
]);

We are using the pipeline function to chain together our streams.

The first pipeline generates the CSV data, formats it, compresses it, and then demultiplexes it. The second pipeline sends the data to S3. The third pipeline sends the data to the SFTP server.

Our second and third streams are receivers for our demultiplexer. The PassThrough is used as both the destination of our compression pipeline, and is also the source for our second and third pipelines that are doing two very different things.

It is also important for us to await the end of the remote streams in our case. That is where the following code comes into play:

await sftp.end();
await upload.done();

Without it, our script would end prematurely before the data is fully uploaded.

Next, we can run our script with the following command:

$ npx tsx src/upload.ts
SFTP connected successfully
File processed and uploaded successfully

At this point, if you jump onto your AWS S3 bucket, you should see a file called output.csv.gz that contains the compressed CSV file as well as a file called sftp-upload.csv.gz that contains the same file that we uploaded through SFTP.

Writing our download script

At this point, we have successfully uploaded a file to S3 and SFTP. Now, let's write a script to download the file from S3 and SFTP, decompress it, and write it to disk.

Add the following to src/download.ts:

import { createWriteStream, readFileSync } from "node:fs";
import { parse } from "fast-csv";
import { createGunzip } from "node:zlib";
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { PassThrough, Readable, Transform } from "node:stream";
import { pipeline } from "node:stream/promises";
import { stringify } from "yaml";
import { Demultiplexer } from "./demultiplexer";
import * as path from "node:path";
import Client = require("ssh2-sftp-client");

const BUCKET_NAME = "stream-to-s3-and-back";
const sftp = new Client();
const s3Client = new S3Client();

const config = {
  host: "cdk-output.server.transfer.ap-southeast-2.amazonaws.com",
  port: 22,
  username: "testuser",
  privateKey: readFileSync(path.join(__dirname, "../sftp_key")),
};

const yamlTransformStream = new Transform({
  objectMode: true,
  transform(chunk, _encoding, callback) {
    this.push(stringify([chunk]) + "\n");
    callback();
  },
});

async function downloadFromS3() {
  try {
    await sftp.connect(config);
    console.log("SFTP connected successfully");

    const { Body: s3UploadBody } = await s3Client.send(
      new GetObjectCommand({
        Bucket: BUCKET_NAME,
        Key: "output.csv.gz",
      })
    );

    if (s3UploadBody instanceof Readable === false) {
      throw new Error("S3 object body is not a readable stream");
    }

    const csvStreamPassThrough = new PassThrough({
      objectMode: true,
    });
    const csvOutput = createWriteStream("downloaded_output.csv");

    const yamlStreamPassThrough = new PassThrough({
      objectMode: true,
    });
    const yamlOutput = createWriteStream("downloaded_output.yaml");

    const demux = new Demultiplexer([
      csvStreamPassThrough,
      yamlStreamPassThrough,
    ]);

    // Create a write stream for the SFTP CSV output
    const sftpCsvOutput = createWriteStream("downloaded_sftp_output.csv");
    const sftpReadableStream = sftp.createReadStream("output.csv.gz");

    // Buffer everything into the demux stream
    await Promise.all([
      // S3 download streams
      pipeline(s3UploadBody, createGunzip(), demux),
      pipeline(csvStreamPassThrough, csvOutput),
      pipeline(
        yamlStreamPassThrough,
        parse({ headers: true, objectMode: true }),
        yamlTransformStream,
        yamlOutput
      ),
      // SFTP download stream
      pipeline(sftpReadableStream, createGunzip(), sftpCsvOutput),
    ]);

    await sftp.end();

    console.log("File downloaded, decompressed, and processed successfully");
  } catch (error) {
    console.error("Error:", error);
  }
}

async function main() {
  await downloadFromS3();
}

main();

Again, you will need to update your SFTP endpoint in the script.

Let's breakdown what we are doing here:

• Import Necessary Modules: Import Node.js and AWS SDK modules for file handling, streaming, CSV parsing, decompression, YAML stringifying, and S3 operations.
• Define Constants: Define the S3 bucket name, SFTP client instance, and S3 client instance.
• Read SFTP Private Key: Read the SFTP private key from a file.
• Set SFTP Configuration: Set up the SFTP connection configuration.
• Define YAML Transform Stream: Create a transform stream to convert CSV data to YAML format.
• Define downloadFromS3 Function: Define an asynchronous function to handle data downloading, decompressing, and processing.
• Connect to SFTP: Connect to the SFTP server using the configuration.
• Download S3 Object: Use AWS SDK to download the compressed CSV file from S3.
• Check S3 Body Type: Ensure the downloaded S3 object body is a readable stream.
• Create PassThrough Streams: Create PassThrough streams to facilitate the pipeline process.
• Create Write Streams: Create write streams for local CSV and YAML files, and for SFTP CSV output.
• Create Demultiplexer: Create a Demultiplexer instance to split the data stream into CSV and YAML streams.
• Setup Pipelines: Set up parallel pipelines for decompressing S3 data, splitting streams, and writing to local files.
• Decompress SFTP Data: Set up a pipeline to download, decompress, and write SFTP CSV data to a local file.
• End SFTP Connection: End the SFTP connection after the download and processing are complete.
• Log Success or Error: Log the success message or handle errors if they occur.
• Define main Function: Define a main function to call downloadFromS3.
• Execute main Function: Execute the main function to start the process.

Another flow chart to help visualize how the code is executing:

Again, it's the pipelines that play the most integral role:

// Buffer everything into the demux stream
await Promise.all([
  // S3 download streams
  pipeline(s3UploadBody, createGunzip(), demux),
  pipeline(csvStreamPassThrough, csvOutput),
  pipeline(
    yamlStreamPassThrough,
    parse({ headers: true, objectMode: true }),
    yamlTransformStream,
    yamlOutput
  ),
  // SFTP download stream
  pipeline(sftpReadableStream, createGunzip(), sftpCsvOutput),
]);

Our first three pipelines work with the multiplexer to download the content from S3, then split that content into streams that write a CSV file and a YAML file respectively.

The fourth pipeline downloads the compressed CSV file from SFTP, decompresses it, and writes it to disk.

We can run this script with the following:

$ npx tsx src/download.ts

SFTP connected successfully
File downloaded, decompressed, and processed successfully

We will end up with three files written to disk:

1. downloaded_output.csv: The CSV file downloaded from S3.
2. downloaded_output.yaml: The YAML file generated from the CSV file.
3. downloaded_sftp_output.csv: The CSV file downloaded from the SFTP server.

The content of (1) and (3) should be identical:

"flavor","topping"
"vanilla","sprinkles"
"chocolate","fudge"
"strawberry","whipped cream"

While the content of (2) should be:

- flavor: vanilla
  topping: sprinkles

- flavor: chocolate
  topping: fudge

- flavor: strawberry
  topping: whipped cream

Please be aware that awaiting the end of the remote streams is important again. We run into the same problem of prematurely ending the stream without it.

Otherwise, we have successfully uploaded a file to S3 and SFTP, downloaded it, decompressed it, and written it to disk!

Teardown

Once you are done with the project, you can tear down the infrastructure with the following command:

$ npx cdk destroy

Please be aware that all the content from the bucket needs to be deleted beforehand. If you get an error teardown, head to the AWS console and manually delete the stack and bucket... and definitely please do. Transfer Family is sadly a bit $$$.

Conclusion

Congratulations on making it through the Node.js Streams blog series! By now, you should have a solid understanding of how to work with streams in Node.js, including the different types of streams and how they can be utilized.

In this post, we introduced the PassThrough stream and demonstrated its use in a practical example involving AWS CDK, AWS S3, and AWS Transfer Family. We explored how to create a pipeline for uploading and downloading files using Node.js streams, showcasing the versatility and power of streams in handling complex data flows.

Our key takeaways and lessons today:

• PassThrough Streams: These streams allow data to pass through without modification, making them useful for observing data flow, delaying piping, or providing placeholders for future transformations.
• AWS Integration: Leveraging AWS CDK, S3, and Transfer Family, we created a robust data pipeline for handling file uploads and downloads.
• Demultiplexer: Implementing a custom demultiplexer stream allowed us to split data streams into multiple outputs efficiently.
• Practical Application: By applying these concepts, we demonstrated how to handle real-world scenarios involving data processing and transfer in a scalable and efficient manner.

This journey through Node.js streams, from understanding the basics to implementing advanced pipelines, equips you with the knowledge to tackle various data processing tasks. Streams provide a powerful abstraction for handling data flows, and with the skills you've gained, you can now confidently apply them to your projects.

Resources and further reading

• Understanding Node.js Streams Blog Post
• Readable Streams in Node.js
• Writable Streams in Node.js
• Duplex Streams in Node.js
• Transform Streams in Node.js
• Node.js Stream Pipeline API Documentation
• Node.js Streams series
• Transform streams
• PassThrough stream
• AWS CDK
• AWS S3
• AWS Transfer Family
• AWS CLI configuration guide
• GitHub repository for the full project code

Disclaimer: This blog post used AI to generate the images used for the analogy.

Photo credit: robanderson72

]]>

During the Node.js Streams blog series, we covered the core fundamentals you should know about Node.js streams and spent a blog post diving into both Readable streams and Writable streams, we've covered the fundamentals of what you should know about streams in Node.js, as well as the four fundamental stream types.

In this post, we are diving into to the pipeline function from the Node.js stream module: how it can be of use and how we can use it for asynchronous operations.

How to think about pipeline streams

The stream.pipeline function is a utility function that allows us to easily pipe streams together. It is very aptly name, so you can think of it as our water network that bridges between the different streams in our analogies that we've used so far.

Creating our first pipeline

We can export the pipeline function from the node:stream module and use it to connect streams together.

const { pipeline, Readable, Writable } = require("node:stream");

const data = ["Hello", "World!"];
const readStream = Readable.from(data);

// Create a writable function
const writeStream = new Writable({
  write(chunk, _encoding, callback) {
    console.log("Writing:", chunk.toString());
    callback();
  },
});

pipeline(readStream, writeStream, (err) => {
  if (err) {
    console.error("Pipeline failed.", err);
  } else {
    console.log("Pipeline succeeded.");
  }
});

In the above example, we create a Readable stream from an array of strings and a Writable stream that logs the data to the console. We then use the pipeline function to connect the two streams together.

Running it gives the following:

$ node pipelines/simple.js
Writing: Hello
Writing: World!
Pipeline succeeded.

This simple example demonstrates how we can use the pipeline function to connect streams together.

For a more complex pipeline (well, more complex than above), we can connect multiple streams together:

const { pipeline, Readable, Writable, Transform } = require("node:stream");

const data = ["Hello", "World!"];
const readStream = Readable.from(data);

// Create a writable function
const writeStream = new Writable({
  write(chunk, _encoding, callback) {
    console.log("Writing:", chunk.toString());
    callback();
  },
});

const lowerCaseTransform = new Transform({
  transform(chunk, _encoding, callback) {
    let str = chunk.toString();
    this.push(`${str.slice(0, 1).toUpperCase()}${str.slice(1)}`);
    callback();
  },
});

const addSuffixTransform = new Transform({
  transform(chunk, _encoding, callback) {
    this.push(`${chunk.toString()}-0123`);
    callback();
  },
});

pipeline(
  readStream,
  lowerCaseTransform,
  addSuffixTransform,
  writeStream,
  (err) => {
    if (err) {
      console.error("Pipeline failed.", err);
    } else {
      console.log("Pipeline succeeded.");
    }
  }
);

In this case, we add in a couple of transforms just to make some string modifications.

The result:

$ node pipelines/slightly-more-complex.js
Writing: hello-0123
Writing: world!-0123
Pipeline succeeded.

The follow sequence diagram can you help you visualize how the pipeline is operating:

Why use the pipeline function?

The pipeline function from the stream module in Node.js offers several important benefits:

1. Automatic error handling:

   • pipeline automatically handles errors from any of the streams in the pipeline.
   • It ensures that if an error occurs in any stream, all streams are properly closed.
   • This prevents potential memory leaks and resource issues.

2. Simplified code:

   • It reduces the boilerplate code needed to properly manage multiple streams.
   • You don't need to manually handle 'error' and 'end' events for each stream.

3. Backpressure management:

   • pipeline automatically manages backpressure between the streams.
   • It ensures that faster streams don't overwhelm slower ones by pausing and resuming as needed.

4. Cleanup on completion:

   • It automatically cleans up resources when the pipeline finishes or errors out.
   • This includes closing file descriptors and freeing up memory.

5. Promise support:

   • When used with util.promisify or when directly imported from node:stream/promises, it can return a Promise, allowing for easier use with async/await syntax.

6. Multiple streams:

   • It can handle any number of streams, making it easy to create complex data processing pipelines.

7. Improved performance:

   • The internal implementation is optimized for performance, especially when dealing with object mode streams.

8. Consistent behavior:

   • It ensures consistent behavior across different types of streams (readable, writable, transform).

9. Error propagation:

   • Errors are properly propagated through the pipeline, ensuring that error handling is centralized and reliable.

10. Ordering guarantee:

    • It guarantees that data will be processed in the order it's received, which is particularly important for certain types of data processing.

Here's another simple example to illustrate its use with other Node.js modules:

Using pipeline is generally considered a best practice when working with streams in Node.js due to these benefits.

Asynchronous streams

What happens if our streams happen to be asynchronous? We can import pipeline from the node:stream/promises module to handle this.

const { Readable, Transform, Writable } = require("node:stream");
const { pipeline } = require("node:stream/promises");

// Simulated async operation
const asyncOperation = (num) => {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(num * 2);
    }, Math.random() * 1000); // Random delay to simulate varying processing times
  });
};

// Source stream
class NumberSource extends Readable {
  constructor(options) {
    super(options);
    this.current = 0;
    this.max = 10;
  }

  _read() {
    if (this.current < this.max) {
      this.push(this.current.toString());
      this.current++;
    } else {
      this.push(null);
    }
  }
}

// Async transform stream
class AsyncDoubler extends Transform {
  constructor(options) {
    super({ ...options, objectMode: true });
  }

  _transform(chunk, encoding, callback) {
    const num = parseInt(chunk, 10);
    asyncOperation(num)
      .then((result) => {
        this.push(`${num} doubled is ${result}\n`);
        callback();
      })
      .catch(callback);
  }
}

// Destination stream
class ConsoleWriter extends Writable {
  _write(chunk, encoding, callback) {
    console.log(chunk.toString().trim());
    callback();
  }
}

async function runPipeline() {
  try {
    await pipeline(new NumberSource(), new AsyncDoubler(), new ConsoleWriter());
    console.log("Pipeline succeeded");
  } catch (err) {
    console.error("Pipeline failed", err);
  }
}

runPipeline();

In this example, we have a NumberSource readable stream that emits numbers from 0 to 9, an AsyncDoubler transform stream that doubles the numbers asynchronously, and a ConsoleWriter writable stream that logs the doubled numbers to the console.

Running the code gives the following:

$ node pipelines/async.js
0 doubled is 0
1 doubled is 2
2 doubled is 4
3 doubled is 6
4 doubled is 8
5 doubled is 10
6 doubled is 12
7 doubled is 14
8 doubled is 16
9 doubled is 18
Pipeline succeeded

We can visualize the asynchronous pipeline as follows:

If we had tried running the pipeline without supporting async, the code would have executed without waiting:

// WHAT NOT TO DO
const { pipeline, Readable, Transform, Writable } = require("node:stream");
// const { pipeline } = require("stream/promises");

// Simulated async operation
const asyncOperation = (num) => {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(num * 2);
    }, Math.random() * 1000); // Random delay to simulate varying processing times
  });
};

// Source stream
class NumberSource extends Readable {
  constructor(options) {
    super(options);
    this.current = 0;
    this.max = 10;
  }

  _read() {
    if (this.current < this.max) {
      this.push(this.current.toString());
      this.current++;
    } else {
      this.push(null);
    }
  }
}

// Async transform stream
class AsyncDoubler extends Transform {
  constructor(options) {
    super({ ...options, objectMode: true });
  }

  _transform(chunk, encoding, callback) {
    const num = parseInt(chunk, 10);
    asyncOperation(num)
      .then((result) => {
        this.push(`${num} doubled is ${result}\n`);
        callback();
      })
      .catch(callback);
  }
}

// Destination stream
class ConsoleWriter extends Writable {
  _write(chunk, encoding, callback) {
    console.log(chunk.toString().trim());
    callback();
  }
}

async function runPipeline() {
  try {
    pipeline(
      new NumberSource(),
      new AsyncDoubler(),
      new ConsoleWriter(),
      (err) => {
        if (err) {
          console.error("Pipeline failed", err);
        } else {
          console.log("Pipeline succeeded");
        }
      }
    );

    console.log("HERE ALREADY");
    process.exit();
  } catch (err) {
    console.error("Pipeline failed", err);
  }
}

runPipeline();

Running the above would exit the code before the streams ran.

$ node pipelines/async-incorrect.js
HERE ALREADY

Multiplexer example

We won't go too deep into other pipeline examples in this post as I will be doing so in some follow-up posts, but we will finish off looking at an example using a Duplex stream to create a multiplexer.

In the following example, we create a multiplexer stream that takes two input streams and multiplexes them into a single output stream.

const { Duplex, pipeline, Transform } = require("node:stream");
const { createReadStream, createWriteStream } = require("node:fs");

class Multiplexer extends Duplex {
  constructor(options) {
    super(options);
    this.sources = [];
    this.currentSource = 0;
    this.activeSourcesCount = 0;
  }

  _read(size) {
    this.readFromSources(size);
  }

  readFromSources(size) {
    if (this.activeSourcesCount === 0) {
      this.push(null);
      return;
    }

    let chunk;
    do {
      const source = this.sources[this.currentSource];
      chunk = source.read(size);

      if (chunk !== null) {
        this.push(chunk);
        return;
      }

      this.currentSource = (this.currentSource + 1) % this.sources.length;
    } while (this.currentSource !== 0);

    // If we've gone through all sources without reading, wait for more data
    this.sources.forEach((source) => {
      source.once("readable", () => this.readFromSources(size));
    });
  }

  _write(chunk, encoding, callback) {
    // For simplicity, we're not implementing write functionality
    callback();
  }

  addSource(source) {
    this.sources.push(source);
    this.activeSourcesCount++;
    source.on("end", () => {
      this.activeSourcesCount--;
      if (this.activeSourcesCount === 0) {
        this.push(null);
      }
    });
  }
}

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());
  },
});

// Create input streams
const input1 = createReadStream("file1.txt");
const input2 = createReadStream("file2.txt");

// Create output stream
const output = createWriteStream("output.txt");

// Create multiplexer
const multiplexer = new Multiplexer();
multiplexer.addSource(input1);
multiplexer.addSource(input2);

// Use pipeline to connect streams
pipeline(multiplexer, upperCaseTransform, output, (err) => {
  if (err) {
    console.error("Pipeline failed:", err);
  } else {
    console.log("Pipeline succeeded");
  }
});

In the example above, we have two sources file1.txt and file2.txt that will be multiplexed into a single output stream. The output stream will be transformed to uppercase before being written to output.txt.

If our files are Hello, world! and Streams are fun for file1.txt and file2.txt respectively, then an example output we can get at output.txt is:

HELLO, WORLD!STREAMS ARE FUN

The sequence diagram for the multiplexer pipeline can be visualized as follows:

In the current implementation:

• The multiplexer attempts to read from each source in turn.
• Whichever source has data available first will be read from and its data pushed to the output.
• This can lead to interleaving of data from different sources in an unpredictable way, especially if the sources become readable at different times or at different rates.

That means that the output may not be in the order that the data is written to output.txt is not guaranteed.

const { pipeline } = require("node:stream");
const { createGzip } = require("node:zlib");
const { createReadStream, createWriteStream } = require("node:fs");

pipeline(
  createReadStream("input.txt"),
  createGzip(),
  createWriteStream("output.txt.gz"),
  (err) => {
    if (err) {
      console.error("Pipeline failed", err);
    } else {
      console.log("Pipeline succeeded");
    }
  }
);

At this point in our Node.js Streams blog series, we covered the core fundamentals you should know about Node.js streams and spent a blog post diving into both Readable streams and Writable streams.

In this shorter blog post, we will cover the fourth and final fundamental stream type Transform streams in this blog post.

How to think about Transform streams

Transform streams are a special type of Duplex stream (which we covered in the last blog post). We use Transform streams for transforming data (very apt naming) as it passes through the stream.

In the Node.js streams fundamentals blog post, the mental model we used for Transform streams was the the water treatment plant.

Some parallels with the analogy to make it memorable:

1. Dual Nature: Transform streams are both readable and writable. A water treatment plant receives untreated water (input) and outputs clean water (output).
2. Data Transformation: Modifies input data before passing it as output. Similarly, various processes in the plant (filtration, chemical treatment) change the water's composition.
3. Streaming Processing: Handles data in chunks, not all at once. We can think of this like how water flows continuously through the plant, not in single large batches.
4. Backpressure Handling: Manages flow when output is slower than input. Reservoirs or holding tanks regulate water flow when treatment is slower than intake.
5. Piping: Can be easily connected to other streams. Treatment plants can also be connected to various water sources and distribution systems
6. Error Handling: Manages issues during transformation. We can think of them like the safety protocols and backup systems in the plant handle treatment failures.

These analogies help tie together the core concepts of Transform streams, as well as a few of the fundamentals that we covered at the start of this series.

With that out of the way, let's see how we can create Transform streams in Node.js.

How to create Transform streams

Similar to all the previous posts, we can create Transform streams using the node:stream module in a few ways:

1. Using the stream.Transform class: We can extend the stream.Transform class to create our Transform stream.
2. Using the stream.Transform constructor: We can use the stream.Transform constructor to create a Transform stream.

const { Transform } = require("stream");

class UpperCaseTransform extends Transform {
  constructor() {
    super();
  }

  _transform(chunk, encoding, callback) {
    try {
      this.push(chunk.toString().toUpperCase());
      callback();
    } catch (error) {
      callback(error);
    }
  }
}

const upperCaseTransform = new UpperCaseTransform();

process.stdin.pipe(upperCaseTransform).pipe(process.stdout);

const { Transform } = require("stream");

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  },
});

process.stdin.pipe(upperCaseTransform).pipe(process.stdout);

const zlib = require("node:zlib");
const gzip = zlib.createGzip();
// gzip is a transform stream (both readable and writable) - we cover this more in a future post

const fs = require("node:fs");
const zlib = require("node:zlib");

const readStream = fs.createReadStream("input.txt");
const writeStream = fs.createWriteStream("input.txt.gz");
const gzip = zlib.createGzip();

readStream.pipe(gzip).pipe(writeStream);

writeStream.on("finish", () => console.log("File successfully compressed"));

const crypto = require("crypto");
const cipher = crypto.createCipher("aes192", "secret");
// cipher is a transform stream (both readable and writable) - we cover this more in a future post

const crypto = require("node:crypto");
const fs = require("node:fs");

const algorithm = "aes-192-cbc";
const password = "Password used to generate key";

const key = crypto.scryptSync(password, "salt", 24);
const iv = Buffer.alloc(16, 0);

const cipher = crypto.createCipheriv(algorithm, key, iv);

const input = fs.createReadStream("input.txt");
const output = fs.createWriteStream("encrypted.txt");

input.pipe(cipher).pipe(output);

output.on("finish", () => console.log("File encrypted successfully"));

At this point in our Node.js Streams blog series, we covered the core fundamentals you should know about Node.js streams and spent a blog post diving into both Readable streams and Writable streams.

The next fundamental stream type that we're going to look into are Duplex streams.

How to think about Duplex streams

Let's recap our mental model on Duplex streams: the two-way water pipe.

To understand this analogy a bit more, let's place flesh it out a little bit more:

• Bidirectional flow: Like a two-way water pipe that allows water to flow in both directions, a Duplex stream enables data to flow both ways.
• Separate channels: Imagine the pipe having two separate channels, one for each direction. This represents how Duplex streams manage separate internal buffers for reading and writing.
• Valves (methods):
  • _read() is like opening a valve to allow water (data) to flow out.
  • _write() is like opening a valve to let water (data) flow in.
• Pressure (backpressure):
  • If water is flowing in faster than it can flow out, pressure builds up. This is analogous to backpressure in streams.
  • The pipe might have a mechanism to signal when it's full, similar to how streams emit events or return values to indicate backpressure.
• Flow control: Imagine regulators on each end of the pipe that can slow down or speed up the flow. This represents how Duplex streams can pause and resume data flow.
• Pipe connections (piping): You can connect multiple two-way pipes to create a complex water system, just as you can pipe multiple Duplex streams together.
• Closing the pipe: The process of properly shutting off both ends of the pipe is similar to the _final() method ensuring all data is processed before closing.
• Leaks (error handling): A leak in the pipe is like an error in the stream, requiring proper handling to prevent data loss.
• Water types (modes): The pipe could carry either a continuous flow of water (binary mode) or discrete objects like fish (object mode).

This analogy can help readers visualize the abstract concepts of Duplex streams in a more concrete, familiar context.

Alternatives to this would be to take the analogy of walkie talkies, or a two-way street with cars coming and going in both directions. Choose whatever helps you visualize things best. I stick with the two-way water pipe analogy to fit in better with all the water-themed analogies I used for Node.js streams.

You can also think of Duplex streams as a combination of Readable and Writable streams, meaning that our previous deep-dives analogies for those can also apply to help you solidify your understanding of Duplex streams.

How to create Duplex streams

We can create duplex streams using the node:stream module. There are a number of ways:

1. Extending the Duplex class.
2. Using the new Duplex() constructor.
3. Using Duplex.from.

Extending the Duplex class

const { Duplex } = require("stream");

class MyDuplex extends Duplex {
  constructor(options) {
    super(options);
    this.data = ["Hello", "World", "Duplex", "Stream"];
  }

  _read(size) {
    const chunk = this.data.shift();
    if (chunk) {
      this.push(chunk);
    } else {
      this.push(null);
    }
  }

  _write(chunk, encoding, callback) {
    console.log("Received:", chunk.toString());
    callback();
  }
}

const myDuplex = new MyDuplex();

myDuplex.on("data", (chunk) => console.log("Read:", chunk.toString()));
myDuplex.write("Some data");

If you have followed along with my previous posts, you will notice that the _read and _write methods are similar to the ones we used for Readable and Writable streams. The key difference is that Duplex streams implement both methods.

The above code creates a Duplex stream that reads data from an internal array and writes data to the console.

When you run node script.js, you should see the following output:

$ node duplex/one.js
Received: Some data
Read: Hello
Read: World
Read: Duplex
Read: Stream

Here's a sequence diagram visual:

Using the new Duplex() constructor

Similar to Readable and Writable, we do not need to extend the class if we choose that we do not want to.

const { Duplex } = require("node:stream");

const duplex = new Duplex({
  read(size) {
    // Implement read logic
  },
  write(chunk, encoding, callback) {
    // Implement write logic
    callback();
  },
});

Using Duplex.from

With Duplex.from, we can write the implementation for the readable and writable properties:

const { Duplex } = require("node:stream");

const duplex = Duplex.from({
  readable: (async function* () {
    yield "data";
    yield "more data";
  })(),
  writable: new WritableStream({
    write(chunk) {
      console.log("Received:", chunk);
    },
  }),
});

Alternatively, we can pass in existing Readable and Writable streams:

const { Duplex } = require("node:stream");
const { PassThrough } = require("node:stream");

const readable = new PassThrough();
const writable = new PassThrough();

const duplex = Duplex.from({ readable, writable });

Use cases for Duplex streams

Duplex streams have several practical use cases in Node.js applications. Here are some common scenarios where Duplex streams are particularly useful:

1. Network sockets:

   • TCP sockets
   • WebSocket connections
   • Implementing custom network protocols

2. Database connections:

   • Streaming queries and results
   • Real-time data synchronization

3. File I/O:

   • Simultaneous reading and writing of files
   • Implementing file-based IPC (Inter-Process Communication)

4. Encryption/Decryption:

   • Creating streams that can both encrypt and decrypt data

5. Compression/Decompression:

   • Implementing streams that can both compress and decompress data

6. Parser/Serializer:

   • Creating streams that can parse incoming data and serialize outgoing data

7. Protocol implementation:

   • HTTP/2 streams
   • Custom bi-directional protocols

8. Proxy servers:

   • Forwarding requests and responses between client and server

9. Telnet-like interfaces:

   • Creating interactive command-line tools

10. Middleware in stream processing pipelines:

    • Implementing transformations that require both read and write capabilities

11. Testing and mocking:

    • Simulating complex I/O scenarios in unit tests

12. Multiplexing/Demultiplexing:

    • Combining multiple streams into one or splitting one stream into many

13. Buffering and flow control:

    • Implementing custom buffering strategies for data flow

Let's go through some simple examples for some of these.

Network sockets

Here we show a TCP server and client operating as Duplex streams:

const net = require("net");

const server = net.createServer((socket) => {
  socket.write("Welcome to the server!\n");
  socket.on("data", (data) => {
    console.log("Received:", data.toString());
    socket.write(`Echo: ${data}`);
  });
});

server.listen(3000, () => {
  const client = new net.Socket();
  client.connect(3000, "localhost", () => {
    client.write("Hello, server!");
  });

  client.on("data", (data) => {
    console.log("Server says:", data.toString());
  });
});

In this example, we create a TCP server and client using Node's net module. The server listens for connections, echoes back any received data, and sends a welcome message. The client connects to the server, sends a message, and logs any responses.

Both the server socket and client socket are Duplex streams, allowing bidirectional communication.

File I/O

In this case we simultaneously read and write:

const fs = require("fs");
const { Duplex } = require("stream");

class FileDuplex extends Duplex {
  constructor(options) {
    super(options);
    this.readFd = fs.openSync("input.txt", "r");
    this.writeFd = fs.openSync("output.txt", "w");
  }

  _read(size) {
    const buf = Buffer.alloc(size);
    fs.read(this.readFd, buf, 0, size, null, (err, bytesRead) => {
      if (bytesRead > 0) {
        this.push(buf.slice(0, bytesRead));
      } else {
        this.push(null);
      }
    });
  }

  _write(chunk, encoding, callback) {
    fs.write(this.writeFd, chunk, callback);
  }

  _final(callback) {
    fs.close(this.readFd, (err) => {
      fs.close(this.writeFd, callback);
    });
  }
}

const fileDuplex = new FileDuplex();
fileDuplex.pipe(process.stdout);
fileDuplex.write("This will be written to output.txt\n");

This example implements a custom Duplex stream for simultaneous file reading and writing. It reads from 'input.txt' and writes to 'output.txt'. The _read method reads chunks from the input file, while the _write method writes data to the output file. This demonstrates how Duplex streams can be used for concurrent file operations.

Encryption/Decryption

const { Duplex } = require("stream");
const crypto = require("crypto");

class CipherDuplex extends Duplex {
  constructor(options) {
    super(options);
    this.key = crypto.randomBytes(32);
    this.iv = crypto.randomBytes(16);
  }

  _read(size) {}

  _write(chunk, encoding, callback) {
    const cipher = crypto.createCipheriv("aes-256-cbc", this.key, this.iv);
    let encrypted = cipher.update(chunk);
    encrypted = Buffer.concat([encrypted, cipher.final()]);
    this.push(encrypted);
    callback();
  }
}

const cipherDuplex = new CipherDuplex();
cipherDuplex.on("data", (data) => {
  console.log("Encrypted:", data.toString("hex"));
});
cipherDuplex.write("Secret message");

We have the CipherDuplex stream that encrypts incoming data using AES-256-CBC encryption. It generates a random key and initialization vector (IV) for the encryption. When data is written to the stream, it's encrypted and then pushed as readable data. This shows how Duplex streams can be used for on-the-fly data transformation.

Compression/Decompression

const zlib = require("zlib");
const { pipeline } = require("stream");

const compressDecompress = zlib.createGzip();

pipeline(
  process.stdin,
  compressDecompress,
  zlib.createGunzip(),
  process.stdout,
  (err) => {
    if (err) {
      console.error("Pipeline failed", err);
    } else {
      console.log("Pipeline succeeded");
    }
  }
);

// Usage: echo "Hello, world!" | node script.js

This example uses zlib's built-in Duplex streams for compression and decompression. It sets up a pipeline that compresses input from stdin, immediately decompresses it, and then writes the result to stdout. This demonstrates how Duplex streams can be chained together for complex data processing.

Parser/serializer

const { Duplex } = require("stream");

class JSONDuplex extends Duplex {
  constructor(options) {
    super({ ...options, objectMode: true });
  }

  _read(size) {}

  _write(chunk, encoding, callback) {
    try {
      const parsed = JSON.parse(chunk);
      this.push(parsed);
      callback();
    } catch (err) {
      callback(err);
    }
  }

  _final(callback) {
    this.push(null);
    callback();
  }
}

const jsonDuplex = new JSONDuplex();

jsonDuplex.on("data", (data) => {
  console.log("Parsed object:", data);
});

jsonDuplex.write('{"name": "John", "age": 30}');
jsonDuplex.write('{"city": "New York", "country": "USA"}');

Here we create a JSONDuplex stream that parses incoming JSON strings into JavaScript objects. It operates in object mode, allowing it to push parsed objects directly. This showcases how Duplex streams can be used for data format conversion in both directions, although this example only implements the parsing direction.

Multiplexing/Demultiplexing

const { Duplex, PassThrough } = require("stream");

class MultiplexDuplex extends Duplex {
  constructor(options) {
    super(options);
    this.streams = new Map();
  }

  _read(size) {}

  _write(chunk, encoding, callback) {
    const { streamId, data } = JSON.parse(chunk);
    if (!this.streams.has(streamId)) {
      this.streams.set(streamId, new PassThrough());
    }
    this.streams.get(streamId).write(data);
    callback();
  }

  getStream(streamId) {
    if (!this.streams.has(streamId)) {
      this.streams.set(streamId, new PassThrough());
    }
    return this.streams.get(streamId);
  }
}

const multiplexer = new MultiplexDuplex();

const stream1 = multiplexer.getStream(1);
const stream2 = multiplexer.getStream(2);

stream1.on("data", (data) => console.log("Stream 1:", data.toString()));
stream2.on("data", (data) => console.log("Stream 2:", data.toString()));

multiplexer.write(JSON.stringify({ streamId: 1, data: "Hello from stream 1" }));
multiplexer.write(JSON.stringify({ streamId: 2, data: "Hello from stream 2" }));

This example implements a MultiplexDuplex stream that can handle multiple logical streams over a single Duplex stream. It uses a streamId to differentiate between different logical streams. Data written to the multiplexer is routed to the appropriate PassThrough stream based on the streamId. This demonstrates how Duplex streams can be used to implement complex communication protocols.

Conclusion

In this journey through the world of Node.js streams, we've delved into the fundamental concepts of streams, explored Readable and Writable streams in detail, and now, we've uncovered the intricacies of Duplex streams. By understanding Duplex streams through the analogy of a two-way water pipe, we've gained insight into how data can flow bidirectionally, the importance of backpressure, and the mechanisms for controlling this flow.

From extending the Duplex class to using the new Duplex() constructor and Duplex.from, we've seen multiple ways to create Duplex streams. These streams offer a powerful way to handle scenarios where both reading and writing are essential, such as network sockets, file I/O, encryption, compression, and complex protocols.

By exploring practical use cases, we've demonstrated how Duplex streams can be applied to real-world problems, making them an indispensable tool in your Node.js toolkit. Whether you're building a custom network protocol, implementing a parser, or creating a multiplexing system, Duplex streams provide the flexibility and control you need.

As we continue this series, we'll look into other advanced stream types and techniques, helping you master the full spectrum of stream capabilities in Node.js. Stay tuned for more insights and practical examples in our next post on Transform streams. Thank you for following along, and happy streaming!

Resources and further reading

• Understanding Node.js Streams Blog Post
• Readable Streams in Node.js
• Writable Streams in Node.js
• Node.js Stream Pipeline API Documentation
• Node.js Streams series
• Node.js Crypto module
• Node.js zlib module

Disclaimer: This blog post used AI to generate the images used for the analogy.

Photo credit: pawel_czerwinski

]]>

Previously, we covered everything we need to know to understand Node.js streams, followed by a look into Readable streams.

In this post, we move onto Writable streams."

How to think about Writable streams

Previously, we started that a mental model for remembering Writable streams is to compare them to a water barrel (or a water bucket if that works better for you).

Here is a little more depth to the analogy and things that you need to know:

• Bucket (Stream Buffer): The bucket represents the internal buffer of a Writable stream. It has a finite capacity to hold water (data).
• Pouring Water (Writing Data): When you write data to a stream, it's like pouring water into the bucket. The write() method is analogous to pouring water.
• Bucket Capacity (highWaterMark): The bucket has a maximum capacity, similar to the highWaterMark in Writable streams. This is the point at which the bucket is considered "full".
• Overflow (Backpressure): If you try to pour more water when the bucket is already full, it will overflow. In streams, this triggers backpressure, and the write() method returns false.
• Draining the Bucket (Processing Data): As the stream processes data, it's like water draining from the bucket. When enough water has drained (data processed), the stream emits a 'drain' event.
• Continuous Flow (Flowing Mode): Imagine a system where water is constantly poured in and drained out. This is similar to a stream in flowing mode, where data is written and processed continuously.
• Pausing the Flow (Paused Mode): If you pause the water flow, it's like pausing a stream. The pause() method is used to pause the stream. This manual process is useful when you need to control the flow of data.
• Sealing the bucket (Ending the Stream): In a Writable stream, calling end() is like saying "This is the last bit of data I'm adding to the bucket, and once it's processed, consider the operation complete." It's more about finalizing the writing process than "closing a source".
• Bucket Emptied (Finish Event): When all the water has drained from the bucket, it's empty. This is analogous to the 'finish' event in a Writable stream, indicating all data has been processed.
• Leaks (Error Handling): This is more for a mental model, but if there's a leak in the bucket, it's like an error in the stream. Proper error handling is crucial in both cases.
• Custom Buckets (Custom Writable Streams): Just as you might have specialized buckets for different liquids, you can create custom Writable streams for specific data types or operations. This is done by extending a class from stream.Writable. See the section on stream.Writable class for more clarification.

Here is a lifecycle sequence diagram to help you visualize the analogy:

How to create Writable streams

The node:stream module provides a Writable class which you use to create Writable streams.

1. Extending the stream.Writable class
2. Creating an instance using the stream.Writable class
3. Using a factory method from stream.Writable

Extending the stream.Writable class

In this scenario, you create your own class and provide a _write method for handling the data processing.

const { Writable } = require("node:stream");
class MyWritable extends Writable {
  _write(chunk, encoding, callback) {
    // Implementation
  }
}

Creating an instance using the stream.Writable class

Alternative, we can just create an instance of a Writable stream directly.

This is similar to extending a class, but we make our implementation for processing data directly in the write method.

const { Writable } = require("node:stream");
const writable = new Writable({
  write(chunk, encoding, callback) {
    // Implementation
  },
});

For example, we could create a LogWriter class that logs the chunks as it processes data:

const { Writable } = require("node:stream");

class LogWriter extends Writable {
  _write(chunk, encoding, callback) {
    console.log(`[${new Date().toISOString()}] ${chunk.toString()}`);
    callback();
  }
}

const logger = new LogWriter();
logger.write("Application started");
logger.write("User logged in");
logger.end("Application closed");

Using a factory method from stream.Writable

const stream = require("node:stream");
const writable = stream.Writable({
  write(chunk, encoding, callback) {
    // Implementation
  },
});

Other examples of Writable streams

In addition to using the stream.Writable class, some Node.js modules provide several built-in Writable streams for common use cases:

4. fs.createWriteStream
5. http response objects
6. process.stdout and process.stderr
7. child_process stdin
8. Compression streams
9. Crypto streams
10. Database write streams (e.g. MongoDB)

fs.createWriteStream

With fs.createWriteStream, you can write data to a file using a Writable stream.

const fs = require("node:fs");
const writableStream = fs.createWriteStream("output.txt");

For example, we could create a log file:

const fs = require("node:fs");

const writableStream = fs.createWriteStream("access.log", { flags: "a" });

function logAccess(ip, route) {
  writableStream.write(`${new Date().toISOString()} ${ip} ${route}\n`);
}

logAccess("192.168.1.1", "/home");
logAccess("192.168.1.2", "/login");

http response objects

The res or "response" argument in an HTTP server callback is a Writable stream.

const http = require("node:http");
const server = http.createServer((req, res) => {
  // res is a writable stream
  res.write("Hello");
  res.end("World");
});

We could use it for streaming back simple responses to the client:

const http = require("node:http");

const server = http.createServer((req, res) => {
  res.writeHead(200, { "Content-Type": "text/html" });
  res.write("<html><body>");
  res.write("<h1>Welcome to my server!</h1>");
  res.write(`<p>You requested: ${req.url}</p>`);
  res.end("</body></html>");
});

server.listen(3000, () => console.log("Server running on port 3000"));

In the above example, we are streaming an HTML response to the client.

process.stdout and process.stderr

Both process.stdout and process.stderr are Writable streams that write to the console.

process.stdout.write("Log to console\n");
process.stderr.write("Log error to console\n");

child_process stdin

child_process provides a way to spawn child processes and interact with them using stdin, stdout and stderr.

const { spawn } = require("node:child_process");
const child = spawn("wc");
// child.stdin is a writable stream
child.stdin.write("Hello World");
child.stdin.end();

For example, we could use child_process to spawn a grep process that searches for the word "hello" in the input stream and logs the output to the console:

const { spawn } = require("node:child_process");

const child = spawn("grep", ["hello"]);

child.stdout.on("data", (data) => {
  console.log(`grep found: ${data}`);
});

child.stdin.write("hello world\n");
child.stdin.write("goodbye world\n");
child.stdin.end();

Compression streams

Although we haven't covered Transform streams in-depth yet, if you've followed along with the series so far then it will be come trivial to understand how they work.

In the case of zlib, we can create a gzip compression Transform stream that is both readable and writable:

const zlib = require("node:zlib");
const gzip = zlib.createGzip();
// gzip is a transform stream (both readable and writable) - we cover this more in a future post

An example of this in action where we compress a file using gzip compression and write it out to a new file:

const fs = require("node:fs");
const zlib = require("node:zlib");

const readStream = fs.createReadStream("input.txt");
const writeStream = fs.createWriteStream("input.txt.gz");
const gzip = zlib.createGzip();

readStream.pipe(gzip).pipe(writeStream);

writeStream.on("finish", () => console.log("File successfully compressed"));

Crypto streams

Similar to above, cipher in our below example is a Transform stream that is both readable and writable.

const crypto = require("crypto");
const cipher = crypto.createCipher("aes192", "secret");
// cipher is a transform stream (both readable and writable) - we cover this more in a future post

An example of this in action where we encrypt a file using AES-192-CBC encryption and write it out to a new file:

const crypto = require("node:crypto");
const fs = require("node:fs");

const algorithm = "aes-192-cbc";
const password = "Password used to generate key";

const key = crypto.scryptSync(password, "salt", 24);
const iv = Buffer.alloc(16, 0);

const cipher = crypto.createCipheriv(algorithm, key, iv);

const input = fs.createReadStream("input.txt");
const output = fs.createWriteStream("encrypted.txt");

input.pipe(cipher).pipe(output);

output.on("finish", () => console.log("File encrypted successfully"));

Database write streams

Finally, some databases provide Writable streams for writing data. For example, MongoDB provides a GridFSBucket class for working with large files.

const mongodb = require("mongodb");
const bucket = new mongodb.GridFSBucket(db);
const uploadStream = bucket.openUploadStream("file.txt");

Here is a more complete example:

const { MongoClient } = require("mongodb");
const fs = require("node:fs");

async function uploadFile(filePath, fileName) {
  const client = await MongoClient.connect("mongodb://localhost:27017");
  const db = client.db("myDatabase");
  const bucket = new MongoClient.GridFSBucket(db);

  const uploadStream = bucket.openUploadStream(fileName);
  const readStream = fs.createReadStream(filePath);

  readStream.pipe(uploadStream);

  return new Promise((resolve, reject) => {
    uploadStream.on("finish", resolve);
    uploadStream.on("error", reject);
  });
}

uploadFile("largefile.zip", "backup.zip")
  .then(() => console.log("File uploaded successfully"))
  .catch(console.error);

You can also write streams to remote sources like S3 buckets, etc.

Writable stream fundamentals in action

To cap off today's blog post, let's take a look at a Writable stream that covers the fundamentals that we have covered in the series previously (backpressure, buffering etc.).

const { Writable } = require("stream");
const fs = require("fs");

class SlowWriter extends Writable {
  constructor(options) {
    // 1. Stream types: This is a Writable stream
    // 7. Object Mode: We're enabling object mode
    super({ ...options, objectMode: true });

    // 2. Buffering: We're setting a custom highWaterMark
    this.highWaterMark = options.highWaterMark || 2;

    this.outputFile = fs.createWriteStream("output.txt", { flags: "a" });
  }

  _write(chunk, encoding, callback) {
    // Simulate slow writing
    setTimeout(() => {
      // 7. Object Mode: We can write objects directly
      const output = `${JSON.stringify(chunk)}\n`;

      // 3. Backpressure: This slow write operation may cause backpressure
      this.outputFile.write(output, (err) => {
        if (err) {
          callback(err);
        } else {
          console.log("Chunk written");
          callback();
        }
      });
    }, 1000);
  }

  _final(callback) {
    this.outputFile.end(callback);
  }
}

// Usage:
const slowWriter = new SlowWriter({ highWaterMark: 2 });

// 6. Events: Listening for events
slowWriter.on("finish", () => console.log("All writes are complete."));
slowWriter.on("drain", () =>
  console.log("Stream is no longer full, can resume writing.")
);

// 5. Piping: Creating a readable stream to pipe into our writable
const { Readable } = require("stream");
const fastReader = new Readable({
  objectMode: true,
  read() {},
});

// 4. Modes of Operation: This is in flowing mode due to the pipe
fastReader.pipe(slowWriter);

// Simulate fast data production
for (let i = 0; i < 10; i++) {
  // 3. Backpressure: This may trigger backpressure due to slow writing
  const canContinue = fastReader.push({ data: `Fast data ${i}` });
  if (!canContinue) {
    console.log("Backpressure detected, waiting for drain");
    // In a real scenario, you'd wait for the 'drain' event before continuing
  }
}

fastReader.push(null); // Signal the end of the readable stream

In the above example, we demonstrate:

1. Stream types: It creates a custom Writable stream (SlowWriter) and uses a Readable stream (fastReader).
2. Buffering: The highWaterMark is set to control the internal buffer size.
3. Backpressure: The slow writing operation simulated by setTimeout may cause backpressure, which is handled by the stream automatically.
4. Modes of Operation: The readable stream is put into flowing mode when piped to the writable stream.
5. Piping: The readable stream is piped to the writable stream, automatically managing data flow.
6. Events: The code listens for 'finish' and 'drain' events on the writable stream.
7. Object Mode: Both the readable and writable streams are set to object mode, allowing them to work with JavaScript objects.

For a visual guide on the above:

Conclusion

Understanding Writable streams in Node.js is crucial for managing data flow in your applications. By drawing parallels to everyday concepts like water barrels and buckets, we've explored how Writable streams handle data, manage backpressure, and process data efficiently. We've also demonstrated various ways to implement and utilize Writable streams, from logging and file writing to more advanced use cases like compression and encryption.

As you continue your journey with Node.js streams, remember that mastering these concepts can greatly enhance your ability to build robust, high-performance applications. Whether you're handling large file uploads, streaming data to a client, or processing logs, Writable streams offer a powerful toolset for efficient data handling. Keep experimenting with different stream types and explore the endless possibilities they provide.

Thank you for following along with this series on Node.js streams. Stay tuned for our next post, where we'll dive into Transform streams and further expand our understanding of Node.js stream fundamentals.

Resources and further reading

• Understanding Node.js Streams Blog Post
• Readable Streams in Node.js
• Node.js Stream Pipeline API Documentation
• Node.js Streams series
• Node.js Crypto module
• Node.js zlib module
• Node.js Blog | Writing files with Node.js

Disclaimer: This blog post used AI to generate the images used for the analogy.

Photo credit: robanderson72

]]>

In my previous post, we learned everything we need to know to understand Node.js streams. We covered the four fundamental stream types in Node.js: Writable, Readable, Duplex, and Transform. We also discussed the benefits of using streams, key concepts like buffering, backpressure, and piping, and the importance of events in stream processing.

In this post, we will dive deeper into Readable streams in particular: how they are created, how we can consume data from them, and some a number of examples to get you comfortable.

How to create Readable streams

In our previous post, we started with the mental model that Readable streams are like water taps. In the same way that we can turn on the tap, change the flow rate, or turn it off, a Readable stream provides data in chunks that we can consume, pause, resume, or end.

But how can we create a Readable stream in Node.js? There are a number of ways, but let's start with what the node:stream module provides.

1. stream.Readable.from()
2. stream.from()
3. stream.Readable class
4. stream.fromWeb()

From top-to-bottom, I've listed out what I think are the most approachable ways to create a Readable stream in Node.js from the node:stream module, so let's go through them each.

stream.Readable.from()

The stream.Readable.from() method creates a Readable stream from an iterable object or an async iterable object. This method is useful when you have an iterable data source, for example an array of strings, that you want to convert into a stream.

const { Readable } = require("stream");
const readableStream = Readable.from(["Hello", " ", "World", "!"]);

In the above example, we create a Readable stream from an array of strings. The Readable.from() method automatically converts the array into a stream that emits each string as a chunk of data.

For the sake of completion, here is a reminder of how that Readable stream could be paired with a Writable stream to pipe data from one to the other:

const { Readable, Writable } = require("node:stream");
const fs = require("node:fs");

// Create a Readable stream from an array of strings
const readableStream = Readable.from(["Hello", " ", "World", "!"]);

// Create a Writable stream (in this case, writing to a file)
const writableStream = fs.createWriteStream("output.txt");

// Pipe the Readable stream to the Writable stream
readableStream.pipe(writableStream);

// Handle the 'finish' event on the Writable stream
writableStream.on("finish", () => {
  console.log("Finished writing to file");
});

// Handle any errors
writableStream.on("error", (err) => {
  console.error("An error occurred:", err);
});

In the above scenario, our readableStream is piped to the writeableStream that writes to a file output.txt.

The array of ["Hello", " ", "World", "!"] here is considered an iterable object with four chunks. Each element of the array is treated as a separate chunk in the resulting stream.

The writeableStream consumes each chunk of data and writes it to the file output.txt. The finish event is emitted when all data has been written to the file, and the error event is emitted if an error occurs during the writing process.

To solidify this, here is a verbose sequence diagram:

stream.from()

This is function that can be imported directly from the node:stream module and is a shorthand for stream.Readable.from().

const { from, Readable } = require("node:stream");

// These two lines are equivalent:
const stream1 = from(["Hello", "World"]);
const stream2 = Readable.from(["Hello", "World"]);

stream.Readable()

The stream.Readable class is a base class for creating custom Readable streams. You can extend this class to create your own Readable streams by implementing the _read() method.

const { Readable } = require("stream");

class MyReadable extends Readable {
  constructor(options, data) {
    super(options);
    this.data = data;
  }

  _read() {
    if (this.data.length) {
      this.push(this.data.shift());
    } else {
      this.push(null);
    }
  }
}

const myReadable = new MyReadable(["Hello", " ", "World", "!"]);

In the above example, we create a custom Readable stream by extending the stream.Readable class. The _read() method is implemented to push data chunks from the data array to the stream. When all data has been pushed, null is pushed to signal the end of the stream.

stream.fromWeb()

stream.fromWeb() is designed to convert a Web API ReadableStream into a Node.js Readable stream. It's not always "from the web" in the sense of network requests; it's about compatibility between Web API streams and Node.js streams.

const { fromWeb } = require("node:stream/web");

// Create a Web API ReadableStream (this could come from a fetch() response, for example)
const webReadableStream = new ReadableStream({
  start(controller) {
    controller.enqueue("Hello");
    controller.enqueue("World");
    controller.close();
  },
});

// Convert it to a Node.js Readable stream
const nodeReadableStream = fromWeb(webReadableStream);

// Use the Node.js stream
nodeReadableStream.on("data", (chunk) => {
  console.log(chunk.toString());
});

nodeReadableStream.on("end", () => {
  console.log("Stream ended");
});

In the above example, we create a Web API ReadableStream that emits two chunks of data ("Hello" and "World") and then closes. We then convert this Web API stream to a Node.js Readable stream using stream.fromWeb().

For advanced more practical example using the web fetch API:

const { fromWeb } = require("node:stream/web");
const { createWriteStream } = require("node:fs");
const { pipeline } = require("node:stream/promises");

async function downloadFile(url, outputPath) {
  const response = await fetch(url);
  if (!response.ok) throw new Error(`HTTP error! status: ${response.status}`);

  // response.body is a Web API ReadableStream
  const nodeStream = fromWeb(response.body);

  // Create a writable stream to the output file
  const fileStream = createWriteStream(outputPath);

  // Use pipeline to pipe the data and handle cleanup
  await pipeline(nodeStream, fileStream);

  console.log(`File downloaded to ${outputPath}`);
}

downloadFile("https://example.com/somefile.txt", "output.txt").catch(
  console.error
);

In the above, we use the fetch API to download a file from a URL. The response body is a Web API ReadableStream, which we convert to a Node.js Readable stream using stream.fromWeb(). We then pipe this stream to a Writable stream that writes the data to a file.

Although we haven't touched on it, we make use of the pipeline API from node:stream in the above example.

A visual look at the download sequence:

Other ways to create a Readable stream

process.stdin

process.stdin provides a built-in Readable stream for standard input that you can hook into thanks streams implementing the EventEmitter class:

process.stdin.on("data", (chunk) => {
  console.log("Received input:", chunk.toString());
});

http.IncomingMessage

Another built-in Readable stream created when receiving an HTTP request from Node.js' node:http module:

const http = require("node:http");

const server = http.createServer((req, res) => {
  // req is an instance of http.IncomingMessage
  req.on("data", (chunk) => {
    console.log("Received data:", chunk.toString());
  });
});

server.listen(3000);

child_process.spawn()

child_process.spawn() creates a child process that provides Readable streams for standard output and standard error:

const { spawn } = require("node:child_process");

const child = spawn("ls", ["-l"]);
child.stdout.on("data", (data) => {
  console.log(`stdout: ${data}`);
});

Compression

Built-in modules like zlib provide Readable streams for compression and decompression:

const zlib = require("node:zlib");
const fs = require("node:fs");

const readStream = fs.createReadStream("file.txt");
const gunzip = zlib.createGunzip();
const uncompressedStream = readStream.pipe(gunzip);

Encryption

Finally for our last example, we can use the crypto module to create streams for cryptographic operations:

const crypto = require("node:crypto");

const key = crypto.randomBytes(32);
const iv = crypto.randomBytes(16);
const decipher = crypto.createDecipheriv("aes-256-cbc", key, iv);

// Use decipher as a Readable stream

For a more complete example for encryption:

const crypto = require("node:crypto");
const { pipeline } = require("node:stream/promises");
const fs = require("node:fs");

// Generate a random key and IV
const key = crypto.randomBytes(32);
const iv = crypto.randomBytes(16);

// Create some sample encrypted data
const algorithm = "aes-256-cbc";
const cipher = crypto.createCipheriv(algorithm, key, iv);
let encrypted = cipher.update("This is a secret message", "utf8", "hex");
encrypted += cipher.final("hex");

// Write the encrypted data to a file
fs.writeFileSync("encrypted.txt", encrypted);

// Create a decipher (Readable stream)
const decipher = crypto.createDecipheriv(algorithm, key, iv);

// Create a Readable stream from the encrypted file
const readStream = fs.createReadStream("encrypted.txt", { encoding: "hex" });

// Create a Writable stream for the decrypted output
const writeStream = fs.createWriteStream("decrypted.txt");

// Use the pipeline to decrypt the data
async function decryptFile() {
  try {
    await pipeline(readStream, decipher, writeStream);
    console.log("File successfully decrypted");
  } catch (err) {
    console.error("Pipeline failed", err);
  }
}

decryptFile();

This last example is a little tough to follow (I've synchronously written the encrypted data to a file), but here is a sequence diagram to help:

Readable stream fundamentals in action

We're going to cap off this blog post by recapping some of the core stream concepts we covered in the previous post Understanding Node.js Streams.

The following will be a quick-fire round of examples to help you get comfortable with the concepts we have covered.

const { Readable } = require("node:stream");

// Create a custom Readable stream
class NumberStream extends Readable {
  constructor(max, options) {
    super(options);
    this.max = max;
    this.current = 1;
  }

  _read() {
    if (this.current <= this.max) {
      const data = { number: this.current };
      this.push(JSON.stringify(data));
      this.current++;
    } else {
      this.push(null); // Signal the end of the stream
    }
  }
}

// Create instances of our custom stream
const numberStream = new NumberStream(5, {
  objectMode: false,
  highWaterMark: 64, // Buffer size
});
const objectStream = new NumberStream(5, {
  objectMode: true,
  highWaterMark: 2, // Buffer size
});

// Example 1: Using events (paused mode)
numberStream.on("readable", () => {
  let chunk;
  while (null !== (chunk = numberStream.read())) {
    console.log("Received chunk:", chunk.toString());
  }
});

numberStream.on("end", () => {
  console.log("Stream ended");
});

// Example 2: Using flowing mode
objectStream.on("data", (chunk) => {
  console.log("Received object:", chunk);
});

objectStream.on("end", () => {
  console.log("Object stream ended");
});

// Example 3: Demonstrating backpressure
const slowConsumer = new Readable({
  read() {},
});

slowConsumer.on("data", (chunk) => {
  console.log("Slow consumer received:", chunk.toString());
});

const fastProducer = new Readable({
  read() {
    for (let i = 0; i < 1000; i++) {
      if (!this.push(`Data ${i}\n`)) {
        console.log("Backpressure applied");
        return;
      }
    }
    this.push(null);
  },
});

// Piping with backpressure handling
fastProducer.pipe(slowConsumer);

In the above example, we cover the following concepts:

1. Buffering: The highWaterMark option is set in the NumberStream constructor, controlling the internal buffer size.

2. Backpressure: In Example 3, we demonstrate backpressure between a fast producer and a slow consumer.

3. Modes of Operation:

   • Example 1 uses the paused mode, manually calling read().
   • Example 2 uses the flowing mode with the 'data' event.

4. Piping: Example 3 uses pipe() to connect the fast producer to the slow consumer.

5. Events: We use readable, data, and end events to interact with the streams.

6. Object Mode: The objectStream is created with objectMode: true, allowing it to emit JavaScript objects.

On top of this:

• The NumberStream class demonstrates how to create a custom Readable stream.
• The _read method is called internally by the stream to generate data.
• In object mode, we directly push JavaScript objects. In non-object mode, we stringify the objects.
• The backpressure example shows how a fast producer will pause when the consumer's buffer fills up.
• Piping automatically handles backpressure and data flow between streams.

Conclusion

Readable streams in Node.js offer a powerful and flexible way to handle data in chunks, making it easier to process large datasets efficiently. By understanding the various methods to create readable streams, such as stream.Readable.from(), stream.from(), stream.Readable, and stream.fromWeb(), you can leverage these tools to build robust data processing applications.

We also explored several practical examples, including piping streams, handling HTTP requests, working with child processes, and performing compression and encryption. Each of these examples highlights the versatility of readable streams and their applicability in different scenarios.

As you continue to work with Node.js streams, remember the key concepts of buffering, backpressure, and the importance of event handling to manage your streams effectively. It all takes practice.

For further reading and more advanced use cases, be sure to check out the official Node.js documentation and other related resources. We'll be looking into Writeable streams next time!

Resources and further reading

• Understanding Node.js Streams Blog Post
• Node.js Stream Pipeline API Documentation
• Node.js Streams series

Disclaimer: This blog post used AI to generate the images used for the analogy.

Photo credit: terminath0r

]]>

Streams can be a less approachable topic for those working with Node.js.

Within Node.js, there are four fundamental stream types that exist:

1. Writeable: Streams that you can write data to.
2. Readable: Streams that you can read data from.
3. Duplex: Streams that are both readable and writeable.
4. Transform: A type of Duplex stream that can modify or transform the data as it is written and read.

Over the course of this post, we will working through the fundamentals that you should understand about Node.js streams and why they exist.

In this post, we will not deep dive into each of the four stream types. Even though I will be introducing stream code in this blog post, we specifically will focus on the foundational concepts that you should understand about streams. In the posts following, will consolidate our knowledge by diving deeper into each stream type.

How to think about the four fundamental Node.js Stream types

To help you understand the four stream types, I will introduce four analogies to think about each stream type orientated around water so that you connect the dots between the four types, but I will also add an alternative analogy to help consolidate your mental model of the four fundamental stream types. These aim to help bridge the gap between the abstract concept of streams and something more tangible.

1. Writable Streams: The Water Bucket, The Notebook

Writable streams are like water buckets that collect data (water). Just as a bucket can be filled with water from various sources, a writable stream can receive data from different inputs. You can pour data into it chunk by chunk, and it will hold that data until it's full or you're done writing. This makes writable streams perfect for scenarios where you need to collect and store data, such as writing to files or sending responses in a web server.

Alternatively, you could think of Writable streams are like a notebook where you can write information. Just as you can open a notebook and jot down thoughts, ideas, or data page by page, a writable stream allows you to input data chunk by chunk. You control when to write, how much to write, and when you're finished.

Examples of Writable streams include:

• File Writing: Writing data to a file using fs.createWriteStream().
• HTTP Responses: Sending responses to clients in a web server.
• Database Inserts: Streaming large amounts of data into a database.
• Console Output: Writing to process.stdout for command-line applications.

A simple example of creating a writable stream to a file output.txt in Node.js and writing in the text Hello, World!:

const fs = require("node:fs");
const writeStream = fs.createWriteStream("output.txt");
writeStream.write("Hello, World!");
writeStream.end();

2. Readable Streams: The Tap, The Fantasy Novel

Readable streams can be thought of as water taps. Like a tap that can be turned on to release a steady flow of water, a readable stream provides a controlled source of data. You can "open" the stream to start receiving data, control its flow, and "close" it when you're done. This makes readable streams ideal for scenarios where you have a source of data that you want to access and process at your own pace, such as reading from a file or receiving a response from an HTTP request. Just as you can adjust a tap to control the water flow, you can also manage how you consume data from a readable stream.

You could also think about a fantasy novel (or in fact, any published book that you read). As you flip through the pages, you're consuming the story bit by bit, much like how a readable stream provides data in chunks. You can pause your reading (pause the stream), continue where you left off (resume the stream), or decide you're done reading for now (end the stream).

Examples of Readable streams include:

• File Reading: Reading data from a file using fs.createReadStream().
• HTTP Requests: Receiving data from client requests in a web server.
• Database Queries: Streaming large result sets from a database.
• User Input: Reading from process.stdin for command-line interfaces.

An example of creating a readable stream from a file input.txt in Node.js:

const fs = require("node:fs");
const readStream = fs.createReadStream("input.txt");
readStream.on("data", (chunk) => {
  console.log(`Received ${chunk.length} bytes of data.`);
});

3. Duplex Streams: The Two-Way Water Pipe, The Walkie-Talkie

Duplex streams are like two-way water pipes. Imagine a pipe system where water can flow in both directions simultaneously. One end of the pipe can receive water (write data), while the other end can release water (read data). This bidirectional flow makes duplex streams perfect for scenarios where you need to both send and receive data, such as in network sockets or inter-process communication.

Admittedly for me, Duplex streams were the ones that took me more time to understand. My alternative is that Duplex streams are akin to a walkie-talkie. With a walkie-talkie, you can both speak (write data) and listen (read data) through the same device. Similarly, a duplex stream can both accept input and provide output, making it perfect for two-way communication scenarios.

Examples of Duplex streams include:

• TCP Sockets: Implementing network communication where data flows both ways.
• SSH Tunnels: Establishing secure, bidirectional communication channels.
• Websockets: Creating real-time, full-duplex communication in web applications.
• Inter-process Communication: Facilitating two-way communication between processes.

An example of creating a duplex stream using a TCP server in Node.js by creating a server that listens on port 8080:

const net = require("node:net");
const server = net.createServer((socket) => {
  socket.write("Server: Hello!");
  socket.on("data", (data) => {
    console.log("Received:", data.toString());
  });
});
server.listen(8080);

In the above code, we have a socket variable that is a duplex stream. We write to the socket using socket.write() and read from the socket using socket.on("data", ...).

4. Transform Streams: The Water Treatment Plant, The Factory Assembly Line

Transform streams you can think of as water treatment plants. Just as a treatment plant takes in raw water, processes it, and outputs clean water, a transform stream receives input data, manipulates or "transforms" it in some way, and then outputs the modified data. This makes transform streams excellent for scenarios where you need to modify data on the fly, such as compression, encryption, or any kind of data parsing or conversion.

Transform streams can be likened to a factory assembly line. In a factory, raw materials enter at one end, undergo various modifications or enhancements as they move along the line, and emerge as finished products at the other end. Similarly, a transform stream takes input data, processes or modifies it in some way, and outputs the transformed data.

Examples of Transform streams include:

• Data Compression: Compressing or decompressing data on-the-fly.
• Encryption/Decryption: Securing data as it's being transmitted.
• Data Parsing: Converting between data formats (e.g., CSV to JSON).
• Text Encoding: Changing character encodings of text data.

Here is a simple example of a transform stream that converts input text to uppercase:

const { Transform } = require("node:stream");
const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  },
});

process.stdin.pipe(upperCaseTransform).pipe(process.stdout);

We could run the following in the terminal to see it in action:

$ echo "Hello" | node basic-transform.js
HELLO

Understanding the difference between Duplex and Transform streams

Transform streams are actually a special type of duplex stream. They can be thought of as duplex streams with a built-in relationship between the input and output. Using out analogy of the factory line or the water treatment plant, we understand that the input and output are related in some way. For Transform streams, data that comes in can be changed in some way before it goes out.

On the other hand, Duplex streams are more general. They can be used for two-way communication, but the input and output are not necessarily related. You can write data in one format and read it in another, or vice versa. In our example of the walkie talkie, what you talk about and what you hear are not necessarily directly related.

The "why" behind Node.js Streams

Node.js streams offer several benefits that make them a powerful tool for data processing and manipulation. Here are some key advantages of using streams in Node.js applications:

1. Memory Efficiency

   • Process large amounts of data without loading it all into memory at once.
   • Ideal for working with files or datasets larger than available Random Access Memory (RAM).

2. Time Efficiency

   • Start processing data as soon as it's available, without waiting for the entire dataset.
   • Reduces overall processing time for large datasets.

3. Composability

   • Easily chain multiple stream operations together using .pipe().
   • Create complex data processing pipelines with modular, reusable components.

4. Backpressure Handling

   • Built-in mechanisms to manage the flow of data between fast producers and slow consumers.
   • Prevents memory overflow and system crashes.

5. Abstraction of Data Sources and Destinations

   • Consistent interface for various types of input/output (files, network, etc.).
   • Write code that works with different data sources without major changes.

6. Event-Driven Architecture

   • Aligns well with Node.js's event-driven, non-blocking I/O model.
   • Enables responsive applications even when dealing with large datasets.

7. Built-in to Core Modules

   • Native support in many Node.js core modules (fs, http, crypto, etc.).
   • Seamless integration with Node.js ecosystem.

8. Asynchronous Processing

   • Natural fit for asynchronous operations, allowing for efficient multitasking.
   • Keeps the event loop free for other operations.

9. Transformations

   • Easy to implement data transformations on-the-fly.
   • Supports operations like compression, encryption, or format conversion as data flows.

10. Improved Performance in Network Operations

    • Efficient for sending and receiving data over networks.
    • Reduces latency in client-server communications.

11. Customizability

    • Can create custom streams for specific use cases.
    • Extend built-in stream classes for tailored functionality.

12. Interoperability

    • Works well with other Node.js features and third-party libraries.
    • Many npm packages provide stream interfaces for various data sources and sinks.

13. Lazy Evaluation

    • Process data only when it's needed, improving application responsiveness.
    • Useful for scenarios where not all data may be used.

14. Standard Interface

    • Provides a consistent API across different types of streams.
    • Reduces the learning curve for working with various data sources and sinks.

These benefits make Node.js streams a powerful tool for efficient, scalable, and flexible data handling in a wide range of applications.

Node.js Streams concepts that you should know

On top of the four fundamental stream types, it is important to keep in mind a few foundational key terms that relate to streams:

1. Stream types: The already-mentioned four foundational stream types (Writeable, Readable, Duplex, and Transform).
2. Buffering: Temporary data storage within streams, controlled by highWaterMark to manage memory usage.
3. Backpressure: Mechanism to balance data flow between fast producers and slow consumers, preventing memory issues.
4. Modes of Operation: Flowing (automatic data reading) vs Paused (manual data reading) modes for controlling data consumption.
5. Piping: Connecting streams to automatically manage data flow and backpressure between them.
6. Events: Asynchronous notifications for stream states and data availability, crucial for stream interaction.
7. Object Mode: An option with streams that enable the capability to process JavaScript objects instead of just buffers or strings.

Understanding these concepts provides a solid foundation for working with Node.js streams effectively. It helps in making informed decisions about stream usage, handling data efficiently, and avoiding common pitfalls in stream-based applications.

Let's walk through each of these concepts in more detail.

Buffering

Buffering in Node.js streams refers to the process of storing chunks of data in memory while they are being transferred between the source and the destination.

You can think of it like a queue of people waiting to enter a room. The room can only hold a certain number of people at a time, so the manager at the front of the queue (buffer) helps manage the flow of people in and out of the room.

When data is read from a stream, it is held temporarily in a buffer until it can be processed or written to another stream. It is a mechanism used by streams to manage data flow efficiently.

We can help example buffering by creating a writable stream with a custom highWaterMark.

const { Writable } = require("node:stream");

const myWritable = new Writable({
  highWaterMark: 1024, // Set buffer size to 1KB
  write(chunk, encoding, callback) {
    console.log("Writing:", chunk.toString());
    callback();
  },
});

1. Create a writable stream with a 1KB buffer.
2. Write data to the stream.
3. If more than 1KB of data is written before it's processed, the write method will return false, indicating the buffer is full.

Here is a more detailed example:

const { Writable } = require("node:stream");

const myWritable = new Writable({
  highWaterMark: 1024, // Set buffer size to 1KB
  write(chunk, encoding, callback) {
    console.log("Writing:", chunk.toString());
    // Simulate processing delay
    setTimeout(() => {
      callback();
    }, 100);
  },
});

// Write data to the writable stream
const data = Buffer.from("This is some data to write to the stream.");

for (let i = 0; i < 10; i++) {
  const canWrite = myWritable.write(data);

  if (!canWrite) {
    console.log("Buffer is full, applying backpressure.");
  } else {
    console.log("Data written to buffer.");
  }
}

// End the writable stream
myWritable.end();

In the above examples, when you call the write method of the writable stream, the data is added to the internal buffer. If the buffer is not full, the write method will immediately process the chunk. If the buffer is full, the stream will handle the backpressure by signaling the source to slow down or stop sending data until there's enough space in the buffer.

Setting the highWaterMark value does not reject chunks smaller than the specified buffer size. Instead, it determines the maximum amount of data that the internal buffer can hold before applying backpressure. I sometimes think of the highWaterMark as a set of go/stop traffic lights.

Backpressure

We touched briefly above on backpressure, but what is it exactly?

Backpressure is a flow control mechanism. It is used in streams to manage the rate at which data is transmitted from the **source (producer) **to the destination (consumer).

Let's continue with our analogy of the queue of people wanting to enter the room. When the room reaches its capacity (buffer is full), the manager stops allowing more people to enter. This is akin to the writable stream applying backpressure because it cannot handle more data at the moment.

It prevents the destination from being overwhelmed by data when it cannot process incoming data as quickly as it is being produced. In the context of Node.js streams, backpressure helps ensure efficient and stable data transfer, preventing memory overflow and maintaining application performance.

For another example, here demonstrate handling backpressure in a pipe operation:

const fs = require("node:fs");

// Create a readable stream from a large file
const readStream = fs.createReadStream("bigfile.txt");

// Create a writable stream to an output file
const writeStream = fs.createWriteStream("output.txt");

// Pipe the readable stream into the writable stream
readStream.pipe(writeStream);

// Listen for the 'drain' event on the writable stream
writeStream.on("drain", () => {
  // This event is emitted when the writable stream buffer is drained
  console.log("Drained. Resuming read.");
  // Resume the readable stream to continue reading data
  readStream.resume();
});

1. Create read and write streams.
2. Pipe data from read to write stream.
3. If writeStream's buffer fills up, it emits 'drain' event.
4. When 'drain' occurs, resume reading to manage backpressure.

Backpressure is managed implicitly by the pipe method, which connects the readable stream (readStream) to the writable stream (writeStream). The pipe method automatically handles the flow control, including backpressure, so you don't need to manually manage it.

When the drain event is emitted, the writeStream signals that it has processed enough data to make room in its buffer. The readStream then resumes reading data and continues the flow.

The two reading modes: Flowing and Paused

Node.js streams can operate in two reading modes: flowing and paused. These modes are separate from object mode (that we will touch on later) and determines how data is read from a readable stream.

1. In flowing mode, data is read automatically from the stream and emitted as soon as it's available. This mode is suitable for scenarios where you want to consume data as it arrives, such as real-time data processing or streaming media.
2. In paused mode, data is read manually by calling the read() method. This mode is useful when you need more control over data consumption, such as processing data in batches or pausing and resuming data flow.

In the following example, we switch between flowing and paused modes.

const fs = require("node:fs");

const readStream = fs.createReadStream("sample.txt");

readStream.on("data", (chunk) => {
  console.log(`Received ${chunk.length} bytes`);
  readStream.pause(); // Switch to paused mode

  setTimeout(() => {
    readStream.resume(); // Switch back to flowing mode
  }, 1000);
});

1. Create a readable stream in flowing mode.
2. Receive data chunks automatically.
3. Pause the stream, switching to paused mode.
4. After a delay, resume the stream, switching back to flowing mode.

Here is a more specific example of flowing mode:

const fs = require("node:fs");

const readStream = fs.createReadStream("bigfile.txt");
const writeStream = fs.createWriteStream("output.txt");

// Pipe the readable stream to the writable stream
readStream.pipe(writeStream);

readStream.on("data", (chunk) => {
  console.log("Processing data chunk:", chunk.length);
});

While here is a more specific example of paused mode:

const fs = require("node:fs");

const readStream = fs.createReadStream("bigfile.txt");

// Start in paused mode (default)
readStream.on("readable", () => {
  let chunk;
  while (null !== (chunk = readStream.read())) {
    console.log("Processing data chunk:", chunk.length);
    // Simulate processing delay
    setTimeout(() => {
      console.log("Finished processing chunk");
    }, 1000);
  }
});

In this scenario, we start in paused mode by listening for the readable event. We then manually read data chunks using the read() method and process them one by one.

Piping

Piping in Node.js streams refers to the process of connecting a readable stream to a writable stream, allowing data to flow automatically from the source (readable stream) to the destination (writable stream). This mechanism simplifies data transfer and handling, making it easy to implement common data processing tasks, such as file copying, HTTP request handling, and more.

const fs = require("node:fs");
const { Transform } = require("node:stream");

const readStream = fs.createReadStream("input.txt");
const writeStream = fs.createWriteStream("output.txt");

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  },
});

readStream.pipe(upperCaseTransform).pipe(writeStream);

1. Create read, transform, and write streams.
2. Pipe data from read stream through transform stream to write stream.
3. Data flows automatically, with backpressure handled by Node.js.

A pipeline can be created by chaining multiple streams together using the .pipe() method or using the pipeline function from the node:stream module. This allows you to create complex data processing workflows with minimal code and effort.

Here is an example using the .pipe() method to create a pipeline:

const fs = require("node:fs");
const zlib = require("node:zlib");
const { Transform } = require("node:stream");

// Create a transform stream to convert data to uppercase
const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  },
});

// Create a readable stream from a source file
const readStream = fs.createReadStream("source.txt");

// Create a writable stream to a destination file
const writeStream = fs.createWriteStream("destination.txt.gz");

// Create a gzip transform stream
const gzip = zlib.createGzip();

// Pipe the streams together: read -> transform -> gzip -> write
readStream
  .pipe(upperCaseTransform) // Transform to uppercase
  .pipe(gzip) // Compress with gzip
  .pipe(writeStream); // Write to the destination file

// Handle events
writeStream.on("finish", () => {
  console.log(
    "File has been copied, transformed, and compressed successfully."
  );
});

writeStream.on("error", (err) => {
  console.error("Error writing to file:", err);
});

readStream.on("error", (err) => {
  console.error("Error reading from file:", err);
});

Here is the alternative using pipeline:

const { pipeline } = require("node:stream");
const fs = require("node:fs");
const zlib = require("node:zlib");
const { Transform } = require("node:stream");

// Create a transform stream to convert data to uppercase
const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  },
});

// Create streams
const readStream = fs.createReadStream("source.txt");
const writeStream = fs.createWriteStream("destination.txt.gz");
const gzip = zlib.createGzip();

// Use pipeline to handle the streams
pipeline(
  readStream, // Read from the source file
  upperCaseTransform, // Transform to uppercase
  gzip, // Compress with gzip
  writeStream, // Write to the destination file
  (err) => {
    if (err) {
      console.error("Pipeline failed:", err);
    } else {
      console.log("Pipeline succeeded.");
    }
  }
);

Events

Events refer to the various events that streams can emit during their lifecycle. These events allow you to handle different stages and aspects of data processing, such as when data is available, when the stream ends, or when an error occurs.

Under the hood, streams in Node.js can emit events because they are instances of the EventEmitter class (which I may retroactively touch on more in depth in a future blog post).

Different types of streams emit different events, but we will highlight some of the more common events that you can listen for in Node.js streams.

Readable

1. data: Emitted when a chunk of data is available to be read.
2. end: Emitted when there is no more data to be read from the stream.
3. error: Emitted when an error occurs while reading data.
4. readable: Emitted when the stream has data available to be read. This event is used in paused mode to indicate that you can call read() to get more data.

Writeable

1. drain: Emitted when the writable stream's internal buffer is emptied and it is ready to receive more data. This is important for handling backpressure.
2. finish: Emitted when the end() method is called and all data has been flushed to the underlying system.
3. error: Emitted when an error occurs while writing data.

Stream Events in action

The following is an simple example of hooking to events on a readable stream.

const fs = require("node:fs");

const readStream = fs.createReadStream("events.txt");

readStream.on("open", () => console.log("Stream opened"));
readStream.on("data", (chunk) => console.log(`Received ${chunk.length} bytes`));
readStream.on("end", () => console.log("Stream ended"));
readStream.on("close", () => console.log("Stream closed"));
readStream.on("error", (err) => console.error("Error:", err));

1. Create a readable stream.
2. Set up event listeners for various stream states.
3. Events fire asynchronously as the stream operates.
4. 'open' event when file is opened, 'data' for each chunk, 'end' when all data is read, 'close' when stream closes.

For a more complex example demonstrating more event types, you can refer to the following:

const fs = require("node:fs");
const zlib = require("node:zlib");
const { Transform } = require("node:stream");

// Create a transform stream to convert data to uppercase
const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  },
});

const readStream = fs.createReadStream("source.txt");
const writeStream = fs.createWriteStream("destination.txt.gz");
const gzip = zlib.createGzip();

// Pipe the streams together
readStream.pipe(upperCaseTransform).pipe(gzip).pipe(writeStream);

// Handle readable stream events
readStream.on("data", (chunk) => {
  console.log("Reading data:", chunk.length);
});

readStream.on("end", () => {
  console.log("Finished reading data.");
});

readStream.on("error", (err) => {
  console.error("Read error:", err);
});

// Handle writable stream events
writeStream.on("finish", () => {
  console.log("Finished writing data.");
});

writeStream.on("drain", () => {
  console.log("Writable stream drained.");
});

writeStream.on("error", (err) => {
  console.error("Write error:", err);
});

// Handle transform stream events
upperCaseTransform.on("error", (err) => {
  console.error("Transform error:", err);
});

gzip.on("error", (err) => {
  console.error("Gzip error:", err);
});

The above hooks into some events for the different streams in the pipeline. This allows you to monitor the progress of the data processing, handle errors, and respond to various stream states.

Object Mode

All streams created by Node.js APIs operate exclusively with strings or Buffer, TypedArray or DataView objects.

Object mode however enables streams to work with other types of JavaScript object values.

In the following, we set objectMode to true in order to process JavaScript objects.

const { Transform } = require("node:stream");

const objectModeTransform = new Transform({
  objectMode: true,
  transform(chunk, encoding, callback) {
    this.push({ length: chunk.length, content: chunk.toString() });
    callback();
  },
});

process.stdin.pipe(objectModeTransform).on("data", (obj) => {
  console.log("Received object:", obj);
});

1. Create a transform stream in object mode.
2. Pipe stdin (buffer mode) to the transform stream.
3. Transform converts Buffer chunks to JavaScript objects.
4. Output stream receives and logs JavaScript objects.

There are trade-offs for using objectMode as it can impact performance overhead, backpressure handling, buffering constraints and serialization requirements.

Here is another example to more clearly demonstrate the use of objectMode between a readable and writeable stream:

const { Readable, Writable } = require("node:stream");

// Create a readable stream in objectMode
const readableStream = new Readable({
  objectMode: true,
  read() {
    this.push({ id: 1, name: "Alice" });
    this.push({ id: 2, name: "Bob" });
    this.push(null); // No more data
  },
});

// Create a writable stream in objectMode
const writableStream = new Writable({
  objectMode: true,
  write(chunk, encoding, callback) {
    console.log("Received object:", chunk);
    callback();
  },
});

// Pipe the readable stream to the writable stream
readableStream.pipe(writableStream);

1. Create a readable stream that emits JavaScript objects.
2. Create a writable stream that receives and logs objects.
3. Pipe the readable stream to the writable stream.

Conclusion

Understanding Node.js streams is crucial for efficient data handling and manipulation in various applications. This blog post introduced the four fundamental types of streams, Writable, Readable, Duplex, and Transform, using relatable analogies to bridge the gap between abstract concepts and practical implementation.

Streams offer numerous benefits, including memory and time efficiency, composability, backpressure handling, and seamless integration with Node.js's event-driven architecture. They enable the creation of powerful, scalable, and flexible data processing pipelines.

By grasping the foundational concepts of streams, such as buffering, backpressure, piping, and modes of operation, you can unlock the full potential of Node.js for handling large datasets, network communication, and real-time data processing.

With this solid foundation, you're now equipped to explore the more complex aspects of Node.js streams and leverage their capabilities to build robust and performant applications.

Future posts will delve deeper into each stream type, providing detailed insights and advanced techniques to further enhance your Node.js applications.

Resources and further reading

• Node.js | Stream | Four fundamental stream types
• Node.js | Stream | Buffering in Node.js streams
• Node.js | Asynchronous work | EventEmitter class
• Node.js | Stream | Two reading modes
• Node.js | Stream | Object mode
• Node.js | Stream | Types of streams
• Node.js homepage

Disclaimer: This blog post used AI to generate the images used for the analogy.

Photo credit: amadejtauses

]]>

Today's post will walk through Node.js' Buffer object. This will be a prelude to the series on demystifying Node.js streams.

What are Node.js Buffer objects?

A Buffer in Node.js is a temporary storage area in memory used to hold raw binary data. It is a fixed-length sequence of bytes, similar to an array of integers, but corresponds to a raw memory allocation outside the V8 JavaScript engine.

Buffers are essential for handling raw binary data efficiently in Node.js. They serve as the backbone for many operations involving file I/O, networking, and streams. Understanding Buffers is key to mastering Node.js streams and dealing with data at a low level.

What does it mean to be allocated outside of the V8 engine?

V8 is the JavaScript engine used by Node.js. It is responsible for executing JavaScript code and managing memory allocation for JavaScript objects, such as strings, arrays, and objects. This memory is managed within a structure called the heap, which the V8 engine controls.

Buffers, on the other hand, are allocated in** Node.js's C++ layer**, which interfaces with the operating system directly. They are designed to handle binary data and are more efficient for certain operations, such as reading files or handling network protocols. To achieve this efficiency, Buffers allocate memory outside the V8 heap.

This means:

1. Direct Memory Access: Buffers provide direct access to memory outside of V8's managed heap, allowing for faster and more efficient handling of raw binary data. This is especially useful for operations that require manipulation of large amounts of data, such as file I/O and networking.
2. Fixed-Size Allocation: When a Buffer is created (as you will see in another section), it allocates a fixed-size block of memory. This block is not subject to the garbage collection process that manages the rest of the JavaScript objects in V8. As a result, Buffers can avoid the overhead associated with garbage collection, leading to better performance in memory-intensive operations.
3. Native Code Interoperability: Allocating memory outside the V8 heap allows Node.js to interact more easily with native code (C/C++ libraries) and system-level resources, which often require access to raw binary data.

When you create a Buffer, the Node.js runtime allocates a block of memory from the system's memory pool. This is done using methods provided by the operating system, such as malloc in C. The allocated memory is then managed by Node.js but remains outside the control of the V8 engine.

How should I think of a Buffer?

To mentally visualize a Buffer, you can think of a water bucket outside of a house.

The house itself represents the V8 heap, where JavaScript objects like strings, arrays, and objects live.

The bucket outside the house represents the Buffer. It is a separate container for raw binary data that is not subject to the same rules and restrictions as the objects inside the house.

As for using the bucket (Buffer), you can think of a hose (data source) that can fill the bucket (Buffer) with water (data).

To start connecting the dots, let's start walking through some examples of what we can do with Buffers and round out our analogy.

Working with Buffers

To create a Buffer, you can use the Buffer class provided by Node.js. The buffer documentation recommends explicitly importing the Buffer class, although it is available in the global scope.

const { Buffer } = require("node:buffer");

// Create a Buffer of 8 bytes
const buf1 = Buffer.alloc(8); // <Buffer 00 00 00 00 00 00 00 00>

// Create a Buffer from a string
const buf2 = Buffer.from("Hello, World!"); // <Buffer 48 65 6c 6c 6f 2c 20 57 6f 72 6c 64 21>

// Create a Buffer from an array of integers
const buf3 = Buffer.from([1, 2, 3, 4, 5]); // <Buffer 01 02 03 04 05>

In the above example, if is buf1 that we can think of as our empty bucket without water, while buf2 and buf3 are our buckets that are already filled with water.

In order to fill buf1, we could also make use of the write method:

const buf1 = Buffer.alloc(8); // <Buffer 00 00 00 00 00 00 00 00>

buf1.write("Hello!!!"); // <Buffer 48 65 6c 6c 6f 21 21 21>

Once a buffer is also created, we can replace "part of the water" directly:

buf1[7] = 0x65; // <Buffer 48 65 6c 6c 6f 21 21 65>

buf1.toString(); // "Hello!e"

Buffer and V8 heap interaction

Note that every time we convert the Buffer back into a JavaScript object data type supported by V8, we are effectively bring it back into the house (V8 heap) which is managed by the V8 JavaScript engine.

This process of conversion has some important implications:

1. Performance: Converting large Buffers to strings or other V8 types can be computationally expensive and may cause memory pressure on V8's heap.
2. Memory usage: The converted data now exists in two places - the original Buffer (outside V8) and the new string or object (inside V8).
3. Garbage collection: While the Buffer itself is managed outside of V8, any strings or objects created from it are subject to V8's garbage collection.
4. Immutability: When you convert a Buffer to a string, you get an immutable JavaScript string. Any changes you make to this string will create a new string, not modify the original Buffer.

To demonstrate, consider the following:

const buf = Buffer.from("Hello, World!");
// buf is allocated outside V8's heap

const str = buf.toString();
// str is a new string in V8's heap

str[0] = "h"; // This doesn't modify str or buf
console.log(str); // Still "Hello, World!"
const str2 = buf.toString(); // Still "Hello, World!" but as a new string in V8's heap

buf[0] = 0x68; // This modifies the Buffer directly
const str3 = buf.toString(); // Now "hello, World!" as a new string in V8's heap

While looking at the above, I believe it is important to keep comparing what is happening to the analogy of the bucket outside the house.

Hexadecimal representation

For all of the code example above, I have added the hexadecimal representation of the Buffer in the comments.

For those unfamiliar with hexadecimal representation, each pair of characters in this representation corresponds to a single byte, represented in hexadecimal (base-16) notation.

In hexadecimal, each digit can be 0-9 or A-F, where A-F represent the decimal values 10-15 respectively.

For the byte values in my buf1 example, we have 48 65 6c 6c 6f 21 21 21 where each of these is a byte value in hex.

These hex values in this example correspond to ASCII characters:

• 48 -> 'H'
• 65 -> 'e'
• 6c -> 'l'
• 6c -> 'l'
• 6f -> 'o'
• 21 -> '!'
• 21 -> '!'
• 21 -> '!'

If we convert this buffer back into a string, it would return to again be "Hello!!!":

const { Buffer } = require("node:buffer");

// Create a Buffer of 8 bytes
const buf1 = Buffer.alloc(8); // <Buffer 00 00 00 00 00 00 00 00>

buf1.write("Hello!!!"); // <Buffer 48 65 6c 6c 6f 21 21 21>
buf1.toString(); // "Hello!!!"

In fact, instead of using the string "Hello!!!" as an argument to write after allocating a buffer, we could even pass in hexadecimal values directly:

const buf1 = Buffer.from([0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x21, 0x21, 0x21]);
console.log(buf1.toString()); // Outputs: Hello!!!

const buf2 = Buffer.from("48656c6c6f212121", "hex");
console.log(buf2.toString()); // Outputs: Hello!!!

const buf3 = Buffer.alloc(8);
[0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x21, 0x21, 0x21].forEach((byte, index) => {
  buf3.writeUInt8(byte, index);
});
console.log(buf3.toString()); // Outputs: Hello!!!

const buf4 = Buffer.alloc(8);
buf4.write("\x48\x65\x6c\x6c\x6f\x21\x21\x21", "hex");
console.log(buf4.toString()); // Outputs: Hello!!!

Note that hexadecimal representation in JavaScript can be denoted by a leading 0x.

Common operations with Buffers

Methods for creating Buffers

Method	Description
Buffer.alloc(size)	Create a new Buffer of specified size
Buffer.from(array)	Create a Buffer from an array of bytes
Buffer.from(string, encoding)	Create a Buffer from a string

Methods for reading and writing from Buffers

Method	Description
buf.toString(encoding, start, end)	Convert Buffer to string
buf.write(string, offset, length, encoding)	Write a string to the Buffer
buf.readUInt8(offset), buf.readUInt16LE(offset), etc.	Read integer values
buf.writeUInt8(value, offset), buf.writeUInt16LE(value, offset), etc.	Write integer values

Buffer methods for comparison and search

Method	Description
buf.equals(otherBuffer)	Compare two Buffers
buf.indexOf(value[, byteOffset][, encoding])	Find the index of a value in the Buffer
buf.includes(value[, byteOffset][, encoding])	Check if Buffer includes a value

Buffer methods for slicing and copying

Method	Description
buf.slice([start[, end]])	Create a new Buffer that references the same memory as the original
buf.copy(target[, targetStart[, sourceStart[, sourceEnd]]])	Copy data between Buffers

Buffer methods for conversion

Method	Description
buf.toJSON()	Convert Buffer to a JSON representation
Buffer.concat(list[, totalLength])	Concatenate a list of Buffers

Common applications of Buffers

File operations

For example, reading from and writing to files, especially when dealing with binary data.

const fs = require("node:fs");

// Reading a file into a Buffer
const buffer = fs.readFileSync("example.bin"); // <Buffer ... >

// Writing a Buffer to a file
fs.writeFileSync("output.bin", buffer);

Network communications

For example, handling raw data in network protocols.

const net = require("node:net");

const server = net.createServer((socket) => {
  socket.on("data", (buffer) => {
    console.log(buffer.toString());
  });
});

Cryptography

This includes things like hashing, encryption, and decryption.

const crypto = require("node:crypto");

const data = Buffer.from("Hello, World!");
const hash = crypto.createHash("sha256").update(data).digest(); // <Buffer df fd 60 21 bb 2b d5 b0 af 67 62 90 80 9e c3 a5 31 91 dd 81 c7 f7 0a 4b 28 68 8a 36 21 82 98 6f>

Image processing

We can use Buffers to manipulate image data.

const sharp = require("sharp");

sharp("input.jpg")
  .resize(300, 200)
  .toBuffer((err, buffer) => {
    // buffer contains the processed image data
  });

Base64 encoding/decoding

Often used for data transfer and storage.

const original = Buffer.from("Hello, World!"); // <Buffer 48 65 6c 6c 6f 2c 20 57 6f 72 6c 64 21>
const encoded = original.toString("base64"); // 'SGVsbG8sIFdvcmxkIQ=='
const decoded = Buffer.from(encoded, "base64"); // <Buffer 48 65 6c 6c 6f 2c 20 57 6f 72 6c 64 21>
decoded.toString(); // "Hello, World!"

Data streaming

Efficiently processing large amounts of data. We will touch more on this throughout the series.

const fs = require("fs");

const readStream = fs.createReadStream("largefile.txt");
readStream.on("data", (chunk) => {
  // chunk is a Buffer
  console.log(chunk.length);
});

The highs and lows of using Buffers

Here a some highs of using Buffers:

• Efficient Binary Data Handling: Buffers are designed to handle raw binary data efficiently, making them ideal for file I/O, network communication, and other scenarios requiring direct manipulation of binary data.
• Direct Memory Allocation: Buffers allocate memory outside the V8 heap, allowing for more efficient use of memory and avoiding the overhead of garbage collection.
• High Performance: Direct access to memory and efficient handling of binary data result in high performance, especially for operations involving large amounts of data.
• Interoperability with Native Code: Buffers facilitate interoperability with native code (C/C++ libraries) and system-level resources, making it easier to integrate with existing systems and perform low-level operations.
• Flexibility in Data Encoding: Buffers support various data encodings (e.g., UTF-8, ASCII, Base64), making it easy to convert between different formats.

In contrast, here are some of the gotchas to look out for:

• Fixed Size: Once a Buffer is created, its size cannot be changed. This means you need to allocate the correct amount of memory upfront, which can be difficult to estimate accurately.
• Complexity: Using Buffers requires a decent understanding of binary data manipulation, memory management, and encoding/decoding, which can add complexity to the code.
• Potential Security Risks: Improper handling of Buffers, such as misuse of Buffer.allocUnsafe, can lead to security vulnerabilities.
• Less Readable Code: Code that deals with Buffers can be harder to read and understand compared to using higher-level abstractions, especially for developers not familiar with binary data manipulation.
• Buffers do not work with objects directly: The type must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. This means that you cannot pass an object directly to a Buffer (nor certain other types like booleans).

Conclusion

Buffers are a fundamental component in Node.js for handling raw binary data efficiently. By allocating memory outside the V8 heap, Buffers provide direct access to memory, enabling high performance for operations involving large data sets, such as file I/O and network communication. Understanding how Buffers work, including their fixed size and manual memory management requirements, is crucial for writing efficient Node.js applications.

In today's blog post, we've covered a the fundamental ideas behind Buffers in Node.js and we gave ourselves an analogy of a bucket outside a house to help us mentally visualize the concepts around Buffers.

While Buffers offer numerous advantages, including interoperability with native code and flexibility in data encoding, they also introduce complexity. Properly managing buffer allocation, avoiding common pitfalls, and ensuring memory is correctly handled are essential practices for developers working with Buffers.

By mastering Buffers, you'll be well-prepared to dive deeper into Node.js streams and handle data at a low level, paving the way for building robust and high-performance applications.

Resources and further reading

• Node.js' Buffer object documentation
• Node.js streams documentation
• The V8 JavaScript engine
• Node.js homepage
• Hexadecimal representation in JavaScript

Disclaimer: This blog post used AI to generate the images used for the analogy.

Photo credit: nervum

]]>