My Blog about Boats

Thoughts on Return on Investment (R.O.I) in Software Engineering

Boaty McBoatFace — Sun, 11 Jun 2023 08:45:52 +0000

Why bother with R.O.I ¶

In order to know the Return on an Investment, you need to know the cost of the investment. More than that, you need to know what you expect the return to be (how you will measure it) and when you expect to see it.

There are some base assumptions here that go unspoken everywhere you go, but I am somewhat of a scientist myself, so let's make them explicit:

You have a finite amount of resources (money, time, people, etc.)
The environment you are operating in will change and if you cannot capitalize now you will be disrupted (or see this synopsis on youtube)

Those two things are enough to suggest you have to make good choices now and they must be based on some kind of evidence. That evidence is the projected return on investment.

N.B. If you find your self calculating the R.O.I of something that has already happened it is probably not for a good reason, but will help you avoid the same mistake in the future.

Armed with this knowledge you can make a decision about what to do next. You can decide to invest in something or not. You can decide to invest in something now or later. You can decide to invest in something else instead.

Measuring things in general ¶

Lets begin with some background on my experience with R.O.I.

In the before times, when I was a Chemical Engineer at X large company the way that things could be measured was much more straight forward. The company had a target IRR that it aimed for and business cases that would operate in the range of decades.

Before that I was a student, mostly working in thermodynamics and established laws of physical systems, where everything was also easy to measure.

Both these previous statements are lies, nothing of course was easy to measure in thermodynamics. People smarter than me had toiled away for centuries to simplify things. I arrived on the scene ready to have it regurgitated at me in a way that I could understand. I was not disappointed.

Do you think that measuring the impact of software engineering is supposed to be harder than this?

Research and truth seeking ¶

I took to the internet to find out if there was a simple set of rules for calculating return on investment for software engineering and I was ~~not~~ disappointed.

I found a book with mixed reviews which I think I will give a hard pass.

I found some news articles about stripe acquiring a company that measures engineer productivity. I'm not sure that this is the same thing as R.O.I but I'm building a mental model here. For anyone that has systems design experience you'll know we are just doing discovery, hold tight, I will get there.

I found a blog post that was a little more interesting. It talks about the fact that R.O.I is a financial metric and that it is hard to measure the impact of software engineering in financial terms. It gives a foot note of advice on why R.O.I is important and talks about what to use it for.

Finally on InfoQ I found a presentation by none other than Kent Beck that was a little more interesting. It starts with a focus on communication (okay Kent, you got me hooked) then it delves a little into the idea of measuring the impact of refactoring. Finally it touches on Net Present Value (NPV) which is a financial metric that I am familiar with.

The basics of Engineering ¶

Kent and I are not on first name terms, as you might imagine, but I think he is on to something here. At around 27 minutes he said something that suddenly crystallized a thought in me that had been brewing for some time. Seeing Kent say this I realized I could be right (yes, here comes a confirmation bias). So here is what he said:

"May you live long enough, that people start doing the dumb things you did when you were young" - Kent Beck source

This quote applies to Software Engineering.

Specifically Chemical Engineering has lived long enough to have a significant equity of experience to look down on Software Engineering and say "give it a try, let's find out what happens?". Which, coincidentally is fine in software engineering, but a terrible idea in chemical engineering for reasons that are obvious to anyone that has ever worked in a chemical plant.

Software engineering is not yet old enough to have a set of best practices that are universally accepted. We are still in the wild west of software engineering. We are still learning how to do things. We are still learning how to measure things.

I'll tell you why. As Kent goes on to talk about the Time Value of Money and the Net Present Value (NPV) I transform Kent Beck into 3 separate talking heads. One from 2012, another form 2014 and a third from 2016. There are about 70 people I know that would have had the exact same reaction to this. Everyone that attended University with me.

That is because the powers that be in my Engineering school decided that before they let loose a bunch of half informed engineers on the world they would teach us about the Time Value of Money.

They would teach us about the Net Present Value (NPV).

They would teach us about the Internal Rate of Return (IRR).

They would teach us about the Discounted Cash Flow (DCF).

The best news here is that I know more than Kent Beck does about something. Which is something I thought I would never say, and likely to be something that will evaporate as soon as I hit publish on this post.

Software engineers, the great truth seekers ¶

You are maybe wondering how we got here?

Accessibility.

Now, I am not talking about the kind of accessibility that you are thinking about. I am talking about the kind of accessibility that means that anyone can write software. Anyone can write software that is accessible to anyone else. And that software can be used by anyone else.

I'm not sending you a link to my latest open source Polystyrene factory that I decided to build on Sunday to avoid cleaning the car now am I!

This flows both ways. Traditional engineering is not accessible!

It takes a huge amount of time and resources to do anything in traditional engineering. That is only one part.

For the price of a small family car you could go to a boot camp in your nearest major city for 12 weeks. You could then get a job; probably paying you too much money (separate rant) and you could have software in front of unwitting users in under 4 months.

The equivalent in traditional engineering is 4 years of university, 2 years of graduate training and 2 years of experience before you are allowed to do anything that might impact the world.

Closing thoughts and a conclusion ¶

I think that the reason that we don't have a good way to measure the impact of software engineering is because we are still learning how to do it. Any time someone comes up with a good attempt, they think they can monetize it into some software to sell to software companies. Once that happens market demand dilutes the pure academic idea into a polarized competition between two companies that are trying to sell the same thing.

I think that there is a missing component in the training of software engineers. Economics and finance. I think that if we taught software engineers about the Time Value of Money, the Net Present Value (NPV), the Internal Rate of Return (IRR) and the Discounted Cash Flow (DCF) then we would have a better understanding of how to measure the impact of software engineering.

I think that all forms of engineering have a similar set of universal constraints. Time, Money, People, Materials, Energy, Information, Equipment, Space, and Environment. Just like all software systems, these are not simply the sum of parts. The outcome is a product of the interactions. In software however you are often able to create time through automation, and simultaneously reduced the required amount of people in a tighter iteration cycle than traditional engineering.

This plays out at a macro level in behaviors that reward failing fast and failing cheap in these fast cycles. In contrast the cost of failure in say, buying the wrong sized pipe for a chemical plant is much higher. In part due to the upfront (Capital) costs but also because the time it takes for the N.P.V to become positive is much longer than software projects.

Finally to those that thought they would learn a quick way to calculate return on investment for software engineering. I'm sorry, I don't have one. I think that the best way to calculate the return on investment for software engineering is to do it the same way that you would calculate the return on investment for any other engineering project.

I can recommend a good book from 1970 on Polystyrene manufacture if that helps?

Better typescript fetch clients

Boaty McBoatFace — Mon, 16 Jan 2023 07:00:00 +0000

Fetch is used in the browser to interact with API's over HTTP(S). Fetch works well for this. It is promise based. These are facts.

When you work with an API you often want type safety.

There is a popular piece by Kent C. Dodds on this. Kent wraps the request and handles coercing with some additional code. Check it out here.

I have my own preferred way to work with this using type intersections and pure functions.

My goal is to create a generic way to handle fetch; but I do not want to reshape the Response object, I want to pass that out.

This helps allow anyone consuming my fetch API client to extend for their own use cases.

If I make a design decision now to return something like { data: any, error: Error } I can paint myself into a corner.

Instead these clients will just return a full Response and when someone tries to use .json() the correct type will be set as if by magic (AKA inference, and narrowing*).

TL;DR ¶

The code is here if you just want to get on with your day...

// ../_snippets/fetch-clients/example.ts

/** For 201 */
type UserCreated = { id: string; name: string };

/** For 400 */
type BadRequest = { code: "bad_request"; message: string };

/** Response type intersection */
type UserResponse =
  | (Omit<Response, "json"> & {
      status: 201;
      json: () => UserCreated | PromiseLike<UserCreated>;
    })
  | (Omit<Response, "json"> & {
      status: 400;
      json: () => BadRequest | PromiseLike<BadRequest>;
    });

/** Marshalling stream to object with narrowing */
const marshalResponse = (res: UserResponse) => {
  if (res.status === 201) return res.json();
  if (res.status === 400) return res.json();
  return Error("Unhandled response code");
};

/** Coerce Response to UserResponse */
const responseHandler = (response: Response) => {
  const res = response as UserResponse;
  return marshalResponse(res);
};

/** Usage returns typed data */
const data = fetch(`https://api.com/v1/user`, {
  method: "POST",
  body: JSON.stringify({ name: "Simon" }),
}).then((res) => responseHandler(res));

The type of data in the above is:

UserCreated | BadRequest | Error;

I will deep dive this below and show how this all works.

An example ¶

If you had an API that could create users you would send a POST request in fetch like below:

// ../_snippets/fetch-clients/request.ts

fetch(`https://user.api.com`, {
  method: "POST",
  body: JSON.stringify({
    name: "Simon",
  }),
});

This would create a promise, and that promise would have the type Promise<Response>.

To access the data that is returned from the API you would want to call .json() on the response.

The type of Response['json'] is a Promise<any> and that is not very helpful (this is a fact).

As part of the domain transfer layer you probably want to get your type back.

Lets suppose the API can return two bodies, one for a 201 code and one for a 400 code. Those will correspond to the Created and BadRequest responses.

// ../_snippets/fetch-clients/domain-transfer-objects.ts

export type UserCreated = {
  id: string;
  name: string;
};

export type BadRequest = {
  code: "bad_request";
  message: string;
};

Our API will always return a 400 error of this kind, and a 201 on success.

Response Typing ¶

We can use an intersection type on the Response type to narrow and restrict response to the two known types I expect.

Our user response look like this:

// ../_snippets/fetch-clients/response-user.ts

/**
 * The UserResponse is a Union of intersection that
 * will narrow when the status is asserted
 */
export type UserResponse =
  | (Omit<Response, "json"> & {
      status: 201;
      json: () => UserCreated | PromiseLike<UserCreated>;
    })
  | (Omit<Response, "json"> & {
      status: 400;
      json: () => BadRequest | PromiseLike<BadRequest>;
    });

I do a few things in this type:

Omit (remove) the standard json that returns any. This is needed or I cannot narrow effectively because all types can be any and because that satisfies all constraints the compiler will helpfully always default to this.
Make a union of intersections (&) that will narrow by relating the specific HTTP status code to a specific response.

I have used PromiseLike instead of promise, because that is the type that the underlying Response wants to have when it chains promises via then or catch.

Narrowing Response ¶

Now I want to use the UserResponse type to handle the response from fetch. We want this type to narrow the possible values of .json() to BadRequest | UserCreated.

To do that I will perform assertions on the status:

// ../_snippets/fetch-clients/marshal.ts

import { UserResponse } from "./response-user";

/**
 * This is the Dto layer. If you want to coerce types
 * like * string representations of dates into real
 * date objects then do it here.
 */
export const marshalResponse = (res: UserResponse) => {
  if (res.status === 201) return res.json();
  if (res.status === 400) return res.json();
  return Error("Unhandled response code");
};

I have returned an error here, but you don't need to, you could expand the interface to return an empty object {} if the response code is unhandled 👍.

Whatever you do, do not throw. Friends don't let friends use errors for flow control!

Next, I need to wrap the marshal function into a response handler.

// ../_snippets/fetch-clients/handlers.ts

import { marshalResponse } from "./marshal";
import { UserResponse } from "./response-user";

export const responseHandler = (response: Response) => {
  const res = response as UserResponse;

  return marshalResponse(res);
};

The responsibility of this function is to type cast the Response to a UserResponse and pass it into the marshal function.

Putting it all together ¶

Let's first look at the await case.

// ../_snippets/fetch-clients/request-await.ts

import { responseHandler } from "./handlers";

const response = await fetch(`https://api.com/v1/user`, {
  method: "POST",
  body: JSON.stringify({
    name: "Simon",
  }),
});

export const body = await responseHandler(response);

So what type do you think body is now?

And when we just use promises:

// ../_snippets/fetch-clients/request-promise.ts

import { responseHandler } from "./handlers";

export const data = fetch(`https://api.com/v1/user`, {
  method: "POST",
  body: JSON.stringify({
    name: "Simon",
  }),
}).then((res) => responseHandler(res));

Wrap up ¶

This concludes the explanation of how to make fetch return the types that you want for your JSON bodies, without masking the underling Response type.

The possibilities for this kind of type are to create general API clients, that have inferred and narrowed response types, without writing lots of wrapper code, and without masking the underlying response object (open for extension).

I hope you find it useful.

Lit elements with Xstate

Boaty McBoatFace — Sat, 14 Jan 2023 07:00:00 +0000

This blog focuses on state management inside of web components.

If you are not familiar with Web Components, the Light DOM (normal DOM) and the Shadow DOM (the encapsulated DOM you can use with web components) then this is an excellent piece of background reading

The TL;DR is:

Web components are HTML Custom Elements that can be adopted by the DOM; they require Javascript to work, and can have their own observable properties.
The Light DOM is standard HTML elements which can be written in plain text.
The Shadow DOM is an encapsulated sub tree with it's own root that maintains it's own styles propagated down, but not back up to the light DOM it was rendered in.

State Management ¶

Unless your are building a purely presentational design system you are likely dealing with state management inside the browser.

In order to make the client experience deterministic when the application grows it becomes good practice to fan all modifications into a central state and resolve the new state. This is common in frameworks like react (with context providers, or redux, zuustand etc.).

Lit has a proposed context implementation here which enables decoration of properties to consume their value from a central store. This will solve passing props to elements via prop drilling.

There is still a need to coordinate state changes within the context, and so I will show how xstate can be used to achieve this.

Xstate ¶

xstate is a tool for state charts and state machines.

It makes it possible to build state machines and coordinate the dispatch of events which make modifications to the state.

Setup a machine in Lit ¶

There will be three components to this:

The machine, which is a state machine with two events to increment and decrement a counter.
A DOM Event adapter for dispatching CustomEvents to an eventListener and sending those events to xstate
A Lit Mediator Element which creates an interpreted xstate machine and listens for CustomEvents. The mediator will send events to the machine, which will invoke a callback to update the observed properties. These properties can be drilled down into child elements.

Lets sketch that out with a handy diagram:

01 - The machine ¶

The below is an example of a state machine to increment and decrement a number. This tool is overkill for such a use case, but I want to use a simple example as not to create too much cognitive load for you to follow.

import { assign, createMachine } from "xstate";

export type Context = { count: number };
export type Events = { type: "INC" } | { type: "DEC" };

export const machine = createMachine(
  {
    predictableActionArguments: true,
    tsTypes: {} as import("./index.typegen").Typegen0,
    schema: {
      context: {} as Context,
      events: {} as Events,
    },
    initial: "active",
    context: {
      count: 0,
    },
    states: {
      active: {
        on: {
          INC: { actions: "increment" },
          DEC: { actions: "decrement" },
        },
      },
    },
  },
  {
    actions: {
      increment: assign({
        count: (context) => context.count + 1,
      }),
      decrement: assign({
        count: (context) => context.count - 1,
      }),
    },
  }
);

The state machine is a declaration of how modification can occur and in what direction. To use the machine we need to create an instance and send events to it.

02 - The DOM Event Adapter ¶

We want to be able to use our state machine, by sending the events that it is listening for.

The idiomatic way to do this inside the DOM is to listen for CustomEvents - and execute an eventListener.

We want to constrain the published events to the set of allowed events that xstate machine is listening for.

So to do that we can create a factory function which returns new events, see below:

import type { Events } from "../machines";

export const DOMEventPublisher = (event: Events) => {
  /**
   * Create a new event that will bubble through
   * multiple shadow roots
   */
  return new CustomEvent<Events>("xstate-broadcast", {
    detail: event,
    composed: true,
    bubbles: true,
  });
};

// This will make TS intellisense handle these event names
declare global {
  interface HTMLElementEventMap {
    "xstate-broadcast": CustomEvent<Events>;
  }
}

We can use the factory now to create events that will:

Be constrained to our expected event types from the machine when writing typescript
Be bubbled through each node in the DOM so any listeners can take action
Be composed so they will propagate out of the Shadow DOM into the standard DOM.

This ensures we will be able to listen on these events wherever we start the xstate machine.

03 - A Lit Mediator Element ¶

We want to be able to both consume CustomEvents, and update xstate state and context, then pass the updated state to all child elements. In order to do this we will use something called the Mediator Method.

Events up, props down - source Event communication between web components - Lit University (Advanced)

I will show what that looks like in practice to enable us to create a generic element that could be used with any xstate machine to subscribe to the set of expected events and pass back down the props to any child elements.

The mediator parent looks like below:

import { css, html, LitElement } from "lit";
import { customElement, property } from "lit/decorators.js";
import { interpret } from "xstate";

// Import our increment machine
import { machine } from "./machines";

@customElement("my-element")
export class MyElement extends LitElement {
  // Scope the service within the element class
  service = interpret(machine);

  // Set the state as a property to make it observable
  @property({ attribute: false })
  state = this.service.initialState.value;

  // Set the context as a property to make it observable
  @property({ attribute: false })
  context = this.service.initialState.context;

  firstUpdated() {
    // Listen for special events for xstate
    this.addEventListener("xstate-broadcast", (e) => {
      this.service.send(e.detail);
    });
  }

  // Start service on connection/adoption into the Light DOM
  connectedCallback(): void {
    super.connectedCallback();
    // Register a callback for all event transitions
    this.service.onTransition((state, _event) => {
      /**
       * Assign state and context REMEMBER if these
       * are objects you need to create a new object
       * for Lit to detect the hasChanged!!
       */
      this.state = state.value;
      this.context = state.context;
    });
    this.service.start();
  }

  // Stop service on removal from the Light DOM
  disconnectedCallback(): void {
    this.service.stop();
  }

  render() {
    html`<p>The current state is <em>${this.state}</em></p>`;
  }
}

This code uses some lifecycle methods from Lit to ensure that the machine will start and stop, when the element is adopted or removed from the DOM respectively.

It also ensures that when the element is first created the properties are assigned to the state machines default values; this is important for a deterministic first paint (reduce Cumulative Layout Shift).

Finally in the firstUpdated event, which fires only once, on the elements first update; we ensure that we establish a listener for all elements in the tree.

We could have an issue with race conditions if we rendered some elements that dispatch events before listener is in place.

We would also have issues if we rendered two of the my-element web components on the same page, because events from one would propagate to the other.

Summary of the pattern ¶

Now we have a mediator setup, we are listening on custom xstate-broadcast events.

When we receive those events in the parent, we call into the local this.service to send the events to the interpreted machine.
When the machine receives those events it will execute a callback, which updates any of our observable properties that have changes.
Lit will run the hasChanged function for each property, and if they have changed it will perform an update.
Any elements below the Mediator will receive the updated state via prop drilling (for now, but later we could tie our callback into a Lit labs context and have all our other elements subscribe).

Wrap up ¶

I have been on quite a journey with Lit to understand how state should be managed.

The resources at Lit university are the best to get started.

I'll be continuing this series on Lit and intend to cover topis like:

Real user monitoring inside the shadow DOM
Production grade internationalization
Design systems

And likely more, so if your interested you can subscribe to my rss feed

Private Singleton for Scoping Sentry SDK

Boaty McBoatFace — Sun, 30 Oct 2022 07:00:00 +0000

Sentry is a great error tracking tool. Sentry browser defaults to using global this as a memoization tool.

This blog explores singleton typescript patterns for scoping sentry down when using it on someone elses website.

This will hopefully help you only capture your own errors, and also not annotate any existing sentry contexts with your embedded applications data.

But why 🤷 ¶

Embedded products are things that bundle into other websites. You might have one???

I'll narrow this down to javascript served (probably from a CDN) onto someone elses website; your customers if you are lucky.

Third party JS is a necessary component of some products. The maintenance and observability of those 3rd party scripts by the authors is crucial.

When you welcome 3rd party JS to your website, you allow it to do a lot. If you are doing this without a Content Security Policy (CSP) then your users data could be at risk and used in any way.

Manning has a book about it; here's a snippet:

Third-party JavaScript applications are self-contained components, typically small scripts or widgets, that add functionality to websites. As the name implies, they're offered by independent organizations, with code and asset files served from a remote web address. Third Party JS, Ben Vinegar and Anton Kovalyov - Manning

Design ¶

To create a global singleton like Sentry I will use the Singleton pattern.

I can make use of a private constructor and static methods that refer to a closured instance to make the API seem like it's mimicking Sentry.

When it's finished it will work like this:

import { ScopedSentry } from "./utils";

// Your code goes here

ScopedSentry.trackSessionId("11-22-33-asd");
ScopedSentry.captureException(new Error("captured"));

Creating a singleton ¶

We need to do four things to create a singleton.

First we need to create a class and declare it to have an instance property that is of it's self as a type.
Second, be sure to declare the constructor to be private so that calls to new ScopedSentry will raise compiler errors in Typescript.
Thirdly we need to write the method that can be used to create or return the instance. Notice that the constructor is called in this private method. This is fine because we can invoke private methods when we are inside a private scope already.
So lastly we will implement the constructor for the Singleton class.

The code for this is below.

// ../_snippets/scoped-sentry/index.ts

import * as Sentry from "@sentry/browser";
import { defaultStackParser, Hub, makeFetchTransport } from "@sentry/browser";

export class ScopedSentry {
  private static instance: ScopedSentry;
  private hub: Sentry.Hub;
  private client: Sentry.BrowserClient;

  private constructor() {
    this.hub = new Hub();
    this.client = this.buildClient();
    this.hub.bindClient(this.client);
  }

  private static getInstance(): ScopedSentry {
    if (!ScopedSentry.instance) {
      ScopedSentry.instance = new ScopedSentry();
    }
    return ScopedSentry.instance;
  }

  public static trackSessionId(sessionId: string) {
    const { hub } = this.getInstance();
    hub.configureScope(function (scope) {
      scope.setTag("session-id", sessionId);
    });
  }

  public static captureMessage(...args: Parameters<Hub["captureMessage"]>) {
    const { hub } = this.getInstance();
    hub.captureMessage(...args);
    this.safeFlush();
  }

  public static captureException(...args: Parameters<Hub["captureException"]>) {
    const { hub } = this.getInstance();
    hub.captureException(...args);
    this.safeFlush();
  }

  public static async safeFlush() {
    const { client } = this.getInstance();
    return client
      .flush(300)
      .then((flushed) => {
        return flushed;
      })
      .catch((e) => {
        console.warn("Flush failed");
      });
  }

  private buildClient() {
    return new Sentry.BrowserClient({
      dsn: process.env.SENTRY_DSN,
      environment: process.env.SENTRY_ENVIRONMENT,
      integrations: [],
      stackParser: defaultStackParser,
      transport: makeFetchTransport,
    });
  }
}

I've added some public methods that fetch the private instance of the closured Hub and then provide additional context. I've also implemented a public methods for captureMessage and captureException that will ensure errors are sent to sentry within 300 milliseconds using the flush.

Wrap up ¶

I've taken inspiration from this thread and read a fair amount of Sentry browser and javascript SDK sort code to arrive at this solution.

Hopefully learning how to create private singletons is useful for you 👍

Cypress component tests for Lit Elements (web components)

Boaty McBoatFace — Thu, 13 Oct 2022 07:00:00 +0000

Recently I have been using lit, a web component framework.

If you are not familiar with web components here is a quick summary:

Web Components is a suite of different technologies allowing you to create reusable custom elements - with their functionality encapsulated away from the rest of your code - and utilize them in your web apps.

MDN - https://developer.mozilla.org/en-US/docs/Web/Web_Components

Lit is a popular framework for these components with over 1 million weekly downloads. (Oct 10th 2022).

Testing Lit ¶

For how to test Lit Elements I had been following some big notable projects,

Spectrum web components by Adobe is really mature design system that makes a lot of usage of Lit Elements. Their testing setup uses the suggested web test runner. Lit's documentation on testing suggests using that library.

Cypress has become one of the most popular test runners in the space. So I was a little surprised to not see some guidance for it on the lit documentation. I think it's an ideal candidate for Web Component testing on modern browsers, and because it recently added support for safari it has great coverage across chromium, webkit, electron and quantum (firefox) based browsers.

Testing Lit Components with Cypress ¶

To test with lit elements you will need a project with cypress.

TL;DR ¶

An example project: https://github.com/simonireilly/cypress-lit

The testing setup consists of writing a cy.mount command for mounting Lit Elements into the DOM.

import { getContainerEl } from "@cypress/mount-utils";
import { LitElement, render, TemplateResult } from "lit";

export const mount = (_element: LitElement, template: TemplateResult) => {
  const target = getContainerEl();

  render(template, target);

  return cy
    .wait(0, { log: false })
    .then(() => {
      const mountMessage = `
        <${String(_element.constructor.name)} ... />
      `;

      Cypress.log({
        name: "mount",
        message: [mountMessage],
      })
        .snapshot("mounted")
        .end();
    })
    .get("[data-cy-root]")
    .children()
    .first()
    .shadow();
};

And this means you can write tess like this:

import { html } from "lit";
import { MyElement } from "../../src/my-element";

describe("my-element.cy.ts", () => {
  it("playground", () => {
    cy.mount(new MyElement(), html`<my-element></my-element>`).as("element");

    cy.get("@element").contains("count").click();
  });
});

And just like that, you can run a cypress test for the component. Here is a quick snapshot of me testing one.

Wrap up ¶

That is a quick example of how to run component tests for Lit with cypress. Hopefully it's just enough information for you to get it setup.

Since cypress components supports so many frameworks already, I don't think it will be long before an official lit component implementation lands. In the meantime, happy testing!

Quick Tip: Hiding local files in Git, just for you

Boaty McBoatFace — Fri, 08 Jul 2022 18:42:55 +0000

The more you code, the more you have tools that help you work quick.

Using those deterministically in open-source can be challenging.

You want to write a small PR, but first, lets install the correct version of pnpm (checks lock file). Ok, now NodeJS to. I use asdf for these (thanks Paul for the hook up), but not everyone does, I'll just put .tool-versions in the .gitignore file.

If this continues you have a huge .gitignore for peoples specific tools, or a bunch of meaningless top level files. In open source projects, you probably want a lighter touch.

Queue the much underutilized .git/info/exclude

This file is your own personal .gitignore, I recommend reading the docs (a little). But basically use this when necessary to exclude your own local files.

Enjoy 👍

Cloudwatch Real User Monitoring (RUM) end to end monitoring

Boaty McBoatFace — Thu, 17 Feb 2022 07:00:00 +0000

I originally blogged about using canaries and RUM with the AWS-CDK here link.

In that blog post I was exploring how to create a website with an AWS Synthetics Canary.

This blog will revisit the Canary Stack I created, and enhance the experience of Real User Monitoring (RUM).

To get a deep dive on RUM checkout this blog post

What has Changed ¶

The cloudformation specification version v50.0.0 includes support for AWS::RUM API's. The AWS-CDK version v2.12.0 added Cloudformation RUM support 🎉.

This is great news, because previously the stack had some manual configuration that was not great for deterministic Infrastructure as Code (IaC) lovers I am sure.

TL; DR ¶

Source code, the ultimate TL; DR https://github.com/simonireilly/canary-stack/blob/main/lib/rum/rum-construct.ts

I create a Rum construct, that uploads the rum.js script to ./rum.js inside the s3 origin bucket for the website
I added default performance budgets for Largest Contentful Paint (LCP), First Input Delay (FID) and Cumulative Layout Shift (CLS).

The stack ends up looking something like this:

Image by cdk-dia

Design ¶

I really like some products, like speedcurve that enable setting performance budgets for the website, so I think I would like something like this (pseudo code):

new Rum(this, 'MyRumResource', {
  topLevelDomain: "*.s3-website-eu-west-1.amazonaws.com"
  appMonitorName: 'simons-website-monitor',
  s3Bucket: websiteBucket,
  // In milliseconds see: https://web.dev/vitals/
  performanceBudgets: {
    CLS: 100,
    LCP: 2500,
    FID: 100
  }
})

The underlying behavior should be that I get alerts when the website performs below these thresholds.

So I need to build this Rum construct to do this for me, ok, here we go.

Updating the stack ¶

I am going to add a custom Rum construct to my stack, that is going to hide the Cfn implementation of RUM.

I am also going to enable AWS XRay, and connect my Canary to my RUM monitor using tags.

First things first, I need the newest version of AWS-CDK installed:

yarn upgrade-interactive --latest

Upgrading the AWS-CDK worked fine for me using yarn interactive. Before I made any changes I did a re-deploy to ensure I wasn't starting from a bad basis, now we can continue.

Creating the App Monitor ¶

With the latest version of the AWS-CDK we can now use the CfnAppMonitor construct to deploy an app monitor.

The App Monitor needs an associated authentication mechanism. I have chosen to use a Cognito Identity Pool, that will get temporary AWS access keys as an unauthenticated user.

That just looks like this:

this.identityPool = new CfnIdentityPool(this, "RumAppIdentityPool", {
  allowUnauthenticatedIdentities: true,
});

So next, I need to allow the Cognito Identity service to assume a role that allows it to put rum events and xray trace segments. This next one is a little more verbose, but here it is:

new Role(this, "UnauthenticatedRumRole", {
  assumedBy: new FederatedPrincipal(
    "cognito-identity.amazonaws.com",
    {
      StringEquals: {
        "cognito-identity.amazonaws.com:aud": this.identityPool.ref,
      },
      "ForAnyValue:StringLike": {
        "cognito-identity.amazonaws.com:amr": "unauthenticated",
      },
    },
    "sts:AssumeRoleWithWebIdentity"
  ),
  inlinePolicies: {
    RUMPutBatchMetrics: new PolicyDocument({
      statements: [
        new PolicyStatement({
          actions: ["rum:PutRumEvents"],
          resources: [
            // We can construct any Arn this way
            Stack.of(this).formatArn({
              service: "rum",
              resource: "appmonitor",
              // This will come in later, but for
              // now its a string 👍
              resourceName: this.appMonitorName,
            }),
          ],
        }),
        // Xray needs access to all resources on
        // PutTraceSegments, you can check
        // the docs here:
        // https://docs.aws.amazon.com/xray/latest/devguide/security_iam_service-with-iam.html#security_iam_service-with-iam-resource-based-policies
        new PolicyStatement({
          actions: ["xray:PutTraceSegments"],
          resources: ["*"],
        }),
      ],
    }),
  },
});

Now with this minimum privileged role, and the identify pool it references, we will perform an attachment. This will mean that the identity pool will have this attached role.

new CfnIdentityPoolRoleAttachment(this, "RumAppRoleAttachment", {
  identityPoolId: this.identityPool.ref,
  roles: {
    unauthenticated: this.unauthenticatedRumRole.roleArn,
  },
});

From here on out it is plain sailing. we can pick the configuration that we want for the CfnAppMonitor. I have decided to go all in with all the telemetries, xray and cookies enabled.

new CfnAppMonitor(this, "RumAppMonitor", {
  name: this.appMonitorName,
  cwLogEnabled: true,
  domain: this.topLevelDomain,
  appMonitorConfiguration: {
    allowCookies: true,
    enableXRay: true,
    sessionSampleRate: 1,
    telemetries: ["errors", "performance", "http"],
    identityPoolId: this.identityPool.ref,
    guestRoleArn: this.unauthenticatedRumRole.roleArn,
  },
});

We can deploy all this into our stack and we will have a great pile of infrastructure, but unfortunately, there won't be any RUM.

That's because cloudwatch console will tell us we need to add a script to our website, so I am going to solve automating the injection of that script.

Automating the RUM script ¶

I am going to assume that I can modify the s3 bucket, to inject a ./rum.js script. Currently the bucket contents are as such:

.
├── images
│   ├── diagram.avif
│   ├── diagram.png
│   └── diagram.webp
└── index.html

I will write an AWS-CDK custom resource that injects a script into the bucket, and then the bucket will look like:

.
├── rum.js
├── images
│   ├── diagram.avif
│   ├── diagram.png
│   └── diagram.webp
└── index.html

So in my website I will add to the <head> tag the following:

<script src="rum.js"></script>

And once that file is in the bucket, the browser will pull it down after it parses the HTML 👍.

Template for rum.js ¶

The RUM script provided by AWS is only available in the dashboard. But on inspection, its fairly simple. So, I've gone and made it a template, here we go:

const RUM_TEMPLATE = (rum: RUM.AppMonitor): string => `
(function (n, i, v, r, s, c, x, z) {
  x = window.AwsRumClient = { q: [], n: n, i: i, v: v, r: r, c: c };
  window[n] = function (c, p) {
    x.q.push({ c: c, p: p });
  };
  z = document.createElement('script');
  z.async = true;
  z.src = s;
  document.head.insertBefore(
    z,
    document.head.getElementsByTagName('script')[0]
  );
})(
  'cwr',
  '${rum.Id}',
  '1.0.0',
  'eu-west-1',
  'https://client.rum.us-east-1.amazonaws.com/1.2.1/cwr.js',
  {
    sessionSampleRate: ${rum.AppMonitorConfiguration?.SessionSampleRate},
    guestRoleArn:
      '${rum.AppMonitorConfiguration?.GuestRoleArn}',
    identityPoolId: '${rum.AppMonitorConfiguration?.IdentityPoolId}',
    endpoint: 'https://dataplane.rum.eu-west-1.amazonaws.com',
    telemetries: [${
      '"' + rum.AppMonitorConfiguration?.Telemetries?.join('","') + '"'
    }],
    allowCookies: ${rum.AppMonitorConfiguration?.AllowCookies},
    enableXRay: ${rum.AppMonitorConfiguration?.EnableXRay},
  }
);
`;

So we have an Immediately Invoked Function Expression (IIFE) here, which, once on our page will:

Run with the arguments we place in the final set of brackets.
Pull from a remote the https://client.rum.us-east-1.amazonaws.com/1.2.1/cwr.js
- A good candidate to add to your Content Security Policy (If you have one 😱).
Insert the cwr.js script as the first element in the page.

Syncing rum.js to s3 ¶

I want to run the template when I know the full App Monitor configuration. However, this is only know after the app monitor has been provisioned.

This can be solved using an AWS-CDK custom resource.

The custom resource will:

Deploy a lambda function which provides its ARN as the serviceToken
Take the App Monitor name as a property
Ensure that once the App Monitor is deployed it invokes the lambda service

That lambda will make a call to get the current configuration of the RUM App Monitor:

const appMonitor = await rum
  .getAppMonitor({
    Name: appMonitorName,
  })
  .promise();

And it will inject that configuration into the above template. That's all looking good so shall we take a look at the end results?

  /**
   * Places a script inside the aws s3 bucket that serves the website
   * using a custom resource
   */
  private uploadRumFile() {
    const fn = new NodejsFunction(this, "UploadRumScriptHandler", {
      handler: "handler",
      entry: path.join(__dirname, "custom", "handler", "index.ts"),
    });

    fn.addToRolePolicy(
      new PolicyStatement({
        actions: ["s3:PutObject*", "s3:DeleteObject*"],
        resources: [`${this.s3Bucket.bucketArn}/rum.js`],
      })
    );

    fn.addToRolePolicy(
      new PolicyStatement({
        actions: ["rum:GetAppMonitor"],
        resources: [
          Stack.of(this).formatArn({
            service: "rum",
            resource: "appmonitor",
            resourceName: this.appMonitorName,
          }),
        ],
      })
    );

    new CustomResource(this, "UploadRumScriptToWebsiteBucket", {
      serviceToken: fn.functionArn,
      properties: {
        s3BucketName: this.s3Bucket.bucketName,
        appMonitorName: this.appMonitorName,
        appMonitorConfiguration: this.appMonitor,
        // The CDK needs to always upload the rum, otherwise the new web
        // deployment erases the file.
        trigger: Date.now(),
      },
    });
  }

Results ¶

There are a number of things for us to checkout, first, lets take a look at Real User Monitoring:

Key takeaways ¶

We have got Real User Monitoring 👍

We can see in the bottom right that there is an associated Canary, that works like this:

Tags.of(canary).add(rum.appMonitor.name, "associated-rum");

Or, configure manually as a tag on your canary like so:

Extra Credit: Adding Web Vital Performance Budgets ¶

It's all well can good to have detailed metrics about your users, but we don't want to sit around and check them all day for regressions.

Instead I am going to add some error budgets that can be used for the site.

Cloudwatch has got some great metrics built into it for this stuff which is being gathered by the App Monitor. I have decided to make a new construct around those which enables creating them inside any stack that has an app monitor like such:

const vitals = new WebVitals(this, "WebVitals", {
  appMonitorName: "canary-stack-monitor",
  /**
   * Cumulative Layout Shift (CLS): measures visual stability. To provide a good
   * user experience, pages should maintain a CLS of 0.1. or less.
   */
  WebVitalsCumulativeLayoutShift: 0.1,
  /**
   * First Input Delay (FID): measures interactivity. To provide a good user
   * experience, pages should have a FID of 100 milliseconds or less.
   */
  WebVitalsFirstInputDelay: 100,
  /**
   * Largest Contentful Paint (LCP): measures loading performance. To provide a
   * good user experience, LCP should occur within 2.5 seconds of when the page
   * first starts loading.
   */
  WebVitalsLargestContentfulPaint: 2500,
});

This construct create the metrics and some alarms that can be added to a dashboard.

Here is the private method that sets the performance budget for each named WebVital metric:

  /**
   * Web Vitals are configured to have alarms and metrics for the average real
   * user experience by default.
   */
  private configureVital(
    name: keyof WebVitalsPerformanceVitals,
    performanceBudget: number,
    statistic: "min" | "max" | "avg" | "p50" | "p90" | "p95" | "p99" = "avg"
  ): ICloudwatchWebVitalProps {
    const metric = new Metric({
      metricName: name,
      namespace: "AWS/RUM",
      dimensionsMap: {
        application_name: this.appMonitorName,
      },
      statistic,
    });

    const alarm = new Alarm(this, `${this.id}-alarm-${name}`, {
      evaluationPeriods: 2,
      threshold: performanceBudget,
      alarmName: name,
      metric,
    });

    return { metric, alarm };
  }

I have included them in a dashboard that also shows the status of the canary:

😱 Looks like I have some issues with First Input Delay! I will need to fix those another time 🤣

Wrap Up ¶

In this blog I showed:

How to configure AWS Real User Monitoring (RUM) App Monitor for your website
I showed how a custom resource can be written to upload the rum script
I showed how to access the metrics created by RUM
I showed how to associate canaries to the RUM agent
I showed how to set performance budgets for the Web Vitals that the RUM agent gathers

That's a whole lot, and most of the detail is available in the source code, so i would encourage you to take a closer look if you are thinking about using AWS RUM

Canary Stack

Paralleling Cypress with Sorry Cypress on Render

Boaty McBoatFace — Wed, 09 Feb 2022 08:45:52 +0000

Cypress is an integrated testing framework for web apps, and web components. It's popular due to it's ease of setup, and visual browser. In this blog I discuss parallel test running with the open source tool sorry cypress.

TL;DR ¶

If you want to host the sorry cypress kit, then https://render.com/ makes it simple.

There is a sample repo here: https://github.com/simonireilly/render-sorry-cypress

Results ¶

When hosting cypress in parallel for the default cypress test suite I performed a comparison on github actions:

Scenario	Cypress time	Total time
Cypress single runner	417 (s)	432 (s)
Cypress five runners	98 (s)	117 (s)

It's really a great deal faster to invest in running the test suite in parallel even with just 20 test files, there is a huge benefit to cycle time.

Sorry cypress running in github actions

The speed benefits are certainly worthwhile, and the scalability unlocked by a coordinator like sorry cypress, over a configured parallel setup, means low configuration complexity.

Background ¶

I have been using Cypress as a test runner since 2019, and enjoyed it's low barrier to entry.

It is probably well know that Cypress itself is not the fastest test runner, because it uses a full browser when running tests instead of other tools that might use a webkit only.

The advantages of Cypress are that it has a fast release cadence, adds new features frequently, and is intuitive.

To combat the slow test runtimes cypress provides a hosted director. The director issues batches of tests to each runner that joins the pool for that build id. It looks something like this:

Sorry cypress architecture, on the assumption that cypress themselves do something similar but at a much larger scale!

Running Sorry Cypress ¶

There are some setup instructions for self hosting sorry cypress here.

I wanted to try hosting Sorry Cypress on render.com because:

I have used render for multi-container apps before and it just makes sense.
Render has a good integration with GitHub which helps streamline setup.
The cost of running could be minute, something like:
- $7/month + $0.25/month to run a mongoDB instance
- $0/month for all other container running on demand; which works well for my use case
- $7.25/month total; well isn't that nice, for 3 docker containers and a DB.

Render Setup ¶

To run sorry cypress on render it's straight forward; here is a blueprint yaml file that will work for you:

The blueprint-spec is here, I recommend a read: render.com/docs/blueprint-spec

services:
  - type: web
    name: director
    env: docker
    healthCheckPath: /
    dockerfilePath: ./Dockerfile.director
    envVars:
      - key: DASHBOARD_URL
        fromService:
          name: dashboard
          type: web
          envVarKey: RENDER_EXTERNAL_URL
      - key: EXECUTION_DRIVER
        value: ../execution/mongo/driver
      - key: SCREENSHOTS_DRIVER
        value: ../screenshots/s3.driver
      - key: MONGODB_DATABASE
        value: sorry-cypress
      - key: MONGODB_URI
        fromService:
          name: mongo
          type: pserv
          envVarKey: MONGODB_URI

  - type: web
    name: dashboard
    env: docker
    healthCheckPath: /
    dockerfilePath: ./Dockerfile.dashboard
    envVars:
      - key: GRAPHQL_SCHemA_URL
        fromService:
          name: api
          type: web
          envVarKey: RENDER_EXTERNAL_URL
      - key: PORT
        value: 8080
      - key: CI_URL
        value: ""

  - type: web
    name: api
    env: docker
    dockerfilePath: ./Dockerfile.api
    envVars:
      - key: MONGODB_URI
        fromService:
          name: mongo
          type: pserv
          envVarKey: MONGODB_URI
      - key: MONGODB_DATABASE
        value: sorry-cypress

  - type: pserv
    name: mongo
    env: docker
    dockerfilePath: ./Dockerfile.mongo
    envVars:
      - key: MONGO_INITDB_ROOT_PASSWORD
        value: <password>
      - key: MONGO_INITDB_ROOT_USERNAME
        value: sorry-cypress
      - key: MONGODB_URI
        value: mongodb://sorry-cypress:<password>@<replace-with-pserv-address>:27017
    disk:
      name: data
      mountPath: /data/db
      sizeGB: 1

You will need to update the <password> before deploying the blueprint, and on first pass it will fail.

After you have mongo deployed you will need to edit the template setting <replace-with-pserv-address> to the network addressable name of your mongo instance:

The addressable endpoint for mongo is not known until deploy time; this would be fine if Sorry Cypress supported MONGO_HOST etc. but it expects a fully qualified URI, and render does not yet support env var interpolation

The benefit of render is that it will pick up on these changes, and notice the dependency chain between env var's exported from other services. It will re-deploy on each change, and reboot those services.

Running tests ¶

I have an example runner setup to use github actions, which was shown in the early part of the blog.

The setup itself was very straight forward.

Sorry Cypress dashboard showing my example project.

The runs themselves are reported to sorry cypress from the test runners and appear inside the examples. I decided to use the git sha for my test runs since it will be unique all of the time:

Sorry Cypress example test run in the sorry cypress dashboard

Wrap up ¶

I loved using sorry cypress, and hosting it on render was a lot easier than I expected.

Since render is setup to scale, I could see this being the most intuitive way to continue to use sorry cypress at a super low cost of $7.25/month; until my use case expands. At that point scaling up on render seems as easy as a few lines of yaml 👍

Typescript with the OpenAPI specification

Boaty McBoatFace — Sun, 23 Jan 2022 07:00:00 +0000

Summary ¶

OpenAPI, formerly swagger is a specification for RESTful interfaces that helps developers and engineers write and consume an API.

Typescript is a strongly typed programming language built on Javascript.

Most Typescript projects that utilise HTTP interfaces rely on an HTTP interface. There are many ways to create an OpenAPI specification for that interface, that is strongly typed to your Typescript.

I have conducted a review of the existing tooling, highlighting three ways that typescript and the OpenAPI specification can interact.

Conclusion ¶

If I had to pick, I would write my OpenAPI specification as a typed object, using typescript. Then I would want typechecking for my request/responses, but I would not want prescriptive code, and I would not use generation in the lifecycle, because it shouldn't be necessary.

Compeller ¶

So, because of that I am writing compeller, here it is in action.

A strong typescript binding for your OpenAPI Schema that doesn't need generation and is not prescriptive in coding style

Typescript - OpenAPI Mapping ¶

At a high level we might say that we can either be OpenAPI specification driven, or Typescript code driven. These two approaches provide the following options.

1️⃣ Define an OpenAPI specification, and from that, generate the Typescript types for the API domain.

2️⃣ Use decorators on Typescript to produce an OpenAPI specification from the API domain.

There is a wildcard, type 3, where you an bind the typescript to the OpenAPI document by placing all or parts of the OpenAPI specification inside typescript itself. This approach does not use generation, but instead uses type inference to ensure the types and their schema are in agreement.

3️⃣ Combine the OpenAPI specification, and API domain types, into a single Typescript object.

1️⃣ OpenAPI => Typescript ¶

The team behind the OpenAPI specification provided an example here.

We have our specification file from the above link. So lets review some of the tools we can ue to get our types from the specification:

OpenAPI Generator ¶

Link: https://github.com/OpenAPITools/openapi-generator

Generate clients, and stub servers in multiple languages from the OpenAPI specification.

Generate client
Generate server - No TS support
Generate types

Usage ¶

The below example creates a typescript API Client.

docker run \
  --rm \
  -v "${PWD}:/local" \
  openapitools/openapi-generator-cli generate \
  -i /local/spec.json \
  -g typescript-axios \
  -o /local/openapi-generator/ts

The created client will dutifully represent your types; here is the NewPet schema as an interface, automatically generated by the tool:

/**
 *
 * @export
 * @interface NewPet
 */
export interface NewPet {
  /**
   *
   * @type {string}
   * @memberof NewPet
   */
  name: string;
  /**
   *
   * @type {string}
   * @memberof NewPet
   */
  tag?: string;
}

There is not yet support for generating a typescript server.

OpenAPI Typescript Codegen ¶

Link: https://github.com/ferdikoomen/openapi-typescript-codegen

Produce typescript api clients from OpenAPI specification. Build tooling is not Java

Supports:

Generate typescript clients

Usage ¶

You can generate a client from the specification with:

npx openapi-typescript-codegen \
  --input ./spec.json \
  --output ./openapi-typescript-codegen \
  --exportSchemas true

The provided client can be used to fetch from the API:

import { PetsService } from "./index";

const pets = await PetsService.listPets(5);

In this instance pets will be typed either as Pet | Error after the fetch promise resolves.

Openapi Typescript ¶

Link: https://github.com/drwpow/openapi-typescript

Produce typescript interfaces, from OpenAPI schema

Supports:

Generate types

Usage ¶

The below example creates Typescript types and interfaces that represent the API Document.

npx openapi-typescript spec.json \
  --output ./openapi-typescript/schema.ts

The created types include paths, operations, and components. Here is an example of the paths:

export interface paths {
  "/pets": {
    get: operations["listPets"];
    post: operations["createPets"];
  };
  "/pets/{petId}": {
    get: operations["showPetById"];
  };
}

The operations section is automatically generated. I can imagine this would be very useful.

You can build a simple response handler, that is not coupled to any particular framework like so:

import { operations } from "./schema";

type showPetById = operations["showPetById"];

const response = <K extends keyof showPetById["responses"]>(
  statusCode: K,
  body: showPetById["responses"][K]["content"]["application/json"]
) => {};

This binds the response codes to the response objects, which are inturn bound to your OpenAPI schema via the generation.

2️⃣ Typescript => OpenAPI ¶

In this scenario we have our Typescript code, for our server, and we want to create an OpenAPI specification.

These decorator and binding methods require you to write your code in a specific way, which is restrictive, and not adoptable for existing large projects.

NestJS ¶

link: https://docs.nestjs.com/openapi/introduction

NestJS is a progressive typescript framework for building backend applications.

Supports:

Rendering documentation
Adding docs to controllers using decorators
Express and Fastify support

The documentation for this tool is extensive, so I will leave it to their talented team to explain the usage.

TSOA ¶

link: https://tsoa-community.github.io/docs/

TSOA generates routes from your Typescript code for koa, fastify or express

Rendering documentation
Adding docs to controllers using decorators
Express and Fastify support

This tool has code bindings and decorators, but also makes use of generation, so there is a lot of configuration overhead.

3️⃣ Typescript <=> OpenAPI ¶

This collection of tools bring the OpenAPI specification domain into the typescript typing system.

This gives some benefits when developers are more used to receiving type hints. Whilst it prevents the schema from being a portable JSON document, you can always export the javascript object to a JSON or YAML file.

This method is in my opinion less opaque, as it does not use generation like method 1️⃣, and less prescriptive than method 2️⃣, because you can write your code however you should like.

Basically I am all in on method 3️⃣!

OpenAPI3 TS ¶

Usage ¶

The extensive type system of OpenAPI3-TS attempts to be true to the OpenAPI specification.

const spec: OpenAPIObject = {
  info: {
    title: "Mandatory",
    version: "1.0.0",
  },
  openapi: "3.0.3",
  paths: {
    "/v1/version": {
      get: {
        responses: {
          "200": {
            description: "example",
            content: {
              "application/json": {
                schema: {
                  type: "string",
                },
              },
            },
          } as ResponseObject,
        },
      },
    } as PathItemObject,
  },
};

This has advantages over other type systems, because it will require a diligent documentation of the types to ensure the compiler agrees.

AJV `JSONSchemaType` ¶

Link: https://ajv.js.org/guide/typescript.html#utility-types-for-schemas

This utility type can take the API models that you declare as type aliases, or interfaces, and infer a fully JSON schema for the type.

Supports

Inferring schema type from object type.

Usage ¶

This example is borrowed from te documentation.

import Ajv, { JSONSchemaType } from "ajv";
const ajv = new Ajv();

interface MyData {
  foo: number;
  bar?: string;
}

const schema: JSONSchemaType<MyData> = {
  type: "object",
  properties: {
    foo: { type: "integer" },
    bar: { type: "string", nullable: true },
  },
  required: ["foo"],
  additionalProperties: false,
};

// validate is a type guard for MyData - type is inferred from schema type
const validate = ajv.compile(schema);

// or, if you did not use type annotation for the schema,
// type parameter can be used to make it type guard:
// const validate = ajv.compile<MyData>(schema)

const data = {
  foo: 1,
  bar: "abc",
};

if (validate(data)) {
  // data is MyData here
  console.log(data.foo);
} else {
  console.log(validate.errors);
}

The benefit of this method, is there is a runtime binding between the schema validation, and the typing system.

JSON Schema to TS ¶

Link: https://github.com/ThomasAribart/json-schema-to-ts

Given a schema object in typescript, the type it represents can be inferred.

Supports:

Inferring Schema to Object type

Usage ¶

The FromSchema type requires the schema to be made a const, and this means the schema itself is missing type hints.

import { FromSchema } from "json-schema-to-ts";

export const VersionSchema = {
  type: "object",
  required: ["version"],
  additionalProperties: false,
  properties: {
    version: {
      type: "string",
    },
  },
} as const;

export type Version = FromSchema<typeof VersionSchema>;

This is a little less convenient as you still need to know the OpenAPI specification, and the type will resolve to never with no error if you get it wrong.

Closing ¶

Thanks for reading this review, if there are any issues with the links let me know 👍

Build your own turborepo remote cache

Boaty McBoatFace — Thu, 30 Dec 2021 07:00:00 +0000

I have been looking at turborepo, the new fast build tool for javascript/typescript monorepos.

Read on for a brief overview of the space, some tools that are also vying for your attention, and a short code example for creating your own remote-cache for turborepo.

What is turborepo ¶

Turborepo is a high-performance build system for JavaScript and TypeScript code bases. https://turborepo.org/

It's a Golang devtool that uses directed acyclic graph this module actually, to accelerate builds with clever caching strategies, and non sequential dependency management.

This DAG space is hotting up, you might know it from GitLab Aug 2019 release, and its about ready to burst onto the scene in a big way with dagger bringing it to the masses.

Turborepo packages this graph technology up nicely, and succeeds in lowering the barrier to entry, targeting Javascript/Typescript mono-repos and their maintainers. Its been so successful that vercel scooped it up this month.

There are still more mature tools in this space, https://nx.dev/ is 4 years old, so there is definitely stiff competition.

So will turborepo blow up in 2022, absolutely it will, it's a smooth developer experience, with some big name backers.

Remote caching ¶

Remote caching is just taking some folders stored locally in ./node_modules/.cache/turbo; creating a zipped tarball (a group of files collected together as one) of those files, and sending it over the wire with a deterministic cache key.

Turborepo itself warns of some dangers when doing this:

Remote Caching is a ... Make sure you are caching correctly first and double check handling of environment variables. Please also remember Turborepo treats logs as artifacts, so be aware of what you are printing to the console. https://turborepo.org/docs/features/remote-caching

We all can be vulnerable to these kinds of mistakes, like accidentally committing an SSH key or API Key to GitHub etc. The stakes are higher when the tools are less transparent, and turborepo understand this, hence the warning.

Remote cache options ¶

Vercel are offering free remote caching at the moment. However, turborepo welcomes us to build our own remote caches, and this can be more secure, as you are controlling the location of your cache, and its encryption.

So a quick remote cache can be built with some serverless technologies but there are other implementations here:

https://github.com/Tapico/tapico-turborepo-remote-cache
- Docker hosted, with Google Cloud Storage or AWS S3
https://github.com/cometkim/turbocache
- A cloudflare workers implementation, this one is really quite neat, and will be super cost effective

Make your own remote cache in AWS ¶

AWS is a big cloud, and since it has a unified auth plane (IAM) it can be a secure place to build your dev-tools. We can then use AWS IAM to enable access to the remote cache(s) locally, or in CI, without having a long lived token. But for the time being that is not an option, so consider a custom authorizer, or a JWT authorizer.

I have used simple Auth using a plaintext header, and for simplicity there is no authorizer being used. The source code is here https://github.com/simonireilly/turbo-remote

Key Takeaways ¶

Tools implementing DAG's are becoming more common, and easier to interface with as a developer.
Turborepo is one of these tools, and it does a good job of lowering the barrier to entry.

Bootstrap the AWS CDK with GitHub Actions

Boaty McBoatFace — Sun, 12 Dec 2021 07:00:00 +0000

In this guide I will talk about getting started with the aws-cdk while following security best practices.

You will need:

A github repository
An aws account
NodeJS 14

Securing Deployments ¶

Open ID Connect can be used to link GitHub Actions to your target AWS account, without sharing long lived credentials.

I explain it below in this diagram:

Open ID Connect provider is added as an identity provider to IAM in your account.
When we wish to deploy to AWS we exchange the GitHub JWT for temporary AWS credentials.

This is more secure that the common approach of storing a super credential in the GitHub Secrets per environment.

Background Reading ¶

There are a couple of guides on this that I have read, you don't need to read these, use them as references.

Setup from GitHub side showing how to control access scope

Setup on AWS Side manually or with CLI (We use CDK)

These should be considered as fairly raw indigestible information.

I will simplify this here and show how to set it up easily with the AWS-CDK.

Setup an AWS CDK V2 Project ¶

Practical Guide using Infrastructure as Code, lets begin with a new cdk v2 app.

mkdir bootstrap

npx aws-cdk@2.x init app --language typescript

If you are just getting started with the AWS-CDK you will need to bootstrap it.

yarn cdk bootstrap

The CDK role and te CDK custom resources should be setup by this deployment.

Add Open ID Connect (OIDC) ¶

You can configure the OIDC provider in AWS CDK but I have explained this in a set of snippets that you need to place in bootstrap/lib/<your-stack>.ts

/**
 * Create an Identity provider for GitHub inside your AWS Account. This
 * allows GitHub to present itself to AWS IAM and assume a role.
 */
const provider = new OpenIdConnectProvider(this, "MyProvider", {
  url: "https://token.actions.githubusercontent.com",
  clientIds: ["sts.amazonaws.com"],
});

Then establish the trust relationship by defining the conditions for this provider to act as a principal.

I will provide an example that assumes you are https://github.com/simonireilly and you want to deploy from all repository branches of a repo called awesome-project

const githubOrganisation = "simonireilly";
// Change this to the repo you want to push code from
const repoName = "awesome-project";
/**
 * Create a principal for the OpenID; which can allow it to assume
 * deployment roles.
 */
const GitHubPrincipal = new OpenIdConnectPrincipal(provider).withConditions({
  StringLike: {
    "token.actions.githubusercontent.com:sub": `repo:${githubOrganisation}/${repoName}:*`,
  },
});

Finally you want to establish the role that can be assumed by the OIDC principal. This will allow GitHub actions to use the AWS Roles, and mutate the AWS Resources you give it access to.

/**
 * Create a deployment role that has short lived credentials. The only
 * principal that can assume this role is the GitHub Open ID provider.
 *
 * This role is granted authority to assume aws cdk roles; which are created
 * by the aws cdk v2.
 */
new Role(this, "GitHubActionsRole", {
  assumedBy: GitHubPrincipal,
  description:
    "Role assumed by GitHubPrincipal for deploying from CI using aws cdk",
  roleName: "github-ci-role",
  maxSessionDuration: Duration.hours(1),
  inlinePolicies: {
    CdkDeploymentPolicy: new PolicyDocument({
      assignSids: true,
      statements: [
        new PolicyStatement({
          effect: Effect.ALLOW,
          actions: ["sts:AssumeRole"],
          resources: [`arn:aws:iam::${this.account}:role/cdk-*`],
        }),
      ],
    }),
  },
});

Deploying Bootstrap ¶

In CI/CD we want the bootstrapping process to be more secure, because it can create a trust relationship.

So for this I create a .github/workflows/bootstrap.yml which requires privileged account credentials to perform the deployment.

name: Bootstrap
on:
  workflow_dispatch:
    inputs:
      AWS_ACCESS_KEY_ID:
        description: "Access Key ID with Permissions to deploy IAM, and OIDC"
        required: true
      AWS_SECRET_ACCESS_KEY:
        description: "Secret Access Key with Permissions to deploy IAM, and OIDC"
        required: true
      AWS_REGION:
        description: "Region to deploy to."
        required: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Git clone the repository
        uses: actions/checkout@v1

      - name: Configure aws credentials
        uses: aws-actions/configure-aws-credentials@master
        with:
          aws-access-key-id: $
          aws-secret-access-key: $
          aws-region: $

      - uses: actions/setup-node@v2
        with:
          node-version: "14"

      - run: yarn install

      - name: Synth stack
        run: yarn --cwd packages/bootstrap cdk synth

      - name: Deploy stack
        run: yarn --cwd packages/bootstrap cdk deploy --require-approval never

This can now be run by someone in your org, that has higher level access.

Post Bootstrap life ¶

With this stack deployed you can now ship any aws-cdk v2 deployments from the trusted repository, to the linked AWS account, without storing long lived credentials.

All you need to do is instruct GitHUb actions to assume the github-ci-role role in your account, and it will get temporary credentials for one hour.

deploy-infrastructure:
  runs-on: ubuntu-latest
  steps:
    - name: Git clone the repository
      uses: actions/checkout@v1

    - name: Assume role using OIDC
      uses: aws-actions/configure-aws-credentials@master
      with:
        role-to-assume: arn:aws:iam::<your-account-id-here>:role/github-ci-role
        aws-region: $

    - uses: actions/setup-node@v2
      with:
        node-version: "14"

    - run: yarn install

    - name: Synth infrastructure stack
      run: yarn --cwd packages/infrastructure cdk synth

    - name: Deploy infrastructure stack
      run: yarn --cwd packages/infrastructure cdk deploy --require-approval never

Next steps might to be:

Create a series of roles for different accounts with trust relationships like:

Only trust main branch for your production account with condition:

{
  StringLike: {
    'token.actions.githubusercontent.com:sub':
      `repo:${githubOrganisation}/${repository}:ref:/refs/head/main`,
  },
}

Follow up ¶

If you are interested in this stuff, you might like microteams!

A guide I am writing for scale ups, that are growing from one person AWS start-ups to multi-team organisations.

Fullstack observability with AWS CDK V2 Canary's and Synthetics

Boaty McBoatFace — Thu, 09 Dec 2021 07:00:00 +0000

Version 2 of the AWS Cloud Development Kit (CDK) shipped a couple weeks ago.

As part of that release AWS have separated stable and experimental cdk libraries. In this blog I am going to build out with some of te new tools and features to learn a little about how to use experimental constructs, and also to share some observability tooling setups for monitoring your websites.

In this blog ¶

Build a website with AWS CDK
Create a canary to continuously monitor the websites up time
Create a cloudwatch dashboard as a single source of truth for observability
Add Real User Monitoring to the website to get detailed user journeys and page load analytics

TL; DR ¶

The canary stack can be found here: https://github.com/simonireilly/canary-stack

Once deployed you will have a Dashboard and Alarm for an S3 bucket website.

The architecture that is built once complete is:

Starting a cdk v2 project ¶

Lets begin.

I do the following to get the project configured:

mkdir canary

cd canary

npx aws-cdk@2.x init app --language typescript

This creates the basic structure.

Previous modules would ship separately, this is still the case for experimental modules.

Adding Experimental modules ¶

To add an experimental module you need to install it with yarn or npm.

yarn add @aws-cdk/aws-synthetics-alpha

You know, I could not recommend cdk-dia enough, with will give you a nice diagram of your stack:

yarn add --dev cdk-dia

Building a Website with canaries ¶

Get started by adding a website to your stack:

mkdir -p lib/website

touch lib/website/index.html

In there you need some HTML, this will do:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>Your website</title>
    <meta name="viewport" content="width=device-width, initial-scale=1" />
  </head>
  <body>
    <main style="display: flex; flex-direction: column; align-items: center;">
      <h1>Welcome to the canary stack</h1>

      <p>This is a simple single page website that show cases the aws-cdk.</p>
    </main>
  </body>
</html>

Now you can deploy your website from the stack using 2 AWS buckets, one to hold the zipped files, and another to deploy as a public website.

import { Bucket } from "aws-cdk-lib/aws-s3";
import { BucketDeployment, Source } from "aws-cdk-lib/aws-s3-deployment";
import { Construct } from "constructs";
import { Stack, StackProps, CfnOutput } from "aws-cdk-lib";
import * as path from "path";

export class CanaryStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const websiteBucket = new Bucket(this, "WebsiteBucket", {
      websiteIndexDocument: "index.html",
      publicReadAccess: true,
    });

    new BucketDeployment(this, "DeployWebsite", {
      sources: [Source.asset(path.join(__dirname, "website"))],
      destinationBucket: websiteBucket,
    });

    new CfnOutput(this, "WebsiteUrl", {
      value: websiteBucket.bucketWebsiteUrl,
    });
  }
}

The output from this will reveal your website URL:

 ✅  CanaryStack

Outputs:
CanaryStack.WebsiteUrl = http://canarystack-websitebucket75c24d94-z8brk94c8nvu.s3-website-eu-west-1.amazonaws.com

Adding a Canary ¶

We want to be able to monitor our websites up time, to do that we can add a canary.

The canary code looks like this:

const canary = new Canary(this, "MyCanary", {
  schedule: Schedule.rate(Duration.minutes(5)),
  test: Test.custom({
    code: Code.fromAsset(path.join(__dirname, "canary")),
    handler: "index.handler",
  }),
  runtime: Runtime.SYNTHETICS_NODEJS_PUPPETEER_3_3,
  environmentVariables: {
    SITE_URL: websiteBucket.bucketWebsiteUrl,
  },
});

You need to be specific about where you put the canary javascript. From what I can tell it follows the same patter as when building AWS layers.

mkdir -p lib/canary/nodejs/node_modules

touch lib/canary/nodejs/node_modules/index.js

Now you want to put the canary javascript into that file like so:

const synthetics = require("Synthetics");
const log = require("SyntheticsLogger");

const pageLoadBlueprint = async function () {
  // Configure the stage of the API using environment variables
  const url = String(process.env.SITE_URL);

  const page = await synthetics.getPage();
  const response = await page.goto(url, {
    waitUntil: "domcontentloaded",
    timeout: 30000,
  });
  // Wait for page to render. Increase or decrease wait time based on endpoint being monitored.
  await page.waitFor(15000);
  // This will take a screenshot that will be included in test output artifacts.
  await synthetics.takeScreenshot("loaded", "loaded");
  const pageTitle = await page.title();
  log.info("Page title: " + pageTitle);
  if (response.status() !== 200) {
    throw "Failed to load page!";
  }
};

exports.handler = async () => {
  return await pageLoadBlueprint();
};

Now we can deploy again:

yarn cdk deploy

This time we will have a canary to go and see in the dashboard which looks like this:

Adding Alerts ¶

Monitoring is hard, but it doesn't need to be hard to implement.

The CDK can make creating a dashboard and an alarm for the canary very easy.

Here is the code you need:

const alarm = new Alarm(this, "CanaryAlarm", {
  metric: canary.metricSuccessPercent(),
  evaluationPeriods: 2,
  threshold: 90,
  comparisonOperator: ComparisonOperator.LESS_THAN_THRESHOLD,
});

const alarmWidget = new AlarmWidget({
  alarm,
  title: "Canary Alarm",
});

const dashboard = new Dashboard(this, "MainDashbaord", {
  dashboardName: "Main-Dashboard",
  periodOverride: PeriodOverride.AUTO,
  widgets: [[alarmWidget]],
});

Thats all it takes to get setup with a dashboard for your site.

Adding Real User Monitoring ¶

AWS just launched Real User Monitoring (RUM). This is a client side tool that tracks the real user journey and page load analytics.

You can read about RUM in Jeff Barr's announcement blog post: https://aws.amazon.com/blogs/aws/cloudwatch-rum/

You should follow the wizard to set this up and chose the option to get a new cognito user pool.

Then you can copy the snippet produced into the <head> tag within the lib/website/index.html.

Now we deploy again:

yarn deploy

When we go to the rum dashboard we will see all the stats for a users session:

Closing ¶

With a canary in place you have:

A simple health check for your web application.
You can now layer more features onto the canary, such as:
- User login using aws-sdk and cognito
- Visual regression testing
In the future

AWS CDK Utils

Boaty McBoatFace — Tue, 16 Nov 2021 07:00:00 +0000

In this blog I am launching a small open source project, aws-cdk-utils.

The aws-cdk-utils is a collection of fringe constructs that I have found useful to re-use.

Vercel Secret Forwarder (Canary ) ¶

🦫 NPM: https://www.npmjs.com/package/@cdk-utils/vercel-secret-forwarder

🦑 GitHub: https://github.com/simonireilly/aws-cdk-utils

The vercel secret forwarder allows sending environment variables to Vercel, directly from your AWS account.

This means we you can follow security best practices, with a minimal configuration.

Features

🔒 Send secrets to Vercel securely from AWS
🧪 Support for preview deploys
🛠️ Utilities for binding Vercel URL(s)

Example ¶

Lets say you have a CDK app, that deploys an API Gateway rest API, and a Cognito user pool.

The secrets you need in Vercel, when you build your app are:

API_GATEWAY_URL
NEXT_PUBLIC_COGNITO_CLIENT_ID

How can we get these into vercel??

Remember, if you are using preview deploys these environment variables will be different, and we want to keep them in sync.

Well, you can just add the forwarder to your aws-cdk or serverless-stack project.

yarn add @cdk-utils/vercel-secret-forwarder@canary

Next, you can import the construct into your own stacks, and forward any configuration or environment variables to your preview deploys.

This enables keeping a preview environment in sync with a Vercel preview deployment, supporting fulls task preview environments with the best experiences from both the CDK and Vercel.

// You will need to get these values
// To get vercel use `vercel link` or starting fresh use `vercel init`
const config = {
  GitBranch: String(process.env.GIT_BRANCH),
  VercelAuthToken: String(process.env.VERCEL_AUTH_TOKEN),
  VercelProjectId: String(process.env.VERCEL_PROJECT_ID),
  VercelProjectName: String(process.env.VERCEL_PROJECT_NAME),
  VercelProjectOrganisation: String(process.env.VERCEL_ORGANISATION_NAME),
};

// Create the construct, and any secrets you need.
const vercel = new VercelSecretSyncConstruct(this, "SendSecretsToVercel", {
  ...config,
  VercelEnvironmentVariables: {
    API_GATEWAY_URL: api.url,
  },
});

// Or add secrets later so you can define secrets across stacks
vercel.addSecret(
  "NEXT_PUBLIC_COGNITO_POOL_ID",
  auth.cognitoUserPool?.userPoolId
);

What does it deploy to my AWS Account ¶

Ok, security is important, the @cdk-utils/vercel-secret-forwarder is aiming to have zero dependencies before moving out of alpha. Additionally it does not need to be granted any IAM access to your AWS account. The lambda runs in AWS's great big VPC play pen, with no access to your own VPC. It takes in messages from cloudformation, and sends secrets out to vercel using HTTPS/TLS over the internet; nothing more.

You perform an AWS-CDK deployment which generates cloudformation for your resources.
This provisions a lambda function, which has no access to any other resources in your AWS Account.
Next, each resolved VercelEnvironmentVariable is sent to vercel to create a preview environment variable.

By default all additional environment variables are configured as preview, and when the stack is torn down these secrets are removed.

Wrap Up ¶

This is an exciting experiment for me, and hopefully it can be useful for others using the constructs.

The @cdk-utils are all in alpha, and I am routinely publishing under canary. When I reach v0.0.1 I will be publishing a latest tag release. The main goal is to remove as many dependencies before getting to that point.

Fine Grained Access Control with Cognito Identity Pools

Boaty McBoatFace — Mon, 15 Nov 2021 00:00:00 +0000

This blog covers configurations I have been using to build the saas-stack. A multi-tenant boiler plate for Serverless Stack apps.

When building that application I wanted to push the boundaries of Cognito multi-tenancy. I want to talk directly to dynamoDB from the browser for reads and be able to cache all my API Gateway GET requests.

But I ask you to think, how could you do this multi-tenant safe? See below for the results:

Example website: https://d1mmk10lcvudwp.cloudfront.net

In the beginning ¶

For a long time my brain has been mulling over multi-tenancy in AWS. There are a couple different ways to do Multi-Tenancy in AWS; the first decision:

Create an account per tenant with no share infrastructure.
Have multiple tenants in one AWS account.

This blog only thinks about number 2.

So if we want to share an AWS Account we still have some options:

Create a new resource (API Gateway, DynamoDB) per tenant.
Share common resources and use IAM to perform strict data segregation.

🧠 Data segregation in an AWS account has tripped up both of my latest engagements, it takes some pre-meditation to say the least!

This blog only thinks about number 2.

So we are going to have shared infrastructure in a single AWS Account. How will we secure everything?

IAM to the rescue 🦸

AWS IAM ¶

AWS IAM has a concept called fine-grained access control.

This type of Role is designed, with regard to allowing specific Principals (what AWS calls the entities assuming the Role), to access specific Resources.

At this point, I think we need to hit reset and introduce:

AWS IAM Roles
AWS IAM Principals
AWS IAM Trust Relationships

AWS IAM - Basics ¶

If we think of a straight forward example, here, is a lambda function.

The lambda function has a Role, that it needs to assume, in order to do all its lambda things. Specifically, if the lambda needs to interact with other AWS services, it needs to have a Role that allows it to do that.

The lambda function, wants to assume a role, and so, we setup a trust relationship, that says, when the Principal is lambda, then it can assumed the Role.

The lambda then gets access to all the Actions in the Role, for all of the Resources in the Role.

AWS IAM - Fine grained Access Control ¶

Fine grained control adds another layer to this relationship.

With fine grained control, we go beyond relying solely on the principal and add conditions. These conditions can be added to both the trust policy, and the iam policy.

Here is an example:

Summarizing the above:

Cognito Identity wants to assume the IAM policy, and this is allowed if:
- The "cognito-identity.amazonaws.com:aud" string is exactly equal to "eu-west-2:e43159e7-b1bd-4cd2-8cbf-xxxxxxxxxxxx".
- The "cognito-identity.amazonaws.com:amr" has any of its values as "authenticated".

AWS IAM - Principal Tag Mapping ¶

There is a final piece to the puzzle, Principal Tags which AWS defines as...

Control what the person making the request (the principal) is allowed to do based on the tags that are attached to that person's IAM user or role. [1]

In this use case with Amazon Cognito, we are specifically going to associate Principal Tags to the JSON Web Token (JWT) claims that are returned by Cognito.

These policies enhance the Fine Grained case with conditions of the type:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Condition": {
        "ForAllValues:StringLike": {
          "dynamodb:LeadingKeys": [
            // Wait wait wait
            // what the heck is this thing?
            "${aws:PrincipalTag/org}#*"
          ]
        }
      },
      "Action": ["dynamodb:GetItem", "dynamodb:Query"],
      "Resource": "::dynamodb:eu-west-2::table/ExampleTableName",
      "Effect": "Allow",
      "Sid": "AllowPrecedingKeysToDynamoDBOrganisation"
    }
  ]
}

Now, all we need to do, is setup cognito to map a principal tag.

Then, when a user assumes this role, they would only be able to see things in DynamoDB table ExampleTableName which have a primary key starting with ${aws:PrincipalTag/org}#

Putting it all together ¶

Please see the saas-stack for a full working Infrastructure as Code example.

A user signs up, and gets saved in cognito user pool.
The user gets the auto-confirm email.
A post-confirmation lambda is triggered
The post-confirmation lambda tells cognito to insert the UUID of the organisation as an immutable custom attribute on the user.
The user exchanges their JWT for temporary credentials associated with the identity pool role.
The policy attached to those credentials is already secured to only allow access to PUBLIC resources and the tenants own resources (using the org uuid); nothing else!
The tenant can make requests to API Gateway, and DynamoDB, but will have restricted access.

Wrap up ¶

I have been working on this for a while, and I am still waiting for an upstream change in the aws-cdk to make it fully supported and easy to configure: https://github.com/aws/aws-cdk/issues/15908

Once that hits the stable CDK builds, I think this will be a much more secure way to perform IAM, and keep your lambda code clean.

There is a big but!

This is very deep integration into AWS, perhaps as deep as you can go, and its not going to be the most flexible approach; so take this with a pinch of salt 🧂

Coding by voice with python

Boaty McBoatFace — Sat, 13 Nov 2021 07:00:00 +0000

In the beginning... ¶

I am not a programmer to trade, but rather a chemical engineer.

I picked up programming for fun in 2014 while at University, and well, I liked it so I kept on doing it.

In May 2018 I got my first programming job, building a full-stack e-commerce platform with some of the most talented people I have ever met.

But in July 2018, I got a Repetitive Strain Injury (RSI). I spent 7 months recovering, but I only spent 5 days off work.

This blog is about RSI, coding by voice, and general well-being.

TL; DR ¶

Let's begin with the typescripts.

const title = "Coding by voice";

Below is a full list of the tools that made me productive when I had limited use of a keyboard and mouse:

Aenea - A dictation toolbox; which enables control of linux machines from a remote windows host
- I keep my own fork, because it is necessary for me to have access, I also keep my own grammars in there for common tasks.
Vimium - A chrome plugin that enables vim based navigation in the browser.
- Using a phonetic alphabet, Vimium will put a character on every hyperlink, and you can call them out to navigate.
VSCode - I still used a rich IDE, in fact, I would recommend not changing from the tools you are most comfortable with if you can.

Some neat references ¶

Hands free coding by Josh Comeau is a good write up about Talon Voice another tool for voice coding
Using python to code by voice by Tavis Rudd is one of the original and best examples of what is possible

Setup ¶

Aenea was fundamentally the key to my success, without it I do not know where I would be.

This server runs locally on a Linux machine, Ubuntu in my case, while a companion client runs inside a windows Virtual Machine (VM). It's outside the scope of this blog to provide the system architecture so, why not instead enjoy the GIF below of me using the system.

Aenea was easy to set up in my case, but I could see how it may prove difficult. I identified early on that the VM was the complex value adding component, and I have it backed up on a physical disk and in the cloud. Now, I can run it on any device, from my super fast USB stick, and pull my Github repository of Aenea and my grammars. I can setup my voice coding environment on any device in just 10 minutes.

Grammar Development ¶

Grammar(s) - A collection of voice commands that map to machine commands or macros to execute My own definition

The Grammar is the important part, and each user seems to develop their own in my experience.

It is not simple to develop, because there are a number of interweaving concerns:

The language you use/speak:
1. In my case English, is not an infinite set, so possibilities are constrained. In reality, your imagine will run dry before you reach the limits of this set.
The programs you use:
1. Aenea is great for this, it has a deep integration in Linux, and can determine the Application context here is an example for how my Ruby grammar works:
```
from aenea import *

ruby_context = aenea.wrappers.AeneaContext(
    ProxyAppContext(match='regex', title='(?i).*(.rb).*'),
    AppContext(executable='Ruby')
    )
```
  This means that my Ruby specific Grammar is only activated when inside ruby files. I don't advocate for having the same command do multiple things in different context, but having specific contexts limits the possible outcomes in the recognition engine running in the windows virtual machine. Meaning it is less likely to misinterpret your intent.

What makes a grammar ¶

In 2018 I was predominantly a Ruby on Rails developer, and so I had some handy utterances I could drop into my microphone. If I was in the Linux terminal, the commands below would be recognised, otherwise, they would do nothing.

# A mapping syntax for applications command line

from aenea import *

class ShiftRule(MappingRule):
    mapping = {
            # Launching the application

            'shifty boot': Text('./bin/dev/boot') + Key('enter'),
            'shifty yawn': Text('./bin/dev/yarn '),
            'shifty post': Text('psql -p 5432 -h localhost -U postgres'),
            'shifty console': Text('./bin/dev/console') + Key('enter'),
            'shifty test': Text('./bin/dev/rspec '),
            'shifty rails': Text('./bin/dev/rails '),
            'shifty bundle': Text('./bin/dev/bundle '),
            'shifty break': Text('./bin/dev/rake '),
        }
    extras = [
        Dictation("text"),
        IntegerRef("n", 1, 100),
    ]
    defaults = {
        "n": 1,
    }

Flow State ¶

As a programmer you may want to attain a flow state, or not, its up to you?!

Popularized by positive psychologists Mihaly Csikszentmihalyi and Jeanne Nakamura, flow state describes a feeling where, under the right conditions, you become fully immersed in whatever you are doing. headspace.com

But when you have to adapt to a new situation, it can be difficult to find that familiar rhythm of productivity.

For me, the approach was all or nothing, and one of the first milestones was being able to extend my voice coding grammars, with my voice. This meant that anytime a new, repetitive task came up I could code a new macro by voice for it; say my keyword "Force field". This would reload the grammars within the remote windows machine in under 2 seconds, and I would now have access to the new command that I needed.

The future of voice coding ¶

I would love to see this level of accessibility begin to bleed into modern Integrated Development Environments.

My own pipe dream is an editor, like VSCode, where intellisense and language servers provide hints when voice coding. Inside VSCode, the intelligence is already in place for what it thinks you may type next. Could it be used to help reason around what it is you are most likely trying to say?

Instead of the up-front investment in grammar development, auto-complete would limit the set of possible values, and constrain the voice model, so we could arrive at likely intent. This would mean more people could dip their toes into coding by voice. Once there is a larger community, there would be faster advancement, and more experimentation, and so on and so forth until coding by voice would be something anyone could try for an hour, enjoy, and go back to the keyboard.

I am hopeful of that future.

cdktf compile optimization with swc

Boaty McBoatFace — Wed, 25 Aug 2021 07:00:00 +0000

Laying the grounds works, there are a number of tools in analysis here:

cdktf - Allows for defining terraform infrastructure as code in Typescript, Python or Golang.
typescript - And in particular tsc, the cli that transpiles typescript to javascript.
swc - A new js/ts transpiler that is gaining traction rapidly, and recently even made its way into NextJS test suite, with awesome results here

I am interested in using the cdktf, with typescript, to enhance infrastructure as code declarations for ease of use.

Seeing large projects take on the challenge of changing their compiler made me think if it was worth it; what are the pros and cons.

Summary ¶

Sample code can be found here: https://github.com/simonireilly/cdktf-swc

Using tsc --incremental is statistically faster than swc.
Using either swc or tsc --incremental will reduce time spent waiting by a factor of 5.

Trade-offs:

Pro - swc is fast on initial build
Pro - tsc --incremental is faster, after the first build
Con - swc does not produce type declarations

SWC vs. TSC ¶

swc was on average 5.6 times faster than tsc for synth(esizing) the terraform cdk stack.
swc was on average 2.5 times faster than tsc for plan(ing) the terraform cdk stack.

Compiler	Command	Mean [s]	Min [s]	Max [s]	Relative
swc	`yarn synth`	4.9 ± 0.2	4.7	5.2	1.0
tsc	`yarn synth`	28.0 ± 3.8	24.9	32.2	5.6 ± 0.8

TSC incremental vs. SWC ¶

tsc --incremental was on average 1.1 times faster than swc for synth(esizing) the terraform cdk stack.
tsc --incremental was on average 1.2 times faster than swc for plan(ing) the terraform cdk stack.

Compiler	Command	Mean [s]	Min [s]	Max [s]	Relative
swc	`yarn synth`	5.098 ± 0.439	4.661	5.540	1.07 ± 0.12
tsc --incremental	`yarn synth`	4.744 ± 0.341	4.360	5.008	1.00

Read on if you want to know about the implementation details

Setup ¶

There is a guide here I followed for getting started with cdktf: https://learn.hashicorp.com/tutorials/terraform/cdktf-build?in=terraform/cdktf

This isn't really the subject of the blog, but it is a nice intro to get you started if you plan on using the cdktf.

Experiencing slow synth ¶

I was doing some development on the train, and I was thinking, wow, this seems slow. When I discovered this was actually the compiler, not the network I was pretty shocked.

Digging under the hood there is actually some fairly good reasons this feels like a slow experience, and there are a number of solutions.

The cdktf was setup to re-compile the code, including all files in the .gen folder, on every synth.
The cdktf was setup with a tsconfig.json that wasn't using incremental builds (tsc keeps a build manifest to help it detect relevant changes and only recompile what is necessary).
The cdktf was setup with a tsconfig.json that would produce type declarations (*.d.ts); which is super useful if you are distributing your code for use in js environments. If you are not, and you are just using typescript, you might not need these.

Solution 1 ¶

Just set incremental to true in the tsconfig.json.

I am not going to lie, this is the simplest, and biggest win for compile time, and improves the overall performance straight away.

{
  "compilerOptions": {
    "incremental": true
  }
}

Solution 2 ¶

Since the initial build is still slow, another transpiler, swc, could be used to speed up builds:

yarn add --dev @swc/core @swc/cli

// .swcrc
{
  "jsc": {
    "parser": {
      "syntax": "typescript"
    },
    "target": "es2018"
  },
  "sourceMaps": true,
  "module": {
    "type": "commonjs",
    "strict": false,
    "noInterop": false,
    "strictMode": false
  }
}

Update the compile script to use swc:

{
  "scripts": {
    "compile": "swc main.ts --out-file dist/main.js & swc .gen/providers/**/*.ts -d dist/.gen/"
  }
}

Analysis ¶

I used hyperfine on my local machine:

$ lshw -short
H/W path       Device     Class          Description
====================================================
                          system         Latitude 7490 (081C)
/0/4d/0                   memory         16GiB SODIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
/0/4d/1                   memory         16GiB SODIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
/0/54                     processor      Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz

TSC --incremental vs. SWC ¶

Testing against incremental, we allow for the build to be present before each run:

hyperfine 'COMMAND=compile yarn synth' \
  'COMMAND=build yarn synth' \
  --export-csv results/complete.csv

TSC vs. SWC ¶

To test without incremental we perform cleaning of the dist directory between runs:

hyperfine 'COMMAND=compile yarn synth' \
  'COMMAND=build yarn synth' \
  --prepare 'yarn clean' \
  --export-csv results/complete.csv

Wrap-up ¶

Thanks for reading the methods, hopefully this is useful analysis if you are considering implementing swc of tsc --incremental.

Serverless Framework with Typescript and Source Maps

Boaty McBoatFace — Tue, 06 Jul 2021 17:51:54 +0000

The serverless framework is the original serverless deployment tool. It was my first introduction to serverless and a solid build tool. Lately I have been doing more terraform and aws-cdk to deploy mostly python and typescript.

In this instance, I would like to look at how to get setup with a typescript deployment to AWS Lambda using serverless framework.

Setup ¶

Run the following to get a serverless project

# Choose node js
npx serverless
# Add a package json
npm init --force
# Add dev dependencies
yarn add --dev serverless serverless-esbuild typescript @types/node @types/aws-lambda
# Setup the tsconfig
npx tsc --init

Next we can update the serverless.yml to include the plugin, and custom configuration for serverless-esbuild

# serverless.yml
service: esbuild-example

frameworkVersion: "2"

provider:
  name: aws
  runtime: nodejs12.x
  lambdaHashingVersion: 20201221
  environment:
    # From Node 12.x this option is available,
    # so support-source-maps package is not needed
    NODE_OPTIONS: "--enable-source-maps"

functions:
  hello:
    handler: handler.hello

plugins:
  # Plugin will register into the
  # serverless framework lifecycle hooks
  - serverless-esbuild

custom:
  # esbuild options go here
  esbuild:
    bundle: true
    minify: false
    sourcemap: external

Change the handler file from handler.js to handler.ts. Now we can deploy the package with this configuration:

yarn sls deploy

Now we can look at the deployed code in the lambda handler and see that there is a source map available:

Typescript Code ¶

So source maps would be useful, for example if we tried to perform a DynamoDB put request, and throw an error because we don't have permissions.

So we can add some code, and look at the debugging experience, with, and without source maps.

Adding some DynamoDB code; we can take the aws-sdk v3 for a spin:

yarn add  @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb @aws-sdk/util-dynamodb

The behavior of TCP Keep-alive is different in the AWS-SDK V3 for JS. Now, the default behavior is to keep connections alive (docs)

The default Node.js HTTP/HTTPS agent creates a new TCP connection for every new request. To avoid the cost of establishing a new connection, the SDK for JavaScript reuses TCP connections.

Previously this behavior was opt in with AWS_NODEJS_CONNECTION_REUSE_ENABLED=1.

Then in our handler we can put the client and document client together like:

import { APIGatewayProxyHandlerV2 } from "aws-lambda";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb";

const client = new DynamoDBClient({});
const DocumentClient = DynamoDBDocumentClient.from(client);

export const hello: APIGatewayProxyHandlerV2 = async (event) => {
  try {
    await DocumentClient.send(
      new PutCommand({
        TableName: "NoTable",
        Item: {
          pk: "not",
          sk: "real",
        },
      })
    );
  } catch (e) {
    // We can log our stack here to check that we get what is expected
    console.error(e);
  }

  return {
    statusCode: 200,
    body: JSON.stringify(
      {
        message: "Go Serverless v2.0! Your function executed successfully!",
        input: event,
      },
      null,
      2
    ),
  };
};

Logs Without Source Maps ¶

Now we can look at our logs and see how our source maps help, the first snippet here has the source maps disabled by removing the custom.esbuild.sourcemap attribute from the serverless.yml file

ERROR	AccessDeniedException: User: arn:aws:sts::322567890963:assumed-role/esbuild-example-dev-eu-west-1-lambdaRole/esbuild-example-dev-hello is not authorized to perform: dynamodb:PutItem on resource: arn:aws:dynamodb:eu-west-1:322567890963:table/NoTable
    at mR (/var/task/handler.js:1:121223)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async /var/task/handler.js:1:217049
    at async /var/task/handler.js:18:2644
    at async F_.retry (/var/task/handler.js:18:31612)
    at async /var/task/handler.js:18:53241
    at async /var/task/handler.js:18:189508
    at async Runtime.dH [as handler] (/var/task/handler.js:18:198204) {
  __type: 'com.amazon.coral.service#AccessDeniedException',
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: 'GAQAJ7NHOELF7P35KI2NQ434D7VV4KQNSO5AEMVJF66Q9ASUAAJG',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

Logs with Source Maps ¶

And with the source maps enabled with custom.esbuild.sourcemap: external the error log looks like below:

ERROR	AccessDeniedException: User: arn:aws:sts::322567890963:assumed-role/esbuild-example-dev-eu-west-1-lambdaRole/esbuild-example-dev-hello is not authorized to perform: dynamodb:PutItem on resource: arn:aws:dynamodb:eu-west-1:322567890963:table/NoTable
    at mR (/var/task/handler.js:1:121223)
        -> /node_modules/@aws-sdk/client-dynamodb/protocols/Aws_json1_0.ts:3625:39
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async /var/task/handler.js:1:217049
        -> /node_modules/@aws-sdk/middleware-serde/src/deserializerMiddleware.ts:18:20
    at async /var/task/handler.js:18:2644
        -> /node_modules/@aws-sdk/middleware-signing/src/middleware.ts:26:22
    at async F_.retry (/var/task/handler.js:18:31612)
        -> /node_modules/@aws-sdk/middleware-retry/src/StandardRetryStrategy.ts:83:38
    at async /var/task/handler.js:18:53241
        -> /node_modules/@aws-sdk/middleware-logger/src/loggerMiddleware.ts:22:22
    at async /var/task/handler.js:18:189508
        -> /node_modules/@aws-sdk/lib-dynamodb/src/commands/PutCommand.ts:72:20
    at async Runtime.dH [as handler] (/var/task/handler.js:18:198204)
        -> /handler.ts:10:5 {
  __type: 'com.amazon.coral.service#AccessDeniedException',
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '26DDD46C5137EV4SIK21F3EHCNVV4KQNSO5AEMVJF66Q9ASUAAJG',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

The second error log is much better, and also, the line numbers correspond to the source typescript code, not the transpiled and minified javascript; making it more easy to debug.

/handler.ts:10:5 ->
  await DocumentClient.send(new PutCommand({

Wrap-Up ¶

Serverless plugins continue to make it easy to use best practices when writing code for lambda. I was impressed with how easy esbuild was to integrate.

My own opinion is that configuring babel/webpack/parcel and others is undifferentiated heavy lifting by the definition Jeff Bezos used in 2006:

all of the hard IT work that companies do that doesn’t add value to the mission of the company

So its good to be able to avoid that (when possible), especially since the best code, is code never written 👍

Fast API with AWS-CDK

Boaty McBoatFace — Thu, 17 Jun 2021 07:00:00 +0000

In this blog we will use serverless-stack; with Typescript and Python to build an API Gateway HTTP API

API Gateway HTTP APIs are roughly a third of the cost of API Gateway REST APIs. They do not support all integrations and features of REST APIs, you can read more here: Choosing between HTTP APIs and REST APIs

Background ¶

I have been writing Python for serverless deployments to AWS Lambda for the last 8 months. One thing I miss from my previous time writing Typescript are type hints.

So I have been exploring ways to bring typing into Python; and looking at Python type hints with extended libraries like pydantic. I ended up reading this great article by Sebastián Ramírez last week, and was instantly inspired to put Python typing into action.

I have also been working a lot with serverless-stack, an abstraction layer over the AWS-CDK that offers an un-paralleled DEV experience (IMO). Out of the box you will have a local development environment that brings you closer to the code; and helps you iterate more quickly.

Problem Statement ¶

Given we are creating an MVP for a project, we have some requirements:

Create a Rest-fully routed HTTP API:
- The API must have documentation.
- The API must have authentication.
- The API should be scalable.

In AWS we have some main staples to use for these kind of tasks:

API Gateway, for creating a HTTP endpoint.
Lambda for handling scalable compute.
AWS IAM for authentication and authorization.

Part 1: Coding the API ¶

The source code is available in full here: https://github.com/simonireilly/sst-python-api

Setup ¶

The tools we need are:

Python
Node
A package manager like Yarn
An AWS account, configured with AWS cli so we have credentials for local deployment

Boot Strapping the Project ¶

We can begin in a new folder; creating a typescript stack for the api.

npx create-serverless-stack@latest \
  api-stack \
  --language typescript \
  --use-yarn

This will give us an initial serverless stack with constructs written in typescript. We do however want to use python as the programming language for our lambda functions. So, update the file api-stack/lib/index.ts, removing lines 6,7 and 8, so it looks like this:

// api-stack/lib/index.ts
import MyStack from "./MyStack";
import * as sst from "@serverless-stack/resources";

export default function main(app: sst.App): void {
  new MyStack(app, "my-stack");
}

Staying at this level, we need to set up our python virtual environment.

python3 -m venv env
source $PWD/env/bin/activate
pip3 install mangum fastapi pydantic

That is everything we need to get started. It is not a lot, but we will cover a great distance with these tools.

Spiking the API ¶

Inside the src folder there is currently a api-stack/src/lambda.ts file. This file is not needed, and we can replace that file with a new one, that will be python based.

.
├── api-stack
│   ├── src
│   │   └── api.py

And in that file, we can create an API like so:

# api-stack/src/api.py

from mangum import Mangum
from fastapi import FastAPI

APP = FastAPI(
    title="Example Test API",
    description="Describe API documentation to be served; types come from "
    "pydantic, routes from the decorators, and docs from the fastapi internal",
    version="0.0.1",
)


@APP.get("/v1/items")
def list_items():
    """
    Return a collection of items
    """
    return {
        'message': 'Hello, world!'
    }


handler = Mangum(APP, lifespan="off")

This code is our serverless-stack for the API; now we can look at deploying it.

In the below we specify that we will be deploying a lambda with the python3.8 runtime from the src directory. We have also define a proxy route, that will route all API requests to our API handler created above.

// api-stack/lib/MyStack.ts

import * as sst from "@serverless-stack/resources";

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props);

    // Create a HTTP API
    const api = new sst.Api(this, "Api", {
      defaultFunctionProps: {
        srcPath: "src",
        runtime: "python3.8",
      },
      routes: {
        "ANY /{proxy+}": "api.handler",
      },
    });

    // Show the endpoint in the output
    this.addOutputs({
      ApiEndpoint: api.url,
    });
  }
}

This architecture follows the Lambda-lith type pattern, where we essential have no AWS specific integration in our src/api.py. Later, we can transition to something that looks more like a Fat-Lambda.

Really there are many schools of thought on this topic. If you want to delve deeper then this is quite a nice round up that includes the thoughts of some serverless Hero's. I personally favour operational agility and every pattern has a use case, but its best to know the pros and cons.

Deploying ¶

Ok, we are going to deploy the stack to AWS using yarn:

yarn start

The output will be an API url. We can test the API using curl; with the url that was provided:

curl https://qpk7n59tz3.execute-api.us-east-1.amazonaws.com/v1/items | jq .
{
  "message": "Hello, world!"
}

How exciting it worked.

Looking under the hood, we have just deployed two cloud formation stacks, and these stacks work together so we can develop locally; read more here to understand how serverless-stack creates its live development environment

Lets test this out, we can add a print statement to our API handler to see it in action:

@APP.get("/v1/items")
def list_items():
    """
    Return a collection of items
    """
    print("Hello, local world!")
    return {
        'message': 'Hello, world!'
    }

And in our terminal we can see the output:

d2755156-e548-4e91-a5f3-e6a1733ef248 REQUEST dev-api-stack-my-stack-ApiLambdaANYproxyB463B5F1-cBeRkQMraTip [src/api.handler] invoked by API GET /v1/items
Hello, local world!
INFO:mangum.http:HTTPCycleState.REQUEST:  'http.response.start' event received from application.
INFO:mangum.http:HTTPCycleState.RESPONSE:  'http.response.body' event received from application.
d2755156-e548-4e91-a5f3-e6a1733ef248 RESPONSE {"isBase64Encoded":false,"statusCode":200,"headers":{"content-length":"27","content-type":"application/json"},"body":"{\"message\":\"Hello, world!\"}"}

Deploying the Docs ¶

Ok, so I lied, we don't need to deploy the docs; you can view the docs at the GET /docs URL of the API:

Adding an Item ¶

Our example API could do with a better use case. We can pretend that we are selling clothes, and write a model using pydantic that will act as a good example. Lets update the api.py file:

# api-stack/src/api.py

from typing import List, Optional
from mangum import Mangum
from fastapi import FastAPI

from pydantic import BaseModel, validator
from datetime import date, timedelta


class Item(BaseModel):
    """
    This model will have validation of the below fields
    """
    name: str
    description: Optional[str] = None
    price: float
    date: date

    @validator('date')
    def check_date_is_valid(cls, v):
        """
        We also have specific validation on the date
        """
        if v > date.today():
            raise ValueError('Date must be in the past.')
        if v < date.today() - timedelta(days=120):
            raise ValueError('Date must be within 120 days')
        return v


APP = FastAPI(
    title="Example Test API",
    description="Describe API documentation to be served; types come from "
    "pydantic, routes from the decorators, and docs from the fastapi internal",
    version="0.0.2",
)


@APP.get("/v1/items", response_model=List[Item])
def list_items():
    """
    Return a collection of items
    """
    return [
        Item(
            name="Purple Jumper",
            price=10.99,
            date=date.today()
        ),
        Item(
            name="Red Jumper",
            price=11.99,
            date=date.today() - timedelta(days=10)
        )
    ]

@APP.post("/v1/items", response_model=Item)
def create_item(item: Item):
    """
    Adds a new Item
    """
    return item


handler = Mangum(APP, lifespan="off")

We have updated our code, and it will all be hot reloaded, so now, we can look at the docs again:

Here we can see that the OpenAPI documentation has also been updated; and we even have a working execute to test the API 🎉.

Adding Authentication ¶

Our API is up and running, it has a schema, and documentation, but every route is currently public.

To secure the API, we can add a Lambda Authorizer to the HTTP API Gateway deployment. I think we should leave our GET /docs publicly accessible to encourage collaborators. Every other route can be secured.

We can use AWS IAM authentication to secure the private routes of the API.

Other options include using a Lambda Authorizer, to write fully custom authentication, it is worth reading the documentation for @aws-cdk/aws-apigatewayv2-authorizers if you have a deeper interest and then exploring the developer guide for AWS API Gateway HTTP API authorizers

We can configure the authorization by updating the stack:

// api-stack/lib/MyStack.ts

import * as sst from "@serverless-stack/resources";

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props);

    // Create a HTTP API
    const api = new sst.Api(this, "Api", {
      defaultFunctionProps: {
        srcPath: "src",
        runtime: "python3.8",
      },
      routes: {
        "ANY /{proxy+}": {
          function: {
            handler: "api.handler",
          },
          authorizationType: sst.ApiAuthorizationType.AWS_IAM,
        },
        "GET /docs": "api.handler",
        "GET /openapi.json": "api.handler",
      },
    });

    this.addOutputs({
      ApiEndpoint: api.url,
    });
  }
}

Part 2: Using the API ¶

This part is optional and has nothing specific to add to the code pattern, read on if you like, Postman, Non-Functional Testing, and graphs.

Now we have a fully scalable API; we can look at how to authenticate with AWS IAM.

We can use Postman to test the API; or we can use the AWS CLI; which will automatically create a signature to sign the requests that we make to invoke our API.

Given we have published our docs to a public link, and have used the OpenAPI specification we can import them directly into Postman.

That was pretty seamless, we can now setup authentication as-well. In the authentication tab, we need to supply AWS Access Key ID, and AWS Secret Access Key values that have permissions to invoke the API Gateway stage ($default). Then, Postman will handle generating the AWS V4 signature for authentication.

Firing a quick test API call, we can even see that our advanced validation returns errors that we have handled in our schema using the pydantic custom validator:

Production Deploy ¶

To get deployed to production, we need allow the Docker image that builds the python lambda to discover the dependencies we have installed.

To get the requirements file we can run the following:

pip3 freeze > ./api-stack/src/requirements.txt

This gives the full list of requirements to the underlying packager, and will ensure that our lambda environment has the correct dependencies.

Then, we can deploy the production stack.

yarn deploy

And we have finished up for now.

Load Testing ¶

I'd like to stress this API now, and look at how it will perform for a medium scale.

API Load Test ¶

K6s was used to stress test for a duration of 1 minute and 30 seconds.

AWS Statistics Under Load Test ¶

A round up using some metrics from AWS:

Metric	Average	Max
Lambda Concurrent executions	N/A	5
Lambda Duration	3.9 ms	45 ms

Wrap Up ¶

I found this to be an interesting pattern to explore. If you are looking to do something similar in NodeJS, then maybe check out serverless express.

Thanks for reading.

Part 2 - Building Realtime Multiplayer Games with React and Roomservice

Boaty McBoatFace — Sun, 25 Apr 2021 07:00:00 +0000

In this blog series I am going to create a realtime multiplayer game.

Features ¶

100% HTML and CSS game running in the browser for mobile, tablet and desktop. PART 1
Realtime Multiplayer and Player vs. Player. PART 2
Authentication and login for players.
Lobbies for private and public games with unique shareable codes.

Part 2 ¶

Welcome back, or, welcome.

If you did miss part 1 of the game then you can grab the source code for the game from here:

Lets have a refresher anyway:

We created a board, player, and controls
Connect the controls to the player
Tested it on a number of devices to ensure it was responsively designed.

Adding Multiplayer ¶

To add multiplayer to the game we will use Room Service

You will need to sign up for Room Service here: https://app.roomservice.dev/register.
Add an API key that we can set inside our .env.local for Next.JS local boot.

The API can be added to the local env like this:

ROOM_SERVICE_API_KEY=<api-key>

Now, we need to add Room Service's dependencies to the application.

yarn add
 @roomservice/browser
 @roomservice/node
 @roomservice/react

Setting Up Room Service API ¶

In Next.JS we can set API routes inside the pages/api folder, so we will do just that to create an authentication endpoint for roomservice.

// ../../roomies/pages/api/roomservice.ts

// Next.js API route support: https://nextjs.org/docs/api-routes/introduction

import { NextApiRequest, NextApiResponse } from "next";

const API_KEY = process.env.ROOM_SERVICE_API_KEY;

// Until we have authentication, that returns a UUID, we will have a pseudo
// random number for users that join the application
function getRandomInt(min, max) {
  min = Math.ceil(min);
  max = Math.floor(max);
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

export default async (req: NextApiRequest, res: NextApiResponse) => {
  const body = req.body;
  const user = "some-user-" + getRandomInt(1, 20000000000);

  // Authenticate with the Roomservice API
  const r = await fetch("https://super.roomservice.dev/provision", {
    method: "post",
    headers: {
      Authorization: `Bearer: ${API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      user: user,
      resources: body.resources,
    }),
  });

  const json = await r.json();
  res.json(json);
};

Now we need to wrap the provider around our application, so that our react application has context that it is connected to the room service API and can update player locations for us.

Setting up the Provider in the _app.tsx is done like so:

// ../../roomies/pages/_app.tsx

import { RoomServiceProvider } from "@roomservice/react";
import Head from "next/head";

import "../styles/globals.css";

function MyApp({ Component, pageProps }) {
  // Wrap the application in the Room Service context provider
  // and declare the auth route to be the route we wrote above.
  const params = { auth: "/api/roomservice" };

  return (
    <RoomServiceProvider clientParameters={...params}>
      <Head>
        <title>ROOMIES!!!</title>
        <meta name="viewport" content="initial-scale=1.0, width=device-width" />
      </Head>
      <Component {...pageProps} />
    </RoomServiceProvider>
  );
}

export default MyApp;

Adding Player Movement Tracking to the Game ¶

Room service has a service known as presence, designed to track transient positions. What Room Services does is, set up a websocket on your device. It then keeps track on an Array of objects. You can specify the shape of these objects to fit your needs. I will have a set of players which have positions on the board.

Import usePresence into our game:

import { usePresence } from "@roomservice/react";

And we need a model for our players, that we can code in typescript as:

export type Player = {
  x: string;
  y: string;
};

Next, we can combine the type with the hook to get a collection of players and an action we can dispatch on our local client to set the position of our own player:

const [players, setMyPlayer] = usePresence<Player>("demo", "players");

Ok, we have everything we need to track our player in Room Service, so finally, we need to control when we send information to room service to update the player location. To do this I will implement a useEffect hook, which will track mutations to the top and, left pieces of our state. That way, when we move locally, we will send an event to Room Service, and Room Service will send that down to everyone else that is in the game with us!

useEffect(() => {
  if (!players) return;

  // Only store relative positions so they
  // can be translated out into the height
  // of the players board
  const x = left / board.current.clientWidth;
  const y = top / board.current.clientHeight;

  setMyPlayer.set({
    x: x.toString(),
    y: y.toString(),
  });
}, [left, top]);

So far so good, we can look at the network tab of our local device and see what is currently happening. It could be a good idea for now to add a console log to the players object, so we can track if everything is connected correctly, and see what happens when we use multiple local devices to connect to the game.

{
  "some-user-161254333": { "x": "0.95725", "y": "0.251486" },
  "some-user-17003492921": { "x": "0", "y": "0" }
}

That looks good, we have an object full of players that we can render on the page.

So we can do that using react like below:

{
  Object.entries(players).map(([key, val]) => {
    const style = {
      top: parseFloat(val.y) * board.current.clientWidth,
      left: parseFloat(val.x) * board.current.clientHeight,
    };
    return (
      <div
        key={key}
        className={styles.otherPlayer}
        // Translate the height back into absolute
        style={style}
      >
        &nbsp;
      </div>
    );
  });
}

That is everything that we need to ensure we can track the players across the multiple devices are get position updates in realtime. Lets take a look:

Deploying the Changes ¶

To deploy the changes to Vercel we will need to:

Set an environment variable of ROOM_SERVICE_API_KEY in vercel
Commit the code
Push the code to GitHub

And that is it, we have a live game, that has multiplayer's.

Adding an Objective ¶

So to make the game competitive I am going to add a scoring mechanism.

To do this I will be adding a single coin to the game board. The players will need to seek out the coin, and when they collect the coin they will receive 10 points!

Roomservice enables us to set persisted items inside the current room by using the Map.

Room Service Maps are simple key-value stores with last-writer-wins resolution.

So to use the map in react we can add a hook to our index component:

const [coin, map] = useMap("demo", "coin");

Here 'demo' is the room name, and the key is 'coin'.

The coin is global to everybody in the room, therefore it will need to appear in the same location for everyone that is playing the game.

Scoring Points ¶

The player scores a point if they reach and touch the coin. SO we need to detect if the div that represents the player is overlapping on the another div which will represent the coin:

useEffect(() => {
  if (!coin?.position) return;

  const interval = setInterval(() => {
    const overlap = isOverlapping(coinElement.current, player.current);

    if (overlap) {
      const x = Math.random();
      const y = Math.random();

      map?.set("position", { x, y });

      setScore((val) => val + 10);
    }
  }, 20);

  return () => clearInterval(interval);
}, [map]);

So this useEffect will ensure that every 20ms we check for an overlap between the ref to the current player and a ref to the coinElement. If the two are overlapping we will increment our local player score and set a new position for the coin.

So we will render the coin into the game board aswell so that it appears on all devices in the same relative position:

{
  coin && board.current && (
    const style = {
        left: parseFloat(coin?.position?.x) * board.current.clientWidth || 0,
        top: parseFloat(coin?.position?.y) * board.current.clientHeight || 0,
      };
    <div
      ref={coinElement}
      className={styles.coin}
      style={style}
    >
      &nbsp;
    </div>
  );
}

Having setup the coin, and given our players the ability to have a score I think we need a leader board so we know who is in the lead.

Adding a scoreboard ¶

A scoreboard will need to display on the mobile, tablet and desktop version of the game.

The react component for the scoreboard looks like:

import { Player } from '../pages'
import styles from './ScoreBoard.module.css'

export const ScoreBoard = ({
  players
}: {
  players: { [key: string]: Player }
}) => (
  <div className={styles.scoreBoard}>
    {
      Object.entries(players)
        .map(([key, player]) => (
          <div
            className={styles.scoreCard}
            key={key}
          >
            {player.name}: {player.score}
          </div>
        )
    }
  </div>
)

The styles for the scoreboard ensure that when we are in desktop mode the board will appear as a distinct div mounted on the left of the game board with a margin between them. Then, on mobile, the scoreboard will appear above the game board.

.scoreBoard {
  display: flex;
  flex-direction: column;
  background-color: gainsboro;
  padding: 10px;
  height: 100%;
  height: 95vw;
  max-height: 70vh;
  margin-right: 2rem;

  border-radius: 1rem;
}

.scoreCard {
  margin: 0.5rem;
}

@media (max-width: 1020px) {
  .scoreBoard {
    flex-direction: row;
    flex-wrap: wrap;
    width: 95vw;
    max-width: 70vh;
    height: 100%;

    margin: 0;

    border-radius: 1rem 1rem 0 0;
  }
}

Who are these players ¶

The final piece of the puzzle is to ensure that the players get a name. This will become something that we can store across session, but for now we have no authentication mechanism.

The name of the player will be set from a text input, with a maximum length of 16:

const [name, setName] = useState<string>('anon')

...

<input
  className={styles.userName}
  onChange={(e) => {
    setName(e.target.value || 'anon')
  }}
  placeholder="Your name max (16)"
  type="text"
  name="username"
  maxLength={16}
  aria-label="Username"
  autoComplete="off"
/>

This mean we have players with names in our scoreboard like below:

Now we can extend the existing position use effect hook, to also send update when the name is changed:

useEffect(() => {
  if (typeof window !== "undefined") {
    if (!players || !board) return;

    // Only store relative positions so they can be
    // translated out into the height of the players
    // board
    const x = left / board.current.clientWidth;
    const y = top / board.current.clientHeight;

    setMyPlayer.set({
      x: x.toString(),
      y: y.toString(),
      name: name,
      score: score,
    });
  }
}, [left, top, name]);

Now we can try out our PvP game:

Wrap Up ¶

That is all for Part 2, I think its a good time to take a break from the application.

If you need the full source code for the application it is here:

In the next Part we will add authentication using NextAuth.js.

PART 3 coming soon...

Part 1 - Building Realtime Multiplayer Games with React and Roomservice

Boaty McBoatFace — Tue, 20 Apr 2021 07:00:00 +0000

In this blog series I am going to create a realtime multiplayer game.

Features ¶

100% HTML and CSS game running in the browser for mobile, tablet and desktop. PART 1
Realtime Multiplayer and Player vs. Player. PART 2
Authentication and login for players.
Lobbies for private and public games with unique shareable codes.

Part 1 ¶

Prerequisites ¶

NodeJS version 14.x
A node build tool, I use yarn

Setup ¶

I am going to use NextJS to build this game, its a production React stack that will ensure the finished application is performant.

I am going to call my game Roomies, you can pick any name 👍 Then, I am going to create it with the create-next-app command:

npx create-next-app roomies

Next, I would like to use typescript, so we will rename the files from js to ts and tsx.

This:

./pages/
├── api
│   └── hello.js
├── _app.js
└── index.js

becomes this:

./pages/
├── api
│   └── hello.ts
├── _app.tsx
└── index.tsx

Now, we can boot the app, for this run:

yarn dev

Since we have started using typescript, NextJS will let us know we have some missing dependencies and that we should install them, so to add those run:

yarn add --dev typescript @types/react @types/node

Designing the Game ¶

How are we going to build a game?

The game is going to be multiplayer, and cross-platform, for desktop and mobile.
We need to design the CSS to work on many different screens
We need to have controls for a keyboard, and controls for a touch screen.

Game Board ¶

Lets start by removing everything else from pages/index.tsx

import Head from "next/head";
import styles from "../styles/Home.module.css";

export default function Home() {
  return (
    <div className={styles.container}>
      <Head>
        <title>NextJs Multiplayer Game</title>
        <link rel="icon" href="/favicon.ico" />
      </Head>

      <main className={styles.main}></main>
    </div>
  );
}

And we can keep just the container and main styles that were in the template in styles/Home.module.css:

.container {
  min-height: 100vh;
  padding: 0 0.5rem;
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
}

.main {
  padding: 5rem 0;
  flex: 1;
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
}

Right now, the screen is a little plain, so lets add a gradient background to the .container class:

.container {
  min-height: 100vh;
  padding: 0 0.5rem;
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
  background: #a01a6a;
  background: -webkit-linear-gradient(top left, #a01a6a, #d83729);
  background: -moz-linear-gradient(top left, #a01a6a, #d83729);
  background: linear-gradient(to bottom right, #a01a6a, #d83729);
}

That looks nice for now:

To add a board we want to have a square, which will contain the game. On desktop, we want the square to be large, and centered. On mobile, we want the square to be as large as the screen width, giving some room at the bottom.

To do this we will add a div with a board style like so:

<div
  className=
  id="board"
>
  { /* Space for players */ }
</div>

The style we need is the following:

/*
The board can be really wide, but it
should never be so wide as to go off
the bottom of the screen
*/
.board {
  width: 95vw;
  max-width: 70vh;
  background-color: white;
  /*
    This comes in handy for specifying the
    position of the players.
   */
  position: absolute;
}

/*
Set a padding bottom on the pseudo
after class to give the .board an
aspect ratio of 1:1
*/
.board:after {
  content: "";
  display: block;
  padding-bottom: 100%;
}

Lets take a look at that:

Players ¶

Putting players into the game will mean adding an element to the div board that we just created. For now we can add a single player to the board:

<div className={styles.board}>
  <div id="current-player" className={styles.currentPlayer}>
    &nbsp;
  </div>
</div>

And the styles for that player are:

.currentPlayer {
  width: 5%;
  height: 5%;
  background-color: hotpink;
  display: inline-block;
  position: absolute;
}

Ok, so we have two squares, a big one, and a little one, lets make this seem more like a game.

Controls ¶

The controls will allow players to move Up, Down, Left, and Right. They need to be close to the edges of the phone or tablet, so that a player holding the device can use them. They are not going to be the main source of control for keyboard and mouse users, because they will be able to use the arrow keys to control the board.

The controls can be added as a single component like so:

// ../../roomies/components/controls.tsx

import { SetStateAction } from "react";
import { Directions } from "../pages";
import styles from "./Controls.module.css";

export const Controls = ({
  directions,
  setDirections,
}: {
  directions: Directions;
  setDirections: React.Dispatch<SetStateAction<Directions>>;
}) => {
  const setDirection = (actionName, value: boolean, e: MouseEvent) => {
    e.preventDefault();

    setDirections((currentDirections) => ({
      ...currentDirections,
      [actionName]: value,
    }));
  };

  const ButtonAction = (actionName: keyof Directions) => ({
    onMouseDown: (e) => setDirection(actionName, true, e),
    onTouchStart: (e) => setDirection(actionName, true, e),
    onMouseUp: (e) => setDirection(actionName, false, e),
    onMouseLeave: (e) => setDirection(actionName, false, e),
    onTouchEnd: (e) => setDirection(actionName, false, e),
  });

  return (
    <div className={styles.controls}>
      <div className={styles.upDownControl}>
        <button className={styles.control} {...ButtonAction("ArrowUp")}>
          &uarr;
        </button>
        <button className={styles.control} {...ButtonAction("ArrowDown")}>
          &darr;
        </button>
      </div>
      <div className="left-right-control">
        <button className={styles.control} {...ButtonAction("ArrowLeft")}>
          &larr;
        </button>
        <button className={styles.control} {...ButtonAction("ArrowRight")}>
          &rarr;
        </button>
      </div>
    </div>
  );
};

I have added a number of event listeners to each of the buttons so that we can account for different types of interaction.

Finally I need to add some CSS for these, which I will do as a module, NextJS supports this, so I can add the file next to the controls component:

├── components
│   ├── Controls.module.css
│   └── controls.tsx

And inside I will add the styles:

/* ../../roomies/components/Controls.module.css  */

.controls {
  background-color: gainsboro;
  width: 100%;

  display: flex;
  flex-direction: row;
  align-items: center;
  justify-content: space-around;
}

.control {
  font-size: 2em;
  height: 2em;
  width: 2em;
  margin: 0.2em;

  /* Prevent touch highlighting */
  -webkit-touch-callout: none;
  -webkit-user-select: none;
  -khtml-user-select: none;
  -moz-user-select: none;
  -ms-user-select: none;
  user-select: none;
  -webkit-tap-highlight-color: rgba(0, 0, 0, 0);
}

.up-down-control {
  display: flex;
  flex-direction: column;
}

Now we can take a look at how our game is shaping up:

Connecting our Player to the Controls ¶

Ok, so we have a board, a player, and a set of controls.

Now, we want to connect our player to the controls, and use the controls to move around the board.

First, we need to calculate the players positions:

The player will start at (0,0) when they join.
The player will be able to move every 20ms depending on which directions they are pressing.

Set the initial player positions and directions:

const [top, setTop] = useState<number>(0);
const [left, setLeft] = useState<number>(0);
const [directions, setDirections] = useState<Directions>({
  ArrowUp: false,
  ArrowDown: false,
  ArrowRight: false,
  ArrowLeft: false,
});

Now we can map the player to those positions using the style in React:

const style = { top, left }

<div
  ref={player}
  id="current-player"
  className={styles.currentPlayer}
  style={style}
></div>

So far so good, we have a player with a fixed position inside the UI. Now, we need to compute the position of the player and set it on the element when the player is pressing the arrow keys of the keyboard.

To calculate the new position we can write a small helper function:

const newPositionOnBoard = (
  currentPositionOnBoard: number,
  movementDistance: number,
  maxPosition: number,
  lowerArrowKeyName: string,
  upperArrowKeyName: string
): number => {
  const calculatedNewPosition =
    currentPositionOnBoard -
    (directions[lowerArrowKeyName] ? movementDistance : 0) +
    (directions[upperArrowKeyName] ? movementDistance : 0);

  if (calculatedNewPosition < 0) return 0;
  if (calculatedNewPosition > maxPosition) return maxPosition;

  return calculatedNewPosition;
};

Now, we need to calculate this position at a regular interval, and then set the state variables of top, and left to update the position on the board:

useEffect(() => {
  if (typeof window !== "undefined") {
    // Register key presses
    document.body.addEventListener("keydown", function (e) {
      directions[e.key] = true;
    });
    document.body.addEventListener("keyup", function (e) {
      directions[e.key] = false;
    });

    // Prevent travel outside the bounds of the box
    const maxPosition = board.current.offsetWidth - player.current.offsetWidth;
    // Travel distance as percentage of the total container size, makes speed
    // the same on all device sizes
    const movementDistance = board.current.clientHeight * 0.005;

    const interval = setInterval(() => {
      setLeft((left) =>
        newPositionOnBoard(
          left,
          movementDistance,
          maxPosition,
          "ArrowLeft",
          "ArrowRight"
        )
      );
      setTop((top) =>
        newPositionOnBoard(
          top,
          movementDistance,
          maxPosition,
          "ArrowUp",
          "ArrowDown"
        )
      );
    }, 20);
    return () => clearInterval(interval);
  }
}, [directions]);

And, we need to ensure that we render our controls, with the directions that we have stored in state. This means that the child component can update the state of the directions. So we need to pass in the directions and the setDirection hook into the <Controls /> component:

<Controls {...{ directions, setDirections }} />

Now that we have put all that together we have a working single player game, although, there is no objective yet 😞

Preview Deployment ¶

I think we should deploy this application now, because, we have a working MVP and we need to ensure that we can ship it to development before we have done too much work.

I am going to use Vercel to deploy the site. You can sign up for an account and connect your github account. Follow the setup instructions to connect your github account to vercel and deploy the application as a Next.JS site.

Wrap Up ¶

That is all for Part 1, I think its a good time to take a break from the application.

If you need the full source code for the application it is here:

In the next Part we will add multiplayer to the game using https://www.roomservice.dev/

PART 2

Serverless search with Meilisearch and Google Cloud Run

Boaty McBoatFace — Tue, 13 Apr 2021 17:51:54 +0000

Serverless is a deployment strategy that is loosely defined by:

Only paying per invocation.
Not requiring any infrastructure management.
Scaling to zero when unused.

While solutions exist for Compute, and Databases few options are available for Search engines. Particularly complex search engines that support faceting, full text search, and type ahead.

Meilisearch is a good option for all of the above, its also open source, and has a vibrant community.

This short blog will focus on a serverless deployment strategy for Meilisearch using Google Cloud Run.

Technology stack ¶

To give a good example we will use a live site as an example:

Using NextJS as a site for testing search.
Using Meilisearch for InstantSearch.
Using google cloud run for 2 million free search requests a month.
Using docker to run locally and build snapshots of our search engine.

The solution will work by building the index when the site is compiled for production.

In order to measure the effectiveness we will perform some testing:

Load testing of the search engine API with https://artillery.io
Global performance testing using https://lighthouse-metrics.com/
Page by page comparison of server rendered pages https://pagespeed.compare/

here we go....

Design ¶

Unlike a continuously running search engine, this one will have pre-computed indexes. This is a serious limitation to a search engine, so, here is a short list of solutions this wouldn't be ideal for:

Fast changing data, like stock levels on an e-commerce website.
Data that is removed, edited or versioned multiple times a day, like the data in a CMS.

So when can this type of search engine be useful:

For Static Websites where the contents are all known ahead of time e.g. Documentation sites or Blogs

To create this pre-computed index we will commit our files to source control, and use Google Cloud Build to create a new docker image.

Meilisearch Setup ¶

First, a multi-stage docker container is setup. The first stage local will be used locally from an accompanying docker-compose.yml.

# ../Dockerfile

FROM getmeili/meilisearch as local

ENTRYPOINT ["tini", "--"]
CMD     ./meilisearch

FROM local

# In production we copy in the dump data, seeding the index at build time
# Dumps: https://docs.meilisearch.com/reference/api/dump.html
COPY ./dumps /dumps

ENV     MEILI_HTTP_ADDR 0.0.0.0:7700
EXPOSE  7700/tcp

ENTRYPOINT ["tini", "--"]

CMD ["./meilisearch", "--import-dump", "dumps/latest.dump"]

The docker-compose.yml will target the local instance, and, mount the local ./data.ms (Meilisearch database).

# ../../../docker-compose.yml

version: "3.7"

services:
  local_search_engine:
    build:
      context: .
      target: local
    ports:
      - 7700:7700
    volumes:
      - ./data.ms:/data.ms
      - ./dumps:/dumps

When going into production, the production image will mount the ./data.ms committed to the repository. This will create a pre-built search container that can scale horizontally.

Seeding Data ¶

Data can be added to Meilisearch using the HTTP API or using one of the client libraries.

The Javascript client library suits this use case. To integrate it with existing tooling you need to consider your existing flow. Here is mine as an example.

Source files are in a directory.
Files are parsed by NextJS at yarn build.
Output from parsing is JSON which can be sent to the search index.

// ../lib/seed-search-index.ts

import { MeiliSearch } from "meilisearch";
import { getSortedPosts } from "./api";

export const seedSearchIndex = async () => {
  const posts = getSortedPosts().map(({ date, ...rest }) => ({
    ...rest,
    date: Date.parse(date),
    objectID: Date.parse(date),
  }));

  try {
    // Create a search client
    const client = new MeiliSearch({
      host: String(process.env.NEXT_PUBLIC_SEARCH_URL),
    });

    // An index is where the documents are stored.
    const index = await client.getOrCreateIndex("blogs");

    const settings = await index.getSettings();

    await index.updateSettings({
      ...settings,
      // New blogs first
      rankingRules: [...settings.rankingRules, "date:desc"],
      // Synonyms are great when items have a shared meaning
      synonyms: {
        docker: ["container"],
        javascript: ["typescript", "node"],
        react: ["nextjs"],
      },
      // We will filter on the tags I use in my blogs
      filterableAttributes: ["tags"],
    });

    // Add the posts to the blogs index
    const response = await index.addDocuments(posts);

    console.info("Completed indexing", response);
  } catch (e) {
    console.warn("No connection found to build index", e);
  }
};

Each post is a javascript object with fields. An ID is created by using the timestamp. This means I can order by ID and have chronological ordering.

By using tags as a facet, I can allow people to filter blogs to subjects they are interested in.

This function is added to the index of the site, so that when the site is built with next build it compiles the search index as well. This is compatible with next dev, for local development, next build for serverless and server bound next production builds, and next export, the option of exporting next JS as a set of static files.

// ../pages/index.tsx

import { GetStaticProps, GetStaticPropsContext } from "next";
import { getSortedPosts } from "../lib/api";
import PostSummary from "../components/PostSummary";
import { seedSearchIndex } from "../lib/seed-search-index";
import { generateRSSFeed } from "../lib/rss";

export const getStaticProps: GetStaticProps = async () => {
  const posts = getSortedPosts();
  // Seed the index at compile
  await seedSearchIndex();

  await generateRSSFeed(posts);

  return {
    props: {
      posts,
    },
  };
};

export default function Home({ posts }) {
  return (
    <main>
      <h1>Blogs</h1>
      {posts.map((post) => (
        <PostSummary {...post} key={post.date} />
      ))}
    </main>
  );
}

Server rendering search pages ¶

Benefits ¶

Server rendered pages work better with SEO.
Server rendered pages support users that have javascript disabled.
Server rendering search with Algolia react-instantsearch means that you can have complex URL's, including, Query strings, Page numbers, Facets.
Rendering on the server means that we won't have to deal with pending states, loading spinners, and skeletons because what is on the page when the HTML is parsed will be the same as what is on the pages once the DOM has been hydrated with the state from React.

An example would be something like this:

https://blog.simonireilly.com/search?query=s&page=2&refinementList%5Btags%5D%5B0%5D=cicd&refinementList%5Btags%5D%5B1%5D=nextjs&refinementList%5Btags%5D%5B2%5D=containers&refinementList%5Btags%5D%5B3%5D=cypress

This server rendered page works by:

Parsing the URL query parameters
Making a request to the search engine
Returning an initial result state to the NextJS render to string process.

// ../pages/search.tsx

import { withRouter } from "next/router";
import { findResultsState } from "react-instantsearch-dom/server";
import qs from "qs";
import { GetServerSideProps } from "next";

import {
  App,
  AlgoliaPage,
  DEFAULT_PROPS,
} from "../page-components/search-page";

// This is a standard SSR implementation of algolia. Next JS exposes the full
// query object, so we can perform full server rendering
export const getServerSideProps: GetServerSideProps = async ({ query }) => {
  const searchState = query ? qs.parse(query) : {};

  const resultsState = await findResultsState(App, {
    ...DEFAULT_PROPS,
    searchState,
  });

  return {
    props: {
      resultsState: JSON.parse(JSON.stringify(resultsState)),
      initialSearchState: searchState,
    },
  };
};

export default withRouter(AlgoliaPage);

Deployment ¶

This is currently a joint deploy between Vercel for hosting my blog site, and google cloud run which will host the search index.

Merge a commit to main.
Trigger a redeploy of the front end with the new blog post.
Trigger a google cloud build of a new docker container with the new blog post in its search index.
Google Cloud Build promotes the container after it builds successfully

Load testing ¶

I am going to perform a load test on Meilisearch deployed using cloud run, to do this I am going to use artillery.

I have a really small container deployed, this is actually the smallest configuration available:

v.CPU	Memory	Maximum Requests Per Container
1	128	250

Artillery Results 💣 ¶

Artillery load tests are written in YAML. so lets take al look at mine:

# ../_snippets/serverless-search/artillery/small/load-test.yml

config:
  target: https://meillisearch-kjsckj2bja-ew.a.run.app
  http:
    timeout: 10
  phases:
    - duration: 20
      arrivalRate: 10
      name: "Normal Traffic"
    - duration: 5
      arrivalRate: 50
      name: "Spike"
    - duration: 20
      arrivalRate: 10
      name: "Normal Traffic"
    - duration: 5
      arrivalRate: 50
      name: "Spike"
    - duration: 20
      arrivalRate: 10
      name: "Normal Traffic"
scenarios:
  - flow:
      - post:
          url: "/indexes/blogs/search"
          json:
            {
              "q": "",
              "limit": 200,
              "facetFilters": [["tags:cicd"]],
              "facetsDistribution": ["tags"],
              "attributesToHighlight": ["*"],
            }

Aggregate ¶

Responses

As expected all fo the HTTP status codes from the 1100 requests have come back with a 200 success.

HTTP Code	Count	Total	Percentage
200	1100	1100	100

Latency

The statistics below show that the free search engine is quite performant!

Avg. RPS	min (ms)	max (ms)	median (ms)	p95 (ms)	p99 (ms)
15.62	50.8	417.1	59.6	148.1	292.5

Performance testing ¶

Performance testing is about measuring the one off delivery of the page to the end user, fully rendered.

There are a number of great tools for this:

First and foremost your own dev tools with light house, be sure to use an incognito tab for this to avoid bundling react dev tools etc.!
Lighthouse Metrics, a tool for global light house metric measuring https://lighthouse-metrics.com/
Page speed compare to compare different pages, and different websites https://pagespeed.compare/

Global Lighthouse Scores ¶

For this test, we want to measure the performance around the globe, for people visiting our search page, and entering a complex query:

The URL: https://blog.simonireilly.com/search?query=s&page=2&refinementList%5Btags%5D%5B0%5D=cicd&refinementList%5Btags%5D%5B1%5D=nextjs&refinementList%5Btags%5D%5B2%5D=containers&refinementList%5Btags%5D%5B3%5D=cypress

Lighthouse version: 6.3.0
Simulated device Moto G4 (2016)
4x CPU Throttling

These scores look good enough to me. What I am most pleased about is that we achieved a max time to interactive of 2.1s. Its always good to keep this under 3 seconds. Running on a 5 year old device, with CPU throttling enabled is a great way to ensure we have a site that is accessible to users with low powered devices, all over the globe.

Page Comparison ¶

Page speed compare is what I will use for this one:

Label	URL	Perf.	FCP (ms)	SI (ms)	LCP (ms)	TTI (ms)	TBT (ms)
Home	Home	100	1230	1314	1230	1372	17
Search	Search	100	1380	1380	1380	2057	126
Query	Query	99	1380	1450	1380	2179	187

The big hitters in this table are Cumulative Layout Shift (CLS) of zero, for all pages. This means we have nothing initially rendered, then shifted after our javascript kicks in. Performance is high for all the pages, which is also really good. Overall these are great numbers.

Speed Index ¶

Largest Contentful Paint ¶

First Contentful Paint ¶

Time to First Byte ¶

Wrap Up ¶

Well that was a bit of a journey, but thanks for sticking around, lets recap.

I showed how we can build a serverless search engine using Meilisearch.
I gave some examples of seeding this search engine into a NextJS site.
I went into detail about testing, both load, and performance testing.

I have been using this with my own blog for a few months, and overall, I am really happy with the setup.

Thanks for reading 👍

DynamoDB Worx

Boaty McBoatFace — Sun, 11 Apr 2021 18:42:55 +0000

DynamoDB is a serverless wide column key value store. It's a great database for Online Transaction Processing (OLTP).

I use this database a lot, and had been looking for a GUI to run locally. Those that I reviewed are either premium products, or open source options that are no longer supported.

The exception to that is NoSQL Workbench which is really good, but does not support my use case of needing to assume a role in another account after authenticating the source account with MFA (see here).

So I decided to build one, here we are 4 weeks later, and I would like to share my experiences with you.

Project Goals ¶

☑️ Multi account support
☑️ MFA Support
☑️ Support to Assume Role in another account
☑️ Edit the JSON documents using a text editor
☑️ Create, Update DynamoDB records
☑️ Perform Scans
☑️ Perform Queries
☑️ Support for DynamoDB local

Finished Product ¶

Check it out over here: https://github.com/simonireilly/dynamoDB-benchworx 👍

☑️ Creating/Updating Records ¶

The editor will detect changes to the primary key and sort key fields. This is a hint that the action will create or update a new record.

☑️ Query Planning ¶

Queries and scan are supported.

Below the planner you can see the parameters passed to the QueryCommandInput.

☑️ MFA Support ¶

MFA support allows assuming roles in separate accounts after MFA auth 🥳.

Making the App ¶

This section is all about what went into the build, and how the DevOps have been set up.

The Stack ¶

Using electron so that I can publish to MacOS, Windows and Ubuntu 💻. I considered flutter, but I have zero experience so it wasn't the right choice for me.

Using React for the UI is just my defacto at this point...

Using typescript to write the code. This is something I think helps make code more legible and portable 👌

Using the AWS SDK for Javascript V3. This is something new I wanted to learn, so a good addition to the mix for me 🤩

Finally for all DevOps using GitHub to collaborate, build, test, deploy and receive feedback (issues).

The Sub Stack ¶

Early on I decided to rely on a User Interface kit, this would help me go faster on the MVP, so the amazing Material UI Core was used.

For an editor, I personally enjoy VSCode, and so, a massive benefit was the @monaco-editor/react, which dropped in nicely. I hope to enhance the app using its diff functionality soon too!

The process ¶

I was aware of electron-forge and electron-builder from previous apps, and so, began with a bootstrapped sample from electron-forge.

I followed the steps for the Electron forge with webpack and Typescript example on electron forge.

Electrons Sharp Edges ¶

The main sticking point that can trip you up in electron is the concept of renderer and main.

Main is the process that is running your application. This main process is responsible for booting, preloading and preparing your application along with any node_modules that you require.

Renderer processes are detached from the main process, and communicate over the IPC bus in electron. This has great security and segregation wins. It enables the launching of multiple windows and keeping windows secure.

Previously I had found the whole concept of forwarding dependencies to the renderer process too tricky. Electron has improved this since I used it in 2019, and now it includes a method to append functions onto the window object. The function is called exposeInMainWorld 👏.

This is a great name, as this is exactly what the function does. It enables you to overload the renderers Window object with additional functions that you would like to call. These functions can be from node_modules as long as they are imported in the preloader.ts. Main handles preloading these and ensures that when the first renderer is opened, all the preloads are accessible on the window.

An example of preloading is below:

// ../_snippets/dynamodb-worx/preloader.ts

// Preloading allows fetching node_modules/native code into the application
//
// In order to use the @aws-sdk we must import those utilities here and add
// those methods to the window.
//
// When writing those methods, we should encapsulate the actual calls to client,
// preventing injection and/or privilege escalation.

console.info("Preloading node modules");

import { contextBridge } from "electron";
import { isDeepStrictEqual } from "util";

import {
  describeTable,
  listTables,
  scan,
  query,
  put,
} from "@src/utils/aws/dynamo/queries";
import { listAwsConfig } from "@src/utils/aws/accounts/config";
import { authenticator } from "@src/utils/aws/credentials";

declare global {
  interface Window {
    aws: {
      authenticator: typeof authenticator;
      describeTable: typeof describeTable;
      listAwsConfig: typeof listAwsConfig;
      listTables: typeof listTables;
      scan: typeof scan;
      query: typeof query;
      put: typeof put;
    };
    util: {
      isDeepStrictEqual: typeof isDeepStrictEqual;
    };
  }
}

// All responses should implement this interface
export type PreloaderResponse<T> = {
  type: "success" | "error" | "warning" | "info";
  data: T | null;
  message: string;
  details: string | null;
};

// Expose the AWS object to over the context bridge to allow invocation of the attached
// methods in the renderer processes.
contextBridge.exposeInMainWorld("aws", {
  authenticator,
  describeTable,
  listAwsConfig,
  listTables,
  scan,
  query,
  put,
});

contextBridge.exposeInMainWorld("util", {
  isDeepStrictEqual,
});

¶

The DevOps ¶

I identified that I wanted:

Visual test automation
Unit test automation
Automated releases

Visual test Automation ¶

I really like vercel's screenshots for new releases as it helps me iterate quickly on my web apps.

I think its important to also recognize the duality of modern front end. While front ends are predominantly visual, modern javascript fat clients have placed a barrier between the designer and end product. I had hoped these review comments would make it possible to qualify the design implementation without needing to get into the code base.

I quickly put something together for this using:

The github action I created would use the github token of the user who submits the pull request to add a snapshot comment. Because of this I had to exclude Dependabot. The token Dependabot uses has less permissions.

# ../_snippets/dynamodb-worx/ci.yml

name: Lint/Test/Build

on:
  push:

jobs:
  screenshot:
    - name: Append current screenshot
      uses: peter-evans/commit-comment@v1
      if: github.actor != 'dependabot[bot]' && github.actor != 'dependabot-preview[bot]'
      with:
        body: |
          ### 📷 Snapshot Tests

          | Previous UI | Current UI |
          | ----------- | ---------- |
          | ![Current snapshot](https://raw.githubusercontent.com/simonireilly/dynamoDB-benchworx/main/cypress/snapshots/end-to-end/index.spec.tsx/latest.snap.png) | ![Current snapshot](https://raw.githubusercontent.com/simonireilly/dynamoDB-benchworx/$/cypress/snapshots/end-to-end/index.spec.tsx/latest.snap.png) |

          Changes to the UI that are applicable after this merge request.

The outcome was exactly what I was after:

Unit Test Automation ¶

Simple unit testing has been more than enough, and for this, Jest is still my preferred runner. My methodology was to test the AWS SDK calls used by my custom context object to ensure they returned the Preloader interfaces defined above.

This ensures that the front end will display a notification of the error. I don't like to obscure API calls behind custom interfaces, so there was a light wrapper that mimics axios/fetch for response.type === 'success' assertions. Errors are logged into the console as well, so users can debug. I assumed most users will be developers, and that my code will break, so if they can debug it I think it will prevent a lot of frustration.

Automated releases ¶

I wanted a low maintenance release process that would work 100% in GitHub.

I ended up using a combination of:

Electron Forge Github Publisher - To create binaries and assets
Release It - To create tags, bump the versions, manage release betas and detect merged changes.
Release It Auto Changelog - To automate the creation of a CHANGELOG.md

I use release-it locally to create the release, then once that is pushed to a tag, a GitHub action picks it up and adds the compiled assets for MacOS, Windows and Ubuntu.

The github action for publishing is really straight forward thanks to Electron Forge

# ../_snippets/dynamodb-worx/release.yml

name: Release

on:
  push:
    keywords:
      - "v*"

jobs:
  build:
    runs-on: $

    strategy:
      matrix:
        os: [macos-latest, ubuntu-latest, windows-latest]

    steps:
      - name: Check out Git repository
        uses: actions/checkout@v1

      - name: Install packages
        run: yarn install

      - name: On Publish
        env:
          GITHUB_TOKEN: $
        run: yarn electron-forge publish

The end result is pretty satisfying, and I am really happy that this process is almost entirely automated.

Going Forward ¶

There are a few features that I cut from the MVP and I will be working on soon:

Code signing for Windows and MacOS, this is a big one and for sure this is something I will need to learn about before implementing.
Deleting Records.
Create Read Update Delete (CRUD) for tables.
Remote to local replication, of table schemas, to create local throw-away copies for designing complex queries.
Two way binding of the Query Builder forms and a editor to allow entering AWS SDK V3 DynamoDB commands without using forms.

And probably many more things!!

Wrap Up ¶

Thanks for reading and if you want to learn more please check out the project here: https://github.com/simonireilly/dynamoDB-benchworx

Typescript patterns for third party dependencies

Boaty McBoatFace — Sun, 28 Mar 2021 08:45:52 +0000

This month I have been creating some electron apps, using typescript and the aws-sdk v3 for JavaScript. I hope to have some exciting outcomes to share in the future.

For now I'd like to share some Typescript patterns I found useful when working with the new SDK.

Return types of functions ¶

The first one is simple, it's the use of ReturnType in combination with the typeof annotations.

Here is an example:

// ../_snippets/typescript-patterns/return-type.ts

import React, { useState, useEffect } from "react";

const listTables = () => ({
  data: {
    message: "I am fully typed",
  },
});

const [credentials, setCredentials] = useState<ReturnType<typeof listTables>>();

useEffect(() => {
  setCredentials(listTables());
}, []);

So to run this through. We can set the type of our useState hook to the return type of the imported function.

This is useful for the following reasons:

The type of credentials will be set correctly to the functions return type.
We are not required to inspect the function from the 3rd party, and we don't need to have knowledge of what it returns.
We have not imported any typing from the third party, which is IMO an improvement over always importing a libraries typedefs just to set them as state variables.
If in the future, if the return type of the function changes, Typescript will complain when we try to access attributes on credentials that have changed.

Resolved return of promises ¶

Next up is an expansion of this pattern, but enabling the use of promises:

// ../_snippets/typescript-patterns/awaited-return-type.ts

import React, { useState, useEffect } from "react";
type Awaited<T> = T extends PromiseLike<infer U> ? Awaited<U> : T;

const listTables = () =>
  Promise.resolve({
    data: {
      message: "I am fully typed",
    },
  });

const [tables, setTables] = useState<Awaited<ReturnType<typeof listTables>>>();

useEffect(() => {
  const init = async () => {
    const results = await listTables();
    setTables(results);
  };
  init();
}, []);

I have included the promise function to make life easier.

So this example can be read as. The type of tables, will be the resolved value, of the promise returned by the listTables function.

This is useful, mainly because, we can say that we have returned a data type, and not a promise type.

Soon, this type will be supported natively in typescript

Wrap Up ¶

That's all for now, I hope you have found this useful 👍

Mozilla Observatory GitHub Action

Boaty McBoatFace — Sat, 13 Feb 2021 18:42:55 +0000

The Mozilla Observatory is a security tool designed to teach developers how to secure their sites.

This GitHub action is designed to help integrate it into your CI/CD workflow, to help raise awareness, and automate some security best practices.

The action is available on Github Marketplaces here: GitHub Observatory Action

Example ¶

Quick Vercel Deploy ¶

We can begin quickly by creating a NextJS site and deploying it to Vercel using their automated deploy system

Setup the GitHub Action ¶

Once the site is live, we can add a github action to the created repository:

# ../../../.github/workflows/deploy.yml

name: "web-security"
on:
  deployment_status:

jobs:
  deployment_status:
    runs-on: ubuntu-latest
    if: github.event.deployment_status.state == 'success'
    steps:
      - uses: actions/checkout@v2

      - name: Test Observatory
        uses: simonireilly/observatory-github-action@v0.0.1-alpha
        id: observatory
        with:
          web_host: $

      - name: Create commit comment
        uses: peter-evans/commit-comment@v1
        with:
          body: "# Deployment Status _$_ $"

Now we can create a new Pull Request for this and see it running on GitHub.

Add Secure Headers ¶

Great, so we have security in our CI/CD.

We can make use of the report link to get a full list of recommendations for security best practices. To implement these, we can use next-secure-headers.

yarn add next-secure-headers

Then into the next.config.js we can put some secure headers and re-deploy that application:

// ../next.config.js

const { createSecureHeaders } = require("next-secure-headers");

module.exports = {
  async headers() {
    return process.env.NODE_ENV === "production"
      ? [
          {
            source: "/(.*)",
            headers: createSecureHeaders({
              contentSecurityPolicy: {
                directives: {
                  objectSrc: ["'self'"],
                  scriptSrc: [
                    "'self'",
                    "'unsafe-inline'",
                    "https://www.googletagmanager.com/",
                    `https://${String(process.env.VERCEL_URL)}/`,
                    "https://blog.simonireilly.com/",
                    "https://static.cloudflareinsights.com/",
                    "https://app.posthog.com/",
                  ],
                  defaultSrc: [
                    "'self'",
                    String(process.env.NEXT_PUBLIC_SEARCH_URL),
                    "https://vitals.vercel-insights.com/v1/vitals",
                    "https://www.google-analytics.com/",
                  ],
                  styleSrc: ["'self'", "'unsafe-inline'"],
                  imgSrc: ["'self'", "www.googletagmanager.com/"],
                  connectSrc: [
                    "'self'",
                    "https://vitals.vercel-insights.com/",
                    "https://www.google-analytics.com/",
                    "https://*.google-analytics.com",
                    "https://app.posthog.com",
                    `${String(process.env.NEXT_PUBLIC_SEARCH_URL)}/`,
                  ],
                  baseUri: "'self'",
                  formAction: "'self'",
                  frameAncestors: true,
                  frameSrc: [""],
                },
              },
              forceHTTPSRedirect: [
                true,
                {
                  maxAge: 60 * 60 * 24 * 4,
                  includeSubDomains: true,
                },
              ],
              referrerPolicy: "same-origin",
            }),
          },
        ]
      : [];
  },
};

We get preview deploys, and the github action intercepts the target_url to ensure we can test security on only our branch.

Now we can see that we are getting a much better outcome.

Additionally we can leave this action in place, preventing security regressions, and helping everyone we collaborate with to understand some of the security best practices that matter to us.

VSCode, Markdown, and NextJS Blogging Stack

Boaty McBoatFace — Sat, 23 Jan 2021 17:51:54 +0000

This blog is written 100% in VSCode, including all the diagrams and images.

My setup is optimized for a fast flow, and a monorepo style by keeping me in my editor.

Text ¶

Text is added to markdown files. The markdown files are converted to HTML using remark and gray-matter.

Example blog:

---
title: VSCode, Markdown, and NextJS Blogging Stack
excerpt: >-
  This blog is about how I built this blog site. It is focussed on a fast flow for
  both writing and drawing/annotating using a single editor, VSCode.
date: "2021-01-23T17:51:54.096Z"
keywords:
  - nextjs
  - vscode
---

# VSCode, Markdown, and NextJS Blogging Stack

This blog is written 100% in VSCode, including all the diagrams and images. Lets describe how.

Images ¶

For images, I use a drawio plugin for vscode. This works really well for diagramming as if files are named *.drawio.png you can embed them directly into your markdown.

![diagram showing architecture](/assets/blog/serverless-search/architecture.drawio.png)

Looks like this

Code Snippets ¶

Most of the code that is shown in snippets is written in normal files. This reduces duplication, and errors. So most of the code is a live piece of code that is referenced using embedme. This CLI runs on a hook to compile markdown files.

Lets have an example, this is the relative path to the page you are currently viewing ../pages/posts/[slug].tsx.

To embed it, I add the path as the first comment to a block of code.

Now I run yarn embed and the below is the outcome.

// ../pages/posts/[slug].tsx

import { GetStaticPaths, GetStaticProps } from "next";
import Head from "next/head";
import markdownToHtml, {
  getPostFileNames,
  getPostMatter,
  getPostSlugs,
} from "../../lib/api";
import "highlight.js/styles/atom-one-dark.css";
import styles from "./Slug.module.css";
import { calculateReadTime } from "../../lib/utils";

export const getStaticPaths: GetStaticPaths = async () => {
  const posts = getPostFileNames();
  const slugs = getPostSlugs(posts);
  const paths = slugs.map((slug) => ({ params: { slug } }));
  return {
    paths,
    fallback: false,
  };
};

export const getStaticProps: GetStaticProps = async ({ params }) => {
  const { data, content } = getPostMatter(params.slug as string);
  const innerContent = await markdownToHtml(content);
  const readTime = calculateReadTime(innerContent);
  return {
    props: {
      data,
      content: innerContent,
      readTime,
    },
  };
};

export default function Post({ data, content, readTime }) {
  const dateTime = new Date(data?.date);
  const html = { __html: content };
  return (
    <div className={styles.lowPadding}>
      <Head>
        <title>{data?.title}</title>
        <meta name="description" content={data?.excerpt} />
        <meta name="keywords" content={[...data?.tags].join(", ")} />
        <meta name="author" content="Simon Reilly" />
      </Head>
      <h1>{data?.title}</h1>
      <div className={styles.titleRow}>
        <em>Time to read: {readTime} minutes</em>
        <time dateTime={data?.date}>{dateTime.toDateString()}</time>
      </div>
      <hr />
      <div dangerouslySetInnerHTML={html} />
    </div>
  );
}

It also supports line numbers...

// ../pages/posts/[slug].tsx#L23-L30

const { data, content } = getPostMatter(params.slug as string);
const innerContent = await markdownToHtml(content);
const readTime = calculateReadTime(innerContent);
return {
  props: {
    data,
    content: innerContent,
    readTime,

Now I can write freely, alter and test code, without copying it across to blogs. This means most of the code snippets are tested and used, so less likely to become stale.

Search Indexing ¶

The search feature for this blog is a pre-computed index. There is no overhead added to indexing pages. Instead the index will insert the gray-matter from each blog on build.

I have dedicated a whole blog to this here

Testing server side rendered (SSR) React with Cypress

Boaty McBoatFace — Sat, 21 Dec 2019 08:45:52 +0000

Cypress is a fully integrated testing suite for doing integration tests on your web application. it comes with pre-installed browsers and a runtime to be able to get started super fast.

Check it out at https://www.cypress.io/

Project Setup ¶

All that code can be found here in its final state https://github.com/simonireilly/cypress-ssr/

Let's begin by creating an example next app using the create-next-app command and node package manager.

NextJS https://github.com/zeit/next.js/ is a server side rendering javascript framework which can be used with react in order to generate server side rendered pages which have client-side interactions.

npx create-next-app cypress-ssr

Next we're going to add all our testing depencies using yarn.

cd cypress-ssr
yarn add cypress

Unlike other testing frameworks where you might be required to include certain libraries and separate runtimes for browsers, with cypress everything is include!

Before we move on you should verify your install by running cypress.

yarn cypress open

Writing you first test ¶

In the root of the application Cypress will have created in cypress.json file. We will modify this file and add the following:

{
  "baseUrl": "http://localhost:3000"
}

This special JSON key will tell Cypress where we are running our application. We can now write our first spec.

Create a folder cypress/integration/next, the folder name doesn't matter, but cypress UI will organise our specs into collections using the folder structure.

In that folder we can put the following test.spec.js file:

context("Next App", () => {
  it("visit the home page", () => {
    cy.visit("/");

    cy.get(".hero").contains("Welcome to Next.js!");
  });
});

Now we can run this test from the Cypress UI. Start the Next JS development server with yarn dev and in a seperate terminal launch cypress with yarn cypress open.

Find and run our new test in the UI and you should get the below.

Server Side Testing ¶

We are going to be learning a few new cypress concepts, so here is a run down:

Support Commands ¶

Cypress allows you to create/register your own commands, into the test suite, just like adding shared functions. When Cypress starts it loads these commands into its global, so you can use them. The below is a command we want to use. It mimics navigating between pages server side, by removing all javascript.

// Request a page over http from the server and mount it in the DOM
//
// To simulate server side rendering remove all scripts so no window:onload
// events can occur and we can test server rendered views

Cypress.Commands.add("serverSideVisit", (url) => {
  cy.reload();
  cy.request(url)
    .its("body")
    .then((html) => {
      html = html.replace(
        /<script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script>/gi,
        ""
      );
      cy.state("document").write(html);
    });
  cy.get("script").should("not.exist");
});

My inspiration for this command comes from this blog https://glebbahmutov.com/blog/ssr-e2e/. So, now we have this command, we can paste it into cypress/support/commands.js and write a new test.

Update our cypress/integration/next/test.spec.js to be:

context("Next App", () => {
  it("visit the home page", () => {
    cy.visit("/");

    cy.get(".hero").contains("Welcome to Next.js!");
  });

  it("visits the home page server side", () => {
    cy.serverSideVisit("/");

    cy.get(".hero").contains("Welcome to Next.js!");
  });
});

And when you save it, if the development server and cypress are still booted, the spec will automatically re-run.

Closing ¶

I hope you have enjoyed setting up Cypress and learning about how to use it for testing Client and Server Side Applications. Remember that when deciding to develop an isomorphic application, that testing its main beneficial feature, should be done at the earliest point.

Here are some useful Cypress resources for continuous delivery:

Cypress documentation: https://docs.cypress.io/guides/overview/why-cypress.html#In-a-nutshell/
Self hosted cypress parallel testing: https://github.com/agoldis/sorry-cypress
Docker images with Cypress and browsers included for CI: https://github.com/cypress-io/cypress-docker-images/

Go Kafka, Cloud Karafka and Heroku

Boaty McBoatFace — Wed, 13 Nov 2019 18:42:55 +0000

A sample for networking kafka in docker using targeted builds. ¶

Deploying to heroku free stack!!

Getting Started ¶

First off, what is all this stuff? Apache kafka is a messaging system with Producers, Consumers and Brokers that publish and subscribe to Topics. There's a little more complexity to it, but you can read a lot more here https://kafka.apache.org/intro

What are we Building ¶

This is just an example of building and crucially, deploying a simple consumer and producer. They are so simple in fact, that were just copied from the official go kafka examples. But this tutorial is more about getting your Kafka deployed, in a free testing environment, and doing that with docker too!

Sample Code ¶

The sample code is available here: https://github.com/simonireilly/go-kafka-docker

The code includes some security settings for rdkafka. It implements the SASL security method here:

	p, err := kafka.NewProducer(&kafka.ConfigMap{
		"bootstrap.servers": os.Getenv("KAFKA_BROKERS"),
		"sasl.username":     os.Getenv("KAFKA_USERNAME"),
		"sasl.password":     os.Getenv("KAFKA_PASSWORD"),
		"security.protocol": "SASL_SSL",
		"sasl.mechanisms":   "SCRAM-SHA-256",
	})

Docker File(s) ¶

So there are multiple dockerfiles, but they are all the same ¯\_(ツ)_/¯ There is a limitation in the heroku.yml (see below) that the dockerfile must be in the same directory as the code.

FROM golang:1.13-alpine3.10 AS builder
# Build with Go
WORKDIR /go/src/github.com/simonireilly/go-kafka-tester
RUN apk add librdkafka-dev build-base
COPY . .# Tell go, to build binaries, as a file called ./main
RUN go build -a -o ./main
# Run main at entry in the builder
ENTRYPOINT ["./main"]
# Production is cleaner, smaller image, no build deps
FROM alpine:edge
# We'll just need librdkafka to actually use kafka
WORKDIR /rootCOPY --from=builder /go/src/github.com/simonireilly/go-kafka-tester/main .
RUN apk --no-cache librdkafka
# Enter at main again
ENTRYPOINT [ "./main" ]

Heroku YAML ¶

The heroku yaml file takes care of prvisioning an instance of Kafka from CloudKarafka. This file will also handle building our consumer and producer, publishing them to a docker registry, handy!

setup:
  addons:
    - plan: cloudkarafka:ducky
    as: KAFKAbuild:
  docker:
    consumer: ./consumer/Dockerfile
    producer: ./producer/Dockerfile

Deploy ¶

To deploy with heroku run:

heroku create your-app-name --manifest

git push heroku master

That's it!

Kafka is included in the heroku.yml and everything is configured to work out of the box.

Docker Compose Cross App Networking

Boaty McBoatFace — Thu, 11 Apr 2019 17:51:54 +0000

Intro ¶

Docker is great for simplifying your local development environment. But, as your workspace grows, lots of apps begin to come online.

Do you have to create a database, background services, messaging services for each of these apps?

This can be a real bug-bear. You might also need to connect to some service that is the backbone of your architecture e.g. Kafka.

Docker to the rescue!

Most people back off from docker abstraction, and begin to expose ports to the client outside of the docker container to bypass this. There is no need - docker can handle this just fine. It can even be setup in your docker-compose.yml to save you from copying and pasting those long command line arguments.

Getting Started ¶

Lets build some Kafka applications inside docker, and connect them up.

Get some Kafka ¶

We are going to build a Kafka docker-compose.yml that will run Lenses using the pre-built image available from Lenses.io ( you need to sign-up for a free license - read ahead to see if it is worth it ).

When I have shared services I always add them to my utilities folder. If you don't have one, lets make one!

mkdir -p docker-utilities/lenses

cd docker-utilities/lenses

touch docker-compose.yml

You can find my example here: lenses setup. The below compose file is all you need, and you can assign the LENSES_LICENSE_URL as an environment variable.

version: "3.5"
services:
  kafka:
    container_name: kafka
    image: landoop/kafka-lenses-dev
    environment:
      SAMPLEDATA: "0"
      RUNNING_SAMPLEDATA: "0"
      EULA: "${LENSES_LICENSE_URL}"
    networks:
      kafka_net:
    ports:
      - "3030:3030"
      - "9092:9092"
      - "2181:2181"
      - "8081:8081"
      - "9581:9581"
      - "9582:9582"
      - "9584:9584"
      - "9585:9585"

networks:
  kafka_net:
    driver: bridge
    name: kafka_net

The real special sauce here is that we are setting up a network called kafka_net. This is the virtual network that our Lenses container belongs to.

I'll show you how to get other apps to join this network as well.

Booting Lenses

We can start lenses now:

docker-compose up

Lenses takes a little while to start, but you will find the admin panel at localhost:3030 once it is ready. The username:password default is admin:admin.

Lenses is up and running, with an Apache Kafka Cluster and Broker. Now we need to create some Apache Kafka apps to use it!

Karafka ¶

Karafka is a framework for building Apache Kafka applications with Ruby, and it supports Ruby on Rails.

We are going to clone the example application to a new directory:

git clone git@github.com:karafka/example-app.git\

It is not a docker app, but we can make it one.

We need a ruby image for this app, so create a Dockerfile in the directory and add:

FROM ruby:2.6.1

RUN apt-get update -qq && apt-get install -y \
  build-essential libpq-dev nodejs

RUN mkdir /myapp

WORKDIR /myapp

COPY Gemfile /myapp/Gemfile

RUN bundle install

COPY . /myapp

To boot the application we will use the following docker-compose.yml:

version: '3.5'
services:

  web:
    build: .
    command: bundle exec karafka server
    volumes:
      - .:/myapp
    ports:
      - "5000:5000"
    networks:
      kafka_net:

See in the above that we specify the kafka_net we created earlier as the containers network. This will mean we can access the other containers in this network using their container names and ports.

One last thing! If you open up karafka.rb you'll see line 17 specifies the lookup for the Apache Kafka broker. We need to point this to our Lenses Apache Cluster, which we named with container_name: kafka

# karafka.rb Line 17
config.kafka.seed_brokers = [ENV['KAFKA_HOST'] || 'kafka://kafka:9092']

Everything comes together to produce this one container address - this is how to read it:

kafka:// Use the Apache Kafka protocol
kafka:9092 Look for a service named kafka on the network and connect on port 9092.

With that sorted lets boot:

docker-compose up

Sending Test Messages ¶

Karafka provides a rake task to start sending messages.

In a new terminal and in the example-app directory run:

 docker-compose exec web bundle exec rake waterdrop:send

Now head over to http://localhost:3030/lenses/#/topics and enjoy the fun.

Wrap-Up ¶

Thanks for reading this introduction to networking between compose files.

My Blog about Boats

Thoughts on Return on Investment (R.O.I) in Software Engineering

Why bother with R.O.I ¶

Measuring things in general ¶

Research and truth seeking ¶

The basics of Engineering ¶

Software engineers, the great truth seekers ¶

Closing thoughts and a conclusion ¶

Better typescript fetch clients

TL;DR ¶

An example ¶

Response Typing ¶

Narrowing Response ¶

Putting it all together ¶

Wrap up ¶

Lit elements with Xstate

State Management ¶

Xstate ¶

Setup a machine in Lit ¶

01 - The machine ¶

02 - The DOM Event Adapter ¶

03 - A Lit Mediator Element ¶

Summary of the pattern ¶

Wrap up ¶

Private Singleton for Scoping Sentry SDK

But why 🤷 ¶

Design ¶

Creating a singleton ¶

Wrap up ¶

Cypress component tests for Lit Elements (web components)

Testing Lit ¶

Testing Lit Components with Cypress ¶

TL;DR ¶

Wrap up ¶

Quick Tip: Hiding local files in Git, just for you

Cloudwatch Real User Monitoring (RUM) end to end monitoring

What has Changed ¶

TL; DR ¶

Design ¶

Updating the stack ¶

Creating the App Monitor ¶

Automating the RUM script ¶

Template for rum.js ¶

Syncing rum.js to s3 ¶

Results ¶

Key takeaways ¶

Extra Credit: Adding Web Vital Performance Budgets ¶

Wrap Up ¶

Paralleling Cypress with Sorry Cypress on Render

TL;DR ¶

Results ¶

Background ¶

Running Sorry Cypress ¶

Render Setup ¶

Running tests ¶

Wrap up ¶

Typescript with the OpenAPI specification

Summary ¶

Conclusion ¶

Compeller ¶

Typescript - OpenAPI Mapping ¶

1️⃣ OpenAPI => Typescript ¶

OpenAPI Generator ¶

Usage ¶

OpenAPI Typescript Codegen ¶

Usage ¶

Openapi Typescript ¶

Usage ¶

2️⃣ Typescript => OpenAPI ¶

NestJS ¶

TSOA ¶

3️⃣ Typescript <=> OpenAPI ¶

OpenAPI3 TS ¶

Usage ¶

AJV JSONSchemaType ¶

Usage ¶

JSON Schema to TS ¶

Usage ¶

Closing ¶

Build your own turborepo remote cache

AJV `JSONSchemaType` ¶