Abhik Pramanik

Culture

Kyra ran our first holiday book exchange, and it's been a hit with the team (to the point people mentioned it in our 1:1s). I added some photos from our recent meetups.


Ben added some improvements to our Slack guidelines. Sylvie added a way for us to learn more about each other in our weekly standup. We've been using both for over a month, and the team has been enjoying the changes!


How we use efficient and deliberate communication at Ashby to keep Slack notifications and meetings to a minimum, focus 36h out of a 40h week, and have fun along the way 😀.


How Thoughtful Communication Makes Us Unreasonably Productive

Engineering

aronow.io

Astronomer

Attentive

Aven

Benchling

Boulevard

Brightline brings virtual behavioral and mental health care to kids and teens, with a focus on also supporting families. They have no waitlists, provide customized plans, and work with families to track progress together.

Brightline

Brigit

Calm

Census

Clever

CloudTrucks

Coder is a remote infrastructure development platform that makes working with code easier than ever. Coder enables everyone from industry rookies to experts to develop code faster than ever before, using templatized starting points and ultra-secure workspaces.

Coder

Cohere

Cointracker

Color

Costanoa Ventures

Decimal

Deel is payroll for modern business. No matter where you're hiring, Deel helps customers run payroll, supporting 150+ countries with hundreds of worldwide experts. Deel helps their customers know that hiring, paying, and managing stayed compliant and easy in their all-in-one platform.

Deel

Deepnote

Deliveroo

Docker

Dune

Duolingo

EightSleep is more than a mattress company - it's a suite of products devoted to sleep fitness. Their goal is to make sure that your sleep is as high-performing as you are, helping you meet and exceed the bar for excellence in your waking life. Athletes and scientists alike love their range of highly customizable sleep tech because it helps them live healthier.

EightSleep

Format One

FullStory

Galileo

Ironclad

Juni

Linear

Marqeta is a modern card issuing platform dedicated to empowering innovators to launch their products to the world. They have over a decade of experience as a trusted, high-tech solution for builders who need simple, scalable payment solutions.

Marqeta

Mastodon

Modern Treasury

Monte Carlo is a Series D data observability company helping its customers accelerate the adoption of data to eliminate downtime. 

Monte Carlo

Multiverse

Notion is for notes, tasks, wikis, and databases. The app blends everyday work apps into one. It serves as an all-in-one workspace for note-taking, task management, and project management. 

Notion

Compensation software that offers 5,000+ companies clarity for every pay decision.

OpenComp

Opendoor

Plato

PostHog

PostHog is an open source analytics platform. Their product is used by product, engineering, and marketing teams to understand user behavior, including product analytics, session recording, user paths, a/b testing, heatmaps, and more. The suite, founded through Y Combinator in 2020, has since been installed by over 10,000 companies.

Pragma Platform

Pulley

Quora

Ramp

Readme

Replit

Retool

Scratchpad

Sequoia

Sift

Snyk

Sourcegraph

Stytch

Talentful

Treasury Prime

VanMoof is a Dutch e-bike company creating stylish, feature-rich city bikes. They're the fastest growing bike company in the world, with an active community in over 50 cities worldwide.

VanMoof

Vantage

Veev

Verkada

WeTransfer

Zippin

Recruiting

Product

Emma Cushman


Shipping is scary. 

Doing our jobs as engineers requires going to work every day and pressing a big green button that could, in principle, break everything. For Ashby to succeed as a company, we must press that button as often as possible.

The systems we employ to make pressing that button safe — static types, automated tests, code review, et cetera — weren't invented here. They are the legacy of decades of software engineers who made terrible mistakes, learned from their failures, and documented what they learned to build a safer environment for all of us.

Ashby's postmortem process is crucial to continuing that legacy of improvement through failure. But just as importantly, our collaborative and blameless postmortem culture also helps us feel safe, knowing that we'll have the team's support when something inevitably goes wrong. Together we'll work through the problem, understand what failed, and build resilient systems that will stop it from happening again.

Google's [SRE Book](https://sre.google/sre-book/postmortem-culture/) summarizes the basic concept of a blameless postmortem: "A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring." If you've worked as an engineer, you're undoubtedly already familiar with this idea.

In this article, I'll talk about the collaborative postmortem process we use: 
1. How we organize postmortems across time zones
2. How we decide who should run a postmortem (and when)
3. How we apply blamelessness to root cause analysis
4. Where we still have room to grow

## Postmortems Are Living Documents

At Ashby, a postmortem is a document created in our internal wiki (we use [Slab](https://slab.com/)), after resolving an incident. It serves as a record that outlines the nature of the incident and provides a timeline of relevant events. It is also the central location for collaborative discussion of the events, root cause analysis, and any learnings or next steps.

Our postmortems live in shared documents. This allows us to contribute to the timeline and evidence collaboratively and use comments to tag participants and discuss root cause analysis. Collaborating through a document helps us avoid costly meetings while ensuring that every stakeholder has a chance to contribute to the process —even those working asynchronously from a different time zone. (For more on our collaborative philosophy, see [How Thoughtful Communication Makes Us Unreasonably Productive](https://www.ashbyhq.com/blog/culture/thoughtful-communication)!)

To kick off the postmortem process, an engineer starts by making a copy of our postmortem template. They'll gather initial evidence, fill out a basic timeline, and tag any participants in the incident. Then, they send the draft around to the team. From there, we work asynchronously to complete the timeline, ask the questions necessary for root cause analysis and document any learnings or next steps.

Once we identify next steps, each one is ticketed as a GitHub Issue and added to our Postmortem Action Items GitHub Project. Then, we link each ticket from the postmortem document to easily see that everything was ticketed. The tickets go to an engineering manager to triage, prioritize, and assign. 

Quick fixes, such as adding a lint rule or a DataDog monitor, are often performed throughout writing the postmortem as soon as they're identified. Often these were already done during the incident response. In that case, we note that the task has already been done and does not need to be ticketed.

## Postmortem Owner Rotation

Who's responsible for doing all that? Ashby's flat organizational structure and focus on autonomy help us move faster with fewer resources than our competitors. Still, it presents a problem for postmortem analysis: we don't have a dedicated SRE team that can afford to investigate and document every incident. To write a postmortem, some engineer needs to take on the task.

In the early days of Ashby, this would often mean the engineer "responsible" for the incident would write up the postmortem. There are a few issues with that habit:

1. To know who should write the postmortem, we need to know what caused the incident. But determining the root cause of the incident is the purpose of the postmortem!
2. These postmortems would often start with a lot of "I" statements: "I shipped commit ead673a", "I misunderstood the documentation," and so on. "I" statements go against the basic tenet of blamelessness, as noted in PagerDuty's [The Blameless Postmortem](https://postmortems.pagerduty.com/culture/blameless/?utm_source=google&utm_medium=cpc&utm_source_platform=gads&utm_campaign=Global-Tr-FreeTrial-PdSrch-Google&utm_creative_format=dsa-tryitfree&utm_content=guides_dsa-1664588210868&utm_term=&matchtype=&network=g&utm_device=c&utm_marketing_tactic=dsa&gclid=Cj0KCQjwrs2XBhDjARIsAHVymmSafg6AQ-wXyiDX1JbzqmQv1MXnSwjLj6d6LmW_njoIReGA6jPmBhcaAiRNEALw_wcB): "The goal of the postmortem is to understand what systemic factors led to the incident and identify actions that can prevent this kind of failure from recurring. A blameless postmortem stays focused on how a mistake was made instead of who made it."
3. We take pride in the quality of our work, and when we ship broken code, it's natural to feel some guilt and personal responsibility for the consequences. Being responsible for organizing the postmortem adds a burden to the person who is already most psychologically affected by the incident. As the [SRE Book](https://sre.google/sre-book/postmortem-culture/) writes: "Writing a postmortem is not punishment —it is a learning opportunity for the entire company."

To address these issues, we decided to switch to a round-robin system for distributing postmortem ownership. When an incident happens, the next engineer in the rotation is responsible for gathering initial evidence, creating the Slab document, and facilitating collaboration and root cause analysis within the team. That helps us distribute the load of organizing postmortems while also engaging more engineers in the postmortem process. 

That has a few additional benefits. For one, it helps us combat confirmation bias —["the tendency to look for and favor information that supports one's own hypothesis"](https://www.infoq.com/articles/postmortems-etsy/). For example, in one recent incident, the on-call engineers noticed suspiciously timed anomalies in our Postgres metrics and concluded that a database issue was likely the culprit. Other evidence supported that theory. During the postmortem, another engineer discovered that the root cause was a network split in our Redis provider, and the database issue was completely unrelated. Often a theory of the root cause is built by first responders during triage, and having a third party reevaluate the sequence of events with fresh eyes can be invaluable.

Furthermore, involving more engineers helps us share knowledge, including specific information about our infrastructure and basic SRE skills. That helps us prepare for future incidents and enables any engineer to help during a response to an incident. The more skill we spread across the team, the better.

## When to Write a Postmortem

Each postmortem involves several engineers gathering evidence, analyzing, and assigning follow-up tasks. We want to reserve postmortems for serious and complex incidents that warrant the investment of engineering time. At Ashby, we don't formally assign a severity level to incidents (unless it's a security incident, which involves the security team and has its own SLA). For incidents that don't have security implications, any member of the team can request a postmortem for an incident. We use some guidelines to help determine when we'd generally want to write a postmortem:

* Incidents that involve loss (or potential loss) of customer data
* Incidents involving substantial or long-lasting degradation of core features, or any app downtime
* Incidents that block key workflows for multiple customers
* "Incidents" that turned out to be false alarms but still paged or caused extra work for the on-call engineer

When a postmortem isn't warranted, we still encourage engineers to write retrospective documents and distribute learnings to the team. However, out of respect for the postmortem process, we consider these documents "Retrospectives" rather than "Postmortems." The word choice makes it clear to everyone when we follow our exact postmortem process.

## Root Cause Analysis

A good root cause analysis should answer the question, *What could we change to prevent a similar incident from happening in the future?* The answers to that question become our follow-up tasks.

We often start our analysis with the [Five Whys](https://en.wikipedia.org/wiki/Five_whys): after answering each question about the incident, ask "Why?" again until you arrive at a root cause. However, we're also sensitive to the tendency for such a linear series of questions to terminate at "first stories," i.e., stories revolving around human error. As John Allspaw writes in [The Infinite Hows](https://www.oreilly.com/radar/the-infinite-hows/):

> "When it comes to decisions and actions, we want to know *how it made sense for someone to do what they did*. And make no mistake: they thought what they were doing made sense. Otherwise, they wouldn't have done it."

So, we don't strictly follow any formal approach. Instead, we ask open-ended questions that seek diverse narratives, digging into the superficial causes of a problem to find the systemic or procedural failures which allowed the problem to become as serious as it did.

Finding the root cause is always a collaborative process. We help each other to arrive at a deeper understanding of the incident by being curious, asking good questions, and using blameless language that allows engineers to acknowledge their mistakes comfortably.

## Blameless Language

Since the purpose of a postmortem is to build better processes, focusing on the actions of individuals can be distracting at best. At worst, it can make engineers feel unsafe when making changes to the codebase or taking decisive action in an emergency.

To combat this, we make a deliberate effort to use blameless language in a postmortem. That means using language that implicitly assumes the good faith of all participants and focuses on failures of systems and documented processes rather than the actions of individual engineers.

In particular, the body of the postmortem never contains the names of individual engineers. Instead, we keep a list of people involved in the incident in a separate *Participants* section. Rather than saying "@person shipped 2aab2c9", we say "2aab2c9 was shipped". The difference may seem inconsequential —anyone who cares can look up which engineer shipped 2aab2c9. But it can make a big difference to that engineer and help them shift from a defensive posture to a collaborative one. 

The goal is twofold: to focus on improvements we can make as a team and provide individuals with a visible sign that their actions will be interpreted in good faith. Using this language tells everyone involved that the team will always support each other in our response to mistakes and emergencies. 

That support is essential for engineers who are early in their careers; shipping a major bug for the first time can shatter the confidence of a new engineer, severely hurting both their productivity and well-being. Careful language choice makes it natural to reframe their mistake as a learning opportunity and can even build their confidence to take measured risks in the future.

## Room to Grow

Like everything at Ashby, our postmortem process isn't set in stone, and we're constantly updating it in response to feedback from stakeholders. For example, our QA team recently communicated that it wasn't always obvious which issues weren't purely infrastructural and could have been prevented by improvements to our QA process. To help, we've added a section to our postmortem template that describes how the incident manifested to users.

One major ongoing work in progress is how we prioritize issues from past postmortems. Usually, the most urgent improvements are made by participants during the postmortem process, if not during the incident itself. Less urgent issues are often moved to product area backlogs to be addressed by individual teams according to their bandwidth. 

But many issues uncovered during postmortems are things that aren't critical *today* but will likely become urgent or even dangerous in the future —maybe after we scale or make a planned migration to a new piece of infrastructure. For those issues, keeping a close eye on them, and allocating the right resources at the right time, is crucial to preventing incidents. That's a part of the process we're still working on getting precisely right.

## Impact

Before formalizing this process, our postmortems were conducted ad hoc and inconsistently. Whether we wrote a postmortem and documented (and completed) follow-ups varied greatly depending on who was involved in the incident.

As a result of a more consistent process, we've implemented many improvements to the systems that protect us from our own mistakes —improvements to developer experience, CI, infrastructure, documentation, and more. At the same time, our blameless culture has provided comfort and psychological support to several engineers who have gone through the rite of passage of shipping their first major bug and being at the center of an incident. 

Shipping is scary. It won't ever be entirely safe to push that big green button. But at Ashby, we're always working to make it *feel* a little bit safer.

## Appendix

### "Blamelessness" in Aerospace and Medicine

The origins of the concept of a "blameless postmortem" are well known in engineering lore, as the [SRE Book](https://sre.google/sre-book/postmortem-culture/) states: 

> "Blameless culture originated in the healthcare and avionics industries where mistakes can be fatal. These industries nurture an environment where every "mistake" is seen as an opportunity to strengthen the system. When postmortems shift from allocating blame to investigating the systematic reasons why an individual or team had incomplete or incorrect information, effective prevention plans can be put in place." 

However, it's worth noting that the concept of "blamelessness" applied in aviation and medicine differs from what we practice: mistakes often have personal consequences. 

The National Transportation Safety Board, or NTSB, conducts a [detailed investigation](https://www.ntsb.gov/investigations/accidentreports/Pages/Reports.aspx) of every aviation accident in the United States, and their reports and recommendations are widely credited for the incredible safety record of US airlines. But where human error is a factor in an accident, [it's fairly common](https://www.govinfo.gov/content/pkg/GAOREPORTS-T-RCED-98-253/pdf/GAOREPORTS-T-RCED-98-253.pdf) for the FAA to revoke or suspend that individual's license to operate an aircraft. 

Likewise, in medicine, where the term "postmortem" has a literal meaning, the role of human error is more nuanced. Preventing future mistakes is essential, but so is punishing negligence or wrongdoing: 

> "Although a culture that does not place blame was a step in the right direction, it was not without its faults. This model failed to confront individuals who willfully and repeatedly made unsafe behavioral clinical practice choices. Disciplining health care workers for honest mistakes is counterproductive, but the failure to discipline workers who are involved in repetitive errors poses a danger to patients. A blame-free culture holds no one accountable and any conduct can be reported without any consequences. Finding a balance between punishment and blamelessness is the basis for developing a [Just Culture](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5424837/)."

As with most software engineering teams, the stakes at Ashby are never life or death. These low stakes allow us to lean into a radically blameless culture that prioritizes psychological safety, risk-taking, and high velocity. Mistakes are an acceptable tradeoff for shipping more code faster.


How Ashby's collaborative postmortem process adapts industry best practices to a global, remote team and helps us move faster, learn from failure, and build a healthier engineering culture.


Async Postmortems in a Global, Remote Team

João Carvalho


Node.js is a fantastic piece of technology, and we use it as the foundation for our application servers.

There is, however, one big caveat when working with Node: our application code runs ([for the most part](https://nodejs.org/en/docs/guides/dont-block-the-event-loop/)) in a single thread (the Event Loop thread). Running in a single thread is fine when the application spends most of its time doing asynchronous I/O since each request consumes only a small amount of CPU. However, when your requests perform CPU-intensive computation, all concurrent requests will be blocked waiting for it to complete!

As the Ashby backend grew in size, we ran into some CPU-intensive code paths, either due to non-optimized code that didn't scale very well or the lack of result set limits (which could cause an endpoint to return an arbitrarily large amount of data).

Join us as we take you through our journey to identify and eliminate requests that block with the help of community NPM packages and our favorite monitoring tool, Datadog.

## The Node Event Loop: a Primer

<Hint type="info">
This section is a brief introduction to the Node Event Loop, meant only to provide the necessary context for the rest of this post. If you want to dive deeper into this topic, we recommend [Node's excellent documentation on the topic](https://nodejs.org/en/docs/guides/dont-block-the-event-loop/).
</Hint>

At a very high level, the Event Loop consists of a single thread running in a continuous loop, going through multiple phases, as described below:

![Node Event Loop Phases](/images/event_loop_blockers/event_loop_phases.png "Node Event Loop phases")

Each phase contains a callback queue of all callbacks that need to be processed in that phase. These include various tasks, such as processing timers (scheduled by `setTimeout` and `setInterval`), processing I/O events, and executing I/O callbacks (including the code that handles incoming HTTP requests).

All callbacks in the Event Loop are processed sequentially. Suppose a specific callback performs a CPU-intensive operation without yielding the Event Loop. In that case, all other callbacks will remain in the queue and not be processed until the Event Loop frees. We call these "Event Loop Blockers."

## Detecting Event Loop Blockers

When we set out to look for solutions to detect Event Loop Blockers, we found two great libraries in npmjs.com: [blocked](https://github.com/tj/node-blocked) and [blocked-at](https://github.com/naugtur/blocked-at) (in both cases, we used version `1.2.0` with Node 16). Both provide a similar interface: a threshold value and a callback that gets invoked when the Event Loop gets blocked for longer than the threshold.

One critical difference between the two, which stems from the fact that their implementations are radically different, is that `blocked-at` provides the callback with the stack trace for the Event Loop Blocker's entry point, whereas `blocked` only provides the Event Loop Blocker's duration. This extra information came at a cost: the performance hit of using `blocked-at` is significant, as indicated in their README:

> It's recommended to detect blocking exists with something without the performance overhead and use `blocked-at` in testing environment to pinpoint where the slowdown happens. Rule of thumb is you should not be running this in production unless desperate.

Let's look at how each of these libraries work and why the cost of capturing the stack trace is so high.

### `blocked v1.2.0`

The mechanism behind `blocked` is ingeniously simple (the whole library comes in at just [518 bytes](https://github.com/tj/node-blocked/blob/master/index.js)). When installed, it registers a timer (using
`setInterval`) to keep track of the time elapsed between invocations. The added delay between invocations can be a good indicator of an Event Loop Blocker since concurrent operations that block the Event Loop may delay timers from being fired. On every execution of the timer, if the drift between the desired and actual invocation times is higher than the configured threshold, the callback is invoked.

Since Event Loop Blocker detection happens in a different context from the running application (within the timer), it cannot gather any information about the source of the Event Loop Blocker. On the other hand, the work on each iteration is so fast that its overhead is minimal.

### `blocked-at v1.2.0`

The `blocked-at` library uses [Async Hooks](https://nodejs.org/api/async_hooks.html) to capture the additional
information about the stack trace. Node recently introduced the Async Hooks API to track asynchronous resources (such as Promises) within a Node process. A developer can register callbacks on asynchronous resource creation (`init`), when its callback is invoked (`before`), and when its callback finishes (`after`).

The `init` hook is called within the context where the resource is created, so we can capture the [current stack trace](https://nodejs.org/api/errors.html#errorcapturestacktracetargetobject-constructoropt). `blocked-at` does precisely that and stores the trace in a global map, indexed by the current `triggerAsyncId` (the mechanism Async Hooks use to identify asynchronous resources). Then, once the resource's callback is ready to run, the `before` hook is invoked, capturing the invocation's timestamp (and once again stored in the global
map where the trace is already stored). Once the callback finishes, the `after` hook is invoked, and the current timestamp is compared against the previously stored timestamp from the `before` invocation. If the difference is greater than the configured threshold, the callback is called, passing along the stack trace captured in `init`.

This approach, while providing valuable information, is too slow to be feasible, due to the following reasons:
- Async Hooks run on *all* asynchronous resources and may cause significant overhead. Recent versions of
  Node have improved this overhead considerably (we are on Node 16, so this isn't much of an issue).
- We can only capture the stack trace in the `init` hook, so before we know whether we need to invoke the callback, we waste a considerable amount of resources. Capturing a stack trace is a relatively expensive operation, and our code will discard it most of the time (when the Event Loop is not blocked).

## Our First Attempt

In our first attempt at finding our Event Loop Blockers, we wanted to get as much information as possible, so we started by measuring the performance impact of `blocked-at`.

After deploying it to our staging environment, we noticed that the response times for our most common GraphQL queries were a few orders of magnitude slower. Our test traffic was enough to consume all the available CPU resources in our staging instances! We also realized that the provided stack trace information wasn't particularly useful. In most cases, it pointed to the Apollo Field Resolver, which is responsible for invoking our code, instead of pointing to the code itself.

With this in mind, we circled back and replaced `blocked-at` with `blocked`. Even without the stack trace, we would at least be able to identify when the Event Loop was blocked and, with some manual investigation, figure out where the Event Loop Blocker occurred in our code.

With `blocked` successfully deployed in production, with minimal impact to performance or resource usage (we noticed approximately 0.5% increased resource usage with the timer running every 100ms), we were, as expected, able to identify critical code paths that were contributing to the blocking issues. However, each event took nearly 5 minutes to investigate, and with hundreds of blocking events per hour, this quickly proved unsustainable.

## Datadog Integration

We use multiple Datadog products in our monitoring stack, particularly [Datadog APM](https://www.datadoghq.com/product/apm/) and [Datadog Log Management](https://www.datadoghq.com/product/log-management/). When using these two products, we can [connect Logs and Traces](https://docs.datadoghq.com/tracing/connect_logs_and_traces/) and view all log lines generated in the context of a specific HTTP Request Trace in a single interface. In our short-lived experiment with `blocked-at`, we noticed that the logs correctly connected with the trace for the request that generated them.

After some investigation, we noticed that the Async Hook used by `blocked-at` allowed us to access the reference to the
"current trace" (as Datadog was also using Async Hooks to propagate its context). What if
instead of capturing the stack trace, we used the Async Hook to surface the blocker in the Datadog Trace?

We started by writing our own Async Hook with heavy inspiration from `blocked-at`. Our first version did little more
than detect when a blocking event happened:

```typescript
const cache = new Map<number, [number, number]>();

function before(asyncId: number) {
  cache.set(asyncId, process.hrtime());
}

function after(asyncId: number) {
  const cached = cache.get(asyncId);
  if (cached == null) {
    return;
  }
  cache.delete(asyncId);

  const diff = process.hrtime(cached);
  const diffNs = diff[0] * 1e9 + diff[1];
  if (diffNs > thresholdNs) {
    const time = diffNs / 1e6;
    logger.warn({
      label: "EventLoopMonitor",
      message: `Event loop was blocked for ${time}ms`,
      metadata: {
        time,
      },
    });
  }
}

const asyncHook = createHook({ before, after });

asyncHook.enable();
```

In the `before` hook, we are capturing the current timestamp (side note: we are using the deprecated `process.hrtime()` method, instead of `process.hrtime.bigint()`, as the latter has a considerable performance impact in Node 16), and storing it in the cache, indexed by the current Execution Async ID (like `blocked-at` does).

Then, in the `after` hook, we fetch the initial timestamp from the cache and compare it against the current timestamp. If the diff exceeds the configured threshold, we log a warning message (like in `blocked-at`, which is associated with the current Datadog Trace).

The final piece of the puzzle is surfacing the block in the trace itself, which we achieved by adding the following to the code path where the block is detected:

```typescript
const span = requestTracer.startSpan("EventLoopBlock", {
  childOf: requestTracer.scope().active() ?? undefined,
  startTime: new Date().getTime() - time,
  tags: {
    [tags.SERVICE_NAME]: "ashby-backend-event-loop-monitor",
  },
});
span.finish();
```

Here, we manually create a trace span named `EventLoopBlock`, which we create as a child of the "current trace." We also set the start time to be the current timestamp minus the total block time (the use of `new Date().getTime()` means we lose sub-millisecond precision, but this is acceptable for our use case). Doing this causes the span to be
rendered as part of our trace:

![Trace with Event Loop Blockers](/images/event_loop_blockers/trace.png "Trace with Event Loop Blockers")

This approach meant that the total amount of work done in the "good" case (where no blockers are detected) is much
smaller than in `blocked-at`: two API calls to fetch the current timestamp, two cache writes, and one cache read. The
performance impact of this approach was much more acceptable to us, ranging between 1% to 2% additional CPU usage (and a comparable increase in total response time) when deployed in production.

## Results

Since we deployed our new Event Loop Monitor, we have been able to identify most of the sources of Event Loop Blocking,
and have both reduced our total number of Blocker events by and p99 latency of said events by up to 80%!

![Event Loop Blocker events over time](/images/event_loop_blockers/events.png "Event Loop Blocker events over time")
![Event Loop Blocker p99 latency over time](/images/event_loop_blockers/latency.png "Event Loop Blocker p99 latency over time.")

## References

- https://nodejs.org/en/docs/guides/dont-block-the-event-loop/
- https://nodejs.org/en/docs/guides/timers-in-node/
- https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick/
- https://nodejs.org/docs/latest-v16.x/api/process.html#processhrtimetime


Our application code runs in a single thread with Node.js. The thread can get blocked by requests with CPU-intensive computation and significantly slow our response times. We'll walk you through how we identify and eliminate these blocking requests.


Detecting Node Event Loop Blockers

Florian Krüger


Once you've picked a rich text library for your application, switching to a new one isn't easy and carries many risks. We recently changed the rich text library we use at Ashby, and, with research and planning, we were able to do it with minimal issues in production. We wanted to share that research and process with the hope that it helps you make a similar switch. 

This post is part one of a two-part series about how we switched from Slate to [tiptap](https://tiptap.dev/). Today, we'll explain the process we used to decide on tiptap, and our follow-up post will describe how we made the switch.

## Time to Switch

We added rich text editing to Ashby in 2019. We evaluated multiple solutions to make our decision (including [Quill](https://quilljs.com/) and [Draft.js](https://draftjs.org/)), and eventually went with [Slate](https://www.slatejs.org/examples/richtext).

Like many libraries back then, Slate was in beta but quickly improving and expanding. We took a chance that Slate would support more browsers and its API would remain stable (despite the maintainers strongly suggesting that both might not happen!). Unfortunately, our gamble didn't pay off, and the maintainers introduced significant breaking changes to the API and document model in `0.50.x`. These changes were the right direction for the future of Slate but would require a lot of work for us to migrate to —work we couldn't do as a small engineering team (even with the [great support the community was providing](https://github.com/ianstormtaylor/slate/issues/3215)).

We stayed on `0.47.x` as long as we could, but our users were growing, and we needed to upgrade Slate to fix issues and support more browsers. There were more rich text editor libraries to choose from since 2019, so we decided to evaluate switching to a new library in addition to upgrading Slate.

## Find All Possible Candidates

We created a comprehensive list of rich text editor libraries to evaluate. While Google is useful, ranking in Google Search is a popularity contest, not a quality test. To be exhaustive, we started with the initial list of libraries we found back in 2019 and used the references to other libraries in their documentation to find more. For instance, Slate lists Draft.js, Prosemirror, and Quill as inspirations [in their introduction document](https://docs.slatejs.org/). Quill [lists](https://quilljs.com/guides/comparison-with-other-rich-text-editors/) CKEditor, TinyMCE, Draft.js, Prosemirror, and Trix. Using this method, we gathered a list of more than ten candidates.

## Find Showstoppers

We went through the candidates and made a list of key library attributes using available documentation, blog posts, and other information we could find. Some key attributes that we collected across the board were:

- **Beta or stable**: It's not a showstopper if a library is in beta, but it's essential to know if using it introduces the risk of breaking changes.
- **Stars on GitHub**: This is not a quality measure by any means, but it gives you a feeling of the library's community and how active it is.
- **React support**: We built our app with React, so the ease of using the library in React is essential.
- **Browser support**: This was one of our main pain points with Slate `0.47.x`, so we wanted to ensure that all major browsers on desktop and mobile had good support.
- **Age of last release**: This shows how actively maintained the library is. Stable or not, the landscape around javascript libraries is constantly evolving, so there's always something to be done. No app is ever "finished."
- **Age of last commit**: Like the age of the last release, this gives you a feeling of how active the community is, even if the latest release has aged, and a proxy for how quickly issues are fixed.
- **Current version**: If we ever looked back at our notes, we knew what version we had evaluated and see if things changed since then.

Typical showstoppers we identified:

- Missing or outdated support for React
- Missing mobile support (or no clear stance in that area)
- Hard to visually customize
- Missing extensibility or unclear documentation around extensions or plugins
- Small community or no recent updates

In addition to the key attributes, there were common features that we paid attention to across all libraries. The number grew as we looked at each library—we'd often find something that stood out in one library and see if other libraries supported it. For instance, the available extensions in tiptap satisfied most of our needs (including @-mentions) and began to compare that across libraries. 

We exited this phase with a shortlist of five libraries: Slate, Quill, Prosemirror, ReMirror, and tiptap.

## Prototyping

We didn't write a single line of code until this point, but it was time to start building quick prototypes to get a better feel for each library.

The goal of each prototype was to come as close as possible to the current state of the editor (visually and functionally). To keep each prototype around half a workday and come to an airtight statement:

- We used our existing frontend components (e.g., buttons, icons, CSS).
- We didn't implement custom extensions —if it wasn't a ready-to-use feature, we left it out (e.g., @mentions). 
- We didn't implement interoperability with the current editor and its document model. Each prototype would only work standalone, with a new document from the library's documentation.

Eventually, we were able to compare a couple more things about our shortlist candidates:

- Developer experience
- Documentation quality
- Customizability
- User experience

## Gather Data

With this experience, we listed the advantages and disadvantages of each of the libraries based on our needs. These might not be valid at the time you read this, or you might consider something that we identified as an advantage as a disadvantage (and vice versa).

We collected the data (including version numbers and GitHub stars) in May 2021. All conclusions have been drawn based on the library version listed next to the names. The libraries might have evolved significantly since then.

### Slate `0.63.0` (20.6k ⭐)

We felt sure that migrating to the latest version was similar to starting from scratch. A significant disadvantage was that we couldn't have both `0.47.x` and `0.63.0` in our application simultaneously. Not using both simultaneously prevented us from gradually migrating features, and we'd need to do a complete rip and replace.

We liked that Slate was built for React (and didn't need a wrapper), had a large user base, was actively maintained, and had prior experience with it. However, it's still in beta, android support is experimental, and the maintainers still warn users about breaking API changes.

### Quill `1.3.7` (29.9k ⭐️)

Quill is popular with well-known brands and officially maintained by Slab, a wiki we use for our internal documentation (and love!). Quill's main advantages were its popularity, its recommendations within our network, and that it had a stable release.

Quill itself doesn't offer any React support, and for our prototype, we used react-quill, a third-party dependency in beta. Although it's actively maintained, the current stable release is from 2019 (there's an ongoing effort for a new major version).

### ProseMirror (5.7k ⭐️)<Footnote>Prosemirror comes in multiple separate packages, so it's hard to list "the version" we looked at here. Back in summer last year, the following tags of the four main components of prosemirror were the current ones: prosemirror-model: `1.4.4`, prosemirror-state: `1.3.4`, prosemirror-view: `1.18.6`, prosemirror-transform: `1.2.5`</Footnote>

We considered going directly with ProseMirror (a "toolkit for building rich-text editors on the web") and not a ready-to-use rich-text editor. ProseMirror would've given us the most control over the editing experience, but we decided the overhead of implementing our editor with the ProseMirror toolkit was too much work.

To us, Prosemirror stood out as the most powerful solution. We wouldn't have needed any third-party dependencies (except for a React integration) and would have been able to define our document schema from the ground up. We didn't go with it because it took a lot of code to get started (even the official ProseMirror guides [state this](https://prosemirror.net/examples/basic/)) and defining our custom schema, while powerful, was not worth the effort since we didn't have many custom requirements.

We still liked ProseMirror, which led us to two available React integrations: ReMirror and tiptap.

### ReMirror`1.0.0-next.60` (1.1k ⭐️)

We liked many things about ReMirror. It's a ProseMirror editor built on React and comes with extensions that covered most of our requirements.

It was hard not to go with ReMirror since we liked it as much as tiptap, but when evaluating both frameworks in May 2021, tiptap felt a little more mature.

<Hint type="info">
Since we evaluated the rich text editor libraries, ReMirror made impressive progress. The library became stable, added many extensions, and the community grew. There's no longer much difference between ReMirror and tiptap, and it's a great example of how quickly a library can change. While we're still delighted with tiptap, today's ReMirror would have met our needs!
</Hint>

### tiptap `2.0.0-beta60` (10.8k ⭐️)

Tiptap ticked a lot of boxes for us. The library (like ReMirror) builds on top of ProseMirror. While it was created initially for Vue and relied on a third-party wrapper for React, it gained official react support in `2.0.0`. Tiptap is in beta, so breaking changes are still possible, and the maintainers almost change things daily. Its API was the simplest of all libraries that we used in the prototyping phase, and we also liked that we could fall back to native ProseMirror code if necessary.

As with ProseMirror and ReMirror, tiptap met most of our requirements with its available extensions, so we only needed to implement a hand full of them. A drawback with all of the ProseMirror-based editors (and ProseMirror itself) is that ProseMirror is not a React library, so some bridging between vanilla javascript and React is necessary.

## Showdown

A library can work great in a sandbox environment, but integrating it with an existing codebase can surface issues. We wanted to find these issues as early as possible, so we implemented the first draft of Ashby's rich text editor with our favorite library, tiptap.

We paired the first draft with our usual writing process and technical specification to tease out more issues and tasks. The specifications covered motivations, goals, research, and a detailed roadmap of how we'd implement the new editor and make the switch in production.

I shared the specification and first draft implementation (via a pull request) with the team to get feedback.

## Decision

Once we answered all questions and responded to all comments, it was clear tiptap would be the best solution for our needs and we were ready to use it as Ashby's new rich text library.

In part two, we'll share our migration plan from one library to the other and the challenges we faced executing that plan and releasing to production.


Switching rich text libraries in a production app with many daily users isn't easy. We recently changed ours with minimal issues and are sharing our research and process with the hope that it helps you make a similar switch. 


Switching Rich Text Editors, Part 1: Picking Tiptap

Ben Asher


At Ashby, we leverage GitHub Actions for lots of small automated tasks. We recently introduced a reminder to deploy changes to production when the number of commits in the deploy passes a specified threshold.

For some background, we deploy by merging `develop` into `master` (via a merge commit). `develop` consists of commits squashed from pull requests. We default to smaller pull requests, so commits may pile up on `develop` if nobody needs to deploy to production right away. Suppose a production issue happens after a deploy and the deploy has lots of commits. In that case, it's harder for our team to roll back (e.g., it has database migrations) and harder to identify the commit(s) that caused the issue. 

To avoid letting deploys accumulate lots of commits, we wanted to figure out a continuous deployment strategy that would keep deploys small. We considered several strategies and, for now, landed on this automated reminder. The reminder has several benefits:

- Deploys are always attended. We wanted to avoid issues where an automated solution might deploy right when everyone signs off for the day in North America or Europe.
- No changes to our deploy process were needed. We're hiring [roles to dedicate more time to improving our deployments](https://jobs.ashbyhq.com/ashby/?departmentId=2c32d70f-d7cb-4a06-bd87-048084e3eb10), but we wanted to reduce deploy size while minimizing immediate effort.

Here's our GitHub Action in its entirety:

```yaml
name: Please Deploy
on:
  push:
    branches:    
      - 'develop'

jobs:
  please-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository code
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Maybe Ask for Deploy
        run: |
          if [ "$(git rev-list --count origin/develop ^origin/master)" -gt "$MAX_COMMITS_BEFORE_NOTIFY" ]; then
            curl -X POST \
              --data '{ "channel": "C02D678LP29", "text": ":rotating_light: The deploy is getting too big! Can someone (on-call?:pray:) please deploy? https://github.com/ashbyhq/Ashby/compare/master...develop :rotating_light:" }'\
              -H 'Content-Type: application/json; charset=utf-8'\
              -H "Authorization: Bearer $PROD_NOTIFY_SLACK_TOKEN"\
              https://slack.com/api/chat.postMessage 
          fi
        env:
          PROD_NOTIFY_SLACK_TOKEN: ${{ secrets.PROD_NOTIFY_SLACK_TOKEN }}
          MAX_COMMITS_BEFORE_NOTIFY: 4
```

The GitHub Action sends an automated message to our #ping-engineering channel when `develop` is behind `master` by five commits, ensuring that we keep the ship shipping throughout the day!

![Slack Message Screenshot](/images/gridhub-actions-deploy-reminder/slack.png "The automated message in Slack")


An automated reminder to deploy as an effective way to keep deploys small and safe.


Continous Deployment via a GitHub Actions Reminder


ES2022 introduces `Array.prototype.at` and we've come to love using it. `Array.prototype.at` can replace using the standard index operator `[]`:

```typescript
const arr: number[] = [];

arr[0].toString();

// Same as above!
arr.at(0).toString();
```

In the example above, even though index 0 may not have anything there, Typescript is happy to allow us to to call `toString()`. With `.at()`, we get a compiler error:

![Typescripts Compiler Error](/images/array-at/compiler_error.png "Typescript's compiler error.")

`Array.prototype.at` is not officially supported in all versions of Node. Check [node.green](https://node.green/) to see if you can use it!

Array.at provides better type safety than the standard index operator, and it works with Node 16!


Type Safety with ES2022's Array.prototype.at


[`jq`](https://github.com/stedolan/jq) is a tool I use at least two to three times a week (maybe daily when it's my week to investigate sentry issues in production), so I thought I'd write about it and its XML offshoot, `xq` (via [yq](https://github.com/kislyuk/yq)).

A lot of our day-to-day work involves reviewing API request and response data in structured text formats, whether it be debugging, reviewing input or output formats for vendor APIs, etc. Often, I find I have a question about a JSON blob, and that blob is too large to inspect by just giving it a "once over." Enter, `jq`. In it's most basic form, `jq` lets you query JSON documents, but it's also useful for linting and formatting minified JSON.

## Basic Usage

Let's say I have the following JSON blob:

```json
{"firstName":"Ben","lastName":"Asher","email":"ben@example.com"}
```

Copy this to your clipboard and run the following command (after you install `jq` with, for example, `brew install jq`<Footnote>Some other ways to install `jq` are [here](https://github.com/stedolan/jq)</Footnote>)

```bash
pbpaste | jq '.'
```

This prints the following:

```json
{
  "firstName": "Ben",
  "lastName": "Asher",
  "email": "ben@example.com"
}
```

What we did is pipe the output of our clipboard into `jq` and use the `jq` command `.` to print it with better formatting (and colors!). If the JSON were invalid, it would tell you where it's invalid. I've found this immensely useful for quickly fixing issues in our public API documentation since our API documentation vendor requires valid JSON in our sample responses that we present readers.

## Querying JSON

Things get even better when you need to query JSON. I often find myself investigating a Sentry error and debugging an issue that requires inspecting JSON involved in the error. Let's say I have a list of people in JSON format:

```json
[
  {
    "firstName": "Ben",
    "lastName": "Asher",
    "email": "ben@example.com"
  },
  {
    "firstName": "Ronald",
    "lastName": "Martin",
    "email": "ronald@example.com"
  }
]
```

If I want see just emails, I can do `jq '.[].email'`. That `jq` command string tells it to look at each array element (you can also do `.[0]` to select just the first one, for example), and from each one, select the `email` attribute. This prints:

```txt
"ben@example.com"
"ronald@example.com"
```

For a recent issue, I had data like the JSON above, and, if one of the object fields were missing, the backend code threw an error. The amount of data was too large to inspect manually. With `jq`, I could query each attribute, and, if `jq` couldn't find it, it printed `null`. So, to find which attribute was the culprit, I did `jq '.[].email' | grep 'null'` and replaced `email` with each sibling attribute until I got a `null`.

## Querying XML

`jq` is super powerful, but recently I wanted to do something like the example above, but with XML. Our customer job feeds for vendors are in XML and often contain thousands of jobs. I found out a new vendor would not ingest jobs that didn't have a valid two digit country code. Before we launched with the vendor, I needed to figure out how many jobs would be impacted. I figured there must be an equivalent for XML, right? There is, and it's a [tool called `yq`](https://github.com/kislyuk/yq) that wraps `jq` and supports multiple other formats. It ships with a tool called `xq` that you can use for XML.

Back to my task, I needed to figure out how many job postings in the feed would be dropped by the vendor. For simplicity, I'll use the snippet below as our "feed" and walk you through what I did. You can install `yq` with `brew install python-yq`.

Let's copy the feed to our clipboard:

```xml
<?xml version="1.0" encoding="utf-8"?>
<source>
  <publisher>Ashby</publisher>
  <job>
    <title><![CDATA[Product Engineer]]></title>
    <country><![CDATA[US]]></country>
  </job>
  <job>
    <title><![CDATA[Full-Stack Engineer]]></title>
    <city><![CDATA[Remote]]></city>
    <country><![CDATA[CA]]></country>
  </job>
  <job>
    <title><![CDATA[Pilot]]></title>
    <city><![CDATA[Remote]]></city>
    <country><![CDATA[PT]]></country>
  </job>
  <job>
    <title><![CDATA[Software Engineer]]></title>
    <city><![CDATA[Remote]]></city>
    <country><![CDATA[DE]]></country>
  </job>
  <job>
    <title><![CDATA[VP of Sales]]></title>
    <city><![CDATA[San Francisco]]></city>
    <country><![CDATA[United States or Canada]]></country>
  </job>
  <job>
    <title><![CDATA[Data Analyst]]></title>
  </job>
</source>
```

Then save the contents of our clipboard to a file:

```bash
pbpaste > feed.xml
```

From here, I want to query countries. Countries are at the path `.source.job[].country`. I can do the following to quickly check them out:

```bash
cat feed.xml | xq '.source.job[].country' | less
```

Looking at that output, I see some `null`s and some country codes, so I know I'm in right place.

Stepping back, our vendor only accepts two digit country codes. The way our feed code works is that we look up the user-inputted country's country code (for example, via [node-countries](https://www.npmjs.com/package/node-countries)). If we can find the country code, we use it, and, if we can't, we use the original value (people sometimes input invalid countries like "Remote" or "United States or Canada"). Also, some won't have countries, so we see some `null`s.

## Operators

To know what the vendor is going to drop, we can use the length of the country field (only ones of length two will work), which `xq` (and `jq`) can calculate! To get the length of all country codes, I can do this:

```bash
cat feed.xml | xq '.source.job[].country | length' | less
```

That's right! `xq` and `jq` have their own operators, which you can pipe output to within the command. The operator above will output the length of each country, and looking at the output, we can see that we're getting `0`s where we had `null`s before, and `2`s for most things (phew!). To get our final answer, we can run the following command:

```bash
cat feed.xml | xq '.source.job[].country | length' | grep -v '2' | wc -l
```

This takes that output that we were viewing in `less` and pipes it to `grep`. With `-v`, we tell `grep` to only give us output that doesn't match, so that gets us all of the "bad" countries. Piping that to `wc -l` will count the number of lines and give us our answer.

## Conclusion

`jq` and its offshoots are super powerful, and I use them all the time, even if it's just the basic path-based searching of JSON files (or formatting to make smaller ones nicer to read). For everything else, I can usually find a useful operator (via Stack Overflow or [`jq`'s docs](https://stedolan.github.io/jq/manual/)) that'll help me put together fancier queries.


How I use command line tools to inspect JSON and XML data that would be impossible to do by eye.


Querying JSON and XML with jq and xq

Graham Murdoch


In Typescript, when defining a type, it's nice to use a union of [literals](https://www.typescriptlang.org/docs/handbook/literal-types.html):

```typescript
type Sizes = "sm" | "md" | "lg"
```

This allows developers to quickly enter a valid value without having to import and expand an enum. In an editor like VSCode, you get a nice autocomplete prompt with the possible values.

However, sometimes you want a union of literals plus some flexibility for custom values. In this example, we want to allow CSS length units (e.g. `"100px"`). 

A simple solution is to combine the literals with a generic type that is a supertype of the literals:

```typescript
type Sizes = "sm" | "md" | "lg" | string
```

Adding the generic type weakens the type coverage since you can enter something potentially invalid that compiles (like `"lgg"`). You've also broken autocompletion for this type. VSCode, for example, won't show the literals as options in a useful way anymore.

One possible approach is to use a generic type that doesn't match the literal values (e.g. an object):

```typescript
type Sizes = "sm" | "md" | "lg" | { custom: string }
```

This restores autocompletion, some type coverage, and preserves a fallback option. However, it isn't quite what you want for CSS length units (and requires some superfluous decoration). 

## LiteralUnion

Prior to Typescript 4.1, a better solution was to use the `LiteralUnion` utility from the [`type-fest` package](https://github.com/sindresorhus/type-fest):

```typescript
import { LiteralUnion } from "type-fest"
type Sizes = LiteralUnion<"sm" | "md" | "lg", string>
```

While it still allows invalid values, autocompletion of the literal values works again!

## Template Literals

As of Typescript 4.1, we can do better. Using [template literal types](https://www.typescriptlang.org/docs/handbook/2/template-literal-types.html), we can more accurately model (and restrict) the range of values:

```typescript
type Sizes = "sm" | "md" | "lg" | `${number}fr` | `${number}px`;

const size: Sizes = "sm"     // ✓
const size: Sizes = "1fr"    // ✓
const size: Sizes = "100px"  // ✓
```

### Working with Enums

At Ashby, some of our older code uses enums and, for consistency, we continue to use them as the basis for possible values.

Historically, you couldn't really combine the approaches:

```typescript
enum DesignSystemSizes {
  sm = "sm",
  md = "md",
  lg = "lg"
}

type Sizes = DesignSystemSizes.sm | DesignSystemSizes.md | DesignSystemSizes.lg

const size: Sizes = "sm"     // 𝒙 - no literals in autocompletion
```

With template literal types we can!

```typescript
enum DesignSystemSizes {
  sm = "sm",
  md = "md",
  lg = "lg"
}

type Sizes = `${DesignSystemSizes}` | `${number}fr` | `${number}px`

const size: Sizes = "sm"     // ✓ - autocompletes literals and DesignSystemSizes!
const size: Sizes = "1fr"    // ✓
const size: Sizes = "100px"  // ✓
```

## Conclusion

With Typescript 4.1 and later, you can do some pretty interesting things to construct accurate types using literals and template literals. Exciting stuff!

Use template literals to create flexible types that accurately check possible values at compile time and support autocomplete in common editors (like VSCode).


Flexible Types that Support Autocomplete with Template Literals (Oh My!)

Arjun Nayini


When writing tests, especially longer integration tests, sometimes you want to run _nearly_ the same test, but with a few key variables changed. Instead of copying and pasting your integration test into another `it` block and changing the small details, you can use the `.each` table syntax [provided by Jest](https://jestjs.io/docs/en/api#describeeachtablename-fn-timeout):

```typescript
describe.each`
  variable1             | variable2
  ${"mystring"}         | ${true}
  ${"otherstring"}      | ${false}          
`(
  "Testing a variety of cases",
  ({
    variable1,
    variable2,
  }: {
    variable1: string;
    variable2: boolean;
  }) => {
    // Test content...
  }
)
```

The `.each` table syntax<Footnote>You can also define the table as an array of arrays or array of dictionaries.</Footnote> allows you to paste in a string that represents the different test cases you want to run.  The first row defines the names of variables that will be passed to each run of the test.  Each subsequent row represents a test run where the provided values will be bound to the variable names.

In example above, two tests will be run where:

1. `variable1 = "mystring", variable2 = true`
1. `variable1 = "otherstring", variable2 = false`

This technique can make tests a lot more readable. Note that in the above snippet, you'll have access to the variables in the body of the test content.

In using `.each`, I've noticed that it is often best when you don't want to change too much of the test logic based on the variables.  If your test ends up having a lot of `if/else` logic based on the variables, especially throughout the entire body of the test, I think it's better to just copy-paste as the additional complexity to read the test isn't worth it.


How to write similar Jest tests without copying and pasting code.  


Testing Using Jest Table Syntax

Ronald Martin


In standard Typescript, you can make an arrow function [generic](https://www.typescriptlang.org/docs/handbook/2/generics.html) by declaring the type variable before the arguments:

```typescript
const identity = <T>(arg: T): T => arg;
```

However, if you try this in a React `.tsx` file, the compiler tries to parse the type variable as a JSX tag:

![Typescripts Compiler Error](https://static.slab.com/prod/uploads/l73blu9g/posts/images/ysOnK_8JL2tvLGuUSS6I__Rh.png "Typescript's compiler error.")

To help the compiler know that this is actually a generic type variable, put a comma (`,`) after the type variable:

```typescript
// Note the comma 👇
const identity = <T,>(arg: T): T => arg;
```


How to fix Typescript compiler errors from generic arrow functions in .tsx files.  


Engineering

Featured

How Thoughtful Communication Makes Us Unreasonably Productive

Async Postmortems in a Global, Remote Team

Detecting Node Event Loop Blockers

Switching Rich Text Editors, Part 1: Picking Tiptap

Snippets

Continous Deployment via a GitHub Actions Reminder

Type Safety with ES2022's Array.prototype.at

Querying JSON and XML with jq and xq

Flexible Types that Support Autocomplete with Template Literals (Oh My!)

Testing Using Jest Table Syntax

Using Generic Arrow Functions in .tsx Files

Product

Resources

Company

Developers

Engineering

Featured

How Thoughtful Communication Makes Us Unreasonably Productive

Async Postmortems in a Global, Remote Team

Detecting Node Event Loop Blockers

Switching Rich Text Editors, Part 1: Picking Tiptap

Snippets

Continous Deployment via a GitHub Actions Reminder

Type Safety with ES2022's Array.prototype.at

Querying JSON and XML with jq and xq

Flexible Types that Support Autocomplete with Template Literals (Oh My!)

Testing Using Jest Table Syntax

Using Generic Arrow Functions in .tsx Files

Subscribe to Updates

Product

Resources

Company

Developers