How I built Shopify Theme Vitals using 11ty and CrUX

For two years at Shopify, I helped merchants and theme developers make their online stores and themes faster. I also applied learnings from those engagements to improve the Shopify platform.

I left in the summer of 2024, but I felt I had unfinished business. I wanted merchants and theme developers to have a better resource for understanding the real-world performance of Shopify themes. And, I knew publicly-available data for this existed in HTTP Archive and the Chrome User Experience Report, or CrUX for short.

Thus began my journey to query and present this data. In this blog post, I will cover how I built Theme Vitals, a site that shows real-user Core Web Vitals performance data by Shopify theme.

Contents:

• Overview diagram
• Background: How Shopify themes are linked to performance and user behavior
• Gathering the data: Publicly-available web performance data
  • Identifying Shopify sites and themes with HTTP Archive
  • Collecting and aggregating real-user data with the Chrome User Experience Report
  • Data processing
  • Charting with SVG
• Creating the Static Site with 11ty
  • Why Use Static Site Generation?
  • Workflow and Automation
• Final Thoughts

If you prefer watching videos over reading text, I presented similar information at the 11ty Meetup:

Overview diagram #

I created a diagram in Figma to better outline how it all works together (open in full screen):

Background: How Shopify themes are linked to performance and user behavior #

Shopify themes are pre-built design templates, or bundles of HTML, CSS, JavaScript, and Shopify’s own templating language, Liquid. Merchants can buy themes in the Shopify theme store, then customize and publish them as their online stores.

Themes contain the core frontend code (though they can also impact backend performance through how Liquid is used). Merchants can also install Shopify apps which are essentially third-party scripts and widgets. Both themes and apps can heavily impact web performance, including the Core Web Vitals.

Typically, how a theme is written sets your maximum achievable performance. Adding apps can further degrade your performance, especially Interaction to Next Paint (INP).

Site performance is important for both search engine rankings and user retention. While at Shopify, time and again I saw how online store bounce rates increased and conversion rates decreased for users with slow performance. Themes would usually be the biggest culprit followed by apps and tags (e.g., Google Tag Manager).

Unfortunately, the Shopify theme store does not yet contain data about web performance so both merchants and theme developers are left in the dark.

My goal was to provide Shopify merchants with insights into the real-world performance of these themes, enabling them to make informed decisions about which theme to choose based on their performance in the wild. I also wanted to provide theme developers with a way to identify their biggest performance bottlenecks so that they could be more effective at optimizing their themes.

Gathering the data: Publicly-available web performance data #

Breaking down the problem into smaller steps, I needed to:

1. Figure out which websites are using Shopify and which themes they use
2. What is the real-user (Core Web Vitals) performance for those websites
3. Aggregate the data by theme

HTTP Archive and the Chrome User Experience Report (CrUX) are my data sources. Both have the ability to run custom queries using BigQuery.

Identifying Shopify sites and themes with HTTP Archive #

HTTP Archive allowed me to achieve step 1. HTTP Archive runs a massive job once per month on the websites with the most traffic. This is currently the top ~16M URLs on mobile and ~13M on desktop. This job runs lab-based tests (a private instance of WebPageTest) on each website to collect detailed data including various performance metrics.

Recently, HTTP Archive added a custom metric that saves the global JavaScript Shopify object when present. This object is present on all Shopify Liquid sites and contains information about the theme. If you open the Dev Tools console on a Shopify site and enter Shopify.theme, here's an example output:

{
    "name": "Loop Offset in Cart-UI",
    "id": 128596803639,
    "schema_name": "Palo Alto",
    "schema_version": "6.0.1",
    "theme_store_id": 777,
    "role": "main",
    "handle": "null",
    "style": {
        "id": null,
        "handle": null
    },
}

The name is a custom name that the merchant can change. They often have multiple copies of the same underlying theme which they will switch out for promotions or seasonal changes. The most important property here is the theme_store_id which tells us the theme came from the Shopify theme store and its ID. schema_name is a recent addition which tells us the name of the theme.

Once I had the list of URLs and what theme they use, I was able to proceed to step 2...

Collecting and aggregating real-user data with the Chrome User Experience Report #

CrUX provides real-world web performance data based on actual users visiting websites in Chrome. Unlike synthetic tools such as Lighthouse, CrUX data is based on real users and reflects how actual visitors experience websites. This is the same data used in:

• Google Search Console and Google search rankings
• The top of PageSpeed Insights in "Discover what your real users are experiencing", if you have enough traffic

Each month, CrUX publishes aggregated monthly data to their database on BigQuery.

To get the real-user performance of Shopify sites, I used SQL to join the URLs from HTTP Archive to the CrUX data set URLs in BigQuery. Then, I aggregated the data by theme within the same query. The full query is a bit gnarly, but you can see it here if you're curious: query.sql.

I exported the result of this query as a JSON file to use in my next step...

Data processing #

The output of the query gives us data for one month. However, Theme Vitals provides detailed data for the previous 6 months. Running queries in BigQuery can get quite expensive, so attempting to do a mega-query for the last 5 months would get prohibitively expensive as well as overly complex.

Thus, I wrote a Node.js script to process each month's data and munge it into one data file with both the aggregation calculations as well as the time series data for each theme.

Charting with SVG #

For the visualizations, I wanted a charting solution that would ensure that the Theme Vitals loading speed and interactions stayed fast. This site is statically generated, so I wanted a static or server-side rendered option. Pre-rendered SVG charts were the perfect choice.

I used Apache eCharts in my Node script to generate SVG strings for each chart which are saved in the same data file by theme. Here's how a basic SVG string is generated:

// Server-side code
const echarts = require('echarts');

// In SSR mode the first container parameter is not required
let chart = echarts.init(null, null, {
  renderer: 'svg', // must use SVG rendering mode
  ssr: true, // enable SSR
  width: 400, // need to specify height and width
  height: 300
});

// use setOption as normal
chart.setOption({
  //...
});

// Output a string
const svgStr = chart.renderToSVGString();

// If chart is no longer useful, consider disposing it to release memory.
chart.dispose();
chart = null;

I have some ideas for new features that would require more dynamic rendering on the client side, and eCharts still has that capability.

Creating the Static Site with 11ty #

To generate the site, I used 11ty (a.k.a., Eleventy), my favorite static site generator. It's Node-based so I don't have to context-switch between JavaScript and other languages (e.g., Ruby). The concepts for how I structured the site may be applicable to other static site generators as well:

1. Global Data: I import the munged data from the previous step into global data objects in 11ty. I don't process and munge the data directly in 11ty because that would make my build time quite slow, and the data is only updated once per month. I create two global data objects:

   • themes: all of the themes and their data and charts
   • metadata: metadata such as the date of the last run and aggregations across all themes

2. Aggregations Page: This page shows how all themes perform as a group, providing minimum, median, and maximum percent of themes passing each Core Web Vital metric. The template renders from metadata directly like so:

{% for metric in metaData.aggregations.mobile %}
  <tr>
    <td>{{ metric.name }}</td>
    <td>{{ metric.min }}%</td>
    <td>{{ metric.median }}%</td>
    <td>{{ metric.max }}%</td>
  </tr>
{% endfor %}

3. All Themes Page: This page lists all themes with their Core Web Vitals data for the current month in a large table for easy comparison. Similar to the aggregations page, the template renders from themes directly like so:

<tbody id="tbody-mobile">
  {% set sortedThemes = themes | sortThemes %}
  {% for theme in sortedThemes %}
    <tr
      data-market-rank="{{ theme.summary.mobile.marketRank }}"
      data-cwv-rank="{{ theme.summary.mobile.cwvRank }}"
      data-alpha-rank="{{ loop.index }}">
      <td><a href="/themes/{{ theme.slug }}/">{{ theme.name }}</a> {% if theme.sunset %}<span class="pill">vintage</span>{% endif %}</td>
      <td>{{ theme.summary.mobile.marketRank }} <small>({{ theme.summary.mobile.marketSharePct }}%)</small></td>
      <td>

4. Theme Pages: These pages provide detailed data for an individual theme, including the Core Web Vitals plus other metrics (TTFB and FCP) as well as the SVG charts. To build this, I use pagination to create one page per theme in themes

---
layout: base.njk
pagination:
  data: themes
  size: 1
  alias: theme
permalink: "/themes/{{ theme.slug }}/"
eleventyComputed:
  title: "{{ theme.name }} Performance Data | Theme Vitals"
  description: "Explore real-user web performance for the Shopify theme {{ theme.name }} including the Core Web Vitals and other metrics, split by desktop and mobile sessions."
---

<section class="banner">
  <div class="content flow">
    <h1>{{ theme.name }}</h1>

Why Use Static Site Generation? #

There were several reasons I chose to use 11ty for this project:

• Speed: Static sites load incredibly fast, as what would typically be server-side processing is done once at build time. When a user loads the page, I only have to deliver static assets.
• Hosting: Because no server is needed except to build the site after any update, I can host for free on a number of platforms. Cloudflare pages and Netlify are two of my favorites. Whenever I push a commit to Github, they automatically rebuild my site. Also, they are both CDNs so my sites hosted on them load fast around the world.
• Fun: Ever since I started building on 11ty and creating sites in mostly plain HTML, CSS, and vanilla JavaScript, I've started having more fun with the web. It's easy to create and manage pages without a lot of overhead so I can go wild with creativity or spend more time learning the latest web platform features.

Workflow and Automation #

Back to business... The monthly workflow is straightforward but involves a few manual steps each month:

1. Query Data (manual): Update and run the query on BigQuery to get the previous month's performance data from CrUX and HTTP Archive.
2. Process Data: (lightly manual) Export the data to JSON and run the Node.js script to munge the data and generate SVG charts. I have another script I run to give me additional statistics like new and dropped themes and which themes improved the most. I use that data for my newsletter.
3. Update Site: (nearly automatic) Run 11ty to pull in the new data and regenerate the site. I confirm everything looks okay on my local dev server before committing and then pushing to Github.
4. Deploy: (automatic) As soon as the main branch is updated in Github, Github sends a webhook to my hosting provider which triggers a rebuild in production.

Final Thoughts #

This project has been very rewarding for me. I love building tools that help others. I'm hoping it will help Shopify merchants in their decision-making process. I'm also hoping it will pressure theme developers (just a light, friendly pressure) to make their themes more performant as well as give them the data they need to do that more effectively.

I'm not stopping here. I'm considering several new features such as:

• showing theme version shares so that viewers can better determine if the performance numbers are more representative of the latest theme version or if there are a lot of un-upgraded instances out there
• adding a theme comparison feature so that you can compare 2 different themes directly, similar to the cwvtech.report
• building a tool for someone to see how their Shopify site compares to other sites with the same theme
• adding distributions of p75 scores rather than just buckets of percent passing

Which one of those would be your preference? Let me know!

Eleventy webperf analytics