Posts tagged #createnode

🗂 🏴‍☠️ ~ Source data (avatars) into the Gatsby Data Layer from a Supabase table

Fri, 26 Aug 2022 00:00:00 GMT

On yesterday's unauthorized and rum-fueled treasure hunt in the sharky waters around the Gatsby islands, we refactored the little avatar section on Prune your follows from client-side to server-side rendering!

PR: If you did (or do) watch the show, you'll see that I have sneakily done some cleaning up and solved our little environment variable problem by making sure I load them in the gatsby-config 🤦‍♀️ 🤪

The What?

Source data from the avatars table on build, so that the little avatars are present as part of the statically generated version of the marketing page.

The Why?

It gets rid of the tiny flicker of no content, and why call Supabase every time someone lands on the marketing page for Prune your follows when there is no need? You can also use your service key when sourcing, allowing you to even information from otherwise locked down table rows.

The How

Fetch data on avatars using the Supabase SKD this time
Create a Gatsby node for each avatar using createNode, createNodeId and createContentDigest.

The Code

// File: gatsby-node.js
// Note: Error handling omitted

const { createClient } = require("@supabase/supabase-js");

const supabaseUrl = process.env.GATSBY_PUBLIC_SUPABASE_URL;
const supabaseServiceKey = process.env.SUPABASE_SERVICE_KEY;

const serviceSupabase = createClient(supabaseUrl, supabaseServiceKey);

exports.sourceNodes = async (gatsbyUtils) => {
  const { actions, createNodeId, createContentDigest, reporter } = gatsbyUtils;
  const { createNode } = actions;

  const { data } = await serviceSupabase
    .from("avatars")
    .select("username, avatar_url")
    .limit(20);

  data.forEach((item) => {
    createNode({
      id: createNodeId(item.username),
      avatarUrl: item.avatar_url,
      username: item.username,
      internal: {
        type: "UserAvatar",
        contentDigest: createContentDigest(item),
      },
    });
  });
};

And voila, we have avatar nodes in the data layer we may use in the Avatars component:

// File: Avatars.js
// Note: Styling omitted

import React from "react";
import { useStaticQuery, graphql } from "gatsby";

export function Avatars() {
  const data = useStaticQuery(graphql`
    query {
      allUserAvatar {
        nodes {
          username
          avatarUrl
        }
      }
    }
  `);

  return (
    <div>
      {data.allUserAvatar.nodes.map((user) => {
        return (
          <img key={user.username} src={user.avatarUrl} alt={user.username} />
        );
      })}
    </div>
  );
}

To see the entire app code, check out its GitHub repository.

All the best,
Queen Raae

Contrary to popular belief, Queen Raae and family have never worked for Gatsby.

📥 🏜 ~ createNode does NOT return anything useful

Tue, 21 Jun 2022 00:00:00 GMT

createNode does not return anything useful:

// gatsby-node.js

// Should be sourced or derived, only for example purposes
const DATA = { id: "unique", hello: "world" };

exports.sourceNodes = async (gatsbyUtils) => {
  const { actions, createNodeId, createContentDigest } = gatsbyUtils;
  const { createNode } = actions;

  const awaitedNode = await createNode({
    ...DATA,
    id: createNodeId(`example-node-${DATA.id}`),
    internal: {
      type: `BadExample`,
      contentDigest: createContentDigest(DATA),
    },
  });
  console.log(">>>> awaitedNode:", awaitedNode); // 👈
  // Console:
  // >>> awaitedNode: []
};

Therefore to create a child-parent link in onCreateNode for instance, you need to keep a reference to the id yourself:

// gatsby-node.js

// Should be sourced or derived, only for example purposes
const DATA = { id: "unique", hello: "world" };

exports.onCreateNode = (gatsbyUtils) => {
  const { node, actions, createNodeId, createContentDigest } = gatsbyUtils;
  const { createNode, createParentChildLink } = actions;

  if (node.internal.type !== "Parent") return;

  const nodeId = createNodeId(`example-node-${DATA.id}`); // 👈

  createNode({
    ...DATA,
    id: nodeId, // 👈
    internal: {
      type: `Child`,
      contentDigest: createContentDigest(DATA),
    },
  });

  // 👇👇👇
  createParentChildLink({ parent: node, child: { id: nodeId } });
};

A more common version of the code above that you see in examples are:

// gatsby-node.js

// Should be sourced or derived, only for example purposes
const DATA = { id: "unique", hello: "world" };

exports.onCreateNode = (gatsbyUtils) => {
  const { node, actions, createNodeId, createContentDigest } = gatsbyUtils;
  const { createNode, createParentChildLink } = actions;

  if (node.internal.type !== "Parent") return;

  const childNode = {
    ...DATA,
    id: createNodeId(`example-node-${DATA.id}`),
    internal: {
      type: `Child`,
      contentDigest: createContentDigest(DATA),
    },
  };

  // 👇👇👇
  createNode(childNode);
  createParentChildLink({ parent: node, child: childNode });
};

However, I'm not too fond of indicating that the data object describing the node is already an actual node in the data layer, so I prefer my initial solution as only the id is needed to create the link.

All the best,
Queen Raae

🗂 🐝 ~ Source content from anywhere with ScrapingBee

Fri, 06 May 2022 00:00:00 GMT

Pierre de Wulf, co-founder of ScrapingBee joined us on yesterday's unauthorized and rum-fueled treasure hunt in the sharky waters around the Gatsby islands.

The What?

Source Crowdcast webinars into the Gatsby Data Layer using ScrapingBee — an API that simplifies web scraping.

The Why?

There is no official Crowdcast API and keeping the data in sync using copy/past is a pain (or at least boring 🤪). To scrape from Crowdcast, we need to load the page in a headless browser such as Puppeteer. Doing so is not possible as part of the Gatsby build process, so we outsource it to ScrapingBee.

The How

We used the Data Extraction-feature from ScrapingBee. It lets us select data on a page using CSS-selector. It felt kinda similar to cheerio if you have ever used that.

As always, I started with copy/pasting the example snippets. It worked almost out of the box, but we had to make use of the wait_for option as the Crowdcast page takes a while to load:

It's sometimes necessary to wait for a particular element to appear in the DOM before ScrapingBee returns the HTML content. <cite>ScrapingBee Docs</cite>

The Code

const axios = require("axios");

const scrapeCrowdcast = async () => {
  const { data } = await axios.get("https://app.scrapingbee.com/api/v1", {
    params: {
      api_key: process.env.SCRAPING_BEE_API_KEY,
      url: "https://www.crowdcast.io/raae",
      // Wait for there to be at least one
      // non-empty .event-tile element
      wait_for: ".event-tile",
      extract_rules: {
        webinars: {
          // Lets create a list with data
          // extracted from the .event-tile element
          selector: ".event-tile",
          type: "list",
          // Each object in the list should
          output: {
            // have a title lifted from
            // the .event-tile__title element
            title: ".event-tile__title",
            // and a path lifted from
            // the href attribute of the first link element
            path: {
              selector: "a",
              output: "@href",
            },
          },
        },
      },
    },
  });

  return data;
};

The resulting data object:

{
  webinars: [
    {
      title: "5 Gatsby Gotchas to look out for as a React developer",
      path: "/e/gatsby-gotchas-react?utm_source=profile&utm_medium=profile_web&utm_campaign=profile",
    },
    {
      title: "Testing your Gatsby Serverless Functions",
      path: "/e/testing-your-functions?utm_source=profile&utm_medium=profile_web&utm_campaign=profile",
    },
    // and more
  ];
}

We then loop through the webinars on the data object creating content nodes:

// gatsby-node.js
exports.sourceNodes = async (gatsbyUtils) => {
  const { actions, createNodeId, createContentDigest } = gatsbyUtils;
  const { createNode } = actions;

  const data = await scrapeCrowdcast();

  for (const webinar of data.webinars) {
    createNode({
      id: createNodeId(webinar.path),
      title: webinar.title,
      url: "https://www.crowdcast.io" + webinar.path,
      rawScrape: webinar,
      internal: {
        type: `CrowdcastWebinar`,
        mediaType: `text/json`,
        content: JSON.stringify(webinar),
        contentDigest: createContentDigest(webinar),
      },
    });
  }
};

And voila, we have webinar nodes in our data layer:

query MyQuery {
  allCrowdcastWebinar {
    nodes {
      title
      url
    }
  }
}

To see the entire demo, check out its GitHub repository.

Should we make this a full-featured plugin? Extracting the cover art and descriptions from the individual webinar pages? Let me know!

All the best,
Queen Raae

PS: ScrapingBee is a paid service, but we are as always neither sponsored nor an affiliate.
PPS: If you want to learn more about web-scraping without ScrapingBee check out their article Web Scraping with Javascript and NodeJS.

🏷 ✨ ~ How to create unique Gatsby node ids

Wed, 04 May 2022 00:00:00 GMT

All Gatsby nodes need a unique id. Luckily for you, a Gatsby Source Plugin developer, you are not responsible for creating unique ids. You are only responsible for supplying a unique-within-your-universe-id to createNodeId. Gatsby is responsible for returning a unique node id.

exports.sourceNodes = async ({ actions, createNodeId }) => {
  const { createNode } = actions;

  createNode({
    id: createNodeId("unique-within-your-universe-id"),

    // More fields
  });
};

In Gatsby Source YouTube oEmbed we use the YouTube id. It is unique across the YouTube universe.

All the best,
Queen Raae

PS: createNodeId is deterministic

📄 🏷 ~ No need to create the content digest yourself

Tue, 03 May 2022 00:00:00 GMT

We see a lot of this type of code when looking at source plugins:

exports.sourceNodes = async ({ actions }) => {
  const { createNode } = actions;
  const data = { hello: "world" };

  createNode({
    // More fields
    internal: {
      // More fields
      contentDigest: crypto
        .createHash(`md5`)
        .update(JSON.stringify(data))
        .digest(`hex`),
    },
  });
};

It is not technically wrong, but I prefer using createContentDigest like so:

exports.sourceNodes = async ({ actions, createContentDigest }) => {
  const { createNode } = actions;
  const data = { hello: "world" };

  createNode({
    // More fields
    internal: {
      // More fields
      contentDigest: createContentDigest(data), // 👈👈👈
    },
  });
};

internal.contentDigest is mandatory and thus part of a Minimal Viable Content Node. If this value has not changed, Gatsby uses the cached node instead of creating a new node, and for instance, onCreateNode is not called.

contentDigest: the digest for the content of this node. Helps Gatsby avoid doing extra work on data that hasn’t changed. <cite>Gatsby Docs</cite>

All the best,
Queen Raae

📥 👩‍👧 ~ Always define a parent when creating nodes in onCreateNode

Fri, 01 Apr 2022 00:00:00 GMT

TLDR: When using the action createNode inside onCreateNode it is super duper important to give the node a parent!

On yesterday's unauthorized and rum-fueled treasure hunt in the sharky waters around the Gatsby islands, we worked through the code from the last two emails:

But we also briefly touched on a bug I had to fix from last week's ImageCDN stream. The YouTubeThumbnail nodes were "lost" between builds... 😱

Gatsby caches nodes between builds. If you call createNode with the same nodeId and content digest as a cached node, Gatsby will not go through the node creation process again. It will use the cached node, skipping onCreateNode for that node.

Since caches can get out of hand, Gatsby throws out all nodes that are not "touched" during the node sourcing phase. And this is where our problem arises.

A node is considered touched either by the use of createNode or touchNode. However, the code "touching" our YouTubeThumbnail nodes is skipped when its YouTube node is cached.

Luckily a node is also considered "touched" if its parent is "touched": problem solved 👩‍👧

createNode({
  id: youTubeThumbnailNodeId,
  parent: node.id, // 👈 👈 👈
  youTubeId: node.youTubeId,
  url: node.oEmbed.thumbnail_url,
  mimeType: "image/jpeg",
  filename: node.youTubeId + ".jpg",
  height: node.oEmbed.thumbnail_height,
  width: node.oEmbed.thumbnail_width,
  internal: {
    type: YOUTUBE_THUMBNAIL_TYPE,
    contentDigest: node.internal.contentDigest,
  },
});

All the best,
Queen Raae

🔴 🏴‍☠️ ~ Source YouTube information using oEmbed, no YouTube API Key needed

Fri, 21 Jan 2022 00:00:00 GMT

In yesterday's unauthorized and rum-fueled treasure hunts in the sharky waters around the Gatsby islands, we looked closely at sourcing content nodes with data from the YouTube oEmbed endpoint.

oEmbed is likely one of the most underappreciated concepts of the open web. I first encountered it when WordPress added support back in my uni days; let's just say it been awhile 👵

A simple yet powerful concept that I think would benefit from more awareness.

Have you heard about or used oEmbed before?

oEmbed Examples

A provider: YouTube, SoundCloud, Issuu, Flickr, or your blog provides an oEmbed endpoint that accepts at a minimum an URL and returns embeddable content in return.

A GET request to https://www.flickr.com/services/oembed/?url=http://www.flickr.com/photos/bees/2341623661/ returns:

{
  "version": "1.0",
  "type": "photo",
  "width": 240,
  "height": 160,
  "title": "ZB8T0193",
  "url": "http://farm4.static.flickr.com/3123/2341623661_7c99f48bbf_m.jpg",
  "author_name": "Bees",
  "author_url": "http://www.flickr.com/photos/bees/",
  "provider_name": "Flickr",
  "provider_url": "http://www.flickr.com/"
}

Or more interestingly in our case, a GET Request to https://www.youtube.com/oembed/?url=https://youtu.be/Bk1jonYPFD4 returns:

{
  "title": "Celebrating POW! (usepow.app)  \u00b7  A day in my life as a developer, founder and mom  \u00b7  March 2021",
  "author_name": "Queen Raae",
  "author_url": "https://www.youtube.com/c/QueenRaae",
  "type": "video",
  "height": 113,
  "width": 200,
  "version": "1.0",
  "provider_name": "YouTube",
  "provider_url": "https://www.youtube.com/",
  "thumbnail_height": 360,
  "thumbnail_width": 480,
  "thumbnail_url": "https://i.ytimg.com/vi/Bk1jonYPFD4/hqdefault.jpg",
  "html": "\u003ciframe width=\u0022200\u0022 height=\u0022113\u0022 src=\u0022https://www.youtube.com/embed/Bk1jonYPFD4?feature=oembed\u0022 frameborder=\u00220\u0022 allow=\u0022accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\u0022 allowfullscreen\u003e\u003c/iframe\u003e"
}

Take note of html on the YouTube response; it's all you need to embed a YouTube player to your site.

Sourcing nodes with oEmbed data

In Gatsby land, we can use oEmbed to ensure we have the correct embed snippet and/or the current title for our YouTube videos.

We do so by using the gatsby-node.js hook sourceNodes where we create one content node per YouTube video and add the information from the oEmbed endpoint onto the node.

// gatsby-node.js
// Full code without error handling

const axios = require("axios");

const YOUTUBE_IDS = ["Bk1jonYPFD4", "TzJfepDjpzM"];

const fetchEmbed = async (id) => {
  const ytUrl = `https://youtu.be/${id}`;
  const { data } = await axios.get("https://www.youtube.com/oembed", {
    params: {
      url: ytUrl,
    },
  });
  return data;
};

const prepYouTubeNode = async (
  id,
  { actions: { createNode }, createNodeId, createContentDigest }
) => {
  const embedData = await fetchEmbed(id);

  createNode({
    id: createNodeId(`you-tube-${id}`),
    oEmbed: embedData,
    internal: {
      type: `YouTube`,
      contentDigest: createContentDigest(embedData),
    },
  });
};

exports.sourceNodes = async (params) => {
  await Promise.all(YOUTUBE_IDS.map((id) => prepYouTubeNode(id, params)));
};

Would you like to see this code refactored into a plugin?

All the best,
Queen Raae

PS: The Pirate Princess and I woke up in this magical tree top hut today...

🥗 🌮 ~ Minimal Viable Content Node

Thu, 20 Jan 2022 00:00:00 GMT

Are you the type who preps all your ingredients in advance of cooking? No?

Sourcing content into the Gatsby Data Layer as content nodes is a little like shopping for ingredients! Chopping up and prepping those ingredients is similar to transforming those sourced content nodes to be ready to go.

If you are the prep type, you might have taken quickly to the Gatsby Data Layer; if you are not, maybe you feel it's an unnecessary step keeping you from making it up while you go and moving quickly?

Either way, if you are going to get the most out of Gatsby, you should probably become intimately familiar with it.

On that note, Ola and I had an interesting conversation last week:

How minimal can a content node be?

It turns out it can be very minimal!

// gatsby-node.js

exports.sourceNodes = ({ actions, createNodeId, createContentDigest }) => {
  const { createNode } = actions;

  const ID = `unique`;

  createNode({
    id: createNodeId(`minimal-content-node-${ID}`),
    internal: {
      type: `Minimal`,
      contentDigest: createContentDigest(ID),
    },
  });
};

A content node of type Minimal is now available in the Gatsby Data Layer and can be queried for like so:

query MinimalQuery {
  minimal {
    internal {
      type
    }
    id
  }
}

Resulting in:

{
  "data": {
    "minimal": {
      "internal": {
        "type": "Minimal"
      },
      "id": "40c03aa4-b334-5ec8-a148-8f42e7bafbb4"
    }
  },
  "extensions": {}
}

Obviously not usable for anything, but in tonight's unauthorized and rum-fueled treasure hunt Deep Dive Edition, we'll build from here and create YouTube content nodes containing the embed code and thumbnail fetched from YouTube.

Have you ever created your own content nodes?
Let me know!

All the best,
Queen Raae

PS: You might feel you have seen this particualar Deep Dive before, but this time we'll move slower, and we'll not need a YouTube API Key 🤯