📃
Graph Codex
  • Welcome to The Graph Codex
  • Getting Started
    • Websites
    • Resources
  • Meetings and Events
    • Core Developers Calls
    • Community Talks
    • Indexer Office Hours
    • NFT Community Calls
  • Workshops
    • Subgraph Development
      • Resources
        • Hackathon Workshops
          • Blockchain Development - Querying with Open APIs Course
          • Building a Custom NFT API with Filtering, Sorting, Full Text Search, and Relationships
          • Building a custom NFT API with The Graph
          • Building a Subgraph on Celo @ The Cross Chain Salon
          • Building a Subgraph with Subgraph Studio
          • Building an NFT API and Subgraph on NEAR with The Graph
          • Building an NFT API with the Graph - Nader Dabit
          • Building an NFT Subgraph - Kuneco April 2021
          • Building and Deploying Subgraphs on TheGraphProtocol
          • Building API's on Ethereum, with Nader Dabit
          • Building Apps on the Decentralized Web with Nader Dabit
          • Building Decentralised GraphQL APIs with The Graph
          • Building on Ethereum with GraphQL, The Graph, and Next.js
          • Building Rich APIs on top of Ethereum with The Graph
          • Building Subgraphs on The Graph - MarketMake
          • Building Subgraphs on The Graph
          • Building Subgraphs with The Graph
          • Defining the Web3 Stack - Nader Dabit - (Next.js Conf 2021)
          • How to build a dApp – Nader Dabit
          • How to Build a Full Stack NFT Marketplace on Ethereum with Polygon and Next.js
          • How to Build an NFT API with The Graph
          • Indexing Smart Contracts with OpenZeppelin Subgraphs & The Graph
          • NFT Dev Talk, GenerativeMasks, and Building NFT APIs with OpenZeppelin, GraphQL, and The Graph
          • Query Ethereum with GraphQL with The Graph
          • The Complete Guide to Full Stack Web3 Development
          • Web3 with Nader Dabit
          • Workshop on How to Build Subgraphs
        • Repositories
      • Developer Highlights
      • Developer Guides
      • Subgraph Testing (Matchstick)
    • Protocol Workshops
  • Ecosystem Updates
    • This Month in Indexing
    • This Month in Curation
    • Council Meeting Notes
    • Governance
      • Governance Resources
      • Graph Improvement Proposals (GIPs)
        • 0000-template
        • 0001-gip-process
        • 0002-gip-withdraw-indexer-rewards
        • 0003-gip-rewards-no-signal
        • 0004-gip-withdraw-indexer-rewards-thawing
        • 0005-gas-costing
        • 0006-gip-withdraw-helper
        • 0007-separate-slashing-percentages
        • 0008-subgraph-api-versioning-and-feature-support
        • 0009-arbitration-charter
        • 0010-rewards-snapshot-empty-poi-fix
        • 0011-stake-to-init-fix
        • 0012-cache-contract-addresses
        • 0013-reduce-curation-tax
        • 0014-batch-gns-transactions
        • 0015-allow-unstake-passing-larger-amount-available
        • 0016-revert-precision-assign-delegation-share
        • 0017-allow-batching-calls-staking-contract
        • 0018-subgraph-ownership-transfer
        • 0019-save-gas-initializing-subgraph-deployment
        • 0020-unattestable-indexer-responses
        • 0023-subgraph-ownership-transfer-nft
        • 0024-query-versioning
        • 0025-principal-protected-bonding-curves
        • 0026-decaying-curation-tax
      • Graph Request for Comments (GRCs)
        • 0001-data-edge
  • Repositories and Documentation
    • Official Repositories
    • Official Documentation
      • About
        • Introduction
        • Network Overview
      • Developer
        • Quick Start
        • Define a Subgraph
        • Create a Subgraph
        • Publish a Subgraph to the Decentralized Network
        • Query The Graph
        • Querying from an Application
        • Querying Best Practices
        • Distributed Systems
        • AssemblyScript API
        • AssemblyScript Migration Guide
        • GraphQL API
        • Unit Testing Framework
        • Deprecating a Subgraph
        • Developer FAQs
      • Indexer
      • Delegator
      • Curator
      • The Graph Explorer
      • Subgraph Studio
        • How to use the Subgraph Studio
        • Deploy a Subgraph to the Subgraph Studio
        • Billing on the Subgraph Studio
        • Managing your API Keys
        • Subgraph Studio FAQs
        • Multisig Users
      • Hosted Service
        • What is Hosted Service?
        • Deploy a Subgraph to the Hosted Service
        • Migrating an Existing Subgraph to The Graph Network
      • Supported Networks
        • NEAR
Powered by GitBook
On this page
  • Polling for updated data
  • Fetching a set of related items

Was this helpful?

Edit on GitHub
  1. Repositories and Documentation
  2. Official Documentation
  3. Developer

Distributed Systems

The Graph is a protocol implemented as a distributed system.

Connections fail. Requests arrive out of order. Different computers with out-of-sync clocks and states process related requests. Servers restart. Re-orgs happen between requests. These problems are inherent to all distributed systems but are exacerbated in systems operating at a global scale.

Consider this example of what may occur if a client polls an Indexer for the latest data during a re-org.

  1. Indexer ingests block 8

  2. Request served to the client for block 8

  3. Indexer ingests block 9

  4. Indexer ingests block 10A

  5. Request served to the client for block 10A

  6. Indexer detects reorg to 10B and rolls back 10A

  7. Request served to the client for block 9

  8. Indexer ingests block 10B

  9. Indexer ingests block 11

  10. Request served to the client for block 11

From the point of view of the Indexer, things are progressing forward logically. Time is moving forward, though we did have to roll back an uncle block and play the block under consensus forward on top of it. Along the way, the Indexer serves requests using the latest state it knows about at that time.

From the point of view of the client, however, things appear chaotic. The client observes that the responses were for blocks 8, 10, 9, and 11 in that order. We call this the "block wobble" problem. When a client experiences block wobble, data may appear to contradict itself over time. The situation worsens when we consider that Indexers do not all ingest the latest blocks simultaneously, and your requests may be routed to multiple Indexers.

It is the responsibility of the client and server to work together to provide consistent data to the user. Different approaches must be used depending on the desired consistency as there is no one right program for every problem.

Reasoning through the implications of distributed systems is hard, but the fix may not be! We've established APIs and patterns to help you navigate some common use-cases. The following examples illustrate those patterns but still elide details required by production code (like error handling and cancellation) to not obfuscate the main ideas.

Polling for updated data

The Graph provides the block: { number_gte: $minBlock } API, which ensures that the response is for a single block equal or higher to $minBlock. If the request is made to a graph-node instance and the min block is not yet synced, graph-node will return an error. If graph-node has synced min block, it will run the response for the latest block. If the request is made to an Edge & Node Gateway, the Gateway will filter out any Indexers that have not yet synced min block and make the request for the latest block the Indexer has synced.

We can use number_gte to ensure that time never travels backward when polling for data in a loop. Here is an example:

/// Updates the protocol.paused variable to the latest
/// known value in a loop by fetching it using The Graph.
async function updateProtocolPaused() {
  // It's ok to start with minBlock at 0. The query will be served
  // using the latest block available. Setting minBlock to 0 is the
  // same as leaving out that argument.
  let minBlock = 0

  for (;;) {
    // Schedule a promise that will be ready once
    // the next Ethereum block will likely be available.
    const nextBlock = new Promise((f) => {
      setTimeout(f, 14000)
    })

    const query = `
        query GetProtocol($minBlock: Int!) {
            protocol(block: { number_gte: $minBlock }  id: "0") {
              paused
            }
            _meta {
                block {
                    number
                }
            }
        }`

    const variables = { minBlock }
    const response = await graphql(query, variables)
    minBlock = response._meta.block.number

    // TODO: Do something with the response data here instead of logging it.
    console.log(response.protocol.paused)

    // Sleep to wait for the next block
    await nextBlock
  }
}

Fetching a set of related items

Another use-case is retrieving a large set or, more generally, retrieving related items across multiple requests. Unlike the polling case (where the desired consistency was to move forward in time), the desired consistency is for a single point in time.

Here we will use the block: { hash: $blockHash } argument to pin all of our results to the same block.

/// Gets a list of domain names from a single block using pagination
async function getDomainNames() {
  // Set a cap on the maximum number of items to pull.
  let pages = 5
  const perPage = 1000

  // The first query will get the first page of results and also get the block
  // hash so that the remainder of the queries are consistent with the first.
  const listDomainsQuery = `
    query ListDomains($perPage: Int!) {
        domains(first: $perPage) {
            name
            id
        }
        _meta {
            block {
                hash
            }
        }
    }`

  let data = await graphql(listDomainsQuery, { perPage })
  let result = data.domains.map((d) => d.name)
  let blockHash = data._meta.block.hash

  let query
  // Continue fetching additional pages until either we run into the limit of
  // 5 pages total (specified above) or we know we have reached the last page
  // because the page has fewer entities than a full page.
  while (data.domains.length == perPage && --pages) {
    let lastID = data.domains[data.domains.length - 1].id
    query = `
        query ListDomains($perPage: Int!, $lastID: ID!, $blockHash: Bytes!) {
            domains(first: $perPage, where: { id_gt: $lastID }, block: { hash: $blockHash }) {
                name
                id
            }
        }`

    data = await graphql(query, { perPage, lastID, blockHash })

    // Accumulate domain names into the result
    for (domain of data.domains) {
      result.push(domain.name)
    }
  }
  return result
}

Note that in case of a re-org, the client will need to retry from the first request to update the block hash to a non-uncle block.

PreviousQuerying Best PracticesNextAssemblyScript API

Last updated 2 years ago

Was this helpful?