Indexing chain genesis state

[vayesy]

Problem statement

There is no possibility to index chain data, populated during genesis block.
This happens because there are no events emitted during genesis.
Since squid indexing service relies on the events to track chain state changes, it will miss those.
This way we will have data stored on chain but missing on the graph side.

Expected result

We would like to define solution for this problem and have objects, generated during genesis to be correctly reflected on the squid indexer side.

Possible solutions

Let's discuss options to solve this issue, pros and cons of each of them.

Option 1: Chain migration

With this approach, nothing is stored during genesis and initial chain objects are created with additional storage migration instead.
They are all populated during additional runtime upgrade, containing storage migration to insert all the objects.
This will allow all events to be triggered and squid process will go usual path, reacting on the events.

Pros:

• No changes involved in other services then the chain

Cons:

• Additional step, runtime upgrade, in order to populate the chain

---

Option 2: Squid fixture/migration

It is known beforehand all the details of each object, which is created during genesis. So we can populate squid database with those objects as initial state for the squid database.

Pros:

• No changes involved in other services then the squid

Cons:

• In case genesis block logic changes, need to reflect it in migration/fixture, otherwise there may be discrepancies between chain and squid data

---

Option 3: Additional script to populate from chain storage

This approach involves usage of additional script prior to starting squid indexer service.
This script will query all the objects from the chain storage via polkadot.js library and insert those in the squid database.

Pros:

• No changes involved in other services then the squid

Cons:

• Additional script to run prior to starting squid service

---

Option 4: Alter squid processor

This is similar to Option 3, but instead of having separate process to populate initial data, this will be part of squid processor.
Once service started, prior to setting up listener for new chain blocks/events, check if there are some objects in the graph database populated. If no, query chain storage for interested objects.

Pros:

• No changes involved in other services then the squid
• No additional services or script to run

Cons:

• There may be data duplicated data as a result of such processing, so need to deal with it

@vovacha @FiberMan @AlexisZero @2075 feel free to add more details, alternative solutions and vote for the preferred one!

[FiberMan]

Can we investigate a bit deeper this issue? Cause it sounds strange that squid can't get info from genesis block.
At least I couldn't find anyone with the same issue.
What we possibly could do more:

• try to use advanced options like --start-block (optional) The block height to start. Note that in order to index the runtime metadata and make the archive compatible with @subsquid/substrate-typegen one must start from the genesis block.
• maybe FireSquid doesn't have this issue?
• Ask for best practices

[vayesy]

Subsquid service does not operate with chain storage (thought it's possible to request storage while handling an event).
It works with indexed chain blocks, extrinsics and events.
But in case of genesis block there are none of these.

As another possible solution, we could subscribe to event, which is generated in each and every block (like timestamp.set) and manually check in handler for that event, is it the first block. If it's not, just skip further processing. But if it is, then fetch all the data we are interested from the chain storage. Generally, this approach is not the best from architectural point of view, but it might work as well. Lets wait for other opinions.

[5-mark]

Hi guys, I will not vote but just wanted to comment on this.

looking at the solutions i think the option 3 is the most reliable.
As i understood:
Block 0 is creating all genesis transactions (creating DAOs, setting balances)
Then a script reads everything from the chain state which is the source of truth and populates into squid.

Option 2 sounds like having a separate file as a source of truth which should contain all the same initial data as during genesis. But since it is not really genesis but another file I would prefer option 3.

Option 4 sounds too much for us.

Option 1 sounds also valid but seems like bypassing the genesis event which I assume is the standard way to go.

[vovacha]

• having one source of proof (not option 2)
• I like having orgs in the genesis (not option 1)
• reduce complexity (no additional parts except for the chain and squid, not option 3)
• share the responsibilities (chain already has actual state, squid needs the data)

Saying this, I choose option 4. If DB is empty, it means it's the first run and nothing was indexed, then insert objects from chain to squid DB.

But option 1 is also good. Maybe we can somehow benefit from having orgs in the migration instead of the genesis, but to me it's not obvious now.

[vayesy]

For me option 4 is also a good choice. I was thinking how to handle chain objects only once and not process it multiple times. This could happen if squid is started after chain was already running for some amount of blocks and new objects were created via extrinsic calls. In this case chain storage would contain initial objects and those created with extrinsics.
But then I realized we can query chain state at specific block (first, to be precise), this will only include objects from genesis state.
So for me this is actually the best approach.

[FiberMan]

Avoiding genesis block (option 1) doesn't look right. But as on PROD it is a one-time action, we can create orgs even manually once, right? On other hand, to have some consistency in all envs, it's better to have the same approach everywhere. But it will require additional manual steps on every deployment.

Option 2 is not good as it requires additional maintenance and may produce discrepancies.

Among 3 and 4 would also prefer 4 to reduce complexity.

[2075]

• option 2: would work if the migration just fires an event, then the genesis state can be properly indexed.
• option 3: sounds legit
• option 5: today i learned subquery provided subscriptions way before squid and does not need an archiver node. ;)

also as yura mentions, it is a one time action, but i assume the genesis should really mint the token as designed on genesis. so not sure, triggering a custom event manually could help.

[vayesy]

@2075 subquery does not provide archiver node - that's the main disadvantage of the service, since indexing chain from the scratch might take ages due to need to query chain for every block instead of caching this information.

[2075]

they dont - need - an archiver node, thats what I understood, but yes, the differences are clear ;)