ParaID 5080: Workplan entry disappeared - Blocks not produced despite successful broker.assign

[jhoyosp]

Bug Report: Parachain Not Producing Blocks Despite Successful Coretime Assignment

Summary

ParaID 5080 (DePow) on Paseo testnet is not producing blocks despite having:

• Successfully purchased bulk coretime
• Successfully assigned coretime to Task 5080 (verified on-chain)
• Workplan showing the assignment
• Collator running correctly with all prerequisites met

Environment

Component	Value
Network	Paseo Testnet
ParaID	5080
Collator Node Version	Polkadot SDK 2509.0.0 (stable2509, Oct 2024)
Collator IP	185.209.229.181:9944
Owner Account	5DfQWud2iTVxoJLZ6PfEHCxh7cSLhxreyY6CV8ZAHSFD9Kv2

Timeline of Events

December 2, 2025 - Coretime Purchase and Assignment

• Transaction: broker.assign
• Block: 2,922,966 (Coretime Paseo)
• Hash: 0xff4727b44b6fd14c687f96ae3c288a6bd33d2a7ebe5aba30f71383e3aca186e3
• Result: ✅ SUCCESS
• Parameters:

  {
    "region_id": {
      "begin": 120572,
      "core": 42,
      "mask": "0xffffffffffffffffffff"
    },
    "task": 5080,
    "finality": "Final"
  }

December 6, 2025 - Verification

• Workplan query showed assignment:

  Workplan for (120,572, 42)
  [
    {
      mask: 0xffffffffffffffffffff
      assignment: { Task: 5,080 }
    }
  ]
  

• paras.paraLifecycles(5080): Parathread
• WASM hash on-chain: 0x011b5fce05e81abd3521f7e3bd604e97a6c9f6a6afe36de39abe443b7ec69505
• Genesis head on-chain: 0x00000000...42aff38567293678d6ac6d25f88005c5d22cbbbbb0a1603f66266313495ffa36...

December 24-25, 2025 - Timeslice Activation

• Timeslice 120,572 = Relay Block 9,645,760
• Relay chain reached block 9,645,760+
• Expected: Collator produces blocks
• Actual: Collator remains idle, no blocks produced

December 25, 2025 - Investigation

• Workplan query: EMPTY
• Region query: DOES NOT EXIST
• Parachain block: #0 (genesis)
• Parachain peers: 0

Collator Configuration

Runtime Configuration (Verified Correct)

// Aura Configuration
impl polkadot_sdk::pallet_aura::Config for Runtime {
    type AuthorityId = AuraId;
    type AllowMultipleBlocksPerSlot = ConstBool<false>;
    type SlotDuration = polkadot_sdk::pallet_aura::MinimumPeriodTimesTwo<Runtime>;
}

// Consensus Hook (Async Backing)
type ConsensusHook = polkadot_sdk::cumulus_pallet_aura_ext::FixedVelocityConsensusHook<
    Runtime,
    6000, // RELAY_CHAIN_SLOT_DURATION_MILLIS
    1,    // BLOCK_PROCESSING_VELOCITY
    3,    // UNINCLUDED_SEGMENT_CAPACITY
>;

Node Configuration (Verified from systemd service)

/opt/depow-collator/blockchain/target/release/depow-node \
  --collator \
  --force-authoring \
  --chain /opt/depow-collator/blockchain/paseo-5080-FINAL.json \
  --base-path /var/lib/depow-collator \
  --port 30333 \
  --rpc-port 9944 \
  --rpc-external \
  --rpc-cors all \
  --rpc-methods Unsafe \
  --prometheus-external \
  --prometheus-port 9615 \
  --name DePow-Collator-5080 \
  --bootnodes /dns/paseo-bootnode-0.parity.io/tcp/30333/p2p/12D3KooWQCkBm1BYtkHpocz3orfw1Qk9qrwJMEWN3D9TMxzhARtC \
  --bootnodes /dns/paseo-bootnode-1.parity.io/tcp/30333/p2p/12D3KooWPEyEfLVxmNRddjKW5SPCFj8g2X9XxyJSJUzXV3aKM5y4 \
  -- \
  --chain paseo \
  --port 30343 \
  --rpc-port 9945 \
  --sync warp

Process running since: Dec 1, 2025 (PID 684407, uptime 24+ days)

Session Keys (Verified in keystore)

/var/lib/depow-collator/chains/depow_paseo/keystore/
├── 6175726156f99f0ae33c500a71bc28806181bbaf3323a9696bb5b230257e6a7965fd3b2b (Dec 1)  ← AURA key in genesis
├── 61757261b2399445c78e7a3e11e530ac1f101e13e676d4077f3560c40eb4e0903e6ee85e (Dec 6)
├── 61757261d019e610c78762750a92467f362ada2410c534f2eaa5c5dc390b14d1bf408e1d (Dec 25)
├── 6772616e857ad28decb01dd9ffb4119a93441e8cb0b1908efb78415827844a180bb48542 (Dec 25)
└── 6772616ef98a55a266d9fe17892af2d0a6ff7661b293c743281d10ba216f55d5bff6c522 (Dec 6)

• AURA authority in genesis: 0x56f99f0ae33c500a71bc28806181bbaf3323a9696bb5b230257e6a7965fd3b2b
• Key file prefix 617572 = "aura" in ASCII hex ✅ MATCH

On-Chain Data Verification (December 25, 2025)

Storage Key	Value	Notes
paras.paraLifecycles(5080)	0x01	Parathread ✅
paras.currentCodeHash(5080)	0x011b5fce05e81abd3521f7e3bd604e97a6c9f6a6afe36de39abe443b7ec69505	WASM hash ✅
paras.heads(5080) stateRoot	42aff38567293678d6ac6d25f88005c5d22cbbbbb0a1603f66266313495ffa36	Genesis match ✅
paras.heads(5080) extrinsicsRoot	03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314	Genesis match ✅

Collator RPC health check (ws://185.209.229.181:9944):

{"peers": 0, "isSyncing": false, "shouldHavePeers": true}
{"currentBlock": 0, "highestBlock": 0}

Verification Checklist

Requirement	Status	Evidence
Collator running	✅	systemctl status shows active for 3+ weeks
Relay chain synced	✅	13 peers, importing blocks 9,648,000+
Session keys configured	✅	Keys in keystore match genesis config
WASM hash matches	✅	On-chain = Collator
Genesis head matches	✅	On-chain = Collator
Coretime purchased	✅	Transaction successful
Coretime assigned	✅	broker.assign SUCCESS
Workplan updated	✅	Verified on Dec 6
--force-authoring	✅	In command line args
--collator	✅	In command line args

Collator Logs (During Expected Block Production)

Dec 25 11:39:19 [Parachain] Update at relay chain block #9645000 - backed: #4111751, #639717, #2689429...
Dec 25 11:39:24 [Parachain] Update at relay chain block #9645001 - backed: #4111752, #2689430...
Dec 25 11:39:30 [Parachain] Update at relay chain block #9645002 - backed: #4111753, #639718...
...
Dec 25 16:41:13 [Parachain] 💤 Idle (0 peers), best: #0 (0xa545…aaba), finalized #0 (0xa545…aaba)

Observation: ParaID 5080 never appears in backed or included lists. No slot claiming attempts. No block building attempts.

Current State (December 25, 2025)

Workplan Query (broker.workplan)

All entries are for timeslice 125,612 (next cycle). No entries for current timeslice range 120,572-125,611.

Task 5080 is NOT in the workplan for any timeslice.

Region Query (broker.regions)

broker.regions(120572, 42, 0xffffffffffffffffffff): None

The region no longer exists on-chain.

Para Lifecycle

paras.paraLifecycles(5080): Parathread

Questions

1. Why did the workplan entry disappear between Dec 6 and Dec 25?
2. Why did the region disappear from broker.regions?
3. Why didn't the collator attempt to produce blocks during active timeslices?
4. Is there a known issue with the Coretime Chain ↔ Relay Chain integration on Paseo?

Similar Issues

• paritytech/polkadot-sdk#3396 - Stalled block production with lookahead collator on Rococo
• paritytech/polkadot-sdk#4226 - Parachains not producing blocks after v1.2 runtime

Requested Support

1. Help understanding why the coretime assignment was lost
2. Guidance on any additional configuration needed for Agile Coretime
3. Verification that the Coretime ↔ Relay Chain integration is working correctly on Paseo

Contact

• GitHub: @jhoyosp
• Project: DePow (Decentralized Power Trading Platform)
• RPC Endpoint: ws://185.209.229.181:9944

Attachments

• Chain spec: paseo-5080-FINAL.json
• Transaction hash: 0xff4727b44b6fd14c687f96ae3c288a6bd33d2a7ebe5aba30f71383e3aca186e3
• Subscan link: https://coretime-paseo.subscan.io/extrinsic/0xff4727b44b6fd14c687f96ae3c288a6bd33d2a7ebe5aba30f71383e3aca186e3

🔗 zboto Link

[jhoyosp]

Update: Collator Code Review Complete

We've reviewed our collator code and configuration:

Configuration Verified ✅

• polkadot-sdk: 2509.0.0 (stable2509)
• Collator type: Lookahead (cumulus_client_consensus_aura::collators::lookahead)
• ConsensusHook: FixedVelocityConsensusHook<6000, 1, 3>
• Runtime APIs: AuraUnincludedSegmentApi, CollectCollationInfo, GetParachainInfo - all implemented
• Flags: --collator --force-authoring

Key Observation

The collator shows NO collation attempts in logs:

• No "Starting collation"
• No "Claiming slot"
• No "Building block"

Only idle messages:

[Parachain] 💤 Idle (0 peers), best: #0, finalized #0

And this recurring error every hour:

[Relaychain] non-existent query failed (QueryId(...))

Questions

Since you confirmed the assignment IS on the relay chain:

1. What logs should we see when the collator recognizes its core assignment?
2. Is there a specific runtime API the collator uses to query scheduled cores with Agile Coretime?
3. Does the lookahead collator work with bulk coretime assignments, or is additional configuration needed?
4. Should AllowMultipleBlocksPerSlot be true for async backing? (We have it as false)

We want to understand what's missing before making any WASM changes that would require re-registration.

[al3mart]

• Why did the workplan entry disappear between Dec 6 and Dec 25?
  

The workplan was correctly committed into the relay chain assignments. Hence the workplan is cleaned after being committed. One can check the storage key coretimeAssignmentProvider.coreDescriptors to see what tasks have been applied into which cores.

• Why did the region disappear from broker.regions?
  

Similar reason as above. Polkadot's docs seem to be going through a revamp atm so not all the information might be at:
https://wiki.polkadot.com/learn/learn-agile-coretime/
or https://docs.polkadot.com/polkadot-protocol/architecture/system-chains/coretime/#bulk-coretime-assignment
But one can always review how the broker operates in detail at https://github.com/paritytech/polkadot-sdk/tree/master/substrate/frame/broker/src

• Is there a known issue with the Coretime Chain ↔ Relay Chain integration on Paseo?
  

No, coretime is working just fine in the testnet.

• Is there a specific runtime API the collator uses to query scheduled cores with Agile Coretime?
  

If you were missing any API implementation the compiler would error at compile time. This shouldn't be your problem.

• Does the lookahead collator work with bulk coretime assignments, or is additional configuration needed?
  

As far as I am aware using the lookahead collator shouldn't present any problems.

• Should AllowMultipleBlocksPerSlot be true for async backing? (We have it as false)
  

You could configure this as true. However I wouldn't expect this to be the origin of your issue.

• Regarding logs

You should check that the collator is erring at block production, you could get more information by activating basic authoring logs with -lbasic-authorship=debug.

• Para Lifecycle

Not really relevant for this case. Now both chains marked as parathreads or parachains have both access to coretime. Understand this categorization more as legacy leftovers from the user facing tooling that isn't really that much differentiating anymore for most operations.

• Other notes

I would think that these two bootnodes:

  --bootnodes /dns/paseo-bootnode-0.parity.io/tcp/30333/p2p/12D3KooWQCkBm1BYtkHpocz3orfw1Qk9qrwJMEWN3D9TMxzhARtC \
  --bootnodes /dns/paseo-bootnode-1.parity.io/tcp/30333/p2p/12D3KooWPEyEfLVxmNRddjKW5SPCFj8g2X9XxyJSJUzXV3aKM5y4 \

are meant to be provided to the "validator part" of your collator node. After the --. If that is not the case and there are other nodes of your parachain operated by Pairty, inquiring them about why your chain is not producing blocks would be the correct approach.

Although less convenient, I'd rather pass a Paseo relay spec you can source from here rather than:

  --chain paseo \

Your checklist points to the basics being just fine. I don't believe the lookahead collator should be a problem, if you find yourselves stuck here for some time exploring a change to a slot based collator might be wort it.
I will suggest trying to see if the collator is having any issues at building blocks by reviewing the basic-authorship logs I suggested above. Per your message the authoring keys seem to be correctly set. It would be better to point the collator to a unique authoring key by specifying --keystore-path.
It is usually a good idea launching a local version first to have a good idea whether the testnet launch will go as expect or not.

[jhoyosp]

Hi @al3mart, thanks for the detailed response!

We applied all your suggestions:

• ✅ Moved bootnodes after --
• ✅ Added -lbasic-authorship=debug (and other debug targets)
• ✅ Added --keystore-path

Result: The problem persists. Logs still show:

ConnectToValidators request peer_set=Collation requested=0

The collator is NOT seeing any core assignment. We now have 11 relay peers (improved from 9), but still 0 parachain peers and stuck at block #0.

Updated service configuration:

Environment="RUST_LOG=info,parachain=debug,aura=debug,cumulus-collator=debug,sc_basic_authorship=debug,basic-authorship=debug"
ExecStart=/opt/depow-collator/blockchain/target/release/depow-node \
  --collator \
  --force-authoring \
  --chain /opt/depow-collator/blockchain/paseo-5080-FINAL.json \
  --base-path /var/lib/depow-collator \
  --keystore-path /var/lib/depow-collator/chains/depow_paseo/keystore \
  ... \
  -- \
  --chain paseo \
  --sync warp \
  --bootnodes /dns/paseo-bootnode-0.parity.io/tcp/30333/p2p/12D3KooWQCkBm1BYtkHpocz3orfw1Qk9qrwJMEWN3D9TMxzhARtC \
  --bootnodes /dns/paseo-bootnode-1.parity.io/tcp/30333/p2p/12D3KooWPEyEfLVxmNRddjKW5SPCFj8g2X9XxyJSJUzXV3aKM5y4

Question: How can we verify that ParaID 5080 is in coretimeAssignmentProvider.coreDescriptors? Is there a Polkadot.js query we can run to confirm the assignment is visible from the relay chain's perspective?

We believe the issue is that the assignment exists in the Broker pallet but the collator's lookahead mechanism isn't detecting it in the claim queue.

[al3mart]

As mentioned here I can see you have identified the offending issue.

To keep the context on both sides, this is the issue that you are hitting: paritytech/polkadot-sdk#10305

The PR hints to adding another collator being a good interim solution until you do the sdk uplift to a version including the fix.

Given that the issue does not appear to be related with the testnet, I'll close this issue now.