Threadbag - Persistance execution

Threadbag lets you run any scenario that you continously in the main interface of Bagpipes.

MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMN0OkO0NMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMKxc';oXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWKOl':0WMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMNd,'';dXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM0d;,xodNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM0d:'odl0MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMk:;:l;lKMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMKl;;od,cXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWx;;;od,oWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMNo,::dd,dWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMXo:clddckWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWKkO0OokxkWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMNkllxOXWkkWMMMMMMMMMMMMNNNWWMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMXd,,cxNWxdWMMMMMMMMMMMXdldxKMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWO::oddxldWMMMMMMMMMMMK:':dXMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMXXWMMMMMMMMMMMMMMMMMMMMMM0c:oxKXx;lXMMWMMMMMMKxc,:ONMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMM0o0MMMMMMMMMMMMMMMMMMMMMWx;;okXMWd;dOKXXXKOkkd:,;ldKWMMMMMMMMMMMMWNWWWMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMWkoKMMMMMMMMMMMMMMMMMMMMNd,:xXNMNllK0OOkkOOOKk:c0NdoXMMMMMMMMMMMNxoxx0WMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMNddNMMMMMMMMMMMMMMMMMMMXo:lOWWWk',OWMMMMMMNOccOWMXdo0NMMMMMMWWNk;':oKWMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMKoxNMMMMMMMMMMMMMMMMMWKkkOXMMWx..xWMMMMMXo:;oXMMMN0xxOOOOOkkx:,;xKNMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMOckWMMMMMMMMMMMMMMMMW0llONMMWo..oNMMMMNd;,cOWMMMMMMNX0O0KKkl:ccxWMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMNo:kWMMMMMMMMMMMMMMMM0:cKMWMNc..:KMMMWO:,:kWMMMMMMMMMMMN0o:lONxlKMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMKlcKMMMMMMMMMMMMMMMMO:dNMWMX:..,0WMWXOdoOWMMMMMMMMMMWOo:;oXWMOl0MMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMM0coXMMMMMMMMMMMMMMWk:kWMWWK;..'kMMXdokKWMMMMMMMMMMXd:,;oKMMMOl0MMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMWkcxNMMMMMMMMMMMMMNoc0MMWMO,...xWWk;lKWMMMMMMMMMW0l;,cONMMMMOoKMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMNdckNMMMMMMMMMMMW0loKMMMMXdcco0WO:cKMMMMMMMMMMNOxdoxXWMMMMMOdXMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMKlc0WMMMMMMMMMMNkdkXWMMMMMWWWW0cc0WMMMMMMMMMXd:oOXWMMMMMMMOxNMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMWOclKWMMMMMMMMMXo:xXMMMMMMMMN0dlOWMMMMMMMMW0c;dKWMMMMMMMMMxdNMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMNxlkXWMMMMMMMMO:;xNMMMMMMMW0ddONMMMMMMMMXd:l0WMMMMMMMMMMWxdWMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMW0odOXMMMMMMMWx,:kWMMMMMMWOc:xXMMMMMMWXOo:xNMMMMMMMMMMMMNodWMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWOclONMMMMMMXo:lOWMMMMMW0:,c0MMMMMMW0dod0WMMMMMMMMMMMMMNodWMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMWWWNKkddkKXXNNN0lco0WMMMMW0:':kWMMWWN0o;:xKWMMMMMMMMMMMMMW0clXMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMWNX0OkkxddooooooddxxdlclOXWWMW0l;;d00OOOxl,';lkKXWWMMMMMMMMMMMWd.,kWMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMWKkolccclollollollllooooooodxkOxc;:oddoool;',:looodk0XWMMMMMMMMMWd..xWMMMMMMMMMM
MMMMMMMMMMMMMMMMMMWKxlccccc::cc::c::c::::lllllloooolc;;clll:,;ccclloodddxkKWMMMMMMMMXc..cXMMMMMMMMMM
MMMMMMMMMMMMMMMMMNOc:::cc::;,;:,;:,';;,,;;ccllloooddoolcclc'..;ccccloodxxdkXMMMMMMMMK;..;0MMMMMMMMMM
MMMMMMMMMMMMMMMMWkc;;::::lodddl,,,..''.'',;;:::clllooooolloc;;;;:::coodddodKWMMMMMMM0,..'OMMMMMMMMMM
MMMMMMMMMMMMMMMMKc,,,;cd0NWWMWNOc........',,,;;::::cllllllloc::;:cc:clloold0WMMMMMMMO'...xWMMMMMMMMM
MMMMMMMMMMMMMMMNx:,;;cONMMMMMMMMXl.........''',;,;;::ccccccllc::;;::::cclldKWMMMMMMMx....dWMMMMMMMMM
MMMMMMMMMMMMMMMKxollo0WMMMMMMMMMMKc'..........',',,;;;;;::::c:;;;,;;;;:cclkNMMMMMMMM0c;;:OWMMMMMMMMM
MMMMMMMMMMMMMMW0lcodkXMMMMMMMMMMMMKc.............',,,,,,;,,;;;;,,,,;;;:ccxNMMMMMMMMMMWNNWWMMMMMMMMMM
MMMMMMMMMMMMMMMOlookNWMMMMMMMMMMMMMKo,...........'''',',,,,;;;;;;;;:::clkNMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMWx:lxXMMMMMMMMMMMMMMMMW0l,.......'''',,,,;;;;:::::::::cox0WMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMk,cKWMMMMMMMMMMMMMMMMMMWXkoc;'',,,,;,,;;;;;::::ccldxOKNWMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMWxc0MMMMMMMMMMMMMMMMMMMMMMMMWX0OkxxdddddddxkkO00KXNWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMXldNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMOcOWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMNolXMMMMMMMMMMMMMMMThreadBagMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMOckWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM

Multi-threaded Persistence work server for Bagpipes.

Made with: A Tokio runtime, Polodb + Sled, Subxt and actix-web.

Basics:

Once we have created the scenario, we can run it continously with threadbag:

• Step 1: Start Job
• Step 2: Query for Logs
• Step 3: Query for transactions

Lets get started by creating a scenario in the main interface at alpha.bagpipes.io, copy the link and copy the scenario id. Then lets start the job:

Start Job:

curl -X POST -H "Content-Type: application/json" -d '{"id": "qSxt94jqC"}' https://th-0.bagpipes.io/job/start -v

Stop Job:

curl -X POST -H "Content-Type: application/json" -d '{"id": "qSxt94jqC"}' https://th-0.bagpipes.io/job/stop -v

Query Job:

After we have started the job we want to continously query for logs. This can be see as the status update/main logger that will give you all output generated by the scenario worker.

Python code example:

base = "https://api.bagpipes.io"
scenario_id = "qSxt94jqC"
logs = requests.post(base+"/scenario/worker/logs", json={"id": scenario_id})
print("Logs returned: ", logs.text)

{"success":true,"result":["Starting worker","Decoding payload..","downloaded scenario data from api","Parsed scenario data","Starting worker","Decoding payload..","downloaded scenario data from api","Parsed scenario data","Starting worker","Decoding payload..","downloaded scenario data from api","Parsed scenario data","Starting worker","Decoding payload..","downloaded scenario data from api","Parsed scenario data","Starting worker","Decoding payload..","downloaded scenario data from api","Parsed scenario data","Starting worker","Decoding payload..","downloaded scenario data from api","Parsed scenario data","Got webhook data from the api: ","Latest data got back from uuid: {\"asset_id\": Number(7), \"address\": String(\"0x2e4ebdfb837f8b9ab416929986184b835a0c39901b4c5ab543e7d283f661fd67\")}","Building webhook request","uuid for webhook is: e5085097-1cc5-45fc-a5ea-c1db571c12a3","Webhook finished, moving on ","ChainQuery Node detected","ChainQuery Node Request built","ChainQuery got response back","workload executed","Sleeping","Starting worker","Decoding payload..","downloaded scenario data from api","Parsed scenario data","Got webhook data from the api: ","Latest data got back from uuid: {\"asset_id\": Number(7), \"address\": String(\"0x2e4ebdfb837f8b9ab416929986184b835a0c39901b4c5ab543e7d283f661fd67\")}","Building webhook request","uuid for webhook is: e5085097-1cc5-45fc-a5ea-c1db571c12a3","Webhook finished, moving on ","ChainQuery Node detected","ChainQuery Node Request built","ChainQuery got response back","workload executed","Sleeping"]}

Transaction Que

All transactions that gets drafted/generated for your scenarios, can be found here(/scenario/tx).

## Query for the transaction que 
txmempool = requests.post(base+"/scenario/tx", json={"id": scenario_id})

{"mempool":[{"chain":"polkadot","amount":"0","txType":"ChainQuery","Date":"2024-06-01T22:35:39.319179786Z","tx":"{\"nonce\":\"8\",\"consumers\":\"0\",\"providers\":\"1\",\"sufficients\":\"0\",\"data\":{\"free\":\"13,096,674,612\",\"reserved\":\"0\",\"frozen\":\"0\",\"flags\":\"170,141,183,460,469,231,731,687,303,715,884,105,728\"}}"}]}

An example for running a scenario with threadbag and your own local signer can be found here: https://github.com/XcmSend/threadbag-examples

Endpoints:

https://th-0.bagpipes.io

Example use case:

0: Navigate to Bagpipes:

Run it locally or use the public instance: https://app.xcmsend.com

1. Create a scenario in the main ui:

Scenario Link: https://app.xcmsend.com/#/create/?diagramData=YDqp6whUK

2. Copy the link and get the scenario id:

Go to Lab in the left side menu and press copy link of the scenario. Copy the last part of the link like in the image, this is your scenario_id.

The scenario id would be "YDqp6whUK" in this example.

3. Start the job worker with the scenario id:

curl -X POST https://th-0.bagpipes.io/job/start -H "Content-Type: application/json" \
-d '{"id":"YDqp6whUK"}'

output:

{'success': True, 'result': 'Job started'}

4: Query for logs:

curl -X POST https://th-0.bagpipes.io/scenario/worker/logs -H "Content-Type: application/json" \
-d '{"id":"YDqp6whUK"}'

output:

{"success":true,"result":["Starting worker","Decoding payload..","Parsed scenario data","Remark transaction generated","Drafting Remark tx from assetHub to assetHub","Building Action request","Building ChainNode request","Building ChainNode request","Building ChainNode request","Could not generate transaction","xTransfer transaction type","Drafting xTransfer tx from assetHub to hydraDx","Building Action request","workload executed","Sleeping"]}

5: Query for generated transactions:

curl -X POST https://th-0.bagpipes.io/scenario/tx-H "Content-Type: application/json" \
-d '{"id":"YDqp6whUK"}'

output:

{"mempool":[{"chain":"assetHub","amount":"0","txType":"Remark","Date":"2024-05-19T22:50:11.827281773Z","tx":"0x2404000714666c756666"},{"chain":"assetHub","amount":"1230000000000","txType":"xTransfer","Date":"2024-05-19T22:50:11.838491720Z","tx":"Could not generate transaction"}]}

Python example:

import requests, time

base = "https://th-0.bagpipes.io"
scenario_id = "set scenario id here"
print("Starting job: ")
js = requests.post(base+"/job/start", json={"id": scenario_id})
# {'success': True, 'result': 'Job started'}
print("start job response:", js.json())
print("sleeping for 10 seconds.. ")
time.sleep(10)

## Query logs for our scenario
print("Quering logs for: ", scenario_id)
logs = requests.post(base+"/scenario/worker/logs", json={"id": scenario_id})

print("Logs returned: ", logs.text)

## Query for the transaction que 
txmempool = requests.post(base+"/scenario/tx", json={"id": scenario_id})
print("Getting transaction queue: ", txmempool.text)

Mempool:

After a worker has started, you can query the /scenario/tx path to get a list of the transactions that have been generated. Query the transaction queue and sign them locally.

Bring your own signer:

Bagpipes is non-custodial, meaning we only generate the transactions for you, you have to sign it yourself. In this typescript example, we will query for the transactions, validate them then sign and broadcast them.

// fetch transaction list from threadbag
async function fetchTxMempool(scenarioId: string): Promise<MempoolItem[]> {
    try {
        const response: AxiosResponse<TxMempoolResponse> = await axios.post(`https://th-0.bagpipes.io/scenario/tx`, {
            id: scenarioId
        });
        return response.data.mempool;
    } catch (error) {
        console.error('Error fetching threadbag transaction mempool:', error);
        throw error;
    }
}

// take a hex encoded tx, sign it
async function string_tx_sign(tx_raw: string): Promise<SubmittableExtrinsic> {
    const api = await get_api(); // polkadot
  // convert the hex encoded transaction to a polkadot type
    const decodedTx = api.registry.createType('Extrinsic', tx_raw) as SubmittableExtrinsic;
    return decodedTx;//signedTx;
}

function get_account(){
    const senderPrivateKey = "";//change me
    const keyring = new Keyring({ type: 'sr25519' }); // substrate ss58 accountid32
	const account = keyring.addFromUri('add seed here');
    return account; 
}




async function main(){
  console.log(`fetching transactions for scenario id: `, scenario_id);

    const tx_list = await fetchTxMempool(scenario_id);
    const raw_tx = tx_list[0].tx; // take the first transaction in the list
    console.log(`Got response: `, tx_list);
    console.log(`Got hex encoded tx: `, raw_tx); // take the first tx as a test
    console.log(`decoding tx...`);
    // decode the xcm tx and validate args
    const l = await string_tx_sign(raw_tx);

    // decode the polkadot tx
    const decodedentry: any = l.toHuman();
    console.log(`tx human: `, l.toHuman());
    // access data like this
    const dest_paraid = decodedentry.method.args.dest.V3.interior.X1.Parachain;
    console.log(`Destination parachain: `, dest_paraid);
    console.log(`Destination account: `, decodedentry.method.args.beneficiary.V3.interior.X1.AccountId32.id);
    //lets verify that the transaction amount is correct
    const fun_amount = decodedentry.method.args.assets.V3[0].fun.Fungible;
    console.log(`Asset Amount: `, fun_amount);
    console.log(`verifying that its one dot`);
    assert.strictEqual(fun_amount, "10,000,000,000"); // one do
    // convert tx to object
 const decodedTx = await api.registry.createType('Extrinsic', raw_tx);
    // get an account and sign and broadcast tx:
    const sender = get_account();
   const tx_out =  await api.tx(decodedTx).signAndSend(sender);
    console.log('tx_out: ', tx_out);
}

Template repo: https://github.com/XcmSend/threadbag-examples

Enterprise/dedicated threadbag endpoints:

For dedicated threadbags endpoint, submit a github issue here:
https://github.com/XcmSend/threadbag-examples/issues/new