Blockchain Engineering: Building a read model
Building a read-model from an open-zeppelin ERC-1155 contract
After many years away from the crypto-sphere I recently got sucked back in by the recent insanity in NFT sales. I thought I would look into NFT generation and see what has changed. The ERC1155 standard quickly caught my attention as it seemed to solve some problems. No longer did you need multiple contracts for multiple tokens! Great, that saves on deployment costs. OpenSea directly integrates with 1155 contracts and we even have meta-data served by traditional APIs showing up on OpenSea stores! In some ways it appears as if crypto has advanced which is super cool.
Sadly this seems to be only on the surface.
Enter my latest project that may or may not go anywhere, NFT Battles.
So I start messing around using Remix (The ethereum IDE for solidity) and in a few hours I have an 1155 contract on the Matic Network. I start minting some tokens using Remix and over a weekend I have an NFT project integrated into OpenSea and I could start selling NFTs. Really super easy. Sounds great right? Well what if I want to do more than just a JPG on the internet…. well let me tell you some stories…
First thing that I didn’t notice when I was deploying my 1155 Contract was that, say I own multiple tokens, but I want to have a nice App that shows them that isnt OpenSea, how would I do that?
…
…
Ya you can’t. Ooooof. The 1155 standard contract gives you 2 public read-only functions, balanceOf and balanceOfBatch. Both of which require address and token Ids. So sure if you want to query all tokens on your contract you can eventually get a balance for a user. That is absolutely horrible for an app developer. For my contract you would need to make 2808 balanceOf calls to find the values for every token… or if you wanted to do the same for something like CryptoKitties it’d be over 2 million calls… Then what? Where do you store this state? In the front-end? … haha No. What happens if someone transfers a token or sells it?
But we have balanceOfBatch! Oh do we? The RPC services that power these API calls all have limits :D and they aren’t documented. So it’s not reliable to be able to batch call balances on all your tokens, yay!
I think you can see how this is just horrible… So while we have built tech to transact NFTs we have yet to build any tech to help developers you know… Use NFTs.
Well I guess I can help there at least a little bit. I sat down tonight intending to figure this problem out and apply my brain cause why not.
The blockchain is more or less exactly like your bank account. In double-entry accounting you always have a debit and a credit. Well that’s actually how blockchains and engineered as well. I got to reading through Web3 documentation to try and figure this out. I eventually found that you can use the 1155s transaction log to read and generate double-entry for an event sourced system.
The following code came after a few hours of drinking beer.
//Item Contract: '0x6558F6462Fdc83048b029AcEB485E85990Dfc514' | |
//Card Contract: '0x339b4e30B8A32b2473E48Aa6560298b1Fd28bb39' | |
const Web3 = require("web3"); | |
let web3 = new Web3(new Web3.providers.HttpProvider("https://polygon-rpc.com")); | |
(async (web3) => { | |
let blockNumber = await web3.eth.getBlockNumber(); | |
let origin = 18858793; | |
let target = origin + 5000; | |
do { | |
console.log(`Starting Loop with values ${origin} and ${target} max ${blockNumber}`); | |
let logs = await web3.eth.getPastLogs({ | |
fromBlock: origin, // block origin | |
toBlock: target, | |
address: "0x6558f6462fdc83048b029aceb485e85990dfc514", | |
topics: ['0xc3d58168c5ae7397731d063d5bbf3d657854427343f4c083240f7aacaa2d0f62'] | |
}) | |
for (let entry of logs) { | |
let data = web3.eth.abi.decodeParameters([{ | |
type: 'uint256', | |
name: 'TokenId' | |
}, { | |
type: 'uint256', | |
name: 'count' | |
}], entry.data); | |
// console.log("Transfer", entry.topics[2], entry.topics[3], data.TokenId, data.count); | |
console.log("Removal", entry.topics[2], data.TokenId, -1 * data.count); | |
console.log("Addition", entry.topics[3], data.TokenId, data.count); | |
} | |
origin += 5001; | |
target = origin+5000; | |
} while(target<blockNumber); | |
})(web3); |
To break this down a little, while it would be great if you could just say, Get All Logs for a contract… Unfortunately the RPC servers restrict you to a maximum of 100,000 blocks. Also the bigger the size of the Log Entry Pull the longer it takes and then you have to fight with some good’ol timeouts… So eventually you will probably end up here, where you just find a good size (in my case I’m just using 5000 size).
The “data” of the transaction is encoded in hex so you need to decode it to understand what is going on. The Topics of the Log are actually very well documented since this is a standard log entry of 1155. Topic [1] is the operator, Topic [2] is Who owned the token, Topic[3] is the account where the token is going. The data contains the token id and the number of those tokens that are being transferred.
So all that is left is to build an event-sourced system and then do some fancy aggregation with queries and you can have your read-model of the blockchain for a certain account :D … man this should exist as a service… maybe that can be my next project.
If you like this kind of content please consider subscribing. If you wanna come talk NFTs or keep up to date on my progress with NFT Battles come join my empty discord.