Dynamic Traversers in TrueBlocks

Custom Ethereum extraction for permissionless access to your data

Ethereum nodes come to consensus on a world-wide global ledger of smart contract invocations every 14 seconds. Everyone know this. This is what we celebrate about the chain. Not only is this data world-wide and consented-to, but it is also permissionless. At least that’s what we’re supposed to believe.

But, is it really permissionless?

I think the answer to that question is a resounding “No.” Ask yourself how you personally get data from the Ethereum blockchain. There’s only two answers: (1) you visit a block explorer, or (2) you go to website built by the developers of the dApp you’re using.

You want to know what’s going on with your ENS names? You go to http://ens.domain (a website).

You want to know how your GitCoin grant is doing? You go to http://gitcoin.co (another website).

You want to invest in DeFi? You go to Zerion (a website).

I got news for you. This is not permissionless.

Every time you go to a website, you are at the mercy of the human being who runs that website. In effect, you have to say please, oh please, great data-giver in the sky…please, please give me access to my own data.

The trouble — and the thing that makes it very obvious that this data is not permissionless — is that at any time the server (or the human) can say “No.” They can choose to not give you your own data.

That is the opposite of permissionlessness!

In my mind, permissionlessness means you can say “Fuck you, Mr. ServerMan. It’s not possible for you to stop me from getting the data I want, no matter what you do.”

If, in the end, blockchains are not permissionless, they are simply very expensive web servers. And if that’s the case, then everything we’re building will be for naught.

Using TrueBlocks to get Permissionless Data

To proceed, visit the installation instructions for downloading and installing TrueBlocks: http://github.com/TrueBlocks/trueblocks-core. Once you finished installing (if you have trouble, join our discord server), try this:

chifra init

This will initialize the TrueBlocks indexes and downloads the most recent collection of bloom filters. Next, run a small test to see if things are working properly

chifra list 0xf503017d7baf7fbc0fff7492b751025c6a78179b

This should return a list of about 600 appearances (blockNumber. transaction_id pairs. If it doesn’t, revisit the installation instructions.

Next, run this command:

chifra export 0xf503017d7baf7fbc0fff7492b751025c6a78179b

This displays the same list of transactions but, for each one, chifra queries the blockchain and presents the full details of each transaction to the screen. Try the above command with your own address.

There are many options to chifra export, all of which are presented with

chifra export --help

Traversers

In the list option, the function simply prints the blockNumber. transaction_id pair to the screen.

For export option, the display function prints the entire transaction and potentially a lot of more data such as articulation and accounting reconciliations.

Traversers are very general, and because they are C++ classes, you may customize them to do practically anything. In fact, this is what this article is about.

Dynamic Traversers

It’s a way of extending the chifra command line (and at the same time, the TrueBlocks API which mirrors the command line). This allows you to preform any task you want on any or all transactions from your accounts.

In the following paragraphs, we describe how you can write and build you own dynamic traversers.

C++ Code

Example Folder

From the top of the TrueBlocks repo.

cd ./src/examples/dynamic
make trav1
cp -p ./libtrav1.dylib $CONFIG/cache/obj/

Where $CONFIG depends on your operating system (it’s ~/Library/ Application Support/TrueBlocks on Mac, ~/.local/share/trublocks on Linux).

This will build a dynamic link library and copy it into the TrueBlocks’ cache where we store dynamic traversers.

Try to run the dynamic traverser

chifra export --load libtrav1.dylib 0xf503017d...6a78179b

In order to run a dynamic traverser, you first put it in the right folder and the simply tell chifra to load it.

This should print the hash for each transaction for the given address. Let’s take a look at the code. Open the file trav1.cpp in the example folder. It looks like this

This is the entirety of the traverser. That’s all you have to do to extract a perfect list of transaction hashes from your own accounts running straight up against your own Ethereum node. You don’t have to ask anyone for anything. Just permissionlessly take as much data as you want.

If you don’t see how powerful this — how upsetting of the web server apple cart decentralized data is — please re-read the above. Getting permissionless access to globally-agreed-to, near-perfect, world-wide data only happens if people actually have permissionless access.

Example GasCost Traverser

cd ./src/examples/dynamic
make trav2
cp -p ./libtrav2.dylib $CONFIG/cache/objs/
chifra export --load libtrav2.dylib 0xf503017d...6a78179b

This prints a table of the gas consumed for each transaction, the accumulated gas used so far, and the accumulated gas used on transactions that ended in error. We leave it as an exercise to the reader to study this fairly straight-forward code. I’ll make a few comments on the code below.

Notice that the makeTraverser function has not changed. Only the header and display function have changed.

Also notice that the code is very simple and very short. That’s the whole beauty of these dynamic traversers. You can write them in a short afternoon and they can do amazing things. For example, you’ll notice we’re accumulating totals as we traverse the data. You could, if you wished, add subtotals once per day or once per month or whatever (notice there’s a timestamp on every transaction which makes this calculation easy).

Another thing to notice is that we’ve derived our own C++ Traverser class in this second example. This gives you infinite flexibility as far as the design of your traverser. This is the holy grail of data access to the Ethereum chain. It’s fast (because it’s fully local and not shared like a web API); it’s 100% private and completely decentralized; it’s fully customizable; it’s not a website; and it’s permissionless.

Conclusion

If you like this article or you simply wish to support our work go to our GitCoin grant https://gitcoin.co/grants/184/trueblocks. Donate to the next matching round. We get the added benefit of a larger matching grant. Even small amounts have a big impact.

If you’d rather, please send ETH to our public Ethereum wallet here 0xf503017d7baf7fbc0fff7492b751025c6a78179b.

Thanks for reading.

Blockchain Enthusiast, Founder TrueBlocks, LLC and Philadelphia Ethereum Meetup, MS Computer Science UPenn