Exploring Optimism
A first exploration of Optimism data
This week, I finally gained access to an Optimism archive node. I found an amazing RPC provider called BlockJoy, who’s done a wonderful job setting me up with a node. BlockJoy uses TestInProduction’s version of Erigon, which means TrueBlocks works well. (TrueBlocks requires Erigon’s trace_ namespace — Geth has trace_, but it’s almost too slow to be useful.)
Using Erigon on Optimism is interesting for a few reasons: (1) we’ve been using Erigon for Ethereum mainnet for a long time, so we’re familiar with it, (2) we love Optimism, and (3) we love finding and reporting bugs. Our software hits the node software very hard, so we can test three different bits of software with one tool: TrueBlocks, BlockJoy’s infrastructure, and TestInProduction’s Erigon node. Four. Plus Optimism.
On with the exploration…
The Exploration
This article discusses my “first impressions” of Optimism’s on-chain data, and I hope to bring many more (and much deeper) investigations in the future.
First impressions:
- Optimism has a large number of blocks: more than 117,000,000;
- Either block times have changed since inception, or block numbers were skipped somewhere in the history;
- Before a certain block, there seems to have been a single transaction per block. After that block, there have been increasingly many transactions per block*;
* - Everyone who is familiar with Optimism knows this, but it becomes obvious early in our investigation. This change is due to the Bedrock fork.
We’ll discuss each of these observations below.
Getting started
If one wishes to explore Optimism data, the first thing one needs to do is get access to an Optimism node. Another way of saying this is that you need an RPC endpoint. The faster, the better. The cheaper, the better. The more convenient, the better—in that order.
Of course, being decentralization maxis, we tried to set up an Optimism node on our local machines. It was hard. As I’ve always feared, this task has exceeded my skill level. Erigon on Mainnet is hard enough. Optimism is harder.
Execution client / schemxecution client. Terabytes / schmerabytes.
Luckily, we found someone to help us: a company called BlockJoy. They were able to set up a fully synced Optimism node very quickly (and they tell us they’re improving their process every day). Their service is not cheap, but it’s well worth it, especially compared to any other provider we’ve tried (and we’ve tried many). I highly recommend them. If you want to set up an Optimism node, connect with them. Tell them TrueBlocks sent you.
Philosophical Sidebar
I have long argued that running the node on your own hardware is the only way to truly gain self-sovereignty, and I am still completely convinced that this is true, but — this is an unfortunate “but” — at a certain point, unless the node software is radically modified, running your own node will become impossible. I fear that time has come, at least for layer 2 chains. Goodbye self-sovereignty.
Another quick sidebar. People ask me, “Why should I run a node?” Multiple answers come to my mind: FTX, privacy, the 2008 banking crisis, and Ronald Regan’s famous quip (as modified): “Do not trust and always verify.” What are we even building if we’re not building permissionless access to the data?
Configuration
I won’t explain how to configure TrueBlocks (see our repo). We’ve written about it multiple times in previous articles. We assume you’ve done that and can, for example, run chifra status successfully.
Assuming you can run this command from your command prompt:
> chifra config editThis will open the configuration file's editor. Near the top of the file, you will see a field called defaultChain. Set this value to optimism.
Next, search for the [optimism] section and set the rpcProvider value to the value for your Optimism RPC endpoint (see BlockJoy). Save your work, and you’re ready to go.
Chifra When
We will limit ourselves in this article to a tool called chifra when. chifra when reports on the timestamps and dates for any block (or range of blocks). This data is surprisingly useful when first encountering a chain and carries a large amount of information.
How many blocks?
For example, finding the earliest and most recent blocks (as of this writing) is easy. Simply run:
> chifra when 0 latest
bn timestamp date
--------------------------------------------
0 1636665386 2021–11–11 21:16:26 UTC
117278316 1710155409 2024–03–11 11:10:09 UTCFrom this, we can see that Optimism was launched at 11:16 AM on November 11, 2021. (Oddly, my birthday. Optimism was a gift to me. Thanks, Retro!)
The chain has been alive for 73,490,023 seconds, which is 1,224,834 minutes, 20,414 hours, 850.6 days, or about 2.3 years. That all makes sense…sort of.
Blocks per second
My intuition told me that 117 million blocks was too many, so I started looking into that.
Can we see how long each block has taken to produce? How many blocks have been produced per second? We already have that information:
(1,710,155,409s - 1,636,665,386s) / 117,278,316 blocks
73,490,023s / 117,278,316
0.627 seconds/blockThis seems odd. I thought block times on Optimism were two seconds. Let’s see if we can dig deeper:
> chifra when 117277300-117278300 --no_header | cut -f2 >timestamps.csv
> awk 'NR>1{print $1-p} {p=$1}' timestamps.csv | sort -u
Result:
> 2The above command runs chifra when against the 1,000 latest blocks on Optimism (and suppresses the header). It then extracts the second column (timestamps) using cut and stores the result in a CSV file.
The next step uses awk to calculate the difference between each successive timestamp. Sorting the result uniquely gives us a list of all the uniq block times. We find there is only one unique block time: 2 seconds. Block times on Optimism are two seconds long.
So, why does the previous calculation say block times are 0.627 seconds? There are two possibilities: (1) the block times changed, or (2) there is a skip in the block numbers.
We will try to figure this out in the remainder of this article.
Did Block Times Change?
We’re going to switch now to using the new TrueBlocks GoLang SDK. It provides the same capabilities and options as the command-line tool but is more flexible.
The code for this article is in the ./src/examples/optimism1 folder of our repo. We assume you’ve cloned the repo and have navigated to the ./build folder as instructed. You can modify the main.go file in the examples folder, run make && optimism1 to see the results.
To get started with the SDK, we’ll use a very simple example that uses the When endpoint to query the latest block:
1. package main
2.
3. import (
4. "fmt"
5.
6. "github.com/TrueBlocks/trueblocks-core/v0/sdk"
7. )
8.
9. func main() {
10. whenOpts := sdk.WhenOptions{
11. BlockIds: []string{"latest"},
12. }
13.
14. blocks, _, _ := whenOpts.Query()
15. for _, block := range blocks {
16. fmt.Println(block.BlockNumber)
17. }
18. }Here are a few comments about this code:
- If you don’t understand the above code, please go back to the computer programmer’s school. Come back when you’re ready.
Whenis just one of the SDK’s endpoints. For each endpoint, one first fills an associatedOptionsstructure, as shown in line 10. The fields of theOptionsstructure correspond to the command-line options for that command. Runchifra when --helpfrom your command line. You will see the options. In this case, we’ve specified “latest” as the block identifier, which means we want to retrieve the head of the chain.- After building the options, call into the
Queryfunction as we’ve done on line 14. Each endpoint has aQueryfunction that returns adataarray, ametaDatapointer, and anerror. (We’ve ignored themetaDataand theerrorin this case.)
That’s as simple as it is, but it’s super powerful. You can access any piece of data you want on any chain.
We’re going to take a short diversion to discuss the GlobalOptions structure, which is available to every command.
Global Options
The GlobalOptions structure is defined as:
...
5. type Globals struct {
6. Ether bool
7. Raw bool
8. Cache bool
9. Decache bool
10. Verbose bool
11. Chain string
12. Output string
13. Append bool
14. }I won’t explain every field other than these two:
Chain: this option allows you to specify which chain to operate against. (Note: You must first obtain an RPC endpoint for the chain.)Cache: This field causes the result of any query to be stored in a very fast binary cache. You’ll want to use this for performance reasons.
We’ve already taken care of the Chain value by specifying defaultChain in the configuration file, so we can ignore that. Note, however, how interesting the Chain option can be. You can use this value to scan multiple chains. For example, you could kick off a Go routine for each of six different chains, checking for balances on each chain. You could even do this for multiple addresses. We’ll leave that intriguing thought there for homework.
The Cache value is a boolean. It is off by default, but you should almost always turn it on. It has a significant effect on the speed of your analysis. Blockchain data is immutable, which means it caches easily. Once you’ve queried the RPC of some data, you should never query it again. The speedup is very noticeable, especially if you’re running against a remote endpoint.
To use the Global options to enable the cache, modify the above code thus:
...
9.
10. whenOpts := sdk.WhenOptions{
11. BlockIds: []string{"latest"},
12. Globals: Globals{
13. Cache: true,
14. },
15. }
...Be careful, though. Storing every query in the cache will lay a fairly large amount of data on your disc. You may use the Decache option to remove things from the cache.
Back to the point
Let’s write some more code. We want to scan the chain’s history to determine where the block times changed. We’ll start by scanning every millionth block.
At each block, we calculate the number of seconds that have passed since the block one million blocks before. We simply report the result to the console. This code:
1. package main
2.
3. import (
4. "fmt"
5.
6. "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/base"
7. "github.com/TrueBlocks/trueblocks-core/src/apps/chifra/pkg/logger"
8. "github.com/TrueBlocks/trueblocks-core/v0/sdk"
9. )
10.
11. const interval = 1000000
12.
13. func main() {
14. meta, err := sdk.MetaData()
15. if err != nil {
16. logger.Fatal("Error getting metadata", "err", err)
17. }
18.
19. blockIds := make([]string, 0, int(meta.Latest/interval))
20. for i := 0; i < int(meta.Latest); i += interval {
21. blockIds = append(blockIds, fmt.Sprintf("%d", i))
22. }
23.
24. lastTs := base.Timestamp(0)
25.
26. whenOpts := sdk.WhenOptions{BlockIds: blockIds}
27. blocks, _, err := whenOpts.Query()
28. if err != nil {
29. logger.Fatal("Error querying blocks with WhenOptions", "err", err)
30. }
31.
32. for _, block := range blocks {
33. diff := block.Timestamp - lastTs
34. secsPerBlock := float64(diff) / float64(interval)
35. fmt.Printf("%d,%d,%d,%f\n", block.BlockNumber, block.Timestamp, diff, secsPerBlock)
36. lastTs = block.Timestamp
37. }
38. }produces this data:
bn ,timestamp ,diff , perBlock
------------------------------------------
...
104000000 ,1685751416 , 280811 , 0.280811
105000000 ,1686011687 , 260271 , 0.260271
106000000 ,1687598777 ,1587090 , 1.587090 <------
107000000 ,1689598777 ,2000000 , 2.000000
108000000 ,1691598777 ,2000000 , 2.000000
...Notice anything? We do.
We found the 1,000,000 block range where the block times changed. Somewhere between 105,000,000 and 106,000,000.
Philosophical Sidebar (Redux)
I want to harken back to my previous philosophical sidebar. The above data was acquired without asking anyone’s permission. No authority figure tells us what happened, nor is there anyone gatekeeping. We can just pick this data out of the ether (as it were). This is the beauty of truly open, permissionless data. Not even Optimism can keep us from seeing this data. (Yes. BlockJoy can gatekeep this data…”so long” to self-sovereignty.)
Some Charts
Let’s see a chart:
It’s pretty clear where the block times changed. In the actual code, we dig deeper and identify the exact block where the timestamps change. The change was scheduled by timestamp (June 6, 2023, at 16:00 UTC), which corresponds to block 105,235,061.
(Note: When we dug into the detailed data, we disabled the cache. We didn’t want to store 1,000,000 blocks on our disc.)
Lasering in
To find what we were looking for, we focused on the block range where the block times changed from obviously fluctuating to perfectly flat. We found some weird shit.
Here’s a sampling of the data surrounding the Bedrock hard fork:
bn ,timestamp ,diff , perBlock
------------------------------------------
...
105235049 ,1686067231 ,0 ,0.000000
105235050 ,1686067231 ,0 ,0.000000
105235051 ,1686067231 ,0 ,0.000000
105235052 ,1686067246 ,15 ,15.000000 <-------
105235053 ,1686067246 ,0 ,0.000000
105235054 ,1686067246 ,0 ,0.000000
105235055 ,1686067246 ,0 ,0.000000
105235056 ,1686067246 ,0 ,0.000000
105235057 ,1686067246 ,0 ,0.000000
105235058 ,1686067246 ,0 ,0.000000
105235059 ,1686067246 ,0 ,0.000000
105235060 ,1686067246 ,0 ,0.000000
105235061 ,1686067246 ,0 ,0.000000
105235062 ,1686067325 ,79 ,79.000000 <-------
105235063 ,1686068903 ,1578 ,1578.000000 <-------
105235064 ,1686068905 ,2 ,2.000000 <-------
105235065 ,1686068907 ,2 ,2.000000
105235066 ,1686068909 ,2 ,2.000000
105235067 ,1686068911 ,2 ,2.000000
105235068 ,1686068913 ,2 ,2.000000
105235069 ,1686068915 ,2 ,2.000000Do you notice anything weird? I do. I see at least five things:
- Before the hard fork block, blocks were produced once every 15 seconds; however, multiple blocks were created at the same time. Many blocks have the same timestamp, as is obvious in the chart below.
- Block
105,235,062(2023–06–06 16:00:46) took78seconds to appear. This is likely the hard fork block. - Block
105,235,063(2023–06–06 16:02:05) took1,578.2seconds to appear, which is more than26minutes. I’m not sure why this happened, but it would be interesting to dig even deeper. - After the hard fork, blocks consistently take two seconds.
- It’s no wonder Optimism has such a high block number.
Here’s another chart (we removed the outliers):
If you can’t see the hard fork from here, you must visit Warby Parker immediately! (Walk, don’t drive.)
The width between successive orange lines indicates the number of transactions in each 15-second period. Before the hard fork, Optimsim processed a group of transactions once every 15 seconds, producing a separate block for each transaction. You can see this in the wider or less wide distances between the bars—more transactions, more blocks. It's very quirky.
I don’t know about you, but I find all this really interesting. And…and this is the point of the article…we gained all this insight using just one command: When. Imagine what we can do with the rest of TrueBlocks.
(A quick note about the cache: the first time we produced this data, it took 188.4 seconds. This was without the cache and hitting against the remote RPC endpoint. When we ran the same code the second time, it took 0.665 seconds—238times faster. Use the cache.)
Fini
Okay, I admit it. The above is not brain science. But…it’s done with 38 lines of very simple, easy-to-understand code. We’ve purposefully limited ourselves to using only one of the 22 different tools available with TrueBlocks.
Here’s a word cloud of the commands available through TrueBlocks and their relative usefulness. Bigger is more useful:
Finally, here’s something I’ve shared on Twitter in the past. This is everything TrueBlocks provides, and it’s a lot. If we missed something, please let us know. Every command and option on this list is available through the SDK, through the command line, and (identically) through the API server.
Conclusion
TrueBlocks is funded from personal funds and grants from Optimism Retro PGF (2022, 2023), The Ethereum Foundation (2018, 2022), Consensys (2019), Moloch DAO (2021), Filecoin/IPFS (2021), and our lovely GitCoin donors.
If you like this article and wish to support our work, please donate to our GitCoin grant using ETH or any token. If you’re an Optimism badge holder, vote for our project. Or send us a tip directly at trueblocks.eth or 0xf503017d7baf7fbc0fff7492b751025c6a78179b.
