TrueBlocks Services SDK

Or How to Watch a Blockchain Without Even Trying

In a previous article, Mist II — Revenge of the Nerds, we discussed Khedra and how, if it had been part of the original Mist browser, that ill-fated piece of software may have stood a better chance of working.

Alas, it did not. Mist has been relegated to the dustbin of history.

But we (TrueBlocks) have not lost faith. We’re still working on what we’ve now tentatively started calling Mist II.

“Mist is dead. Long live Mist II.”

Obligiatory DalleDress Identicon for the phrase “TrueBlocks Services Architecture”

In this brief (AI-assisted) document, we outline a component of our system that facilitates the “watch-the-tip-of-the-chain-with-a-long-running-process” aspect of this work. We refer to it as the TrueBlocks Services Architecture. This article is a description of the same.

TrueBlocks Services Architecture

In our TrueBlocks Core code (at https://github.com/TrueBlocks/trueblocks-core), there is a submodule called the SDK: https://github.com/TrueBlocks/trueblocks-sdk.

Part of the SDK is called the TrueBlocks Services Architecture (./sdk/services). It’s the basis of a software program called Khedra, described in depth in the article mentioned above.

Here are the key components:

Core Service Framework

1. Service Interface (service.go)

Defines the Servicer interface that all services must implement:

• Name() — Service identifier
• Initialize() — Set up logic
• Process(chan bool) — Main service loop
• Cleanup() — Graceful shutdown
• Logger() — Logging interface

2. Extended Interfaces

• Pauser — Services that can be paused/un-paused
• ChildManager — Services that manage child processes

3. Service Manager (service_manager.go)

• Orchestrates multiple services
• Handles graceful shutdown with signal management
• Supports pause/unpause operations across all services
• Provides concurrent cleanup with wait groups

Individual Services

1. Scraper Service (service_scraper.go)

Purpose: Continuously scrapes blockchain data and maintains the Unchained Index that enables fast queries across the entire blockchain history. This service monitors the blockchain for new blocks, processes transactions, receipts, logs, and traces, and updates the local index structures that power TrueBlocks’ rapid data retrieval capabilities.

Key Features:

• Supports multiple initialization modes: “none”, “blooms”, “all”
• Multi-chain support with configurable targets
• Pausable operation
• Configurable sleep intervals and batch sizes
• Reports scraping progress (blocks behind, staged, finalized)
• Handles “caught up” detection (within 28+4 blocks of head)

2. Monitor Service (service_monitor.go)

Purpose: Provides blockchain address monitoring with automated change detection and configurable data extraction. This service continuously monitors all existing monitors in the system, detects when new transactions appear for tracked addresses, and automatically extracts and caches specified data types based on per-address configuration. A frontend desktop application, like Mist II, can use this service to keep its backend data “fresh”.

Key Features:

• Continuous Monitoring: Automatically discovers and monitors all existing address monitors
• Change Detection: Uses the SDK’s version of chifra list — count to detect new appearances for tracked addresses
• Channel-Based Processing: Asynchronous pipeline for handling detected changes
• Configurable Data Extraction: Per-address configuration for logs, traces, transactions, receipts, and balances, or any combination
• Automatic Caching: Extracted data is automatically cached to the TrueBlocks cache system
• Batch Processing: Efficient batch checking of multiple addresses (default: 200 per batch)
• Thread-Safe: Concurrent-safe design with proper mutex protection
• Pausable Operation: Can be paused and resumed without losing state

Data Types Extracted:

• Event logs with optional articulation
• Transaction traces
• Transaction details
• Transaction receipts
• Balance snapshots
• Reconciled profit and loss statements

Configuration Options:

• ExtractLogs Extract and cache event logs
• ExtractTraces Extract and cache traces
• ExtractTransactions Extract and cache transactions
• ExtractReceipts Extract and cache receipts
• ExtractBalances Extract and cache balance snapshots
• Articulate Apply articulation to extracted data
• CacheResults Enable automatic caching

For detailed documentation, see [MONITOR_SERVICE].

3. IPFS Service (service_ipfs.go)

Purpose: Manages IPFS daemon for distributed storage and retrieval of blockchain index data across the TrueBlocks network. This service enables decentralized sharing of index chunks, allowing users to download pre-computed index data rather than building it from scratch, significantly reducing initial sync times.

Key Features:

• Detects if IPFS is already running
• Starts the IPFS daemon if needed
• Reads the IPFS configuration from ~/.ipfs/config
• Extracts the Api port from the multi-addr format
• Graceful process management with timeouts

4. Control Service (service_control.go)

Purpose: Provides HTTP REST API for service management and runtime control of the TrueBlocks service ecosystem. This service enables administrators and monitoring systems to pause, resume, and query the status of individual services without requiring direct access to the underlying processes or configuration files.

Key Features:

• /status or /isPaused — Check service status
• /pause — Pause services
• /unpause — Resume services
• JSON responses
• Find available control ports (8338, 8337, 8336, 8335)

5. API Service (service_api.go)

Purpose: Runs the TrueBlocks API daemon that provides RESTful HTTP endpoints as a fallback interface for remote access to blockchain data. This service is intended for scenarios where the scraper and monitor must run on a remote machine, mimicking all features of the SDK and command line chifra tool, though using a remote API significantly reduces the speed advantages of TrueBlocks’ local, streaming data architecture.

Key Features:

• Automatically finds available ports (8080, 8088, 9090, 9099)
• Uses the TrueBlocks SDK DaemonBytes function
• Handles context-based cancellation
• Provides HTTP API endpoints for blockchain data

Service Coordination

Scraper Reporting (service_scraper_report.go)

Tracks blockchain synchronization status

Reports metrics like:

• Head block number
• Blocks behind (unripe, staged, finalized)
• Processing batch sizes
• Timestamps

Integration Points

The services are primarily used by:

1. Khedra daemon (khedra/app/action_daemon.go) — Main orchestrator runs scraper, monitor, api, and IPFS services
2. Test runner (src/dev_tools/testRunner/main.go) — Testing infrastructure to start the API for testing

Service Lifecycle

1. Initialization: Each service sets up its dependencies and configuration
2. Process: Services run their main loops with context-based cancellation
3. Ready Signal: Services signal when they’re ready to process requests
4. Graceful Shutdown: Services clean up resources on termination signals
5. Pause/Resume: Compatible services can be paused and resumed during operation through controller api

Key Design Patterns

• Interface-based design for extensibility
• Context-based cancellation for clean shutdowns
• Signal handling for graceful process management
• Port discovery for avoiding conflicts
• Structured logging with slog
• Concurrent execution with goroutines for speed
• Error handling with proper cleanup (standard GoLang)

Usage Example

// Create services
apiSvc := NewApiService(logger)
scraperSvc := NewScrapeService(logger, “download”, “all”, []string{“mainnet”}, 13, 2000)
controlSvc := NewControlService(logger)

// Create service manager
services := []Servicer{apiSvc, scraperSvc, controlSvc}
manager := NewServiceManager(services, logger)

// Start all services
manager.StartAllServices()
manager.HandleSignals()
```

API Endpoints

The Control Service provides REST endpoints for runtime management:

• GET /status?name=<service_name> — Get the status of specific services or all services
• POST /pause?name=<service_name> — Pause specific service or all pausable services
• POST /unpause?name=<service_name> — Resume specific service or all paused services

For comprehensive API documentation including all available endpoints, request/response formats, and usage examples, please refer to the TrueBlocks API Documentation.

Architecture Benefits

This architecture provides a robust foundation for running TrueBlocks as a distributed system with multiple coordinated services handling different aspects of blockchain indexing, API serving, and data management.

Your Support is Welcome

TrueBlocks is funded by personal funds and grants from The Ethereum Foundation (2018, 2022, 2024), Optimism Retro PGF (2022, 2023), Moloch DAO (2021), Filecoin/IPFS (2021), ConsenSys (2019), and our generous GitCoin donors.

If you like this article and wish to support our work, please donate to our GitCoin grant using ETH or any token. If you’re an Optimism badge holder, vote for our project in future Retro rounds. Or send us a tip directly at trueblocks.eth or 0xf503017d7baf7fbc0fff7492b751025c6a78179b.