Build and Run Filecoin Venus in Testnet

11 min readAug 2, 2024

A step-by-step guide on running Filecoin node implementation Venus on your local machine with multipass.

Intro & Overview

Venus is an interoperable alternative Filecoin network node implementation, primarily based on Go language. Built by the IPFS Force team, it works like the the main implementation Lotus (based on Rust and Go), but with the added benefit of achieving high availability and miner modules designed for larger scale (for example more than 10PiB) operations of Filecoin storage nodes.

Venus is in a collection of products that is evolving over time, as of June 2024, there are eight modules that one can install for a full implementation that takes care of all life cycles that are needed of a storage provider and storage client.

Here is a full list:

|      Product     |                             Role                            |
|:----------------:|:-----------------------------------------------------------:|
|       Venus      |         Daemon node for Filecoin chain interactions         |
|   Sophon Miner   |              Proving and winning block rewards              |
| Sophon Messenger |                   Chain message management                  |
|    Sophon Auth   |                   Authorization utilities                   |
|  Sophon Gateway  | Provides an external interface to control all the services. |
|   Venus Wallet   |              Account addresses and wallet keys              |
|     Damocles     |       Task scheduling, sealing, proving & taking deals      |
|      Droplet     |              Data client tool for making deals              |

Venus setup is slightly more complicated because it is designed for Filecoin storage providers that run a larger operation to provide enterprise grade services. Learn more about Filecoin storage deals here and watch this detailed video created by Patrick from factor8 on Filecoin storage provision.

What this tutorial covers

In this tutorial, we will set up a full node service on Filecoin’s Calibration testnet on Mac to allow querying of the chain information. This only requires the installation of a sophon-auth service and the latest stable Venus node service This is a fundamental step to deploy a storage node or a client node. For information & assistance on setting up a storage providing node check out VenusHub (Venus’ community incubator program).

Prerequisites & Prerequisites

Node Versions

Network upgrades are an important procedure for the Filecoin nodes to keep up with the latest implementation of Filecoin Improvement Proposals (FIPs) which might include important protocol updates such as features and economics changes.

Before starting, check Venus project’s Github discussion where it highlights the latest stable version against Filecoin Network Version (NV). For the duration of this tutorial, NV22 was the latest stable upgrade so these versions will be used.

The current version of this tutorial (June 2024), was written for Network Version 22

Hardware

To run this, a PC or MAC that at least has 8 cores, 12GB of memory and 100GB free storage space on disk is needed. Since we will be spinning up at least two ubuntu with one of them requiring 8 cores, 8GB of memory & 30 GB of disk space.

Multipass

Because more than one ubuntu machine is needed, a hypervisor is needed to run Virtual Machines (VMs) on a single computer. The choice for this tutorial is Multipass, which is a lightweight VM manager developed by Canonical.

Follow the guides to install on the machine, then confirm successful installation by opening terminal or cli to check:

$ multipass launch - name primary
$ Starting primary

This might take some time.

Next use the `list` command to see if VMs are running:

The shell can be accessed via `$ multipass shell venus-one` for example.

Golang 1.20+

Venus requires a Go version later than 1.20. So in each VM install the latest golang by following commands.

sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt update
sudo apt install golang-go

Check the golang is installed correctly via `$go version`.

$ go version
go version go1.22.3 linux/arm64

Step by Step Guide

Node Planning

Since Venus is designed for high availability, the sensible installation will involve not just one node. Let’s have a few nodes and assign roles to them as if this is a full setup:

Create Nodes

Create the VMs on multipass with the following commands:

$ multipass launch lts - name venus-one - memory 2G - disk 10G - cpus 2;
$ multipass launch lts - name venus-two - memory 2G - disk 10G - cpus 2;
$ multipass launch lts - name venus-three - memory 8G - disk 30G - cpus 8

Note: set a higher resource for `venus-three` nodes to allow import of snapshot and hosting of the entire calibration net’s chain information.

Accidentally created VMs can be deleted here`$ multipass delete xxx` to delete and clear it from the list by using `$ multipass purge`.

Download Snapshot

Once the machines are all up, access the shell of `venus-three` via `$multipass shell venus-three` and download the calibration net snapshot image from ChainSafe Forest team’s Snapshot Services. This holds the chain information needed to kick-start Venus node so it does not have to sync from the beginning (2022 November) which will take several months to complete.

The current snapshot size for calibration net is ~3.5GB which could take some time. You can always find the latest snapshot here. Learn more about the Filecoin blockchain snapshot from this video made by the Forest team.

Notice that the link `https://forest-archive.chainsafe.dev/latest/calibnet/` is a 302 redirect to the latest compressed .car file. Using curly, we can find the actual file address:

$ curl -IL https://forest-archive.chainsafe.dev/latest/calibnet/
HTTP/2 302
date: Wed, 29 May 2024 09:56:04 GMT
content-type: text/plain;charset=UTF-8
content-length: 6
location: /archive/calibnet/latest/forest_snapshot_calibnet_2024–05–29_height_1654931.forest.car.zst
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=cIWDl7I2ZFL3KnnXCvhM2wUCpRI60nhI83XTH%2FCze5PdGJL9UFltLEwsB%2FGAU8JztzJwjGyV3vYYlQqffbZVPy7G0kQKWpG9tuZpabEdRoieFvfdy13AJDsRRJN%2BpCUEfdhADxwuoZCJ%2FxWyoixl"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 88b59cc3dfe91f67-MEL
alt-svc: h3=":443"; ma=86400
HTTP/2 200
date: Wed, 29 May 2024 09:56:05 GMT
content-length: 4017798704
accept-ranges: bytes
content-disposition: attachment; filename*=UTF-8''forest_snapshot_calibnet_2024–05–29_height_1654931.forest.car.zst; filename="forest_snapshot_calibnet_2024–05–29_height_1654931.forest.car.zst"
etag: "6eeb89e1164dccd7bd6baf6b71971aaf-1"
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=PQ4qelxQOU7PILPIyK8N2eTeYfa5vULd23ZJTS%2BKsHONo60j2U%2BJlxVejir0qUTVq6JsMlMMGaFhnLHrRXo3p1GgTsI%2BZW1nDESt2URu85lowlXus3lG0gdw9ZVo7QfnsvaKb8vSqk05xYO%2BrBvh"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 88b59ccadc8c1f67-MEL
alt-svc: h3=":443"; ma=86400

On the HTTP 203 response, we can find the redirect address is `/archive/calibnet/latest/forest_snapshot_calibnet_2024–05–29_height_1654931.forest.car.zst`, concatinating it with the domain we get the actual download address `https://forest-archive.chainsafe.dev/archive/calibnet/latest/forest_snapshot_calibnet_2024-05-29_height_1654931.forest.car.zst`.

To download, in `venus-three` use curl with the new url. The downloaded file should be around 3.9GB large.

curl -O https://forest-archive.chainsafe.dev/archive/calibnet/latest/forest_snapshot_calibnet_2024-05-29_height_1654931.forest.car.zst

This will take a while depending on your network speed, other setups can continue in the meantime.

Install Sophon-auth

Access `venus-one` VM and install sophon-auth. This is a module that will allow management of all other services in the venus cluster.

To install, clone the repo, enter the directory `sophon-auth` use `make` to build sophon-auth from source.

$ git clone https://github.com/ipfs-force-community/sophon-auth.git
$ cd sophon-auth
$ git checkout $ v1.15.0
$ make

Once successfully installed, you can start the sophon-auth by `nohup ./sophon-auth run > auth.log 2>&1 &`. This starts sophon-auth in the background with logs sent to the auth.log file. To follow the log `tail -f auth.log`.

By default sophon-auth will not use a database connection and runs on port 8989. But for actual usage, users can change them at `~/.sophon-auth/config.toml`.

Example:

Listen = "127.0.0.1:8989"
ReadTimeout = 60000000000
WriteTimeout = 60000000000
IdleTimeout = 60000000000
[Log]
LogLevel = "trace"
Type = 0
HookSwitch = false
[DB]
Type = "badger"
DSN = ""
MaxOpenConns = 64
MaxIdleConns = 128
MaxLifeTime = 120000000000
MaxIdleTime = 60000000000
Debug = false
[Trace]
JaegerTracingEnabled = false
ProbabilitySampler = 1.0
JaegerEndpoint = "localhost:6831"
ServerName = "sophon-auth"
[Metrics]
Enabled = false
[Metrics.Exporter]
Type = "prometheus"
[Metrics.Exporter.Prometheus]
RegistryType = "define"
Namespace = "sophon_auth"
EndPoint = "/ip4/0.0.0.0/tcp/4568"
Path = "/debug/metrics"
ReportingPeriod = "10s"
[Metrics.Exporter.Graphite]
Namespace = "sophon_auth"
Host = "127.0.0.1"
Port = 4568
ReportingPeriod = "10s"

Create API Token for Venus

In order for Venus to access this service, the `venus-one`’s IP address and an user with API token need to be passed in the venus launch command.

The IP address can be obtained by `$ multipass info venus-one`, where the IPv4 address in my case `192.168.64.11` needs to be saved for later.

Name: venus-one
State: Running
Snapshots: 0
IPv4: 192.168.64.10
Release: Ubuntu 24.04 LTS
Image hash: 54f6b62cc8d3 (Ubuntu 24.04 LTS)
CPU(s): 2
Load: 0.13 0.05 0.01
Disk usage: 1.5GiB out of 9.6GiB
Memory usage: 219.4MiB out of 1.9GiB
Mounts: -

Create a user with `user add` command name it `venus-user`. Confirm the user is created using `user get` and make the user active:

./sophon-auth user add venus-user
Add user success: 0a5c6366-f84c-4e1f-81cb-35a53e452b78, next can add miner for this user
$ ./sophon-auth user get venus-user
name: venus-user
state enabled // 2: disable, 1: enable
comment:
createTime: Fri, 24 May 2024 21:15:28 AEST
updateTime: Fri, 24 May 2024 21:15:28 AEST
$ ./sophon-auth user active venus-user
active user success

Then generate a token via `token gen`.

./sophon-auth token gen — perm admin venus-user
generate token success: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYW1lIjoidmVudXMtdXNlciIsInBlcm0iOiJhZG1pbiIsImV4dCI6IiJ9.Xys05DPL499lQSsAdnPhDkPF0uOaWZE6bHDxu7a2T_4

Save the user token for later use.

Install Venus Daemon

Go back to `venus-three` VM’s shell.

First, install dependencies needed for venus:

sudo apt install mesa-opencl-icd ocl-icd-opencl-dev gcc git bzr jq pkg-config curl clang build-essential hwloc libhwloc-dev wget -y && sudo apt upgrade -y

Then install venus daemon by cloning github and build it from source just like installing `venus-auth`:

git clone https://github.com/filecoin-project/venus.git
cd venus
git checkout v1.15.0
make deps
make

To confirm installation, go into the venus directory and check for venus version.

$ ./venus version
{
"Version": "1.15.0+git.aabda75e"
}

Start Venus with Imported Snapshot

This step is very important as initially Venus will need to import the snapshot to catch up with the block information, otherwise it will take a long time to sync.

Things needed to start Venus initially:

IP address and port of `venus-one` VM’s auth service,
the `sophon-auth` token for venus-user created earlier
the path of the earlier snapshot you downloaded on this VM.

Use the following long command to start Venus daemon with logs sent to `venus.log`.

nohup ./venus/venus daemon \
 - network=calibrationnet \
 - auth-url=http://192.168.65.4:8989 \
 - auth-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYW1lIjoidmVudXMtdXNlciIsInBlcm0iOiJhZG1pbiIsImV4dCI6IiJ9.Xys05DPL499lQSsAdnPhDkPF0uOaWZE6bHDxu7a2T_4 \
 - import-snapshot /home/ubuntu/snapshots/calibnet/forest_snapshot_calibnet_2024–05–29_height_1654931.forest.car \
> venus.log 2>&1 &

Following `venus.log` the following should be observed:

$ tail -f venus.log
2024–05–29T20:40:17.611+1000 INFO badger v2@v2.2007.4/levels.go:183 All 0 tables opened in 0s
2024–05–29T20:40:17.623+1000 INFO chain.store chain/store.go:174 check point value: { bafy2bzacecyaggy24wol5ruvs6qm73gjibs2l2iyhcqmvi7r7a4ph7zx3yqd4 }
2024–05–29T20:40:17.624+1000 INFO chain.store chain/store.go:592 SetHead { bafy2bzacecyaggy24wol5ruvs6qm73gjibs2l2iyhcqmvi7r7a4ph7zx3yqd4 } 0
2024–05–29T20:40:17.725+1000 INFO chain.store chain/store.go:174 check point value: { }
176.57 MiB / 5.82 GiB [==> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ] 2.96% 67.78 MiB/s 01m24s2024–05–29T20:40:20.412+1000 INFO bufbs v2@v2.2007.4/db.go:1027 Storing value log head: {Fid:0 Len:31 Offset:206667252}
325.60 MiB / 5.82 GiB [====> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ] 5.46% 60.07 MiB/s 01m28s2024–05–29T20:40:23.259+1000 INFO bufbs v2@v2.2007.4/db.go:1027 Storing value log head: {Fid:0 Len:31 Offset:382637425}
436.64 MiB / 5.82 GiB [======> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ] 7.33% 54.38 MiB/s 01m38s2024–05–29T20:40:25.835+1000 INFO bufbs v2@v2.2007.4/db.go:1027 Storing value log head: {Fid:1 Len:32 Offset:72795797}
903.31 MiB / 5.82 GiB [============> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ] 15.16% 74.99 MiB/s 01m03s2024–05–29T20:40:29.868+1000 INFO bufbs v2@v2.2007.4/db.go:1027 Storing value log head: {Fid:1 Len:32 Offset:588758025}
1.41 GiB / 5.82 GiB [=====================> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -] 24.19% 84.42 MiB/s 00m52s2024–05–29T20:40:34.902+1000 INFO bufbs v2@v2.2007.4/db.go:1027 Storing value log head: {Fid:2 Len:32 Offset:346973246}
1.44 GiB / 5.82 GiB [=====================> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -] 24.75% 84.39 MiB/s 00m51s2024–05–29T20:40:35.202+1000 INFO bufbs v2@v2.2007.4/levels.go:1000 [Compactor: 0] Running compaction: {level:0 score:1 dropPrefixes:[]} for level: 0

After a while, when venus is syncing blocks, check if the blockheight is at the latest using `chain head`. It should be around 16,000+ with no blocks to catch up on:

$ ./venus/venus chain head
{
"Height": 1655122,
"ParentWeight": "30889013177",
"Cids": [
{
"/": "bafy2bzaceavzeo247nximwihtl7r2vuctkixwg476ey4mem2ws35xsfm2445i"
},
{
"/": "bafy2bzaceaxbh4tj4jh5xipow6an425q2mntlae6gzt75n6tpruwnhclmbsa4"
}
],
"Timestamp": "2024–05–29 20:54:00"
}

Also `sync status` should be `Done!`:

sync status
{sender:12D3KooWKaD4rUzxNWVty62M7xJFAZqMizbjWBvwr4HZYxj52bJj height=1655121 head={ bafy2bzacedaouyh3uq5ngpudtsqreirz52f4wqtp33ts7kwry6an7cmamkigu bafy2bzaceauo4fbxslg2nts375q6ge5sn6p5afrn7uwhx62rrytai624otjxw bafy2bzaceccqcjx7ya6ycav6gyazjkgdww4plj762plwiyfybcfvrfqgkejic }}
Done!

If seeing otherwise like the below case, then that means either the snapshot is nor important properly or the process has crashed and you are syncing from block 0.

$ ./venus/venus sync status
Syncing:
SyncTarget: 1
Base: 0 { bafy2bzacecyaggy24wol5ruvs6qm73gjibs2l2iyhcqmvi7r7a4ph7zx3yqd4 }
Target: 1655049 { bafy2bzaceak7ckzk65eetv7rc2umovpinfejq25znpkkmh663jjbn433vxh3e bafy2bzaceaqsjs2z5mupoih7oejmnjifj2es5q3xpyosjr7vsqctu7h3ohdy2 bafy2bzacecayfsl2le224njbow57wiofnouizohsyuwswd7h7nit543vwouh4 }
Current: 0 { bafy2bzacecyaggy24wol5ruvs6qm73gjibs2l2iyhcqmvi7r7a4ph7zx3yqd4 }
HeightDiff: 1655049
Status: message sync
Err: <nil>
Waiting:
SyncTarget: 2
Base: 0 { bafy2bzacecyaggy24wol5ruvs6qm73gjibs2l2iyhcqmvi7r7a4ph7zx3yqd4 }
Target: 1655050 { bafy2bzaceayjupb52krrpgxgwob3w2fp7eprulhfh7n5sbf3viuy2vrox6yhk bafy2bzaceafqziy2rs6vvg73yjavtqcrelhsc3r5owgktk3vqtojp57rtb7vs }
Current: 0 { bafy2bzacecyaggy24wol5ruvs6qm73gjibs2l2iyhcqmvi7r7a4ph7zx3yqd4 }
HeightDiff: 1655050
Status: idle
Err: <nil>
#check sync height
$ ./venus/venus chain head
{
"Height": 0,
"ParentWeight": "0",
"Cids": [
{
"/": "bafy2bzacecyaggy24wol5ruvs6qm73gjibs2l2iyhcqmvi7r7a4ph7zx3yqd4"
}
],
"Timestamp": "2022–11–02 05:13:00"
}
```
To start over again, find venus process `ps ef | grep venus` and kills the existing venus process gracefully via `kill -9 <PID>`. Then delete the venus user profile under user directory `rm -F ~/.venus`. Use the long command earlier to retry.
Note that if the node has not enough RAM (e.g. less than 8GB) or CPU (e.g. 4 cores) it will likely crash during the import process.
Venus APIs ports are specified in the venus config file `$cat ~/.venus/config.toml`. Port 1234 is default so avoid using this for other services.
[API]
ListenAddress = "/ip4/127.0.0.1/tcp/1234/http"
…

This can be changed via command line without restarting:

./venus daemon — api-listen /ip4/0.0.0.0/tcp/3453

Venus API is compatible with Lotus so use a combination of `.venus daemon — -help` and Lotus reference doc to navigate it.

Running the node

Automatic Restarts

Automatic Start of Venus & Sophon Auth at startup is convenient for cases of unexpected shutdown of the node. One way is to use systemd:

Create a service file for Venus Daemon in `/etc/systemd/system` named venus.service.

sudo nano /etc/systemd/system/venus.service

Edit the file with following configurations.

[Unit]
Description=Venus Daemon
After=network.target
[Service]
User=<username>
ExecStart=nohup ~/venus/venus daemon - network=calibrationnet - auth-url=<sophon-auth-ip>:<port> - auth-token=<sophon-auth-api-token> > venus.log 2>&1 &
Restart=on-failure
Environment=VENUS_PATH=/home/<username>/.venus
LimitNOFILE=4096
[Install]
WantedBy=multi-user.target

Replace <username>, <sophon-auth-ip>:<port> & <sophon-auth-api-token> with actual values.

Reload Systemd and Enable the Service

$sudo systemctl daemon-reload
$sudo systemctl enable venus
$sudo systemctl start venus

Check service status by `sudo systemctl status venus`.

Static Network Configuration

Multipass will assign dynamic IP to nodes under the host, so to ensure sophon-auth’s IP does not change after restart. A bridged network on the host network will need to be created.

For ubuntu host, follow guide here.

There is currently a bug on mac, so multipass users cannot create a virtual interface for a bridged network. I am still looking for a solution for this by the time of authoring this tutorial

Logging and debugging

Most of the logging will be kept in venus.log that were configured earlier. While the process is running, follow logs via `$ tail -f venus.log` to inspect the health of the node.

Conclusion

This guide provides a way to setup and configure multiple VMs with appropriate roles to run Filecoin Venus node. The important steps such as importing calibrationnet (testnet) snapshot to expedite blockchain synchronization, manage network configuration and automatic start of the service.

Running a Venus node can help developers explore ways to interact with Filecoin chain information for developments, testing or operations of decentralized storage nodes.

Here are some useful links::

Sophon Documentation

Venus GitHub Repository

Multipass Documentation

Filecoin Community Slack (#fil-venus & #fil-venus-announcements)

Filecoin Discord Channel

Feel free to reach out to the community forums or refer to the official documentation for more detailed information and support.