App Install

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 08 Oct 2026 08:11:50 UTC

Duration

300s

Peer count

1

Behaviours

large (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_large_37734232741_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.151687999

s

Install app timing

The time taken to install an app.

mean

0.065075

s

SD

0.005639

s

p50

0.063775

s

<

p95

0.078018

s

<

p99

0.082275

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'large::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.00008911s

Not enough time data to show a trend.

WASM call duration: 'large::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004224

s

SD

0.001386

s

p50

0.003807

s

<

p95

0.007418

s

<

p99

0.007653

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000111

s

SD

7.7e-05

s

p50

8.6e-05

s

<

p95

0.000347

s

<

p99

0.000411

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

7.16

s

<

mean

17.16

s

SD

8.17

s

<

p95

27.16

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 20.000177882s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001635

s

SD

0.00182

s

p50

0.001671

s

<

p95

0.005439

s

≤

p99

0.005439

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001882

s

SD

0.001801

s

p50

0.001886

s

<

p95

0.005652

s

≤

p99

0.005652

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.006913

s

SD

0.007892

s

p50

0.006841

s

<

p95

0.023496

s

≤

p99

0.023496

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002607

s

SD

0.002198

s

p50

0.002598

s

<

p95

0.00719

s

≤

p99

0.00719

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001242

s

SD

0.000431

s

p50

0.001111

s

<

p95

0.00133

s

≤

p99

0.00133

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

7.9e-05

s

SD

1.6e-05

s

p50

7.9e-05

s

<

p95

0.000112

s

≤

p99

0.000112

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001737

s

SD

0.001938

s

p50

0.002255

s

<

p95

0.00704

s

≤

p99

0.00704

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000416

s

SD

3.2e-05

s

p50

0.000418

s

<

p95

0.000485

s

≤

p99

0.000485

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

66.41

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

156.27

KiB/s

<

mean

447.19

KiB/s

SD

396.78

KiB/s

<

p95

1008.20

KiB/s

Not enough time data to show a trend.

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

23.84

KiB/s

<

mean

44.57

KiB/s

SD

20.91

KiB/s

<

p95

73.20

KiB/s

Not enough time data to show a trend.

Total bytes received

Total bytes received on primary network interface

count

13.10

MiB

mean

447.19

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.31

MiB

mean

44.57

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.4%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

81.44

%

<

mean

86.2

%

SD

5.8

%

<

p95

94.38

%

Not enough time data to show a trend.

System

19.80

%

CPU percentiles

CPU usage percentiles

p50

82.80

%

p95

94.38

%

p99

94.38

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

300.00

s

Memory used percentage

Percentage of memory used

p5

10.03

%

<

mean

11.66

%

SD

1.35

%

<

p95

13.78

%

Not enough time data to show a trend.

Memory available percentage

Percentage of available memory

p5

86.22

%

<

mean

88.34

%

SD

1.35

%

<

p95

89.97

%

Not enough time data to show a trend.

Max host memory used

Maximum memory usage percentage across all hosts

max

11.66

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-0.73

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1.71

MiB/s

Disk write throughput

Disk write throughput in MiB/s

14.84

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

7.76

5 min

6.38

15 min

3.67

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

42.52

%

<

mean

53.3225

%

SD

7.4301

%

<

p95

62.26

%

Not enough time data to show a trend.

60 second average

51.8900

%

300 second average

37.7800

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

18.65

%

<

mean

26.12

%

SD

8.36

%

<

p95

37.78

%

Not enough time data to show a trend.

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.24

KiB

<

mean

376.15

KiB

SD

482.41

KiB

<

p95

1.35

MiB

Not enough time data to show a trend.

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

60.83

threads

SD

81.94

threads

<

p95

236

threads

Not enough time data to show a trend.

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

135.33

file descriptors

SD

174.8

file descriptors

<

p95

508

file descriptors

Not enough time data to show a trend.

App Install

↑ Back to index

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 08 Oct 2026 08:13:34 UTC

Duration

300s

Peer count

1

Behaviours

minimal (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_minimal_37734232741_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.323416111

s

Install app timing

The time taken to install an app.

mean

0.039983

s

SD

0.00225

s

p50

0.039812

s

<

p95

0.043803

s

<

p99

0.04662

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'callback::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000117601s

Not enough time data to show a trend.

WASM usage: 'callback_integrity::genesis_self_check_2'

Metered usage count of the WASM ribosome for this zome function.

total

114480

over 10.000117601s

mean rate

11447.87

/s

std

0

/s

p5

11447.87

/s

p95

11447.87

/s

peak

11447.87

/s

Not enough time data to show a trend.

WASM call duration: 'callback::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000753

s

SD

0.000175

s

p50

0.000703

s

<

p95

0.000872

s

<

p99

0.001547

s

WASM call duration: 'callback_integrity::genesis_self_check_2'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000719

s

SD

0

s

p50

0.000719

s

≤

p95

0.000719

s

≤

p99

0.000719

s

Post-commit duration

Time spent executing post-commit workflows.

mean

6.7e-05

s

SD

9e-06

s

p50

6.6e-05

s

<

p95

8.5e-05

s

<

p99

8.8e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

7.51

s

<

mean

12.51

s

SD

5

s

<

p95

17.51

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 10.00002882s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001197

s

SD

6.9e-05

s

p50

0.001304

s

≤

p95

0.001304

s

≤

p99

0.001304

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001047

s

SD

1e-06

s

p50

0.001048

s

≤

p95

0.001048

s

≤

p99

0.001048

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.003633

s

SD

0.00011

s

p50

0.003808

s

≤

p95

0.003808

s

≤

p99

0.003808

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002076

s

SD

0.00014

s

p50

0.002295

s

≤

p95

0.002295

s

≤

p99

0.002295

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001298

s

SD

0.000182

s

p50

0.001583

s

≤

p95

0.001583

s

≤

p99

0.001583

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

5.2e-05

s

SD

0

s

p50

5.3e-05

s

≤

p95

5.3e-05

s

≤

p99

5.3e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000942

s

SD

0.000202

s

p50

0.001213

s

<

p95

0.001246

s

≤

p99

0.001246

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000267

s

SD

1e-06

s

p50

0.000267

s

≤

p95

0.000267

s

≤

p99

0.000267

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

49.40

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

629.60

KiB/s

<

mean

635.22

KiB/s

SD

5.61

KiB/s

<

p95

640.83

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

75.29

KiB/s

<

mean

77.44

KiB/s

SD

2.15

KiB/s

<

p95

79.59

KiB/s

Total bytes received

Total bytes received on primary network interface

count

12.41

MiB

mean

635.22

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.51

MiB

mean

77.44

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 5.66 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

59.55

%

<

mean

65.78

%

SD

6.23

%

<

p95

72.01

%

System

16.38

%

CPU percentiles

CPU usage percentiles

p50

72.01

%

p95

72.01

%

p99

72.01

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

9.29

%

<

mean

10.58

%

SD

1.6

%

<

p95

12.84

%

Memory available percentage

Percentage of available memory

p5

87.16

%

<

mean

89.42

%

SD

1.6

%

<

p95

90.71

%

Max host memory used

Maximum memory usage percentage across all hosts

max

10.58

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

5.66

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

7.35

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.35

5 min

6.00

15 min

3.80

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

19.77

%

<

mean

22.8367

%

SD

2.1788

%

<

p95

24.63

%

60 second average

30.8100

%

300 second average

34.7467

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

28.32

%

<

mean

37.14

%

SD

8.82

%

<

p95

45.96

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

20.82

KiB

<

mean

736.94

KiB

SD

704.34

KiB

<

p95

1.81

MiB

Holochain process threads

Number of threads in Holochain process

p5

8

threads

<

mean

132.25

threads

SD

145.05

threads

<

p95

363

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

285.25

file descriptors

SD

313.58

file descriptors

<

p95

781

file descriptors

Conductor Startup

↑ Back to index

This scenario measures Holochain conductor startup time when many cells are installed. The agent starts an empty conductor, installs a configurable number of apps without enabling them (disabled cells), restarts the conductor, and then enables cells one by one — timing each enable call and periodically restarting the conductor to measure startup time as the share of enabled cells grows.

Started

Thu, 08 Oct 2026 08:02:00 UTC

Duration

900s

Peer count

30

Peer count at end

17

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

conductor_startup_37734232741_1

Initial conductor startup timing

Startup time with an empty conductor before any cells are installed.

mean

2.791135

s

SD

0.901343

s

p50

2.698613

s

<

p95

4.188078

s

<

p99

4.510328

s

Disabled-cell conductor startup timing

Startup time after all configured cells are installed but remain disabled.

mean

2.180516

s

SD

0.62495

s

p50

2.41232

s

<

p95

3.391929

s

≤

p99

3.391929

s

Progressive conductor startup timing

Startup time sampled periodically as more installed cells are enabled.

mean

11.759227

s

SD

6.342291

s

p50

10.481699

s

<

p95

23.698758

s

<

p99

26.134601

s

Cell enable timing

Time to enable one previously disabled cell via the admin interface.

mean

0.142249

s

SD

0.067033

s

p50

0.139205

s

<

p95

0.233019

s

<

p99

0.312802

s

Installed cells

Number of cells installed on the conductor during setup.

200

cells

Maximum enabled cells

Highest percentage of installed cells enabled during the run.

100

%

Holochain MetricsClick to toggle visibility.

WASM usage: 'callback_integrity::genesis_self_check_2'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.329935615s

Not enough time data to show a trend.

WASM call duration: 'callback_integrity::genesis_self_check_2'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001197

s

SD

0.000578

s

p50

0.000838

s

<

p95

0.00238

s

<

p99

0.002463

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

0.93

s

<

mean

98.33

s

SD

90.6

s

<

p95

256.53

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 403.009448362s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

4.778862

s

SD

5.885012

s

p50

1.004125

s

<

p95

18.088097

s

<

p99

22.750786

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001737

s

SD

0.00156

s

p50

0.001485

s

<

p95

0.003563

s

<

p99

0.005224

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.007557

s

SD

0.013011

s

p50

0.001974

s

<

p95

0.004059

s

<

p99

0.00686

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.030512

s

SD

0.185645

s

p50

0.002242

s

<

p95

0.007223

s

<

p99

1.162029

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.007531

s

SD

0.059379

s

p50

0.002248

s

<

p95

0.004955

s

<

p99

0.007946

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001357

s

SD

0.001856

s

p50

0.000866

s

<

p95

0.003815

s

<

p99

0.005773

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000121

s

SD

7.4e-05

s

p50

9.8e-05

s

<

p95

0.000275

s

<

p99

0.000322

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.010681

s

SD

0.061885

s

p50

0.001248

s

<

p95

0.002985

s

<

p99

0.246336

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000229

s

SD

0.000151

s

p50

0.00021

s

<

p95

0.000582

s

<

p99

0.000772

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.085873

s

SD

0.102715

s

p50

0.053335

s

<

p95

0.243592

s

≤

p99

0.243592

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

19.33

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

91

B/s

<

mean

134.02

KiB/s

SD

383.10

KiB/s

<

p95

269.70

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

48

B/s

<

mean

53.73

KiB/s

SD

49.20

KiB/s

<

p95

123.21

KiB/s

Total bytes received

Total bytes received on primary network interface

count

556.69

MiB

mean

1.27

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

201.80

MiB

mean

469.65

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (75.0% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 4% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

2.23

%

<

mean

30.94

%

SD

25.94

%

<

p95

72.61

%

System

11.61

%

CPU percentiles

CPU usage percentiles

p50

31.45

%

p95

72.61

%

p99

78.09

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

64.29

s

Memory used percentage

Percentage of memory used

p5

8.39

%

<

mean

16.43

%

SD

12.58

%

<

p95

27.23

%

Memory available percentage

Percentage of available memory

p5

72.77

%

<

mean

83.57

%

SD

12.58

%

<

p95

91.61

%

Max host memory used

Maximum memory usage percentage across all hosts

max

82.92

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

75.00

%

Memory growth rate

Rate of memory growth over time

growth

0.40

MiB/s

Disk read throughput

Disk read throughput in MiB/s

3.20

MiB/s

Disk write throughput

Disk write throughput in MiB/s

116.93

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/10

hosts

Mount Point /etc/hosts

0/10

hosts

Mount Point /etc/nomad.d/client-meta.json

0/10

hosts

Mount Point /etc/resolv.conf

0/10

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.07

5 min

1.73

15 min

0.72

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

4.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

9.7187

%

SD

14.0645

%

<

p95

41.74

%

60 second average

6.7814

%

300 second average

2.6707

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0622

%

SD

0.4467

%

<

p95

0.18

%

60 second average

0.0640

%

300 second average

0.0363

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0602

%

SD

0.4349

%

<

p95

0.15

%

60 second average

0.0615

%

300 second average

0.0351

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

23.2061

%

SD

24.1554

%

<

p95

58.94

%

60 second average

21.8817

%

300 second average

13.2101

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

21.2444

%

SD

24.1284

%

<

p95

58.28

%

60 second average

20.4572

%

300 second average

12.6388

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

16.27

%

SD

23.77

%

<

p95

67.18

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

58.87

KiB

<

mean

287.90

KiB

SD

150.21

KiB

<

p95

553.65

KiB

Holochain process threads

Number of threads in Holochain process

p5

7

threads

<

mean

376.68

threads

SD

457.11

threads

<

p95

1406

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

23

file descriptors

<

mean

865.35

file descriptors

SD

999.72

file descriptors

<

p95

3136

file descriptors

DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.

Started

Thu, 08 Oct 2026 08:15:24 UTC

Duration

1800s

Peer count

30

Peer count at end

27

Behaviours

record_lag (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

dht_sync_lag_37734232741_1

Create rate

The average number of created records per agent.

Mean

mean

20.05

/s

Max

Highest per-partition mean rate

max

31.44

/s

Min

Lowest per-partition mean rate

min

8.95

/s

Sync lag timing

The average time between when a record was created and when it is first seen by an agent.

Mean

mean

651.492703

s

SD

404.250992

s

Max

Highest per-partition mean latency

max mean

1190.50761

s

Min

Lowest per-partition mean latency

min mean

49.613578

s

Sync lag rate

The average number of created records discovered per agent.

Mean

mean

45.86

/s

Max

Highest per-partition mean rate

max

181.44

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully received by all record_lag agents.

0.0041

Sent count

The total number of entries sent (created) by write agents, by agent.

Total

82502

Agents affected

22 / 22

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3750

Max

Highest count in any single partition

5669

Min

Lowest count in any single partition

1623

Received count

The total number of entries received by record_lag agents, by agent.

Total

1682

Agents affected

4 / 5

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

336

Max

Highest count in any single partition

1645

Min

Lowest count in any single partition

0

Error count

The number of errors encountered during the scenario.

5

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000288

s

SD

0.000741

s

p50

0.000635

s

<

p95

0.002527

s

<

p99

0.002652

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

7

over 1720.916395406s

mean rate

2.87

/s

std

3.8

/s

p5

0

/s

p95

12.07

/s

peak

12.12

/s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

277515840

over 1787.970895416s

mean rate

3.88457523011e+09

/s

std

8.38704809767e+10

/s

p5

3.23031315e+06

/s

p95

1.35142518489e+09

/s

peak

3.46527935918e+12

/s

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

1304320339

over 1782.224643579s

mean rate

1.07457712166e+10

/s

std

1.57162062898e+10

/s

p5

8.4902174099e+08

/s

p95

5.1065860234e+10

/s

peak

6.10696349871e+10

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

51719400

over 1788.03409281s

mean rate

5.6179247246e+08

/s

std

1.01949192832e+10

/s

p5

475098

/s

p95

2.6904957095e+08

/s

peak

4.67390430554e+11

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.014189

s

SD

0.018772

s

p50

0.010937

s

<

p95

0.059088

s

<

p99

0.092855

s

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.01637

s

SD

0.024113

s

p50

0.012118

s

<

p95

0.082373

s

<

p99

0.123492

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.013315

s

SD

0.018595

s

p50

0.009977

s

<

p95

0.057775

s

<

p99

0.091232

s

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.015724

s

SD

0.024098

s

p50

0.011407

s

<

p95

0.081811

s

<

p99

0.122921

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001934

s

SD

0.001723

s

p50

0.001698

s

<

p95

0.006872

s

<

p99

0.008237

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.002369

s

SD

0.005798

s

p50

0.001085

s

<

p95

0.015101

s

<

p99

0.027062

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.002259

s

SD

0.005542

s

p50

0.001068

s

<

p95

0.014595

s

<

p99

0.026403

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000274

s

SD

0.000858

s

p50

0.000825

s

<

p95

0.002916

s

<

p99

0.003001

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001236

s

SD

0.001318

s

p50

0.001379

s

<

p95

0.004479

s

<

p99

0.006659

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.003101

s

SD

0.002958

s

p50

0.002652

s

<

p95

0.010671

s

<

p99

0.015041

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000319

s

SD

0.000596

s

p50

0.000201

s

<

p95

0.001257

s

<

p99

0.003609

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

87.97

s

<

mean

887.88

s

SD

516.71

s

<

p95

1697.88

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

8

over 1788.034103545s

mean rate

681476

/s

std

1.253844568e+07

/s

p5

0

/s

p95

1.07975376e+06

/s

peak

5.8059546117e+08

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

87.285526

s

SD

70.249259

s

p50

0.087193

s

<

p95

178.984274

s

<

p99

335.038441

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.231284

s

SD

0.702116

s

p50

0.266462

s

<

p95

1.869214

s

<

p99

3.91359

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.424895

s

SD

0.519155

s

p50

0.700372

s

<

p95

1.521266

s

<

p99

1.648261

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.356361

s

SD

0.214235

s

p50

0.26726

s

<

p95

0.682065

s

<

p99

0.883886

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

27.069148

s

SD

58.223999

s

p50

31.11631

s

<

p95

156.363263

s

<

p99

282.773223

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

35.333959

s

SD

62.295469

s

p50

0.041698

s

<

p95

179.056185

s

<

p99

294.479892

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001711

s

SD

0.001145

s

p50

0.001603

s

<

p95

0.00426

s

<

p99

0.005767

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.006003

s

SD

0.011792

s

p50

0.001686

s

<

p95

0.039946

s

<

p99

0.054039

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.001876

s

SD

0.005002

s

p50

0.000815

s

<

p95

0.013174

s

<

p99

0.024293

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.219586

s

SD

28.930522

s

p50

0.338951

s

<

p95

60.006056

s

<

p99

60.048082

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.172087

s

SD

13.372624

s

p50

0.147151

s

<

p95

60.000157

s

<

p99

60.002812

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

68.78

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

3.36

KiB/s

<

mean

795.69

KiB/s

SD

541.50

KiB/s

<

p95

1.45

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

3.32

KiB/s

<

mean

714.43

KiB/s

SD

466.81

KiB/s

<

p95

1.45

MiB/s

Total bytes received

Total bytes received on primary network interface

count

34.56

GiB

mean

19.77

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

31.03

GiB

mean

17.75

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (77.1% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 70% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

41.92

%

<

mean

83.16

%

SD

16.93

%

<

p95

97.6

%

System

14.38

%

CPU percentiles

CPU usage percentiles

p50

90.82

%

p95

97.60

%

p99

98.00

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

24

hosts

mean time

1366.34

s

Memory used percentage

Percentage of memory used

p5

8.07

%

<

mean

22.39

%

SD

15.79

%

<

p95

44.19

%

Memory available percentage

Percentage of available memory

p5

55.88

%

<

mean

77.61

%

SD

15.79

%

<

p95

91.94

%

Max host memory used

Maximum memory usage percentage across all hosts

max

87.94

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

77.12

%

Memory growth rate

Rate of memory growth over time

growth

0.17

MiB/s

Disk read throughput

Disk read throughput in MiB/s

3.65

MiB/s

Disk write throughput

Disk write throughput in MiB/s

144.13

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/9

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.85

5 min

6.07

15 min

4.36

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

70.37

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

1.09

%

<

mean

44.1627

%

SD

22.96

%

<

p95

73.71

%

60 second average

42.9527

%

300 second average

37.6449

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0032

%

SD

0.0519

%

≤

p95

0

%

60 second average

0.0026

%

300 second average

0.0012

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0019

%

SD

0.0415

%

≤

p95

0

%

60 second average

0.0014

%

300 second average

0.0006

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

8.7432

%

SD

16.2719

%

<

p95

48.26

%

60 second average

8.5183

%

300 second average

7.8386

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.8282

%

SD

4.1983

%

<

p95

12.27

%

60 second average

1.7923

%

300 second average

1.8997

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

35.74

%

SD

35.98

%

<

p95

95.05

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

237.53

KiB

<

mean

401.60

KiB

SD

141.55

KiB

<

p95

605.92

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

31.9

threads

SD

16.64

threads

<

p95

59

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

69.4

file descriptors

SD

34

file descriptors

<

p95

109

file descriptors

First Call

↑ Back to index

This scenario installs a simple app which implements the init callback, and measures the time elapsed between start of install and calling init. It will uninstall the app it installed so that it can re-install it on the next iteration. This is required to re-run the initialisation callback.

Started

Thu, 08 Oct 2026 07:54:51 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

first_call_37734232741_1

Zome call timing

The time elapsed between start of install and calling init.

mean

0.005878

s

SD

0.001162

s

p50

0.005792

s

<

p95

0.008308

s

<

p99

0.00975

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 0s

Not enough time data to show a trend.

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 0s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 0s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.002501

s

SD

0

s

p50

0.002501

s

≤

p95

0.002501

s

≤

p99

0.002501

s

Not enough time data to show a trend.

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002029

s

SD

0.000348

s

p50

0.001944

s

<

p95

0.002608

s

<

p99

0.003042

s

Not enough time data to show a trend.

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000829

s

SD

3.1e-05

s

p50

0.000828

s

<

p95

0.000888

s

<

p99

0.000901

s

Not enough time data to show a trend.

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000622

s

SD

0

s

p50

0.000622

s

≤

p95

0.000622

s

≤

p99

0.000622

s

Not enough time data to show a trend.

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000237

s

SD

6e-06

s

p50

0.000244

s

≤

p95

0.000244

s

≤

p99

0.000244

s

Not enough time data to show a trend.

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00104

s

SD

0

s

p50

0.00104

s

≤

p95

0.00104

s

≤

p99

0.00104

s

Not enough time data to show a trend.

Post-commit duration

Time spent executing post-commit workflows.

mean

4.7e-05

s

SD

1.1e-05

s

p50

4.4e-05

s

<

p95

6.7e-05

s

<

p99

8.1e-05

s

Not enough time data to show a trend.

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

8.11

s

≤

mean

8.11

s

SD

0

s

≤

p95

8.11

s

Not enough time data to show a trend.

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 0s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Not enough time data to show a trend.

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Not enough time data to show a trend.

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000821

s

SD

0

s

p50

0.000821

s

≤

p95

0.000821

s

≤

p99

0.000821

s

Not enough time data to show a trend.

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Not enough time data to show a trend.

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.000657

s

SD

0

s

p50

0.000657

s

≤

p95

0.000657

s

≤

p99

0.000657

s

Not enough time data to show a trend.

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.002073

s

SD

0

s

p50

0.002073

s

≤

p95

0.002073

s

≤

p99

0.002073

s

Not enough time data to show a trend.

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.001584

s

SD

0

s

p50

0.001584

s

≤

p95

0.001584

s

≤

p99

0.001584

s

Not enough time data to show a trend.

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001334

s

SD

0

s

p50

0.001334

s

≤

p95

0.001334

s

≤

p99

0.001334

s

Not enough time data to show a trend.

DHT DB connection use time

Time spent holding DHT database connections.

mean

2.6e-05

s

SD

0

s

p50

2.6e-05

s

≤

p95

2.6e-05

s

≤

p99

2.6e-05

s

Not enough time data to show a trend.

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000567

s

SD

0.000179

s

p50

0.000846

s

≤

p95

0.000846

s

≤

p99

0.000846

s

Not enough time data to show a trend.

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000187

s

SD

0

s

p50

0.000187

s

≤

p95

0.000187

s

≤

p99

0.000187

s

Not enough time data to show a trend.

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

24.37

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

125.08

KiB/s

≤

mean

125.08

KiB/s

SD

0

B/s

≤

p95

125.08

KiB/s

Not enough time data to show a trend.

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

37.71

KiB/s

≤

mean

37.71

KiB/s

SD

0

B/s

≤

p95

37.71

KiB/s

Not enough time data to show a trend.

Total bytes received

Total bytes received on primary network interface

count

1.22

MiB

mean

125.08

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

377.08

KiB

mean

37.71

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 11.80 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

30.28

%

≤

mean

30.28

%

SD

0

%

≤

p95

30.28

%

Not enough time data to show a trend.

System

5.91

%

CPU percentiles

CPU usage percentiles

p50

30.28

%

p95

30.28

%

p99

30.28

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

4.97

%

<

mean

7.75

%

SD

2.78

%

<

p95

10.53

%

Not enough time data to show a trend.

Memory available percentage

Percentage of available memory

p5

89.47

%

<

mean

92.25

%

SD

2.78

%

<

p95

95.03

%

Not enough time data to show a trend.

Max host memory used

Maximum memory usage percentage across all hosts

max

7.75

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

11.80

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

8.44

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.25

5 min

0.12

15 min

0.17

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

1.03

%

<

mean

1.11

%

SD

0.08

%

<

p95

1.19

%

Not enough time data to show a trend.

60 second average

0.2950

%

300 second average

0.0600

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

29.81

%

≤

mean

29.81

%

SD

0

%

≤

p95

29.81

%

Not enough time data to show a trend.

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

229.25

KiB

<

mean

1.07

MiB

SD

866.71

KiB

<

p95

1.92

MiB

Not enough time data to show a trend.

Holochain process threads

Number of threads in Holochain process

p5

25

threads

<

mean

189.5

threads

SD

164.5

threads

<

p95

354

threads

Not enough time data to show a trend.

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

62

file descriptors

<

mean

411.5

file descriptors

SD

349.5

file descriptors

<

p95

761

file descriptors

Not enough time data to show a trend.

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 08 Oct 2026 10:52:52 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

full_arc_create_validated_zero_arc_read--high_zero_37734232741_1

Create rate

The number of timed entries created by the full-arc node(s) per second.

Mean

mean

38.76

/s

Max

Highest per-partition mean rate

max

48.8

/s

Min

Lowest per-partition mean rate

min

28.72

/s

Fetch lag timing

For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.

Mean

mean

88.199574

s

SD

7.980889

s

Max

Highest per-partition mean latency

max mean

1182.54335

s

Min

Lowest per-partition mean latency

min mean

4.298184

s

Fetch rate

The number of entries per second fetched by zero-arc nodes.

Mean

mean

10.12

/s

Max

Highest per-partition mean rate

max

172

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc nodes.

0.029

Entry created count

Total number of entries created by full-arc nodes, by agent.

Total

7203

Agents affected

2 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3602

Max

Highest count in any single partition

4549

Min

Lowest count in any single partition

2654

Received count

Total number of entries fetched by zero-arc nodes, by agent.

Total

5635

Agents affected

17 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

209

Max

Highest count in any single partition

1191

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

27

<

mean

27.62

SD

2.51

<

p95

28

zero-arc

p5

0

<

mean

5.86

SD

5.66

<

p95

16

Retrieval errors

Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.

Total

364

Agents affected

27 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

13

Max

Highest count in any single partition

14

Min

Lowest count in any single partition

3

Error count

The number of errors accumulated across all nodes.

364

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.06038

s

SD

0.063547

s

p50

0.060395

s

<

p95

0.193878

s

<

p99

0.349688

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

8

over 800.321519091s

mean rate

356.39

/s

std

1113.39

/s

p5

0

/s

p95

3580.06

/s

peak

6616.58

/s

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.006517

s

SD

0.002379

s

p50

0.005133

s

<

p95

0.010379

s

<

p99

0.010497

s

Zome call duration: 'timed_and_validated::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

0.033574

s

SD

0.112116

s

p50

0.020591

s

<

p95

0.46497

s

<

p99

0.509839

s

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005909

s

SD

0.002243

s

p50

0.004604

s

<

p95

0.009564

s

<

p99

0.009679

s

WASM call duration: 'timed_and_validated::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.332192

s

SD

0.695145

s

p50

0.356879

s

<

p95

2.059134

s

<

p99

3.007625

s

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001036

s

SD

0.000757

s

p50

0.000876

s

<

p95

0.003193

s

<

p99

0.003339

s

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.007406

s

SD

0.002479

s

p50

0.000954

s

<

p95

0.007587

s

<

p99

0.014275

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000466

s

SD

0.000187

s

p50

0.000369

s

<

p95

0.00078

s

<

p99

0.000797

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000467

s

SD

0.000196

s

p50

0.000367

s

<

p95

0.000783

s

<

p99

0.000794

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.072422

s

SD

0.219982

s

p50

0.090357

s

<

p95

0.510481

s

<

p99

1.151332

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.012696

s

SD

0.044604

s

p50

0.008668

s

<

p95

0.128577

s

<

p99

0.249629

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.017876

s

SD

0.009569

s

p50

0.014736

s

<

p95

0.03617

s

<

p99

0.03714

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001702

s

SD

0.001075

s

p50

0.00135

s

<

p95

0.004645

s

<

p99

0.004919

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.0001

s

SD

7.3e-05

s

p50

9.8e-05

s

<

p95

0.000266

s

<

p99

0.000434

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.98

s

<

mean

446.72

s

SD

256.61

s

<

p95

846.99

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 888.321998908s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

90.595313

s

SD

22.462855

s

p50

0

s

<

p95

33.198546

s

<

p99

95.044595

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

3.84658

s

SD

1.959397

s

p50

0.001433

s

<

p95

2.463026

s

<

p99

11.64892

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.027133

s

SD

0.007915

s

p50

0.001534

s

<

p95

0.017735

s

<

p99

0.056584

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.145201

s

SD

0.036808

s

p50

0.016918

s

<

p95

0.090636

s

<

p99

0.200808

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

1.054804

s

SD

5.065335

s

p50

0.002747

s

<

p95

0.639581

s

<

p99

38.840618

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

4.637166

s

SD

4.183644

s

p50

0.002279

s

<

p95

5.137763

s

<

p99

24.19638

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002213

s

SD

0.001837

s

p50

0.000114

s

<

p95

0.005193

s

<

p99

0.009604

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.011882

s

SD

0.005557

s

p50

0.001583

s

<

p95

0.010982

s

<

p99

0.032447

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000269

s

SD

7.7e-05

s

p50

0.000306

s

<

p95

0.00041

s

<

p99

0.00043

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.117075

s

SD

29.913839

s

p50

0.161262

s

<

p95

60.001565

s

<

p99

60.001839

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

2.326421

s

SD

17.184602

s

p50

0.900523

s

<

p95

60.000762

s

<

p99

60.001614

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

1.269943

s

SD

22.449222

s

p50

2.293438

s

<

p95

60.001364

s

≤

p99

60.001364

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

7.58

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

520

B/s

<

mean

60.14

KiB/s

SD

188.74

KiB/s

<

p95

269.23

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

437

B/s

<

mean

56.24

KiB/s

SD

265.94

KiB/s

<

p95

263.95

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.38

GiB

mean

1.59

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.29

GiB

mean

1.48

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.5%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (75.5% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 14% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.55

%

<

mean

14.49

%

SD

23.23

%

<

p95

77.34

%

System

6.91

%

CPU percentiles

CPU usage percentiles

p50

3.39

%

p95

77.34

%

p99

94.52

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

4

hosts

mean time

293.26

s

Memory used percentage

Percentage of memory used

p5

8.23

%

<

mean

19.69

%

SD

15.18

%

<

p95

38.53

%

Memory available percentage

Percentage of available memory

p5

61.48

%

<

mean

80.31

%

SD

15.18

%

<

p95

91.77

%

Max host memory used

Maximum memory usage percentage across all hosts

max

87.12

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

75.51

%

Memory growth rate

Rate of memory growth over time

growth

0.13

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.71

MiB/s

Disk write throughput

Disk write throughput in MiB/s

31.33

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/10

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.01

5 min

1.10

15 min

1.48

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

14.29

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

3.3394

%

SD

11.6956

%

<

p95

28.95

%

60 second average

3.5070

%

300 second average

5.0539

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.1857

%

SD

6.9645

%

<

p95

12.37

%

60 second average

2.0683

%

300 second average

1.7800

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.2306

%

SD

4.2802

%

<

p95

6.37

%

60 second average

1.1762

%

300 second average

1.1025

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.16

%

<

mean

7.93

%

SD

21.51

%

<

p95

77.7

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

343.80

KiB

<

mean

422.63

KiB

SD

62.33

KiB

<

p95

555.47

KiB

Holochain process threads

Number of threads in Holochain process

p5

20

threads

<

mean

27.19

threads

SD

6.16

threads

<

p95

40

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

53

file descriptors

<

mean

61.26

file descriptors

SD

8.67

file descriptors

<

p95

79

file descriptors

Local Signals

↑ Back to index

This scenario is for testing the speed and reliability of local signals. It uses a zome with a function that will emit 10,000 signals. The scenario measures how long it takes to send the signals and how many have been received by the time the zome call ends.

Started

Thu, 08 Oct 2026 07:56:37 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

local_signals_37734232741_1

Send timing

The time taken to emit a batch of 10,000 signals.

mean

1.178156

s

SD

0.890628

s

p50

1.127703

s

<

p95

3.440703

s

≤

p99

3.440703

s

Receive timing

The time taken to receive the completed batch, to the nearest 250ms.

mean

28.377919

s

SD

4.870076

s

p50

30.001236

s

<

p95

30.002344

s

≤

p99

30.002344

s

Success ratio

The proportion of signals successfully received in a batch.

min

0.755299985408783

<

mean

0.8232

SD

0.0363

<

max

0.8963000178337097

Holochain MetricsClick to toggle visibility.

WASM usage: 'signal::emit_10k_signals'

Metered usage count of the WASM ribosome for this zome function.

total

278239986

over 280.003894663s

mean rate

993698.45

/s

std

1.44381038e+06

/s

p5

0

/s

p95

3.09152909e+06

/s

peak

3.09153369e+06

/s

Zome call duration: 'signal::emit_10k_signals'

Duration of this zome call as measured by Holochain internally.

mean

0.134726

s

SD

0.025968

s

p50

0.112391

s

<

p95

0.169313

s

≤

p99

0.169313

s

WASM call duration: 'signal::emit_10k_signals'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.133728

s

SD

0.025895

s

p50

0.111371

s

<

p95

0.168327

s

≤

p99

0.168327

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

9e-06

s

SD

2e-06

s

p50

7e-06

s

<

p95

1.2e-05

s

≤

p99

1.2e-05

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

90000

over 280.003839791s

mean rate

321.42

/s

std

465.13

/s

p5

0

/s

p95

999.99

/s

peak

999.99

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

5e-05

s

SD

0

s

p50

5e-05

s

≤

p95

5e-05

s

≤

p99

5e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

18.11

s

<

mean

148.11

s

SD

83.67

s

<

p95

278.11

s

Dropped WebSocket signals

Count of signals dropped from the app WebSocket due to channel overload.

total

1699

over 280.003840471s

mean rate

6.07

/s

std

9.7

/s

p5

0

/s

p95

27.6

/s

peak

30.1

/s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.003849225s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.00085

s

SD

0

s

p50

0.00085

s

≤

p95

0.00085

s

≤

p99

0.00085

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.000426

s

SD

0

s

p50

0.000426

s

≤

p95

0.000426

s

≤

p99

0.000426

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.002864

s

SD

0.000272

s

p50

0.002759

s

<

p95

0.003136

s

≤

p99

0.003136

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002514

s

SD

0

s

p50

0.002514

s

≤

p95

0.002514

s

≤

p99

0.002514

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000165

s

SD

0

s

p50

0.000165

s

≤

p95

0.000165

s

≤

p99

0.000165

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000166

s

SD

3.7e-05

s

p50

0.000174

s

<

p95

0.000182

s

<

p99

0.000183

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000603

s

SD

0.000144

s

p50

0.000799

s

≤

p95

0.000799

s

≤

p99

0.000799

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000182

s

SD

0

s

p50

0.000182

s

≤

p95

0.000182

s

≤

p99

0.000182

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

8.21

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

162

B/s

<

mean

637.92

KiB/s

SD

1.88

MiB/s

<

p95

7.11

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

156

B/s

<

mean

22.19

KiB/s

SD

35.99

KiB/s

<

p95

129.25

KiB/s

Total bytes received

Total bytes received on primary network interface

count

180.66

MiB

mean

637.92

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

6.29

MiB

mean

22.19

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 3.65 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.52

%

<

mean

13.08

%

SD

11.82

%

<

p95

41.09

%

System

4.87

%

CPU percentiles

CPU usage percentiles

p50

8.46

%

p95

41.09

%

p99

48.47

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

4.64

%

<

mean

5.96

%

SD

1.29

%

<

p95

8.12

%

Memory available percentage

Percentage of available memory

p5

91.88

%

<

mean

94.04

%

SD

1.29

%

<

p95

95.36

%

Max host memory used

Maximum memory usage percentage across all hosts

max

5.96

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

3.65

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

3.51

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.23

5 min

0.56

15 min

0.32

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.18

%

<

mean

3.407

%

SD

4.9247

%

<

p95

14.34

%

60 second average

2.4463

%

300 second average

0.9193

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.32

%

<

mean

3.15

%

SD

4.1

%

<

p95

16.42

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

127.70

KiB

<

mean

202.71

KiB

SD

21.86

KiB

<

p95

214.84

KiB

Holochain process threads

Number of threads in Holochain process

p5

22

threads

<

mean

27.77

threads

SD

6.61

threads

<

p95

34

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

57

file descriptors

<

mean

64.97

file descriptors

SD

13.86

file descriptors

<

p95

78

file descriptors

Mixed Arc `get_agent_activity`

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

zero_write: A zero-arc conductor that creates entries on its source chain.
full_write: A full-arc conductor that creates entries on its source chain.
zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.

Started

Thu, 08 Oct 2026 11:08:21 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full_write (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_get_agent_activity_37734232741_1

Entry created count

The total number of entries created by write agents over the run, partitioned by agent and behaviour.

Total

105585

Agents affected

19 / 19

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

5557

Max

Highest count in any single partition

9390

Min

Lowest count in any single partition

2228

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

2565.5

Max

Highest chain head value observed for any single write agent.

2912

Write agents observed

2

Chain head delay timing

The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.

Mean

mean

0.001666

s

SD

0.027376

s

Max

Highest per-partition mean latency

max mean

0.005945

s

Min

Lowest per-partition mean latency

min mean

0.000242

s

Chain head delay rate

The rate at which readers detect new chain heads appearing on writers' chains.

Mean

mean

54.58

/s

Max

Highest per-partition mean rate

max

79.74

/s

Min

Lowest per-partition mean rate

min

26.43

/s

get_agent_activity_full zome call timing

The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.

Mean

mean

0.08537

s

SD

0.072592

s

Max

Highest per-partition mean latency

max mean

0.212167

s

Min

Lowest per-partition mean latency

min mean

0.01121

s

Retrieval errors

A running accumulation of errors encountered by read agents when calling get_agent_activity.

Total

19

Agents affected

6 / 9

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

2

Max

Highest count in any single partition

5

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors, partitioned by agent behaviour.

behaviour: full_write

p5

27

<

mean

27.25

SD

1.42

<

p95

28

behaviour: zero_read

p5

0

<

mean

13.9

SD

5.03

<

p95

19

behaviour: zero_write

p5

10

<

mean

16.53

SD

3.33

<

p95

21

Final error count

The total number of errors accumulated across all agents over the run.

19

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.06137

s

SD

0.127133

s

p50

0.11539

s

<

p95

0.4168

s

≤

p99

0.4168

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

2

over 767.134648165s

mean rate

1.72

/s

std

3.53

/s

p5

0

/s

p95

9.84

/s

peak

14.12

/s

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.006661

s

SD

0.005338

s

p50

0.004887

s

<

p95

0.026198

s

≤

p99

0.026198

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004596

s

SD

0.002553

s

p50

0.004754

s

<

p95

0.009308

s

<

p99

0.016457

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.057263

s

SD

0.042156

s

p50

0.053017

s

<

p95

0.16665

s

<

p99

0.198732

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.023437

s

SD

0.12726

s

p50

0.232689

s

<

p95

0.419423

s

≤

p99

0.419423

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00569

s

SD

0.005075

s

p50

0.003702

s

<

p95

0.024736

s

≤

p99

0.024736

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003932

s

SD

0.002471

s

p50

0.004023

s

<

p95

0.008523

s

<

p99

0.015562

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.06318

s

SD

0.042862

s

p50

0.057865

s

<

p95

0.165263

s

<

p99

0.197354

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.022435

s

SD

0.127319

s

p50

0.23166

s

<

p95

0.418568

s

≤

p99

0.418568

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001299

s

SD

0.000587

s

p50

0.001179

s

<

p95

0.002376

s

<

p99

0.003413

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

5e-05

s

SD

4.3e-05

s

p50

3.3e-05

s

<

p95

0.000166

s

≤

p99

0.000166

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000563

s

SD

0.001151

s

p50

0.000444

s

<

p95

0.002884

s

<

p99

0.006528

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000601

s

SD

0.000691

s

p50

0.000459

s

<

p95

0.003404

s

≤

p99

0.003404

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.061862

s

SD

0.042424

s

p50

0.056882

s

<

p95

0.162162

s

<

p99

0.194229

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.018961

s

SD

0.127822

s

p50

0.228843

s

<

p95

0.416835

s

≤

p99

0.416835

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1e-05

s

SD

3e-06

s

p50

1.1e-05

s

<

p95

1.5e-05

s

≤

p99

1.5e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002026

s

SD

0.00179

s

p50

0.001826

s

<

p95

0.005757

s

<

p99

0.011983

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000158

s

SD

0.000153

s

p50

0.000119

s

<

p95

0.000301

s

<

p99

0.000962

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.1

s

<

mean

444.92

s

SD

255.66

s

<

p95

845.71

s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

30.328155

s

SD

46.31972

s

p50

0

s

<

p95

71.095914

s

<

p99

88.579217

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.020623

s

SD

0.069081

s

p50

0.001365

s

<

p95

0.224608

s

<

p99

0.282597

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.05235

s

SD

0.250544

s

p50

0.001414

s

<

p95

0.646732

s

<

p99

1.176118

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.09607

s

SD

0.066

s

p50

0.058389

s

<

p95

0.210985

s

<

p99

0.247321

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

2.858518

s

SD

9.955747

s

p50

0.003379

s

<

p95

20.969915

s

<

p99

59.603255

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

5.936079

s

SD

20.432389

s

p50

0.002059

s

<

p95

8.243128

s

<

p99

99.975746

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002601

s

SD

0.001577

s

p50

0.001527

s

<

p95

0.004839

s

<

p99

0.006481

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00379

s

SD

0.005223

s

p50

0.001177

s

<

p95

0.016625

s

<

p99

0.027755

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000534

s

SD

0.000977

s

p50

0.000339

s

<

p95

0.000931

s

<

p99

0.006187

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

15.766367

s

SD

29.695675

s

p50

0.163146

s

<

p95

60.003878

s

≤

p99

60.003878

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.03366

s

SD

0.122581

s

p50

0.170286

s

<

p95

0.409087

s

≤

p99

0.409087

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.077773

s

SD

28.162674

s

p50

0.065606

s

<

p95

60.001592

s

<

p99

60.001837

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.125945

s

SD

16.208088

s

p50

0.130331

s

<

p95

60.001569

s

≤

p99

60.001569

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

43.27

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

5.33

KiB/s

<

mean

333.89

KiB/s

SD

332.22

KiB/s

<

p95

942.07

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

2.04

KiB/s

<

mean

321.95

KiB/s

SD

283.47

KiB/s

<

p95

840.74

KiB/s

Total bytes received

Total bytes received on primary network interface

count

7.37

GiB

mean

8.47

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.10

GiB

mean

8.17

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.3%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (72.3% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 44% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.71

%

<

mean

55.4

%

SD

35.71

%

<

p95

97.77

%

System

12.13

%

CPU percentiles

CPU usage percentiles

p50

59.12

%

p95

97.77

%

p99

98.30

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

13

hosts

mean time

716.24

s

Memory used percentage

Percentage of memory used

p5

7.26

%

<

mean

18.82

%

SD

15.66

%

<

p95

34.49

%

Memory available percentage

Percentage of available memory

p5

65.51

%

<

mean

81.18

%

SD

15.66

%

<

p95

92.74

%

Max host memory used

Maximum memory usage percentage across all hosts

max

88.62

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

72.35

%

Memory growth rate

Rate of memory growth over time

growth

0.29

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.19

MiB/s

Disk write throughput

Disk write throughput in MiB/s

79.83

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/12

hosts

Mount Point /etc/hosts

0/12

hosts

Mount Point /etc/nomad.d/client-meta.json

0/10

hosts

Mount Point /etc/resolv.conf

0/12

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.76

5 min

3.02

15 min

2.24

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

44.44

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

26.239

%

SD

28.0395

%

<

p95

79.68

%

60 second average

24.9004

%

300 second average

19.2866

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

4.1479

%

SD

8.9048

%

<

p95

26.39

%

60 second average

4.1005

%

300 second average

3.6016

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.9257

%

SD

5.1776

%

<

p95

16.45

%

60 second average

1.9527

%

300 second average

1.7711

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.95

%

<

mean

49.49

%

SD

37.53

%

<

p95

96.63

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

227.81

KiB

<

mean

310.26

KiB

SD

78.43

KiB

<

p95

460.36

KiB

Holochain process threads

Number of threads in Holochain process

p5

25

threads

<

mean

33.63

threads

SD

6.25

threads

<

p95

46

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

57

file descriptors

<

mean

74.5

file descriptors

SD

11.92

file descriptors

<

p95

89

file descriptors

Mixed Arc `must_get_agent_activity`

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started

Thu, 08 Oct 2026 11:24:00 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full_write (1 agent)
zero_must_get_agent_activity (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_must_get_agent_activity_37734232741_1

Highest observed action_seq

The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

1545

Max

Highest chain head value observed for any single write agent.

1787

Write agents observed

3

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

101.033772

s

SD

79.679981

s

Max

Highest per-partition mean latency

max mean

315.200277

s

Min

Lowest per-partition mean latency

min mean

8.634802

s

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

1.99

/s

Max

Highest per-partition mean rate

max

2.42

/s

Min

Lowest per-partition mean rate

min

1.45

/s

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

0.49806

s

SD

0.584097

s

Max

Highest per-partition mean latency

max mean

0.837885

s

Min

Lowest per-partition mean latency

min mean

0.313456

s

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

23939

Agents affected

9 / 9

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

2660

Max

Highest count in any single partition

6065

Min

Lowest count in any single partition

928

Open connections

The average number of open connections maintained by an agent over the run, broken down by behaviour.

behaviour: full_write

p5

28

≤

mean

27.87

SD

1.03

<

p95

28

behaviour: zero_must_get_agent_activity

p5

11

<

mean

15.62

SD

3.2

<

p95

20

behaviour: zero_write

p5

0

<

mean

15.3

SD

5.91

<

p95

23

Final error count

The total number of all types of error accumulated over the run by all agents.

36967

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.057249

s

SD

0.154679

s

p50

0.064173

s

<

p95

0.494937

s

<

p99

0.634808

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 812.641746215s

Not enough time data to show a trend.

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007551

s

SD

0.004555

s

p50

0.006657

s

<

p95

0.021799

s

≤

p99

0.021799

s

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.070085

s

SD

0.039681

s

p50

0.043987

s

<

p95

0.140485

s

<

p99

0.219065

s

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.005034

s

SD

0.002313

s

p50

0.00492

s

<

p95

0.009892

s

<

p99

0.011854

s

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.190624

s

SD

0.066578

s

p50

0.177834

s

<

p95

0.279353

s

<

p99

0.338311

s

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.006025

s

SD

0.207901

s

p50

0.049793

s

<

p95

0.586637

s

≤

p99

0.586637

s

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00643

s

SD

0.004175

s

p50

0.005735

s

<

p95

0.019941

s

≤

p99

0.019941

s

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.069238

s

SD

0.039641

s

p50

0.042943

s

<

p95

0.139149

s

<

p99

0.218137

s

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004153

s

SD

0.001976

s

p50

0.004101

s

<

p95

0.008278

s

<

p99

0.009937

s

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.217774

s

SD

0.165923

s

p50

0.174412

s

<

p95

0.624057

s

<

p99

0.695018

s

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00569

s

SD

0.207424

s

p50

0.048952

s

<

p95

0.584515

s

≤

p99

0.584515

s

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001264

s

SD

0.000523

s

p50

0.001095

s

<

p95

0.0025

s

<

p99

0.002977

s

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.015959

s

SD

0.019869

s

p50

0.004888

s

<

p95

0.059009

s

<

p99

0.08459

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2.3e-05

s

SD

0.000366

s

p50

2.4e-05

s

<

p95

0.000654

s

<

p99

0.002252

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000394

s

SD

0.000186

s

p50

0.000367

s

<

p95

0.000751

s

<

p99

0.000939

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000442

s

SD

0.000196

s

p50

0.000414

s

<

p95

0.000808

s

<

p99

0.000996

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.141637

s

SD

0.19195

s

p50

0.143478

s

<

p95

0.578215

s

<

p99

0.664113

s

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.044876

s

SD

0.067479

s

p50

0.04913

s

<

p95

0.220925

s

<

p99

0.284935

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.030252

s

SD

0.025574

s

p50

0.011427

s

<

p95

0.075255

s

<

p99

0.118463

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

3e-06

s

SD

8e-06

s

p50

8e-06

s

<

p95

2.9e-05

s

≤

p99

2.9e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001953

s

SD

0.001203

s

p50

0.002068

s

<

p95

0.004245

s

<

p99

0.008787

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000135

s

SD

7e-05

s

p50

0.000111

s

<

p95

0.000302

s

<

p99

0.00033

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47

s

<

mean

445.97

s

SD

256.32

s

<

p95

846.98

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

21087

over 888.653496467s

mean rate

937299.28

/s

std

1.312761724e+07

/s

p5

0

/s

p95

467592.07

/s

peak

4.7632100396e+08

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

146.243315

s

SD

76.875549

s

p50

0

s

<

p95

219.86081

s

<

p99

346.445679

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

2.527817

s

SD

1.562785

s

p50

0.001977

s

<

p95

4.111357

s

<

p99

6.568594

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.114637

s

SD

0.113628

s

p50

0.001324

s

<

p95

0.305483

s

<

p99

0.420154

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.142206

s

SD

0.1774

s

p50

0.019904

s

<

p95

0.530243

s

<

p99

0.792193

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

11.015908

s

SD

15.979219

s

p50

0.003999

s

<

p95

38.592198

s

<

p99

77.63377

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

22.618045

s

SD

24.846145

s

p50

0.003293

s

<

p95

82.89723

s

<

p99

115.319471

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003096

s

SD

0.002128

s

p50

0.000256

s

<

p95

0.006111

s

<

p99

0.007772

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002865

s

SD

0.007312

s

p50

0.00176

s

<

p95

0.014347

s

<

p99

0.039365

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000377

s

SD

0.000131

s

p50

0.000308

s

<

p95

0.000599

s

<

p99

0.00066

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.202961

s

SD

29.912265

s

p50

0.266962

s

<

p95

60.001336

s

<

p99

60.001476

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.119786

s

SD

26.159259

s

p50

0.171392

s

<

p95

60.001675

s

<

p99

60.001957

s

P2P request roundtrip: 'must_get_agent_activity'

Time spent sending a must_get_agent_activity request and awaiting its response.

mean

0.076687

s

SD

25.801134

s

p50

0.078562

s

<

p95

60.001372

s

<

p99

60.001778

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.111085

s

SD

6.49066

s

p50

0.106509

s

<

p95

15.23996

s

<

p99

15.279242

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

22.81

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

1.88

KiB/s

<

mean

127.27

KiB/s

SD

433.69

KiB/s

<

p95

238.03

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

1.52

KiB/s

<

mean

81.66

KiB/s

SD

106.96

KiB/s

<

p95

237.42

KiB/s

Total bytes received

Total bytes received on primary network interface

count

2.92

GiB

mean

3.35

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.87

GiB

mean

2.15

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (74.9% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 29% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.27

%

<

mean

31.98

%

SD

33.42

%

<

p95

97.55

%

System

9.17

%

CPU percentiles

CPU usage percentiles

p50

11.98

%

p95

97.55

%

p99

98.20

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

10

hosts

mean time

420.67

s

Memory used percentage

Percentage of memory used

p5

8.17

%

<

mean

20.56

%

SD

15.69

%

<

p95

39.33

%

Memory available percentage

Percentage of available memory

p5

60.67

%

<

mean

79.44

%

SD

15.69

%

<

p95

91.83

%

Max host memory used

Maximum memory usage percentage across all hosts

max

89.69

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

74.87

%

Memory growth rate

Rate of memory growth over time

growth

0.20

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1.88

MiB/s

Disk write throughput

Disk write throughput in MiB/s

86.78

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/10

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.99

5 min

2.38

15 min

2.47

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

28.57

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

11.3982

%

SD

21.5451

%

<

p95

69.85

%

60 second average

11.6764

%

300 second average

14.5040

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.7647

%

SD

7.4945

%

<

p95

25.35

%

60 second average

2.8704

%

300 second average

3.1669

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.8677

%

SD

5.6758

%

<

p95

11.56

%

60 second average

1.9465

%

300 second average

1.9468

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.65

%

<

mean

25.79

%

SD

31.78

%

<

p95

96.36

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

371.76

KiB

<

mean

441.06

KiB

SD

80.01

KiB

<

p95

648.92

KiB

Holochain process threads

Number of threads in Holochain process

p5

27

threads

<

mean

34.07

threads

SD

8.37

threads

<

p95

51

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

59

file descriptors

<

mean

73.91

file descriptors

SD

11.51

file descriptors

<

p95

93

file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started

Thu, 08 Oct 2026 07:58:32 UTC

Duration

900s

Peer count

10

Peer count at end

8

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_call_rate_37734232741_1

Dispatch timing

The time between sending a remote call and the remote handler being invoked.

Mean

mean

0.081528

s

SD

0.044767

s

Max

Highest per-partition mean latency

max mean

0.106642

s

Min

Lowest per-partition mean latency

min mean

0.064216

s

Round-trip timing

The total elapsed time to get a response to the client.

Mean

mean

0.165098

s

SD

0.065644

s

Max

Highest per-partition mean latency

max mean

0.220445

s

Min

Lowest per-partition mean latency

min mean

0.122451

s

Error count

The total number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_call::call_echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

184534482

over 883.264959362s

mean rate

1.1379718778e+08

/s

std

1.2891129162e+08

/s

p5

456253.08

/s

p95

3.5644889932e+08

/s

peak

4.5069420207e+08

/s

WASM usage: 'remote_call::echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

36131970

over 883.264959362s

mean rate

1.19761738e+06

/s

std

1.82850123e+06

/s

p5

23961.21

/s

p95

6.0707827e+06

/s

peak

8.00852409e+06

/s

WASM usage: 'remote_call::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 883.264959362s

Not enough time data to show a trend.

Zome call duration: 'remote_call::call_echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.160753

s

SD

0.033853

s

p50

0.150289

s

<

p95

0.215027

s

<

p99

0.226314

s

Zome call duration: 'remote_call::echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.002869

s

SD

0.001114

s

p50

0.002566

s

<

p95

0.005347

s

<

p99

0.005397

s

WASM call duration: 'remote_call::call_echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.159703

s

SD

0.034017

s

p50

0.148929

s

<

p95

0.21417

s

<

p99

0.225297

s

WASM call duration: 'remote_call::echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001899

s

SD

0.000783

s

p50

0.001656

s

<

p95

0.003608

s

<

p99

0.003645

s

WASM call duration: 'remote_call::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001846

s

SD

0.000748

s

p50

0.001513

s

<

p95

0.003259

s

≤

p99

0.003259

s

Host function call duration: '__hc__call_1'

Duration of this host function call invoked from within WASM.

mean

0.157481

s

SD

0.03422

s

p50

0.146329

s

<

p95

0.212066

s

<

p99

0.223149

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000304

s

SD

0.000107

s

p50

0.000296

s

<

p95

0.000489

s

≤

p99

0.000489

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

9e-06

s

SD

5e-06

s

p50

9e-06

s

<

p95

2e-05

s

≤

p99

2e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00026

s

SD

8.8e-05

s

p50

0.000231

s

<

p95

0.000395

s

<

p99

0.000408

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000197

s

SD

0.000281

s

p50

0.000103

s

<

p95

0.000989

s

≤

p99

0.000989

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.99

s

<

mean

446.6

s

SD

256.6

s

<

p95

847

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

54

over 883.264825342s

mean rate

9.37

/s

std

31.7

/s

p5

0

/s

p95

54.01

/s

peak

228.67

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.941896

s

SD

1.564142

s

p50

1.100056

s

<

p95

4.584951

s

<

p99

7.01894

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002256

s

SD

0.001112

s

p50

0.002404

s

<

p95

0.003388

s

<

p99

0.005636

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.076232

s

SD

0.041434

s

p50

0.069067

s

<

p95

0.165349

s

<

p99

0.177618

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.005032

s

SD

0.001852

s

p50

0.004672

s

<

p95

0.007869

s

<

p99

0.011306

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.714941

s

SD

0.577005

s

p50

0.521909

s

<

p95

1.944486

s

<

p99

2.395639

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.145236

s

SD

0.062575

s

p50

0.1505

s

<

p95

0.228683

s

<

p99

0.275891

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000139

s

SD

3.6e-05

s

p50

0.000136

s

<

p95

0.00022

s

<

p99

0.000222

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000991

s

SD

0.000423

s

p50

0.001116

s

<

p95

0.00216

s

≤

p99

0.00216

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000388

s

SD

0.000149

s

p50

0.000358

s

<

p95

0.000706

s

<

p99

0.000715

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.102658

s

SD

0.069848

s

p50

0.136327

s

<

p95

0.229049

s

<

p99

0.235564

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.165543

s

SD

0.03053

s

p50

0.171347

s

<

p95

0.204399

s

<

p99

0.204401

s

P2P request roundtrip: 'call_remote'

Time spent sending a call_remote request and awaiting its response.

mean

0.156968

s

SD

0.034295

s

p50

0.145688

s

<

p95

0.211627

s

<

p99

0.22253

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

13.36

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

10.59

KiB/s

<

mean

253.57

KiB/s

SD

1.08

MiB/s

<

p95

401.69

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

8.65

KiB/s

<

mean

28.36

KiB/s

SD

35.03

KiB/s

<

p95

108.36

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.72

GiB

mean

1.98

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

197.16

MiB

mean

226.84

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.58 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 11% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

2.08

%

<

mean

20.45

%

SD

23.73

%

<

p95

76.17

%

System

7.09

%

CPU percentiles

CPU usage percentiles

p50

10.97

%

p95

76.17

%

p99

88.87

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

3

hosts

mean time

117.98

s

Memory used percentage

Percentage of memory used

p5

8.72

%

<

mean

19.48

%

SD

9.9

%

<

p95

43.89

%

Memory available percentage

Percentage of available memory

p5

56.11

%

<

mean

80.52

%

SD

9.9

%

<

p95

91.28

%

Max host memory used

Maximum memory usage percentage across all hosts

max

42.77

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

7.57

%

Memory growth rate

Rate of memory growth over time

growth

1.58

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.08

MiB/s

Disk write throughput

Disk write throughput in MiB/s

19.12

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

0/3

hosts

Mount Point /etc/hosts

0/3

hosts

Mount Point /etc/nomad.d/client-meta.json

0/6

hosts

Mount Point /etc/resolv.conf

0/3

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.06

5 min

1.64

15 min

0.97

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

11.11

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

7.4558

%

SD

13.5042

%

<

p95

44.86

%

60 second average

7.2511

%

300 second average

5.4928

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.1641

%

SD

0.5078

%

<

p95

1.11

%

60 second average

0.1652

%

300 second average

0.1384

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.1095

%

SD

0.3636

%

<

p95

0.67

%

60 second average

0.1089

%

300 second average

0.0897

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

6.03

%

SD

13.74

%

<

p95

37.74

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

93.04

KiB

<

mean

227.35

KiB

SD

80.17

KiB

<

p95

310.04

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

85.3

threads

SD

279.46

threads

<

p95

618

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

190.67

file descriptors

SD

617.02

file descriptors

<

p95

1381

file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started

Thu, 08 Oct 2026 08:00:11 UTC

Duration

900s

Peer count

10

Peer count at end

8

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_signals_37734232741_1

Round trip time

The time from origin signal dispatch to origin receive of the remote side's response signal.

mean

0.198765

s

SD

0.406883

s

p50

0.207062

s

<

p95

0.238122

s

<

p99

0.303854

s

Timeout rate

The proportion of remote signal round-trips that timed out.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_signal::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 888.035019011s

Not enough time data to show a trend.

WASM usage: 'remote_signal::recv_remote_signal'

Metered usage count of the WASM ribosome for this zome function.

total

31359976

over 886.453601015s

mean rate

8.054697314e+07

/s

std

6.4219256711e+08

/s

p5

0

/s

p95

2.6381509194e+08

/s

peak

1.17447006755e+10

/s

WASM usage: 'remote_signal::signal_request'

Metered usage count of the WASM ribosome for this zome function.

total

13153672

over 888.035019011s

mean rate

2.182752293e+07

/s

std

2.3366968275e+08

/s

p5

0

/s

p95

1.879393395e+07

/s

peak

3.52993244817e+09

/s

Zome call duration: 'remote_signal::recv_remote_signal'

Duration of this zome call as measured by Holochain internally.

mean

0.002243

s

SD

0.000597

s

p50

0.002446

s

<

p95

0.003146

s

<

p99

0.003776

s

Zome call duration: 'remote_signal::signal_request'

Duration of this zome call as measured by Holochain internally.

mean

0.002202

s

SD

0.000524

s

p50

0.002399

s

<

p95

0.002926

s

<

p99

0.003767

s

WASM call duration: 'remote_signal::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002826

s

SD

0.001692

s

p50

0.002121

s

<

p95

0.006604

s

≤

p99

0.006604

s

WASM call duration: 'remote_signal::recv_remote_signal'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001489

s

SD

0.000426

s

p50

0.001487

s

<

p95

0.002186

s

<

p99

0.002461

s

WASM call duration: 'remote_signal::signal_request'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001437

s

SD

0.000349

s

p50

0.001471

s

<

p95

0.001983

s

<

p99

0.002401

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000632

s

SD

0.000607

s

p50

0.000394

s

<

p95

0.002194

s

≤

p99

0.002194

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

5.8e-05

s

SD

3.5e-05

s

p50

5.4e-05

s

<

p95

0.000107

s

<

p99

0.000117

s

Host function call duration: '__hc__send_remote_signal_1'

Duration of this host function call invoked from within WASM.

mean

6.2e-05

s

SD

3.3e-05

s

p50

5.5e-05

s

<

p95

0.000121

s

<

p99

0.000145

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

4e-06

s

SD

1e-06

s

p50

4e-06

s

<

p95

5e-06

s

<

p99

7e-06

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000356

s

SD

0.000413

s

p50

0.000185

s

<

p95

0.001556

s

≤

p99

0.001556

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

1739

over 886.453592418s

mean rate

2536.53

/s

std

15283.24

/s

p5

0

/s

p95

10332.98

/s

peak

312940.07

/s

Sent remote signals

Count of remote signals sent via the send_remote_signal host function.

total

1739

over 888.035040602s

mean rate

1967.05

/s

std

11062.65

/s

p5

0

/s

p95

9335.96

/s

peak

230149.6

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000198

s

SD

0.000212

s

p50

0.000102

s

<

p95

0.000804

s

≤

p99

0.000804

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.8

s

<

mean

446.86

s

SD

256.91

s

<

p95

846.82

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

86

over 888.035062201s

mean rate

3116.93

/s

std

17917.07

/s

p5

0

/s

p95

13121.35

/s

peak

288788.3

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.965209

s

SD

1.508693

s

p50

1.192049

s

<

p95

4.256737

s

<

p99

5.1117

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002005

s

SD

0.000976

s

p50

0.002031

s

<

p95

0.003989

s

<

p99

0.004513

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.070445

s

SD

0.032852

s

p50

0.084834

s

<

p95

0.116977

s

<

p99

0.133298

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.005924

s

SD

0.003729

s

p50

0.005653

s

<

p95

0.013892

s

<

p99

0.018166

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.119926

s

SD

0.134306

s

p50

0.066526

s

<

p95

0.321023

s

<

p99

0.521472

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.24152

s

SD

0.08336

s

p50

0.26899

s

<

p95

0.330407

s

<

p99

0.447487

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000124

s

SD

4.4e-05

s

p50

0.00013

s

<

p95

0.000213

s

<

p99

0.000296

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001005

s

SD

0.000656

s

p50

0.001265

s

<

p95

0.003093

s

≤

p99

0.003093

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000417

s

SD

0.000127

s

p50

0.000452

s

<

p95

0.000595

s

<

p99

0.000776

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.122562

s

SD

0.026124

s

p50

0.125983

s

<

p95

0.147659

s

<

p99

0.172446

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.171626

s

SD

0.025222

s

p50

0.176888

s

<

p95

0.205199

s

≤

p99

0.205199

s

P2P received remote signals

Count of remote signals received.

total

1738

over 886.453684765s

mean rate

3171.27

/s

std

18315.67

/s

p5

0

/s

p95

12767.23

/s

peak

330114.71

/s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

15.60

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

6.84

KiB/s

<

mean

321.27

KiB/s

SD

1.21

MiB/s

<

p95

820.35

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

5.68

KiB/s

<

mean

32.32

KiB/s

SD

36.28

KiB/s

<

p95

108.10

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.09

GiB

mean

1.25

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

112.35

MiB

mean

129.27

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.3%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 40% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.92

%

<

mean

23.38

%

SD

26.89

%

<

p95

86.32

%

System

7.78

%

CPU percentiles

CPU usage percentiles

p50

11.81

%

p95

86.32

%

p99

94.35

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

3

hosts

mean time

117.98

s

Memory used percentage

Percentage of memory used

p5

8.22

%

<

mean

21.2

%

SD

12

%

<

p95

43.96

%

Memory available percentage

Percentage of available memory

p5

56.04

%

<

mean

78.8

%

SD

12

%

<

p95

91.78

%

Max host memory used

Maximum memory usage percentage across all hosts

max

43.23

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

7.57

%

Memory growth rate

Rate of memory growth over time

growth

0.67

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.07

MiB/s

Disk write throughput

Disk write throughput in MiB/s

13.15

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

0/2

hosts

Mount Point /etc/hosts

0/2

hosts

Mount Point /etc/nomad.d/client-meta.json

0/3

hosts

Mount Point /etc/resolv.conf

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.26

5 min

2.72

15 min

1.65

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

40.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

9.4841

%

SD

15.3205

%

<

p95

47.67

%

60 second average

9.4865

%

300 second average

7.3973

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.1396

%

SD

0.5028

%

<

p95

0.91

%

60 second average

0.1506

%

300 second average

0.1426

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.0894

%

SD

0.3482

%

<

p95

0.6

%

60 second average

0.0981

%

300 second average

0.0944

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

6.52

%

SD

13.5

%

<

p95

34.13

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

77.26

KiB

<

mean

236.55

KiB

SD

120.21

KiB

<

p95

355.61

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

77.45

threads

SD

249.17

threads

<

p95

494

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

172.83

file descriptors

SD

550.47

file descriptors

<

p95

1111

file descriptors

Single-write, many-read

↑ Back to index

Creates an entry in the agent setup, then read one's own record back. This tests the maximum read performance for reading back local data when the chain is short.

Started

Thu, 08 Oct 2026 08:03:38 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

single_write_many_read_37734232741_1

read call timing

The time it takes to complete the zome call that retrieves the record.

mean

0.003972

s

SD

0.0024

s

p50

0.003387

s

<

p95

0.007051

s

<

p99

0.010943

s

read call rate

The number of read zome calls completed per second.

mean

2514.21

/s

Error count

The number of errors collected.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000268

s

SD

3.1e-05

s

p50

0.000263

s

<

p95

0.000345

s

<

p99

0.000358

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.00191863s

Not enough time data to show a trend.

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

2369720052

over 280.00191863s

mean rate

8.46323009e+06

/s

std

2.36311336e+06

/s

p5

5.80565065e+06

/s

p95

1.299598886e+07

/s

peak

1.301933532e+07

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.00191863s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.00191863s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.002882

s

SD

0

s

p50

0.002882

s

≤

p95

0.002882

s

≤

p99

0.002882

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.001613

s

SD

0.000111

s

p50

0.001607

s

<

p95

0.001872

s

<

p99

0.001888

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002392

s

SD

0

s

p50

0.002392

s

≤

p95

0.002392

s

≤

p99

0.002392

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001132

s

SD

8.3e-05

s

p50

0.001127

s

<

p95

0.001335

s

<

p99

0.001343

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001191

s

SD

0

s

p50

0.001191

s

≤

p95

0.001191

s

≤

p99

0.001191

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000783

s

SD

0

s

p50

0.000783

s

≤

p95

0.000783

s

≤

p99

0.000783

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000241

s

SD

2e-06

s

p50

0.000243

s

≤

p95

0.000243

s

≤

p99

0.000243

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000293

s

SD

3.2e-05

s

p50

0.000289

s

<

p95

0.000374

s

<

p99

0.000386

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001268

s

SD

0

s

p50

0.001268

s

≤

p95

0.001268

s

≤

p99

0.001268

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.00012

s

SD

0

s

p50

0.00012

s

≤

p95

0.00012

s

≤

p99

0.00012

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.46

s

<

mean

147.46

s

SD

83.67

s

<

p95

277.47

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.002003424s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001279

s

SD

0

s

p50

0.001279

s

≤

p95

0.001279

s

≤

p99

0.001279

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.000828

s

SD

0

s

p50

0.000828

s

≤

p95

0.000828

s

≤

p99

0.000828

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004864

s

SD

0.000894

s

p50

0.005055

s

<

p95

0.006578

s

≤

p99

0.006578

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002114

s

SD

0

s

p50

0.002114

s

≤

p95

0.002114

s

≤

p99

0.002114

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000827

s

SD

0

s

p50

0.000827

s

≤

p95

0.000827

s

≤

p99

0.000827

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

5.3e-05

s

SD

5e-06

s

p50

5.1e-05

s

<

p95

6.3e-05

s

<

p99

6.6e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000981

s

SD

0.000297

s

p50

0.001417

s

≤

p95

0.001417

s

≤

p99

0.001417

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000314

s

SD

0

s

p50

0.000314

s

≤

p95

0.000314

s

≤

p99

0.000314

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

46.03

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.02

KiB/s

<

mean

609.67

KiB/s

SD

1.79

MiB/s

<

p95

6.93

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.71

KiB/s

<

mean

41.29

KiB/s

SD

41.85

KiB/s

<

p95

151.25

KiB/s

Total bytes received

Total bytes received on primary network interface

count

166.71

MiB

mean

609.67

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

11.29

MiB

mean

41.29

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.25 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

33.83

%

<

mean

65.81

%

SD

16.05

%

<

p95

83.76

%

System

19.79

%

CPU percentiles

CPU usage percentiles

p50

65.25

%

p95

83.76

%

p99

88.57

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

85.71

s

Memory used percentage

Percentage of memory used

p5

7.55

%

<

mean

8.96

%

SD

0.75

%

<

p95

9.92

%

Memory available percentage

Percentage of available memory

p5

90.08

%

<

mean

91.04

%

SD

0.75

%

<

p95

92.45

%

Max host memory used

Maximum memory usage percentage across all hosts

max

8.96

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

1.25

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.11

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.41

5 min

4.08

15 min

1.97

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

3.52

%

<

mean

27.4728

%

SD

14.1851

%

<

p95

45.06

%

60 second average

23.0417

%

300 second average

12.7241

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

26.53

%

SD

13.64

%

<

p95

44.53

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.67

KiB

<

mean

229.98

KiB

SD

96.86

KiB

<

p95

320.85

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

113.03

threads

SD

345.85

threads

<

p95

1069

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

248.84

file descriptors

SD

749.74

file descriptors

<

p95

2325

file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started

Thu, 08 Oct 2026 08:17:02 UTC

Duration

900s

Peer count

20

Peer count at end

19

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

validation_receipts_37734232741_1

Receipts complete timing

The amount of time between publishing a record and receiving the required number of validation receipts.

Mean

mean

6.090876

s

SD

2.074772

s

Max

Highest per-partition mean latency

max mean

6.885278

s

Min

Lowest per-partition mean latency

min mean

5.670182

s

Receipts complete rate

The number of complete validation receipt sets collected per second.

Mean

mean

1.54

/s

Max

Highest per-partition mean rate

max

1.65

/s

Min

Lowest per-partition mean rate

min

1.33

/s

Error count

The total number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

4322430

over 886.588798336s

mean rate

4.04082176e+06

/s

std

8.74700655e+06

/s

p5

0

/s

p95

2.520642105e+07

/s

peak

4.627417231e+07

/s

WASM usage: 'crud::get_sample_entry_validation_receipts'

Metered usage count of the WASM ribosome for this zome function.

total

15359748

over 886.588798336s

mean rate

9.37424534e+06

/s

std

2.107833832e+07

/s

p5

30860.51

/s

p95

6.790118369e+07

/s

peak

8.969454113e+07

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 886.588798336s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

540666

over 886.588798336s

mean rate

499121.11

/s

std

1.08846958e+06

/s

p5

0

/s

p95

3.15291511e+06

/s

peak

5.78814964e+06

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.006012

s

SD

0.002017

s

p50

0.005187

s

<

p95

0.010161

s

<

p99

0.011946

s

Zome call duration: 'crud::get_sample_entry_validation_receipts'

Duration of this zome call as measured by Holochain internally.

mean

0.020854

s

SD

0.010958

s

p50

0.01371

s

<

p95

0.035572

s

<

p99

0.051096

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005161

s

SD

0.001815

s

p50

0.004456

s

<

p95

0.008889

s

<

p99

0.010565

s

WASM call duration: 'crud::get_sample_entry_validation_receipts'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.019835

s

SD

0.010749

s

p50

0.012779

s

<

p95

0.033922

s

<

p99

0.049687

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002449

s

SD

0.001778

s

p50

0.00211

s

<

p95

0.004689

s

<

p99

0.009386

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001734

s

SD

0.000658

s

p50

0.001416

s

<

p95

0.003172

s

<

p99

0.00367

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.00053

s

SD

0.000834

s

p50

0.000467

s

<

p95

0.001992

s

<

p99

0.005437

s

Host function call duration: '__hc__get_validation_receipts_1'

Duration of this host function call invoked from within WASM.

mean

0.017653

s

SD

0.010285

s

p50

0.010989

s

<

p95

0.031044

s

<

p99

0.045957

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00274

s

SD

0.000993

s

p50

0.002376

s

<

p95

0.004645

s

<

p99

0.005895

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000218

s

SD

8.8e-05

s

p50

0.000197

s

<

p95

0.000367

s

<

p99

0.000402

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.12

s

<

mean

444.19

s

SD

255.94

s

<

p95

844.84

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

6813

over 886.588783195s

mean rate

120.85

/s

std

182.26

/s

p5

4.49

/s

p95

384.18

/s

peak

2638.05

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.90907

s

SD

2.466134

s

p50

1.687993

s

<

p95

5.574284

s

<

p99

15.554518

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.00449

s

SD

0.006942

s

p50

0.002683

s

<

p95

0.022065

s

<

p99

0.031979

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.041622

s

SD

0.027003

s

p50

0.045943

s

<

p95

0.094409

s

<

p99

0.149274

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.044401

s

SD

0.02252

s

p50

0.027734

s

<

p95

0.074333

s

<

p99

0.097045

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.051626

s

SD

1.366812

s

p50

0.050554

s

<

p95

2.774689

s

<

p99

5.674896

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

2.380996

s

SD

5.472289

s

p50

2.984566

s

<

p95

17.087944

s

<

p99

21.394619

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000786

s

SD

0.000376

s

p50

0.000529

s

<

p95

0.001402

s

<

p99

0.001697

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.006426

s

SD

0.007014

s

p50

0.001836

s

<

p95

0.024922

s

<

p99

0.026883

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000329

s

SD

0.000123

s

p50

0.000291

s

<

p95

0.000614

s

<

p99

0.00068

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.164519

s

SD

19.441466

s

p50

0.051859

s

<

p95

60.001418

s

<

p99

60.002231

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.127093

s

SD

0.036438

s

p50

0.111288

s

<

p95

0.201635

s

<

p99

0.214543

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

71.34

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

0

B/s

<

mean

791.00

KiB/s

SD

356.39

KiB/s

<

p95

1.35

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

0

B/s

<

mean

727.11

KiB/s

SD

450.43

KiB/s

<

p95

1.37

MiB/s

Total bytes received

Total bytes received on primary network interface

count

12.74

GiB

mean

14.66

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

11.71

GiB

mean

13.48

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Warning Moderate swap usage (17.2% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 80% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

58.81

%

<

mean

85.78

%

SD

12.26

%

<

p95

97.6

%

System

14.44

%

CPU percentiles

CPU usage percentiles

p50

90.94

%

p95

97.60

%

p99

97.95

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

19

hosts

mean time

689.74

s

Memory used percentage

Percentage of memory used

p5

6.82

%

<

mean

23.49

%

SD

9.37

%

<

p95

42.14

%

Memory available percentage

Percentage of available memory

p5

57.86

%

<

mean

76.51

%

SD

9.37

%

<

p95

93.18

%

Max host memory used

Maximum memory usage percentage across all hosts

max

44.36

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

17.21

%

Memory growth rate

Rate of memory growth over time

growth

0.31

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.08

MiB/s

Disk write throughput

Disk write throughput in MiB/s

115.59

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/9

hosts

Mount Point /etc/hosts

0/9

hosts

Mount Point /etc/nomad.d/client-meta.json

0/6

hosts

Mount Point /etc/resolv.conf

0/9

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

7.65

5 min

6.06

15 min

3.72

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

80.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

11.97

%

<

mean

48.7029

%

SD

19.5698

%

<

p95

75.96

%

60 second average

47.7844

%

300 second average

37.6541

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0011

%

SD

0.0472

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0011

%

SD

0.0472

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

12.4569

%

SD

19.1565

%

<

p95

54.32

%

60 second average

12.0141

%

300 second average

9.5631

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.6172

%

SD

5.1137

%

<

p95

15.05

%

60 second average

2.5350

%

300 second average

2.3714

%

Holochain process CPU usage

CPU usage by Holochain process

p5

1.04

%

<

mean

6.24

%

SD

6.02

%

<

p95

14.48

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

229.10

KiB

<

mean

256.30

KiB

SD

50.34

KiB

<

p95

305.29

KiB

Holochain process threads

Number of threads in Holochain process

p5

30

threads

<

mean

33.27

threads

SD

7.85

threads

<

p95

47

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

71

file descriptors

<

mean

76.03

file descriptors

SD

14.98

file descriptors

<

p95

89

file descriptors

Write/`get_agent_activity`

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 08 Oct 2026 08:45:58 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_37734232741_1

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

3834.57

Max

Highest chain head value observed for any single write agent.

5338

Write agents observed

7

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.354785

s

SD

0.155046

s

Max

Highest per-partition mean latency

max mean

1.038414

s

Min

Lowest per-partition mean latency

min mean

0.121021

s

Error count

The number of errors accumulated.

17

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.010506

s

SD

0.410769

s

p50

0.000403

s

<

p95

0.32365

s

<

p99

0.600156

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 780.213351299s

Not enough time data to show a trend.

WASM usage: 'agent_activity::announce_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 881.933412913s

Not enough time data to show a trend.

WASM usage: 'agent_activity::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

133703856

over 881.933412913s

mean rate

6.7386576467e+08

/s

std

1.76264401796e+09

/s

p5

1.04344884e+06

/s

p95

5.79380998436e+09

/s

peak

9.72029295112e+09

/s

WASM usage: 'agent_activity::get_agent_activity_full'

Metered usage count of the WASM ribosome for this zome function.

total

558136252

over 851.604765729s

mean rate

2.73446504049e+09

/s

std

3.80957947329e+09

/s

p5

2.178838275e+07

/s

p95

1.17467987667e+10

/s

peak

1.84481577107e+10

/s

WASM usage: 'agent_activity::get_random_agent_with_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

317543696

over 888.404916597s

mean rate

1.58741342252e+09

/s

std

3.32019827018e+09

/s

p5

5.553052351e+07

/s

p95

1.10720598334e+10

/s

peak

1.64474515343e+10

/s

WASM usage: 'agent_activity_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

23002800

over 888.404916597s

mean rate

1.5014839565e+08

/s

std

2.5018904624e+08

/s

p5

4.03546515e+06

/s

p95

8.0232317633e+08

/s

peak

1.24158075926e+09

/s

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.004826

s

SD

0.002045

s

p50

0.004177

s

<

p95

0.008131

s

≤

p99

0.008131

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004388

s

SD

0.002806

s

p50

0.004514

s

<

p95

0.011571

s

<

p99

0.013187

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.118054

s

SD

0.697669

s

p50

0.1233

s

<

p95

0.500991

s

<

p99

0.812398

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.003645

s

SD

0.001939

s

p50

0.003729

s

<

p95

0.006993

s

<

p99

0.010336

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003849

s

SD

0.001536

s

p50

0.003497

s

<

p95

0.006478

s

≤

p99

0.006478

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003689

s

SD

0.002658

s

p50

0.00376

s

<

p95

0.010524

s

<

p99

0.012165

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.117216

s

SD

0.697693

s

p50

0.122663

s

<

p95

0.500552

s

<

p99

0.811701

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003008

s

SD

0.001691

s

p50

0.003163

s

<

p95

0.005733

s

<

p99

0.00913

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00104

s

SD

0.000701

s

p50

0.00119

s

<

p95

0.003051

s

<

p99

0.003823

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2.5e-05

s

SD

8e-06

s

p50

2.5e-05

s

<

p95

4e-05

s

≤

p99

4e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000441

s

SD

0.000497

s

p50

0.000445

s

<

p95

0.001673

s

<

p99

0.002382

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000341

s

SD

0.000128

s

p50

0.000298

s

<

p95

0.000519

s

≤

p99

0.000519

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.115219

s

SD

0.697702

s

p50

0.120713

s

<

p95

0.499245

s

<

p99

0.809625

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000306

s

SD

0.000193

s

p50

0.000306

s

<

p95

0.000893

s

<

p99

0.000931

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

8e-06

s

SD

8e-06

s

p50

8e-06

s

<

p95

2.6e-05

s

<

p99

2.9e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00162

s

SD

0.001003

s

p50

0.001748

s

<

p95

0.00483

s

<

p99

0.005174

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000135

s

SD

8.3e-05

s

p50

0.000109

s

<

p95

0.000206

s

<

p99

0.000506

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.1

s

<

mean

446.66

s

SD

256.71

s

<

p95

847.12

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

18012

over 888.404865217s

mean rate

170717.61

/s

std

819045.25

/s

p5

6.09

/s

p95

354873.62

/s

peak

7.7221216e+06

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

244.481261

s

SD

206.140833

s

p50

31.264502

s

<

p95

523.465027

s

<

p99

558.688194

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.057337

s

SD

0.100635

s

p50

0.106709

s

<

p95

0.290531

s

<

p99

0.372993

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.376373

s

SD

0.450112

s

p50

0.986406

s

<

p95

1.18528

s

<

p99

1.222491

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.121396

s

SD

0.06031

s

p50

0.029078

s

<

p95

0.171226

s

<

p99

0.251913

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

6.963272

s

SD

19.024459

s

p50

12.760915

s

<

p95

54.749474

s

<

p99

72.437565

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

16.086936

s

SD

31.400558

s

p50

0.060369

s

<

p95

87.366492

s

<

p99

124.729673

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002389

s

SD

0.001316

s

p50

0.002288

s

<

p95

0.00481

s

<

p99

0.005672

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.003088

s

SD

0.005269

s

p50

0.001354

s

<

p95

0.018671

s

<

p99

0.023307

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000355

s

SD

0.000262

s

p50

0.000268

s

<

p95

0.000629

s

<

p99

0.00168

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.761508

s

SD

26.428759

s

p50

0.104767

s

<

p95

60.001249

s

<

p99

60.001422

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.284825

s

SD

0.135658

s

p50

0.274204

s

<

p95

0.53966

s

≤

p99

0.53966

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.06343

s

SD

25.924079

s

p50

0.099022

s

<

p95

60.001608

s

<

p99

60.002002

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.155835

s

SD

21.644205

s

p50

0.168822

s

<

p95

60.001373

s

<

p99

60.001765

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

50.82

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

32.29

KiB/s

<

mean

307.65

KiB/s

SD

422.87

KiB/s

<

p95

611.22

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

27.95

KiB/s

<

mean

259.65

KiB/s

SD

299.00

KiB/s

<

p95

772.94

KiB/s

Total bytes received

Total bytes received on primary network interface

count

6.77

GiB

mean

7.79

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.71

GiB

mean

6.57

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.3%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (76.2% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 67% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

21.37

%

<

mean

62.62

%

SD

23.88

%

<

p95

97.39

%

System

11.80

%

CPU percentiles

CPU usage percentiles

p50

65.20

%

p95

97.39

%

p99

98.30

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

15

hosts

mean time

511.39

s

Memory used percentage

Percentage of memory used

p5

7.39

%

<

mean

19.03

%

SD

15.25

%

<

p95

36.73

%

Memory available percentage

Percentage of available memory

p5

63.27

%

<

mean

80.97

%

SD

15.25

%

<

p95

92.61

%

Max host memory used

Maximum memory usage percentage across all hosts

max

86.25

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

76.15

%

Memory growth rate

Rate of memory growth over time

growth

0.23

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2.27

MiB/s

Disk write throughput

Disk write throughput in MiB/s

133.88

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/12

hosts

Mount Point /etc/hosts

0/12

hosts

Mount Point /etc/nomad.d/client-meta.json

0/10

hosts

Mount Point /etc/resolv.conf

0/12

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.17

5 min

4.74

15 min

5.20

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

66.67

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

20.4804

%

SD

21.8565

%

<

p95

67.39

%

60 second average

21.0559

%

300 second average

26.2068

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

7.9751

%

SD

14.3969

%

<

p95

42.96

%

60 second average

7.9952

%

300 second average

7.9296

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.8884

%

SD

8.9902

%

<

p95

27.2

%

60 second average

3.8988

%

300 second average

3.3771

%

Holochain process CPU usage

CPU usage by Holochain process

p5

20.15

%

<

mean

57.33

%

SD

22.87

%

<

p95

95.36

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

275.73

KiB

<

mean

349.70

KiB

SD

64.98

KiB

<

p95

482.46

KiB

Holochain process threads

Number of threads in Holochain process

p5

28

threads

<

mean

36.27

threads

SD

9.47

threads

<

p95

52

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

71

file descriptors

<

mean

79.75

file descriptors

SD

12.87

file descriptors

<

p95

97

file descriptors

Write/Query

↑ Back to index

Creates an entry, then queries the source chain and performs a simple operation on the entries, then repeats.

Started

Thu, 08 Oct 2026 08:05:23 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_query_37734232741_1

Write timing

The time it takes to complete the zome function call that writes the entry.

mean

0.035922

s

SD

0.021808

s

p50

0.029886

s

<

p95

0.073286

s

<

p99

0.096671

s

Write rate

The number of writes completed per second.

mean

79.36

/s

Query timing

The time it takes to complete the chain_query_count_len zome call that queries the source chain.

mean

0.080017

s

SD

0.05457

s

p50

0.065898

s

<

p95

0.169095

s

<

p99

0.240302

s

Query rate

The number of chain queries completed per second.

mean

79.32

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::chain_query_count_len'

Metered usage count of the WASM ribosome for this zome function.

total

68373344528

over 280.005915382s

mean rate

2.4418555469e+08

/s

std

2.888682965e+07

/s

p5

1.9152294717e+08

/s

p95

2.9004727296e+08

/s

peak

2.9053161012e+08

/s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

79587600

over 280.005915382s

mean rate

284237.11

/s

std

193302.27

/s

p5

133788.59

/s

p95

692954.58

/s

peak

998266.12

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.005915382s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

19914531

over 280.005915382s

mean rate

71122.25

/s

std

48385.81

/s

p5

33040.48

/s

p95

172925.7

/s

peak

250162.95

/s

Zome call duration: 'crud::chain_query_count_len'

Duration of this zome call as measured by Holochain internally.

mean

0.030871

s

SD

0.009974

s

p50

0.0284

s

<

p95

0.040821

s

<

p99

0.041755

s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.00307

s

SD

0.000185

s

p50

0.002999

s

<

p95

0.003283

s

<

p99

0.003302

s

WASM call duration: 'crud::chain_query_count_len'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.030214

s

SD

0.009921

s

p50

0.027758

s

<

p95

0.040112

s

<

p99

0.04104

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002494

s

SD

0.000158

s

p50

0.002429

s

<

p95

0.002681

s

<

p99

0.002697

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001279

s

SD

0

s

p50

0.001279

s

≤

p95

0.001279

s

≤

p99

0.001279

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000812

s

SD

5.4e-05

s

p50

0.000788

s

<

p95

0.000876

s

<

p99

0.000884

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000286

s

SD

7.3e-05

s

p50

0.000416

s

≤

p95

0.000416

s

≤

p99

0.000416

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.013472

s

SD

0.004505

s

p50

0.012425

s

<

p95

0.017797

s

<

p99

0.018239

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001285

s

SD

9.3e-05

s

p50

0.001249

s

<

p95

0.001405

s

<

p99

0.001434

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.5e-05

s

SD

1.1e-05

s

p50

8e-05

s

<

p95

9.7e-05

s

<

p99

9.8e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.57

s

<

mean

147.57

s

SD

83.67

s

<

p95

277.57

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.005903596s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001812

s

SD

0

s

p50

0.001812

s

≤

p95

0.001812

s

≤

p99

0.001812

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001222

s

SD

0.000181

s

p50

0.00115

s

<

p95

0.001446

s

<

p99

0.001474

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.025284

s

SD

0.009315

s

p50

0.022703

s

<

p95

0.034906

s

<

p99

0.036117

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.003841

s

SD

0

s

p50

0.003841

s

≤

p95

0.003841

s

≤

p99

0.003841

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002781

s

SD

0

s

p50

0.002781

s

≤

p95

0.002781

s

≤

p99

0.002781

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00252

s

SD

0.000887

s

p50

0.002319

s

<

p95

0.00338

s

<

p99

0.003466

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00095

s

SD

0.00017

s

p50

0.001197

s

≤

p95

0.001197

s

≤

p99

0.001197

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000253

s

SD

2.6e-05

s

p50

0.000241

s

<

p95

0.000286

s

<

p99

0.000289

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

60.72

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.02

KiB/s

<

mean

813.08

KiB/s

SD

1.93

MiB/s

<

p95

6.93

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.56

KiB/s

<

mean

35.21

KiB/s

SD

37.86

KiB/s

<

p95

149.11

KiB/s

Total bytes received

Total bytes received on primary network interface

count

230.27

MiB

mean

813.08

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

9.97

MiB

mean

35.21

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 97.6%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 2.64 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

64.78

%

<

mean

79.86

%

SD

10.51

%

<

p95

96.57

%

System

19.14

%

CPU percentiles

CPU usage percentiles

p50

82.90

%

p95

96.57

%

p99

97.60

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

196.55

s

Memory used percentage

Percentage of memory used

p5

8.55

%

<

mean

9.74

%

SD

0.86

%

<

p95

11.59

%

Memory available percentage

Percentage of available memory

p5

88.41

%

<

mean

90.26

%

SD

0.86

%

<

p95

91.45

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.74

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

2.64

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.73

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.05

5 min

4.60

15 min

2.38

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

18.74

%

<

mean

37.5687

%

SD

15.6432

%

<

p95

61.66

%

60 second average

32.1853

%

300 second average

18.8887

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

23

%

<

mean

31.12

%

SD

6.88

%

<

p95

49.9

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

23.28

KiB

<

mean

273.40

KiB

SD

119.35

KiB

<

p95

447.32

KiB

Holochain process threads

Number of threads in Holochain process

p5

8

threads

<

mean

25.61

threads

SD

8.44

threads

<

p95

34

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

58

file descriptors

SD

19.52

file descriptors

<

p95

74

file descriptors

Write/Read

↑ Back to index

Creates an entry and reads it back, then repeats.

Started

Thu, 08 Oct 2026 08:07:03 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_read_37734232741_1

Write timing

The time taken to write the entry.

mean

0.046706

s

SD

0.025669

s

p50

0.043605

s

<

p95

0.08503

s

<

p99

0.118422

s

Write rate

The number of writes completed per second.

mean

97.96

/s

Read timing

The time taken to read the entry.

mean

0.051403

s

SD

0.030666

s

p50

0.047944

s

<

p95

0.096416

s

<

p99

0.148099

s

Read rate

The number of reads completed per second.

mean

97.96

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.00082

s

SD

0.000136

s

p50

0.000811

s

<

p95

0.000909

s

<

p99

0.000918

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

97288980

over 280.00740225s

mean rate

347453.78

/s

std

216518.63

/s

p5

185245.77

/s

p95

799300.67

/s

peak

1.18351543e+06

/s

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

94830852

over 280.00740225s

mean rate

338674.93

/s

std

210153.19

/s

p5

184028.78

/s

p95

779612.31

/s

peak

1.14819234e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.00740225s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

12169276

over 280.00740225s

mean rate

43460.84

/s

std

27082.97

/s

p5

23171.25

/s

p95

99979.57

/s

peak

148038.62

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.003774

s

SD

0.000164

s

p50

0.003771

s

<

p95

0.003864

s

<

p99

0.003875

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.002699

s

SD

0.000214

s

p50

0.002689

s

<

p95

0.002843

s

<

p99

0.002864

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003119

s

SD

0.000146

s

p50

0.003118

s

<

p95

0.003199

s

<

p99

0.003207

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002034

s

SD

0.000196

s

p50

0.002024

s

<

p95

0.002163

s

<

p99

0.002181

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002758

s

SD

0

s

p50

0.002758

s

≤

p95

0.002758

s

≤

p99

0.002758

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000983

s

SD

3.3e-05

s

p50

0.000981

s

<

p95

0.001006

s

<

p99

0.001009

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000402

s

SD

0.000713

s

p50

0.001816

s

≤

p95

0.001816

s

≤

p99

0.001816

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000855

s

SD

0.000137

s

p50

0.000845

s

<

p95

0.000944

s

<

p99

0.000953

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001591

s

SD

7.1e-05

s

p50

0.001588

s

<

p95

0.001632

s

<

p99

0.001637

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000124

s

SD

1.2e-05

s

p50

0.000122

s

<

p95

0.000131

s

<

p99

0.000132

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.29

s

<

mean

147.29

s

SD

83.67

s

<

p95

277.29

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.007388525s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001299

s

SD

0

s

p50

0.001299

s

≤

p95

0.001299

s

≤

p99

0.001299

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001743

s

SD

0.000195

s

p50

0.001728

s

<

p95

0.001845

s

<

p99

0.001848

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.035488

s

SD

0.012427

s

p50

0.032321

s

<

p95

0.046579

s

<

p99

0.047663

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.003698

s

SD

0

s

p50

0.003698

s

≤

p95

0.003698

s

≤

p99

0.003698

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000726

s

SD

0

s

p50

0.000726

s

≤

p95

0.000726

s

≤

p99

0.000726

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002821

s

SD

0.000996

s

p50

0.002591

s

<

p95

0.003703

s

<

p99

0.003783

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001306

s

SD

0.000188

s

p50

0.000924

s

<

p95

0.001361

s

<

p99

0.001371

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000366

s

SD

2.6e-05

s

p50

0.000368

s

<

p95

0.000381

s

≤

p99

0.000381

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

67.97

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.02

KiB/s

<

mean

868.41

KiB/s

SD

1.94

MiB/s

<

p95

7.08

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.40

KiB/s

<

mean

36.69

KiB/s

SD

37.43

KiB/s

<

p95

149.11

KiB/s

Total bytes received

Total bytes received on primary network interface

count

245.94

MiB

mean

868.41

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.39

MiB

mean

36.69

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 97.6%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

82.14

%

<

mean

87.25

%

SD

3.84

%

<

p95

96.57

%

System

19.28

%

CPU percentiles

CPU usage percentiles

p50

86.79

%

p95

96.57

%

p99

97.60

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

300.00

s

Memory used percentage

Percentage of memory used

p5

9.62

%

<

mean

10.27

%

SD

0.53

%

<

p95

11.59

%

Memory available percentage

Percentage of available memory

p5

88.41

%

<

mean

89.73

%

SD

0.53

%

<

p95

90.38

%

Max host memory used

Maximum memory usage percentage across all hosts

max

10.27

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

0.85

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

6.00

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

7.03

5 min

5.34

15 min

2.88

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

40.75

%

<

mean

48.6373

%

SD

7.4151

%

<

p95

62.94

%

60 second average

43.4753

%

300 second average

26.5247

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.02

%

<

mean

24.75

%

SD

8.24

%

<

p95

44.82

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.66

KiB

<

mean

249.80

KiB

SD

126.81

KiB

<

p95

446.19

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

24.76

threads

SD

9.25

threads

<

p95

34

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

56.32

file descriptors

SD

21.17

file descriptors

<

p95

74

file descriptors

Write validated

↑ Back to index

Creates an entry, updates it and links the create to the update, then repeats. Each of the three actions is validated by the included hApp.

Started

Thu, 08 Oct 2026 08:08:33 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_37734232741_1

Write timing

The time it takes to complete the zome function call that writes the entry, updates it, and links to the update.

mean

0.086222

s

SD

0.035645

s

p50

0.092039

s

<

p95

0.137145

s

<

p99

0.200789

s

Write rate

The number of write zome function calls that complete per second.

mean

44.81

/s

Read timing

The time it takes to complete the zome function call that reads the data.

mean

0.129049

s

SD

0.048168

s

p50

0.135863

s

<

p95

0.196367

s

<

p99

0.304013

s

Read rate

The number of read zome function calls that complete per second.

mean

44.78

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.006128

s

SD

0.002054

s

p50

0.005707

s

<

p95

0.008848

s

<

p99

0.0091

s

WASM usage: 'validated::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

40818777

over 270.003918577s

mean rate

151178.73

/s

std

56171.25

/s

p5

102959.58

/s

p95

275666.07

/s

peak

335450.28

/s

WASM usage: 'validated::update_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

48309480

over 270.003918577s

mean rate

178921.72

/s

std

66613.89

/s

p5

121754.75

/s

p95

329916.2

/s

peak

396686.39

/s

WASM usage: 'validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

79284268

over 280.003957902s

mean rate

283154.59

/s

std

103080.46

/s

p5

191566.51

/s

p95

518200.15

/s

peak

623254.9

/s

WASM usage: 'validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

715577766

over 270.003918577s

mean rate

2.6502542e+06

/s

std

984880.36

/s

p5

1.8038239e+06

/s

p95

4.8830941e+06

/s

peak

5.86796315e+06

/s

Zome call duration: 'validated::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004653

s

SD

0.00025

s

p50

0.004662

s

<

p95

0.004821

s

<

p99

0.004838

s

Zome call duration: 'validated::update_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.011965

s

SD

0.002388

s

p50

0.011502

s

<

p95

0.013693

s

<

p99

0.01379

s

WASM call duration: 'validated::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003905

s

SD

0.000234

s

p50

0.003915

s

<

p95

0.004054

s

<

p99

0.004072

s

WASM call duration: 'validated::update_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.011233

s

SD

0.002375

s

p50

0.010763

s

<

p95

0.012966

s

<

p99

0.013066

s

WASM call duration: 'validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001078

s

SD

4.6e-05

s

p50

0.001082

s

<

p95

0.001105

s

<

p99

0.001108

s

WASM call duration: 'validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00414

s

SD

0.000356

s

p50

0.004093

s

<

p95

0.00435

s

<

p99

0.004362

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000513

s

SD

4.3e-05

s

p50

0.000507

s

<

p95

0.000544

s

<

p99

0.000548

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000457

s

SD

1.6e-05

s

p50

0.00046

s

<

p95

0.000471

s

<

p99

0.000472

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.005638

s

SD

0.001591

s

p50

0.005272

s

<

p95

0.006921

s

<

p99

0.007028

s

Host function call duration: '__hc__update_1'

Duration of this host function call invoked from within WASM.

mean

0.007641

s

SD

0.002293

s

p50

0.00713

s

<

p95

0.009448

s

<

p99

0.009572

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001818

s

SD

0.000125

s

p50

0.001851

s

<

p95

0.0021

s

<

p99

0.002144

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000137

s

SD

5e-06

s

p50

0.000138

s

<

p95

0.000142

s

<

p99

0.00016

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.91

s

<

mean

146.91

s

SD

83.67

s

<

p95

276.91

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.003935762s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.003321

s

SD

0

s

p50

0.003321

s

≤

p95

0.003321

s

≤

p99

0.003321

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.002004

s

SD

0.000279

s

p50

0.002019

s

<

p95

0.002151

s

<

p99

0.002192

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.052747

s

SD

0.018678

s

p50

0.047004

s

<

p95

0.067152

s

<

p99

0.068132

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.003006

s

SD

0

s

p50

0.003006

s

≤

p95

0.003006

s

≤

p99

0.003006

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000694

s

SD

0

s

p50

0.000694

s

≤

p95

0.000694

s

≤

p99

0.000694

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.0038

s

SD

0.001446

s

p50

0.0034

s

<

p95

0.004899

s

<

p99

0.004977

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002026

s

SD

0.000174

s

p50

0.001682

s

<

p95

0.002087

s

<

p99

0.002093

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000388

s

SD

8.7e-05

s

p50

0.000389

s

<

p95

0.000407

s

<

p99

0.000837

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

66.29

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

21.05

KiB/s

<

mean

690.45

KiB/s

SD

1.54

MiB/s

<

p95

4.50

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.58

KiB/s

<

mean

35.65

KiB/s

SD

31.40

KiB/s

<

p95

101.40

KiB/s

Total bytes received

Total bytes received on primary network interface

count

195.54

MiB

mean

690.45

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.10

MiB

mean

35.65

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 97.6%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

72.21

%

<

mean

85.18

%

SD

7.31

%

<

p95

96.58

%

System

18.89

%

CPU percentiles

CPU usage percentiles

p50

86.66

%

p95

96.58

%

p99

97.60

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

237.93

s

Memory used percentage

Percentage of memory used

p5

9.19

%

<

mean

10.38

%

SD

0.92

%

<

p95

11.62

%

Memory available percentage

Percentage of available memory

p5

88.38

%

<

mean

89.62

%

SD

0.92

%

<

p95

90.81

%

Max host memory used

Maximum memory usage percentage across all hosts

max

10.38

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-1.12

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.18

MiB/s

Disk write throughput

Disk write throughput in MiB/s

6.31

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

7.32

5 min

5.89

15 min

3.31

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

23.18

%

<

mean

46.3173

%

SD

11.8498

%

<

p95

62.94

%

60 second average

47.2093

%

300 second average

32.4567

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.05

%

<

mean

22.73

%

SD

7.27

%

<

p95

37.49

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.60

KiB

<

mean

285.19

KiB

SD

234.97

KiB

<

p95

513.41

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

32.62

threads

SD

37.35

threads

<

p95

73

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

73.03

file descriptors

SD

80.45

file descriptors

<

p95

165

file descriptors

Write Validated `must_get_agent_activity`

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started

Thu, 08 Oct 2026 09:03:32 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

must_get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_must_get_agent_activity_37734232741_1

Highest observed action_seq

The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

1510.43

Max

Highest chain head value observed for any single write agent.

1864

Write agents observed

7

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

57.157704

s

SD

59.470628

s

Max

Highest per-partition mean latency

max mean

453.183716

s

Min

Lowest per-partition mean latency

min mean

-882.238231

s

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

2.98

/s

Max

Highest per-partition mean rate

max

7.5

/s

Min

Lowest per-partition mean rate

min

1.6

/s

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

0.290605

s

SD

0.349441

s

Max

Highest per-partition mean latency

max mean

0.885703

s

Min

Lowest per-partition mean latency

min mean

0.088951

s

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

3973

Agents affected

20 / 20

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

199

Max

Highest count in any single partition

781

Min

Lowest count in any single partition

3

Final error count

The total number of all types of error accumulated over the run by all agents.

64575

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.022244

s

SD

0.016525

s

p50

0.000873

s

<

p95

0.04732

s

<

p99

0.067658

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

1

over 731.42095451s

mean rate

0.27

/s

std

0.35

/s

p5

0

/s

p95

0.71

/s

peak

0.71

/s

WASM usage: 'validated_must_get_agent_activity::announce_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 882.883467587s

Not enough time data to show a trend.

WASM usage: 'validated_must_get_agent_activity::create_sample_entries_batch'

Metered usage count of the WASM ribosome for this zome function.

total

2390721691

over 882.883467587s

mean rate

3.766578434e+07

/s

std

4.843867228e+07

/s

p5

0

/s

p95

1.474808452e+08

/s

peak

2.361916267e+08

/s

WASM usage: 'validated_must_get_agent_activity::create_validated_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

13693369

over 880.744465991s

mean rate

6.708792881e+07

/s

std

9.453984819e+07

/s

p5

48448.19

/s

p95

2.5371443784e+08

/s

peak

3.263551571e+08

/s

WASM usage: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Metered usage count of the WASM ribosome for this zome function.

total

351668457

over 880.744465991s

mean rate

1.0939368187e+09

/s

std

1.98366010441e+09

/s

p5

0

/s

p95

7.48907856438e+09

/s

peak

7.96581781893e+09

/s

WASM usage: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

630921

over 889.235366071s

mean rate

2.52579878e+06

/s

std

5.07103055e+06

/s

p5

0

/s

p95

1.740485762e+07

/s

peak

1.967151735e+07

/s

WASM usage: 'validated_must_get_agent_activity_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

302268276

over 889.235366071s

mean rate

3.3991354731e+08

/s

std

5.986314215e+08

/s

p5

2.72634875e+06

/s

p95

1.9905821131e+09

/s

peak

2.75442411864e+09

/s

WASM usage: 'validated_must_get_agent_activity_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

1682594105

over 889.235366071s

mean rate

5.96955598233e+10

/s

std

1.42940572017e+11

/s

p5

1.190184252e+07

/s

p95

3.25376473594e+11

/s

peak

1.11363840749e+12

/s

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.004289

s

SD

0.001749

s

p50

0.003719

s

<

p95

0.008249

s

≤

p99

0.008249

s

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.068197

s

SD

0.025176

s

p50

0.055865

s

<

p95

0.10427

s

<

p99

0.115564

s

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004869

s

SD

0.002024

s

p50

0.004741

s

<

p95

0.00975

s

<

p99

0.010558

s

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.006822

s

SD

0.006547

s

p50

0.005836

s

<

p95

0.016833

s

<

p99

0.037391

s

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007312

s

SD

0.011176

s

p50

0.005622

s

<

p95

0.05416

s

≤

p99

0.05416

s

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003585

s

SD

0.001573

s

p50

0.003048

s

<

p95

0.007134

s

≤

p99

0.007134

s

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.067305

s

SD

0.025008

s

p50

0.05487

s

<

p95

0.103212

s

<

p99

0.114757

s

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004126

s

SD

0.001862

s

p50

0.003926

s

<

p95

0.008887

s

<

p99

0.009368

s

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004497

s

SD

0.002772

s

p50

0.004753

s

<

p95

0.012521

s

<

p99

0.012911

s

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006319

s

SD

0.011222

s

p50

0.004633

s

<

p95

0.053723

s

≤

p99

0.053723

s

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001127

s

SD

0.000444

s

p50

0.001068

s

<

p95

0.002165

s

<

p99

0.002328

s

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.0097

s

SD

0.008093

s

p50

0.00819

s

<

p95

0.024251

s

<

p99

0.040117

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2e-05

s

SD

5e-06

s

p50

2e-05

s

<

p95

2.9e-05

s

≤

p99

2.9e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000442

s

SD

0.000229

s

p50

0.000395

s

<

p95

0.000808

s

<

p99

0.001396

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000506

s

SD

0.000246

s

p50

0.000319

s

<

p95

0.001139

s

≤

p99

0.001139

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000728

s

SD

0.008197

s

p50

0.000545

s

<

p95

0.001343

s

<

p99

0.051245

s

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.025185

s

SD

0.016842

s

p50

0.020601

s

<

p95

0.05451

s

<

p99

0.079725

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.025974

s

SD

0.012826

s

p50

0.018393

s

<

p95

0.043357

s

<

p99

0.051694

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1.1e-05

s

SD

8e-06

s

p50

7e-06

s

<

p95

1.7e-05

s

<

p99

4.2e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001772

s

SD

0.001049

s

p50

0.001963

s

<

p95

0.004537

s

<

p99

0.006222

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.00016

s

SD

8.8e-05

s

p50

0.000126

s

<

p95

0.000311

s

<

p99

0.000512

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47

s

<

mean

446.58

s

SD

256.61

s

<

p95

847.02

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

41767

over 889.235339074s

mean rate

61198.86

/s

std

134853.48

/s

p5

219.47

/s

p95

349880.01

/s

peak

756539.79

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

28.402405

s

SD

23.594467

s

p50

22.671499

s

<

p95

60.159239

s

<

p99

137.339454

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

2.008113

s

SD

1.535167

s

p50

2.007236

s

<

p95

5.461628

s

<

p99

7.467008

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.080293

s

SD

0.177865

s

p50

0.10589

s

<

p95

0.504577

s

<

p99

0.936639

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.164013

s

SD

0.106711

s

p50

0.086014

s

<

p95

0.336251

s

<

p99

0.50384

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.352968

s

SD

5.417566

s

p50

0.600224

s

<

p95

16.266526

s

<

p99

25.868609

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

17.920008

s

SD

21.919407

s

p50

14.202605

s

<

p95

61.617263

s

<

p99

155.141523

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003425

s

SD

0.001832

s

p50

0.002917

s

<

p95

0.005963

s

<

p99

0.008603

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.008402

s

SD

0.006073

s

p50

0.00182

s

<

p95

0.020675

s

<

p99

0.026351

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000352

s

SD

0.000106

s

p50

0.000345

s

<

p95

0.000548

s

<

p99

0.000616

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.085334

s

SD

28.139293

s

p50

0.086692

s

<

p95

60.002236

s

<

p99

60.045018

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.219328

s

SD

0

s

p50

0.219328

s

≤

p95

0.219328

s

≤

p99

0.219328

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.142751

s

SD

2.151452

s

p50

0.13546

s

<

p95

0.24064

s

<

p99

15.077093

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

44.02

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

50.26

KiB/s

<

mean

99.57

KiB/s

SD

215.33

KiB/s

<

p95

161.95

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

18.33

KiB/s

<

mean

81.03

KiB/s

SD

69.08

KiB/s

<

p95

207.82

KiB/s

Total bytes received

Total bytes received on primary network interface

count

2.28

GiB

mean

2.63

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.86

GiB

mean

2.14

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.4%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (77.2% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 46% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

12.77

%

<

mean

56.46

%

SD

25.75

%

<

p95

96.74

%

System

12.45

%

CPU percentiles

CPU usage percentiles

p50

60.11

%

p95

96.74

%

p99

98.40

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

19

hosts

mean time

270.37

s

Memory used percentage

Percentage of memory used

p5

8.69

%

<

mean

20.69

%

SD

15.28

%

<

p95

39.66

%

Memory available percentage

Percentage of available memory

p5

60.35

%

<

mean

79.31

%

SD

15.28

%

<

p95

91.32

%

Max host memory used

Maximum memory usage percentage across all hosts

max

87.78

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

77.24

%

Memory growth rate

Rate of memory growth over time

growth

0.19

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.50

MiB/s

Disk write throughput

Disk write throughput in MiB/s

81.59

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/10

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.26

5 min

3.14

15 min

3.76

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

46.43

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

16.7014

%

SD

19.9971

%

<

p95

60.6

%

60 second average

15.8064

%

300 second average

15.2942

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

6.738

%

SD

11.8569

%

<

p95

30.79

%

60 second average

6.4968

%

300 second average

6.4850

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.1688

%

SD

6.2108

%

<

p95

15.99

%

60 second average

3.1429

%

300 second average

3.3142

%

Holochain process CPU usage

CPU usage by Holochain process

p5

5.16

%

<

mean

49.87

%

SD

26.22

%

<

p95

95.23

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

379.44

KiB

<

mean

464.46

KiB

SD

59.51

KiB

<

p95

566.79

KiB

Holochain process threads

Number of threads in Holochain process

p5

32

threads

<

mean

37.85

threads

SD

10.64

threads

<

p95

53

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

73

file descriptors

<

mean

79.03

file descriptors

SD

17.37

file descriptors

<

p95

95

file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
full: A full-arc conductor that is just here to serve entries to zero arc nodes.

Started

Thu, 08 Oct 2026 10:21:35 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_and_read_37734232741_1

Create rate

The number of timed entries created by the zero-arc write node(s) per second.

Mean

mean

63.81

/s

Max

Highest per-partition mean rate

max

81.11

/s

Min

Lowest per-partition mean rate

min

49.05

/s

Fetch lag timing

For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.

Mean

mean

327.512804

s

SD

160.907163

s

Max

Highest per-partition mean latency

max mean

659.197489

s

Min

Lowest per-partition mean latency

min mean

119.082837

s

Fetch lag rate

The number of entries per second fetched by zero-arc read nodes.

Mean

mean

129.57

/s

Max

Highest per-partition mean rate

max

295.43

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc read nodes.

0.0203

Entry created count

Total number of entries created by zero-arc write nodes, by agent.

Total

70544

Agents affected

12 / 12

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

5879

Max

Highest count in any single partition

7617

Min

Lowest count in any single partition

4489

Received count

Total number of entries fetched by zero-arc read nodes, by agent.

Total

15729

Agents affected

11 / 11

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

1430

Max

Highest count in any single partition

2998

Min

Lowest count in any single partition

346

Open connections

The number of currently open connections to other conductors.

full-arc

p5

24

<

mean

27.19

SD

3.3

<

p95

28

zero-arc

p5

4

<

mean

9.46

SD

4.07

<

p95

17

Error count

The number of errors accumulated across all nodes.

128

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.007746

s

SD

0.087339

s

p50

0.032112

s

<

p95

0.160128

s

<

p99

0.445754

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 370.008628966s

Not enough time data to show a trend.

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

398020900

over 877.554645901s

mean rate

9.278874676e+07

/s

std

4.346580477e+07

/s

p5

3.379710389e+07

/s

p95

1.9287834561e+08

/s

peak

2.2655173003e+08

/s

WASM usage: 'timed::get_timed_entries_network'

Metered usage count of the WASM ribosome for this zome function.

total

132514982

over 878.06716627s

mean rate

8.56688081968e+10

/s

std

1.9194454919e+11

/s

p5

1.404342411e+07

/s

p95

5.54066436354e+11

/s

peak

9.24077006683e+11

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 888.172949162s

Not enough time data to show a trend.

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004941

s

SD

0.001615

s

p50

0.004995

s

<

p95

0.009435

s

<

p99

0.009506

s

Zome call duration: 'timed::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

1.080589

s

SD

4.947407

s

p50

2.948759

s

<

p95

16.325229

s

≤

p99

16.325229

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.0044

s

SD

0.001532

s

p50

0.004408

s

<

p95

0.0087

s

<

p99

0.008771

s

WASM call duration: 'timed::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

1.079827

s

SD

4.947362

s

p50

2.947658

s

<

p95

16.324202

s

≤

p99

16.324202

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000945

s

SD

0.000409

s

p50

0.000975

s

<

p95

0.001777

s

<

p99

0.002064

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000259

s

SD

7.6e-05

s

p50

0.000254

s

<

p95

0.000451

s

<

p99

0.000471

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000255

s

SD

8.3e-05

s

p50

0.000268

s

<

p95

0.000454

s

<

p99

0.000476

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.007565

s

SD

0.056389

s

p50

0.031455

s

<

p95

0.154853

s

<

p99

0.187907

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.066168

s

SD

0.449444

s

p50

0.159795

s

<

p95

1.638872

s

<

p99

1.949416

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001392

s

SD

0.000569

s

p50

0.001439

s

<

p95

0.002711

s

<

p99

0.002766

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.9e-05

s

SD

4.5e-05

s

p50

9.7e-05

s

<

p95

0.000209

s

≤

p99

0.000209

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.08

s

<

mean

443.87

s

SD

255.3

s

<

p95

842.28

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

4174

over 888.172951006s

mean rate

32840.74

/s

std

337550.84

/s

p5

0

/s

p95

46752.23

/s

peak

8.03259248e+06

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

42.295515

s

SD

31.686921

s

p50

0

s

<

p95

117.233136

s

<

p99

167.381767

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.189211

s

SD

0.299849

s

p50

0.001355

s

<

p95

0.341066

s

<

p99

1.975485

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

1e-06

s

p50

0

s

≤

p95

0

s

<

p99

3e-06

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.033814

s

SD

0.417013

s

p50

0.001418

s

<

p95

1.156876

s

<

p99

1.482743

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.096719

s

SD

0.044164

s

p50

0.018829

s

<

p95

0.131023

s

<

p99

0.158211

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

13.408897

s

SD

11.256273

s

p50

0.003174

s

<

p95

27.072425

s

<

p99

58.561864

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

11.562676

s

SD

15.254398

s

p50

0.002555

s

<

p95

50.046356

s

<

p99

72.425823

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001244

s

SD

0.000911

s

p50

0.001167

s

<

p95

0.002589

s

<

p99

0.002883

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.004556

s

SD

0.006552

s

p50

0.001444

s

<

p95

0.016812

s

<

p99

0.03862

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000282

s

SD

0.00039

s

p50

0.000239

s

<

p95

0.000625

s

<

p99

0.002324

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.165512

s

SD

29.733284

s

p50

0.221237

s

<

p95

60.001981

s

<

p99

60.012932

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.036324

s

SD

16.091342

s

p50

0.131429

s

<

p95

60.000452

s

≤

p99

60.000452

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.110438

s

SD

0.076195

s

p50

0.107911

s

<

p95

0.162231

s

<

p99

0.393132

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

31.91

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

996

B/s

<

mean

256.98

KiB/s

SD

429.97

KiB/s

<

p95

956.93

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

840

B/s

<

mean

230.90

KiB/s

SD

340.63

KiB/s

<

p95

596.78

KiB/s

Total bytes received

Total bytes received on primary network interface

count

5.86

GiB

mean

6.75

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.27

GiB

mean

6.06

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 96.3%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (76.1% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 29% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.88

%

<

mean

42.52

%

SD

33.58

%

<

p95

94.89

%

System

10.61

%

CPU percentiles

CPU usage percentiles

p50

37.35

%

p95

94.89

%

p99

96.35

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

6

hosts

mean time

829.21

s

Memory used percentage

Percentage of memory used

p5

6.87

%

<

mean

18.47

%

SD

14.99

%

<

p95

34.91

%

Memory available percentage

Percentage of available memory

p5

65.09

%

<

mean

81.53

%

SD

14.99

%

<

p95

93.13

%

Max host memory used

Maximum memory usage percentage across all hosts

max

85.10

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

76.10

%

Memory growth rate

Rate of memory growth over time

growth

0.24

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.60

MiB/s

Disk write throughput

Disk write throughput in MiB/s

76.55

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/10

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.63

5 min

2.66

15 min

2.74

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

28.57

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

13.6818

%

SD

19.2056

%

<

p95

54.43

%

60 second average

13.3822

%

300 second average

12.6771

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0004

%

SD

0.02

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0004

%

SD

0.02

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.5839

%

SD

8.236

%

<

p95

21.78

%

60 second average

3.7070

%

300 second average

3.9531

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.5188

%

SD

6.7192

%

<

p95

17.86

%

60 second average

2.6567

%

300 second average

2.9030

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.33

%

<

mean

36.09

%

SD

31.97

%

<

p95

92.78

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

230.94

KiB

<

mean

312.08

KiB

SD

65.05

KiB

<

p95

435.61

KiB

Holochain process threads

Number of threads in Holochain process

p5

25

threads

<

mean

32.29

threads

SD

5.51

threads

<

p95

40

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

58

file descriptors

<

mean

71.32

file descriptors

SD

10.96

file descriptors

<

p95

85

file descriptors

Zero-Arc Create Data

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 08 Oct 2026 09:34:45 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data--high_zero_37734232741_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

64.09

/s

Max

Highest per-partition mean rate

max

81.41

/s

Min

Lowest per-partition mean rate

min

50.56

/s

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

4.076518

s

SD

0

s

Max

Highest per-partition mean latency

max mean

4.076518

s

Min

Lowest per-partition mean latency

min mean

4.076518

s

Not enough time data to show a trend.

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

0

/s

Max

Highest per-partition mean rate

max

0

/s

Min

Lowest per-partition mean rate

min

0

/s

Not enough time data to show a trend.

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

158600

Agents affected

27 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

5874

Max

Highest count in any single partition

7677

Min

Lowest count in any single partition

4608

Received count

Total number of entries received by full-arc nodes, by agent.

Total

1

Agents affected

1 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

1

Max

Highest count in any single partition

1

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

0

<

mean

20.54

SD

10.41

<

p95

28

zero-arc

p5

2

<

mean

9.03

SD

5.44

<

p95

19

Error count

The number of errors accumulated across all nodes.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000507

s

SD

0.000372

s

p50

0.00019

s

<

p95

0.000997

s

<

p99

0.001006

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

3

over 770.011962607s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

407500680

over 874.242822971s

mean rate

2.3486087088e+08

/s

std

3.3031679922e+08

/s

p5

4.61899276e+06

/s

p95

9.8696938574e+08

/s

peak

5.28431517971e+09

/s

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

454859455

over 885.01997093s

mean rate

1.695767201e+07

/s

std

8.00843844e+06

/s

p5

8.56704944e+06

/s

p95

3.340530458e+07

/s

peak

3.759122732e+07

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

54208026

over 888.579982751s

mean rate

3.927533104e+07

/s

std

4.956897798e+07

/s

p5

539215.92

/s

p95

1.4332462026e+08

/s

peak

8.3224098302e+08

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004958

s

SD

0.001176

s

p50

0.00487

s

<

p95

0.008069

s

<

p99

0.008303

s

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.003891

s

SD

0.002439

s

p50

0.0041

s

<

p95

0.007814

s

<

p99

0.007882

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004394

s

SD

0.001089

s

p50

0.004284

s

<

p95

0.007245

s

<

p99

0.007583

s

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003326

s

SD

0.002116

s

p50

0.003203

s

<

p95

0.00678

s

<

p99

0.006847

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000929

s

SD

0.000283

s

p50

0.000918

s

<

p95

0.001637

s

<

p99

0.001786

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000265

s

SD

5.8e-05

s

p50

0.000254

s

<

p95

0.00035

s

<

p99

0.000459

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.00026

s

SD

6.1e-05

s

p50

0.000264

s

<

p95

0.000375

s

<

p99

0.000451

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.00144

s

SD

0.000138

s

p50

0.001447

s

<

p95

0.001513

s

<

p99

0.001519

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000328

s

SD

0.000226

s

p50

0.000205

s

<

p95

0.000692

s

<

p99

0.0007

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001397

s

SD

0.000409

s

p50

0.00135

s

<

p95

0.002392

s

<

p99

0.002779

s

Post-commit duration

Time spent executing post-commit workflows.

mean

9.9e-05

s

SD

4.4e-05

s

p50

9.6e-05

s

<

p95

0.000172

s

<

p99

0.000262

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.08

s

<

mean

445.54

s

SD

256.04

s

<

p95

846.17

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 888.579924938s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0.000343

s

SD

0.330284

s

p50

0

s

≤

p95

0

s

<

p99

1.793516

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.093571

s

SD

0.094148

s

p50

0.001425

s

<

p95

0.236203

s

<

p99

0.641837

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.008835

s

SD

0.272662

s

p50

0.000833

s

<

p95

1.047148

s

<

p99

1.222418

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.093602

s

SD

0.039669

s

p50

0.080756

s

<

p95

0.140439

s

<

p99

0.158347

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

10.166714

s

SD

16.345072

s

p50

0.00323

s

<

p95

16.462938

s

<

p99

132.78794

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002687

s

SD

0.00163

s

p50

0.00234

s

<

p95

0.004923

s

<

p99

0.007396

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.004615

s

SD

0.001923

s

p50

0.004666

s

<

p95

0.007089

s

<

p99

0.008087

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.005372

s

SD

0.008259

s

p50

0.001431

s

<

p95

0.026693

s

<

p99

0.039888

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000211

s

SD

5.6e-05

s

p50

0.000205

s

<

p95

0.000351

s

<

p99

0.000382

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

4.269452

s

SD

27.699478

s

p50

0.217165

s

<

p95

60.001037

s

≤

p99

60.001037

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.021818

s

SD

0

s

p50

0.021818

s

≤

p95

0.021818

s

≤

p99

0.021818

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

43.19

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

7.52

KiB/s

<

mean

87.73

KiB/s

SD

365.89

KiB/s

<

p95

108.57

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

17.43

KiB/s

<

mean

73.35

KiB/s

SD

70.39

KiB/s

<

p95

192.32

KiB/s

Total bytes received

Total bytes received on primary network interface

count

2.01

GiB

mean

2.31

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.68

GiB

mean

1.93

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 96.5%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (75.2% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 29% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

19.56

%

<

mean

55.19

%

SD

24.12

%

<

p95

94.4

%

System

12.00

%

CPU percentiles

CPU usage percentiles

p50

55.87

%

p95

94.40

%

p99

96.55

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

11

hosts

mean time

442.19

s

Memory used percentage

Percentage of memory used

p5

7.25

%

<

mean

18.57

%

SD

15.11

%

<

p95

38.08

%

Memory available percentage

Percentage of available memory

p5

61.92

%

<

mean

81.43

%

SD

15.11

%

<

p95

92.75

%

Max host memory used

Maximum memory usage percentage across all hosts

max

85.71

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

75.24

%

Memory growth rate

Rate of memory growth over time

growth

0.26

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.94

MiB/s

Disk write throughput

Disk write throughput in MiB/s

96.40

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/10

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.87

5 min

2.93

15 min

3.13

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

28.57

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

10.126

%

SD

12.4357

%

<

p95

31.99

%

60 second average

10.1861

%

300 second average

11.0711

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

5.1224

%

SD

9.8565

%

<

p95

27.84

%

60 second average

5.2134

%

300 second average

5.4556

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

4.2793

%

SD

8.5344

%

<

p95

23.62

%

60 second average

4.3716

%

300 second average

4.4620

%

Holochain process CPU usage

CPU usage by Holochain process

p5

17.6

%

<

mean

49.6

%

SD

23.05

%

<

p95

93.56

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

238.44

KiB

<

mean

321.93

KiB

SD

65.21

KiB

<

p95

445.39

KiB

Holochain process threads

Number of threads in Holochain process

p5

32

threads

<

mean

34.7

threads

SD

5.51

threads

<

p95

46

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

69

file descriptors

<

mean

76.38

file descriptors

SD

9.46

file descriptors

<

p95

83

file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 08 Oct 2026 10:06:00 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data_validated--high_zero_37734232741_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

60.35

/s

Max

Highest per-partition mean rate

max

83.36

/s

Min

Lowest per-partition mean rate

min

47.14

/s

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

186.510716

s

SD

139.753701

s

Max

Highest per-partition mean latency

max mean

324.714724

s

Min

Lowest per-partition mean latency

min mean

48.306708

s

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

59.21

/s

Max

Highest per-partition mean rate

max

118.42

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0.007

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

144517

Agents affected

27 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

5352

Max

Highest count in any single partition

7366

Min

Lowest count in any single partition

4235

Received count

Total number of entries received by full-arc nodes, by agent.

Total

2034

Agents affected

2 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

1017

Max

Highest count in any single partition

1773

Min

Lowest count in any single partition

261

Open connections

The number of currently open connections to other conductors.

full-arc

p5

21

<

mean

25.24

SD

2.75

<

p95

28

zero-arc

p5

2

<

mean

9.4

SD

5.33

<

p95

20

Error count

The number of errors accumulated across all nodes.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.001425

s

SD

0.003508

s

p50

0.007465

s

<

p95

0.012469

s

<

p99

0.015

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

1

over 795.575531643s

mean rate

95.39

/s

std

59

/s

p5

0

/s

p95

183.95

/s

peak

202.96

/s

WASM usage: 'timed_and_validated::create_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

280668672

over 871.446643477s

mean rate

4.0532257421e+08

/s

std

5.7944171676e+08

/s

p5

1.37478068e+06

/s

p95

1.77026695343e+09

/s

peak

2.14629899442e+09

/s

WASM usage: 'timed_and_validated::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

4335671734

over 870.0487424s

mean rate

6.07465458568e+09

/s

std

3.65817983449e+09

/s

p5

2.5845917321e+08

/s

p95

1.15733799967e+10

/s

peak

1.21507384722e+10

/s

WASM usage: 'timed_and_validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

165258060

over 888.085126802s

mean rate

2.269040801e+08

/s

std

3.2735488258e+08

/s

p5

382591.28

/s

p95

1.02793509942e+09

/s

peak

1.26162986776e+09

/s

WASM usage: 'timed_and_validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

1292983351

over 888.085126802s

mean rate

1.81634376822e+09

/s

std

2.56527315585e+09

/s

p5

4.04178759e+06

/s

p95

8.04864472346e+09

/s

peak

9.87147859995e+09

/s

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.005093

s

SD

0.001488

s

p50

0.005069

s

<

p95

0.008178

s

<

p99

0.011119

s

Zome call duration: 'timed_and_validated::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.185724

s

SD

0.131793

s

p50

0.17537

s

<

p95

0.45184

s

<

p99

0.482799

s

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004503

s

SD

0.001364

s

p50

0.004523

s

<

p95

0.007456

s

<

p99

0.009984

s

WASM call duration: 'timed_and_validated::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.184852

s

SD

0.131678

s

p50

0.174273

s

<

p95

0.45071

s

<

p99

0.481666

s

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000944

s

SD

0.000859

s

p50

0.00093

s

<

p95

0.002027

s

<

p99

0.005445

s

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004231

s

SD

0.003173

s

p50

0.003869

s

<

p95

0.006711

s

<

p99

0.021065

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000266

s

SD

6.7e-05

s

p50

0.000271

s

<

p95

0.000368

s

<

p99

0.000546

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000263

s

SD

7.7e-05

s

p50

0.000273

s

<

p95

0.000385

s

<

p99

0.000586

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000665

s

SD

0.001945

s

p50

0.000447

s

<

p95

0.004953

s

<

p99

0.005094

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.006421

s

SD

0.003599

s

p50

0.006471

s

<

p95

0.012943

s

<

p99

0.01358

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.008631

s

SD

0.0033

s

p50

0.007845

s

<

p95

0.01265

s

<

p99

0.015114

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001459

s

SD

0.001422

s

p50

0.001469

s

<

p95

0.002688

s

<

p99

0.010384

s

Post-commit duration

Time spent executing post-commit workflows.

mean

9.9e-05

s

SD

4.3e-05

s

p50

0.000101

s

<

p95

0.000167

s

<

p99

0.00026

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.05

s

<

mean

444.23

s

SD

255.33

s

<

p95

841.37

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 888.085183882s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

3.454444

s

SD

10.553427

s

p50

0

s

<

p95

41.741462

s

<

p99

43.293264

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

2.67039

s

SD

1.208219

s

p50

0.001284

s

<

p95

2.571359

s

<

p99

6.198753

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.005565

s

SD

0.227266

s

p50

0.000932

s

<

p95

0.515018

s

<

p99

1.27201

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.081193

s

SD

0.032555

s

p50

0.070418

s

<

p95

0.121798

s

<

p99

0.134932

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

11.052103

s

SD

6.103629

s

p50

0.003149

s

<

p95

17.678504

s

<

p99

36.212936

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

12.326912

s

SD

19.388541

s

p50

0.001948

s

<

p95

61.994822

s

<

p99

65.784211

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002049

s

SD

0.001182

s

p50

0.003143

s

<

p95

0.004276

s

<

p99

0.004765

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.006582

s

SD

0.008087

s

p50

0.001552

s

<

p95

0.025801

s

<

p99

0.041905

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000249

s

SD

0.001227

s

p50

0.000218

s

<

p95

0.000406

s

<

p99

0.007199

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

5.483838

s

SD

29.858505

s

p50

0.143664

s

<

p95

60.007315

s

≤

p99

60.007315

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.119763

s

SD

0.030177

s

p50

0.107268

s

<

p95

0.167622

s

<

p99

0.167712

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

38.96

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

12.51

KiB/s

<

mean

142.61

KiB/s

SD

396.15

KiB/s

<

p95

690.39

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

27.58

KiB/s

<

mean

121.54

KiB/s

SD

129.66

KiB/s

<

p95

222.12

KiB/s

Total bytes received

Total bytes received on primary network interface

count

3.26

GiB

mean

3.75

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

2.78

GiB

mean

3.20

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.4%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (74.9% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 29% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

20.82

%

<

mean

51.97

%

SD

22.25

%

<

p95

89.54

%

System

13.01

%

CPU percentiles

CPU usage percentiles

p50

51.36

%

p95

89.54

%

p99

95.43

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

6

hosts

mean time

588.20

s

Memory used percentage

Percentage of memory used

p5

8.99

%

<

mean

20.64

%

SD

15.49

%

<

p95

38.87

%

Memory available percentage

Percentage of available memory

p5

61.13

%

<

mean

79.36

%

SD

15.49

%

<

p95

91.01

%

Max host memory used

Maximum memory usage percentage across all hosts

max

89.28

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

74.92

%

Memory growth rate

Rate of memory growth over time

growth

0.23

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.31

MiB/s

Disk write throughput

Disk write throughput in MiB/s

85.44

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/5

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/10

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/5

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.70

5 min

2.76

15 min

2.86

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

28.57

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

10.5609

%

SD

14.0538

%

<

p95

31.02

%

60 second average

10.4610

%

300 second average

10.8187

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

4.3876

%

SD

8.7186

%

<

p95

25.46

%

60 second average

4.4417

%

300 second average

4.4129

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.3156

%

SD

6.9951

%

<

p95

19.32

%

60 second average

3.4084

%

300 second average

3.4877

%

Holochain process CPU usage

CPU usage by Holochain process

p5

21.29

%

<

mean

46.45

%

SD

20.82

%

<

p95

88.69

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

383.02

KiB

<

mean

463.61

KiB

SD

69.53

KiB

<

p95

588.87

KiB

Holochain process threads

Number of threads in Holochain process

p5

32

threads

<

mean

34.87

threads

SD

4.02

threads

<

p95

41

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

67

file descriptors

<

mean

75.8

file descriptors

SD

7.34

file descriptors

<

p95

83

file descriptors

Zome Call, Single Value

↑ Back to index

Calls a zome function that returns a fixed value. This tests the maximum performance of zome calls when the zome function is not calling into Holochain other otherwise doing any work.

Started

Thu, 08 Oct 2026 08:10:07 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zome_call_single_value_37734232741_1

Call timing

The time taken for the call to complete.

mean

0.00356

s

SD

0.00233

s

p50

0.002845

s

<

p95

0.007075

s

<

p99

0.011769

s

Call rate

The number of zome calls completed per second.

mean

2809.21

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'return_single_value::get_value'

Metered usage count of the WASM ribosome for this zome function.

total

304221663

over 270.004238666s

mean rate

1.12673152e+06

/s

std

431290.58

/s

p5

535308.13

/s

p95

1.84034725e+06

/s

peak

1.84442389e+06

/s

Zome call duration: 'return_single_value::get_value'

Duration of this zome call as measured by Holochain internally.

mean

0.001445

s

SD

0.000132

s

p50

0.001594

s

<

p95

0.001666

s

<

p99

0.001725

s

WASM call duration: 'return_single_value::get_value'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000908

s

SD

9.2e-05

s

p50

0.00101

s

<

p95

0.001067

s

<

p99

0.001125

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.8e-05

s

SD

0

s

p50

8.8e-05

s

≤

p95

8.8e-05

s

≤

p99

8.8e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.09

s

<

mean

147.09

s

SD

83.67

s

<

p95

277.09

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.004263723s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.003613

s

SD

0

s

p50

0.003613

s

≤

p95

0.003613

s

≤

p99

0.003613

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.002084

s

SD

0.000588

s

p50

0.002046

s

≤

p95

0.002046

s

<

p99

0.005266

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.006771

s

SD

0.0019

s

p50

0.007156

s

<

p95

0.008973

s

<

p99

0.014885

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.004651

s

SD

0

s

p50

0.004651

s

≤

p95

0.004651

s

≤

p99

0.004651

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.00397

s

SD

0

s

p50

0.00397

s

≤

p95

0.00397

s

≤

p99

0.00397

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

8.3e-05

s

SD

1.4e-05

s

p50

0.0001

s

<

p95

0.00011

s

<

p99

0.000113

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001951

s

SD

0.000288

s

p50

0.002131

s

≤

p95

0.002131

s

<

p99

0.003909

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000516

s

SD

2.6e-05

s

p50

0.000519

s

≤

p95

0.000519

s

≤

p99

0.000519

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

53.45

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

8.17

KiB/s

<

mean

885.66

KiB/s

SD

1.91

MiB/s

<

p95

7.14

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

5.25

KiB/s

<

mean

37.11

KiB/s

SD

38.48

KiB/s

<

p95

145.15

KiB/s

Total bytes received

Total bytes received on primary network interface

count

250.82

MiB

mean

885.66

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.51

MiB

mean

37.11

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 96.6%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

32.11

%

<

mean

71.34

%

SD

21.29

%

<

p95

94.38

%

System

17.90

%

CPU percentiles

CPU usage percentiles

p50

73.16

%

p95

94.38

%

p99

96.58

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

144.83

s

Memory used percentage

Percentage of memory used

p5

6.95

%

<

mean

9.68

%

SD

1.68

%

<

p95

12.84

%

Memory available percentage

Percentage of available memory

p5

87.16

%

<

mean

90.32

%

SD

1.68

%

<

p95

93.05

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.68

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-3.92

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.18

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.58

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.37

5 min

5.99

15 min

3.58

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

1.28

%

<

mean

34.168

%

SD

20.0877

%

<

p95

62.94

%

60 second average

40.1180

%

300 second average

34.2837

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.06

%

<

mean

22.83

%

SD

6.89

%

<

p95

37.64

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

22.62

KiB

<

mean

320.34

KiB

SD

365.14

KiB

<

p95

1.35

MiB

Holochain process threads

Number of threads in Holochain process

p5

8

threads

<

mean

46

threads

SD

70.37

threads

<

p95

236

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

102.85

file descriptors

SD

151.08

file descriptors

<

p95

508

file descriptors

App Install

App Install

Conductor Startup

DHT Sync Lag

First Call

Full-Arc Create (Validated) / Zero-Arc Read

Local Signals

Mixed Arc get_agent_activity

Mixed Arc must_get_agent_activity

Remote Call Rate

Remote Signals

Single-write, many-read

Validation Receipts

Write/get_agent_activity

Write/Query

Write/Read

Write validated

Write Validated must_get_agent_activity

Zero-Arc Create and Read

Zero-Arc Create Data

Zero-arc Create Data (Validated)

Zome Call, Single Value

Mixed Arc `get_agent_activity`

Mixed Arc `must_get_agent_activity`

Write/`get_agent_activity`

Write Validated `must_get_agent_activity`