App Install

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 01 Oct 2026 11:09:25 UTC

Duration

300s

Peer count

1

Behaviours

minimal (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_minimal_36830042121_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.412165107

s

Install app timing

The time taken to install an app.

mean

0.087423

s

SD

0.163782

s

p50

0.048764

s

<

p95

0.200207

s

<

p99

0.888566

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'callback::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000095126s

Not enough time data to show a trend.

WASM usage: 'callback_integrity::genesis_self_check_2'

Metered usage count of the WASM ribosome for this zome function.

total

119451

over 10.000095126s

mean rate

11944.99

/s

std

0

/s

p5

11944.99

/s

p95

11944.99

/s

peak

11944.99

/s

Not enough time data to show a trend.

WASM call duration: 'callback::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000983

s

SD

0.000223

s

p50

0.000946

s

<

p95

0.001921

s

≤

p99

0.001921

s

WASM call duration: 'callback_integrity::genesis_self_check_2'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000988

s

SD

4e-05

s

p50

0.001056

s

≤

p95

0.001056

s

≤

p99

0.001056

s

Post-commit duration

Time spent executing post-commit workflows.

mean

9.1e-05

s

SD

0.00011

s

p50

6.4e-05

s

<

p95

0.000334

s

<

p99

0.000701

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

6.83

s

<

mean

11.83

s

SD

5

s

<

p95

16.83

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 10.000111619s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001255

s

SD

0.000265

s

p50

0.001334

s

≤

p95

0.001334

s

≤

p99

0.001334

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001272

s

SD

3.6e-05

s

p50

0.001282

s

≤

p95

0.001282

s

≤

p99

0.001282

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004477

s

SD

7.7e-05

s

p50

0.004499

s

≤

p95

0.004499

s

≤

p99

0.004499

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002887

s

SD

7e-06

s

p50

0.002899

s

≤

p95

0.002899

s

≤

p99

0.002899

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001977

s

SD

0.000145

s

p50

0.002224

s

≤

p95

0.002224

s

≤

p99

0.002224

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

3.7e-05

s

SD

1e-06

s

p50

3.8e-05

s

≤

p95

3.8e-05

s

≤

p99

3.8e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001114

s

SD

0.000395

s

p50

0.001684

s

<

p95

0.001688

s

≤

p99

0.001688

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000287

s

SD

1.9e-05

s

p50

0.00032

s

≤

p95

0.00032

s

≤

p99

0.00032

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

54.80

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

2.59

MiB/s

<

mean

3.33

MiB/s

SD

753.73

KiB/s

<

p95

4.06

MiB/s

Not enough time data to show a trend.

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

47.00

KiB/s

<

mean

76.34

KiB/s

SD

29.34

KiB/s

<

p95

105.67

KiB/s

Not enough time data to show a trend.

Total bytes received

Total bytes received on primary network interface

count

66.54

MiB

mean

3.33

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.49

MiB

mean

76.34

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 5.15 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

57.04

%

<

mean

66.04

%

SD

9

%

<

p95

75.04

%

Not enough time data to show a trend.

System

11.24

%

CPU percentiles

CPU usage percentiles

p50

75.04

%

p95

75.04

%

p99

75.04

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

20.05

%

<

mean

26.02

%

SD

5.43

%

<

p95

33.18

%

Not enough time data to show a trend.

Memory available percentage

Percentage of available memory

p5

66.82

%

<

mean

73.98

%

SD

5.43

%

<

p95

79.95

%

Not enough time data to show a trend.

Max host memory used

Maximum memory usage percentage across all hosts

max

26.02

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

5.15

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

11.49

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.64

5 min

0.96

15 min

1.13

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

4.09

%

<

mean

10.06

%

SD

5.1792

%

<

p95

16.72

%

Not enough time data to show a trend.

60 second average

4.8500

%

300 second average

8.2233

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.77

%

SD

0.5479

%

<

p95

1.23

%

Not enough time data to show a trend.

60 second average

0.2567

%

300 second average

0.3133

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.6833

%

SD

0.4853

%

<

p95

1.08

%

Not enough time data to show a trend.

60 second average

0.2233

%

300 second average

0.2000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

22.57

%

SD

22.57

%

<

p95

45.14

%

Not enough time data to show a trend.

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

24.23

KiB

<

mean

568.59

KiB

SD

603.88

KiB

<

p95

1.55

MiB

Not enough time data to show a trend.

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

95.5

threads

SD

129.12

threads

<

p95

316

threads

Not enough time data to show a trend.

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

208.5

file descriptors

SD

281.23

file descriptors

<

p95

686

file descriptors

Not enough time data to show a trend.

App Install

↑ Back to index

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 01 Oct 2026 11:07:18 UTC

Duration

300s

Peer count

1

Behaviours

large (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_large_36830042121_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.135978503

s

Install app timing

The time taken to install an app.

mean

0.067473

s

SD

0.011411

s

p50

0.064975

s

<

p95

0.10752

s

<

p99

0.112467

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'large::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000302203s

Not enough time data to show a trend.

WASM call duration: 'large::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004109

s

SD

0.001348

s

p50

0.003497

s

<

p95

0.0072

s

<

p99

0.008716

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000152

s

SD

0.000304

s

p50

8e-05

s

<

p95

0.000409

s

<

p99

0.001606

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

7.24

s

<

mean

17.24

s

SD

8.17

s

<

p95

27.24

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 20.000422062s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001333

s

SD

0.000194

s

p50

0.001265

s

<

p95

0.001381

s

≤

p99

0.001381

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001774

s

SD

0.001317

s

p50

0.001765

s

<

p95

0.004551

s

≤

p99

0.004551

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.005764

s

SD

0.00309

s

p50

0.005719

s

<

p95

0.012242

s

≤

p99

0.012242

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002588

s

SD

0.000931

s

p50

0.002556

s

<

p95

0.004526

s

≤

p99

0.004526

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001291

s

SD

0.000177

s

p50

0.001293

s

<

p95

0.001307

s

≤

p99

0.001307

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

7.7e-05

s

SD

9e-06

s

p50

7.8e-05

s

<

p95

9.7e-05

s

≤

p99

9.7e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001678

s

SD

0.000789

s

p50

0.00239

s

<

p95

0.003379

s

≤

p99

0.003379

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000454

s

SD

6.4e-05

s

p50

0.000451

s

<

p95

0.000458

s

≤

p99

0.000458

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

71.37

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

82.25

KiB/s

<

mean

885.82

KiB/s

SD

803.57

KiB/s

<

p95

1.65

MiB/s

Not enough time data to show a trend.

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

35.27

KiB/s

<

mean

39.62

KiB/s

SD

4.35

KiB/s

<

p95

43.97

KiB/s

Not enough time data to show a trend.

Total bytes received

Total bytes received on primary network interface

count

17.30

MiB

mean

885.82

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

792.34

KiB

mean

39.62

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.5%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 6.28 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

84.98

%

<

mean

89.75

%

SD

4.77

%

<

p95

94.52

%

Not enough time data to show a trend.

System

18.38

%

CPU percentiles

CPU usage percentiles

p50

94.52

%

p95

94.52

%

p99

94.52

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

300.00

s

Memory used percentage

Percentage of memory used

p5

10.1

%

<

mean

11.88

%

SD

1.63

%

<

p95

14.05

%

Not enough time data to show a trend.

Memory available percentage

Percentage of available memory

p5

85.95

%

<

mean

88.12

%

SD

1.63

%

<

p95

89.9

%

Not enough time data to show a trend.

Max host memory used

Maximum memory usage percentage across all hosts

max

11.88

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

6.28

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

4.86

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.13

5 min

5.01

15 min

2.76

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

29.96

%

<

mean

41.2233

%

SD

8.118

%

<

p95

48.78

%

Not enough time data to show a trend.

60 second average

38.3667

%

300 second average

32.3800

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

22.26

%

<

mean

34.45

%

SD

12.19

%

<

p95

46.64

%

Not enough time data to show a trend.

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

83.24

KiB

<

mean

637.61

KiB

SD

531.24

KiB

<

p95

1.32

MiB

Not enough time data to show a trend.

Holochain process threads

Number of threads in Holochain process

p5

21

threads

<

mean

98.33

threads

SD

83.94

threads

<

p95

215

threads

Not enough time data to show a trend.

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

65

file descriptors

<

mean

221.33

file descriptors

SD

173.95

file descriptors

<

p95

464

file descriptors

Not enough time data to show a trend.

Conductor Startup

↑ Back to index

This scenario measures Holochain conductor startup time when many cells are installed. The agent starts an empty conductor, installs a configurable number of apps without enabling them (disabled cells), restarts the conductor, and then enables cells one by one — timing each enable call and periodically restarting the conductor to measure startup time as the share of enabled cells grows.

Started

Thu, 01 Oct 2026 10:56:18 UTC

Duration

900s

Peer count

30

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

conductor_startup_36830042121_1

Initial conductor startup timing

Startup time with an empty conductor before any cells are installed.

mean

2.630766

s

SD

0.774249

s

p50

2.323125

s

<

p95

4.19286

s

<

p99

4.506967

s

Disabled-cell conductor startup timing

Startup time after all configured cells are installed but remain disabled.

mean

2.120345

s

SD

0.572334

s

p50

2.442044

s

<

p95

3.356724

s

≤

p99

3.356724

s

Progressive conductor startup timing

Startup time sampled periodically as more installed cells are enabled.

mean

11.342727

s

SD

6.207949

s

p50

10.228008

s

<

p95

23.823086

s

<

p99

27.471581

s

Cell enable timing

Time to enable one previously disabled cell via the admin interface.

mean

0.138528

s

SD

0.061066

s

p50

0.130613

s

<

p95

0.229178

s

<

p99

0.311131

s

Installed cells

Number of cells installed on the conductor during setup.

200

cells

Maximum enabled cells

Highest percentage of installed cells enabled during the run.

100

%

Holochain MetricsClick to toggle visibility.

WASM usage: 'callback_integrity::genesis_self_check_2'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 303.721256166s

Not enough time data to show a trend.

WASM call duration: 'callback_integrity::genesis_self_check_2'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00126

s

SD

0.000589

s

p50

0.000915

s

<

p95

0.002277

s

<

p99

0.002639

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

0.65

s

<

mean

74.59

s

SD

92.49

s

<

p95

266.41

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 417.924724824s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

3.505458

s

SD

4.28856

s

p50

1.003766

s

<

p95

12.092067

s

<

p99

15.889136

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.00179

s

SD

0.002284

s

p50

0.001421

s

<

p95

0.004087

s

<

p99

0.005747

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.012988

s

SD

0.0191

s

p50

0.002036

s

<

p95

0.004318

s

<

p99

0.113025

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.015812

s

SD

0.140764

s

p50

0.002201

s

<

p95

0.006081

s

<

p99

0.208449

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.008257

s

SD

0.071426

s

p50

0.002223

s

<

p95

0.005322

s

<

p99

0.008289

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001452

s

SD

0.002255

s

p50

0.000891

s

<

p95

0.004254

s

<

p99

0.005845

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000115

s

SD

7.4e-05

s

p50

9.4e-05

s

<

p95

0.000266

s

<

p99

0.000324

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.007362

s

SD

0.054686

s

p50

0.001373

s

<

p95

0.003269

s

<

p99

0.014526

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000276

s

SD

0.000161

s

p50

0.000345

s

<

p95

0.000661

s

<

p99

0.000786

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.108984

s

SD

0.078982

s

p50

0.125625

s

<

p95

0.233884

s

<

p99

0.249035

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

25.50

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

69

B/s

<

mean

437.09

KiB/s

SD

1.13

MiB/s

<

p95

1.75

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

136

B/s

<

mean

79.52

KiB/s

SD

50.75

KiB/s

<

p95

143.54

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.28

GiB

mean

2.92

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

239.18

MiB

mean

544.27

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.13 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (73.4% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 4% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

2.43

%

<

mean

39.74

%

SD

26.53

%

<

p95

76.52

%

System

14.23

%

CPU percentiles

CPU usage percentiles

p50

42.56

%

p95

76.52

%

p99

79.37

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

2

hosts

mean time

80.00

s

Memory used percentage

Percentage of memory used

p5

3.9

%

<

mean

18.57

%

SD

13.72

%

<

p95

30.69

%

Memory available percentage

Percentage of available memory

p5

69.31

%

<

mean

81.43

%

SD

13.72

%

<

p95

96.1

%

Max host memory used

Maximum memory usage percentage across all hosts

max

82.82

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

73.43

%

Memory growth rate

Rate of memory growth over time

growth

1.13

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2.84

MiB/s

Disk write throughput

Disk write throughput in MiB/s

184.09

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/11

hosts

Mount Point /etc/hosts

0/11

hosts

Mount Point /etc/nomad.d/client-meta.json

0/14

hosts

Mount Point /etc/resolv.conf

0/11

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.42

5 min

1.10

15 min

0.47

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

3.70

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

14.2538

%

SD

16.9693

%

<

p95

45.82

%

60 second average

9.7731

%

300 second average

3.9068

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0148

%

SD

0.1065

%

≤

p95

0

%

60 second average

0.0120

%

300 second average

0.0037

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0122

%

SD

0.0952

%

≤

p95

0

%

60 second average

0.0104

%

300 second average

0.0032

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

12.3571

%

SD

20.7656

%

<

p95

57.81

%

60 second average

11.5885

%

300 second average

7.1198

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

11.1411

%

SD

20.1845

%

<

p95

56.74

%

60 second average

10.6794

%

300 second average

6.7387

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

17.16

%

SD

24.98

%

<

p95

67.29

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

58.05

KiB

<

mean

277.07

KiB

SD

141.93

KiB

<

p95

529.20

KiB

Holochain process threads

Number of threads in Holochain process

p5

5

threads

<

mean

460.06

threads

SD

505.99

threads

<

p95

1482

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

11

file descriptors

<

mean

1037.13

file descriptors

SD

1103.02

file descriptors

<

p95

3248

file descriptors

DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.

Started

Thu, 01 Oct 2026 11:11:44 UTC

Duration

1800s

Peer count

30

Peer count at end

27

Behaviours

record_lag (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

dht_sync_lag_36830042121_1

Create rate

The average number of created records per agent.

Mean

mean

22.55

/s

Max

Highest per-partition mean rate

max

32.47

/s

Min

Lowest per-partition mean rate

min

10.13

/s

Sync lag timing

The average time between when a record was created and when it is first seen by an agent.

Mean

mean

1145.680873

s

SD

99.20109

s

Max

Highest per-partition mean latency

max mean

1709.926516

s

Min

Lowest per-partition mean latency

min mean

27.118488

s

Sync lag rate

The average number of created records discovered per agent.

Mean

mean

81.89

/s

Max

Highest per-partition mean rate

max

243

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully received by all record_lag agents.

0.0051

Sent count

The total number of entries sent (created) by write agents, by agent.

Total

99827

Agents affected

24 / 24

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

4159

Max

Highest count in any single partition

5907

Min

Lowest count in any single partition

1845

Received count

The total number of entries received by record_lag agents, by agent.

Total

2048

Agents affected

3 / 4

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

512

Max

Highest count in any single partition

1691

Min

Lowest count in any single partition

0

Error count

The number of errors encountered during the scenario.

24

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000393

s

SD

8.3e-05

s

p50

0.000349

s

<

p95

0.000513

s

<

p99

0.000515

s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

345692448

over 1788.249064872s

mean rate

1.50364325012e+09

/s

std

8.15985688983e+09

/s

p5

6.85546778e+06

/s

p95

4.40790428436e+09

/s

peak

2.24189871015e+11

/s

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

14166795072

over 1782.680153499s

mean rate

7.84595802901e+09

/s

std

6.76194805252e+09

/s

p5

6.0956151859e+08

/s

p95

1.8245052632e+10

/s

peak

2.97602934465e+10

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

46626354

over 1788.249064872s

mean rate

1.8461023803e+08

/s

std

1.02533150396e+09

/s

p5

1.24629795e+06

/s

p95

4.7898736775e+08

/s

peak

3.02383125511e+10

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.012527

s

SD

0.015157

s

p50

0.009524

s

<

p95

0.043197

s

<

p99

0.079345

s

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.015309

s

SD

0.007599

s

p50

0.013443

s

<

p95

0.026411

s

<

p99

0.026611

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.011714

s

SD

0.015013

s

p50

0.008713

s

<

p95

0.042333

s

<

p99

0.07788

s

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.014637

s

SD

0.00784

s

p50

0.012931

s

<

p95

0.02591

s

<

p99

0.026105

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001734

s

SD

0.001232

s

p50

0.001567

s

<

p95

0.003904

s

<

p99

0.006947

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.002028

s

SD

0.00478

s

p50

0.000884

s

<

p95

0.012293

s

<

p99

0.022939

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.001958

s

SD

0.00466

s

p50

0.000882

s

<

p95

0.012068

s

<

p99

0.022565

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000364

s

SD

7.8e-05

s

p50

0.00034

s

<

p95

0.000484

s

<

p99

0.000485

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001476

s

SD

0.001009

s

p50

0.001239

s

<

p95

0.003071

s

<

p99

0.003121

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002784

s

SD

0.002291

s

p50

0.002481

s

<

p95

0.007237

s

<

p99

0.013051

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000258

s

SD

0.000419

s

p50

0.000174

s

<

p95

0.000926

s

<

p99

0.002499

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

87.65

s

<

mean

884.09

s

SD

515.91

s

<

p95

1696.96

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

19767

over 1788.249123827s

mean rate

64967.44

/s

std

149463.49

/s

p5

0

/s

p95

418371.98

/s

peak

911167.38

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

232.217884

s

SD

232.291904

s

p50

0.002809

s

<

p95

301.968749

s

<

p99

1202.332182

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.123345

s

SD

0.62302

s

p50

0.273804

s

<

p95

1.70272

s

<

p99

2.779903

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.513872

s

SD

0.451304

s

p50

0.756608

s

<

p95

1.292549

s

<

p99

1.448218

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.327954

s

SD

0.179011

s

p50

0.285503

s

<

p95

0.567989

s

<

p99

0.781696

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

17.469441

s

SD

61.686502

s

p50

30.621624

s

<

p95

179.034639

s

<

p99

304.722854

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

26.825989

s

SD

52.543975

s

p50

0.003213

s

<

p95

174.485131

s

<

p99

223.507422

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001849

s

SD

0.001078

s

p50

0.001641

s

<

p95

0.004308

s

<

p99

0.005729

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.003893

s

SD

0.007166

s

p50

0.001528

s

<

p95

0.01763

s

<

p99

0.043079

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.001671

s

SD

0.004164

s

p50

0.000643

s

<

p95

0.010948

s

<

p99

0.020815

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.47255

s

SD

27.865883

s

p50

0.624762

s

<

p95

60.002194

s

<

p99

60.005401

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.196943

s

SD

11.426514

s

p50

0.261987

s

<

p95

20.79027

s

<

p99

60.001289

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

64.62

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

4.05

KiB/s

<

mean

550.25

KiB/s

SD

513.24

KiB/s

<

p95

1.18

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

3.35

KiB/s

<

mean

516.08

KiB/s

SD

455.14

KiB/s

<

p95

1.28

MiB/s

Total bytes received

Total bytes received on primary network interface

count

23.89

GiB

mean

13.67

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

22.40

GiB

mean

12.82

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.6%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (75.9% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 66% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

28.77

%

<

mean

77.06

%

SD

22.64

%

<

p95

97.75

%

System

12.44

%

CPU percentiles

CPU usage percentiles

p50

85.52

%

p95

97.75

%

p99

98.60

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

22

hosts

mean time

1273.80

s

Memory used percentage

Percentage of memory used

p5

7.31

%

<

mean

21.53

%

SD

15.99

%

<

p95

42.01

%

Memory available percentage

Percentage of available memory

p5

57.99

%

<

mean

78.47

%

SD

15.99

%

<

p95

92.69

%

Max host memory used

Maximum memory usage percentage across all hosts

max

87.77

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

75.90

%

Memory growth rate

Rate of memory growth over time

growth

0.41

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1.33

MiB/s

Disk write throughput

Disk write throughput in MiB/s

152.53

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/15

hosts

Mount Point /etc/hosts

0/15

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/15

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.64

5 min

5.07

15 min

3.64

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

65.52

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

33.8301

%

SD

24.844

%

<

p95

70.13

%

60 second average

33.2636

%

300 second average

30.1982

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0019

%

SD

0.0695

%

≤

p95

0

%

60 second average

0.0031

%

300 second average

0.0025

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0015

%

SD

0.0561

%

≤

p95

0

%

60 second average

0.0027

%

300 second average

0.0022

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

4.2516

%

SD

10.5511

%

<

p95

27.69

%

60 second average

4.0720

%

300 second average

3.4886

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.1953

%

SD

4.4193

%

<

p95

6.54

%

60 second average

1.1332

%

300 second average

0.9948

%

Holochain process CPU usage

CPU usage by Holochain process

p5

27.83

%

<

mean

71.98

%

SD

21.97

%

<

p95

96.11

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

320.45

KiB

<

mean

515.61

KiB

SD

152.90

KiB

<

p95

747.98

KiB

Holochain process threads

Number of threads in Holochain process

p5

35

threads

<

mean

45.23

threads

SD

26.62

threads

<

p95

67

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

79

file descriptors

<

mean

91.71

file descriptors

SD

25.69

file descriptors

<

p95

111

file descriptors

First Call

↑ Back to index

This scenario installs a simple app which implements the init callback, and measures the time elapsed between start of install and calling init. It will uninstall the app it installed so that it can re-install it on the next iteration. This is required to re-run the initialisation callback.

Started

Thu, 01 Oct 2026 10:49:10 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

first_call_36830042121_1

Zome call timing

The time elapsed between start of install and calling init.

mean

0.011517

s

SD

0.002397

s

p50

0.01081

s

<

p95

0.016059

s

<

p99

0.018605

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000387259s

Not enough time data to show a trend.

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000387259s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

242592

over 10.000387259s

mean rate

24258.26

/s

std

0

/s

p5

24258.26

/s

p95

24258.26

/s

peak

24258.26

/s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.005031

s

SD

0.000124

s

p50

0.005058

s

≤

p95

0.005058

s

≤

p99

0.005058

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004268

s

SD

0.000742

s

p50

0.003949

s

<

p95

0.005971

s

<

p99

0.006198

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001367

s

SD

0.000263

s

p50

0.001299

s

<

p95

0.002464

s

≤

p99

0.002464

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00121

s

SD

4.7e-05

s

p50

0.001221

s

≤

p95

0.001221

s

≤

p99

0.001221

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000316

s

SD

3.9e-05

s

p50

0.000352

s

<

p95

0.000363

s

≤

p99

0.000363

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002226

s

SD

7.5e-05

s

p50

0.002242

s

≤

p95

0.002242

s

≤

p99

0.002242

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.7e-05

s

SD

2.5e-05

s

p50

8.1e-05

s

<

p95

0.000167

s

<

p99

0.000172

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

6.84

s

<

mean

11.84

s

SD

5

s

<

p95

16.84

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 10.000024672s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000963

s

SD

0.000193

s

p50

0.001298

s

≤

p95

0.001298

s

≤

p99

0.001298

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001357

s

SD

7.3e-05

s

p50

0.001485

s

≤

p95

0.001485

s

≤

p99

0.001485

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004105

s

SD

0.000234

s

p50

0.004515

s

≤

p95

0.004515

s

≤

p99

0.004515

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002611

s

SD

0.000301

s

p50

0.003134

s

≤

p95

0.003134

s

≤

p99

0.003134

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002002

s

SD

0.000479

s

p50

0.002834

s

≤

p95

0.002834

s

≤

p99

0.002834

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

3.4e-05

s

SD

2e-06

s

p50

3.9e-05

s

≤

p95

3.9e-05

s

≤

p99

3.9e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001091

s

SD

0.000348

s

p50

0.001454

s

<

p95

0.001753

s

≤

p99

0.001753

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000267

s

SD

2e-06

s

p50

0.00027

s

≤

p95

0.00027

s

≤

p99

0.00027

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

44.52

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

37.15

KiB/s

<

mean

47.14

KiB/s

SD

9.99

KiB/s

<

p95

57.13

KiB/s

Not enough time data to show a trend.

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

20.61

KiB/s

<

mean

24.84

KiB/s

SD

4.24

KiB/s

<

p95

29.08

KiB/s

Not enough time data to show a trend.

Total bytes received

Total bytes received on primary network interface

count

942.74

KiB

mean

47.14

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

496.85

KiB

mean

24.84

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

46.69

%

<

mean

55.52

%

SD

8.83

%

<

p95

64.35

%

Not enough time data to show a trend.

System

11.01

%

CPU percentiles

CPU usage percentiles

p50

64.35

%

p95

64.35

%

p99

64.35

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

11.7

%

<

mean

17.31

%

SD

5.17

%

<

p95

24.18

%

Not enough time data to show a trend.

Memory available percentage

Percentage of available memory

p5

75.82

%

<

mean

82.69

%

SD

5.17

%

<

p95

88.3

%

Not enough time data to show a trend.

Max host memory used

Maximum memory usage percentage across all hosts

max

17.31

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-1.70

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.08

MiB/s

Disk write throughput

Disk write throughput in MiB/s

15.92

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.65

5 min

0.18

15 min

0.05

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

4.79

%

<

mean

8.33

%

SD

2.7032

%

<

p95

11.35

%

Not enough time data to show a trend.

60 second average

2.6533

%

300 second average

0.6233

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.89

%

SD

5.0008

%

<

p95

10.95

%

Not enough time data to show a trend.

60 second average

0.7733

%

300 second average

0.2133

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.4867

%

SD

3.0219

%

<

p95

6.74

%

Not enough time data to show a trend.

60 second average

0.5133

%

300 second average

0.1600

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

31.16

%

SD

31.16

%

<

p95

62.32

%

Not enough time data to show a trend.

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

98.38

KiB

<

mean

543.98

KiB

SD

445.59

KiB

<

p95

989.57

KiB

Not enough time data to show a trend.

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

66.33

threads

SD

76.71

threads

<

p95

174

threads

Not enough time data to show a trend.

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

149.67

file descriptors

SD

169.72

file descriptors

<

p95

387

file descriptors

Not enough time data to show a trend.

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 01 Oct 2026 14:33:47 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

full_arc_create_validated_zero_arc_read--high_zero_36830042121_1

Create rate

The number of timed entries created by the full-arc node(s) per second.

Mean

mean

31.4

/s

Max

Highest per-partition mean rate

max

33.85

/s

Min

Lowest per-partition mean rate

min

28.95

/s

Fetch lag timing

For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.

Mean

mean

565.110158

s

SD

39.433061

s

Max

Highest per-partition mean latency

max mean

1124.72984

s

Min

Lowest per-partition mean latency

min mean

5.490475

s

Fetch rate

The number of entries per second fetched by zero-arc nodes.

Mean

mean

56.63

/s

Max

Highest per-partition mean rate

max

113.25

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc nodes.

0.0133

Entry created count

Total number of entries created by full-arc nodes, by agent.

Total

5943

Agents affected

2 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

2972

Max

Highest count in any single partition

3169

Min

Lowest count in any single partition

2774

Received count

Total number of entries fetched by zero-arc nodes, by agent.

Total

2138

Agents affected

2 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

79

Max

Highest count in any single partition

1069

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

28

≤

mean

27.48

SD

2.96

<

p95

28

zero-arc

p5

0

<

mean

5.44

SD

5.84

<

p95

18

Retrieval errors

Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.

Total

366

Agents affected

27 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

14

Max

Highest count in any single partition

14

Min

Lowest count in any single partition

3

Error count

The number of errors accumulated across all nodes.

366

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.054166

s

SD

0.093822

s

p50

0.071974

s

<

p95

0.248215

s

<

p99

0.498769

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

6

over 798.176512753s

mean rate

291.6

/s

std

841.02

/s

p5

0

/s

p95

2296.06

/s

peak

9846.6

/s

WASM usage: 'timed_and_validated::create_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

198988989

over 870.355694717s

mean rate

5.424453032e+07

/s

std

2.579276691e+07

/s

p5

1.5280188e+06

/s

p95

8.714882906e+07

/s

peak

9.032132017e+07

/s

WASM usage: 'timed_and_validated::get_timed_entries_network'

Metered usage count of the WASM ribosome for this zome function.

total

331423148

over 888.596570598s

mean rate

8.71234396922e+09

/s

std

2.68694189794e+10

/s

p5

4.109496799e+07

/s

p95

7.36274019063e+10

/s

peak

1.76737059443e+11

/s

WASM usage: 'timed_and_validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

14808970

over 888.596570598s

mean rate

2.02884558241e+09

/s

std

1.72546618379e+10

/s

p5

5.67309948e+06

/s

p95

1.20500143052e+10

/s

peak

4.76311575354e+11

/s

WASM usage: 'timed_and_validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 888.596570598s

Not enough time data to show a trend.

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.008668

s

SD

0.001711

s

p50

0.008258

s

<

p95

0.010948

s

<

p99

0.011106

s

Zome call duration: 'timed_and_validated::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

0.007725

s

SD

0.035873

s

p50

0.003463

s

<

p95

0.046639

s

<

p99

0.296086

s

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.007999

s

SD

0.0017

s

p50

0.007564

s

<

p95

0.010296

s

<

p99

0.010451

s

WASM call duration: 'timed_and_validated::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.10345

s

SD

0.142534

s

p50

0.099269

s

<

p95

0.479639

s

<

p99

0.642019

s

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001109

s

SD

0.00079

s

p50

0.000914

s

<

p95

0.003283

s

<

p99

0.00334

s

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.009358

s

SD

0.002966

s

p50

0.000914

s

<

p95

0.009242

s

<

p99

0.014077

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000635

s

SD

0.000214

s

p50

0.000544

s

<

p95

0.000947

s

<

p99

0.000966

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000597

s

SD

0.000157

s

p50

0.000551

s

<

p95

0.000816

s

<

p99

0.000834

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.075792

s

SD

6.982433

s

p50

0.386705

s

<

p95

10.950414

s

<

p99

60.001951

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.036331

s

SD

0.085179

s

p50

0.028671

s

<

p95

0.2468

s

<

p99

0.441758

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.020941

s

SD

0.009157

s

p50

0.017358

s

<

p95

0.032444

s

<

p99

0.039569

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001765

s

SD

0.001137

s

p50

0.001415

s

<

p95

0.004824

s

<

p99

0.004881

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000176

s

SD

4.9e-05

s

p50

8.7e-05

s

<

p95

0.000227

s

<

p99

0.000239

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.67

s

<

mean

447.16

s

SD

256.71

s

<

p95

847.72

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 888.596456018s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

115.437464

s

SD

26.981661

s

p50

0

s

<

p95

73.03975

s

<

p99

147.089884

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

5.253774

s

SD

3.256296

s

p50

0.001116

s

<

p95

9.654888

s

<

p99

16.080607

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.024599

s

SD

0.005992

s

p50

0.00117

s

<

p95

0.018602

s

<

p99

0.031801

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.156423

s

SD

0.03862

s

p50

0.011323

s

<

p95

0.091268

s

<

p99

0.219656

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

17.631099

s

SD

15.890439

s

p50

0.002791

s

<

p95

32.056104

s

<

p99

77.99579

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

3.655868

s

SD

3.278303

s

p50

0.002259

s

<

p95

3.27219

s

<

p99

22.270893

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001478

s

SD

0.002248

s

p50

7.5e-05

s

<

p95

0.005945

s

<

p99

0.012165

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.013932

s

SD

0.005176

s

p50

0.001652

s

<

p95

0.015442

s

<

p99

0.026884

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000357

s

SD

8.2e-05

s

p50

0.000312

s

<

p95

0.000487

s

<

p99

0.000494

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.840094

s

SD

29.918822

s

p50

0.212808

s

<

p95

60.001736

s

<

p99

60.002017

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

25.085082

s

SD

29.221141

s

p50

1.365417

s

<

p95

60.001444

s

<

p99

60.002334

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

1.106312

s

SD

4.816501

s

p50

1.362708

s

<

p95

2.558703

s

≤

p99

2.558703

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

7.65

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

601

B/s

<

mean

61.74

KiB/s

SD

390.99

KiB/s

<

p95

157.49

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

459

B/s

<

mean

23.52

KiB/s

SD

127.12

KiB/s

<

p95

59.22

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.41

GiB

mean

1.63

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

551.52

MiB

mean

634.55

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.6%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (72.8% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 7% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.44

%

<

mean

14.51

%

SD

25.16

%

<

p95

88.73

%

System

6.86

%

CPU percentiles

CPU usage percentiles

p50

2.54

%

p95

88.73

%

p99

98.60

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

6

hosts

mean time

260.81

s

Memory used percentage

Percentage of memory used

p5

4.09

%

<

mean

19.15

%

SD

15.59

%

<

p95

39.02

%

Memory available percentage

Percentage of available memory

p5

60.98

%

<

mean

80.85

%

SD

15.59

%

<

p95

95.91

%

Max host memory used

Maximum memory usage percentage across all hosts

max

86.24

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

72.76

%

Memory growth rate

Rate of memory growth over time

growth

0.09

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.88

MiB/s

Disk write throughput

Disk write throughput in MiB/s

28.51

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.91

5 min

1.07

15 min

1.32

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

7.14

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

4.5927

%

SD

17.0262

%

<

p95

41.37

%

60 second average

4.5757

%

300 second average

4.9079

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0155

%

SD

0.3325

%

≤

p95

0

%

60 second average

0.0091

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0131

%

SD

0.2794

%

≤

p95

0

%

60 second average

0.0083

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.6185

%

SD

6.5296

%

<

p95

10.82

%

60 second average

1.4740

%

300 second average

1.1607

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.7335

%

SD

3.8133

%

<

p95

3.13

%

60 second average

0.6571

%

300 second average

0.5783

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.12

%

<

mean

8.08

%

SD

23.4

%

<

p95

87.89

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

358.27

KiB

<

mean

425.86

KiB

SD

92.29

KiB

<

p95

528.63

KiB

Holochain process threads

Number of threads in Holochain process

p5

20

threads

<

mean

28.91

threads

SD

13.87

threads

<

p95

39

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

53

file descriptors

<

mean

60.03

file descriptors

SD

8.46

file descriptors

<

p95

79

file descriptors

Local Signals

↑ Back to index

This scenario is for testing the speed and reliability of local signals. It uses a zome with a function that will emit 10,000 signals. The scenario measures how long it takes to send the signals and how many have been received by the time the zome call ends.

Started

Thu, 01 Oct 2026 10:50:51 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

local_signals_36830042121_1

Send timing

The time taken to emit a batch of 10,000 signals.

mean

1.619953

s

SD

1.562581

s

p50

1.266903

s

<

p95

5.881079

s

≤

p99

5.881079

s

Receive timing

The time taken to receive the completed batch, to the nearest 250ms.

mean

27.574019

s

SD

7.281348

s

p50

30.001113

s

<

p95

30.001672

s

≤

p99

30.001672

s

Success ratio

The proportion of signals successfully received in a batch.

min

0.47110000252723694

<

mean

0.5894

SD

0.0791

<

max

0.7384999990463257

Holochain MetricsClick to toggle visibility.

WASM usage: 'signal::emit_10k_signals'

Metered usage count of the WASM ribosome for this zome function.

total

245084656

over 280.003957952s

mean rate

875289.09

/s

std

1.38395356e+06

/s

p5

0

/s

p95

3.06353899e+06

/s

peak

3.06354456e+06

/s

Zome call duration: 'signal::emit_10k_signals'

Duration of this zome call as measured by Holochain internally.

mean

0.178947

s

SD

0.022232

s

p50

0.161218

s

<

p95

0.212521

s

≤

p99

0.212521

s

WASM call duration: 'signal::emit_10k_signals'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.178092

s

SD

0.0222

s

p50

0.160278

s

<

p95

0.211625

s

≤

p99

0.211625

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

1.2e-05

s

SD

2e-06

s

p50

1.1e-05

s

<

p95

1.5e-05

s

≤

p99

1.5e-05

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

80000

over 280.003994211s

mean rate

285.71

/s

std

451.75

/s

p5

0

/s

p95

999.99

/s

peak

999.99

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000137

s

SD

0

s

p50

0.000137

s

≤

p95

0.000137

s

≤

p99

0.000137

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.92

s

<

mean

146.92

s

SD

83.67

s

<

p95

276.93

s

Dropped WebSocket signals

Count of signals dropped from the app WebSocket due to channel overload.

total

3827

over 280.003964822s

mean rate

13.67

/s

std

25.27

/s

p5

0

/s

p95

72.2

/s

peak

79.6

/s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.003964287s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000613

s

SD

0

s

p50

0.000613

s

≤

p95

0.000613

s

≤

p99

0.000613

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001561

s

SD

0

s

p50

0.001561

s

≤

p95

0.001561

s

≤

p99

0.001561

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004988

s

SD

0.001033

s

p50

0.005341

s

<

p95

0.006969

s

≤

p99

0.006969

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002554

s

SD

0

s

p50

0.002554

s

≤

p95

0.002554

s

≤

p99

0.002554

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.003347

s

SD

0

s

p50

0.003347

s

≤

p95

0.003347

s

≤

p99

0.003347

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000102

s

SD

1.6e-05

s

p50

9.4e-05

s

<

p95

0.000121

s

≤

p99

0.000121

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001171

s

SD

0.000118

s

p50

0.001332

s

≤

p95

0.001332

s

≤

p99

0.001332

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.00052

s

SD

0

s

p50

0.00052

s

≤

p95

0.00052

s

≤

p99

0.00052

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

14.92

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

1.01

KiB/s

<

mean

917.12

KiB/s

SD

2.50

MiB/s

<

p95

8.32

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

43

B/s

<

mean

26.95

KiB/s

SD

38.74

KiB/s

<

p95

132.50

KiB/s

Total bytes received

Total bytes received on primary network interface

count

259.73

MiB

mean

917.12

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.63

MiB

mean

26.95

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 96.9%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 3.81 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.84

%

<

mean

21.88

%

SD

21.85

%

<

p95

77.33

%

System

6.96

%

CPU percentiles

CPU usage percentiles

p50

15.14

%

p95

77.33

%

p99

96.90

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

10.34

s

Memory used percentage

Percentage of memory used

p5

14.77

%

<

mean

19.36

%

SD

4.54

%

<

p95

27.32

%

Memory available percentage

Percentage of available memory

p5

72.68

%

<

mean

80.64

%

SD

4.54

%

<

p95

85.23

%

Max host memory used

Maximum memory usage percentage across all hosts

max

19.36

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

3.81

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

4.41

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.49

5 min

0.63

15 min

0.25

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.41

%

<

mean

10.6557

%

SD

13.8642

%

<

p95

47.93

%

60 second average

7.8583

%

300 second average

3.3950

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.526

%

SD

0.5807

%

<

p95

1.81

%

60 second average

0.6767

%

300 second average

0.6310

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.3547

%

SD

0.3545

%

<

p95

1.04

%

60 second average

0.5287

%

300 second average

0.5117

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.35

%

<

mean

7.44

%

SD

10.76

%

<

p95

37.34

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

89.74

KiB

<

mean

214.69

KiB

SD

42.92

KiB

<

p95

250.14

KiB

Holochain process threads

Number of threads in Holochain process

p5

19

threads

<

mean

26.77

threads

SD

5.82

threads

<

p95

35

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

59

file descriptors

<

mean

72.43

file descriptors

SD

10.3

file descriptors

<

p95

88

file descriptors

Mixed Arc `get_agent_activity`

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

zero_write: A zero-arc conductor that creates entries on its source chain.
full_write: A full-arc conductor that creates entries on its source chain.
zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.

Started

Thu, 01 Oct 2026 14:49:28 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full_write (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_get_agent_activity_36830042121_1

Entry created count

The total number of entries created by write agents over the run, partitioned by agent and behaviour.

Total

120709

Agents affected

20 / 20

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

6035

Max

Highest count in any single partition

10671

Min

Lowest count in any single partition

2105

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

2826.33

Max

Highest chain head value observed for any single write agent.

3725

Write agents observed

3

Chain head delay timing

The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.

Mean

mean

0.005727

s

SD

0.0768

s

Max

Highest per-partition mean latency

max mean

0.021285

s

Min

Lowest per-partition mean latency

min mean

0.000617

s

Chain head delay rate

The rate at which readers detect new chain heads appearing on writers' chains.

Mean

mean

22.78

/s

Max

Highest per-partition mean rate

max

44.19

/s

Min

Lowest per-partition mean rate

min

11.97

/s

get_agent_activity_full zome call timing

The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.

Mean

mean

0.136234

s

SD

0.097454

s

Max

Highest per-partition mean latency

max mean

0.266337

s

Min

Lowest per-partition mean latency

min mean

0.01377

s

Retrieval errors

A running accumulation of errors encountered by read agents when calling get_agent_activity.

Total

37

Agents affected

6 / 9

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

4

Max

Highest count in any single partition

9

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors, partitioned by agent behaviour.

behaviour: full_write

p5

28

≤

mean

27.8

SD

1.72

<

p95

28

behaviour: zero_read

p5

0

<

mean

13.48

SD

5.64

<

p95

21

behaviour: zero_write

p5

11

<

mean

15.75

SD

4.04

<

p95

25

Final error count

The total number of errors accumulated across all agents over the run.

37

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.141362

s

SD

0.178279

s

p50

0.186685

s

<

p95

0.657774

s

≤

p99

0.657774

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 680.012263717s

Not enough time data to show a trend.

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.006409

s

SD

0.002487

s

p50

0.006298

s

<

p95

0.010319

s

<

p99

0.010874

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.00389

s

SD

0.003374

s

p50

0.003503

s

<

p95

0.013688

s

<

p99

0.016109

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.146783

s

SD

0.070904

s

p50

0.167636

s

<

p95

0.261547

s

<

p99

0.308358

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.029623

s

SD

0.227126

s

p50

0.192677

s

<

p95

0.662421

s

≤

p99

0.662421

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005388

s

SD

0.002147

s

p50

0.005745

s

<

p95

0.008951

s

<

p99

0.009334

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003308

s

SD

0.003239

s

p50

0.002877

s

<

p95

0.012916

s

<

p99

0.015291

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.145563

s

SD

0.070843

s

p50

0.165764

s

<

p95

0.260264

s

<

p99

0.306932

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.028732

s

SD

0.226915

s

p50

0.191926

s

<

p95

0.661707

s

≤

p99

0.661707

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001088

s

SD

0.000776

s

p50

0.001163

s

<

p95

0.002724

s

<

p99

0.003465

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

4.5e-05

s

SD

2.4e-05

s

p50

4.2e-05

s

<

p95

7.6e-05

s

<

p99

0.000111

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000492

s

SD

0.001327

s

p50

0.000302

s

<

p95

0.004806

s

<

p99

0.006015

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000515

s

SD

0.000242

s

p50

0.000474

s

<

p95

0.000909

s

<

p99

0.001016

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.143109

s

SD

0.070628

s

p50

0.162775

s

<

p95

0.258594

s

<

p99

0.30424

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.024632

s

SD

0.226833

s

p50

0.188299

s

<

p95

0.657826

s

≤

p99

0.657826

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1e-05

s

SD

8e-06

s

p50

1.2e-05

s

<

p95

3.3e-05

s

≤

p99

3.3e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001678

s

SD

0.001236

s

p50

0.001948

s

<

p95

0.004736

s

<

p99

0.006332

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000125

s

SD

0.000187

s

p50

9.6e-05

s

<

p95

0.000699

s

<

p99

0.00089

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.68

s

<

mean

446.65

s

SD

256.53

s

<

p95

847.66

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

56

over 887.971252569s

mean rate

1760.84

/s

std

5346.86

/s

p5

0

/s

p95

19210.57

/s

peak

25699.04

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

10.220782

s

SD

13.098042

s

p50

0

s

<

p95

0.631931

s

<

p99

89.168729

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.013584

s

SD

0.133404

s

p50

0.002465

s

<

p95

0.411014

s

<

p99

0.522323

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.053307

s

SD

0.312885

s

p50

0.001316

s

<

p95

1.021026

s

<

p99

1.292451

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.080425

s

SD

0.058853

s

p50

0.053341

s

<

p95

0.189471

s

<

p99

0.227557

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

1.906543

s

SD

15.770306

s

p50

0.003394

s

<

p95

53.005424

s

<

p99

69.900054

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

5.727658

s

SD

15.032842

s

p50

0.002511

s

<

p95

9.222786

s

<

p99

34.207816

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002319

s

SD

0.001278

s

p50

0.001522

s

<

p95

0.004159

s

<

p99

0.004677

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002364

s

SD

0.002761

s

p50

0.001079

s

<

p95

0.003498

s

<

p99

0.019912

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000467

s

SD

0.001093

s

p50

0.000276

s

<

p95

0.003569

s

<

p99

0.00562

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

6.197649

s

SD

29.782729

s

p50

0.249919

s

<

p95

60.004518

s

<

p99

60.071136

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.208353

s

SD

0.143322

s

p50

0.186278

s

<

p95

0.537958

s

≤

p99

0.537958

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.126525

s

SD

28.411248

s

p50

0.181511

s

<

p95

60.001305

s

<

p99

60.001881

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.128578

s

SD

0.099054

s

p50

0.192702

s

<

p95

0.359485

s

≤

p99

0.359485

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

38.03

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

2.93

KiB/s

<

mean

373.02

KiB/s

SD

511.42

KiB/s

<

p95

986.31

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

1.55

KiB/s

<

mean

316.46

KiB/s

SD

265.39

KiB/s

<

p95

780.48

KiB/s

Total bytes received

Total bytes received on primary network interface

count

8.52

GiB

mean

9.80

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.23

GiB

mean

8.32

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (74.3% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 25% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

2.31

%

<

mean

48.45

%

SD

34.49

%

<

p95

97.45

%

System

10.43

%

CPU percentiles

CPU usage percentiles

p50

58.68

%

p95

97.45

%

p99

98.15

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

10

hosts

mean time

597.28

s

Memory used percentage

Percentage of memory used

p5

3.65

%

<

mean

18.07

%

SD

15.38

%

<

p95

35.89

%

Memory available percentage

Percentage of available memory

p5

64.11

%

<

mean

81.93

%

SD

15.38

%

<

p95

96.35

%

Max host memory used

Maximum memory usage percentage across all hosts

max

85.54

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

74.28

%

Memory growth rate

Rate of memory growth over time

growth

0.34

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.62

MiB/s

Disk write throughput

Disk write throughput in MiB/s

84.55

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.35

5 min

2.68

15 min

1.97

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

25.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

18.7401

%

SD

26.0259

%

<

p95

75.51

%

60 second average

17.8440

%

300 second average

14.2761

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0004

%

SD

0.0101

%

≤

p95

0

%

60 second average

0.0016

%

300 second average

0.0072

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0002

%

SD

0.0062

%

≤

p95

0

%

60 second average

0.0013

%

300 second average

0.0061

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.4741

%

SD

5.0018

%

<

p95

6.43

%

60 second average

1.5669

%

300 second average

1.6869

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.4572

%

SD

2.1343

%

<

p95

2.55

%

60 second average

0.5706

%

300 second average

0.7214

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.72

%

<

mean

43.3

%

SD

32.2

%

<

p95

96.31

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

227.07

KiB

<

mean

340.11

KiB

SD

137.26

KiB

<

p95

568.78

KiB

Holochain process threads

Number of threads in Holochain process

p5

22

threads

<

mean

38.99

threads

SD

28.59

threads

<

p95

66

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

57

file descriptors

<

mean

80.05

file descriptors

SD

31

file descriptors

<

p95

109

file descriptors

Mixed Arc `must_get_agent_activity`

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started

Thu, 01 Oct 2026 15:05:11 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full_write (1 agent)
zero_must_get_agent_activity (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_must_get_agent_activity_36830042121_1

Highest observed action_seq

The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

1668.75

Max

Highest chain head value observed for any single write agent.

1754

Write agents observed

4

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

55.758142

s

SD

46.987918

s

Max

Highest per-partition mean latency

max mean

147.022129

s

Min

Lowest per-partition mean latency

min mean

14.583321

s

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

2.28

/s

Max

Highest per-partition mean rate

max

2.57

/s

Min

Lowest per-partition mean rate

min

1.69

/s

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

0.404139

s

SD

0.482608

s

Max

Highest per-partition mean latency

max mean

0.697985

s

Min

Lowest per-partition mean latency

min mean

0.235762

s

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

15749

Agents affected

8 / 8

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

1969

Max

Highest count in any single partition

6187

Min

Lowest count in any single partition

665

Open connections

The average number of open connections maintained by an agent over the run, broken down by behaviour.

behaviour: full_write

p5

28

≤

mean

27.9

SD

0.97

<

p95

28

behaviour: zero_must_get_agent_activity

p5

12

<

mean

17.09

SD

4.04

<

p95

24

behaviour: zero_write

p5

0

<

mean

15.42

SD

5.97

<

p95

27

Final error count

The total number of all types of error accumulated over the run by all agents.

28435

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.065389

s

SD

0.122087

s

p50

0.066192

s

<

p95

0.344693

s

<

p99

0.541307

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

1

over 771.768859239s

mean rate

2.01

/s

std

3.78

/s

p5

0

/s

p95

10.68

/s

peak

11.02

/s

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007603

s

SD

0.003659

s

p50

0.006403

s

<

p95

0.018082

s

≤

p99

0.018082

s

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.073072

s

SD

0.037449

s

p50

0.054773

s

<

p95

0.134544

s

<

p99

0.185646

s

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004458

s

SD

0.002126

s

p50

0.003694

s

<

p95

0.008421

s

<

p99

0.008549

s

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.126159

s

SD

0.081895

s

p50

0.07123

s

<

p95

0.255056

s

<

p99

0.266247

s

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.025567

s

SD

0.099247

s

p50

0.014559

s

<

p95

0.323921

s

≤

p99

0.323921

s

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00635

s

SD

0.003323

s

p50

0.005542

s

<

p95

0.016708

s

≤

p99

0.016708

s

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.072165

s

SD

0.037281

s

p50

0.053806

s

<

p95

0.133383

s

<

p99

0.184411

s

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003622

s

SD

0.00183

s

p50

0.002973

s

<

p95

0.006961

s

<

p99

0.007032

s

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.182191

s

SD

0.158092

s

p50

0.157203

s

<

p95

0.540701

s

<

p99

0.654133

s

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.025027

s

SD

0.099085

s

p50

0.013651

s

<

p95

0.322825

s

≤

p99

0.322825

s

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001311

s

SD

0.000566

s

p50

0.001152

s

<

p95

0.002401

s

<

p99

0.003016

s

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.013507

s

SD

0.022094

s

p50

0.004515

s

<

p95

0.076738

s

<

p99

0.104266

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2.3e-05

s

SD

2e-05

s

p50

2.6e-05

s

<

p95

6.6e-05

s

<

p99

8.1e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000404

s

SD

0.000185

s

p50

0.000404

s

<

p95

0.000741

s

<

p99

0.000856

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000463

s

SD

0.000262

s

p50

0.000487

s

<

p95

0.001011

s

<

p99

0.001343

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.129216

s

SD

0.14933

s

p50

0.085874

s

<

p95

0.473936

s

<

p99

0.58534

s

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.050561

s

SD

0.079169

s

p50

0.041222

s

<

p95

0.239206

s

<

p99

0.340527

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.031256

s

SD

0.025054

s

p50

0.013535

s

<

p95

0.074519

s

<

p99

0.109347

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

5e-06

s

SD

6e-06

s

p50

6e-06

s

<

p95

2.2e-05

s

≤

p99

2.2e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002055

s

SD

0.001134

s

p50

0.002191

s

<

p95

0.004382

s

<

p99

0.008046

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000139

s

SD

8.3e-05

s

p50

0.000116

s

<

p95

0.000333

s

<

p99

0.000414

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.68

s

<

mean

446.94

s

SD

256.61

s

<

p95

847.69

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

20198

over 888.519869027s

mean rate

1.3926378e+06

/s

std

3.731167834e+07

/s

p5

0

/s

p95

241678.61

/s

peak

1.49646756583e+09

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

157.293801

s

SD

88.182177

s

p50

0

s

<

p95

242.373947

s

<

p99

348.004965

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

2.932115

s

SD

1.553825

s

p50

0.001318

s

<

p95

4.181164

s

<

p99

5.70389

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.168179

s

SD

0.168916

s

p50

0.001101

s

<

p95

0.467636

s

<

p99

0.715995

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.133078

s

SD

0.169499

s

p50

0.015026

s

<

p95

0.472079

s

<

p99

0.781216

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

13.07245

s

SD

16.931574

s

p50

0.002896

s

<

p95

41.719256

s

<

p99

99.508608

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

22.56666

s

SD

24.783159

s

p50

0.002493

s

<

p95

63.152529

s

<

p99

108.090126

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002673

s

SD

0.001804

s

p50

0.000299

s

<

p95

0.005268

s

<

p99

0.006592

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002906

s

SD

0.003786

s

p50

0.001526

s

<

p95

0.007376

s

<

p99

0.025773

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000385

s

SD

0.000147

s

p50

0.000345

s

<

p95

0.000621

s

<

p99

0.000646

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

2.423606

s

SD

29.896875

s

p50

0.241229

s

<

p95

60.001562

s

≤

p99

60.001562

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.099225

s

SD

26.950771

s

p50

0.125325

s

<

p95

60.001752

s

<

p99

60.00199

s

P2P request roundtrip: 'must_get_agent_activity'

Time spent sending a must_get_agent_activity request and awaiting its response.

mean

0.091063

s

SD

25.438037

s

p50

0.076404

s

<

p95

60.0013

s

<

p99

60.001796

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.140469

s

SD

6.102933

s

p50

0.17517

s

<

p95

15.209156

s

<

p99

15.231991

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

23.13

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

1.83

KiB/s

<

mean

143.96

KiB/s

SD

520.16

KiB/s

<

p95

268.11

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

1.52

KiB/s

<

mean

79.03

KiB/s

SD

90.26

KiB/s

<

p95

209.77

KiB/s

Total bytes received

Total bytes received on primary network interface

count

3.30

GiB

mean

3.79

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.81

GiB

mean

2.08

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (75.0% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 21% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.94

%

<

mean

32.48

%

SD

33.81

%

<

p95

96.2

%

System

9.35

%

CPU percentiles

CPU usage percentiles

p50

9.94

%

p95

96.20

%

p99

98.05

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

10

hosts

mean time

340.79

s

Memory used percentage

Percentage of memory used

p5

4.74

%

<

mean

20

%

SD

15.39

%

<

p95

40.71

%

Memory available percentage

Percentage of available memory

p5

59.29

%

<

mean

80

%

SD

15.39

%

<

p95

95.26

%

Max host memory used

Maximum memory usage percentage across all hosts

max

84.92

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

74.99

%

Memory growth rate

Rate of memory growth over time

growth

0.22

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1.12

MiB/s

Disk write throughput

Disk write throughput in MiB/s

93.62

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.86

5 min

2.14

15 min

2.18

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

21.43

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

11.5223

%

SD

20.3203

%

<

p95

66.1

%

60 second average

11.2480

%

300 second average

11.9542

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0002

%

SD

0.0046

%

≤

p95

0

%

60 second average

0.0001

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0002

%

SD

0.0046

%

≤

p95

0

%

60 second average

0.0001

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.867

%

SD

5.9321

%

<

p95

9.76

%

60 second average

1.8372

%

300 second average

1.7357

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.8571

%

SD

3.2064

%

<

p95

5.67

%

60 second average

0.8377

%

300 second average

0.7355

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.47

%

<

mean

26.23

%

SD

32.03

%

<

p95

94.6

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

376.10

KiB

<

mean

458.36

KiB

SD

94.80

KiB

<

p95

569.47

KiB

Holochain process threads

Number of threads in Holochain process

p5

26

threads

<

mean

35.18

threads

SD

12.98

threads

<

p95

44

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

59

file descriptors

<

mean

70.83

file descriptors

SD

9.49

file descriptors

<

p95

83

file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started

Thu, 01 Oct 2026 10:52:46 UTC

Duration

900s

Peer count

10

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_call_rate_36830042121_1

Dispatch timing

The time between sending a remote call and the remote handler being invoked.

Mean

mean

0.070623

s

SD

0.060343

s

Max

Highest per-partition mean latency

max mean

0.089147

s

Min

Lowest per-partition mean latency

min mean

0.031972

s

Round-trip timing

The total elapsed time to get a response to the client.

Mean

mean

0.142037

s

SD

0.085695

s

Max

Highest per-partition mean latency

max mean

0.182531

s

Min

Lowest per-partition mean latency

min mean

0.06503

s

Error count

The total number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_call::call_echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

429757350

over 887.52420462s

mean rate

1.3681824023e+08

/s

std

1.7758608688e+08

/s

p5

1.62615386e+06

/s

p95

6.0297129046e+08

/s

peak

8.002893451e+08

/s

WASM usage: 'remote_call::echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

36608208

over 887.52420462s

mean rate

838723.54

/s

std

832606.7

/s

p5

32189.98

/s

p95

2.45935636e+06

/s

peak

2.86794331e+06

/s

WASM usage: 'remote_call::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 887.52420462s

Not enough time data to show a trend.

Zome call duration: 'remote_call::call_echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.138954

s

SD

0.04457

s

p50

0.161128

s

<

p95

0.204017

s

<

p99

0.279367

s

Zome call duration: 'remote_call::echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.00286

s

SD

0.000641

s

p50

0.002762

s

<

p95

0.004133

s

<

p99

0.004484

s

WASM call duration: 'remote_call::call_echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.137968

s

SD

0.044548

s

p50

0.160116

s

<

p95

0.203012

s

<

p99

0.278447

s

WASM call duration: 'remote_call::echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001925

s

SD

0.000479

s

p50

0.001888

s

<

p95

0.002822

s

<

p99

0.003098

s

WASM call duration: 'remote_call::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00271

s

SD

0.000701

s

p50

0.002705

s

<

p95

0.003545

s

≤

p99

0.003545

s

Host function call duration: '__hc__call_1'

Duration of this host function call invoked from within WASM.

mean

0.1357

s

SD

0.044512

s

p50

0.157648

s

<

p95

0.200832

s

<

p99

0.276272

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000469

s

SD

9.5e-05

s

p50

0.000482

s

<

p95

0.000603

s

≤

p99

0.000603

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

9e-06

s

SD

4e-06

s

p50

9e-06

s

<

p95

1.4e-05

s

<

p99

1.6e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000271

s

SD

5.8e-05

s

p50

0.000248

s

<

p95

0.00036

s

≤

p99

0.00036

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.00014

s

SD

4e-05

s

p50

0.000157

s

<

p95

0.00019

s

≤

p99

0.00019

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.95

s

<

mean

446.02

s

SD

256.34

s

<

p95

846.92

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

85

over 887.524368544s

mean rate

6.07

/s

std

35.32

/s

p5

0

/s

p95

19.84

/s

peak

442.55

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.233974

s

SD

0.775238

s

p50

0.913628

s

<

p95

2.756233

s

<

p99

3.004992

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002539

s

SD

0.000651

s

p50

0.002714

s

<

p95

0.003416

s

<

p99

0.004129

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.081284

s

SD

0.039317

s

p50

0.074639

s

<

p95

0.16697

s

<

p99

0.16899

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.006219

s

SD

0.002023

s

p50

0.006206

s

<

p95

0.009838

s

<

p99

0.011739

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.190366

s

SD

0.238964

s

p50

0.049754

s

<

p95

0.769499

s

<

p99

0.933

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.164099

s

SD

0.081906

s

p50

0.161111

s

<

p95

0.280392

s

<

p99

0.351742

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000129

s

SD

2.8e-05

s

p50

0.000133

s

<

p95

0.000184

s

<

p99

0.000223

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001132

s

SD

0.000365

s

p50

0.001111

s

<

p95

0.00195

s

<

p99

0.001955

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000379

s

SD

6.6e-05

s

p50

0.000362

s

<

p95

0.000525

s

<

p99

0.000576

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.113645

s

SD

0.049563

s

p50

0.137786

s

<

p95

0.179698

s

<

p99

0.212595

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.151332

s

SD

0.037836

s

p50

0.153935

s

<

p95

0.194625

s

≤

p99

0.194625

s

P2P request roundtrip: 'call_remote'

Time spent sending a call_remote request and awaiting its response.

mean

0.135206

s

SD

0.044502

s

p50

0.157108

s

<

p95

0.200411

s

<

p99

0.275853

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

14.98

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

10.45

KiB/s

<

mean

390.26

KiB/s

SD

1.42

MiB/s

<

p95

1.63

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

8.78

KiB/s

<

mean

34.05

KiB/s

SD

40.46

KiB/s

<

p95

119.97

KiB/s

Total bytes received

Total bytes received on primary network interface

count

2.96

GiB

mean

3.41

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

264.72

MiB

mean

304.58

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.81 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.99

%

<

mean

22.9

%

SD

25.82

%

<

p95

78.78

%

System

7.92

%

CPU percentiles

CPU usage percentiles

p50

11.18

%

p95

78.78

%

p99

89.65

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

5

hosts

mean time

75.52

s

Memory used percentage

Percentage of memory used

p5

3.13

%

<

mean

17.18

%

SD

11.3

%

<

p95

47.1

%

Memory available percentage

Percentage of available memory

p5

52.9

%

<

mean

82.82

%

SD

11.3

%

<

p95

96.87

%

Max host memory used

Maximum memory usage percentage across all hosts

max

45.88

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

5.38

%

Memory growth rate

Rate of memory growth over time

growth

1.81

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.22

MiB/s

Disk write throughput

Disk write throughput in MiB/s

26.29

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

0/2

hosts

Mount Point /etc/hosts

0/2

hosts

Mount Point /etc/nomad.d/client-meta.json

0/8

hosts

Mount Point /etc/resolv.conf

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.33

5 min

1.05

15 min

0.57

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

9.4645

%

SD

14.4665

%

<

p95

45.37

%

60 second average

9.1934

%

300 second average

7.2322

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.4361

%

SD

1.4635

%

<

p95

2.35

%

60 second average

0.4274

%

300 second average

0.3838

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.3053

%

SD

1.1794

%

<

p95

1.52

%

60 second average

0.2942

%

300 second average

0.2647

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

7.45

%

SD

16.17

%

<

p95

56.2

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

96.22

KiB

<

mean

238.21

KiB

SD

90.25

KiB

<

p95

421.12

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

113.49

threads

SD

329.2

threads

<

p95

875

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

249.02

file descriptors

SD

721.29

file descriptors

<

p95

1927

file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started

Thu, 01 Oct 2026 10:54:34 UTC

Duration

900s

Peer count

10

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_signals_36830042121_1

Round trip time

The time from origin signal dispatch to origin receive of the remote side's response signal.

mean

0.134038

s

SD

0.375072

s

p50

0.137599

s

<

p95

0.218748

s

<

p99

0.259279

s

Timeout rate

The proportion of remote signal round-trips that timed out.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_signal::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 883.759677481s

Not enough time data to show a trend.

WASM usage: 'remote_signal::recv_remote_signal'

Metered usage count of the WASM ribosome for this zome function.

total

31080761

over 883.759677481s

mean rate

5.392156054e+07

/s

std

2.1345328429e+08

/s

p5

0

/s

p95

3.0988079319e+08

/s

peak

2.49874772704e+09

/s

WASM usage: 'remote_signal::signal_request'

Metered usage count of the WASM ribosome for this zome function.

total

12965220

over 883.759677481s

mean rate

4.33286527e+06

/s

std

3.933938467e+07

/s

p5

0

/s

p95

172403.56

/s

peak

8.270069808e+08

/s

Zome call duration: 'remote_signal::recv_remote_signal'

Duration of this zome call as measured by Holochain internally.

mean

0.002712

s

SD

0.000535

s

p50

0.002521

s

<

p95

0.003575

s

<

p99

0.003805

s

Zome call duration: 'remote_signal::signal_request'

Duration of this zome call as measured by Holochain internally.

mean

0.002688

s

SD

0.000565

s

p50

0.002496

s

<

p95

0.003652

s

<

p99

0.003763

s

WASM call duration: 'remote_signal::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002501

s

SD

0.000921

s

p50

0.002649

s

<

p95

0.003977

s

≤

p99

0.003977

s

WASM call duration: 'remote_signal::recv_remote_signal'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001824

s

SD

0.000429

s

p50

0.001725

s

<

p95

0.002478

s

<

p99

0.002624

s

WASM call duration: 'remote_signal::signal_request'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001788

s

SD

0.000447

s

p50

0.001571

s

<

p95

0.002546

s

<

p99

0.002591

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000498

s

SD

0.000352

s

p50

0.00042

s

<

p95

0.001497

s

≤

p99

0.001497

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

6.3e-05

s

SD

2.1e-05

s

p50

5.6e-05

s

<

p95

0.000107

s

<

p99

0.000118

s

Host function call duration: '__hc__send_remote_signal_1'

Duration of this host function call invoked from within WASM.

mean

7.1e-05

s

SD

3.9e-05

s

p50

6e-05

s

<

p95

0.00015

s

<

p99

0.000204

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

5e-06

s

SD

1e-06

s

p50

5e-06

s

<

p95

6e-06

s

<

p99

7e-06

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000261

s

SD

0.000128

s

p50

0.000236

s

<

p95

0.000581

s

≤

p99

0.000581

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

1719

over 883.759818148s

mean rate

47118.18

/s

std

856447

/s

p5

0

/s

p95

10303.79

/s

peak

1.694915254e+07

/s

Sent remote signals

Count of remote signals sent via the send_remote_signal host function.

total

1719

over 883.759819161s

mean rate

3613.81

/s

std

31189.83

/s

p5

0

/s

p95

8571.99

/s

peak

506136.91

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000125

s

SD

4.9e-05

s

p50

0.000131

s

<

p95

0.0002

s

≤

p99

0.0002

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.56

s

<

mean

444.89

s

SD

256.34

s

<

p95

844.39

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

90

over 883.759792488s

mean rate

1719.42

/s

std

9489.72

/s

p5

0

/s

p95

1152.76

/s

peak

104720.66

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.538233

s

SD

1.082276

s

p50

0.849851

s

<

p95

3.735143

s

<

p99

4.474246

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002521

s

SD

0.001091

s

p50

0.0029

s

<

p95

0.004167

s

<

p99

0.007083

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.094377

s

SD

0.03284

s

p50

0.089057

s

<

p95

0.146666

s

<

p99

0.174492

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.00597

s

SD

0.00181

s

p50

0.005703

s

<

p95

0.009127

s

<

p99

0.011961

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.197022

s

SD

0.308829

s

p50

0.025695

s

<

p95

0.836405

s

<

p99

1.061878

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.186106

s

SD

0.058498

s

p50

0.186391

s

<

p95

0.249706

s

<

p99

0.367676

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000129

s

SD

4.2e-05

s

p50

0.000137

s

<

p95

0.000231

s

<

p99

0.000296

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00118

s

SD

0.00072

s

p50

0.001457

s

<

p95

0.003774

s

≤

p99

0.003774

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000462

s

SD

0.000117

s

p50

0.000446

s

<

p95

0.000626

s

<

p99

0.00086

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.058127

s

SD

0.032486

s

p50

0.040664

s

<

p95

0.097892

s

<

p99

0.12818

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.121768

s

SD

0.021122

s

p50

0.130727

s

<

p95

0.149406

s

<

p99

0.162524

s

P2P received remote signals

Count of remote signals received.

total

1721

over 883.759789403s

mean rate

3733.45

/s

std

28407.35

/s

p5

0

/s

p95

19566.41

/s

peak

544959.13

/s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

16.80

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

8.68

KiB/s

<

mean

587.18

KiB/s

SD

1.58

MiB/s

<

p95

2.87

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

5.75

KiB/s

<

mean

39.88

KiB/s

SD

41.69

KiB/s

<

p95

125.65

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.99

GiB

mean

2.29

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

138.26

MiB

mean

159.08

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.5%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 20% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.91

%

<

mean

24.62

%

SD

28.11

%

<

p95

85.27

%

System

7.81

%

CPU percentiles

CPU usage percentiles

p50

12.56

%

p95

85.27

%

p99

94.52

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

4

hosts

mean time

96.99

s

Memory used percentage

Percentage of memory used

p5

3.08

%

<

mean

20.61

%

SD

15.09

%

<

p95

47.47

%

Memory available percentage

Percentage of available memory

p5

52.53

%

<

mean

79.39

%

SD

15.09

%

<

p95

96.92

%

Max host memory used

Maximum memory usage percentage across all hosts

max

46.48

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

5.38

%

Memory growth rate

Rate of memory growth over time

growth

0.73

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.03

MiB/s

Disk write throughput

Disk write throughput in MiB/s

17.92

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

0/2

hosts

Mount Point /etc/hosts

0/2

hosts

Mount Point /etc/nomad.d/client-meta.json

0/3

hosts

Mount Point /etc/resolv.conf

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.63

5 min

1.30

15 min

0.73

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

20.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

9.9273

%

SD

14.9391

%

<

p95

47.33

%

60 second average

9.8800

%

300 second average

7.7920

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.1657

%

SD

0.5859

%

<

p95

0.92

%

60 second average

0.1785

%

300 second average

0.2072

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.0955

%

SD

0.34

%

<

p95

0.51

%

60 second average

0.1039

%

300 second average

0.1320

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

6.79

%

SD

13.59

%

<

p95

35.89

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

72.86

KiB

<

mean

251.50

KiB

SD

116.53

KiB

<

p95

422.17

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

93.04

threads

SD

272.94

threads

<

p95

598

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

202.38

file descriptors

SD

594.89

file descriptors

<

p95

1269

file descriptors

Single-write, many-read

↑ Back to index

Creates an entry in the agent setup, then read one's own record back. This tests the maximum read performance for reading back local data when the chain is short.

Started

Thu, 01 Oct 2026 10:58:02 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

single_write_many_read_36830042121_1

read call timing

The time it takes to complete the zome call that retrieves the record.

mean

0.004038

s

SD

0.002295

s

p50

0.003456

s

<

p95

0.007044

s

<

p99

0.010747

s

read call rate

The number of read zome calls completed per second.

mean

2460.36

/s

Error count

The number of errors collected.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000282

s

SD

3.1e-05

s

p50

0.000278

s

<

p95

0.00036

s

<

p99

0.000379

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.002165713s

Not enough time data to show a trend.

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

2314832982

over 280.002165713s

mean rate

8.26719793e+06

/s

std

2.20948511e+06

/s

p5

5.58876896e+06

/s

p95

1.248214306e+07

/s

peak

1.268231347e+07

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.002165713s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.002165713s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.00502

s

SD

0

s

p50

0.00502

s

≤

p95

0.00502

s

≤

p99

0.00502

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.001668

s

SD

0.000122

s

p50

0.001654

s

<

p95

0.001957

s

<

p99

0.002047

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004436

s

SD

0

s

p50

0.004436

s

≤

p95

0.004436

s

≤

p99

0.004436

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001169

s

SD

9.5e-05

s

p50

0.001157

s

<

p95

0.00139

s

<

p99

0.00149

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004161

s

SD

0

s

p50

0.004161

s

≤

p95

0.004161

s

≤

p99

0.004161

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001803

s

SD

0

s

p50

0.001803

s

≤

p95

0.001803

s

≤

p99

0.001803

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.001109

s

SD

0.00087

s

p50

0.001979

s

≤

p95

0.001979

s

≤

p99

0.001979

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000309

s

SD

3.3e-05

s

p50

0.000304

s

<

p95

0.00039

s

<

p99

0.000411

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00325

s

SD

0

s

p50

0.00325

s

≤

p95

0.00325

s

≤

p99

0.00325

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.5e-05

s

SD

0

s

p50

8.5e-05

s

≤

p95

8.5e-05

s

≤

p99

8.5e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.43

s

<

mean

147.43

s

SD

83.67

s

<

p95

277.43

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.002173527s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001731

s

SD

0

s

p50

0.001731

s

≤

p95

0.001731

s

≤

p99

0.001731

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.002659

s

SD

0

s

p50

0.002659

s

≤

p95

0.002659

s

≤

p99

0.002659

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.007937

s

SD

0.00153

s

p50

0.008246

s

<

p95

0.010883

s

≤

p99

0.010883

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.005607

s

SD

0

s

p50

0.005607

s

≤

p95

0.005607

s

≤

p99

0.005607

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001429

s

SD

0

s

p50

0.001429

s

≤

p95

0.001429

s

≤

p99

0.001429

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

5.5e-05

s

SD

6e-06

s

p50

5.4e-05

s

<

p95

6.5e-05

s

<

p99

7.7e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00207

s

SD

0.000594

s

p50

0.002387

s

≤

p95

0.002387

s

≤

p99

0.002387

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000562

s

SD

0

s

p50

0.000562

s

≤

p95

0.000562

s

≤

p99

0.000562

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

48.53

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.14

KiB/s

<

mean

1.12

MiB/s

SD

2.19

MiB/s

<

p95

8.40

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.70

KiB/s

<

mean

52.99

KiB/s

SD

50.55

KiB/s

<

p95

177.50

KiB/s

Total bytes received

Total bytes received on primary network interface

count

323.38

MiB

mean

1.12

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

15.01

MiB

mean

52.99

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.23 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

36.05

%

<

mean

69.44

%

SD

15.5

%

<

p95

88.38

%

System

20.91

%

CPU percentiles

CPU usage percentiles

p50

68.94

%

p95

88.38

%

p99

88.83

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

82.76

s

Memory used percentage

Percentage of memory used

p5

7.58

%

<

mean

9.05

%

SD

0.81

%

<

p95

10.07

%

Memory available percentage

Percentage of available memory

p5

89.93

%

<

mean

90.95

%

SD

0.81

%

<

p95

92.42

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.05

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

1.23

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

6.14

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.18

5 min

2.41

15 min

1.06

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

4.69

%

<

mean

28.0863

%

SD

13.9697

%

<

p95

45.77

%

60 second average

23.9113

%

300 second average

13.0087

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

26.11

%

SD

12.81

%

<

p95

38.52

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

108.07

KiB

<

mean

253.14

KiB

SD

76.42

KiB

<

p95

316.41

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

141.52

threads

SD

358.29

threads

<

p95

1046

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

309.58

file descriptors

SD

776.55

file descriptors

<

p95

2269

file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started

Thu, 01 Oct 2026 11:42:02 UTC

Duration

900s

Peer count

20

Peer count at end

20

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

validation_receipts_36830042121_1

Receipts complete timing

The amount of time between publishing a record and receiving the required number of validation receipts.

Mean

mean

5.392732

s

SD

1.172609

s

Max

Highest per-partition mean latency

max mean

6.356687

s

Min

Lowest per-partition mean latency

min mean

5.167634

s

Receipts complete rate

The number of complete validation receipt sets collected per second.

Mean

mean

1.84

/s

Max

Highest per-partition mean rate

max

1.93

/s

Min

Lowest per-partition mean rate

min

1.61

/s

Error count

The total number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

5385885

over 888.326443071s

mean rate

1.74924423e+06

/s

std

3.07788997e+06

/s

p5

0

/s

p95

9.09123953e+06

/s

peak

1.543517854e+07

/s

WASM usage: 'crud::get_sample_entry_validation_receipts'

Metered usage count of the WASM ribosome for this zome function.

total

24322660

over 888.326443071s

mean rate

7.49286689e+06

/s

std

8.50690965e+06

/s

p5

19808.45

/s

p95

2.693764798e+07

/s

peak

3.479592372e+07

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 888.326443071s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

668192

over 888.326443071s

mean rate

216192.6

/s

std

381286.35

/s

p5

0

/s

p95

1.12789143e+06

/s

peak

1.91494301e+06

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.005635

s

SD

0.002488

s

p50

0.004876

s

<

p95

0.010457

s

<

p99

0.010874

s

Zome call duration: 'crud::get_sample_entry_validation_receipts'

Duration of this zome call as measured by Holochain internally.

mean

0.014413

s

SD

0.007447

s

p50

0.008869

s

<

p95

0.026185

s

<

p99

0.03206

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004706

s

SD

0.002106

s

p50

0.004058

s

<

p95

0.008786

s

<

p99

0.009127

s

WASM call duration: 'crud::get_sample_entry_validation_receipts'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.013392

s

SD

0.007339

s

p50

0.007792

s

<

p95

0.024779

s

<

p99

0.030522

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001798

s

SD

0.000665

s

p50

0.001545

s

<

p95

0.003247

s

≤

p99

0.003247

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001617

s

SD

0.000698

s

p50

0.001307

s

<

p95

0.003032

s

<

p99

0.003123

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000438

s

SD

0.000155

s

p50

0.000361

s

<

p95

0.000707

s

<

p99

0.000823

s

Host function call duration: '__hc__get_validation_receipts_1'

Duration of this host function call invoked from within WASM.

mean

0.011394

s

SD

0.007089

s

p50

0.005796

s

<

p95

0.021841

s

<

p99

0.027196

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002367

s

SD

0.001074

s

p50

0.002039

s

<

p95

0.004376

s

<

p99

0.004558

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000154

s

SD

8.2e-05

s

p50

0.00013

s

<

p95

0.000349

s

<

p99

0.0004

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.38

s

<

mean

442.64

s

SD

254.61

s

<

p95

837.76

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

6095

over 888.326440337s

mean rate

504.23

/s

std

1012.83

/s

p5

0

/s

p95

2491.74

/s

peak

9980.25

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.533071

s

SD

1.78472

s

p50

1.465533

s

<

p95

4.901614

s

<

p99

8.382745

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.003087

s

SD

0.004104

s

p50

0.001975

s

<

p95

0.015172

s

<

p99

0.017868

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.038492

s

SD

0.0243

s

p50

0.044291

s

<

p95

0.085358

s

<

p99

0.137885

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.023861

s

SD

0.009791

s

p50

0.014511

s

<

p95

0.036918

s

<

p99

0.042347

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.055318

s

SD

0.752794

s

p50

0.058234

s

<

p95

1.690326

s

<

p99

4.158088

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.692315

s

SD

3.398482

s

p50

0.669909

s

<

p95

10.136352

s

<

p99

14.236939

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000658

s

SD

0.000252

s

p50

0.000505

s

<

p95

0.001042

s

<

p99

0.001229

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00296

s

SD

0.003406

s

p50

0.00143

s

<

p95

0.011011

s

<

p99

0.018988

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000416

s

SD

0.000148

s

p50

0.000352

s

<

p95

0.000684

s

<

p99

0.000699

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.089431

s

SD

12.504883

s

p50

0.041771

s

<

p95

0.18526

s

<

p99

60.002242

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.117991

s

SD

0.040897

s

p50

0.096593

s

<

p95

0.195329

s

<

p99

0.204954

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

10.37

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

0

B/s

<

mean

98.75

KiB/s

SD

639.21

KiB/s

≤

p95

62.63

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

0

B/s

<

mean

21.97

KiB/s

SD

19.55

KiB/s

<

p95

47.32

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.50

GiB

mean

1.73

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

342.42

MiB

mean

393.98

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Warning Moderate swap usage (17.2% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 5% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.94

%

<

mean

17.19

%

SD

16.45

%

<

p95

63.74

%

System

6.82

%

CPU percentiles

CPU usage percentiles

p50

12.55

%

p95

63.74

%

p99

64.91

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

4

hosts

mean time

10.11

s

Memory used percentage

Percentage of memory used

p5

2.33

%

<

mean

18.69

%

SD

8.98

%

<

p95

37.09

%

Memory available percentage

Percentage of available memory

p5

62.91

%

<

mean

81.31

%

SD

8.98

%

<

p95

97.67

%

Max host memory used

Maximum memory usage percentage across all hosts

max

37.71

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

17.22

%

Memory growth rate

Rate of memory growth over time

growth

0.16

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.17

MiB/s

Disk write throughput

Disk write throughput in MiB/s

32.32

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/9

hosts

Mount Point /etc/hosts

0/9

hosts

Mount Point /etc/nomad.d/client-meta.json

0/8

hosts

Mount Point /etc/resolv.conf

0/9

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.09

5 min

1.87

15 min

2.94

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

5.26

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

3.8452

%

SD

8.224

%

<

p95

31.79

%

60 second average

4.8629

%

300 second average

10.0894

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.848

%

SD

11.433

%

<

p95

39.51

%

60 second average

4.0250

%

300 second average

4.5346

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.4093

%

SD

10.2728

%

<

p95

35.61

%

60 second average

3.4307

%

300 second average

3.1182

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.67

%

<

mean

8.73

%

SD

6.75

%

<

p95

19.83

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

229.67

KiB

<

mean

271.59

KiB

SD

64.06

KiB

<

p95

463.31

KiB

Holochain process threads

Number of threads in Holochain process

p5

27

threads

<

mean

37.02

threads

SD

21.24

threads

<

p95

125

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

71

file descriptors

<

mean

77.13

file descriptors

SD

12.99

file descriptors

<

p95

117

file descriptors

Write/`get_agent_activity`

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 01 Oct 2026 11:57:30 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_36830042121_1

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

3872

Max

Highest chain head value observed for any single write agent.

5200

Write agents observed

4

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.229636

s

SD

0.151596

s

Max

Highest per-partition mean latency

max mean

0.57806

s

Min

Lowest per-partition mean latency

min mean

0.015341

s

Error count

The number of errors accumulated.

46

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.053139

s

SD

0.336291

s

p50

0.100672

s

<

p95

1.040458

s

<

p99

1.576365

s

WASM usage: 'agent_activity::announce_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 885.91387994s

Not enough time data to show a trend.

WASM usage: 'agent_activity::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

169258320

over 885.91387994s

mean rate

4.71777525e+07

/s

std

8.40647308e+07

/s

p5

2.06886605e+06

/s

p95

2.4541321923e+08

/s

peak

4.9548134095e+08

/s

WASM usage: 'agent_activity::get_agent_activity_full'

Metered usage count of the WASM ribosome for this zome function.

total

287046473

over 887.234863645s

mean rate

7.70176704577e+09

/s

std

1.14721131742e+10

/s

p5

3.079059149e+07

/s

p95

2.89813467118e+10

/s

peak

6.21009484742e+10

/s

WASM usage: 'agent_activity::get_random_agent_with_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

250362178

over 888.042648557s

mean rate

6.5405551812e+08

/s

std

1.40907473075e+09

/s

p5

0

/s

p95

4.63355940609e+09

/s

peak

7.1244928348e+09

/s

WASM usage: 'agent_activity_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

18126450

over 888.338300452s

mean rate

5.422008724e+07

/s

std

9.372459097e+07

/s

p5

0

/s

p95

3.0035284242e+08

/s

peak

5.1629851581e+08

/s

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.005085

s

SD

0.002149

s

p50

0.005305

s

<

p95

0.008742

s

≤

p99

0.008742

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004495

s

SD

0.002124

s

p50

0.004382

s

<

p95

0.008173

s

<

p99

0.008256

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.104873

s

SD

0.16053

s

p50

0.148202

s

<

p95

0.459695

s

<

p99

0.662728

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.005041

s

SD

0.41818

s

p50

0.00692

s

<

p95

1.044585

s

<

p99

1.579641

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004374

s

SD

0.001879

s

p50

0.004696

s

<

p95

0.007404

s

≤

p99

0.007404

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003772

s

SD

0.001823

s

p50

0.003767

s

<

p95

0.006983

s

<

p99

0.007105

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.104125

s

SD

0.160514

s

p50

0.147554

s

<

p95

0.458976

s

<

p99

0.662062

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004333

s

SD

0.418169

s

p50

0.005849

s

<

p95

1.043902

s

<

p99

1.578891

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00135

s

SD

0.00062

s

p50

0.001146

s

<

p95

0.002609

s

<

p99

0.002786

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

3.1e-05

s

SD

1.8e-05

s

p50

1.8e-05

s

<

p95

6.6e-05

s

≤

p99

6.6e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000462

s

SD

0.000229

s

p50

0.000473

s

<

p95

0.000853

s

<

p99

0.000895

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.00038

s

SD

0.000164

s

p50

0.000306

s

<

p95

0.000691

s

≤

p99

0.000691

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.102481

s

SD

0.160256

s

p50

0.145941

s

<

p95

0.457603

s

<

p99

0.660271

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000639

s

SD

0.418478

s

p50

0.000548

s

<

p95

1.040491

s

<

p99

1.576413

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1.3e-05

s

SD

7e-06

s

p50

1e-05

s

<

p95

3.1e-05

s

≤

p99

3.1e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002108

s

SD

0.001033

s

p50

0.001908

s

<

p95

0.004461

s

<

p99

0.004709

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000126

s

SD

0.000109

s

p50

9.8e-05

s

<

p95

0.000422

s

<

p99

0.000495

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.61

s

<

mean

444.35

s

SD

255.2

s

<

p95

838.43

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

4876

over 888.338227885s

mean rate

53087.05

/s

std

124118.58

/s

p5

55.3

/s

p95

316665.4

/s

peak

896112.46

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

209.863075

s

SD

161.824665

s

p50

2.668801

s

<

p95

417.022287

s

<

p99

598.611736

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.026027

s

SD

0.179577

s

p50

0.123309

s

<

p95

0.438149

s

<

p99

0.877019

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.553394

s

SD

0.396347

s

p50

1.012373

s

<

p95

1.18919

s

<

p99

1.313656

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.129575

s

SD

0.061116

s

p50

0.029431

s

<

p95

0.183156

s

<

p99

0.242956

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

4.143993

s

SD

30.010677

s

p50

11.008008

s

<

p95

86.713531

s

<

p99

107.822962

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

16.72482

s

SD

29.518691

s

p50

0.181322

s

<

p95

78.239303

s

<

p99

140.49602

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002726

s

SD

0.001192

s

p50

0.002582

s

<

p95

0.004571

s

<

p99

0.005108

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002151

s

SD

0.00379

s

p50

0.001206

s

<

p95

0.007782

s

<

p99

0.022079

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000362

s

SD

0.000209

s

p50

0.000317

s

<

p95

0.000792

s

<

p99

0.000936

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.402917

s

SD

27.95898

s

p50

0.23595

s

<

p95

60.001741

s

<

p99

60.001878

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.593357

s

SD

0.239528

s

p50

0.523108

s

<

p95

1.039251

s

≤

p99

1.039251

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.07102

s

SD

28.550482

s

p50

0.150345

s

<

p95

60.00252

s

<

p99

60.006021

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.083818

s

SD

0.772152

s

p50

0.107374

s

<

p95

1.100116

s

<

p99

3.686613

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

49.46

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

75.41

KiB/s

<

mean

289.99

KiB/s

SD

430.00

KiB/s

<

p95

567.45

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

28.01

KiB/s

<

mean

258.90

KiB/s

SD

289.36

KiB/s

<

p95

901.68

KiB/s

Total bytes received

Total bytes received on primary network interface

count

6.64

GiB

mean

7.64

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.93

GiB

mean

6.82

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (74.8% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 43% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

14.81

%

<

mean

61.39

%

SD

28.28

%

<

p95

97.09

%

System

11.93

%

CPU percentiles

CPU usage percentiles

p50

64.20

%

p95

97.09

%

p99

98.65

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

13

hosts

mean time

718.44

s

Memory used percentage

Percentage of memory used

p5

3.92

%

<

mean

19.27

%

SD

15.3

%

<

p95

39.03

%

Memory available percentage

Percentage of available memory

p5

60.97

%

<

mean

80.73

%

SD

15.3

%

<

p95

96.08

%

Max host memory used

Maximum memory usage percentage across all hosts

max

84.34

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

74.77

%

Memory growth rate

Rate of memory growth over time

growth

0.27

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.70

MiB/s

Disk write throughput

Disk write throughput in MiB/s

145.53

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.83

5 min

3.07

15 min

2.73

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

42.86

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

21.7267

%

SD

22.9758

%

<

p95

64.32

%

60 second average

20.7792

%

300 second average

16.4550

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0005

%

SD

0.012

%

≤

p95

0

%

60 second average

0.0004

%

300 second average

0.0001

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0004

%

SD

0.0115

%

≤

p95

0

%

60 second average

0.0004

%

300 second average

0.0001

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.7514

%

SD

9.3518

%

<

p95

28.6

%

60 second average

3.6809

%

300 second average

3.5035

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.6889

%

SD

5.5714

%

<

p95

8.23

%

60 second average

1.7185

%

300 second average

1.9494

%

Holochain process CPU usage

CPU usage by Holochain process

p5

13.81

%

<

mean

54.95

%

SD

26

%

<

p95

95.16

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

273.88

KiB

<

mean

366.30

KiB

SD

99.16

KiB

<

p95

486.19

KiB

Holochain process threads

Number of threads in Holochain process

p5

30

threads

<

mean

41.71

threads

SD

26.91

threads

<

p95

68

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

71

file descriptors

<

mean

85.94

file descriptors

SD

26.87

file descriptors

<

p95

113

file descriptors

Write/`get_agent_activity` with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 01 Oct 2026 12:13:15 UTC

Duration

1800s

Peer count

30

Peer count at end

26

Behaviours

get_agent_activity_volatile (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_volatile_36830042121_1

Highest observed action_seq

The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.

Mean

mean

42.79

/s

Max

Highest per-partition mean rate

max

86.01

/s

Min

Lowest per-partition mean rate

min

20.47

/s

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.184897

s

SD

0.141109

s

Max

Highest per-partition mean latency

max mean

0.329411

s

Min

Lowest per-partition mean latency

min mean

0.068692

s

Running volatile conductors count

The number of conductors running by get_agent_activity_volatile peers.

p5

9

conductors

<

mean

13.39

conductors

SD

2.39

conductors

<

p95

17

conductors

Volatile conductor total running duration

The total running duration of a get_agent_activity_volatile conductor.

Mean

mean

604.729191

s

SD

328.343406

s

Max

Highest per-partition mean latency

max mean

671.446809

s

Min

Lowest per-partition mean latency

min mean

508.175

s

Volatile conductor running duration

The duration that a get_agent_activity_volatile conductor was running before being stopped.

Mean

mean

27.940363

s

SD

19.275376

s

Max

Highest per-partition mean latency

max mean

33.637988

s

Min

Lowest per-partition mean latency

min mean

23.654695

s

Volatile conductor stopped duration

The duration that a get_agent_activity_volatile conductor was stopped before being started again.

Mean

mean

21.990913

s

SD

18.274361

s

Max

Highest per-partition mean latency

max mean

31.603267

s

Min

Lowest per-partition mean latency

min mean

15.237821

s

Reached target arc

Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.

get_agent_activity_volatile_agent: uhCAk-8c8Bwqe5BHbjRgFjKOXsgHo1ME6pknOUyU7zfHXAgVv0Duy

p5

0

<

mean

0.08

SD

0.26

<

p95

1

get_agent_activity_volatile_agent: uhCAk-RZUtwWKfce9RWPRTvFCxtRRHi9PxdUVU4au8JajGhSRHSFE

p5

0

<

mean

0.02

SD

0.14

≤

p95

0

get_agent_activity_volatile_agent: uhCAk62slmjCyMIPH10ooKZKyhIHfiOnjKRL7peOobvFmiR_Lv8fi

p5

0

<

mean

0.05

SD

0.22

≤

p95

0

get_agent_activity_volatile_agent: uhCAkBF4uEIElc-yxMo4K8OW0jyYdMWgxL_qmY1lkd-0XKL9gu1MT

p5

0

<

mean

0.05

SD

0.22

≤

p95

0

get_agent_activity_volatile_agent: uhCAkF0UVMdPR8OyVJG_FjNlpsa3OHsK76bqsNqm6uCyNVtHTU9nC

p5

0

<

mean

0.06

SD

0.24

<

p95

1

get_agent_activity_volatile_agent: uhCAkKKjJf-n1Htt9LQmAfOpdG2VSfMT2KCc-XqSrq-vLcr5FmgmX

p5

0

<

mean

0.04

SD

0.2

≤

p95

0

get_agent_activity_volatile_agent: uhCAkL0ZpvG88c85desiKhcEj8YOx6GS2TBe3bJrUpCN_6SWZ_qen

p5

0

<

mean

0.04

SD

0.19

≤

p95

0

get_agent_activity_volatile_agent: uhCAkNdwoETjLpMpm5UeN8MEqEJ9c3uuG2c2NDpLMuG4_mxEWfI0n

p5

0

<

mean

0.03

SD

0.16

≤

p95

0

get_agent_activity_volatile_agent: uhCAkOE7bYDmTJ7Kw9Gcxc3tIAvh3ovks-76UaErBnTYpDPQVjayi

p5

0

<

mean

0.03

SD

0.16

≤

p95

0

get_agent_activity_volatile_agent: uhCAkR9y9-IiF2hPSOSVxKIFnewWQKHVx0kC-kKtjZ3wIR-5U6eji

p5

0

<

mean

0.08

SD

0.27

<

p95

1

get_agent_activity_volatile_agent: uhCAkUr45wNU1twY42exmgdZlolsAUOz44EmTXO67-VvQtakPdbTA

p5

0

<

mean

0.04

SD

0.2

≤

p95

0

get_agent_activity_volatile_agent: uhCAkWDueYyuBN5mRpwgU9MLHfVu06VvsiztS5RenQs1vnK8-rZgw

p5

0

<

mean

0.04

SD

0.2

≤

p95

0

get_agent_activity_volatile_agent: uhCAkWapRcyc3Buzdn9SLUL18f79FmuKsqU2SpExCCl25D0QdEVje

p5

0

<

mean

0.06

SD

0.23

<

p95

1

get_agent_activity_volatile_agent: uhCAkavpusK6NXszqaJ6y1TdP4wgdD8RkfackiGqk3qWm5xF1wF3p

p5

0

<

mean

0.05

SD

0.22

<

p95

1

get_agent_activity_volatile_agent: uhCAkjFq2bhTZS2AzQd0uNm0mTVdFhoXED9DKYLVQ4XhOuLihgIwo

p5

0

<

mean

0.03

SD

0.17

≤

p95

0

get_agent_activity_volatile_agent: uhCAkki0FgjieMxf3Bcd5XZTWqVLmnvq7cJPp4h4bURQnYDmEe-1x

p5

0

<

mean

0.07

SD

0.25

<

p95

1

get_agent_activity_volatile_agent: uhCAkrKEL2skL3ZhKheThpBaYommHPHuf7lJ6NmjvoWMEdRU1K064

p5

0

<

mean

0.07

SD

0.26

<

p95

1

get_agent_activity_volatile_agent: uhCAkrZw3mI6lqxTLBQ9I3U-l2a346b6IhvyAyozOHcHzFNjkeAmS

p5

0

<

mean

0.04

SD

0.2

≤

p95

0

get_agent_activity_volatile_agent: uhCAksm18q9biW0QhO0CcCMdeLaTkIPFgRKl4G3E1PQKVqDYtLwTB

p5

0

<

mean

0.04

SD

0.2

≤

p95

0

get_agent_activity_volatile_agent: uhCAkuHVD5IzI1S5-6BCt0a4sSyN2b95IHXlf_cEzo8Na-VA-qUTg

p5

0

<

mean

0.08

SD

0.26

<

p95

1

Error count

The number of errors accumulated.

145

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.074944

s

SD

0.178455

s

p50

0.192389

s

<

p95

0.508745

s

<

p99

0.821512

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

1

over 1698.515934836s

mean rate

0.79

/s

std

1.23

/s

p5

0

/s

p95

2.78

/s

peak

7.05

/s

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.004166

s

SD

0.001341

s

p50

0.003562

s

<

p95

0.006398

s

≤

p99

0.006398

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.005518

s

SD

0.001826

s

p50

0.005431

s

<

p95

0.009036

s

<

p99

0.009482

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.106418

s

SD

0.16544

s

p50

0.206914

s

<

p95

0.555386

s

<

p99

0.814573

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.00353

s

SD

0.39903

s

p50

0.004094

s

<

p95

0.483909

s

<

p99

2.651841

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003526

s

SD

0.001203

s

p50

0.003092

s

<

p95

0.005558

s

≤

p99

0.005558

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00476

s

SD

0.001714

s

p50

0.004643

s

<

p95

0.008225

s

<

p99

0.008668

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.105608

s

SD

0.163131

s

p50

0.19879

s

<

p95

0.517983

s

<

p99

0.779074

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003

s

SD

0.398987

s

p50

0.003481

s

<

p95

0.483371

s

<

p99

2.65099

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001601

s

SD

0.000582

s

p50

0.001568

s

<

p95

0.002692

s

<

p99

0.002787

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2e-05

s

SD

7e-06

s

p50

1.8e-05

s

<

p95

3.5e-05

s

≤

p99

3.5e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000636

s

SD

0.000339

s

p50

0.00057

s

<

p95

0.001473

s

<

p99

0.001577

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000346

s

SD

0.00011

s

p50

0.000293

s

<

p95

0.000526

s

≤

p99

0.000526

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.103753

s

SD

0.162068

s

p50

0.19653

s

<

p95

0.512281

s

<

p99

0.771652

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000486

s

SD

0.398848

s

p50

0.000401

s

<

p95

0.481653

s

<

p99

2.647364

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

6e-06

s

SD

3e-06

s

p50

7e-06

s

<

p95

1e-05

s

<

p99

1.4e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00256

s

SD

0.000922

s

p50

0.002112

s

<

p95

0.004052

s

<

p99

0.004793

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000151

s

SD

6.3e-05

s

p50

0.000134

s

<

p95

0.000271

s

<

p99

0.000278

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

3.13

s

<

mean

340.33

s

SD

518.09

s

<

p95

1534.92

s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

200.022067

s

SD

261.810575

s

p50

19.547388

s

<

p95

788.540521

s

<

p99

1267.763837

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.038099

s

SD

0.117976

s

p50

0.005853

s

<

p95

0.361695

s

<

p99

0.454313

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.278088

s

SD

0.259516

s

p50

0.008033

s

<

p95

0.719064

s

<

p99

0.926279

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.252803

s

SD

0.370123

s

p50

0.092641

s

<

p95

1.293673

s

<

p99

1.678824

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

7.107868

s

SD

24.456284

s

p50

3.040623

s

<

p95

71.867632

s

<

p99

89.349083

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

48.873129

s

SD

54.236223

s

p50

22.515175

s

<

p95

157.098159

s

<

p99

160.825533

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.007473

s

SD

0.003341

s

p50

0.001728

s

<

p95

0.010803

s

<

p99

0.013208

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.008486

s

SD

0.01454

s

p50

0.000977

s

<

p95

0.020149

s

<

p99

0.045066

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000523

s

SD

0.000254

s

p50

0.000452

s

<

p95

0.001019

s

<

p99

0.001529

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

5.182278

s

SD

28.4753

s

p50

0.415784

s

<

p95

60.001762

s

<

p99

60.001984

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.238255

s

SD

0.525252

s

p50

0.416548

s

<

p95

2.645475

s

≤

p99

2.645475

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.100312

s

SD

3.643042

s

p50

0.196316

s

<

p95

0.524165

s

<

p99

0.83057

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.240259

s

SD

26.05173

s

p50

0.229153

s

<

p95

60.00174

s

<

p99

60.001902

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

27.84

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

1.62

KiB/s

<

mean

198.43

KiB/s

SD

437.73

KiB/s

<

p95

485.02

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

1.82

KiB/s

<

mean

166.29

KiB/s

SD

252.23

KiB/s

<

p95

666.36

KiB/s

Total bytes received

Total bytes received on primary network interface

count

9.28

GiB

mean

5.31

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.78

GiB

mean

4.45

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (72.5% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 24% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.43

%

<

mean

36.6

%

SD

34.91

%

<

p95

98.33

%

System

8.76

%

CPU percentiles

CPU usage percentiles

p50

21.61

%

p95

98.33

%

p99

98.75

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

9

hosts

mean time

1209.37

s

Memory used percentage

Percentage of memory used

p5

3.56

%

<

mean

18.34

%

SD

14.74

%

<

p95

36.11

%

Memory available percentage

Percentage of available memory

p5

63.89

%

<

mean

81.66

%

SD

14.74

%

<

p95

96.44

%

Max host memory used

Maximum memory usage percentage across all hosts

max

82.26

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

72.54

%

Memory growth rate

Rate of memory growth over time

growth

0.18

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1.32

MiB/s

Disk write throughput

Disk write throughput in MiB/s

47.55

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/14

hosts

Mount Point /etc/hosts

0/14

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/14

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.79

5 min

2.88

15 min

2.90

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

24.14

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

18.1271

%

SD

31.2284

%

<

p95

92.84

%

60 second average

17.9251

%

300 second average

17.6442

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.4328

%

SD

7.0358

%

<

p95

20.81

%

60 second average

2.5105

%

300 second average

2.8301

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.7077

%

SD

3.2635

%

<

p95

3.76

%

60 second average

0.7831

%

300 second average

1.0565

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

28.46

%

SD

36.35

%

<

p95

97.53

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

52.26

KiB

<

mean

228.00

KiB

SD

156.25

KiB

<

p95

510.05

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

26.52

threads

SD

16.11

threads

<

p95

44

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

63.45

file descriptors

SD

28.63

file descriptors

<

p95

86

file descriptors

Write/Query

↑ Back to index

Creates an entry, then queries the source chain and performs a simple operation on the entries, then repeats.

Started

Thu, 01 Oct 2026 10:59:45 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_query_36830042121_1

Write timing

The time it takes to complete the zome function call that writes the entry.

mean

0.037686

s

SD

0.0214

s

p50

0.034033

s

<

p95

0.073967

s

<

p99

0.102452

s

Write rate

The number of writes completed per second.

mean

77.57

/s

Query timing

The time it takes to complete the chain_query_count_len zome call that queries the source chain.

mean

0.077136

s

SD

0.05

s

p50

0.066466

s

<

p95

0.162618

s

<

p99

0.21787

s

Query rate

The number of chain queries completed per second.

mean

77.57

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::chain_query_count_len'

Metered usage count of the WASM ribosome for this zome function.

total

69132126486

over 280.003782535s

mean rate

2.4689719346e+08

/s

std

2.639708078e+07

/s

p5

1.9372175124e+08

/s

p95

2.8016125545e+08

/s

peak

2.8437453194e+08

/s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

80925495

over 280.003782535s

mean rate

289016.54

/s

std

192121.9

/s

p5

144066.49

/s

p95

682665.3

/s

peak

1.01885142e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.003782535s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

20075552

over 280.003782535s

mean rate

71697.64

/s

std

47626.52

/s

p5

35321.24

/s

p95

169387.76

/s

peak

252379.05

/s

Zome call duration: 'crud::chain_query_count_len'

Duration of this zome call as measured by Holochain internally.

mean

0.029191

s

SD

0.009253

s

p50

0.027025

s

<

p95

0.038333

s

<

p99

0.039124

s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.003075

s

SD

0.000143

s

p50

0.003035

s

<

p95

0.003244

s

<

p99

0.003264

s

WASM call duration: 'crud::chain_query_count_len'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.028528

s

SD

0.009205

s

p50

0.02637

s

<

p95

0.037626

s

<

p99

0.038414

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002495

s

SD

0.000123

s

p50

0.002462

s

<

p95

0.002643

s

<

p99

0.002661

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00119

s

SD

0

s

p50

0.00119

s

≤

p95

0.00119

s

≤

p99

0.00119

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00081

s

SD

4.5e-05

s

p50

0.000796

s

<

p95

0.000867

s

<

p99

0.000872

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000286

s

SD

1.6e-05

s

p50

0.000288

s

<

p95

0.000311

s

<

p99

0.000316

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.012042

s

SD

0.003932

s

p50

0.01118

s

<

p95

0.015821

s

<

p99

0.016156

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001287

s

SD

7.7e-05

s

p50

0.001266

s

<

p95

0.001399

s

<

p99

0.001417

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.8e-05

s

SD

6e-06

s

p50

8.8e-05

s

<

p95

9.4e-05

s

<

p99

9.5e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.52

s

<

mean

147.52

s

SD

83.67

s

<

p95

277.52

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.003750597s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000463

s

SD

0

s

p50

0.000463

s

≤

p95

0.000463

s

≤

p99

0.000463

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001144

s

SD

0.000117

s

p50

0.001108

s

<

p95

0.001281

s

<

p99

0.001296

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.024742

s

SD

0.008925

s

p50

0.022413

s

<

p95

0.034067

s

<

p99

0.034804

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.00331

s

SD

0

s

p50

0.00331

s

≤

p95

0.00331

s

≤

p99

0.00331

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.004924

s

SD

0

s

p50

0.004924

s

≤

p95

0.004924

s

≤

p99

0.004924

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002507

s

SD

0.000876

s

p50

0.00232

s

<

p95

0.003353

s

<

p99

0.00342

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000944

s

SD

0.000414

s

p50

0.00172

s

≤

p95

0.00172

s

≤

p99

0.00172

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000253

s

SD

2.1e-05

s

p50

0.000248

s

<

p95

0.00028

s

<

p99

0.000283

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

58.65

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.14

KiB/s

<

mean

958.35

KiB/s

SD

2.13

MiB/s

<

p95

8.17

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.69

KiB/s

<

mean

37.57

KiB/s

SD

43.55

KiB/s

<

p95

170.62

KiB/s

Total bytes received

Total bytes received on primary network interface

count

262.05

MiB

mean

958.35

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.27

MiB

mean

37.57

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.7%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.88 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

67.26

%

<

mean

79.11

%

SD

9.22

%

<

p95

91.64

%

System

20.46

%

CPU percentiles

CPU usage percentiles

p50

81.32

%

p95

91.64

%

p99

95.68

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

160.71

s

Memory used percentage

Percentage of memory used

p5

8.56

%

<

mean

9.51

%

SD

0.7

%

<

p95

10.49

%

Memory available percentage

Percentage of available memory

p5

89.51

%

<

mean

90.49

%

SD

0.7

%

<

p95

91.44

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.51

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

1.88

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

6.93

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.45

5 min

3.01

15 min

1.42

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

18.36

%

<

mean

32.7107

%

SD

12.7757

%

<

p95

47.87

%

60 second average

28.9217

%

300 second average

17.9438

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

24.5

%

<

mean

31.93

%

SD

4.4

%

<

p95

38.55

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

134.92

KiB

<

mean

280.79

KiB

SD

75.13

KiB

<

p95

394.32

KiB

Holochain process threads

Number of threads in Holochain process

p5

9

threads

<

mean

28.43

threads

SD

6

threads

<

p95

35

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

12

file descriptors

<

mean

64.13

file descriptors

SD

15.06

file descriptors

<

p95

76

file descriptors

Write/Read

↑ Back to index

Creates an entry and reads it back, then repeats.

Started

Thu, 01 Oct 2026 11:01:37 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_read_36830042121_1

Write timing

The time taken to write the entry.

mean

0.045118

s

SD

0.023936

s

p50

0.041467

s

<

p95

0.082209

s

<

p99

0.114438

s

Write rate

The number of writes completed per second.

mean

100.11

/s

Read timing

The time taken to read the entry.

mean

0.048911

s

SD

0.027328

s

p50

0.044368

s

<

p95

0.093617

s

<

p99

0.1209

s

Read rate

The number of reads completed per second.

mean

100.11

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.00078

s

SD

9.9e-05

s

p50

0.000769

s

<

p95

0.000873

s

<

p99

0.000879

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

101782935

over 280.007435185s

mean rate

363502

/s

std

219274.79

/s

p5

195536.21

/s

p95

799298.01

/s

peak

1.22125116e+06

/s

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

99178020

over 280.007435185s

mean rate

354198.94

/s

std

212687.45

/s

p5

187373.81

/s

p95

782955.68

/s

peak

1.18138938e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.007435185s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

12627552

over 280.007435185s

mean rate

45097.35

/s

std

27204.01

/s

p5

24258.92

/s

p95

99163.75

/s

peak

151512.75

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.003643

s

SD

6.9e-05

s

p50

0.00364

s

<

p95

0.003698

s

≤

p99

0.003698

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.002596

s

SD

0.000136

s

p50

0.002584

s

<

p95

0.002724

s

<

p99

0.002733

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002995

s

SD

6.3e-05

s

p50

0.002994

s

<

p95

0.003044

s

<

p99

0.003045

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001944

s

SD

0.000124

s

p50

0.001935

s

<

p95

0.002061

s

<

p99

0.00207

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001218

s

SD

0

s

p50

0.001218

s

≤

p95

0.001218

s

≤

p99

0.001218

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000941

s

SD

1.3e-05

s

p50

0.00094

s

<

p95

0.000951

s

<

p99

0.000953

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000387

s

SD

5.8e-05

s

p50

0.000362

s

<

p95

0.000398

s

<

p99

0.0004

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000815

s

SD

9.9e-05

s

p50

0.000805

s

<

p95

0.000908

s

<

p99

0.000914

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001521

s

SD

3.5e-05

s

p50

0.001522

s

<

p95

0.001546

s

<

p99

0.00155

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000109

s

SD

2e-06

s

p50

0.000109

s

<

p95

0.000112

s

<

p99

0.000113

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.33

s

<

mean

147.34

s

SD

83.67

s

<

p95

277.34

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.007439123s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001191

s

SD

0

s

p50

0.001191

s

≤

p95

0.001191

s

≤

p99

0.001191

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001669

s

SD

0.000117

s

p50

0.001637

s

<

p95

0.001753

s

≤

p99

0.001753

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.033543

s

SD

0.011263

s

p50

0.03089

s

<

p95

0.044057

s

<

p99

0.045016

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002321

s

SD

0

s

p50

0.002321

s

≤

p95

0.002321

s

≤

p99

0.002321

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001209

s

SD

0

s

p50

0.001209

s

≤

p95

0.001209

s

≤

p99

0.001209

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00274

s

SD

0.000945

s

p50

0.002531

s

<

p95

0.003581

s

<

p99

0.003653

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001253

s

SD

9.8e-05

s

p50

0.00138

s

≤

p95

0.00138

s

≤

p99

0.00138

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000351

s

SD

1.4e-05

s

p50

0.000354

s

<

p95

0.000363

s

≤

p99

0.000363

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

66.56

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

21.14

KiB/s

<

mean

1.06

MiB/s

SD

1.88

MiB/s

<

p95

5.71

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.34

KiB/s

<

mean

40.04

KiB/s

SD

37.88

KiB/s

<

p95

111.48

KiB/s

Total bytes received

Total bytes received on primary network interface

count

296.96

MiB

mean

1.06

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.95

MiB

mean

40.04

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

79.18

%

<

mean

85.96

%

SD

4.39

%

<

p95

93.68

%

System

19.40

%

CPU percentiles

CPU usage percentiles

p50

86.04

%

p95

93.68

%

p99

95.69

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

278.57

s

Memory used percentage

Percentage of memory used

p5

9.03

%

<

mean

9.95

%

SD

0.46

%

<

p95

10.56

%

Memory available percentage

Percentage of available memory

p5

89.44

%

<

mean

90.05

%

SD

0.46

%

<

p95

90.97

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.95

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

0.22

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

7.52

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.51

5 min

3.81

15 min

1.89

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

26.26

%

<

mean

40.7969

%

SD

8.8979

%

<

p95

49.44

%

60 second average

36.9524

%

300 second average

24.1331

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.45

%

<

mean

29.25

%

SD

7.09

%

<

p95

38.58

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

20.25

KiB

<

mean

272.55

KiB

SD

120.57

KiB

<

p95

403.04

KiB

Holochain process threads

Number of threads in Holochain process

p5

7

threads

<

mean

26.39

threads

SD

8.72

threads

<

p95

35

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

58.42

file descriptors

SD

21.55

file descriptors

<

p95

76

file descriptors

Write validated

↑ Back to index

Creates an entry, updates it and links the create to the update, then repeats. Each of the three actions is validated by the included hApp.

Started

Thu, 01 Oct 2026 11:03:41 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_36830042121_1

Write timing

The time it takes to complete the zome function call that writes the entry, updates it, and links to the update.

mean

0.083462

s

SD

0.035707

s

p50

0.086714

s

<

p95

0.136274

s

<

p99

0.195967

s

Write rate

The number of write zome function calls that complete per second.

mean

46.89

/s

Read timing

The time it takes to complete the zome function call that reads the data.

mean

0.124068

s

SD

0.048284

s

p50

0.125233

s

<

p95

0.190082

s

<

p99

0.313028

s

Read rate

The number of read zome function calls that complete per second.

mean

46.89

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.005899

s

SD

0.001971

s

p50

0.005489

s

<

p95

0.008732

s

<

p99

0.008995

s

WASM usage: 'validated::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

42313362

over 270.004549422s

mean rate

156713.28

/s

std

58041.52

/s

p5

102959.57

/s

p95

272344.89

/s

peak

335449.32

/s

WASM usage: 'validated::update_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

49998348

over 270.004549422s

mean rate

185175.68

/s

std

68308.82

/s

p5

121754.74

/s

p95

318133.6

/s

peak

396685.25

/s

WASM usage: 'validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

82515316

over 280.004596973s

mean rate

294692.34

/s

std

107218.86

/s

p5

192890.44

/s

p95

501427.25

/s

peak

627667.09

/s

WASM usage: 'validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

741315953

over 270.004549422s

mean rate

2.74556427e+06

/s

std

1.01625027e+06

/s

p5

1.81924627e+06

/s

p95

4.7244212e+06

/s

peak

5.91029413e+06

/s

Zome call duration: 'validated::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004151

s

SD

0.000105

s

p50

0.004155

s

<

p95

0.004241

s

<

p99

0.004252

s

Zome call duration: 'validated::update_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.011308

s

SD

0.002216

s

p50

0.010907

s

<

p95

0.013252

s

<

p99

0.013375

s

WASM call duration: 'validated::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003457

s

SD

9.2e-05

s

p50

0.003459

s

<

p95

0.003539

s

<

p99

0.00355

s

WASM call duration: 'validated::update_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.010613

s

SD

0.00221

s

p50

0.010209

s

<

p95

0.012559

s

<

p99

0.012685

s

WASM call duration: 'validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000982

s

SD

0.000144

s

p50

0.000982

s

<

p95

0.001027

s

<

p99

0.00177

s

WASM call duration: 'validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003825

s

SD

0.000248

s

p50

0.0038

s

<

p95

0.004028

s

<

p99

0.004037

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000436

s

SD

1.7e-05

s

p50

0.000437

s

<

p95

0.000445

s

<

p99

0.000446

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000406

s

SD

1.9e-05

s

p50

0.000404

s

<

p95

0.000454

s

<

p99

0.000463

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.005433

s

SD

0.001517

s

p50

0.005104

s

<

p95

0.006783

s

<

p99

0.006894

s

Host function call duration: '__hc__update_1'

Duration of this host function call invoked from within WASM.

mean

0.00732

s

SD

0.002192

s

p50

0.006879

s

<

p95

0.009297

s

<

p99

0.009434

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001636

s

SD

8.2e-05

s

p50

0.001681

s

<

p95

0.001846

s

<

p99

0.001882

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000129

s

SD

3e-06

s

p50

0.000129

s

<

p95

0.000137

s

<

p99

0.000138

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.08

s

<

mean

147.08

s

SD

83.67

s

<

p95

277.09

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.004586789s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.00124

s

SD

0

s

p50

0.00124

s

≤

p95

0.00124

s

≤

p99

0.00124

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001788

s

SD

0.000173

s

p50

0.001828

s

<

p95

0.002022

s

<

p99

0.002041

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.050632

s

SD

0.017909

s

p50

0.044913

s

<

p95

0.065561

s

<

p99

0.06666

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002778

s

SD

0

s

p50

0.002778

s

≤

p95

0.002778

s

≤

p99

0.002778

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000749

s

SD

0

s

p50

0.000749

s

≤

p95

0.000749

s

≤

p99

0.000749

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003699

s

SD

0.0014

s

p50

0.003298

s

<

p95

0.00483

s

<

p99

0.004914

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00185

s

SD

0.000137

s

p50

0.001846

s

<

p95

0.001887

s

<

p99

0.001925

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.00034

s

SD

1.9e-05

s

p50

0.00034

s

<

p95

0.000349

s

<

p99

0.000432

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

64.98

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

19.10

KiB/s

<

mean

1.26

MiB/s

SD

1.89

MiB/s

<

p95

5.58

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

11.88

KiB/s

<

mean

42.21

KiB/s

SD

37.63

KiB/s

<

p95

109.28

KiB/s

Total bytes received

Total bytes received on primary network interface

count

364.94

MiB

mean

1.26

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

11.95

MiB

mean

42.21

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

74.33

%

<

mean

84.14

%

SD

6.69

%

<

p95

94.51

%

System

19.16

%

CPU percentiles

CPU usage percentiles

p50

83.58

%

p95

94.51

%

p99

95.68

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

217.24

s

Memory used percentage

Percentage of memory used

p5

8.19

%

<

mean

9.91

%

SD

1.11

%

<

p95

11.51

%

Memory available percentage

Percentage of available memory

p5

88.49

%

<

mean

90.09

%

SD

1.11

%

<

p95

91.81

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.91

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-0.98

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.06

MiB/s

Disk write throughput

Disk write throughput in MiB/s

7.24

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.01

5 min

4.58

15 min

2.42

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

22.41

%

<

mean

38.4077

%

SD

10.2112

%

<

p95

49.44

%

60 second average

39.0460

%

300 second average

29.5877

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.27

%

<

mean

25.46

%

SD

8.05

%

<

p95

43.93

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

25.32

KiB

<

mean

301.30

KiB

SD

221.46

KiB

<

p95

475.39

KiB

Holochain process threads

Number of threads in Holochain process

p5

9

threads

<

mean

33.31

threads

SD

32.8

threads

<

p95

59

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

74.44

file descriptors

SD

71.16

file descriptors

<

p95

135

file descriptors

Write Validated `must_get_agent_activity`

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started

Thu, 01 Oct 2026 12:43:58 UTC

Duration

900s

Peer count

30

Peer count at end

28

Behaviours

must_get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_must_get_agent_activity_36830042121_1

Highest observed action_seq

The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

1721

Max

Highest chain head value observed for any single write agent.

1831

Write agents observed

3

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

105.470508

s

SD

50.230873

s

Max

Highest per-partition mean latency

max mean

1047.0726

s

Min

Lowest per-partition mean latency

min mean

7.473339

s

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

2.32

/s

Max

Highest per-partition mean rate

max

4.52

/s

Min

Lowest per-partition mean rate

min

0

/s

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

0.240236

s

SD

0.248821

s

Max

Highest per-partition mean latency

max mean

0.680506

s

Min

Lowest per-partition mean latency

min mean

0.038647

s

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

3399

Agents affected

21 / 21

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

162

Max

Highest count in any single partition

674

Min

Lowest count in any single partition

9

Final error count

The total number of all types of error accumulated over the run by all agents.

70371

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.017607

s

SD

0.086765

s

p50

0.000872

s

<

p95

0.061593

s

<

p99

0.553738

s

WASM usage: 'validated_must_get_agent_activity::announce_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 882.793471254s

Not enough time data to show a trend.

WASM usage: 'validated_must_get_agent_activity::create_sample_entries_batch'

Metered usage count of the WASM ribosome for this zome function.

total

2335714250

over 882.793471254s

mean rate

2.5515264901e+08

/s

std

5.866763903e+08

/s

p5

2064.35

/s

p95

2.00817693521e+09

/s

peak

2.88230799492e+09

/s

WASM usage: 'validated_must_get_agent_activity::create_validated_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

12766131

over 881.716879729s

mean rate

5.493754638e+07

/s

std

1.1338767164e+08

/s

p5

93505.7

/s

p95

3.5958318035e+08

/s

peak

4.1336802734e+08

/s

WASM usage: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Metered usage count of the WASM ribosome for this zome function.

total

468770020

over 881.716879729s

mean rate

3.1529417464e+08

/s

std

2.9236877499e+08

/s

p5

3.16886215e+06

/s

p95

8.0973990797e+08

/s

peak

1.13544374522e+09

/s

WASM usage: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

74682

over 888.189388927s

mean rate

2.02903836e+06

/s

std

5.95219257e+06

/s

p5

0

/s

p95

1.748949833e+07

/s

peak

3.461911787e+07

/s

WASM usage: 'validated_must_get_agent_activity_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

420877125

over 888.189388927s

mean rate

1.67103064776e+09

/s

std

1.00337522643e+10

/s

p5

4.79263546e+06

/s

p95

5.05989325676e+09

/s

peak

1.56795213003e+11

/s

WASM usage: 'validated_must_get_agent_activity_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

2671806943

over 888.189388927s

mean rate

9.83189545503e+10

/s

std

4.45303837596e+11

/s

p5

6.54400355e+06

/s

p95

4.76947648631e+11

/s

peak

5.77062152962e+12

/s

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.005323

s

SD

0.002885

s

p50

0.004178

s

<

p95

0.011846

s

≤

p99

0.011846

s

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.061103

s

SD

0.018765

s

p50

0.055733

s

<

p95

0.090952

s

<

p99

0.101723

s

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004764

s

SD

0.001607

s

p50

0.005036

s

<

p95

0.008207

s

<

p99

0.008657

s

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.006205

s

SD

0.007539

s

p50

0.00501

s

<

p95

0.010588

s

<

p99

0.057354

s

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.023893

s

SD

0.151684

s

p50

0.008189

s

<

p95

0.556355

s

≤

p99

0.556355

s

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004465

s

SD

0.002443

s

p50

0.003617

s

<

p95

0.009988

s

≤

p99

0.009988

s

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.060256

s

SD

0.018646

s

p50

0.054795

s

<

p95

0.09006

s

<

p99

0.100491

s

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003997

s

SD

0.00141

s

p50

0.004209

s

<

p95

0.007145

s

<

p99

0.007668

s

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00449

s

SD

0.001891

s

p50

0.004348

s

<

p95

0.0089

s

<

p99

0.01076

s

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.022804

s

SD

0.15182

s

p50

0.006927

s

<

p95

0.555596

s

≤

p99

0.555596

s

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00108

s

SD

0.000409

s

p50

0.001103

s

<

p95

0.002095

s

<

p99

0.002333

s

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00806

s

SD

0.006163

s

p50

0.00712

s

<

p95

0.022096

s

<

p99

0.028376

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2e-05

s

SD

1.3e-05

s

p50

2.1e-05

s

<

p95

6.3e-05

s

≤

p99

6.3e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000413

s

SD

0.000153

s

p50

0.000419

s

<

p95

0.000645

s

<

p99

0.000838

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.00045

s

SD

0.000216

s

p50

0.0004

s

<

p95

0.000946

s

≤

p99

0.000946

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000736

s

SD

0.112204

s

p50

0.000647

s

<

p95

0.450051

s

<

p99

0.553774

s

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.020203

s

SD

0.015925

s

p50

0.018407

s

<

p95

0.052759

s

<

p99

0.067101

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.022998

s

SD

0.010531

s

p50

0.017292

s

<

p95

0.037014

s

<

p99

0.041459

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1.1e-05

s

SD

5e-06

s

p50

1e-05

s

<

p95

3e-05

s

≤

p99

3e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.0017

s

SD

0.000919

s

p50

0.001983

s

<

p95

0.003954

s

<

p99

0.004804

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000143

s

SD

8.3e-05

s

p50

0.000122

s

<

p95

0.000284

s

<

p99

0.000388

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.57

s

<

mean

443.66

s

SD

254.8

s

<

p95

838.15

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

50872

over 888.18916676s

mean rate

51680.57

/s

std

122920.4

/s

p5

150.63

/s

p95

254312.42

/s

peak

2.14477581e+06

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

22.155619

s

SD

29.872788

s

p50

11.702757

s

<

p95

72.901883

s

<

p99

138.196526

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.74209

s

SD

1.264047

s

p50

1.080949

s

<

p95

3.974761

s

<

p99

6.242992

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.066176

s

SD

0.116519

s

p50

0.080944

s

<

p95

0.401465

s

<

p99

0.564736

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.134475

s

SD

0.078437

s

p50

0.089733

s

<

p95

0.248389

s

<

p99

0.373353

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.263075

s

SD

4.01768

s

p50

0.305745

s

<

p95

8.961415

s

<

p99

26.767331

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

23.697186

s

SD

24.099245

s

p50

14.819255

s

<

p95

74.534929

s

<

p99

105.548504

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00287

s

SD

0.001469

s

p50

0.002634

s

<

p95

0.005047

s

<

p99

0.006655

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.006314

s

SD

0.003583

s

p50

0.001818

s

<

p95

0.010186

s

<

p99

0.018502

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000342

s

SD

9.8e-05

s

p50

0.000352

s

<

p95

0.000537

s

<

p99

0.000604

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.431479

s

SD

26.269659

s

p50

0.071317

s

<

p95

60.001686

s

<

p99

60.001859

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.29698

s

SD

0.211949

s

p50

0.163796

s

<

p95

0.552155

s

≤

p99

0.552155

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.132238

s

SD

2.745284

s

p50

0.120356

s

<

p95

0.278246

s

<

p99

15.228287

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

39.77

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

48.48

KiB/s

<

mean

139.01

KiB/s

SD

492.61

KiB/s

<

p95

158.25

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

21.57

KiB/s

<

mean

83.77

KiB/s

SD

68.39

KiB/s

<

p95

208.80

KiB/s

Total bytes received

Total bytes received on primary network interface

count

3.16

GiB

mean

3.63

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.90

GiB

mean

2.19

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 96.9%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (73.1% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 29% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

8.33

%

<

mean

51.61

%

SD

24.64

%

<

p95

89.97

%

System

11.83

%

CPU percentiles

CPU usage percentiles

p50

52.44

%

p95

89.97

%

p99

96.88

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

12

hosts

mean time

335.78

s

Memory used percentage

Percentage of memory used

p5

4.47

%

<

mean

20.92

%

SD

16.27

%

<

p95

43.16

%

Memory available percentage

Percentage of available memory

p5

56.84

%

<

mean

79.08

%

SD

16.27

%

<

p95

95.53

%

Max host memory used

Maximum memory usage percentage across all hosts

max

89.52

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

73.08

%

Memory growth rate

Rate of memory growth over time

growth

0.33

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.36

MiB/s

Disk write throughput

Disk write throughput in MiB/s

97.72

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.85

5 min

2.74

15 min

2.69

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

28.57

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

13.2291

%

SD

15.5623

%

<

p95

46.19

%

60 second average

13.2176

%

300 second average

14.3184

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0004

%

SD

0.02

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0004

%

SD

0.02

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.6011

%

SD

7.029

%

<

p95

23.19

%

60 second average

2.5601

%

300 second average

2.3059

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.217

%

SD

3.6272

%

<

p95

10.34

%

60 second average

1.1947

%

300 second average

0.9339

%

Holochain process CPU usage

CPU usage by Holochain process

p5

7.33

%

<

mean

46.29

%

SD

23.6

%

<

p95

89.3

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

368.27

KiB

<

mean

490.36

KiB

SD

137.72

KiB

<

p95

614.97

KiB

Holochain process threads

Number of threads in Holochain process

p5

33

threads

<

mean

44.26

threads

SD

27.23

threads

<

p95

68

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

73

file descriptors

<

mean

86.78

file descriptors

SD

28.21

file descriptors

<

p95

111

file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
full: A full-arc conductor that is just here to serve entries to zero arc nodes.

Started

Thu, 01 Oct 2026 14:02:23 UTC

Duration

900s

Peer count

30

Peer count at end

28

Behaviours

full (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_and_read_36830042121_1

Create rate

The number of timed entries created by the zero-arc write node(s) per second.

Mean

mean

68.76

/s

Max

Highest per-partition mean rate

max

81.49

/s

Min

Lowest per-partition mean rate

min

50.34

/s

Fetch lag timing

For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.

Mean

mean

104.628385

s

SD

11.116206

s

Max

Highest per-partition mean latency

max mean

1028.881044

s

Min

Lowest per-partition mean latency

min mean

9.365646

s

Fetch lag rate

The number of entries per second fetched by zero-arc read nodes.

Mean

mean

0.18

/s

Max

Highest per-partition mean rate

max

2

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc read nodes.

0

Entry created count

Total number of entries created by zero-arc write nodes, by agent.

Total

75765

Agents affected

12 / 12

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

6314

Max

Highest count in any single partition

7646

Min

Lowest count in any single partition

4504

Received count

Total number of entries fetched by zero-arc read nodes, by agent.

Total

38

Agents affected

11 / 11

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3

Max

Highest count in any single partition

4

Min

Lowest count in any single partition

1

Open connections

The number of currently open connections to other conductors.

full-arc

p5

17

<

mean

26.12

SD

4.73

<

p95

28

zero-arc

p5

0

<

mean

8.41

SD

5.33

<

p95

16

Error count

The number of errors accumulated across all nodes.

99

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.011627

s

SD

0.05505

s

p50

0.050977

s

<

p95

0.156907

s

<

p99

0.224785

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 610.012453569s

Not enough time data to show a trend.

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

372079200

over 878.475127857s

mean rate

1.0954303681e+08

/s

std

1.3647631186e+08

/s

p5

1.216099132e+07

/s

p95

4.5001846457e+08

/s

peak

5.1767778838e+08

/s

WASM usage: 'timed::get_timed_entries_network'

Metered usage count of the WASM ribosome for this zome function.

total

986630118

over 878.539231087s

mean rate

5.48312939686e+09

/s

std

9.22094193175e+09

/s

p5

2.76421388e+06

/s

p95

2.45993797638e+10

/s

peak

2.85445550709e+10

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

23124708

over 888.613675538s

mean rate

1.36854488951e+09

/s

std

9.99747619501e+09

/s

p5

1.5912461e+06

/s

p95

1.52126312547e+09

/s

peak

2.0225188079e+11

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004628

s

SD

0.000939

s

p50

0.004583

s

<

p95

0.006032

s

<

p99

0.006177

s

Zome call duration: 'timed::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

0.046492

s

SD

0.163108

s

p50

0.233999

s

<

p95

0.547672

s

<

p99

0.599022

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004102

s

SD

0.000863

s

p50

0.004099

s

<

p95

0.005349

s

<

p99

0.005484

s

WASM call duration: 'timed::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.046005

s

SD

0.162968

s

p50

0.232121

s

<

p95

0.546704

s

<

p99

0.597988

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000836

s

SD

0.000647

s

p50

0.000973

s

<

p95

0.002342

s

<

p99

0.003493

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000262

s

SD

5.9e-05

s

p50

0.000273

s

<

p95

0.000344

s

<

p99

0.000352

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000253

s

SD

6e-05

s

p50

0.000279

s

<

p95

0.000327

s

<

p99

0.00033

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.003796

s

SD

0.032072

s

p50

0.001194

s

<

p95

0.096833

s

<

p99

0.151686

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.042964

s

SD

0.16229

s

p50

0.217476

s

<

p95

0.541169

s

<

p99

0.589293

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001241

s

SD

0.000993

s

p50

0.001441

s

<

p95

0.003616

s

<

p99

0.005007

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.9e-05

s

SD

4.4e-05

s

p50

8.7e-05

s

<

p95

0.000203

s

<

p99

0.000211

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.68

s

<

mean

447.11

s

SD

256.7

s

<

p95

847.69

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 888.613708242s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

21.389572

s

SD

55.006912

s

p50

0

s

<

p95

6.564341

s

<

p99

419.704127

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.186709

s

SD

0.296061

s

p50

0.001175

s

<

p95

0.601588

s

<

p99

1.972846

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.039947

s

SD

0.452052

s

p50

0.001127

s

<

p95

1.255557

s

<

p99

1.423367

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.091573

s

SD

0.043654

s

p50

0.012968

s

<

p95

0.122363

s

<

p99

0.138667

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

10.762506

s

SD

13.175314

s

p50

0.002872

s

<

p95

14.649481

s

<

p99

67.888734

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.010544

s

SD

0.017581

s

p50

0.002205

s

<

p95

0.020669

s

<

p99

0.094058

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001655

s

SD

0.001142

s

p50

0.001178

s

<

p95

0.003082

s

<

p99

0.003797

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002352

s

SD

0.004216

s

p50

0.001097

s

<

p95

0.003646

s

<

p99

0.028118

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000412

s

SD

0.005208

s

p50

0.000254

s

<

p95

0.000723

s

<

p99

0.038158

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.591664

s

SD

26.80209

s

p50

0.2031

s

<

p95

60.01064

s

<

p99

60.022788

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.046452

s

SD

29.702784

s

p50

0.226597

s

<

p95

60.001344

s

<

p99

60.001644

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.145621

s

SD

0.053765

s

p50

0.071278

s

<

p95

0.184555

s

≤

p99

0.184555

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

35.36

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

990

B/s

<

mean

204.95

KiB/s

SD

504.49

KiB/s

<

p95

714.56

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

863

B/s

<

mean

186.77

KiB/s

SD

387.72

KiB/s

<

p95

376.61

KiB/s

Total bytes received

Total bytes received on primary network interface

count

4.52

GiB

mean

5.20

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

4.12

GiB

mean

4.74

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.8%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (71.1% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 22% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.8

%

<

mean

46.46

%

SD

33.54

%

<

p95

94.99

%

System

11.11

%

CPU percentiles

CPU usage percentiles

p50

55.61

%

p95

94.99

%

p99

95.80

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

9

hosts

mean time

543.82

s

Memory used percentage

Percentage of memory used

p5

3.86

%

<

mean

18.65

%

SD

15.55

%

<

p95

38.03

%

Memory available percentage

Percentage of available memory

p5

61.97

%

<

mean

81.35

%

SD

15.55

%

<

p95

96.14

%

Max host memory used

Maximum memory usage percentage across all hosts

max

85.22

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

71.13

%

Memory growth rate

Rate of memory growth over time

growth

0.22

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.09

MiB/s

Disk write throughput

Disk write throughput in MiB/s

68.37

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.28

5 min

1.79

15 min

1.62

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

22.22

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

12.7563

%

SD

16.9848

%

<

p95

43.33

%

60 second average

12.1019

%

300 second average

9.1964

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.41

%

SD

4.9867

%

<

p95

6.04

%

60 second average

1.3074

%

300 second average

0.9796

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.9127

%

SD

4.0726

%

<

p95

2.47

%

60 second average

0.8551

%

300 second average

0.6690

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.17

%

<

mean

39.88

%

SD

33.85

%

<

p95

93.17

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

226.37

KiB

<

mean

314.02

KiB

SD

69.25

KiB

<

p95

453.47

KiB

Holochain process threads

Number of threads in Holochain process

p5

24

threads

<

mean

34.68

threads

SD

13.12

threads

<

p95

41

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

57

file descriptors

<

mean

71.63

file descriptors

SD

11.5

file descriptors

<

p95

85

file descriptors

Zero-Arc Create Data

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 01 Oct 2026 13:15:26 UTC

Duration

900s

Peer count

30

Peer count at end

28

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data--high_zero_36830042121_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

67.28

/s

Max

Highest per-partition mean rate

max

85.51

/s

Min

Lowest per-partition mean rate

min

42.94

/s

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

188.571592

s

SD

102.438897

s

Max

Highest per-partition mean latency

max mean

188.571592

s

Min

Lowest per-partition mean latency

min mean

188.571592

s

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

0

/s

Max

Highest per-partition mean rate

max

0

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0.0072

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

163913

Agents affected

27 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

6071

Max

Highest count in any single partition

7701

Min

Lowest count in any single partition

3855

Received count

Total number of entries received by full-arc nodes, by agent.

Total

2357

Agents affected

1 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

1179

Max

Highest count in any single partition

2357

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

2

<

mean

21.89

SD

9.09

<

p95

28

zero-arc

p5

2

<

mean

11.24

SD

6.26

<

p95

24

Error count

The number of errors accumulated across all nodes.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000311

s

SD

5.1e-05

s

p50

0.000283

s

<

p95

0.000332

s

<

p99

0.00035

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

5

over 790.004752107s

mean rate

0.01

/s

std

0.04

/s

p5

0

/s

p95

0

/s

peak

0.3

/s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

359665728

over 876.68747996s

mean rate

5.991096673e+08

/s

std

8.0703548445e+08

/s

p5

8.65962736e+06

/s

p95

2.68341308953e+09

/s

peak

4.31669179321e+09

/s

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

72892033173

over 884.434727419s

mean rate

7.72058136623e+09

/s

std

5.06300979555e+09

/s

p5

1.534865478e+07

/s

p95

1.56200052193e+10

/s

peak

1.64291255996e+10

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

49291218

over 888.757735332s

mean rate

7.700944828e+07

/s

std

1.0373141035e+08

/s

p5

842593.35

/s

p95

3.4385935029e+08

/s

peak

5.8222735505e+08

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004949

s

SD

0.00159

s

p50

0.004998

s

<

p95

0.008391

s

<

p99

0.011075

s

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.029451

s

SD

0.045733

s

p50

0.003595

s

<

p95

0.129969

s

<

p99

0.139277

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004383

s

SD

0.001445

s

p50

0.004455

s

<

p95

0.00753

s

<

p99

0.009967

s

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.028918

s

SD

0.045688

s

p50

0.003089

s

<

p95

0.129357

s

<

p99

0.138664

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000928

s

SD

0.000336

s

p50

0.000978

s

<

p95

0.001571

s

<

p99

0.002352

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000271

s

SD

7.2e-05

s

p50

0.000262

s

<

p95

0.000448

s

<

p99

0.000488

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000263

s

SD

8e-05

s

p50

0.000273

s

<

p95

0.000453

s

<

p99

0.00053

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000303

s

SD

1.6e-05

s

p50

0.000302

s

<

p95

0.000355

s

<

p99

0.000364

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001868

s

SD

0.00286

s

p50

0.000302

s

<

p95

0.008208

s

<

p99

0.008825

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00139

s

SD

0.000449

s

p50

0.001434

s

<

p95

0.002313

s

<

p99

0.003218

s

Post-commit duration

Time spent executing post-commit workflows.

mean

9.6e-05

s

SD

4.3e-05

s

p50

9.8e-05

s

<

p95

0.00015

s

<

p99

0.000257

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.67

s

<

mean

446.78

s

SD

256.63

s

<

p95

847.68

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 888.757491017s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

14.517547

s

SD

25.470811

s

p50

0

s

<

p95

0.003133

s

<

p99

162.791972

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.066094

s

SD

0.053034

s

p50

0.001179

s

<

p95

0.095732

s

<

p99

0.332882

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.006193

s

SD

0.252016

s

p50

0.000833

s

<

p95

0.959913

s

<

p99

1.099554

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.091308

s

SD

0.036752

s

p50

0.080304

s

<

p95

0.132795

s

<

p99

0.154068

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

12.064946

s

SD

11.124586

s

p50

0.002778

s

<

p95

18.99899

s

<

p99

86.452706

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002206

s

SD

0.001464

s

p50

0.002441

s

<

p95

0.004217

s

<

p99

0.007331

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002706

s

SD

0.001933

s

p50

0.004546

s

<

p95

0.006819

s

<

p99

0.008196

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.003203

s

SD

0.005981

s

p50

0.001242

s

<

p95

0.005473

s

<

p99

0.040367

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000215

s

SD

6.1e-05

s

p50

0.00021

s

<

p95

0.000346

s

<

p99

0.000416

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.02735

s

SD

27.755009

s

p50

0.262696

s

<

p95

60.000821

s

≤

p99

60.000821

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.244842

s

SD

0.114633

s

p50

0.015642

s

<

p95

0.257525

s

<

p99

0.26737

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

44.36

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

9.91

KiB/s

<

mean

124.42

KiB/s

SD

394.02

KiB/s

<

p95

770.35

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

26.16

KiB/s

<

mean

89.87

KiB/s

SD

82.77

KiB/s

<

p95

245.85

KiB/s

Total bytes received

Total bytes received on primary network interface

count

2.85

GiB

mean

3.28

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

2.06

GiB

mean

2.37

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.0%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (73.5% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 21% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

19.74

%

<

mean

56.28

%

SD

23.97

%

<

p95

93.95

%

System

11.93

%

CPU percentiles

CPU usage percentiles

p50

57.29

%

p95

93.95

%

p99

94.99

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

10

hosts

mean time

422.70

s

Memory used percentage

Percentage of memory used

p5

3.84

%

<

mean

18.38

%

SD

15.67

%

<

p95

37.22

%

Memory available percentage

Percentage of available memory

p5

62.78

%

<

mean

81.62

%

SD

15.67

%

<

p95

96.16

%

Max host memory used

Maximum memory usage percentage across all hosts

max

86.83

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

73.54

%

Memory growth rate

Rate of memory growth over time

growth

0.23

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.06

MiB/s

Disk write throughput

Disk write throughput in MiB/s

90.69

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.74

5 min

2.66

15 min

2.66

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

21.43

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

9.6426

%

SD

11.4568

%

<

p95

30.98

%

60 second average

9.3673

%

300 second average

9.2473

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.2636

%

SD

7.5655

%

<

p95

23.27

%

60 second average

2.2079

%

300 second average

2.1488

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.8432

%

SD

6.646

%

<

p95

20.01

%

60 second average

1.8132

%

300 second average

1.7847

%

Holochain process CPU usage

CPU usage by Holochain process

p5

14.48

%

<

mean

50.78

%

SD

23.49

%

<

p95

93.42

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

250.62

KiB

<

mean

340.85

KiB

SD

81.94

KiB

<

p95

500.56

KiB

Holochain process threads

Number of threads in Holochain process

p5

31

threads

<

mean

37.41

threads

SD

13.82

threads

<

p95

67

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

64

file descriptors

<

mean

76.57

file descriptors

SD

10.91

file descriptors

<

p95

83

file descriptors

Zome Call, Single Value

↑ Back to index

Calls a zome function that returns a fixed value. This tests the maximum performance of zome calls when the zome function is not calling into Holochain other otherwise doing any work.

Started

Thu, 01 Oct 2026 11:05:32 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zome_call_single_value_36830042121_1

Call timing

The time taken for the call to complete.

mean

0.003186

s

SD

0.001769

s

p50

0.002587

s

<

p95

0.006143

s

<

p99

0.009943

s

Call rate

The number of zome calls completed per second.

mean

3149.36

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'return_single_value::get_value'

Metered usage count of the WASM ribosome for this zome function.

total

351249858

over 280.003526864s

mean rate

1.25445066e+06

/s

std

432370.59

/s

p5

729754.49

/s

p95

1.81426283e+06

/s

peak

1.81588458e+06

/s

Zome call duration: 'return_single_value::get_value'

Duration of this zome call as measured by Holochain internally.

mean

0.001341

s

SD

0.000145

s

p50

0.00148

s

<

p95

0.001638

s

<

p99

0.001648

s

WASM call duration: 'return_single_value::get_value'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000835

s

SD

9.9e-05

s

p50

0.000926

s

<

p95

0.001037

s

<

p99

0.001074

s

Post-commit duration

Time spent executing post-commit workflows.

mean

7.7e-05

s

SD

0

s

p50

7.7e-05

s

≤

p95

7.7e-05

s

≤

p99

7.7e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.23

s

<

mean

147.23

s

SD

83.67

s

<

p95

277.23

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.003507886s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001662

s

SD

0

s

p50

0.001662

s

≤

p95

0.001662

s

≤

p99

0.001662

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001483

s

SD

0

s

p50

0.001483

s

≤

p95

0.001483

s

≤

p99

0.001483

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004649

s

SD

0.000941

s

p50

0.005011

s

<

p95

0.00642

s

≤

p99

0.00642

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002222

s

SD

0

s

p50

0.002222

s

≤

p95

0.002222

s

≤

p99

0.002222

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.00017

s

SD

0

s

p50

0.00017

s

≤

p95

0.00017

s

≤

p99

0.00017

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

6.9e-05

s

SD

1.5e-05

s

p50

8.2e-05

s

<

p95

0.000102

s

<

p99

0.000103

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001466

s

SD

0.000335

s

p50

0.001927

s

≤

p95

0.001927

s

≤

p99

0.001927

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000339

s

SD

0

s

p50

0.000339

s

≤

p95

0.000339

s

≤

p99

0.000339

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

47.78

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

3.10

KiB/s

<

mean

1.24

MiB/s

SD

1.84

MiB/s

<

p95

5.66

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

681

B/s

<

mean

39.28

KiB/s

SD

39.96

KiB/s

<

p95

110.48

KiB/s

Total bytes received

Total bytes received on primary network interface

count

360.82

MiB

mean

1.24

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

11.12

MiB

mean

39.28

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.5%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

31.08

%

<

mean

65.13

%

SD

24.18

%

<

p95

93.68

%

System

17.36

%

CPU percentiles

CPU usage percentiles

p50

74.79

%

p95

93.68

%

p99

94.52

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

103.45

s

Memory used percentage

Percentage of memory used

p5

5.12

%

<

mean

8.52

%

SD

2.11

%

<

p95

11.51

%

Memory available percentage

Percentage of available memory

p5

88.49

%

<

mean

91.48

%

SD

2.11

%

<

p95

94.88

%

Max host memory used

Maximum memory usage percentage across all hosts

max

8.52

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-6.24

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.06

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.99

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.18

5 min

4.78

15 min

2.74

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

1.18

%

<

mean

25.2683

%

SD

18.3166

%

<

p95

48.78

%

60 second average

30.3313

%

300 second average

29.8193

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.27

%

<

mean

23.64

%

SD

5.84

%

<

p95

27.42

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

24.65

KiB

<

mean

245.87

KiB

SD

214.36

KiB

<

p95

475.39

KiB

Holochain process threads

Number of threads in Holochain process

p5

9

threads

<

mean

31.81

threads

SD

33.54

threads

<

p95

59

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

72.94

file descriptors

SD

72.51

file descriptors

<

p95

135

file descriptors

App Install

App Install

Conductor Startup

DHT Sync Lag

First Call

Full-Arc Create (Validated) / Zero-Arc Read

Local Signals

Mixed Arc get_agent_activity

Mixed Arc must_get_agent_activity

Remote Call Rate

Remote Signals

Single-write, many-read

Validation Receipts

Write/get_agent_activity

Write/get_agent_activity with volatile conductors

Write/Query

Write/Read

Write validated

Write Validated must_get_agent_activity

Zero-Arc Create and Read

Zero-Arc Create Data

Zome Call, Single Value

Mixed Arc `get_agent_activity`

Mixed Arc `must_get_agent_activity`

Write/`get_agent_activity`

Write/`get_agent_activity` with volatile conductors

Write Validated `must_get_agent_activity`