App Install

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 27 Aug 2026 09:24:41 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

minimal (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_minimal_33056872755_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.548831063

s

Install app timing

The time taken to install an app.

mean

0.101346

s

SD

0.378797

s

p50

0.045334

s

<

p95

0.056537

s

<

p99

2.642164

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'callback::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000084067s

Not enough time data to show a trend.

WASM usage: 'callback_integrity::genesis_self_check_2'

Metered usage count of the WASM ribosome for this zome function.

total

132277

over 10.000084067s

mean rate

13227.59

/s

std

0

/s

p5

13227.59

/s

p95

13227.59

/s

peak

13227.59

/s

Not enough time data to show a trend.

WASM call duration: 'callback::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000963

s

SD

0.000227

s

p50

0.000904

s

<

p95

0.001895

s

≤

p99

0.001895

s

WASM call duration: 'callback_integrity::genesis_self_check_2'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000906

s

SD

7.5e-05

s

p50

0.001038

s

≤

p95

0.001038

s

≤

p99

0.001038

s

Post-commit duration

Time spent executing post-commit workflows.

mean

9.3e-05

s

SD

8.2e-05

s

p50

7.1e-05

s

<

p95

0.000276

s

<

p99

0.000518

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

6.94

s

<

mean

11.94

s

SD

5

s

<

p95

16.94

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 10.000114584s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000956

s

SD

0.000158

s

p50

0.001232

s

≤

p95

0.001232

s

≤

p99

0.001232

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001191

s

SD

3.5e-05

s

p50

0.001199

s

≤

p95

0.001199

s

≤

p99

0.001199

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004042

s

SD

4e-05

s

p50

0.004113

s

≤

p95

0.004113

s

≤

p99

0.004113

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002416

s

SD

0.000427

s

p50

0.003163

s

≤

p95

0.003163

s

≤

p99

0.003163

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001622

s

SD

0.000405

s

p50

0.002331

s

≤

p95

0.002331

s

≤

p99

0.002331

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

3.3e-05

s

SD

5e-06

s

p50

4.3e-05

s

≤

p95

4.3e-05

s

≤

p99

4.3e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001005

s

SD

0.000263

s

p50

0.001231

s

<

p95

0.001459

s

≤

p99

0.001459

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000293

s

SD

3e-06

s

p50

0.000294

s

≤

p95

0.000294

s

≤

p99

0.000294

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

40.53

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

354

B/s

<

mean

521.81

KiB/s

SD

1.09

MiB/s

<

p95

825.96

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

16

B/s

<

mean

190.84

KiB/s

SD

167.26

KiB/s

<

p95

453.70

KiB/s

Total bytes received

Total bytes received on primary network interface

count

147.78

MiB

mean

521.81

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

54.05

MiB

mean

190.84

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

52.84

%

<

mean

78.09

%

SD

18.7

%

<

p95

95.02

%

System

37.56

%

CPU percentiles

CPU usage percentiles

p50

92.99

%

p95

95.02

%

p99

95.70

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

175.86

s

Memory used percentage

Percentage of memory used

p5

14.99

%

<

mean

19.73

%

SD

5.25

%

<

p95

31.71

%

Memory available percentage

Percentage of available memory

p5

68.29

%

<

mean

80.27

%

SD

5.25

%

<

p95

85.01

%

Max host memory used

Maximum memory usage percentage across all hosts

max

19.73

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-0.50

MiB/s

Disk read throughput

Disk read throughput in MiB/s

204.91

MiB/s

Disk write throughput

Disk write throughput in MiB/s

6.62

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.71

5 min

3.28

15 min

2.66

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

8.69

%

<

mean

44.0477

%

SD

31.8225

%

<

p95

78.95

%

60 second average

32.8853

%

300 second average

17.5413

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

3.8

%

<

mean

5.268

%

SD

2.0282

%

<

p95

9.46

%

60 second average

5.4440

%

300 second average

5.8080

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0.37

%

<

mean

2.2627

%

SD

2.4575

%

<

p95

7.63

%

60 second average

3.0590

%

300 second average

4.4087

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

45.19

%

<

mean

49.124

%

SD

5.5939

%

<

p95

62.42

%

60 second average

55.5147

%

300 second average

65.8633

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0.02

%

<

mean

16.066

%

SD

16.6367

%

<

p95

40.89

%

60 second average

29.0643

%

300 second average

48.7567

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

29.22

%

SD

23.07

%

<

p95

53.82

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.82

KiB

<

mean

359.43

KiB

SD

420.72

KiB

<

p95

1.18

MiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

36.12

threads

SD

69.65

threads

<

p95

202

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

83.94

file descriptors

SD

152.33

file descriptors

<

p95

450

file descriptors

App Install

↑ Back to index

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 27 Aug 2026 09:22:41 UTC

Duration

300s

Peer count

1

Behaviours

large (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_large_33056872755_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.118150339

s

Install app timing

The time taken to install an app.

mean

0.065955

s

SD

0.00437

s

p50

0.065459

s

<

p95

0.075753

s

<

p99

0.078701

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'large::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.00012803s

Not enough time data to show a trend.

WASM call duration: 'large::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004302

s

SD

0.001147

s

p50

0.003838

s

<

p95

0.006575

s

<

p99

0.00696

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000117

s

SD

0.000134

s

p50

7.8e-05

s

<

p95

0.000456

s

<

p99

0.000814

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

7.3

s

<

mean

17.3

s

SD

8.17

s

<

p95

27.3

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 20.000104395s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001394

s

SD

0.000105

s

p50

0.001298

s

<

p95

0.001449

s

≤

p99

0.001449

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001894

s

SD

0.001386

s

p50

0.001899

s

<

p95

0.004805

s

≤

p99

0.004805

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.005847

s

SD

0.000957

s

p50

0.005953

s

<

p95

0.007776

s

≤

p99

0.007776

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002277

s

SD

0.000375

s

p50

0.002316

s

<

p95

0.00302

s

≤

p99

0.00302

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001269

s

SD

0.000695

s

p50

0.001334

s

<

p95

0.002734

s

≤

p99

0.002734

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

7.6e-05

s

SD

9e-06

s

p50

7.7e-05

s

<

p95

9.5e-05

s

≤

p99

9.5e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00157

s

SD

0.000198

s

p50

0.001726

s

<

p95

0.001961

s

≤

p99

0.001961

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000406

s

SD

5e-05

s

p50

0.000417

s

<

p95

0.0005

s

≤

p99

0.0005

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

67.94

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

39.05

KiB/s

<

mean

55.56

KiB/s

SD

16.51

KiB/s

<

p95

72.08

KiB/s

Not enough time data to show a trend.

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

19.48

KiB/s

<

mean

24.80

KiB/s

SD

5.33

KiB/s

<

p95

30.13

KiB/s

Not enough time data to show a trend.

Total bytes received

Total bytes received on primary network interface

count

1.09

MiB

mean

55.56

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

496.03

KiB

mean

24.80

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 5.61 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

84.63

%

<

mean

87.26

%

SD

2.63

%

<

p95

89.89

%

Not enough time data to show a trend.

System

19.32

%

CPU percentiles

CPU usage percentiles

p50

89.89

%

p95

89.89

%

p99

89.89

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

300.00

s

Memory used percentage

Percentage of memory used

p5

9.75

%

<

mean

11.25

%

SD

1.49

%

<

p95

13.28

%

Memory available percentage

Percentage of available memory

p5

86.72

%

<

mean

88.75

%

SD

1.49

%

<

p95

90.25

%

Max host memory used

Maximum memory usage percentage across all hosts

max

11.25

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

5.61

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.45

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.70

5 min

3.92

15 min

1.88

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

36.51

%

<

mean

40.44

%

SD

2.817

%

<

p95

42.97

%

60 second average

39.1200

%

300 second average

25.7900

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

23.27

%

<

mean

31.02

%

SD

7.75

%

<

p95

38.77

%

Not enough time data to show a trend.

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

139.43

KiB

<

mean

778.64

KiB

SD

571.40

KiB

<

p95

1.49

MiB

Holochain process threads

Number of threads in Holochain process

p5

26

threads

<

mean

136

threads

SD

102.63

threads

<

p95

273

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

63

file descriptors

<

mean

297.67

file descriptors

SD

218.45

file descriptors

<

p95

589

file descriptors

DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.

Started

Thu, 27 Aug 2026 09:26:52 UTC

Duration

1800s

Peer count

30

Peer count at end

27

Behaviours

record_lag (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

dht_sync_lag_33056872755_1

Create rate

The average number of created records per agent.

Mean

mean

18.45

/s

Max

Highest per-partition mean rate

max

31.69

/s

Min

Lowest per-partition mean rate

min

8.9

/s

Sync lag timing

The average time between when a record was created and when it is first seen by an agent.

Mean

mean

628.540066

s

SD

306.498759

s

Max

Highest per-partition mean latency

max mean

1639.518161

s

Min

Lowest per-partition mean latency

min mean

1.427859

s

Sync lag rate

The average number of created records discovered per agent.

Mean

mean

136.72

/s

Max

Highest per-partition mean rate

max

336.6

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully received by all record_lag agents.

0.0503

Sent count

The total number of entries sent (created) by write agents, by agent.

Total

79463

Agents affected

23 / 23

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3455

Max

Highest count in any single partition

5746

Min

Lowest count in any single partition

1611

Received count

The total number of entries received by record_lag agents, by agent.

Total

20002

Agents affected

5 / 5

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

4000

Max

Highest count in any single partition

6255

Min

Lowest count in any single partition

153

Error count

The number of errors encountered during the scenario.

14

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000629

s

SD

0.000241

s

p50

0.000606

s

<

p95

0.00102

s

<

p99

0.001057

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

1

over 1672.319627424s

mean rate

6.4

/s

std

6.44

/s

p5

0

/s

p95

20.7

/s

peak

24.52

/s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

168570938

over 1789.237803371s

mean rate

6.29546566607e+09

/s

std

2.54023649016e+11

/s

p5

2.952768506e+07

/s

p95

2.10935655196e+09

/s

peak

1.1567444186e+13

/s

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

56303483239

over 1785.891312744s

mean rate

2.22994047834e+10

/s

std

1.48458150192e+10

/s

p5

4.4656635858e+08

/s

p95

4.66121435082e+10

/s

peak

4.9328013328e+10

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

21494112

over 1789.237803371s

mean rate

7.2454742019e+08

/s

std

3.02245901333e+10

/s

p5

825190.3

/s

p95

2.6854566152e+08

/s

peak

1.47493953488e+12

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.023304

s

SD

0.02505

s

p50

0.016998

s

<

p95

0.076049

s

<

p99

0.123813

s

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.121725

s

SD

0.110759

s

p50

0.127172

s

<

p95

0.361207

s

<

p99

0.455909

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.022097

s

SD

0.024524

s

p50

0.015741

s

<

p95

0.070912

s

<

p99

0.121674

s

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.121102

s

SD

0.110704

s

p50

0.126489

s

<

p95

0.360498

s

<

p99

0.455181

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002629

s

SD

0.002135

s

p50

0.002329

s

<

p95

0.008128

s

<

p99

0.010453

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.004538

s

SD

0.008052

s

p50

0.002127

s

<

p95

0.020007

s

<

p99

0.037883

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.004279

s

SD

0.007411

s

p50

0.001866

s

<

p95

0.018684

s

<

p99

0.034181

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000628

s

SD

0.000221

s

p50

0.000612

s

<

p95

0.000966

s

<

p99

0.000998

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.00461

s

SD

0.003958

s

p50

0.004757

s

<

p95

0.011251

s

<

p99

0.014277

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.004397

s

SD

0.004019

s

p50

0.003744

s

<

p95

0.013918

s

<

p99

0.019672

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000591

s

SD

0.001046

s

p50

0.000284

s

<

p95

0.003725

s

<

p99

0.005027

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

87.85

s

<

mean

886.88

s

SD

512.92

s

<

p95

1695.61

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

117

over 1789.237839592s

mean rate

77475.49

/s

std

866877.19

/s

p5

0

/s

p95

265953.97

/s

peak

4.031113206e+07

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

117.30083

s

SD

167.876407

s

p50

0.026389

s

<

p95

145.376

s

<

p99

1192.136052

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.119763

s

SD

1.248639

s

p50

0.211866

s

<

p95

2.789309

s

<

p99

6.476939

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.519127

s

SD

0.67424

s

p50

0.864324

s

<

p95

1.877216

s

<

p99

3.336056

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.435067

s

SD

0.31707

s

p50

0.343365

s

<

p95

0.963837

s

<

p99

1.494798

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

11.434053

s

SD

64.954306

s

p50

23.447414

s

<

p95

192.310685

s

<

p99

285.929876

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

40.128366

s

SD

82.695148

s

p50

0.043722

s

<

p95

183.803075

s

<

p99

331.710613

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001824

s

SD

0.00241

s

p50

0.002001

s

<

p95

0.007786

s

<

p99

0.012766

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00907

s

SD

0.014607

s

p50

0.002166

s

<

p95

0.046915

s

<

p99

0.062255

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.003601

s

SD

0.006782

s

p50

0.001136

s

<

p95

0.016579

s

<

p99

0.032796

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.036076

s

SD

29.153995

s

p50

1.588498

s

<

p95

60.0224

s

<

p99

60.06193

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.58591

s

SD

22.909505

s

p50

0.569543

s

<

p95

60.001707

s

<

p99

60.00243

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

72.86

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

27.57

KiB/s

<

mean

563.34

KiB/s

SD

672.31

KiB/s

<

p95

1.24

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

11.41

KiB/s

<

mean

475.18

KiB/s

SD

402.92

KiB/s

<

p95

1.18

MiB/s

Total bytes received

Total bytes received on primary network interface

count

25.67

GiB

mean

14.69

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

21.66

GiB

mean

12.39

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 93% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

52.85

%

<

mean

87.9

%

SD

15.15

%

<

p95

98.5

%

System

15.04

%

CPU percentiles

CPU usage percentiles

p50

95.44

%

p95

98.50

%

p99

98.80

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

27

hosts

mean time

1486.46

s

Memory used percentage

Percentage of memory used

p5

11.78

%

<

mean

24.81

%

SD

10

%

<

p95

44.7

%

Memory available percentage

Percentage of available memory

p5

55.3

%

<

mean

75.19

%

SD

10

%

<

p95

88.22

%

Max host memory used

Maximum memory usage percentage across all hosts

max

48.98

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.53

%

Memory growth rate

Rate of memory growth over time

growth

0.33

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2216.98

MiB/s

Disk write throughput

Disk write throughput in MiB/s

154.73

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/8

hosts

Mount Point /efi-boot

0/7

hosts

Mount Point /etc/hostname

0/11

hosts

Mount Point /etc/hosts

0/11

hosts

Mount Point /etc/nomad.d/client-meta.json

0/9

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

0/11

hosts

Mount Point /nix/store

0/8

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

9.97

5 min

8.99

15 min

6.24

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

92.86

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

7.72

%

<

mean

59.2216

%

SD

28.1134

%

<

p95

96.35

%

60 second average

58.0131

%

300 second average

52.0699

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.2216

%

SD

0.8483

%

<

p95

2.15

%

60 second average

0.2279

%

300 second average

0.2564

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.1088

%

SD

0.4861

%

<

p95

0.74

%

60 second average

0.1144

%

300 second average

0.1497

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

14.5818

%

SD

19.2237

%

<

p95

52.77

%

60 second average

14.3011

%

300 second average

12.8185

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.1781

%

SD

4.872

%

<

p95

12.51

%

60 second average

2.1918

%

300 second average

2.2060

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

31.58

%

SD

27.46

%

<

p95

88.69

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

23.42

KiB

<

mean

315.09

KiB

SD

146.40

KiB

<

p95

559.81

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

27.45

threads

SD

15.35

threads

<

p95

48

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

63.14

file descriptors

SD

34.55

file descriptors

<

p95

89

file descriptors

First Call

↑ Back to index

This scenario installs a simple app which implements the init callback, and measures the time elapsed between start of install and calling init. It will uninstall the app it installed so that it can re-install it on the next iteration. This is required to re-run the initialisation callback.

Started

Thu, 27 Aug 2026 09:05:47 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

first_call_33056872755_1

Zome call timing

The time elapsed between start of install and calling init.

mean

0.011787

s

SD

0.003303

s

p50

0.01073

s

<

p95

0.018618

s

<

p99

0.02036

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000151784s

Not enough time data to show a trend.

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000151784s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

250914

over 10.000151784s

mean rate

25091.02

/s

std

0

/s

p5

25091.02

/s

p95

25091.02

/s

peak

25091.02

/s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004703

s

SD

0.000325

s

p50

0.005298

s

≤

p95

0.005298

s

≤

p99

0.005298

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003989

s

SD

0.000921

s

p50

0.003867

s

<

p95

0.005714

s

<

p99

0.006401

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001331

s

SD

0.000231

s

p50

0.001284

s

<

p95

0.00215

s

<

p99

0.002347

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001204

s

SD

2e-05

s

p50

0.001208

s

≤

p95

0.001208

s

≤

p99

0.001208

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000304

s

SD

4.6e-05

s

p50

0.000331

s

<

p95

0.000368

s

≤

p99

0.000368

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00215

s

SD

0.000152

s

p50

0.002428

s

≤

p95

0.002428

s

≤

p99

0.002428

s

Post-commit duration

Time spent executing post-commit workflows.

mean

9.9e-05

s

SD

5.9e-05

s

p50

7.7e-05

s

<

p95

0.000208

s

<

p99

0.000341

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

6.88

s

<

mean

11.88

s

SD

5

s

<

p95

16.88

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 10.000195339s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000967

s

SD

0.000114

s

p50

0.001169

s

≤

p95

0.001169

s

≤

p99

0.001169

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001445

s

SD

5.6e-05

s

p50

0.001457

s

≤

p95

0.001457

s

≤

p99

0.001457

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004162

s

SD

2e-06

s

p50

0.004163

s

≤

p95

0.004163

s

≤

p99

0.004163

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002616

s

SD

0.000262

s

p50

0.002675

s

≤

p95

0.002675

s

≤

p99

0.002675

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001717

s

SD

0.000424

s

p50

0.002471

s

≤

p95

0.002471

s

≤

p99

0.002471

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

3.2e-05

s

SD

3e-06

s

p50

3.8e-05

s

≤

p95

3.8e-05

s

≤

p99

3.8e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001071

s

SD

0.000293

s

p50

0.001402

s

<

p95

0.001582

s

≤

p99

0.001582

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000261

s

SD

2e-06

s

p50

0.000261

s

≤

p95

0.000261

s

≤

p99

0.000261

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

48.76

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.81

KiB/s

<

mean

40.36

KiB/s

SD

19.56

KiB/s

<

p95

59.92

KiB/s

Not enough time data to show a trend.

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

10.98

KiB/s

<

mean

21.24

KiB/s

SD

10.26

KiB/s

<

p95

31.50

KiB/s

Not enough time data to show a trend.

Total bytes received

Total bytes received on primary network interface

count

807.27

KiB

mean

40.36

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

424.85

KiB

mean

21.24

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 6.49 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

53.78

%

<

mean

62.21

%

SD

8.43

%

<

p95

70.63

%

Not enough time data to show a trend.

System

13.45

%

CPU percentiles

CPU usage percentiles

p50

70.63

%

p95

70.63

%

p99

70.63

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

18.1

%

<

mean

24.29

%

SD

7.38

%

<

p95

34.65

%

Memory available percentage

Percentage of available memory

p5

65.35

%

<

mean

75.71

%

SD

7.38

%

<

p95

81.9

%

Max host memory used

Maximum memory usage percentage across all hosts

max

24.29

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

6.49

MiB/s

Disk read throughput

Disk read throughput in MiB/s

254.55

MiB/s

Disk write throughput

Disk write throughput in MiB/s

4.77

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.62

5 min

1.22

15 min

1.13

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

4.76

%

<

mean

7.9033

%

SD

3.3839

%

<

p95

12.6

%

60 second average

4.4200

%

300 second average

3.3733

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

1.96

%

<

mean

3.2233

%

SD

1.403

%

<

p95

5.18

%

60 second average

4.5067

%

300 second average

4.7267

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

1.94

%

<

mean

3.1367

%

SD

1.3352

%

<

p95

5

%

60 second average

4.4733

%

300 second average

4.7067

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

51.5

%

<

mean

62.5

%

SD

10.5393

%

<

p95

76.71

%

60 second average

75.0767

%

300 second average

77.6567

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

33.28

%

<

mean

49.08

%

SD

16.9736

%

<

p95

72.63

%

60 second average

70.8133

%

300 second average

76.3500

%

Holochain process CPU usage

CPU usage by Holochain process

p5

47.1

%

<

mean

56.54

%

SD

9.44

%

<

p95

65.98

%

Not enough time data to show a trend.

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

272.31

KiB

<

mean

759.82

KiB

SD

580.68

KiB

<

p95

1.54

MiB

Holochain process threads

Number of threads in Holochain process

p5

5

threads

<

mean

126.33

threads

SD

132.71

threads

<

p95

311

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

11

file descriptors

<

mean

279

file descriptors

SD

285.18

file descriptors

<

p95

674

file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 27 Aug 2026 10:25:41 UTC

Duration

900s

Peer count

30

Peer count at end

27

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

full_arc_create_validated_zero_arc_read--high_zero_33056872755_1

Create rate

The number of timed entries created by the full-arc node(s) per second.

Mean

mean

38.26

/s

Max

Highest per-partition mean rate

max

59.79

/s

Min

Lowest per-partition mean rate

min

16.73

/s

Fetch lag timing

For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.

Mean

mean

185.644393

s

SD

16.74699

s

Max

Highest per-partition mean latency

max mean

1241.588797

s

Min

Lowest per-partition mean latency

min mean

12.113399

s

Fetch rate

The number of entries per second fetched by zero-arc nodes.

Mean

mean

30.88

/s

Max

Highest per-partition mean rate

max

226

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc nodes.

0.0287

Entry created count

Total number of entries created by full-arc nodes, by agent.

Total

6775

Agents affected

2 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3388

Max

Highest count in any single partition

5366

Min

Lowest count in any single partition

1409

Received count

Total number of entries fetched by zero-arc nodes, by agent.

Total

5057

Agents affected

8 / 26

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

195

Max

Highest count in any single partition

1515

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

25

<

mean

26.59

SD

2.28

<

p95

27

zero-arc

p5

0

<

mean

5.35

SD

5.35

<

p95

16

Retrieval errors

Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.

Total

353

Agents affected

26 / 26

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

14

Max

Highest count in any single partition

15

Min

Lowest count in any single partition

4

Error count

The number of errors accumulated across all nodes.

355

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.097148

s

SD

0.335328

s

p50

0.121688

s

<

p95

1.008207

s

<

p99

1.78914

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

12

over 813.728074235s

mean rate

364.04

/s

std

1396.08

/s

p5

0

/s

p95

2260.32

/s

peak

14709.13

/s

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.008881

s

SD

0.009644

s

p50

0.005665

s

<

p95

0.025153

s

<

p99

0.025748

s

Zome call duration: 'timed_and_validated::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

0.12367

s

SD

23.412224

s

p50

0.096238

s

<

p95

66.201333

s

<

p99

96.818104

s

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008233

s

SD

0.009332

s

p50

0.005113

s

<

p95

0.023882

s

<

p99

0.024539

s

WASM call duration: 'timed_and_validated::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

1.017362

s

SD

60.414564

s

p50

1.097622

s

<

p95

168.379624

s

<

p99

207.861864

s

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001784

s

SD

0.001941

s

p50

0.001508

s

<

p95

0.004272

s

<

p99

0.010006

s

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.009034

s

SD

0.004108

s

p50

0.001318

s

<

p95

0.015474

s

<

p99

0.019771

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000724

s

SD

0.001175

s

p50

0.000335

s

<

p95

0.003006

s

<

p99

0.003215

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000657

s

SD

0.001007

s

p50

0.000319

s

<

p95

0.002526

s

<

p99

0.002672

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.10417

s

SD

0.731589

s

p50

0.198291

s

<

p95

1.106582

s

<

p99

3.516531

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.047116

s

SD

2.245271

s

p50

0.032405

s

<

p95

6.60205

s

<

p99

10.224188

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.014286

s

SD

0.007248

s

p50

0.012481

s

<

p95

0.024542

s

<

p99

0.025355

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002962

s

SD

0.001653

s

p50

0.002275

s

<

p95

0.007044

s

<

p99

0.007665

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000157

s

SD

0.000234

s

p50

0.000122

s

<

p95

0.000525

s

<

p99

0.001311

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.93

s

<

mean

443.87

s

SD

255.3

s

<

p95

846.01

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 888.499600421s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

130.356023

s

SD

46.916602

s

p50

0

s

<

p95

0.013653

s

<

p99

45.703246

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

4.01466

s

SD

2.538887

s

p50

0.001277

s

<

p95

2.949789

s

<

p99

14.868262

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.110075

s

SD

0.221158

s

p50

0.001411

s

<

p95

0.007066

s

<

p99

1.307064

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.129447

s

SD

0.054254

s

p50

0.014036

s

<

p95

0.087218

s

<

p99

0.353202

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

2.865233

s

SD

7.518716

s

p50

0.002908

s

<

p95

2.13076

s

<

p99

49.120503

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

4.895893

s

SD

4.535092

s

p50

0.00206

s

<

p95

0.004672

s

<

p99

27.975827

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001996

s

SD

0.001814

s

p50

0.000143

s

<

p95

0.004543

s

<

p99

0.010613

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.004821

s

SD

0.002306

s

p50

0.002022

s

<

p95

0.006735

s

<

p99

0.013758

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000443

s

SD

0.000983

s

p50

0.00032

s

<

p95

0.001735

s

<

p99

0.00546

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

2.179804

s

SD

29.836832

s

p50

0.594613

s

<

p95

60.001895

s

<

p99

60.003814

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

3.094619

s

SD

20.998873

s

p50

1.218329

s

<

p95

60.001216

s

<

p99

60.001689

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.978951

s

SD

5.183992

s

p50

0.956115

s

<

p95

15.266263

s

≤

p99

15.266263

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

18.15

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

617

B/s

<

mean

164.80

KiB/s

SD

565.66

KiB/s

<

p95

598.79

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

469

B/s

<

mean

130.65

KiB/s

SD

315.09

KiB/s

<

p95

761.80

KiB/s

Total bytes received

Total bytes received on primary network interface

count

3.77

GiB

mean

4.33

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

2.99

GiB

mean

3.44

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/7 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 19% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.08

%

<

mean

24.61

%

SD

35.28

%

<

p95

94.28

%

System

6.47

%

CPU percentiles

CPU usage percentiles

p50

3.78

%

p95

94.28

%

p99

98.80

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

15

hosts

mean time

278.96

s

Memory used percentage

Percentage of memory used

p5

6.67

%

<

mean

15.48

%

SD

8.03

%

<

p95

29.05

%

Memory available percentage

Percentage of available memory

p5

70.95

%

<

mean

84.52

%

SD

8.03

%

<

p95

93.33

%

Max host memory used

Maximum memory usage percentage across all hosts

max

40.96

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.01

%

Memory growth rate

Rate of memory growth over time

growth

0.26

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.05

MiB/s

Disk write throughput

Disk write throughput in MiB/s

58.45

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/6

hosts

Mount Point /boot

0/1

hosts

Mount Point /efi-boot

0/5

hosts

Mount Point /etc/hostname

1/7

hosts

Mount Point /etc/hosts

1/7

hosts

Mount Point /etc/nomad.d/client-meta.json

1/15

hosts

Mount Point /etc/resolv.conf

1/7

hosts

Mount Point /nix/store

0/6

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.03

5 min

2.19

15 min

3.14

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

18.52

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

12.0238

%

SD

24.5048

%

<

p95

70.27

%

60 second average

11.5562

%

300 second average

12.1877

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.8581

%

SD

3.3157

%

<

p95

2.99

%

60 second average

0.7501

%

300 second average

0.9083

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.4882

%

SD

1.4186

%

<

p95

2.46

%

60 second average

0.4649

%

300 second average

0.5208

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.18

%

<

mean

9.02

%

SD

21.11

%

<

p95

69.48

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

216.43

KiB

<

mean

383.16

KiB

SD

116.58

KiB

<

p95

490.51

KiB

Holochain process threads

Number of threads in Holochain process

p5

14

threads

<

mean

28.29

threads

SD

11.91

threads

<

p95

40

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

41

file descriptors

<

mean

60.61

file descriptors

SD

14.72

file descriptors

<

p95

82

file descriptors

Local Signals

↑ Back to index

This scenario is for testing the speed and reliability of local signals. It uses a zome with a function that will emit 10,000 signals. The scenario measures how long it takes to send the signals and how many have been received by the time the zome call ends.

Started

Thu, 27 Aug 2026 09:07:56 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

local_signals_33056872755_1

Send timing

The time taken to emit a batch of 10,000 signals.

mean

1.250435

s

SD

0.934546

s

p50

1.217448

s

<

p95

3.355866

s

≤

p99

3.355866

s

Receive timing

The time taken to receive the completed batch, to the nearest 250ms.

mean

27.818327

s

SD

6.548548

s

p50

30.001039

s

<

p95

30.002413

s

≤

p99

30.002413

s

Success ratio

The proportion of signals successfully received in a batch.

min

0.05869999900460243

<

mean

0.4669

SD

0.2095

<

max

0.6804999709129333

Holochain MetricsClick to toggle visibility.

WASM usage: 'signal::emit_10k_signals'

Metered usage count of the WASM ribosome for this zome function.

total

277609437

over 280.003163732s

mean rate

991451.16

/s

std

1.44054515e+06

/s

p5

0

/s

p95

3.08452178e+06

/s

peak

3.08453541e+06

/s

Zome call duration: 'signal::emit_10k_signals'

Duration of this zome call as measured by Holochain internally.

mean

0.195107

s

SD

0.012288

s

p50

0.195925

s

<

p95

0.219564

s

≤

p99

0.219564

s

WASM call duration: 'signal::emit_10k_signals'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.194138

s

SD

0.012407

s

p50

0.194937

s

<

p95

0.219039

s

≤

p99

0.219039

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

1.4e-05

s

SD

1e-06

s

p50

1.4e-05

s

<

p95

1.7e-05

s

≤

p99

1.7e-05

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

90000

over 280.00316041s

mean rate

321.42

/s

std

454.79

/s

p5

0

/s

p95

999.99

/s

peak

1000

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000197

s

SD

0

s

p50

0.000197

s

≤

p95

0.000197

s

≤

p99

0.000197

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.84

s

<

mean

146.84

s

SD

83.67

s

<

p95

276.84

s

Dropped WebSocket signals

Count of signals dropped from the app WebSocket due to channel overload.

total

5542

over 280.003164752s

mean rate

19.79

/s

std

33.74

/s

p5

0

/s

p95

102.5

/s

peak

109.9

/s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.003168847s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.00139

s

SD

0

s

p50

0.00139

s

≤

p95

0.00139

s

≤

p99

0.00139

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001768

s

SD

0

s

p50

0.001768

s

≤

p95

0.001768

s

≤

p99

0.001768

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.01288

s

SD

0.003243

s

p50

0.013417

s

<

p95

0.019489

s

≤

p99

0.019489

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002183

s

SD

0

s

p50

0.002183

s

≤

p95

0.002183

s

≤

p99

0.002183

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002526

s

SD

0

s

p50

0.002526

s

≤

p95

0.002526

s

≤

p99

0.002526

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000172

s

SD

3e-05

s

p50

0.000181

s

<

p95

0.000188

s

≤

p99

0.000188

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002007

s

SD

0.00047

s

p50

0.002249

s

<

p95

0.002499

s

≤

p99

0.002499

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000278

s

SD

0

s

p50

0.000278

s

≤

p95

0.000278

s

≤

p99

0.000278

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

12.69

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

344

B/s

<

mean

299.70

KiB/s

SD

1.16

MiB/s

<

p95

2.12

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

35

B/s

<

mean

14.05

KiB/s

SD

17.39

KiB/s

<

p95

39.68

KiB/s

Total bytes received

Total bytes received on primary network interface

count

84.87

MiB

mean

299.70

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

3.98

MiB

mean

14.05

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.84 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

20.18

%

<

mean

36.64

%

SD

13.85

%

<

p95

75.38

%

System

23.95

%

CPU percentiles

CPU usage percentiles

p50

34.31

%

p95

75.38

%

p99

77.41

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

17.29

%

<

mean

21.12

%

SD

3.02

%

<

p95

27.54

%

Memory available percentage

Percentage of available memory

p5

72.46

%

<

mean

78.88

%

SD

3.02

%

<

p95

82.71

%

Max host memory used

Maximum memory usage percentage across all hosts

max

21.12

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

1.84

MiB/s

Disk read throughput

Disk read throughput in MiB/s

149.12

MiB/s

Disk write throughput

Disk write throughput in MiB/s

4.56

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.47

5 min

2.98

15 min

1.88

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

8.77

%

<

mean

19.5317

%

SD

6.9038

%

<

p95

30.16

%

60 second average

18.2483

%

300 second average

11.2717

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

17.8543

%

SD

13.6298

%

<

p95

33.26

%

60 second average

18.8257

%

300 second average

12.9763

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

16.3107

%

SD

12.637

%

<

p95

31.23

%

60 second average

17.4010

%

300 second average

12.2233

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

12.25

%

<

mean

60.9143

%

SD

32.628

%

<

p95

92.1

%

60 second average

68.3070

%

300 second average

75.6097

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

9.67

%

<

mean

38.5177

%

SD

20.3886

%

<

p95

64

%

60 second average

46.5583

%

300 second average

62.9233

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.35

%

<

mean

8.39

%

SD

12.38

%

<

p95

50.46

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

192.05

KiB

<

mean

218.78

KiB

SD

48.06

KiB

<

p95

261.04

KiB

Holochain process threads

Number of threads in Holochain process

p5

5

threads

<

mean

25.61

threads

SD

9.35

threads

<

p95

36

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

23

file descriptors

<

mean

68.32

file descriptors

SD

19.37

file descriptors

<

p95

88

file descriptors

Mixed Arc `get_agent_activity`

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

zero_write: A zero-arc conductor that creates entries on its source chain.
full_write: A full-arc conductor that creates entries on its source chain.
zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.

Started

Thu, 27 Aug 2026 10:39:14 UTC

Duration

900s

Peer count

30

Peer count at end

28

Behaviours

full_write (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_get_agent_activity_33056872755_1

Entry created count

The total number of entries created by write agents over the run, partitioned by agent and behaviour.

Total

105006

Agents affected

20 / 20

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

5250

Max

Highest count in any single partition

12684

Min

Lowest count in any single partition

2081

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

2958.25

Max

Highest chain head value observed for any single write agent.

3422

Write agents observed

4

Chain head delay timing

The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.

Mean

mean

0.00244

s

SD

0.049343

s

Max

Highest per-partition mean latency

max mean

0.018067

s

Min

Lowest per-partition mean latency

min mean

-0.002443

s

Chain head delay rate

The rate at which readers detect new chain heads appearing on writers' chains.

Mean

mean

31.51

/s

Max

Highest per-partition mean rate

max

48.9

/s

Min

Lowest per-partition mean rate

min

18.87

/s

get_agent_activity_full zome call timing

The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.

Mean

mean

0.152235

s

SD

0.125044

s

Max

Highest per-partition mean latency

max mean

0.2158

s

Min

Lowest per-partition mean latency

min mean

0.029207

s

Retrieval errors

A running accumulation of errors encountered by read agents when calling get_agent_activity.

Total

23

Agents affected

4 / 9

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3

Max

Highest count in any single partition

11

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors, partitioned by agent behaviour.

behaviour: full_write

p5

26

<

mean

27.75

SD

1.39

<

p95

28

behaviour: zero_read

p5

0

<

mean

15.41

SD

5.43

<

p95

22

behaviour: zero_write

p5

11

<

mean

14.92

SD

2.87

<

p95

20

Final error count

The total number of errors accumulated across all agents over the run.

24

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.119345

s

SD

0.182556

s

p50

0.111204

s

<

p95

0.523628

s

<

p99

0.7016

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

5

over 703.42766941s

mean rate

6.21

/s

std

7.15

/s

p5

0

/s

p95

15.72

/s

peak

15.9

/s

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007452

s

SD

0.004832

s

p50

0.006059

s

<

p95

0.013852

s

<

p99

0.024094

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.005283

s

SD

0.004089

s

p50

0.004938

s

<

p95

0.017379

s

<

p99

0.019833

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.128932

s

SD

0.187965

s

p50

0.142781

s

<

p95

0.657429

s

<

p99

0.70562

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.012826

s

SD

0.170872

s

p50

0.021176

s

<

p95

0.421798

s

≤

p99

0.421798

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006351

s

SD

0.00441

s

p50

0.004973

s

<

p95

0.011674

s

<

p99

0.022183

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004547

s

SD

0.003792

s

p50

0.004201

s

<

p95

0.016194

s

<

p99

0.017674

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.129098

s

SD

0.189221

s

p50

0.141429

s

<

p95

0.65644

s

<

p99

0.754093

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.011962

s

SD

0.170757

s

p50

0.020115

s

<

p95

0.420627

s

≤

p99

0.420627

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001493

s

SD

0.001023

s

p50

0.001518

s

<

p95

0.004099

s

<

p99

0.005571

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

5.2e-05

s

SD

5.2e-05

s

p50

3.1e-05

s

<

p95

0.000131

s

<

p99

0.000252

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000664

s

SD

0.001127

s

p50

0.000489

s

<

p95

0.003149

s

<

p99

0.005954

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000535

s

SD

0.000276

s

p50

0.000453

s

<

p95

0.001066

s

≤

p99

0.001066

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.126898

s

SD

0.188813

s

p50

0.139261

s

<

p95

0.653433

s

<

p99

0.751122

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.008052

s

SD

0.170449

s

p50

0.015635

s

<

p95

0.415514

s

≤

p99

0.415514

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

9e-06

s

SD

5e-06

s

p50

1.2e-05

s

<

p95

1.8e-05

s

≤

p99

1.8e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002368

s

SD

0.001984

s

p50

0.00245

s

<

p95

0.006508

s

<

p99

0.011535

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000194

s

SD

0.000243

s

p50

0.000186

s

<

p95

0.000842

s

<

p99

0.001136

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.47

s

<

mean

443.01

s

SD

255

s

<

p95

837.95

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 888.829164698s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

43.154405

s

SD

34.786255

s

p50

0

s

<

p95

85.567501

s

<

p99

174.866881

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.064827

s

SD

0.117783

s

p50

0.002005

s

<

p95

0.341734

s

<

p99

0.507435

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.047874

s

SD

0.317247

s

p50

0.001772

s

<

p95

1.079791

s

<

p99

1.188606

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.105058

s

SD

0.075472

s

p50

0.065863

s

<

p95

0.231447

s

<

p99

0.278033

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

7.599859

s

SD

20.028706

s

p50

0.004696

s

<

p95

48.31051

s

<

p99

118.355035

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

13.630693

s

SD

27.583701

s

p50

0.002759

s

<

p95

58.121122

s

<

p99

165.741366

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002655

s

SD

0.00168

s

p50

0.001706

s

<

p95

0.00496

s

<

p99

0.007671

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.008201

s

SD

0.007008

s

p50

0.001487

s

<

p95

0.022709

s

<

p99

0.035218

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000717

s

SD

0.00093

s

p50

0.000396

s

<

p95

0.00263

s

<

p99

0.005456

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

10.510827

s

SD

29.737236

s

p50

0.907629

s

<

p95

60.002836

s

<

p99

60.019241

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.021139

s

SD

0.153344

s

p50

0.140281

s

<

p95

0.41206

s

≤

p99

0.41206

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.11668

s

SD

27.10013

s

p50

0.136112

s

<

p95

60.001322

s

<

p99

60.00156

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.228267

s

SD

0.138292

s

p50

0.240457

s

<

p95

0.396592

s

<

p99

0.660096

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

52.73

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

9.56

KiB/s

<

mean

487.91

KiB/s

SD

693.41

KiB/s

<

p95

1.08

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

9.16

KiB/s

<

mean

378.41

KiB/s

SD

298.95

KiB/s

<

p95

914.67

KiB/s

Total bytes received

Total bytes received on primary network interface

count

11.15

GiB

mean

12.83

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

8.65

GiB

mean

9.95

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.5%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.10 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/7 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 68% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

6.29

%

<

mean

64.97

%

SD

34.84

%

<

p95

98

%

System

12.23

%

CPU percentiles

CPU usage percentiles

p50

83.05

%

p95

98.00

%

p99

98.55

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

20

hosts

mean time

677.22

s

Memory used percentage

Percentage of memory used

p5

10.7

%

<

mean

20.92

%

SD

8.08

%

<

p95

41.07

%

Memory available percentage

Percentage of available memory

p5

58.93

%

<

mean

79.08

%

SD

8.08

%

<

p95

89.3

%

Max host memory used

Maximum memory usage percentage across all hosts

max

43.41

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

1.03

%

Memory growth rate

Rate of memory growth over time

growth

1.10

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2792.25

MiB/s

Disk write throughput

Disk write throughput in MiB/s

110.06

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/12

hosts

Mount Point /boot

0/1

hosts

Mount Point /efi-boot

0/10

hosts

Mount Point /etc/hostname

1/7

hosts

Mount Point /etc/hosts

1/7

hosts

Mount Point /etc/nomad.d/client-meta.json

1/10

hosts

Mount Point /etc/resolv.conf

1/7

hosts

Mount Point /nix/store

0/12

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.80

5 min

4.88

15 min

4.65

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

67.86

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.03

%

<

mean

37.0561

%

SD

30.6198

%

<

p95

88.61

%

60 second average

35.1120

%

300 second average

28.8807

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.3565

%

SD

1.4077

%

<

p95

4.34

%

60 second average

0.3586

%

300 second average

0.3286

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.2786

%

SD

1.093

%

<

p95

3.37

%

60 second average

0.2852

%

300 second average

0.2716

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

9.5468

%

SD

14.889

%

<

p95

39.09

%

60 second average

9.3100

%

300 second average

9.1542

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.6744

%

SD

3.3806

%

<

p95

8.15

%

60 second average

1.8817

%

300 second average

2.8814

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.4

%

<

mean

19.54

%

SD

23.07

%

<

p95

68.41

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

136.19

KiB

<

mean

391.43

KiB

SD

162.85

KiB

<

p95

524.61

KiB

Holochain process threads

Number of threads in Holochain process

p5

12

threads

<

mean

35.62

threads

SD

21.28

threads

<

p95

41

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

38

file descriptors

<

mean

75.31

file descriptors

SD

25.58

file descriptors

<

p95

85

file descriptors

Mixed Arc `must_get_agent_activity`

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started

Thu, 27 Aug 2026 10:41:12 UTC

Duration

900s

Peer count

30

Peer count at end

28

Behaviours

full_write (1 agent)
zero_must_get_agent_activity (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_must_get_agent_activity_33056872755_1

Highest observed action_seq

The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

1265.6

Max

Highest chain head value observed for any single write agent.

1710

Write agents observed

5

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

138.916478

s

SD

103.34094

s

Max

Highest per-partition mean latency

max mean

274.465892

s

Min

Lowest per-partition mean latency

min mean

8.273232

s

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

2.09

/s

Max

Highest per-partition mean rate

max

2.96

/s

Min

Lowest per-partition mean rate

min

1.6

/s

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

0.629619

s

SD

0.861457

s

Max

Highest per-partition mean latency

max mean

0.950449

s

Min

Lowest per-partition mean latency

min mean

0.292835

s

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

23629

Agents affected

9 / 9

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

2625

Max

Highest count in any single partition

5557

Min

Lowest count in any single partition

332

Open connections

The average number of open connections maintained by an agent over the run, broken down by behaviour.

behaviour: full_write

p5

0

<

mean

25.29

SD

7.76

<

p95

28

behaviour: zero_must_get_agent_activity

p5

12

<

mean

19.09

SD

4.05

<

p95

26

behaviour: zero_write

p5

12

<

mean

18.47

SD

4.08

<

p95

26

Final error count

The total number of all types of error accumulated over the run by all agents.

30619

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.073355

s

SD

0.182168

s

p50

0.139812

s

<

p95

0.479733

s

<

p99

0.787655

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

3

over 716.641053028s

mean rate

16.57

/s

std

54.81

/s

p5

0

/s

p95

63.05

/s

peak

390.86

/s

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.006816

s

SD

0.003806

s

p50

0.005235

s

<

p95

0.010414

s

<

p99

0.019531

s

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.085596

s

SD

0.054524

s

p50

0.055513

s

<

p95

0.200316

s

<

p99

0.262701

s

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004892

s

SD

0.000649

s

p50

0.004908

s

<

p95

0.006492

s

<

p99

0.006516

s

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.240287

s

SD

0.228233

s

p50

0.213594

s

<

p95

0.584772

s

<

p99

1.199086

s

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.207437

s

SD

0.126504

s

p50

0.278124

s

<

p95

0.321155

s

≤

p99

0.321155

s

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005749

s

SD

0.003488

s

p50

0.004561

s

<

p95

0.009106

s

<

p99

0.018102

s

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.084581

s

SD

0.054298

s

p50

0.054489

s

<

p95

0.198775

s

<

p99

0.261581

s

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004082

s

SD

0.000564

s

p50

0.004119

s

<

p95

0.005501

s

<

p99

0.005525

s

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.324187

s

SD

0.224095

s

p50

0.303479

s

<

p95

0.722691

s

<

p99

1.282688

s

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.206571

s

SD

0.126481

s

p50

0.277609

s

<

p95

0.319887

s

≤

p99

0.319887

s

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001605

s

SD

0.000733

s

p50

0.001422

s

<

p95

0.003533

s

<

p99

0.004187

s

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.016896

s

SD

0.033099

s

p50

0.007055

s

<

p95

0.102769

s

<

p99

0.162735

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

3.4e-05

s

SD

2.3e-05

s

p50

2.6e-05

s

<

p95

8e-05

s

<

p99

0.000121

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000527

s

SD

0.000348

s

p50

0.000445

s

<

p95

0.001372

s

<

p99

0.001624

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000685

s

SD

0.00035

s

p50

0.000405

s

<

p95

0.00132

s

<

p99

0.001835

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.31929

s

SD

0.193985

s

p50

0.279294

s

<

p95

0.583547

s

<

p99

1.255934

s

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.054254

s

SD

0.12467

s

p50

0.055335

s

<

p95

0.41519

s

<

p99

0.500131

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.03049

s

SD

0.029936

s

p50

0.012696

s

<

p95

0.098777

s

<

p99

0.148361

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1.3e-05

s

SD

8e-06

s

p50

1.1e-05

s

<

p95

3.5e-05

s

≤

p99

3.5e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002582

s

SD

0.001209

s

p50

0.002374

s

<

p95

0.00507

s

<

p99

0.007021

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000196

s

SD

8.2e-05

s

p50

0.000186

s

<

p95

0.000328

s

<

p99

0.000367

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

45.12

s

<

mean

441.36

s

SD

254.61

s

<

p95

836.95

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

13515

over 888.702916297s

mean rate

27492.33

/s

std

51267.28

/s

p5

0

/s

p95

129179.12

/s

peak

338942.62

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

117.390972

s

SD

62.58636

s

p50

0

s

<

p95

187.518465

s

<

p99

292.1564

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

2.697958

s

SD

2.147399

s

p50

0.002051

s

<

p95

5.876326

s

<

p99

10.564031

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.208002

s

SD

0.205103

s

p50

0.002385

s

<

p95

0.622255

s

<

p99

0.868532

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.145106

s

SD

0.199932

s

p50

0.025973

s

<

p95

0.436489

s

<

p99

1.172323

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

5.215487

s

SD

9.141225

s

p50

0.004211

s

<

p95

24.752873

s

<

p99

43.014675

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

16.09948

s

SD

19.11846

s

p50

0.00179

s

<

p95

47.711478

s

<

p99

109.523858

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002719

s

SD

0.002019

s

p50

0.000259

s

<

p95

0.006003

s

<

p99

0.007639

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.007228

s

SD

0.012672

s

p50

0.002602

s

<

p95

0.030402

s

<

p99

0.071999

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000517

s

SD

0.000265

s

p50

0.000403

s

<

p95

0.001069

s

<

p99

0.001386

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.031072

s

SD

29.758125

s

p50

0.353437

s

<

p95

60.001774

s

<

p99

60.001905

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.155987

s

SD

24.716733

s

p50

0.189709

s

<

p95

60.001785

s

<

p99

60.001884

s

P2P request roundtrip: 'must_get_agent_activity'

Time spent sending a must_get_agent_activity request and awaiting its response.

mean

0.111811

s

SD

28.477722

s

p50

0.16842

s

<

p95

60.001964

s

<

p99

60.002311

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.191563

s

SD

6.600868

s

p50

0.194205

s

<

p95

15.266335

s

<

p99

16.474348

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

50.49

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

5.83

KiB/s

<

mean

378.77

KiB/s

SD

356.75

KiB/s

<

p95

952.73

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

5.79

KiB/s

<

mean

338.85

KiB/s

SD

279.61

KiB/s

<

p95

874.06

KiB/s

Total bytes received

Total bytes received on primary network interface

count

8.65

GiB

mean

9.95

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.74

GiB

mean

8.90

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.5%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/7 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 68% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

5.53

%

<

mean

62

%

SD

36.64

%

<

p95

97.95

%

System

11.51

%

CPU percentiles

CPU usage percentiles

p50

82.07

%

p95

97.95

%

p99

98.55

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

21

hosts

mean time

622.06

s

Memory used percentage

Percentage of memory used

p5

10.67

%

<

mean

20.82

%

SD

8.14

%

<

p95

41.07

%

Memory available percentage

Percentage of available memory

p5

58.93

%

<

mean

79.18

%

SD

8.14

%

<

p95

89.33

%

Max host memory used

Maximum memory usage percentage across all hosts

max

43.76

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

1.12

%

Memory growth rate

Rate of memory growth over time

growth

0.13

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2645.38

MiB/s

Disk write throughput

Disk write throughput in MiB/s

96.42

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/12

hosts

Mount Point /boot

0/1

hosts

Mount Point /efi-boot

0/10

hosts

Mount Point /etc/hostname

1/7

hosts

Mount Point /etc/hosts

1/7

hosts

Mount Point /etc/nomad.d/client-meta.json

1/10

hosts

Mount Point /etc/resolv.conf

1/7

hosts

Mount Point /nix/store

0/12

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.03

5 min

5.29

15 min

4.83

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

67.86

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.01

%

<

mean

35.4895

%

SD

30.986

%

<

p95

88.28

%

60 second average

36.1272

%

300 second average

31.5075

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.334

%

SD

1.3637

%

<

p95

4.24

%

60 second average

0.3431

%

300 second average

0.3338

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.2581

%

SD

1.0397

%

<

p95

3.3

%

60 second average

0.2665

%

300 second average

0.2697

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

9.5469

%

SD

14.9779

%

<

p95

38.95

%

60 second average

9.5462

%

300 second average

9.3114

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.7054

%

SD

3.752

%

<

p95

8.25

%

60 second average

1.6944

%

300 second average

2.4632

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.43

%

<

mean

16.84

%

SD

20.51

%

<

p95

57.5

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

206.52

KiB

<

mean

414.76

KiB

SD

167.06

KiB

<

p95

539.25

KiB

Holochain process threads

Number of threads in Holochain process

p5

12

threads

<

mean

36.28

threads

SD

22.27

threads

<

p95

43

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

28

file descriptors

<

mean

76.16

file descriptors

SD

26.68

file descriptors

<

p95

85

file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started

Thu, 27 Aug 2026 09:09:56 UTC

Duration

900s

Peer count

10

Peer count at end

9

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_call_rate_33056872755_1

Dispatch timing

The time between sending a remote call and the remote handler being invoked.

Mean

mean

0.078224

s

SD

0.172049

s

Max

Highest per-partition mean latency

max mean

0.125368

s

Min

Lowest per-partition mean latency

min mean

-0.221219

s

Round-trip timing

The total elapsed time to get a response to the client.

Mean

mean

0.155225

s

SD

0.224725

s

Max

Highest per-partition mean latency

max mean

0.198076

s

Min

Lowest per-partition mean latency

min mean

0.079724

s

Error count

The total number of errors accumulated during the run.

254

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_call::call_echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

8774901

over 888.373502713s

mean rate

2.138808725e+07

/s

std

2.139798052e+07

/s

p5

57503.25

/s

p95

5.431720272e+07

/s

peak

5.694986973e+07

/s

WASM usage: 'remote_call::echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

1602357

over 888.373502713s

mean rate

3.19474755e+06

/s

std

2.95870861e+06

/s

p5

38191.84

/s

p95

9.36711726e+06

/s

peak

9.40252589e+06

/s

WASM usage: 'remote_call::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 888.373502713s

Not enough time data to show a trend.

Zome call duration: 'remote_call::call_echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.147688

s

SD

0.051578

s

p50

0.166151

s

<

p95

0.190607

s

<

p99

0.256457

s

Zome call duration: 'remote_call::echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.003279

s

SD

0.001042

s

p50

0.003453

s

<

p95

0.004416

s

<

p99

0.004441

s

WASM call duration: 'remote_call::call_echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.260705

s

SD

0.415373

s

p50

0.238196

s

<

p95

1.559982

s

<

p99

2.127476

s

WASM call duration: 'remote_call::echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002139

s

SD

0.000705

s

p50

0.002165

s

<

p95

0.003041

s

<

p99

0.003073

s

WASM call duration: 'remote_call::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002711

s

SD

0.001168

s

p50

0.002602

s

<

p95

0.005133

s

≤

p99

0.005133

s

Host function call duration: '__hc__call_1'

Duration of this host function call invoked from within WASM.

mean

0.258196

s

SD

0.41545

s

p50

0.234684

s

<

p95

1.557809

s

<

p99

2.125292

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000442

s

SD

0.000215

s

p50

0.000392

s

<

p95

0.000863

s

≤

p99

0.000863

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

1.1e-05

s

SD

5e-06

s

p50

1e-05

s

<

p95

2.6e-05

s

<

p99

2.7e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000308

s

SD

0.00024

s

p50

0.000295

s

<

p95

0.001194

s

≤

p99

0.001194

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000206

s

SD

0.00031

s

p50

0.000103

s

<

p95

0.001126

s

≤

p99

0.001126

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.76

s

<

mean

435.8

s

SD

254.06

s

<

p95

845.22

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

127

over 888.373591417s

mean rate

89.85

/s

std

382.14

/s

p5

0

/s

p95

166.57

/s

peak

2183.99

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.640377

s

SD

1.081832

s

p50

1.205245

s

<

p95

3.757855

s

<

p99

5.354194

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002583

s

SD

0.001719

s

p50

0.002963

s

<

p95

0.005771

s

<

p99

0.00653

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.088105

s

SD

0.035254

s

p50

0.097583

s

<

p95

0.150287

s

<

p99

0.159644

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.006458

s

SD

0.002745

s

p50

0.005866

s

<

p95

0.012243

s

<

p99

0.015593

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.402144

s

SD

0.495037

s

p50

0.232366

s

<

p95

1.542098

s

<

p99

1.773017

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.186354

s

SD

0.10445

s

p50

0.177793

s

<

p95

0.36099

s

<

p99

0.45966

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000156

s

SD

4.6e-05

s

p50

0.000171

s

<

p95

0.000234

s

<

p99

0.000241

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001136

s

SD

0.000724

s

p50

0.001255

s

<

p95

0.003426

s

≤

p99

0.003426

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000469

s

SD

0.000147

s

p50

0.000559

s

<

p95

0.000698

s

<

p99

0.000703

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.098106

s

SD

0.059916

s

p50

0.083407

s

<

p95

0.287033

s

≤

p99

0.287033

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.151615

s

SD

0.040232

s

p50

0.163992

s

<

p95

0.184585

s

<

p99

0.190587

s

P2P request roundtrip: 'call_remote'

Time spent sending a call_remote request and awaiting its response.

mean

0.148115

s

SD

18.738026

s

p50

0.165591

s

<

p95

60.001768

s

≤

p99

60.001768

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

10.09

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

1.50

KiB/s

<

mean

111.25

KiB/s

SD

693.93

KiB/s

≤

p95

32.65

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

1.55

KiB/s

<

mean

28.09

KiB/s

SD

280.91

KiB/s

≤

p95

28.08

KiB/s

Total bytes received

Total bytes received on primary network interface

count

837.66

MiB

mean

963.78

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

211.47

MiB

mean

243.31

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 2.14 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 10% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

2.01

%

<

mean

17.4

%

SD

20.81

%

<

p95

74.31

%

System

7.30

%

CPU percentiles

CPU usage percentiles

p50

11.28

%

p95

74.31

%

p99

89.91

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

4

hosts

mean time

73.31

s

Memory used percentage

Percentage of memory used

p5

6.52

%

<

mean

21.19

%

SD

13.44

%

<

p95

48.43

%

Memory available percentage

Percentage of available memory

p5

51.57

%

<

mean

78.81

%

SD

13.44

%

<

p95

93.48

%

Max host memory used

Maximum memory usage percentage across all hosts

max

46.49

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.47

%

Memory growth rate

Rate of memory growth over time

growth

2.14

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2767.11

MiB/s

Disk write throughput

Disk write throughput in MiB/s

16.43

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

0/3

hosts

Mount Point /etc/hosts

0/3

hosts

Mount Point /etc/nomad.d/client-meta.json

0/7

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

0/3

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.11

5 min

0.95

15 min

0.69

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

10.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

5.8023

%

SD

9.6929

%

<

p95

26.71

%

60 second average

5.8076

%

300 second average

4.8192

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

1.2999

%

SD

3.2293

%

<

p95

6.1

%

60 second average

1.4320

%

300 second average

1.5914

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

1.1713

%

SD

2.8849

%

<

p95

6.02

%

60 second average

1.3017

%

300 second average

1.4702

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

7.9222

%

SD

21.353

%

<

p95

76.94

%

60 second average

8.0941

%

300 second average

8.7011

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

6.004

%

SD

17.6945

%

<

p95

48.07

%

60 second average

5.9454

%

300 second average

6.2056

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

5.92

%

SD

11.36

%

<

p95

38.38

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

193.50

KiB

<

mean

219.65

KiB

SD

60.89

KiB

<

p95

250.20

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

27.5

threads

SD

14.1

threads

<

p95

46

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

66.48

file descriptors

SD

31.09

file descriptors

<

p95

87

file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started

Thu, 27 Aug 2026 09:11:47 UTC

Duration

900s

Peer count

10

Peer count at end

8

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_signals_33056872755_1

Round trip time

The time from origin signal dispatch to origin receive of the remote side's response signal.

mean

0.253123

s

SD

1.037461

s

p50

0.165944

s

<

p95

0.231901

s

<

p99

2.459382

s

Timeouts

The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).

total

265

over 870.202829962s

mean rate

21.38

/s

std

185.82

/s

p5

0

/s

p95

57.41

/s

peak

4713.71

/s

Timeout rate

The proportion of remote signal round-trips that timed out.

0.0357

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_signal::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 885.831788774s

Not enough time data to show a trend.

WASM usage: 'remote_signal::recv_remote_signal'

Metered usage count of the WASM ribosome for this zome function.

total

27014316

over 885.831788774s

mean rate

1.06275286419e+09

/s

std

1.30079659367e+10

/s

p5

0

/s

p95

5.9660256083e+08

/s

peak

2.57136783734e+11

/s

WASM usage: 'remote_signal::signal_request'

Metered usage count of the WASM ribosome for this zome function.

total

13591440

over 885.831788774s

mean rate

5.84115638e+06

/s

std

2.427740813e+07

/s

p5

0

/s

p95

3.952389765e+07

/s

peak

5.0149021641e+08

/s

Zome call duration: 'remote_signal::recv_remote_signal'

Duration of this zome call as measured by Holochain internally.

mean

0.002628

s

SD

0.001056

s

p50

0.002511

s

<

p95

0.004895

s

<

p99

0.005478

s

Zome call duration: 'remote_signal::signal_request'

Duration of this zome call as measured by Holochain internally.

mean

0.002559

s

SD

0.000966

s

p50

0.00248

s

<

p95

0.004621

s

<

p99

0.004799

s

WASM call duration: 'remote_signal::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001973

s

SD

0.001094

s

p50

0.001301

s

<

p95

0.003907

s

≤

p99

0.003907

s

WASM call duration: 'remote_signal::recv_remote_signal'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001625

s

SD

0.000652

s

p50

0.001618

s

<

p95

0.002852

s

<

p99

0.003287

s

WASM call duration: 'remote_signal::signal_request'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001574

s

SD

0.000531

s

p50

0.00156

s

<

p95

0.002586

s

<

p99

0.002647

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000355

s

SD

0.000198

s

p50

0.000303

s

<

p95

0.000769

s

≤

p99

0.000769

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

5.7e-05

s

SD

1.9e-05

s

p50

5.8e-05

s

<

p95

8.8e-05

s

<

p99

9.7e-05

s

Host function call duration: '__hc__send_remote_signal_1'

Duration of this host function call invoked from within WASM.

mean

5.9e-05

s

SD

2.6e-05

s

p50

5.3e-05

s

<

p95

0.000102

s

<

p99

0.000116

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

5e-06

s

SD

4e-06

s

p50

5e-06

s

<

p95

1.3e-05

s

<

p99

2.1e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00032

s

SD

0.000313

s

p50

0.000188

s

<

p95

0.001192

s

≤

p99

0.001192

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

1491

over 885.83170679s

mean rate

61130.83

/s

std

990870.93

/s

p5

0

/s

p95

38470.42

/s

peak

2.209944751e+07

/s

Sent remote signals

Count of remote signals sent via the send_remote_signal host function.

total

1552

over 885.831702259s

mean rate

18099.22

/s

std

108210.84

/s

p5

0

/s

p95

47454.09

/s

peak

1.70164492e+06

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000201

s

SD

0.000211

s

p50

8.6e-05

s

<

p95

0.00075

s

≤

p99

0.00075

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

44.7

s

<

mean

430.14

s

SD

251.29

s

<

p95

837.58

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

109

over 887.387024408s

mean rate

4566.42

/s

std

53719.52

/s

p5

0

/s

p95

32.47

/s

peak

960711.94

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.586075

s

SD

1.99045

s

p50

0.642526

s

<

p95

7.301022

s

≤

p99

7.301022

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.003465

s

SD

0.007722

s

p50

0.002771

s

<

p95

0.022572

s

<

p99

0.030217

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.0811

s

SD

0.068455

s

p50

0.083914

s

<

p95

0.275504

s

<

p99

0.285725

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.007388

s

SD

0.004258

s

p50

0.006637

s

<

p95

0.016597

s

<

p99

0.022559

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.121295

s

SD

0.31267

s

p50

0.019521

s

<

p95

1.184209

s

≤

p99

1.184209

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.263738

s

SD

0.163237

s

p50

0.292839

s

<

p95

0.463083

s

<

p99

0.811735

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000158

s

SD

7.9e-05

s

p50

0.000162

s

<

p95

0.000323

s

<

p99

0.000459

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001484

s

SD

0.001375

s

p50

0.001237

s

<

p95

0.00481

s

<

p99

0.004942

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000518

s

SD

0.000226

s

p50

0.000474

s

<

p95

0.001001

s

<

p99

0.001218

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.070481

s

SD

0.051811

s

p50

0.073145

s

<

p95

0.223537

s

≤

p99

0.223537

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.158334

s

SD

0.031215

s

p50

0.172892

s

<

p95

0.189141

s

<

p99

0.206272

s

P2P received remote signals

Count of remote signals received.

total

1491

over 885.831657598s

mean rate

9614.16

/s

std

39297.99

/s

p5

0

/s

p95

33664.93

/s

peak

478697.94

/s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

13.38

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

5.89

KiB/s

<

mean

119.60

KiB/s

SD

701.73

KiB/s

≤

p95

39.05

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

7.78

KiB/s

<

mean

15.78

KiB/s

SD

15.16

KiB/s

<

p95

38.37

KiB/s

Total bytes received

Total bytes received on primary network interface

count

393.61

MiB

mean

452.88

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

51.94

MiB

mean

59.76

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 93.2%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

4.07

%

<

mean

23.06

%

SD

23.25

%

<

p95

84.63

%

System

9.68

%

CPU percentiles

CPU usage percentiles

p50

15.80

%

p95

84.63

%

p99

93.23

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

4

hosts

mean time

72.85

s

Memory used percentage

Percentage of memory used

p5

5.23

%

<

mean

28.37

%

SD

14.88

%

<

p95

48.5

%

Memory available percentage

Percentage of available memory

p5

51.5

%

<

mean

71.63

%

SD

14.88

%

<

p95

94.77

%

Max host memory used

Maximum memory usage percentage across all hosts

max

48.04

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.47

%

Memory growth rate

Rate of memory growth over time

growth

0.86

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2635.93

MiB/s

Disk write throughput

Disk write throughput in MiB/s

10.16

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

0/3

hosts

Mount Point /etc/hosts

0/3

hosts

Mount Point /etc/nomad.d/client-meta.json

0/2

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

0/3

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.43

5 min

1.21

15 min

0.88

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

8.2719

%

SD

11.4625

%

<

p95

42.97

%

60 second average

8.3371

%

300 second average

6.7374

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

1.378

%

SD

2.4326

%

<

p95

6.18

%

60 second average

1.3769

%

300 second average

1.3662

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

1.3663

%

SD

2.4149

%

<

p95

6.12

%

60 second average

1.3666

%

300 second average

1.3574

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.4218

%

SD

4.4548

%

<

p95

9.61

%

60 second average

1.3286

%

300 second average

2.1627

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.2026

%

SD

3.4617

%

<

p95

8.56

%

60 second average

1.1179

%

300 second average

1.1969

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

5.17

%

SD

10.75

%

<

p95

36.27

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

193.22

KiB

<

mean

215.30

KiB

SD

82.93

KiB

<

p95

246.70

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

28.49

threads

SD

19.02

threads

<

p95

46

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

62.52

file descriptors

SD

40.94

file descriptors

<

p95

87

file descriptors

Single-write, many-read

↑ Back to index

Creates an entry in the agent setup, then read one's own record back. This tests the maximum read performance for reading back local data when the chain is short.

Started

Thu, 27 Aug 2026 09:13:39 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

single_write_many_read_33056872755_1

read call timing

The time it takes to complete the zome call that retrieves the record.

mean

0.003922

s

SD

0.006881

s

p50

0.00326

s

<

p95

0.005935

s

<

p99

0.00853

s

read call rate

The number of read zome calls completed per second.

mean

2525.36

/s

Error count

The number of errors collected.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000305

s

SD

9e-06

s

p50

0.000307

s

<

p95

0.000323

s

<

p99

0.000325

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.001964187s

Not enough time data to show a trend.

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

2351931120

over 280.001964187s

mean rate

8.39969524e+06

/s

std

285034.02

/s

p5

7.79038284e+06

/s

p95

8.78161151e+06

/s

peak

8.79831874e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.001964187s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.001964187s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004379

s

SD

0

s

p50

0.004379

s

≤

p95

0.004379

s

≤

p99

0.004379

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.001881

s

SD

2e-05

s

p50

0.001881

s

<

p95

0.001922

s

<

p99

0.001926

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00373

s

SD

0

s

p50

0.00373

s

≤

p95

0.00373

s

≤

p99

0.00373

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001366

s

SD

1.8e-05

s

p50

0.001368

s

<

p95

0.001405

s

<

p99

0.001406

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002661

s

SD

0

s

p50

0.002661

s

≤

p95

0.002661

s

≤

p99

0.002661

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001592

s

SD

0

s

p50

0.001592

s

≤

p95

0.001592

s

≤

p99

0.001592

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000402

s

SD

4.6e-05

s

p50

0.000448

s

≤

p95

0.000448

s

≤

p99

0.000448

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000337

s

SD

9e-06

s

p50

0.000339

s

<

p95

0.000355

s

<

p99

0.000358

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001938

s

SD

0

s

p50

0.001938

s

≤

p95

0.001938

s

≤

p99

0.001938

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000108

s

SD

0

s

p50

0.000108

s

≤

p95

0.000108

s

≤

p99

0.000108

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.88

s

<

mean

146.88

s

SD

83.67

s

<

p95

276.88

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.001970518s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001413

s

SD

0

s

p50

0.001413

s

≤

p95

0.001413

s

≤

p99

0.001413

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001816

s

SD

0

s

p50

0.001816

s

≤

p95

0.001816

s

≤

p99

0.001816

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004895

s

SD

0.000877

s

p50

0.00509

s

<

p95

0.00657

s

≤

p99

0.00657

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.003106

s

SD

0

s

p50

0.003106

s

≤

p95

0.003106

s

≤

p99

0.003106

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000598

s

SD

0

s

p50

0.000598

s

≤

p95

0.000598

s

≤

p99

0.000598

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

4.8e-05

s

SD

2e-06

s

p50

4.8e-05

s

<

p95

5.1e-05

s

<

p99

5.7e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001504

s

SD

0.000127

s

p50

0.001572

s

≤

p95

0.001572

s

≤

p99

0.001572

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000359

s

SD

0

s

p50

0.000359

s

≤

p95

0.000359

s

≤

p99

0.000359

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

33.79

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

1.71

KiB/s

<

mean

194.19

KiB/s

SD

997.92

KiB/s

≤

p95

14.29

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

1.13

KiB/s

<

mean

8.85

KiB/s

SD

17.06

KiB/s

<

p95

18.05

KiB/s

Total bytes received

Total bytes received on primary network interface

count

55.00

MiB

mean

194.19

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

2.51

MiB

mean

8.85

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

54.47

%

<

mean

56.92

%

SD

2.92

%

<

p95

58.88

%

System

23.13

%

CPU percentiles

CPU usage percentiles

p50

56.46

%

p95

58.88

%

p99

71.23

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

21.28

%

<

mean

21.64

%

SD

0.7

%

<

p95

23.06

%

Memory available percentage

Percentage of available memory

p5

76.94

%

<

mean

78.36

%

SD

0.7

%

<

p95

78.72

%

Max host memory used

Maximum memory usage percentage across all hosts

max

21.64

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-0.12

MiB/s

Disk read throughput

Disk read throughput in MiB/s

219.84

MiB/s

Disk write throughput

Disk write throughput in MiB/s

1.75

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.06

5 min

3.14

15 min

2.32

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

13.48

%

<

mean

24.3387

%

SD

4.0818

%

<

p95

29.56

%

60 second average

22.8590

%

300 second average

17.9270

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

1.49

%

<

mean

3.7897

%

SD

1.2548

%

<

p95

5.82

%

60 second average

3.5467

%

300 second average

7.4923

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

1.05

%

<

mean

2.0557

%

SD

0.5549

%

<

p95

3.1

%

60 second average

2.1570

%

300 second average

6.4397

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

49.3

%

<

mean

51.6247

%

SD

2.8941

%

<

p95

56.43

%

60 second average

50.3140

%

300 second average

56.9950

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

22

%

<

mean

24.176

%

SD

3.5886

%

<

p95

27.19

%

60 second average

25.6923

%

300 second average

38.3170

%

Holochain process CPU usage

CPU usage by Holochain process

p5

40.76

%

<

mean

43.18

%

SD

7.32

%

<

p95

46.11

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.74

KiB

<

mean

186.79

KiB

SD

60.97

KiB

<

p95

209.23

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

21.5

threads

SD

7.65

threads

<

p95

25

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

58.56

file descriptors

SD

19.41

file descriptors

<

p95

68

file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started

Thu, 27 Aug 2026 09:28:41 UTC

Duration

900s

Peer count

20

Peer count at end

19

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

validation_receipts_33056872755_1

Receipts complete timing

The amount of time between publishing a record and receiving the required number of validation receipts.

Mean

mean

7.488372

s

SD

3.078432

s

Max

Highest per-partition mean latency

max mean

9.1662

s

Min

Lowest per-partition mean latency

min mean

6.628176

s

Receipts complete rate

The number of complete validation receipt sets collected per second.

Mean

mean

1.27

/s

Max

Highest per-partition mean rate

max

1.4

/s

Min

Lowest per-partition mean rate

min

1.14

/s

Error count

The total number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

3217050

over 887.163826928s

mean rate

1.28503666e+06

/s

std

2.32263304e+06

/s

p5

0

/s

p95

8.00446464e+06

/s

peak

1.164701571e+07

/s

WASM usage: 'crud::get_sample_entry_validation_receipts'

Metered usage count of the WASM ribosome for this zome function.

total

11987135

over 887.163826928s

mean rate

4.01692482e+06

/s

std

7.78015988e+06

/s

p5

126521.2

/s

p95

2.616905842e+07

/s

peak

4.086207095e+07

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 887.163826928s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

400582

over 888.289085336s

mean rate

158151.05

/s

std

285244.19

/s

p5

0

/s

p95

973271.54

/s

peak

1.43433104e+06

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.0098

s

SD

0.008129

s

p50

0.006804

s

<

p95

0.029984

s

<

p99

0.040822

s

Zome call duration: 'crud::get_sample_entry_validation_receipts'

Duration of this zome call as measured by Holochain internally.

mean

0.026634

s

SD

0.023888

s

p50

0.015507

s

<

p95

0.061023

s

<

p99

0.145022

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008723

s

SD

0.00751

s

p50

0.005956

s

<

p95

0.025549

s

<

p99

0.038735

s

WASM call duration: 'crud::get_sample_entry_validation_receipts'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.025289

s

SD

0.023091

s

p50

0.014431

s

<

p95

0.058124

s

<

p99

0.139934

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003185

s

SD

0.002279

s

p50

0.00246

s

<

p95

0.008644

s

<

p99

0.009078

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002779

s

SD

0.002274

s

p50

0.002077

s

<

p95

0.006915

s

<

p99

0.012452

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.001032

s

SD

0.001021

s

p50

0.000672

s

<

p95

0.003702

s

<

p99

0.004995

s

Host function call duration: '__hc__get_validation_receipts_1'

Duration of this host function call invoked from within WASM.

mean

0.021781

s

SD

0.021439

s

p50

0.011526

s

<

p95

0.050842

s

<

p99

0.131333

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.004682

s

SD

0.004037

s

p50

0.00343

s

<

p95

0.013064

s

<

p99

0.021717

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000281

s

SD

0.000292

s

p50

0.000165

s

<

p95

0.000703

s

<

p99

0.001557

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.09

s

<

mean

438.15

s

SD

254.83

s

<

p95

841.77

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

5202

over 888.289075479s

mean rate

381.9

/s

std

850.9

/s

p5

1.05

/s

p95

2197.41

/s

peak

4791.16

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.192526

s

SD

1.231416

s

p50

1.032109

s

<

p95

4.069691

s

<

p99

6.916586

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.004052

s

SD

0.008442

s

p50

0.002557

s

<

p95

0.026261

s

<

p99

0.037428

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.048334

s

SD

0.063411

s

p50

0.051009

s

<

p95

0.194565

s

<

p99

0.326635

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.05551

s

SD

0.049331

s

p50

0.032608

s

<

p95

0.124609

s

<

p99

0.295897

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.032794

s

SD

0.314258

s

p50

0.029297

s

<

p95

0.365697

s

<

p99

1.823779

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

3.777966

s

SD

14.890478

s

p50

6.798537

s

<

p95

41.441025

s

<

p99

52.979197

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00109

s

SD

0.001012

s

p50

0.000674

s

<

p95

0.002862

s

<

p99

0.006002

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002682

s

SD

0.002936

s

p50

0.001825

s

<

p95

0.010726

s

<

p99

0.014902

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000472

s

SD

0.000631

s

p50

0.000355

s

<

p95

0.002182

s

<

p99

0.002952

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.140405

s

SD

0.082452

s

p50

0.097165

s

<

p95

0.218568

s

<

p99

0.329441

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.185434

s

SD

0.045412

s

p50

0.168833

s

<

p95

0.251871

s

<

p99

0.337109

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

74.55

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

0

B/s

<

mean

600.41

KiB/s

SD

631.45

KiB/s

<

p95

872.30

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

0

B/s

<

mean

497.04

KiB/s

SD

326.85

KiB/s

<

p95

1.08

MiB/s

Total bytes received

Total bytes received on primary network interface

count

9.46

GiB

mean

10.88

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.83

GiB

mean

9.01

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 85% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

62.05

%

<

mean

91.97

%

SD

12.48

%

<

p95

98.53

%

System

17.42

%

CPU percentiles

CPU usage percentiles

p50

97.25

%

p95

98.53

%

p99

98.80

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

19

hosts

mean time

825.64

s

Memory used percentage

Percentage of memory used

p5

8.66

%

<

mean

29.63

%

SD

11.19

%

<

p95

50.36

%

Memory available percentage

Percentage of available memory

p5

49.64

%

<

mean

70.37

%

SD

11.19

%

<

p95

91.34

%

Max host memory used

Maximum memory usage percentage across all hosts

max

52.41

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.47

%

Memory growth rate

Rate of memory growth over time

growth

0.98

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1896.04

MiB/s

Disk write throughput

Disk write throughput in MiB/s

148.77

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/1

hosts

Mount Point /efi-boot

0/1

hosts

Mount Point /etc/hostname

0/11

hosts

Mount Point /etc/hosts

0/11

hosts

Mount Point /etc/nomad.d/client-meta.json

0/8

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

0/11

hosts

Mount Point /nix/store

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

10.90

5 min

8.30

15 min

4.63

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

85.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

17.66

%

<

mean

71.1871

%

SD

25.5262

%

<

p95

97.42

%

60 second average

69.2458

%

300 second average

53.4979

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.1158

%

SD

0.52

%

<

p95

1.3

%

60 second average

0.1233

%

300 second average

0.1665

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.1133

%

SD

0.5064

%

<

p95

1.3

%

60 second average

0.1205

%

300 second average

0.1631

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.8565

%

SD

7.2287

%

<

p95

24.78

%

60 second average

3.7323

%

300 second average

2.9347

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.656

%

SD

2.6311

%

<

p95

3.94

%

60 second average

0.6664

%

300 second average

0.6069

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

26.63

%

SD

17.16

%

<

p95

45.68

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

93.08

KiB

<

mean

268.20

KiB

SD

84.94

KiB

<

p95

372.97

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

31.55

threads

SD

12.06

threads

<

p95

48

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

72.72

file descriptors

SD

25.21

file descriptors

<

p95

90

file descriptors

Write/`get_agent_activity`

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 27 Aug 2026 09:30:29 UTC

Duration

900s

Peer count

30

Peer count at end

27

Behaviours

get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_33056872755_1

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

2012.5

Max

Highest chain head value observed for any single write agent.

3529

Write agents observed

6

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.343996

s

SD

0.350855

s

Max

Highest per-partition mean latency

max mean

0.895586

s

Min

Lowest per-partition mean latency

min mean

0.11335

s

Error count

The number of errors accumulated.

47

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.066779

s

SD

0.210296

s

p50

0.001402

s

<

p95

0.693337

s

<

p99

0.903877

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

5

over 700.941180437s

mean rate

2.42

/s

std

3.51

/s

p5

0

/s

p95

6.29

/s

peak

17.52

/s

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.021115

s

SD

0.029258

s

p50

0.007229

s

<

p95

0.078176

s

≤

p99

0.078176

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.012409

s

SD

0.018836

s

p50

0.011762

s

<

p95

0.058501

s

<

p99

0.060036

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.117079

s

SD

0.238823

s

p50

0.119612

s

<

p95

0.774476

s

<

p99

0.970567

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.00808

s

SD

0.14516

s

p50

0.006199

s

<

p95

0.014131

s

<

p99

0.699016

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.02001

s

SD

0.028675

s

p50

0.005616

s

<

p95

0.075958

s

≤

p99

0.075958

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.011059

s

SD

0.017097

s

p50

0.010189

s

<

p95

0.051633

s

<

p99

0.053793

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.115902

s

SD

0.23838

s

p50

0.118243

s

<

p95

0.77171

s

<

p99

0.966944

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00712

s

SD

0.145144

s

p50

0.005346

s

<

p95

0.012899

s

<

p99

0.698022

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002655

s

SD

0.002184

s

p50

0.002038

s

<

p95

0.009586

s

<

p99

0.011006

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2.7e-05

s

SD

1.1e-05

s

p50

2.4e-05

s

<

p95

5.6e-05

s

≤

p99

5.6e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.002912

s

SD

0.008048

s

p50

0.002014

s

<

p95

0.022034

s

<

p99

0.023325

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.010091

s

SD

0.019777

s

p50

0.00064

s

<

p95

0.061601

s

≤

p99

0.061601

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.112181

s

SD

0.236726

s

p50

0.113309

s

<

p95

0.763075

s

<

p99

0.953038

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001064

s

SD

0.145255

s

p50

0.00053

s

<

p95

0.004391

s

<

p99

0.693372

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

3e-05

s

SD

1.7e-05

s

p50

9e-06

s

<

p95

6.3e-05

s

≤

p99

6.3e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.004352

s

SD

0.006351

s

p50

0.003287

s

<

p95

0.020116

s

<

p99

0.035082

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000671

s

SD

0.001349

s

p50

0.000161

s

<

p95

0.003572

s

<

p99

0.006965

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

45.34

s

<

mean

435.78

s

SD

252.51

s

<

p95

835.2

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

878

over 887.070467186s

mean rate

102412.6

/s

std

613423.18

/s

p5

62.66

/s

p95

388502.4

/s

peak

1.51440007e+07

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

113.432854

s

SD

104.632829

s

p50

33.289751

s

<

p95

342.680487

s

<

p99

398.899961

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.08057

s

SD

0.267052

s

p50

0.162609

s

<

p95

0.861083

s

<

p99

1.21775

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.589927

s

SD

0.556181

s

p50

1.03494

s

<

p95

1.96257

s

<

p99

2.495684

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.179051

s

SD

0.301885

s

p50

0.081016

s

<

p95

0.808792

s

<

p99

2.039945

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

6.582577

s

SD

21.956423

s

p50

15.363959

s

<

p95

67.069982

s

<

p99

101.368887

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

23.996439

s

SD

41.737707

s

p50

0.19526

s

<

p95

119.792626

s

<

p99

171.662363

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003996

s

SD

0.002223

s

p50

0.003836

s

<

p95

0.00708

s

<

p99

0.012337

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.007924

s

SD

0.00978

s

p50

0.002681

s

<

p95

0.030125

s

<

p99

0.045192

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.00161

s

SD

0.005055

s

p50

0.000892

s

<

p95

0.020045

s

<

p99

0.021919

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

4.26569

s

SD

28.858429

s

p50

0.314341

s

<

p95

60.008646

s

<

p99

60.023867

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.576439

s

SD

0

s

p50

0.576439

s

≤

p95

0.576439

s

≤

p99

0.576439

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.105412

s

SD

28.235254

s

p50

0.144226

s

<

p95

60.015163

s

<

p99

60.065059

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.310133

s

SD

0.445673

s

p50

0.423991

s

<

p95

0.907298

s

<

p99

1.364467

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

75.34

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

270.42

KiB/s

<

mean

698.38

KiB/s

SD

511.94

KiB/s

<

p95

1.34

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

141.63

KiB/s

<

mean

609.51

KiB/s

SD

386.75

KiB/s

<

p95

1.24

MiB/s

Total bytes received

Total bytes received on primary network interface

count

16.20

GiB

mean

18.64

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

14.14

GiB

mean

16.27

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.03 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 89% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

62.6

%

<

mean

91.28

%

SD

12.31

%

<

p95

98.55

%

System

15.95

%

CPU percentiles

CPU usage percentiles

p50

97.02

%

p95

98.55

%

p99

98.82

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

28

hosts

mean time

782.25

s

Memory used percentage

Percentage of memory used

p5

13.8

%

<

mean

26.55

%

SD

9.81

%

<

p95

48.78

%

Memory available percentage

Percentage of available memory

p5

51.22

%

<

mean

73.45

%

SD

9.81

%

<

p95

86.2

%

Max host memory used

Maximum memory usage percentage across all hosts

max

52.23

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.47

%

Memory growth rate

Rate of memory growth over time

growth

1.03

MiB/s

Disk read throughput

Disk read throughput in MiB/s

175.27

MiB/s

Disk write throughput

Disk write throughput in MiB/s

174.94

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/10

hosts

Mount Point /efi-boot

0/9

hosts

Mount Point /etc/hostname

0/10

hosts

Mount Point /etc/hosts

0/10

hosts

Mount Point /etc/nomad.d/client-meta.json

0/8

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

0/10

hosts

Mount Point /nix/store

0/10

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

12.40

5 min

9.65

15 min

5.42

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

89.29

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

13.02

%

<

mean

71.1603

%

SD

25.5462

%

<

p95

97.32

%

60 second average

69.2503

%

300 second average

55.3911

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.1335

%

SD

0.7174

%

≤

p95

0

%

60 second average

0.1386

%

300 second average

0.1554

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0163

%

SD

0.0974

%

≤

p95

0

%

60 second average

0.0177

%

300 second average

0.0425

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

17.3204

%

SD

19.3203

%

<

p95

52.43

%

60 second average

16.8731

%

300 second average

13.7603

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.5429

%

SD

4.8748

%

<

p95

13.6

%

60 second average

2.5734

%

300 second average

2.5450

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.47

%

<

mean

29.34

%

SD

15.12

%

<

p95

47.19

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.47

KiB

<

mean

247.43

KiB

SD

106.56

KiB

<

p95

364.02

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

29.56

threads

SD

12.35

threads

<

p95

39

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

5

file descriptors

<

mean

69.76

file descriptors

SD

27.13

file descriptors

<

p95

87

file descriptors

Write/`get_agent_activity` with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 27 Aug 2026 09:32:14 UTC

Duration

1800s

Peer count

30

Peer count at end

25

Behaviours

get_agent_activity_volatile (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_volatile_33056872755_1

Highest observed action_seq

The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.

Mean

mean

45.48

/s

Max

Highest per-partition mean rate

max

97.2

/s

Min

Lowest per-partition mean rate

min

11.99

/s

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.189248

s

SD

0.241659

s

Max

Highest per-partition mean latency

max mean

0.577301

s

Min

Lowest per-partition mean latency

min mean

0.065024

s

Running volatile conductors count

The number of conductors running by get_agent_activity_volatile peers.

p5

9

conductors

<

mean

12.83

conductors

SD

2.12

conductors

<

p95

16

conductors

Volatile conductor total running duration

The total running duration of a get_agent_activity_volatile conductor.

Mean

mean

758.217253

s

SD

412.044951

s

Max

Highest per-partition mean latency

max mean

939.714286

s

Min

Lowest per-partition mean latency

min mean

653.529412

s

Volatile conductor running duration

The duration that a get_agent_activity_volatile conductor was running before being stopped.

Mean

mean

42.67122

s

SD

25.482028

s

Max

Highest per-partition mean latency

max mean

138.357785

s

Min

Lowest per-partition mean latency

min mean

29.520625

s

Volatile conductor stopped duration

The duration that a get_agent_activity_volatile conductor was stopped before being started again.

Mean

mean

29.106236

s

SD

28.280098

s

Max

Highest per-partition mean latency

max mean

43.095231

s

Min

Lowest per-partition mean latency

min mean

8.156302

s

Reached target arc

Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.

get_agent_activity_volatile_agent: uhCAk56ROvu_6ErvlU6IjQisbNKGUSZdXrRWfsJycjcTpemOTmYlJ

p5

0

<

mean

0.32

SD

0.47

<

p95

1

get_agent_activity_volatile_agent: uhCAk61l0L33REWAAAJ50Yl6VB1rEVTpHaQzv8swLFED_-xZmnKc0

p5

0

<

mean

0.44

SD

0.5

<

p95

1

get_agent_activity_volatile_agent: uhCAk9YY3BqbbIW2DKpXhwKy8FIuZJb4hc2u0ZIchJySGJQ9WwmSM

p5

0

<

mean

0.27

SD

0.45

<

p95

1

get_agent_activity_volatile_agent: uhCAkJFCjSmhMmai0HCBfRR0sGlnj2KqSs7h-0SCupMOSg9G_m5K1

p5

0

<

mean

0.21

SD

0.4

<

p95

1

get_agent_activity_volatile_agent: uhCAkJzFH8Dw2I0-FtTc1fPjQz_2cFEW6BqiQQZwEIpxZL4ZMWxe0

p5

0

<

mean

0.3

SD

0.46

<

p95

1

get_agent_activity_volatile_agent: uhCAkL8PGJomPnbK3CQYChc2XuKpYAS4kMaeK4h6gGffKRQ-cBjrZ

p5

0

<

mean

0.28

SD

0.45

<

p95

1

get_agent_activity_volatile_agent: uhCAkU2JmyM7qj5gTqzbnzHAzVM0RwFGkMeIMwulkKnmOd6CMsOLm

p5

0

<

mean

0.23

SD

0.42

<

p95

1

get_agent_activity_volatile_agent: uhCAkVijnzQdU4-Uw0UdeK_EPnZwmRevftdZEAIA2YGjqUF1agupq

p5

0

<

mean

0.49

SD

0.5

<

p95

1

get_agent_activity_volatile_agent: uhCAkb2Xz39CHVP0qSs5eqwfKf3m8iRJgwGAvhPvnszL6f0lm_ItI

p5

0

<

mean

0.27

SD

0.45

<

p95

1

get_agent_activity_volatile_agent: uhCAkdH-KGYHU3OUSfajdf48tEVp3pkit0OpF-EXjBiyEpgOJQB7f

p5

0

<

mean

0.31

SD

0.46

<

p95

1

get_agent_activity_volatile_agent: uhCAkefJy-78wku_bJhiKiQMZJnca9KsqMzOA-RqCGCOMwBteVALz

p5

0

≤

mean

0

SD

0

≤

p95

0

get_agent_activity_volatile_agent: uhCAkhWQuJZ6-GOvQIMOi6cqx6diAR89quHc6QS1SbHIBExTZYcw5

p5

0

<

mean

0.36

SD

0.48

<

p95

1

get_agent_activity_volatile_agent: uhCAkqGs0W0UP3SMlQqZoleK60y9yeUx0ZZeIpCp7KtdgxzwS9CdN

p5

0

<

mean

0.31

SD

0.46

<

p95

1

get_agent_activity_volatile_agent: uhCAkvkrjVj9CMboHhxG-1Hknik0GBk2WBGrbIxeFepd0oKl1ybhh

p5

0

<

mean

0.25

SD

0.43

<

p95

1

get_agent_activity_volatile_agent: uhCAkw0oZMWcnXFpexrTBE6RA4gukP2wIOZPFdLKHMTfc7VJ0z2KO

p5

0

<

mean

0.41

SD

0.49

<

p95

1

get_agent_activity_volatile_agent: uhCAkx37trM9staqU7s1Klde3QGjKLWQMsL8XmAOpOjy_FGxTLTye

p5

0

<

mean

0.38

SD

0.49

<

p95

1

get_agent_activity_volatile_agent: uhCAkxtf3OIGe2rCbxF2DJqGyEOneOHbOUaFZ5zapfyq-qUc1O1l5

p5

0

<

mean

0.24

SD

0.43

<

p95

1

get_agent_activity_volatile_agent: uhCAkzpinzCQzGE90rycZ_3U-OmLGZXFW9JA2YI9CcqrbNWpKmKRE

p5

0

<

mean

0.42

SD

0.49

<

p95

1

Error count

The number of errors accumulated.

180

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.135438

s

SD

8.473259

s

p50

0.259012

s

<

p95

0.886589

s

<

p99

65.813532

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

13

over 1634.590186355s

mean rate

9.07

/s

std

13.55

/s

p5

0

/s

p95

31.67

/s

peak

77.48

/s

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007812

s

SD

0.003834

s

p50

0.007156

s

<

p95

0.014241

s

≤

p99

0.014241

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.009929

s

SD

0.00552

s

p50

0.011083

s

<

p95

0.021193

s

<

p99

0.021897

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.114677

s

SD

0.289343

s

p50

0.270042

s

<

p95

0.808091

s

<

p99

1.111462

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.012567

s

SD

0.416828

s

p50

0.012216

s

<

p95

1.186696

s

<

p99

1.315348

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006655

s

SD

0.003445

s

p50

0.006085

s

<

p95

0.01364

s

≤

p99

0.01364

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008719

s

SD

0.005082

s

p50

0.009671

s

<

p95

0.019151

s

<

p99

0.019714

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.141114

s

SD

8.616069

s

p50

0.269652

s

<

p95

0.882335

s

<

p99

65.815354

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.011538

s

SD

0.410425

s

p50

0.010054

s

<

p95

1.185239

s

<

p99

1.207258

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.0029

s

SD

0.001614

s

p50

0.002992

s

<

p95

0.005841

s

<

p99

0.006175

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

3.2e-05

s

SD

6e-06

s

p50

3e-05

s

<

p95

4.4e-05

s

≤

p99

4.4e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.001103

s

SD

0.000791

s

p50

0.001155

s

<

p95

0.002668

s

<

p99

0.00275

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000644

s

SD

0.000452

s

p50

0.000482

s

<

p95

0.001726

s

≤

p99

0.001726

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.137812

s

SD

8.616455

s

p50

0.264745

s

<

p95

0.878469

s

<

p99

65.814084

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.007878

s

SD

0.410015

s

p50

0.000611

s

<

p95

1.177854

s

<

p99

1.20401

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

7e-06

s

SD

4e-06

s

p50

7e-06

s

<

p95

1.6e-05

s

≤

p99

1.6e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.004647

s

SD

0.002338

s

p50

0.003555

s

<

p95

0.008622

s

<

p99

0.009904

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000267

s

SD

0.000172

s

p50

0.000288

s

<

p95

0.000592

s

<

p99

0.000628

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

4.88

s

<

mean

346.12

s

SD

505.2

s

<

p95

1500.89

s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

145.624939

s

SD

248.367774

s

p50

53.32038

s

<

p95

802.071595

s

<

p99

1062.128533

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.034232

s

SD

0.249272

s

p50

0.016185

s

<

p95

0.527447

s

<

p99

0.765154

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

<

p99

2e-06

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.102898

s

SD

0.195116

s

p50

0.029082

s

<

p95

0.521015

s

<

p99

0.768147

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.26536

s

SD

0.287802

s

p50

0.094147

s

<

p95

0.544118

s

<

p99

1.047126

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

4.586494

s

SD

40.681614

s

p50

7.145217

s

<

p95

93.138854

s

<

p99

175.840778

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

56.570197

s

SD

58.633112

s

p50

27.738634

s

<

p95

172.767994

s

<

p99

244.426226

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.008228

s

SD

0.004262

s

p50

0.001141

s

<

p95

0.013036

s

<

p99

0.018974

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.026232

s

SD

0.049211

s

p50

0.001594

s

<

p95

0.053098

s

<

p99

0.201239

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000912

s

SD

0.000618

s

p50

0.000437

s

<

p95

0.001964

s

<

p99

0.002329

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

8.599365

s

SD

27.633947

s

p50

0.478418

s

<

p95

60.002581

s

<

p99

60.012693

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.550917

s

SD

0.374716

s

p50

0.596711

s

<

p95

1.202756

s

≤

p99

1.202756

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.175944

s

SD

11.213685

s

p50

0.258289

s

<

p95

0.93734

s

<

p99

60.001641

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.316308

s

SD

20.226976

s

p50

0.337131

s

<

p95

60.001218

s

<

p99

60.005058

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

38.18

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

2.93

KiB/s

<

mean

398.00

KiB/s

SD

661.57

KiB/s

<

p95

1.26

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

3.48

KiB/s

<

mean

322.91

KiB/s

SD

418.08

KiB/s

<

p95

1.16

MiB/s

Total bytes received

Total bytes received on primary network interface

count

17.18

GiB

mean

9.83

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

13.94

GiB

mean

7.97

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 42% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.43

%

<

mean

47.65

%

SD

37.38

%

<

p95

98.07

%

System

9.47

%

CPU percentiles

CPU usage percentiles

p50

41.33

%

p95

98.07

%

p99

98.65

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

14

hosts

mean time

1071.79

s

Memory used percentage

Percentage of memory used

p5

6.15

%

<

mean

15.44

%

SD

6.95

%

<

p95

25.99

%

Memory available percentage

Percentage of available memory

p5

74.01

%

<

mean

84.56

%

SD

6.95

%

<

p95

93.85

%

Max host memory used

Maximum memory usage percentage across all hosts

max

33.40

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.04

%

Memory growth rate

Rate of memory growth over time

growth

0.36

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.33

MiB/s

Disk write throughput

Disk write throughput in MiB/s

62.67

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/18

hosts

Mount Point /boot

0/1

hosts

Mount Point /efi-boot

0/16

hosts

Mount Point /etc/hostname

0/5

hosts

Mount Point /etc/hosts

0/5

hosts

Mount Point /etc/nomad.d/client-meta.json

0/3

hosts

Mount Point /etc/resolv.conf

0/5

hosts

Mount Point /nix/store

0/18

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.04

5 min

5.61

15 min

4.07

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

42.31

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

25.1269

%

SD

32.7579

%

<

p95

84.9

%

60 second average

24.9892

%

300 second average

23.1298

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

15.4524

%

SD

18.9635

%

<

p95

51.94

%

60 second average

15.3168

%

300 second average

14.1267

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

4.6734

%

SD

8.4266

%

<

p95

23.02

%

60 second average

4.6041

%

300 second average

4.1660

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

19.47

%

SD

22.08

%

<

p95

63.31

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

22.51

KiB

<

mean

195.20

KiB

SD

116.41

KiB

<

p95

392.33

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

28.17

threads

SD

10.67

threads

<

p95

39

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

7

file descriptors

<

mean

69.25

file descriptors

SD

23.25

file descriptors

<

p95

87

file descriptors

Write/Query

↑ Back to index

Creates an entry, then queries the source chain and performs a simple operation on the entries, then repeats.

Started

Thu, 27 Aug 2026 09:15:29 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_query_33056872755_1

Write timing

The time it takes to complete the zome function call that writes the entry.

mean

0.031014

s

SD

0.020732

s

p50

0.025637

s

<

p95

0.070603

s

<

p99

0.088998

s

Write rate

The number of writes completed per second.

mean

91.14

/s

Query timing

The time it takes to complete the chain_query_count_len zome call that queries the source chain.

mean

0.074907

s

SD

0.05329

s

p50

0.062464

s

<

p95

0.174475

s

<

p99

0.212992

s

Query rate

The number of chain queries completed per second.

mean

91.14

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::chain_query_count_len'

Metered usage count of the WASM ribosome for this zome function.

total

82283886378

over 280.004101274s

mean rate

2.9386690815e+08

/s

std

4.474386765e+07

/s

p5

2.2778810289e+08

/s

p95

3.5322271244e+08

/s

peak

3.6797603135e+08

/s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

88004190

over 280.004101274s

mean rate

314296.91

/s

std

227267.56

/s

p5

142979.14

/s

p95

822127.6

/s

peak

1.16885154e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.004101274s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

21679850

over 280.004101274s

mean rate

77427.11

/s

std

56007.39

/s

p5

35215.79

/s

p95

202930.37

/s

peak

287888.35

/s

Zome call duration: 'crud::chain_query_count_len'

Duration of this zome call as measured by Holochain internally.

mean

0.030233

s

SD

0.009948

s

p50

0.028119

s

<

p95

0.040472

s

<

p99

0.041311

s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.002577

s

SD

0.000155

s

p50

0.002538

s

<

p95

0.00278

s

<

p99

0.0028

s

WASM call duration: 'crud::chain_query_count_len'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.029657

s

SD

0.009896

s

p50

0.027553

s

<

p95

0.039842

s

<

p99

0.040676

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002073

s

SD

0.000127

s

p50

0.002043

s

<

p95

0.002244

s

<

p99

0.002262

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000811

s

SD

0

s

p50

0.000811

s

≤

p95

0.000811

s

≤

p99

0.000811

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000671

s

SD

4.2e-05

s

p50

0.00066

s

<

p95

0.00073

s

<

p99

0.000735

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000263

s

SD

2e-05

s

p50

0.000232

s

<

p95

0.000283

s

<

p99

0.000286

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.014246

s

SD

0.004661

s

p50

0.013518

s

<

p95

0.01864

s

<

p99

0.01895

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001049

s

SD

7.2e-05

s

p50

0.001028

s

<

p95

0.001157

s

<

p99

0.001179

s

Post-commit duration

Time spent executing post-commit workflows.

mean

5.5e-05

s

SD

7e-06

s

p50

5.3e-05

s

<

p95

6.6e-05

s

≤

p99

6.6e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

18.11

s

<

mean

148.11

s

SD

83.67

s

<

p95

278.11

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.00409243s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000287

s

SD

0

s

p50

0.000287

s

≤

p95

0.000287

s

≤

p99

0.000287

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.00083

s

SD

0.000177

s

p50

0.000758

s

<

p95

0.001062

s

<

p99

0.001082

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.0206

s

SD

0.007855

s

p50

0.018456

s

<

p95

0.029627

s

<

p99

0.030364

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.00132

s

SD

0

s

p50

0.00132

s

≤

p95

0.00132

s

≤

p99

0.00132

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002067

s

SD

0

s

p50

0.002067

s

≤

p95

0.002067

s

≤

p99

0.002067

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002246

s

SD

0.000799

s

p50

0.002077

s

<

p95

0.003076

s

<

p99

0.003143

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000623

s

SD

9e-05

s

p50

0.000651

s

<

p95

0.000755

s

<

p99

0.000782

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000236

s

SD

1.4e-05

s

p50

0.000231

s

<

p95

0.000256

s

<

p99

0.000258

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

52.30

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

8.98

KiB/s

<

mean

457.25

KiB/s

SD

1.10

MiB/s

<

p95

3.70

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

8.82

KiB/s

<

mean

18.75

KiB/s

SD

18.21

KiB/s

<

p95

64.07

KiB/s

Total bytes received

Total bytes received on primary network interface

count

125.03

MiB

mean

457.25

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.13

MiB

mean

18.75

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 93.5%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 3.60 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

39.55

%

<

mean

67.21

%

SD

20.29

%

<

p95

91.43

%

System

14.91

%

CPU percentiles

CPU usage percentiles

p50

76.94

%

p95

91.43

%

p99

93.45

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

75.00

s

Memory used percentage

Percentage of memory used

p5

6.26

%

<

mean

7.52

%

SD

1.12

%

<

p95

9.46

%

Memory available percentage

Percentage of available memory

p5

90.54

%

<

mean

92.48

%

SD

1.12

%

<

p95

93.74

%

Max host memory used

Maximum memory usage percentage across all hosts

max

7.52

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

3.60

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

4.28

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.68

5 min

1.35

15 min

0.59

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

1.57

%

<

mean

18.5772

%

SD

16.3416

%

<

p95

45.72

%

60 second average

12.5817

%

300 second average

5.2569

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

25.04

%

<

mean

34.74

%

SD

5.19

%

<

p95

39.17

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

228.61

KiB

<

mean

283.65

KiB

SD

80.31

KiB

<

p95

441.00

KiB

Holochain process threads

Number of threads in Holochain process

p5

24

threads

<

mean

26.23

threads

SD

5.29

threads

<

p95

32

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

57

file descriptors

<

mean

61.3

file descriptors

SD

11.05

file descriptors

<

p95

72

file descriptors

Write/Read

↑ Back to index

Creates an entry and reads it back, then repeats.

Started

Thu, 27 Aug 2026 09:17:18 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_read_33056872755_1

Write timing

The time taken to write the entry.

mean

0.04427

s

SD

0.026763

s

p50

0.036954

s

<

p95

0.0855

s

<

p99

0.117787

s

Write rate

The number of writes completed per second.

mean

106.45

/s

Read timing

The time taken to read the entry.

mean

0.047702

s

SD

0.029407

s

p50

0.04125

s

<

p95

0.09374

s

<

p99

0.118911

s

Read rate

The number of reads completed per second.

mean

106.45

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.0006

s

SD

0.000101

s

p50

0.000581

s

<

p95

0.0007

s

<

p99

0.000705

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

105161790

over 280.004183371s

mean rate

375572.94

/s

std

247371.07

/s

p5

207314.84

/s

p95

879321.69

/s

peak

1.29753736e+06

/s

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

98106826

over 280.004183371s

mean rate

350376.98

/s

std

230465.8

/s

p5

190134.95

/s

p95

820601.85

/s

peak

1.20764869e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.004183371s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

12950684

over 280.004183371s

mean rate

46251.84

/s

std

30461.95

/s

p5

25530.84

/s

p95

108288.55

/s

peak

159791.84

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.003221

s

SD

0.000155

s

p50

0.003204

s

<

p95

0.003359

s

<

p99

0.003368

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.002203

s

SD

0.000173

s

p50

0.002166

s

<

p95

0.002379

s

<

p99

0.002393

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002632

s

SD

0.000134

s

p50

0.002617

s

<

p95

0.002754

s

<

p99

0.002761

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001617

s

SD

0.000152

s

p50

0.001585

s

<

p95

0.001773

s

<

p99

0.001785

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001193

s

SD

0

s

p50

0.001193

s

≤

p95

0.001193

s

≤

p99

0.001193

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000842

s

SD

3.7e-05

s

p50

0.000837

s

<

p95

0.000875

s

<

p99

0.000877

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000321

s

SD

2.6e-05

s

p50

0.000272

s

<

p95

0.000343

s

<

p99

0.000345

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000632

s

SD

0.000102

s

p50

0.000613

s

<

p95

0.000733

s

<

p99

0.000739

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001343

s

SD

6.9e-05

s

p50

0.001335

s

<

p95

0.001404

s

<

p99

0.001409

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.8e-05

s

SD

9e-06

s

p50

8.6e-05

s

<

p95

9.8e-05

s

<

p99

9.9e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.59

s

<

mean

147.59

s

SD

83.67

s

<

p95

277.59

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.004161337s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000987

s

SD

0

s

p50

0.000987

s

≤

p95

0.000987

s

≤

p99

0.000987

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001543

s

SD

0.000168

s

p50

0.001518

s

<

p95

0.001653

s

<

p99

0.001656

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.030799

s

SD

0.010952

s

p50

0.028468

s

<

p95

0.041503

s

<

p99

0.042413

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.001858

s

SD

0

s

p50

0.001858

s

≤

p95

0.001858

s

≤

p99

0.001858

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.005914

s

SD

0

s

p50

0.005914

s

≤

p95

0.005914

s

≤

p99

0.005914

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00262

s

SD

0.000932

s

p50

0.002422

s

<

p95

0.003507

s

<

p99

0.003578

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001085

s

SD

0.000392

s

p50

0.001803

s

≤

p95

0.001803

s

≤

p99

0.001803

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000288

s

SD

2.2e-05

s

p50

0.000285

s

<

p95

0.00031

s

≤

p99

0.00031

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

66.32

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

9.05

KiB/s

<

mean

571.24

KiB/s

SD

1.45

MiB/s

<

p95

3.88

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

8.82

KiB/s

<

mean

21.01

KiB/s

SD

23.89

KiB/s

<

p95

77.65

KiB/s

Total bytes received

Total bytes received on primary network interface

count

161.78

MiB

mean

571.24

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.95

MiB

mean

21.01

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 93.5%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 2.55 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

74.4

%

<

mean

83.77

%

SD

6.32

%

<

p95

93.23

%

System

17.45

%

CPU percentiles

CPU usage percentiles

p50

87.11

%

p95

93.23

%

p99

93.46

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

175.86

s

Memory used percentage

Percentage of memory used

p5

7.34

%

<

mean

8.44

%

SD

0.89

%

<

p95

9.46

%

Memory available percentage

Percentage of available memory

p5

90.54

%

<

mean

91.56

%

SD

0.89

%

<

p95

92.66

%

Max host memory used

Maximum memory usage percentage across all hosts

max

8.44

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

2.55

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

4.79

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.18

5 min

2.24

15 min

0.99

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

15.74

%

<

mean

32.3927

%

SD

13.5049

%

<

p95

45.82

%

60 second average

25.8137

%

300 second average

11.9200

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.89

%

<

mean

30.07

%

SD

8.26

%

<

p95

39.16

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

19.17

KiB

<

mean

279.79

KiB

SD

116.77

KiB

<

p95

447.59

KiB

Holochain process threads

Number of threads in Holochain process

p5

8

threads

<

mean

26

threads

SD

6.89

threads

<

p95

32

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

59.81

file descriptors

SD

15.78

file descriptors

<

p95

72

file descriptors

Write validated

↑ Back to index

Creates an entry, updates it and links the create to the update, then repeats. Each of the three actions is validated by the included hApp.

Started

Thu, 27 Aug 2026 09:19:08 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_33056872755_1

Write timing

The time it takes to complete the zome function call that writes the entry, updates it, and links to the update.

mean

0.082429

s

SD

0.034417

s

p50

0.087061

s

<

p95

0.128363

s

<

p99

0.179227

s

Write rate

The number of write zome function calls that complete per second.

mean

47.46

/s

Read timing

The time it takes to complete the zome function call that reads the data.

mean

0.122794

s

SD

0.045407

s

p50

0.128826

s

<

p95

0.182727

s

<

p99

0.269065

s

Read rate

The number of read zome function calls that complete per second.

mean

47.46

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.005312

s

SD

0.001941

s

p50

0.004905

s

<

p95

0.008411

s

<

p99

0.008673

s

WASM usage: 'validated::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

43886520

over 270.008498375s

mean rate

162537.97

/s

std

57492.55

/s

p5

103019.38

/s

p95

285020.3

/s

peak

346831.78

/s

WASM usage: 'validated::update_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

51674158

over 270.008498375s

mean rate

191380.23

/s

std

67669.77

/s

p5

121205.27

/s

p95

335334.61

/s

peak

408057.58

/s

WASM usage: 'validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

82504464

over 280.008550916s

mean rate

294650.66

/s

std

105579.06

/s

p5

182096.3

/s

p95

504567.6

/s

peak

614520.46

/s

WASM usage: 'validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

772600127

over 270.008498375s

mean rate

2.86139907e+06

/s

std

1.01138725e+06

/s

p5

1.80093556e+06

/s

p95

5.0098799e+06

/s

peak

6.10207218e+06

/s

Zome call duration: 'validated::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004183

s

SD

0.000128

s

p50

0.004176

s

<

p95

0.004249

s

<

p99

0.004273

s

Zome call duration: 'validated::update_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.011203

s

SD

0.00213

s

p50

0.010775

s

<

p95

0.012962

s

<

p99

0.013075

s

WASM call duration: 'validated::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003483

s

SD

0.000114

s

p50

0.003482

s

<

p95

0.003545

s

<

p99

0.003562

s

WASM call duration: 'validated::update_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.010509

s

SD

0.002106

s

p50

0.010087

s

<

p95

0.012253

s

<

p99

0.012368

s

WASM call duration: 'validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000983

s

SD

0.000267

s

p50

0.000987

s

<

p95

0.001007

s

<

p99

0.002446

s

WASM call duration: 'validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003665

s

SD

0.000218

s

p50

0.003619

s

<

p95

0.003828

s

<

p99

0.003834

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000434

s

SD

1.8e-05

s

p50

0.000433

s

<

p95

0.000444

s

<

p99

0.000447

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000409

s

SD

1.9e-05

s

p50

0.000412

s

<

p95

0.000459

s

<

p99

0.000484

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.004616

s

SD

0.001246

s

p50

0.004274

s

<

p95

0.005777

s

<

p99

0.005885

s

Host function call duration: '__hc__update_1'

Duration of this host function call invoked from within WASM.

mean

0.007193

s

SD

0.002108

s

p50

0.006751

s

<

p95

0.00898

s

<

p99

0.009111

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001634

s

SD

9.1e-05

s

p50

0.001724

s

<

p95

0.001864

s

<

p99

0.001879

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000122

s

SD

4e-06

s

p50

0.000122

s

<

p95

0.000128

s

≤

p99

0.000128

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.22

s

<

mean

147.23

s

SD

83.67

s

<

p95

277.23

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.008559502s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.00185

s

SD

0

s

p50

0.00185

s

≤

p95

0.00185

s

≤

p99

0.00185

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001836

s

SD

0.001075

s

p50

0.001875

s

<

p95

0.001988

s

<

p99

0.007701

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.050731

s

SD

0.01743

s

p50

0.045161

s

<

p95

0.065305

s

<

p99

0.066407

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.004776

s

SD

0

s

p50

0.004776

s

≤

p95

0.004776

s

≤

p99

0.004776

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.003705

s

SD

0

s

p50

0.003705

s

≤

p95

0.003705

s

≤

p99

0.003705

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003689

s

SD

0.00138

s

p50

0.003294

s

<

p95

0.004795

s

<

p99

0.004879

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001821

s

SD

0.000797

s

p50

0.003304

s

≤

p95

0.003304

s

<

p99

0.004606

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000344

s

SD

5.1e-05

s

p50

0.000343

s

<

p95

0.000352

s

<

p99

0.000616

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

66.42

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

8.81

KiB/s

<

mean

584.80

KiB/s

SD

1.66

MiB/s

<

p95

6.04

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

9.35

KiB/s

<

mean

23.73

KiB/s

SD

27.28

KiB/s

<

p95

112.42

KiB/s

Total bytes received

Total bytes received on primary network interface

count

165.62

MiB

mean

584.80

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

6.72

MiB

mean

23.73

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 93.5%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

73.61

%

<

mean

84.8

%

SD

6.31

%

<

p95

93.23

%

System

18.38

%

CPU percentiles

CPU usage percentiles

p50

88.00

%

p95

93.23

%

p99

93.46

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

227.59

s

Memory used percentage

Percentage of memory used

p5

8.1

%

<

mean

9.11

%

SD

0.97

%

<

p95

10.72

%

Memory available percentage

Percentage of available memory

p5

89.28

%

<

mean

90.89

%

SD

0.97

%

<

p95

91.9

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.11

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

0.21

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.14

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.29

5 min

3.22

15 min

1.49

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

21.41

%

<

mean

37.964

%

SD

9.0536

%

<

p95

45.82

%

60 second average

35.5693

%

300 second average

19.7927

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.81

%

<

mean

25.17

%

SD

7.28

%

<

p95

38.75

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

22.79

KiB

<

mean

321.84

KiB

SD

257.24

KiB

<

p95

670.13

KiB

Holochain process threads

Number of threads in Holochain process

p5

9

threads

<

mean

36.09

threads

SD

45.35

threads

<

p95

109

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

80.91

file descriptors

SD

97.83

file descriptors

<

p95

241

file descriptors

Write Validated `must_get_agent_activity`

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started

Thu, 27 Aug 2026 09:57:20 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

must_get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_must_get_agent_activity_33056872755_1

Highest observed action_seq

The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

1211.33

Max

Highest chain head value observed for any single write agent.

1710

Write agents observed

6

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

185.123939

s

SD

67.804843

s

Max

Highest per-partition mean latency

max mean

652.644295

s

Min

Lowest per-partition mean latency

min mean

16.922

s

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

2.91

/s

Max

Highest per-partition mean rate

max

5.1

/s

Min

Lowest per-partition mean rate

min

1.96

/s

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

0.375705

s

SD

0.434868

s

Max

Highest per-partition mean latency

max mean

0.932541

s

Min

Lowest per-partition mean latency

min mean

0.080906

s

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

4013

Agents affected

18 / 18

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

223

Max

Highest count in any single partition

614

Min

Lowest count in any single partition

11

Final error count

The total number of all types of error accumulated over the run by all agents.

57953

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.020532

s

SD

0.054908

s

p50

0.001203

s

<

p95

0.100736

s

<

p99

0.388124

s

WASM usage: 'validated_must_get_agent_activity::announce_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 887.697825582s

Not enough time data to show a trend.

WASM usage: 'validated_must_get_agent_activity::create_sample_entries_batch'

Metered usage count of the WASM ribosome for this zome function.

total

2042436689

over 887.697825582s

mean rate

1.95871453989e+09

/s

std

5.36322002697e+09

/s

p5

0

/s

p95

6.90697072718e+09

/s

peak

5.40248447792e+10

/s

WASM usage: 'validated_must_get_agent_activity::create_validated_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

11484828

over 885.654393858s

mean rate

2.195281867e+07

/s

std

4.555443306e+07

/s

p5

89726.89

/s

p95

7.842722178e+07

/s

peak

1.04459755084e+09

/s

WASM usage: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Metered usage count of the WASM ribosome for this zome function.

total

371645068

over 885.654393858s

mean rate

1.19234368314e+09

/s

std

3.41695380769e+09

/s

p5

217946.28

/s

p95

6.62407463838e+09

/s

peak

4.50290797707e+10

/s

WASM usage: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

35330

over 885.832321531s

mean rate

661672.82

/s

std

753910

/s

p5

0

/s

p95

1.73766572e+06

/s

peak

7.47009876e+06

/s

WASM usage: 'validated_must_get_agent_activity_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

204493836

over 888.135279319s

mean rate

5.104446924e+08

/s

std

1.85308834149e+09

/s

p5

2.62757272e+06

/s

p95

1.98122487885e+09

/s

peak

4.98529308623e+10

/s

WASM usage: 'validated_must_get_agent_activity_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

5864715785

over 887.697825582s

mean rate

3.14680193105e+10

/s

std

9.22890159748e+10

/s

p5

4.384076106e+07

/s

p95

1.41115653847e+11

/s

peak

1.47760191057e+12

/s

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.00543

s

SD

0.001232

s

p50

0.006115

s

<

p95

0.006486

s

≤

p99

0.006486

s

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.162699

s

SD

0.133858

s

p50

0.089761

s

<

p95

0.416584

s

<

p99

0.726536

s

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.007619

s

SD

0.003845

s

p50

0.006036

s

<

p95

0.011265

s

<

p99

0.026621

s

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.007738

s

SD

0.004804

s

p50

0.006038

s

<

p95

0.019499

s

<

p99

0.024406

s

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.010331

s

SD

0.083018

s

p50

0.006678

s

<

p95

0.011204

s

<

p99

0.390867

s

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004588

s

SD

0.001085

s

p50

0.005136

s

<

p95

0.005728

s

≤

p99

0.005728

s

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.161289

s

SD

0.133308

s

p50

0.088532

s

<

p95

0.414191

s

<

p99

0.723129

s

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00644

s

SD

0.003143

s

p50

0.00505

s

<

p95

0.009953

s

<

p99

0.021413

s

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005973

s

SD

0.003501

s

p50

0.005742

s

<

p95

0.016094

s

<

p99

0.016236

s

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00928

s

SD

0.083102

s

p50

0.005658

s

<

p95

0.009068

s

<

p99

0.39027

s

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001641

s

SD

0.001005

s

p50

0.001455

s

<

p95

0.003475

s

<

p99

0.005218

s

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.009846

s

SD

0.016136

s

p50

0.008712

s

<

p95

0.037532

s

<

p99

0.073476

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

5.9e-05

s

SD

4.5e-05

s

p50

2.4e-05

s

<

p95

0.000136

s

<

p99

0.000268

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.001383

s

SD

0.001308

s

p50

0.000591

s

<

p95

0.002628

s

<

p99

0.008586

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.001641

s

SD

0.001372

s

p50

0.000538

s

<

p95

0.002912

s

<

p99

0.008485

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000873

s

SD

0.059953

s

p50

0.000697

s

<

p95

0.002015

s

<

p99

0.388165

s

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.025097

s

SD

0.043457

s

p50

0.019502

s

<

p95

0.134711

s

<

p99

0.217694

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.061368

s

SD

0.056277

s

p50

0.027273

s

<

p95

0.166333

s

<

p99

0.284332

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1e-05

s

SD

6e-06

s

p50

7e-06

s

<

p95

2.1e-05

s

<

p99

2.4e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002754

s

SD

0.001952

s

p50

0.002583

s

<

p95

0.006022

s

<

p99

0.011249

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000271

s

SD

0.000182

s

p50

0.0002

s

<

p95

0.00065

s

<

p99

0.000955

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.18

s

<

mean

439.84

s

SD

253.59

s

<

p95

836.51

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

17312

over 887.697643079s

mean rate

116574.86

/s

std

513882.85

/s

p5

463.13

/s

p95

477465.36

/s

peak

1.408773571e+07

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

41.435973

s

SD

32.186041

s

p50

34.207947

s

<

p95

96.538725

s

<

p99

171.590287

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.963798

s

SD

1.832759

s

p50

1.829982

s

<

p95

5.211608

s

<

p99

9.644927

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.160044

s

SD

0.314991

s

p50

0.190777

s

<

p95

1.001977

s

<

p99

1.654889

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.259906

s

SD

0.283492

s

p50

0.100748

s

<

p95

0.712668

s

<

p99

1.36176

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.82454

s

SD

7.341949

s

p50

3.46736

s

<

p95

20.569378

s

<

p99

29.818507

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

17.137866

s

SD

23.600368

s

p50

9.105293

s

<

p95

67.217434

s

<

p99

122.904756

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003367

s

SD

0.003116

s

p50

0.002378

s

<

p95

0.006762

s

<

p99

0.020826

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.011688

s

SD

0.008039

s

p50

0.002438

s

<

p95

0.025885

s

<

p99

0.031666

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000841

s

SD

0.00108

s

p50

0.000462

s

<

p95

0.002148

s

<

p99

0.007099

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.989441

s

SD

29.110409

s

p50

0.161393

s

<

p95

60.001743

s

<

p99

60.00311

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.356405

s

SD

0

s

p50

0.356405

s

≤

p95

0.356405

s

≤

p99

0.356405

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.24812

s

SD

2.544667

s

p50

0.24552

s

<

p95

0.384198

s

<

p99

15.379147

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

61.18

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

57.65

KiB/s

<

mean

379.75

KiB/s

SD

878.17

KiB/s

<

p95

1.52

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

26.48

KiB/s

<

mean

218.87

KiB/s

SD

168.84

KiB/s

<

p95

593.84

KiB/s

Total bytes received

Total bytes received on primary network interface

count

9.28

GiB

mean

10.68

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.35

GiB

mean

6.16

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 4.67 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/8 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 90% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

20.51

%

<

mean

75.03

%

SD

26.66

%

<

p95

98.6

%

System

13.86

%

CPU percentiles

CPU usage percentiles

p50

86.85

%

p95

98.60

%

p99

98.75

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

29

hosts

mean time

543.01

s

Memory used percentage

Percentage of memory used

p5

13.58

%

<

mean

24.83

%

SD

9.24

%

<

p95

46.05

%

Memory available percentage

Percentage of available memory

p5

53.95

%

<

mean

75.17

%

SD

9.24

%

<

p95

86.42

%

Max host memory used

Maximum memory usage percentage across all hosts

max

50.54

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.81

%

Memory growth rate

Rate of memory growth over time

growth

4.67

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2557.06

MiB/s

Disk write throughput

Disk write throughput in MiB/s

182.83

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/13

hosts

Mount Point /efi-boot

0/12

hosts

Mount Point /etc/hostname

1/8

hosts

Mount Point /etc/hosts

1/8

hosts

Mount Point /etc/nomad.d/client-meta.json

1/10

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/8

hosts

Mount Point /nix/store

0/13

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

8.58

5 min

7.57

15 min

7.05

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

90.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.99

%

<

mean

45.4217

%

SD

32.4358

%

<

p95

97.15

%

60 second average

43.9752

%

300 second average

40.3808

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.2883

%

SD

1.1933

%

<

p95

2.94

%

60 second average

0.2839

%

300 second average

0.2685

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.2014

%

SD

0.9832

%

<

p95

1.05

%

60 second average

0.2008

%

300 second average

0.1887

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

18.1996

%

SD

21.1515

%

<

p95

62.74

%

60 second average

17.7998

%

300 second average

16.5045

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

4.884

%

SD

8.8181

%

<

p95

24.91

%

60 second average

4.8757

%

300 second average

4.4843

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.37

%

<

mean

24.29

%

SD

15.3

%

<

p95

49.11

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

22.09

KiB

<

mean

285.62

KiB

SD

142.92

KiB

<

p95

455.34

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

32.01

threads

SD

17.11

threads

<

p95

43

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

69.68

file descriptors

SD

26.29

file descriptors

<

p95

85

file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
full: A full-arc conductor that is just here to serve entries to zero arc nodes.

Started

Thu, 27 Aug 2026 10:21:18 UTC

Duration

900s

Peer count

30

Peer count at end

28

Behaviours

full (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_and_read_33056872755_1

Create rate

The number of timed entries created by the zero-arc write node(s) per second.

Mean

mean

41.88

/s

Max

Highest per-partition mean rate

max

59.78

/s

Min

Lowest per-partition mean rate

min

21.04

/s

Fetch lag timing

For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.

Mean

mean

294.725364

s

SD

71.009434

s

Max

Highest per-partition mean latency

max mean

563.291913

s

Min

Lowest per-partition mean latency

min mean

23.807767

s

Fetch lag rate

The number of entries per second fetched by zero-arc read nodes.

Mean

mean

48.58

/s

Max

Highest per-partition mean rate

max

254

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc read nodes.

0.0075

Entry created count

Total number of entries created by zero-arc write nodes, by agent.

Total

40513

Agents affected

11 / 11

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3683

Max

Highest count in any single partition

5330

Min

Lowest count in any single partition

1817

Received count

Total number of entries fetched by zero-arc read nodes, by agent.

Total

3643

Agents affected

10 / 12

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

304

Max

Highest count in any single partition

598

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

21

<

mean

26.9

SD

3.6

<

p95

28

zero-arc

p5

6

<

mean

12.31

SD

4.28

<

p95

20

Error count

The number of errors accumulated across all nodes.

136

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.012886

s

SD

4.765433

s

p50

0.118535

s

<

p95

10.457133

s

<

p99

30.029644

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 786.881601145s

Not enough time data to show a trend.

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

172182199

over 875.57957989s

mean rate

5.5308793367e+08

/s

std

9.3664346473e+08

/s

p5

2.03439646e+06

/s

p95

3.48774016849e+09

/s

peak

4.51472237769e+09

/s

WASM usage: 'timed::get_timed_entries_network'

Metered usage count of the WASM ribosome for this zome function.

total

85826869

over 876.532714708s

mean rate

8.06951517611e+10

/s

std

1.60587055683e+11

/s

p5

2.339178095e+07

/s

p95

4.41448820447e+11

/s

peak

4.86104606876e+11

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

22640928

over 886.803779857s

mean rate

2.6015319146e+08

/s

std

1.33412010948e+09

/s

p5

0

/s

p95

9.3847724498e+08

/s

peak

2.65795550641e+10

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.009704

s

SD

0.00959

s

p50

0.007469

s

<

p95

0.039437

s

<

p99

0.040183

s

Zome call duration: 'timed::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

0.503249

s

SD

43.950839

s

p50

1.011608

s

<

p95

63.509981

s

<

p99

304.274328

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008827

s

SD

0.008654

s

p50

0.006725

s

<

p95

0.035429

s

<

p99

0.036101

s

WASM call duration: 'timed::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.571308

s

SD

45.242586

s

p50

1.466512

s

<

p95

77.08507

s

<

p99

304.273586

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001614

s

SD

0.001043

s

p50

0.001123

s

<

p95

0.003668

s

<

p99

0.006147

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000661

s

SD

0.000885

s

p50

0.000444

s

<

p95

0.003449

s

<

p99

0.00365

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000651

s

SD

0.000902

s

p50

0.000459

s

<

p95

0.003524

s

<

p99

0.003717

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.012373

s

SD

7.996125

s

p50

0.117689

s

<

p95

11.400072

s

<

p99

45.069756

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.05877

s

SD

1.226083

s

p50

0.149808

s

<

p95

3.401928

s

<

p99

6.539963

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002538

s

SD

0.002022

s

p50

0.001784

s

<

p95

0.005928

s

<

p99

0.01105

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000255

s

SD

0.00036

s

p50

0.000123

s

<

p95

0.000545

s

<

p99

0.002083

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

45.56

s

<

mean

439

s

SD

253.48

s

<

p95

836.12

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 886.803781406s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

53.44129

s

SD

41.322601

s

p50

0

s

<

p95

39.885939

s

<

p99

106.773309

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.117981

s

SD

0.106489

s

p50

0.001645

s

<

p95

0.275192

s

<

p99

0.500978

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

<

p99

1e-06

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.043935

s

SD

0.378243

s

p50

0.001564

s

<

p95

1.166864

s

<

p99

1.346204

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.144202

s

SD

0.087756

s

p50

0.025861

s

<

p95

0.254056

s

<

p99

0.368626

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

11.575645

s

SD

10.295798

s

p50

0.00261

s

<

p95

35.0983

s

<

p99

43.255469

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

8.229393

s

SD

14.77301

s

p50

0.001863

s

<

p95

28.635159

s

<

p99

99.223248

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001337

s

SD

0.001462

s

p50

0.00106

s

<

p95

0.0038

s

<

p99

0.007106

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.006112

s

SD

0.016769

s

p50

0.001945

s

<

p95

0.048758

s

<

p99

0.086004

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000522

s

SD

0.000437

s

p50

0.000352

s

<

p95

0.001005

s

<

p99

0.00259

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.459184

s

SD

29.73791

s

p50

0.253913

s

<

p95

60.001563

s

<

p99

60.002269

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.077622

s

SD

24.690361

s

p50

0.089982

s

<

p95

60.001455

s

<

p99

60.001757

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.150132

s

SD

0.04836

s

p50

0.100546

s

<

p95

0.182981

s

<

p99

0.196219

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

47.58

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

10.31

KiB/s

<

mean

453.55

KiB/s

SD

899.00

KiB/s

<

p95

1.95

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

7.49

KiB/s

<

mean

334.51

KiB/s

SD

531.20

KiB/s

<

p95

982.39

KiB/s

Total bytes received

Total bytes received on primary network interface

count

10.69

GiB

mean

12.30

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.89

GiB

mean

9.07

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.5%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.41 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/12 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 72% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

9.06

%

<

mean

63.45

%

SD

29.95

%

<

p95

97.87

%

System

15.87

%

CPU percentiles

CPU usage percentiles

p50

70.84

%

p95

97.87

%

p99

98.45

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

19

hosts

mean time

594.63

s

Memory used percentage

Percentage of memory used

p5

14.29

%

<

mean

28.75

%

SD

11.42

%

<

p95

51.7

%

Memory available percentage

Percentage of available memory

p5

48.3

%

<

mean

71.25

%

SD

11.42

%

<

p95

85.71

%

Max host memory used

Maximum memory usage percentage across all hosts

max

53.36

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.87

%

Memory growth rate

Rate of memory growth over time

growth

1.41

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2409.61

MiB/s

Disk write throughput

Disk write throughput in MiB/s

162.39

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/13

hosts

Mount Point /boot

0/1

hosts

Mount Point /efi-boot

0/11

hosts

Mount Point /etc/hostname

1/12

hosts

Mount Point /etc/hosts

1/12

hosts

Mount Point /etc/nomad.d/client-meta.json

1/5

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/12

hosts

Mount Point /nix/store

0/13

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.28

5 min

6.17

15 min

6.21

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

72.41

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

29.9694

%

SD

28.7011

%

<

p95

81.49

%

60 second average

29.7404

%

300 second average

28.1123

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.2091

%

SD

0.8716

%

<

p95

1.84

%

60 second average

0.2162

%

300 second average

0.2375

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.1651

%

SD

0.7888

%

<

p95

0.89

%

60 second average

0.1702

%

300 second average

0.1915

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

14.8143

%

SD

19.7303

%

<

p95

54.25

%

60 second average

15.0387

%

300 second average

15.4892

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

7.5816

%

SD

13.1815

%

<

p95

38.16

%

60 second average

7.8507

%

300 second average

8.2829

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.02

%

<

mean

13.13

%

SD

20.01

%

<

p95

52.09

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

21.65

KiB

<

mean

364.28

KiB

SD

144.93

KiB

<

p95

538.06

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

30.85

threads

SD

14.17

threads

<

p95

48

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

61.98

file descriptors

SD

22.11

file descriptors

<

p95

83

file descriptors

Zero-Arc Create Data

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 27 Aug 2026 10:03:18 UTC

Duration

900s

Peer count

30

Peer count at end

27

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data--high_zero_33056872755_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

49.91

/s

Max

Highest per-partition mean rate

max

94.55

/s

Min

Lowest per-partition mean rate

min

21.61

/s

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

13.625062

s

SD

0.26689

s

Max

Highest per-partition mean latency

max mean

13.625062

s

Min

Lowest per-partition mean latency

min mean

13.625062

s

Not enough time data to show a trend.

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

0

/s

Max

Highest per-partition mean rate

max

0

/s

Min

Lowest per-partition mean rate

min

0

/s

Not enough time data to show a trend.

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0.0009

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

116309

Agents affected

26 / 26

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

4473

Max

Highest count in any single partition

8565

Min

Lowest count in any single partition

1814

Received count

Total number of entries received by full-arc nodes, by agent.

Total

201

Agents affected

1 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

101

Max

Highest count in any single partition

201

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

1

<

mean

18.95

SD

9.67

<

p95

27

zero-arc

p5

4

<

mean

12.69

SD

5.64

<

p95

22

Error count

The number of errors accumulated across all nodes.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.003007

s

SD

0.001538

s

p50

0.000202

s

<

p95

0.003514

s

<

p99

0.00355

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 600.013437789s

Not enough time data to show a trend.

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

266006848

over 875.982407711s

mean rate

3.57032159548e+09

/s

std

2.7081640198e+10

/s

p5

1.244303473e+07

/s

p95

4.94403790548e+09

/s

peak

5.19576690584e+11

/s

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

7300280722

over 883.237772591s

mean rate

9.7341587865e+08

/s

std

5.9820395523e+08

/s

p5

2.893110831e+07

/s

p95

1.99825380902e+09

/s

peak

2.15196618269e+09

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

36307152

over 886.782160417s

mean rate

7.9546229746e+08

/s

std

1.14091121019e+10

/s

p5

593155.84

/s

p95

8.9719792809e+08

/s

peak

3.59765037926e+11

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.009351

s

SD

0.008582

s

p50

0.008353

s

<

p95

0.033575

s

<

p99

0.040994

s

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.037821

s

SD

0.186831

s

p50

0.002758

s

<

p95

0.451424

s

<

p99

0.456887

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008572

s

SD

0.00802

s

p50

0.007588

s

<

p95

0.031578

s

<

p99

0.037415

s

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.037303

s

SD

0.18642

s

p50

0.002275

s

<

p95

0.45004

s

<

p99

0.455497

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001706

s

SD

0.001481

s

p50

0.001516

s

<

p95

0.006062

s

<

p99

0.006321

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000611

s

SD

0.000697

s

p50

0.0005

s

<

p95

0.002549

s

<

p99

0.003326

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000593

s

SD

0.000693

s

p50

0.00051

s

<

p95

0.002559

s

<

p99

0.003379

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.003226

s

SD

0.000299

s

p50

0.003247

s

<

p95

0.003482

s

<

p99

0.003513

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001473

s

SD

0.006986

s

p50

0.000223

s

<

p95

0.017788

s

<

p99

0.018011

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002666

s

SD

0.002563

s

p50

0.002409

s

<

p95

0.010015

s

<

p99

0.011445

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000199

s

SD

0.000339

s

p50

0.000169

s

<

p95

0.00076

s

<

p99

0.002045

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

45.36

s

<

mean

441.36

s

SD

254.8

s

<

p95

837.95

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 886.78215173s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0.1489

s

SD

1.591547

s

p50

0

s

<

p95

0.417047

s

<

p99

8.585021

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.114037

s

SD

0.082867

s

p50

0.001438

s

<

p95

0.08817

s

<

p99

0.59692

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.011385

s

SD

0.281327

s

p50

0.001697

s

<

p95

1.008834

s

<

p99

1.393936

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.130652

s

SD

0.066687

s

p50

0.116044

s

<

p95

0.205907

s

<

p99

0.343238

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

8.822609

s

SD

5.568274

s

p50

0.003106

s

<

p95

14.676426

s

<

p99

43.867857

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

3.96936

s

SD

7.883805

s

p50

0.001416

s

<

p95

0.135895

s

<

p99

48.096142

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.005237

s

SD

0.003278

s

p50

0.005547

s

<

p95

0.010674

s

<

p99

0.015684

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.009538

s

SD

0.012777

s

p50

0.002659

s

<

p95

0.031716

s

<

p99

0.069551

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.00045

s

SD

0.000522

s

p50

0.000381

s

<

p95

0.002045

s

<

p99

0.002216

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.982295

s

SD

26.176232

s

p50

0.215528

s

<

p95

60.0007

s

≤

p99

60.0007

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.182822

s

SD

0.016774

s

p50

0.216305

s

≤

p95

0.216305

s

≤

p99

0.216305

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

66.93

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

14.29

KiB/s

<

mean

215.08

KiB/s

SD

570.80

KiB/s

<

p95

709.49

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

41.64

KiB/s

<

mean

192.96

KiB/s

SD

188.19

KiB/s

<

p95

419.45

KiB/s

Total bytes received

Total bytes received on primary network interface

count

5.07

GiB

mean

5.83

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

4.55

GiB

mean

5.23

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 99.0%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/11 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 79% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

34.34

%

<

mean

81.49

%

SD

20.24

%

<

p95

98.7

%

System

14.56

%

CPU percentiles

CPU usage percentiles

p50

87.91

%

p95

98.70

%

p99

98.95

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

28

hosts

mean time

626.63

s

Memory used percentage

Percentage of memory used

p5

10.69

%

<

mean

24.64

%

SD

10.41

%

<

p95

45.84

%

Memory available percentage

Percentage of available memory

p5

54.17

%

<

mean

75.36

%

SD

10.41

%

<

p95

89.33

%

Max host memory used

Maximum memory usage percentage across all hosts

max

50.54

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.47

%

Memory growth rate

Rate of memory growth over time

growth

0.95

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2.34

MiB/s

Disk write throughput

Disk write throughput in MiB/s

188.33

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/10

hosts

Mount Point /boot

0/1

hosts

Mount Point /efi-boot

0/9

hosts

Mount Point /etc/hostname

1/11

hosts

Mount Point /etc/hosts

1/11

hosts

Mount Point /etc/nomad.d/client-meta.json

1/8

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/11

hosts

Mount Point /nix/store

0/10

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

8.50

5 min

7.51

15 min

6.22

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

78.57

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.82

%

<

mean

45.0832

%

SD

32.2116

%

<

p95

97.2

%

60 second average

45.3350

%

300 second average

40.5678

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

10.5251

%

SD

16.2905

%

<

p95

44.28

%

60 second average

10.4627

%

300 second average

9.0012

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.9048

%

SD

8.3491

%

<

p95

20.69

%

60 second average

3.7569

%

300 second average

3.1051

%

Holochain process CPU usage

CPU usage by Holochain process

p5

2.17

%

<

mean

30.72

%

SD

16.08

%

<

p95

52.13

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

24.20

KiB

<

mean

320.00

KiB

SD

152.85

KiB

<

p95

579.75

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

33.58

threads

SD

14.21

threads

<

p95

51

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

71.4

file descriptors

SD

23.42

file descriptors

<

p95

84

file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 27 Aug 2026 10:18:56 UTC

Duration

900s

Peer count

30

Peer count at end

28

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data_validated--high_zero_33056872755_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

48.12

/s

Max

Highest per-partition mean rate

max

93.44

/s

Min

Lowest per-partition mean rate

min

29.74

/s

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

166.331763

s

SD

76.678578

s

Max

Highest per-partition mean latency

max mean

318.447298

s

Min

Lowest per-partition mean latency

min mean

14.216227

s

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

236.65

/s

Max

Highest per-partition mean rate

max

473.29

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0.0146

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

115227

Agents affected

27 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

4268

Max

Highest count in any single partition

8309

Min

Lowest count in any single partition

2532

Received count

Total number of entries received by full-arc nodes, by agent.

Total

3369

Agents affected

1 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

1685

Max

Highest count in any single partition

3369

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

6

<

mean

21.37

SD

7.38

<

p95

28

zero-arc

p5

2

<

mean

8.66

SD

4.73

<

p95

17

Error count

The number of errors accumulated across all nodes.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.002802

s

SD

0.004144

s

p50

0.007277

s

<

p95

0.015633

s

<

p99

0.018717

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

7864

over 770.019239219s

mean rate

514.51

/s

std

616.04

/s

p5

3.94

/s

p95

1982.37

/s

peak

2241.72

/s

WASM usage: 'timed_and_validated::create_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

422676506

over 854.225486173s

mean rate

1.10464771937e+09

/s

std

2.97142240396e+09

/s

p5

1.817790168e+07

/s

p95

6.08886587903e+09

/s

peak

3.08759833505e+10

/s

WASM usage: 'timed_and_validated::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

22191934179

over 870.037857828s

mean rate

3.12225339532e+09

/s

std

1.93415628698e+09

/s

p5

7.634735696e+07

/s

p95

6.01147098621e+09

/s

peak

6.30026413114e+09

/s

WASM usage: 'timed_and_validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

231937328

over 887.23728217s

mean rate

5.6775219244e+08

/s

std

1.52597166827e+09

/s

p5

1.21452402e+06

/s

p95

2.5483889039e+09

/s

peak

1.68955333245e+10

/s

WASM usage: 'timed_and_validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

1940571946

over 887.23728217s

mean rate

4.90792632216e+09

/s

std

1.29204803892e+10

/s

p5

2.504375591e+07

/s

p95

2.24341224812e+10

/s

peak

1.41060883769e+11

/s

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.006459

s

SD

0.002263

s

p50

0.006042

s

<

p95

0.011026

s

<

p99

0.014596

s

Zome call duration: 'timed_and_validated::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.088635

s

SD

0.058614

s

p50

0.039297

s

<

p95

0.195676

s

<

p99

0.214418

s

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005751

s

SD

0.002117

s

p50

0.005359

s

<

p95

0.010112

s

<

p99

0.013428

s

WASM call duration: 'timed_and_validated::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.086631

s

SD

0.059709

s

p50

0.03482

s

<

p95

0.194748

s

<

p99

0.21348

s

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001222

s

SD

0.000891

s

p50

0.001081

s

<

p95

0.002556

s

<

p99

0.005761

s

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005372

s

SD

0.003819

s

p50

0.004234

s

<

p95

0.011065

s

<

p99

0.023057

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000351

s

SD

0.00014

s

p50

0.000319

s

<

p95

0.000607

s

<

p99

0.000874

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000346

s

SD

0.000132

s

p50

0.000332

s

<

p95

0.000597

s

<

p99

0.000849

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.001037

s

SD

0.002939

s

p50

0.001328

s

<

p95

0.008697

s

<

p99

0.008979

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.003709

s

SD

0.001498

s

p50

0.003274

s

<

p95

0.006254

s

<

p99

0.006733

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.008976

s

SD

0.004037

s

p50

0.007602

s

<

p95

0.015839

s

<

p99

0.01892

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001944

s

SD

0.001943

s

p50

0.001707

s

<

p95

0.003956

s

<

p99

0.013157

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000129

s

SD

6.7e-05

s

p50

0.000122

s

<

p95

0.000275

s

<

p99

0.000355

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

45.62

s

<

mean

438.75

s

SD

253.19

s

<

p95

835.23

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 887.237335815s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

25.214803

s

SD

26.836205

s

p50

0

s

<

p95

2.359036

s

<

p99

179.389386

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

3.725671

s

SD

1.450466

s

p50

0.001471

s

<

p95

2.562522

s

<

p99

8.100551

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

<

p99

1e-06

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.008309

s

SD

0.297046

s

p50

0.001125

s

<

p95

0.984213

s

<

p99

1.453927

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.096183

s

SD

0.30657

s

p50

0.085786

s

<

p95

0.1747

s

<

p99

2.474403

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

11.236018

s

SD

7.238613

s

p50

0.002713

s

<

p95

10.896753

s

<

p99

48.951216

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

3.356285

s

SD

4.26526

s

p50

0.001629

s

<

p95

0.023669

s

<

p99

26.506944

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002962

s

SD

0.001781

s

p50

0.003192

s

<

p95

0.006308

s

<

p99

0.007562

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.013301

s

SD

0.01675

s

p50

0.002015

s

<

p95

0.055244

s

<

p99

0.078078

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000283

s

SD

0.00067

s

p50

0.000271

s

<

p95

0.000666

s

<

p99

0.003952

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

6.111236

s

SD

29.815612

s

p50

0.314576

s

<

p95

60.007429

s

≤

p99

60.007429

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.089358

s

SD

0.036222

s

p50

0.013562

s

<

p95

0.095156

s

<

p99

0.097865

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

47.71

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

13.67

KiB/s

<

mean

493.55

KiB/s

SD

1010.86

KiB/s

<

p95

2.13

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

30.82

KiB/s

<

mean

312.70

KiB/s

SD

493.74

KiB/s

<

p95

889.18

KiB/s

Total bytes received

Total bytes received on primary network interface

count

11.66

GiB

mean

13.41

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.39

GiB

mean

8.50

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.5%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.87 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/12 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 79% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

17.06

%

<

mean

63.78

%

SD

28.26

%

<

p95

97.8

%

System

16.07

%

CPU percentiles

CPU usage percentiles

p50

69.94

%

p95

97.80

%

p99

98.45

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

21

hosts

mean time

509.18

s

Memory used percentage

Percentage of memory used

p5

11.98

%

<

mean

28.47

%

SD

11.59

%

<

p95

51.64

%

Memory available percentage

Percentage of available memory

p5

48.36

%

<

mean

71.53

%

SD

11.59

%

<

p95

88.02

%

Max host memory used

Maximum memory usage percentage across all hosts

max

52.41

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.86

%

Memory growth rate

Rate of memory growth over time

growth

1.87

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2450.30

MiB/s

Disk write throughput

Disk write throughput in MiB/s

181.95

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/13

hosts

Mount Point /boot

0/1

hosts

Mount Point /efi-boot

0/11

hosts

Mount Point /etc/hostname

1/12

hosts

Mount Point /etc/hosts

1/12

hosts

Mount Point /etc/nomad.d/client-meta.json

1/5

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/12

hosts

Mount Point /nix/store

0/13

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.12

5 min

6.09

15 min

6.18

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

79.31

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

29.1044

%

SD

28.1482

%

<

p95

81.35

%

60 second average

28.2180

%

300 second average

27.0039

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.2258

%

SD

0.9339

%

<

p95

1.88

%

60 second average

0.2370

%

300 second average

0.2506

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.1813

%

SD

0.8531

%

<

p95

0.89

%

60 second average

0.1936

%

300 second average

0.2050

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

15.1112

%

SD

19.52

%

<

p95

53.14

%

60 second average

15.2354

%

300 second average

15.7404

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

8.1178

%

SD

13.4493

%

<

p95

37.96

%

60 second average

8.4517

%

300 second average

8.5427

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.02

%

<

mean

14.89

%

SD

18.97

%

<

p95

51.08

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.81

KiB

<

mean

359.98

KiB

SD

146.01

KiB

<

p95

538.20

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

30.44

threads

SD

14.42

threads

<

p95

47

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

61.95

file descriptors

SD

22.66

file descriptors

<

p95

83

file descriptors

Zome Call, Single Value

↑ Back to index

Calls a zome function that returns a fixed value. This tests the maximum performance of zome calls when the zome function is not calling into Holochain other otherwise doing any work.

Started

Thu, 27 Aug 2026 09:20:54 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zome_call_single_value_33056872755_1

Call timing

The time taken for the call to complete.

mean

0.003222

s

SD

0.001607

s

p50

0.002665

s

<

p95

0.006157

s

<

p99

0.009042

s

Call rate

The number of zome calls completed per second.

mean

3064

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'return_single_value::get_value'

Metered usage count of the WASM ribosome for this zome function.

total

340705562

over 280.003327452s

mean rate

1.2167934e+06

/s

std

418508.42

/s

p5

751978.87

/s

p95

1.79509103e+06

/s

peak

1.89783937e+06

/s

Zome call duration: 'return_single_value::get_value'

Duration of this zome call as measured by Holochain internally.

mean

0.001362

s

SD

0.000129

s

p50

0.001496

s

<

p95

0.0016

s

≤

p99

0.0016

s

WASM call duration: 'return_single_value::get_value'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000853

s

SD

8.7e-05

s

p50

0.000941

s

<

p95

0.001014

s

≤

p99

0.001014

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.7e-05

s

SD

0

s

p50

8.7e-05

s

≤

p95

8.7e-05

s

≤

p99

8.7e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.23

s

<

mean

147.23

s

SD

83.67

s

<

p95

277.24

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.003335811s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001259

s

SD

0

s

p50

0.001259

s

≤

p95

0.001259

s

≤

p99

0.001259

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001177

s

SD

0

s

p50

0.001177

s

≤

p95

0.001177

s

≤

p99

0.001177

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004597

s

SD

0.000916

s

p50

0.004862

s

<

p95

0.006391

s

≤

p99

0.006391

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.00263

s

SD

0

s

p50

0.00263

s

≤

p95

0.00263

s

≤

p99

0.00263

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.00075

s

SD

0

s

p50

0.00075

s

≤

p95

0.00075

s

≤

p99

0.00075

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

7e-05

s

SD

1.4e-05

s

p50

8.2e-05

s

<

p95

9.6e-05

s

<

p99

9.7e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001051

s

SD

6.3e-05

s

p50

0.001132

s

≤

p95

0.001132

s

≤

p99

0.001132

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000637

s

SD

0

s

p50

0.000637

s

≤

p95

0.000637

s

≤

p99

0.000637

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

48.84

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

5.04

KiB/s

<

mean

234.04

KiB/s

SD

1.12

MiB/s

≤

p95

71.74

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

7.38

KiB/s

<

mean

18.35

KiB/s

SD

19.01

KiB/s

<

p95

30.24

KiB/s

Total bytes received

Total bytes received on primary network interface

count

66.28

MiB

mean

234.04

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.20

MiB

mean

18.35

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 93.3%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

32.34

%

<

mean

65.78

%

SD

24.04

%

<

p95

89.9

%

System

16.94

%

CPU percentiles

CPU usage percentiles

p50

74.27

%

p95

89.90

%

p99

93.26

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

124.14

s

Memory used percentage

Percentage of memory used

p5

5.82

%

<

mean

8.2

%

SD

1.65

%

<

p95

10.72

%

Memory available percentage

Percentage of available memory

p5

89.28

%

<

mean

91.8

%

SD

1.65

%

<

p95

94.18

%

Max host memory used

Maximum memory usage percentage across all hosts

max

8.20

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-4.41

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

3.89

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.85

5 min

3.73

15 min

1.87

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

1.04

%

<

mean

24.957

%

SD

17.5413

%

<

p95

45.48

%

60 second average

29.9543

%

300 second average

23.1897

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.77

%

<

mean

24.01

%

SD

5.66

%

<

p95

37.93

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

21.95

KiB

<

mean

245.04

KiB

SD

247.43

KiB

<

p95

670.13

KiB

Holochain process threads

Number of threads in Holochain process

p5

8

threads

<

mean

32.53

threads

SD

44.82

threads

<

p95

109

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

73.82

file descriptors

SD

96.81

file descriptors

<

p95

241

file descriptors

App Install

App Install

DHT Sync Lag

First Call

Full-Arc Create (Validated) / Zero-Arc Read

Local Signals

Mixed Arc get_agent_activity

Mixed Arc must_get_agent_activity

Remote Call Rate

Remote Signals

Single-write, many-read

Validation Receipts

Write/get_agent_activity

Write/get_agent_activity with volatile conductors

Write/Query

Write/Read

Write validated

Write Validated must_get_agent_activity

Zero-Arc Create and Read

Zero-Arc Create Data

Zero-arc Create Data (Validated)

Zome Call, Single Value

Mixed Arc `get_agent_activity`

Mixed Arc `must_get_agent_activity`

Write/`get_agent_activity`

Write/`get_agent_activity` with volatile conductors

Write Validated `must_get_agent_activity`