First Call

↑ Back to index

This scenario installs a simple app which implements the init callback, and measures the time elapsed between start of install and calling init. It will uninstall the app it installed so that it can re-install it on the next iteration. This is required to re-run the initialisation callback.

Started

Wed, 26 Aug 2026 16:41:35 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

first_call_32989406575_1

Zome call timing

The time elapsed between start of install and calling init.

mean

0.011585

s

SD

0.003619

s

p50

0.011639

s

<

p95

0.015983

s

<

p99

0.023058

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.00026964s

Not enough time data to show a trend.

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.00026964s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

246512

over 10.00026964s

mean rate

24650.54

/s

std

0

/s

p5

24650.54

/s

p95

24650.54

/s

peak

24650.54

/s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004671

s

SD

0.000168

s

p50

0.004708

s

≤

p95

0.004708

s

≤

p99

0.004708

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003937

s

SD

0.001065

s

p50

0.003691

s

<

p95

0.005747

s

<

p99

0.006436

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001433

s

SD

0.000424

s

p50

0.00128

s

<

p95

0.002983

s

≤

p99

0.002983

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001135

s

SD

2.2e-05

s

p50

0.00114

s

≤

p95

0.00114

s

≤

p99

0.00114

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.00032

s

SD

1.8e-05

s

p50

0.000339

s

<

p95

0.000343

s

≤

p99

0.000343

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002028

s

SD

6.4e-05

s

p50

0.002042

s

≤

p95

0.002042

s

≤

p99

0.002042

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.7e-05

s

SD

3.4e-05

s

p50

7.4e-05

s

<

p95

0.000173

s

<

p99

0.000227

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

6.94

s

<

mean

11.94

s

SD

5

s

<

p95

16.94

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 10.000083449s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000935

s

SD

0.000135

s

p50

0.001175

s

≤

p95

0.001175

s

≤

p99

0.001175

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001262

s

SD

3.9e-05

s

p50

0.001332

s

≤

p95

0.001332

s

≤

p99

0.001332

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004056

s

SD

0.000134

s

p50

0.004295

s

≤

p95

0.004295

s

≤

p99

0.004295

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002443

s

SD

0.000275

s

p50

0.002503

s

≤

p95

0.002503

s

≤

p99

0.002503

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001484

s

SD

0.000389

s

p50

0.001568

s

≤

p95

0.001568

s

≤

p99

0.001568

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

3.2e-05

s

SD

2e-06

s

p50

3.5e-05

s

≤

p95

3.5e-05

s

≤

p99

3.5e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001012

s

SD

0.000306

s

p50

0.00128

s

<

p95

0.001569

s

≤

p99

0.001569

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000259

s

SD

8e-06

s

p50

0.000273

s

≤

p95

0.000273

s

≤

p99

0.000273

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

49.56

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

60.09

KiB/s

<

mean

243.11

KiB/s

SD

183.02

KiB/s

<

p95

426.13

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

17.25

KiB/s

<

mean

24.41

KiB/s

SD

7.16

KiB/s

<

p95

31.57

KiB/s

Total bytes received

Total bytes received on primary network interface

count

4.75

MiB

mean

243.11

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

488.22

KiB

mean

24.41

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 6.93 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

53.71

%

<

mean

62.93

%

SD

9.23

%

<

p95

72.16

%

System

13.38

%

CPU percentiles

CPU usage percentiles

p50

72.16

%

p95

72.16

%

p99

72.16

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

18.38

%

<

mean

25.29

%

SD

7.7

%

<

p95

36.04

%

Memory available percentage

Percentage of available memory

p5

63.96

%

<

mean

74.71

%

SD

7.7

%

<

p95

81.62

%

Max host memory used

Maximum memory usage percentage across all hosts

max

25.29

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

6.93

MiB/s

Disk read throughput

Disk read throughput in MiB/s

286.41

MiB/s

Disk write throughput

Disk write throughput in MiB/s

7.26

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.23

5 min

1.11

15 min

1.10

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

6.46

%

<

mean

9.4367

%

SD

2.8235

%

<

p95

13.23

%

60 second average

4.9667

%

300 second average

3.6133

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

2.22

%

<

mean

3.2767

%

SD

1.0338

%

<

p95

4.68

%

60 second average

4.5900

%

300 second average

4.9300

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

1.72

%

<

mean

3.0333

%

SD

1.2258

%

<

p95

4.67

%

60 second average

4.4900

%

300 second average

4.8767

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

49.51

%

<

mean

57.8767

%

SD

8.2772

%

<

p95

69.15

%

60 second average

67.3233

%

300 second average

71.4667

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

31.09

%

<

mean

43.55

%

SD

14.3678

%

<

p95

63.68

%

60 second average

62.5733

%

300 second average

69.9500

%

Holochain process CPU usage

CPU usage by Holochain process

p5

46.93

%

<

mean

56.5

%

SD

9.57

%

<

p95

66.07

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.83

KiB

<

mean

583.64

KiB

SD

631.34

KiB

<

p95

1.61

MiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

101.75

threads

SD

131.09

threads

<

p95

323

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

225.5

file descriptors

SD

283.58

file descriptors

<

p95

701

file descriptors

Local Signals

↑ Back to index

This scenario is for testing the speed and reliability of local signals. It uses a zome with a function that will emit 10,000 signals. The scenario measures how long it takes to send the signals and how many have been received by the time the zome call ends.

Started

Wed, 26 Aug 2026 16:43:49 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

local_signals_32989406575_1

Send timing

The time taken to emit a batch of 10,000 signals.

mean

1.335495

s

SD

0.628594

s

p50

1.823887

s

<

p95

1.979215

s

≤

p99

1.979215

s

Receive timing

The time taken to receive the completed batch, to the nearest 250ms.

mean

27.852231

s

SD

6.446219

s

p50

30.000951

s

<

p95

30.001436

s

≤

p99

30.001436

s

Success ratio

The proportion of signals successfully received in a batch.

min

0.4659999907016754

<

mean

0.5866

SD

0.0822

<

max

0.7105000019073486

Holochain MetricsClick to toggle visibility.

WASM usage: 'signal::emit_10k_signals'

Metered usage count of the WASM ribosome for this zome function.

total

277609437

over 280.004108253s

mean rate

991446.87

/s

std

1.4405389e+06

/s

p5

0

/s

p95

3.08452861e+06

/s

peak

3.0845318e+06

/s

Zome call duration: 'signal::emit_10k_signals'

Duration of this zome call as measured by Holochain internally.

mean

0.223641

s

SD

0.037939

s

p50

0.226515

s

<

p95

0.271914

s

≤

p99

0.271914

s

WASM call duration: 'signal::emit_10k_signals'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.222782

s

SD

0.037942

s

p50

0.225672

s

<

p95

0.27103

s

≤

p99

0.27103

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

1.5e-05

s

SD

2e-06

s

p50

1.5e-05

s

<

p95

1.8e-05

s

≤

p99

1.8e-05

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

90000

over 280.004117905s

mean rate

321.42

/s

std

460.21

/s

p5

0

/s

p95

999.99

/s

peak

999.99

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000342

s

SD

0

s

p50

0.000342

s

≤

p95

0.000342

s

≤

p99

0.000342

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.93

s

<

mean

146.93

s

SD

83.67

s

<

p95

276.93

s

Dropped WebSocket signals

Count of signals dropped from the app WebSocket due to channel overload.

total

4216

over 280.00403839s

mean rate

15.06

/s

std

25.93

/s

p5

0

/s

p95

67

/s

peak

82.9

/s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.004049877s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000995

s

SD

0

s

p50

0.000995

s

≤

p95

0.000995

s

≤

p99

0.000995

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001991

s

SD

0

s

p50

0.001991

s

≤

p95

0.001991

s

≤

p99

0.001991

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004988

s

SD

0.000944

s

p50

0.005281

s

<

p95

0.006823

s

≤

p99

0.006823

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.001865

s

SD

0

s

p50

0.001865

s

≤

p95

0.001865

s

≤

p99

0.001865

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002903

s

SD

0

s

p50

0.002903

s

≤

p95

0.002903

s

≤

p99

0.002903

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000113

s

SD

2.3e-05

s

p50

0.000113

s

<

p95

0.000129

s

<

p99

0.00013

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001367

s

SD

0.000161

s

p50

0.001586

s

≤

p95

0.001586

s

≤

p99

0.001586

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000331

s

SD

0

s

p50

0.000331

s

≤

p95

0.000331

s

≤

p99

0.000331

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

8.75

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

434

B/s

<

mean

282.54

KiB/s

SD

1.00

MiB/s

<

p95

2.77

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

143

B/s

<

mean

13.51

KiB/s

SD

17.30

KiB/s

<

p95

42.13

KiB/s

Total bytes received

Total bytes received on primary network interface

count

77.26

MiB

mean

282.54

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

3.69

MiB

mean

13.51

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 2.81 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

9.42

%

<

mean

21.97

%

SD

12.95

%

<

p95

34.29

%

System

13.22

%

CPU percentiles

CPU usage percentiles

p50

21.10

%

p95

34.29

%

p99

78.94

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

17.21

%

<

mean

22.94

%

SD

3.91

%

<

p95

28.34

%

Memory available percentage

Percentage of available memory

p5

71.66

%

<

mean

77.06

%

SD

3.91

%

<

p95

82.79

%

Max host memory used

Maximum memory usage percentage across all hosts

max

22.94

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

2.81

MiB/s

Disk read throughput

Disk read throughput in MiB/s

278.40

MiB/s

Disk write throughput

Disk write throughput in MiB/s

2.21

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.76

5 min

1.45

15 min

1.24

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

3.49

%

<

mean

9.1097

%

SD

4.057

%

<

p95

16.35

%

60 second average

8.0493

%

300 second average

5.5831

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

2.66

%

<

mean

3.3976

%

SD

0.736

%

<

p95

4.83

%

60 second average

3.7824

%

300 second average

4.5738

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

2.41

%

<

mean

3.3062

%

SD

0.7776

%

<

p95

4.74

%

60 second average

3.6762

%

300 second average

4.4783

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

44.33

%

<

mean

54.3424

%

SD

8.4278

%

<

p95

70.08

%

60 second average

58.2290

%

300 second average

65.5034

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

33.38

%

<

mean

46.5386

%

SD

11.3766

%

<

p95

65.18

%

60 second average

51.5700

%

300 second average

61.4759

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.45

%

<

mean

6.14

%

SD

4.16

%

<

p95

10.34

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.75

KiB

<

mean

196.41

KiB

SD

68.34

KiB

<

p95

251.19

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

23.48

threads

SD

9.35

threads

<

p95

31

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

63.64

file descriptors

SD

22.88

file descriptors

<

p95

80

file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started

Wed, 26 Aug 2026 16:45:28 UTC

Duration

900s

Peer count

10

Peer count at end

9

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_call_rate_32989406575_1

Dispatch timing

The time between sending a remote call and the remote handler being invoked.

Mean

mean

0.071967

s

SD

0.140009

s

Max

Highest per-partition mean latency

max mean

0.447657

s

Min

Lowest per-partition mean latency

min mean

-0.005891

s

Round-trip timing

The total elapsed time to get a response to the client.

Mean

mean

0.126319

s

SD

0.110874

s

Max

Highest per-partition mean latency

max mean

0.165867

s

Min

Lowest per-partition mean latency

min mean

0.06079

s

Error count

The total number of errors accumulated during the run.

1

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_call::call_echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

185985645

over 884.49618553s

mean rate

5.986297803e+07

/s

std

1.149883871e+08

/s

p5

443370.32

/s

p95

3.6073968306e+08

/s

peak

5.9627491004e+08

/s

WASM usage: 'remote_call::echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

47157975

over 886.907885655s

mean rate

1.948650724e+07

/s

std

5.409458037e+07

/s

p5

8712.05

/s

p95

1.7232758133e+08

/s

peak

2.4626392053e+08

/s

WASM usage: 'remote_call::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 886.907885655s

Not enough time data to show a trend.

Zome call duration: 'remote_call::call_echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.126209

s

SD

0.154642

s

p50

0.158757

s

<

p95

0.223469

s

<

p99

0.56441

s

Zome call duration: 'remote_call::echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.002779

s

SD

0.001096

s

p50

0.002775

s

<

p95

0.004357

s

<

p99

0.004365

s

WASM call duration: 'remote_call::call_echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.126914

s

SD

0.154464

s

p50

0.157893

s

<

p95

0.224236

s

<

p99

0.563368

s

WASM call duration: 'remote_call::echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001792

s

SD

0.000699

s

p50

0.0019

s

<

p95

0.002985

s

<

p99

0.002992

s

WASM call duration: 'remote_call::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002517

s

SD

0.001148

s

p50

0.002681

s

<

p95

0.004291

s

≤

p99

0.004291

s

Host function call duration: '__hc__call_1'

Duration of this host function call invoked from within WASM.

mean

0.124904

s

SD

0.154539

s

p50

0.155906

s

<

p95

0.222033

s

<

p99

0.561472

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000468

s

SD

0.000265

s

p50

0.000466

s

<

p95

0.000955

s

≤

p99

0.000955

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

9e-06

s

SD

6e-06

s

p50

9e-06

s

<

p95

1.5e-05

s

<

p99

2.7e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000224

s

SD

0.000257

s

p50

0.000223

s

<

p95

0.000645

s

<

p99

0.001331

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000267

s

SD

0.000342

s

p50

0.000176

s

<

p95

0.00129

s

≤

p99

0.00129

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.79

s

<

mean

445.56

s

SD

256.09

s

<

p95

846.62

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

117

over 886.907982095s

mean rate

16.3

/s

std

128.28

/s

p5

0

/s

p95

7.71

/s

peak

1304.65

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

2.401072

s

SD

2.331256

s

p50

1.644974

s

<

p95

8.631886

s

<

p99

9.112653

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001933

s

SD

0.001379

s

p50

0.002061

s

<

p95

0.005202

s

<

p99

0.005366

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.075543

s

SD

0.03029

s

p50

0.075695

s

<

p95

0.139504

s

≤

p99

0.139504

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.005741

s

SD

0.003473

s

p50

0.006265

s

<

p95

0.01247

s

<

p99

0.015312

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.256792

s

SD

0.419996

s

p50

0.196476

s

<

p95

1.282237

s

<

p99

1.707421

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.164306

s

SD

0.11577

s

p50

0.137226

s

<

p95

0.361921

s

<

p99

0.439415

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000125

s

SD

5.6e-05

s

p50

0.000123

s

<

p95

0.000223

s

<

p99

0.000247

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001037

s

SD

0.000917

s

p50

0.001149

s

<

p95

0.004421

s

≤

p99

0.004421

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000363

s

SD

0.000171

s

p50

0.000384

s

<

p95

0.000646

s

<

p99

0.000651

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.074692

s

SD

0.06377

s

p50

0.053933

s

<

p95

0.237657

s

<

p99

0.260302

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.137654

s

SD

0.059714

s

p50

0.151652

s

<

p95

0.214373

s

<

p99

0.219491

s

P2P request roundtrip: 'call_remote'

Time spent sending a call_remote request and awaiting its response.

mean

0.124433

s

SD

16.69352

s

p50

0.157662

s

<

p95

60.000594

s

≤

p99

60.000594

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

6.47

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

10.69

KiB/s

<

mean

137.80

KiB/s

SD

844.55

KiB/s

≤

p95

59.30

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

8.92

KiB/s

<

mean

16.39

KiB/s

SD

21.66

KiB/s

<

p95

48.96

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.05

GiB

mean

1.21

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

127.59

MiB

mean

146.80

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 3.85 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/4 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (79.2% swap used)

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

4.29

%

<

mean

11.79

%

SD

8.78

%

<

p95

22.39

%

System

5.32

%

CPU percentiles

CPU usage percentiles

p50

10.75

%

p95

22.39

%

p99

47.45

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

2

hosts

mean time

15.34

s

Memory used percentage

Percentage of memory used

p5

3.71

%

<

mean

21.3

%

SD

13.57

%

<

p95

44

%

Memory available percentage

Percentage of available memory

p5

56

%

<

mean

78.7

%

SD

13.57

%

<

p95

96.29

%

Max host memory used

Maximum memory usage percentage across all hosts

max

44.85

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

79.23

%

Memory growth rate

Rate of memory growth over time

growth

3.85

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2512.76

MiB/s

Disk write throughput

Disk write throughput in MiB/s

60.57

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

1/4

hosts

Mount Point /etc/hosts

1/4

hosts

Mount Point /etc/nomad.d/client-meta.json

1/7

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/4

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.07

5 min

0.97

15 min

0.80

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

2.9052

%

SD

2.9764

%

<

p95

7.72

%

60 second average

2.8187

%

300 second average

2.4524

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.8944

%

SD

1.7222

%

<

p95

4.89

%

60 second average

0.9063

%

300 second average

0.9670

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.8076

%

SD

1.5395

%

<

p95

4.29

%

60 second average

0.8192

%

300 second average

0.8970

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

8.4597

%

SD

16.4167

%

<

p95

48.16

%

60 second average

8.5314

%

300 second average

9.0544

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

5.5396

%

SD

12.7533

%

<

p95

40.59

%

60 second average

5.6706

%

300 second average

6.3562

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

4.26

%

SD

4.53

%

<

p95

10.05

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

200.78

KiB

<

mean

235.75

KiB

SD

63.56

KiB

<

p95

405.91

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

31.36

threads

SD

24.88

threads

<

p95

97

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

60.22

file descriptors

SD

30.77

file descriptors

<

p95

91

file descriptors