Make all atomic operations explicit

Allows mkdocs to discover broken links when the pointed document has
moved.

doc/dev/channels.md

@@ -65,5 +65,5 @@ to write the duplicated value with no error.
 A unique function can be set to each channel which will be called once a channel
 becomes dirty with `chan_set_dirty_cb()`. This callback will be called before
 `chan_set()`, `chan_push()` or `chan_pop()` returns. The [patch
-bay](../patchbay) uses this callback to detect when a channel is modified an run
+bay](patchbay.md) uses this callback to detect when a channel is modified an run
 other callbacks.

doc/dev/model.md

@@ -22,7 +22,7 @@ If the model is not enabled, no other function will be called.
 
 The create function is called for each enabled model to allow them to allocate
 all the required structures to perform the emulation using the
-[extend](../extend) mechanism. All the required channels must be created and
+[extend](extend.md) mechanism. All the required channels must be created and
 registered in the patch bay in this function, so other models can found them in
 the next stage.
 

doc/dev/mux.md

@@ -1,6 +1,6 @@
 # Mux
 
-The emulator provides a mechanism to interconnect [channels](../channels) in a
+The emulator provides a mechanism to interconnect [channels](channels.md) in a
 similar way as an [analog
 multiplexer](https://en.wikipedia.org/wiki/Multiplexer) by using the `mux`
 module.
@@ -19,7 +19,7 @@ selected. This allows a multiplexer to act as a filter too.
 
 The typical use of multiplexers is to implement the tracking modes of channels.
 As an example, the following diagram shows two multiplexers used to implement
-the subsystem view of [Nanos6](../nanos6):
+the subsystem view of [Nanos6](../user/emulation/nanos6.md):
 
 ![Mux example](fig/mux.svg)
 
@@ -51,5 +51,5 @@ Multiplexers allow models to interact with each other in a controlled way. In
 the example, the blue channel (*nanos6.thread0.subsystem*) is directly modified by
 the Nanos6 model when a new event is received. While the red channels are
 controlled by the ovni model.  The rest of the channels are automatically updated
-in the propagation phase of the [bay](../patchbay) allowing the ovni model to
+in the propagation phase of the [bay](patchbay.md) allowing the ovni model to
 modify the Nanos6 Paraver view of the subsystems.

doc/dev/paraver.md

@@ -16,7 +16,7 @@ A channel can be connected to each row in a trace with `prv_register()`, so the
 new values of the channel get written in the trace. Only null and int64 data
 values are supported for now.
 
-The emission phase is controlled by the [patch bay](../patchbay) and runs all
+The emission phase is controlled by the [patch bay](patchbay.md) and runs all
 the emit callbacks at once for all dirty channels.
 
 ## Duplicate values

doc/dev/patchbay.md

@@ -1,6 +1,6 @@
 # Patch bay
 
-The patch bay (or simply bay) allows [channels](../channels/) to be registered
+The patch bay (or simply bay) allows [channels](channels.md) to be registered
 with their name so they are visible to all parts of the emulator and provides a
 way to run callbacks when the channels update their values.
 

doc/index.md

@@ -13,8 +13,9 @@ The ovni project implements a fast instrumentation library that records
 small events (starting at 12 bytes) during the execution of programs to
 later investigate how the execution happened.
 
-The instrumentation process is split in two stages: [runtime](runtime)
+The instrumentation process is split in two stages:
-tracing and [emulation](emulation/).
+[runtime](user/runtime/index.md)
+tracing and [emulation](user/emulation/index.md).
 
 During runtime, very short binary events are stored on disk which
 describe what is happening. Once the execution finishes, the events are

doc/user/concepts.md

@@ -1,31 +0,0 @@
-# Overview
-
-The objective of the ovni project is to provide insight into what
-happened at execution of a program.
-
-![Instrumentation process](fig/instrumentation.svg)
-
-The key pieces of software involved are instrumented so they emit events
-during the execution which allow the reconstruction of the execution
-later on.
-
-During the execution phase, the information gathered in the events is
-kept very short and simple, so the overhead is kept at minimum to avoid
-disturbing the execution process. Here is an example of a single event
-emitted during the execution phase, informing the current thread to
-finish the execution:
-
-	00 4f 48 65 52 c0 27 b4 d3 ec 01 00
-
-During the emulation phase, the events are read and processed in the
-emulator, reconstructing the execution. State transitions are recorded
-in a Paraver trace. Here is an example of the same thread ceasing the
-execution:
-
-	2:0:1:1:1:50105669:1:0
-
-Finally, loading the trace in the Paraver program, we can generate a
-timeline visualization of the state change. Here is the example for the
-same state transition of the thread stopping the execution:
-
-![Visualization](fig/visualization.png)

doc/user/concepts/part-model.md

@@ -0,0 +1,62 @@
+# Part model
+
+Ovni has a model to represent the hardware components as well as the software
+concepts like threads or processes. Each concept is considered to be a *part*.
+Here is an example diagram depicting the part hierarchy:
+
+![Part model](part-model.svg)
+
+Notice how a loom can restrict the CPUs of the node to its child processes.
+
+## Software parts
+
+These are not physical parts, but they abstract common concepts.
+
+### Thread
+
+A thread in ovni is directly mapped to a [POSIX
+thread](https://en.wikipedia.org/wiki/Pthreads) and they are identified by a
+`TID` which must be unique in a [node](#node). Threads in ovni have [a model with
+an internal state](../emulation/ovni.md/#thread_model) that tries to tracks the
+state of the real thread.
+
+### Process
+
+A process is directly mapped to a UNIX
+[process](https://en.wikipedia.org/wiki/Process_(computing)) and they are
+identified by a `PID` number which must be unique in a [node](#node).
+
+### Loom
+
+A loom has no direct mapping to a usual concept. It consists of a set of
+[CPUs](#cpu) from the same node and a set of processes that can *only run in
+those CPUs*. Each CPUs must belong to one and only one loom. It is often used
+to group CPUs that belong to the same process when running workloads with
+multiple processes (like with MPI).
+
+Each loom has a virtual CPU which collects running threads that are not
+exclusively assigned to a physical CPU, so we cannot determine on which CPU they
+are running.
+
+## Hardware parts
+
+These parts have a physical object assigned.
+
+### CPU
+
+A CPU is a hardware thread that can execute at most one thread at a time. Each
+CPU must have a physical ID that is unique in a node. In ovni there is also a
+virtual CPU, which simply is used to collect threads that are not tied to an
+specific physical CPU, so it cannot be easily determined where they are running.
+
+### Node
+
+A *node* refers to a compute node, often a physical machine with memory and
+network which may contain one or more
+[sockets](https://en.wikipedia.org/wiki/CPU_socket), where each socket has one
+or more CPUs.
+
+### System
+
+A system represents the complete set of hardware parts and software parts that
+are known to ovni in a given trace.

doc/user/concepts/part-model.svg

@@ -0,0 +1,516 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   width="306.91663mm"
+   height="127.00001mm"
+   viewBox="0 0 306.91663 127.00001"
+   version="1.1"
+   id="svg1"
+   inkscape:version="1.3.2 (091e20ef0f, 2023-11-25, custom)"
+   sodipodi:docname="part-model.svg"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg">
+  <sodipodi:namedview
+     id="namedview1"
+     pagecolor="#ffffff"
+     bordercolor="#999999"
+     borderopacity="1"
+     inkscape:showpageshadow="0"
+     inkscape:pageopacity="0"
+     inkscape:pagecheckerboard="0"
+     inkscape:deskcolor="#d1d1d1"
+     inkscape:document-units="mm"
+     showgrid="false"
+     inkscape:zoom="0.73049944"
+     inkscape:cx="629.70616"
+     inkscape:cy="280.62992"
+     inkscape:window-width="1920"
+     inkscape:window-height="1031"
+     inkscape:window-x="0"
+     inkscape:window-y="24"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="layer1">
+    <inkscape:grid
+       id="grid1"
+       units="px"
+       originx="-74.083327"
+       originy="-89.958345"
+       spacingx="0.26458333"
+       spacingy="0.26458334"
+       empcolor="#7a7aff"
+       empopacity="0.25098039"
+       color="#6a6aff"
+       opacity="0.1254902"
+       empspacing="5"
+       dotted="false"
+       gridanglex="30"
+       gridanglez="30"
+       visible="false" />
+  </sodipodi:namedview>
+  <defs
+     id="defs1" />
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-74.083328,-89.958344)">
+    <rect
+       style="fill:none;stroke:none;stroke-width:0.264999;stroke-linecap:square;stroke-dasharray:none;stroke-dashoffset:0"
+       id="rect33"
+       width="306.91663"
+       height="127.00001"
+       x="74.083328"
+       y="89.958344" />
+    <rect
+       style="fill:#ffeeaa;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none;stroke-dashoffset:0"
+       id="rect32"
+       width="121.70832"
+       height="84.666679"
+       x="253.99998"
+       y="127.00001" />
+    <text
+       xml:space="preserve"
+       style="font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="290.19363"
+       y="202.25124"
+       id="text32"><tspan
+         sodipodi:role="line"
+         id="tspan32"
+         style="font-size:6.35px;stroke-width:0.264583"
+         x="290.19363"
+         y="202.25124">Hardware parts</tspan></text>
+    <rect
+       style="fill:#eeffaa;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none;stroke-dashoffset:0"
+       id="rect31"
+       width="169.33333"
+       height="84.666679"
+       x="79.374992"
+       y="127.00001" />
+    <rect
+       style="fill:#fff6d5;stroke:#000000;stroke-width:0.3;stroke-linecap:square;stroke-dasharray:none;stroke-dashoffset:0"
+       id="rect27"
+       width="47.625004"
+       height="21.166676"
+       x="269.87497"
+       y="148.16667"
+       ry="2.645834" />
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect1"
+       width="21.166662"
+       height="10.583329"
+       x="84.566666"
+       y="174.52502"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="87.947342"
+       y="181.39488"
+       id="text1"><tspan
+         sodipodi:role="line"
+         id="tspan1"
+         style="stroke-width:0.264583"
+         x="87.947342"
+         y="181.39488">Thread</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect3"
+       width="21.166664"
+       height="10.583337"
+       x="111.12499"
+       y="153.45834"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="113.57446"
+       y="160.26309"
+       id="text3"><tspan
+         sodipodi:role="line"
+         id="tspan3"
+         style="stroke-width:0.264583"
+         x="113.57446"
+         y="160.26309">Process</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect4"
+       width="21.166664"
+       height="10.583333"
+       x="111.12499"
+       y="174.62502"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="114.50567"
+       y="181.49487"
+       id="text4"><tspan
+         sodipodi:role="line"
+         id="tspan4"
+         style="stroke-width:0.264583"
+         x="114.50567"
+         y="181.49487">Thread</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect5"
+       width="21.166658"
+       height="10.583333"
+       x="137.58333"
+       y="174.62502"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="140.964"
+       y="181.49487"
+       id="text5"><tspan
+         sodipodi:role="line"
+         id="tspan5"
+         style="stroke-width:0.264583"
+         x="140.964"
+         y="181.49487">Thread</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect6"
+       width="21.166672"
+       height="10.583333"
+       x="169.33331"
+       y="174.62502"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="172.71399"
+       y="181.49487"
+       id="text6"><tspan
+         sodipodi:role="line"
+         id="tspan6"
+         style="stroke-width:0.264583"
+         x="172.71399"
+         y="181.49487">Thread</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect7"
+       width="20.966642"
+       height="10.38336"
+       x="195.99167"
+       y="153.55833"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="198.24112"
+       y="160.16312"
+       id="text7"><tspan
+         sodipodi:role="line"
+         id="tspan7"
+         style="stroke-width:0.264583"
+         x="198.24112"
+         y="160.16312">Process</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect8"
+       width="21.16667"
+       height="10.583333"
+       x="195.79164"
+       y="174.62502"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="199.17232"
+       y="181.49487"
+       id="text8"><tspan
+         sodipodi:role="line"
+         id="tspan8"
+         style="stroke-width:0.264583"
+         x="199.17232"
+         y="181.49487">Thread</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect9"
+       width="21.166666"
+       height="10.583333"
+       x="222.24997"
+       y="174.62502"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="225.63065"
+       y="181.49487"
+       id="text9"><tspan
+         sodipodi:role="line"
+         id="tspan9"
+         style="stroke-width:0.264583"
+         x="225.63065"
+         y="181.49487">Thread</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect10"
+       width="21.166658"
+       height="10.583337"
+       x="153.45833"
+       y="132.29167"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="158.21979"
+       y="139.09644"
+       id="text10"><tspan
+         sodipodi:role="line"
+         id="tspan10"
+         style="stroke-width:0.264583"
+         x="158.21979"
+         y="139.09644">Loom</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect11"
+       width="21.166651"
+       height="10.58334"
+       x="306.91666"
+       y="132.29167"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="311.88068"
+       y="139.16154"
+       id="text11"><tspan
+         sodipodi:role="line"
+         id="tspan11"
+         style="stroke-width:0.264583"
+         x="311.88068"
+         y="139.16154">Node</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect12"
+       width="15.875001"
+       height="10.583341"
+       x="275.16666"
+       y="153.45834"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="278.86566"
+       y="160.291"
+       id="text12"><tspan
+         sodipodi:role="line"
+         id="tspan12"
+         style="stroke-width:0.264583"
+         x="278.86566"
+         y="160.291">CPU</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect13"
+       width="15.874983"
+       height="10.583341"
+       x="296.33334"
+       y="153.45834"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="300.03235"
+       y="160.291"
+       id="text13"><tspan
+         sodipodi:role="line"
+         id="tspan13"
+         style="stroke-width:0.264583"
+         x="300.03235"
+         y="160.291">CPU</tspan></text>
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 121.70832,164.04168 0,10.58334"
+       id="path15"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect3"
+       inkscape:connection-end="#rect4" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 206.45046,163.94169 -0.0505,10.68333"
+       id="path16"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect7"
+       inkscape:connection-end="#rect8" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 212.94006,163.94169 13.30366,10.68333"
+       id="path17"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-end="#rect9"
+       inkscape:connection-start="#rect7" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 199.96086,163.94169 -13.40463,10.68333"
+       id="path18"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect7"
+       inkscape:connection-end="#rect6" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 128.32291,164.04168 13.22917,10.58334"
+       id="path19"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect3"
+       inkscape:connection-end="#rect5" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 115.03723,164.04168 -13.21614,10.48334"
+       id="path20"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect3"
+       inkscape:connection-end="#rect1" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 153.45833,142.87501 -21.16667,10.58333"
+       id="path21"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect10"
+       inkscape:connection-end="#rect3" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 174.62499,142.86254 21.44212,10.69579"
+       id="path22"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect10"
+       inkscape:connection-end="#rect7" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 314.19269,142.87501 -6.61457,10.58333"
+       id="path24"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect11"
+       inkscape:connection-end="#rect13" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="M 308.90102,142.87501 291.04166,153.8654"
+       id="path25"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect11"
+       inkscape:connection-end="#rect12" />
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect25"
+       width="15.875013"
+       height="10.583337"
+       x="322.79163"
+       y="153.45834"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="326.49063"
+       y="160.291"
+       id="text25"><tspan
+         sodipodi:role="line"
+         id="tspan25"
+         style="stroke-width:0.264583"
+         x="326.49063"
+         y="160.291">CPU</tspan></text>
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect26"
+       width="15.874991"
+       height="10.583337"
+       x="343.95831"
+       y="153.45834"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="347.65732"
+       y="160.291"
+       id="text26"><tspan
+         sodipodi:role="line"
+         id="tspan26"
+         style="stroke-width:0.264583"
+         x="347.65732"
+         y="160.291">CPU</tspan></text>
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 320.80727,142.87501 6.61457,10.58333"
+       id="path26"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect11"
+       inkscape:connection-end="#rect25" />
+    <path
+       style="display:inline;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 326.09894,142.87501 17.85937,10.99039"
+       id="path27"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect11"
+       inkscape:connection-end="#rect26" />
+    <path
+       style="display:inline;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.265;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       d="m 174.62499,139.70001 95.24998,19.05"
+       id="path28"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect10" />
+    <rect
+       style="fill:#ffffff;stroke:#000000;stroke-width:0.2;stroke-linecap:square;stroke-dasharray:none"
+       id="rect29"
+       width="21.166666"
+       height="10.583335"
+       x="216.95831"
+       y="95.250008"
+       ry="0" />
+    <text
+       xml:space="preserve"
+       style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="219.6889"
+       y="101.67236"
+       id="text29"><tspan
+         sodipodi:role="line"
+         id="tspan29"
+         style="stroke-width:0.264583"
+         x="219.6889"
+         y="101.67236">System</tspan></text>
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 238.12498,104.89952 68.79168,28.32598"
+       id="path29"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect29"
+       inkscape:connection-end="#rect11" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m 218.47022,105.83334 -45.35713,26.45833"
+       id="path30"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect29"
+       inkscape:connection-end="#rect10" />
+    <text
+       xml:space="preserve"
+       style="font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;stroke-width:0.264583"
+       x="140.81671"
+       y="202.25124"
+       id="text31"><tspan
+         sodipodi:role="line"
+         id="tspan31"
+         style="font-size:6.35px;stroke-width:0.264583"
+         x="140.81671"
+         y="202.25124">Software parts</tspan></text>
+  </g>
+</svg>

doc/user/concepts/trace.md

@@ -0,0 +1,123 @@
+# Trace concepts
+
+When using libovni to generate traces or the emulator to process them, there are
+several concepts to keep in mind.
+
+## Trace elements
+
+The information generated by a program or later processed by other ovni tools is
+known as a trace. A runtime trace stores the information as-is in disk from a
+program execution. While a emulation trace is generated from the runtime trace
+for visualization with Paraver.
+
+Both runtime and emulation traces are always stored inside the same directory,
+by default `ovni/`, which is known as the *trace directory*.
+
+Here are the components of a runtime trace, as generated by libovni:
+
+<p align="center">
+<img alt="Trace concepts" src="../trace.svg">
+</p>
+
+### Stream
+
+A stream is a directory which contains a binary stream and the associated stream
+metadata file. Each stream is associated with a given part of a system. As of
+now, libovni can only generate streams associated to [threads](part-model.md#thread).
+
+### Stream metadata
+
+The stream metadata is a JSON file named `stream.json` which holds information
+about the stream itself.
+
+### Binary stream
+
+A binary stream is a file named `stream.obs` (.obs stands for Ovni Binary
+Stream) composed of a header and a concatenated array of events without padding.
+Notice that each event may have different length.
+
+### Event
+
+An event is a point in time that has some information associated. Events written
+at runtime by libovni have at MCV, a clock and a optional payload. The list of
+all events recognized by the emulator can be found [here](../emulation/events.md).
+
+Events can be displayed by ovnidump, which shows an explanation of what the
+event means:
+
+```txt
+$ ovnidump ovni/loom.hop.nosv-u1000/proc.1121064 | grep -A 10 VTx | head
+517267929632815  VTx  thread.1121064  executes the task 1 with bodyid 0
+517267930261672  VYc  thread.1121064  creates task type 2 with label "task"
+517267930875858  VTC  thread.1121064  creates parallel task 2 with type 2
+517267930877789  VU[  thread.1121064  starts submitting a task
+517267930877990  VU]  thread.1121064  stops  submitting a task
+517267930878098  VTC  thread.1121064  creates parallel task 3 with type 2
+517267930878196  VU[  thread.1121064  starts submitting a task
+517267930878349  VU]  thread.1121064  stops  submitting a task
+517267930878432  VTC  thread.1121064  creates parallel task 4 with type 2
+517267930878494  VU[  thread.1121064  starts submitting a task
+```
+
+There are two types or events: normal and jumbo events, the latter can hold
+large attached payloads.
+
+### MCV
+
+The MCV acronym is the abbreviation of Model-Class-Value, which are a three
+characters that identify any event. The MCV is shown in the ovnitop and ovnidump
+tools and allows easy filtering with grep, for a single or related events:
+
+```
+$ ovnitop ovni | grep VT
+VTe      20002
+VTx      20002
+VTC        200
+VTc          2
+VTp          1
+VTr          1
+```
+
+### Clock
+
+A clock is a 64 bit counter, which counts the number of nanoseconds from an
+arbitrary point in time in the past. Each event has the value of the clock
+stored inside, to indicate when that event happened. In a given trace there can
+be multiple clocks which don't refer to the same point in the past and must be
+corrected so they all produce an ordered sequence of events. The ovnisync
+program performs this correction by measuring the difference across clocks of
+different nodes.
+
+### Payload
+
+Events may have associated additional information which is stored in the stream.
+Normal events can hold up to 16 bytes, otherwise the jumbo events must be used
+to hold additional payload.
+
+## Other related concepts
+
+Apart from the trace itself, there are other concepts to keep in mind when the
+trace is being processed by the emulator.
+
+### Event model
+
+Each event belongs to an event model, as identified by the model character in
+the MCV. An event model is composed of several components:
+
+- A set of [events](#event) all with the same model identifier in the
+  [MCV](#mcv)
+- The emulator code that processes those events.
+- A human readable name, like `ovni` or `nanos6`.
+- A semantic version.
+
+### State
+
+A state is a discrete value that can change over time based on the events the
+emulator receives. Usually a single event causes a single state change, which is
+then written to the Paraver traces. An example is the thread state, which can
+change over time based on the events `OH*` that indicate a state transition
+of the current thread.
+
+In contrast with an event, states have a duration associated which can usually
+be observed in Paraver. Notice that the trace only contains events, the states
+are computed at emulation.

doc/user/concepts/trace.svg

@@ -0,0 +1,474 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   width="116.41666mm"
+   height="105.83334mm"
+   viewBox="0 0 116.41666 105.83334"
+   version="1.1"
+   id="svg1"
+   inkscape:version="1.3.2 (091e20ef0f, 2023-11-25, custom)"
+   sodipodi:docname="trace.svg"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg">
+  <sodipodi:namedview
+     id="namedview1"
+     pagecolor="#ffffff"
+     bordercolor="#999999"
+     borderopacity="1"
+     inkscape:showpageshadow="0"
+     inkscape:pageopacity="0"
+     inkscape:pagecheckerboard="0"
+     inkscape:deskcolor="#d1d1d1"
+     inkscape:document-units="mm"
+     showgrid="true"
+     inkscape:zoom="1.4609989"
+     inkscape:cx="248.80238"
+     inkscape:cy="199.17879"
+     inkscape:window-width="1920"
+     inkscape:window-height="1031"
+     inkscape:window-x="0"
+     inkscape:window-y="24"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="layer1">
+    <inkscape:grid
+       id="grid1"
+       units="px"
+       originx="-39.687498"
+       originy="-15.875"
+       spacingx="0.26458333"
+       spacingy="0.26458334"
+       empcolor="#7a7aff"
+       empopacity="0.25098039"
+       color="#6a6aff"
+       opacity="0.1254902"
+       empspacing="5"
+       dotted="false"
+       gridanglex="30"
+       gridanglez="30"
+       visible="true" />
+  </sodipodi:namedview>
+  <defs
+     id="defs1">
+    <rect
+       x="45"
+       y="370"
+       width="209.99998"
+       height="35"
+       id="rect15" />
+  </defs>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-39.687498,-15.875)">
+    <g
+       id="g59"
+       transform="translate(5.5781353,-0.74017783)"
+       style="stroke-width:0.15875;stroke-dasharray:none">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="100.23113"
+         y="24.896034"
+         id="text22"><tspan
+           sodipodi:role="line"
+           id="tspan22"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="100.23113"
+           y="24.896034">Trace</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect32"
+         width="15.875"
+         height="7.9375005"
+         x="92.281235"
+         y="19.4142" />
+    </g>
+    <g
+       id="g57"
+       transform="translate(-0.84260657,-0.98969722)"
+       style="stroke-width:0.15875;stroke-dasharray:none">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="106.39689"
+         y="46.186943"
+         id="text24"><tspan
+           sodipodi:role="line"
+           id="tspan24"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="106.39689"
+           y="46.186943">Stream</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect33"
+         width="21.166666"
+         height="7.9375019"
+         x="96.092606"
+         y="40.6772" />
+    </g>
+    <g
+       id="g56"
+       transform="translate(0.32292488,-0.89336269)"
+       style="stroke-width:0.15875;stroke-dasharray:none">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="78.773026"
+         y="46.090603"
+         id="text23"><tspan
+           sodipodi:role="line"
+           id="tspan23"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="78.773026"
+           y="46.090603">Stream</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect34"
+         width="21.166666"
+         height="7.9375"
+         x="68.468742"
+         y="40.580864" />
+    </g>
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="M 100.80665,26.611522 84.365216,39.687501"
+       id="path35"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect32"
+       inkscape:connection-end="#rect34" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="m 105.80376,26.611522 0.0227,13.075981"
+       id="path36"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect32"
+       inkscape:connection-end="#rect33" />
+    <g
+       id="g54"
+       style="stroke-width:0.15875;stroke-dasharray:none"
+       transform="translate(-0.99999828,-3.5391978)">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="58.792854"
+         y="72.586143"
+         id="text25"><tspan
+           sodipodi:role="line"
+           id="tspan25"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="58.792854"
+           y="72.586143">Metadata</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect36"
+         width="26.458336"
+         height="7.9375"
+         x="45.979164"
+         y="67.0392" />
+    </g>
+    <g
+       id="g55"
+       style="stroke-width:0.15875;stroke-dasharray:none"
+       transform="translate(-0.99999828,-3.5391978)">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="97.157433"
+         y="72.175835"
+         id="text26"><tspan
+           sodipodi:role="line"
+           id="tspan26"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="97.157433"
+           y="72.175835">Binary stream</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect37"
+         width="34.395832"
+         height="7.9375"
+         x="80.374992"
+         y="67.0392" />
+    </g>
+    <rect
+       style="stroke-width:0.15875;stroke-dasharray:none;fill:none;stroke:none;stroke-linecap:square"
+       id="rect60"
+       width="116.41666"
+       height="105.83334"
+       x="40.687496"
+       y="19.414198"
+       inkscape:label="background"
+       transform="translate(-0.99999735,-3.5391979)" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="M 75.847223,47.625001 61.736111,63.500002"
+       id="path37"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect34"
+       inkscape:connection-end="#rect36" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="M 82.241318,47.625001 93.706592,63.500002"
+       id="path38"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect34"
+       inkscape:connection-end="#rect37" />
+    <g
+       id="g53"
+       style="stroke-width:0.15875;stroke-dasharray:none"
+       transform="translate(-8.2507438,-3.0742007)">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="62.074863"
+         y="95.933655"
+         id="text27"><tspan
+           sodipodi:role="line"
+           id="tspan27"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="62.074863"
+           y="95.933655">Header</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect38"
+         width="23.812506"
+         height="7.9375014"
+         x="50.584076"
+         y="90.386703" />
+    </g>
+    <g
+       id="g52"
+       transform="translate(-8.2507438,-3.3066989)"
+       style="stroke-width:0.15875;stroke-dasharray:none">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="91.411552"
+         y="96.101036"
+         id="text28"><tspan
+           sodipodi:role="line"
+           id="tspan28"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="91.411552"
+           y="96.101036">Event</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect39"
+         width="23.812506"
+         height="7.9375014"
+         x="79.920769"
+         y="90.619202" />
+    </g>
+    <g
+       id="g51"
+       style="stroke-width:0.15875;stroke-dasharray:none"
+       transform="translate(-8.2507438,-3.0742007)">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="120.28321"
+         y="95.868538"
+         id="text29"><tspan
+           sodipodi:role="line"
+           id="tspan29"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="120.28321"
+           y="95.868538">Event</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect40"
+         width="23.812506"
+         height="7.9375014"
+         x="108.79243"
+         y="90.386703" />
+    </g>
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="M 89.517356,71.437502 61.29514,87.312503"
+       id="path40"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect37"
+       inkscape:connection-end="#rect38" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="M 94.406805,71.437502 85.742383,87.312503"
+       id="path41"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect37"
+       inkscape:connection-end="#rect39" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="M 99.218748,71.437502 109.8021,87.312503"
+       id="path42"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect37"
+       inkscape:connection-end="#rect40" />
+    <g
+       id="g46"
+       transform="translate(-68.659367,2.7781237)"
+       style="stroke-width:0.15875;stroke-dasharray:none">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="123.93869"
+         y="113.98891"
+         id="text30"><tspan
+           sodipodi:role="line"
+           id="tspan30"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="123.93869"
+           y="113.98891">MCV</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect42"
+         width="15.875001"
+         height="7.9375038"
+         x="116.41666"
+         y="108.47917" />
+    </g>
+    <g
+       id="g45"
+       transform="translate(-68.659367,5.4239515)"
+       style="stroke-width:0.15875;stroke-dasharray:none">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="147.92894"
+         y="111.38029"
+         id="text31"><tspan
+           sodipodi:role="line"
+           id="tspan31"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="147.92894"
+           y="111.38029">Clock</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect43"
+         width="15.875001"
+         height="7.9375038"
+         x="140.22916"
+         y="105.83334" />
+    </g>
+    <g
+       id="g44"
+       transform="translate(-66.112857,13.610614)"
+       style="stroke-width:0.15875;stroke-dasharray:none">
+      <g
+         id="g60"
+         style="stroke-width:0.15875;stroke-dasharray:none">
+        <text
+           xml:space="preserve"
+           style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="172.85362"
+           y="102.65102"
+           id="text32"><tspan
+             sodipodi:role="line"
+             id="tspan32"
+             style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+             x="172.85362"
+             y="102.65102">Payload</tspan></text>
+        <rect
+           style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+           id="rect44"
+           width="23.812506"
+           height="7.9375014"
+           x="161.36285"
+           y="97.514389" />
+      </g>
+    </g>
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="M 83.366344,95.250004 80.27399,111.2573"
+       id="path47"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-end="#g45" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="M 80.082847,95.250004 60.540101,111.2573"
+       id="path48"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-end="#g46" />
+    <g
+       id="g58"
+       transform="translate(-1.9073076,-1.4153422)"
+       style="stroke-width:0.15875;stroke-dasharray:none">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="132.47266"
+         y="46.496338"
+         id="text49"><tspan
+           sodipodi:role="line"
+           id="tspan49"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="132.47266"
+           y="46.496338">...</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect49"
+         width="18.619415"
+         height="7.9374995"
+         x="123.61564"
+         y="41.102844" />
+    </g>
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="m 110.56031,26.611522 15.69429,13.07598"
+       id="path49"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-end="#rect49"
+       inkscape:connection-start="#rect32" />
+    <g
+       id="g50"
+       transform="translate(-8.2507438,-2.9293636)"
+       style="stroke-width:0.15875;stroke-dasharray:none">
+      <text
+         xml:space="preserve"
+         style="font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+         x="149.35014"
+         y="95.635361"
+         id="text50"><tspan
+           sodipodi:role="line"
+           id="tspan50"
+           style="text-align:center;text-anchor:middle;stroke-width:0.15875;stroke-dasharray:none"
+           x="149.35014"
+           y="95.635361">...</tspan></text>
+      <rect
+         style="fill:none;stroke:#000000;stroke-width:0.15875;stroke-linecap:square;stroke-dasharray:none"
+         id="rect40-5"
+         width="23.812506"
+         height="7.9375014"
+         x="137.89658"
+         y="90.241867" />
+    </g>
+    <path
+       style="display:inline;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="m 104.06944,71.437502 29.98612,15.875001"
+       id="path50"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#rect37"
+       inkscape:connection-end="#g50" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.15875;stroke-linecap:butt;stroke-linejoin:miter;stroke-dasharray:none;stroke-opacity:1"
+       d="M 87.506274,95.250004 103.22625,111.125"
+       id="path60"
+       inkscape:connector-type="polyline"
+       inkscape:connector-curvature="0"
+       inkscape:connection-start="#g52"
+       inkscape:connection-end="#g44" />
+  </g>
+</svg>

doc/user/emulation/index.md

@@ -47,7 +47,7 @@ the following elements:
 
 - A single byte model identification (for example `O`).
 - A set of runtime events with that model identification (see the [list
-  of events](events)).
+  of events](events.md)).
 - Rules that determine which sequences of events are valid.
 - The emulation hooks that process each event and modify the state of
   the emulator.

doc/user/emulation/nosv.md

@@ -60,7 +60,7 @@ For more details, see [this MR][1].
 
 The subsystem view provides a simplified view on what is the nOS-V
 runtime doing over time. The view follows the same rules described in
-the [subsystem view of Nanos6](../nanos6/#subsystem_view).
+the [subsystem view of Nanos6](nanos6.md/#subsystem_view).
 
 
 ## Idle view

doc/user/emulation/ovni.md

@@ -60,4 +60,4 @@ will set all the channels to an error state.
 The emulator automatically switches the channels from one thread to
 another when a thread is switched from the CPU. So the different models
 don't need to worry about thread transitions. See the
-[channels](../channels) section for more information.
+[channels](../../dev/channels.md) section for more information.

doc/user/runtime/env.md

@@ -0,0 +1,56 @@
+# Environment variables
+
+Some environment variables can be used to adjust settings during the execution
+of libovni, they all begin with the `OVNI_` prefix. Be sure that all threads of
+the same node use the same environment variables.
+
+## OVNI_TMPDIR
+
+During the execution of your program, a per-thread buffer is kept where the new
+events are being recorded. When this buffer is full, it is written to disk and
+emptied, an operation known as flush. This may take a while depending on the
+underliying filesystem.
+
+Keep in mind that the thread will be blocked until the flush ends, so if your
+filesystem is slow it would interrupt the execution of your program for a long
+time. It is advisable to use the fastest filesystem available (see the tmpfs(5)
+and df(1) manual pages).
+
+You can select a temporary trace directory where the buffers will be flushed
+during the execution by setting the environment variable `OVNI_TMPDIR`. The last
+directory will be created if doesn't exist. In that case, as soon as a process
+calls `ovni_proc_fini()`, the traces of all its threads will be moved to the
+final directory at `$PWD/ovni`. Example:
+
+	OVNI_TMPDIR=$(mktemp -u /dev/shm/ovni.XXXXXX) srun ./your-app
+
+To test the different filesystem speeds, you can use hyperfine and dd. Take a
+closer look at the max time:
+
+```
+$ hyperfine 'dd if=/dev/zero of=/gpfs/projects/bsc15/bsc15557/kk bs=2M count=10'
+Benchmark 1: dd if=/dev/zero of=/gpfs/projects/bsc15/bsc15557/kk bs=2M count=10
+  Time (mean ± σ):      71.7 ms ± 130.4 ms    [User: 0.8 ms, System: 10.2 ms]
+  Range (min … max):    14.7 ms … 1113.2 ms    162 runs
+ 
+  Warning: Statistical outliers were detected. Consider re-running this
+  benchmark on a quiet PC without any interferences from other programs. It
+  might help to use the '--warmup' or '--prepare' options.
+
+$ hyperfine 'dd if=/dev/zero of=/tmp/kk bs=2M count=10'
+Benchmark 1: dd if=/dev/zero of=/tmp/kk bs=2M count=10
+  Time (mean ± σ):      56.2 ms ±   5.7 ms    [User: 0.6 ms, System: 14.8 ms]
+  Range (min … max):    45.8 ms …  77.8 ms    63 runs
+ 
+$ hyperfine 'dd if=/dev/zero of=/dev/shm/kk bs=2M count=10'
+Benchmark 1: dd if=/dev/zero of=/dev/shm/kk bs=2M count=10
+  Time (mean ± σ):      11.4 ms ±   0.4 ms    [User: 0.5 ms, System: 11.1 ms]
+  Range (min … max):     9.7 ms …  12.5 ms    269 runs
+```
+
+## OVNI_TRACEDIR
+
+By default, the runtime trace will be placed in the `ovni` directory, inside the
+working directory. You can specify a different location to place the trace by
+setting the `OVNI_TRACEDIR` environment variable. It accepts a relative or
+absolute path, which will be created if it doesn't exist.

doc/user/runtime/index.md

@@ -0,0 +1,109 @@
+# Introduction
+
+To use *libovni* to instrument a program, follow the next instructions
+carefully, or you may end up with an incomplete trace that is rejected at
+emulation.
+
+You can also generate a valid trace from your own software or hardware
+directly, but be sure to follow the [trace specification](trace_spec.md).
+
+## Initialization
+
+To initialize libovni follow these steps in all threads:
+
+1. **Check the version**. Call `ovni_version_check()` once before calling any
+   ovni function. It can be called multiple times from any thread, but only one
+   is required.
+
+2. **Init the process**. Call `ovni_proc_init()` to initialize the process. It
+   can only be called **once per process** and it must be called before the
+   thread is initialized.
+
+3. **Init the thread**. Call `ovni_thread_init()` to initialize the thread.
+   Multiple attempts to initialize the same thread are ignored with a warning.
+
+The `ovni_proc_init()` arguments are as follows:
+
+```c
+void ovni_proc_init(int app, const char *loom, int pid);
+```
+
+The `app` defines the "appid" of the program, which must be a number >0. This is
+useful to run multiple processes some of which run the same "app", so you can
+tell which one is which. The `loom` argument defines the
+[loom](../concepts/part-model.md#loom) name and maps the process to that
+loom. It must be compose of the host name, a dot and a suffix. The PID is the
+one obtained by `getpid(2)`.
+
+The `ovni_thread_init()` function only accepts one argument, the TID as returned
+by `gettid(2)`.
+
+## Setup metadata
+
+Once the process and thread are initialized, you can begin adding metadata to
+the thread stream.
+
+1. **Require models**. Call `ovni_thread_require()` with the required model
+   version before emitting events for a given model. Only required once from a
+   thread in a given trace.
+
+2. **Emit loom CPUs**. Call `ovni_add_cpu()` to register each CPU in the loom. It can
+   be done from a single thread or multiple threads, in the latter the list of
+   CPUs is merged.
+
+3. **Set the rank**. If you use MPI, call `ovni_proc_set_rank()` to register the
+   rank and number of ranks of the current execution. Only once per process.
+
+## Start the execution
+
+The current thread must switch to the "Running" state before any event can be
+processed by the emulator. Do so by emitting a [`OHx`
+event](../emulation/events.md#OHx) in the stream with the appropriate payload:
+
+```c
+static void thread_execute(int32_t cpu, int32_t ctid, uint64_t tag)
+{
+    struct ovni_ev ev = {0};
+    ovni_ev_set_clock(&ev, ovni_clock_now());
+    ovni_ev_set_mcv(&ev, "OHx");
+    ovni_payload_add(&ev, (uint8_t *) &cpu, sizeof(cpu));
+    ovni_payload_add(&ev, (uint8_t *) &ctid, sizeof(ctid));
+    ovni_payload_add(&ev, (uint8_t *) &tag, sizeof(tag));
+    ovni_ev_emit(&ev);
+}
+```
+
+The `cpu` is the logical index (not the physical ID) of the loom CPU at which
+this thread will begin the execution. Use -1 if it is not known. The `ctid` and
+`tag` allow you to track the exact point at which a given thread was created and
+by which thread but they are not relevant for the first thread, so they can be
+set to -1.
+
+## Emit events
+
+After this point you can emit any other event from this thread. Use the
+`ovni_ev_*` set of functions to create and emit events. Notice that all events
+are refer to the current thread that emits them.
+
+If you need to store metadata information, use the `ovni_attr_*` set of
+functions. The metadata is stored in disk by `ovni_attr_fluch()` and when the
+thread is freed by `ovni_thread_free()`.
+
+Attempting to emit events or writing metadata without having a thread
+initialized will cause your program to abort.
+
+## Finishing the execution
+
+To finalize the execution **every thread** must perform the following steps,
+otherwise the trace **will be rejected**.
+
+1. **End the current thread**. Emit a [`OHe` event](../emulation/events.md#OHe) to inform the current thread ends.
+2. **Flush the buffer**. Call `ovni_flush()` to be sure all events are written
+   to disk.
+3. **Free the thread**. Call `ovni_thread_free()` to complete the stream and
+   free the memory used by the buffer.
+4. **Finish the process**. If this is the last thread, call `ovni_proc_fini()`
+   to set the process state to finished.
+
+If a thread fails to perform these steps, the complete trace will be rejected by
+the emulator as it cannot guarantee the trace to be consistent.

doc/user/runtime/mark.md

@@ -1,4 +1,4 @@
-# Mark API
+# Mark events
 
 The mark API allows you to add arbitrary events in a trace to mark regions of
 interest while debugging or developing a new program or library. The events are
@@ -80,6 +80,68 @@ void ovni_mark_pop(int32_t type, int64_t value);
 
 The value in the pop call must match the previous pushed value.
 
+<details>
+<summary>Example OmpSs-2 program</summary>
+<br>
+<div style="padding-left: 1em; border-left: 3px solid #ddd">
+<p>
+Here is a dummy program showing how to use the mark API with an OmpSs-2 program.
+Notice that there is no initialization of the current thread or process, as it
+already occurs inside the OmpSs-2 runtime before reaching the main.
+
+```c
+/* Build with:
+ *     $ clang -fompss-2 -lovni dummy.c -o dummy
+ * Enable instrumentation in nanos6:
+ *     $ echo 'version.instrument = "ovni"' > nanos6.toml
+ * Run:
+ *     $ ./dummy
+ * Emulate:
+ *     $ ovniemu ovni
+ * View timeline:
+ *     $ wxparaver ovni/cpu.prv ovni/cfg/cpu/ovni/mark.cfg
+ */
+#include <ovni.h>
+#include <unistd.h>
+
+enum { INDEX = 0, RUN = 1 };
+
+static void process(int run, int i)
+{
+    ovni_mark_push(RUN, run + 1);
+    ovni_mark_push(INDEX, i + 1);
+    usleep(10000); // Dummy operation for 10 ms
+    ovni_mark_pop(INDEX, i + 1);
+    ovni_mark_pop(RUN, run + 1);
+}
+
+int main(void)
+{
+    ovni_mark_type(INDEX, OVNI_MARK_STACK, "Index");
+    ovni_mark_type(RUN, OVNI_MARK_STACK, "Run");
+
+    for (int run = 0; run < 10; run++) {
+        for (int i = 0; i < 50; i++) {
+            #pragma oss task
+            process(run, i);
+        }
+    }
+
+    #pragma oss taskwait
+
+    return 0;
+}
+```
+
+<!-- Images don't seem to work via markdown -->
+<p>Here is the resulting timeline loaded in Paraver with the gradient color
+configuration, showing the first mark type (the index):
+<img style="margin-top: 1em" alt="" src="../fig/mark.png"></p>
+
+</div>
+</details>
+<p></p>
+
 ## Usage in Paraver
 
 Each thread holds a channel for each mark type that you have defined. The

doc/user/runtime/trace_spec.md

@@ -1,127 +1,149 @@
-# Trace specification
+# Trace specification v3
 
 !!! Important
 
 	This document refers to the trace specification for
-	the version 2
+	the version 3
 
-The ovni instrumentation library stores the information collected in a
+The ovni instrumentation library libovni stores the information
-trace following the specification of this document.
+collected in a runtime trace following the specification of this document.
+
+## Structure
+
+An ovni runtime trace (or simply, a trace) is composed of one or more
+[streams](../concepts/trace.md#stream), which are directories containing
+two mandatory files:
+
+- `stream.json` the stream metadata in JSON format.
+- `stream.obs` the binary stream with events.
+
+Each stream is assigned to a single *part* in the [part
+model](../concepts/part-model.md), usually assigned to a given thread.
+
+There are no imposed rules on how to organize the several streams into
+directories, but libovni uses the following approach for thread streams:
 
 The complete trace is stored in a top-level directory named `ovni`.
-Inside this directory you will find the loom directories with the prefix
+Inside this directory you will find the loom directories. The name of
-`loom.`. The name of the loom is built from the `loom` parameter of
+the loom directory is built from the `loom` parameter of `ovni_proc_init()`,
-`ovni_proc_init()`, prefixing it with `loom.`.
+prefixing it with `loom.`.
 
 Each loom directory contains one directory per process of that loom. The
 name is composed of the `proc.` prefix and the PID of the process
 specified in the `pid` argument to `ovni_proc_init()`.
 
-Each process directory contains:
+Inside each process there is one directory for each thread, composed by
+the `thread.` prefix and the TID, which are the streams. The files
+`stream.json` and `stream.obs` reside inside. Example:
 
-- The process metadata file `metadata.json`.
+```
-- The thread streams, composed of:
+ovni/loom.mio.nosv-u1000/proc.89719/thread.89719/stream.json
-    - The binary stream like `thread.123.obs`
+ovni/loom.mio.nosv-u1000/proc.89719/thread.89719/stream.obs
-    - The thread metadata like `thread.123.json`
+```
 
-## Process metadata
+This structure prevents collisions among threads with the same TID among nodes,
+while allowing dumping events from a single thread, process or loom with
+ovnidump.
 
-!!! Important
+## Stream metadata
 
-	Process metadata has version 2
+The `stream.json` metadata file contains information about the part that
+the stream is assigned to. This is generally used to determine the
+hierarchy of the part model.
 
-The process metadata file contains important information about the trace
+The JSON must be an object (dictionary) with the following mandatory
-that is invariant during the complete execution, and generally is
+keys:
-required to be available prior to processing the events in the trace.
-
-The metadata is stored in the JSON file `metadata.json` inside each
-process directory and contains the following keys:
 
 - `version`: a number specifying the version of the metadata format.
-  Must have the value 2 for this version.
+  Must have the value 3 for this version.
-- `app_id`: the application ID, used to distinguish between applications
+
-  running on the same loom.
+The rest of information is stored for each model.
-- `rank`: the rank of the MPI process (optional).
+
-- `nranks`: number of total MPI processes (optional).
+In particular, the `ovni` model enforces the use of:
-- `cpus`: the array of $`N_c`$ CPUs available in the loom. Only one
+
-  process in the loom must contain this mandatory key. Each element is a
+- `ovni.part`: the type of part this stream is assigned to, usually
-  dictionary with the keys:
+  `thread`.
-  - `index`: containing the logical CPU index from 0 to $`N_c - 1`$.
+- `ovni.require`: a dictionary of model name and version which will
+  determine which models are enabled at emulation and the required
+  version.
+- `ovni.finished`: must be 1 to ensure the stream is complete (mandatory
+  in all streams).
+
+### Thread stream metadata
+
+For `thread` streams, the following attributes are used.
+
+- `ovni.tid`: the TID of the thread (mandatory, per-thread).
+- `ovni.pid`: the PID of the process that the thread belongs to (mandatory, per-thread).
+- `ovni.app_id`: the application ID of the process (optional, per-process).
+- `ovni.rank`: the rank of the MPI process (optional, per-process).
+- `ovni.nranks`: number of total MPI processes (optional, per-process).
+- `ovni.loom`: the name of the loom that the process belongs to (mandatory, per-process).
+- `ovni.loom_cpus`: the array of N CPUs available in the loom
+  (mandatory, per-loom). Each element is a dictionary with the keys:
+    - `index`: containing the logical CPU index from 0 to N - 1.
     - `phyid`: the number of the CPU as given by the operating system
-    (which can exceed $`N_c`$).
+      (which can exceed N).
 
-Here is an example of the `metadata.json` file:
+Notice that some attributes don't need to be present in all thread
+streams. For example, per-process requires that at least one thread
+contains the attribute for each process. Similarly, per-loom requires
+that at least one thread of the loom emits the attribute.
 
-```
+The final attribute value will be computed by merging all the values from the
-{
+children metadata. Simple values like numbers or strings must match exactly if
-    "version": 2,
+they appear duplicated, arrays are appended.
-    "app_id": 1,
-    "rank": 0,
-    "nranks": 4,
-    "cpus": [
-        {
-            "index": 0,
-            "phyid": 0
-        },
-        {
-            "index": 1,
-            "phyid": 1
-        },
-        {
-            "index": 2,
-            "phyid": 2
-        },
-        {
-            "index": 3,
-            "phyid": 3
-        }
-    ]
-}
-```
 
-## Thread metadata
+Other attributes can be used for other models.
 
-!!! Important
+Here is an example of the `stream.json` file for a thread of a nOS-V
-
+program:
-	Thread metadata has version 2
-
-The thread metadata stores constant information per thread, like the
-process metadata. The information is stored in a dictionary, where the
-name of the emulation models are used as keys. In particular, the
-libovni library writes information in the "ovni" key, such as the
-model requirements, and other information like the version of libovni
-used. Example:
 
 ```json
 {
-    "version": 2,
+    "version": 3,
     "ovni": {
         "lib": {
-            "version": "1.4.0",
+            "version": "1.10.0",
-            "commit": "unknown"
+            "commit": "dirty"
         },
+        "part": "thread",
+        "tid": 89719,
+        "pid": 89719,
+        "loom": "mio.nosv-u1000",
+        "app_id": 1,
         "require": {
-            "ovni": "1.0.0"
+            "ovni": "1.1.0",
-        }
+            "nosv": "2.3.0"
+        },
+        "loom_cpus": [
+            { "index": 0, "phyid": 0 },
+            { "index": 1, "phyid": 1 },
+            { "index": 2, "phyid": 2 },
+            { "index": 3, "phyid": 3 }
+        ],
+        "finished": 1
+    },
+    "nosv": {
+        "can_breakdown": false,
+        "lib_version": "2.3.1"
     }
 }
 ```
 
-The metadata is written to disk when the thread is first initialized
+## Binary stream
-and when the thread finishes.
-
-## Thread binary streams
 
 !!! Important
 
-	Thread binary stream has version 1
+	Binary streams have version 1
 
-Streams are a binary files that contains a succession of events with
+A binary stream is a binary file named `stream.obs` that contains a
-monotonically increasing clock values. Streams have a small header and
+succession of events with monotonically increasing clock values. They
-the variable size events just after the header.
+have a small header and the variable size events just after the header.
 
 The header contains the magic 4 bytes of "ovni" and a version number of
-4 bytes too. Here is a figure of the data stored in disk:
+4 bytes too. Here is a figure of the data stored in disk on a little
+endian machine:
 
 ![Stream](fig/stream.svg)
 
@@ -145,7 +167,7 @@ payload:
 - Normal events: with a payload up to 16 bytes
 - Jumbo events: with a payload up to $`2^{32}`$ bytes
 
-## Normal events
+### Normal events
 
 The normal events are composed of:
 
@@ -178,7 +200,7 @@ In the following figure you can see each field annotated:
 
 ![Normal event with payload content](fig/event-normal-payload.svg)
 
-## Jumbo events
+### Jumbo events
 
 The jumbo events are just like normal events but they can hold large
 data. The size of the jumbo data is stored as a 32 bits integer as a
@@ -203,10 +225,10 @@ In the following figure you can see each field annotated:
 
 ![Jumbo event](fig/event-jumbo.svg)
 
-## Design considerations
+### Design considerations
 
-The stream format has been designed to be very simple, so writing a
+The binary stream format has been designed to be very simple, so writing
-parser library would take no more than 2 days for a single developer.
+a parser library would take no more than 2 days for a single developer.
 
 The size of the events has been designed to be small, with 12 bytes per
 event when no payload is used.
@@ -239,11 +261,7 @@ raw stream in binary, as the MCV codes can be read as ASCII characters:
 This allows a human to detect signs of corruption by visually inspecting
 the streams.
 
-## Limitations
+### Limitations
 
 The streams are designed to be read only forward, as they only contain
 the size of each event in the header.
-
-Currently, we only support using the threads as sources of events, using
-one stream per thread. However, adding support for more streams from
-multiple sources is planned for the future.

doc/user/runtime/tracing.md

@@ -1,94 +0,0 @@
-# Tracing a new program
-
-Read carefully this document before using libovni to instrument a new
-component. There are a few rules you must follow to ensure the runtime
-trace is correct.
-
-## Trace processes and threads
-
-- Call `ovni_version_check()` once before calling any ovni function.
-
-- Call `ovni_proc_init()` when a new process begins the execution.
-
-- Call `ovni_thread_init()` when a new thread begins the execution
-  (including the main process thread).
-
-- Call `ovni_thread_require()` with the required model version before
-  emitting events for that model.
-
-- Call `ovni_flush()` and `ovni_thread_free()` when it finishes (in that
-  order).
-
-- Call `ovni_proc_fini()` when a process ends, after all threads have
-  finished.
-
-You can use `ovni_ev_emit()` to record a new event. If you need more
-than 16 bytes of payload, use `ovni_ev_jumbo_emit()`. See the [trace
-specification](../trace_spec) for more details.
-
-Compile and link with libovni. When you run your program, a new
-directory ovni will be created in the current directory `$PWD/ovni`
-which contains the execution trace.
-
-You can change the trace directory by defining the `OVNI_TRACEDIR`
-environment variable. The envar accepts a trace directory name, a
-relative path to the trace directory, or its absolute path. In the
-first case, the trace directory will be created in the current
-directory `$PWD`.
-
-## Rules
-
-Follow these rules to avoid losing events:
-
-1. No event may be emitted until the process is initialized with
-`ovni_proc_init()` and the thread with `ovni_thread_init()`.
-
-2. When a thread ends the execution, it must call `ovni_flush()` to write the
-events in the buffer to disk.
-
-3. All threads must have flushed its buffers before calling `ovni_proc_fini()`.
-
-## Select a fast directory
-
-During the execution of your program, a per-thread buffer is kept where the new
-events are being recorded. When this buffer is full, it is written to disk and
-emptied, an operation known as flush. This may take a while depending on the
-underliying filesystem.
-
-Keep in mind that the thread will be blocked until the flush ends, so if your
-filesystem is slow it would interrupt the execution of your program for a long
-time. It is advisable to use the fastest filesystem available (see the tmpfs(5)
-and df(1) manual pages).
-
-You can select the trace directory where the buffers will be flushed during the
-execution by setting the environment variable `OVNI_TMPDIR`. The last directory
-will be created if doesn't exist. In that case, as soon as a process calls
-`ovni_proc_fini()`, the traces of all its threads will be moved to the final
-directory at `$PWD/ovni`. Example:
-
-	OVNI_TMPDIR=$(mktemp -u /dev/shm/ovni.XXXXXX) srun ./your-app
-
-To test the different filesystem speeds, you can use hyperfine and dd. Take a
-closer look at the max time:
-
-```
-$ hyperfine 'dd if=/dev/zero of=/gpfs/projects/bsc15/bsc15557/kk bs=2M count=10'
-Benchmark 1: dd if=/dev/zero of=/gpfs/projects/bsc15/bsc15557/kk bs=2M count=10
-  Time (mean ± σ):      71.7 ms ± 130.4 ms    [User: 0.8 ms, System: 10.2 ms]
-  Range (min … max):    14.7 ms … 1113.2 ms    162 runs
- 
-  Warning: Statistical outliers were detected. Consider re-running this
-  benchmark on a quiet PC without any interferences from other programs. It
-  might help to use the '--warmup' or '--prepare' options.
-
-$ hyperfine 'dd if=/dev/zero of=/tmp/kk bs=2M count=10'
-Benchmark 1: dd if=/dev/zero of=/tmp/kk bs=2M count=10
-  Time (mean ± σ):      56.2 ms ±   5.7 ms    [User: 0.6 ms, System: 14.8 ms]
-  Range (min … max):    45.8 ms …  77.8 ms    63 runs
- 
-$ hyperfine 'dd if=/dev/zero of=/dev/shm/kk bs=2M count=10'
-Benchmark 1: dd if=/dev/zero of=/dev/shm/kk bs=2M count=10
-  Time (mean ± σ):      11.4 ms ±   0.4 ms    [User: 0.5 ms, System: 11.1 ms]
-  Range (min … max):     9.7 ms …  12.5 ms    269 runs
-```
-

include/ovni.h.in

@@ -18,7 +18,7 @@ extern "C" {
 #include <string.h>
 #include <sys/types.h>
 
-#define OVNI_METADATA_VERSION 2
+#define OVNI_METADATA_VERSION 3
 
 #define OVNI_TRACEDIR "ovni"
 #define OVNI_MAX_HOSTNAME 512
@@ -31,6 +31,9 @@ extern "C" {
 
 #define OVNI_STREAM_EXT ".obs"
 
+/* Version of the ovni model for events */
+#define OVNI_MODEL_VERSION "1.1.0"
+
 /* Follow https://semver.org rules for versioning */
 #define OVNI_LIB_VERSION "@PROJECT_VERSION@"
 #define OVNI_GIT_COMMIT  "@OVNI_GIT_COMMIT@"

mkdocs.yml

@@ -23,10 +23,13 @@ markdown_extensions:
 nav:
   - index.md
   - 'User guide':
-    - user/concepts.md
     - user/installation.md
+    - 'Concepts':
+      - user/concepts/part-model.md
+      - user/concepts/trace.md
     - 'Runtime':
-      - user/runtime/tracing.md
+      - user/runtime/index.md
+      - user/runtime/env.md
       - user/runtime/mark.md
       - user/runtime/distributed.md
       - user/runtime/kernel.md

src/emu/CMakeLists.txt

@@ -30,7 +30,6 @@ add_library(emu STATIC
   stream.c
   trace.c
   loom.c
-  metadata.c
   mux.c
   sort.c
   path.c

src/emu/loom.c

@@ -1,4 +1,4 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2021-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #include "loom.h"
@@ -8,15 +8,13 @@
 #include "cpu.h"
 #include "path.h"
 #include "proc.h"
+#include "stream.h"
 #include "uthash.h"
 
-static const char *loom_prefix = "loom.";
-
 static void
 set_hostname(char host[PATH_MAX], const char name[PATH_MAX])
 {
-	/* Skip prefix */
+	const char *start = name;
-	const char *start = name + strlen(loom_prefix);
 
 	/* Copy until dot or end */
 	int i;
@@ -30,10 +28,19 @@ set_hostname(char host[PATH_MAX], const char name[PATH_MAX])
 	host[i] = '\0';
 }
 
-int
+const char *
-loom_matches(const char *path)
+loom_name(struct stream *s)
 {
-	return path_has_prefix(path, loom_prefix);
+	JSON_Object *meta = stream_metadata(s);
+	const char *loom = json_object_dotget_string(meta, "ovni.loom");
+
+	if (loom == NULL) {
+		err("cannot get attribute ovni.loom for stream: %s",
+				s->relpath);
+		return NULL;
+	}
+
+	return loom;
 }
 
 int
@@ -41,11 +48,6 @@ loom_init_begin(struct loom *loom, const char *name)
 {
 	memset(loom, 0, sizeof(struct loom));
 
-	if (!path_has_prefix(name, loom_prefix)) {
-		err("loom name must start with '%s': %s", loom_prefix, name);
-		return -1;
-	}
-
 	if (strchr(name, '/') != NULL) {
 		err("loom name cannot contain '/': %s", name);
 		return -1;
@@ -68,6 +70,84 @@ loom_init_begin(struct loom *loom, const char *name)
 	return 0;
 }
 
+/* Merges the metadata CPUs with the ones in the loom */
+static int
+load_cpus(struct loom *loom, JSON_Object *meta)
+{
+	JSON_Array *cpuarray = json_object_dotget_array(meta, "ovni.loom_cpus");
+
+	/* It may not have the CPUs defined */
+	if (cpuarray == NULL)
+		return 0;
+
+	size_t ncpus = json_array_get_count(cpuarray);
+	if (ncpus == 0) {
+		err("empty 'cpus' array in metadata");
+		return -1;
+	}
+
+	for (size_t i = 0; i < ncpus; i++) {
+		JSON_Object *jcpu = json_array_get_object(cpuarray, i);
+		if (jcpu == NULL) {
+			err("json_array_get_object() failed for cpu");
+			return -1;
+		}
+
+		/* Cast from double */
+		int index = (int) json_object_get_number(jcpu, "index");
+		int phyid = (int) json_object_get_number(jcpu, "phyid");
+
+		struct cpu *cpu = loom_find_cpu(loom, phyid);
+
+		if (cpu) {
+			/* Ensure they have the same index */
+			if (cpu->index != index) {
+				err("mismatch index in existing cpu: %d", index);
+				return -1;
+			}
+
+			/* Duplicated, ignore */
+			continue;
+		}
+
+		cpu = calloc(1, sizeof(struct cpu));
+		if (cpu == NULL) {
+			err("calloc failed:");
+			return -1;
+		}
+
+		cpu_init_begin(cpu, index, phyid, 0);
+
+		if (loom_add_cpu(loom, cpu) != 0) {
+			err("loom_add_cpu() failed");
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+/** Merges the given metadata with the one stored.
+ *
+ * It is an error to provide metadata that doesn't match with the already stored
+ * in the process.
+ *
+ * Precondition: The stream ovni.part must be "thread".
+ * Precondition: The stream version must be ok.
+ */
+int
+loom_load_metadata(struct loom *loom, struct stream *s)
+{
+	JSON_Object *meta = stream_metadata(s);
+
+	if (load_cpus(loom, meta) != 0) {
+		err("cannot load loom cpus");
+		return -1;
+	}
+
+	return 0;
+}
+
 void
 loom_set_gindex(struct loom *loom, int64_t gindex)
 {
@@ -181,6 +261,44 @@ by_phyid(struct cpu *c1, struct cpu *c2)
 		return 0;
 }
 
+int
+loom_set_rank_min(struct loom *loom)
+{
+	if (loom->rank_min != INT_MAX) {
+		err("rank_min already set");
+		return -1;
+	}
+
+	/* Ensure that all processes have a rank */
+	for (struct proc *p = loom->procs; p; p = p->hh.next) {
+		if (p->rank >= 0) {
+			loom->rank_enabled = 1;
+			break;
+		}
+	}
+
+	if (!loom->rank_enabled) {
+		dbg("loom %s has no rank information", loom->name);
+		return 0;
+	}
+
+	/* Ensure that all processes have a rank */
+	for (struct proc *p = loom->procs; p; p = p->hh.next) {
+		if (p->rank < 0) {
+			err("process %s has no rank information", p->id);
+			return -1;
+		}
+
+		/* Compute rank_min for CPU sorting */
+		if (p->rank < loom->rank_min)
+			loom->rank_min = p->rank;
+	}
+
+	dbg("loom %s has rank_min %d", loom->name, loom->rank_min);
+
+	return 0;
+}
+
 void
 loom_sort(struct loom *loom)
 {
@@ -198,22 +316,22 @@ loom_sort(struct loom *loom)
 int
 loom_init_end(struct loom *loom)
 {
-	/* Set rank enabled */
+	/* rank_min must be set */
-	for (struct proc *p = loom->procs; p; p = p->hh.next) {
+	if (loom->rank_enabled && loom->rank_min == INT_MAX) {
-		if (p->rank >= 0) {
+		err("rank_min not set");
-			loom->rank_enabled = 1;
-			break;
-		}
-	}
-
-	/* Ensure that all processes have a rank */
-	if (loom->rank_enabled) {
-		for (struct proc *p = loom->procs; p; p = p->hh.next) {
-			if (p->rank < 0) {
-				err("process %s has no rank information", p->id);
 		return -1;
 	}
+
+	/* It is not valid to define a loom without CPUs */
+	if (loom->ncpus == 0) {
+		err("loom %s has no physical CPUs", loom->name);
+		return -1;
 	}
+
+	/* Or without processes */
+	if (loom->nprocs == 0) {
+		err("loom %s has no processes", loom->name);
+		return -1;
 	}
 
 	/* Populate cpus_array */
@@ -222,6 +340,7 @@ loom_init_end(struct loom *loom)
 		err("calloc failed:");
 		return -1;
 	}
+
 	for (struct cpu *c = loom->cpus; c; c = c->hh.next) {
 		int index = cpu_get_index(c);
 		if (index < 0 || (size_t) index >= loom->ncpus) {
@@ -277,34 +396,6 @@ loom_add_proc(struct loom *loom, struct proc *proc)
 		return -1;
 	}
 
-	if (!proc->metadata_loaded) {
-		err("process %d hasn't loaded metadata", pid);
-		return -1;
-	}
-
-	if (loom->rank_enabled && proc->rank < 0) {
-		err("missing rank in process %d", pid);
-		return -1;
-	}
-
-	/* Check previous ranks if any */
-	if (!loom->rank_enabled && proc->rank >= 0) {
-		loom->rank_enabled = 1;
-
-		for (struct proc *p = loom->procs; p; p = p->hh.next) {
-			if (p->rank < 0) {
-				err("missing rank in process %d", p->pid);
-				return -1;
-			}
-
-			if (p->rank < loom->rank_min)
-				loom->rank_min = p->rank;
-		}
-	}
-
-	if (loom->rank_enabled && proc->rank < loom->rank_min)
-		loom->rank_min = proc->rank;
-
 	HASH_ADD_INT(loom->procs, pid, proc);
 	loom->nprocs++;
 

src/emu/loom.h

@@ -12,6 +12,7 @@
 #include "cpu.h"
 #include "extend.h"
 struct proc;
+struct stream;
 
 struct loom {
 	int64_t gindex;
@@ -50,8 +51,10 @@ struct loom {
 	struct extend ext;
 };
 
-USE_RET int loom_matches(const char *relpath);
+USE_RET const char *loom_name(struct stream *s);
 USE_RET int loom_init_begin(struct loom *loom, const char *name);
+USE_RET int loom_load_metadata(struct loom *loom, struct stream *s);
+USE_RET int loom_set_rank_min(struct loom *loom);
 USE_RET int loom_init_end(struct loom *loom);
 USE_RET int loom_add_cpu(struct loom *loom, struct cpu *cpu);
 USE_RET int64_t loom_get_gindex(struct loom *loom);

src/emu/metadata.c

@@ -1,159 +0,0 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
- * SPDX-License-Identifier: GPL-3.0-or-later */
-
-#include "metadata.h"
-#include <stdlib.h>
-#include <string.h>
-#include "cpu.h"
-#include "loom.h"
-#include "ovni.h"
-#include "parson.h"
-#include "proc.h"
-#include "thread.h"
-
-static JSON_Object *
-load_json(const char *path)
-{
-	JSON_Value *vmeta = json_parse_file_with_comments(path);
-	if (vmeta == NULL) {
-		err("json_parse_file_with_comments() failed");
-		return NULL;
-	}
-
-	JSON_Object *meta = json_value_get_object(vmeta);
-	if (meta == NULL) {
-		err("json_value_get_object() failed");
-		return NULL;
-	}
-
-	return meta;
-}
-
-static int
-check_version(JSON_Object *meta)
-{
-	JSON_Value *version_val = json_object_get_value(meta, "version");
-	if (version_val == NULL) {
-		err("missing attribute \"version\"");
-		return -1;
-	}
-
-	int version = (int) json_number(version_val);
-
-	if (version != OVNI_METADATA_VERSION) {
-		err("metadata version mismatch %d (expected %d)",
-				version, OVNI_METADATA_VERSION);
-		return -1;
-	}
-
-	return 0;
-}
-
-static int
-has_cpus(JSON_Object *meta)
-{
-	/* Only check for the "cpus" key, if it has zero elements is an error
-	 * that will be reported later */
-	if (json_object_get_array(meta, "cpus") != NULL)
-		return 1;
-
-	return 0;
-}
-
-static int
-load_cpus(struct loom *loom, JSON_Object *meta)
-{
-	JSON_Array *cpuarray = json_object_get_array(meta, "cpus");
-	if (cpuarray == NULL) {
-		err("cannot find 'cpus' array");
-		return -1;
-	}
-
-	size_t ncpus = json_array_get_count(cpuarray);
-	if (ncpus == 0) {
-		err("empty 'cpus' array in metadata");
-		return -1;
-	}
-
-	if (loom->ncpus > 0) {
-		err("loom %s already has cpus", loom->id);
-		return -1;
-	}
-
-	for (size_t i = 0; i < ncpus; i++) {
-		JSON_Object *jcpu = json_array_get_object(cpuarray, i);
-		if (jcpu == NULL) {
-			err("json_array_get_object() failed for cpu");
-			return -1;
-		}
-
-		/* Cast from double */
-		int index = (int) json_object_get_number(jcpu, "index");
-		int phyid = (int) json_object_get_number(jcpu, "phyid");
-
-		struct cpu *cpu = calloc(1, sizeof(struct cpu));
-		if (cpu == NULL) {
-			err("calloc failed:");
-			return -1;
-		}
-
-		cpu_init_begin(cpu, index, phyid, 0);
-
-		if (loom_add_cpu(loom, cpu) != 0) {
-			err("loom_add_cpu() failed");
-			return -1;
-		}
-	}
-
-	return 0;
-}
-
-int
-metadata_load_proc(const char *path, struct loom *loom, struct proc *proc)
-{
-	JSON_Object *meta = load_json(path);
-	if (meta == NULL) {
-		err("cannot load proc metadata from file %s", path);
-		return -1;
-	}
-
-	if (check_version(meta) != 0) {
-		err("version check failed");
-		return -1;
-	}
-
-	/* The appid is populated from the metadata */
-	if (proc_load_metadata(proc, meta) != 0) {
-		err("cannot load process attributes");
-		return -1;
-	}
-
-	if (has_cpus(meta) && load_cpus(loom, meta) != 0) {
-		err("cannot load loom cpus");
-		return -1;
-	}
-
-	return 0;
-}
-
-int
-metadata_load_thread(const char *path, struct thread *thread)
-{
-	JSON_Object *meta = load_json(path);
-	if (meta == NULL) {
-		err("cannot load thread metadata from file %s", path);
-		return -1;
-	}
-
-	if (check_version(meta) != 0) {
-		err("version check failed");
-		return -1;
-	}
-
-	if (thread_load_metadata(thread, meta) != 0) {
-		err("cannot load thread attributes");
-		return -1;
-	}
-
-	return 0;
-}

src/emu/metadata.h

@@ -1,15 +0,0 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
- * SPDX-License-Identifier: GPL-3.0-or-later */
-
-#ifndef METADATA_H
-#define METADATA_H
-
-#include "common.h"
-struct loom;
-struct proc;
-struct thread;
-
-USE_RET int metadata_load_proc(const char *path, struct loom *loom, struct proc *proc);
-USE_RET int metadata_load_thread(const char *path, struct thread *thread);
-
-#endif /* METADATA_H */

src/emu/model.c

@@ -235,13 +235,6 @@ model_finish(struct model *model, struct emu *emu)
 static int
 should_enable(int have[3], struct model_spec *spec, struct thread *t)
 {
-	static int compat = 0;
-
-	/* Enable all models if we are in compatibility model. Don't check other
-	 * threads metadata */
-	if (compat)
-		return 1;
-
 	if (t->meta == NULL) {
 		err("missing metadata for thread %s", t->id);
 		return -1;
@@ -249,12 +242,8 @@ should_enable(int have[3], struct model_spec *spec, struct thread *t)
 
 	JSON_Object *require = json_object_dotget_object(t->meta, "ovni.require");
 	if (require == NULL) {
-		warn("missing 'ovni.require' key in thread %s", t->id);
+		err("missing 'ovni.require' key in thread %s", t->id);
-		warn("loading trace in compatibility mode");
+		return -1;
-		warn("all models will be enabled (expect slowdown)");
-		warn("use ovni_thread_require() to enable only required models");
-		compat = 1;
-		return 1;
 	}
 
 	/* May not have the current model */

src/emu/ovni/setup.c

@@ -13,6 +13,7 @@
 #include "model_cpu.h"
 #include "model_pvt.h"
 #include "model_thread.h"
+#include "ovni.h"
 #include "pv/pcf.h"
 #include "pv/prv.h"
 #include "system.h"
@@ -47,7 +48,7 @@ static struct ev_decl model_evlist[] = {
 
 struct model_spec model_ovni = {
 	.name    = model_name,
-	.version = "1.1.0",
+	.version = OVNI_MODEL_VERSION,
 	.evlist  = model_evlist,
 	.model   = model_id,
 	.create  = model_ovni_create,

src/emu/ovnisort.c

@@ -361,7 +361,7 @@ execute_sort_plan(struct sortplan *sp)
 static int
 stream_winsort(struct stream *stream, struct ring *r)
 {
-	char *fn = stream->path;
+	char *fn = stream->obspath;
 	int fd = open(fn, O_WRONLY);
 
 	if (fd < 0)

src/emu/path.c

@@ -112,3 +112,47 @@ path_filename(const char *path)
 
 	return start;
 }
+
+int
+path_append(char dst[PATH_MAX], const char *src, const char *extra)
+{
+	if (snprintf(dst, PATH_MAX, "%s/%s", src, extra) >= PATH_MAX) {
+		err("path too long: %s/%s", src, extra);
+		return -1;
+	}
+
+	return 0;
+}
+
+/** Copy the path src into dst. */
+int
+path_copy(char dst[PATH_MAX], const char *src)
+{
+	if (snprintf(dst, PATH_MAX, "%s", src) >= PATH_MAX) {
+		err("path too long: %s", src);
+		return -1;
+	}
+
+	return 0;
+}
+
+/** Strip last component from path */
+void
+path_dirname(char path[PATH_MAX])
+{
+	path_remove_trailing(path);
+	int n = (int) strlen(path);
+	int i;
+	for (i = n - 1; i >= 0; i--) {
+		if (path[i] == '/') {
+			break;
+		}
+	}
+	/* Remove all '/' */
+	for (; i >= 0; i--) {
+		if (path[i] != '/')
+			break;
+		else
+			path[i] = '\0';
+	}
+}

src/emu/path.h

@@ -1,9 +1,10 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2021-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #ifndef PATH_H
 #define PATH_H
 
+#include <limits.h>
 #include "common.h"
 
 USE_RET int path_has_prefix(const char *path, const char *prefix);
@@ -13,5 +14,8 @@ USE_RET int path_keep(char *path, int n);
 USE_RET int path_strip(const char *path, int n, const char (**next));
         void path_remove_trailing(char *path);
 USE_RET const char *path_filename(const char *path);
+USE_RET int path_append(char dst[PATH_MAX], const char *src, const char *extra);
+USE_RET int path_copy(char dst[PATH_MAX], const char *src);
+        void path_dirname(char path[PATH_MAX]);
 
 #endif /* PATH_H */

src/emu/proc.c

@@ -1,4 +1,4 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2021-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #include "proc.h"
@@ -6,85 +6,42 @@
 #include <stdlib.h>
 #include <string.h>
 #include "path.h"
+#include "stream.h"
 #include "thread.h"
 
-static int
+int
-get_pid(const char *id, int *pid)
+proc_stream_get_pid(struct stream *s)
 {
-	/* TODO: Store the PID the metadata.json instead */
+	JSON_Object *meta = stream_metadata(s);
 
-	/* The id must be like "loom.host01.123/proc.345" */
+	double pid = json_object_dotget_number(meta, "ovni.pid");
-	if (path_count(id, '/') != 1) {
+
-		err("proc id can only contain one '/': %s", id);
+	/* Zero is used for errors, so forbidden for pid too */
+	if (pid == 0) {
+		err("cannot get attribute ovni.pid for stream: %s",
+				s->relpath);
 		return -1;
 	}
 
-	/* Get the proc.345 part */
+	return (int) pid;
-	const char *procname;
-	if (path_next(id, '/', &procname) != 0) {
-		err("cannot get proc name");
-		return -1;
-	}
-
-	/* Ensure the prefix is ok */
-	const char prefix[] = "proc.";
-	if (!path_has_prefix(procname, prefix)) {
-		err("proc name must start with '%s': %s", prefix, id);
-		return -1;
-	}
-
-	/* Get the 345 part */
-	const char *pidstr;
-	if (path_next(procname, '.', &pidstr) != 0) {
-		err("cannot find proc dot in '%s'", id);
-		return -1;
-	}
-
-	*pid = atoi(pidstr);
-
-	return 0;
 }
 
 int
-proc_relpath_get_pid(const char *relpath, int *pid)
+proc_init_begin(struct proc *proc, int pid)
-{
-	char id[PATH_MAX];
-
-	if (snprintf(id, PATH_MAX, "%s", relpath) >= PATH_MAX) {
-		err("path too long");
-		return -1;
-	}
-
-	if (path_keep(id, 2) != 0) {
-		err("cannot delimite proc dir");
-		return -1;
-	}
-
-	return get_pid(id, pid);
-}
-
-int
-proc_init_begin(struct proc *proc, const char *relpath)
 {
 	memset(proc, 0, sizeof(struct proc));
 
 	proc->gindex = -1;
+	proc->appid = 0;
+	proc->rank = -1;
+	proc->nranks = 0;
+	proc->pid = pid;
 
-	if (snprintf(proc->id, PATH_MAX, "%s", relpath) >= PATH_MAX) {
+	if (snprintf(proc->id, PATH_MAX, "proc.%d", pid) >= PATH_MAX) {
 		err("path too long");
 		return -1;
 	}
 
-	if (path_keep(proc->id, 2) != 0) {
-		err("cannot delimite proc dir");
-		return -1;
-	}
-
-	if (get_pid(proc->id, &proc->pid) != 0) {
-		err("cannot parse proc pid");
-		return -1;
-	}
-
 	dbg("created proc %s", proc->id);
 
 	return 0;
@@ -102,38 +59,105 @@ proc_set_loom(struct proc *proc, struct loom *loom)
 	proc->loom = loom;
 }
 
-int
+static int
-proc_load_metadata(struct proc *proc, JSON_Object *meta)
+load_appid(struct proc *proc, struct stream *s)
 {
-	if (proc->metadata_loaded) {
+	JSON_Object *meta = stream_metadata(s);
-		err("process %s already loaded metadata", proc->id);
+	JSON_Value *appid_val = json_object_dotget_value(meta, "ovni.app_id");
+
+	/* May not be present in all thread streams */
+	if (appid_val == NULL)
+		return 0;
+
+	int appid = (int) json_number(appid_val);
+	if (proc->appid && proc->appid != appid) {
+		err("mismatch previous appid %d with stream: %s",
+				proc->appid, s->relpath);
 		return -1;
 	}
 
-	JSON_Value *version_val = json_object_get_value(meta, "version");
+	if (appid <= 0) {
-	if (version_val == NULL) {
+		err("appid must be >0, stream: %s", s->relpath);
-		err("missing attribute 'version' in metadata");
 		return -1;
 	}
 
-	proc->metadata_version = (int) json_number(version_val);
+	proc->appid = appid;
+	return 0;
+}
 
-	JSON_Value *appid_val = json_object_get_value(meta, "app_id");
+static int
-	if (appid_val == NULL) {
+load_rank(struct proc *proc, struct stream *s)
-		err("missing attribute 'app_id' in metadata");
+{
+	JSON_Object *meta = stream_metadata(s);
+	JSON_Value *rank_val = json_object_dotget_value(meta, "ovni.rank");
+
+	/* Optional */
+	if (rank_val == NULL) {
+		dbg("process %s has no rank", proc->id);
+		return 0;
+	}
+
+	int rank = (int) json_number(rank_val);
+
+	if (rank < 0) {
+		err("rank %d must be >=0, stream: %s", rank, s->relpath);
 		return -1;
 	}
 
-	proc->appid = (int) json_number(appid_val);
+	if (proc->rank >= 0 && proc->rank != rank) {
+		err("mismatch previous rank %d with stream: %s",
+				proc->rank, s->relpath);
+		return -1;
+	}
 
-	JSON_Value *rank_val = json_object_get_value(meta, "rank");
+	/* Same with nranks, but it is not optional now */
+	JSON_Value *nranks_val = json_object_dotget_value(meta, "ovni.nranks");
+	if (nranks_val == NULL) {
+		err("missing ovni.nranks attribute: %s", s->relpath);
+		return -1;
+	}
 
-	if (rank_val != NULL)
+	int nranks = (int) json_number(nranks_val);
-		proc->rank = (int) json_number(rank_val);
-	else
-		proc->rank = -1;
 
-	proc->metadata_loaded = 1;
+	if (nranks <= 0) {
+		err("nranks %d must be >0, stream: %s", nranks, s->relpath);
+		return -1;
+	}
+
+	if (proc->nranks > 0 && proc->nranks != nranks) {
+		err("mismatch previous nranks %d with stream: %s",
+				proc->nranks, s->relpath);
+		return -1;
+	}
+
+	/* Ensure rank fits in nranks */
+	if (rank >= nranks) {
+		err("rank %d must be lower than nranks %d: %s",
+				rank, nranks, s->relpath);
+		return -1;
+	}
+
+	dbg("process %s rank=%d nranks=%d",
+			proc->id, rank, nranks);
+	proc->rank = rank;
+	proc->nranks = nranks;
+
+	return 0;
+}
+
+/** Merges the metadata from the stream in the process. */
+int
+proc_load_metadata(struct proc *proc, struct stream *s)
+{
+	if (load_appid(proc, s) != 0) {
+		err("load_appid failed for stream: %s", s->relpath);
+		return -1;
+	}
+
+	if (load_rank(proc, s) != 0) {
+		err("load_rank failed for stream: %s", s->relpath);
+		return -1;
+	}
 
 	return 0;
 }
@@ -197,8 +221,8 @@ proc_init_end(struct proc *proc)
 		return -1;
 	}
 
-	if (!proc->metadata_loaded) {
+	if (proc->appid <= 0) {
-		err("metadata not loaded");
+		err("appid not set");
 		return -1;
 	}
 

src/emu/proc.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2021-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #ifndef PROC_H
@@ -8,9 +8,9 @@
 #include <stdint.h>
 #include "common.h"
 #include "extend.h"
-#include "parson.h"
 #include "uthash.h"
 struct loom;
+struct stream;
 struct thread;
 
 struct proc {
@@ -18,12 +18,11 @@ struct proc {
 	char id[PATH_MAX];
 	int is_init;
 
-	int metadata_loaded;
-	int metadata_version;
 	int pid;
 	int index;
 	int appid;
 	int rank;
+	int nranks;
 
 	int nthreads;
 	struct thread *threads;
@@ -45,14 +44,14 @@ struct proc {
 	struct extend ext;
 };
 
-USE_RET int proc_relpath_get_pid(const char *relpath, int *pid);
+USE_RET int proc_stream_get_pid(struct stream *s);
-USE_RET int proc_init_begin(struct proc *proc, const char *id);
+USE_RET int proc_init_begin(struct proc *proc, int pid);
 USE_RET int proc_init_end(struct proc *proc);
 USE_RET int proc_get_pid(struct proc *proc);
         void proc_set_gindex(struct proc *proc, int64_t gindex);
         void proc_set_loom(struct proc *proc, struct loom *loom);
         void proc_sort(struct proc *proc);
-USE_RET int proc_load_metadata(struct proc *proc, JSON_Object *meta);
+USE_RET int proc_load_metadata(struct proc *proc, struct stream *s);
 USE_RET struct thread *proc_find_thread(struct proc *proc, int tid);
 USE_RET int proc_add_thread(struct proc *proc, struct thread *thread);
         void proc_sort(struct proc *proc);

src/emu/stream.c

@@ -71,41 +71,22 @@ load_stream_fd(struct stream *stream, int fd)
 	return 0;
 }
 
-int
+static int
-stream_load(struct stream *stream, const char *tracedir, const char *relpath)
+load_obs(struct stream *stream, const char *path)
 {
-	memset(stream, 0, sizeof(struct stream));
-
-	if (snprintf(stream->path, PATH_MAX, "%s/%s", tracedir, relpath) >= PATH_MAX) {
-		err("path too long: %s/%s", tracedir, relpath);
-		return -1;
-	}
-
-	/* Allow loading a trace with empty relpath */
-	path_remove_trailing(stream->path);
-
-	if (snprintf(stream->relpath, PATH_MAX, "%s", relpath) >= PATH_MAX) {
-		err("path too long: %s", relpath);
-		return -1;
-	}
-
-	dbg("loading %s", stream->relpath);
-
 	int fd;
-	if ((fd = open(stream->path, O_RDWR)) == -1) {
+	if ((fd = open(path, O_RDWR)) == -1) {
-		err("open %s failed:", stream->path);
+		err("open %s failed:", path);
 		return -1;
 	}
 
 	if (load_stream_fd(stream, fd) != 0) {
-		err("load_stream_fd failed for stream '%s'",
+		err("load_stream_fd failed for: %s", path);
-				stream->path);
 		return -1;
 	}
 
 	if (check_stream_header(stream) != 0) {
-		err("stream '%s' has bad header",
+		err("stream has bad header: %s", path);
-				stream->path);
 		return -1;
 	}
 
@@ -132,6 +113,97 @@ stream_load(struct stream *stream, const char *tracedir, const char *relpath)
 	return 0;
 }
 
+static int
+check_version(JSON_Object *meta)
+{
+	JSON_Value *version_val = json_object_get_value(meta, "version");
+	if (version_val == NULL) {
+		err("missing attribute \"version\"");
+		return -1;
+	}
+
+	int version = (int) json_number(version_val);
+
+	if (version != OVNI_METADATA_VERSION) {
+		err("metadata version mismatch %d (expected %d)",
+				version, OVNI_METADATA_VERSION);
+		return -1;
+	}
+
+	return 0;
+}
+
+static JSON_Object *
+load_json(const char *path)
+{
+	JSON_Value *vmeta = json_parse_file_with_comments(path);
+	if (vmeta == NULL) {
+		err("json_parse_file_with_comments() failed");
+		return NULL;
+	}
+
+	JSON_Object *meta = json_value_get_object(vmeta);
+	if (meta == NULL) {
+		err("json_value_get_object() failed");
+		return NULL;
+	}
+
+	if (check_version(meta) != 0) {
+		err("check_version failed");
+		return NULL;
+	}
+
+	return meta;
+}
+
+/** Loads a stream from disk.
+ *
+ * The relpath must be pointing to a directory with the stream.json and
+ * stream.obs files.
+ */
+int
+stream_load(struct stream *stream, const char *tracedir, const char *relpath)
+{
+	memset(stream, 0, sizeof(struct stream));
+
+	if (snprintf(stream->path, PATH_MAX, "%s/%s", tracedir, relpath) >= PATH_MAX) {
+		err("path too long: %s/%s", tracedir, relpath);
+		return -1;
+	}
+
+	/* Allow loading a trace with empty relpath */
+	path_remove_trailing(stream->path);
+
+	if (snprintf(stream->relpath, PATH_MAX, "%s", relpath) >= PATH_MAX) {
+		err("path too long: %s", relpath);
+		return -1;
+	}
+
+	dbg("loading %s", stream->relpath);
+
+	if (path_append(stream->jsonpath, stream->path, "stream.json") != 0) {
+		err("path_append failed");
+		return -1;
+	}
+
+	if ((stream->meta = load_json(stream->jsonpath)) == NULL) {
+		err("load_json failed for: %s", stream->jsonpath);
+		return -1;
+	}
+
+	if (path_append(stream->obspath, stream->path, "stream.obs") != 0) {
+		err("path_append failed");
+		return -1;
+	}
+
+	if (load_obs(stream, stream->obspath) != 0) {
+		err("load_obs failed");
+		return -1;
+	}
+
+	return 0;
+}
+
 void
 stream_data_set(struct stream *stream, void *data)
 {
@@ -144,6 +216,16 @@ stream_data_get(struct stream *stream)
 	return stream->data;
 }
 
+/* Is never NULL */
+JSON_Object *
+stream_metadata(struct stream *stream)
+{
+	if (stream->meta == NULL)
+		die("stream metadata is NULL: %s", stream->relpath);
+
+	return stream->meta;
+}
+
 int
 stream_clkoff_set(struct stream *stream, int64_t clkoff)
 {

src/emu/stream.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2021-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #ifndef STREAM_H
@@ -8,6 +8,7 @@
 #include <stdint.h>
 #include "common.h"
 #include "heap.h"
+#include "parson.h"
 struct ovni_ev;
 
 struct stream {
@@ -27,13 +28,17 @@ struct stream {
 	int active;
 	int unsorted;
 
-	char path[PATH_MAX];
+	char path[PATH_MAX]; /* To stream dir */
 	char relpath[PATH_MAX]; /* To tracedir */
+	char obspath[PATH_MAX]; /* To obs file */
+	char jsonpath[PATH_MAX]; /* To json file */
 
 	int64_t usize; /* Useful size for events */
 	int64_t offset;
 
 	double progress;
+
+	JSON_Object *meta;
 };
 
 USE_RET int stream_load(struct stream *stream, const char *tracedir, const char *relpath);
@@ -46,5 +51,6 @@ USE_RET int64_t stream_lastclock(struct stream *stream);
         void stream_allow_unsorted(struct stream *stream);
         void stream_data_set(struct stream *stream, void *data);
 USE_RET void *stream_data_get(struct stream *stream);
+USE_RET JSON_Object *stream_metadata(struct stream *stream);
 
 #endif /* STREAM_H */

src/emu/system.c

@@ -11,7 +11,6 @@
 #include "cpu.h"
 #include "emu_args.h"
 #include "loom.h"
-#include "metadata.h"
 #include "proc.h"
 #include "pv/prf.h"
 #include "pv/pvt.h"
@@ -24,11 +23,11 @@
 struct bay;
 
 static struct thread *
-create_thread(struct proc *proc, const char *tracedir, const char *relpath)
+create_thread(struct proc *proc, struct stream *s)
 {
 	int tid;
-	if (thread_relpath_get_tid(relpath, &tid) != 0) {
+	if ((tid = thread_stream_get_tid(s)) < 0) {
-		err("cannot get thread tid from %s", relpath);
+		err("cannot get thread tid from stream: %s", s->relpath);
 		return NULL;
 	}
 
@@ -45,21 +44,13 @@ create_thread(struct proc *proc, const char *tracedir, const char *relpath)
 		return NULL;
 	}
 
-	if (thread_init_begin(thread, relpath) != 0) {
+	if (thread_init_begin(thread, tid) != 0) {
-		err("cannot init thread");
+		err("thread_init_begin failed: %s", s->relpath);
 		return NULL;
 	}
 
-	/* Build metadata path */
+	if (thread_load_metadata(thread, s) != 0) {
-	char mpath[PATH_MAX];
+		err("thread_load_metadata failed: %s", s->relpath);
-	if (snprintf(mpath, PATH_MAX, "%s/%s/thread.%d.json",
-				tracedir, proc->id, tid) >= PATH_MAX) {
-		err("path too long");
-		return NULL;
-	}
-
-	if (metadata_load_thread(mpath, thread) != 0) {
-		err("cannot load metadata from %s", mpath);
 		return NULL;
 	}
 
@@ -72,18 +63,17 @@ create_thread(struct proc *proc, const char *tracedir, const char *relpath)
 }
 
 static struct proc *
-create_proc(struct loom *loom, const char *tracedir, const char *relpath)
+create_proc(struct loom *loom, struct stream *s)
 {
-	int pid;
+	int pid = proc_stream_get_pid(s);
-	if (proc_relpath_get_pid(relpath, &pid) != 0) {
+	if (pid < 0) {
-		err("cannot get proc pid from %s", relpath);
+		err("cannot get proc pid from stream: %s", s->relpath);
 		return NULL;
 	}
 
 	struct proc *proc = loom_find_proc(loom, pid);
-
+	if (proc == NULL) {
-	if (proc != NULL)
+		/* Create a new process */
-		return proc;
 
 		proc = malloc(sizeof(struct proc));
 
@@ -92,23 +82,8 @@ create_proc(struct loom *loom, const char *tracedir, const char *relpath)
 			return NULL;
 		}
 
-	if (proc_init_begin(proc, relpath) != 0) {
+		if (proc_init_begin(proc, pid) != 0) {
-		err("proc_init_begin failed");
+			err("proc_init_begin failed: %s", s->relpath);
-		return NULL;
-	}
-
-	/* Build metadata path */
-	char mpath[PATH_MAX];
-
-	if (snprintf(mpath, PATH_MAX, "%s/%s/metadata.json",
-				tracedir, proc->id) >= PATH_MAX) {
-		err("path too long");
-		return NULL;
-	}
-
-	/* Load metadata too */
-	if (metadata_load_proc(mpath, loom, proc) != 0) {
-		err("cannot load metadata from %s", mpath);
 			return NULL;
 		}
 
@@ -116,6 +91,13 @@ create_proc(struct loom *loom, const char *tracedir, const char *relpath)
 			err("loom_add_proc failed");
 			return NULL;
 		}
+	}
+
+	/* The appid is populated from the metadata */
+	if (proc_load_metadata(proc, s) != 0) {
+		err("proc_load_metadata failed");
+		return NULL;
+	}
 
 	return proc;
 }
@@ -131,16 +113,11 @@ find_loom(struct system *sys, const char *id)
 }
 
 static struct loom *
-create_loom(struct system *sys, const char *relpath)
+create_loom(struct system *sys, struct stream *s)
 {
-	char name[PATH_MAX];
+	const char *name = loom_name(s);
-	if (snprintf(name, PATH_MAX, "%s", relpath) >= PATH_MAX) {
+	if (name == NULL) {
-		err("path too long: %s", relpath);
+		err("loom_name failed");
-		return NULL;
-	}
-
-	if (strtok(name, "/") == NULL) {
-		err("cannot find first '/': %s", relpath);
 		return NULL;
 	}
 
@@ -163,6 +140,11 @@ create_loom(struct system *sys, const char *relpath)
 		sys->nlooms++;
 	}
 
+	if (loom_load_metadata(loom, s) != 0) {
+		err("loom_load_metadata failed for stream: %s", s->relpath);
+		return NULL;
+	}
+
 	return loom;
 }
 
@@ -220,11 +202,33 @@ report_libovni_version(struct system *sys)
 	return 0;
 }
 
+static int
+is_thread_stream(struct stream *s)
+{
+	JSON_Object *meta = stream_metadata(s);
+	if (meta == NULL) {
+		err("no metadata for stream: %s", s->relpath);
+		return -1;
+	}
+
+	/* All streams must have a ovni.part attribute */
+	const char *part_type = json_object_dotget_string(meta, "ovni.part");
+
+	if (part_type == NULL) {
+		err("cannot get attribute ovni.part for stream: %s",
+				s->relpath);
+		return -1;
+	}
+
+	if (strcmp(part_type, "thread") == 0)
+		return 1;
+
+	return 0;
+}
+
 static int
 create_system(struct system *sys, struct trace *trace)
 {
-	const char *dir = trace->tracedir;
-
 	/* Allocate the lpt map */
 	sys->lpt = calloc((size_t) trace->nstreams, sizeof(struct lpt));
 	if (sys->lpt == NULL) {
@@ -234,25 +238,28 @@ create_system(struct system *sys, struct trace *trace)
 
 	size_t i = 0;
 	for (struct stream *s = trace->streams; s ; s = s->next) {
-		if (!loom_matches(s->relpath)) {
+		int ok = is_thread_stream(s);
+		if (ok < 0) {
+			err("is_thread_stream failed");
+			return -1;
+		} else if (ok == 0) {
 			warn("ignoring unknown stream %s", s->relpath);
 			continue;
 		}
 
-		struct loom *loom = create_loom(sys, s->relpath);
+		struct loom *loom = create_loom(sys, s);
 		if (loom == NULL) {
 			err("create_loom failed");
 			return -1;
 		}
 
-		/* Loads metadata too */
+		struct proc *proc = create_proc(loom, s);
-		struct proc *proc = create_proc(loom, dir, s->relpath);
 		if (proc == NULL) {
 			err("create_proc failed");
 			return -1;
 		}
 
-		struct thread *thread = create_thread(proc, dir, s->relpath);
+		struct thread *thread = create_thread(proc, s);
 		if (thread == NULL) {
 			err("create_thread failed");
 			return -1;
@@ -267,14 +274,6 @@ create_system(struct system *sys, struct trace *trace)
 		stream_data_set(s, lpt);
 	}
 
-	/* Ensure all looms have at least one CPU */
-	for (struct loom *l = sys->looms; l; l = l->next) {
-		if (l->ncpus == 0) {
-			err("loom %s has no physical CPUs", l->id);
-			return -1;
-		}
-	}
-
 	return 0;
 }
 
@@ -552,6 +551,12 @@ set_sort_criteria(struct system *sys)
 	int some_have = 0;
 	int all_have = 1;
 	for (struct loom *l = sys->looms; l; l = l->next) {
+		/* Set the rank_min for later sorting */
+		if (loom_set_rank_min(l) != 0) {
+			err("loom_set_rank_min failed");
+			return -1;
+		}
+
 		if (l->rank_enabled)
 			some_have = 1;
 		else

src/emu/thread.c

@@ -15,6 +15,7 @@
 #include "pv/prv.h"
 #include "pv/pvt.h"
 #include "recorder.h"
+#include "stream.h"
 #include "value.h"
 struct proc;
 
@@ -59,75 +60,37 @@ static const struct pcf_value_label (*pcf_labels[TH_CHAN_MAX])[] = {
 	[TH_CHAN_STATE] = &state_name,
 };
 
-static int
+int
-get_tid(const char *id, int *tid)
+thread_stream_get_tid(struct stream *s)
 {
-	/* The id must be like "loom.host01.123/proc.345/thread.567" */
+	JSON_Object *meta = stream_metadata(s);
-	if (path_count(id, '/') != 2) {
+
-		err("proc id can only contain two '/': %s", id);
+	double tid = json_object_dotget_number(meta, "ovni.tid");
+
+	/* Zero is used for errors, so forbidden for tid too */
+	if (tid == 0) {
+		err("cannot get attribute ovni.tid for stream: %s",
+				s->relpath);
 		return -1;
 	}
 
-	/* Get the thread.567 part */
+	return (int) tid;
-	const char *thname;
-	if (path_strip(id, 2, &thname) != 0) {
-		err("cannot get thread name");
-		return -1;
-	}
-
-	/* Ensure the prefix is ok */
-	const char prefix[] = "thread.";
-	if (!path_has_prefix(thname, prefix)) {
-		err("thread name must start with '%s': %s", prefix, thname);
-		return -1;
-	}
-
-	/* Get the 567 part */
-	const char *tidstr;
-	if (path_next(thname, '.', &tidstr) != 0) {
-		err("cannot find thread dot in '%s'", id);
-		return -1;
-	}
-
-	char *endptr;
-	errno = 0;
-	*tid = (int) strtol(tidstr, &endptr, 10);
-	if (errno != 0) {
-		err("strtol failed for '%s':", tidstr);
-		return -1;
-	}
-	if (endptr == tidstr) {
-		err("no digits in tid string '%s'", tidstr);
-		return -1;
-	}
-
-	return 0;
 }
 
 int
-thread_relpath_get_tid(const char *relpath, int *tid)
+thread_init_begin(struct thread *thread, int tid)
-{
-	return get_tid(relpath, tid);
-}
-
-int
-thread_init_begin(struct thread *thread, const char *relpath)
 {
 	memset(thread, 0, sizeof(struct thread));
 
 	thread->state = TH_ST_UNKNOWN;
 	thread->gindex = -1;
+	thread->tid = tid;
 
-	if (snprintf(thread->id, PATH_MAX, "%s", relpath) >= PATH_MAX) {
+	if (snprintf(thread->id, PATH_MAX, "thread.%d", tid) >= PATH_MAX) {
 		err("relpath too long");
 		return -1;
 	}
 
-	if (get_tid(thread->id, &thread->tid) != 0) {
-		err("cannot parse thread tid");
-		return -1;
-	}
-
 	return 0;
 }
 
@@ -445,20 +408,19 @@ thread_migrate_cpu(struct thread *th, struct cpu *cpu)
 }
 
 int
-thread_load_metadata(struct thread *thread, JSON_Object *meta)
+thread_load_metadata(struct thread *thread, struct stream *s)
 {
-	if (meta == NULL) {
+	JSON_Object *meta = stream_metadata(s);
-		err("metadata is null");
-		return -1;
-	}
 
 	if (thread->meta != NULL) {
 		err("thread %s already loaded metadata", thread->id);
 		return -1;
 	}
 
-	if (json_object_dotget_number(meta, "ovni.finished") != 1)
+	if (json_object_dotget_number(meta, "ovni.finished") != 1) {
-		warn("thread didn't finish properly: %s", thread->id);
+		err("missing ovni.finished: %s", s->relpath);
+		return -1;
+	}
 
 	thread->meta = meta;
 

src/emu/thread.h

@@ -20,6 +20,7 @@ struct pcf;
 struct proc;
 struct recorder;
 struct value;
+struct stream;
 
 /* Emulated thread runtime status */
 enum thread_state {
@@ -82,10 +83,10 @@ struct thread {
 	UT_hash_handle hh; /* threads in the process */
 };
 
-USE_RET int thread_relpath_get_tid(const char *relpath, int *tid);
+USE_RET int thread_stream_get_tid(struct stream *s);
-USE_RET int thread_init_begin(struct thread *thread, const char *relpath);
+USE_RET int thread_init_begin(struct thread *thread, int tid);
 USE_RET int thread_init_end(struct thread *thread);
-USE_RET int thread_load_metadata(struct thread *thread, JSON_Object *meta);
+USE_RET int thread_load_metadata(struct thread *thread, struct stream *s);
 USE_RET int thread_set_state(struct thread *th, enum thread_state state);
 USE_RET int thread_set_cpu(struct thread *th, struct cpu *cpu);
 USE_RET int thread_unset_cpu(struct thread *th);

src/emu/trace.c

@@ -27,7 +27,7 @@ add_stream(struct trace *trace, struct stream *stream)
 }
 
 static int
-load_stream(struct trace *trace, const char *path)
+load_stream(struct trace *trace, const char *json_path)
 {
 	struct stream *stream = calloc(1, sizeof(struct stream));
 
@@ -36,6 +36,14 @@ load_stream(struct trace *trace, const char *path)
 		return -1;
 	}
 
+	/* The json_path must end in .../stream.json, so remove it */
+	char path[PATH_MAX];
+	if (path_copy(path, json_path) != 0) {
+		err("path_copy failed");
+		return -1;
+	}
+	path_dirname(path);
+
 	int offset = (int) strlen(trace->tracedir);
 	const char *relpath = path + offset;
 
@@ -52,45 +60,15 @@ load_stream(struct trace *trace, const char *path)
 	return 0;
 }
 
-static int
-has_suffix(const char *str, const char *suffix)
-{
-	if (!str || !suffix)
-		return 0;
-
-	int lenstr = (int) strlen(str);
-	int lensuffix = (int) strlen(suffix);
-
-	if (lensuffix > lenstr)
-		return 0;
-
-	const char *p = str + lenstr - lensuffix;
-	if (strncmp(p, suffix, (size_t) lensuffix) == 0)
-		return 1;
-
-	return 0;
-}
-
 static int
 is_stream(const char *fpath)
 {
-	if (has_suffix(fpath, OVNI_STREAM_EXT))
-		return 1;
-
-	/* For compatibility load the old streams too */
 	const char *filename = path_filename(fpath);
 
-	const char prefix[] = "thread.";
+	if (strcmp(filename, "stream.json") == 0)
-	if (!path_has_prefix(filename, prefix))
-		return 0;
-
-	const char *tid = filename + strlen(prefix);
-	for (int i = 0; tid[i]; i++) {
-		if (tid[i] < '0' || tid[i] > '9')
-			return 0;
-	}
-
 		return 1;
+
+	return 0;
 }
 
 static int
@@ -127,6 +105,10 @@ trace_load(struct trace *trace, const char *tracedir)
 		return -1;
 	}
 
+	/* Remove trailing slashes from tracedir */
+	path_remove_trailing(trace->tracedir);
+	tracedir = trace->tracedir;
+
 	/* Try to open the directory to catch permission errors */
 	DIR *dir = opendir(tracedir);
 	if (dir == NULL) {

src/rt/ovni.c

@@ -5,6 +5,8 @@
 #include <fcntl.h>
 #include <inttypes.h>
 #include <limits.h>
+#include <stdatomic.h>
+#include <stdbool.h>
 #include <stdint.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -17,6 +19,21 @@
 #include "ovni.h"
 #include "parson.h"
 #include "version.h"
+#include "utlist.h"
+
+enum {
+	ST_UNINIT = 0,
+	ST_INIT,
+	ST_READY,
+	ST_GONE,
+};
+
+struct ovni_rcpu {
+	int index;
+	int phyid;
+	struct ovni_rcpu *next;
+	struct ovni_rcpu *prev;
+};
 
 /* State of each thread on runtime */
 struct ovni_rthread {
@@ -35,6 +52,16 @@ struct ovni_rthread {
 	/* Buffer to write events */
 	uint8_t *evbuf;
 
+	struct ovni_rcpu *cpus;
+
+	int rank_set;
+	int rank;
+	int nranks;
+
+	/* Where the stream dir is finally copied */
+	char thdir_final[PATH_MAX];
+	char thdir[PATH_MAX];
+
 	JSON_Value *meta;
 };
 
@@ -54,10 +81,9 @@ struct ovni_rproc {
 	int app;
 	int pid;
 	char loom[OVNI_MAX_HOSTNAME];
-	int ncpus;
 	clockid_t clockid;
 
-	int ready;
+	atomic_int st;
 
 	JSON_Value *meta;
 };
@@ -104,17 +130,40 @@ void ovni_version_check_str(const char *version)
 	/* Ignore the patch number */
 }
 
+/* Create dir $procdir/thread.$tid and return it in path. */
+static void
+mkdir_thread(char *path, const char *procdir, int tid)
+{
+	if (snprintf(path, PATH_MAX, "%s/thread.%d",
+				procdir, tid) >= PATH_MAX) {
+		die("path too long: %s/thread.%d", procdir, tid);
+	}
+
+	if (mkpath(path, 0755, /* subdir */ 1))
+		die("mkpath %s failed:", path);
+}
+
+static void
+create_thread_dir(int tid)
+{
+	/* The procdir must have been created earlier */
+	mkdir_thread(rthread.thdir, rproc.procdir, tid);
+	if (rproc.move_to_final)
+		mkdir_thread(rthread.thdir_final, rproc.procdir_final, tid);
+}
+
 static void
 create_trace_stream(void)
 {
 	char path[PATH_MAX];
 
-	int written = snprintf(path, PATH_MAX, "%s/thread.%d%s",
+	int written = snprintf(path, PATH_MAX, "%s/thread.%d/stream.obs",
-			rproc.procdir, rthread.tid, OVNI_STREAM_EXT);
+			rproc.procdir, rthread.tid);
 
-	if (written >= PATH_MAX)
+	if (written >= PATH_MAX) {
-		die("thread trace path too long: %s/thread.%d%s",
+		die("path too long: %s/thread.%d/stream.obs",
-				rproc.procdir, rthread.tid, OVNI_STREAM_EXT);
+				rproc.procdir, rthread.tid);
+	}
 
 	rthread.streamfd = open(path, O_WRONLY | O_CREAT, 0644);
 
@@ -122,31 +171,6 @@ create_trace_stream(void)
 		die("open %s failed:", path);
 }
 
-static void
-proc_metadata_init(struct ovni_rproc *proc)
-{
-	proc->meta = json_value_init_object();
-
-	if (proc->meta == NULL)
-		die("failed to create metadata JSON object");
-}
-
-static void
-proc_metadata_store(JSON_Value *meta, const char *procdir)
-{
-	char path[PATH_MAX];
-
-	if (meta == NULL)
-		die("process metadata not initialized");
-
-	if (snprintf(path, PATH_MAX, "%s/metadata.json", procdir) >= PATH_MAX)
-		die("metadata path too long: %s/metadata.json",
-				procdir);
-
-	if (json_serialize_to_file_pretty(meta, path) != JSONSuccess)
-		die("failed to write process metadata");
-}
-
 void
 ovni_add_cpu(int index, int phyid)
 {
@@ -156,101 +180,34 @@ ovni_add_cpu(int index, int phyid)
 	if (phyid < 0)
 		die("cannot use negative CPU id %d", phyid);
 
-	if (!rproc.ready)
+	if (atomic_load(&rproc.st) != ST_READY)
-		die("process not yet initialized");
+		die("process not ready");
 
-	if (rproc.meta == NULL)
+	if (!rthread.ready)
-		die("metadata not initialized");
+		die("thread not yet initialized");
 
-	JSON_Object *meta = json_value_get_object(rproc.meta);
+	struct ovni_rcpu *cpu = malloc(sizeof(*cpu));
-
-	if (meta == NULL)
-		die("json_value_get_object() failed");
-
-	int first_time = 0;
-
-	/* Find the CPU array and create it if needed */
-	JSON_Array *cpuarray = json_object_dotget_array(meta, "cpus");
-
-	if (cpuarray == NULL) {
-		JSON_Value *value = json_value_init_array();
-		if (value == NULL)
-			die("json_value_init_array() failed");
-
-		cpuarray = json_array(value);
-		if (cpuarray == NULL)
-			die("json_array() failed");
-
-		first_time = 1;
-	}
-
-	JSON_Value *valcpu = json_value_init_object();
-	if (valcpu == NULL)
-		die("json_value_init_object() failed");
-
-	JSON_Object *cpu = json_object(valcpu);
 	if (cpu == NULL)
-		die("json_object() failed");
+		die("malloc failed:");
 
-	if (json_object_set_number(cpu, "index", index) != 0)
+	cpu->index = index;
-		die("json_object_set_number() failed");
+	cpu->phyid = phyid;
 
-	if (json_object_set_number(cpu, "phyid", phyid) != 0)
+	DL_APPEND(rthread.cpus, cpu);
-		die("json_object_set_number() failed");
-
-	if (json_array_append_value(cpuarray, valcpu) != 0)
-		die("json_array_append_value() failed");
-
-	if (first_time) {
-		JSON_Value *value = json_array_get_wrapping_value(cpuarray);
-		if (value == NULL)
-			die("json_array_get_wrapping_value() failed");
-
-		if (json_object_set_value(meta, "cpus", value) != 0)
-			die("json_object_set_value failed");
-	}
-}
-
-static void
-proc_set_app(int appid)
-{
-	JSON_Object *meta = json_value_get_object(rproc.meta);
-
-	if (meta == NULL)
-		die("json_value_get_object failed");
-
-	if (json_object_set_number(meta, "app_id", appid) != 0)
-		die("json_object_set_number for app_id failed");
-}
-
-static void
-proc_set_version(void)
-{
-	JSON_Object *meta = json_value_get_object(rproc.meta);
-
-	if (meta == NULL)
-		die("json_value_get_object failed");
-
-	if (json_object_set_number(meta, "version", OVNI_METADATA_VERSION) != 0)
-		die("json_object_set_number for version failed");
 }
 
 void
 ovni_proc_set_rank(int rank, int nranks)
 {
-	if (!rproc.ready)
+	if (atomic_load(&rproc.st) != ST_READY)
-		die("process not yet initialized");
+		die("process not ready");
 
-	JSON_Object *meta = json_value_get_object(rproc.meta);
+	if (!rthread.ready)
+		die("thread not yet initialized");
 
-	if (meta == NULL)
+	rthread.rank_set = 1;
-		die("json_value_get_object failed");
+	rthread.rank = rank;
-
+	rthread.nranks = nranks;
-	if (json_object_set_number(meta, "rank", rank) != 0)
-		die("json_object_set_number for rank failed");
-
-	if (json_object_set_number(meta, "nranks", nranks) != 0)
-		die("json_object_set_number for nranks failed");
 }
 
 /* Create $tracedir/loom.$loom/proc.$pid and return it in path. */
@@ -292,10 +249,19 @@ create_proc_dir(const char *loom, int pid)
 void
 ovni_proc_init(int app, const char *loom, int pid)
 {
-	if (rproc.ready)
+	/* Protect against two threads calling at the same time */
-		die("pid %d already initialized", pid);
+	int st = ST_UNINIT;
+	bool was_uninit = atomic_compare_exchange_strong(&rproc.st,
+			&st, ST_INIT);
 
-	memset(&rproc, 0, sizeof(rproc));
+	if (!was_uninit) {
+		if (st == ST_INIT)
+			die("pid %d already being initialized", pid);
+		else if (st == ST_READY)
+			die("pid %d already initialized", pid);
+		else if (st == ST_GONE)
+			die("pid %d has finished, cannot init again", pid);
+	}
 
 	if (strlen(loom) >= OVNI_MAX_HOSTNAME)
 		die("loom name too long: %s", loom);
@@ -307,17 +273,13 @@ ovni_proc_init(int app, const char *loom, int pid)
 
 	create_proc_dir(loom, pid);
 
-	proc_metadata_init(&rproc);
+	atomic_store(&rproc.st, ST_READY);
-
-	rproc.ready = 1;
-
-	proc_set_version();
-	proc_set_app(app);
 }
 
 static int
 move_thread_to_final(const char *src, const char *dst)
 {
+	info("moving src=%s to dst=%s", src, dst);
 	char buffer[1024];
 
 	FILE *infile = fopen(src, "r");
@@ -350,38 +312,38 @@ move_thread_to_final(const char *src, const char *dst)
 }
 
 static void
-move_procdir_to_final(const char *procdir, const char *procdir_final)
+move_thdir_to_final(const char *thdir, const char *thdir_final)
 {
 	DIR *dir;
 	int ret = 0;
 
-	if ((dir = opendir(procdir)) == NULL) {
+	if ((dir = opendir(thdir)) == NULL) {
-		err("opendir %s failed:", procdir);
+		err("opendir %s failed:", thdir);
 		return;
 	}
 
 	struct dirent *dirent;
-	const char *prefix = "thread.";
+	const char *prefix = "stream.";
 	while ((dirent = readdir(dir)) != NULL) {
-		/* It should only contain thread.* directories, skip others */
+		/* It should only contain stream.* directories, skip others */
 		if (strncmp(dirent->d_name, prefix, strlen(prefix)) != 0)
 			continue;
 
 		char thread[PATH_MAX];
-		if (snprintf(thread, PATH_MAX, "%s/%s", procdir,
+		if (snprintf(thread, PATH_MAX, "%s/%s", thdir,
 				    dirent->d_name)
 				>= PATH_MAX) {
-			err("snprintf: path too large: %s/%s", procdir,
+			err("snprintf: path too large: %s/%s", thdir,
 					dirent->d_name);
 			ret = 1;
 			continue;
 		}
 
 		char thread_final[PATH_MAX];
-		if (snprintf(thread_final, PATH_MAX, "%s/%s", procdir_final,
+		if (snprintf(thread_final, PATH_MAX, "%s/%s", thdir_final,
 				    dirent->d_name)
 				>= PATH_MAX) {
-			err("snprintf: path too large: %s/%s", procdir_final,
+			err("snprintf: path too large: %s/%s", thdir_final,
 					dirent->d_name);
 			ret = 1;
 			continue;
@@ -395,7 +357,7 @@ move_procdir_to_final(const char *procdir, const char *procdir_final)
 
 	/* Warn the user, but we cannot do much at this point */
 	if (ret)
-		err("errors occurred when moving the trace to %s", procdir_final);
+		err("errors occurred when moving the thread dir to %s", thdir_final);
 }
 
 static void
@@ -409,20 +371,18 @@ try_clean_dir(const char *dir)
 void
 ovni_proc_fini(void)
 {
-	if (!rproc.ready)
+	/* Protect against two threads calling at the same time */
-		die("process not initialized");
+	int st = ST_READY;
+	bool was_ready = atomic_compare_exchange_strong(&rproc.st,
+			&st, ST_GONE);
 
-	/* Mark the process no longer ready */
+	if (!was_ready)
-	rproc.ready = 0;
+		die("process not ready");
 
 	if (rproc.move_to_final) {
-		proc_metadata_store(rproc.meta, rproc.procdir_final);
-		move_procdir_to_final(rproc.procdir, rproc.procdir_final);
 		try_clean_dir(rproc.procdir);
 		try_clean_dir(rproc.loomdir);
 		try_clean_dir(rproc.tmpdir);
-	} else {
-		proc_metadata_store(rproc.meta, rproc.procdir);
 	}
 }
 
@@ -465,11 +425,11 @@ static void
 thread_metadata_store(void)
 {
 	char path[PATH_MAX];
-	int written = snprintf(path, PATH_MAX, "%s/thread.%d.json",
+	int written = snprintf(path, PATH_MAX, "%s/thread.%d/stream.json",
 			rproc.procdir, rthread.tid);
 
 	if (written >= PATH_MAX)
-		die("thread trace path too long: %s/thread.%d.json",
+		die("thread trace path too long: %s/thread.%d/stream.json",
 				rproc.procdir, rthread.tid);
 
 	if (json_serialize_to_file_pretty(rthread.meta, path) != JSONSuccess)
@@ -530,6 +490,21 @@ thread_metadata_populate(void)
 
 	if (json_object_dotset_string(meta, "ovni.lib.commit", OVNI_GIT_COMMIT) != 0)
 		die("json_object_dotset_string failed");
+
+	if (json_object_dotset_string(meta, "ovni.part", "thread") != 0)
+		die("json_object_dotset_string failed");
+
+	if (json_object_dotset_number(meta, "ovni.tid", (double) rthread.tid) != 0)
+		die("json_object_dotset_number failed");
+
+	if (json_object_dotset_number(meta, "ovni.pid", (double) rproc.pid) != 0)
+		die("json_object_dotset_number failed");
+
+	if (json_object_dotset_string(meta, "ovni.loom", rproc.loom) != 0)
+		die("json_object_dotset_string failed");
+
+	if (json_object_dotset_number(meta, "ovni.app_id", rproc.app) != 0)
+		die("json_object_dotset_number for ovni.app_id failed");
 }
 
 static void
@@ -560,8 +535,8 @@ ovni_thread_init(pid_t tid)
 	if (tid == 0)
 		die("cannot use tid=%d", tid);
 
-	if (!rproc.ready)
+	if (atomic_load(&rproc.st) != ST_READY)
-		die("process not yet initialized");
+		die("process not ready");
 
 	memset(&rthread, 0, sizeof(rthread));
 
@@ -572,12 +547,59 @@ ovni_thread_init(pid_t tid)
 	if (rthread.evbuf == NULL)
 		die("malloc failed:");
 
+	create_thread_dir(tid);
 	create_trace_stream();
 	write_stream_header();
 
 	thread_metadata_init();
 
 	rthread.ready = 1;
+
+	ovni_thread_require("ovni", OVNI_MODEL_VERSION);
+}
+
+static void
+set_thread_rank(JSON_Object *meta)
+{
+	if (json_object_dotset_number(meta, "ovni.rank", rthread.rank) != 0)
+		die("json_object_set_number for rank failed");
+
+	if (json_object_dotset_number(meta, "ovni.nranks", rthread.nranks) != 0)
+		die("json_object_set_number for nranks failed");
+}
+
+static void
+set_thread_cpus(JSON_Object *meta)
+{
+	JSON_Value *value = json_value_init_array();
+	if (value == NULL)
+		die("json_value_init_array() failed");
+
+	JSON_Array *cpuarray = json_array(value);
+	if (cpuarray == NULL)
+		die("json_array() failed");
+
+	for (struct ovni_rcpu *c = rthread.cpus; c; c = c->next) {
+		JSON_Value *valcpu = json_value_init_object();
+		if (valcpu == NULL)
+			die("json_value_init_object() failed");
+
+		JSON_Object *cpu = json_object(valcpu);
+		if (cpu == NULL)
+			die("json_object() failed");
+
+		if (json_object_set_number(cpu, "index", c->index) != 0)
+			die("json_object_set_number() failed");
+
+		if (json_object_set_number(cpu, "phyid", c->phyid) != 0)
+			die("json_object_set_number() failed");
+
+		if (json_array_append_value(cpuarray, valcpu) != 0)
+			die("json_array_append_value() failed");
+	}
+
+	if (json_object_dotset_value(meta, "ovni.loom_cpus", value) != 0)
+		die("json_object_dotset_value failed");
 }
 
 void
@@ -594,6 +616,14 @@ ovni_thread_free(void)
 	if (meta == NULL)
 		die("json_value_get_object failed");
 
+	if (rthread.rank_set)
+		set_thread_rank(meta);
+
+	/* It can happen there are no CPUs defined if there is another
+	 * process in the loom that defines them. */
+	if (rthread.cpus)
+		set_thread_cpus(meta);
+
 	/* Mark it finished so we can detect partial streams */
 	if (json_object_dotset_number(meta, "ovni.finished", 1) != 0)
 		die("json_object_dotset_string failed");
@@ -606,8 +636,14 @@ ovni_thread_free(void)
 	close(rthread.streamfd);
 	rthread.streamfd = -1;
 
-	rthread.ready = 0;
+	if (rproc.move_to_final) {
+		/* The dir rthread.thdir_final must exist in the FS */
+		move_thdir_to_final(rthread.thdir, rthread.thdir_final);
+		try_clean_dir(rthread.thdir);
+	}
+
 	rthread.finished = 1;
+	rthread.ready = 0;
 }
 
 int
@@ -734,8 +770,8 @@ ovni_flush(void)
 	if (!rthread.ready)
 		die("thread is not initialized");
 
-	if (!rproc.ready)
+	if (atomic_load(&rproc.st) != ST_READY)
-		die("process is not initialized");
+		die("process not ready");
 
 	ovni_ev_set_clock(&pre, ovni_clock_now());
 	ovni_ev_set_mcv(&pre, "OF[");

test/emu/common/instr.h

@@ -102,9 +102,11 @@ instr_start(int rank, int nranks)
 
 	ovni_version_check();
 	ovni_proc_init(1, rankname, getpid());
-	ovni_proc_set_rank(rank, nranks);
 	ovni_thread_init(get_tid());
 
+	if (nranks > 0)
+		ovni_proc_set_rank(rank, nranks);
+
 	/* All ranks inform CPUs */
 	for (int i = 0; i < nranks; i++)
 		ovni_add_cpu(i, i);

test/emu/nosv/multiple-segment.c

@@ -27,11 +27,12 @@ main(void)
 	int app = rank / N;
 
 	char loom[128];
-	if (snprintf(loom, 128, "loom.%04d", app) >= 128)
+	if (snprintf(loom, 128, "node0.%04d", app) >= 128)
 		die("snprintf failed");
 
 	ovni_proc_init(1 + app, loom, getpid());
 	ovni_proc_set_rank(rank, nranks);
+	ovni_thread_init(get_tid());
 
 	/* Leader of the segment, must emit CPUs */
 	if (rank % N == 0) {
@@ -39,13 +40,13 @@ main(void)
 		for (int i = 0; i < N; i++) {
 			cpus[i] = app * N + i;
 			ovni_add_cpu(i, cpus[i]);
+			info("adding cpu %d to rank %d", i, rank);
 		}
 	}
 
 	int nlooms = nranks / N;
 	int lcpu = rank % N;
 
-	ovni_thread_init(get_tid());
 	instr_require("ovni");
 	instr_nosv_init();
 	instr_thread_execute(lcpu, -1, 0);

test/emu/ovni/CMakeLists.txt

@@ -11,6 +11,7 @@ test_emu(sort-first-and-full-ring.c SORT
   SHOULD_FAIL REGEX "cannot find a event previous to clock")
 test_emu(burst-stats.c REGEX "burst stats: median/avg/max =  33/ 33/ 33 ns")
 test_emu(mp-simple.c MP)
+test_emu(merge-cpus-loom.c MP)
 test_emu(version-good.c)
 test_emu(version-bad.c SHOULD_FAIL REGEX "incompatible .* version")
 test_emu(clockgate.c MP SHOULD_FAIL REGEX "detected large clock gate")
@@ -18,11 +19,11 @@ test_emu(no-cpus.c SHOULD_FAIL REGEX "loom .* has no physical CPUs")
 test_emu(sort-cpus-by-loom.c MP)
 test_emu(sort-cpus-by-rank.c MP)
 test_emu(tracedir-subdir.c MP DRIVER "tracedir-subdir.driver.sh")
-test_emu(empty-stream.c SHOULD_FAIL REGEX "model_ovni_finish: thread .* is not dead")
+test_emu(empty-stream.c SHOULD_FAIL REGEX "missing ovni.finished")
 test_emu(require-bad-version.c SHOULD_FAIL REGEX "unsupported ovni model version (want 666.66.6, have .*)")
-test_emu(require-compat.c REGEX "loading trace in compatibility mode")
+test_emu(require-compat.c)
 test_emu(require-repeated.c)
-test_emu(thread-crash.c SHOULD_FAIL REGEX "incomplete stream")
+test_emu(thread-crash.c SHOULD_FAIL REGEX "missing ovni.finished")
 test_emu(thread-free-isready.c)
 test_emu(flush-tmpdir.c MP DRIVER "flush-tmpdir.driver.sh")
 test_emu(tmpdir-metadata.c MP DRIVER "tmpdir-metadata.driver.sh")

test/emu/ovni/clockgate.c

@@ -37,8 +37,8 @@ start_delayed(int rank, int nranks)
 
 	ovni_version_check();
 	ovni_proc_init(1, rankname, getpid());
-	ovni_proc_set_rank(rank, nranks);
 	ovni_thread_init(get_tid());
+	ovni_proc_set_rank(rank, nranks);
 	instr_require("ovni");
 
 	/* All ranks inform CPUs */

test/emu/ovni/merge-cpus-loom.c

@@ -0,0 +1,52 @@
+/* Copyright (c) 2024 Barcelona Supercomputing Center (BSC)
+ * SPDX-License-Identifier: GPL-3.0-or-later */
+
+#include <stdlib.h>
+#include "compat.h"
+#include "instr.h"
+
+/* Ensure we can emit CPUs from multiple threads of the same loom */
+
+static inline void
+start(int rank, int nranks)
+{
+	char hostname[OVNI_MAX_HOSTNAME];
+
+	if (gethostname(hostname, OVNI_MAX_HOSTNAME) != 0)
+		die("gethostname failed");
+
+	ovni_version_check();
+
+	/* Only one loom */
+	ovni_proc_init(1, hostname, getpid());
+	ovni_thread_init(get_tid());
+	ovni_proc_set_rank(rank, nranks);
+
+	/* Only emit a subset of CPUs up to the rank number */
+	for (int i = 0; i <= rank; i++)
+		ovni_add_cpu(i, i);
+
+	int curcpu = rank;
+
+	dbg("thread %d has cpu %d (ncpus=%d)",
+			get_tid(), curcpu, nranks);
+
+	instr_require("ovni");
+	instr_thread_execute(curcpu, -1, 0);
+}
+
+
+int
+main(void)
+{
+	int rank = atoi(getenv("OVNI_RANK"));
+	int nranks = atoi(getenv("OVNI_NRANKS"));
+
+	start(rank, nranks);
+
+	sleep_us(50 * 1000);
+
+	instr_end();
+
+	return 0;
+}

test/emu/ovni/sort-cpus-by-loom.c

@@ -1,4 +1,4 @@
-/* Copyright (c) 2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2023-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #include <stdio.h>
@@ -35,6 +35,7 @@ main(void)
 	for (int i = 0; i < N; i++)
 		ovni_add_cpu(i, cpus[i]);
 
+	instr_require("ovni");
 	instr_thread_execute(-1, -1, 0);
 
 	instr_end();

test/emu/ovni/sort-cpus-by-rank.c

@@ -1,4 +1,4 @@
-/* Copyright (c) 2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2023-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #include <stdio.h>
@@ -36,6 +36,7 @@ main(void)
 	for (int i = 0; i < N; i++)
 		ovni_add_cpu(i, cpus[i]);
 
+	instr_require("ovni");
 	instr_thread_execute(-1, -1, 0);
 
 	instr_end();

test/emu/ovni/tmpdir-metadata.driver.sh

@@ -8,9 +8,9 @@ test_files() {
   test -e "$dst"
   test -e "$dst/loom.node.1"
   test -e "$dst/loom.node.1/proc.123"
-  test -e "$dst/loom.node.1/proc.123/metadata.json"
+  test -e "$dst/loom.node.1/proc.123/thread.123"
-  test -e "$dst/loom.node.1/proc.123/thread.123.json"
+  test -e "$dst/loom.node.1/proc.123/thread.123/stream.json"
-  test -e "$dst/loom.node.1/proc.123/thread.123.obs"
+  test -e "$dst/loom.node.1/proc.123/thread.123/stream.obs"
 }
 
 test_no_files() {
@@ -18,9 +18,9 @@ test_no_files() {
   test '!' -e "$dst"
   test '!' -e "$dst/loom.node.1"
   test '!' -e "$dst/loom.node.1/proc.123"
-  test '!' -e "$dst/loom.node.1/proc.123/metadata.json"
+  test '!' -e "$dst/loom.node.1/proc.123/thread.123"
-  test '!' -e "$dst/loom.node.1/proc.123/thread.123.json"
+  test '!' -e "$dst/loom.node.1/proc.123/thread.123/stream.json"
-  test '!' -e "$dst/loom.node.1/proc.123/thread.123.obs"
+  test '!' -e "$dst/loom.node.1/proc.123/thread.123/stream.obs"
 }
 
 # Test setting OVNI_TMPDIR

test/rt/nanos6/spawn-task-external.c

@@ -1,4 +1,4 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2021-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 /* Spawn a task from an external thread that calls some nanos6
@@ -67,7 +67,7 @@ instr_thread_start(int32_t cpu, int32_t creator_tid, uint64_t tag)
 	ovni_payload_add(&ev, (uint8_t *) &tag, sizeof(tag));
 	ovni_ev_emit(&ev);
 
-	/* Flush the events to disk before killing the thread */
+	/* Flush the events to disk after creating the thread */
 	ovni_flush();
 }
 
@@ -82,6 +82,9 @@ instr_thread_end(void)
 
 	/* Flush the events to disk before killing the thread */
 	ovni_flush();
+
+	/* Finish the thread */
+	ovni_thread_free();
 }
 
 /* Call the nanos6_spawn_function from an external thread */

test/unit/CMakeLists.txt

@@ -17,7 +17,6 @@ unit_test(mux.c)
 unit_test(prv.c)
 unit_test(stream.c)
 unit_test(task.c)
-unit_test(thread.c)
 unit_test(value.c)
 unit_test(version.c)
 unit_test(path.c)

test/unit/cpu.c

@@ -1,4 +1,4 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2021-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #include "emu/cpu.h"
@@ -27,18 +27,16 @@ test_oversubscription(void)
 
 	struct proc proc;
 
-	if (proc_init_begin(&proc, "loom.0/proc.0") != 0)
+	if (proc_init_begin(&proc, 1) != 0)
 		die("proc_init_begin failed");
 
+	proc.appid = 1;
+
 	proc_set_gindex(&proc, 0);
 
-	/* FIXME: We shouldn't need to recreate a full process to test the CPU
-	 * affinity rules */
-	proc.metadata_loaded = 1;
-
 	struct thread th0, th1;
 
-	if (thread_init_begin(&th0, "loom.0/proc.0/thread.0.obs") != 0)
+	if (thread_init_begin(&th0, 1) != 0)
 		die("thread_init_begin failed");
 
 	thread_set_gindex(&th0, 0);
@@ -47,7 +45,7 @@ test_oversubscription(void)
 	if (thread_init_end(&th0) != 0)
 		die("thread_init_end failed");
 
-	if (thread_init_begin(&th1, "loom.1/proc.1/thread.1.obs") != 0)
+	if (thread_init_begin(&th1, 2) != 0)
 		die("thread_init_begin failed");
 
 	thread_set_gindex(&th1, 1);

test/unit/loom.c

@@ -1,4 +1,4 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2021-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #include "emu/loom.h"
@@ -8,19 +8,17 @@
 #include "emu/proc.h"
 #include "unittest.h"
 
-char testloom[] = "loom.0";
+char testloom[] = "node1";
-char testproc[] = "loom.0/proc.1";
+int testproc = 1;
 
 static void
 test_bad_name(struct loom *loom)
 {
-	ERR(loom_init_begin(loom, "blah"));
 	ERR(loom_init_begin(loom, "loom/blah"));
-	ERR(loom_init_begin(loom, "loom.123/testloom"));
-	ERR(loom_init_begin(loom, "loom.123/"));
 	ERR(loom_init_begin(loom, "/loom.123"));
 	ERR(loom_init_begin(loom, "./loom.123"));
 	OK(loom_init_begin(loom, "loom.123"));
+	OK(loom_init_begin(loom, "foo"));
 
 	err("ok");
 }
@@ -28,7 +26,7 @@ test_bad_name(struct loom *loom)
 static void
 test_hostname(struct loom *loom)
 {
-	OK(loom_init_begin(loom, "loom.node1.blah"));
+	OK(loom_init_begin(loom, "node1.blah"));
 
 	if (strcmp(loom->hostname, "node1") != 0)
 		die("wrong hostname: %s", loom->hostname);
@@ -67,7 +65,6 @@ test_duplicate_procs(struct loom *loom)
 	struct proc proc;
 	OK(loom_init_begin(loom, testloom));
 	OK(proc_init_begin(&proc, testproc));
-	proc.metadata_loaded = 1;
 	OK(loom_add_proc(loom, &proc));
 	ERR(loom_add_proc(loom, &proc));
 

test/unit/stream.c

@@ -1,4 +1,4 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
+/* Copyright (c) 2021-2024 Barcelona Supercomputing Center (BSC)
  * SPDX-License-Identifier: GPL-3.0-or-later */
 
 #include <stdio.h>
@@ -11,10 +11,27 @@
 #include "ovni.h"
 #include "unittest.h"
 
+static void
+write_dummy_json(const char *path)
+{
+	const char *json = "{ \"version\" : 3 }";
+	FILE *f = fopen(path, "w");
+
+	if (f == NULL)
+		die("fopen json failed:");
+
+	if (fwrite(json, strlen(json), 1, f) != 1)
+		die("fwrite json failed:");
+
+	fclose(f);
+}
+
 static void
 test_ok(void)
 {
-	const char *fname = "stream-ok.obs";
+	OK(mkdir("ok", 0755));
+
+	const char *fname = "ok/stream.obs";
 	FILE *f = fopen(fname, "w");
 
 	if (f == NULL)
@@ -30,8 +47,10 @@ test_ok(void)
 
 	fclose(f);
 
+	write_dummy_json("ok/stream.json");
+
 	struct stream stream;
-	OK(stream_load(&stream, ".", fname));
+	OK(stream_load(&stream, ".", "ok"));
 
 	if (stream.active)
 		die("stream is active");
@@ -42,7 +61,9 @@ test_ok(void)
 static void
 test_bad(void)
 {
-	const char *fname = "stream-bad.obs";
+	OK(mkdir("bad", 0755));
+
+	const char *fname = "bad/stream.obs";
 	FILE *f = fopen(fname, "w");
 
 	if (f == NULL)
@@ -58,8 +79,10 @@ test_bad(void)
 
 	fclose(f);
 
+	write_dummy_json("bad/stream.json");
+
 	struct stream stream;
-	ERR(stream_load(&stream, ".", fname));
+	ERR(stream_load(&stream, ".", "bad"));
 
 	err("OK");
 }

test/unit/thread.c

@@ -1,35 +0,0 @@
-/* Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC)
- * SPDX-License-Identifier: GPL-3.0-or-later */
-
-#include "common.h"
-#include "emu/thread.h"
-#include "unittest.h"
-
-/* Ensure we can load the old trace format */
-static void
-test_old_trace(void)
-{
-	struct thread th;
-
-	OK(thread_init_begin(&th, "loom.0/proc.0/thread.1.obs"));
-	if (th.tid != 1)
-		die("wrong tid");
-
-	OK(thread_init_begin(&th, "loom.0/proc.0/thread.2"));
-	if (th.tid != 2)
-		die("wrong tid");
-
-	ERR(thread_init_begin(&th, "loom.0/proc.0/thread.kk"));
-	ERR(thread_init_begin(&th, "loom.0/proc.0/thread."));
-	ERR(thread_init_begin(&th, "loom.0/proc.0/thread"));
-	ERR(thread_init_begin(&th, "thread.prv"));
-
-	err("ok");
-}
-
-int main(void)
-{
-	test_old_trace();
-
-	return 0;
-}