From 6141c2e303f260decea42a5034d5464dd243c810 Mon Sep 17 00:00:00 2001
From: Rodrigo Arias <rodrigo.arias@bsc.es>
Date: Mon, 29 Aug 2022 13:41:28 +0200
Subject: [PATCH] Document nanos6 subsystem view

---
 doc/emu_nanos6.md | 117 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
 create mode 100644 doc/emu_nanos6.md

diff --git a/doc/emu_nanos6.md b/doc/emu_nanos6.md
new file mode 100644
index 0000000..8033dbc
--- /dev/null
+++ b/doc/emu_nanos6.md
@@ -0,0 +1,117 @@
+# Nanos6 Emulation
+
+The Nanos6 emulator generates four different Paraver views, which are
+explained in this document.
+
+## Task id
+
+The task id view represents the id of the Nanos6 task instance that is
+currently executing on each thread/cpu. This id is a monotonically
+increasing identifier assigned on task creation. Lower ids correspond to
+tasks created at an earlier point than higher ids.
+
+## Task type
+
+Every task in Nanos6 contains a task type, which roughly corresponds to
+the actual location in the code a task was declared. For example if a
+function fn() is declared as a Nanos6 task, and it is called multiple
+times in a program, every created task will have a different id, but the
+same type.
+
+In the view, each type is shown with a label declared in the source with
+the label() attribute of the task. If no label was specified, one is
+automatically generated for each type.
+
+Note that in this view, event value is a hash function of the type
+label, so two distinct types (tasks declared in different parts of the
+code) with the same label will share event value and will hence be
+indistinguishable.
+
+## MPI Rank
+
+Represents the current MPI rank for the currently running task in a
+thread or cpu.
+
+## Subsystem view
+
+The subsystem view attempts to provide a general overview of what the
+runtime is doing at any point in time. This view is more complex to
+understand than the others but is generally the most useful to
+understand what is happening and debug problems related with the
+runtime.
+
+The view shows the state of the runtime for each thread (and for each
+CPU, the state of the running thread in that CPU).
+
+The state is computed by the following method: the runtime code is
+completely divided into sections of code (machine instructions) S1, S2,
+..., SN, which are instrumented (an event is emitted when entering and
+exiting each section), and one common section of code which is shared
+across the subsystems, U, of no interest. We also assume any other code
+not belonging to the runtime to be in the U section.
+
+Every instruction of the runtime belongs to *exactly one section*.
+
+To determine the state of a thread, we look into the stack to see what
+is the top-most instrumented section.
+
+At any given point in time, a thread may be executing code with a stack
+that spawns multiple sections, for example \[ S1, U, S2, S3, U \] (the
+last is on top). The subsystem view selects the last subsystem section
+from the stack ignoring the common section U, and presents that section
+as the current state of the execution, in this case the S3.
+
+Additionally, the runtime sections are grouped together in systems,
+which form a group of closely related functions. A complete set of
+states for the subsystem view is listed below. The system is listed
+first and then the subsystem:
+
+- **No subsystem**: There is no instrumented section in the stack of the
+thread.
+
+The **Scheduler** system groups the actions that relate to the queueing
+and dequeueing of ready tasks. The subsystems are:
+
+- **Scheduler: Waiting for tasks**: Actively waiting for tasks inside the
+scheduler subsystem, registered but not holding the scheduler lock
+
+- **Scheduler: Serving tasks**: Inside the scheduler lock, serving tasks
+to other threads
+
+- **Scheduler: Adding ready tasks**: Adding tasks to the scheduler queues,
+but outside of the scheduler lock.
+
+The **Task** system contains the code that controls the lifecycle of
+tasks.
+
+- **Task: Running**: Executing the body of the task (user defined code).
+
+- **Task: Spawning function**: Registering a new spawn function
+(programmatically created task)
+
+- **Task: Creating**: Creating a new task, through `nanos6_create_task`
+
+- **Task: Submitting**: Submitting a recently created task, through
+`nanos6_submit_task`
+
+The **Dependency** group only contains the dependency code:
+
+- **Dependency: Registering**: Registering a task's dependencies
+
+- **Dependency: Unregistering**: Releasing a task's dependencies because
+it has ended
+
+- **Blocking: Taskwait**: Task is blocked while inside a taskwait
+
+- **Blocking: Blocking current task**: Task is blocked through the Nanos6
+blocking API
+
+- **Blocking: Unblocking remote task**: Unblocking a different task using
+the Nanos6 blocking API
+
+- **Blocking: Wait For**: Blocking a deadline task, which will be
+re-enqueued when a certain amount of time has passed
+
+- **Threading: Attached as external thread**: External/Leader thread
+(which has registered to Nanos6) is running
+