From cd7eb7179f37d81d227b01ccb5913c16ad9a6c01 Mon Sep 17 00:00:00 2001 From: Rodrigo Arias Mallo Date: Thu, 1 Aug 2024 19:29:04 +0200 Subject: [PATCH] Update journal with PLIC experiments --- JOURNAL.md | 665 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 665 insertions(+) diff --git a/JOURNAL.md b/JOURNAL.md index 333b33a..154f370 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -2588,4 +2588,669 @@ This one yes: [ 165.066220] mm_page_alloc: page=(____ptrval____) pfn=0x83d28 order=2 migratetype=0 gfp_flags=GFP_KERNEL [ 165.066540] console: mm_page_alloc: page=(____ptrval____) pfn=0x83d28 order=2 migratetype=0 gfp_flags=GFP_KERNEL +## 2024-08-01 +Now that we have a new bitstream with a CLINT connected to a PLIC input, we may +be able to generate an interrupt. + +Here is the comment where I gather the pieces: + +---8<--- + +From https://gitlab.bsc.es/hwdesign/rtl/core-tile/sa-fpga/ I can see that the +auxiliary timer [is in fact another +CLINT](https://gitlab.bsc.es/hwdesign/rtl/core-tile/sa-fpga/-/blob/10ba8b2a11ef105d7cda065e13838a3d28f3c951/fpga_core_bridge/rtl/fpga_core_bridge.sv#L685). + +I don't have access to the [hlib +repository](https://gitlab.bsc.es/hwdesign/hlib.git) (@jmendoza can I get access +to it?) to see the CLINT definition, but based on [this +CLINT](https://github.com/openhwgroup/cva6/blob/master/corev_apu/clint/clint.sv) +and [this one](https://github.com/pulp-platform/clint/blob/master/src/clint.sv) +I can estimate some of the previous information: + +> - The information on which port number of the PLIC the timer is connected to. + +https://gitlab.bsc.es/hwdesign/rtl/core-tile/sa-fpga/-/blob/main/fpga_core_bridge/rtl/fpga_core_bridge.sv#L1114 + +``` + plic #( + .PARAMETER_BITWIDTH (7), + .NUM_TARGETS (1), + .NUM_SOURCES (4) + ) plic_inst ( + .clk_i (clk_i), + .rstn_i (reset), + .irq_sources_i ({plic_timer_eirq,eth_irq,uart1_irq}), + .eip_targets_o (irq), +``` + +If I read it from right to left starting at 1, it should be **at 4**, as the +`eth_irq` has two "wires". + + +> - The memory address of the timer and the mapped registers, so I can see it +> increasing its value. I think the `aux_timer` you had in the past would be +> fine. + +https://gitlab.bsc.es/hwdesign/rtl/core-tile/sa-fpga/-/blob/main/fpga_core_bridge/rtl/local_includes/defines.svh#L33-36 + +``` +//Size: 64KB +`define AUX_TIMER_XBAR_ID 2 +`define AUX_TIMER_BASE_ADDR 64'h0000_0000_4001_0000 // Need to be this space because we use a clint as aux timer +`define AUX_TIMER_END_ADDR 64'h0000_0000_4001_FFFF +``` + +> - The specific operations I need to do in machine mode to configure the timer +> to fire at 1 Hz (probably setting two registers). + +Based on the source of the CLINT, **only one interrupt will be generated** after +setting the mtimecmp register to something larger than the mtime register. Then +I suspect I would have to make the interrupt run some code to rearm it again by +modifying the mtimecmp register to some value in the future: + +``` +// ----------------------------- +// IRQ Generation +// ----------------------------- +// The mtime register has a 64-bit precision on all RV32, RV64, and RV128 systems. Platforms provide a 64-bit +// memory-mapped machine-mode timer compare register (mtimecmp), which causes a timer interrupt to be posted when the +// mtime register contains a value greater than or equal (mtime >= mtimecmp) to the value in the mtimecmp register. +// The interrupt remains posted until it is cleared by writing the mtimecmp register. The interrupt will only be taken +// if interrupts are enabled and the MTIE bit is set in the mie register. +always_comb begin : irq_gen + // check that the mtime cmp register is set to a meaningful value + for (int unsigned i = 0; i < NR_CORES; i++) begin + if (mtime_q >= mtimecmp_q[i]) begin + timer_irq_o[i] = 1'b1; + end else begin + timer_irq_o[i] = 1'b0; + end + end +end +``` + +I could ensure that an interrupt has been fired by reading the mtime and +mtimecmp values, and checking that mtime > mtimecmp. + +Now I only need to find a bitstream that has been generated with +https://gitlab.bsc.es/hwdesign/rtl/core-tile/sa-fpga/-/commit/10ba8b2a11ef105d7cda065e13838a3d28f3c951. + + +This may work: + +https://gitlab.bsc.es/hwdesign/fpga/integration-lab/fpga-shell/-/jobs/968583/raw + +> Submodule path 'sa-fpga': checked out '12b77cb50cf1c416f107d4c7ab1c52d7b5e59056' + +Which is based on fpga-shell https://gitlab.bsc.es/hwdesign/fpga/integration-lab/fpga-shell/-/commit/01265d197f256bce2c7e82d21c7f4bf5dcb44e68 + +Here is the bitstream job: https://gitlab.bsc.es/hwdesign/fpga/integration-lab/fpga-shell/-/jobs/968585 + +And the bitstream: [artifacts.zip](/uploads/d8240a779cd485771b9e3d0147e342d1/artifacts.zip) + +And full log: [job.log](/uploads/a4215e4d039065b77f7a2d2b1403e475/job.log) + +The memory map would need a bit of adjustment in the device tree, but to play with the timer in machine mode not much is needed. + +I think I have all the pieces now. + +---8<--- + +I will try with the last bitstream that I already had compiled, as I will have +to rebuild the required packages in nix. + +To compute the memory position of the registers: + + `define AUX_TIMER_XBAR_ID 2 + `define AUX_TIMER_BASE_ADDR 64'h0000_0000_4001_0000 // Need to be this space because we use a clint as aux timer + `define AUX_TIMER_END_ADDR 64'h0000_0000_4001_FFFF + + localparam logic [15:0] MSIP_BASE = 16'h0; + localparam logic [15:0] MTIMECMP_BASE = 16'h4000; + localparam logic [15:0] MTIME_BASE = 16'hbff8; + +So, the base address 0x40010000 and the first MTIME at 0xbff8 would give us a +timer at 0x4001bff8. + +Here it is: + + => md 0x4001bff8 1 + 4001bff8: 006e65b8 .en. + => md 0x4001bff8 1 + 4001bff8: 006e9a26 &.n. + => md 0x4001bff8 1 + 4001bff8: 006ebae1 ..n. + => md 0x4001bff8 1 + 4001bff8: 006eda45 E.n. + => md 0x4001bff8 1 + 4001bff8: 006ef9d4 ..n. + => md 0x4001bff8 1 + 4001bff8: 006f1abb ..o. + +Now, the MTIMECMP should be at 0x40014000, which should be 0. + + => md 0x40014000 1 + 40014000: 00000000 .... + +Good. + +Now, I suspect the MSIP is not used, so it should be 0 at 0x40010000 too: + + => md 0x40010000 1 + 40010000: 00000000 .... + +Nice. + +Just for testing, let's see if I can make the timer cause any change in the MSIP +register by setting the MTIMECMP to a value: + + => mw 0x40014000 0x01700000 # Write the MTIMECMP + => md 0x40014000 1 + 40014000: 01700000 ..p. + => md 0x4001bff8 1 + 4001bff8: 016da81a ..m. + => md 0x40010000 1 + 40010000: 00000000 .... + => md 0x4001bff8 1 + 4001bff8: 016f947c |.o. + => md 0x4001bff8 1 + 4001bff8: 016fff96 ..o. + => md 0x4001bff8 1 + 4001bff8: 01704367 gCp. # Now we passed it + => md 0x40010000 1 + 40010000: 00000000 .... # But MSIP is still 0 + +As expected, nothing happens. We cannot monitor the interrupt line from the +timer itself. + +Now, let see if we can inspect the state of the PLIC. + +From the `plic_interface` I can see where are the memory addresses of the +registers exposed. + +The PLIC is mapped here: + + //Size: 4MB + `define PLIC_XBAR_ID 5 + `define PLIC_BASE_ADDR 64'h0000_0000_4080_0000 + `define PLIC_END_ADDR 64'h0000_0000_40BF_FFFF + +There are several ways in which the interrupts are not forwarded to the +destination, and several destinations. The PLIC specification is a good resource +to understand it: + + https://github.com/riscv/riscv-plic-spec + +This is important: + +> The interrupt gateways are responsible for converting global interrupt signals +> into a common interrupt request format, and for controlling the flow of +> interrupt requests to the PLIC core. At most one interrupt request per +> interrupt source can be pending in the PLIC core at any time, indicated by +> setting the source’s IP bit. The gateway only forwards a new interrupt request +> to the PLIC core after receiving notification that the interrupt handler +> servicing the previous interrupt request from the same source has completed. + +So, there cannot be any pending interrupt, otherwise no more interrupts will be +sent to the core. + +Assuming the PLIC uses the standard memory layout, we should find: + + base + 0x000000: Reserved (interrupt source 0 does not exist) + base + 0x000004: Interrupt source 1 priority + base + 0x000008: Interrupt source 2 priority + +Which they should begin at 0x40800000. + + => md 0x40800000 8 + 40800000: 00000000 00000000 00000000 00000000 ................ + 40800010: 00000000 00000000 00000000 00000000 ................ + +All the priorities are set to 0. + +Let's see the pending interrupts: + + base + 0x000FFC: Interrupt source 1023 priority + base + 0x001000: Interrupt Pending bit 0-31 + base + 0x00107C: Interrupt Pending bit 992-1023 + +They should be at 0x40801000: + + => md 0x40801000 8 + 40801000: 00000010 00000000 00000000 00000000 ................ + 40801010: 00000000 00000000 00000000 00000000 ................ + +Whoa, look at that. + + 4321 + 0x00000010 = 10000 + | | + | int 0 (reserved) + int 4 = timer + +We got the interrupt 4 pending in context 0! + +Other context don't seem to see anything: + + => md 0x40801080 1 + 40801080: 00000000 .... + => md 0x40801100 1 + 40801100: 00000000 .... + => md 0x40801180 1 + 40801180: 00000000 .... + => md 0x40801200 1 + 40801200: 00000000 .... + => md 0x40801280 1 + 40801280: 00000000 .... + => md 0x40801300 1 + 40801300: 00000000 .... + => md 0x40801380 1 + 40801380: 00000000 .... + +So, as the priority is 0, this means it is ignored: + +> If PLIC supports Interrupt Priorities, then each PLIC interrupt source can be +> assigned a priority by writing to its 32-bit memory-mapped priority register. +> A priority value of 0 is reserved to mean "never interrupt" and effectively +> disables the interrupt. Priority 1 is the lowest active priority while the +> maximum level of priority depends on PLIC implementation. Ties between global +> interrupts of the same priority are broken by the Interrupt ID; interrupts +> with the lowest ID have the highest effective priority. + +Let's claim the interrupt, by just performing a read from 0x40a00004: + + => md 0x40801000 1 + 40801000: 00000010 .... + => md 0x40a00004 1 + 40a00004: 00000000 .... + => md 0x40801000 1 + 40801000: 00000010 .... + +So, it continues to be pending. + +We have to write the completed interrupt, by writing the number 4 to the same +register: + + => mw 0x40a00004 4 + => md 0x40801000 1 + 40801000: 00000010 .... + +Still not cleared. + +Let's try making the MTIMECMP value much higher than MTIME: + + => md 0x40014000 1 + 40014000: 01700000 ..p. + => md 0x4001bff8 1 + 4001bff8: 03a4584b KX.. + => mw 0x40014000 0xaaaaaaaa + => md 0x40014000 1 + 40014000: aaaaaaaa .... + => md 0x4001bff8 1 + 4001bff8: 03abc84d M... + +So... the ID that must be written to the completion register is not the +interrupt number, but the value read from the claim register, which is 0. + + => mw 0x40a00004 0 + => md 0x40801000 1 + 40801000: 00000010 .... + +Still, nothing. + +All interrupts are disabled: + + => md 0x40802000 4 + 40802000: 00000000 00000000 00000000 00000000 ................ + +Let's try enabling the interrupt 4, by writting: + + => mw 0x40802000 0x10 + => md 0x40802000 1 + 40802000: 00000010 .... + => md 0x40801000 1 + 40801000: 00000010 .... + +Now, let's set the priority to something else than 0. + +First, lets make sure that the context 0 threshold priority is set to 0, so we +allow all interrupts: + + 0x200000: Priority threshold for context 0 + + => md 0x40a00000 1 + 40a00000: 00000007 .... + +Oh, so we are only receiving interrupts with priority 7 or higher. But our +interrupt has priority 0! + + => md 0x40800004 1 + 40800004: 00000000 .... + +Let's make the threshold 0 and our interrupt have priority 1. + + => mw 0x40a00000 0 + => mw 0x40800004 1 + => md 0x40800004 1 + 40800004: 00000001 .... + => md 0x40a00000 + 40a00000: 00000000 .... + +Not let's see again the interrupt state: + + => md 0x40801000 1 + 40801000: 00000010 .... + +Still on. + +Let's read the claim register again. + + => md 0x40a00004 + 40a00004: 00000000 .... + +Still 0, let's try to complete it: + + => mw 0x40a00004 0 + => md 0x40801000 1 + 40801000: 00000010 .... + +Nope, still pending. + +What, what the hell. The threshold value has changed to 1: + + => md 0x40800004 1 + 40800004: 00000001 .... + => md 0x40a00000 1 + 40a00000: 00000001 .... <-- this was 0 + +Let's configure the interruption priority to something bigger than 1. + +Wait, I put the priority in the wrong source: + + 0x000000: Reserved (interrupt source 0 does not exist) + 0x000004: Interrupt source 1 priority + 0x000008: Interrupt source 2 priority + +Our timer should be the source 4, so 12 or 0xc: + + => md 0x4080000c 1 + 4080000c: 00000000 .... + +(This is wrong, should be 0x40800010, see below) + +Let's make it have priority 0xd: + + => mw 0x4080000c 0xd + => md 0x4080000c 1 + 4080000c: 0000000d .... + +Something weird is going on with the priority register? + + => md 0x40a00000 1 + 40a00000: 00000000 .... + => md 0x40a00000 1 + 40a00000: 0000000d .... + => md 0x40a00000 1 + 40a00000: 0000000d .... + => md 0x40a00000 1 + 40a00000: 0000000d .... + => md 0x40a00000 1 + 40a00000: 0000000d .... + +Let's see the claim register, which should be in the next word: + + => md 0x40a00004 1 + 40a00004: 00000004 .... + +Yes! Now I can see the claim register with a proper ID. Let's complete this +interrupt by writing the 4 back to that register: + + => mw 0x40a00004 4 + => md 0x40801000 1 + 40801000: 00000000 .... + +Perfect! It properly caused the pending interrupt to disappear. + +Let's try now setting the MTIMECMP to something smaller than the MTIME, so it +causes an interrupt. With a value 0 should always work, but lets choose a non +zero value: + + => md 0x40014000 + 40014000: aaaaaaaa .... + => mw 0x40014000 00aaaaaa + => md 0x40014000 + 40014000: 00aaaaaa .... + => md 0x4001bff8 + 4001bff8: 06211a0c ..!. + => md 0x40801000 1 + 40801000: 00000010 .... + +Perfect! It causes the interrupt to appear as pending. + +So, using the context 0, we can properly see the interrupt pending, claim it and +complete it. But the context 0 is not used in OpenSBI, only the 9 and 11: + +From `include/sbi/riscv_encoding.h`: + + #define IRQ_S_SOFT 1 + #define IRQ_VS_SOFT 2 + #define IRQ_M_SOFT 3 + #define IRQ_S_TIMER 5 + #define IRQ_VS_TIMER 6 + #define IRQ_M_TIMER 7 + #define IRQ_S_EXT 9 + #define IRQ_VS_EXT 10 + #define IRQ_M_EXT 11 + #define IRQ_S_GEXT 12 + #define IRQ_PMU_OVF 13 + +And from `lib/utils/irqchip/fdt_irqchip_plic.c`: + + static int irqchip_plic_update_hartid_table(void *fdt, int nodeoff, + struct plic_data *pd) + { + const fdt32_t *val; + u32 phandle, hwirq, hartid; + struct sbi_scratch *scratch; + int i, err, count, cpu_offset, cpu_intc_offset; + + val = fdt_getprop(fdt, nodeoff, "interrupts-extended", &count); + if (!val || count < sizeof(fdt32_t)) + return SBI_EINVAL; + count = count / sizeof(fdt32_t); + + for (i = 0; i < count; i += 2) { + phandle = fdt32_to_cpu(val[i]); + hwirq = fdt32_to_cpu(val[i + 1]); + + cpu_intc_offset = fdt_node_offset_by_phandle(fdt, phandle); + if (cpu_intc_offset < 0) + continue; + + cpu_offset = fdt_parent_offset(fdt, cpu_intc_offset); + if (cpu_offset < 0) + continue; + + err = fdt_parse_hart_id(fdt, cpu_offset, &hartid); + if (err) + continue; + + scratch = sbi_hartid_to_scratch(hartid); + if (!scratch) + continue; + + plic_set_hart_data_ptr(scratch, pd); + switch (hwirq) { + case IRQ_M_EXT: + plic_set_hart_mcontext(scratch, i / 2); + break; + case IRQ_S_EXT: + plic_set_hart_scontext(scratch, i / 2); + break; + } + } + + return 0; + } + +So, lets try to do the same, but with the context 11 for machine mode +`IRQ_M_EXT`. + +Let's compute the address of the input source for context 11: + + base + 0x002000: Enable bits for sources 0-31 on context 0 + base + 0x002004: Enable bits for sources 32-63 on context 0 + ... + base + 0x00207C: Enable bits for sources 992-1023 on context 0 + base + 0x002080: Enable bits for sources 0-31 on context 1 + base + 0x002084: Enable bits for sources 32-63 on context 1 + ... + base + 0x0020FC: Enable bits for sources 992-1023 on context 1 + base + 0x002100: Enable bits for sources 0-31 on context 2 + base + 0x002104: Enable bits for sources 32-63 on context 2 + ... + base + 0x00217C: Enable bits for sources 992-1023 on context 2 + ... + base + 0x1F1F80: Enable bits for sources 0-31 on context 15871 + base + 0x1F1F84: Enable bits for sources 32-63 on context 15871 + base + 0x1F1FFC: Enable bits for sources 992-1023 on context 15871 + ... + +It should be: + + >>> hex(0x40800000 + 0x2000 + (11 * 0x80)) + '0x40802580' + +They are all disabled: + + => md 0x40802580 + 40802580: 00000000 .... + +So, let's enable the source 4 by writing 0x10 + + => mw 0x40802580 0x10 + => md 0x40801000 1 + 40801000: 00000010 .... + +Now, let's check the context 11 priority threshold: + + 0x200000: Priority threshold for context 0 + 0x201000: Priority threshold for context 1 + 0x202000: Priority threshold for context 2 + 0x203000: Priority threshold for context 3 + +The priority threshold for context 11 should be at: + + >>> hex(0x40800000 + 0x200000 + (11 * 0x1000)) + '0x40a0b000' + + => md 0x40a0b000 + 40a0b000: 00000000 .... + +It has value 0, so all interrupts with non-zero priority should pass: + +> For example, a threshold value of zero permits all interrupts with non-zero +> priority. + +Let's see the priority of source 4 in context 11: + + 0x000000: Reserved (interrupt source 0 does not exist) + 0x000004: Interrupt source 1 priority + 0x000008: Interrupt source 2 priority + ... + 0x000FFC: Interrupt source 1023 priority + +The address should be at: + + >>> hex(0x40800000 + (4 * 0x4)) + + => md 0x40800010 + 40800010: 00000000 .... + +It has priority 0, so it would never work. Let's make it priority 1: + + => mw 0x40800010 1 + => md 0x40800010 1 + 40800010: 00000001 + +Let's check the pending interrupts: + + => md 0x40801000 1 + 40801000: 00000010 .... + +It is still pending, so let's clear it my setting the MTIMECMP to a large value. + + => md 0x40014000 + 40014000: 00aaaaaa .... + => mw 0x40014000 0xaaaaaaaa + => md 0x40014000 + 40014000: aaaaaaaa .... + => md 0x4001bff8 + 4001bff8: 0e8e6066 f`.. + => md 0x4001bff8 + 4001bff8: 0e8ea4c9 .... + => md 0x4001bff8 + 4001bff8: 0e8ece24 $... + +Now, let's claim and complete it for the context 0 which was already enabled +from the test before. + + => md 0x40a00004 1 + 40a00004: 00000004 .... + => mw 0x40a00004 4 + => md 0x40801000 1 + 40801000: 00000000 .... + +Perfect, now it is not pending anymore. + +Now, the context 0 is still enabled, so the interruptions may be sent there +instead of context 11. So let's disable the context 0 first. + + => mw 0x40802000 0 + => md 0x40802000 1 + 40802000: 00000000 .... + +Now let's fire the MTIMECMP and see if OpenSBI sees a machine trap. + + => md 0x40014000 1 + 40014000: aaaaaaaa .... + => mw 0x40014000 00aaaaaa + => md 0x40014000 1 + 40014000: 00aaaaaa .... + +Nothing happened. + +The interrupt is pending: + + => md 0x40801000 1 + 40801000: 00000010 .... + +The claim on context 0 returns 0, so not interrupt there which is expected: + + => md 0x40a00004 1 + 40a00004: 00000000 .... + +Let's compute the claim register on context 11: + + 0x200004: Interrupt Claim Process for context 0 + 0x201004: Interrupt Claim Process for context 1 + 0x202004: Interrupt Claim Process for context 2 + 0x203004: Interrupt Claim Process for context 3 + ... + + >>> hex(0x40800000 + 0x200004 + (11 * 0x1000)) + '0x40a0b004' + + => md 0x40a0b004 1 + 40a0b004: 00000000 .... + +Hmm, there is no claim ID. + +So, I checked again, and I cannot enable the interrupt on context 11: + + => md 0x40802580 1 + 40802580: 00000000 .... + => mw 0x40802580 0x10 + => md 0x40802580 1 + 40802580: 00000000 ....