rarias/nixos-riscv
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Use headings to allow hrefs

Browse Source
This commit is contained in:
Rodrigo Arias 2024-07-05 17:05:28 +02:00
parent 540780e508
commit a7c460b034
1 changed files with 49 additions and 38 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

87
JOURNAL.md
View File

@ -53,17 +53,17 @@ CPU.
Let's go back and try to get the initrd shell, so we can systematically hang it Let's go back and try to get the initrd shell, so we can systematically hang it
in the `switch_root` in the `switch_root`
<!--}}}--> <!--}}}-->
**Observation**: The riscv-timer seems to be causing interrupts with IRQ 5:<!--{{{--> ### OBSERVATION: The riscv-timer seems to be causing interrupts with IRQ 5:<!--{{{-->
``` ```
[ 62.439060] irq_handler_entry: irq=5 name=riscv-timer [ 62.439060] irq_handler_entry: irq=5 name=riscv-timer
[ 62.444980] irq_handler_exit: irq=5 ret=handled [ 62.444980] irq_handler_exit: irq=5 ret=handled
``` ```
<!--}}}--> <!--}}}-->
**Observation**: Rohan reports the serial startup routine being running *after*{{{ ### OBSERVATION: Rohan reports the serial startup routine being running *after* <!--{{{-->
the init begins. the init begins.
<!--}}}--> <!--}}}-->
**Observation**: Only interrupts in timer, others are zero.<!--{{{--> ### OBSERVATION: Only interrupts in timer, others are zero.<!--{{{-->
With: With:
@ -95,7 +95,7 @@ I can see this:
IPI4: 0 IRQ work interrupts IPI4: 0 IRQ work interrupts
IPI5: 0 Timer broadcast interrupts IPI5: 0 Timer broadcast interrupts
<!--}}}--> <!--}}}-->
**Observation**: There is a timer configured in 0x40170000 but in the device<!--{{{--> ### OBSERVATION: There is a timer configured in 0x40170000 but in the device<!--{{{-->
tree we only have one at `timer@40002000`. tree we only have one at `timer@40002000`.
#define OX_ALVEO_TIMER_BASE 0x40170000 #define OX_ALVEO_TIMER_BASE 0x40170000
@ -106,7 +106,7 @@ tree we only have one at `timer@40002000`.
https://gitlab.bsc.es/hwdesign/bsc-linux/-/blob/d6d194bd30d9a8fe49c2a278ffb3c3ae7852e75d/bsc_tree/patches/ox_alveo/opensbi/0001-opensbi-ox_alveo-platform.patch#L63 https://gitlab.bsc.es/hwdesign/bsc-linux/-/blob/d6d194bd30d9a8fe49c2a278ffb3c3ae7852e75d/bsc_tree/patches/ox_alveo/opensbi/0001-opensbi-ox_alveo-platform.patch#L63
<!--}}}--> <!--}}}-->
**Observation**: When the serial console starts, the speed of the serial port<!--{{{--> ### OBSERVATION: When the serial console starts, the speed of the serial port<!--{{{-->
changes to 9600: changes to 9600:
[ 6.845400] io scheduler mq-deadline registered [ 6.845400] io scheduler mq-deadline registered
@ -181,13 +181,14 @@ That was my mistake as I need to put the baud speed in the ttyS0, like this:
console=ttyS0,115200n8 console=ttyS0,115200n8
<!--}}}--> <!--}}}-->
**Observation**: Trying to read from the serial console /dev/ttyS0 causes no<!--{{{--> ### OBSERVATION: Trying to read from the serial console /dev/ttyS0 causes no<!--{{{-->
more messages in the console (or a hang). more messages in the console (or a hang).
<!--}}}--> <!--}}}-->
**Question**: Can we make a heartbeat for the kernel? The idea is to keep a<!--{{{--> ### QUESTION: Can we make a heartbeat for the kernel? <!--{{{-->
counter in some memory of the kernel so we can see it from the host being moved. The idea is to keep a counter in some memory of the kernel so we can see it from
the host being moved.
<!--}}}--> <!--}}}-->
**Question**: Can we disable the serial driver 8250 from loading?<!--{{{--> ### QUESTION: Can we disable the serial driver 8250 from loading?<!--{{{-->
initcall_blacklist=<driver_init> initcall_blacklist=<driver_init>
@ -208,7 +209,7 @@ Yes, but that doesn't seem to do anything. It is hanging:
[ 629.733920] stage-1-init: [Thu Jan 1 00:10:29 UTC 1970] + echo /nix/store/snvvqpxmryw1szlllk0bxpm37p8vj8sw-extra-utils/bin/modprobe [ 629.733920] stage-1-init: [Thu Jan 1 00:10:29 UTC 1970] + echo /nix/store/snvvqpxmryw1szlllk0bxpm37p8vj8sw-extra-utils/bin/modprobe
<!--}}}--> <!--}}}-->
**Question**: What happens if we remap the interruptions?<!--{{{--> ### QUESTION: What happens if we remap the interruptions?<!--{{{-->
- Move the serial from 0 to 1 - Move the serial from 0 to 1
- Move the plic from 3 to 2 and remove 7 - Move the plic from 3 to 2 and remove 7
@ -225,14 +226,14 @@ Rather than two:
[ 0.000000] plic: plic@40800000: mapped 3 interrupts with 0 handlers for 2 contexts. [ 0.000000] plic: plic@40800000: mapped 3 interrupts with 0 handlers for 2 contexts.
[ 0.000000] riscv: providing IPIs using SBI IPI extension [ 0.000000] riscv: providing IPIs using SBI IPI extension
<!--}}}--> <!--}}}-->
**Question**: What happens if we block the `sbi_ipi` driver?<!--{{{--> ### QUESTION: What happens if we block the `sbi_ipi` driver?<!--{{{-->
initcall_blacklist=sbi_ipi_init initcall_blacklist=sbi_ipi_init
Nothing, it cannot be disabled it seems. I will remove SMP support so it won't Nothing, it cannot be disabled it seems. I will remove SMP support so it won't
be compiled in. be compiled in.
<!--}}}--> <!--}}}-->
**Observation**: Searching for 'riscv,plic0' only matches irq-sifive-plic driver.<!--{{{--> ### OBSERVATION: Searching for 'riscv,plic0' only matches irq-sifive-plic driver.<!--{{{-->
hut% rg 'riscv,plic0' hut% rg 'riscv,plic0'
Documentation/devicetree/bindings/interrupt-controller/sifive,plic-1.0.0.yaml Documentation/devicetree/bindings/interrupt-controller/sifive,plic-1.0.0.yaml
@ -244,7 +245,7 @@ be compiled in.
So it looks that the only driver that setups the plic is the one used by SiFive. So it looks that the only driver that setups the plic is the one used by SiFive.
Here is the doc: https://static.dev.sifive.com/U54-MC-RVCoreIP.pdf Here is the doc: https://static.dev.sifive.com/U54-MC-RVCoreIP.pdf
<!--}}}--> <!--}}}-->
**Observation**: The number of handlers is 0, so there are no interruptions.<!--{{{--> ### OBSERVATION: The number of handlers is 0, so there are no interruptions.<!--{{{-->
It seems the number next to the phandle of the interrupts-extended attribute in It seems the number next to the phandle of the interrupts-extended attribute in
the plic follows a different convention of values. Using 9 and 11: the plic follows a different convention of values. Using 9 and 11:
@ -257,7 +258,7 @@ polled, otherwise it hangs.<!--}}}-->
## 2024-07-04 ## 2024-07-04
**Observation**: I saw they changed this option in Cinco Ranch DTS for the<!--{{{--> ### OBSERVATION: I saw they changed this option in Cinco Ranch DTS for the<!--{{{-->
serial: serial:
> reg-shift = <0>; // regs are spaced on 8 bit boundary (modified from Xilinx UART16550 to be ns16550 compatible) > reg-shift = <0>; // regs are spaced on 8 bit boundary (modified from Xilinx UART16550 to be ns16550 compatible)
@ -265,7 +266,7 @@ serial:
Tested booting with debug1 and the ttyS0 console, and it goes extremely slow Tested booting with debug1 and the ttyS0 console, and it goes extremely slow
(but still outputs at 115200) and then continues to fail to read keyboard input. (but still outputs at 115200) and then continues to fail to read keyboard input.
<!--}}}--> <!--}}}-->
**Question**: Let's try setting the console in poll mode.<!--{{{--> ### QUESTION: Let's try setting the console in poll mode.<!--{{{-->
setenv bootargs "root=/dev/ram0 loglevel=7 debug rw earlycon=uart,io,0x40001000,115200n8 boot.trace console=uart,io,0x40001000,115200n8 debug1 init=/nix/store/wavmnv6wjj8y10ha07wxd5f0sqacivj8-nixos-system-nixos-riscv-23.11pre-git/init" setenv bootargs "root=/dev/ram0 loglevel=7 debug rw earlycon=uart,io,0x40001000,115200n8 boot.trace console=uart,io,0x40001000,115200n8 debug1 init=/nix/store/wavmnv6wjj8y10ha07wxd5f0sqacivj8-nixos-system-nixos-riscv-23.11pre-git/init"
@ -304,17 +305,17 @@ setenv bootargs "root=/dev/ram0 loglevel=7 debug rw earlycon=uart,io,0x40001000,
Also found: `no_console_suspend` Also found: `no_console_suspend`
<!--}}}--> <!--}}}-->
**Observation**: There are messages of address space being assigned to<!--{{{--> ### OBSERVATION: There are messages of address space being assigned to<!--{{{-->
registers: registers:
Slave segment '/MEEP_uart_0/S_AXI/Reg' is being assigned into address space '/m_axi_uart0' at <0x0000_0000 [ 4K ]>. Slave segment '/MEEP_uart_0/S_AXI/Reg' is being assigned into address space '/m_axi_uart0' at <0x0000_0000 [ 4K ]>.
Slave segment '/MEEP_uart_1/S_AXI/Reg' is being assigned into address space '/m_axi_uart1' at <0x0000_0000 [ 4K ]>. Slave segment '/MEEP_uart_1/S_AXI/Reg' is being assigned into address space '/m_axi_uart1' at <0x0000_0000 [ 4K ]>.
<!--}}}--> <!--}}}-->
**Question**: What happens if I enable `CONFIG_CONSOLE_POLL`?<!--{{{--> ### QUESTION: What happens if I enable `CONFIG_CONSOLE_POLL`?<!--{{{-->
With `console=ttyS0,115200n8 debug1` I cannot type. With `console=ttyS0,115200n8 debug1` I cannot type.
<!--}}}--> <!--}}}-->
**Observation**: I can dump iomem memory with the tool devmem:<!--{{{--> ### OBSERVATION: I can dump iomem memory with the tool devmem:<!--{{{-->
But it seems I cannot dump the registers of the serial io mapped region: But it seems I cannot dump the registers of the serial io mapped region:
@ -364,7 +365,7 @@ It works!
~ # devmem 0x40001000 ~ # devmem 0x40001000
0x0000000D 0x0000000D
<!--}}}--> <!--}}}-->
**Observation**: The interrupt register of the serial console is 0x0:<!--{{{--> ### OBSERVATION: The interrupt register of the serial console is 0x0:<!--{{{-->
Assuming the console registers follow AXI UART 16550, here is the IER: Assuming the console registers follow AXI UART 16550, here is the IER:
@ -380,8 +381,7 @@ The line control register is 0x3:
~ # devmem 0x4000100C ~ # devmem 0x4000100C
0x00000003 0x00000003
<!--}}}--> <!--}}}-->
**Question**: Can I write to some memory address and see the result from the<!--{{{--> ### QUESTION: Can I write to some memory address and see the result from the host?<!--{{{-->
host?
For that I would need to find some address that is mapped to the DMA or to the For that I would need to find some address that is mapped to the DMA or to the
pmem. Xavi recommended `0x6000_0000` as it is uncached. pmem. Xavi recommended `0x6000_0000` as it is uncached.
@ -467,18 +467,22 @@ But we don't see the same:
[bsc015557@fpgan02 nixos]$ dd if=/dev/qdma34000-MM-1 count=16 bs=1 skip=$FPGACTL_KERNEL_ADDR 2>/dev/null | xxd [bsc015557@fpgan02 nixos]$ dd if=/dev/qdma34000-MM-1 count=16 bs=1 skip=$FPGACTL_KERNEL_ADDR 2>/dev/null | xxd
00000000: 9797 9797 9797 9797 9797 9797 9797 9797 ................ 00000000: 9797 9797 9797 9797 9797 9797 9797 9797 ................
[bsc015557@fpgan02 nixos]$ dd if=/dev/qdma34000-MM-0 count=16 bs=1 skip=$FPGACTL_KERNEL_ADDR 2>/dev/null | xxd [bsc015557@fpgan02 nixos]$ dd if=/dev/qdma34000-MM-0 count=16 bs=1 skip=$FPGACTL_KERNEL_ADDR 2>/dev/null | xxd
00000000: 9797 9797 9797 9797 9797 9797 9797 9797 ................<!--}}}--> 00000000: 9797 9797 9797 9797 9797 9797 9797 9797 ................
**Question**: Missing forward M to S via Mideleg?<!--{{{-->
<!--}}}-->
### QUESTION: Missing forward M to S via Mideleg?<!--{{{-->
Can it be happening that he MEDELEG is not forwarding the interruptions to the Can it be happening that he MEDELEG is not forwarding the interruptions to the
Supervisor (kernel)? Supervisor (kernel)?
Boot HART MIDELEG : 0x0000000000000222 Boot HART MIDELEG : 0x0000000000000222
Boot HART MEDELEG : 0x000000000000b109 Boot HART MEDELEG : 0x000000000000b109
<!--}}}--> <!--}}}-->
**Question**: Can we add a timer to the PLIC to test the interrupts?<!--{{{--> ### QUESTION: Can we add a timer to the PLIC to test the interrupts?<!--{{{-->
<!--}}}--> <!--}}}-->
**Observation**: Here is the PLIC register dump:<!--{{{--> ### OBSERVATION: Here is the PLIC register dump:<!--{{{-->
~ # for i in `seq 0 16`; do addr=$((0x40600000 + $i)); printf '%08x: ' $addr; devmem $addr; done ~ # for i in `seq 0 16`; do addr=$((0x40600000 + $i)); printf '%08x: ' $addr; devmem $addr; done
40600000: 0x00010002 40600000: 0x00010002
@ -499,12 +503,12 @@ Supervisor (kernel)?
4060000f: 0x00000000 4060000f: 0x00000000
40600010: 0x00000000 40600010: 0x00000000
<!--}}}--> <!--}}}-->
**Question**: Can we boot with the new bitstream that includes the second UART?<!--{{{--> ### QUESTION: Can we boot with the new bitstream that includes the second UART?<!--{{{-->
The interruptions are enabled for the UART 1, not the default UART 0. The interruptions are enabled for the UART 1, not the default UART 0.
<!--}}}--> <!--}}}-->
**Observation**: I'm using 0x100 not 0x1000 in the serial range:<!--{{{--> ### OBSERVATION: I'm using 0x100 not 0x1000 in the serial range:<!--{{{-->
reg = <0x0 0x40003000 0x0 0x100>; reg = <0x0 0x40003000 0x0 0x100>;
reg = <0x0 0x40003000 0x0 0x1000>; reg = <0x0 0x40003000 0x0 0x1000>;
@ -512,12 +516,13 @@ The interruptions are enabled for the UART 1, not the default UART 0.
Can this produce any problem? Can this produce any problem?
It doesn't seem to change anything, still unable to send any bytes. It doesn't seem to change anything, still unable to send any bytes.
<!--}}}--> <!--}}}-->
**Question**: Can we use virtio to mount a FS in the DMA shared memory? ### QUESTION: Can we use virtio to mount a FS in the DMA shared memory?
## 2024-07-05 ## 2024-07-05
**Observation**: The kernel continues working when the console hangs.<!--{{{--> ### OBSERVATION: The kernel continues working when the console hangs.<!--{{{-->
Switching to 0x100000000 as 0x60000000 shows: Switching to 0x100000000 as 0x60000000 shows:
@ -540,8 +545,9 @@ Shows the kernel works:
a0000000: 6700 0000 g... a0000000: 6700 0000 g...
a0000000: 6800 0000 h... a0000000: 6800 0000 h...
a0000000: 6900 0000 i... a0000000: 6900 0000 i...
<!--}}}--> <!--}}}-->
**Question**: Can we reproduce it with `switch_root`?<!--{{{--> ### QUESTION: Can we reproduce it with `switch_root`?<!--{{{-->
For that I would have to ensure the process continues to operate, even if we For that I would have to ensure the process continues to operate, even if we
exit the console. Maybe I can make a double fork? exit the console. Maybe I can make a double fork?
@ -563,10 +569,11 @@ Yes, it seems to be working. Let's load the rootfs too.
I added a loop in the stage1 script.<!--}}}--> I added a loop in the stage1 script.<!--}}}-->
**Question**: Can we see any clock in memory? This will allow us to check if the ### QUESTION: Can we see any clock in memory?
AXI still works.
**Observation**: The kernel stops updating the counter in the mount phase.<!--{{{--> This will allow us to check if the AXI still works.
### OBSERVATION: The kernel stops updating the counter in the mount phase.<!--{{{-->
Managed to reach the mount and hang there: Managed to reach the mount and hang there:
@ -595,7 +602,7 @@ hardware clock from the DMA region too, so we can discard problems in the AXI.<!
+ loadkmap + loadkmap
[ 266.301040] stage-1-init: [Thu Jan 1 00:04:25 UTC 1970] + kbd_mode -u -C /dev/console [ 266.301040] stage-1-init: [Thu Jan 1 00:04:25 UTC 1970] + kbd_mode -u -C /dev/console
**Assumption**: The kernel hangs.<!--{{{--> ### ASSUMPTION: The kernel hangs.<!--{{{-->
If the kernel hangs, there must be an instruction or sequence of instructions If the kernel hangs, there must be an instruction or sequence of instructions
that causes it. First I need to determine what is being executed by the kernel. that causes it. First I need to determine what is being executed by the kernel.
@ -607,16 +614,18 @@ hangs.
(prev_comm != 2 && next_comm != 2) (prev_comm != 2 && next_comm != 2)
So, we can just enable the `tp_printk` but not the tracer. Then in the initrd So, we can just enable the `tp_printk` but not the tracer. Then in the initrd
script, I enable the function tracer and the filter.<!--}}}--> script, I enable the function tracer and the filter.
**Observation**: It takes a long time to init the pty:<!--{{{--> <!--}}}-->
### OBSERVATION: It takes a long time to init the pty:<!--{{{-->
Interesting timing: Interesting timing:
[ 12.612620] initcall_start: func=pty_init+0x0/0x3f4 [ 12.612620] initcall_start: func=pty_init+0x0/0x3f4
[ 20.962640] initcall_finish: func=pty_init+0x0/0x3f4 ret=0 [ 20.962640] initcall_finish: func=pty_init+0x0/0x3f4 ret=0
<!--}}}--> <!--}}}-->
**Observation**: The kcompactd0 daemon is using the CPU:<!--{{{--> ### OBSERVATION: The kcompactd0 daemon is using the CPU:<!--{{{-->
[ 290.394920] sched_switch: prev_comm=devmem prev_pid=129 prev_prio=120 prev_state=R ==> next_comm=init next_pid=69 next_prio=120 [ 290.394920] sched_switch: prev_comm=devmem prev_pid=129 prev_prio=120 prev_state=R ==> next_comm=init next_pid=69 next_prio=120
[ 290.408160] sched_switch: prev_comm=init prev_pid=69 prev_prio=120 prev_state=R ==> next_comm=tee next_pid=68 next_prio=120 [ 290.408160] sched_switch: prev_comm=init prev_pid=69 prev_prio=120 prev_state=R ==> next_comm=tee next_pid=68 next_prio=120
@ -644,8 +653,9 @@ Interesting timing:
[ 290.699720] sched_switch: prev_comm=ksoftirqd/0 prev_pid=12 prev_prio=120 prev_state=R ==> next_comm=init next_pid=1 next_prio=120 [ 290.699720] sched_switch: prev_comm=ksoftirqd/0 prev_pid=12 prev_prio=120 prev_state=R ==> next_comm=init next_pid=1 next_prio=120
[ 290.712880] sched_switch: prev_comm=init prev_pid=1 prev_prio=120 prev_state=R ==> next_comm=khvcd next_pid=31 next_prio=120 [ 290.712880] sched_switch: prev_comm=init prev_pid=1 prev_prio=120 prev_state=R ==> next_comm=khvcd next_pid=31 next_prio=120
[ 290.725500] sched_switch: prev_comm=khvcd prev_pid=31 prev_prio=120 prev_state=R ==> next_comm=kcompactd0 next_pid=22 next_prio=120 [ 290.725500] sched_switch: prev_comm=khvcd prev_pid=31 prev_prio=120 prev_state=R ==> next_comm=kcompactd0 next_pid=22 next_prio=120
<!--}}}--> <!--}}}-->
**Question**: Can we reproduce this hang with 6.9.7? ### QUESTION: Can we reproduce this hang with 6.9.7?<!--{{{-->
Disabling clang as it is failing to build: Disabling clang as it is failing to build:
@ -669,3 +679,4 @@ Disabling clang as it is failing to build:
error: 1 dependencies of derivation '/nix/store/b13shgqj7128rdsdzzp4qicqbzl0wnfw-system-path.drv' failed to build error: 1 dependencies of derivation '/nix/store/b13shgqj7128rdsdzzp4qicqbzl0wnfw-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/6qghlihqcyg6155309ldj5xm9m0v835i-nixos-system-nixos-riscv-24.11pre-git.drv' failed to build error: 1 dependencies of derivation '/nix/store/6qghlihqcyg6155309ldj5xm9m0v835i-nixos-system-nixos-riscv-24.11pre-git.drv' failed to build
error: 1 dependencies of derivation '/nix/store/l2x18cih29r1kn6vi8imwhkyk98yhw4i-nix-shell-riscv64-unknown-linux-gnu-env.drv' failed to build error: 1 dependencies of derivation '/nix/store/l2x18cih29r1kn6vi8imwhkyk98yhw4i-nix-shell-riscv64-unknown-linux-gnu-env.drv' failed to build
<!--}}}-->