Emulating a RK3328 SoC ?

[Daimyo]

Hi, I am currently learning about the SoC RK3328 from Rockchip and would like to emulate this card via QEMU. I have no idea how to do this. I did a lot of research and came across an article that explained how to emulate a SoC on QEMU, using the virtual machine "virt" (https://wiki.qemu.org/Documentation/Platforms/ARM). This technique seems to me to be a good one but I have no idea how to use it and adapt it to my needs. And I can't get used to the idea that a simple virtual machine with specified peripherals can behave more or less the same way as an SoC. Could someone point me in the right direction? Thank you.

[zaval]

qemu does emulate some ARM boards, the page you posted lists them and also - instructs you how to get that list interactively from your qemu installation. virtualized board won't behave 1:1 as the real one. I suggest you use "virt" as a generic, easy to run target and not be lazy to use the real board for testing. I honestly don't understand people, wanting to "emulate" ARM SBCs. are you guys afraid to attach that UART or plug the PSU in? why you avoid touching the real board? and don't tell "it's hard", because you came up here to develop OS. programming qemu is fun, of course, but if you "want to learn rk3328" as you say, then you need to touch it. ARM is not as unified as x86, your "written for virt" OS won't run on, say, Rock64 (which I have), you need to test on that board, that you supposedly want to learn. finally, a cherry on top - there is no a single rk3328 board in the qemu list (the latter might be horribly incomplete and outdated though).

[nexos]

@zaval
The problem with running on real hardware is not that it's hard. It's that you get no debugging info at all on real hardware. If you triple fault, you don't have -d int, cpu_reset to save the day.

[rdos]

@zaval
The problem with running on real hardware is not that it's hard. It's that you get no debugging info at all on real hardware. If you triple fault, you don't have -d int, cpu_reset to save the day.

That's part of the challenge. I didn't have an emulator when I started with my brand new 386 processor in 1988. I only had the Intel manuals and a PC board with a 386SX processor on it.

[nexos]

How would you debug without a register state? QEMU gives you -d int,cpu_reset for that. I personally want to work smarter not harder .

[bzt]

How would you debug without a register state? QEMU gives you -d int,cpu_reset for that. I personally want to work smarter not harder .

Why don't you add a debugger? Here's an example. The basic idea is, you set up a handler with VBAR, and you put a minimal interactive debugger interface in the handler (about 300 SLoC). This way when an exception happens, you can examine the real hardware's state over a serial terminal. This example is written for AArch64 and also includes a disassembler. But you could use the same approach for AArch32 too (I'm not sure about RK3328 SoC, is it ARMv8?)

Cheers,
bzt

[rdos]

How would you debug without a register state? QEMU gives you -d int,cpu_reset for that. I personally want to work smarter not harder .

Initially, you will get a long row of fatal errors where you have no idea what is wrong. The first priority is to create exception handlers and let those print register state and then freeze. Before this works you need to use either reboots or freezes to see if some code is executed or not. Once you can print stuff on screen, you can print a lot of stuff and put freeze code at different positions to figure out where it stops working.

The next step is to create an integrated debugger where you can view memory & registers and single step the code after an exception has occured.

The final step is to create an embedded emulator that can show memory & register states of cores and emulate instructions so you can step the code even after the OS has crashed completely.

I have all of this. I have a simple register dump function that will dump registers if a fatal error occurs before the basic environment is setup. I also have a crash debugger that sets up a completely isolated environment that can dump core states and allows tracing the code using an emulator. This code can be activated by faults early in the boot process before the scheduler is initialized, and when fatal errors occurs in the kernel, like the kernel stack becoming exhausted by interrupt loops or faults in the scheduler code. Lastly, I have a thread-aware kernel debugger that can trace threads. The Watcom application debugger can also be used to trace into syscalls and debug kernel space code at source level, which is a must for C based drivers.

[Daimyo]

I honestly don't understand people, wanting to "emulate" ARM SBCs. are you guys afraid to attach that UART or plug the PSU in? why you avoid touching the real board? and don't tell "it's hard", because you came up here to develop OS. programming qemu is fun, of course, but if you "want to learn rk3328" as you say, then you need to touch it. ARM is not as unified as x86, your "written for virt" OS won't run on, say, Rock64 (which I have), you need to test on that board, that you supposedly want to learn. finally, a cherry on top - there is no a single rk3328 board in the qemu list (the latter might be horribly incomplete and outdated though).

The problem is not here. I am looking for an approximate solution to emulate that chip because I do not have the means to buy one real. If I could have one I would not think about emulating it.

[crosssans]

From what `qemu-system-aarch64 -machine help` gives us, I can say that it can't specifically emulate your SoC. However, by looking at the specifications, we can get closer to your target by passing these options:

Code: Select all

qemu-system-aarch64                     \
    -cpu cortex-a53                     \
    -machine virt                       \
    -device virtio-gpu-pci              \
    -vga std                            \
    -nic user,model=virtio-net-pci      \
    -kernel startup_image.img           \
    -drive if=sd,file=sd.img,format=raw

I'll briefly explain what these options do:

• `-cpu cortex-a53` indicates to QEMU to execute AARCH64 code as if it was a ARM Cortex A53 processor (this exactly matches your SoC CPU!)
• `-machine virt` is a mandatory option saying that we're not going to target a particular system board (we don't really have a choice here!)
• `-device virtio-gpu-pci` creates us a "GPU" for this virtual system (QEMU doesn't support specific GPU emulation, and I'm pretty sure that this will not let you do accelerated graphics on this, maybe this option has to be tinkered with other ones?)
• `-vga std` lets us actually see the rendered display
• `-nic user,model=virtio-net-pci` creates us a network access for this virtual system
• `-kernel startup_image.img` sets the startup image that will be used to initiate and load an operating system (it could just be a loader or an actual kernel, as the option stands for). Don't forget to rename `startup_image.img` to your actual image file that contains the boot code or the kernel for your operating system
• `-drive if=sd,file=sd.img` lets us use `sd.img` as an SD card/flash chip. Rename `sd.img` to your actual image that contains the rest of your operating system

This should help you get close to the SoC specifications, but it's still not identical to what you're going to target unfortunately

[Daimyo]

From what `qemu-system-aarch64 -machine help` gives us, I can say that it can't specifically emulate your SoC. However, by looking at the specifications, we can get closer to your target by passing these options:

Code: Select all

qemu-system-aarch64                     \
    -cpu cortex-a53                     \
    -machine virt                       \
    -device virtio-gpu-pci              \
    -vga std                            \
    -nic user,model=virtio-net-pci      \
    -kernel startup_image.img           \
    -drive if=sd,file=sd.img,format=raw

I'll briefly explain what these options do:

• `-cpu cortex-a53` indicates to QEMU to execute AARCH64 code as if it was a ARM Cortex A53 processor (this exactly matches your SoC CPU!)
• `-machine virt` is a mandatory option saying that we're not going to target a particular system board (we don't really have a choice here!)
• `-device virtio-gpu-pci` creates us a "GPU" for this virtual system (QEMU doesn't support specific GPU emulation, and I'm pretty sure that this will not let you do accelerated graphics on this, maybe this option has to be tinkered with other ones?)
• `-vga std` lets us actually see the rendered display
• `-nic user,model=virtio-net-pci` creates us a network access for this virtual system
• `-kernel startup_image.img` sets the startup image that will be used to initiate and load an operating system (it could just be a loader or an actual kernel, as the option stands for). Don't forget to rename `startup_image.img` to your actual image file that contains the boot code or the kernel for your operating system
• `-drive if=sd,file=sd.img` lets us use `sd.img` as an SD card/flash chip. Rename `sd.img` to your actual image that contains the rest of your operating system

This should help you get close to the SoC specifications, but it's still not identical to what you're going to target unfortunately

Thank you very much for your help crosssans ! I will take a deeper look into the Qemu documentation. I just have another stupid question: Does the memory mapping of the SoC is the same as the memory mapping of the emulator ? Or, differently said: Does the memory mapping of a peripheral device depend on it's specification or on the SoC architecture in itself ? Sorry if my question does not look logical or if the answer is obvious.

[zaval]

it's not even close to rk3328. rk3328 doesn't have PCI. rk3328 doesn't have vga, but it does have some weird 4ss Synopsys HDMI display controller, several SD host controllers and eMMC host controller, where the former are SDA compliant, the latter isn't, due to, well, - there is no eMMC HC standard. and it has ARM Mali 450 GPU, some mysterious video processing unit, image "enhancement" stuff and loads of other SoC specific controllers that aren't present in the above example. like for example, a quirky USB3 XHCI thing. I'd say, the only thing from the above example that is common with rk3328 is Cortex-A53 cores. you could pull the rk3328 Device Tree off and looking at it, try to find the closest set from what qemu can suggest, but still, it won't be an "rk3328 emulation", too many discrepancies. I mentioned specifically display and storage controllers here, because they are very important for an early OS development phase. what you would do without storage, right? and now, with rk3328, you are facing the need to have a vendor specific HC driver for eMMC. qemu won't give you that controller (most probably).

yes, memory map is SoC specific, since qemu doesn't emulate this SoC, you won't have the same memory map with that variant crosssans thinks is rk3328 close. Honestly, I don't know what qemu will be using as firmware in that setup, because all qemu I did, I did with UEFI, so there is another story, yet more different what rockchip and co do for now, I mean, I get ACPI and UEFI for the virt target, whereas these chinese SoC vendors can't do that yet. they use uboot as underfirmware. if you could feed qemu with that... but then you would need to fool uboot it's really an rk3328 machine, and again, it comes to the qemu not supporting this. I am sorry, that you can't afford a rk3328 board, I, too, can't afford everything I want. however 1GB Rock64 is about 25$, maybe you don't know, so I tell, and getting an SBC is better, than a set top box, even if the latter can be cheaper, because with an SBC, you can easier debug your code - the simplest is UART as I've mentioned.

[Daimyo]

it's not even close to rk3328. rk3328 doesn't have PCI. rk3328 doesn't have vga, but it does have some weird 4ss Synopsys HDMI display controller, several SD host controllers and eMMC host controller, where the former are SDA compliant, the latter isn't, due to, well, - there is no eMMC HC standard. and it has ARM Mali 450 GPU, some mysterious video processing unit, image "enhancement" stuff and loads of other SoC specific controllers that aren't present in the above example. like for example, a quirky USB3 XHCI thing. I'd say, the only thing from the above example that is common with rk3328 is Cortex-A53 cores. you could pull the rk3328 Device Tree off and looking at it, try to find the closest set from what qemu can suggest, but still, it won't be an "rk3328 emulation", too many discrepancies. I mentioned specifically display and storage controllers here, because they are very important for an early OS development phase. what you would do without storage, right? and now, with rk3328, you are facing the need to have a vendor specific HC driver for eMMC. qemu won't give you that controller (most probably).

yes, memory map is SoC specific, since qemu doesn't emulate this SoC, you won't have the same memory map with that variant crosssans thinks is rk3328 close. Honestly, I don't know what qemu will be using as firmware in that setup, because all qemu I did, I did with UEFI, so there is another story, yet more different what rockchip and co do for now, I mean, I get ACPI and UEFI for the virt target, whereas these chinese SoC vendors can't do that yet. they use uboot as underfirmware. if you could feed qemu with that... but then you would need to fool uboot it's really an rk3328 machine, and again, it comes to the qemu not supporting this. I am sorry, that you can't afford a rk3328 board, I, too, can't afford everything I want. however 1GB Rock64 is about 25$, maybe you don't know, so I tell, and getting an SBC is better, than a set top box, even if the latter can be cheaper, because with an SBC, you can easier debug your code - the simplest is UART as I've mentioned.

Thank you for your complete answer zaval. Finally, and because of what you mentioned in your message, I will not try to emulate that specific SoC. I will not buy one too, because shipping costs are going to be to high (nevertheless, I do not forget that buying a SBC is a better choice than emulating a SBC, to manipulate it). What do you mean when you say that UART is the simplest choice ?

[zaval]

simpler, than JTAG and sh1t. I meant, that UART is the simplest (ever) user I/O channel. simpler, than HDMI out or USB HID in. however, on set top boxes, UART header often isn't soldered or even exposed. with SBCs, which are "developers friendly", you have it made for you. you just need to connect your USB-UART adapter and enjoy your low level programming - with UART, you can have at least printing what's going on, and if you are nimble enough, you can embed the whole in-kernel debugger there.

[crosssans]

(remarks about emulation inexactitude)

Oh well, at last I tried As I said at the bottom of my post, "this should help you get close to the SoC specifications, but it's still not identical to what you're going to target unfortunately". This is why I specified "PCI" devices on the argument list so that the emulated machine had network access and one display output - As PCI is really a PC thing, I don't think they are actually hooked up as PCI devices, but rather as devices that would be integrated to the system although you specify them with names that have the word "PCI" in it (correct me if I'm wrong here!) - but otherwise you're right

(question about memory mapping)

As zaval said in his post, it won't have the same mapping of the SoC unfortunately. Maybe QEMU could let you change the mapped location of the attached devices with specific arguments that you pass - but I have no idea if it is possible or not, maybe a lookup in QEMU's documentation could give clues

[zaval]

Oh well, at last I tried As I said at the bottom of my post, "this should help you get close to the SoC specifications, but it's still not identical to what you're going to target unfortunately". This is why I specified "PCI" devices on the argument list so that the emulated machine had network access and one display output - As PCI is really a PC thing, I don't think they are actually hooked up as PCI devices, but rather as devices that would be integrated to the system although you specify them with names that have the word "PCI" in it (correct me if I'm wrong here!) - but otherwise you're right

you are wrong. the "pci" suffix there is to indicate, those devices are sitting on the PCI interface. what this "I don't think they are actually hooked up as PCI devices, but rather as devices that would be integrated to the system" mean, I honestly don't know. do you?