ESP32 board support?

[dgtlrift]

I see there's ESP8266 support, any chance of the new ESP32 support being added?

[jcmvbkbc]

You might have noticed that ESP8266 support is very rudimentary.

Espressif haven't provided any feedback to my request for the hardware specifications of ESP32, I haven't been following ESP32 development/reverse engineering closely and I don't have the hardware ATM.
For the ESP32 support to appear something of the following need to happen: either somebody contributes the implementation (ideal, but doesn't sound likely), or somebody publishes hardware specifications (at least the subset with the memory layout and core peripherals description/register maps)/ROM dumps. In addition somebody willing to test that QEMU model behavior is somewhat close to the real hardware behavior is needed.

[Ebiroll]

I got my ESP32 board from adafruit. I have managed to dump the ROM and use it to start an elf file compiled with esp-idf in my patched version of QEMU. You can see the results here, https://github.com/Ebiroll/qemu_esp32 . However, I was wondering, what is the difference of this repository and the xtensa support in QEMU? (https://github.com/qemu/qemu) Thank you.

[jcmvbkbc]

Hi @Ebiroll
the master branch of this tree follows (with some noticeable lag) the master branch of the QEMU mainline.
Other branches are experimental (e.g. windowed*, xtensa-perf or xtensa-esp8266), work in progress (e.g. xtensa-ccount, xtensa-tcg, xtensa-user), not currently upstreamable for some reason, but having somewhat useful functionality (e.g. xtensa-nommu, xtensa-smp*) or some kind of helpers (e.g. xtensa-cores, xtensa-fp-test, xtensa-staging*, xtensa-trace, xtensa-xtfpga).

[Ebiroll]

Thanks a lot. I guess the xtensa-esp8266 branch would be most helpful for esp32 emulation work.
I had to make some rom patches to make it run, but proper hardware emulation would be much better.

Running gdb with qemu on assembly level works quite well, however when stepping through the c-code I get this assertion in gdb: /home/ivan/e/crosstool-NG/.build/src/gdb-7.10/gdb/regcache.c:697: internal-error: regcache_raw_read_unsigned: Assertion `regnum >= 0 && regnum < regcache->descr->nr_raw_registers' failed.
This happens sometimes at return from a function. Maybe it has something to do with a new task being created.

Any chance you have an idea of whats the cause of this problem? Has it something to do with the windowed call mechanism or interrupts screwing up the registers? Its really annoying and discourages any further work on this. If not stepping in but rather just setting a breakpoint or running (next) it runs well.

Here is your toolchain that espresstif are using.
https://github.com/espressif/crosstool-NG
https://github.com/espressif/esp-idf/blob/master/docs/linux-setup.rst

Thanks for your great work anyways.
/Olof

[jcmvbkbc]

I guess the xtensa-esp8266 branch would be most helpful for esp32 emulation work.

Sure, feel free to reuse it if it helps.

Running gdb with qemu on assembly level works quite well, however when stepping through the c-code I get this assertion in gdb: /home/ivan/e/crosstool-NG/.build/src/gdb-7.10/gdb/regcache.c:697: internal-error: regcache_raw_read_unsigned: Assertion `regnum >= 0 && regnum < regcache->descr->nr_raw_registers' failed.
This happens sometimes at return from a function. Maybe it has something to do with a new task being created.
Any chance you have an idea of whats the cause of this problem?

I believe it's gdb issue. Have seen it multiple times in different contexts with different xtensa cores in recent gdb versions, I'll likely have a chance to look deeper into it this week.

[Ebiroll]

Have seen it multiple times in different contexts with different xtensa cores in recent gdb versions,
I looked closer at the assert.
Assertion `regnum >= 0 && regnum < regcache->descr->nr_raw_registers' failed.
I added some printouts to gdb and it seems that regnum is 116 and nr_raw_registers is 105. My guess is that 116 is the EPC1 register and that one is not included in the 'raw' registers.

If you look for register 116 you find it here
From gdb xtensa-config.c
XTREG(116,464,32, 4, 4,0x02b1,0x0007,-2, 2,0x1000,epc1, 0,0,0,0,0,0)

The only location that it is accessed by gdb is here,
In xtensa-tdep.c
static void
xtensa_window_interrupt_frame_cache (struct frame_info this_frame,
xtensa_frame_cache_t *cache,
CORE_ADDR pc)
{
...
/ Read PC of interrupted function from EPC1 register. */
epc1_regnum = xtensa_find_register_by_name (gdbarch,"epc1");
if (epc1_regnum < 0)
error(_("Unable to read Xtensa register EPC1"));

cache->ra = xtensa_read_register (epc1_regnum);
cache->pc = get_frame_func (this_frame);
}

I also did a analysis of the gdb-protocol that was sent, you can see it here,
https://github.com/Ebiroll/qemu_esp32/blob/master/gdbanalysis.md

When patching the xtensa_window_interrupt_frame_cache() functions it seems like qemu gets into to the interrupt function , _WindowUnderflow8, however I am not sure what to put in cache->ra.

It could be an error in the file core-esp32/gdb-config.c but when I add more than 105 registers then gdb complaints about Remote 'g' packet reply is too long....

[jcmvbkbc]

@Ebiroll looks like there are two issues here: first is that gdb overlay in the espressif crosstool-NG is not patched to mark all registers as unprivileged ( & ~1 in the 7th column of XTREG list in gdb/xtensa-config.c), second is that you've specified incorrect num_regs in https://github.com/Ebiroll/qemu_esp32/blob/master/qemu-patch/target-xtensa/core-esp32.c . It should work correctly with gdb corrected as said above if you just drop this line.

I've pushed a branch xtensa-esp32 with my version of the esp32 core to this repository, please take a look.
The combination of these two fixes works fine in my limited testing. I'll look at it some more.

[Ebiroll]

gdb overlay in the espressif crosstool-NG is not patched to mark all registers as unprivileged

Thanks, the gdb patch worked! That really improved the debugging experience when debugging with qemu. I also tested your xtensa-esp32 branch but had to comment out //gen_exception_cause(dc, ILLEGAL_INSTRUCTION_CAUSE); in the function gen_check_sr(), translate.c,
(register 97) SR 97 is not implemented
Other missing instructions,
WUR 234 not implemented, TBD(pc = 400971db)
WUR 235 not implemented, TBD(pc = 400971e0)
WUR 236 not implemented, TBD(pc = 400971e5)
But they do not generate exceptions. I guess these instructions are related to synchronisation between the cores.

static bool gen_check_sr(DisasContext *dc, uint32_t sr, unsigned access)
{
if (!xtensa_option_bits_enabled(dc->config, sregnames[sr].opt_bits)) {
if (sregnames[sr].name) {
qemu_log_mask(LOG_GUEST_ERROR, "SR %s is not configured\n", sregnames[sr].name);
} else {
qemu_log_mask(LOG_UNIMP, "SR %d %s is not implemented\n", sr);
}
//gen_exception_cause(dc, ILLEGAL_INSTRUCTION_CAUSE);

Keep the good work up and have a nice weekend!

[jcmvbkbc]

(register 97) SR 97 is not implemented

Oh, right, MEMCTL. I'll implement it.

WUR 234 not implemented, TBD(pc = 400971db)
WUR 235 not implemented, TBD(pc = 400971e0)
WUR 236 not implemented, TBD(pc = 400971e5)

These are the registers of the double precision floating point accelerator. I'll look at implementing it too.

BTW, this core (esp32) uses new hardware floating point, slightly different from what's in the QEMU right now, and with additional instructions for division and square root. It's still in WIP state at the xtensa-new-ops branch.

[Ebiroll]

Thanks for MEMCTL. It works fine.
Espressif have published a reference manual, updated with MMU chapter on the 9/11, 2016
http://espressif.com/sites/default/files/documentation/esp32_technical_reference_manual_en.pdf
Do you have the hardware now? I can make a pull request with ROM0 & ROM1 if you want.
Otherwise making a romdump is easy with esptool.py, esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 -b 921600 -p /dev/ttyUSB0 dump_mem 0x40000000 0x000C2000 rom0.bin

Also I have a question about the mapfile generated in the build directory by esp-idf. Is there anyway to use this information directly in gdb? The purpose of this is to get the name of the rom function in gdb when running in qemu. From the mapfile we know that uartAttach is here 0x40008fd0.
Rather than this,
(gdb) where
0x40008fd0 in ?? ()
0x40080990 in call_start_cpu0
I would like to see this,
0x40008fd0 in uartAttach
0x40080990 in call_start_cpu0
I tried adding -defsym=uartAttach=0x40008fd0 when linking but not with any success.
Try using, make VERBOSE=1 , with esp-idf to see how its compiled and linked. It would help a lot when trying to understand the code in the romdump.

[jcmvbkbc]

Espressif have published a reference manual, updated with MMU chapter on the 9/11, 2016

Thanks, I'll take a look.

Do you have the hardware now?

No, not yet. Looks like I won't have time to play with it for the following several weeks anyway.

I can make a pull request with ROM0 & ROM1 if you want.

Ok, I can take a look.

Also I have a question about the mapfile generated in the build directory by esp-idf. Is there anyway to use this information directly in gdb? The purpose of this is to get the name of the rom function in gdb when running in qemu.

Not that I know of. OTOH I did this for ESP8266 for the same purpose. The resulting elf may be loaded into gdb with 'add-symbol-file' command.

[Ebiroll]

Thanks, almost works for me, not sure how to modify this though

rom-functions.s: esp32.rom.ld
sed -n 's/PROVIDE[[:space:]]([[:space:]]([^[:space:]=]+)[^0]+(0x4.......).*/\1 = \2 - 0x40000000 + _stext\n.global \1\n.type \1, @function/p' < $^ &gt; $@
Whe I run I get this,
(gdb) add-symbol-file rom.elf
The address where rom.elf has been loaded is missing

Output from make
xtensa-esp32-elf-ld -M -T bootrom.ld -r bootrom.o -o bootrom.elf
xtensa-esp32-elf-ld: bootrom.elf section .text' will not fit in regionbootrom0'

Discarded input sections

.data 0x0000000000000000 0x0 bootrom.o
.bss 0x0000000000000000 0x0 bootrom.o

Memory Configuration

Name Origin Length Attributes
bootrom0 0x0000000040000000 0x0000000000010000
default 0x0000000000000000 0xffffffffffffffff

Linker script and memory map

.text 0x0000000040000000 0xc2000
*(.text)
.text 0x0000000040000000 0xc2000 bootrom.o
0x0000000040000000 _WindowOverflow4
0x0000000040000010 _xtos_alloca_handler
0x0000000040000040 _WindowUnderflow4
0x0000000040000080 _WindowOverflow8

[Ebiroll]

Sorry, I was in a hurry.
(gdb) add-symbol-file rom.elf 0x40000000 did the trick.

[Ebiroll]

@jcmvbkbc any time to update xtensa-esp32 yet?

I tried to look at the new-ops branch and tried to add the missing instructions here. https://github.com/Ebiroll/qemu-xtensa-esp32
There is not support for DFP accelerator in xtensa-gcc at the moment so I guess there is no need to add all floating point coprocessor instructions. I guess it is also not really necessary to add the RUR and WUR instructions either as its not used in the code. After the rom lets the control to the app.elf file xthal_restore_extra_nw is not called and these instructions are not called at all.

I have also added an open core lwip ethernet driver, but never got interrupts to work. I need to look closer at this. I did something like this.
xtensa_get_extint(env, 9)
s = SYS_BUS_DEVICE(dev);
sysbus_connect_irq(s, 0, irq);

But when I called qemu_irq_raise(s->irq); no interrupt was generated. Any ideas?
Probably some numbering mismatch or maybe I should try to call qemu_irq_pulse() in case the interrupt is edge triggered.

This is how interrupt handler is added by the real esp32 ether-net driver.
static void emac_enable_intr()
{
//init emac intr
REG_SET_FIELD(DPORT_PRO_EMAC_INT_MAP_REG, DPORT_PRO_EMAC_INT_MAP, ETS_EMAC_INUM);
xt_set_interrupt_handler(ETS_EMAC_INUM, emac_process_intr, NULL);
xt_ints_on(1 << ETS_EMAC_INUM);

REG_WRITE(EMAC_DMAINTERRUPT_EN_REG, EMAC_INTR_ENABLE_BIT);

}

[jcmvbkbc]

irq = xtensa_get_extint(env, 9)

AFAICS 9 is the internal IRQ number, not the external. Should be irq = env->irq_inputs[9];

[Ebiroll]

Thanks. It worked! I also spent a few extra hours because I forgot to add the argument -net nic,model=vlan0
:-P

Now I try to start both cores, but I am stuck here
x40009a69 <Cache_Flush+85> memw
0x40009a6c <Cache_Flush+88> l32i.n a10, a8, 0
0x40009a6e <Cache_Flush+90> bnone a10, a9, 0x40009a69 <Cache_Flush+85>
io read 58 DPORT_APP_CACHE_CTRL_REG 3ff00058=0x01

I guess returning 0 could do the trick. Any other advice for running 2 cores?
This is the startup code,
#if !CONFIG_FREERTOS_UNICORE
ESP_EARLY_LOGI(TAG, "Starting app cpu, entry point is %p", call_start_cpu1);
//Flush and enable icache for APP CPU
Cache_Flush(1);
Cache_Read_Enable(1);
esp_cpu_unstall(1);
//Enable clock gating and reset the app cpu.
SET_PERI_REG_MASK(DPORT_APPCPU_CTRL_B_REG, DPORT_APPCPU_CLKGATE_EN);
CLEAR_PERI_REG_MASK(DPORT_APPCPU_CTRL_C_REG, DPORT_APPCPU_RUNSTALL);
SET_PERI_REG_MASK(DPORT_APPCPU_CTRL_A_REG, DPORT_APPCPU_RESETTING);
CLEAR_PERI_REG_MASK(DPORT_APPCPU_CTRL_A_REG, DPORT_APPCPU_RESETTING);
ets_set_appcpu_boot_addr((uint32_t)call_start_cpu1);

while (!app_cpu_started) {
    ets_delay_us(100);
}

#else
(gdb) info threads
Id Target Id Frame

• 2 Thread 2 (CPU#1 [running]) 0x400003c0 in _DoubleExceptionVector ()
  1 Thread 1 (CPU#0 [running]) 0x40009a69 in Cache_Flush ()

Any thoughts, advice? I guess
SET_PERI_REG_MASK(DPORT_APPCPU_CTRL_A_REG, DPORT_APPCPU_RESETTING);
should cause a reset of the app cpu.
Thanks for all your great advice anyways.

[Ebiroll]

Ivan sent me updated info regarding inter core communication.

Communication between cores is done using an atomic compare-and-set instruction (S32C1I) , which is implemented as a transaction over the AHB bus. We also use shared memory for some things, mainly in FreeRTOS scheduler. Once FreeRTOS is running, we use its communication primitives (queues, semaphores) to communicate between tasks running on different CPUs.

A simple transaction on the AHB consists of an address phase and a subsequent data phase (without wait states: only two bus-cycles). Access to the target device is controlled through a MUX (non-tristate), thereby admitting bus-access to one bus-master at a time.

Any thoughts? Is S32C1I possible to implement?

How about Flash MMU support? Would that be difficult to implement? Would it help to look at the esp8266 implementation? (esp8266_spi)

[jcmvbkbc]

Any other advice for running 2 cores?
esp_cpu_unstall(1);
//Enable clock gating and reset the app cpu.
SET_PERI_REG_MASK(DPORT_APPCPU_CTRL_B_REG, DPORT_APPCPU_CLKGATE_EN);
CLEAR_PERI_REG_MASK(DPORT_APPCPU_CTRL_C_REG, DPORT_APPCPU_RUNSTALL);
SET_PERI_REG_MASK(DPORT_APPCPU_CTRL_A_REG, DPORT_APPCPU_RESETTING);
CLEAR_PERI_REG_MASK(DPORT_APPCPU_CTRL_A_REG, DPORT_APPCPU_RESETTING);

Looks like you need proper runstall signal to prevent app CPU from running too early. I've had it implemented in my xtensa-smp* branch, but IIRC it got broken in qemu-2.7. I'll look at it.

I guess SET_PERI_REG_MASK(DPORT_APPCPU_CTRL_A_REG, PORT_APPCPU_RESETTING); should cause a reset of the app cpu.

I agree.

Is S32C1I possible to implement?

It's there.

How about Flash MMU support? Would that be difficult to implement?

It doesn't look too difficult, but one part of it isn't entirely clear to me:

The decision an MPU can make, based on this information, is to allow or deny a process to access
the memory region or peripheral

in case of CPU being denied memory access, will it get any exception? Or only write would be ignored/read would return garbage?

[Ebiroll]

It worked! APP & PRO cpu running!
I (97142) cpu_start: Starting scheduler on PRO CPU.
I (68253) cpu_start: Starting scheduler on APP CPU.

Not sure about how the runstall stuff should have worked and I was not able to do a proper reset,
I tried this,
XtensaCPU *APPcpu = NULL; // Is set at init and saved for reset,

 CPUClass *cc = CPU_CLASS(APPcpu);
 XtensaCPUClass *xcc = XTENSA_CPU_GET_CLASS(APPcpu);
 CPUXtensaState *env = &APPcpu->env;
 cc->reset(env);   // core-dump. 

Instead I just set pc register like this instead of cc->reset(env);
env->pc = env->config->exception_vector[EXC_RESET];
Maybe you can tell me how to call reset_cpu(APPcpu)?

At startup, The app cpu (2) hangs in a _double exception until the pro cpu calls,
SET_PERI_REG_MASK(DPORT_APPCPU_CTRL_A_REG, PORT_APPCPU_RESETTING);
It seems like they communicate the app-cpu flash code startup adress with DPORT_APPCPU_CTRL_D_REG.

in case of CPU being denied memory access, will it get any exception? Or only write would be ignored/read would return garbage?

My guess would be that it would be ignored and maybe return the same value as previous memory access. You could probably test it on the actual hardware. See table 35. I could test if I knew how,

I am also very unsure of how to do MMU flash emulation. Do you have any example code?
I saw this,
c4ec11a

If you want to try running and implement SPI-flash emulation, I have implemented unpatch rom function that restores the original rom
int *unpatch=(int *) 0x3ff00088;
*unpatch=0x42;

Have a nice weekend anyways.

[Ebiroll]

One small observation regarding interrupts and -smp.
If starting 2 cores the lan driver only gets interrupts on the last created core.
How should interrupts work when running two cores?
Anyway, I changed the order so that PRO cpu is initated last. Maybe I am doing it wrong.

for (n = 0; n < smp_cpus; n++) {
    cpu = cpu_xtensa_init(cpu_model);
    if (cpu == NULL) {
        error_report("unable to find CPU definition '%s'",
                     cpu_model);
        exit(EXIT_FAILURE);
    }
    env = &cpu->env;

    if (smp_cpus==2) {
        if (n==1) {
            env->sregs[PRID] = 0xCDCD;
        } else {
            env->sregs[PRID] = 0xABAB;
            APPcpu = cpu;            
        }
    } else {
            // If only running one core, it should be PRO-CPU 
            env->sregs[PRID] = 0xCDCD;
    }

    qemu_register_reset(lx60_reset, cpu);
    /* Need MMU initialized prior to ELF loading,
     * so that ELF gets loaded into virtual addresses
     */
    cpu_reset(CPU(cpu));
}

I was doing it wrong. I was working too late.
I used the env = &cpu->env; variable, thats why only the last created core got the interrupts.

 open_net_init(system_memory,0x3ff69000,0x3ff69400 , 0x3FFF8000,
                  env->irq_inputs[9]
                  , nd_table);   //  

https://github.com/Ebiroll/qemu-xtensa-esp32/blob/master/hw/xtensa/esp32.c

[Ebiroll]

I saw your runstall changes, and I meged them.
Any plans on doing spi/flash support? Otherwise I will give it a try.
I will delete the pull request.

[Ebiroll]

Hello again, @jcmvbkbc
I used the code from the esp8266 branch and added flash support. It worked fine.
I use the values written to the MMU table to map from flash to physical memory, with
cpu_physical_memory_write . I also store the written data in a memory mapped flash file.
/* Flash MMU table for PRO CPU /
#define DPORT_PRO_FLASH_MMU_TABLE ((volatile uint32_t) 0x3FF10000)
Just to let you know, still work in progress. https://github.com/Ebiroll/qemu_esp32
Have fun & Happy new year.