ESP32-C3 memory model

[marinjurjevic]

Hi,

I am evaluating ESP32-C3 for a project which will use WiFi, TLS and MQTT. Since these components tend to use a lot of resources, my first step is understanding ESP32-C3 memory model and try to get a rough estimation of RAM usage.

I have tried getting help on Discord, but the conversation there is moving really fast and the thread got left behind, so I'll try get more coverage here on GH.

As a starting point, I'm using a sample mentioned in one of the previous issues (link).

Environment

Board: esp32c3_devkitm
Toolchain: zephyr-sdk-0.16.0 (riscv64-zephyr-elf-gcc)
Zephyr version: zephyr-v3.3.0-1372-g4939463dc2

Build Report

After building the sample, here's the report I get. One of the first things that confused me is RAM size. Per datasheet, 384KB is available on C3, but here it's reported 320KB - that's a significant difference of 64KB missing.

Memory region         Used Size  Region Size  %age Used
     mcuboot_hdr:          32 B         32 B    100.00%
        metadata:          28 B         32 B     87.50%
             ROM:      597278 B    4194240 B     14.24%
     iram0_0_seg:       85696 B       320 KB     26.15%
     irom0_0_seg:      517322 B    4194272 B     12.33%
     drom0_0_seg:      118172 B    4194240 B      2.82%
     dram0_0_seg:      320028 B       320 KB     97.66%
    rtc_iram_seg:          0 GB         8 KB      0.00%
        IDT_LIST:          0 GB         8 KB      0.00%
esptool.py v3.3

RAM report

I attached the complete report.

RAM report shows 450728B as 100% which makes no sense since I would expect 320KB (or 384KB). Maybe it's due to IRAM/DRAM overlaps so something gets added twice.
Also, I don't know what (hidden) stands for?

RAM report

Root                                                                                           450728 100.00%
├── (hidden)                                                                                   217437  48.24%
├── WORKSPACE                                                                                   10036   2.23%
│   └── modules                                                                                 10036   2.23%
│       └── hal                                                                                  9873   2.19%
│           └── espressif                                                                        9873   2.19%
└── ZEPHYR_BASE                                                                                205415  45.57%
    ├── drivers                                                                                  9600   2.13%
    │   └── wifi                                                                                 9314   2.07%
    │       └── esp32                                                                            9314   2.07%
    │          
    ├── kernel                                                                                 107094  23.76%
    │   ├── init.c                                                                               4613   1.02%
    │   │   ├── log_dynamic_os                                                                      4   0.00%
    │   │   ├── z_idle_stacks                                                                     512   0.11%
    │   │   ├── z_main_stack                                                                     4096   0.91%
    │   │   └── z_sys_post_kernel                                                                   1   0.00%
    │   ├── mempool.c                                                                           98324  21.81%
    │   │   ├── _system_heap                                                                       20   0.00%
    │   │   └── kheap__system_heap                                                              98304  21.81%
    │   ├── sched.c                                                                                33   0.01%
    │   │   ├── slice_expired                                                                       1   0.00%
    │   │   ├── slice_max_prio                                                                      4   0.00%
    │   │   ├── slice_ticks                                                                         4   0.00%
    │   │   └── slice_timeouts                                                                     24   0.01%
    │   ├── system_work_q.c                                                                      4096   0.91%
    │   │   └── sys_work_q_stack                                                                 4096   0.91%
    │   ├── timeout.c                                                                              20   0.00%
    │   │   ├── announce_remaining                                                                  4   0.00%
    │   │   ├── curr_tick                                                                           8   0.00%
    │   │   └── timeout_list                                                                        8   0.00%
    │   └── work.c                                                                                  8   0.00%
    │       └── pending_cancels                                                                     8   0.00%
    ├── lib                                                                                      8428   1.87%
    ├── modules                                                                                 32768   7.27%
    │           └── _mbedtls_heap                                                               32768   7.27%
    ├── samples    (ACTUAL APPLICATION)                                                         6130   1.36%
    └── subsys                                                                                  41395   9.18%
        ├── logging                                                                              9366   2.08%
        ├── net                                                                                 25532   5.66%
        └── shell                                                                                6497   1.44%

DTS

I then started digging into DTS and found esp32c3.dtsi.

zephyr/dts/riscv/espressif/esp32c3.dtsi

Lines 55 to 58 in 42d9cf1

	sram0: memory@3fc7c000 {
	compatible = "mmio-sram";
	reg = <0x3fc7c000 0x50000>;
	};

Why does DTS state starting address 0x3fc7c000 and size 0x50000 (320KB)?
Datasheet shows DRAM is mapped from 0x3fc8000 and addresses before are reserved.
Only IRAM is mapped to first 0x4000 (16KB) and that's used as cache for external memory.
Subtracting these 16KB from 400KB yields 384KB available RAM:

• 0x3fc80000 - 0x3fcdffff are DRAM addresses
• 0x40380000 - 0x403Dffff are IRAM adresses

Linker

Linker script for esp32c3 is a bit confusing.

Just like in dts above, address 0x3FC7C000 is defined as SRAM_DRAM_START. I'm assuming this is to make calculations consistent with two different address spaces regarding offsets (I_D_SRAM_OFFSET variable is defined as difference of two).
Actual addresses used for memory are SRAM_IRAM_ORG and SRAM_DRAM_ORG which are correct.

One more important note is the comment on SRAM_DRAM_END where it's mentioned that bootloader starts at 0x403D0000, so this could explain 64KB vanishing.

zephyr/soc/riscv/esp32c3/linker.ld

Lines 18 to 32 in 42d9cf1

	#define RAMABLE_REGION dram0_0_seg
	#define RODATA_REGION drom0_0_seg
	#define IRAM_REGION iram0_0_seg
	#define FLASH_CODE_REGION irom0_0_seg
	#define ROMABLE_REGION ROM
	#define SRAM_IRAM_START 0x4037C000
	#define SRAM_DRAM_START 0x3FC7C000
	#define ICACHE_SIZE 0x4000 /* ICache size is fixed to 16KB on ESP32-C3 */
	#define I_D_SRAM_OFFSET (SRAM_IRAM_START - SRAM_DRAM_START)
	#define SRAM_DRAM_END 0x403D0000 - I_D_SRAM_OFFSET /* 2nd stage bootloader iram_loader_seg start address */
	#define SRAM_IRAM_ORG (SRAM_IRAM_START + ICACHE_SIZE)
	#define SRAM_DRAM_ORG (SRAM_DRAM_START + ICACHE_SIZE)
	#define I_D_SRAM_SIZE SRAM_DRAM_END - SRAM_DRAM_ORG

Bootloader

The last thing I checked is bootloader linker script. The iram_loader_seg is consistent with application linker script comment and it has size 0x6000 (24K), followed by dram_seg of size 0x4000 (16K). This adds up to 0xa000 (40K) which leaves 24K unspecified.

At last, there is iram_seg defined at 0x403CE000. This is 0x2000 (8KB) before end of application SRAM_DRAM_END.

https://github.com/zephyrproject-rtos/hal_espressif/blob/f18f680fa010cef4a50072a11d418296f467594a/components/bootloader/subproject/main/ld/esp32c3/bootloader.ld#L7-L12

At the end of the linker script ROM bootloader memory usage is mentioned. I'm not sure how to interpret this since part of address space is before 0x403D0000 and part is after.

https://github.com/zephyrproject-rtos/hal_espressif/blob/f18f680fa010cef4a50072a11d418296f467594a/components/bootloader/subproject/main/ld/esp32c3/bootloader.ld#L177-L193

Summary

So that's what I collected for up to this point. I understand ESP32C3 has specific memory model with two different address spaces addressing the same physical memory.

I created a diagram to visualize how are all these things connected and I'd like to make sense of it.

---

Questions

I have a lot of questions. I'll list most important and number them for easier referencing in discussion.

1. What is (hidden) segment in RAM report and why does ram_report give wrong numbers?
2. Why is RAM address in esp32c3.dtsi defined as 0x3fc7c000 and its size 0x50000 (320KB)?
3. Why is RAM end address (calculated) equal to 0x3fcd000 -> 64KB before actual end? Is this for bootloader IRAM?
4. Assuming bootloader uses last 64KB, why is this space not available to application?
5. Why is iram_seg located before application RAM end address (SRAM_DRAM_END) ?
6. How does ROM (first-stage) bootloader use RAM?

[xyzzy42]

As I understand it, the ESP32 2nd stage bootloader (e.g. mcuboot or esp-idf bootloader) does not consume any I/DRAM once the application starts. I don't have an ESP32-C3 yet but it seems very similar.

[ 1] .iram_loader.text PROGBITS 40078000 003000 003b6e 00 AX 0 0 4
The esp-idf 2nd stage bootloader is linked to put its instruction code at 4007_8000. This is inside the 64kB section of SRAM0 used as the cache. The application will enable the cache (since it will almost certainly run from external flash and need the cache) and so it won't use this as IRAM anyway. So there is no issue with wasted IRAM at the point where both bootloader and application are mapped into IRAM.

[ 3] .dram0.bss NOBITS 3fff0000 001000 000030 00 WA 0 0 4
The bootloader's data is segment is linked to start at 3fff_0000. This is the last 64kB of SRAM1's data bus mapping. Zephyr will link so that the bss and noinit sections are at the end of its data segment. This means the app does not need to load any data into the end of the data segment before it starts. The space that was used by the bootloader's DRAM will be re-used as empty space for e.g. the heap and thread stacks once the application starts.

[marekmatej]

[ 1] .iram_loader.text PROGBITS 40078000 003000 003b6e 00 AX 0 0 4
The esp-idf 2nd stage bootloader is linked to put its instruction code at 4007_8000. This is inside the 64kB section of SRAM0 used as the cache. The application will enable the cache (since it will almost certainly run from external flash and need the cache) and so it won't use this as IRAM anyway. So there is no issue with wasted IRAM at the point where both bootloader and application are mapped into IRAM.

I think that technique is not used in the C3 as it is in ESP32 as you correctly noticed. I have tried to put the iram_loader to SRAM0 before to save some space in SRAM1, but I was not successful.

[marekmatej]

Hi @marinjurjevic,
thanks for your interest and great elaboration on the problem.
Let me try to answer your questions to some extent.

1. You are probably referring to west build -t ram_report which is based on elf-size-analyze. I think the (hidden) attributes do not refer to actual code. Try to use the original elf-size-analyze -R <file.elf> Maybe it will suit you more.

2. It is probably a typo/bug which comes from the calculus in the linker.ld. The correct number should be 0x3fc80000 as C3 does not have a data cache like other ESP32 which is placed just next to another(bigger) SRAM.

3. The symbol SRAM_DRAM_START from the C3 linker script is actually a helper variable used to calculate actual SRAM_DRAM_ORG which is actually used in memory sections. Also, it is used to create I_D_SRAM_OFFSET which is a nice number 0x700000.

4. There are safety margins for the bootloader so it won't hurt itself during the application loading.
   The ESP32 memory model is quite complex but not perfect and there is certainly room for improvements.

5. The IDF bootloader (but any other 2nd stage bootloader) during its lifetime, is trying to maneuver not to be in conflict with the ROM bootloader NOR the application being loaded.

6. The calculation of SRAM_DRAM_END is giving us some hints on how the ROM bootloader uses the ram. You can probably find more details if you dig deep into Espressif documentation.

[xyzzy42]

Why is RAM address in esp32c3.dtsi defined as 0x3fc7c000 and its size 0x50000 (320KB)?

From what I can see, the address in the dtsi file is not used anywhere. So the answer is, the dtsi is just wrong but is not used so it doesn't matter. On other platforms, the dts specified address is used as a macro in the linker script. But the esp32c3 linker script hard codes the value.

Just like in dts above, address 0x3FC7C000 is defined as SRAM_DRAM_START. I'm assuming this is to make calculations consistent with two different address spaces regarding offsets

Yes, I think that's it too. They could use the correct address and adjust some other macros to take that into account. Like this:

#define SRAM_IRAM_START     0x4037C000
#define SRAM_DRAM_START     0x3FC80000  /* same address as datasheet now */
#define ICACHE_SIZE         0x4000 /* ICache size is fixed to 16KB on ESP32-C3 */
#define SRAM_IRAM_ORG       (SRAM_IRAM_START + ICACHE_SIZE)
#define SRAM_DRAM_ORG       (SRAM_DRAM_START)
#define I_D_SRAM_SIZE       SRAM_DRAM_END - SRAM_DRAM_ORG
#define I_D_SRAM_OFFSET     (SRAM_IRAM_ORG - SRAM_DRAM_ORG)
#define SRAM_DRAM_END       (0x403D0000 - I_D_SRAM_OFFSET)  /* 2nd stage bootloader iram_loader_seg start address */

Having done that, the resulting linker script produces the same result as before when it had the incorrect dram start address.

Something interesting to me about the dram segment, is that contains in order the sections bss, noinit, .data. The first two can be full of other data (i.e., bootloader) when starting and the app init will zero bss when it starts. But .data needs to be initialized before the app starts. Having it at the end means the end of the data segment can not overlap the bootloader if the bootloader is placed at the end of the SRAM region. I think ESP32 uses this trick to avoid losing RAM to the bootloader.

I see that the division between IRAM and DRAM is aligned to 16 bytes, i.e. in the example it is 0x1_4EC0. But from the ESP32-C3 TRM §14.4.2.2, "Note that the split address must be aligned to 512 bytes, meaning you can only write the integral multiples of 0x200 to the SPLITADDR field." So how does it work when IRAM and DRAM are not being split at a 512 byte aligned boundary?

[xyzzy42]

From what I can see, Zephyr not aligning the IRAM/DRAM split to 512 bytes is a bug. The esp-idf does align to 512 on the ESP32-C3:

#define SOC_MEMPROT_MEM_ALIGN_SIZE          512
_esp_memprot_align_size = CONFIG_SOC_MEMPROT_MEM_ALIGN_SIZE;
  .iram0.text_end (NOLOAD) :
  {
    /* ESP32-C3 memprot requires 16B padding for possible CPU prefetch and 512B alignment for PMS split lines */
    . += _esp_memprot_prefetch_pad_size;
    . = ALIGN(_esp_memprot_align_size);
  } > iram0_0_seg

The reason Zephyr runs instead crashing on a invalid I/DBUS access, is that the PMS is never programmed for any of the IRAM or DRAM permissions! Effectively, all of SRAM is accessible from the IRAM region and the DRAM region at the same time. Not just mapped into the virtual address space, which is always how the the ESP32-C3 works, bus accessible and usable as both code and data. There's no protection. It's possible to write data into IRAM with a buffer overflow or to execute code on the stack with return address corruption.

The esp-idf has a similar issue to @marinjurjevic 's points 2-4, about the missing 64kB. In esp-idf, there is even a bit more missing SRAM, #define SRAM_DRAM_END 0x403CE710. Some space before the last 64 kB (at 403C_D000) is also reserved. But on esp-idf this does not result in totally lost RAM like it does on Zephyr.

This is because the ESP-IDF also puts the sections into DRAM in a different order than Zephyr. It uses the order .data, .noinit, .bss, heap. It requires the first three to fit in into dram0_0_seg segment, but not the last: the heap. So the heap can use memory previously occupied by the bootloader. Basically, what I said would be a better way in the my previous comment. I do not know why noinit and bss are not allowed to overlap the bootloader, as they are noload sections that don't need to write into RAM until the app is running.

On Zephyr, unless I have missed something, the heap is allocated out of space reserved in the noinit section.

I tried re-ordering the Zephyr sections and it seems to work fine. So perhaps this is a bug in the Zephyr linker setup for ESP32-C3 and the bss, noinit, and heap could use the bootloader's SRAM.

[ankayca]

Hey,
Have you solved this problem?
I tried to run an "hello world" HTTP server on zephyr rtos with esp32c3 then i realized that a basic HTTP server takes %80 of dram.

[marekmatej]

@marinjurjevic, @ankayca, can you please rebase your work to use the latest changes? The PR with heap optimization was merged recently and it allows more statical allocation for the user code.