ARM: Start/Wakeup/Bringup the other CPU cores/APs and pass execution start address?

Question

I've been banging my head with this for the last 3-4 days and I can't find a DECENT explanatory documentation (from ARM or unofficial) to help me. I've got an ODROID-XU board (big.LITTLE 2 x Cortex-A15 + 2 x Cortex-A7) board and I'm trying to understand a bit more about the ARM architecture. In my "experimenting" code I've now arrived at the stage where I want to WAKE UP THE OTHER CORES FROM THEIR WFI (wait-for-interrupt) state.

The missing information I'm still trying to find is:

1. When getting the base address of the memory-mapped GIC I understand that I need to read CBAR; But no piece of documentation explains how the bits in CBAR (the 2 PERIPHBASE values) should be arranged to get to the final GIC base address

2. When sending an SGI through the GICD_SGIR register, what interrupt ID between 0 and 15 should I choose? Does it matter?

3. When sending an SGI through the GICD_SGIR register, how can I tell the other cores WHERE TO START EXECUTION FROM?

4. How does the fact that my code is loaded by the U-BOOT bootloader affect this context?

The Cortex-A Series Programmer's Guide v3.0 (found here: link) states the following in section 22.5.2 (SMP boot in Linux, page 271):

While the primary core is booting, the secondary cores will be held in a standby state, using the WFI instruction. It (the primary core) will provide a startup address to the secondary cores and wake them using an Inter-Processor Interrupt(IPI), meaning an SGI signalled through the GIC

How does Linux do that? The documentation-S don't give any other details regarding "It will provide a startup address to the secondary cores".

My frustration is growing and I'd be very grateful for answers. Thank you very much in advance!

EXTRA DETAILS

Documentation I use:

• ARMv7-A&R Architecture Reference Manual
• Cortex-A15 TRM (Technical Reference Manual)
• Cortex-A15 MPCore TRM
• Cortex-A Series Programmer's Guide v3.0
• GICv2 Architecture Specification

What I've done by now:

• UBOOT loads me at 0x40008000; I've set-up Translation Tables (TTBs), written TTBR0 and TTBCR accordingly and mapped 0x40008000 to 0x8000_0000 (2GB), so I also enabled the MMU
• Set-up exception handlers of my own
• I've got Printf functionality over the serial (UART2 on ODROID-XU)

All the above seems to work properly.

What I'm trying to do now:

• Get the GIC base address => at the moment I read CBAR and I simply AND (&) its value with 0xFFFF8000 and use this as the GIC base address, although I'm almost sure this ain't right
• Enable the GIC distributor (at offset 0x1000 from GIC base address?), by writting GICD_CTLR with the value 0x1
• Construct an SGI with the following params: Group = 0, ID = 0, TargetListFilter = "All CPUs Except Me" and send it (write it) through the GICD_SGIR GIC register
• Since I haven't passed any execution start address for the other cores, nothing happens after all this

....UPDATE....

I've started looking at the Linux kernel and QEMU source codes in search for an answer. Here's what I found out (please correct me if I'm wrong):

• When powering up the board ALL THE CORES start executing from the reset vector
• A software (firmware) component executes WFI on the secondary cores and some other code that will act as a protocol between these secondary cores and the primary core, when the latter wants to wake them up again
• For example, the protocol used on the EnergyCore ECX-1000 (Highbank) board is as follows:

**(1)** the secondary cores enter WFI and when

**(2)** the primary core sends an SGI to wake them up

**(3)** they check if the value at address (0x40 + 0x10 * coreid) is non-null;

**(4)** if it is non-null, they use it as an address to jump to (execute a BX)

**(5)** otherwise, they re-enter standby state, by re-executing WFI

**(6)** So, if I had an EnergyCore ECX-1000 board, I should write (0x40 + 0x10 * coreid) with the address I want each of the cores to jump to and send an SGI

Questions:

• 1. What is the software component that does this? Is it the BL1 binary I've written on the SD Card, or is it U-BOOT?
• 2. From what I understand, this software protocol differs from board to board. Is it so, or does it only depend on the underlying processor?
• 3. Where can I find information about this protocol for a pick-one ARM board? - can I find it on the official ARM website or on the board webpage?

Pontine

I believe that this kind of initialization sequence (i.e. waking-up the secondary cores from standby) is done by ROM code which is pre-loaded by the chip manufacturer (i.e. TI, Samsung, ...). For example, this is how it is done in TI OMAP's. omappedia.org/wiki/Bootloader_Project – Anacrusis

I believe (obviously) it must be one of: ROM code, BL1 code => both inaccessible (BL1 is encrypted [signed]), or the bootloader...I've searched a bit through the bootloader sources but I haven't found any relation to this SMP protocol; plus, Linux doesn't seem to rely on arguments from the bootloader to discover the protocol (maybe through an ATAG), which leads me to believe that what I'm looking for is either in the ROM code or in the BL1 code...the question is which and, since they're inaccesible directly, where can I find details on what they do for my board (ODROID-XU)... – Pontine

I suppose you meant "i.e. putting the secondary cores in standby" – Pontine

This kind of details could be provided by some kind of manufacturer's SoC technical datasheet. That's why I would first try to look at the Samsung resources, specifically Exynos 5. I couldn'f find the appropriate sheet fast enough, so I provided a link to similar OMAP chip just to give you an idea how TI handles these things. – Anacrusis

oh, believe me, I tried that too; that list I've printed there of documentations is not complete: I also have Exynos Dual 5 User's Manual, searched for other relevant samsung documents...but unfortunately I couldn't find the needed information in their docs either; it's all a big mess...so I was hoping someone else that has already done this would help me out, maybe someone that knows the intrinsics of the linux kernel on ARM architectures... and unfortunately here's what the internet tells me link – Pontine

I'm still searching and trying to get something out of the kernel source code, I hope I'll find more relevant information there.. – Pontine

Unfortunately some manufacturers do not give away ALL of the relevant SoC information as a free-lunch (read it as: they do but under the NDA). This SoC, as you say, appears to be in that group. But again, this kind of functionality is not ought to be exposed to a user. That's why the TI handles it as a ROM code. I would expect from Samsung to have done it the same. Therefore, I would not expect that you will find any of such implementation details in Linux kernel. – Anacrusis

But again if you have enough time, you could browse through the platform specific (Exynos 5) stuff and see if you can find something indicative. Or try to get support directly from Samsung. Good luck :) – Anacrusis

I had a quick look at the Exynos related code I found in the Android related release. There you can find some SMP related code. There are quite few things related to booting the secondary cores. So, have a look at the following files: android.googlesource.com/kernel/exynos/+/… android.googlesource.com/kernel/exynos/+/… – Anacrusis

Could I ask you to update your question a bit to highlight what exactly are your current questions? You've clearly done a lot of work in investigating this, and have updated the question with your findings, so I no longer know for example whether the GIC base address is a question anymore (if I recall correctly, it is GICbase = ((CBAR & 0x000000FF) << 32) | (CBAR & 0xFFFF0000), but it's been more than a year since I've worked on any ARM cores). – Libertylibia

Answer 1

Ok, I'm back baby. Here are the conclusions:

• The software component that puts the CPUs to sleep is the bootloader (in my case U-Boot)
• Linux somehow knows how the bootloader does this (hardcoded in the Linux kernel for each board) and knows how to wake them up again

For my ODROID-XU board the sources describing this process are UBOOT ODROID-v2012.07 and the linux kernel found here: LINUX ODROIDXU-3.4.y (it would have been better if I looked into kernel version from the branch odroid-3.12.y since the former doesn't start all of the 8 processors, just 4 of them but the latter does).

Anyway, here's the source code I've come up with, I'll post the relevant source files from the above source code trees that helped me writing this code afterwards:

typedef unsigned int DWORD;
typedef unsigned char BOOLEAN;
#define FAILURE (0)
#define SUCCESS (1)
#define NR_EXTRA_CPUS (3) // actually 7, but this kernel version can't wake them up all -> check kernel version 3.12 if you need this

// Hardcoded in the kernel and in U-Boot; here I've put the physical addresses for ease
// In my code (and in the linux kernel) these addresses are actually virtual
// (thus the 'VA' part in S5P_VA_...); note: mapped with memory type DEVICE
#define S5P_VA_CHIPID (0x10000000)
#define S5P_VA_SYSRAM_NS (0x02073000)
#define S5P_VA_PMU (0x10040000)
#define EXYNOS_SWRESET ((DWORD) S5P_VA_PMU + 0x0400)
// Other hardcoded values
#define EXYNOS5410_REV_1_0 (0x10)
#define EXYNOS_CORE_LOCAL_PWR_EN (0x3)

BOOLEAN BootAllSecondaryCPUs(void* CPUExecutionAddress){

// 1. Get bootBase (the address where we need to write the address where the woken CPUs will jump to)
//    and powerBase (we also need to power up the cpus before waking them up (?))
DWORD bootBase, powerBase, powerOffset, clusterID;

asm volatile ("mrc p15, 0, %0, c0, c0, 5" : "=r" (clusterID));
clusterID = (clusterID >> 8);
powerOffset = 0;
if( (*(DWORD*)S5P_VA_CHIPID & 0xFF) < EXYNOS5410_REV_1_0 )
{
    if( (clusterID & 0x1) == 0 ) powerOffset = 4;
}
else if( (clusterID & 0x1) != 0 ) powerOffset = 4;

bootBase = S5P_VA_SYSRAM_NS + 0x1C;
powerBase = (S5P_VA_PMU + 0x2000) + (powerOffset * 0x80);

// 2. Power up each CPU, write bootBase and send a SEV (they are in WFE [wait-for-event] standby state)
for (i = 1; i <= NR_EXTRA_CPUS; i++)
{
    // 2.1 Power up this CPU
    powerBase += 0x80;
    DWORD powerStatus = *(DWORD*)( (DWORD) powerBase + 0x4);

    if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == 0)
    {
        *(DWORD*) powerBase = EXYNOS_CORE_LOCAL_PWR_EN;
        for (i = 0; i < 10; i++) // 10 millis timeout
        {
            powerStatus = *(DWORD*)((DWORD) powerBase + 0x4);
            if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == EXYNOS_CORE_LOCAL_PWR_EN)
                break;
            DelayMilliseconds(1); // not implemented here, if you need this, post a comment request 
        }
        if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) != EXYNOS_CORE_LOCAL_PWR_EN)
            return FAILURE;
    }
    if ( (clusterID & 0x0F) != 0 )
    {
        if ( *(DWORD*)(S5P_VA_PMU + 0x0908) == 0 )
        do { DelayMicroseconds(10); } // not implemented here, if you need this, post a comment request
        while (*(DWORD*)(S5P_VA_PMU + 0x0908) == 0);
        *(DWORD*) EXYNOS_SWRESET = (DWORD)(((1 << 20) | (1 << 8)) << i);
    }

    // 2.2 Write bootBase and execute a SEV to finally wake up the CPUs
    asm volatile ("dmb" : : : "memory");
    *(DWORD*) bootBase = (DWORD) CPUExecutionAddress;
    asm volatile ("isb");
    asm volatile ("\n   dsb\n   sev\n   nop\n");
}
return SUCCESS;
}

This successfully wakes 3 of 7 of the secondary CPUs.

And now for that short list of relevant source files in u-boot and the linux kernel:

• UBOOT: lowlevel_init.S - notice lines 363-369, how the secondary CPUs wait in a WFE for the value at _hotplug_addr to be non-zeroed and to jump to it; _hotplug_addr is actually bootBase in the above code; also lines 282-285 tell us that _hotplug_addr is to be relocated at CONFIG_PHY_IRAM_NS_BASE + _hotplug_addr - nscode_base (_hotplug_addr - nscode_base is 0x1C and CONFIG_PHY_IRAM_NS_BASE is 0x02073000, thus the above hardcodings in the linux kernel)

• LINUX KERNEL: generic - smp.c (look at function __cpu_up), platform specific (odroid-xu): platsmp.c (function boot_secondary, called by generic __cpu_up; also look at platform_smp_prepare_cpus [at the bottom] => that's the function that actually sets the boot base and power base values)

Pontine

Answer 2

For clarity and future reference, there's a subtle piece of information missing here thanks to the lack of proper documentation of the Exynos boot protocol (n.b. this question should really be marked "Exynos 5" rather than "Cortex-A15" - it's a SoC-specific thing and what ARM says is only a general recommendation). From cold boot, the secondary cores aren't in WFI, they're still powered off.

The simpler minimal solution (based on what Linux's hotplug does), which I worked out in the process of writing a boot shim to get a hypervisor running on the XU, takes two steps:

1. First write the entry point address to the Exynos holding pen (0x02073000 + 0x1c)
2. Then poke the power controller to switch on the relevant core(s): That way, they drop out of the secure boot path into the holding pen to find the entry point waiting for them, skipping the WFI loop and obviating the need to even touch the GIC at all.

Unless you're planning a full-on CPU hotplug implementation you can skip checking the cluster ID - if we're booting, we're on cluster 0 and nowhere else (the check for pre-production chips with backwards cluster registers should be unnecessary on the Odroid too - certainly was for me).

From my investigation, firing up the A7s is a little more involved. Judging from the Exynos big.LITTLE switcher driver, it seems you need to poke a separate set of power controller registers to enable cluster 1 first (and you may need to mess around with the CCI too, especially to have the MMUs and caches on) - I didn't get further since by that point it was more "having fun" than "doing real work"...

As an aside, Samsung's mainline patch for CPU hotplug on the 5410 makes the core power control stuff rather clearer than the mess in their downstream code, IMO.

Kraal

thanks for the clarifications, I just wanted to say that I removed "Cortex-A15" from the title but haven't added the "Exynos" part, since the discussion can help ARM programmers in general, no matter the SoC, i.e. the procedure of waking up the other cores can be found similarly for other specific SoCs; thus, google searches can direct those here as well – Pontine

Answer 3

QEMU uses PSCI

The ARM Power State Coordination Interface (PSCI) is documented at: https://developer.arm.com/docs/den0022/latest/arm-power-state-coordination-interface-platform-design-document and controls things such as powering on and off of cores.

TL;DR this is the aarch64 snippet to wake up CPU 1 on QEMU v3.0.0 ARMv8 aarch64:

/* PSCI function identifier: CPU_ON. */
ldr w0, =0xc4000003
/* Argument 1: target_cpu */
mov x1, 1
/* Argument 2: entry_point_address */
ldr x2, =cpu1_entry_address
/* Argument 3: context_id */
mov x3, 0
/* Unused hvc args: the Linux kernel zeroes them,
 * but I don't think it is required.
 */
hvc 0

and for ARMv7:

ldr r0, =0x84000003
mov r1, #1
ldr r2, =cpu1_entry_address
mov r3, #0
hvc 0

A full runnable example with a spinlock is available on the ARM section of this answer: What does multicore assembly language look like?

The hvc instruction then gets handled by an EL2 handler, see also: the ARM section of: What are Ring 0 and Ring 3 in the context of operating systems?

Linux kernel

In Linux v4.19, that address is informed to the Linux kernel through the device tree, QEMU for example auto-generates an entry of form:

    psci {
            method = "hvc";
            compatible = "arm,psci-0.2", "arm,psci";
            cpu_on = <0xc4000003>;
            migrate = <0xc4000005>;
            cpu_suspend = <0xc4000001>;
            cpu_off = <0x84000002>;
    };

The hvc instruction is called from: https://github.com/torvalds/linux/blob/v4.19/drivers/firmware/psci.c#L178

static int psci_cpu_on(unsigned long cpuid, unsigned long entry_point)

which ends up going to: https://github.com/torvalds/linux/blob/v4.19/arch/arm64/kernel/smccc-call.S#L51

Tiebout

Answer 4

Go to www.arm.com and download there evaluation copy of DS-5 developement suite. Once installed, under the examples there will be a startup_Cortex-A15MPCore directory. Look at startup.s.

Garpike

This could be a lot more helpful with a little more elaboration. What does startup.s illustrate? – Countercheck

hi, sorry for not posting this earlier, I already succeeded solving this; I'll post my conclusions and solution tomorrow, right now I don't really have the time – Pontine