Skip to content

Return Slot Elimination

Doesn't put the return address on the stack for function calls where possible.

Where this happens

run in codegen/optimizations/eliminate_return_slot.rs.

How it works

On x86, function calling works by putting the return address on the stack.

call pushes it, and ret pops it to jump back.

So when the called function starts, the stack looks like this:

On entry Holds
[esp] return address
[esp+4] argument 0
[esp+8] argument 1

Recompiled code doesn't need this.

Since compiler will just use your CPU native calling convention.

So if every caller skips those 4 bytes, the arguments move down:

On entry Holds
[esp] argument 0
[esp+4] argument 1

It applies when:

Before and after

sub_2EE070 from Shadow the Hedgehog.
Finds the highest set bit of a positive argument, or -1 for 0.

Generated with Stack Argument Elimination and speed hacks off, so only this pass changes.

Recompiled C++

// push ecx
X86_PUSH(ctx, base, ctx.ecx.u32);
// call 0x002EE070
ctx.SubEsp(1); //Adjust stack pointer for call instruction
sub_2EE070(ctx, base);
// push ecx
X86_PUSH(ctx, base, ctx.ecx.u32);
// call 0x002EE070
X86_EXTERN_FUNC(sub_2EE070_fastcall);
sub_2EE070_fastcall(ctx, base);
X86_PROTECTED_FUNC(sub_2EE070) {
    // mov ecx, [esp+0x04]
    ctx.ecx.u32 = X86_LOAD_U32(ctx.esp.u32 + 0x4);
    // ...
loc_2EE087:
    // ret
    ctx.esp.u32 += 0x4;  //Adjust esp for return instruction
    return;
}
X86_FUNC_IMPL(sub_2EE070_fastcall) {
    // mov ecx, [esp]
    ctx.ecx.u32 = X86_LOAD_U32(ctx.esp.u32);
    // ...
loc_2EE087:
    // ret
    return;
}

X86_PROTECTED_FUNC(sub_2EE070) {
    ctx.esp.u32 += 0x4;
    sub_2EE070_fastcall(ctx, base);
}

Calls we rewrote go to sub_2EE070_fastcall. sub_2EE070 keeps the old layout for every other caller. See Known Callers.

Host assembly

Clang 21.1.8, x86-64, -O2.

The caller is cut off right after the call, so Clang turns it into a jmp.
Flag math is unchanged, so it's cut out.

mov    ecx, DWORD PTR [rdi+56]       ; guest ESP
lea    edx, [rcx-4]
mov    DWORD PTR [rsi+rdx], eax      ; push ecx
add    ecx, -8                       ; push + return address
mov    DWORD PTR [rdi+56], ecx       ; update guest ESP
jmp    sub_2EE070
mov    ecx, DWORD PTR [rdi+56]       ; guest ESP
add    ecx, -4                       ; push only
mov    DWORD PTR [rsi+rcx], eax      ; push ecx
mov    DWORD PTR [rdi+56], ecx       ; update guest ESP
jmp    sub_2EE070_fastcall
mov    eax, DWORD PTR [rdi+56]       ; guest ESP
add    eax, 4
mov    ecx, DWORD PTR [rsi+rax]      ; load argument 0
; ... the loop
mov    DWORD PTR [rdi+56], eax       ; ret: guest ESP += 4
mov    eax, DWORD PTR [rdi+56]       ; guest ESP
mov    eax, DWORD PTR [rsi+rax]      ; load argument 0
; ... the loop

ret no longer writes guest ESP.