Emulating Applications with Intel® SDE and Control Flow Enforcement Technology

Background

Intel® control-flow enforcement technology (CET) is a feature for security to guard from ROP (return oriented programming) and JOP (indirect jump oriented programming). Intel SDE provides an emulation to this feature that include:

• Stack checks
• Indirect branch checks

Running Intel SDE with CET emulation

The emulation of CET requires some initialization and additional emulation for existing (legacy) instructions. Therefore, it requires using a special knob (-cet) to tell Intel SDE to apply the additional functionality to these instructions.

The basic command line invocation is:

> sde -cet -- application

This knob enables only stack-checks and it checks the correlation between call and return instructions across the entire process. This means that if there are mismatch issues in the system libraries, they will be reports. Intel SDE provides a filtering mechanism to ignore and recover from failures. See below for OS specific considerations. 

Stack Checks

Running Intel SDE with the -cet knob turns on the stack checks. For each thread a shadow stack at the size of 1 page (4Kb) is allocated and the top of this page as is set as the shadow stack pointer (SSP). If this size is not enough, then users can use the shadow stack size knob to change it, see the knobs section below for the full knob name. Intel SDE reports the detected issues to a file. Intel SDE provides a knob to dump the errors to the standard error and a knob to control the output file name other than the default. 

> sde -cet -cet-stderr -- application
Control flow error: IP: 0x400764 expected (shadow stack): 0x2aaac06d6c36 got (actual return address): 0x2aaac06ea6d3
INS: ret

Intel SDE also provides an option to dump the thread's call stack at the time of the failure. This makes it easier to debug the problem. The default call stack depth is 10 entries, but you can control it with a knob.

When the application also has debug information then the call stack will also report source file and source line for each entry in the stack.

> sde -cet -cet-stderr -cet-call-stack -- application
Control flow error: IP: 0x0000000000400764 expected (shadow stack): 0x00002aaac064dc36 got (actual return address): 0x00002aaac06616d3
INS: ret

Call stack:
# IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
0# 0x00002aaac064db50 __libc_start_main /lib64/libc.so.6:0x00001eb50
1# 0x0000000000400520 __libc_start_main@plt<full-path-to-application>/application:0x000000520
2# 0x0000000000400550 _start <full-path-to-application>/application:0x000000550 at /usr/src/packages/BUILD/glibc-2.11.3/csu/../sysdeps/x86_64/elf/start.S:65

See below in the OS specific information for more details.

Indirect Branch Checks

The indirect jump checks part of control-flow enforcement technology consists of a state machine and a special instruction ENDBRANCH. This instruction (endoded as a NOP instruction in older CPUs) marks the indirect jump target address as safe. It is emitted by the compiler  and therefore requires a special compilation. Since applications often use dynamic libraries which were not compiled with ENDBRACH, a special legacy compatibility mode was added. 

Intel SDE provides two options to turn on indirect branch checks: include images or exclude images. When including an image, the checks happen only in the address range of the included images. When excluding images, the check happen on all the images except for the addresses in the excluded images. The -cet-endbr-exe knob is a shortcut to the include image knob and the name of the executable.

The ENDBRANCH instruction itself is considered new instruction which is legal only starting from a future CPU, therefore you need to specify the -future chip knob to indicate that it is legal.

> sde -future -cet -cet-stderr -cet-endbr-exe -- application.exe

Control flow ENDBRANCH error detected at IP: 0x400550 INS: xor ebp, ebp
Last branch IP: 0x2aaaaaaabb44 INS: jmp r12

Control flow ENDBRANCH error detected at IP: 0x400780 INS: mov qword ptr [rsp-0x28], rbp
Last branch IP: 0x2aabc0a2abc0 INS: call rbx

 Control flow error: IP: 0x400764 expected (shadow stack): 0x2aabc0a2ac36 got (actual return address): 0x2aabc0a3e6d3

INS: ret

Control flow ENDBRANCH error detected at IP: 0x400620 INS: cmp byte ptr [rip+0x200659], 0x0
Last branch IP: 0x2aaaaaab9865 INS: call qword ptr [r12+rax*8]

Control flow ENDBRANCH error detected at IP: 0x40080c INS: push rbp
Last branch IP: 0x2aaaaaab9882 INS: call rax

In this case Intel SDE checks for both the stack matches and indirect branch. The indirect branch checks happen only to the executable, and it found a few places where the indirect jumps where from external libraries into the executable. You can see in the report what is the indirect branch target address and what was the branch that cause this report.

Linux

There are known issues and considerations that exists when running with legacy runtime on Linux:

• Running 64 bits applications with stack checks has problems with C++ exceptions and setjmp/longjmp use. 
• Running 32 bits applications with stack checks has problems with the way the dynamic loader resolves external symbols (i.e. calls between images). These problem can bypassed by set the environment variable LD_BIND_NOW. As in the example below. However, this does not resolve issues with C++ exceptions and setjmp/longjmp.

  > sde -env LD_BIND_NOW 1 -cet -- app ...

• Running applications with indirect branch checks suffers from a few issues even when checking only the executable and it was compiled with CET checks. The main issues are indirect calls from the CRT files (init and fini calls). Here is an example of running "hello world" application.

  > sde -future -cet -cet-stderr -cet-endbr-exe -cet-call-stack -- hw64
  
  Control flow ENDBRANCH error detected at IP: 0x0000000000400410 INS: xor ebp, ebp 
  Last branch IP: 0x00002b27ba199c74 INS: jmp r12 
  Call stack:
  # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
  0# 0x0000000000400410 _start <path-to-exe>/hw64:0x000000410
  
  
  Control flow ENDBRANCH error detected at IP: 0x0000000000400500 INS: push r15
  Last branch IP: 0x00002b28cf74f7bd INS: call rbp
  Call stack:
  # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
  0# 0x0000000000400500 __libc_csu_init <path-to-exe>/hw64:0x000000500
  1# 0x00002b28cf74f7bf __libc_start_main /lib/x86_64-linux-gnu/libc.so.6:0x0000207bf
  2# 0x00002b28cf74f740 __libc_start_main /lib/x86_64-linux-gnu/libc.so.6:0x000020740
  3# 0x00000000004003f0 __libc_start_main@plt<path-to-exe>/hw64:0x0000003f0
  4# 0x0000000000400410 _start <path-to-exe>/hw64:0x000000410
  
  
  Hello
  Control flow ENDBRANCH error detected at IP: 0x0000000000400574 INS: sub rsp, 0x8
  Last branch IP: 0x00002b27ba1a9e03 INS: call rax
  Call stack:
  # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
  0# 0x0000000000400574 _fini <path-to-exe>/hw64:0x000000574
  1# 0x00002b27ba1a9e05 _dl_rtld_di_serinfo /lib64/ld-linux-x86-64.so.2:0x000010e05
  2# 0x00002b27ba1a9b00 _dl_rtld_di_serinfo /lib64/ld-linux-x86-64.so.2:0x000010b00
  3# 0x00002b27ba1a9ab0 _dl_rtld_di_serinfo /lib64/ld-linux-x86-64.so.2:0x000010ab0
  ...

   

Windows

There are known issues and limitation when running application with CET checks. Even running small console application (on Win10) might result in multiple error messages even when only doing stack checks. For example:

> sde -cet -cet-stderr -cet-call-stack -- hello.exe

Control flow error: IP: 0x7ff92fff0523 expected (shadow stack): 0x7ff959039600 got (actual return address): 0x7ff95900a052

INS: ret
Call stack:
# IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
0# 0x7ff959039600 NtTestAlert C:\WINDOWS\SYSTEM32\ntdll.dll:0x0000a9600

Could not unwind to previous frame: IP: 0x00007ff92fff0523 INS: ret
Call stack:
# IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
0# 0x7ff959039600 NtTestAlert C:\WINDOWS\SYSTEM32\ntdll.dll:0x0000a9600


hello

Intel SDE can exclude images also from stack checks, this is useful when there are stack check violations in system libraries. 

> sde -cet -cet-exclude-image ntdll.dll -- hello.exe

As in the Linux case, applications on Windows also have indirect branches from the C runtime code (linked into the executable) even if the application was compiled with control-flow protection.

When running Intel SDE with indirect branch checks on the executable you might get errors like the following:

> sde -future -cet -cet-call-stack -cet-stderr -cet-endbr-exe -cet-exclude-image ntdll.dll -- app.exe

Control flow ENDBRANCH error detected at IP: 0x00007ff76e5014ac INS: sub rsp, 0x28
Last branch IP: 0x00007ff956935290 INS: jmp rax
Call stack:
# IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
0# 0x00007ff76e5014ac unnamedImageEntryPoint <path>\app.exe:0x0000014ac
1# 0x00007ff95691836d BaseThreadInitThunk C:\WINDOWS\System32\KERNEL32.DLL:0x00000836d
2# 0x00007ff956918350 BaseThreadInitThunk C:\WINDOWS\System32\KERNEL32.DLL:0x000008350
3# 0x00007ff958ff7093 RtlUserThreadStart C:\WINDOWS\SYSTEM32\ntdll.dll:0x000067093

Control flow ENDBRANCH error detected at IP: 0x00007ff76e501ff0 INS: ret 0x0
Last branch IP: 0x00007ff76e501e20 INS: jmp qword ptr [rip+0x44479]
Call stack:
# IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
0# 0x00007ff76e503480 unnamedImageEntryPoint <path>\app.exe:0x000003480
1# 0x00007ff76e502ba4 unnamedImageEntryPoint <path>\app.exe:0x000002ba4
2# 0x00007ff76e502290 unnamedImageEntryPoint <path>\app.exe:0x000002290
3# 0x00007ff76e5016fc unnamedImageEntryPoint <path>\app.exe:0x0000016fc
4# 0x00007ff76e50129c .text <path>\app.exe:0x00000129c
5# 0x00007ff95691836d BaseThreadInitThunk C:\WINDOWS\System32\KERNEL32.DLL:0x00000836d
6# 0x00007ff956918350 BaseThreadInitThunk C:\WINDOWS\System32\KERNEL32.DLL:0x000008350
7# 0x00007ff958ff7093 RtlUserThreadStart C:\WINDOWS\SYSTEM32\ntdll.dll:0x000067093
...

Knobs

Intel SDE provides the following knobs to control the emulation of control flow enforcement technology:

-cet [default 0]                      Enable Intel(R) Control-flow enforcement emulation
-cet_abort [default 0]                Abort the run on the first control-flow ennforcement error
-cet_call_next [default 1]            Ignore call to next instruction followed with pop instruction
-cet_call_stack [default 0]           Present call stack with control flow enforcement errors
-cet_call_stack_depth [default 10]    Specify control flow enforcement errors call-stack max depth
-cet_endbr_exclude_image              Specify images to exclude from ENDBRANCH checking (repeatable)
-cet_endbr_exe [default 0]            Add the main executable to the ENDBRANCH checking
-cet_endbr_include_image              Specify images to enable ENDBRANCH checking (repeatable)
-cet_exclude_image                    Specify images to exclude shadow stack checks (repeatable)
-cet_output_file [default sde-cet-checker-out.txt]
                                      File name for control flow enforcement errors
-cet_shadow_stack_size [default 4096] Specify the size of the initial shadow stack
-cet_stderr [default 0]               Report control flow exception to standard error