Run the unmodified iOS kernel image on the VIM3 development board

Run the unmodified iOS kernel image on the VIM3 development board

I have seen articles on the Internet that have used QEMU to successfully run the IOS kernel and successfully mount the root file system. In theory, it can be successfully run on QEMU, and it is not a problem to run on a real ARMv8 development board. With the purpose of studying the IOS kernel startup process and the relevant security measures of the IOS kernel, after more than a week of research and debugging, the IOS kernel was successfully run on the Khadas VIM3 development board through virtualization. The following records are made. Some problems encountered during debugging.

Preparation and development stage

Debugging environment

IOS uses the xnu kernel. At present, Apple has open sourced the xnu source code (but the related driver code of the A series chip is not open). The specific source code can be found in the following Github repository: https://github.com/apple/darwin-xnu.git

The following are some articles experimented on QEMU. Through the following articles, you can learn a lot about the basic knowledge of XNU and IOS startup. You can directly enjoy the research results of Niuren to avoid repetitive wheel creation:

Zhuowei : https://worthdoingbadly.com/xnuqemu2/

Jonathan Afek： https://alephsecurity.com/2019/06/25/xnu-qemu-arm64-2/

The above articles are written in great detail. Each series is divided into two parts. The first part introduces how to do it. The second batch records the problems encountered and how to solve the problems. Both of them provide many tools for Extract the required firmware from the IOS upgrade package.

Next, based on the research results of Zhuowei and Afek, the process of running the IOS kernel on the Khadas VIM3 development board through the hypervisor, the first step is to build a basic debugging environment:

• Hypervisor uses Minos- https://github.com/minosproject/minos-hypervisor

• As the kernel files uncompressed by IOS keep fewer and fewer kernel symbols, and the remaining symbols are not helpful for debugging, they are basically the addresses of syscall. If all the problems have been recorded by Zhuowei and Afek, the theory Only need to transplant the relevant implementation code, you can run. But often the situation is more complicated, so there must be a tool that can be debugged to support, so take the following ideas

• 1. Use ARM DS5 as a debugging tool. DS5 is a very powerful integrated development and debugging tool of ARM architecture produced by ARM. Users can apply for a 30-day trial for free
  2. Debug on ARM FVP in the early stage, and can basically be directly transplanted to VIM3 after FVP runs through

IOS OTA file selection

Apple began to use the 64-bit ARMv8 architecture from the A7 chip. The following is some basic information of each generation of Apple chips

Screenshot from 2020-02-12 17-05-38 1332×749

Each upgrade of Apple’s SOC is quite fast, and keeps up with the latest ARM instruction set. The latest SOC uses the ARMv8.3 instruction set. Since the currently downloaded emulator of DS5 only supports the ARMv8.0 instruction set (in fact, the current development boards on the market are also ARMv8.0 instruction set, including VIM3), so only the first three can be selected. Here I choose Apple Typhoon, which is Apple A8 processor, and Iphone 6 uses A8 processor, so Iphone 6 IOS 12.2 firmware was downloaded from the following address as the debugging firmware

http://updates-http.cdn-apple.com/2019SpringFCS/fullrestores/041-06917/ECD402F4-499F-11E9-85D3-E64576CE070F/iPhone_4.7_12.2_16E227_Restore.ipsw

1. According to the tutorial in the first article of Jonathan Afek, download the relevant script and get the required image. Not all images mentioned in the tutorial are fully used here, there are two images and the difference mentioned by Afek

   1. secure monitor-Since we will start the ios kernel in the form of a virtual machine, here we will simulate the relevant SMC calls and some register access simulation in the hypervisor
   2. trust cache-This part of the Afek tutorial is a bit complicated, and needs to be switched to the Mac system (the relevant tools also provide the linux version, but the test is not easy to use), so I chose to directly load the recovery ramdisk as the root file system.

   Follow the steps in the Afek tutorial to get the following image

   1. kernel.out MACH-O format ios kernel image
   2. The device tree of dt.out device, the device tree of IOS kernel is also used, but the format of device-tree used by ios is not the same as the format under linux, so it is impossible to use the dtc tool to decompile directly. The general purpose of the device tree script in the Afek tutorial is to change the compatible field of almost all peripherals to 0, not to allow the system to load the corresponding driver, only to leave the uart and the most basic platform information, in fact, we only need for pre-debugging You can drive uart and timer interrupts together
   3. ramdisk.out ramdisk image

Bootloader

The first step we need to do is to load the extracted three images into memory, that is, we want to implement a simple IOS bootloader, IOS uses iboot as the bootloader, this part of the code is not open. Fortunately, Zhuowei has implemented the relevant code for loading MACH-O files in qemu. We only need to port these codes in qemu to Minos. There are two solutions:

1. Use U-boot as the bootloader, realize the analysis and loading of MACH-O files in uboot, and use IOS as the Native VM to start
2. Use Minos’s own virtual machine management program mvm to start, start a Guest VM

Obviously, the second method above will be simpler and more practical, and the operation of reading files will be more convenient. The code mainly implements the following functions

1. Parse the kernel file to get the entry address of the kernel
2. Calculate the addresses of ranmdisk, device-tree and bootarg according to the memory layout information provided by the kernel MACH-O file
3. Load the kernel image to the corresponding memory area
4. Set kernel startup parameters, and command line

In order to reduce as few problems as possible during the migration process, here I have allocated the VM as much memory as iphone6, which is 1G

With the configuration of 1G memory, I encountered a problem during the actual debugging process. Since the official Android image of VIM3 uses 32-bit and LPAE mode is not turned on, in order to simulate this situation, ARM FVP also uses 32-bit non-LPAE mode. The address space of the process under the bit is only 4G. By default, mvm will map all the memory space of the VM into its own process space. This means that it is necessary to find a 1G continuous virtual address space in the mvm process. This situation is almost not Too easy to satisfy. To this end, by modifying the mvm and hypervisor to achieve the function of mapping the VM specific area to the memory space, how much mapping is used. The specific implementation of bootloader can refer to https://github.com/minosproject/minos-hypervisor/blob/master/tools/mvm/os/os_xnu.c

Support of XNU operating system in Hypervisor

The xnu system startup requires that the bootloader needs to pass the startup parameters to the kernel. By reading the xnu arm64 startup code, it is found that the processing of this block is relatively simple. You only need to store the hardware address of the kernel startup parameters in the x0 register. The xnu arm64 startup code You can refer to https://github.com/apple/darwin-xnu/blob/master/osfmk/arm64/start.s , the definition of startup parameters is as follows, the settings of startup parameters can refer to the relevant code in mvm

struct xnu_arm64_boot_args {
        uint16_t revision;              /* Revision of boot_args structure */
        uint16_t version;               /* Version of boot_args structure */
        uint64_t virt_base;             /* Virtual base of memory */
        uint64_t phys_base;             /* Physical base of memory */
        uint64_t mem_size;              /* Size of memory */
        uint64_t top_of_kdata;          /* Highest physical address used in kernel data area */
        struct xnu_arm64_boot_video video; /* Video Information */
        uint32_t machine_type;          /* Machine Type */
        uint64_t dtb;                   /* Base of flattened device tree */
        uint32_t dtb_length;            /* Length of flattened tree */
        char cmdline[XNU_ARM64_BOOT_LINE_LENGTH]; /* Passed in command line */
        uint64_t boot_flags;            /* Additional flags specified by the bootloader */
        uint64_t mem_size_actual;       /* Actual size of memory */
};

Added https://github.com/minosproject/minos-hypervisor/blob/master/virt/os/os_xnu.c file in Minos to support xnu virtual machine startup

Interrupt controller support

Apple’s A series processors did not use ARM’s GIC series, but their own AIC. After searching the entire XNU code, only the AIC header file definition was found, and only the relevant register offset was defined in the header file. I do n’t know the principle of AIC ’s terminal controller. How can I send the interrupt message to the virtual machine? Fortunately, by looking at the previous two articles and the related code in xnu:

1. The system tick timer in xnu uses FIQ and does not use irq
2. The processing of FIQ in xnu is simple and rough, and there is no need to operate AIC at all.

void
sleh_fiq(arm_saved_state_t *state)
{
        unsigned int type   = DBG_INTR_TYPE_UNKNOWN;
#if DEVELOPMENT || DEBUG
        int preemption_level = get_preemption_level();
#endif
#if MONOTONIC
        uint64_t pmsr = 0, upmsr = 0; 
#endif /* MONOTONIC */

#if MONOTONIC
        if (mt_pmi_pending(&pmsr, &upmsr)) {
                type = DBG_INTR_TYPE_PMI;
        } else 
#endif /* MONOTONIC */
        if (ml_get_timer_pending()) {
                type = DBG_INTR_TYPE_TIMER;
        }

        sleh_interrupt_handler_prologue(state, type);

#if MONOTONIC
        if (type == DBG_INTR_TYPE_PMI) {
                mt_fiq(getCpuDatap(), pmsr, upmsr);
        } else 
#endif /* MONOTONIC */
        {
                /*
                 * We don't know that this is a timer, but we don't have insight into
                 * the other interrupts that go down this path.
                 */

                cpu_data_t *cdp = getCpuDatap();

                cdp->cpu_decrementer = -1; /* Large */

                /*
                 * ARM64_TODO: whether we're coming from userland is ignored right now.
                 * We can easily thread it through, but not bothering for the
                 * moment (AArch32 doesn't either).
                 */
                rtclock_intr(TRUE);
        }

        sleh_interrupt_handler_epilogue();
#if DEVELOPMENT || DEBUG
        if (preemption_level != get_preemption_level())
                panic("fiq type %u changed preemption level from %d to %d", type, preemption_level, get_preemption_level());
#endif
}

The entire FIQ process has only one big function. Ml_get_timer_pending in the code is used to determine whether there is a timer pending, and the implementation is also very simple. By reading the CNTP_CTL register and then judging the ISTATUS bit to determine whether there is a timer terminal pending, the reason for this implementation may be The software architecture of IOS / MAC is related to security. The AIC driver is implemented as a kext.

Because we only need a timer interrupt at the beginning, the support for AIC in Minos is very simple. You only need to send FIQ to the kernel. This can be done by setting the VFI flag of HCR_EL2. For specific implementation, please refer to https: //github.com/minosproject/minos/blob/master/virt/virq_chips/vaic.c

Timer support

Like the interrupt controller, the support for the timer is also placed in the hypervisor. The IOS also uses the standard timer of the ARM architecture, but unfortunately the Physical Timer is used instead of the Virtual Timer. Fortunately, Minos already has a simulation of the Physical timer. , Only need to make a few changes, there are two specific changes:

1. Change interrupt type from irq to fiq
2. Add code to clear the interrupt action

Serial port

Finally, in order to view the log output by the kernel, we also need to implement a virtual serial port device. The ios kernel uses Samsung’s serial port IP. The relevant code can be found in the xnu source code. The simulation of the serial port is implemented in mvm, which is currently only implemented. The most basic logging function is provided. The specific code is in the https://github.com/minosproject/minos-hypervisor/blob/master/tools/mvm/devices/s3c_uart.c file

ARM-FVP debugging stage

Through the above work, we have the required image and the basic environment code that supports the IOS kernel. Everything is ready, but we owe it to the east wind. Next, we will carry out the first stage of debugging in ARM-FVP. We can start the VM with the following command

./mvm vcpus=1 memory=1024M vm_name=ios12 tc_file=tc.out kernel_image=kernel.out dtb_image=dt.out ramdisk_image=ramdisk.out os-64bit vm_os=xnu run_as_daemon debug_enable cmdline="debug=0x8 kextlog=0xfff cpus=1 rd=md0 serial=2" device@s3c_uart

ios-os-boot-01 513×768

Through the above information, we can see that all the images have been loaded successfully. The entry address of the kernel is 0x470a5098. We set a breakpoint at this address

ios-boot-02 726×1174

By disassembling and comparing with xnu start.S, it is found that the first instruction is the same, and it can be matched after a few instructions in a single step, indicating that our bootloader function is ok.

16K page table support

The first problem encountered on the ARM FVP is that the kernel is hung in the mmu configuration, as long as the MMU system is turned on, and later found that the kernel configuration of TCR_EL1 is as follows

#define TCR_EL1_BASE    (TCR_IPS_40BITS | \
                                                 TCR_SH0_OUTER | TCR_ORGN0_WRITEBACK |  TCR_IRGN0_WRITEBACK | (T0SZ_BOOT << TCR_T0SZ_SHIFT) | (TCR_TG0_GRANULE_SIZE) |\
                                                 TCR_SH1_OUTER | TCR_ORGN1_WRITEBACK |  TCR_IRGN1_WRITEBACK | (TCR_TG1_GRANULE_SIZE))

The kernel uses a 16K page table. This is not supported by FVP. It is found through the register that ARM FVP does not support 16K page table by default. You need to add the following parameters to the startup parameters.

-C cluster0.has_16k_granule=1

Processing of CPACR_EL1 register

Solve the problem of page table, continue to run, and find that the serial port still has no log output, and hangs again. Through single-step and various debugging techniques (it must be said that it is really difficult to reverse a software without any symbol). Just find that the system has accessed SMID and floating-point registers are hung up, and later found that CPACR_EL1 is initialized to 0, that is, EL1 and EL0 can not access the floating-point module, this is simple, directly set the register value of CPACR_EL1 in the hypervisor 0x300000. But think about it later is definitely not so simple, finally found this article online

Tick (FPU) Tock (IRQ) https://xerub.github.io/ios/kpp/2017/04/13/tick-tock.html

It turns out that IOS has implemented a set of kernel protection mechanisms through the secure monitor using floating-point modules. The kernel-related code can be found in the source code of xnu. This means that we need to simulate the entire process in Minos, but think about it since the value configuration bit 0x300000 can be bypassed first, use it first.

DC ZVA instruction simulation

After solving the above two problems, I feel that victory is in sight, but the kernel hangs again. Through repeated debugging, I found that each hang is after executing the function 0xfffffff0070996c0.

ios-boot-04 831×730

Through repeated research on disassembly and existing kernel code, it is found that this function is actually the bzero function in the xnu kernel, and the implementation logic of this function is: it will determine whether the size of bzero is an integer multiple of 64 bytes, if If yes, call the dc zva instruction to directly operate the cache, then the size of each dc zva operation is 64 bytes, the size of the dc zva operation will be specified in the DCZID_EL0 register, and the xnu kernel does not judge the value of DCZID_EL0, use The fixed 64 bytes, in ARM FVP, the size is 8, which is 2 to the 8th power of 256 power, which results in the actual size of each DC ZVA execution is 256 bytes, which leads to the execution of this instruction The memory area that should not be cleared is also cleared.

So here we need to simulate the DC ZVA instruction, but fortunately arm v8 architecture provides this support

1. Set the 28 position in HCR_EL2 to 1
2. Add the following code in Hypervisor

static int apple_soc_dczva_trap(struct vcpu *vcpu, unsigned long va) 
{
        unsigned long pa = guest_va_to_pa(va, 0); 

		    memset((void *)pa, 0, ASOC_DCZVA_SIZE);

        return 0;
}

SHA256 instruction support

It seems that the road of debugging is difficult, but after solving the DC ZVA problem, I feel that it will be successful soon, because the code related to the processing architecture is basically run through the tracking code, and the actual situation is indeed the case. Finally, the kernel log has output

But panic, the report instruction is not supported, and it is found through disassembly that the sha256 instruction is executed. Doesn’t ARM FVP implement the sha256 related instruction?

FVP command line parameter display

minle@minle-Z840:/usr/local/DS-5_v5.27.0/bin$ ./FVP_Base_AEMv8A --list-params | grep sha256
cluster0.cpu0.crypto_sha256=0x1                       # (int   , init-time) default = '0x1'    : SHA-256 instructions supported (requires CryptoPlugin to be loaded).

What is CryptoPlugin? No information has been found online?