# NVIDIA H100 LibOS Kernel Security Analysis Report

**Target**: kernel_gh100.elf (LibOS v3.1.0, RISC-V 64-bit, 199 functions, ~197KB)
**Method**: Ghidra MCP binary analysis + recovered C source code review
**Date**: 2026-05-09

---

## Part 1: Anti-Reverse Engineering Design Analysis

### 1.1 Automated Detection Results

| Tool | Result |
|------|--------|
| find_anti_analysis_techniques | **0 findings** — no classic anti-RE patterns (no opaque predicates, anti-debug, CFF) |
| detect_malware_behaviors | 3 findings (false positives): ProcessPendingPortOps, HandleFatalException, PortDmaSend flagged as "data_exfiltration" — these are legitimate kernel IPC/DMA operations |
| detect_crypto_constants | Not implemented |
| PMP CSR byte search | No direct PMP access patterns found in kernel code |

**Conclusion**: The kernel uses **no traditional anti-RE obfuscation**. This is consistent with a production firmware kernel where performance and reliability outweigh obfuscation benefits.

### 1.2 Anti-RE Mechanism Catalog

#### A1: M-Mode Boot Validation

- **Implementation**: `if (param_1 == 0x30)` in KernelInit (0x93002444)
- **Ghidra verification**: When `boot_hart != 0x30`, code jumps to `LAB_ram_ffffffff93003512` (WFI idle loop), bypassing ALL initialization
- **Enforcement**: Software only — constant 0x30 is hardcoded
- **Effectiveness**: **Low** — trivially patchable (single branch instruction), and non-M-mode boots simply idle rather than panicking
- **Note**: This is a design guard, not an anti-RE measure per se

#### A2: Root Partition ID Check

- **Implementation**: `if (param_6 == 3)` in KernelInit, triggers `KernelPanic(0xb, 0x26f)`
- **Assertion string**: `"callingPartition == LIBOS_CONFIG_ROOT_PARTITION_ID"`
- **Enforcement**: Software — constant 3 is hardcoded
- **Effectiveness**: **Low** — constant comparison, trivially bypassed

#### A3: WPR Shadow Descriptors (Hardware-Backed)

- **Implementation**: Parses up to 9 WPR shadow descriptors from `ram:ffffffffa3009168`
- Type 0x02 descriptors: identity-mapped at offset `0xc00000000000000`
- Other types: aperture-mapped at `0x9000000000000000`+ (2MB aligned)
- **Ghidra verification**: Memory copy from `(param_6 << 0x3a) + param_7 + 0x200` into `DAT_ram_ffffffffa300b558`, then aperture lookup against table at `0x9300c4e0`
- **Enforcement**: Hardware (PMP registers set by M-mode firmware before kernel starts)
- **Effectiveness**: **Medium** — PMP is hardware-enforced, but the kernel itself cannot modify PMP (it runs in S-mode). Bypassing requires M-mode access.
- **Weakness**: The descriptor parsing has an overflow check (`uVar22 + uVar7 < uVar22` for wrap-around), but the aperture table lookup iterates up to 8 entries without validating the table bounds

#### A4: ASID-Based Access Control (0x491 Bitmask)

- **Implementation**: `0x491 = 0b10010010001` in KernelPortOperation (0x93003694)
- **Ghidra-verified code**: `(uVar5 == 5) || (uVar5 < 0xb && ((0x491UL >> (uVar5 & 0x3f) & 1) != 0))`
- **Granted ASIDs**: 0 (kernel), 2, 5 (special bypass), 8, 10
- **Blocked ASIDs**: 1, 3, 4, 6, 7, 9, 11+
- **Hardcoded syscall ASID checks**:
  - SyscallMapMemory (low): ASID == 10
  - SyscallMapMemory (high): ASID == 3 (dst), ASID == 8 (src), ASID == 0xe (as)
  - SyscallSetInterruptPriority: ASID == 7
  - SyscallPartitionConfig: ASID == 0
  - KernelServerMainLoop: ASID == 6
- **Enforcement**: Software
- **Effectiveness**: **Medium** — ASID==5 always bypasses the bitmask; hardcoded checks in syscalls may not match actual handle ASIDs, creating permission gaps

#### A5: Handle Table Grant Bits

- **Implementation**: Each handle has a `grant` field checked before operations
- Grant bit 0: read access, bit 1: write/send, bit 2: control
- **Enforcement**: Software
- **Effectiveness**: **Medium** — depends on correct grant assignment at object registration time

#### A6: Kernel/Server Task Separation (Architectural)

- **Implementation**: KernelInit creates two separate tasks:
  - Init task: HIGH priority (0x40), user address space, 128-entry handle table
  - Kernel server: NORMAL priority (0x80), kernel address space, 8192-entry handle table
- Each task has its own handle table with different grants
- **Enforcement**: Architectural (separate address spaces via Sv39 page tables)
- **Effectiveness**: **High** — kernel server runs in kernel AS with full access; init task runs in user AS with restricted access

#### A7: Partition Isolation (Hardware-Backed)

- **Implementation**: Separate address spaces per partition, Sv39 page table isolation
- IPC via port+shuttle model with DMA
- IPI messages for cross-partition communication (targeted at partition 3)
- **Enforcement**: Hardware (Sv39 MMU) + software (handle table grants)
- **Effectiveness**: **High** — hardware page table isolation prevents direct cross-partition memory access

#### A8: GDMA Permission Fields

- **Implementation**: GdmaCopyMemory (0x93009896) encodes `(bVar2 & 3) << 5 | (bVar3 & 3) << 0x15`
- **Ghidra-verified**: Only 2 bits of the 3-bit hardware permission field are used
- Permission values: 0-3 (out of possible 0-7)
- **Enforcement**: Hardware (GDMA engine validates permissions)
- **Effectiveness**: **Medium** — the unused permission bit could allow unauthorized DMA if hardware accepts permission values 4-7 that the kernel never sets

#### A9: Assertion String Leakage (Anti-RE Negative)

- **Finding**: The binary contains ~50 assertion message strings and ~15 source file paths
- **Examples**:
  - `/gpu_drv/uproc/os/libos-v3.1.0/kernel/full/init.c`
  - `"callingPartition == LIBOS_CONFIG_ROOT_PARTITION_ID"`
  - `"shuttleObject->owner != NULL"`
  - `"pools[poolId].objectSize == objectSize"`
- **Special string**: `"LNRKGOL"` at 0x9300c738 — rootfs filesystem magic number
- **Effectiveness**: **Very Low (negative)** — these strings significantly aid reverse engineering by revealing internal variable names, design intent, and source tree structure

### 1.3 Anti-RE Effectiveness Summary

| Mechanism | Static Analysis | Dynamic Analysis | Overall |
|-----------|----------------|-----------------|---------|
| A1: M-mode boot check | Bypassed | Bypassed | Low |
| A2: Root partition ID | Bypassed | Bypassed | Low |
| A3: WPR shadow descriptors | Bypassed | Partially effective | Medium |
| A4: ASID bitmask | Bypassed | Partially effective | Medium |
| A5: Handle grant bits | Bypassed | Effective | Medium |
| A6: Task separation | Bypassed | Effective | High |
| A7: Partition isolation | Bypassed | Effective | High |
| A8: GDMA permissions | Bypassed | Effective | Medium |
| A9: Assertion strings | **Helps attacker** | **Helps attacker** | Negative |

**Key insight**: The kernel's anti-RE strategy relies primarily on **hardware-enforced isolation** (PMP, Sv39 MMU, GDMA permissions) rather than software obfuscation. This is architecturally sound but assumes the hardware trust chain is intact. If an attacker gains M-mode access (e.g., via JTAG or boot ROM exploit), all software protections are bypassable.

---

## Part 2: Vulnerability Analysis

### Critical Severity

#### V1: Non-Atomic Reference Count Decrement in KernelPortClose (CWE-362/CWE-416)

- **Address**: `ram:ffffffff930055f8`
- **Ghidra-verified code**:
  ```c
  lVar4 = *(long *)(param_1 + -8) + -1;   // read refcount, subtract 1
  *(long *)(param_1 + -8) = lVar4;          // write back
  if (lVar4 == 0) {                          // if zero, free the object
      // call destructor, return to free list
  }
  ```
- **Attack**: Two concurrent calls to KernelPortClose on the same object (e.g., from a syscall handler interrupted by a timer interrupt) can both read the refcount as 1, both decrement to 0, and both attempt to free the object — **use-after-free**
- **No underflow check**: Double-close causes refcount to become 0xFFFFFFFFFFFFFFFF, which is != 0, so the destructor is never called but the refcount is corrupted
- **37 callers** of KernelPortClose identified, including all syscall handlers
- **Exploitability**: Requires precise timing (interrupt between read and write), but timer interrupts are predictable at 1.5M-cycle intervals

#### V2: Non-Atomic Reference Count Increment in Syscall Handlers (CWE-362)

- **Addresses**:
  - SyscallMapMemory (0x93000afa): `*(long *)(lVar3 + -8) = *(long *)(lVar3 + -8) + 1`
  - SyscallPartitionConfig (0x9300101e): `plVar3[-1] = plVar3[-1] + 1`
  - SyscallPartitionIpi (0x9300117c): `port[-1] += 1`
  - SyscallPartitionPortClose (0x93001212): `port[-1] += 1`
  - SyscallSetInterruptPriority (0x93000c4e): `*(long *)(handler - 8) += 1`
  - SyscallMemorySetInsert (0x93005bf6): `*(long *)(segment_a - 8) += 1`
  - PageTableMapEntry (0x93006c8c): `*(long *)(uVar7 - 8) = *(long *)(uVar7 - 8) + 1`
- **Attack**: Same as V1 — concurrent increment/decrement races lead to incorrect refcounts
- **Exploitability**: High — any of these syscalls can be interrupted by the timer, creating a TOCTOU window

### High Severity

#### V3: Hardcoded ASID Checks in SyscallMapMemory (CWE-863)

- **Address**: `ram:ffffffff93007180` and `ram:ffffffff93000afa`
- **Ghidra-verified**: `GetAddressSpaceId() != 10` (low-level variant), `!= 3/8/0xe` (high-level variant)
- **Attack**: These ASID checks are hardcoded constants rather than derived from the handle's actual ASID. If a handle is assigned an unexpected ASID (e.g., due to a bug in handle table management), the check fails open — either granting access when it shouldn't or denying access when it should grant
- **Note**: The ASID==10 check in the low-level SyscallMapMemory is particularly suspicious — why would a mapping operation require ASID 10 specifically?

#### V4: User-Controlled Data Written to Global Page (CWE-1236)

- **Address**: `ram:ffffffff9300101e` (SyscallPartitionConfig)
- **Ghidra-verified**:
  ```c
  _DAT_ram_ffffffff70000440 = param_5;  // user-controlled 64-bit value
  _DAT_ram_ffffffff70000448 = param_6;  // user-controlled 64-bit value
  ```
- **Xref analysis**: Global page offset 0x440 is:
  - **WRITTEN** by SyscallPartitionConfig
  - **READ** by KernelInit (at 0x9300339e) during boot configuration
- **Attack**: A user task with the right handle can write arbitrary 64-bit values to the global page, which persists across partition restarts. This enables **persistence attacks** — malicious configuration injected before a reboot will be read by KernelInit during the next boot
- **Impact**: Integrity violation of boot-time kernel configuration

#### V9: HandleTableRelease Non-Atomic Refcount (CWE-362/CWE-416)

- **Address**: `ram:ffffffff9300768c`
- **Ghidra-verified**:
  ```c
  lVar4 = *(long *)(param_1 + 0x10) + -1;
  *(long *)(param_1 + 0x10) = lVar4;
  if (lVar4 == 0) {
      // walk all entries, close live objects, free pages
  }
  ```
- **Attack**: Concurrent HandleTableRelease calls can race on the handle table refcount, causing premature page freeing while entries are still being walked
- **Integer overflow in size**: `*(long *)(param_1 + 8) * 0x10 + 0x1017U & 0xfffffffffffff000` — if handle count is very large, the multiplication could overflow, leading to releasing too few pages

### Medium Severity

#### V5: Kernel Pointer Leakage to Global Page (CWE-200)

- **Address**: SyscallPartitionConfig (0x9300101e)
- **Ghidra-verified**:
  ```c
  _DAT_ram_ffffffff70000478 = local_c0;                          // channel port pointer
  _DAT_ram_ffffffff70000468 = DAT_ram_ffffffffa30191f0;          // kernel server pointer
  ```
- **Attack**: Kernel heap addresses are written to the globally-accessible shared page, defeating KASLR if present
- **Impact**: Information disclosure aiding further exploitation

#### V6: GDMA Bounce Buffer Cross-Partition Data Leakage (CWE-200)

- **Address**: GdmaBounceBufferCopy (0x93003b3e)
- **Ghidra-verified**: The function copies data through the bounce buffer using two GdmaCopyMemory calls (src→bounce, bounce→dst) without clearing the buffer between uses
- **Attack**: If partition A's DMA transfer is followed by partition B's DMA transfer before the bounce buffer is overwritten, partition B can read residual data from partition A
- **Impact**: Cross-partition information leakage via shared DMA bounce buffer

#### V7: Controlled Crash Dump via SyscallTriggerCrash (CWE-200)

- **Address**: `ram:ffffffff9300184c`
- **Ghidra-verified**: `GetAddressSpaceId() != 4` and `(uStack_18 & 7) != 7` — if both checks pass, the task can force WriteCrashDump which dumps kernel state to WPR memory
- **Attack**: A user task with ASID 4 and full grant bits can trigger a controlled crash dump, leaking register state, stack traces, and kernel data structures
- **Impact**: Information disclosure of kernel internals

#### V8: Pointer Comparison Ownership Check in SyscallUnmapMemory (CWE-863)

- **Address**: `ram:ffffffff93000b88`
- **Ghidra-verified**: `*(long *)(param_1 + 0x80) == DAT_ram_ffffffffa3008008` (compares owner field to current task)
- **Attack**: If a task can predict or influence the kernel heap address of another task's mapping object, it can forge the ownership check. The kernel uses a deterministic slab allocator, making address prediction feasible
- **Impact**: Unauthorized unmapping of another task's memory

#### V11: SyscallReadMemory Limited Validation (CWE-125)

- **Address**: `ram:ffffffff93003a0a`
- **Ghidra-verified**: `param_2 - 1U < 7` allows counts 1-7 (4-28 bytes); CopyWithAddressTranslation uses the task's page table
- **Attack**: The function reads arbitrary virtual addresses from the task's address space. While the count is limited, there is no validation that `src_addr` falls within the task's legitimate memory regions
- **Stack buffer concern**: `local_40 + local_38` provides 16 bytes of stack space, but up to 28 bytes can be written — however, the `lVar1 = -(local_38 + 0xfU & 0xfffffffffffffff0)` alignment adjustment may provide adequate spacing
- **Impact**: Potential memory disclosure within the task's address space

#### V12: Weak code_offset Validation in KernelTaskCreate (CWE-787)

- **Address**: `ram:ffffffff930072be`
- **Source code**: `if (code_offset > 0x1ff000)` — only validates offset < ~2MB
- **Attack**: A large but valid code_offset could cause out-of-bounds access when computing `map_size = (code_offset != 0 ? 0x200000 : 0) + 0x200000`
- **Impact**: Potential out-of-bounds memory mapping

### Low Severity

#### V10: GdmaInterruptHandler Infinite Loop (CWE-835)

- **Address**: `ram:ffffffff93008d3e`
- **Ghidra-verified**: `do { SyscallPartitionConfig(...); ... } while(true)` — the loop at `switchD_ram_ffffffff93008d64_caseD_34` continues until `param_2 == param_4`
- **Attack**: If partition cleanup never completes (param_2 never equals param_4), the handler loops forever
- **Impact**: Denial of service — the interrupt handler never returns, blocking the hart

---

## Part 3: IPC/DMA/Hardware Attack Surface

### IPI Message Injection Surface

**IpiSend callers** (Ghidra-verified):
1. HandleFatalException — exception path
2. KernelServerMainLoop — kernel server
3. ScheduleNextTask — scheduler
4. SendIpiMessage — wrapper
5. **SyscallPartitionConfig** — user-accessible
6. **SyscallPartitionIpi** — user-accessible
7. **SyscallPartitionPortClose** — user-accessible
8. WriteCrashDump — crash dump

Three syscall handlers can inject IPI messages to partition 3 (root partition). While each validates handle grants and ASID, the **message content is user-controlled** (arg1, arg2, message type). This creates a privilege escalation surface if the root partition's IPI handler doesn't fully validate incoming messages.

### GDMA Permission Model Gap

Ghidra-verified in GdmaCopyMemory (0x93009896):
```c
*(uint *)(lVar4 + -0x9febefd8) = (bVar2 & 3) << 5 | (bVar3 & 3) << 0x15 | 0x30009;
```

The kernel uses only 2 bits (values 0-3) for source and destination permissions, but the hardware GDMA control register field is 3 bits wide (values 0-7). Permission values 4-7 are never set by the kernel but may be accepted by hardware, potentially enabling DMA operations that bypass intended restrictions.

### Global Page Shared Memory Attack Surface

| Offset | Writer | Reader | Content | Risk |
|--------|--------|--------|---------|------|
| 0x410 | Boot config | KernelInit | Bounce buffer PA | Info leak |
| 0x440 | SyscallPartitionConfig | KernelInit | User-controlled arg | **Persistence** |
| 0x448 | SyscallPartitionConfig | KernelInit | User-controlled arg | **Persistence** |
| 0x458 | Boot config | KernelInit | Boot hart ID | — |
| 0x468 | SyscallPartitionConfig | Boot config | Kernel server ptr | **KASLR bypass** |
| 0x478 | SyscallPartitionConfig | Boot config | Channel port ptr | **KASLR bypass** |
| 0x4a0 | SyscallPartitionConfig | Boot config | Fixed value 2 | — |

The global page at `0xffffffff70000000` is the most critical secondary attack surface. User-controlled writes at offsets 0x440/0x448 persist across reboots and are read by KernelInit, enabling persistent compromise.

---

## Part 4: Chained Attack Paths

### Path 1: Global Page Persistence → Boot Compromise
1. Attacker gains access to a handle with ASID==0 (via partition configuration)
2. Calls SyscallPartitionConfig to write malicious data to GLOBAL_PAGE(0x440/0x448)
3. On next boot, KernelInit reads these values as boot configuration
4. **Impact**: Persistent rootkit via global page manipulation

### Path 2: Refcount Race → UAF → Privilege Escalation
1. Attacker triggers concurrent syscalls that increment/decrement the same object's refcount
2. Timer interrupt creates TOCTOU window between read and write of refcount
3. Double-free leads to use-after-free in kernel heap
4. **Impact**: Arbitrary kernel code execution

### Path 3: KASLR Bypass → Targeted Exploitation
1. Attacker calls SyscallPartitionConfig to read kernel pointers from global page
2. Reads GLOBAL_PAGE(0x468) and GLOBAL_PAGE(0x478) for kernel heap addresses
3. Uses known addresses to craft targeted exploits
4. **Impact**: Defeating kernel address space randomization

### Path 4: Crash Dump → Full Kernel State Extraction
1. Attacker obtains a handle with ASID==4 and grant==7
2. Calls SyscallTriggerCrash to force kernel crash dump
3. Reads crash dump from WPR memory (if accessible)
4. **Impact**: Complete kernel state extraction including registers, stacks, and data structures

---

## Appendix: Vulnerability Summary Table

| # | Vulnerability | Address | CWE | Severity | Ghidra Verified |
|---|--------------|---------|-----|----------|----------------|
| V1 | KernelPortClose non-atomic refcount | 0x930055f8 | CWE-362/416 | Critical | Yes |
| V2 | Syscall refcount++ non-atomic (7 sites) | 0x93000afa+ | CWE-362 | Critical | Yes |
| V3 | Hardcoded ASID checks | 0x93007180 | CWE-863 | High | Yes |
| V4 | User data to global page | 0x9300101e | CWE-1236 | High | Yes |
| V5 | Kernel ptr leak to global page | 0x9300101e | CWE-200 | Medium | Yes |
| V6 | GDMA bounce buffer data leak | 0x93003b3e | CWE-200 | Medium | Yes |
| V7 | Controlled crash dump | 0x9300184c | CWE-200 | Medium | Yes |
| V8 | Pointer comparison ownership | 0x93000b88 | CWE-863 | Medium | Yes |
| V9 | HandleTableRelease race | 0x9300768c | CWE-362/416 | High | Yes |
| V10 | GdmaInterruptHandler loop | 0x93008d3e | CWE-835 | Low | Yes |
| V11 | SyscallReadMemory weak validation | 0x93003a0a | CWE-125 | Medium | Yes |
| V12 | Weak code_offset check | 0x930072be | CWE-787 | Medium | Source only |

## Appendix: Anti-RE Mechanism Summary Table

| # | Mechanism | Address | Type | Effectiveness |
|---|-----------|---------|------|--------------|
| A1 | M-mode boot check | 0x93002444 | Software | Low |
| A2 | Root partition ID | 0x93002444 | Software | Low |
| A3 | WPR shadow descriptors | 0x93002444 | Hardware-backed | Medium |
| A4 | ASID bitmask 0x491 | 0x93003694 | Software | Medium |
| A5 | Handle grant bits | 0x930052f0 | Software | Medium |
| A6 | Kernel/server task separation | 0x93002444 | Architectural | High |
| A7 | Partition address space isolation | 0x93005a9c | Hardware-backed | High |
| A8 | GDMA permission fields | 0x93009896 | Hardware-backed | Medium |
| A9 | Assertion string leakage | 0x9300b000+ | Anti-RE negative | Very Low |