Just-in-Time (JIT) Compilation Architecture and Internal Working Dynamics

Just-in-Time (JIT) compilers are highly complex hybrid systems in the software world, bridging the gap between the static nature of Ahead-of-Time (AOT) compilation and the flexible yet slow structure of interpreters. The primary goal of a JIT compiler is to analyze the program's behavior during runtime (profiling), directly translate the most frequently executed parts of the code (hotspots) into machine code, and apply dynamic optimizations on the fly.

In this article, we will examine how modern JIT compilers (e.g., JVM HotSpot or the V8 JavaScript Engine) work under the hood, their memory-level operations, and their runtime optimization strategies.

Modern JIT architectures do not translate code into machine language all at once. Instead, they follow a phased process called "Tiered Compilation." This architecture typically consists of multiple compilation tiers:

• Tier 0 (Interpreter): The source code (or Bytecode/Intermediate Representation - IR) is executed directly, line by line. The startup time is near zero, but the execution speed is quite low. At this stage, the JIT compiler acts as an observer; it collects branch profiling, type feedback, and loop frequencies.

• Tier 1 (Baseline Compiler): When the code crosses a certain execution threshold (e.g., a function is called 100 times), the baseline JIT steps in. Heavy optimizations are not performed at this stage; the goal is to quickly convert the code into machine code to offload the interpreter (like C1 in JVM or the older Sparkplug in V8).

• Tier 2 (Optimizing Compiler): When a function is executed tens of thousands of times, it is marked as a "HotSpot." Using the collected profiling data, aggressive optimizations (such as inlining and loop unrolling) are applied, and extremely fast machine code is generated (like C2 in JVM or TurboFan in V8).

The major difference that distinguishes JIT compilers from AOT compilers is their ability to perform speculative optimizations based on precise runtime information (runtime types, actual control flow).

a. Inline Caching (IC)

In dynamically typed languages, calling a method or property of an object requires an expensive lookup (hash map lookup) in memory. JIT remembers the types of objects that have previously arrived at a call site. If an object of the same type arrives again, it skips the lookup process and jumps directly to the offset address in memory. This converts polymorphic calls into monomorphic calls, providing a massive performance boost.

b. On-Stack Replacement (OSR)

This is one of the most fascinating engineering achievements of JIT architecture. When a program enters a long-running, "never-ending" loop (e.g., while(true)), waiting for the function to be called again is a loss for JIT. Thanks to OSR, JIT compiles the machine code of this loop in the background and, while the loop is still running, freezes the local variable states (registers and stack frames) in the processor's current call stack, replaces it with the stack frame of the new, optimized machine code, and allows the execution to continue from the optimized version.

c. Speculative Optimization and Deoptimization (Bailout)

JIT gambles on the future of the code based on limited data (e.g., "This variable has always received an Integer so far, so let me compile the rest of the code assuming an Integer"). But if a String suddenly arrives, does the system crash? No.

JIT embeds hidden "Guard" instructions inside the generated machine code. When expectations fail (Guard failure), the current register states are captured, and the execution thread is instantly thrown back from the optimized machine code to the slower but safer Interpreter. This event is called Deoptimization (or Bailout).

In an AOT-compiled program, the .text (code) section already exists on disk and is loaded into memory as read-only and executable (R-X). However, JIT generates brand new machine code at runtime. This requires dynamic memory allocation from the operating system.

• Code Cache: The machine code generated by JIT is stored in a special memory area called the "Code Cache."

• W^X (Write XOR Execute) Principle: For security reasons in modern operating systems, a memory page cannot be simultaneously writable and executable (RWX). When JIT generates code, it marks the memory as Writable (RW-), copies the code bits into this area, and then changes the page permissions to Executable (R-X) via operating system calls (e.g., mprotect() in Linux, VirtualProtect() in Windows).

The fact that JIT compilers emit machine code at runtime creates a highly critical attack surface in terms of information security.

Particularly in techniques called JIT Spraying, an attacker attempts to manipulate the JIT compiler (for example, by writing massive XOR operations with specific byte sequences) to inject malicious machine code (shellcode) into the Code Cache area. The complex state-recovery mechanisms of JIT during "Deoptimization" or race conditions in speculative execution are among the most critical areas for cybersecurity researchers, often serving as triggers for memory corruption vulnerabilities.

JIT compilers represent a perfect intersection of hardware, operating system memory management, and advanced algorithmic optimizations. Thanks to techniques like fast startup, adaptive profiling, speculative optimizations, and Deoptimization/OSR, they enable even high-level abstraction languages to run at performances very close to native languages like C/C++. The cost of this high performance is higher RAM consumption (due to profiling data and the Code Cache) and increased CPU overhead from background compiling threads.