1 Introduction↩︎

Rust is a rapidly growing systems programming language due to its focus on enhancing memory safety [1] and addressing memory safety vulnerabilities [2], [3]. For example, Rust has been employed in system software such as Mozilla Firefox, Google Chrome [4], Linux kernel modules [5], Windows kernel components [6], and various other device drivers [7].

To achieve memory safety while providing flexibility, Rust is divided into safe and unsafe code regions. Safe Rust is a strongly typed language that ensures memory safety through compile-time checks. However, its strict rules can limit the ability to implement certain features (e.g., resource sharing, low-level assembly) commonly needed in systems programming. To overcome this, unsafe Rust is employed, shifting the responsibility of memory safety checks from the compiler to the developers. In particular, unsafe Rust enables the execution of hazardous operations [8] such as dereferencing pointers or interfacing with external C libraries.

Rust Memory Safety Bugs: Source Code Analysis. Despite the memory safety mechanism in Rust, more than 360 memory safety bugs have been found in Rust programs in the last five years [9]. The primary reason is that the strict rules of safe Rust often necessitate developers to engage with unsafe Rust, which can lead to memory safety bugs when developers fail to manually verify unsafe regions [10], [11]. For example, cyclic types (e.g., doubly-linked lists) cannot be implemented without resorting to unsafe Rust. Additionally, developers might inadvertently introduce unsafe Rust code when utilizing libraries that contain unsafe regions.

To find and fix memory-safe bugs in Rust programs, a rich line of prior work (e.g.,  [12]–[21]) has focused on analyzing the unsafe regions in Rust source code. For example, Rudra [12] presented three important patterns of memory safety bugs in unsafe Rust and identified these bugs through static analysis of the unsafe regions; RPG [13] prioritized fuzzing unsafe regions in Rust libraries to more efficiently identify memory safety bugs; ERASAN [14] proposed a more efficient address sanitizer [22] for Rust programs, leveraging the characteristics of unsafe regions.

Our Focus: Rust Binary Analysis. While source-level analysis has advanced significantly in detecting memory safety issues, it is fundamentally constrained by the requirement for source code accessibility. This limitation is increasingly acute as Rust is adopted in commercial and security-critical sectors where proprietary codebases are the norm. For example, when auditing proprietary Rust-based firmware, safety-critical embedded systems, or commercial drivers such as Ferrocene [23] and Windows kernel components [6], security analysts must rely exclusively on the binary.

Furthermore, binary analysis remains essential even when source code is available, as it provides the ultimate ground truth of the executable’s behavior. It can unmask vulnerabilities introduced during the compilation pipeline, such as aggressive compiler optimizations that alter memory access patterns [24] or complex macro expansions that generate unforeseen direct unsafe operations [25]. These instruction-level risks are often elided or obscured at the source level, making their detection in stripped binaries a vital necessity for robust security assurance.

For Rust binary analysis, pinpointing unsafe regions is critical since these regions bypass the Rust compiler’s memory safety checks and are susceptible to vulnerabilities. However, as the unsafe keyword is absent from Rust binaries after compilation, identifying these unsafe regions becomes challenging for the following reasons:

\(\bullet\) Diverse unsafe Rust operations: In practice, the rustc compiler checks 12 unsafe operations [10] outlined in 1. Each operation presents a distinct root cause, and addressing each unsafe operation constitutes a subproblem of the broader challenge of recovering unsafe Rust from binaries.

\(\bullet\) Ambiguity between safe/unsafe Rust: Some unsafe operations can closely resemble safe operations, making them challenging to identify. Examples include dereferencing pointers or references, traversing unions or structs, and accessing mutable or immutable static variables. These operations exhibit minimal differences in the compiled binary; e.g., in [list:deref], there exists merely a one-instruction difference between the two functions.

\(\bullet\) Diverse architectures and compiler toolchains: Unlike source code, binary programs are closely bound to the target architectures. Rust relies on LLVM as its backend to generate machine code, which does not preserve unsafe information. Therefore, different architectures and compiler toolchains can produce distinct binary programs from the same codebase.

// a is &i32, dereferencing is safe
*a;         // => ; deref op is optimized out
// a is *i32, dereferencing is unsafe
unsafe {*a} // => mov eax,DWORD PTR [rbx]

Our Approach. Although the diverse unsafe operations and subtle differences from safe operations make it challenging to unmask unsafe Rust, our insight is: there are binary representation differences (see 4) between safe and unsafe Rust due to compiler optimizations, because safe Rust provides more constraints. These subtle differences can be learned by machine learning models during the large scale training phase [26], [27] and captured through inference. We propose Ruby, which leverages ML to address the challenges mentioned above and identify unsafe regions within Rust binaries.

As the workflow of Ruby shown in 1, we first construct CrateU dataset, which contains the unsafe labels on functions from the entire crates.io. Next, Ruby collects the RustSecU and RustSecB datasets containing real-world five-year memory safety bugs from Rustsec, with each bug’s root cause manually labeled to evaluate Ruby’s performance. We trained Ruby on CrateU and evaluated it on RustSecU and RustSecB. With the help of the classification confidence score, Ruby outputs a prioritized list as targets, which can be used by reverse engineers to focus on only 7.43% of programs, with an 88.92% chance of finding potential memory safety bugs. In comparison, a default analysis requires examining 41.38% of the binaries, demonstrating a slowdown of more than 5.5\(\times\). We also conducted an evaluation of Ruby against SOTA LLMs including GPT-5.2, Claude-4.5, and Gemini-3; Ruby achieves higher accuracy than these general-purpose models.

To illustrate the real-world application of Ruby, we incorporate Ruby into the program analysis workflow, using its results as guidance for static analysis by Angr [28] and directed fuzzing by AFLGo [29]. Utilizing Ruby, we can achieve a time reduction of 57.95% in symbolic execution by Angr and 21.26% in directed fuzzing efforts to identify bugs. By applying Ruby’s results with Android patch testing for Rust libraries, we successfully identified five bugs in Android’s Rust libraries, which have been confirmed and patched by Google.

In summary, our contributions are as follows:

We collect Rust packages from crates.io and bug information from RustSec reports [9]. All data collected is publicly available, and we only use this data for research purposes. For the five bugs we identified in Android, we reported them to Google, and all bugs have been patched.

2 Background↩︎

2.1 Rust Programming Language↩︎

Safe Rust. Rust language [31] provides a memory-safety guaranty at compile time while allowing control over low-level access to resources. The claimed memory-safety guaranty is demonstrated in [32], [33] under certain assumptions regarding language models. Specifically, safe Rust is defined as: Safe Rust will never cause undefined behavior [34].

To achieve memory safety: safe Rust leverages ownership and borrowing to ensure that there are no undefined behaviors in the compiled programs [32]. Ownership is a relationship between a value and a variable. Each value in Rust is owned by only one variable, and the memory associated with the value is automatically freed if the variable goes out of scope. This simple memory management mechanism provides compile-time memory safety without the need for a run-time garbage collector. In particular, since a value’s associated memory is freed only via drop(), double-free or use-after-free bugs are avoided through the compiler’s lifetime checks. Although ownership provides a vital safeguard to avoid memory-safety bugs, it is too strict to allow only one owner for each value. Thus, borrowing is introduced to address this issue.

Borrowing allows us to reference a variable without ownership. Specifically, Rust provides two types of borrowing: immutable and mutable. Each variable can have either multiple immutable references or only one mutable reference. Through this restriction, safe Rust ensures that no other party can write to the variable when it is borrowed as a mutable reference.

Unsafe Rust. Although safe Rust provides a strong guaranty of memory safety and relatively flexible restrictions, there are many scenarios where programmers need to maintain a shared mutable reference in system programming. For example, memory can be shared among multiple threads with well-defined synchronization.

impl<T> Vec<T> {
    ...
    pub unsafe fn set_len(&mut self, 
        new_len: usize) {
        self.len = new_len;
    }
    pub const unsafe fn get_unchecked<I>(&self,
        index: I) -> &I::Output
    {
        unsafe { &*index.get_unchecked(self) }
    }
}
fn get(self, slice: &[T]) -> Option<&T> {
    if self < slice.len() {
        unsafe{Some(slice_get_unchecked(slice,
            self))}
    }
    ...
}

To support such cases, unsafe Rust is introduced to escape from the Rust compiler’s checks inside safe regions and requires programmers to ensure memory safety inside the unsafe regions. In particular, the Rust compiler defines five operations as unsafe operations, which help programmers identify unsafe regions and ensure memory safety [8]: dereference a raw pointer, call an unsafe function or method, access or modify a mutable static variable, implement an unsafe trait, and access fields of unions. As these memory-related operations within unsafe regions are not checked by a compiler for memory safety, they are likely to cause memory safety bugs. In practice, the Rust compiler (rustc 1.67) performs explicit checks for 12 distinct unsafe operations [30], as summarized with their relative frequencies in 1. These 12 explicit operations are derived from the five canonical categories of unsafe operations, and several of them (5, 10, 11) are already deprecated. Among the remaining nine categories of active unsafe operations, we designate those that do not involve (1) and (2) the invocation of unsafe functions as direct unsafe operations, because direct unsafe operations are, by themselves, capable of inducing undefined behavior.

Because Rust’s safety guarantees are defined at the module level [35], certain functions, such as set_len, are declared unsafe yet only cause undefined behavior indirectly, via other direct unsafe operations. An illustrative example is given in [list:direct-unsafe]: the two unsafe functions shown there do not themselves contain any direct unsafe operations in their bodies but instead rely on the direct unsafe operation in the third function, get, to potentially violate memory safety. In this work, Ruby primarily targets these direct unsafe operations as exemplified in get.

The direct unsafe Rust takes 24.6% of the codebase across all crates in the Rust community [10], and even if the target project does not have any unsafe code, the usage of third-party and the standard libraries can introduce implicit unsafe regions.

2.2 Memory Safety Error↩︎

Memory safety is the state of being protected from various software bugs and security vulnerabilities when dealing with memory access, such as buffer overflows and dangling pointers. Operations that violate this protection are considered memory safety errors [36]. Among these errors, access errors, uninitialized variables, and use-after-free can directly enable malicious users to hijack the program, leading to serious consequences. Therefore, searching for memory safety errors is important for reverse engineers to ensure the program’s safety.

On the other hand, these memory safety errors (excluding the memory leaks) are considered undefined behaviors in Rust [34], which are designed to be encompassed by unsafe Rust. Ideally, because of the safe Rust definition, undefined behavior (operations) is considered to be triggered only through direct unsafe operations. Therefore, by helping to reduce the search space for reverse engineers in locating direct unsafe operations, Ruby can accelerate the memory safety error hunting process.

3 Threat Model↩︎

We assume the possession of a Rust binary compiled with the release profile [37], without debugging symbols. Furthermore, we are aware of all the function boundaries, which can be analyzed using reverse engineering tools such as IDA or Ghidra. Besides function boundaries, we do not require any other knowledge about the binary.

Ruby’s objective is to identify the code operations in the binary that are written using unsafe Rust and may directly trigger undefined behavior, referred to as direct unsafe operations. Ruby outputs such direct unsafe operations as a sub-region of the whole program operations to minimize manual effort in exploitation. Once these direct unsafe regions are identified, we can focus on those areas and employ additional intensive analysis (e.g., symbolic execution, directed fuzzing) to uncover potential vulnerabilities (6.4).

4 Binary Representations of Unsafe Rust↩︎

We first study each of the unsafe operations in detail and show the unique binary representations².

4.1 Calling Unsafe Functions↩︎

The first category of unsafe operations in Rust is calling unsafe functions. Based on the owner of the unsafe functions, there are two cases for calling unsafe functions: calling unsafe FFI (Foreign Function Interface) functions or calling unsafe Rust functions. An unsafe function indicates that there is an unsafe operation inside the function body, and by adding strict constraints, developers can convert an unsafe function into a safe function.

4.1.1 Calling unsafe Rust Function↩︎

Calling unsafe Rust functions (i.e., label 1) indicates that both the caller and callee are from Rust. It is a language-level definition of unsafe because calling an unsafe Rust function does not directly trigger undefined behaviors or memory safety errors, but it is the unsafe operations inside the unsafe functions that trigger these bugs.

To correctly infer such functions, there are two challenges:

Instead of directly identifying such cases, Ruby tries to identify the root unsafe operations inside the unsafe function and skip the unsafe calls. Ruby’s solution is based on an observation under Ruby’s threat model:

Direct Unsafe Delegation: Every unsafe Rust function must eventually ground its unsafety in a direct unsafe operation other than the call itself.

We separate this property into two individual problems:

Problem 1. The Rust unsafe calling function must have a different innermost unsafe operation.

Suppose we have an unsafe function \(F\) in Rust without root cause operation, which means that the Rustc compiler cannot detect any unsafe operations inside \(F\). Then we can safely remove the unsafe keyword from \(F\) without causing compilation errors. Thus, for any unsafe Rust functions, there must be an inner unsafe operation inside the function body, referring to its root cause.

Problem 2. The innermost unsafe operation within the unsafe function is not calling to an unsafe Rust function.

Suppose that we have a function \(F_{n+1}\) calling an unsafe function \(F_{n}\), and the caller \(F_{n+1}\) will have unsafe regions around the calling instructions. For the callee function \(F_{n}\), its unsafe label can be either because it calls other unsafe Rust functions (i.e., label 1) or it has other unsafe operations (i.e., label 2-12) in 1. For the second case, we show that the root cause is different from the callee itself. For the first case, there must exist another unsafe Rust function \(F_{n-1}\) called \(F_n\), so the root cause of \(F_n\) will be delegated to \(F_{n-1}\). Due to Problem 1’s proof, there must exist an innermost function \(F_{0}\) that could not be further delegated, and the unsafe operation \(F_0\) belongs to the second case.

Guided by this observation, Ruby intentionally skips the identification of the call instruction itself. Instead, it attempts to locate the concrete direct unsafe operations that serve as the root cause of the unsafety. By pinpointing these "root" locations, downstream tools like directed fuzzing can more effectively target the specific instructions where memory corruption is physically possible, rather than wasting resources on high-level function entries.

4.1.2 Calling unsafe FFI Function↩︎

FFI functions enable Rust to interact with existing C/C++ libraries and are inherently unsafe. These functions can originate from either statically or dynamically linked libraries. For dynamically linked functions, Ruby embeds the dynamic symbol table entries corresponding to the called functions as part of the input, aiding the model in recognizing such cases. In the case of statically linked functions, Rust functions often exhibit different stack prologues from their C/C++ counterparts. Due to ownership tracking and move semantics, Rust tends to allocate more local variables, resulting in larger stack frames compared to typical C/C++ functions. An example can be found at Godbolt. These differences can be captured by ML models to distinguish the function sources.

4.2 Using Inline Assembly↩︎

let mut x: u64 = 4;
asm!(
    "mov {tmp}, {x}",
    "shl {tmp}, 1",
    "shl {x}, 2",
    "add {x}, {tmp}",
    x = inout(reg) x,
    tmp = out(reg) _,
);           // Assembly preserved in binary
let mut y: u64 = 4;
let tmp = y << 1;
y = y << 2;
y = y + tmp; // Compiler optimizes computation

Rust, similar to C/C++, enables developers to integrate inline assembly for precise control over low-level behavior. These inline instructions are treated as unsafe since they bypass the Rust compiler’s type and memory safety checks. As shown in [list:use-asm], developer-written assembly is preserved verbatim in the final binary, whereas semantically equivalent Rust code is typically optimized away or transformed. Moreover, manually crafted assembly usually varies from compiler-generated code, and these discrepancies can be utilized to detect potential unsafe operations during binary analysis. However, we note that developers may write arbitrary inline assembly due to its flexibility, creating challenges for Ruby to identify.

4.3 Initializing Type with Constraints↩︎

#[rustc_layout_scalar_valid_range_start(1)]
#[rustc_layout_scalar_valid_range_end(5)]
struct NonZeroI64(i64);
size_of::<Option<NonZeroI64>> // 8
#[repr(transparent)]
struct PlainI64(i64);
size_of::<Option<PlainI64>> // 16

To further support compiler optimization, Rust provides rustc_layout_scalar_valid_range attributes that enable users to specify the valid range of a given variable. Any value outside of the valid range can lead to invalid values, which is considered unsafe in Rust. The compiler can then assume this valid range and perform niche optimizations accordingly. As shown in [list:init-type], niche optimization [39] in Rust attempts to compress the actual memory size of enums with values by reusing invalid values of the wrapped type. In the example, NonZeroI64 is defined with a valid range from 1 to 5, making values outside this range—including 0—invalid. This enables the compiler to use 0 as a niche value to represent None, reducing the size of Option<NonZeroI64> to 8 bytes, the same as i64. In contrast, PlainI64 does not provide such a range guarantee, thus Option<PlainI64> cannot reuse any value as a niche and requires 16 bytes to store both the value and the discriminant. By inspecting the memory access patterns, Ruby can analyze the memory size of the types and attempt to identify this type of unsafe operation.

4.4 Mutating Static Variables↩︎

Mutating global variables can lead to data races, which are considered unsafe in Rust. To mitigate this, Rustc explicitly checks two types of unsafe operations: (1) mutations to static variables defined in Rust code, and (2) accesses to static variables originating from FFI libraries (e.g., in C/C++).

To support detection in binary analysis, Ruby embeds information from static variable sections (i.e., .data and .bss in ELF binaries) into the corresponding functions, enabling the model to identify and reason about global variable usage.

For FFI static variables, Rustc cannot analyze their mutability, as their definitions reside outside of its analysis scope. Consequently, it conservatively assumes all FFI static variables are mutable and treats any access to them as unsafe. As discussed in 4.1, Ruby can leverage characteristic differences between Rust and C/C++ binaries to infer the origins of these static variables and assess their safety.

4.5 Dereferencing Raw Pointers↩︎

<reference>:
    ... ; omitted 13 instructions for simplicity
    ; deref reference is optimized and removed
    8eb4:   add    rsp,0x40
    8eb8:   pop    rbx
    8eb9:   ret 

<pointer>:
    ... ; omitted 13 instructions for simplicity
    8f33:   mov    eax,DWORD PTR [rbx]; deref ptr
    8f35:   add    rsp,0x40
    8f39:   pop    rbx
    8f3a:   ret

Dereferencing raw pointers, a common operation in C/C++, can lead to severe memory safety issues, such as use-after-free and out-of-bounds access. To uphold memory safety, Rust enforces the use of references instead of raw pointers through its borrow checker, as illustrated in [list:deref].

However, from the perspective of binary instructions, both references and raw pointers are represented as memory addresses, making them indistinguishable at the instruction level. This poses a key challenge in binary analysis: correctly identifying whether a given memory access corresponds to a safe reference or an unsafe raw pointer.

To address this, we observe that Rust references act as ‘checked pointers’ with guaranteed valid access. This property enables the compiler to optimize references more aggressively, frequently promoting them to registers and eliminating redundant memory loads. In contrast, raw pointer accesses lack these guarantees and are generally preserved explicitly in the binary. As demonstrated in [list:deref-x64], the compiler optimizes reference-based access into register operations, whereas raw pointer dereferencing produces additional memory access instructions.

4.6 Accessing Union Fields↩︎

union MyUnion {
    f1: u32,
    f2: i64,
}
let mut x = MyUnion { f1: 1 };
x.f1 = x.f1 + 1; // mov DWORD PTR [rsp],0x2
x.f2 = x.f2 + 1; // inc QWORD PTR [rsp]

Unions are special types that allow multiple fields of different types to share the same memory location. While union types are powerful and flexible, improper initialization or access can lead to undefined behavior, such as reading uninitialized memory or interpreting bits with an incorrect type.

At the binary level, union accesses can be distinguished by the operand size of the generated instructions, which reflects the accessed field’s type. As shown in [list:union-access], accessing the 32-bit field results in a DWORD instruction, while accessing the 64-bit field results in a QWORD instruction. Ruby leverages these differences in operand size to infer possible field types in union-based memory locations and to detect potentially unsafe or type-violating operations.

4.7 Calling Functions with Hardware Features↩︎

#[target_feature(enable = "aes")]
...
for &key in &keys[1..KEYS - 1] {
    b = _mm_aesenc_si128(b, key);
} // aesenclast xmm0, XMMWORD PTR [rip+0x3d533]

Rust provides the std::arch module to enable direct use of specialized hardware instructions, such as SIMD and AES-NI, for performance-critical operations. However, these instructions depend on specific CPU features and may not be supported across all hardware platforms. To manage compatibility, Rust uses the target_feature attribute at the function level to indicate the required hardware capabilities. Executing such instructions on unsupported hardware can result in undefined behavior. In [list:hardware], the AES-NI instruction _mm_aesenc_si128 is utilized to enhance the performance of AES encryption. This leads to the inclusion of the aesenclast instruction within the binary, which is tailored for the x86_64 architecture. These hardware-dependent instructions are preserved in the binary and can be identified by Ruby.

4.8 Phantom Unsafe Operations↩︎

Certain unsafe operations in Rust, such as CastPointerToInteger, MutLayoutConstrainedField, and BorrowLayoutConstrainedField, are considered phantom—they exist in the Rust compiler’s internal semantics (e.g., in the rustc implementation) but are virtually absent in real-world Rust code. Our dataset, which includes over 150K crates from crates.io, contains no observed instances of these operations. Due to their rarity and limited practical use in Rust development, Ruby is not trained or designed to detect these phantom unsafe operations in binary programs and excludes them from its analysis scope.

5 Proposed Approach↩︎

Building on the insights of binary representations in 4, we propose Ruby, a machine learning-based tool for detecting unsafe operations. In this section, we present the design choices and implementation details of Ruby.

5.1 Dataset Collection↩︎

2 summarizes the statistics of the datasets in the number of functions in binaries along with their labels.

The Crate dataset, denoted by CrateU, is a set of pairs consisting of a function in binary and the corresponding unsafe labels, i.e., \(\{(x_1, u_1), \dots,\) \((x_m, u_m)\}\), where \(x_i\) is a function in binary, \(u_i\) is a set of safe or unsafe labels (i.e., \(u_i=\{ 0 \}\) for “safe” and \(u_i \subseteq \{ 1, ..., 12\}\) for “unsafe” root causes in 1), \(m\) is the total number of function and label pairs. The dataset is generated from all the Rust crates from crates.io, encompassing a total of 107,460 crates.

To further demonstrate the connection between unsafe Rust and memory safety errors, we created a novel dataset that contains real cases of memory safety bugs from the RustSec Advisory Database [9]. The RustSec dataset is a set of tuples of a function in assembly code, unsafe labels, and a bug label, i.e.,  \(\{(x_1, u_1, y_1), \dots, (x_n, u_n, y_n)\}\), where \(x_i\) is a function, \(u_i\) is a set of unsafe labels as in CrateU, \(y_i\) is a bug label, and \(n\) is the total number of labeled functions. The combination of labeled functions only with unsafe labels is denoted as the RustSecU dataset, and only with bug labels is denoted as the RustSecB dataset. During the construction of these datasets, we exclude the relevant crates from CrateU to ensure that the model will not be trained with them.

To evaluate Ruby on real-world vulnerabilities, we collected 360 advisories from the RustSec Advisory Database [9] spanning five years. Two people independently audited these reports to identify reproducible memory-safety vulnerabilities, filtering the set to 121 unique root-cause bugs by excluding logic-only issues. We downloaded the 257 corresponding crates and compiled them to generate our evaluation dataset, RustSecB. To prevent data leakage, these 257 crates were strictly excluded from the 11M-function CrateU training set.

While the dataset contains 121 unique vulnerabilities, they manifest as 301 buggy function entries in the compiled stripped binaries. This expansion is a technical consequence of the Rust compilation model: a single generic vulnerability is often specialized into multiple concrete instances via monomorphization or propagated across function boundaries through inlining. In total, the RustSecB evaluation set comprises 1.8M functions, of which only 301 are buggy (\(<0.02\%\)). This extreme class imbalance underscores the "needle-in-a-haystack" challenge of binary-level bug hunting and the necessity of Ruby’s search-space reduction.

Generation. To generate the CrateU dataset for training purposes, Ruby utilizes two components: a custom Rust toolchain to record unsafe locations and a binary analyzer to map instructions back to the source. The process begins by modifying the Rustc to log the locations of all unsafe operations during the compilation process. Subsequently, the modified toolchains are applied to recompile the input crates and extract the unsafe location information from both the compiler and binary programs as compilation output. Additionally, the configuration files are modified to include debugging output for all compilation targets. After obtaining the binary programs and compilation logs, binary analyzer can utilize the DWARF debugging information present in the binary programs to map the instructions back to the source code. By comparing the source code locations of unsafe regions, the binary analyzer outputs the unsafe region addresses and labels as part of the training dataset. Through this automated approach, Ruby builds the CrateU dataset for training and RustSecU evaluation.

5.2 Preprocessing↩︎

As discussed in 4, detecting certain unsafe operations related to global variables and FFI functions necessitates a comprehensive understanding of the binary program’s metadata and memory layout. Thus, Ruby incorporates the binary’s metadata information: (1) the static analysis of global variables is represented as a special token <GLB> when instructions access the global variables in the .data and .bss, and (2) external function calls are signified as <EXT> when the calling instruction attempts to invoke an external function.

Ruby embeds these two metadata as special tokens in the same line as the assembly instruction and utilizes machine learning models to capture subtle differences in assembly instructions, inferring correct unsafe operations.

Before the training step, Ruby first trains a customized tokenizer based on the input architecture and assembly language. Since assembly is a specialized programming language, a customized tokenizer can effectively split the assembly instructions into meaningful tokens without compromising their integrity. Ruby by default applies its analysis at the function granularity; however, for functions longer than the token limitation of the model, Ruby will segment the instructions into pieces that conform to the token limitation and perform analysis on each piece independently. After analyzing all the pieces, Ruby will consolidate the results to produce the final output.

Due to the large number of records in CrateU dataset and limited computing resources, Ruby we cannot perform training and evaluation on the entire dataset. Consequently, we sampled 10 million records from the entire CrateU dataset (approximately 806 million in total) for training purposes. During the sampling process, Ruby we prioritized the minor labels by attempting to include all cases while maintaining a 1:1 ratio of safe to unsafe records (the biased distribution is preserved in the validation and test datasets). Specifically, based on the training dataset, Ruby we calculate the weights of each label to support a weighted loss function and mitigate bias during the training process.

5.3 Model Training↩︎

We define our problem as a multi-class, multi-label classification task, where the model input is a sequence of binary instructions in assembly language, and the expected output consists of labels for the input sequences that indicate their unsafe status and reasons. We note from Rust’s unsafe definition that the labels can overlap (e.g. accessing a global mutable union structure); hence, the expected output can be either safe label or unsafe label, with at least one reason representing the root cause of the unsafe regions.

The goal of unsafe classification is to design a classifier that predicts whether a given function embeds unsafe blocks. In particular, let \(x \in \mathcal{{X}}\) be a sequence of instructions represented in assembly code,

let \(\mathcal{{U}}\mathrel{\vcenter{:}}= \{2,3,4,6,7,8,9,12\}\) be a set of unsafe labels, where the corresponding unsafe notations are defined in 1,

\(u \in 2^{\mathcal{{U}}\cup \{0\}}\) be a subset of safe or unsafe labels, where a safe label is denoted by \(0\), \(\hat{{s}}: \mathcal{{X}}\times \mathcal{{U}}\cup \{0\} \rightarrow \mathbb{R}_{\geq 0}\) be an unsafe scoring function, and \(\hat{{u}}: \mathcal{{X}}\rightarrow \{0, 1\}\) be the binary unsafe classifier.

Lastly, labeled functions from a distribution are split into train, validation, and test sets i.e.,  \(S \mathrel{\vcenter{:}}= (S_\text{train}, S_\text{val}, S_\text{test})\).

We consider the following parameterized function of the binary unsafe classifier based on \(\hat{{s}}\): \[\begin{align} \hat{{u}}(x) \mathrel{\vcenter{:}}= \begin{cases} 1 &\text{if~} 1 - \hat{{s}}(x, 0) \ge \hat{\tau} \\ 0 &\text{otherwise} \end{cases}. \end{align}\] Here, \(\hat{{s}}(x, 0)\) is the safe score for a given function \(x\) and \(\hat{\tau} \in \mathbb{R}_{\ge 0}\) is a threshold for unsafe classifier; thus, if the risk \(1 - \hat{{s}}(x, 0)\) is greater than the threshold \(\hat{\tau}\), we consider a function \(x\) to be unsafe.

We use RoBERTa-large [40] as the backbone of the unsafe classifier, which is the long-standing stable masked language model based on transformers [41] and its input is assembly code. On top of this backbone model, we attach a classification header for the unsafe classifier. The entire unsafe classifier is trained by minimizing the loss of cross entropy in the training set \(S_\text{train}\) for each unsafe and safe label. In addition, we weighted each label’s loss by considering the distribution in the training dataset to eliminate the effects of the unfair label distribution.

5.4 Trustworthy Thresholding↩︎

After the training process, we get the unsafe classifier, noted at \(\hat{{u}}\). Now we describe how to pick the threshold \(\hat{\tau}\) with a guarantee of correctness on the recall of \(\hat{{u}}\). In particular, choosing a threshold for a classifier is a classic problem [42], where heuristic methods are mostly considered. We consider a rigorous thresholding approach that comes with a PAC guarantee based on conformal prediction [43], [44].

We adopt the PAC conformal set algorithm [45], [46] for thresholding. Let \(\bar{\theta}\) be the upper Clopper-Pearson (CP) bound [47], where the binomial parameter \(\mu\) is included with high probability, i.e.,  \(\bar{\theta}(k; m, \delta) \mathrel{\vcenter{:}}= \inf\{ \theta \in [0, 1] \mid F(k; m, \theta) \le \delta\} \cup \{1\},\) where \(\mathbb{P}_{k \sim \text{Binomial}(m, \mu)}\left[ \mu \le \bar{\theta}(k; m, \delta)\right] \ge 1 - \delta\). Here, \(F(k; m, \theta)\) is the cumulative distribution function of the binomial distribution with trials \(m\) and the probability of success \(\theta\). The threshold \(\hat{\tau}\) is obtained by solving the following optimization: \[\begin{align} \hat{\tau} = \arg\max_{\tau \in \mathbb{R}_{\ge 0}}~ \tau \qquad \text{subj. to} \qquad \bar{\theta}(k; |S_\text{cal}|, \delta) \le \varepsilon, \label{eq:alg} \end{align}\tag{1}\] where \(S_\text{cal}\) is the set of unsafe functions in \(S_\text{val}\), i.e.,  \(S_\text{cal}\mathrel{\vcenter{:}}= \{ (x, u) \in S_\text{val}\mid u \neq \{0\} \}\), and \(k\) is the number of unsafe functions that are missed by a threshold, i.e.,  \(k \mathrel{\vcenter{:}}= \sum_{(x, u) \in S_\text{cal}} \mathbb{1}\left( 1 - \hat{{s}}(x, 0) < \tau \right)\). Intuitively, the interval \([\hat{\tau}, \infty)\) contains the most unsafe scores \(1 - \hat{{s}}(x, 0)\) for \(x \in S_\text{cal}\). If an downstream analyzer wants to have \(90\%\) recall on unsafe functions, \(\varepsilon\) is set by \(0.1\); if the analyzer wants this desired recall level to be strictly satisfied, \(\delta\) needs to be small, where we use \(\delta = 10^{-3}\).

6 Evaluation↩︎

To demonstrate the effectiveness of Ruby, we conducted a comprehensive evaluation motivated by the following questions:

Hardware Settings. We launch our experiment on a machine with 128-core AMD EPYC 7452 processors and 8 NVIDIA RTX A6000 GPUs running under the Ubuntu 22.04 operating system. For ARM evaluation, we use an ARM Neoverse-N1.

6.1 RQ1: Unsafe Classifier Evaluation↩︎

Setup. We utilize the CrateU dataset for both training and evaluation. The dataset is partitioned into 60% for training, 20% for validation during training, and 20% for testing. We also conduct evaluation on RustSecU, which are excluded from CrateU to ensure that Ruby never learned from those data.

6.1.1 Evaluation on CrateU↩︎

We first evaluate Ruby over CrateU for the unsafe classification task. 2 presents the precision-recall curve in RustSecU for each type of unsafe.

Unsafe Classifier. The area under the precision recall curve (AUPRC) of an unsafe classifier is 0.98 for the overall safe/unsafe classification, demonstrating that unsafe blocks in the Rust binary are identifiable. Besides, Ruby applied weights to eliminate the biased distribution of unsafe labels, achieving higher scores in all unsafe Rust classification tasks.

Trustworthy Thresholding. The precision-recall curve shows the trend of precision and recall with a varying threshold \(\hat{\tau}\); however, this threshold should be chosen in practice. We use the trusted thresholding algorithm proposed in (1 ) for \(\hat{\tau}\), and the chosen threshold provides \(91.75\%\) recall of unsafe functions, which is larger than the desired recall of \(90\%\), as expected. This suggests that reliable thresholding provides the desired guaranty of recall, controlled by \(\varepsilon\). In practice, we desire to have a list of functions containing the desired rate of unsafe functions, so we empirically demonstrate that the proposed algorithm achieves this goal.

Comparison with SOTA LLMs. We compare Ruby model with thresholding to the SOTA LLMs, including chat models: GPT-5.2 [48], Claude-Sonnet-4.5 [49], Gemini-3-flash, and reasoning models: Claude-Opus-4.6 [50]. We conduct few-shot (K) and best-of-N evaluations to validate the abilities of the SOTA models with gradually increased context. The few-shot evaluation involves gradually adding K examples for each candidate category in the prompt and evaluating the model’s inference based on existing examples. For the best-of-N evaluation, we ask the model N times for the same question and check if there is a chance that the model outputs the correct answer. The evaluation results are shown in 3.

Overall, Ruby successfully lead the accuracy by around 20% as an dedicated model for unsafe Rust classification. Although few-shot and best of N evaluations help SOTA LLMs to better understand the problem, they still suffer from lack of context and requires further improvement.

Comparison with Fine-tuned Models. To further demonstrate Ruby’s benefit in the specific domain of the Rust binary unsafe classification task, we conduct fine-tuning using OpenAI’s GPT-4³. For each label, we sampled 20 cases for each unsafe label and combined them into our fine-tuning dataset, reusing the same benchmark that we tested with the vanilla models. The result is shown in the GPT-4.1-FT row in the 3. By comparing the result, we believe it’s more efficient for Ruby to adopt a dedicated model instead of general LLM models.

6.1.2 Evaluation on RustSecU↩︎

To further demonstrate the classification performance of Ruby, we evaluated Ruby in the RustSecU dataset. The crates in RustSecU are excluded from CrateU prior to training, so Ruby has never seen them before. The evaluation result is shown in 3. In general, Ruby achieves the 0.98 AUPRC score with precision 95.21% and recall 95.09%, showing its ability to recover unsafe Rust operations from unknown assembly.

6.2 RQ2: Assistance in Reverse Engineering↩︎

The ultimate goal of Ruby is to accelerate the bug hunting process for reverse engineers; therefore, to further demonstrate the benefit of Ruby in RQ2, we evaluate Ruby in the RustSecB dataset to show its ability to accelerate the bug hunting process. Then we measure the analysis speed to show its practical applications.

6.2.1 Assisting Reverse Engineering↩︎

The ultimate objective of Ruby is to help reverse engineers narrow the potential search space. Therefore, to show Ruby’s efficiency, we compare Ruby’s guidance with following methods on the RustSecB dataset:

We set our target recall to be at least 80% and the result shows Ruby can minimize the searching space to only \(7.43\%\) and guarantees \(91.75\%\) of unsafe operations inside, details are in 4. Among the four methods, Ruby performs close to the optimal case: with only 1.25x coverage overhead. Compared to randomly searching or prioritizing third-party crates, Ruby achieves 4-6x benefits.

6.2.2 Performance Overhead↩︎

Ruby’s performance is affected by the bootstrapping process like decompiling stage and model loading stage. We sampled 200 binaries based on their size and applies Ruby on them with single CPU core and single GPU card. It takes Ruby around 10 hours to finish the analyses. Considering CrateU has around 100K binary programs, it only take Ruby for around one week to analyze all the binary programs in crate.io with 32 single CPU and GPU processes.

6.2.3 Real World Applications↩︎

We evaluated Ruby in real-world applications to demonstrate its capability in analyzing large binary programs that exhibit complex logic. In particular, we choose three types of binaries: servo as a web browser, artichoke, rustpython as high-level language interpreters, Ruby and Python accordingly, deno as a JavaScript and TypeScript runtime⁴, where these binaries are not in our dataset, while related packages might be included (e.g., smallvec for servo). We additionally count the lines of Rust code in the target repository to evaluate our model’s performance. Since Rust utilizes cargo to manage its dependencies, we employ cargo vendor to download all dependencies and subsequently count the code lines, which encompass both the project’s source code and all its dependencies.

4 shows the evaluation results for each application and Ruby’s AUPRC score. Overall, Ruby performs well in AUPRC, achieving over 0.907 for deno and above 0.83 for the other applications. For language interpreters like artichoke and rustpython, Ruby’s performance is influenced by the various system calls and low-level APIs supported by the target language. We note that all of these applications contain more than 1M lines of Rust code, and Ruby can complete the analysis of these projects in three hours.

6.3 RQ3: Robustness Evaluation↩︎

By the definition of unsafe Rust in 2.2, it is a language-level definition that is expected to be independent of different compiler toolchains, architectures, etc. To show the robustness of Ruby, we evaluate Ruby across different compiler toolchains (Rustc 1.57.0, 1.67.0, and 1.75.0) and architectures(x64 and ARM).

6.3.1 Different Compiler Toolchains↩︎

We first demonstrate how fine-tuning aids Ruby in managing various toolchains of Rust and in recovering the unsafe regions.

Setup. The CrateU dataset is built on customized Rustc 1.67.0 [54], which uses LLVM-15 as the backend. To further demonstrate Ruby’s robustness, we collected the RustSecU dataset using Rustc 1.57.0 [55] with the LLVM-13 backend and Rustc 1.75.0 [56] with the LLVM-17 backend, fine-tuning and evaluating Ruby’s performance on both versions and comparing it with the 1.67.0 version. Since Rustc converts MIR into LLVM IR and leverages LLVM to optimize and generate instructions, different LLVM backend versions can produce varying outputs. We sampled only 20% of the data from both Rustc 1.57.0 and 1.75.0, fine-tuned for two epochs in less than an hour.

Result. The comparison of the unsafe classification task is presented in 5. The results indicate that prior to fine-tuning, the model’s score diminishes by approximately 0.10 due to the compiler toolchain changes. With the help of the fine-tuning process, Ruby is able to capture minor changes across different compiler versions, achieving high scores among all compiler versions: 0.91 for 1.57.0 and 0.92 for 1.75.0.

6.3.2 Different Architectures↩︎

Besides the toolchains, we further explore Ruby’s performance under different architectures.

Setup. According to the definition of unsafe Rust, unsafe information is lost in the early stages of compilation when Rustc ports its MIR to the LLVM IR, indicating the architecture independence of unsafe betrayal. To validate this important property, we conduct a similar machine learning pipeline: dataset collection, preprocessing, model training, and evaluation on ARM architectures.

Result. The results are shown in 6. Ruby demonstrate high performance on both ARM and x64 architectures, indicating the architectural independence of unsafe Rust. Some improvements observed in certain labels can be attributed to ARM’s fixed-length instruction set, which offers a simpler assembly language relative to the variable-length instruction set of x64.

6.3.3 Ablation Study↩︎

Setup. Ruby embeds the static analysis result in the input of the model to help the model identify the inner connections. To demonstrate the effectiveness of this approach, we picked the UseOfMutableStatic label and retrained it without embedding the static analysis using the same dataset CrateU.

Result. The comparison is shown in 7, the static analysis helps Ruby improve the AUPRC by 0.61, showing the benefit of embedding the static analysis result into the model input.

6.4 RQ4: Bug Hunting Applications↩︎

We applied Ruby on the Angr, directed fuzzing and Android system fuzzing to assess its end-to-end effectiveness.

6.4.1 Angr Analysis↩︎

Angr [28] is a powerful binary analysis tool that finds bugs through symbolic execution. However, performing symbolic execution is time and resource-consuming, as it involves the exploration of different paths in the program. Therefore, Ruby can be integrated as a guiding framework, enabling the prioritization of suspect functions.

For instance, in the predefined binary of RUSTSEC-2021-0094, there are 1,434 functions in total, and under the default ordering the actual buggy function is ranked 783. Ruby reorders functions according to their estimated unsafe probabilities, elevating the buggy function to rank 140 and accelerating the bug localization process.

We collected the benchmark from the RustSec dataset, manually filtered out bugs that were not shown in the binary programs, and successfully built 14 vulnerable binaries. We set up our Angr analyzes based on QueryX [57]’s memory safety analysis script, identifying heap/stack overflows, use-after-free, and out of bounds access errors, and added detection of uninitialized memory access. We established targets for each function in the binaries with a total limitation of 12 hours per binary and 5 minutes per function. We compared our approach with the baseline, which provided no guidance, and the unsafe oracle from the source code; the results are shown in 5.

Ruby is close to the unsafe oracle result, with only 1.48x overhead to reach the oracle case with additional source code information. Compared to the baseline approach, Ruby saves 57.95% of time to find the same bugs, largely saving time and resources during the vulnerability hunting process.We also performed the same evaluation on the ARM binaries, and Ruby saves 61.4% on ARM architectures.

6.4.2 Fuzzing Analysis↩︎

Directed fuzzers are emerging dynamic analysis tools that leverage control flow graphs and static analysis to compute the instruction’s distance to the target function and prioritize the corpus that reaches closer locations.

We apply the result of Ruby to a directed fuzzer as its target function to show the direction of Ruby for dynamic analysis. Specifically, we use AFLGO [29] as our directed fuzzer, and we use the trophy cases found by cargo fuzz as our benchmark. Among the 7 reproducible bugs with their harnesses, we first launch the undirected AFL, then apply Ruby to these binaries and launch the AFLGO with Ruby’s output as the target. We count the wall time of the fuzzers that encounter the first crash as a result and repeat the fuzzing process three times to obtain the mean value.

The result is shown in 6; on average, with the help of Ruby, AFLGO can save 21.26% of the time to find the same bug without guidance.

6.4.3 Android Rust Library Fuzzing↩︎

To evaluate Ruby’s end-to-end effectiveness, we applied Ruby to the Android system and provided guidance for fuzzing Rust libraries.

As of Android 16.0.0, there are 93 Rust crates serving as fundamental OS libraries, containing 1,534 function APIs. Testing all of the functions requires a lot of engineering effort. We consider the following steps: given a crate library in .dylib or .so, (1) we feed all the functions into Ruby and get a prioritized target list; manually develop the fuzzing harnesses for each function, and (2) conduct black box fuzzing with AFL++ in Frida mode [58] for the top 50 targets. For each target, we run for a maximum of 24 hours.

We identified and confirmed five different bugs: two stemming from character boundary issues, one concerning an out-of-bounds access, one related to an unexpected unwrap in the library, and one resulting in a panic abort⁵. We reported all the PoC code with corresponding inputs to Google, and all the bugs were confirmed and patched by the maintainers. Compared with the default approach of fuzzing each function individually, Ruby saves 74.6% of time to find these bugs.

7 Limitation↩︎

Ruby’s limitation can be categorized into two parts: the limitation inherited from the toolsets and methodologies used by Ruby, and the limitation related to our implementation.

Ruby relies on Rustc to generate the corresponding dataset for training purposes. For unsafe operations that Rustc cannot detect, Ruby cannot identify them as well. Second, Ruby leverages machine learning to recover unsafe Rust, the loss during the training process cannot be recovered by Ruby. Ruby experiences instances of false positives and false negatives compared to the oracle method.

Ruby’s implementation limitations are mainly from the data collection and model training steps. Currently Ruby leverages the specific Rustc to automatically generate the CrateU dataset. Because of the compiler differences and architecture requirements, CrateU dataset may miss several crates and the unsafe functions are missed by Ruby as well. Furthermore, constrained by hardware capacity, Ruby utilizes only 10M out of a 903M dataset for training. Finally, as a prototype, Ruby uses RoBERTa [40], which is a classic BERT model specialized for classification tasks. Finally, due to the context window limitation, Ruby may be inaccurate because of different context splits.

8 Related work↩︎

Unsafe Rust and its usage. Although software engineers try to avoid unsafe regions in their program to avoid potential memory safety problems [59], [60], many Rust crates are using "unsafe" more frequently and lead to implicit usage and wide spread of unsafe blocks in Rust binaries [11]. Also, studies [10], [61] show that these unsafe usages of are often for good or unavoidable reasons, which are not easy to remove. While unsafe catches the attention of programmers on memory safety, it can be also used by reverse engineers to find the weakness of the given Rust binaries. To our knowledge, Ruby is the first tool to recover the unsafe regions from raw binary instructions.

Binary bug hunting. Approaches to find bugs or vulnerabilities have been proposed through binary analyses [28], [62]–[68]. Angr [28] is a powerful framework that combines static and dynamic analyses to automatically find general vulnerabilities in binary executable. In contrast, other tools are designed to find specific bugs in binaries; oo7 [66] is designed for spectre attacks, KEPLER [67] is targeted for control-flow hijacking, and DTaint [63] aims to detect taint-style vulnerabilities. Furthermore, machine learning has been applied to program analyses for bug finding [69]–[71]. VulDeePecker is a system that leverages deep learning to automatically detect bugs inside programs. However, [72] also proposed several open questions that may interfere with the performance of applying machine learning to the search for bugs.

General binary analysis. Beyond bug hunting, general binary analysis is an essential task in computer security. Some of this research (e.g., function boundary detection) can be regarded as a basis of Ruby. While Nova [73] utilizes hierarchical attention and contrastive learning for general semantic representation, Ruby is a specialized classifier designed for the security-critical task of identifying direct unsafe operations. In binary-binary code matching, a binary function or an entire program is represented in a vector to retrieve functions or programs similar to a binary target given [74]–[76]. The core of known approaches is learning binary code representation to summarize the instructions of a function or a program into a vector based on recurrent neural networks [77], graph neural networks [78], or transformer-based models [41]. Similarly, binary-source code matching finds similar binary or source code given source or binary code [79]. Function prototype inference is predicting the type of function (e.g., the number of arguments and the type of argument) given instructions of a function [80]. Function boundary detection enumerates the list of the start and end of a function in a binary [27], [81]–[83], and malware classification classifies each binary regardless of whether it embeds malware [84].

9 Conclusion↩︎

In this paper, we present Ruby, the first tool to leverage machine learning and static analysis to identify unsafe regions in Rust binaries. Our evaluation shows that Ruby can identify unsafe regions from Rust binaries with precision 93.84% and recall 91.75%. Ruby can accelerate the vulnerability finding process in symbolic execution (static) and directed fuzzing (dynamic) by 57.95% and 21.26% respectively. By applying Ruby to blackbox fuzzing in Android, we successfully identified 5 unknown bugs in the Rust libraries and all the bugs were confirmed by the developers.

10 Acknowledgment↩︎

We thank our shepherd Hui Xu and the anonymous reviewers for their valuable feedback and suggestions. We also thank Mingyu Guan and Zhuoran Yu for their assistance with model training. This research was supported, in part, by the ONR under grant N00014-23-1-2095, IITP grant (No. RS-2024-00509258 and No. RS-2024-00469482), NRF grant (RS-2025-00560062), and gifts from Facebook, Mozilla, Intel, VMware and Google.

References↩︎