Report for Rustc Testsuite Adapter

GCCRS Logo

Rustc Testsuite Adapter for GCC Rust (GCCRS)

Description:

This project is the follow-up of the previous Google Summer of Code project Improving user errors & Error Code Support for GCC Rust Frontend. We have to adapt a copy of the rustc testsuite to make use of the error code framework implemented in gccrs as part of GSoC 2023. We need to develop a test case runner similar to rustc’s one, in order to match error codes and line numbers to the output of gccrs. Specifically, we need to ensure that gccrs is emitting the correct error code consistent with rustc-1.49. This project requires investigating the current test framework of gcc/gccrs (dejagnu) and also the official rustc one. The main exact goal of this project is to have access to a tool which enables us to run gccrs on the rustc test cases and assert that we emit the correct error codes/messages w.r.t to their line numbers. Furthermore, the extended deliverable of this project is to integrate this tool in gccrs CI/CD pipeline.

What was Done:

  • Researched the Rust testing framework “compiletest” to understand how test cases are organized and Rust processes and executes test cases.
  • Learned “Rust” programming language and created a new tool, named rusttest-to-dg to convert the Rust test cases to DejaGnu format.

For gccrs, we only want to check the ui tests — these tests are a collection of general-purpose tests which primarily focus on validating the console output of the compiler.

The general structure of rustc test cases consists of a Rust source file located anywhere in the “tests/ui” directory. For example, tests/ui/hello.rs is a basic hello-world test. The rust compiler uses a tool called “compiletest” which will test, and compare the compiler output against the expected output which is stored in a .stdout or .stderr file located next to the test. See output comparison for more details.

The errors and warnings in the rust test cases was annotated with comments within the source file.

For explaning, let’s take a random example of a Rust test case:

trait Trait<A> {}

trait Assoc {
    type Ty;
}

impl<A> Assoc for dyn Trait<A> {
    type Ty = i32;
}

fn main() {
    let _x: <dyn Trait<i32>>::Ty; //~ ERROR ambiguous associated type
}

It’s stderr file was:

error[E0223]: ambiguous associated type
  --> $DIR/ambiguous-associated-type-with-generics.rs:13:13
   |
LL |     let _x: <dyn Trait<i32>>::Ty;
   |             ^^^^^^^^^^^^^^^^^^^^ help: use fully-qualified syntax: `<dyn Trait<i32> as Assoc>::Ty`

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0223`.

The error annotation needs to match with the line of the diagnostic. The rust compiletest tool uses these methods to match the message with the line

  • ~: Associates the error level and message with the current line
  • ~^: Associates the error level and message with the previous error annotation line. Each caret (^) that you add adds a line to this, so ~^^^ is three lines above the error annotation line.
  • ~|: Associates the error level and message with the same line as the previous comment. This is more convenient than using multiple carets when there are multiple messages associated with the same line.

See Error annotations for more.

It’s equivalent dejagnu version after passing this to rusttest-to-dg tool, we get:

➜  rusttest-to-dg git:(feat/add-additional-options) ✗ cargo run -- -f test2.rs -e test2.stderr
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/rusttest-to-dg -f test2.rs -e test2.stderr`
//@ run-rustfix
trait Trait<A> {}

trait Assoc {
    type Ty;
}

impl<A> Assoc for dyn Trait<A> {
    type Ty = i32;
}

fn main() {
    let _x: <dyn Trait<i32>>::Ty; // { dg-error ".E0223." "" { target *-*-* } }
}

Note that, we are only adding error codes, not the error messages. We are not comparing error messages, as gccrs and rustc error messages differs in grammatical terms but have same context. We are just making sure, that gccrs and rustc emits same rustc error codes. Therefore, as long as the error code is correct, the regex pattern in dg-error will pass.

Contributions Overview:

Pull Requests:

For a comprehensive overview of the pull requests I’ve submitted, visit pull requests page.

Issues:

To delve deeper into the intricacies of the issues I’ve engaged with, feel free to explore the dedicated issue page.

Work for the future:

  • Write the invocation script for the tool in the rust language.
  • Add support for more header directives in the rust test cases.
  • Add more error codes and fix ICE issues that are still present in the latest version of currentgccrs.

Learning Journey:

  • Learned a new programming language - Rust and written a tool “rusttest-to-dg” in Rust.
  • Learning how to effectively navigate into large and complex codebases.
  • Prioritizing progress over perfection. A piece of advice from my mentor, has really helped me to keep moving forward and not get stuck on small details.
  • Overall, I have learned a lot about the GCC Rust Frontend and the Rust programming language.

Acknowledgements:

I am very grateful to the following people who made this project possible and supported me throughout the process. Their expertise, feedback and encouragement were invaluable for me.

Pulls →