Finding the Higgs on RISC-V 2023-01-28

RISC-V is a new computer architecture. It currently receives a lot of attention from open source developers because the ISA itself is royalty free and vendors can create hardware without paying licensing fees. As with any new architecture, the software ecosystem has to catch up once actual hardware is available. In this post, I would like to share the journey of making Cling and ROOT work on RISC-V.

ROOT is a data analysis framework developed at CERN that is widely used in High Energy Physics. One central component is Cling, an interactive C++ interpreter based on LLVM and Clang. After booting Debian on the StarFive VisionFive in June, I set myself the goal to make enough of ROOT work to run a (simplified) physics analysis and (re-)discover the Higgs boson. I will come back to the results of this goal at the end of the post.

As my day job also involves working on ROOT and Cling, I already gave a presentation on this topic a few weeks ago. If you are curious, you can take a look at the slides, but I will rehash the main points below. The talk was also recorded and I will share a link to the video once it is available (update: the recording has been uploaded to YouTube).

LLVM JIT and clang-repl

As mentioned above, Cling is built on top of LLVM and Clang, and in particular just-in-time compilation (JIT). So in order to “build up the stack”, the first step is to make these components work on RISC-V. Luckily, compiler support in LLVM (and GCC, for that matter) was already in decent shape. JIT support also already existed in the form of a backend for JITLink by StephenFan (luxufan), which will hopefully replace the “legacy” RuntimeDyld at some point in the future. My only contribution in this area was to make JITLink the default for RISC-V, so the JIT works out-of-the-box on RISC-V (D129092).

As the next step in the stack, I chose clang-repl which are (generic) parts of Cling being upstreamed into LLVM. To my surprise, this also worked pretty nicely with two contributions from my side: To avoid warnings, the interpreter has to pass target features to the JIT so the backend can select the right ISA extensions and calling conventions (D128853). After that change, the generated code uses some more relocations. Fortunately, these can be ignored in the specific circumstances (D129159).

ROOT and Cling

With the ground work being done, I moved to bring up a “minimal” ROOT including Cling. At that time, ROOT was based on LLVM 9 which did not yet support just-in-time compilation for RISC-V. However, as part of my day job, I was involved in the upgrade to LLVM 13, on which I based my port to RISC-V¹. The port itself consisted of two parts: First, I needed to add support for RISC-V to the build system and deal with the new architecture in configuration headers. The second part consisted of backporting a number of changes for RISC-V and JITLink that were developed after LLVM 13. This also included working with upstream (thanks to Lang Hames) and implementing two relocations for compressed instructions myself (D140827).

The final two problems were related to the LLVM JIT and its complex setup with the Clang frontend: For “global” C++ objects, Clang registers their destructors atexit, which is intercepted by the JIT. This also passes the value of the special __dso_handle symbol, which the JIT “abuses” to pass a context. However, because Clang marked __dso_handle as “local” (which it is in separate compilation), LLVM was using the “wrong” relocation. As a matter of fact, we hit the same problem one week later on macOS, and it is now worked around in Cling itself. The second problem again had to do with the modularity of RISC-V, resulting in the LLVM backend choosing the wrong calling convention. I have to confess that I did not yet find a satisfying solution here, and resorted to hacking the default calling convention for now.

Update 2023-02-19
In the end, it turns out that the calling convention can be fixed by adding a single line. This change makes sure to pass down the ABI lp64d computed by the Clang driver down all the way to the machine code generator. With that, floating point arguments are passed via registers, as is the default for Linux on RISC-V and expected by compiled code.

Finding the Higgs

In the end, though, the work proved to be worth it and Cling is working on RISC-V. With that, I was able to gradually enable more and more parts and features of ROOT. Eventually, I arrived at RDataFrame, which “offers a modern, high-level interface for analysis”. It can be used from C++ (using Cling internally) or from Python, where Cling is also used for interoperability.

So for the final challenge, I used the working port of ROOT on RISC-V to run the tutorial df103_NanoAODHiggsAnalysis.py. It combines a simplified, but still complex analysis written in Python with a C++ header file to JIT a number of functions. These are used to filter and analyze publicly available OpenData recorded in 2012 with the CMS detector at the LHC. At the end, the reconstructed mass of the Higgs boson is plotted as a histogram that you can see below – showing a bump at around 125 GeV, representing the Higgs.

For people curious to try out ROOT on RISC-V themselves: I am currently working to clean up my port (rebase it on master) and publish the changes. I will then proceed to submit pull requests to add support in the build system and a small number of other changes. For the backports in LLVM, it is not yet clear if they should be merged into ROOT right now. My current preferred approach is to keep them in a separate branch: They will come automatically with the next upgrade of LLVM as I worked to submit all changes upstream. But this is still work-in-progress, and I will update this post with more details as soon as possible.

Update 2023-02-19
The pull request for ROOT is now up as root-project/root#12351. As mentioned above and in the PR, this only includes the changes to the build system and ROOT’s configuration files and utilities. The necessary backports to ROOT’s copy of LLVM 13 are available separately as a branch in my fork on GitHub. (Note that the branch includes the three commits of the PR linked above, and then 12 backports to make the interpreter actually functional.)

You do not need to agree with my opinions expressed in this blog post, and I'm fine with different views on certain topics. However, if there is a technical fault please send me a message so that I can correct it!