Analysis of large binaries and games in Ghidra-SRE

Ghidra is a free and open source reverse engineering suite. It is flexible with scripting and plugins and can be used for almost any architecture. If one does not exist you can add it yourself, if there’s a bug you can fix it yourself, which is the beauty unlike competitors such as IDA Pro. (Did I mention it was free as well???) I have been checking out Ghidra since the initial public release and for “real work” situations it was less than ideal. I kept watching process on the bug tracker, and noticed that 10.x had gotten many improvements and bug-fixes over the 9.x release and thought to give it a second try.

The initial results were not spectacular, running into the same Swing timeout errors (which can be solved by looking here), and overall taking 24+ hours to analyze a 300MB executable with symbols just to finally crash about about a day and half later, leaving me back at square one. This is the experience of many people that I have talked to who are into games reverse engineering where binary sizes can balloon very fast, even without symbols. I will go over the steps for PC executables that are large without symbols, but the process should be about the same for other platforms (PSX, PS2, PS3, PS4, PS5) provided you have the loaders/scripts and follow the instructions for them.

Prerequisites

Before beginning we will need a few things, first you will need to have Ghidra downloaded and extracted, at the time of writing this is version 10.0.1. Since most games are written in C++ and may have some form of RTTI (Run-Time Type Information) having a plugin to handle this is ESSENTIAL. Luckily there is, made by someone who is one of the most amazing developers that contributes to Ghidra that I have met astrelsky! His work in this area is some of the greatest I’ve seen come out of the OSS community. We will be using his plugin for the C++ Class Analyzer which handles a plethora of C++ specific things. You can download the C++ Class Analyzer on the releases section of GitHub. At the time of writing 10.0.1 is not “supported”, but it is just a very minor tweak to get it to work and will probably be updated in the future. Open the download zip file (at time of writing, ghidra_10.0_PUBLIC_20210623_Ghidra-Cpp-Class-Analyzer.zip) in your favorite archival tool (I use 7-Zip) and find the file extension.properties in the folder Ghidra-Cpp-Class-Analyzer and double click to edit it.

extension.properties file in the archive

You will need to edit the version information to say version=10.0.1 instead of the default version=10.0 save the file, and 7-zip usually will ask if you want to save to the archive, select yes.

Updated version information

Launch Ghidra and click on File -> Install Extensions click the + and select the Ghidra-Cpp-Class-Analyzer zip file.

It should be added to the list and check the checkbox to enable it and restart Ghidra.

Once Ghidra is restarted, create a new project, give it a name (I use GhidraTutorial) for this and drag an drop in your binaries. If you are loading PS1/2/3/4 ensure that the right architecture is selected. For PC binaries it usually detects correctly on the first try. In this case I am using the game Horizon Zero Dawn GOG release on PC. Allow the import to complete, but do not launch the CodeBrowser in Ghidra quite yet.

Import prompt for Horizon Zero Dawn

You will need to close Ghidra at this point because the GUI for CodeBrowser when analyzing causes nothing but issues. This is where the normal usage of Ghidra will come to a pause. We will be using the Ghidra Headless Analyzer which is a CLI tool for Analyzing or Importing files into a Ghidra database. Open up a new Terminal window (I use Powershell on Windows, Bash on Ubuntu/Linux) and navigate to the Ghidra support directory. In my case this is PS C:\Users\godiwik\Documents_tools\Ghidra\ghidra_10.0.1_PUBLIC\support .

Since we have already imported the executable with the correct architecture we will just need to analyze using the -process flag. The command we will use is .\analyzeHeadless <directory to the Ghidra database> <DatabaseName> -process "*" -recursive which should handle analyzing anything we have already imported. If this does not work for you, try adding some to the wildcard such as Horizon*, or if you have it in a folder, supply the folder name with a wildcard behind.

The example that I am using is .\analyzeHeadless ../../ GhidraTutorial -process "/*" -recursive since my Ghidra database (GhidraTutorial.gpr/GhidraTutorial.rep) is one directory above the Ghidra root directory. If no errors appear, just go take a bathroom break, get some coffee, play with your favorite pet, feed your kids whatever you have to do because this will take awhile…

In process of analyzing… Waiting for completion

After everything is all completed, you can re-open Ghidra again. And then double click on the executable to open up the CodeBrowser.

Headless Analyzer Completed in ~30m

After this point everything should be analyzed, and the C++ Analyzer should also have been ran during the headless analyzer. If you aren’t seeing good results, then click Analysis->All Open->Deselect All->Windows (or GCC) C++ Class Analyzer (prototype), increase the Decompiler timeout from 30 seconds to 120 or 240, and click Locate Constructors then click Analyze. Wait for that to complete and everything should be ready for you to explore!

Additional C++ Class Analyzer options

All Done!

I hope that this was able to help someone, I know I was frustrated with attempting to use the GUI for large binaries, this is a workaround that is pretty awesome and you could even automate this for automatic analysis.

I’d like to thank OSM, Seremo, Shiro, krystalgamer_, Z80 in no particular order for taking interest and or helping proofread.

kiwicon – Modern Warfare Remastered Console Source

Just something I threw together in 5 minutes. Do note that a bunch of dvars have been hashed, and no longer are able to be changed by their original commands. Some work, some don’t. There is no output from the game, but you can input console commands.

Some commands that work:

  • spdevmap
  • cg_fov
  • cg_fovScale
  • jump_height
  • g_gravity
  • g_speed
  • fx_enable

 

kiwicon for Call of Duty: Modern Warfare Remastered SINGLEPLAYER Source Only (version 1.0.0.1 ONLY!!!!!)

Rime: November Progress Report – Preparing and Automation

It’s been a wild few months, and I would like to thank all of the testers of Rime that have been reporting bugs, suggestions and usability problems.

Rime: Operation Metro WIP

One main issue that I have with Rime is the lack of automation. It had gotten an auto-updater quite awhile back, but it was very primitive and needs an overhaul so I can push updates faster to the testers and eventually the entire community. With this documentation is key for such a complicated and large project and before hand-generating the documentation or even using a tool like DoxyGen wasn’t exactly up-to-par. I’ve spent the last few days creating DokuGen which takes the C# assemblies and XML documentation and creates a DokuWiki format output so you can just drop it in your dokuwiki/data/pages/ directory and have it reflect instantly online!

Continue reading “Rime: November Progress Report – Preparing and Automation”

Rime: June Progress Report

3 Months Later, much progress has been made. Here is another dump of the git repository change-log. I won’t be able to continue development on Rime until after July due to myself switching apartments again (I move every year =[ ). With that being said, I have set up a Continuous Integration server for Rime so current alpha testers can get bleeding edge builds if needed. There’s only one problem, deploying the builds 😛 I will be writing an open source deployment tool that I will upload on my github when it is completed.

With that being said, I have a software recommendation, If you haven’t heard PVS-Studio is a hell of a tool catching many programmer errors and mistakes. Rime didn’t suffer from many errors, only 1 high level issue, ~40 medium (mostly serialization issues), and 200 low priority ones (caused by floating point arithmetic) Read more to see full change-log.

Continue reading “Rime: June Progress Report”

Rime: March Update Report

Good news, Rime is becoming more and more functional every day that it gets worked on (busy with school primarily). The last month has been both productive and a ton of work has gone into the project. Here are some of the changes from the git history.

Implementations:

  • Implemented “-cleanup” option for the auto-updater in order to clean up old files on next launch.

Additions:

  • Added more editing abilities to the PartitionBrowserControl.
  • Added logging of version on startup (the UI is currently broken and doesn’t show it)
  • Added “Dashes” member to GUID in order to get the dashes format as well.
  • Added checking of instances on CtrRef ToString in order to show “*null*” if the partition or instance is null. (may change to partition only)
  • Also made it where if you double click on an CtrRef it will automatically open up a new editor with that instance loaded.

Bug Fixes:

  • Fixed bug with the PluginManager in where multiple instances would be removed due to old-style name checks. This was resolved by properly removing that one instance.
  • Fixed crash in the OutputPlugin due to the parameters being null.
  • Fixed threading related crash on updating the UI when a new update was made available.
  • Fixed crash in the logger when parameters were null.

Changes and Corrections:

  • Corrected List DataTemplate, by changing the member type from TreeView to ItemsControl.
  • Refactored update system.
  • Major refactoring to allow plugins to load properly. Also disabled the Theme in order to properly see what is going on in the ebx editor.

Thanks to all of the testers we have, and special shoutouts to Powback for assisting with the documentation and preparation for the public build.

For usage with VeniceEXT
For usage with VeniceEXT

Rime: January Progress Report

Time flies, I really thought I wrote the last entry last month, but apparently its been quite awhile. Here’s a bit of a status update…

I’ve been swamped with work and fixing issues and researching more cool things for Venice Unleashed. So work on Rime has been slow, but after reading the last blog post, there has been a ton of work behind the scenes that have been done with Rime.

  • Converted quite a few parts to use the MVVM for raw speed and performance
  • Added chunk reading via the VFS
  • Implemented a “new” ebx reader written by NoFaTe almost 3 years ago (+- a year)
  • Corrected issues with said EBX reader
  • Implemented ebx VFS and cross-ebx reading
  • Ebx reading and parsing is all async and threaded now and done in the background without affecting initial map loading times. (Maxed out my laptops i5 for ~9s before finishing)
  • Updated all projects to Visual C++ 14.0 and .NET 4.6.1 (fixes all UI issues using C# 6.0)
  • The bundle browser is now sorted alphabetically
  • Updated bundle browser plugin to use the VFS instead of manually parsing bundles itself.

And many many more changes to increase performance etc.

Rime: September Progress Report

Hello there, I know it’s been awhile since I’ve updated this and a few people thought the project was dead (I don’t know why, but they did lol).  Here’s a quick update of what has happened over the last few months. Most of the work has been done by the other member of Venice Unleashed NoFaTe!

Fixes:

  • Fixed a crash that would occur due to plugins without UI’s being added as UI elements.
  • Fixed various threading issues and collection changed issues due to the asynchronous nature of Rime.
  • Fixed a ton of issues dealing with the UI and how they interface with plugins.

Updates:

  • Updated WebLink security (cross communication to modders link to prepare for the community)
  • Updated the BinaryReader/Writer with more efficient versions.

Refactoring:

  • Removal of a lot of deprecated, unused, commented out code.
  • Deprecated all previous (semi-functional) mounting code (due to the new readers/virtual file system).
  • Moved a TON of shared code (used between Lib/Rime itself) to the RimeCommon and restructured it.
  • Moved a TON of Frostbite specific utilities and classes to RimeLib.
  • Refactored the plugin system again (Will have to update the RimeExamplePlugin github soon).
  • Refactored the handling for Frostbite Guid class.

Features:

  • Added the ability to auto-start certain plugins on launch
  • Added all available plugins to be launched from a menu if you are unsure how to launch them (only ones with UI).
  • Added auto-detection of game builds and information.
  • Added BuildInfoProxy for x86 and x64 versions of Frostbite to read out the engine’s build info natively.
  • Added detection for Frostbite2014.4 (Star Wars Battlefront Alpha Trial).
  • Implemented a new Virtual File System.
  • Loading of patch files and patched superbundles completed.

Changes:

  • Added Rime to a CI solution to prepare for testing builds for those who have already been selected for testing.

 

Rime: Closed Alpha

Well, it’s been a month since the last update. But in that month, Rime went from being non-functional, but powerful tool to a semi-functional extraction for Frostbite 2 (and 3) games. There has been a huge overhaul in the back-end to allow it to function in a modular fashion.

Currently there are many bugs, but to poke around and open up various resources has been implemented and soon(TM) integration with Venice Unleashed will be finished and the community will have the first taste of modding the Frostbite engine.

Continue reading “Rime: Closed Alpha”

Rime: May Progress Report

Sup,

Just a few updates. As read in my previous post, security is something I take very seriously as a developer, and I was going to allow plugins to run unrestricted until the beta. Luckily with enough debugging and trial and error, I finally got proper locked down sandboxed plugins that can be loaded from any directory! With that being said as well, I have resolved issues with the internal messaging structure for Rime as well as changed the way that plugins function. I also will be updating the example plugin git repository.

Fixes:

  • Overhauled and fixed internal communication of all parts of Rime
  • Fixed loading of plugins
  • Implemented plugin security sandbox
  • Cleanup of old code, refactored entire plugin system

Removals:

  • Removal of SharpDX and all DirectX 11 related mesh rendering
  • Removal of IPlugin interface.

Implementations:

  • Moved all plugin logic and abstract classes to a common module (RimeCommon)
  • Moved most message logic and listeners to a common module.
  • Setup an OpenGL based renderer to replace SharpDX. Proof of concept finished.

Also I’m moving, so there may not be any work done until I find a place to live. Peace.

Rime: April Progress Report

Hey,

There have been a few things that have been going on in the last few months. I just wanted to give everyone an update on the current status of Rime and the Venice Unleashed interop. First things first, plugin’s are hard, if you have checked the previous post on plugins and open source, I am making use of a plugin system for all external and internal plugins. That way you, the hobby modder can look at what we have to offer, and the functionality of it and be able to write your own code to interact with the Frostbite engine. That’s all fine and dandy, but I am a security-oriented coder. I have been working on for the last week a system to overlay the plugin interface’s to allow the code to be sandboxed, only allowing read/write access to the folder where the plugin resides. This will lessen the chance of someone writing a malicious plugin to steal sensitive information, or execute code that could potentially be harmful.

The way that I was going to get around these issues, is force the sandbox to run as lower-level privileges than Rime itself. (Even though Rime does not require to be ran as administrator, some people will anyway). That a malicious plugin writer won’t be able to gain privileges via Rime. And all of this is where the problem comes in, I have been debugging and researching the best approach to this in C#, and there just isn’t one (publicly at least). So I am making the decision to scrap the secure plugin system for the closed/open alpha of Rime until we get a functional system for the beta. With that being said, I highly recommend that any plugin creator open source their plugins so people can compile it from scratch. If you download a pre-built executable, use caution and only download from trustworthy people.

Fixes:

  • Fixed incorrect TextureHeader structure
  • Fixed DXT1-3 native C# processing.
  • Fixed EBX reading code
  • Entire application structure refactoring for prep for release (removing unused/old code etc, that’s why we have source control ain’t it?)
  • Abstracted multiple classes to prepare for multi-game Frostbite support.

Broke:

  • Internal communication of all parts of Rime. (Yeah… I fudged up bad)
  • Plugins in their entirety (which means all the built-in editors for Rime have been disabled as well, Rime’s internals uses the same plugin system)
  • DirectX 11 Mesh Renderer (Instead of using laggy WPF for everything)

Additions:

  • Battlefield 4 Resource Types
  • Proper loading of all game files and resources. (Just like how it gets done in-engine, no longer needs to decompile the entire map/game to mod resources)
  • Retrieving encryption information, and real time modding via Venice Unleashed.