Category: Research and Development

Preface

Before starting, I will note that I am aware that Unreal Horde is being used internally at Epic Games, and currently is in an experimental stage and support is not provided. I’m hoping that this can be of use to some other sysadmins, or indie game developers to get their studio running “The Epic Way” (insert guitar rift).

Acknowledgements

I’d also like to acknowledge a few people for answering some questions about their workflow.

1. DANNYonPC
   • I asked DANNYonPC if there was anyone he knew that could answer some internal CI/CD, Task Tracking, and other internal systems gamedev questions without breaking NDA. He referred me to GameDevMicah!
2. GameDevMicah
   • Thanks for answering my questions and immediately pointing me to @flassari, they were one of the only people to reply to my requests.
3. Ari Arnbjörnsson
   • The presenter, and promoter of the “Setting up an Unreal Engine Studio the Epic Way” [X/Twitter, YouTube, Epic], without there insights and overview this would never have been possible, and is the rough guide of how to do things “properly” for the best results.
4. Developer from The Farm 51
   • Could not answer most of my questions due to NDA, but at least let me know that Jira is the industry standard for issue tracking

In order to keep things straight from a configuration area, here is my (at the time of writing) configuration for the Unreal Horde setup.

Hardware Specifications

• Primary Virtualization Server
  • 16-Core Epyc with 128GB of RAM, Installed with Proxmox VE
• VM1
  • Hostname: docker.example.com
  • 64GB RAM allocated
  • Ubuntu 22.04 LTS
  • Docker + Portainer Installed
• Perforce Container
  • Hostname: p4.example.com
• Postgres Container (From Horde Docker Compose)
  • Hostname: db.example.com
• Redis Container (From Horde Docker Compose)
  • Hostname: redis.example.com
• VM2
  • Hostname: winbuilds.example.com
  • 32GB RAM allocated
  • 512GB Storage
  • Windows Server 2022 Core
• Build Server 2 (not installed yet)
  • Hostname: winbuilds2.example.com
  • 64GB RAM
  • 1TB SSD Storage
  • Windows Server 2022 Core
• Build Server 3 (not installed yet)
  • Hostname: winbuilds3.example.com
  • 64GB RAM
  • 1TB SSD Storage
  • Windows Server 2022 Core
• Laptop
  • Hostname: win-laptop.example.com
  • 40GB RAM
  • 2TB SSD Storage
  • Windows 10

Installing the Horde Server

This was quite frankly one of the easiest parts of installing the Horde server. I use Portainer, so I went to my admin control panel, went to registries, and added the ghcr-packages-readonly for ghcr.io on GitHub using a Read-Only packages token as the password to get access to the Epic Games Unreal Engine packages repository. Then I selected my local docker server, went to Stacks then created a new one for ue-horde. I won’t re-iterate this portion here because it is located in the Unreal Engine Horde Source Code documentation.

Issues with the Horde Server install

For whatever reason when installing the Horde Server using the docker-compose method that is suggested by Epic themselves. When you go to Download->Tools there were none available. I made note of this to the author Ari, and they mentioned:

I’m not entirely sure if this lines up with 1:1 to the expected experience. Is the GitHub package not the latest release? Did I mis-configure something, or didn’t set something up properly? I’m not entirely sure, this may be something to look into in the future. The workaround I did for getting the Horde Agent installed was to install the Horde Server on my laptop, then download the installer and manually copy it over to the build server and install it. This was the most confusing part of the install, and even building the binaries from source, I still had no reference on where to put them, what registry keys need to be set, or what configuration files needed to exist. I could just be a idiot at this step and missed something.

Installing the WinBuilds Server

I chose to use Windows Server 2022 Core, which does not contain the typical GUI interface for Windows Server. This would lead to a few roadblocks. All of these can be overcome.

Installing the software

One of the first roadblocks that I encountered was involving the Windows Server 2022 Core image and getting all of the requirements for the build server in order. Preparing the build server was done with a pre-made PowerShell script, but the relevant parts have been clipped below.

# Disable defender.
Write-Output "Disabling Windows Defender..."
dism /Online /NoRestart /Disable-Feature /FeatureName:Windows-Defender /Remove /Quiet

# Download cacerts
mkdir C:\certs
mkdir C:\certs\out
Invoke-WebRequest https://curl.se/ca/cacert.pem -OutFile C:\certs\cacert.pem

# Split CA cert bundle into individual certs.
$content = Get-Content -Path "C:\certs\cacert.pem" -Raw
$certificates = $content -split '(?<=-----END CERTIFICATE-----\s+)' -ne ''

$index = 1
foreach ($certificate in $certificates) {
    if ($certificate -match '-----BEGIN CERTIFICATE-----') {
        $title = ($certificate -split '\r?\n')[0].Trim()
        $fileName = "C:\certs\out\certificate_$index.pem"
        $certificate | Out-File -FilePath $fileName
        Write-Host "Certificate saved to: $fileName"
        $index++

        # Import the cert into the root trust store.
        try {
            $cert = Import-Certificate -FilePath $fileName -CertStoreLocation Cert:\LocalMachine\Root
            Write-Host "Successfully imported: $fileName"
        } catch {
            Write-Host "Failed to import: $fileName. Error: $_"
        }
    }
}

# Install Chocolatey
Write-Output "Installing Chocolatey..."
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

Write-Output "Configuring Chocolatey..."
choco feature enable -n allowGlobalConfirmation
choco feature enable -n useRememberedArgumentsForUpgrades

# Install packages
Write-Output "Installing packages..."
choco install -y docker-engine

# Install additional dependencies using chocolatey:
# - git
# - gzip
# - ninja
# - Node.js LTS
# - 7-Zip
# - CMake
# - Python 3
# - Visual Studio 2022 Build Tools (w/ all workloads)
# - nuget
choco install gzip ninja nodejs-lts python3 7zip
choco install cmake --installargs '"ADD_CMAKE_TO_PATH=System"'
choco install -y git.install --params '"/NoAutoCrlf /SChannel"'
choco install -y visualstudio2022buildtools
choco install -y visualstudio2022-workload-webbuildtools --package-parameters '"--includeRecommended --locale en-US"'
choco install -y visualstudio2022-workload-manageddesktopbuildtools --package-parameters '"--includeRecommended --locale en-US"'
choco install -y visualstudio2022-workload-vctools --package-parameters '"--includeRecommended --locale en-US --add Microsoft.VisualStudio.Component.VC.ATL --add Microsoft.VisualStudio.Component.VC.ATLMFC --add Microsoft.VisualStudio.Component.Windows10SDK.20348"'
choco install -y nuget.commandline
choco install -y awscli

Resolving issues with the Perforce client

Something that was not noted, but maybe I had missed is Horde requires Perforce to be installed, and the server to be rebooted before build agents will function properly. After downloading the P4 client from Perforce, the installer refused to operate properly. It would create a blank window then disappear later. This was resolved by opening up PowerShell to where the installer was located and running the installer in quiet mode which can be done via:

helix-p4-x64.exe /q

Additional Configuration

In order for p4 (the Perforce client) to function at all it requires a bit more of a setup. This can be found on Perforce’s Environment and registry variables page. You must set the system-wide/global environment variables on the build server. This can be done via PowerShell once again and requires these variables:

P4USER=<user that has access to perforce, I use ldap for this so this is the username of that user>
P4TRUST=<location of p4trust.txt file, to allow self-signed perforce certs over SSL>
P4PASSWD=<password of P4USER>
P4PORT=ssl:p4.example.com:1666 <perforce server hostname:port>
P4CLIENT=C:\x\h\b <location of where to build, should be a short path>

You should be able to test this configuration after another restart to ensure that it works properly with a “p4 info” command. I would also try listing the repositories with “p4 repos” to ensure that proper connection is established, and the p4trust.txt file is working as intended.

Now you have a fully configured Windows based build server which is a requirement for Unreal Engine 5.

Additional Notes on the Perforce Server (p4d) side

In order for the build server to work properly, the p4 account MUST have super access to the repository. I tried this with read/list permissions only, and Horde would fail to operate properly. I don’t believe this should be a requirement, but I’m not sure what p4 command Horde tries issuing that fails without this. You can check the log files in C:\ProgramData\Epic\Horde\Agent to debug issues going on. I’m hoping that Epic Games can shed some light on this in the future.

Potential Additional P4API issues on Build Server

I am not certain if this was required, but as I was attempting to debug issues on why Horde was not dealing with Perforce correctly, I was getting an C++ crash exception in Horde Agent which would crash the agent all together. It was dealing with the Interop with the P4API, so I downloaded the P4API files and copied them to the same directory as the Horde Agent (located C:\Program Files\Epic Games\Horde\Agent). I don’t believe this is required, but I was just troubleshooting.

Modifying the Horde Server address in the Agent build server

I’m not going to lie, I typo’d my server address and needed to change it for the Horde Agent that was installed on the build server. I scraped through the Horde Agent source code and found that registry key location is: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Epic Games\Horde Key: Url Value: http://horde.example.com:13340 after changing this and giving the build server another restart, it was connected properly to the server.

Final Results

Well I am not at the point of actually building anything yet, but that’s just configuration files which are well documented in the Tutorial: Setting up and Unreal Engine Studio the Epic Way video and documentation, which again we have to thank Ari so much for. You are a lifesaver in this situation and I cannot wait to see how Epic and Unreal Horde progress in the later versions. Expect a Unreal Engine 5 indie title eventually from me 🙂

General Information

Install paths are located

Registry Location: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Epic Games\Horde
Key: Url, Value: http://horde.example.com:13340

Registry Location: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Epic Games\Horde\Agent
Key: WorkingDir, Value: C:\x\h\

Horde Agent (the tool) logs
PS C:\ProgramData\Epic\Horde\Agent

Horde "run output" logs (this will be the directory set for WorkingDir in registry)
C:\x\h\

Horde .dll/exe/whatever files
C:\Program Files\Epic Games\Horde\Agent