Simulation / Modeling / Design

Implementing NVIDIA Highlights Plugin for Unreal Engine 4

NVIDIA Highlights (or just Highlights) is a feature of NVIDIA Shadowplay that allows players to capture in-game moments automatically based on in-game events. (More information about Shadowplay can be found here)This tutorial focuses on the Unreal Engine plugin which adds support for NVIDIA Highlights to Blueprints and C++, allowing developers to integrate the SDK into their game with very little effort!

We’ll cover blueprints first and give you an idea of how Highlights works. Then we’ll dive into the C++ interfaces (just a bit!) and go over how PLAYERUNKNOWN’S BATTLEGROUNDS implemented NVIDIA Highlights into their very popular battle royale shooter.

NVIDIA Highlights SDK Overview

Players often wish they could capture and save memorable moments during gameplay. A game’s developer knows those key moments that deserve to be recorded and shared with other gamers. NVIDIA Highlights allows developers to add triggers in their game code that activate automatic video recording and save short videos during a gameplay session. A C++ SDK (upon which the UE4 plugin is based) can be found on the NVIDIA public GIT.  You can also find a more detailed overview of how NVIDIA HIghlights works on the GIT readme. Figure 1 shows the overall flow of Highlights in a game.

NVIDIA Highlights
Figure 1. High level flow of NVIDIA Highlights in a game

Unreal Engine Plugin Usage

Once you obtain the plugin, you install it as an engine plugin and rebuild. After that, it can be switched on or off in the plugin menu, the “other” section in the list, as shown in Figure 2.

NVIDIA Highlights Plugin Unreal Engine 4
Figure 2. HIghlights plugin location

Let’s look at the three main stages for using the plugin in your game:

  1. Init and configure. This is where compatibility checks, the Highlights permission requests, and types of highlights setup happen.
  2. Capture Highlights. People want to save spectacular events which occur frequently in single-player game levels, quests, multiplayer rounds, and raids. Highlights event groups are opened and closed here, screenshots taken, and videos recorded.
  3. Review Highlights. All event groups are closed after temp videos and screenshots have been taken. It’s time to open the summary and let the user decide what are the most interesting moments people want to share.

Blueprints

Developers using the plugin can choose between using the C++ interfaces or the Unreal Engine 4 Blueprints visual scripting engine. The two can interoperate as well. The general usage pattern is the same for either.  We’ll talk about blueprints first and then move on to C++ with some anecdotes. After enabling the plugin, new functions become available by right-clicking the blueprint space and finding the “Shadowplay Highlights” group, shown in Figure 3.

Figure 3. Blueprints initialization tree

Initialization and Configuration

Let’s start with initialization.

The game needs to do two main things:

  1. Create the SDK interface and register to GFE
  2. Request Permissions to use Highlights system

Init Highlights action takes care of #1.  The call should be issued only once at application start; the right place to call this should be the game instance, shown in Figure 4. 

The Highlights system will use In Game Name to identify your game and store the Highlights options. The application uses Video and Screenshots operations; the Init Highlights function will ask the backend for permissions to use them.

Figure 4. Initializing NVIDIA Highlights in blueprints.

OnSuccess and OnFailure are the two triggers out.

The OnFailure execution path triggers if the Highlights system fails to start and the reason gets transferred to the GFE SDK Return Code.

Note: There are also some error codes that are not translated to the blueprints nodes. See the Debugging section to find out how to solve problems.

This function outputs the structure “GFE SDK Properties” with SDK and GFE versions as well as available operations permitted by the system. A helper function “Check if Highlights Available” parses this structure and provides an interface to handle possible main cases, listed in Figure 5.

Figure 5: Checking video / screenshots granted.
  • Video Granted and Screenshots Granted will be “true” value if the requested operation status is “Granted”.
  • Video and Screenshots pins are an enum for reasons that a permission might be false.  The can be:
    • Denied:  the user is not interested in the highlights in this game and disabled it in the GeForce Experience Shadowplay menu.
    • MustAsk: the application must ask for permissions explicitly with the function GFERequest permissions:

The GFERequest Permissions function gets called in case if one of the returned values for Video or Screenshot operations are in MustAsk state, shown below in Figure 7. 

Figure 7. Permissions

Of course, the developer must handle other cases, like Denied case. For example, the game can ignore the Highlights triggering in the case if both Screenshots and Videos are disabled.

Now we have all the permissions and the system is activated. Let’s now configure the Highlights.

Configure

We need to tell the system what type of Highlights we have in our game and what are their names. The Highlights Configure function helps us here, receiving the Config Properties structure. This mainly consists of a list of highlight definitions. Table 1 contains explanations of the definitions.

 

Id A string token that uniquely identifies the event
User Default Interest Sets the default value of a highlight to be enabled/disabled.  Users may manually change this dynamically at runtime via the GFE Share overlay.  This value initializes the default. Should probably be set to True for most highlights.
Highlight Tags This is a classification of the type of highlight.  Set it to what makes sense to you.
Significance This is an indication of how significant this type of highlight is relative to others.  A range from very bad to very good.
Name Translation Table This is a name/value pair listing locale names and a localized string of the highlight type for that locale.  If empty or target locale is not found, will use the default locale or Id as the localized description.

Table 1 Config Properties

Polling and Callbacks

Every command is asynchronous while the game runs and issues calls against the Highlights SDK. This minimizes or eliminates stutter during gameplay. Thus, all returns codes and results are obtained via callbacks.

A Poll function must be called periodically to get these callbacks from the GFE SDK. If this call isn’t issued periodically, the OnSuccess and OnFailure events will never be triggered! Please use the Timer or Tick event to periodically call the Poll function.

Figure 8. Make sure to poll!

Capture Highlights

Opening Capture Groups

First, all the screenshots and videos need to be grouped. Groups allow the Highlights to arrange the videos and screenshots. Figure 9 shows how first the new group should be opened:

Figure 9. Grouping and Group ID

The Group ID is the string name of the group that represents the group for internal usage. Group Description Translation is the array of strings and appropriate locales. They represent the right group title for different localizations.

Recording videos and taking the screenshots

After the group is opened, we can record videos and take screenshots, shown in Figure 10:

Figure 10. Now we can capture!

A variety of parameters need to be set for capturing video or screen shots. Table 2 lists these parameters.

Group Id ID of the opened group that will hold the recorded video
Highlight ID ID of the Highlight entered in the Config Params structure at the beginning of the setup process.
Start Delta Millisecond offset of the video start from the moment in time when the highlight was triggered (for example, start recording several seconds before a boss kill). Negative values are allowed and most highlights will capture with a negative start deltaThis is a classification of the type of highlight.  Set it to what makes sense to you.
End Delta Millisecond offset of the video end from the moment in time when the highlight was triggered.  Will likely be a positive value, but can be negative as well. End must be greater than start or an error will return via callback.

Table 2 Key parameters for video and screenshots

Reviewing Highlights

When the game level or match ends or the player exits to a lobby, they can take time to review the temporary saved highlights and decide which ones to save or post to social media.

First, ensure that all the opened groups are closed. Use the Highlights Close Group to close groups with the Group ID to be closed. Destroy Highlights shows if the system should delete the videos and screenshots that were not checked for saving by the user.

The Highlights Open Summary function opens the GFE overlay with all the Highlights summary, as we see in Figure 11.

Figure 11. Highlights summary and filters

Open Summary Highlights contains the filter for highlights. The filters comprise group IDs, tags and significance values that the taken highlights will be matched to.

C++ Interfaces a0nd PUBG use case

PLAYERUNKNOWN’S BATTLEGROUNDS (PUBG) is one of the most popular games that utilizes Nvidia  Highlights to allow a user to record video of significant gameplay. We’ll work through a very brief overview of the C++ interfaces in this section, then describe how PUBG makes use of the Highlights SDK via the Unreal Engine 4 Plugin to track their game’s events and highlights.

C++ Interfaces Quick Overview

The UE4 Plugin defines a few interface classes which wrap the GFE SDK C++ DLL interfaces.  Your best bet to understand them is to look at the source code.  It’s relatively straightforward as shown in figure 13 below.

Figure 12. C++ class hierarchy for the plugin

The FGfeSDKHighlights function class contains most of the functions you’ll use and directly correspond to the C++ SDK methods — just wrapped with UE4 types.  (Please see the official C++ SDK documentation in the GIT repository or in the “ThirdParty/NVIDIAGfeSDK\docs” for details on the function calls.  Usage is the same as in C++).

PUBG Usage flow

Since PUBG implements most of the game logic in C++, the game uses the SDK C++ interface in UE4. Figure 14 depicts the GfeSDK API in-game usage flow. The game creates the GfeSDK instance and configures the Highlight definition table when the game starts up. It opens/closes a new SPH group when a match starts/ends respectively and shows highlight summary when the user exits to the lobby screen after closing the SPH group. During a match, game events trigger highlights conditions through the plugin to save videos by calling SetVideoHighlight. Figure 13 outlines the flow.

Figure 13. Plugin API usage flow

PUBG Highlight types

The developer defined 7 types of highlights, shown i1n Table 1. The game configures the highlight definition table before getting into the main lobby. As per Highlights SDK requirements, an event comprises 2 times, “timeBeforeEvent” and “timeAfterEvent”, which represent temporal ranges of the recorded video.  PUBG also includes an additional, developer-defined used internally called “waitAfterEvent”, defines a time interval between consecutive event occurrence. Table 3 outlines the event types recorded.

typeOfEvent timeBeforeEvent timeAfterEvent waitAfterEvent
Kill -30 +10 30
Knockout -30 +30 30
Die -30 +5 5
Knocked Out -30 +10 30
Match End s-30 +5 5
DeathCam 35
Replay

Table 3. Event types recorded via NVIDIA Highlights.

Figure 14 shows an example of what happens when a player kills another player. The game will wait “waitAfterEvent” amount of time and call SetVideoHighlightAsync for a single kill video. If a new kill occurs within waitAfterEvent period, the game ignores waitAfterEvent#1 and will save a ‘double kill’ video after waiting another waitAfterEvent of 30 seconds.  

Figure 14. Example function call timing for single (a) or double (b) kill events

Global event stack

Event structure

The PUBG defines a stack structure to manage the timing of Highlight function calls and length of videos. The function instantiates one global stack that stores all types of events except the replay type. When an event occurs, the engine allocates and pushes a highlight_event stack element declared as the following type definition.

typedef struct {
     int timeOfEvent; // time of event occurence     char* typeOfEvent; // type(name) of event: kill, death, …
     int timeAfterEvent; //post-event range of video(msec)
     int timeBeforeEvent; // pre-event range of video (msec)
     int waitAfterEvent; // time to wait for further events (msec)
} highlightEvent;

Event stack handler

The stack handler monitors the stack and calls SetVideoHighlightAsync function at the proper. The stack handler checks whether waitAfterEvent amount of time has passed since last event by testing timeOfEventtop+waitAfterEventtop < currentTime, as shown in Figure 16. Pushing a new event onto the stack resets the wait timer, since the stack handler will check  the new timeOfEvent and waitAfterEvent on top. When a waitAfterEvent time has passed, the handler calls SetVideoHighlightAsync with an temporal range from first (stack-bottom) to last (stack-top) events.

Figure 15. Event stack handler checks stacktop to determine when to call SetVideoHighlight

Figure 16 illustrates the startDelta, endDelta to be input to SetVideoHighlightAsync call. The length of the highlight video is determined by the difference between them. You can see how this can be handled in the code sample below the figure.

Figure 16. startDelta and endDelta calculated from currentTime
if(stack.isEmpty()) return;
if( currentTime > ) {
    int startDelta = -(currentTime - stack.bottom().timeOfEvent + stack.bottom().timeBeforeEvent);
  int endDelta = -(currentTime - stack.top().timeOfEvent - stack.top()timeAfterEvent);
  NVGSDK_VideoHighlightParams videoHighlight = {stack.top().typeOfEvent, startDelta, endDelta };
  NVGSDK_Highlights_SetScreenshotHighlightAsync(... , videoHighlight, ... , ...);
  stack.flush();
}

Replay/deathcam event

PUBG also utilizes NVIDIA Highlights in deathcam and replay mode. Like other types of events, the engine pushes one event onto stack when the deathcam is started. Since a user is not able to input any command during deathcam, the stack handler will monitor only one event in the stack and save a video clip for the event after waitAfterEvent (35 for deathcam).

Figure 17. Record toggle button in replay mode

The game also provides NVIDIA Highlights recording in replay playback. As figure 18 shows, a circular record button appears when the Highlight feature is enabled. A user can start/finish recording by toggling the button. PUBG opens a new group (OpenGroup()) when entering replay mode and the replay can be played back after exiting the lobby. Replay module does not exploit the stack scheme and directly calls SetVideoHighlightAsync function, unlike the other event type.

Debugging Highlights

The plugin relies on the GeForce Experience SDK functionality. The log files left by the SDK can be inspected when error determination is not possible with the legal plugin options.  You can often find the source of the problems by inspecting the logs.

The GFE SDK logs are in the following folder:

C:\Users\<MyProfile>\AppData\Local\NVIDIA Corporation\GfeSDK

NVIDIA Highlights Improves Game Experiences

As we’ve shown, implementing NVIDIA Highlights automates the collection of screenshots and video during climactic gameplay moments. Users deeply engaged in their gaming don’t need to remember to activate Highlights; it’s all handled for them. Implementing NVIDIA Highlights in Unreal Engine 4 games is relatively straightforward to implement, allowing developers to give players another reason to stay immersed in their titles.


Discuss (2)

Tags