Arm/neural-super-sampling

Neural Super Sampling (NSS)

⚡ NSS v1 is now available • New architecture and trained weights. • Higher image quality compared to NSS v0.1, with comparable runtime performance. • Three quality modes (High, Mid, and Low) let you balance image quality and performance. • Updated example scenarios using fragment shaders.

Neural Super Sampling (NSS) is an innovative, efficient network for temporal super sampling on mobile devices. Content rendered at a low resolution can be upscaled to a higher resolution, resulting in up to 50% GPU savings.

With our retraining tools content creators and game studios can build derivatives of the model suited to artwork style and performance requirements.

🎥 Neural Super Sampling Demo

Model Details

Neural Super Sampling (NSS) is a parameter prediction model for real-time temporal super sampling developed by Arm, optimized for execution on Neural Accelerators (NX) in mobile GPUs. It enables high-resolution rendering at a lower compute cost by reconstructing high-quality output frames from low-resolution temporal inputs. NSS is particularly suited for mobile gaming, XR, and other power-constrained graphics use cases.

NSS v1 provides three quality modes that allow you to balance image quality and runtime performance:

• high: highest runtime cost but best-quality option. Checks the current frame and depth/motion detail more thoroughly, uses a larger image filter for clean-up, and samples previous frames more accurately.
• low: lowest runtime cost but lowest-quality option. Uses lighter current-frame and depth/motion checks, and samples previous frames less accurately. May show more flicker or motion artifacts around fine detail and moving objects.
• mid: balanced option. Similar to low but samples previous frames more accurately, like high.

For a more detailed explanation of these quality modes, refer to the Neural Graphics Model Gym documentation.

Developed By	Arm Limited
Model Type	Temporal image super sampling
License	Other
Repository	Neural Graphics Model Gym
Blog	How Neural Super Sampling Works
Paper	Mobile Neural Super Sampling
Quickstart with ML Extensions for Vulkan®	ML Extensions for Vulkan® Learning Path
Quickstart with Unreal®	NSS integration into Unreal® Engine Learning Path

NSS is under active development with regular updates planned. As we increase the size and diversity of the training dataset we expect to see significant quality improvements. Follow Arm to stay up to date on the latest releases.

The model is released under Arm's permissive AI Model Community License, designed to foster innovation in the graphics industry and provide differentiation to content creators. It allows NSS to be retrained on datasets captured from your own content.

A plugin for capturing data in Unreal Engine is available to capture your own content for use in (re)training or evaluation in the Neural Graphics Model Gym.

---

Integration

NSS can be directly integrated into graphics pipelines using ML Extensions for Vulkan.

The recommended way of integrating the model into a graphics pipeline is by using the VGF Library from the ML SDK for Vulkan.

Integration in Game Engines

NSS has been integrated into Unreal Engine via the NSS Plugin for Unreal Engine. See the Learning Path for step-by-step instructions on how to use NSS in Unreal Engine.

It is also available to use as part of the Neural Graphics SDK for Game Engines. This is designed to help integrate NSS across a wide variety of game engines.

---

Getting Started

This repository contains pre-trained weights and compiled NSS models in VGF format, ready for integration with Vulkan applications. These VGF models are unshaped, so will require developers to shape-specialize them before use. An example of how to do this can be seen in the Neural Graphics SDK for Game Engines.

See the included ML SDK for Vulkan scenarios for the simplest way to evaluate the model. The scenarios include the necessary pre- and post-processing fragment shaders along with a single frame worth of input data.

They demonstrate full execution of NSS on a Vulkan compute-capable system. The Emulation Layer can be used to implement ML Extensions for Vulkan where it is not supported by the native Vulkan driver.

Three different scenarios for high, mid, and low quality modes of NSS are provided. For demonstration purposes these work on a fixed size input, performing upsampling from 540p to 1080p.

1. Clone the Repository

These instructions will work on both Windows and Linux. On Windows, we strongly recommend the use of PowerShell over cmd.exe ("Command Prompt").

1. Ensure that Git LFS is installed. See the project's website for more details.

2. Clone the NSS model repository from Hugging Face:

git clone https://huggingface.co/Arm/neural-super-sampling
cd neural-super-sampling
git lfs pull

2. Install the Scenario Runner and Emulation Layers

1. Follow the steps in the Python documentation's "Create a new virtual environment" guide. We will assume below that you installed this environment into a .venv subdirectory.

2. Follow the steps in the Python documentation's "Activate a virtual environment" guide.

   Note for Windows users: It is not necessary to determine the location of the Python interpreter as described in this guide. For reference, however, the correct PowerShell command to do this is gcm python rather than the given where python.

3. Install Arm's Scenario Runner and Emulation Layer into your new virtual environment:

   pip install ai-ml-sdk-scenario-runner ai-ml-emulation-layer-for-vulkan
   

4. The Emulation Layer package provides a helper, that prints the commands to set environment variables for your current platform and virtual environment:

   • Enter the following command:

   emulation_layer --help
   

   Copy and run the commands it prints in the same shell you will use for scenario-runner.

   On Linux this will set LD_LIBRARY_PATH, VK_LAYER_PATH, and VK_INSTANCE_LAYERS. On Windows it will set the equivalent PowerShell $env:VK_LAYER_PATH and $env:VK_INSTANCE_LAYERS variables.

   For more help, the Emulation Layer's documentation explains how its environment variables should be configured on Linux and on Windows.

3. Run the Scenario

1. Change your working directory to the scenario folder:

   cd scenario
   

2. Execute the Scenario Runner with your scenario of choice:

   scenario-runner --scenario configs/960x540_1920x1080_high_fragment.json --output out_high
   scenario-runner --scenario configs/960x540_1920x1080_mid_fragment.json --output out_mid
   scenario-runner --scenario configs/960x540_1920x1080_low_fragment.json --output out_low
   

3. Use RenderDoc to view these images. Output images are encoded as B10G11R11_UFLOAT. This format is common for framebuffers but not widely supported by image viewers.

---

Training and Evaluation

For background on NSS architecture and training read our blog: How Neural Super Sampling Works

Training and evaluation details, including model architecture code, training pipeline, test configurations and dataset capture plugins, are available at:

• Model training code: https://github.com/arm/neural-graphics-model-gym
• Examples and tutorials: https://github.com/arm/neural-graphics-model-gym-examples
• Sample dataset: https://huggingface.co/datasets/Arm/neural-graphics-dataset
• Dataset capture plugin: https://github.com/arm/neural-graphics-data-capture-for-unreal

🔎 Model Explorer VGF Extension

The VGF extension to Model Explorer provides a simple interface to visualize and analyse the NSS model and any other VGF model.

Biases, Risks, Limitations and Out-of-Scope Use

• Not suited for non-temporal tasks such as a standalone image upsampling
• Requires accurate motion vectors and frame history for stable output
• May underperform in extremely low framerate scenarios (<10 FPS) with fast camera movement
• Padding of the input is needed if input dimensions are not divisible by 8

Recommendations

For ultra-low-FPS use cases, reduce the camera speed, acceleration, or both so that the relative motion between frames mimics the application running at a higher frame rate.

License

• The license for the model source code can be found here.
• The license for the content of this repository can be found here

More Information

🧑‍🔬 More technical details about the model can be found in the NSS Guide.

👩🏽‍💻 Our Neural Graphics Development Kit contains engine plugins, model training tools, code examples and extensive developer documentation.

🙋🏻‍♀️ For questions or feedback please start a discussion

---

Trademark Notice

• Arm® is a registered trademark of Arm Limited (or its subsidiaries) in the US and/or elsewhere.
• Python® is a registered trademark of the Python Software Foundation.
• Unreal® is a trademark or registered trademark of Epic Games, Inc. in the United States of America and elsewhere.
• Vulkan and the Vulkan logo are registered trademarks of the Khronos Group Inc..
• Windows® is a trademark of the Microsoft group of companies.