Skip to content

The .nx Archive Format (Rust Edition)



High Performance File Format
For Loading and Sharing Mods.

GitHub Workflow Status

About

The Nexus (.nx) format is a semi-SOLID archive format, using modern compression technologies such as ZStandard and LZ4 under the hood.

flowchart TD
    subgraph Block 2
        BigFile1.bin
    end

    subgraph Block 1
        BigFile0.bin
    end

    subgraph Block 0
        ModConfig.json -.-> Updates.json
        Updates.json -.-> more["... more .json files"]
    end

Between size optimized SOLID archives like .7z w/ LZMA and non-SOLID archives like .zip w/ Deflate, the .nx format bridges the gap; reaping most of the benefits from both worlds.

Nx aims to be a simple format, appropriate for both local storage of mods and for downloading from the web.

Characteristics

What can you expect from Nx?

  • High Performance (constrained by RAM speeds, bottlenecks any NVMe)
  • File Sizes between .zip and .7z.
  • Lightweight Dependencies
  • Dictionary Compression
  • Self Contained Format
    • e.g. Copy of Dictionaries used always available in archive if not available.
  • Low Latency File Access
  • Partial Downloads
  • Optimized for Fast Parsing (Whole Header in Front, with Chunk Size)
  • Hardened for Server Usage (by default)
    • Passes miri (memory safety & no undefined behaviour).
    • Tests cover all known possible invalid memory acceeses.
    • Can disable client side via feature if you trust the data for parsing speedups.

Ways to use Nx

These are available as 'presets' in the Nx API

For Archiving Mods

For storage of mods:

  • Use standard 16MiB SOLID Blocks to save space by grouping small files together.
  • Chunk large files using >128MiB blocks where compression efficiency deficit is negligible.
  • Use Dictionary Compression for small SOLID blocks.

This basically turns the format into a ZStandard based archive with per-file multithreading & per-chunk for huge files.

For Read Only File Systems

  • Use dictionary compression for instant access to files.
  • Use no SOLID blocks
  • Use small chunks (128KiB) for large files.

This gets you multithreaded decompression for files of any size, and minimal latency for tiny files.

For Hosting Mod Downloads

For hosting mod downloads:

  • Use non-SOLID blocks
  • Nx blocks can be deduplicated server side, avoiding duplicate data between mod versions in CDN.
  • No dictionary compression (unless standardised).

With Nx, users can perform partial file downloads, only downloading the actual files which have changed between the current version of the archive and the last version of the archive.

Where is Nx used?

Original C# version, in the Nexus Mods App

Nx was originally designed by me for the Nexus Mods App; to allow recompressing of legacy uploaded mods for faster deployment and re-deployment.

In Reloaded3 for mod packaging

For distributing mods on mod websites, either as Nx directly, or Nx wrapped in a .zip container.

Why Rust?

I originally designed and created Nx for the Nexus Mods App

The transition to Rust here is mainly for technical reasons involving my own required use cases.

  • .NET cannot be safely used for hooking Operating System APIs.

    • This is due to GC Transitions.
    • I need this for building Nx Accelerated Virtual File Systems.

  • The Reloaded3 Project requires more portability than what .NET targets.

    • R3 aims to support esoteric platforms like embedded devices and consoles wherever possible.
    • .NET (C#) is not available on these platforms, but LLVM (and thus Rust) often are.
    • Using NativeAOT would massively bloat the binary size.
    • The rest of the R3 ecosystem is in Rust, which leads to binary savings thanks to code reuse.

Technical Questions

If you have technical questions, direct them to library & docs author @Sewer56 via one of these methods:

Happy Hacking ❤️