273 lines
8.4 KiB
Markdown
273 lines
8.4 KiB
Markdown
# `nix-serve-ng`
|
||
|
||
`nix-serve-ng` is a faster, more reliable, drop-in replacement for `nix-serve`.
|
||
|
||
## Quick start
|
||
|
||
There are three main approaches you can use to configure a NixOS system to replace
|
||
the old `nix-serve` with `nix-serve-ng`:
|
||
|
||
- **A**: Set `services.nix-serve.package = pkgs.nix-serve-ng;` in your NixOS configuration
|
||
- `nix-serve-ng` is [packaged in nixpkgs](https://search.nixos.org/packages) already
|
||
- There is no need to consume this repository directly
|
||
- **B**: Include `nix-serve-ng.nixosModules.default` in your NixOS configuration
|
||
- `nix-serve-ng` refers to this repository being a flake input
|
||
- Requires consume this repository / this flake
|
||
- Overlays `pkgs.nix-serve` with `pkgs.nix-serve-ng`
|
||
- **C**: Like **B** but not requiring a flake
|
||
|
||
We recommend approach **A**. Only use **B** or **C** if you need a bleeding edge
|
||
upstream version of the project.
|
||
|
||
### Variant A:
|
||
|
||
_The code snippet below shows a `flake.nix`._
|
||
|
||
```nix
|
||
{
|
||
inputs.nixpkgs.url = "github:NixOS/nixpkgs";
|
||
|
||
outputs = { nixpkgs, ... }: {
|
||
nixosConfigurations.default = nixpkgs.lib.nixosSystem {
|
||
modules = [
|
||
/* ... */
|
||
{
|
||
services.nix-serve.enable = true;
|
||
services.nix-serve.package = pkgs.nix-serve-ng;
|
||
/* ... */
|
||
}
|
||
/* ... */
|
||
];
|
||
};
|
||
};
|
||
}
|
||
```
|
||
|
||
### Variant B:
|
||
|
||
_The code snippet below shows a `flake.nix`._
|
||
|
||
```nix
|
||
{
|
||
inputs.nixpkgs.url = "github:NixOS/nixpkgs";
|
||
inputs.nix-serve-ng.url = "github:aristanetworks/nix-serve-ng";
|
||
|
||
outputs = { nixpkgs, nix-serve-ng, ... }: {
|
||
nixosConfigurations.default = nixpkgs.lib.nixosSystem {
|
||
modules = [
|
||
nix-serve-ng.nixosModules.default
|
||
/* ... */
|
||
{
|
||
services.nix-serve.enable = true;
|
||
/* ... */
|
||
}
|
||
/* ... */
|
||
];
|
||
};
|
||
};
|
||
}
|
||
```
|
||
|
||
### Variant C:
|
||
|
||
_The code snippet below shows a NixOS module file._
|
||
|
||
```nix
|
||
{ config, pkgs, lib, ... }:
|
||
|
||
let
|
||
nix-serve-ng-src = builtins.fetchTarball {
|
||
# Replace the URL and hash with whatever you actually need
|
||
url = "https://github.com/aristanetworks/nix-serve-ng/archive/1937593598bb1285b41804f25cd6f9ddd4d5f1cb.tar.gz";
|
||
|
||
sha256 = "1lqd207gbx1wjbhky33d2r8xi6avfbx4v0kpsvn84zaanifdgz2g";
|
||
};
|
||
|
||
nix-serve-ng = import nix-serve-ng-src;
|
||
in
|
||
{
|
||
/* ... */
|
||
imports = [ nix-serve-ng.nixosModules.default ];
|
||
|
||
config = {
|
||
services.nix-serve.enable = true;
|
||
};
|
||
/* ... */
|
||
}
|
||
```
|
||
|
||
## Lix compatability
|
||
|
||
The default `nix-serve-ng` should work on top of lix, but if you want to build
|
||
it against lix for development or to remove the default nix dependency, you can
|
||
pass `-f lix` to cabal or use the `lix-serve-ng` package from the flake.
|
||
|
||
## Motivation
|
||
|
||
Our requirements for this project were:
|
||
|
||
* Improve reliability
|
||
|
||
… since `nix-serve` would intermittently hang and require restarts
|
||
|
||
* Improve efficiency
|
||
|
||
… since `nix-serve` was doing some obviously inefficient things which we
|
||
felt we could improve upon
|
||
|
||
* Be backwards-compatible
|
||
|
||
Our replacement would need to be a drop-in replacement for the original
|
||
`nix-serve`, supporting the same command-line options and even sharing the
|
||
same executable name
|
||
|
||
The only exception is logging: we provide more detailed logging than before
|
||
|
||
Did we satisfy those requirements?
|
||
|
||
## Results
|
||
|
||
* Reliability
|
||
|
||
We have test-driven this internally under heavy load with stable memory
|
||
usage and without any failures but it's probably premature to declare victory.
|
||
|
||
In particular, we have *not* done the following things:
|
||
|
||
* Memory leak detection
|
||
|
||
In other words, we haven't put our `nix-serve` through, say, `valgrind`
|
||
|
||
* Exploit detection
|
||
|
||
In other words, we haven't attempted to crash or sabotage the service with
|
||
maliciously-crafted payload
|
||
|
||
* Performance
|
||
|
||
We have improved significantly on efficiency, not only compared to `nix-serve`
|
||
but also compared to other `nix-serve` rewrites. We are more efficient than:
|
||
|
||
* The original `nix-serve`
|
||
|
||
* [`eris`](https://github.com/thoughtpolice/eris) - A Perl rewrite of
|
||
`nix-serve`
|
||
|
||
* [`harmonia`](https://github.com/nix-community/harmonia) - A Rust rewrite
|
||
of `nix-serve`
|
||
|
||
See the Benchmarks section below for more details
|
||
|
||
* Backwards-compatibility
|
||
|
||
We have excellent backwards-compatibility, so in the vast majority of cases,
|
||
you can simply replace `pkgs.nix-serve` with `pkgs.nix-serve-ng` and make no
|
||
other changes.
|
||
|
||
* Our executable shares the same name (`nix-serve`) as the original program
|
||
|
||
* We support most the original command-line options
|
||
|
||
The options that we're aware of that we do not currently support fall into
|
||
two categories:
|
||
|
||
* Useless options which are only relevant to `starman`:
|
||
|
||
Upon request, we can still parse and ignore the following irrelevant
|
||
options for extra backwards compatibility:
|
||
|
||
* `--workers`
|
||
|
||
We do not use worker subprocess like `starman` does. Instead we use
|
||
`warp` which internally uses Haskell green threads to service a much
|
||
larger number of requests with less overhead and lower footprint when
|
||
idle.
|
||
|
||
* `--preload-app`
|
||
|
||
This optimization is meaningless for a compiled Haskell executable.
|
||
|
||
* `--disable-proctitle`
|
||
|
||
* Useful options
|
||
|
||
We might accept requests to support the following options, but we might
|
||
explore other alternatives first before supporting them:
|
||
|
||
* `--max-requests`
|
||
|
||
`warp` itself is unlikely to be a bottleneck to servicing a large number
|
||
of requests but there may still be Nix-specific or disk-specific
|
||
reasons to cap the number of requests.
|
||
|
||
* `--disable-keepalive`
|
||
|
||
* `--keepalive-timeout`
|
||
|
||
* `--read-timeout`
|
||
|
||
* `--user`
|
||
|
||
* `--group`
|
||
|
||
* `--pid`
|
||
|
||
* `--error-log`
|
||
|
||
Because of this backwards-compatibility you only need to replace the old
|
||
`nix-serve` executable with the `nix-serve` executable built by this package
|
||
(which is what the included NixOS module does).
|
||
|
||
You don't need to define or use any new NixOS options. You continue to use
|
||
the old `services.nix-serve` options hierarchy to configure the upgraded
|
||
service.
|
||
|
||
## Benchmarks
|
||
|
||
The test environment is a large server machine:
|
||
|
||
* CPU: 24 × Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|
||
* RAM: 384 GB (24 × 16 GB @ 2133 MT/s)
|
||
* Disk (`/nix/store`): ≈4 TB SSD
|
||
|
||
Legend:
|
||
|
||
* **Fetch present NAR info ×10**: Time to fetch the NAR info for 10 different files that are present
|
||
* **Fetch absent NAR info ×1**: Time to fetch the NAR info a single file that is absent
|
||
* **Fetch empty NAR ×10**: Time to fetch the NAR for the same empty file 10 times
|
||
* **Fetch 10 MB NAR ×10**: Time to fetch the NAR for the same 10 MB file 10 times
|
||
|
||
Raw numbers:
|
||
|
||
| Benchmark | `nix-serve` | `eris` | `harmonia` | `nix-serve-ng` |
|
||
|----------------------------|------------------|------------------|------------------|------------------|
|
||
| Fetch present NAR info ×10 | 2.09 ms ± 66 μs | 41.5 ms ± 426 μs | 1.57 ms ± 91 μs | 1.32 ms ± 33 μs |
|
||
| Fetch absent NAR info ×1 | 212 μs ± 18 μs | 3.42 ms ± 113 μs | 139 μs ± 11 μs | 115 μs ± 6.2 μs |
|
||
| Fetch empty NAR ×10 | 164 ms ± 8.5 ms | 246 ms ± 20 ms | 279 ms ± 10 ms | 5.16 ms ± 368 μs |
|
||
| Fetch 10 MB NAR ×10 | 291 ms ± 8.7 ms | 453 ms ± 19 ms | 487 ms ± 41 ms | 86.9 ms ± 3.0 ms |
|
||
|
||
Speedups (compared to `nix-serve`):
|
||
|
||
| Benchmark | `nix-serve` | `eris` | `harmonia` | `nix-serve-ng` |
|
||
|----------------------------|------------------|------------------|------------------|------------------|
|
||
| Fetch present NAR info ×10 | 1.0 | 0.05 | 1.33 | 1.58 |
|
||
| Fetch absent NAR info ×1 | 1.0 | 0.06 | 1.53 | 1.84 |
|
||
| Fetch empty NAR ×10 | 1.0 | 0.67 | 0.59 | 31.80 |
|
||
| Fetch 10 MB NAR ×10 | 1.0 | 0.64 | 0.60 | 3.35 |
|
||
|
||
We can summarize `nix-serve-ng`'s performance like this:
|
||
|
||
* Time to handle a NAR info request: ≈ 100 μs
|
||
* Time to serve a NAR: ≈ 500 μs + 800 μs / MB
|
||
|
||
You can reproduce these benchmarks using the benchmark suite. See the
|
||
instructions in [`./benchmark/Main.hs`](./benchmark/Main.hs) for running your
|
||
own benchmarks.
|
||
|
||
Caveats:
|
||
|
||
* We haven't used any of these services' tuning options, including:
|
||
* Tuning garbage collection (for `nix-serve-ng`)
|
||
* Tuning concurrency/parallelism/workers
|
||
* We haven't benchmarked memory utilization
|