Tags

Changed

* [#220](https://github.com/firecracker-microvm/private-firecracker-staging/pull/220):
  Made MMDS unwrap a string of JSON and respond a JSON object for IMDS security
  credentials paths so that some clients that requesting with
  `Accept: application/json` is able to parse the response.

Added

* [#5048](https://github.com/firecracker-microvm/firecracker/pull/5048): Added
  support for [PVH boot mode](docs/pvh.md). This is used when an x86 kernel
  provides the appropriate ELF Note to indicate that PVH boot mode is supported.
  Linux kernels newer than 5.0 compiled with `CONFIG_PVH=y` set this ELF Note,
  as do FreeBSD kernels.
* [#5065](https://github.com/firecracker-microvm/firecracker/pull/5065) Added
  support for Intel AMX (Advanced Matrix Extensions). To be able to take and
  restore a snapshot of Intel AMX state, `Xsave` is used instead of `kvm_xsave`,
  so users need to regenerate snapshots.
* [#4731](https://github.com/firecracker-microvm/firecracker/pull/4731): Added
  support for modifying the host TAP device name during snapshot restore.
* [#5146](https://github.com/firecracker-microvm/firecracker/pull/5146): Added
  Intel Sapphire Rapids as a supported and tested platform for Firecracker.
* [#5148](https://github.com/firecracker-microvm/firecracker/pull/5148): Added
  ARM Graviton4 as a supported and tested platform for Firecracker.

Changed

* [#5118](https://github.com/firecracker-microvm/firecracker/pull/5118): Cleared
  WAITPKG CPUID bit in CPUID normalization. The feature enables a guest to put a
  physical processor into an idle state, which is undesirable in a FaaS
  environment since that is what the host wants to decide.
* [#5142](https://github.com/firecracker-microvm/firecracker/pull/5142):
  Clarified what CPU models are supported by each existing CPU template.
  Firecracker exits with an error if a CPU template is used on an unsupported
  CPU model.

Deprecated

* [#4948](https://github.com/firecracker-microvm/firecracker/pull/4948):
  Deprecated the `page_size_kib` field in the
  [UFFD handshake](docs/snapshotting/handling-page-faults-on-snapshot-resume.md#registering-memory-to-be-handled-via-userfault-file-descriptors),
  and replaced it with a `page_size` field. The `page_size_kib` field is
  misnamed, as the value Firecracker sets it to is actually the page size in
  _bytes_, not KiB. It will be removed in Firecracker 2.0.

Fixed

* [#5074](https://github.com/firecracker-microvm/firecracker/pull/5074) Fix the
  `SendCtrlAltDel` command not working for ACPI-enabled guest kernels, by
  dropping the i8042.nopnp argument from the default kernel command line
  Firecracker constructs.
* [#5122](https://github.com/firecracker-microvm/firecracker/pull/5122): Keep
  the UFFD Unix domain socket open to prevent the race condition between the
  guest memory mappings message and the shutdown event that was sometimes
  causing arrival of an empty message on the UFFD handler side.
* [#5143](https://github.com/firecracker-microvm/firecracker/pull/5143): Fixed
  to report `process_startup_time_us` and `process_startup_time_cpu_us` metrics
  for `api_server` right after the API server starts, while previously reported
  before applying seccomp filter and starting the API server. Users may observe
  a bit longer startup time metrics.

Started development of v1.13

Added

* [#4987](https://github.com/firecracker-microvm/firecracker/pull/4987): Reset
  physical counter register (`CNTPCT_EL0`) on VM startup. This avoids VM reading
  the host physical counter value. This is only possible on 6.4 and newer
  kernels. For older kernels physical counter will still be passed to the guest
  unmodified. See more info
  [here](https://github.com/firecracker-microvm/firecracker/blob/main/docs/prod-host-setup.md#arm-only-vm-physical-counter-behaviour)
* [#5088](https://github.com/firecracker-microvm/firecracker/pull/5088): Added
  AMD Genoa as a supported and tested platform for Firecracker.

Changed

* [#4913](https://github.com/firecracker-microvm/firecracker/pull/4913): Removed
  unnecessary fields (`max_connections` and `max_pending_resets`) from the
  snapshot format, bumping the snapshot version to 5.0.0. Users need to
  regenerate snapshots.
* [#4926](https://github.com/firecracker-microvm/firecracker/pull/4926): Replace
  underlying implementation for seccompiler from in house one in favor of
  `libseccomp` which produces smaller and more optimized BPF code.

Deprecated

Removed

Fixed

* [#4921](https://github.com/firecracker-microvm/firecracker/pull/4921): Fixed
  swagger `CpuConfig` definition to include missing aarch64-specific fields.
* [#4916](https://github.com/firecracker-microvm/firecracker/pull/4916): Fixed
  `IovDeque` implementation to work with any host page size. This fixes
  virtio-net device on non 4K host kernels.
* [#4991](https://github.com/firecracker-microvm/firecracker/pull/4991): Fixed
  `mem_size_mib` and `track_dirty_pages` being mandatory for all
  `PATCH /machine-config` requests. Now, they can be omitted which leaves these
  parts of the machine configuration unchanged.
* [#5007](https://github.com/firecracker-microvm/firecracker/pull/5007): Fixed
  watchdog softlockup warning on x86_64 guests when a vCPU is paused during GDB
  debugging.
* [#5021](https://github.com/firecracker-microvm/firecracker/pull/5021) If a
  balloon device is inflated post UFFD-backed snapshot restore, Firecracker now
  causes `remove` UFFD messages to be sent to the UFFD handler. Previously, no
  such message would be sent.
* [#5034](https://github.com/firecracker-microvm/firecracker/pull/5034): Fix an
  integer underflow in the jailer when computing the value it passes to
  Firecracker's `--parent-cpu-time-us` values, which caused development builds
  of Firecracker to crash (but production builds were unaffected as underflows
  do not panic in release mode).
* [#5045](https://github.com/firecracker-microvm/firecracker/pull/5045): Fixed
  an issue where firecracker intermittently receives SIGHUP when using jailer
  with `--new-pid-ns` but without `--daemonize`.
* [#4995](https://github.com/firecracker-microvm/firecracker/pull/4995):
  Firecracker no longer overwrites CPUID leaf 0x80000000 when running AMD
  hardware, meaning the guest can now discover a greater range of CPUID leaves
  in the extended function range (this range is host kernel dependent).
* [#5046](https://github.com/firecracker-microvm/firecracker/pull/5046): Retry
  KVM_CREATE_VM on EINTR that occasionally happen on heavily loaded hosts to
  improve reliability of microVM creation.
* [#5052](https://github.com/firecracker-microvm/firecracker/pull/5052): Build
  the empty seccomp policy as default for debug builds to avoid crashes on
  syscalls introduced by debug assertions from Rust 1.80.0.

Started development of v1.12

Changed

* [#4907](https://github.com/firecracker-microvm/firecracker/pull/4907): Bump
  snapshot version to 4.0.0.

Added

* [#4834](https://github.com/firecracker-microvm/firecracker/pull/4834): Add
  `VIRTIO_NET_F_RX_MRGBUF` support to the `virtio-net` device. When this feature
  is negotiated, guest `virtio-net` driver can perform more efficient memory
  management which in turn improves RX and TX performance.
* [#4460](https://github.com/firecracker-microvm/firecracker/pull/4460): Add a
  call to
  [`KVM_KVMCLOCK_CTRL`](https://docs.kernel.org/virt/kvm/api.html#kvm-kvmclock-ctrl)
  after pausing vCPUs on x86_64 architectures. This ioctl sets a flag in the KVM
  state of the vCPU indicating that it has been paused by the host userspace. In
  guests that use kvmclock, the soft lockup watchdog checks this flag. If it is
  set, it won't trigger the lockup condition. Calling the ioctl for guests that
  don't use kvmclock will fail. These failures are not fatal. We log the failure
  and increase the `vcpu.kvmclock_ctrl_fails` metric.
* [#4869](https://github.com/firecracker-microvm/firecracker/pull/4869): Added
  support for Aarch64 systems which feature CPU caches with a number of sets
  higher than `u16::MAX`.
* [#4797](https://github.com/firecracker-microvm/firecracker/pull/4797),
  [#4854](https://github.com/firecracker-microvm/firecracker/pull/4854): Added
  GDB debugging support for a microVM guest kernel. Please see our
  [GDB debugging documentation](docs/gdb-debugging.md) for more information.

Changed

* [#4844](https://github.com/firecracker-microvm/firecracker/pull/4844): Upgrade
  `virtio-net` device to use `readv` syscall to avoid unnecessary memory copies
  on RX path, increasing the RX performance.

Deprecated

Removed

* [#4804](https://github.com/firecracker-microvm/firecracker/pull/4804): Drop
  Support for guest kernel 4.14. Linux 4.14 reached end-of-life in
  [January 2024](https://lore.kernel.org/lkml/2024011046-ecology-tiptoeing-ce50@gregkh/)
  The minimum supported guest kernel now is 5.10.

Fixed

* [#4796](https://github.com/firecracker-microvm/firecracker/pull/4796): Fixed
  Vsock not notifying guest about `TRANSPORT_RESET_EVENT` event after snapshot
  restore. This resulted in guest waiting indefinitely on a connection which was
  reset during snapshot creation.
* [#4790](https://github.com/firecracker-microvm/firecracker/pull/4790): v1.9.0
  was missing most of the debugging information in the debuginfo file, due to a
  change in the Cargo defaults. This has been corrected.
* [#4826](https://github.com/firecracker-microvm/firecracker/pull/4826): Add
  missing configuration of tap offload features when restoring from a snapshot.
  Setting the features was previously
  [moved](https://github.com/firecracker-microvm/firecracker/pull/4680/commits/49ed5ea4b48ccd98903da037368fa3108f58ac1f)
  from net device creation to device activation time, but it was not reflected
  in the restore path. This was leading to inability to connect to the restored
  VM if the offload features were used.

Started development of v1.11.0

Fixed

* [#4824](https://github.com/firecracker-microvm/firecracker/pull/4824): Add
  missing configuration of tap offload features when restoring from a snapshot.
  Setting the features was previously
  [moved](https://github.com/firecracker-microvm/firecracker/pull/4680/commits/49ed5ea4b48ccd98903da037368fa3108f58ac1f)
  from net device creation to device activation time, but it was not reflected
  in the restore path. This was leading to inability to connect to the restored
  VM if the offload features were used.
* [#4829](https://github.com/firecracker-microvm/firecracker/pull/4829): v1.9.0
  was missing most of the debugging information in the debuginfo file, due to a
  change in the Cargo defaults. This has been corrected.

Added

* [#4687](https://github.com/firecracker-microvm/firecracker/pull/4687): Added
  VMGenID support for microVMs running on ARM hosts with 6.1 guest kernels.
  Support for VMGenID via DeviceTree bindings exists only on mainline 6.10 Linux
  onwards. Users of Firecracker will need to backport the relevant patches on
  top of their 6.1 kernels to make use of the feature.
* [#4732](https://github.com/firecracker-microvm/firecracker/pull/4732),
  [#4733](https://github.com/firecracker-microvm/firecracker/pull/4733),
  [#4741](https://github.com/firecracker-microvm/firecracker/pull/4741),
  [#4746](https://github.com/firecracker-microvm/firecracker/pull/4746): Added
  official support for 6.1 microVM guest kernels.

Changed

Deprecated

* Support for guest kernel 4.14 is now deprecated. We will completely remove
  4.14 support with Firecracker version v1.10

Removed

* [#4689](https://github.com/firecracker-microvm/firecracker/pull/4689): Drop
  support for host kernel 4.14. Linux 4.14 reached end-of-life in
  [January 2024](https://lore.kernel.org/lkml/2024011046-ecology-tiptoeing-ce50@gregkh/).
  The minimum supported kernel now is 5.10. Guest kernel 4.14 is still
  supported.

Fixed

* [4680](https://github.com/firecracker-microvm/firecracker/pull/4680): Fixed an
  issue
  ([#4659](https://github.com/firecracker-microvm/firecracker/issues/4659))
  where the virtio-net device implementation would always assume the guest
  accepts all VirtIO features the device offers. This is always true with the
  Linux guest kernels we are testing but other kernels, like FreeBSD make
  different assumptions. This PR fixes the emulation code to set the TAP
  features based on the features accepted by the guest.

Started development of v1.10

Added

* [#4428](https://github.com/firecracker-microvm/firecracker/pull/4428): Added
  ACPI support to Firecracker for x86_64 microVMs. Currently, we pass ACPI
  tables with information about the available vCPUs, interrupt controllers,
  VirtIO and legacy x86 devices to the guest. This allows booting kernels
  without MPTable support. Please see our
  [kernel policy documentation](docs/kernel-policy.md) for more information
  regarding relevant kernel configurations.
* [#4487](https://github.com/firecracker-microvm/firecracker/pull/4487): Added
  support for the Virtual Machine Generation Identifier (VMGenID) device on
  x86_64 platforms. VMGenID is a virtual device that allows VMMs to notify
  guests when they are resumed from a snapshot. Linux includes VMGenID support
  since version 5.18. It uses notifications from the device to reseed its
  internal CSPRNG. Please refer to
  [snapshot support](docs/snapshotting/snapshot-support.md) and
  [random for clones](docs/snapshotting/random-for-clones.md) documention for
  more info on VMGenID. VMGenID state is part of the snapshot format of
  Firecracker. As a result, Firecracker snapshot version is now 2.0.0.

Changed

* [#4492](https://github.com/firecracker-microvm/firecracker/pull/4492): Changed
  `--config` parameter of `cpu-template-helper` optional. Users no longer need
  to prepare kernel, rootfs and Firecracker configuration files to use
  `cpu-template-helper`.
* [#4537](https://github.com/firecracker-microvm/firecracker/pull/4537) Changed
  T2CL template to pass through bit 27 and 28 of `MSR_IA32_ARCH_CAPABILITIES`
  (`RFDS_NO` and `RFDS_CLEAR`) since KVM consider they are able to be passed
  through and T2CL isn't designed for secure snapshot migration between
  different processors.
* [#4537](https://github.com/firecracker-microvm/firecracker/pull/4537) Changed
  T2S template to set bit 27 of `MSR_IA32_ARCH_CAPABILITIES` (`RFDS_NO`) to 1
  since it assumes that the fleet only consists of processors that are not
  affected by RFDS.
* [#4388](https://github.com/firecracker-microvm/firecracker/pull/4388): Avoid
  setting `kvm_immediate_exit` to 1 if are already handling an exit, or if the
  vCPU is stopped. This avoids a spurious KVM exit upon restoring snapshots.
* [#4567](https://github.com/firecracker-microvm/firecracker/pull/4567): Do not
  initialize vCPUs in powered-off state upon snapshot restore. No functional
  change, as vCPU initialization is only relevant for the booted case (where the
  guest expects CPUs to be powered off).

Deprecated

* Firecracker's `--start-time-cpu-us` and `--start-time-us` parameters are
  deprecated and will be removed in v2.0 or later. They are used by the jailer
  to pass the value that should be subtracted from the (CPU) time, when emitting
  the `start_time_us` and `start_time_cpu_us` metrics. These parameters were
  never meant to be used by end customers, and we recommend doing any such time
  adjustments outside Firecracker.
* Booting with microVM kernels that rely on MPTable on x86_64 is deprecated and
  support will be removed in v2.0 or later. We suggest to users of Firecracker
  to use guest kernels with ACPI support. For x86_64 microVMs, ACPI will be the
  only way Firecracker passes hardware information to the guest once MPTable
  support is removed.

Fixed

* [#4526](https://github.com/firecracker-microvm/firecracker/pull/4526): Added a
  check in the network TX path that the size of the network frames the guest
  passes to us is not bigger than the maximum frame the device expects to
  handle. On the TX path, we copy frames destined to MMDS from guest memory to
  Firecracker memory. Without the check, a mis-behaving virtio-net driver could
  cause an increase in the memory footprint of the Firecracker process. Now, if
  we receive such a frame, we ignore it and increase `Net::tx_malformed_frames`
  metric.
* [#4536](https://github.com/firecracker-microvm/firecracker/pull/4536): Make
  the first differential snapshot taken after a full snapshot contain only the
  set of memory pages changed since the full snapshot. Previously, these
  differential snapshots would contain all memory pages. This will result in
  potentially much smaller differential snapshots after a full snapshot.
* [#4578](https://github.com/firecracker-microvm/firecracker/pull/4578): Fix
  UFFD support not being forward-compatible with new ioctl options introduced in
  Linux 6.6. See also
  https://github.com/bytecodealliance/userfaultfd-rs/issues/61.
* [#4630](https://github.com/firecracker-microvm/firecracker/pull/4630): On
  x86_64, when taking a snapshot, if a vCPU has MSR_IA32_TSC_DEADLINE set to 0,
  Firecracker will replace it with the MSR_IA32_TSC value from the same vCPU.
  This is to guarantee that the vCPU will continue receiving TSC interrupts
  after restoring from the snapshot even if an interrupt is lost when taking a
  snapshot.
* [#4666](https://github.com/firecracker-microvm/firecracker/pull/4666): Fixed
  Firecracker sometimes restoring `MSR_IA32_TSC_DEADLINE` before `MSR_IA32_TSC`.
  Now it always restores `MSR_IA32_TSC_DEADLINE` MSR after `MSR_IA32_TSC`, as
  KVM relies on the guest TSC for correct restoration of
  `MSR_IA32_TSC_DEADLINE`. This fixed guests using the `TSC_DEADLINE` hardware
  feature receiving incorrect timer interrupts after snapshot restoration, which
  could lead to them seemingly getting stuck in sleep-related syscalls (see also
  https://github.com/firecracker-microvm/firecracker/pull/4099).

Started development of v1.9

Added

* [#4346](https://github.com/firecracker-microvm/firecracker/pull/4346): Added
  support to emit aggregate (minimum/maximum/sum) latency for
  `VcpuExit::MmioRead`, `VcpuExit::MmioWrite`, `VcpuExit::IoIn` and
  `VcpuExit::IoOut`. The average for these VM exits is not emitted since it can
  be deduced from the available emitted metrics.
* [#4360](https://github.com/firecracker-microvm/firecracker/pull/4360): Added
  dev-preview support for backing a VM's guest memory by 2M hugetlbfs pages.
  Please see the [documentation](docs/hugepages.md) for more information
* [#4490](https://github.com/firecracker-microvm/firecracker/pull/4490): Added
  block and net device metrics for file/tap access latencies and queue backlog
  lengths, which can be used to analyse saturation of the Firecracker VMM thread
  and underlying layers. Queue backlog length metrics are flushed periodically.
  They can be used to esimtate an average queue length by request by dividing
  its value by the number of requests served.

Changed

* [#4230](https://github.com/firecracker-microvm/firecracker/pull/4230): Changed
  microVM snapshot format version strategy. Firecracker snapshot format now has
  a version that is independent of Firecracker version. The current version of
  the snapshot format is v1.0.0. From now on, the Firecracker binary will define
  the snapshot format version it supports and it will only be able to load
  snapshots with format that is backwards compatible with that version. Users
  can pass the `--snapshot-version` flag to the Firecracker binary to see its
  supported snapshot version format. This change renders all previous
  Firecracker snapshots (up to Firecracker version v1.6.0) incompatible with the
  current Firecracker version.
* [#4449](https://github.com/firecracker-microvm/firecracker/pull/4449): Added
  information about page size to the payload Firecracker sends to the UFFD
  handler. Each memory region object now contains a `page_size_kib` field. See
  also the [hugepages documentation](docs/hugepages.md).
* [#4501](https://github.com/firecracker-microvm/firecracker/pull/4501): Only
  use memfd to back guest memory if a vhost-user-blk device is configured,
  otherwise use anonymous private memory. This is because serving page faults of
  shared memory used by memfd is slower and may impact workloads.

Fixed

* [#4409](https://github.com/firecracker-microvm/firecracker/pull/4409): Fixed a
  bug in the cpu-template-helper that made it panic during conversion of cpu
  configuration with SVE registers to the cpu template on aarch64 platform. Now
  cpu-template-helper will print warnings if it encounters SVE registers during
  the conversion process. This is because cpu templates are limited to only
  modify registers less than 128 bits.
* [#4413](https://github.com/firecracker-microvm/firecracker/pull/4413): Fixed a
  bug in the Firecracker that prevented it to restore snapshots of VMs that had
  SVE enabled.
* [#4414](https://github.com/firecracker-microvm/firecracker/pull/4360): Made
  `PATCH` requests to the `/machine-config` endpoint transactional, meaning
  Firecracker's configuration will be unchanged if the request returns an error.
  This fixes a bug where a microVM with incompatible balloon and guest memory
  size could be booted, due to the check for this condition happening after
  Firecracker's configuration was updated.
* [#4259](https://github.com/firecracker-microvm/firecracker/pull/4259): Added a
  double fork mechanism in the Jailer to avoid setsid() failures occurred while
  running Jailer as the process group leader. However, this changed the
  behaviour of Jailer and now the Firecracker process will always have a
  different PID than the Jailer process.
  [#4436](https://github.com/firecracker-microvm/firecracker/pull/4436): Added a
  "Known Limitations" section in the Jailer docs to highlight the above change
  in behaviour introduced in PR#4259.
  [#4442](https://github.com/firecracker-microvm/firecracker/pull/4442): As a
  solution to the change in behaviour introduced in PR#4259, provided a
  mechanism to reliably fetch Firecracker PID. With this change, Firecracker
  process's PID will always be available in the Jailer's root directory
  regardless of whether new_pid_ns was set.
* [#4468](https://github.com/firecracker-microvm/firecracker/pull/4468): Fixed a
  bug where a client would hang or timeout when querying for an MMDS path whose
  content is empty, because the 'Content-Length' header field was missing in a
  response.

Started development of v1.8

Added

* [#4145](https://github.com/firecracker-microvm/firecracker/pull/4145):
  Added support for per net device metrics. In addition to aggregate metrics `net`,
  each individual net device will emit metrics under the label `"net_{iface_id}"`.
  E.g. the associated metrics for the endpoint `"/network-interfaces/eth0"` will
  be available under `"net_eth0"` in the metrics json object.
* [#4202](https://github.com/firecracker-microvm/firecracker/pull/4202):
  Added support for per block device metrics. In addition to aggregate metrics `block`,
  each individual block device will emit metrics under the label `"block_{drive_id}"`.
  E.g. the associated metrics for the endpoint `"/drives/{drive_id}"` will
  be available under `"block_drive_id"` in the metrics json object.
* [#4205](https://github.com/firecracker-microvm/firecracker/pull/4205):
  Added a new `vm-state` subcommand to `info-vmstate` command in the
  `snapshot-editor` tool to print MicrovmState of vmstate snapshot file in
  a readable format. Also made the `vcpu-states` subcommand available on
  x86_64.
* [#4063](https://github.com/firecracker-microvm/firecracker/pull/4063):
  Added source-level instrumentation based tracing. See
  [tracing](./docs/tracing.md) for more details.
* [#4138](https://github.com/firecracker-microvm/firecracker/pull/4138),
  [#4170](https://github.com/firecracker-microvm/firecracker/pull/4170),
  [#4223](https://github.com/firecracker-microvm/firecracker/pull/4223),
  [#4247](https://github.com/firecracker-microvm/firecracker/pull/4247),
  [#4226](https://github.com/firecracker-microvm/firecracker/pull/4226):
  Added **developer preview only** (NOT for production use) support for
  vhost-user block devices.
  Firecracker implements a vhost-user frontend. Users are free to choose
  from existing open source backend solutions or their own implementation.
  Known limitation: snapshotting is not currently supported for microVMs
  containing vhost-user block devices.
  See the [related doc page](./docs/api_requests/block-vhost-user.md) for details.
  The device emits metrics under the label `"vhost_user_{device}_{drive_id}"`.

Changed

* [#4309](https://github.com/firecracker-microvm/firecracker/pull/4309): The
  jailer's option `--parent-cgroup` will move the process to that cgroup if no
  `cgroup` options are provided.
* Simplified and clarified the removal policy of deprecated API elements
  to follow semantic versioning 2.0.0. For more information, please refer to
  [this GitHub discussion](https://github.com/firecracker-microvm/firecracker/discussions/4135).
* [#4180](https://github.com/firecracker-microvm/firecracker/pull/4180):
  Refactored error propagation to avoid logging and printing an error on
  exits with a zero exit code. Now, on successful exit
  "Firecracker exited successfully" is logged.
* [#4194](https://github.com/firecracker-microvm/firecracker/pull/4194):
  Removed support for creating Firecracker snapshots targeting older versions
  of Firecracker. With this change, running 'firecracker --version' will not
  print the supported snapshot versions.
* [#4301](https://github.com/firecracker-microvm/firecracker/pull/4301):
  Allow merging of diff snapshots into base snapshots by directly writing
  the diff snapshot on top of the base snapshot's memory file. This can be
  done by setting the `mem_file_path` to the path of the pre-existing full
  snapshot.

Deprecated

* [#4209](https://github.com/firecracker-microvm/firecracker/pull/4209):
  `rebase-snap` tool is now deprecated. Users should use `snapshot-editor`
  for rebasing diff snapshots.

Fixed

* [#4171](https://github.com/firecracker-microvm/firecracker/pull/4171):
  Fixed a bug that ignored the `--show-log-origin` option, preventing it from
  printing the source code file of the log messages.
* [#4178](https://github.com/firecracker-microvm/firecracker/pull/4178):
  Fixed a bug reporting a non-zero exit code on successful shutdown when
  starting Firecracker with `--no-api`.
* [#4261](https://github.com/firecracker-microvm/firecracker/pull/4261): Fixed
  a bug where Firecracker would log "RunWithApiError error: MicroVMStopped
  without an error: GenericError" when exiting after encountering an emulation
  error. It now correctly prints "RunWithApiError error: MicroVMStopped *with* an
  error: GenericError".
* [#4242](https://github.com/firecracker-microvm/firecracker/pull/4242):
  Fixed a bug introduced in #4047 that limited the `--level` option of logger
  to Pascal-cased values (e.g. accepting "Info", but not "info"). It now
  ignores case again.
* [#4286](https://github.com/firecracker-microvm/firecracker/pull/4286):
  Fixed a bug in the asynchronous virtio-block engine that rendered the device
  non-functional after a PATCH request was issued to Firecracker for updating
  the path to the host-side backing file of the device.
* [#4301](https://github.com/firecracker-microvm/firecracker/pull/4301):
  Fixed a bug where if Firecracker was instructed to take a snapshot of a
  microvm which itself was restored from a snapshot, specifying `mem_file_path`
  to be the path of the memory file from which the microvm was restored would
  result in both the microvm and the snapshot being corrupted. It now instead
  performs a "write-back" of all memory that was updated since the snapshot
  was originally loaded.

Started development of v1.7

Added

* [#4287](https://github.com/firecracker-microvm/firecracker/issues/4287)
  Document a caveat to the jailer docs when using the `--parent-cgroup` option,
  which results in it being ignored by the jailer. Refer to the [jailer
  documentation](./docs/jailer.md#caveats) for a workaround.

Changed

* [#4191](https://github.com/firecracker-microvm/firecracker/pull/4191):
  Refactored error propagation to avoid logging and printing an error on
  exits with a zero exit code. Now, on successful exit
  "Firecracker exited successfully" is logged.

Fixed

* [#4277](https://github.com/firecracker-microvm/firecracker/pull/4277): Fixed a
  bug that ignored the `--show-log-origin` option, preventing it from printing
  the source code file of the log messages.
* [#4179](https://github.com/firecracker-microvm/firecracker/pull/4179):
  Fixed a bug reporting a non-zero exit code on successful shutdown when
  starting Firecracker with `--no-api`.
* [#4271](https://github.com/firecracker-microvm/firecracker/pull/4271): Fixed
  a bug where Firecracker would log "RunWithApiError error: MicroVMStopped
  without an error: GenericError" when exiting after encountering an emulation
  error. It now correctly prints "RunWithApiError error: MicroVMStopped *with* an
  error: GenericError".
* [#4270](https://github.com/firecracker-microvm/firecracker/pull/4270):
  Fixed a bug introduced in #4047 that limited the `--level` option of logger
  to Pascal-cased values (e.g. accepting "Info", but not "info"). It now
  ignores case again.
* [#4295](https://github.com/firecracker-microvm/firecracker/pull/4295):
  Fixed a bug in the asynchronous virtio-block engine that rendered the device
  non-functional after a PATCH request was issued to Firecracker for updating
  the path to the host-side backing file of the device.

Started development of v1.6