Private Subnet IpsecTunnel rekeying

With this commit, we introduce the mechanism to replace the encryption
keys for existing ipsec tunnels. Here is the detailed explanation of the
work;
1. Private Subnet nexus hops to the refresh_keys state when needed. Here
we are going through a preparation phase. We start producing new
encryption keys, SPIs and reqids for the new state and policy objects.
At the end of the day, we write the new encryption keys, SPIs and the
reqids to the nic entities.  So, for every nic in the system, we prepare
the new parameters at the private subnet level.
2. Trigger NicNexus to start rekeying.
3. NicNexus buds RekeyNicTunnel to create the SA objects for inbound
packets.  This part is important. Let me explain the current state of
the tunnelling in detail here. Please refer to the end of the commit
message.
3. The RekeyNic prog in  setup_inbound state, first finds all the state
objects required to be created at the destination end. This way, no
matter when the source policy/state is updated, the receiving end will
be ready with the encryption key.
4. Once everyone creates their receiving end state objects, we switch to
update the sender (aka outbound). This ping-pong is managed via
SubnetNexus by checking the state of NicNexus strands and using
semaphores to progress the state machine. This part of the code means
any packet that is being created at this moment, will be encrypted with
the new keys.
5. Once everyone in the mesh has created/updated their sender policies
and state objects. We go ahead and drop the state objects that are not
referenced by any policy.

Ipsec Tunneling
Each tunnel has two ends <source_nic> and <destination_nic>. For a
tunnel to work, both ends need to know about the encryption key so that
they can either encrypt or decrypt the packet. However, the encryption
key is not sent over with the packet. Therefore, the key must exist
locally in both systems. How does it work? Firstly, I should tell you
about the 2 objects we use to implement ipsec tunnels.
1. Policy:
Policies are used to capture packages and apply encryption. The way it
works is easy. All the packets are observed and if there is a packet
that matches one of the policies we created, we capture the reqid from
the policy. The reqid is the unique identifier of a state object that
has the encryption key information. Therefore, if a policy is matched,
we get the reqid from the policy, find the matching state object and
encrypt the packet using the encryption key, send it to the networking
stack again.
2. State:
States are there to store the encryption key information. They also have
fields like spi and reqid. reqid is mentioned in the policy part, so,
for outgoing connections, reqid is used to identify the encryption key.
For incoming connections, that is spi because spi is sent with the
packet in the header so that when receiving end gets the packet, finds a
matching policy, the spi is read from the header and the encryption key
is identified. This gives us the flexibility of using multiple state
objects for 1 tunnel. Also, handy for the rekeying process.

migrate/20230811_add_nic_rekey_payload.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    alter_table(:nic) do
+      add_column :rekey_payload, :jsonb
+    end
+  end
+end

model/nic.rb

@@ -10,7 +10,7 @@ class Nic < Sequel::Model
   one_to_one :strand, key: :id, class: Strand
   include ResourceMethods
   include SemaphoreMethods
-  semaphore :destroy, :refresh_mesh, :detach_vm
+  semaphore :destroy, :refresh_mesh, :detach_vm, :start_rekey, :trigger_outbound_update, :old_state_drop_trigger
 
   plugin :column_encryption do |enc|
     enc.column :encryption_key

model/private_subnet.rb

@@ -36,7 +36,7 @@ class PrivateSubnet < Sequel::Model
   end
 
   include SemaphoreMethods
-  semaphore :destroy, :refresh_mesh
+  semaphore :destroy, :refresh_mesh, :refresh_keys
 
   def self.random_subnet
     PRIVATE_SUBNET_RANGES.sample

prog/vnet/nic_nexus.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 class Prog::Vnet::NicNexus < Prog::Base
-  semaphore :destroy, :refresh_mesh, :detach_vm
+  semaphore :destroy, :refresh_mesh, :detach_vm, :start_rekey, :trigger_outbound_update, :old_state_drop_trigger
 
   def self.assemble(private_subnet_id, name: nil, ipv6_addr: nil, ipv4_addr: nil)
     unless (subnet = PrivateSubnet[private_subnet_id])
@@ -41,9 +41,61 @@ class Prog::Vnet::NicNexus < Prog::Base
       hop :detach_vm
     end
 
+    when_start_rekey_set? do
+      hop :start_rekey
+    end
+
     nap 30
   end
 
+  def start_rekey
+    bud Prog::Vnet::RekeyNicTunnel, {}, :setup_inbound
+    hop :wait_rekey_inbound
+  end
+
+  def wait_rekey_inbound
+    reap
+    if leaf?
+      decr_start_rekey
+      hop :wait_rekey_outbound_trigger
+    end
+    donate
+  end
+
+  def wait_rekey_outbound_trigger
+    when_trigger_outbound_update_set? do
+      bud Prog::Vnet::RekeyNicTunnel, {}, :setup_outbound
+      hop :wait_rekey_outbound
+    end
+    donate
+  end
+
+  def wait_rekey_outbound
+    reap
+    if leaf?
+      decr_trigger_outbound_update
+      hop :wait_rekey_old_state_drop_trigger
+    end
+    donate
+  end
+
+  def wait_rekey_old_state_drop_trigger
+    when_old_state_drop_trigger_set? do
+      bud Prog::Vnet::RekeyNicTunnel, {}, :drop_old_state
+      hop :wait_rekey_old_state_drop
+    end
+    donate
+  end
+
+  def wait_rekey_old_state_drop
+    reap
+    if leaf?
+      decr_old_state_drop_trigger
+      hop :wait
+    end
+    donate
+  end
+
   def refresh_mesh
     if nic.vm_id.nil?
       decr_refresh_mesh

prog/vnet/rekey_nic_tunnel.rb

@@ -0,0 +1,91 @@
+# frozen_string_literal: true
+
+class Prog::Vnet::RekeyNicTunnel < Prog::Base
+  subject_is :nic
+
+  def setup_inbound
+    nic.dst_ipsec_tunnels.each do |tunnel|
+      args = tunnel.src_nic.rekey_payload
+      create_state(tunnel, args)
+    end
+
+    pop "inbound_setup is complete"
+  end
+
+  def setup_outbound
+    nic.src_ipsec_tunnels.each do |tunnel|
+      args = tunnel.src_nic.rekey_payload
+      create_state(tunnel, args)
+      policy_update(tunnel, "out")
+    end
+
+    pop "outbound_setup is complete"
+  end
+
+  def drop_old_state
+    new_spis = [nic.rekey_payload["spi4"], nic.rekey_payload["spi6"]]
+    new_spis += nic.dst_ipsec_tunnels.map do |tunnel|
+      [tunnel.src_nic.rekey_payload["spi4"], tunnel.src_nic.rekey_payload["spi6"]]
+    end.flatten
+
+    state_data = sshable_cmd("sudo ip -n #{nic.src_ipsec_tunnels.first.vm_name(nic)} xfrm state")
+
+    # Extract SPIs along with src and dst from state data
+    states = state_data.scan(/^src (\S+) dst (\S+).*?proto esp spi (0x[0-9a-f]+)/m)
+
+    # Identify which states to drop
+    states_to_drop = states.reject { |(_, _, spi)| new_spis.include?(spi) }
+    states_to_drop.each do |src, dst, spi|
+      sshable_cmd("sudo ip -n #{nic.src_ipsec_tunnels.first.vm_name(nic)} xfrm state delete src #{src} dst #{dst} proto esp spi #{spi}")
+    end
+
+    pop "drop_old_state is complete"
+  end
+
+  def sshable_cmd(cmd)
+    nic.vm.vm_host.sshable.cmd(cmd)
+  end
+
+  def create_state(tunnel, args)
+    namespace = tunnel.vm_name(nic)
+    src = subdivide_network(tunnel.src_nic.vm.ephemeral_net6).network
+    dst = subdivide_network(tunnel.dst_nic.vm.ephemeral_net6).network
+    reqid = args["reqid"]
+    key = tunnel.src_nic.encryption_key
+
+    sshable_cmd("sudo ip -n #{namespace} xfrm state add " \
+      "src #{src} dst #{dst} proto esp spi #{args["spi4"]} reqid #{reqid} mode tunnel " \
+      "aead 'rfc4106(gcm(aes))' #{key} 128 sel src 0.0.0.0/0 dst 0.0.0.0/0 ")
+    sshable_cmd("sudo ip -n #{namespace} xfrm state add " \
+      "src #{src} dst #{dst} proto esp spi #{args["spi6"]} reqid #{reqid} mode tunnel " \
+      "aead 'rfc4106(gcm(aes))' #{key} 128")
+  end
+
+  def policy_update_cmd(namespace, src, dst, tmpl_src, tmpl_dst, reqid, spi, dir)
+    sshable_cmd("sudo ip -n #{namespace} xfrm policy update " \
+      "src #{src} dst #{dst} dir #{dir} tmpl src #{tmpl_src} dst #{tmpl_dst} " \
+      "proto esp reqid #{reqid} mode tunnel")
+  end
+
+  def policy_update(tunnel, dir)
+    namespace = tunnel.vm_name(nic)
+    tmpl_src = subdivide_network(tunnel.src_nic.vm.ephemeral_net6).network
+    tmpl_dst = subdivide_network(tunnel.dst_nic.vm.ephemeral_net6).network
+    reqid = tunnel.src_nic.rekey_payload["reqid"]
+    src4 = tunnel.src_nic.private_ipv4
+    dst4 = tunnel.dst_nic.private_ipv4
+    src6 = tunnel.src_nic.private_ipv6
+    dst6 = tunnel.dst_nic.private_ipv6
+    spi4 = tunnel.src_nic.rekey_payload["spi4"]
+    spi6 = tunnel.src_nic.rekey_payload["spi6"]
+
+    policy_update_cmd(namespace, src4, dst4, tmpl_src, tmpl_dst, reqid, spi4, dir)
+    policy_update_cmd(namespace, src6, dst6, tmpl_src, tmpl_dst, reqid, spi6, dir)
+  end
+
+  def subdivide_network(net)
+    prefix = net.netmask.prefix_len + 1
+    halved = net.resize(prefix)
+    halved.next_sib
+  end
+end

prog/vnet/subnet_nexus.rb

@@ -2,7 +2,7 @@
 
 class Prog::Vnet::SubnetNexus < Prog::Base
   subject_is :private_subnet
-  semaphore :refresh_mesh, :destroy
+  semaphore :refresh_mesh, :destroy, :refresh_keys
 
   def self.assemble(project_id, name: nil, location: "hetzner-hel1", ipv6_range: nil, ipv4_range: nil)
     project = Project[project_id]
@@ -35,13 +35,79 @@ class Prog::Vnet::SubnetNexus < Prog::Base
       hop :refresh_mesh
     end
 
+    when_refresh_keys_set? do
+      private_subnet.update(state: "refreshing_keys")
+      hop :refresh_keys
+    end
+
     nap 30
   end
 
+  def gen_encryption_key
+    "0x" + SecureRandom.bytes(36).unpack1("H*")
+  end
+
+  def gen_spi
+    "0x" + SecureRandom.bytes(4).unpack1("H*")
+  end
+
+  def gen_reqid
+    SecureRandom.random_number(100000) + 1
+  end
+
+  def refresh_keys
+    payload = {}
+
+    private_subnet.nics.each do |nic|
+      payload = {
+        spi4: gen_spi,
+        spi6: gen_spi,
+        reqid: gen_reqid
+      }
+      nic.update(encryption_key: gen_encryption_key, rekey_payload: payload)
+    end
+
+    private_subnet.nics.each do |nic|
+      nic.incr_start_rekey
+    end
+
+    hop :wait_inbound_setup
+  end
+
+  def wait_inbound_setup
+    if private_subnet.nics.all? { |nic| nic.strand.label == "wait_rekey_outbound_trigger" }
+      private_subnet.nics.each(&:incr_trigger_outbound_update)
+      hop :wait_outbound_setup
+    end
+
+    donate
+  end
+
+  def wait_outbound_setup
+    if private_subnet.nics.all? { |nic| nic.strand.label == "wait_rekey_old_state_drop_trigger" }
+      private_subnet.nics.each(&:incr_old_state_drop_trigger)
+      hop :wait_old_state_drop
+    end
+
+    donate
+  end
+
+  def wait_old_state_drop
+    if private_subnet.nics.all? { |nic| nic.strand.label == "wait" }
+      private_subnet.update(state: "waiting")
+      private_subnet.nics.each do |nic|
+        nic.update(encryption_key: nil, rekey_payload: nil)
+      end
+      decr_refresh_keys
+      hop :wait
+    end
+    donate
+  end
+
   def refresh_mesh
     DB.transaction do
       private_subnet.nics.each do |nic|
-        nic.update(encryption_key: "0x" + SecureRandom.bytes(36).unpack1("H*"))
+        nic.update(encryption_key: gen_encryption_key)
         nic.incr_refresh_mesh
       end
     end

spec/prog/vnet/nic_nexus_spec.rb

@@ -85,6 +85,98 @@ RSpec.describe Prog::Vnet::NicNexus do
         nx.wait
       }.to hop("detach_vm")
     end
+
+    it "hops to start rekey if needed" do
+      expect(nx).to receive(:when_start_rekey_set?).and_yield
+      expect {
+        nx.wait
+      }.to hop("start_rekey")
+    end
+  end
+
+  describe "#rekey" do
+    it "buds rekey with setup_inbound and hops to wait_rekey_inbound" do
+      expect(nx).to receive(:bud).with(Prog::Vnet::RekeyNicTunnel, {}, :setup_inbound).and_return(true)
+      expect {
+        nx.start_rekey
+      }.to hop("wait_rekey_inbound")
+    end
+
+    it "reaps and donates if setup_inbound is continuing" do
+      expect(nx).to receive(:leaf?).and_return(false)
+      expect(nx).to receive(:reap).and_return(true)
+      expect(nx).to receive(:donate).and_return(true)
+      nx.wait_rekey_inbound
+    end
+
+    it "reaps and hops to wait_rekey_outbound_trigger if setup_inbound is completed" do
+      expect(nx).to receive(:leaf?).and_return(true)
+      expect(nx).to receive(:reap).and_return(true)
+      expect(nx).to receive(:decr_start_rekey).and_return(true)
+      expect {
+        nx.wait_rekey_inbound
+      }.to hop("wait_rekey_outbound_trigger")
+    end
+
+    it "if outbound setup is not triggered, just donate" do
+      expect(nx).to receive(:when_trigger_outbound_update_set?).and_return(false)
+      expect(nx).to receive(:donate).and_return(true)
+      nx.wait_rekey_outbound_trigger
+    end
+
+    it "if outbound setup is triggered, hops to setup_outbound" do
+      expect(nx).to receive(:when_trigger_outbound_update_set?).and_yield
+      expect(nx).to receive(:bud).with(Prog::Vnet::RekeyNicTunnel, {}, :setup_outbound).and_return(true)
+      expect {
+        nx.wait_rekey_outbound_trigger
+      }.to hop("wait_rekey_outbound")
+    end
+
+    it "wait_rekey_outbound reaps and donates if setup_outbound is continuing" do
+      expect(nx).to receive(:leaf?).and_return(false)
+      expect(nx).to receive(:reap).and_return(true)
+      expect(nx).to receive(:donate).and_return(true)
+      nx.wait_rekey_outbound
+    end
+
+    it "wait_rekey_outbound reaps and hops to wait_rekey_old_state_drop_trigger if setup_outbound is completed" do
+      expect(nx).to receive(:leaf?).and_return(true)
+      expect(nx).to receive(:reap).and_return(true)
+      expect(nx).to receive(:decr_trigger_outbound_update).and_return(true)
+      expect {
+        nx.wait_rekey_outbound
+      }.to hop("wait_rekey_old_state_drop_trigger")
+    end
+
+    it "wait_rekey_old_state_drop_trigger donates if trigger is not set" do
+      expect(nx).to receive(:when_old_state_drop_trigger_set?).and_return(false)
+      expect(nx).to receive(:donate).and_return(true)
+      nx.wait_rekey_old_state_drop_trigger
+    end
+
+    it "wait_rekey_old_state_drop_trigger hops to wait_rekey_old_state_drop if trigger is set" do
+      expect(nx).to receive(:when_old_state_drop_trigger_set?).and_yield
+      expect(nx).to receive(:bud).with(Prog::Vnet::RekeyNicTunnel, {}, :drop_old_state).and_return(true)
+      expect {
+        nx.wait_rekey_old_state_drop_trigger
+      }.to hop("wait_rekey_old_state_drop")
+    end
+
+    it "wait_rekey_old_state_drop reaps and donates if drop_old_state is continuing" do
+      expect(nx).to receive(:leaf?).and_return(false)
+      expect(nx).to receive(:reap).and_return(true)
+      expect(nx).to receive(:donate).and_return(true)
+      nx.wait_rekey_old_state_drop
+    end
+
+    it "wait_rekey_old_state_drop reaps and hops to wait if drop_old_state is completed" do
+      expect(nx).to receive(:leaf?).and_return(true)
+      expect(nx).to receive(:reap).and_return(true)
+      expect(nx).to receive(:decr_old_state_drop_trigger).and_return(true)
+      expect {
+        nx.wait_rekey_old_state_drop
+      }.to hop("wait")
+    end
   end
 
   describe "#refresh_mesh" do

spec/prog/vnet/rekey_nic_tunnel_spec.rb

@@ -0,0 +1,138 @@
+# frozen_string_literal: true
+
+RSpec.describe Prog::Vnet::RekeyNicTunnel do
+  subject(:nx) {
+    described_class.new(st)
+  }
+
+  let(:st) { Strand.new }
+  let(:ps) {
+    PrivateSubnet.create_with_id(name: "ps", location: "hetzner-hel1", net6: "fd10:9b0b:6b4b:8fbb::/64",
+      net4: "1.1.1.0/26", state: "waiting")
+  }
+  let(:tunnel) {
+    n_src = Nic.create_with_id(private_subnet_id: ps.id,
+      private_ipv6: "fd10:9b0b:6b4b:8fbb:abc::",
+      private_ipv4: "10.0.0.1",
+      mac: "00:00:00:00:00:00",
+      encryption_key: "0x736f6d655f656e6372797074696f6e5f6b6579",
+      name: "default-nic",
+      rekey_payload: {"reqid" => 86879, "spi4" => "0xe2222222", "spi6" => "0xe3333333"})
+    n_dst = Nic.create_with_id(private_subnet_id: ps.id,
+      private_ipv6: "fd10:9b0b:6b4b:8fbb:def::",
+      private_ipv4: "10.0.0.2",
+      mac: "00:00:00:00:00:00",
+      encryption_key: "0x736f6d655f656e6372797074696f6e5f6b6579",
+      name: "default-nic",
+      rekey_payload: {"reqid" => 14329, "spi4" => "0xe0000000", "spi6" => "0xe1111111"})
+    IpsecTunnel.create_with_id(src_nic_id: n_src.id, dst_nic_id: n_dst.id).tap { _1.id = "0a9a166c-e7e7-4447-ab29-7ea442b5bb0e" }
+  }
+
+  before do
+    nx.instance_variable_set(:@nic, tunnel.src_nic)
+  end
+
+  describe ".sshable_cmd" do
+    let(:sshable) { instance_double(Sshable) }
+    let(:vm) {
+      vmh = instance_double(VmHost, sshable: sshable)
+      instance_double(Vm, vm_host: vmh)
+    }
+
+    it "returns vm_host sshable of source nic" do
+      expect(nx.nic).to receive(:vm).and_return(vm)
+      expect(sshable).to receive(:cmd).with("echo hello")
+      nx.sshable_cmd("echo hello")
+    end
+  end
+
+  describe "#setup_inbound" do
+    let(:vm) {
+      instance_double(Vm, name: "hellovm", id: "0a9a166c-e7e7-4447-ab29-7ea442b5bb0e",
+        ephemeral_net6: NetAddr.parse_net("2a01:4f8:10a:128b:4919::/80"))
+    }
+
+    before do
+      expect(tunnel).to receive(:vm_name).with(tunnel.src_nic).and_return("hellovm")
+      expect(tunnel.src_nic).to receive(:vm).and_return(vm)
+      expect(tunnel.dst_nic).to receive(:vm).and_return(vm)
+      expect(tunnel.src_nic).to receive(:dst_ipsec_tunnels).and_return([tunnel])
+    end
+
+    it "pops with inbound_setup is complete" do
+      expect(nx).to receive(:sshable_cmd).with("sudo ip -n hellovm xfrm state add src 2a01:4f8:10a:128b:4919:8000:: dst 2a01:4f8:10a:128b:4919:8000:: proto esp spi 0xe2222222 reqid 86879 mode tunnel aead 'rfc4106(gcm(aes))' 0x736f6d655f656e6372797074696f6e5f6b6579 128 sel src 0.0.0.0/0 dst 0.0.0.0/0 ").and_return(true)
+      expect(nx).to receive(:sshable_cmd).with("sudo ip -n hellovm xfrm state add src 2a01:4f8:10a:128b:4919:8000:: dst 2a01:4f8:10a:128b:4919:8000:: proto esp spi 0xe3333333 reqid 86879 mode tunnel aead 'rfc4106(gcm(aes))' 0x736f6d655f656e6372797074696f6e5f6b6579 128").and_return(true)
+      expect(nx).to receive(:pop).with("inbound_setup is complete").and_return(true)
+      nx.setup_inbound
+    end
+  end
+
+  describe "#setup_outbound" do
+    let(:vm) {
+      instance_double(Vm, name: "hellovm", id: "0a9a166c-e7e7-4447-ab29-7ea442b5bb0e",
+        ephemeral_net6: NetAddr.parse_net("2a01:4f8:10a:128b:4919::/80"), inhost_name: "inhost")
+    }
+
+    before do
+      expect(tunnel).to receive(:vm_name).with(tunnel.src_nic).and_return("hellovm").at_least(:once)
+      expect(tunnel.src_nic).to receive(:vm).and_return(vm).at_least(:once)
+      expect(tunnel.dst_nic).to receive(:vm).and_return(vm).at_least(:once)
+      expect(tunnel.src_nic).to receive(:src_ipsec_tunnels).and_return([tunnel]).at_least(:once)
+    end
+
+    it "creates new state and policy for src" do
+      expect(nx).to receive(:sshable_cmd).with("sudo ip -n hellovm xfrm state add src 2a01:4f8:10a:128b:4919:8000:: dst 2a01:4f8:10a:128b:4919:8000:: proto esp spi 0xe2222222 reqid 86879 mode tunnel aead 'rfc4106(gcm(aes))' 0x736f6d655f656e6372797074696f6e5f6b6579 128 sel src 0.0.0.0/0 dst 0.0.0.0/0 ").and_return(true)
+      expect(nx).to receive(:sshable_cmd).with("sudo ip -n hellovm xfrm state add src 2a01:4f8:10a:128b:4919:8000:: dst 2a01:4f8:10a:128b:4919:8000:: proto esp spi 0xe3333333 reqid 86879 mode tunnel aead 'rfc4106(gcm(aes))' 0x736f6d655f656e6372797074696f6e5f6b6579 128").and_return(true)
+      expect(nx).to receive(:sshable_cmd).with("sudo ip -n hellovm xfrm policy update src 10.0.0.1/32 dst 10.0.0.2/32 dir out tmpl src 2a01:4f8:10a:128b:4919:8000:: dst 2a01:4f8:10a:128b:4919:8000:: proto esp reqid 86879 mode tunnel").and_return(true)
+      expect(nx).to receive(:sshable_cmd).with("sudo ip -n hellovm xfrm policy update src fd10:9b0b:6b4b:8fbb:abc::/128 dst fd10:9b0b:6b4b:8fbb:def::/128 dir out tmpl src 2a01:4f8:10a:128b:4919:8000:: dst 2a01:4f8:10a:128b:4919:8000:: proto esp reqid 86879 mode tunnel").and_return(true)
+      expect(nx).to receive(:pop).with("outbound_setup is complete").and_return(true)
+      nx.setup_outbound
+    end
+  end
+
+  describe "#drop_old_state" do
+    let(:vm) {
+      instance_double(Vm, name: "hellovm", id: "0a9a166c-e7e7-4447-ab29-7ea442b5bb0e",
+        ephemeral_net6: NetAddr.parse_net("2a01:4f8:10a:128b:4919::/80"), inhost_name: "inhost")
+    }
+    let(:states_data) {
+      "src 2a01:4f8:10a:128b:7537:: dst 2a01:4f8:10a:128b:4919::
+proto esp spi 0xe3333333 reqid 49966 mode tunnel
+replay-window 0
+aead rfc4106(gcm(aes)) 0x6c838df72ba3abe1a2643ee104e21d617830f1b765addced5e26d17a4cc5048d1468ac54 128
+anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
+sel src ::/0 dst ::/0
+src 2a01:4f8:10a:128b:7537:: dst 2a01:4f8:10a:128b:4919::
+proto esp spi 0xe2222222 reqid 49966 mode tunnel
+replay-window 0
+aead rfc4106(gcm(aes)) 0x6c838df72ba3abe1a2643ee104e21d617830f1b765addced5e26d17a4cc5048d1468ac54 128
+anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
+sel src 0.0.0.0/0 dst 0.0.0.0/0
+src 2a01:4f8:10a:128b:4919:: dst 2a01:4f8:10a:128b:7537::
+proto esp spi 0x610a9eb5 reqid 29850 mode tunnel
+replay-window 0
+aead rfc4106(gcm(aes)) 0xc0c6485e1020fd7178cf9bed74b91cfee06bc5b19066db12ec0d801737954296f1894134 128
+anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
+sel src ::/0 dst ::/0
+src 2a01:4f8:10a:128b:4919:: dst 2a01:4f8:10a:128b:7537::
+proto esp spi 0x059e11e6 reqid 29850 mode tunnel
+replay-window 0
+aead rfc4106(gcm(aes)) 0xc0c6485e1020fd7178cf9bed74b91cfee06bc5b19066db12ec0d801737954296f1894134 128
+anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
+sel src 0.0.0.0/0 dst 0.0.0.0/0"
+    }
+
+    before do
+      expect(tunnel.src_nic).to receive(:vm).and_return(vm).at_least(:once)
+      expect(tunnel.src_nic).to receive(:dst_ipsec_tunnels).and_return([tunnel]).at_least(:once)
+    end
+
+    it "drops old states and pops" do
+      expect(nx).to receive(:sshable_cmd).with("sudo ip -n inhost xfrm state").and_return(states_data)
+      expect(nx).to receive(:sshable_cmd).with("sudo ip -n inhost xfrm state delete src 2a01:4f8:10a:128b:4919:: dst 2a01:4f8:10a:128b:7537:: proto esp spi 0x610a9eb5").and_return(true)
+      expect(nx).to receive(:sshable_cmd).with("sudo ip -n inhost xfrm state delete src 2a01:4f8:10a:128b:4919:: dst 2a01:4f8:10a:128b:7537:: proto esp spi 0x059e11e6").and_return(true)
+      expect(nx).to receive(:pop).with("drop_old_state is complete").and_return(true)
+      nx.drop_old_state
+    end
+  end
+end

spec/prog/vnet/subnet_nexus_spec.rb

@@ -64,6 +64,20 @@ RSpec.describe Prog::Vnet::SubnetNexus do
     end
   end
 
+  describe ".gen_spi" do
+    it "generates a random spi" do
+      expect(SecureRandom).to receive(:bytes).with(4).and_return("e3af3a04")
+      expect(nx.gen_spi).to eq("0x6533616633613034")
+    end
+  end
+
+  describe ".gen_reqid" do
+    it "generates a random reqid" do
+      expect(SecureRandom).to receive(:random_number).with(100000).and_return(10)
+      expect(nx.gen_reqid).to eq(11)
+    end
+  end
+
   describe "#wait" do
     it "hops to destroy if when_destroy_set?" do
       expect(nx).to receive(:when_destroy_set?).and_yield
@@ -80,6 +94,14 @@ RSpec.describe Prog::Vnet::SubnetNexus do
       }.to hop("refresh_mesh")
     end
 
+    it "hops to refresh_keys if when_refresh_keys_set?" do
+      expect(nx).to receive(:when_refresh_keys_set?).and_yield
+      expect(ps).to receive(:update).with(state: "refreshing_keys").and_return(true)
+      expect {
+        nx.wait
+      }.to hop("refresh_keys")
+    end
+
     it "naps if nothing to do" do
       expect {
         nx.wait
@@ -87,6 +109,100 @@ RSpec.describe Prog::Vnet::SubnetNexus do
     end
   end
 
+  describe "#refresh_keys" do
+    let(:nic) {
+      instance_double(Nic, id: "57afa8a7-2357-4012-9632-07fbe13a3133")
+    }
+
+    it "refreshes keys and hops to wait_refresh_keys" do
+      expect(ps).to receive(:nics).and_return([nic]).twice
+      expect(nx).to receive(:gen_spi).and_return("0xe3af3a04").twice
+      expect(nx).to receive(:gen_reqid).and_return(86879)
+      expect(nx).to receive(:gen_encryption_key).and_return("0x0a0b0c0d0e0f10111213141516171819")
+      expect(nic).to receive(:update).with(encryption_key: "0x0a0b0c0d0e0f10111213141516171819", rekey_payload:
+        {
+          spi4: "0xe3af3a04",
+          spi6: "0xe3af3a04",
+          reqid: 86879
+        }).and_return(true)
+      expect(nic).to receive(:incr_start_rekey).and_return(true)
+      expect {
+        nx.refresh_keys
+      }.to hop("wait_inbound_setup")
+    end
+  end
+
+  describe "#wait_inbound_setup" do
+    let(:nic) {
+      st = instance_double(Strand, label: "start")
+      instance_double(Nic, strand: st)
+    }
+
+    it "donates if state creation is ongoing" do
+      expect(ps).to receive(:nics).and_return([nic]).at_least(:once)
+      expect(nx).to receive(:donate).and_call_original
+
+      expect { nx.wait_inbound_setup }.to nap(0)
+    end
+
+    it "hops to wait_policy_updated if state creation is done" do
+      expect(nic.strand).to receive(:label).and_return("wait_rekey_outbound_trigger")
+      expect(ps).to receive(:nics).and_return([nic]).at_least(:once)
+      expect(nic).to receive(:incr_trigger_outbound_update).and_return(true)
+      expect {
+        nx.wait_inbound_setup
+      }.to hop("wait_outbound_setup")
+    end
+  end
+
+  describe "#wait_outbound_setup" do
+    let(:nic) {
+      st = instance_double(Strand, label: "wait_rekey_outbound")
+      instance_double(Nic, strand: st)
+    }
+
+    it "donates if policy update is ongoing" do
+      expect(ps).to receive(:nics).and_return([nic]).at_least(:once)
+      expect(nx).to receive(:donate).and_call_original
+
+      expect { nx.wait_outbound_setup }.to nap(0)
+    end
+
+    it "hops to wait_state_dropped if policy update is done" do
+      expect(nic.strand).to receive(:label).and_return("wait_rekey_old_state_drop_trigger")
+      expect(ps).to receive(:nics).and_return([nic]).at_least(:once)
+      expect(nic).to receive(:incr_old_state_drop_trigger).and_return(true)
+      expect {
+        nx.wait_outbound_setup
+      }.to hop("wait_old_state_drop")
+    end
+  end
+
+  describe "#wait_old_state_drop" do
+    let(:nic) {
+      st = instance_double(Strand, label: "wait_rekey_old_state_drop")
+      instance_double(Nic, strand: st)
+    }
+
+    it "donates if policy update is ongoing" do
+      expect(ps).to receive(:nics).and_return([nic]).at_least(:once)
+      expect(nx).to receive(:donate).and_call_original
+
+      expect { nx.wait_old_state_drop }.to nap(0)
+    end
+
+    it "hops to wait if all is done" do
+      expect(nic.strand).to receive(:label).and_return("wait")
+      expect(ps).to receive(:update).with(state: "waiting").and_return(true)
+      expect(ps).to receive(:nics).and_return([nic]).at_least(:once)
+      expect(nic).to receive(:update).with(encryption_key: nil, rekey_payload: nil).and_return(true)
+      expect(nx).to receive(:decr_refresh_keys).and_return(true)
+      expect {
+        nx.wait_old_state_drop
+      }.to hop("wait")
+    end
+  end
+
   describe "#refresh_mesh" do
     let(:nic) {
       instance_double(Nic)