Accessing the Internet from Oracle Cloud VMware Solution

Okay, so we have our Oracle Cloud VMware Solution (OCVS) up and running, we can connect to the vCenter, NSX Manager etc. and have started to deploy workloads inside the SDDC, that’s awesome! But, now we want to be able to connect to the Internet from those workloads, and, just maybe, connect to them from the Internet. In this post, we’ll look at the steps needed to enable our workload VMs to access the Internet. Then, in the second post in this series, we’ll enhance our Internet access to allow us to publish services to the Internet, from the SDDC.

OCVS is a first-class citizen in Oracle Cloud Infrastructure (OCI), and has direct access to all the native services, which is great. Although the flip side of that “great” is that OCVS needs to use the native OCI Internet access services to reach the err… Internet. This makes the task of giving general Internet access to our workload VMs, slightly more complex than in some other environments. It’s not difficult when you know the steps, just not super obvious how you go about it until you do.

OCI Internet Access

OCI offers two types of Internet access, the NAT Gateway and the Internet Gateway. We’ll look at the NAT Gateway first. We’ll illustrate its use with a simple diagram, then call out a couple of important notes. In the diagram below we see an OCI virtual machine on an OCI subnet. Its route to the Internet follows the flow shown.

In order to reach the Internet, we need:

1. An Outbound Rule in the Subnet’s associated Security List which permits our flow.
2. A Route Rule in the Subnet’s associated Route Table matching our Internet destination and pointing to:
3. An OCI “NAT Gateway” which provides our Internet Access.

The way the NAT Gateway works is nice and simple. We use a Route Table rule, like the one below, to direct traffic heading for the Internet using the `0.0.0.0/0` destination. We set the Target Type to “NAT Gateway”, and select the NAT Gateway deployed in our SDDC. Here’s an example of a Route Rule like that, in a Route Table, targeting a NAT Gateway (which is helpfully named “NAT Gateway”).

Workloads on any Subnet or VLAN using a Route Table with this rule in will, (subject to the Security List or Network Security Group rules) be able to access the Internet.

OCVS SDDC Workloads’ Internet Access

Let’s take a look at the same thing but from the point of view of an OCVS SDDC workload VM. We’ll start off by revisiting the flow diagram, but this time, we’ll add in the SDDC components. We’ll have a workload VM on an NSX-T Logical Segment which is connected to a Tier-1 Gateway. The Tier-1 Gateway is automatically uplinked to a Tier-0 Gateway, which is connected to OCI’s networking layer through a VLAN. From there, things start to resemble the native flow diagram from earlier in the post. In picture form, it looks like this.

This time, in order to reach the Internet, we will need:

1. Our workload VM connected to a Tier-1 Segment.
2. Unless we’re heading to another segment, we will take the T1’s default route to it’s parent Tier-0.
3. For the Internet, we will take the T0’s default route to the VLAN’s next-hop gateway.

Now, back on familiar territory:

4. An Outbound Rule in the VLAN‘s associated Network Security Group which permits our flow.
5. A Route Rule in the VLAN’s associated Route Table matching our Internet destination and pointing to:
6. An OCI “NAT Gateway” which provides our Internet Access.

Not quite there yet…

All good so far, except, what we’ve shown in the diagram won’t work (yet). The issue is the OCI NAT Gateway! As we saw earlier, all we need to do is send traffic to it and the NAT gateway does most of the hard work. But, it does rely on seeing that traffic come from a recognizable OCI device, which it identifies by the OCI device interface’s Oracle Cloud IDentifier (OCID).

Unfortunately, SDDC VMs routing through the Tier-0 don’t have an OCID. But, in order to send traffic into any device connected to a VLAN, OCI needs to loan that device an OCID. The T0 is one such device, and because of that, Source NATing SDDC VMs to the Interface address of the T0 (with its loaned OCID), lets us work around the NAT Gateway limitation. So, all we need to do is ensure all traffic from the SDDC heading for the Internet gets SNAT’d behind the T0 (the .130 address in the diagram above) as it leaves the SDDC. Let’s go and do that now.

First, let’s take a closer look at that OCID loaning business. The SDDC vMotion “network” for example, is a VLAN, but, for the most part, OCI doesn’t need to know what is happening inside it. The hosts can talk to each other and that’s sufficient. However, when it does need to know, like say for a connection to vCenter on the vSphere VLAN, to allow OCI to keep track of a Layer 3 IP address inside the VLAN, we assign an External Access (V)IP. This anchors the IP address of some interface of a device inside the VLAN to an OCID. The rest of OCI can then use the OCID to find that device interface even in an opaque Layer 2 VLAN. Here’s the “VLAN Details” view of the Uplink-1 VLAN showing OCI’s view of the Tier-0’s “address”:

In the image above, we can see that the External Access VIP has been assigned the address 10.76.9.130, and, in NSX-T, if we check the Tier-0 Gateway config (following the orange arrows), in the left hand side red oval we can see that we’re using that same address for the HA VIP.

As an aside, notice the Interface names in the second red oval? The two interfaces forming the T0 HA uplink are called “NSX-Edge-Uplink-1” and “NSX-Edge-Uplink-2“. These two interfaces are both connected to the OCI VLAN called “VLAN-OCVS-NSX Edge Uplink-1” and, to confuse things further, there is also a (currently unused) VLAN called “VLAN-OCVS-NSX-Edge Uplink-2“. Try not to confuse these NSX-T T0 interfaces with the similarly named VLANs!

Let’s look at that last flow diagram, but this time we’ll add some IP addresses so we can see the NAT(s) in action:

In (7) we can see the packet has the vSphere VM’s private source address of 10.76.124.10 and the destination public address of 8.8.8.8. As the packet passes through the Tier-0 Gateway in (8) we see the Tier-0 swaps the original source address to its own, 10.76.9.130. As packets then pass through the OCI NAT Gateway in (9), it swaps the Source Address (from the dot-130 address of the Tier-0), to that of its own Public IP address, 132.226.122.232. The Tier-0 and OCI NAT Gateways both keep track of which private source addresses are talking to which public addresses, so that they can reverse this process on returning packets.

Configuring the NSX-T NAT rules

Let’s see that configured in our NSX-T Tier-0 NAT rules:

Choosing Networking -> NAT, and then selecting our Tier-0 in the Gateway selector we see the NAT rules. We have four in this test environment, but in a production environment there could be more. The rule which we’re most interested in right now is the “SNAT of Last Resort” at the bottom of the list. What we want to do, is make sure that traffic from our SDDC workloads, when heading for the Internet, is Source NAT’d behind our Tier-0 VIP address. The important parts of that rule are:

• The Name and Action fields are hopefully self-explanatory.
• The Source in our lab example is the 10.76.124.0/24 subnet we have allocated for the workloads we’ll build inside the SDDC. In our example this is only small, as we don’t need much space in this particular lab. Often, that will require more sources. In an ideal world, we  would allocate a single, large subnet to the SDDC which we could pre-arrange the SNAT for, and then consume parts of as we deploy SDDC subnets from within it. However, in a less ideal world, we would add non-contiguous subnets as we build each workload segment, and need to repeat this rule for each of the discrete source subnets we deploy.
• The Destination is the catch-all “quad-zero” address which is the CIDR equivalent of “Any”.
• The Translated field is the address we want to replace the Source with, in this case the T0 VIP Address.

You can’t see the priorities of the rules without clicking the small “>” arrow next to each rule, but this rule should come last. That’s because, with just this rule in place, traffic heading for the Internet will match the quad-zero route, but not just traffic bound for the Internet, all traffic leaving the SDDC will be affected, and Source NAT’d!

This might be what we want, but usually isn’t. Normally, we want to NAT Internet traffic, and Route on-net (the VCN, the rest of OCI and to all our on-prem locations) traffic. In order to do this, we need to tell NSX not to SNAT traffic to those destinations, and we need to do that before it gets ahead of itself and slaps down the Internet SNAT. This is where the additional rules in the NAT table screenshot come in, and in particular, their priorities. NAT rules in NSX-T have a numeric priority, with lower numbers having a higher priority (so being actioned first). As we’ve seen, having the SNAT of Last Resort… well… last, is very important.

In this lab, we have three “No_SNAT” rules, one for each subnet in RFC1918. Together these tell NSX-T not to SNAT traffic from our SDDC when it is heading towards private addresses in any of the RFC1918 spaces, typically the addresses we find in customer networks. It is of course possible to find a customer using non-1918 addresses, and if that were the case, we’d just duplicate more No_SNAT rules with those destination addresses in, in addition to, or instead of, these three. To save squinting at the picture, here’s those four rules in table form.

The only differences between these three No SNAT rules and the SNAT rule are the Action of course, and the Any in the Translated field. The latter just means we’re not actually translating, and Any original destination will make it through.

And that’s it. With these steps completed, our OCVS SDDC workload VMs should now be able to reach the Internet.

Wrapping up

In this post we saw how native OCI VMs can access the Internet through a NAT Gateway. We saw that the SDDC as a whole, follows the same steps to reach the Internet. We also saw that to enable the SDDC’s workload VMs to use the NAT Gateway, we had to hide their real addresses behind the Tier-0 Gateway’s HA VIP address. As always, Routing and Security played their part, but, once we got all the bits lined up, using the NAT Gateway from the SDDC was pretty straight forward, even if we had to be careful with the order of the NSX NAT rules.

That’s great, but, what if as well as having our VMs go out to the Internet, we need to offer services to the Internet which means unsolicited inbound connections? How does that work? We’ll expand on what we learned here and cover that in the second post in this series.

The full version of this post originally appeared on https://notthe.blog and this slightly abridged version is re-published with permission.