🎶 Underlay, overlay … networking free, the network engineers of the Internet are networking free 🎶

Intro

One of the more ‘modern’ concepts that is used predominantly amongst marketing of a technology businesses, is what network engineers have been using for decades to separate the ‘core’ Interior Gateway Protocol (IGP) ‘the underlay’, from the ‘services’ that need to be supported over the top of it, ‘the overlay’.

So What is an Underlay?

An underlay is the key protocol(s) which connect the routers and devices to the network (the Autonomous System Number (ASN)), and allow routing to occur within the network, i.e., exchanging loopback information of the service routers/ devices, example for Pseudowire Headend Termination (PWHT).

This is usually an IGP such as IS-IS or OSPF, however it can be MPLS, Segment Routing (SR), MPLS-SR, Flex Algo … etc.

Overall the Underlay is usually split into ‘areas’/ regions based on either technical limitations and/ or business requirements with the key driver to reduce ‘blast radii’ should anything go wrong in the ASN.

Example: There would be several regions for a tier1 operator, the diagram below is quick image to show the relationships between the boundaries.

Example from Metro Fabric and Broadband Edge—Juniper Validated Design (JVD)

Metro

This is typically the ‘area’ which covers the ‘last mile’ (the connections to the subscribers/ end-users including splitters, street cabinets … etc.), and is all physically connected back to a central location in a hub and spoke topology.

If the Metro is located inside say a city, then there will be multiple Metro’s (as needed), to connect the subscribers/ end-users, just like postcodes.

Usually the Metro Headend will live somewhere central to the deployment, and for many years in the UK, since the 2000’s telephone exchanges have been used (see Local Loop Unbundling).

Unfortunately, with the closure of the exchanges in the UK, this means that UK operators/ ISPs need to invest in facilities of their own, example CityFibre build FEX’s.

Metro Aggregation (AGG)/ Headend Aggregation (AGG)

This is where multiple Metro’s are connected to an aggregation point. This will facilitate connectivity between the Metro and the Region. And if required house some of the service termination devices, i.e., Broadband Network Gateway (BNG)/ Carrier Grade Network Address Translation (CG-NAT), NNI handoff (A10-NSP), … etc.

This can be a city level AGG, and requires a high degree of resilience and redundancy both environmentally, and connectivity (both downstream to the Metro and upstream to the Region).

Region

This covers a geographical area and is the next level of aggregation for the Metro AGGs. This will typically follow a country’s geographical mapping, as it will depend on the literal ‘lay of the land’.

Core

This is where the ‘magic’ happens, as this comprises of the highest bandwidth links, dense optical equipment, and other than upgrading capacity and devices, should be the lowest touch point in any network (that’s not to say that several jobs aren’t carried on in any one day/ week, however it should have less adds, moves and/ or changes (CRUD) compared to the other ‘areas’.

So if that’s an Underlay, What’s is an Overlay?

An Overlay is the service, it’s the layer that is used to transport the subscriber/ end-users service from the first device which can create the overlay (usually a Provider Edge Router (PE router) or Customer Edge Router (CE) in the case of SDN), to the service termination device (usually a BNG or an A10-NSP).

To do this, you need to build either:

Pseudowire(s)

A pseudowire(s) can we be created with:

• L2Circuits.

• VPLS VPWS.

• EVPN VPWS (E-LINE).

However, they require the endpoints to be distributed via the ‘underlay’, i.e., loopback IPs via the IGP.

Example EVPN-VPWS (E-LINE) from Metro Fabric and Broadband Edge—Juniper Validated Design (JVD)

MP-BGP VPN(s)

Multiprotocol BGP (MP-BGP) can be used to carry several protocols to create L2 or L3 VPN:

• VPLS.

• EVPN E-LAN.

• VXLAN.

However, they require the endpoints to be distributed via the ‘underlay’, i.e., loopback IPs via the IGP.

SDWAN(s)

Software Defined Wide Area Networking were (for me) what popularised the term ‘overlay’, as they automatically build VPN tunnels based on policies and controllers.

By using a controller to coordinate the endpoints and build the ‘overlay’ of tunnels ‘on-demand’. they remove any dependancy on the ‘underlay’.

Summary

Whilst pseudowire(s) and MP-BGP have a dependancy on the ‘underlay’, the benefit comes from not needing a 3rd party controller, as per SDWAN, which is unaware of the topology.

SDWAN compensates for this by using jitter and latency between the end-points to ‘monitor’ the underlay and where necessary steer traffic, however this only works on a point-to-point basis, whereas pseudowire(s) and MP-BGP can leverage the IGP which is calculated link-by-link/ hop-by-hop.