DIQ'S DEN - WWW.COUYON.NET
  • Home
  • Blog
  • Photos

Anycast Your Infrastructure IP's

2/14/2018

 
A while back, I tasked one of my team members to update the NTP servers used in one of our datacenters. We were using standard pool NTP services and decided to move away from them for various reasons. We found that stable time was more important than accurate time, and the pools definitely didn't add stability. NTP uses UDP by default, and we wanted to turn off/ACL-off UDP in certain networks. So we grabbed a few CDMA-based time servers off of Ebay, fronted them with our typical Juniper SRX firewalls, and set up clients to use the SRX's as time sources.

After setting up a few devices, this employee suggested, "Hey why don't we set this up on a loopback and anycast it?" I thought about it for a second, something else came up, we moved on, and the suggestion was forgotten (by both of us). We had not finished moving everything from flat, layer 2 networks to a true Clos L3LS setup, so the timing wasn't just right. After finishing the L3LS migration, I looked at this again......and we're very happy with the results.
The idea of BGP anycasting isn't novel. CDN's like BitGravity proved it worked with TCP as well as UDP. It's a well accepted method of distributing traffic across geographic regions on the Internet. But what about inside your own datacenter? It works just as well if not better! The idea is very simple:
  • Set up a service to run on a loopback IP (process should bind to that IP instead of 0.0.0.0 or ::/0)
  • Install bird or another routing daemon
  • Announce loopback IP via routing daemon to your Top-of-Rack (ToR) switch
  • ToR switch announces out to spine via BGP
  • Use native IP routing to find the closest, best, and possibly ECMP route
One nice benefit is that you get built-in load balancing with ECMP! DNS server has too much traffic? Just setup another with the identical loopback and go. ECMP is supported in even the most basic switches now (thanks Broadcom!), so there's no reason why you can't accomplish this even on a budget. No more dedicated appliances or clunky HA-proxy setups. Here are some screenshots to illustrate our process:
Picture
Add a new IP to your loopback
Picture
Run bird (we use daemontools)
Picture
Announce the route! (We use OSPF inside each rack)
Picture
Route as seen by ToR switch
Picture
Route as seen by a spine switch
As you can see, it's not terribly complex in operation. It works extremely well, even across datacenters. Use the same loopback IP value across all of your datacenters for a service (here, NTP) and your clients (and routers) will automatically send you to the closest one. If the service in your DC goes down for whatever reason, you'll be routed to another DC and everything keeps humming along. We do this for stuff like NTP and DNS, as well as services like Postgres and Ceph. Here's a shot of our Ceph RGW's with 3 ECMP routes:
Picture
For simplicity, we chose to use Bird and OSPF as the route advertisement mechanism. Zero knowledge and setup is required there. Just fire up OSPF on 0.0.0.0 as you only run OSPF inside the rack. The ToR switch exports OSPF routes into the BGP L3 Leaf-Spine mesh, but no OSPF is used in the Leaf-Spine level. This cuts down on the administrative overhead required to use BGP inside the rack.

I realize that this post isn't novel or groundbreaking to many, but I wanted to share what we found with everyone.
papajwl
2/15/2018 12:30:34 pm

The simplicity is brilliant. Nice write up.

Diq link
2/19/2018 03:38:57 pm

Thanks! The simplicity was a major (internal) selling point. Routing daemons are mature and stable enough that we feel very comfortable with such a simple setup.

Shadow fight 3 unlimited money link
4/9/2018 10:20:58 pm

Very Nice site thanks a lot Really amazing site I will visit again this lovely site Thanks <3


Comments are closed.

    Author

    A NOLA native just trying to get by. I live in San Francisco and work as a digital plumber for the joint that runs this thing. (Square/Weebly) Thoughts are mine, not my company's.

    Picture
    Moi

    Archives

    May 2021
    April 2021
    March 2021
    February 2021
    December 2019
    April 2019
    July 2018
    February 2018
    January 2017
    August 2016
    May 2016
    June 2015
    May 2015
    December 2014
    July 2014
    March 2014
    December 2013
    November 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    October 2012
    July 2012
    June 2012
    April 2012
    March 2012
    February 2012
    November 2011
    September 2011
    August 2011
    June 2011
    May 2011

    Categories

    All
    Cajun
    Computers And Internet
    Cooking
    Creole
    Food
    Gumbo
    Nola
    Recipe
    Sf

    RSS Feed

Proudly powered by Weebly
  • Home
  • Blog
  • Photos