The access to the market(s) is interesting enough to warrant it’s own page.<\/p>\n
For near exchange work, it’s all about proximity. \u00c2\u00a0If you can get a cross connect to an exchange, \u00c2\u00a0you want to be as physically close to them as you can. \u00c2\u00a0In the same datacenter is a minimum. \u00c2\u00a0On the same floor. \u00c2\u00a0In the same area. \u00c2\u00a0As physically close as possible. \u00c2\u00a0On a trip to a datacenter, one of the jobs is scoping out what’s going on. \u00c2\u00a0That new expansion is where the exchange is going? \u00c2\u00a0You should be able to get access to pick your spot. \u00c2\u00a0All that info is secret, of course, but that doesn’t matter. \u00c2\u00a0You know where the exchange is going, that giant cage with custom power and cooling next to it? \u00c2\u00a0That’s Citadel, I heard. \u00c2\u00a0But you can’t compete with Citadel that way, but you can be the next best spot.<\/p>\n
This is the area where you’re measuring in microseconds. \u00c2\u00a0 The differences here can also be made up in other ways. \u00c2\u00a0Matching your routers to the exchange you’re connecting to. \u00c2\u00a0Will a Cisco or a Juniper have better time? \u00c2\u00a0Tune the algorithm. \u00c2\u00a0Measure, measure, measure.<\/p>\n
If you can’t get a cross connect directly, or you have lines coming in from elsewhere, your line makes a trip to the “meet me room” or some variation on the theme. \u00c2\u00a0That’s where all the customers of the datacenter have their patches (that don’t go directly to someone else), and get connected to other folks. \u00c2\u00a0Your outside line, your other cages maybe, exchanges, whoever it is you have to talk to. \u00c2\u00a0Then somewhere in your cage or cabinet you have a patch panel where you can finally connect it to your equipment.<\/p>\n
Of course, sometimes you get buildings like where the TSX floor is located. \u00c2\u00a0You can’t just connect your equipment on one floor to another floor – that would be too easy. \u00c2\u00a0You have to connect at your floor, it goes all the way down to the meet me room in the basement, and then all the way back up to the other location you’re connecting to. \u00c2\u00a0All at a fee, of course.<\/p><\/blockquote>\n
The datacenter itself<\/h3>\n
The datacenters themselves are amazing in almost every way. \u00c2\u00a0The way they’re not amazing? \u00c2\u00a0Appearances from the outside. \u00c2\u00a0Unmarked. \u00c2\u00a0Actively nondescript. \u00c2\u00a0Could be a warehouse, could be an office building. \u00c2\u00a0Some \u00c2\u00a0are multiple-block-long buildings out in New Jersey. \u00c2\u00a0Some are closets in skyscrapers. \u00c2\u00a0Some are multiple story former printing press buildings. \u00c2\u00a0Some you can almost throw a stone to one of the busiest roads in the world.<\/p>\n
One has a great view of the Statue of Liberty.<\/p><\/blockquote>\n
There tend to be similarities, though. \u00c2\u00a0There are rows and rows of either racks, cages, or a combination of the two. \u00c2\u00a0There’s alternating hot and cold aisles. \u00c2\u00a0Cold air is forced either through the floor or from above via vents into the cold aisle where the servers draw it in. \u00c2\u00a0The hot aisles collect the output from both sides and either take it in or let it go naturally up into an open space. \u00c2\u00a0There are overhead ladders with network wiring that then breaks off and goes into the desired cages. \u00c2\u00a0The wiring for the power is the same story as the cold air. \u00c2\u00a0Either above or below.<\/p>\n
The biggest limiting factor? \u00c2\u00a0Power and a\/c. \u00c2\u00a0You can usually get space, and with that space comes a little bit of power. \u00c2\u00a0How much? \u00c2\u00a0Between 15 and 30 amps for the whole cabinet. \u00c2\u00a0A 42 U space with between 8\u00c2\u00a0 and 18 or so used. \u00c2\u00a0Hope you’re using 220v power for that extra efficiency!<\/p><\/blockquote>\n
Working in a datacenter is, like most things in tech and trading, an exercise in contradictions. \u00c2\u00a0They’re super cold so you’re freezing, but if you’re moving equipment, you’re still sweating. \u00c2\u00a0They’re also humidity controlled, so you’re usually thirsty – but there’s absolutely no liquid allowed inside. \u00c2\u00a0You’re probably hooked up to one of the most advanced networks in the world, but getting online is a pain. \u00c2\u00a0Wifi is strictly prohibited except for very temporary setups for working. \u00c2\u00a0You brought your laptop, but there’s no place to plug it in!<\/p>\n
That’s legit – no place to plug it in. \u00c2\u00a0The cage has a PDU designed for use in cages like this, and with servers and networking equipment. \u00c2\u00a0Like this<\/a>: \u00c2\u00a0If you didn’t have the foresight to order an adapter and\/or your laptop won’t work on the voltage your cage is at – \u00c2\u00a0then you have no power!<\/p><\/blockquote>\n
There’s also no food allowed, but there’s usually a break room where people stash goodies for later. \u00c2\u00a0Bring drinks for there too, and label your bag. \u00c2\u00a0The other problem is snack food does not a meal make, and they usually (on purpose) make these things as far from civilization as possible! \u00c2\u00a0Usually there’s a small group working at the same time, so when you’re not going flat out as fast as you can, you’re sitting and waiting to be useful again. \u00c2\u00a0Of course, after a few trips out you generally just start sneaking candy and drinks in anyway. \u00c2\u00a0“Rules” eh?<\/p>\n
If you’re lucky enough to have a whole cage of whatever size, you don’t have enough power to fill it up anyway. \u00c2\u00a0So make yourself a work area. \u00c2\u00a0Get a folding table and chair, keep tools around, have all the lengths and colors of cabling you use on hand, have a usable power strip and a couple extra ports. \u00c2\u00a0It’s like a cold, loud home away from home!<\/p>\n
Latency<\/h3>\n
When working with two or more exchanges, it’s more about latency than proximity. \u00c2\u00a0 Verizon guarantees 8.5 milliseconds per 1,000 miles – but that’s as the wire lays. \u00c2\u00a0 \u00c2\u00a0It’s 710 miles from Chicago to New York. \u00c2\u00a0What time will your packet make the round trip? \u00c2\u00a0Well, if you have a good line, 18ms is pretty good. \u00c2\u00a0Less than that and you can make a business out of it. These folks<\/a> for example advertises 17.2ms. \u00c2\u00a0There’s rumors about 16.8 and less. \u00c2\u00a0I want to know what the bandwidth along I-80<\/a> is.<\/p>\n
When in a group of industry folks, someone will talk about a new path to NJ they’ve found. \u00c2\u00a0“Does it follow I-80?” “Better!” \u00c2\u00a0But you won’t get anything else out of them.<\/p><\/blockquote>\n
Within metropolitan areas, you still have to find the best paths between places. \u00c2\u00a0A few hundred microseconds are too many to waste.<\/p>\n
What is \u00c2\u00a0dark fiber exactly? \u00c2\u00a0Well, it’s a piece of fiber optic cable that you connect to your equipment at both ends. \u00c2\u00a0It’s literally dark until you “light” it up. There’s no telephone company routers in between to change the pathing or add latency.<\/p><\/blockquote>\n
Measure twice. \u00c2\u00a0Actually, just keep measuring.<\/h3>\n
Everything – everything everything depends on your measurements. \u00c2\u00a0When did we send it, when do they say they got it, when did we get it back. \u00c2\u00a0Does that all match what the router says?<\/p>\n
I used to ask family members what the most important part of the measuring operations was. \u00c2\u00a0No-one guessed knowing the time.<\/p><\/blockquote>\n
You have got to have a good time infrastructure. \u00c2\u00a0NTP is okay, but where are you getting your TP from? \u00c2\u00a0The internet is way too slow to be accurate enough for what we’re measuring here. \u00c2\u00a0You need a network time appliance in every metropolitan area. \u00c2\u00a0Every single datacenter would be awesome. \u00c2\u00a0Meinberg<\/a> is the gold standard. \u00c2\u00a0It’s cool technology. \u00c2\u00a0The other problem with NTP is it’s not accurate enough in practice (in theory it is, but that’s a nice theory). \u00c2\u00a0PTP. \u00c2\u00a0How accurate is accurate enough? \u00c2\u00a0When you’re measuring the time it takes for an update to get into your decision engine until the time it takes to make a decision to measure how good your software is, you need in the nanos. \u00c2\u00a0I’m not saying single-digit nanosecond resolution. \u00c2\u00a0(Yet)<\/p>\n
And throughout the day, things can change. \u00c2\u00a0Your monitoring infrastructure has to keep up not only with the hardware running your network, but the Quality of the Service on it, too. \u00c2\u00a0Things happen. \u00c2\u00a0Fiber gets cut. \u00c2\u00a0The telco re-routes you without any notice. Suddenly you’re getting picked off because you’re a millisecond slower than everyone else.<\/p>\n
It’s better to drop out of the market and take the opportunity cost than to be in and making other people money.<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"
The access to the market(s) is interesting enough to warrant it’s own page. Proximity For near exchange work, it’s all about proximity. \u00c2\u00a0If you can get a cross connect to an exchange, \u00c2\u00a0you want to be as physically close to them as you can. \u00c2\u00a0In the same datacenter is a minimum. \u00c2\u00a0On the same floor. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[22],"tags":[34],"_links":{"self":[{"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=\/wp\/v2\/posts\/107"}],"collection":[{"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=107"}],"version-history":[{"count":6,"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=\/wp\/v2\/posts\/107\/revisions"}],"predecessor-version":[{"id":249,"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=\/wp\/v2\/posts\/107\/revisions\/249"}],"wp:attachment":[{"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=107"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=107"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.imaginarybillboards.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=107"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}