2 Located atop Mt. Hamilton and commencing operations in 1888, Lick Observatory was UC’s first major scientific research facility. It is now overseen by the University of California Observatories (or UCO) which is headquartered on the UC Santa Cruz campus. The Observatory’s connection to the Internet is jointly managed by UCO and the Information Technology Services department of the UCSC campus.
3 Lick’s four main telescopes are actively used for research nearly every clear night. Lick is a systemwide facility that serves astronomers, researchers, and students throughout the UC system.
4 Most astronomical observations performed with the Lick Telescopes are now conducted remotely over the Internet from the observer’s home campus. Remote Observing Control rooms have been established at eight of the ten UC campuses, at CSU San Luis Obispo, and at Seoul National University in South Korea. These rooms are equipped with multi-screen interactive displays that mimic those located in the control rooms on Mt. Hamilton. A multi-point videoconferencing system enables the observers on campus to coordinate with the operations staff and support astronomers at the telescopes
5 Although Lick is only about 20 miles east of downtown San Jose, high speed connectivity to the CalREN network is not readily available via commercial providers. Starting in the late 1980s, a single T1 circuit connected Lick to the UCSC campus and provided both campus telephone service and low-speed network connectivity. Over time, 3 more T1 circuits were added, increasing Lick’s Internet bandwidth to about 5 Mbps. Given its mountaintop location and the constraints imposed by the local topography and vegetation, Lick has a clear line of sight -> to a significant fraction of the Silicon Valley. That would make Lick a good site for a high-speed microwave link.
6 The Blue Oak Ranch Reserve (or BORR) was added to UC’s Natural Reserve System in 2007. That system consists of 39 wildland sites, located throughout California. BORR spans 3,280 acres of oak woodlands located along the northwest slope of Mt. Hamilton and serves as a biological field station and ecological reserve
7 Installation of facilities commenced in 2008. -> A series of small cabins, provide lodging for visitors. -> The Cedar Barn (show in the foreground) houses labs, classrooms, and offices used by visiting students and researchers who are involved in on-site classes and who conduct field research at the reserve. Because of its isolated location, BORR has hardly any access to external utilities. Instead, it generates its own “off the grid” -> solar power, backed up by a bank of batteries and a propane generator. [Large tanks provide storage for on-site well-water.]
8 BORR operates a state-of-the art, solar-powered distributed, sensor and camera network. At various locations throughout the reserve it monitors weather and environmental conditions as well as the movements of wildlife.
9 Although it is only about 10 miles east of downtown San Jose, because of the foothills to its west, most of BORR lacks any line-of-sight to Silicon Valley. However, many parts of BORR have a clear line-of-sight to the the Lick facilities on Mt. Hamilton, as can be seen in the next slide
10 Thus, Lick could potentially serve as a relay station for BORR.
11 Accordingly, Lick was an obvious choice for connecting BORR to CalREN. And despite the limited bandwidth of our T1 circuits, Lick agreed to be BORR’s ISP. We provided them a slice of that limited bandwidth and allowed them to establish a modest 5 GHz microwave link between BORR and Lick. It utilizes small flat-panel antennas and now provides a bandwidth of about 50 Mbps. However, from the start it was clear that Lick’s T1-based connection to the UCSC campus was inadequate to meet the growing bandwidth needs of either Lick or BORR and that a better solution was needed.
12 Due to their limited bandwidth and reliability, adding more T1 circuits would not have been practical nor cost-effective. Installing a fiber link between Mt. Hamilton and the closest connection point in San Jose would cost over $1 million while providing only a single, non-diverse path. -> Also, fiber links can be severed in many ways, such as by fallen trees, landslides, or road washouts. However, deploying dual microwave links would cost less than $100 thousand and provide both redundancy and frequency diversity. But first, we would need to secure capital funding and find a downlink site.
13 In 2010, the UC Natural Reserve System applied for a major NSF cyber-infrastructure upgrade grant to improve connectivity to many of its reserves, including BORR. As part of that application, Lick and BORR proposed a high-speed microwave link between Lick and a CalREN-connected site in the Santa Clara Valley. The proposed link would provide 250 Mbps of bandwidth that Lick and BORR would share. That grant was approved in 2011 and funded -> via the American Recovery and Reinvestment Act of 2009 (a.k.a. “stimulus funds”). That funding included a “Buy American” requirement which constrained our of choice equipment vendors.
14 For the downlink site, we selected a 3-story office building in Santa Clara that was already being leased by UC Santa Cruz ’s University Extension Program (a.k.a. UCSC UNEX). That lease agreement included a rooftop use clause and the building’s rooftop provided a clear line of sight to Lick. The lease was scheduled to run through the year 2029. Accordingly, it should have provided a stable home for our downlink antennas for the next 17 years, and perhaps longer. A Gigaman fiber link already connected this building to the CENIC hub in Sunnyvale and UCSC ITS had an established a network presence. The server room provided UPS backup power and a conditioned space for our radio and network gear. [The path length from this site to Lick was an acceptable 18.6 miles. Thus, this building met our requirements.]
15 In 2012 we used the NSF funding to install an “American-made” high-speed primary link that operated in the licensed 11 GHz band. That link provided a solid 250 Mbps with extremely low latency and packet loss. It employed Trango Systems ApexPlus radios and 4-foot parabolic dish antennas. The antenna atop the UCSC UNEX building was mounted to a non-penetrating rooftop mount. -> The antenna at the Lick endpoint was attached to a pipe mount assembly bolted to the 10-inch-thick concrete ring wall of the Shane Telescope Dome.
16 To provide redundancy and frequency diversity, we installed an inexpensive, moderate-speed, secondary link that operated in the unlicensed 5 GHz band. This link is only used if the primary 11 GHz link goes offline. To avoid the “Buy American” restriction imposed by the ARRA funding, we used UCO funds for the secondary link. While it exhibited somewhat variable bandwidth, packet loss, and latency, it still provided a minimum bandwidth of about 25 Mbps and low susceptibility to rain fade. It employed Ubiquiti Rocket M5 radios and light-weight 2-foot parabolic dish antennas, which were simply -> clamped to existing structures at each endpoint (UNEX at left, Lick at right) and thus easy to relocate if needed.
17 Both links became fully operational in June 2012. For the next several years, the primary 11 GHz microwave link was extremely reliable. It operated at full bandwidth even during the harsh winter weather at Lick, including heavy rain, snow, ice storms, and winds gusting as high as 93 MPH. Both Lick and BORR were very satisfied with its performance. All was well until we learned that the rug was soon going to pulled out from under our feet … or, more precisely… that the UCSC UNEX Building was going to be demolished.
18 In September 2014 a major real estate development company purchased the UCSC UNEX Building, along with ALL of the dozens of other buildings on both sides of a half-mile long stretch of Augustine Drive in Santa Clara. The existing two and three-story office buildings were to be demolished and replaced by a combination of 6 to 8 story office buildings and a shopping center. This architect’s conceptual drawing shows what a portion of redevelopment area would look like when Phase 1 of the project is completed in 2016. -> The UCSC UNEX building was located near the center of the shaded portion of that drawing, which shows the buildings slated for demolition and replacement during Phases 2 and 3 of the project.
19 The development company planned to terminate all leases for the existing buildings and to induce the tenants to move out promptly. Hence, the UCSC UNEX program urgently needed to find a new building for its classrooms, offices, and labs, while UCO needed to find a new downlink site for its rooftop-mounted antennas. The pale yellow rectangles on this diagram show the proposed location of the new buildings superimposed over an aerial view of the existing buildings. The dashed red line indicates the construction fencing that would surround the -> UCSC UNEX Building during Phase 2. New Buildings -> 3, 4, and 5 -> would eventually block the line of sight to Lick, with construction of the latter two slated to commence near the end of 2015.
20 In the meantime, both UCSC UNEX and UCO needed to implement mitigations to enable their operations to continue. For UCSC UNEX, this meant mitigating noise from nearby demolition and construction activities by installing transparent sound-absorbing plastic blankets over the building’s windows. For UCO, this meant finding ways to keep its microwave links operating despite obstruction of the microwave beam lines by construction equipment and the erection of 6 story buildings nearby
21 In late 2015, a large crane arrived on site, and either it or its payload would sometimes obstruct the microwave beam lines. Fortunately, we were able to get the construction company to agree to park their crane out of the antenna beam lines at the end of each workday. This ensured that the primary 11 GHz link would not be obstructed at night when our telescopes would be operating. This mid-January 2016 photo shows that much of the framework for the -> first 4 floors of Building 4 was now in place. Once construction commenced on the 6th floor of this building, the antenna’s line of site to Mt. Hamilton would be blocked.
22 When the 11 and 5 GHz antennas were first installed on the rooftop in 2012, they were placed side-by-side to comply with the terms of the rooftop-use clause in the lease agreement. As a result, the beam lines from our two antennas were very close together. Both beams lines could be obstructed at the same time by the crane and -> both would be blocked once the steel framework for Building 4 neared completion.
23 To address that problem, we persuaded the new owner to amend the lease so that we could increase the horizontal separation between our two antennas. We proposed moving one of the two antennas 100 feet towards the north end of the building. For several reasons, moving the 11 GHz 4-foot dish antenna proved impractical while moving the smaller and lighter-weight 5 GHz 2-foot dish was quite simple. After the move, the resulting 5 GHz beam line (shown by the purple arrow) would no longer be at risk of being obstructed by -> Buildings 4 and 5, and construction of -> Building 3 was not slated to begin until after the UCSC UNEX Building had been vacated and demolished. This mitigation strategy enabled us to maintain connectivity to Lick until we were finally able to vacate the UCSC UNEX rooftop in March 2016.
24 In addition to developing such mitigation strategies, UCO continued to look for a new downlink site, giving priority to buildings that were owned by their occupants. We also gave preference to buildings occupied by long-term government entities that would be unlikely to move any time soon. We made extensive use of Google Earth to compare each building’s line of sight to Lick to determine the risk that it might become blocked by future construction. Ideally, the portion of that line of sight closest to the building should traverse land on which there would be little likelihood of tall buildings or structures being built. For example, tall buildings would be unlikely to be built in the middle of a major highway or a regional park. In this image, the -> yellow line shows the line-of-sight to Lick from the UCSC UNEX Building, while the -> shorter green line shows the line-of-sight from the new building ultimately selected. [The runways from San Jose Airport can be seen at right.]
25 While Google Earth provided us an initial estimate of a building’s line of sight to Lick, neither it, nor similar web-based elevation-profile tools from microwave vendors like Solwise and Trango, accounts for vegetation. For that, we relied on photographic verification of the beam line as viewed from the proposed downlink antenna location. This telephoto image taken in March 2015 shows the line of sight to the Shane Telescope Dome at Lick as viewed from Independence High School in San Jose, located 12.7 miles away. In this photo we were able to identify various features, including the ->11 GHz 4-foot parabolic dish mounted on the west face of that dome. [To obtain photos of sufficient clarity across such distances is challenging due to air turbulence.]
26 By mid-2015, UCO had identified the Santa Clara County Office Education’s building as our top choice for a replacement site for our downlink antennas. As a K12 institution, SCCOE is connected to CENIC and they have an excellent IT and networking support staff. And they own their building.
27 Its rooftop is easily accessible and it has some existing antenna masts on which space could be made available for our antennas.
28 A nearby data closet inside the building’s top floor had rack space available for mounting our radio and networking gear. That closet is -> supplied with an E-power circuit connected to a large UPS backed up by a diesel generator.
29 The -> SCCOE building is located in North San Jose near the -> E. Brokaw Rd. exit from -> Interstate 880, not far from -> the tree-lined Coyote Creek. In this overhead view, the -> line of sight to Lick (shown in yellow) aligns with E. Brokaw Road for the first quarter mile, at which point -> the beam lines are already 120 feet above ground level. They soon pass over a well-established neighborhood of single-family homes. Accordingly, there is a relatively low probability of a tall building or structure being erected in the future that would block the beam lines from our antennas. However, we dodged a bullet last month when heavy rains caused -> severe flooding along Coyote Creek in southeast San Jose, leading to the evacuation of 14,000 residents. Fortunately, the portion of the Creek near SCCOE did not flood
30 In late 2015, UCO and SCCOE completed a rooftop-use license agreement and we began installation of cabling to support our radios. Without disturbing the existing links still operating between Lick and the UCSC UNEX Building, we installed a new 5 GHz link between Lick and SCCOE, using 3-foot dish antennas and Ubiquiti AF-5X radios. In the meantime, staff from UCSC ITS coordinated with SCCOE IT staff on the installation of networking gear and fiber circuits inside the SCCOE building. By late February 2016, we were able to route test traffic from the UCSC campus to Lick via the new 5 GHz link between SCCOE and Lick.
31 Meanwhile, construction continued near the UCSC UNEX Building. By early February 2016, the steel framework for Building 4 was mostly complete -> and was now blocking the visual line of sight for our 11 GHz link, as seen in this photo. We had expected such blockage would render the 11 GHz link completely inoperative and were greatly surprised and puzzled when it did not. Although the signal strength was significantly reduced, the link continued to carry traffic reliably, albeit at a somewhat reduced bandwidth. Our theory is that the metal flooring of the new building constituted a set of reflective, evenly-spaced horizontal plates that acted as a wave guide to convey the 11 GHz signal through the open steel framework. Even in this degraded state, the 11 GHz link was able to carry operational traffic for Lick and BORR until mid-March, when such traffic was re-routed onto the new 5 GHz link at SCCOE.
32 At that point, both microwave links at the UCSC UNEX endpoint were shut down and -> our equipment removed from the building. The 11 GHz -> radio and antenna were transported to the SCCOE Building, -> installed, and co-aligned with the antenna at Lick ->. That completed our move to SCCOE. By late March 2016, the router configurations were finalized and both the 5 and 11 GHz links between Lick and SCCOE were fully operational. Over the past year, both microwave links have performed extremely well, with no unscheduled downtime on the primary link thus far, despite this year’s very harsh winter weather.
33 George Peek of UCSC ITS will now describe how these microwave links are interfaced to CalREN What you see here is our network topology of how Mt. Hamilton and BORR are able to reach UCSC and the internet via UCSC's border routers. In addition to the private circuit connected by the carrier, we have two GRE tunnels terminated to both cores going over the internet. At SCCOE we run a pair of Cisco switches in a stack config and connect our private circuit and tunneled circuit as well as the radios in a redundant manner so if we lost one switch, Mt. Hamilton remains operational. On the mountain, the radios are connected to a Cisco switch at the Shane Dome, and through firewalls, able to reach the Main Building which has another wireless link to reach BORR.
34 When the radios were being moved to SCCOE we had no way to extend them to UCSC without a private ethernet circuit. It seems every time you order a circuit in the Bay Area, you rarely get it installed on time. So, SCCOE was kind enough to work with us and allow us to tunnel our traffic across their internet pipe using GRE
35 We ran into an interesting problem where certain larger websites would not load over GRE tunnels and finally narrowed it down to using proper MTU and TCP configuration settings under the tunnel interface config. If you plan to use GRE in the future, you will need these. -> Another best practice was to use BFD, bidirectional forwarding detection, along with OSPF for dynamic routing, in the event your wireless link was not stable it would failover much faster. -> For troubleshooting performance links we used iPerf (can be installed by itself or deployed part of PerfSonar), installed on a server at SCCOE, which helped us pinpoint any performance issues on the GRE tunnel, the ATT circuit, and the wireless radios to Mt. Hamilton.
36 ->The 5 GHz secondary link let us maintain connectivity to Lick and BORR during the UCSC UNEX construction and enabled a phased relocation to SCCOE. ->Overall, while the technical challenges of the move were significant, the legal, regulatory, and administrative challenges were the most difficult, risky, and time consuming. For example: amending the lease agreement so that we could move the 5 GHz link to the north end of the building took two months. Actually moving the link took less than a day. ->As we learned the hard way, planning any microwave link that traverses a city skyline requires one to think carefully about how that skyline might change over time and how best to cope with such changes if and when they occur. ->Finally, this endeavor shows how CENIC members can forge effective collaborations that achieve practical solutions to difficult problems.
37 The individuals and their respective organizations listed here all made significant contributions to this success of this effort. Any questions? [After leaving this slide up for 60 seconds, advance to the final “Author Contact Information” slide.]