This morning I was invited to give a presentation to the Carlsbad Chamber of Commerce’s Sustainability Committee.  If you are interested, the presentation is located here.  It was a nice meeting with several questions from participants.  Was interesting to try to take a presentation about a data center and change it for people who do not have data centers on the fly.

Thanks to @ecostreaming (http://www.ecostreaming.com) for the invitation.  I am happy to say that I now represent NCTD on this committee, and I am looking forward to actively participating.

Well today is the big move of the network and server equipment into our new infrastructure system.  Not only are we moving servers around, we’re also firing up the new UPS and cooling systems, and wiring the generator into the building power.  It is a dicey day for us, and we kick off at 4:00am.  The team is getting tired – several of the Logicalis and NCTD IT team members were here until 11:00 last night.  The power is scheduled to go offline at 4:00am, and unfortunately we have some mission critical systems that simply have to be live again by 7:00am.  This is a short, and frankly nerve-wracking window.
I am sitting here at the office and it is the quiet before the storm.  3:40am.  T-minus 20 minutes.

4:12am
All systems are powered down, and the power will be cut in a few moments at the main breaker.  They will have just over 2 hours to complete all wiring.

6:37am
Power cut over successful!  The new APC UPS is live – time to bring up the mainframe.  Hard to believe we still say that here.  We have 20 minutes to be live for our Bus Operations team.

6:52am
APC UPS engineer:  “Can you turn that big machine off for a minute?  We need to test the phase on the UPS.”  Oops.  No I cannot.  You will have to shoot me before my hands will push the off button.  First lesson of the day:  test power before you turn on computers.

7:02am
Mainframe operational.  UPS test shows power phasing correctly – crisis averted.

8:13am
The team has a saws-all out and they are cutting away an old cabinet.  Slightly disconcerting.  We have 4 cabinets still up on the old raised floor.  They have to come out, raised floor demolished and removed, new ESD tile laid, and new cabinets installed by 9:00am if we’re to stay on schedule.  Uh.

9:15am
Roel has the ESD tile installed.  APC team moving cabinets in now.  Simply amazing.

11:45am
All server and network equipment has been removed from legacy racks.  The old Mitsubishi UPS is sitting in the hallway feeling sorry for itself.  We’ll be recycling all of the older equipment, and reusing some of the racks at remote locations.

1:31pm
Delivering Pizza to the office for the team.  Some things just never change in the IT business.  Most of the equipment is in the new racks.  Cabling is underway.

5:15pm
All but 10 units put into racks.  The team is making custom cables.  May be done this evening with staging, leaving tomorrow for restart.

8:10pm
Shutting down for the night.  Cable testing starts in the morning.

5:30am
Most people won’t be here until 8:00, but I am enjoying the calm before the storm.

8:00am
Cabling testing and final patching begins.

2:45pm
Starting to bring up servers.

5:48pm
Well there were several bumps in the road, but the environment is successfully up.  Thanks to our vendors Logicalis, Prime, APC, and Roel for helping us transition to the new power and equipment.

8:00am
Well no surprise, we’re having some issues this morning.  But most are manageable, so I feel like we had a successful project.

The LEED saga continues.  I’ve mentioned some of the challenges that we’ve faced, but today presented another interesting one.  I assumed because our purchasing specifications included statements like “systems that comply with the US EPA Energy Star requirements” that we would qualify for the Energy Star points on the LEED rating system.  Today, I learned how naive I am.  In fact, while several vendors have machines that in fact to have an EPA stamp of Energy Star compliance, they are few and far between right now.

Energy Star was a program started initially around residential power use.  As a result, most items that have the Energy Star seal are appliances or electronics in the consumer space.  A check of the EPA website shows fewer than 15 enterprise, server class machines that qualify for the rating.  So while many vendors state that they have Energy Star-compliant equipment, they do not in fact have too many machines that actually went through and successfully completed the process.

Let’s compare some from my data center as examples.  We run an HP shop (this is not an endorsement of their product or a sales pitch, just disclosure that we have them as an architectural standard).  So I have a wide variety of their equipment.  For our Microsoft Exchange upgrade, we installed Energy Star certified HP Proliant DL380 G6 rack-optimized servers.  These are currently the only series of HP machines that have the seal.  For most of my purchasing, however, I prefer HP BL460c G6 or HP BL680c G6 machines that slot into a blade chassis.  What is nice about blades is that they share components like power supplies and fans.  So this reduces the power pull, and reduces the amount of waste in the product.  So from a product life cycle perspective, they are a better choice. 

In spite of our choice to generally rely on the more energy-efficient and therefore more eco-friendly choice of the blade servers, we actually cannot claim the LEED energy star credit because these servers are merely “EnergyStar Compliant” instead of certified.  This is needless-t0-say an unfortunate outcome as we are inching closer to a possible Gold certification and any point that we miss now keeps us from that nearly impossible goal.

I’ve uploaded my APTA presentation about building a sustainable data center to slideshare if anyone is interested.  Clearly NCTD is a pioneer in the transit industry, even if building a sustainable data center is cliche in other industries.  Not one participant in the room had added sustainability as a design criteria when building their data centers.  We’ll see if I made any impression with the community on this issue.

Well today was an interesting meeting with our LEED consultant Brandon Smith.  Based on our meeting, it is clear that specific LEED requirements for data centers do not yet exist.  As a result, we’re pursuing a classification for an Interior Space (for reference, the requirements are here).  There are a few items from this list that simply do not apply to data centers, but are ‘gatekeepers’ that must be addressed in order for us to pursue any certification.

The first of these is that no part of the interior space up for LEED certification can be cooled by a CFC-based air handling system.  Unfortunately our building was constructed years before the non-CFC requirements came into practice and therefore the majority of the facility is in fact in violation of this requirement.  However, we had previously installed two CFC-free air conditioners dedicated for the data center.  Our original plan was to reuse these air conditioners to supplement the cooling in our board room; however, given the LEED requirements we are now going to use one air conditioner for the ambient air handling in the Data Center commercial space, and the other for the Board room.  Had we not possessed these air conditioning units, this would have been the end of pursuing LEED for us.

The second set of requirements that are unforeseen gatekeepers are those around reduction in water use.  Interestingly, we use absolutely no water in the data center as none of our equipment is water-cooled.  We made the mistaken assumption that no water use would be considered a good thing.  We were wrong.  As a result, we are now forced to add in the public shared spaces on the floor for consideration in the commercial space, and then to show not only a reduction down to the required baseline water usage, but then an incremental reduction from the baseline.

Again, this is an older facility, so we have older toilets and water fixtures.  So Mr. Smith is now working on the 5 toilets, 2 urinals, and 5 water faucets we will need to replace to be considered for certification.  I am wondering how the price will impact my overall return on investment calculation.  The cost for the LEED certification itself is approximately $15,000, and now the incremental cost of the water use reduction could be an additional $5,000.   I will refrain from discussing how this is a crappy situation.

To answer this question, I made the decision that logically we would want to spend no more than 1 advertising campaign would cost.  This would be the net cost – in other words, if the LEED portion of this project costs us $20,000 to go through the process, and if our average advertising campaign costs us $5000, then in order to have a $0 incremental cost the investments through LEED would need to save $15,000 through their total lifetime in order to justify the expenditure.  My logic here is that by successfully obtaining an LEED certification for a commercial interior space – the first such certification for NCTD, and for northern San Diego County – I would likely be generating some press and attention through the investment perhaps equivalent to one small marketing push.  The rest of the investment must be justified by some other tangible return.

Brandon and I are working on this question now.  Just for edification, here are the other items we’re considering in order to meet the basic LEED certification requirements:

• Designating some of our parking spaces for carpools or vanpools
• Reusing our interior door from the project instead of purchasing a new one
• Measuring the Solar Reflective Index of the concrete around the building
• Measuring the shade of the trees on the parking lot

To be fair, I understand the need to consider the overall building in this project.   Given that this is a ‘green data center’ that we are building inside of an existing and rather dated facility, the task of addressing all of the needs of LEED may be insurmountable.  Especially considering my desire to also establish the business case for the project beyond the tangible benefits of the data center itself.  In other words – building the case for green instead of conventional data center practice.

Once we have finished the complete ROI analysis, I will post.  Until then, dual flush or low flow?  That is the question…

Now that we’ve faced down the crises from our various storms these last two weeks, the team can again turn attention to the final elements of planning around the NCTD data center project.  Our current data center has an elevated floor and a standard office dropped ceiling; however, the physical constraints of the room prevented installing either of these at their recommended heights.  The raised floor is very shallow, and does not provide enough space to be utilized for air handling.  In fact, the primary original purpose of the raised floor was to allow for piping for the fire suppression system.

The floor has no routing or conduit for cabling, resulting in a haphazard approach to cabling in the room:  some racks have cabling in ad-hoc runs constructed above racks, while other systems have random cables under the floor.  This resulted in a relative rats’ nest of cables discovered as we temporarily relocated some of the systems this month. 

Originally we investigated re-using the raised floor; however, to accomnmodate the growth in equipment we desired to extend the room by 2.5 feet resulting in the need to procure raised flooring for another 50 sq ft.  The original manufacturer does not exist anymore, and the product was not something we were able to find on the open market.  I am frankly happy that we were unable to locate a suitable product to go with this system, as it is simply not ideal for our situation.

This work resulted in the team deciding to remove both the existing ceiling and flooring systems.  Our installation approach now is to instead install electrostatic dissipative (ESD) tiles on the subfloor, and a new dropped ceiling system that will provide for a more logical approach to cabling, air handling, and fire suppression.

My first stop in researching materials was again the US Green Building Council website, which described a program from the Resilient Floor Covering Institute(RFCI) called FloorScore.  Much like the scores provided around sustainable forestry processes, the RFCI FloorScore provides guidelines and a mechanism for an independent assessment (through Scientific Certification Systems) of the volatile organic compound (VOC) emissions from the materials.  Similar to the VOCs from paints, floor and ceiling tiles – and other industrial products – can come with a large VOC emissions load.  From both sustainability and worker comfort perspectives, choosing products with low VOCs are preferable where possible.

In addition to low VOC, we’re requiring that the selected products contain a substantial percentage of recycled or renewable materials.  We also would like to work with a company that will allow us to recycle all of the flooring and ceiling material from the existing room.

Surprisingly, there are now a wide variety of products on the market that will meet all of our needs, from ESD, to material composition, to recycling services.  My original fear that was in requiring the additional environmental criteria, we would be facing a substantial increase in the project cost.  However, that has not proven to be the case.  Instead we’ve discovered a wide array of companies that have embraced the need for these products and who are delivering a variety of choices to the marketplace.

We have selected a flooring product manufactured by Armstrong for our project.  Not only do these products meet all of the above criteria, but one of their manufacturing facilities is within 500 miles of the NCTD project site.

The purpose of this posting isn’t to market the specific product we’ve selected, but to instead point out that it was possible for us to impose additional purchasing criteria on the project, find products that met the criteria, and to do so in a manner that did not substantially increase our costs.

Our purchasing criteria included:

• Must meet all anti-static or static dissipative requirements for a data center
• Must contain at least 15% recycled materials (would prefer higher percentage)
• Must be FloorScore certified
• Desirable to be sourced within 500 miles of the project site
• Must have low VOC emissions

Dig Deeper

There are a wide array of manufacturers who provide flooring and ceiling products designed specifically to address both the needs of a data center and the environmental purchasing criteria we selected.  Rather than providing product links, instead I would encourage that people review the RFCI and USGBC sites for information about establishing the appropriate criteria for your project.

I spoke today with Bob Mobach – one of my technology gurus at Logicalis – about my concerns about the non-green generator.  After much discussion and research, we have again validated our decision to move forward with the generator.  He had some excellent insights about why we are in this position, and when some of the alternative technologies are more appropriate.

NCTD requires a high-availability environment now.  There is no getting around the fact that we run important, mission-critical systems on our infrastructure, and as such we can no longer afford to have unexpected and unplanned outages.  To meet this basic requirement, we must have the capability of creating power on-demand, nearly instantaneously should our power fail.  There simply are few choices on the market today to address this need for a facility of our size.

We could have taken the approach that we would provide our own ‘co-lo’ power via a set of natural gas generators.  This scenario would have allowed us to failover to SDG&E power should the natural gas units fail.  This option does not really make sense for our data center because firstly we’ve invested in solar pv power to offset the power demands, and secondly because our data center power draw is really too small to make this economical.  So from a business case, this is a poor choice for our scenario.

Another option was the fuel-cell based UPS.  The problem with this approach is simply the cost and the durability of the power.  We have had now two outages this year over 6 hours in duration.  There is simply no way we could sustain the entire data center on the fuel cells for this long.  And again, the business case is not there.

We are now investigating alternative fuel sources for the Kohler generator – either ‘clean diesel’ or biofuels.  While the biodiesel seems like a natural choice, the biggest issue is the long-term storage of the fuel.  Fuel that sits around for months can degrade due to algae or condensation.  Research indicates that not many entities are yet using biodiesel as their fuel choice for standby scenarios where the generator is not going to be used frequently.  So my next step is to contact the manufacturer to determine whether a biodiesel mix is appropriate in the generator, and what they would believe the ideal mix to contain.  We will then compare this to the emissions, performance, and price of ‘clean diesel.’

So while I am still not 100% reconciled that installing a generator is moving us in the right direction for a green data center, I am convinced that we’ve made the business decision which balances our environmental needs with our business realities.

This week we started investigating the LEED certification process and it was illuminating. For my data center project we only have two options for obtaining the certification- and both are frankly uphill battles.

The first option is for an interior commercial space. Our LEED consultant Brandon Smith believes we may be able to qualify under this program at a basic certified level. Our data center is part of a shared use space in the basement of our building. We lease out the third floor of our facility to at least three other companies. We also share a lunch room in the basement, and lease out another office in the basement to the same companies.

The data center is a separately-accessed space that is one small office on it’s own. I think this is our best shot at certification. Mr. Smith believes our project will garner 47 of the 100 possible points, which would put us right at the certified level.

Our other option is to pursue an Existing Facility Operations and Mainentance certification. This is a more rigorous review that would require us to change out our old whole-building air conditioning units. I cannot see how I can justify this large capital investment on the data center project alone.

We will investigate the possibility of the second certification as part of our larger sustainability plan.

There were some surprises in this evaluation:

• The rather large solar investment in the project- enough to offset 100% of the power needs of the data center – garnered us all of 1 point in the evaluation. The same as reusing our door.
• We received the bulk of our points because of the location of the site and the nature of our business. Being a public transit district and the free transit we offer out employees brought us 6 points. The thought is that commuters will generate a much larger carbon footprint than that of the data center itself.
• We receive no points for reusing the air conditioners. Even though they will be used infrequently in another part of the building, they clearly still pull a great deal of power.

Pursuing the second certification will require significant investment on our part- from changing the toilets and water fixtures, to the cleaning products we use, to changes in out parking lot. It will be an interesting process to determine whether these investments will make sense to our financially strapped organization.

And that brings us to the question of ROI. The certification process will probably cost around $7500 with fees and consulting hours. This is a very small cost on the overall project, but is equivalent to purchasing 3 servers. Will it return as much as 3 servers would? Is there a monetary value to LEED?

One focus of our data center design activities this week has been around the question of backup power. Any high-availability data center must face this question, and the reality is that there are not too many options to have a green approach to backup power.

The first consideration is the Uninterruptible Power Supply (UPS). We looked at three options for providing the immediate, on-demand UPS capability: traditional battery UPS, flywheel UPS, and Fuel Cell UPS.

The flywheel UPS seemed to provide the greenest option; however, it was not available to integrate with our chosen ‘pod’ manufacturer. This leaves us with the fuel cell or the battery options. While the Fuel Cell is a solid option for the future, I felt that the cost/benefit was not yet there for us. Which leaves us with a traditional battery for our UPS. Not the greenest option overall, but when you factor in the full APC Infrastruxure solution, we still come out ahead of the game.

The second major decision was on backup power to support the UPS for extended potential power failures. Most SMB data centers do not really face this issue. However, in the last two years of my tenure, we have had no fewer than three extended power outages that have taken down my data center for over 8 hours. Now that we’re running the SPRINTER train and the fare collection systems using the network, extended downtime is simply not an option.

We are therefore going to install an onsite generator. I initially was not happy with this idea because clearly a diesel generator is simply not green. We were initially pursuing a natural gas generator under the assumption that this would be a more environmentally-friendly option. However, in working with our generator installation firm (Bay City Electric), it became clear that natural gas would not be an option for us due to potential issues from earthquakes interrupting the supply lines.

So we’re going with the traditional diesel generator unless some other option presents itself. Our strategy will be to mitigate as many environmental concerns as possible through process:

• Whenever possible, we will test the generator under load -meaning that we will run the data center 100% on the power the generator creates. This means that we will not be wasting the fuel.
• We will install filtration on the emissions to ensure that we’re not discharging significant particulates
• We will test as infrequently as possible to validate that the system in functional

I spent hours working with our IT firm Logicalis to try to find alternatives to these two decisions. I have to say that I am disappointed that there were not other options readily available in the market place for us. In reviewing some of the most successful green data center project case studies from the last two years, the preponderance of them make reference to their use of generators for backup power. Even my green-web hosting firm AISO.net talks about their use of generators in times of need.

While we are making great strides on designing our green data center, I have to admit that I was disheartened with these two decisions I made in the design; however, I did not feel I had the business case, other case studies, nor the viable alternatives that would lead us down a different path.

by Angela Miller
One of the biggest issues for my little data center at NCTD is handling the heat load of the room.  I was reading an article online at the Georgia Institute of Technology that said that cooling the data center has become more complicated as the average heat load per cabinet has moved from  1-5 Kilowatts of heat to 28 in the last 5 years.  We can easily see this in our room – we have a cabinet with 10 rack-optimized HP DL360 servers each with a 1 unit space between for air flow sitting next to another rack with a C3000 8-server blade chassis and a SAN sitting next to a rack with a C7000 16-server chassis with the Cisco VOIP equipment and no spaces between servers.  This little example shows how in just 5 years the density of the typical server rack in our room has increased immensely.

We also have first-hand experience with the problems this increased heat load can cause for the facility itself.  While we have a raised floor in the room, it was designed primarily for cable management instead of heat mitigation.  So the floor tiles are solid and do not allow the cooler air to be funneled through to the cabinets.  About 3 years ago the heat in the room exceeded the capacity of the air conditioners resulting in both a flood of condensation in the room and a blown ac that took down the data center.

I have previously blogged about our current poor cooling solution installed as a result of this outage – two residential-class air handling units on the floor of the data center with a fabricated venting system designed to pull in the hot air from the floor (?) and push the cold air from the top vents directed at various angles throughout the room.  This system makes it extremely uncomfortable for my staff to work — and who can blame them?  I myself moved one of the vents just an inch higher so I could stand in the room for an hour and forgot to move it back.  This one little adjustment resulted in the temperature in the racks increasing on average 3 degrees while the vent was moved.

Clearly this is not a sustainable solution.  So as we embark on the redesign of the data center, cooling solutions have been front-and-center in the conversation.  The Green Grid has published seven steps to consider when designing a cooling solution for the green data center:

1. Developing an air management strategy
2. Moving cooling systems closer to the load
3. Operating at a higher delta-T
4. Installing economizers
5. Using higher-specifi cation and performance equipment
6. Using dynamic controls
7. Maintaining higher operating temperatures

We kept these guidelines in mind when evaluating the options for cooling the data center.  We have looked at a variety of cooling solutions for the facility, but there are several things which make it difficult to be innovative in this space.  The first is that this room is in the basement of a former bank building.  We are strictly limited on the height and footprint of the room.  Therefore we cannot be more creative with our raised floor – it is only 8 inches high, but going higher to allow for venting under the floor is not in the cards.

We also must contend with walls that support vaults on two floors above the data center, limiting what we can do with the venting and air handling outside of the room.  Given these constraints, the recommendation by Logicalis and Roel was to install a pod system.  This approach will allow us to encapsulate the racks, create hot and cold zones, and provide in-line cooling for the racks right where the need is.  This is not ideal for every data center – for example, we almost were unable to use this solution because the footprint of the room was 1 foot too short for the necessary clearance around the pod.  Fortunately, we were able to recapture some space by moving an internal wall out slightly allowing us to just fit the equipment into the design.

It is also important to understand the tradeoff with an encapsulated system like the APC pods:  once it is in within the walls of this room, I will not have the ability to grow the data center past this size.  This will be the finite number of racks we can install in the foreseeable future.  I cannot move walls again, nor can I migrate to a new space within this building.  So we must be smart in the design phase in order to get us a full ten-year investment and growth opportunity in this space.  Choosing a pod also increases the budget versus sticking with the current approach of open racks in the space.  But given the other design criteria, the pod solution is the clear winner on energy efficiency and heat handling.

While this is the basic design we’ve elected to pursue, I also requested that we begin by first performing an air flow analysis of the current facility, both with the air conditioners running and without.  Such a study might reveal some interesting design criteria for us to keep in mind as we move forward.  I have a feeling that we might find we have significant bypass airflow issues to deal with (basically this means air that is infiltrating the room through gaps and openings in the walls).  Our initial monitoring of the humidity in the room using simple environmental monitors shows that we actually have a moisture problem in the room in addition to the cooling issues.

I will post more as we get into this project so that you can see the practical realities of how we make decisions that consider the energy efficiency, sustainability, and practical design considerations through the project.

Digg Deeper on the Issues:

I relied on the following sites for this post:

The Green Grid
American Power Conversion
Georgia Institute of Technology

None of the entities in this post has provided compensation or incentive to discuss their products or services.