Today the data center project paid off big time. Yes, for those in Southern California it was a near disaster- we had an extensive power outage that reached from Mexico to Los Angeles, and from San Diego to Arizona.

The good news for my team- the data center stayed up 100% of the time. In fact, we lost only 1 rail communications cabinet during 10 hours of an outage.

This means that our phones were up, even when cell phones were not. Our Internet was online and we could communicate with our customers. Our emergency operations center (EOC) was live and effective. I was proud to see NCTD keep bus and rail service in play hours into an outage that was so significant for the region.

More than anything, I know that without the data center project, we would not have achieved this success.

Not everything was smooth and I don’t want to overstate the positives… but I am confident without our green data center, our EOC team would have struggled that much more. So here is to investment in technology and the ability to see a ROI.

This morning I was invited to give a presentation to the Carlsbad Chamber of Commerce’s Sustainability Committee.  If you are interested, the presentation is located here.  It was a nice meeting with several questions from participants.  Was interesting to try to take a presentation about a data center and change it for people who do not have data centers on the fly.

Thanks to @ecostreaming (http://www.ecostreaming.com) for the invitation.  I am happy to say that I now represent NCTD on this committee, and I am looking forward to actively participating.

The LEED saga continues.  I’ve mentioned some of the challenges that we’ve faced, but today presented another interesting one.  I assumed because our purchasing specifications included statements like “systems that comply with the US EPA Energy Star requirements” that we would qualify for the Energy Star points on the LEED rating system.  Today, I learned how naive I am.  In fact, while several vendors have machines that in fact to have an EPA stamp of Energy Star compliance, they are few and far between right now.

Energy Star was a program started initially around residential power use.  As a result, most items that have the Energy Star seal are appliances or electronics in the consumer space.  A check of the EPA website shows fewer than 15 enterprise, server class machines that qualify for the rating.  So while many vendors state that they have Energy Star-compliant equipment, they do not in fact have too many machines that actually went through and successfully completed the process.

Let’s compare some from my data center as examples.  We run an HP shop (this is not an endorsement of their product or a sales pitch, just disclosure that we have them as an architectural standard).  So I have a wide variety of their equipment.  For our Microsoft Exchange upgrade, we installed Energy Star certified HP Proliant DL380 G6 rack-optimized servers.  These are currently the only series of HP machines that have the seal.  For most of my purchasing, however, I prefer HP BL460c G6 or HP BL680c G6 machines that slot into a blade chassis.  What is nice about blades is that they share components like power supplies and fans.  So this reduces the power pull, and reduces the amount of waste in the product.  So from a product life cycle perspective, they are a better choice. 

In spite of our choice to generally rely on the more energy-efficient and therefore more eco-friendly choice of the blade servers, we actually cannot claim the LEED energy star credit because these servers are merely “EnergyStar Compliant” instead of certified.  This is needless-t0-say an unfortunate outcome as we are inching closer to a possible Gold certification and any point that we miss now keeps us from that nearly impossible goal.

I’ve uploaded my APTA presentation about building a sustainable data center to slideshare if anyone is interested.  Clearly NCTD is a pioneer in the transit industry, even if building a sustainable data center is cliche in other industries.  Not one participant in the room had added sustainability as a design criteria when building their data centers.  We’ll see if I made any impression with the community on this issue.

Well today was an interesting meeting with our LEED consultant Brandon Smith.  Based on our meeting, it is clear that specific LEED requirements for data centers do not yet exist.  As a result, we’re pursuing a classification for an Interior Space (for reference, the requirements are here).  There are a few items from this list that simply do not apply to data centers, but are ‘gatekeepers’ that must be addressed in order for us to pursue any certification.

The first of these is that no part of the interior space up for LEED certification can be cooled by a CFC-based air handling system.  Unfortunately our building was constructed years before the non-CFC requirements came into practice and therefore the majority of the facility is in fact in violation of this requirement.  However, we had previously installed two CFC-free air conditioners dedicated for the data center.  Our original plan was to reuse these air conditioners to supplement the cooling in our board room; however, given the LEED requirements we are now going to use one air conditioner for the ambient air handling in the Data Center commercial space, and the other for the Board room.  Had we not possessed these air conditioning units, this would have been the end of pursuing LEED for us.

The second set of requirements that are unforeseen gatekeepers are those around reduction in water use.  Interestingly, we use absolutely no water in the data center as none of our equipment is water-cooled.  We made the mistaken assumption that no water use would be considered a good thing.  We were wrong.  As a result, we are now forced to add in the public shared spaces on the floor for consideration in the commercial space, and then to show not only a reduction down to the required baseline water usage, but then an incremental reduction from the baseline.

Again, this is an older facility, so we have older toilets and water fixtures.  So Mr. Smith is now working on the 5 toilets, 2 urinals, and 5 water faucets we will need to replace to be considered for certification.  I am wondering how the price will impact my overall return on investment calculation.  The cost for the LEED certification itself is approximately $15,000, and now the incremental cost of the water use reduction could be an additional $5,000.   I will refrain from discussing how this is a crappy situation.

To answer this question, I made the decision that logically we would want to spend no more than 1 advertising campaign would cost.  This would be the net cost – in other words, if the LEED portion of this project costs us $20,000 to go through the process, and if our average advertising campaign costs us $5000, then in order to have a $0 incremental cost the investments through LEED would need to save $15,000 through their total lifetime in order to justify the expenditure.  My logic here is that by successfully obtaining an LEED certification for a commercial interior space – the first such certification for NCTD, and for northern San Diego County – I would likely be generating some press and attention through the investment perhaps equivalent to one small marketing push.  The rest of the investment must be justified by some other tangible return.

Brandon and I are working on this question now.  Just for edification, here are the other items we’re considering in order to meet the basic LEED certification requirements:

• Designating some of our parking spaces for carpools or vanpools
• Reusing our interior door from the project instead of purchasing a new one
• Measuring the Solar Reflective Index of the concrete around the building
• Measuring the shade of the trees on the parking lot

To be fair, I understand the need to consider the overall building in this project.   Given that this is a ‘green data center’ that we are building inside of an existing and rather dated facility, the task of addressing all of the needs of LEED may be insurmountable.  Especially considering my desire to also establish the business case for the project beyond the tangible benefits of the data center itself.  In other words – building the case for green instead of conventional data center practice.

Once we have finished the complete ROI analysis, I will post.  Until then, dual flush or low flow?  That is the question…

Now that we’ve faced down the crises from our various storms these last two weeks, the team can again turn attention to the final elements of planning around the NCTD data center project.  Our current data center has an elevated floor and a standard office dropped ceiling; however, the physical constraints of the room prevented installing either of these at their recommended heights.  The raised floor is very shallow, and does not provide enough space to be utilized for air handling.  In fact, the primary original purpose of the raised floor was to allow for piping for the fire suppression system.

The floor has no routing or conduit for cabling, resulting in a haphazard approach to cabling in the room:  some racks have cabling in ad-hoc runs constructed above racks, while other systems have random cables under the floor.  This resulted in a relative rats’ nest of cables discovered as we temporarily relocated some of the systems this month. 

Originally we investigated re-using the raised floor; however, to accomnmodate the growth in equipment we desired to extend the room by 2.5 feet resulting in the need to procure raised flooring for another 50 sq ft.  The original manufacturer does not exist anymore, and the product was not something we were able to find on the open market.  I am frankly happy that we were unable to locate a suitable product to go with this system, as it is simply not ideal for our situation.

This work resulted in the team deciding to remove both the existing ceiling and flooring systems.  Our installation approach now is to instead install electrostatic dissipative (ESD) tiles on the subfloor, and a new dropped ceiling system that will provide for a more logical approach to cabling, air handling, and fire suppression.

My first stop in researching materials was again the US Green Building Council website, which described a program from the Resilient Floor Covering Institute(RFCI) called FloorScore.  Much like the scores provided around sustainable forestry processes, the RFCI FloorScore provides guidelines and a mechanism for an independent assessment (through Scientific Certification Systems) of the volatile organic compound (VOC) emissions from the materials.  Similar to the VOCs from paints, floor and ceiling tiles – and other industrial products – can come with a large VOC emissions load.  From both sustainability and worker comfort perspectives, choosing products with low VOCs are preferable where possible.

In addition to low VOC, we’re requiring that the selected products contain a substantial percentage of recycled or renewable materials.  We also would like to work with a company that will allow us to recycle all of the flooring and ceiling material from the existing room.

Surprisingly, there are now a wide variety of products on the market that will meet all of our needs, from ESD, to material composition, to recycling services.  My original fear that was in requiring the additional environmental criteria, we would be facing a substantial increase in the project cost.  However, that has not proven to be the case.  Instead we’ve discovered a wide array of companies that have embraced the need for these products and who are delivering a variety of choices to the marketplace.

We have selected a flooring product manufactured by Armstrong for our project.  Not only do these products meet all of the above criteria, but one of their manufacturing facilities is within 500 miles of the NCTD project site.

The purpose of this posting isn’t to market the specific product we’ve selected, but to instead point out that it was possible for us to impose additional purchasing criteria on the project, find products that met the criteria, and to do so in a manner that did not substantially increase our costs.

Our purchasing criteria included:

• Must meet all anti-static or static dissipative requirements for a data center
• Must contain at least 15% recycled materials (would prefer higher percentage)
• Must be FloorScore certified
• Desirable to be sourced within 500 miles of the project site
• Must have low VOC emissions

Dig Deeper

There are a wide array of manufacturers who provide flooring and ceiling products designed specifically to address both the needs of a data center and the environmental purchasing criteria we selected.  Rather than providing product links, instead I would encourage that people review the RFCI and USGBC sites for information about establishing the appropriate criteria for your project.

This week we started investigating the LEED certification process and it was illuminating. For my data center project we only have two options for obtaining the certification- and both are frankly uphill battles.

The first option is for an interior commercial space. Our LEED consultant Brandon Smith believes we may be able to qualify under this program at a basic certified level. Our data center is part of a shared use space in the basement of our building. We lease out the third floor of our facility to at least three other companies. We also share a lunch room in the basement, and lease out another office in the basement to the same companies.

The data center is a separately-accessed space that is one small office on it’s own. I think this is our best shot at certification. Mr. Smith believes our project will garner 47 of the 100 possible points, which would put us right at the certified level.

Our other option is to pursue an Existing Facility Operations and Mainentance certification. This is a more rigorous review that would require us to change out our old whole-building air conditioning units. I cannot see how I can justify this large capital investment on the data center project alone.

We will investigate the possibility of the second certification as part of our larger sustainability plan.

There were some surprises in this evaluation:

• The rather large solar investment in the project- enough to offset 100% of the power needs of the data center – garnered us all of 1 point in the evaluation. The same as reusing our door.
• We received the bulk of our points because of the location of the site and the nature of our business. Being a public transit district and the free transit we offer out employees brought us 6 points. The thought is that commuters will generate a much larger carbon footprint than that of the data center itself.
• We receive no points for reusing the air conditioners. Even though they will be used infrequently in another part of the building, they clearly still pull a great deal of power.

Pursuing the second certification will require significant investment on our part- from changing the toilets and water fixtures, to the cleaning products we use, to changes in out parking lot. It will be an interesting process to determine whether these investments will make sense to our financially strapped organization.

And that brings us to the question of ROI. The certification process will probably cost around $7500 with fees and consulting hours. This is a very small cost on the overall project, but is equivalent to purchasing 3 servers. Will it return as much as 3 servers would? Is there a monetary value to LEED?

One focus of our data center design activities this week has been around the question of backup power. Any high-availability data center must face this question, and the reality is that there are not too many options to have a green approach to backup power.

The first consideration is the Uninterruptible Power Supply (UPS). We looked at three options for providing the immediate, on-demand UPS capability: traditional battery UPS, flywheel UPS, and Fuel Cell UPS.

The flywheel UPS seemed to provide the greenest option; however, it was not available to integrate with our chosen ‘pod’ manufacturer. This leaves us with the fuel cell or the battery options. While the Fuel Cell is a solid option for the future, I felt that the cost/benefit was not yet there for us. Which leaves us with a traditional battery for our UPS. Not the greenest option overall, but when you factor in the full APC Infrastruxure solution, we still come out ahead of the game.

The second major decision was on backup power to support the UPS for extended potential power failures. Most SMB data centers do not really face this issue. However, in the last two years of my tenure, we have had no fewer than three extended power outages that have taken down my data center for over 8 hours. Now that we’re running the SPRINTER train and the fare collection systems using the network, extended downtime is simply not an option.

We are therefore going to install an onsite generator. I initially was not happy with this idea because clearly a diesel generator is simply not green. We were initially pursuing a natural gas generator under the assumption that this would be a more environmentally-friendly option. However, in working with our generator installation firm (Bay City Electric), it became clear that natural gas would not be an option for us due to potential issues from earthquakes interrupting the supply lines.

So we’re going with the traditional diesel generator unless some other option presents itself. Our strategy will be to mitigate as many environmental concerns as possible through process:

• Whenever possible, we will test the generator under load -meaning that we will run the data center 100% on the power the generator creates. This means that we will not be wasting the fuel.
• We will install filtration on the emissions to ensure that we’re not discharging significant particulates
• We will test as infrequently as possible to validate that the system in functional

I spent hours working with our IT firm Logicalis to try to find alternatives to these two decisions. I have to say that I am disappointed that there were not other options readily available in the market place for us. In reviewing some of the most successful green data center project case studies from the last two years, the preponderance of them make reference to their use of generators for backup power. Even my green-web hosting firm AISO.net talks about their use of generators in times of need.

While we are making great strides on designing our green data center, I have to admit that I was disheartened with these two decisions I made in the design; however, I did not feel I had the business case, other case studies, nor the viable alternatives that would lead us down a different path.

by Angela Miller
One of the biggest issues for my little data center at NCTD is handling the heat load of the room.  I was reading an article online at the Georgia Institute of Technology that said that cooling the data center has become more complicated as the average heat load per cabinet has moved from  1-5 Kilowatts of heat to 28 in the last 5 years.  We can easily see this in our room – we have a cabinet with 10 rack-optimized HP DL360 servers each with a 1 unit space between for air flow sitting next to another rack with a C3000 8-server blade chassis and a SAN sitting next to a rack with a C7000 16-server chassis with the Cisco VOIP equipment and no spaces between servers.  This little example shows how in just 5 years the density of the typical server rack in our room has increased immensely.

We also have first-hand experience with the problems this increased heat load can cause for the facility itself.  While we have a raised floor in the room, it was designed primarily for cable management instead of heat mitigation.  So the floor tiles are solid and do not allow the cooler air to be funneled through to the cabinets.  About 3 years ago the heat in the room exceeded the capacity of the air conditioners resulting in both a flood of condensation in the room and a blown ac that took down the data center.

I have previously blogged about our current poor cooling solution installed as a result of this outage – two residential-class air handling units on the floor of the data center with a fabricated venting system designed to pull in the hot air from the floor (?) and push the cold air from the top vents directed at various angles throughout the room.  This system makes it extremely uncomfortable for my staff to work — and who can blame them?  I myself moved one of the vents just an inch higher so I could stand in the room for an hour and forgot to move it back.  This one little adjustment resulted in the temperature in the racks increasing on average 3 degrees while the vent was moved.

Clearly this is not a sustainable solution.  So as we embark on the redesign of the data center, cooling solutions have been front-and-center in the conversation.  The Green Grid has published seven steps to consider when designing a cooling solution for the green data center:

1. Developing an air management strategy
2. Moving cooling systems closer to the load
3. Operating at a higher delta-T
4. Installing economizers
5. Using higher-specifi cation and performance equipment
6. Using dynamic controls
7. Maintaining higher operating temperatures

We kept these guidelines in mind when evaluating the options for cooling the data center.  We have looked at a variety of cooling solutions for the facility, but there are several things which make it difficult to be innovative in this space.  The first is that this room is in the basement of a former bank building.  We are strictly limited on the height and footprint of the room.  Therefore we cannot be more creative with our raised floor – it is only 8 inches high, but going higher to allow for venting under the floor is not in the cards.

We also must contend with walls that support vaults on two floors above the data center, limiting what we can do with the venting and air handling outside of the room.  Given these constraints, the recommendation by Logicalis and Roel was to install a pod system.  This approach will allow us to encapsulate the racks, create hot and cold zones, and provide in-line cooling for the racks right where the need is.  This is not ideal for every data center – for example, we almost were unable to use this solution because the footprint of the room was 1 foot too short for the necessary clearance around the pod.  Fortunately, we were able to recapture some space by moving an internal wall out slightly allowing us to just fit the equipment into the design.

It is also important to understand the tradeoff with an encapsulated system like the APC pods:  once it is in within the walls of this room, I will not have the ability to grow the data center past this size.  This will be the finite number of racks we can install in the foreseeable future.  I cannot move walls again, nor can I migrate to a new space within this building.  So we must be smart in the design phase in order to get us a full ten-year investment and growth opportunity in this space.  Choosing a pod also increases the budget versus sticking with the current approach of open racks in the space.  But given the other design criteria, the pod solution is the clear winner on energy efficiency and heat handling.

While this is the basic design we’ve elected to pursue, I also requested that we begin by first performing an air flow analysis of the current facility, both with the air conditioners running and without.  Such a study might reveal some interesting design criteria for us to keep in mind as we move forward.  I have a feeling that we might find we have significant bypass airflow issues to deal with (basically this means air that is infiltrating the room through gaps and openings in the walls).  Our initial monitoring of the humidity in the room using simple environmental monitors shows that we actually have a moisture problem in the room in addition to the cooling issues.

I will post more as we get into this project so that you can see the practical realities of how we make decisions that consider the energy efficiency, sustainability, and practical design considerations through the project.

Digg Deeper on the Issues:

I relied on the following sites for this post:

The Green Grid
American Power Conversion
Georgia Institute of Technology

None of the entities in this post has provided compensation or incentive to discuss their products or services.

by Angela Miller
On Friday we kicked off the data center project at NCTD, and rarely have I been so excited about the probable success of a project.  Our vendors are Roel Construction (Rob Netzer) and Logicalis (Bob Mobach).  We were lucky to find vendors that have such competency and experience in data center design, and specifically in the requirements for attaining LEED certification.

This is a large undertaking for a small entity like NCTD.  While I can justify the project purely on the long-term anticipated Return-on-Investment, the deal was sweetened by meeting all of the requirements for the Federal Stimulus program.

The overall project has several elements that will hopefully qualify it for LEED certification, including:

• The anticipated reduction in power demand and increase in power utilization efficiency in the facility
• The re-use of the building, and materials within the facility for the project (for example, we are supplementing our green fire system instead of replacing it, we’re going to reuse doors instead of purchasing new ones, etc.
• The ability to reset the ambient temperature of the data center much higher – we expect to set it around 80 degrees instead of the 68 we maintain today
• The installation of over 220 Kw solar system onsite to more than offset the power draw of the data center
• The use of in-line cooling instead of the two air handling units currently in the room (these will be recycled for other purposes in the District)

In addition, as I’ve mentioned in previous posts, the District has been committed over the last two years to slowly introducing greener, more sustainable approaches to our information technology infrastructure, including virtualization and consolidation, switching to blade server technology, replacing older equipment with more energy-efficient equipment, and testing desktop virtualization.

All of these steps make solid financial sense for the District – an especially important consideration given the tight financial times we are currently facing.  I could not in good conscience recommend these investments if I could not show solid ROI for our troubles.  While we want to be an agency with a priority on sustainability as part of our mission, it is logical that we could not choose to make these investments if they required a corresponding reduction in service or positions at the District.  Instead, choosing to follow the more sustainable path will realize direct operating cost savings on a monthly basis for the District.

We of course face some challenges to our ambitious timeline.  One of those challenges is the delay of a system migration project by 90 days after its anticipated completion.  This project is attempting to move our Prime System applications (installed in 1986) from a minicomputer to a client-server web interface.  We cannot risk downtime on this environment and therefore out-of-the gate may see a 90-day delay.  But every IT project faces challenges, and I am confident the team will find an approach that gives us what we need.

Over the next several weeks I will be blogging about the adventures of upgrading our data center.  Hopefully in March I will be able to say that we’ve completed the work and that we are on track with our ROI.