Blog

data center, green computing, infrastructure
Comments Off on Evaluating Backup Power Strategies

One focus of our data center design activities this week has been around the question of backup power. Any high-availability data center must face this question, and the reality is that there are not too many options to have a green approach to backup power.

 

The first consideration is the Uninterruptible Power Supply (UPS). We looked at three options for providing the immediate, on-demand UPS capability: traditional battery UPS, flywheel UPS, and Fuel Cell UPS.

 

The flywheel UPS seemed to provide the greenest option; however, it was not available to integrate with our chosen ‘pod’ manufacturer. This leaves us with the fuel cell or the battery options. While the Fuel Cell is a solid option for the future, I felt that the cost/benefit was not yet there for us. Which leaves us with a traditional battery for our UPS. Not the greenest option overall, but when you factor in the full APC Infrastruxure solution, we still come out ahead of the game.

 

The second major decision was on backup power to support the UPS for extended potential power failures. Most SMB data centers do not really face this issue. However, in the last two years of my tenure, we have had no fewer than three extended power outages that have taken down my data center for over 8 hours. Now that we’re running the SPRINTER train and the fare collection systems using the network, extended downtime is simply not an option.

 

We are therefore going to install an onsite generator. I initially was not happy with this idea because clearly a diesel generator is simply not green. We were initially pursuing a natural gas generator under the assumption that this would be a more environmentally-friendly option. However, in working with our generator installation firm (Bay City Electric), it became clear that natural gas would not be an option for us due to potential issues from earthquakes interrupting the supply lines.

 

So we’re going with the traditional diesel generator unless some other option presents itself. Our strategy will be to mitigate as many environmental concerns as possible through process:

  • Whenever possible, we will test the generator under load -meaning that we will run the data center 100% on the power the generator creates. This means that we will not be wasting the fuel.
  • We will install filtration on the emissions to ensure that we’re not discharging significant particulates
  • We will test as infrequently as possible to validate that the system in functional

 

I spent hours working with our IT firm Logicalis to try to find alternatives to these two decisions. I have to say that I am disappointed that there were not other options readily available in the market place for us. In reviewing some of the most successful green data center project case studies from the last two years, the preponderance of them make reference to their use of generators for backup power. Even my green-web hosting firm AISO.net talks about their use of generators in times of need.

 

While we are making great strides on designing our green data center, I have to admit that I was disheartened with these two decisions I made in the design; however, I did not feel I had the business case, other case studies, nor the viable alternatives that would lead us down a different path.

data center, green computing
Comments Off on Designing the cooling system for the NCTD Data Center

by Angela Miller
One of the biggest issues for my little data center at NCTD is handling the heat load of the room.  I was reading an article online at the Georgia Institute of Technology that said that cooling the data center has become more complicated as the average heat load per cabinet has moved from  1-5 Kilowatts of heat to 28 in the last 5 years.  We can easily see this in our room – we have a cabinet with 10 rack-optimized HP DL360 servers each with a 1 unit space between for air flow sitting next to another rack with a C3000 8-server blade chassis and a SAN sitting next to a rack with a C7000 16-server chassis with the Cisco VOIP equipment and no spaces between servers.  This little example shows how in just 5 years the density of the typical server rack in our room has increased immensely.

We also have first-hand experience with the problems this increased heat load can cause for the facility itself.  While we have a raised floor in the room, it was designed primarily for cable management instead of heat mitigation.  So the floor tiles are solid and do not allow the cooler air to be funneled through to the cabinets.  About 3 years ago the heat in the room exceeded the capacity of the air conditioners resulting in both a flood of condensation in the room and a blown ac that took down the data center.

I have previously blogged about our current poor cooling solution installed as a result of this outage – two residential-class air handling units on the floor of the data center with a fabricated venting system designed to pull in the hot air from the floor (?) and push the cold air from the top vents directed at various angles throughout the room.  This system makes it extremely uncomfortable for my staff to work — and who can blame them?  I myself moved one of the vents just an inch higher so I could stand in the room for an hour and forgot to move it back.  This one little adjustment resulted in the temperature in the racks increasing on average 3 degrees while the vent was moved.

Clearly this is not a sustainable solution.  So as we embark on the redesign of the data center, cooling solutions have been front-and-center in the conversation.  The Green Grid has published seven steps to consider when designing a cooling solution for the green data center:

  1. Developing an air management strategy
  2. Moving cooling systems closer to the load
  3. Operating at a higher delta-T
  4. Installing economizers
  5. Using higher-specifi cation and performance equipment
  6. Using dynamic controls
  7. Maintaining higher operating temperatures

We kept these guidelines in mind when evaluating the options for cooling the data center.  We have looked at a variety of cooling solutions for the facility, but there are several things which make it difficult to be innovative in this space.  The first is that this room is in the basement of a former bank building.  We are strictly limited on the height and footprint of the room.  Therefore we cannot be more creative with our raised floor – it is only 8 inches high, but going higher to allow for venting under the floor is not in the cards.

We also must contend with walls that support vaults on two floors above the data center, limiting what we can do with the venting and air handling outside of the room.  Given these constraints, the recommendation by Logicalis and Roel was to install a pod system.  This approach will allow us to encapsulate the racks, create hot and cold zones, and provide in-line cooling for the racks right where the need is.  This is not ideal for every data center – for example, we almost were unable to use this solution because the footprint of the room was 1 foot too short for the necessary clearance around the pod.  Fortunately, we were able to recapture some space by moving an internal wall out slightly allowing us to just fit the equipment into the design.

It is also important to understand the tradeoff with an encapsulated system like the APC pods:  once it is in within the walls of this room, I will not have the ability to grow the data center past this size.  This will be the finite number of racks we can install in the foreseeable future.  I cannot move walls again, nor can I migrate to a new space within this building.  So we must be smart in the design phase in order to get us a full ten-year investment and growth opportunity in this space.  Choosing a pod also increases the budget versus sticking with the current approach of open racks in the space.  But given the other design criteria, the pod solution is the clear winner on energy efficiency and heat handling.

While this is the basic design we’ve elected to pursue, I also requested that we begin by first performing an air flow analysis of the current facility, both with the air conditioners running and without.  Such a study might reveal some interesting design criteria for us to keep in mind as we move forward.  I have a feeling that we might find we have significant bypass airflow issues to deal with (basically this means air that is infiltrating the room through gaps and openings in the walls).  Our initial monitoring of the humidity in the room using simple environmental monitors shows that we actually have a moisture problem in the room in addition to the cooling issues.

I will post more as we get into this project so that you can see the practical realities of how we make decisions that consider the energy efficiency, sustainability, and practical design considerations through the project.

Digg Deeper on the Issues:

I relied on the following sites for this post:

The Green Grid
American Power Conversion

Georgia Institute of Technology

None of the entities in this post has provided compensation or incentive to discuss their products or services.

data center, energy consumption, green computing, hardware, infrastructure
Comments Off on Data Center Redesign Kickoff

by Angela Miller
On Friday we kicked off the data center project at NCTD, and rarely have I been so excited about the probable success of a project.  Our vendors are Roel Construction (Rob Netzer) and Logicalis (Bob Mobach).  We were lucky to find vendors that have such competency and experience in data center design, and specifically in the requirements for attaining LEED certification.

This is a large undertaking for a small entity like NCTD.  While I can justify the project purely on the long-term anticipated Return-on-Investment, the deal was sweetened by meeting all of the requirements for the Federal Stimulus program.

The overall project has several elements that will hopefully qualify it for LEED certification, including:

  • The anticipated reduction in power demand and increase in power utilization efficiency in the facility
  • The re-use of the building, and materials within the facility for the project (for example, we are supplementing our green fire system instead of replacing it, we’re going to reuse doors instead of purchasing new ones, etc.
  • The ability to reset the ambient temperature of the data center much higher – we expect to set it around 80 degrees instead of the 68 we maintain today
  • The installation of over 220 Kw solar system onsite to more than offset the power draw of the data center
  • The use of in-line cooling instead of the two air handling units currently in the room (these will be recycled for other purposes in the District)

In addition, as I’ve mentioned in previous posts, the District has been committed over the last two years to slowly introducing greener, more sustainable approaches to our information technology infrastructure, including virtualization and consolidation, switching to blade server technology, replacing older equipment with more energy-efficient equipment, and testing desktop virtualization.

All of these steps make solid financial sense for the District – an especially important consideration given the tight financial times we are currently facing.  I could not in good conscience recommend these investments if I could not show solid ROI for our troubles.  While we want to be an agency with a priority on sustainability as part of our mission, it is logical that we could not choose to make these investments if they required a corresponding reduction in service or positions at the District.  Instead, choosing to follow the more sustainable path will realize direct operating cost savings on a monthly basis for the District.

We of course face some challenges to our ambitious timeline.  One of those challenges is the delay of a system migration project by 90 days after its anticipated completion.  This project is attempting to move our Prime System applications (installed in 1986) from a minicomputer to a client-server web interface.  We cannot risk downtime on this environment and therefore out-of-the gate may see a 90-day delay.  But every IT project faces challenges, and I am confident the team will find an approach that gives us what we need.

Over the next several weeks I will be blogging about the adventures of upgrading our data center.  Hopefully in March I will be able to say that we’ve completed the work and that we are on track with our ROI.

by Angela Miller
Well today came the unfortunate news that my web hosting service provider is going out of business. Greenesthost.com was the only entity that I could find 2 years ago focused on delivering a solar-powered, truly green hosting environment. All of the other players I researched were meeting their sustainability target through purchased carbon offsets. This seemed to me – while a positive step – far less progressive than the greenesthost model of constructing a facility powered exclusively with alternative energy.

But alas not all that is green is gold. The company stopped accepting new customers a little over a month ago, and today the letter came with the sad news that in September all lights will turn off.

I have chosen to move to AISO.net. They also now offset all of their power demand with their own solar installation. They have provided a nice analysis of their estimated reduction of carbon output based on the solar installation which was certified by the USEPA.

This was interesting to me because not only would we like to provide some analysis of what we’ll offset with our solar installation at NCTD, we would also like to investigate some consistent methology for comparing an individual ride with a public transit ride between destinations.

alternative energy, carbon offset, data center, energy consumption
Comments Off on IT Department to Offset Power with Solar Installation

by Angela Miller
Today we inked the deal that was two-years in the making:  we are officially installing solar panels at our administrative offices.  The intent is to at least offset the estimated power demands of the data center over the next 10 years of anticipated growth.  In speaking with various solar vendors, and with our data center installation firm, we feel this is an achievable goal.

With the incentives still offered at the federal and state level, and working with our local utility company SDG&E, the business case for solar is a viable one.  In a time when my agency – like so many across the State of California – is facing dramatic pressures on our operating budget, any capital investment that potentially decreases monthly operating expenses deserves a serious look. 

We have two phases of this plan – we’ll install solar panels on the roof of the building, which turns out to be an ideal location.  Not every building meets our criteria – we have significant open space, our roof was repaired recently with a material that is reflective, and our utility sheds are relatively short.  These design elements make our roof highly suitable for a solar installation.

The second phase of our project will install carport solar panels in our public parking lot.  Our existing lot was designed at a time when there were fewer requirements for heat dissipation.  Our black asphalt parking lot gets quite hot during most months, and installation of solar panels will mitigate some of this problem.  We are also making good use of otherwise non-revenue generating space.

This project is a stimulus project through the American Recovery and Reinvestment Act.  There is significant pressure on governmental entities to use this money responsibly, transparently, and according to the terms of the intent of the act.  I believe this project meets all of the objectives of the program:  it is a shovel-ready construction project, a long-term capital improvement, beneficially changes the operations of public transit in the community, installs technology that decreases operating costs, provides jobs during the project, invests in green technology, and completes in a timely fashion to stimulate the local economy.

While the solar projects added cost to our data center redesign project, the investment had a solid business case, with a six-year payback period.  An additional consideration was the increasing carbon footprint of the technology resources at the District.  This investment not only pays off from a fiscal perspective, but also an environmental one.

My message to IT people is to think bigger than our data centers.  We have the opportunity to improve our company’s bottom line not only through our core IT investments, but also in how we choose to construct and improve the buildings that house that technology.  Work with your facility managers to become more creative in solving data center problems.

alternative energy, data center, green computing
Comments Off on APTA TransITech Presentation

by Angela Miller
A few months ago I wrote a post about attending an American Public Transportation Association meeting on Sustainability and how I felt out of place as the only IT person in the room.  Today I had yet another lonely experience.  I was the only person in the collection of IT professionals in public transit focused on green technology. 

I gave a presentation on how transit entities can take small steps to make incremental progress toward greening their data center.   I’m not sure that my observations could compete with the more exciting presentations like iPhone applications and Google Transit.  But I think the slide that showed how making a simple change in our desktops and monitors had actual dollar savings and a measurable reduction in our carbon footprint made some impression.

data center
Comments Off on Fire in the hole…

by Angela Miller
We had some excitement today at my office.  We noticed around 4:30 this morning that we dropped our connectivity to some of the servers – never a good sign when you are seeing multiple notifications from the automated system.  Turns out that the air handling I’ve discussed in other posts just wasn’t up to the job.  One of the two air conditioners blew, and the second one did not take over as designed.  In under 20 minutes the room hit 117 degrees, and servers started shutting down.

Walking into that environment this morning was a little heart-stopping.  The room was filled with smoke, but fortunately nothing was on fire.  Things generally performed as they should after an incident like this, and most of the servers shut down as they were designed to do.

This incident could have been far worse.  We could have lost equipment, or data, or actually had a fire in the room.  But it is still a wake-up call that we must invest in remediating the data center.  Fortunately, we’re hearing about the possibility of stimulus money from the federal government for transit.  So my goal over the next week will be to brush off the grandiose plans for that green data center and start working on getting together biddable requirements for a project.

data center, energy consumption, infrastructure
Comments Off on The Jet Engines in the Data Center.

blowersby Angela Miller
I tend to assume that IT people (like me) have a fabulous sense of hyperbole. We toss around words like “disaster” and “outage” and “negative business impacts” effortlessly. And many IT Managers panic when we are less than say 97% perfect. We have a sense of the dramatic.

So I figure people might take umbrage with my characterization of my data center ‘blower problems.’ Perhaps you think I am again exaggerating. The picture on the left should correct this.

When I say blowers in my data center, I mean just that – a line of 6 vents resembling aircraft engines pointed different directions in the room. People who enter this room do so at their own risk, especially when the blowers are churning at full speed. Smaller people on the team probably should have carabiners and safety lines.

Since my arrival at NCTD, this device has been one of the more discussed items on the team. All kidding aside, this is not the right solution for a data center. Admitedly, we were in a crisis mode when this modification was made. And nothing against the people who chose to install these, because frankly we would be in far worse shape had they not done so. This installation bought us a year … maybe two.

In my mind, this is a good example of the dilemma facing many SMB entities in their data centers — how do we evolve to deal with our ever-increasing energy, cooling, and storage demands? How do we make upgrades to our facilities in a way that meets the burgeoning needs? And should sustainability be a consideration in our design choices.

One thing is certain – choosing the wrong solution is far from sustainable and misses our return on investment. In our case, we are now facing not only cooling problems since the air flow with this device does not address the hot spot problems in the room, but also humidity issues, and power issues since we’ve used almost all the rest of the power in the room on these air handling units. On the plus side, if I installed a wind turbine in the room I could probably dramatically improve my Power Utilization Effectiveness.

Over the next several weeks we’ll look at some of the options we’ll have to choose from in implementing a different approach to the cooling needs. Will we be able to choose a green option given our budget constraints? Stay tuned…

data center, public transit
Comments Off on Facing my new challenge…

by Angela Miller
Not every company gets to invest in the top-of-the-line data center to run their operations.  In fact, I would imagine that most IT people reading this would agree that we face serious constraints in our data centers.  Unless the company’s core business is technology, spending money on the rooms and spaces that house the technology infrastructure often ranks far lower in the executive mind than other operational concerns.

So as I walked into my new role as CIO for the NCTD, I knew that there were going to be issues with my core infrastructure and facilities.  On the one hand, NCTD has invested wisely in the basic facility infrastructure.  Unlike some SMB and governmental agencies, we do in fact have a true data center.  We have fire suppression, and a nice footprint, and standard data center design from the 1980s.  I was pleasantly surprised to have this as a foundation on which to build.

But we also face significant challenges in that space – we have a clash of the old and the new – with both a Prime mini-computer occupying a significant portion (almost 30% of the footprint) of the room, and racks full of rack-optimized computers.  There has been no investment in newer thinking about the data center – blades, virtualization, storage area networks, etc.  Of greatest concern for me are power and air handling.  My data center has two Carrier units that intake the air on floor-level and blow out the cooler air through a large set of blowers.  The torrent of air that whooshes through this small room is impressive.

We installed some environmental monitoring devices and discovered that in addition to the obvious problems with the air handling, we also now have a significant humidity issue.

So we have decisions to face at the company- do we invest in upgrading the data center?  Do we move instead to a co-location facility?  How are we going to handle disaster recovery?  These questions form an excellent backdrop for a question more pertinent to this blog – is it possible to employ greener approaches to the technology while still living within our means?  Will I have a business case for a more sustainable data center?

My intent is to focus over the next several months on my data center challenges in this blog to see whether I can make that business case.  It will be challenging, especially for a small team on a tight capital investment budget.   But I believe that making small, but greener, choices at NCTD will cumulatively lead to a more sustainable technology program for the District.

data center, public transit
Comments Off on New Postion in Public Transit

by Angela Miller 
I am pleased to announce that I have accepted a position as Chief Information Officer with the North County Transit District (www.gonctd.com).  I am excited about the opportunity to combine my technology and environmental skills in an industry in which I believe in the product.  Public transit will grow more important over time as our communities start to practice more sustainable development, and I am excited to become part of this movement.