This is a little off topic, but I thought I would share a presentation I did about my NCTD IT Team for a Leadership program I am in through the American Public Transportation Association. This shows I am perhaps a little overly-proud of my team, but they do earn that respect.
Today the data center project paid off big time. Yes, for those in Southern California it was a near disaster- we had an extensive power outage that reached from Mexico to Los Angeles, and from San Diego to Arizona.
The good news for my team- the data center stayed up 100% of the time. In fact, we lost only 1 rail communications cabinet during 10 hours of an outage.
This means that our phones were up, even when cell phones were not. Our Internet was online and we could communicate with our customers. Our emergency operations center (EOC) was live and effective. I was proud to see NCTD keep bus and rail service in play hours into an outage that was so significant for the region.
More than anything, I know that without the data center project, we would not have achieved this success.
Not everything was smooth and I don’t want to overstate the positives… but I am confident without our green data center, our EOC team would have struggled that much more. So here is to investment in technology and the ability to see a ROI.
The LEED saga continues. I’ve mentioned some of the challenges that we’ve faced, but today presented another interesting one. I assumed because our purchasing specifications included statements like “systems that comply with the US EPA Energy Star requirements” that we would qualify for the Energy Star points on the LEED rating system. Today, I learned how naive I am. In fact, while several vendors have machines that in fact to have an EPA stamp of Energy Star compliance, they are few and far between right now.
Energy Star was a program started initially around residential power use. As a result, most items that have the Energy Star seal are appliances or electronics in the consumer space. A check of the EPA website shows fewer than 15 enterprise, server class machines that qualify for the rating. So while many vendors state that they have Energy Star-compliant equipment, they do not in fact have too many machines that actually went through and successfully completed the process.
Let’s compare some from my data center as examples. We run an HP shop (this is not an endorsement of their product or a sales pitch, just disclosure that we have them as an architectural standard). So I have a wide variety of their equipment. For our Microsoft Exchange upgrade, we installed Energy Star certified HP Proliant DL380 G6 rack-optimized servers. These are currently the only series of HP machines that have the seal. For most of my purchasing, however, I prefer HP BL460c G6 or HP BL680c G6 machines that slot into a blade chassis. What is nice about blades is that they share components like power supplies and fans. So this reduces the power pull, and reduces the amount of waste in the product. So from a product life cycle perspective, they are a better choice.
In spite of our choice to generally rely on the more energy-efficient and therefore more eco-friendly choice of the blade servers, we actually cannot claim the LEED energy star credit because these servers are merely “EnergyStar Compliant” instead of certified. This is needless-t0-say an unfortunate outcome as we are inching closer to a possible Gold certification and any point that we miss now keeps us from that nearly impossible goal.
One focus of our data center design activities this week has been around the question of backup power. Any high-availability data center must face this question, and the reality is that there are not too many options to have a green approach to backup power.
The first consideration is the Uninterruptible Power Supply (UPS). We looked at three options for providing the immediate, on-demand UPS capability: traditional battery UPS, flywheel UPS, and Fuel Cell UPS.
The flywheel UPS seemed to provide the greenest option; however, it was not available to integrate with our chosen ‘pod’ manufacturer. This leaves us with the fuel cell or the battery options. While the Fuel Cell is a solid option for the future, I felt that the cost/benefit was not yet there for us. Which leaves us with a traditional battery for our UPS. Not the greenest option overall, but when you factor in the full APC Infrastruxure solution, we still come out ahead of the game.
The second major decision was on backup power to support the UPS for extended potential power failures. Most SMB data centers do not really face this issue. However, in the last two years of my tenure, we have had no fewer than three extended power outages that have taken down my data center for over 8 hours. Now that we’re running the SPRINTER train and the fare collection systems using the network, extended downtime is simply not an option.
We are therefore going to install an onsite generator. I initially was not happy with this idea because clearly a diesel generator is simply not green. We were initially pursuing a natural gas generator under the assumption that this would be a more environmentally-friendly option. However, in working with our generator installation firm (Bay City Electric), it became clear that natural gas would not be an option for us due to potential issues from earthquakes interrupting the supply lines.
So we’re going with the traditional diesel generator unless some other option presents itself. Our strategy will be to mitigate as many environmental concerns as possible through process:
I spent hours working with our IT firm Logicalis to try to find alternatives to these two decisions. I have to say that I am disappointed that there were not other options readily available in the market place for us. In reviewing some of the most successful green data center project case studies from the last two years, the preponderance of them make reference to their use of generators for backup power. Even my green-web hosting firm AISO.net talks about their use of generators in times of need.
While we are making great strides on designing our green data center, I have to admit that I was disheartened with these two decisions I made in the design; however, I did not feel I had the business case, other case studies, nor the viable alternatives that would lead us down a different path.
by Angela Miller
On Friday we kicked off the data center project at NCTD, and rarely have I been so excited about the probable success of a project. Our vendors are Roel Construction (Rob Netzer) and Logicalis (Bob Mobach). We were lucky to find vendors that have such competency and experience in data center design, and specifically in the requirements for attaining LEED certification.
This is a large undertaking for a small entity like NCTD. While I can justify the project purely on the long-term anticipated Return-on-Investment, the deal was sweetened by meeting all of the requirements for the Federal Stimulus program.
The overall project has several elements that will hopefully qualify it for LEED certification, including:
In addition, as I’ve mentioned in previous posts, the District has been committed over the last two years to slowly introducing greener, more sustainable approaches to our information technology infrastructure, including virtualization and consolidation, switching to blade server technology, replacing older equipment with more energy-efficient equipment, and testing desktop virtualization.
All of these steps make solid financial sense for the District – an especially important consideration given the tight financial times we are currently facing. I could not in good conscience recommend these investments if I could not show solid ROI for our troubles. While we want to be an agency with a priority on sustainability as part of our mission, it is logical that we could not choose to make these investments if they required a corresponding reduction in service or positions at the District. Instead, choosing to follow the more sustainable path will realize direct operating cost savings on a monthly basis for the District.
We of course face some challenges to our ambitious timeline. One of those challenges is the delay of a system migration project by 90 days after its anticipated completion. This project is attempting to move our Prime System applications (installed in 1986) from a minicomputer to a client-server web interface. We cannot risk downtime on this environment and therefore out-of-the gate may see a 90-day delay. But every IT project faces challenges, and I am confident the team will find an approach that gives us what we need.
Over the next several weeks I will be blogging about the adventures of upgrading our data center. Hopefully in March I will be able to say that we’ve completed the work and that we are on track with our ROI.
by Angela Miller
Well today came the unfortunate news that my web hosting service provider is going out of business. Greenesthost.com was the only entity that I could find 2 years ago focused on delivering a solar-powered, truly green hosting environment. All of the other players I researched were meeting their sustainability target through purchased carbon offsets. This seemed to me – while a positive step – far less progressive than the greenesthost model of constructing a facility powered exclusively with alternative energy.
But alas not all that is green is gold. The company stopped accepting new customers a little over a month ago, and today the letter came with the sad news that in September all lights will turn off.
I have chosen to move to AISO.net. They also now offset all of their power demand with their own solar installation. They have provided a nice analysis of their estimated reduction of carbon output based on the solar installation which was certified by the USEPA.
This was interesting to me because not only would we like to provide some analysis of what we’ll offset with our solar installation at NCTD, we would also like to investigate some consistent methology for comparing an individual ride with a public transit ride between destinations.
by Angela Miller
One of the systems we’re building at NCTD is a security system for our COASTER commuter rail that extends for over 40 miles from Oceanside to San Diego, California. The geek in me is excited about this project from a pure technology perspective: the ability to monitor activities in and around the vehicle while it is moving at 60 miles an hour, well that’s a real technology challenge. Our solution along the rail is a mesh network.
We have a series of ‘nodes’ that we’re installing at about 1 mile intervals that provide a wireless infrastructure along the rail. With that network, we can then install other wireless devices – for example cameras, or VOIP devices. But what would this have to do with being green?
Our approach to these nodes has been to install solar panels at each node. On the one hand this is simple practicality because some sites are very remote and far from existing power sources. But, for more than half of the nodes we could have chosen to plug into wired power sources. So as a CIO, I had to make a choice- use solar or use grid power?
The business case is not so straight forward. While we will save power using the solar, we incur higher installation and maintenance costs to ensure the solar systems provide consistent, reliable power to each node. We cannot claim that we chose to use solar simply because it was the greener choice – that would be a deceptive message. However, I can say that as the CIO I had the architectural design choice to go with either solution and chose solar – it seemed to be the more stable choice, and the choice that would still operate if we had a power outage, and yes when all solutions were compared it was the more sustainable, environmentally-friendly approach.
So I leave it in the reader’s mind – perhaps security at NCTD is green, and perhaps not. You be the judge.
by Angela Miller
Today we kicked off our desktop virtualization test at NCTD. I have a few reasons for wanting to look at virtualization – the first is that the workspaces for some of our users are too crowded and we would like to provide them more space within their constraints without requiring a larger footprint on their desk. This would keep us from needing to expand the offices into additional spaces.
The second compelling reason is to try to extend the lifetime of some of our desktop investment. I previously discussed my choice to upgrade user machines with memory and larger hard disks in an attempt to get at least 2 more years out of the machines. My hope is that desktop virtualization will allow us to contemplate an operating system upgrade without needing to replace every machine at the office.
Our test environment consists of an HPBlade 460, 3 small HP thin clients, 1 HP 6910p laptop, and 2 HP 5100dct desktops. We acquired a VMWare VDI bundle for a relatively small investment and will run our test with these tools. Our plan is to test a variety of elements: stability of the desktop environment, ease of deploying new desktops, the performance of the desktops with a variety of virtual server builds, and performance on the different types of hardware.
We will also be analyzing and researching the power requirements for each of these scenarios. Our hope is that we might see a reduced power requirement even on the older hardware because the server will be doing the ‘heavy lifting’ on the desktop.
After our test, I’ll post the different results we see, including an analysis of whether pursuing a desktop virtualization strategy is ‘greener’ than continuing with our current approach.
I relied on the following sites for analysis in support of this post:
by Angela Miller
I tend to assume that IT people (like me) have a fabulous sense of hyperbole. We toss around words like “disaster” and “outage” and “negative business impacts” effortlessly. And many IT Managers panic when we are less than say 97% perfect. We have a sense of the dramatic.
So I figure people might take umbrage with my characterization of my data center ‘blower problems.’ Perhaps you think I am again exaggerating. The picture on the left should correct this.
When I say blowers in my data center, I mean just that – a line of 6 vents resembling aircraft engines pointed different directions in the room. People who enter this room do so at their own risk, especially when the blowers are churning at full speed. Smaller people on the team probably should have carabiners and safety lines.
Since my arrival at NCTD, this device has been one of the more discussed items on the team. All kidding aside, this is not the right solution for a data center. Admitedly, we were in a crisis mode when this modification was made. And nothing against the people who chose to install these, because frankly we would be in far worse shape had they not done so. This installation bought us a year … maybe two.
In my mind, this is a good example of the dilemma facing many SMB entities in their data centers — how do we evolve to deal with our ever-increasing energy, cooling, and storage demands? How do we make upgrades to our facilities in a way that meets the burgeoning needs? And should sustainability be a consideration in our design choices.
One thing is certain – choosing the wrong solution is far from sustainable and misses our return on investment. In our case, we are now facing not only cooling problems since the air flow with this device does not address the hot spot problems in the room, but also humidity issues, and power issues since we’ve used almost all the rest of the power in the room on these air handling units. On the plus side, if I installed a wind turbine in the room I could probably dramatically improve my Power Utilization Effectiveness.
Over the next several weeks we’ll look at some of the options we’ll have to choose from in implementing a different approach to the cooling needs. Will we be able to choose a green option given our budget constraints? Stay tuned…