August 12, 2005

Stock Exchange CIO: Real-Time Means Real-Time

By Evan Schuman, Ziff Davis Internet

At a time when IT executives are finding users less and less tolerant of network delays, CIO William Morgan has perhaps the least tolerant set of users in the world.

Morgan runs technology operations for the 215-year-old Philadelphia Stock Exchange, the nation's first stock exchange. But its age doesn't make its daily load any easier, with networks having to handle 120,000 messages per second and peaks of 200,000 messages per second.

But because it is a financial exchange, any delay—even a half-second—is not acceptable.

"We measure transactions in milliseconds these days. This business can't tolerate delays: We're pricing customer orders," Morgan said. "It's survival for us."

In recent years, financial activity has pushed that IT demand much higher. "If you go back five or six years, probably the number [of messages per second] was 10,000 or less," which is one-twelfth today's volume, Morgan said.

Morgan delivers that real-time speed with some homegrown applications sitting atop Sun Microsystems Solaris 10 servers, Stratus fault-tolerant servers in a Nortel network.

The CIO argues strongly for using as much standardized software as possible; the exchange's Web site runs on Windows, and e-mail is using Microsoft Outlook.

"We use the standard Windows environment for all that, but not for our trading. On the trading side, there simply aren't many packages," he said.

"There are many for broker dealers, but a select for exchanges. There aren't that many exchanges and, because of the custom nature of each exchange's business, it's very hard to find an off-the-shelf" package.

Having the network deliver all of those messages per second—Morgan's people stress test their system with 200,000 messages per second—is only part of the battle.

After the messages are delivered, they have to be stored, catalogued and archived. These days, that's about one-half billion messages every day.

All things considered, Morgan said, the storage is the easy part. "Data storage for us during the day is not the challenge. The challenge for us is retention," he said. "This is more about cost, given our size and the challenges."

At about 490 million messages a day, the government-required seven years of message retention adds up quickly.

The exchange handles messages in two ways, splitting them into recent messages (about three months' worth) where the data needs to be readily accessible and the remainder that can be held in offsite storage about 25 miles away from headquarters.

Much of the data is managed in SANs (Storage Area Networks) with about 10TB of storage at headquarters.

The balancing act of when to have data no longer be so readily available is primarily a budget issue.

The data that is kept super-accessible costs a lot more to maintain. "The longer you wait for it, the cheaper the storage," Morgan said.

The 120,000 messages a second are primarily being managed by some Sun Fire 6800 series servers. Morgan estimates that a typical day handles about 400 million quotes.

"They were the largest servers we could get at the time," he said. "We try and leave as much spare capacity as possible. The key is to constantly be proactive, to be monitoring and measuring these systems. You have to always be watching, measuring."

In the never-ending argument of whether it's better to have a small number of big servers or a large number of medium or small servers, Morgan finds himself in the large server camp, opting for horizontal growth (adding CPU capacity to existing large servers) over vertical growth (adding more servers).

"It's so much easier to plop a board in and run," Morgan said.

The Philadelphia Stock Exchange is clearly focused on security, but it has advantages that not all companies have.

One key advantage is that the systems are only booted up from 7 a.m. to 4:15 p.m.

This sidesteps a typical problem that plagues many networks that need to run 24-7. Those systems must have extensive redundant systems in order for backups to be run and patches and updates installed. The fact that the Philadelphia systems only run during the day makes those issues a lot simpler to deal with.

Another examples is that the Philadelphia Stock Exchange offers no Web access to any operational systems.

"We're not conducting business over the Internet. They can log in and get information that they need," but no trades are permitted using the Internet, Morgan said. "Our trading systems for the most part are closed networks. No public domain. No one is logging in. It's all dedicated point-to-point."

The Philadelphia Stock Exchange is involved in stocks, futures and options trading and therefore competes with a wide range of financial markets in the United States and abroad.

But the bulk of its business is in options trading and it only has five rivals in that segment: the Boston Options Exchange, the Pacific Coast Exchange, the American Stock Exchange, the Chicago Board Options Exchange and the International Securities Exchange.

Compared with most of those other exchanges, Philadelphia's IT operations are "very progressive and much more proactive than some of the others," said Kristin Lovejoy, the chief technology officer for a data auditing firm called Consul Risk Management.

"I have worked with two of the other exchanges, and their attitude toward [security] compliance is much different."

She gave a security example. Technically, the exchanges are not governed by Sarbanes-Oxley. Most exchanges do not comply with those regulations, but she said Philadelphia voluntarily does.

"Philadelphia gets it. They don't have to comply, but they interpret it in the spirit of what Sarbanes-Oxley is all about."

Lovejoy said that the Philadelphia Stock Exchange's IT attitude is also reflected in its approach to technology audits.

"Their attitude toward audits is unique. They don't want the auditors to drive them. They drive the auditors," she said, adding that when auditors report problems with other exchanges, those other exchange IT departments "would scramble to implement the fix and to find the cheapest software possible to address the need. (The Philadelphia Stock Exchange's IT people) instead look at the business overall to see if the changes make sense and the best way to make them happen."

She compared the security audit to a house inspection. The typical exchange response to an inspection that tells of a specific leak in the corner of one room is to patch that leak.

The Philadelphia IT response, she said, is to trace the leak back to find where the water is coming from and fix the problem's cause.

"Another of the things that Philly does that is much more effective than what others do is that they are very focused on the software change management process," Lovejoy said. "They look at every change that is made to every production system."

Beyond taking precautions such as capturing full system snapshots before any software change, the exchange has rigid procedures for installing any application or upgrade, to make it easier to roll back any changes if a problem crops up. "They look at security as including system availability," Lovejoy said. "Others don't."

David Schehr, a Gartner research director, sees the short-term future of exchanges such as Philadelphia's being very demanding on IT resources.

Philadelphia "will be facing a situation in a few years where they have to become more nimble, look for external partners and have a system flexibility that can sustain them in the long term," Schehr said.

"So, for IT, it's not just the ability to handle the volume today and tomorrow, but can the systems be set up in a way that can manage that" in the long term?

Schehr also referenced the New York Stock Exchange's plans to purchase Archipelago Holdings—along with Nasdaq's purchase of the Instinet Group—as proof that the markets are changing and that technology flexibility will be key.

"Both exchanges want to trade more than just equities. There's going to be more fluidity in what's traded," Schehr said.

"During a week or week and a half in April, the announcement about Archipelago was made, and then a few days later Nasdaq made their announcement. It's the first two rounds of a much bigger fight. Other exchanges are going to have to deal with that regardless of what they're trading."