June 28, 2005

Cingular CIO: Consolidating 50 Million Users Is Not Fun

By Evan Schuman, Ziff Davis Internet

When Cingular announced it was buying AT&T Wireless last year, top executives promised a seamless integration for users. Easy for them to say.

The top execs of the $32 billion combined wireless phone company who were making those announcements didn't have to wrestle with 2.5 petabytes of data, more than 600 overlapping applications, and tons of redundant and incompatible systems. That task fell to CIO Thaddeus Arroyo and his team of more than 6,000 IT professionals.

"As it relates to planning, the level of effort and the complexity of integration was more demanding than we had anticipated," Arroyo said.

As it happens, Arroyo had some experience consolidating and rationalizing wireless telecom mergers. He joined Cingular shortly after it was formed by a merger of the cell phone operations of Bell South and SBC Communications.

When Arroyo joined Cingular in February 2001, the Bell South/SBS merger was only a few months old and he found himself managing about 1,400 overlapping applications. By August 2003, he had whittled that number down to 275. Put another way, he rationalized away 80 percent of the companies' applications.

When Arroyo joined the company, "I had 11 IT groups running their own applications," he said. "When you have three or four different systems that do the same thing, it's difficult to move ahead efficiently."

This time around, the combined companies are carrying about 600 applications, with 325 on the AT&T Wireless side and about 275 with Cingular. The CIO's new goal? Slash that 600 to less than half, with a specific target of "fewer than 300" applications by the end of 2006.

But overlapping applications was just one of the issues his team had to deal with. Because the combined company would have to support more than 50 million customers (21.5 million coming from AT&T Wireless), Arroyo quickly realized that his infrastructure was simply insufficient to support that large an operation.

"The complexity and the demands were substantially larger," he said. "We have to scale to a different level, and you have to address the scale points in software and hardware."

One decision was to move both companies quickly to a combined database that was native to neither company. "All of our POS service, it goes to this enterprise directory with a common view of all customers," Arroyo said.

That database "was based on an Oracle database with feeds from all of the disparate sources: about 56 different platforms with the right data elements," he said. "The first step was to create a common experience. We have not yet moved the complete base of customers" to the database that will be the permanent repository.

The combined databases took up more than 2.5 petabytes of space (each of which equals 1,024 terabytes, or the equivalent of 500 billion pages of printed text). Of that, the AT&T Wireless database—although smaller—took up slightly more space. "We had rationalized more of our system, had reduced more duplicate customer information. [AT&T Wireless] had not invested as much as we had," he said.

The first trouble spot was an area near and dear to any CEO: getting paid. "The platforms that supported our billings, they all were outgrown," he said.

The reason Arroyo makes application consolidation such a high priority is that it has a severe financial ripple affect throughout IT, to a much greater extent than some IT execs realize.

"Our primary focus is turning off applications" and then realizing savings from "the server infrastructure, the associated support, licensing and all of the third-party expenses associated with the maintenance of that application and its equipment," Arroyo said.

To support all of that data, the CIO is making modest upgrades (such as sticking with HP Unix servers but moving to a higher-end version of those servers), bringing in more Linux and pushing more applications on fewer—but larger—servers. "We'll get better economies," he said.

One example is what Arroyo is doing with backup and disaster recovery plans. Until the merger, Cingular had two disaster recovery backup sites (in Atlanta and Dallas). The merger brought in a third, which happened to be in Seattle. The convenient geographic positioning of those three data centers made Arroyo comfortable that most disasters would be unlikely to impact all three or, for that matter, even two of them. "They're pretty well separated," he said.

His cost-cutting move: He only needs to back up one-third of the data at a time.