• Home
  • About
  • Team
  • Experience
  • Engagements
  • Papers
• Focus
  • Virtualization
  • High Availability
  • High Performance
  • Cloud Computing
    • Challenge
  • Information Management
  • Data Integration
    • Financial Services
  • Business Intelligence
    • Column Oriented Database
  • Software Development
    • Overview
• Services
  • Voice & Data Infrastructure Management
  • Data Center Analysis
  • Supply Chain Management
  • Channel Enablement
  • Analyst Relations & Influencer Marketing
  • Go To Market
  • Professional Services
  • Sales & Marketing
  • White Papers & Articles
  • Primary Research
  • Market Share Analysis
  • Market Sizing & Forecasting
• Contact

Financial Services - Meeting the Data Integration Challenge

Challenge

Data in corporations is in every conceivable format, structured and unstructured, and in diverse heterogeneous environments.  Adding to the complexity data now comes in all speeds, real-time, near time, historical, archival.   The challenge is to integrate, centralize, and make useful to the business / organization all available data regardless of complexity or speed.

Opportunity

Effective data integration will drive cost reductions, operational efficiency, competitive advantage, regulatory compliance, and productivity.  Next Generation  Business Intelligence (BI)  and Operational Intelligence (OI)  applications today are starting to leverage cross enterprise and in some cases cross industry data integration efforts to provide an enhanced customer experience there by ensuring customer retention and the opportunity to efficiently and effectively up-sell and cross-sell by providing a single view of the customer.

Operational intelligence is more of an intra day and intra hour process while Business Intelligence is more strategic leveraging more historical or static data.

Who benefits?

• Business users who will be able to have a single view of the customer and products across the organization 

• Businesses that  will be able to meet compliance requirements without having to create more work / cost by pushing compliance down to individual divisions

• Business analysts who are looking for trends and correlations between products, sales, promotions, and other relevant events

• Business users who are trying to come up with better products and services or to become more competitive can use all available data to benefit of the business

• Businesses will benefit as business value is increased due to the exposure of  previously hidden shared creative intellect or  "good ideas” (value) that is imbedded in data and transactions and can now be extracted and used.

Meeting the Challenge

In Financial Services firms across the globe a continually growing tsunami of data must be analyzed sometimes in near real time in order to come up with the answers that are required to conduct day-to-day business.  Market prices, historical experience, interest rates, and numerous risk factors feed complex models, which will run hundreds of times for each assessment.  The data must be reliable, instantly accessible, continually updated, and selectively replicated.  This requires a strong information management foundation, a unified, adaptable approach that enables central management and distributed operation.  Effective solutions must protect vast investments in proprietary applications, while enabling continual evolution to new technologies and architectures.

The intensity of data access, analysis, and continual flow presents the information technology organization with numerous challenges.  At the top of the list:  get the information to process into a centrally managed database.  The database itself may be distributed, but it must function as a unified system, to ensure:

• Maximum performance:  with potentially millions of simulations run daily, high volume data access and processing is critical.

• Data integrity and quality:  data must be complete, current, correct, and readable.

• Continuous availability:  all of the data must be readily accessible, all of the time; there can be no downtime.

• Data Security:  data must be protected from tampering and unauthorized access.

This represents an enormous challenge to many of the broker/dealers and investment banks that grew so rapidly in the ‘80s and ‘90s.  There just wasn’t time for integration.  The result was the independent creation of numerous systems and databases, all acting as individual data silos, unrelated to one another, in different formats, based on products from multiple vendors. 

There are really two issues.  The first is to bring the data together under one logical roof.   Middleware offers the vehicle to get this done.  Since the massive amounts of product and market data comprise many different types of data, they are most likely contained in multiple, special purpose databases.  Data replication software provides the vehicle for intelligently moving needed data to the right location, in the right format, and in near real-time as values change, without compromising the function of the special-purpose databases. 

There is a second, much larger issue:  how to provide data access to legacy applications without reengineering them?  Most applications currently in place were custom developed, and they are often at the heart of the firm’s competitive differentiation.  Countless staff-years go into development and quality assessment to ensure that they are completely reliable and function non-stop.  Any change will require repeating the arduous quality assurance process.  Is there an alternative?

Again, the answer can be found in middleware solutions.  Some large firms are turning to a sort of “hub and spokes” model (Figure 1), creating a central database and using middleware adapters to connect with the legacy system and application “spokes.”  Middleware applications handle format differences between different databases and programs, serving as protocol and format “adapters,” without impacting the legacy applications.  Some IT organizations choose a very deliberate, case by case approach to adapters, to minimize performance impact and ensure quality.

The creation of centralized databases raises a critical need for scalabiliy and overall system performance. By definition, these are very large, dynamic databases.  They need to be able to grow flexibly, without disruption, and deliver the rapid response required by real-time applications and the trading functions they serve. 

Not all of this needs to be implemented in-house, although larger firms, with their scale advantage, typically do.  In an outsourced model, the firm becomes the spoke to the service bureau’s hub.  A decision to outsource really comes down to the need for scale and competitive cost structures.  A service bureau can take on the problem of managing market data, integrating multiple feeds, handling data arbitration, achieving economies of scale that can be passed along to outsourcing clients. 

Figure 1.  Hub and Spokes Model – Integration across diverse applications and platforms

Multinational or global operations require the same integrated data environment, encompassing multiple databases in major business centers:  New York, London, Hong Kong and Tokyo, for example.  Data networks offer the necessary bandwidth and redundancy, so that they can be reliably connected.  Again, our difficulty is with the volume and time-critical nature of the data required by all locations.  The solution is to selectively replicate only that information that really needs to be in multiple locations, at frequencies that suit the dynamic nature of the data and timeliness requirements of applications.  Some categories of data, like a trade, or update of a market price, may need to be replicated instantly, while others may occur hourly or at the end of the trading day.

Finally, data security presents significant operational risk to the organization.  While security overall is a major topic at all levels, and is generally addressed in numerous application vehicles, data security must also be addressed within the database itself, and in all data communications. Row level access controls, security level classification and access protection, change management and tracking, data encryption, and much more need to be factored into the overall data management schema. 

Conclusions/Recommendations

For many IT managers in the financial services industry the following areas are seen as particularly important:

• Knowledgeable, dedicated people – who understand the financial services business environment and application arena, and will work as partners. 

• Support in times of crisis:  When disaster strikes, as it occasionally does, IT managers need a vendor that will do what it takes to keep the operation going.  People are still talking about the aftermath of  “Black Friday” in 1987, when trading volumes stressed systems literally to the breaking point.  Some vendors came through, some did not.

• Access to and partnership with leading technical talent – Financial services information technologists push the limits of data management technology.  They value peer-to-peer interaction and collaboration with top vendor talent.

• Tools and Middleware – of all the product information we discussed, development tools and middleware jumped out as areas of critical interest.  Data and protocol adapters, and tools to create these and other middleware applications, were seen as critical to solving data integration and “many to many” interface needs.

• Standards support and multi-vendor interoperability – look for a business partner that knows how to create a unified data management environment, and drive the effective flow of information, spanning diverse technologies, vendors, locations and geographies.  Seek solutions that maximize the use of open standards and interfaces, maximizing future flexibility to grow and change.

Considering these factors, and the product needs discussed above, any viable long term solution needs to provide for the unimpeded flow of information across diverse technologies, vendors and locations. Further more, an effective risk management system requires, at its foundation, a unified, adaptable approach to information management that allows central management and distributed operation, working across diverse technologies and vendors.