Handling the flow of growing amounts of messy data from multiple sources throughout the enterprise is a complex process. Whether moving transactional data so that it can be reported upon, migrating application data from old systems to new ones, or integrating data from external suppliers or partners, the reliable management of data streams can present a challenge to IT organizations. Because these movements of data can be scheduled on a regular basis or executed based on specific triggers, ETL and data warehousing processes inherently lend themselves to automation. Automation brings a host of benefits to these processes including faster delivery times, improved productivity, reduced cost, decreased risk of error, and higher levels of data quality, among others. But the automation tools commonly available often don’t provide the flexibility that is needed for granular and end-to-end automation. For most organizations a different approach, a unified workload automation solution, is needed to achieve all the benefits that automation can provide.
The Challenge of Data Overload
Data moves through organizations in a constant state, going from one individual, department, or office to the next. As new and different data sources like social media and wearables become part of the data pool that businesses collect and analyze for decision-making, data warehousing processes are becoming increasingly complex.
Back in 1930, economist John Maynard Keynes predicted that the working week would be cut to 15 hours as technology advanced and living standards rose. But despite the technological innovation over the past 85 years, people are working longer hours and taking fewer vacations than ever before.
Similarly, despite the advances made in the computing industry, IT is more pressed for time than ever before. The sheer number of technologies, applications, and products that IT organizations must maintain and update has surged over the years. In fact, according to research by Gartner, most large organizations have more than three workload automation tools implemented in their environment.
We had the opportunity to talk with many PowerCenter users at Informatica World in Las Vegas in May and they shared a lot of information about their IT Automation requirements, at the application and enterprise level, and how it could be improved. Application vendors, as we have previously addressed in our earlier posting, focus their resources (money and time) in developing the core capabilities of their products rather than enhancing their solutions with a robust automation system necessary to address today’s business and operational requirements.
In recent years, the democratization of analytic, reporting and BI solutions has become a driving force in the growing complexity of data integration and data warehousing models. Add to the equation the growing complexity and volume of information thanks to Big Data, and it’s no surprise that the underlying ETL and data warehousing processes to integrate and access data from multiple sources is becoming increasingly complex.
Attending Informatica World presents the opportunity to speak with IT professionals about one of the strongest use cases for job scheduling and workload automation: the end-to-end automation of ETL, data warehousing and business intelligence (BI) processes.
Last month Gartner announced the retirement of the Magic Quadrant for Workload Automation. The announcement has created significant buzz amongst various social media channels, such as LinkedIn’s Enterprise Job Scheduling & Workload Automation group. Here’s the opening summary courtesy of the Gartner announcement: