Authorized System House of
Transactions have always been the beating heart of business. But in the digital age, the volume and nature of those transactions has changed considerably. The ability to process a growing amount of increasingly diverse transactions has become a competitive necessity.
Until recently, only certain companies, such as high-frequency trading firms or digitally native companies like Facebook and Google, built their business models around extreme transaction
processing—the ability to process at least 2 million transactions per hour.1 Today, however, enterprises across industries must figure out how to ingest and process extraordinary numbers of transactions
from an expanding variety of sources concurrently, quickly and reliably, with no tolerance for latency, loss, inconsistency or failure.
“Transaction processing is becoming far more distributed, the data involved far more heterogeneous, and it’s blowing out the limits of what traditional transaction processing systems are capable of handling,” says Tony Baer, principal analyst covering information management at Ovum Research, a market research and consulting firm.
In this report, readers will learn:
BEYOND ORDERS AND PAYMENTS
In many ways, digital business innovation depends on transactions that are measured in milliseconds or microseconds—millions of them per hour. Companies ranging from industrial giants like GE to
startups like Ripple Labs—which sells services based on block-chain technology, the distributed ledger system that underpins Bitcoin transactions—require near-real-time processing of billions of transactions to deliver their digital products and services. “The speed of transaction processing directly impacts business success overall, and in some cases, the results of the transactions affect other transactions at nearly the same time,” observes Carl Olofson,
research vice president at market research firm IDC.
Furthermore, today’s transactions go beyond traditional orders and payments. Even payments have expanded: They now include machine-to-machine transactions, micro-payments, peer-to-peer money transfers, pay-per-second products and services, mobile apps and purchases through TV set-top boxes.2 And any other data-generating event— sensor readings, streaming information from
Internet-connected products and activity recorded by in-store beacons, to name just a few—can potentially be a critical transaction (see Figure 1, “More Devices, More Transactions,” above).
To compete in the digital economy, most enterprises will need to transform their transaction processing systems, rethinking their transaction processing architecture and databases for extreme speed and extremely diverse workloads. No single technology solution will fit every organization or business case, and the evolution to extreme transaction processing is as much a change management challenge as a technology one. Companies will have to establish the value of extreme transaction processing for their businesses to develop the right processes and systems to meet their particular high-volume processing requirements.
DISTRIBUTED, DIVERSE— AND DAUNTING—TRANSACTIONS
The digital economy is already generating data at an exponential rate. A study by data storage systems vendor EMC (now Dell EMC) and IDC has predicted that 44 zettabytes of data will be generated by 2020, a tenfold increase compared to 2013.3 The amount of information that organizations must process will grow dramatically, driven in part by the increased volume of transactions that will be performed not just by humans but by machines, as well.
“We’ve got more data and more pipes that can transmit it faster than ever, so we can start to perform more actions in real time,” notes Ovum Research’s Baer. That makes certain types of new real-time interactions—each a digital transaction—“not just possible, but expected. There’s an expectation of instancy,” he said.
And expectations of intimacy, too. Transactions can be used to deliver a more relevant customer experience. Location-based marketing used to mean a billboard by the side of the road. Today, retailers that can process and analyze masses of consumer transactions—individuals’ GPS locations or check ins, recent purchases, retail beacon information, etc.—to deliver customized marketing messages more efficiently and effectively in the moment.
Meanwhile, manufacturers like GE4 and Ford Motor Company5 are reinventing themselves to sell data-based services, such as transportation services, maintenance management and real-time performance enhancements, along with their products. Such changes increase their universe of digital transactions exponentially. “There’s a huge shift in the product economy to delivering everything as a service,” Baer says. “And that is creating a demand
for real-time transaction processing.”
CONNECTED DEVICES: A NEW SOURCE OF TRANSACTION DEMAND
Internet-connected devices are another source of demand. Research and advisory firm Gartner predicts there will be 20.8 billion Internet-connected things in 20206, and these will generate potentially trillions more transactions. “Companies can now pull
telemetry data from large earth movers or operational data from smart energy meters,” says Mark Peacock, principal and leader of the IT transformation practice at consulting firm The Hackett Group.
In addition, global smartphone penetration will surge to around 49 percent in 2020 (and to 70 percent in developed markets), up from 33 percent in 2014, according the GSMA.7 Each one of those mobile devices has multiple sensors, including a magnetometer, barometer, thermometer, gyroscope, proximity sensor, accelerometer and light sensor. What’s more, IDC predicts 237.1 million wearable devices will be shipped by 2020.8 In other words, it’s no longer just the NASDAQs and Googles of the world in the billions-of-transactions business.
“Companies are trying to figure out what the return on investment will be if they invest in the next generation of technology to facilitate these tremendous new transactional data flows,” says Oliver Marks,
vice president of digital and IoT at analyst firm HfS Research. They are facing “incredible pressure to be far more agile and fast moving,” he adds— pressure that has become a catalyst for new digital
business models.
A GROWTH ENGINE FOR THE DIGITAL ECONOMY
The number of new business approaches enabled by extreme transaction processing are at least as great as the number of vertical industries that can benefit from them—“which is pretty much all of them,” says IDC’s Olofson. “This could involve things like streaming
data from smart meters impacting power rates in real time for a utility or traffic data combined with scheduled pick up and deliveries (including last minute changes) affecting chosen trucks and routes for a logistics company” (see Figure 2, “Digital Transactions of the Future,” right).
Two examples of transactions that could become increasingly important for enabling new business models in the near future are the Internet of Things (IoT) and blockchain.
IoT
IoT devices are endpoints that may generate a string of transactions. Consider these steps in Monsanto’s seed management platform:9 A sensor continually reads the temperature on a truck full of seeds. Another reads the truck’s location. Each reading is a transaction, processed by an application that checks whether the seeds are becoming heat stressed. If the seeds are at risk of being compromised, the truck is dynamically rerouted—in another transaction—to either the nearest cooling station or the head of the grain processing line.
Amazon Dash10 offers another example. The small, Wi-Fi enabled device is designed to be kept near frequently purchased household products like laundry detergent and paper towels. Individuals can
repurchase those items with a single tap. Each tap launches several transactions. First, Amazon sends a message to the Dash user’s phone indicating an order has been initiated. If the customer doesn’t cancel the order, a payment transaction is then completed. After that, the transactions required to ship the item from the distribution center to the customer’s door begin.
Blockchain
A blockchain is a decentralized digital ledger of transactions that are processed by a distributed or peer-to-peer network. To create a secure record, a transaction—called a “block”—is transmitted to each node on the network. Using cryptography, each node approves the transaction as valid, and then encodes the block in a “chain” of multiple transactions that each node stores.11 In other words, a single transaction must be validated, synchronized and recorded across the entire blockchain network instantaneously (see Figure 3, “Modern Transactions Are Distributed,” on page 4).
Blockchain technology first made headlines as the foundation for new types of financial transactions, beginning with Bitcoin in 2009. According to financial consultancy Greenwich Associates, financial and technology companies will invest an estimated $1 billion in blockchain technology in 2016.12 By the 2020s, professional services network PwC expects blockchain-based systems will reduce or eliminate many points of friction for a variety of business
transactions; individuals and companies will be able to exchange a wide range of digitized or digitally represented assets and value with anyone else.
For example, blockchain-enabled transactions could revolutionize supply chains14, which at a basic level are a series of transactions required to get a product from place to place. Because the system
enables users to document transactions safely and securely, a blockchain could be used to create and document each transaction within an extended supply chain, from the manufacture of a product to its distribution and sale, thereby reducing the time delays, added costs and human error that plague supply chains today.
THE CHALLENGES OF TRANSACTION PROCESSING TRANSFORMATION
According to research by The Hackett Group published in 2016, companies that are top performers at managing transactional data collect more data about their customers than other firms. The
additional data, Peacock says, drives exponentially more analytics, insights and activities, leading to potentially more transactions through incremental promotions, sales and customer service interactions (see Figure 4, “Top Performers Handle More Data,” on page 5).
But capturing enormous amounts of concurrent transactions in large and rapidly growing databases with full data consistency, high availability, instant failover and zero data loss takes new technology,
people and processes that most large enterprises have not employed before. Doing so presents significant challenges for the average large enterprise, whose existing IT infrastructure is built to process traditional transactions such as orders, payments, enrollment, account creation or customer cases from centralized applications in batches.
Modern transactions increasingly can and must be processed where the data is received (in the initial recipient database), whether the source is a sensor, an Internet-connected device or a smartphone.
Doing so is faster, and it reduces network traffic and transmission costs. What’s more, a transaction may span multiple databases, and each database must be synchronized. But what sort of database is
appropriate as an initial recipient of such data? “A classic database management system cannot accept streaming data fast enough to process it immediately and cannot perform transactions fast enough,” says IDC’s Olofson.
With distributed, fast moving transactions come requirements for improved performance, reliability and efficiency of transaction processing applications and databases. Modern transaction processing systems must also support real-time monitoring
of the transaction stream to maintain compliance, traceability and data integrity, to ensure data is backed up and secure, and to prevent disruptions or breaches. Companies also need to control the
administrative costs associated with managing exponentially more data and faster networks.
It’s not just new technology. In addition to investing in new core systems and capabilities, companies will need to enlist skilled professionals who have experience with the technologies for real-time processing of large transaction volumes. Extreme transaction
processing can very rapidly create outsized crises. As Peacock, who was the CIO of a travel technology and services firm prior to joining The Hackett Group, points out, “minor performance problems that
wouldn’t typically have a huge impact when you’re processing a couple hundred thousand transactions suddenly become very critical when you’re processing millions of them.” So far, there aren’t many IT professionals skilled at finding the source of a small
problem in a sea of transactions; until recently, processing millions of transactions per hour was a niche capability.
Early adopters have developed such talent internally. Other companies can expect to do the same. “It will be a challenge to find the people to architect, build, run and improve these systems,” says Bryan Degraw, senior director at The Hackett Group. So will finding professionals who have experience managing, and extracting value from, these very large data sets.
Change management can also be a hurdle. “The biggest challenges generally involve people organization and internal politics,” Olofson says. “People have to get used to working in a different way, with
different processes. Some organizations have a tough time with that transition.”
EXTREME TRANSACTION MAKEOVERS: WHERE TO START
For startups, putting the people, processes and systems in place to support extreme transaction processing can be more straight forward, as these companies are starting from scratch. For enterprises with decades of legacy technology business processes, and employees in service, it’s much more difficult. A successful transformation depends on understanding the business problems
extreme transaction processing would solve, and then putting the appropriate technology in place. Here are five steps:
1. Define which transactions have value. “Consider the types of transactions and data that are relevant to the products and services you deliver and the demands of your customers,” advises Ovum Research’s Baer.
A sanitation company, for example, may save half a million dollars annually by installing $150,000 worth of IoT devices in garbage bins and dumpsters to monitor when they need to be emptied, says HfS Research’s Marks. But in many cases, the business outcomes from extreme transaction processing—and therefore its value—may not be instantly clear. “There are nuggets of valuable information,” he says, “but unless you know what those nuggets look like, it makes no sense to buy the spades and start digging.”
2. Start small, and experiment. Once the potential business case for extreme transaction processing is clear, companies should begin to experiment with new transaction processing technologies. “The only way to learn is by doing, putting together pilots and tests and proofs of concepts,” says The Hackett Group’s Peacock.
Key technologies to evaluate, according to analysts and other experts, may include:
3. Consider the cloud. Because companies can payonly for the computing they use, the public cloud offers flexibility that supports the shifting workload levels and volumes of data inherent in extreme transaction processing, Olofson says.
4. Examine your business processes, as well as the IT infrastructure, databases and applications that support them. Pilot projects may point to ways to simplify and streamline, Marks adds. “Otherwise, despite the best of intentions, you put layers of new complexity on top of old complexity and you’ll choke the new data to death.”
5. Choose your solutions. The last step, as with any major transformation, Baer says, is to select the appropriate technologies to support extreme transaction processing and deliver timely intelligence to the enterprise.
“Any business that does not evolve its transaction processing technology runs the risk of being unable to make use of live data in the moment, as it is needed,” Olofson says. “That risk is compounded by the fact that competitors are probably developing
that capability, so those that don’t will be seen as slow and unresponsive to market needs and customer requirements.”
Ultimately, different businesses will approach extreme transaction processing in unique ways. But the agenda is clear: It’s time to figure out what business value new data-producing transactions
can deliver, and then put in place the systems and processes to capture it. Extreme transaction processing capabilities are the foundation for the digital transformation that will deliver the differentiating products and services of the future. Those companies
that aren’t investing in these new systems and processes will not be able to keep up with competitors that meet today’s transaction processing demands head on.
