Once they get a foot in the door, enterprise software vendors will predictably start pushing pilots of their latest and greatest wares, from hybrid cloud to in-memory databases.
For customers, testing these extensions and enhancements can make sense, provided there’s a strong business case, supportive leadership, and the right success metrics. Innovative organizations are always looking for technologies that promise to streamline business processes or create new business opportunities.
But it’s also true that pilots eat up finite resources like people, time, and infrastructure. In the best case, they inevitably take attention away from other opportunities. On the other hand, piloting a new technology in some limited context is preferable to putting it into production prematurely. Consider the case of Under Armour, which last October blamed its third quarter results on “change management issues” related to a system migration to SAP’s S/4HANA, which it said disrupted its supply chain, causing “delayed shipments and loss of productivity.”
The point of a pilot is to test new technology in a non-critical context before making a go / no-go decision on putting it into production — and to be willing to kill or rethink a system if the technology is not ready or the organization is not ready to put it to work.
While most organizations have some process for the due diligence around greenlighting development, “what’s not so pervasive is when to kill a project,” says Paul Dalpiaz, Senior Director Of Technology at United Way for Southeastern Michigan. And once something starts rolling forward, this momentum makes it “really tough” to stop or rethink the project, Dalpiaz says.
It’s understandable why those who approved the pilot, the project leaders and senior executives, have a hard time objectively evaluating its success (or failure). They are, after all, the same people who thought it was a superb idea in the first place. As Alan Zucker, founding principal of Project Management Essentials, a project management consulting firm based in Arlington, Va., told CIO last September: “There is the ostrich bias, where we tend to ignore negative information, or the confirmation bias where we look for information that supports our point of view.” Muddying the waters more, Zucker says, is overconfidence bias, “where we are too confident about our abilities.”
The point is, once a pilot is approved, funded, and underway, time and resources have already been expended. And anything that pulls away from the IT budget, pulls always from innovation.
The vast majority of decision makers (89%) believe their organizations should be spending more on innovation, and most (77%) believe one major obstacle is spending too much “keeping the lights on” — basic maintenance and support of systems and IT infrastructure — according to to “The State of IT Innovation: Priorities and Challenges,” a global survey of IT and finance decision-makers, sponsored by Rimini Street.
The trick is, even when organizations try to innovate with a new technology pilot, they don’t always make the right choices.
When should you cut bait, count your losses, and move on?
1. Missing Metrics and Use Cases
Subjective opinion dooms a fair assessment. That’s why project management experts stress the importance of creating specific, agreed-upon success metrics before starting any pilot.
“Without that pre-assigned success criteria, and an intention to measure it, you can’t know if you added value,” says Rich Mironov, a project management guru who provides full-time and short-term product management help to large and small tech companies. Maybe this is why, he speculates, IT often shifts its goal from delivering value to “delivering the thing we promised.” “They don’t circle back to value,” he says, adding that this unfortunate habit diminishes IT’s reputation.
Pre-established metrics will also blunt after-the-fact finger-pointing and blame games. Notwithstanding the laudable corporate mantra about “risk-taking organizations,” few people really crave being associated with a failure. As JFK answered a reporter’s question about the Bay of Pigs: “Victory has 100 fathers, and defeat is an orphan.”
Consider using a “premortum,” instead of a postmortum. Get everyone to write down two reasons why the project will fail, Dalpiaz says. This accomplishes two goals: It identifies potential problems, and gets everybody on record, so they can’t do an “I told you so.” The premortum list can also serve a checkpoint with the team, he adds.
2. New Software Isn’t The Only Way to Innovate
Enterprise software vendors are quick to promote their wares as the best way for you to reach any given business objective. But even if they have identified a goal that makes sense for your business, is their new software or their latest upgrade the best way to achieve it? Or could an existing platform serve the purpose? It’s a question that needs to be asked, each time.
Enterprise IT vendors could do more to help. As Rimini Street’s “The State of IT Innovation” research found, 81% said they would like to see more clarity about the product roadmap so they can understand how it fits within their own plans for IT architecture. In addition, 80% would like more help optimizing their existing IT infrastructure. Fewer than three in ten (28%) respondents reported being completely satisfied that their organization’s enterprise software providers are helping them innovate faster and accelerate their business strategy.
Much as enterprise software vendors portray themselves as fountains of innovation, many organizations achieve innovation, not with the base software they purchase, but with their own customizations. The only trick is those customizations must be properly supported. Rimini Street receives roughly 30,000 cases to resolve a year from clients of third-party enterprise support business. Sixty-five percent, or almost 20,000 of those cases, are on custom code or interfaces, integrations, or performance tuning. These customization issues are not resolved at all by vendors like SAP and Oracle, because they only support issues with the base software.
3. Watch Your Language
The core problem may be that different constituencies have different-often undeclared-definitions for the word “pilot.” For some, the word translates to “investigation” or “experiment,” and they therefore welcome either outcome (success or failure). ‘Gosh, it didn’t work out, and aren’t we glad we avoided building that infrastructure or whatever,” Mironov says.
For others, often on the business side, “pilot” denotes “the first implementation of our solution or the first customer in a live setting.” This definition makes it extraordinary difficult to close-up shop. As Mironov puts it: “Once you’ve signed up a customer, there’s never an opportunity to tell them, ‘Just kidding.'”
Senior management changes can advance or block a pilot, so it’s worth considering the stability of the organization’s leadership, and how changes there could impact the pilot.
“We were doing a pilot for analytics,” remembers Tery Lockitski, who at the time was Managing Partner with Ideology. “We provided an excellent business case that demonstrated a quantified supply chain value meeting very well-vetted KPI’s.” But there was a change in senior leadership, with a new team who had different opinions about the initiative and the pilot, she said. They agreed with the business case, just not the approach. So the work was put on hold. Eventually, the pilot was set back on path, but with different sponsorship, scope, and funding.
4. Too Many Dependencies or Won’t Scale
A successful pilot that has too many external dependencies is not a success, says Zahiri. “In some cases, a project requires technology from multiple partners. Each has its own development roadmap and deployment protocols, and if a new feature on our pilot requires one of them to push their timeline, that’s not always possible.” These dependencies become a maintenance and development issue too, he adds.
Officially authorized pilots are at least visible (part of IT, approved by the board, metrics-driven) and can be dispatched if they aren’t achieving goals or have other issues (say, security flaws). But then there are skunkwork projects that may look like sanctioned line-of-business (LOB) applications but aren’t.
In such “shadow IT” projects, Dalpiaz says, “people create their own or buy one, and create something off to the side,” adding a layer of ERP processes to a subgroup of people to get a job done. These pilots linger, and are problematic. “A LOB that takes that on is running with scissors,” he says.
Sometimes, a pilot can be a success in a constrained environment, but won’t scale or meet complex business process dependencies, and has to be abandoned for that reason. Several years ago, when she was Chief Technology Officer at Pepsico International, Lockitski was working on an S&OP pilot with SAP and IBM. The pilot was successful in one business unit within one country, but was ultimately abandoned “because it wouldn’t scale or meet the needs of the other business units in other countries,” she says. “The level of effort to globalize this initiative would have involved significant upfront and support costs.”
On the other side of the coin, when she was the Vice President of Supply Chain at Hitachi, Lockitski ultimately scrapped a custom, third-party transportation management system at the midway point. “We realized a much better transportation management solution worked with their existing ERP.
The costs associated with supporting a large SAP footprint was already astronomical, so the total cost of ownership (TCO) to make it bigger by adding a full transportation management solution was too high at the time. However, by diligently tracking their existing transportation and logistics costs over a 3-year span, and comparing it the TCO of what the pilot was trying to prove out, the SAP management platform was eventually implemented.
Nevertheless, she says pilot are a worthwhile venture and money well spent. “It’s always good for future requirements gathering and strategic planning,” she says.
In Lockitski’s experience, smaller companies (<$500M) tend to have more success with pilots, since they can often adopt a tactical solution that doesn’t require “investing with an enterprise software company.” “Many times,” she says, “the pilot turns into the production solution.”
5. Total Cost of Pilot (TCP)
As noted above, pilots cost time, money and people. As most IT organizations struggle to “keep the lights on,” this resource drain is a real downside, even if the pilot at hand isn’t particularly ambitious.
What about those pilots involving the enterprise platform vendor? These tend to be sold in terms of “shared risk,” by which the ERP vendor means little or no direct, up-front costs. However, it’s worth remember that being involved in a vendor’s pilot means becoming one of many early adopters (read, “beta testers”). Also beware of contract terms that may lock you into paying for the software, whether it’s working for you or not, once the pilot period ends. Viewed in this light, there may be far less risk to an organization to exit vendor’s product cycle, and redirect these savings toward innovations that are specifically needed by the organization.
And what about those ambitious “rip and replace” deployments, such as SAP S/4 HANA or Oracle Cloud? Can a pilot dependably demonstrate that such big moves will make a big difference? Is it customer need that prompts a vendor to upgrade its robust, stable, 30- to 40-year-old back office solution? Is a forced march to a “Mandatory Evolution” really in the customer’s interest?
A Good Death
Finally, when a pilot needs to end, there are good and bad ways to handle the execution.
Internal consensus is important, since there will be other projects in the future requiring buy-in. So the politics of this situation demand a gentle touch.
Dalpiaz recently found that an application that was just about to built almost from scratch was basically obviated by a commercial product being deployed by another group. “We had to go to that first team and say, ‘Open your eyes and look at this.’ It was obvious, and the homegrown application, which had already been greenlighted, was halted.
What would have happened if the first group hadn’t killed its pilot? “I would have pulled in finance or an executive from their area,” says Dalpiaz, who doesn’t recommend such a heavy-handed move unless absolutely necessary.
Over the years, Mironov says, he has done a lot of coaching about “how to politely and humbly thank someone for their good idea” but then not pursue the idea.
He says it’s important to separate the request from the requester. One diplomatic approach is to praise the goals of the pilot, but point out its shortcomings. “This separates goals from implementation, and gives everybody a face-saving moment.”