Lots of chat about fraud but none about the Bayesian influence and the architecture of a system which would influence decades of content processing systems.
And in a similar regard, the statistical machine translation software that would be the basis of all machine translation and analysis software to come.
4 February 2022 (Zurich, Switzerland) – I have followed Autonomy since 2009 when Nick Patience (former co-founder of The 451 Group) introduced me to the company. I still have my duck from Legaltech 2009:
But my follow intensified when the now infamous Autonomy/Hewlett Packard litigation blew up the eDiscovery ecosystem, a story that might now be reaching its conclusion. UK Home Secretary Priti Patel has approved the extradition of British tech tycoon and ex-CEO of Autonomy, Mike Lynch, to the U.S. to face criminal fraud charges. The decision comes after Lynch, the founder of Autonomy, lost a multibillion-dollar fraud action in London last week. As most of my readers know, Autonomy was sold to Hewlett Packard (HP) for $11bn (£8bn) in 2011. It would not be too long before HP announced an $8.8 billion write-down of its investment in Autonomy – citing “serious accounting improprieties and outright misrepresentations amounting to fraud.” In 2015, HP filed a lawsuit in the UK against Lynch, alleging he was involved in the publication of false accounts that overstated how valuable Autonomy’s business was.
The Autonomy/HP story is well summarized in this Wired magazine article but I want to make a few additional points that have been missed, and talk about the tech – and the nefarious people behind it – and how they laid the building blocks for today’s e-discovery industry.
In September 2016 HP announced an $8.8 billion deal to sell the bulk of its software division to Micro Focus International PLC. The deal represented the divestiture of Autonomy. In 2017 the Harvard Business School added it to its famous “Case Study” series and titled it “Failed M&As — An Introduction to HP’s Acquisition of Autonomy”. HP has always claimed that the write down was due to fraud but the Harvard case study shows there were many other factors belying this failed merger.
The start was rather simple. In 2010, shortly after HP appointed Leo Apotheker to lead the company, the board approved the acquisition of the UK-based software company, Autonomy. HP, well known for its computer hardware, thought the synergies it could have with Autonomy coupled with its brand recognition would give it a strong presence in the software market.
The Harvard case study discusses all of this, and the fraud committed by Autonomy. But it also provides insight into the impact of cultural differences on the HP-Autonomy debacle, offering a multitude of examples, and providing guidance about how-not-to-do-due-diligence-for-cross-border-mergers. It addresses the cultural factors that can screw up a “M&A synergies framework”: poor communication, mismanagement, the difficulties of merging accounting and finance teams, wrongly placed marketing optimism, and earnings mismanagement. The failure of the Autonomy acquisition was due to more than just fraud.
The marriage of the two firms was rocky from the onset, with HP’s shareholders decrying the 79% premium HP paid as abhorrently high. It was not too long before HP announced an $8.8 billion write-down of its investment in Autonomy – citing “serious accounting improprieties and outright misrepresentations amounting to fraud.” The allegations of fraud were of course rebutted by Lynch and marked the beginning of a long, disastrously messy public relations battle fought by Autonomy’s ousted CEO and members of HP management.
But lost in all of this was what Autonomy brought to the table: Bayesian influence and the architecture of a system which would influence decades of content processing systems. The following is a simplified version of a more detailed piece I had written a few years ago and so it leaves out some detail and nuances. But its premise remains the same: Autonomy changed the rules of the game.
Autonomy’s management seized upon the “automatic” function of Bayesian and other numerical recipes. Competitors’ search and content processing systems were not positioned as “automatic.” From 1996 when Autonomy opened for business, the company differentiated its approach to content processing by suggesting its method was lower cost, more efficient, and more technologically sophisticated. The competition was caught by surprise. Autonomy then layered on other benefits that drew inquiries from prospects. For example:
• Autonomy had invented a “dynamic reasoning engine” pictured in an early patent as a sophisticated, complex system
• The “black box” in the dynamic reasoning engine could understand content and automatically assign a document to a category even though the specific word for that category did not appear in a source document. It, in effect, introduced the “black box” concept to hundreds of applications, and it was adopted by scores of technology vendors.
• The system was based on advanced statistical procedures, which in the late 1980s, were not widely known nor understood by such search industry leaders at the time as Fulcrum Technologies, IBM, OpenText, Teratext, and Verity.
• Autonomy’s engineering allowed distributed processing, painless scaling, and such advanced features as dynamic caching. Every “information governance” vendor copied the approach.
• The Autonomy technologies were immediately applicable to fraud detection, customer support, Web portals, traditional research including business intelligence, content archiving, and dozens of other problem areas with which most businesses struggle on a daily basis.
Autonomy was a competitive firm. Search and content processing companies continued to focus on complex methods used to allow an employee to locate a document. Autonomy offered a black box and strong arguments for the cost effectiveness of Autonomy’s sophisticated system and methods.
The history? The founders – Michael Lynch and Richard Gaunt – were from Cambridge University where Autonomy began as a class project. Lynch and Gaunt (and their numerous sales professionals) had emphasized that Autonomy’s strength lies “in a unique combination of technologies that employs advanced pattern-matching techniques utilizing Bayesian Inference and Claude Shannon’s principles of information theory”. Autonomy’s technology has its roots in Thomas Bayes, and the 18th century Presbyterian minister’s mathematics.
Bayes’s Theorem sets forth a method by which one can derive inferences about what is analyzed. When the Bayesian numerical recipes are applied to information retrieval, the system “learns”; that is, its technology automatically (autonomously) forms an understanding of the concepts of the processed content. It makes inferences about the information. The method makes “meaning-based computing” work.
Autonomy based its approach to search on math, filed patent applications, and locked up its core methods in a black box. Other search vendors had access to the good minister’s math, and focused on adding value via linguistics, semantics, and knowledge bases. Autonomy used these methods as well, but — and the but is important — emphasized that its system was “automatic”, and Lynch named the company’s name “Autonomy” for a reason. In a head-to-head comparison of search systems, Autonomy was as good or bad as other systems. However, Autonomy’s sales and marketing approach focused on:
• Mathematics in the sense that 2+2=4. One can have difficulty arguing with mathematical truths.
• The IDOL system (Intelligent Data Operating Layer) was “automatic,” and “automation” was desirable in certain types of labor-intensive work.
• Autonomy bundled Bayes, various technologies, and acquisitions far afield from basic keyword search into a wonderful catchphrase; to wit: “meaning based computing”.
____________________________________________________________________________
Bayes for Beginners: How I Learned to Love Recursion and Stop Counting
Dr. Kate Cowles in her “must read” texts on applied Bayesian statistics uses the explanation below and it is a clear statement of the recursive nature of the Bayesian approach to data analysis:
1. Ask a question or pose a problem.
2. Assemble and evaluate the relevant information.
3. Based on current information, design an investigation or experiment to address the question posed in step 1. Consider costs and benefits of the available experiments, including the value of any information they may contain. Recognize that step 6 is coming.
4. Carry out the investigation or experiment.
5. Use the evidence from step 4 to update the previously available information; draw conclusions, if only tentative ones.
6. Repeat steps 3 through 5.
_______________________________________________________________________
As anyone in any element of content processing systems can tell you, it became difficult to sort out the differences among search and retrieval solutions. Every enterprise search provider began using numerical recipes, including Bayesian methods, or claimed that it already had. If one digs deep enough Bayesian statistics turns up in virtually all of these vendors’ products. However, search engine vendors fall into different philosophical schools. Some focus on linguistic methods, others use keyword indexing and metadata outputs as inputs to ranking procedures. The anti-Bayesian vendors put more emphasis on dictionaries, algorithms that attempt to figure out relevance from what were then XML tags, natural language processing, and dozens of other approaches.
But it was Autonomy that set the pace for everybody.
And then we have Robert Mercer.
Robert Mercer, the hedge fund tycoon and computer scientist, was the major financier behind the creation of alt-right firebrand Breitbart News, and was the founder and financier of Cambridge Analytica. He is a right-wing product of the military-intelligence complex, owing to his early work on artificial intelligence for the US Air Force Weapons Laboratory at Kirtland Air Force Base in New Mexico. It was there that Mercer developed the artificial intelligence to be used (and is still used) for weapons systems and then transformed that artificial intelligence to predict winners and losers in the stock market. The stock market programs he developed made him a multi-billionaire.
But he also learned that you can use that same artificial intelligence techniques to develop real-time propaganda. His alt-right network dumps thousands of cross-narratives across social media and main stream media, hoping that each one takes 0.5% of the attention away from the truth. As Gary Kasparov said so eloquently in his book Winter Is Coming which is an examination of this process:
Propaganda today is not a wall, not a dike holding back information from reaching the people. It is a flood, overwhelming our critical thinking. It is not to promote a narrative or agenda but to create doubt and to make people believe that the truth is unknowable. Its purpose is to exhaust our critical thinking, to hide the truth by seeding a thousand falsehoods around it.
But it was also Mercer who, beginning in 1972 at IBM, with a select team, invented the statistical machine translation software that would be the basis of all machine translation software to come.
And as that technology and those techniques propagated through tech, it would lay the basis for all advanced statistical methods and machine logic to review text – and would give birth to eDiscovery’s computer assisted review/technology assisted review.
Oh, and yes. Mercer and his crew made millions of dollars in the process. Leading to billions of dollars, really, because the team would go on to use those same techniques in the stock market as I noted above.
Yes, other human-readable and machine-readable languages for structuring and exchanging data have since displaced much of what Mercer and his crew developed, especially in cloud-sourced Big Data applications. However, it was the adoption and influence of the Mercer techniques that helped us to confront the new dimensions of the complex historical and philosophical relationship between content and form, between the substance of things and the shapes they take. It ushered in the virtual world that now both mimes and shapes our reality. Its impact also represents a paradigmatic instance of the process by which technology revolutions can bend the arc of technology history.
Two rather “colorful” characters, Lynch and Mercer, created Archimedean moments, creating entirely new tool kits that entirely transformed our relationships with data and information.