Revisiting Insight Generation in the Age of Generative AI

Ten years ago, I explored the process of insight generation. I presented its key stages and emphasized the characteristics distinguishing true data-driven insights from mere correlations. While the foundations of insight generation remain relevant, since the publication of my original article, the methods, tools, and implications have expanded dramatically. Generative AI is reshaping how insights are derived, validated, and applied across industries.

The Core Tenets of Insight Generation

In an era where data-driven decision-making has become a competitive necessity, insight generation serves as a cornerstone of strategic advantage. In my original piece, I defined insight as a novel, interesting, plausible, and understandable relation, or set of associated relations, that is selected from a more extensive set of patterns derived from a data set. I argued that insights must be actionable, measurable, stable, reproducible, robust, and enduring. These qualities set insights apart from mere correlations. I also presented a framework for generating insights. 

Insight generation requires pattern recognition, information synthesis, and the creation of meaningful causal connections that lead to actionable knowledge. In the original framework, human domain expertise was crucial for selecting the data that would be provided to the insight generation system, providing seed domain knowledge, assessing the generated insights-action plan pairs, and evaluating the decisions that result from the application of these plans. 

While the overall structure of the proposed framework does not change, generative AI accelerates the insight generation process and can potentially improve the quality and quantity of the generated insights. However, the human role in contextualizing, evaluating, and, in certain cases, acting upon the generated insights remains essential. 

Generative AI’s Role in Insight Generation

Generative AI introduces three enhancements to my insight generation framework:

1. AI-Enhanced Data Exploration: The framework’s Knowledge Extractors’ effectiveness to generate models, establish causal relations, identify outliers, and develop benchmarks can improve significantly with generative AI. Large Language Models (LLMs) can process structured and unstructured real and synthetic data to quickly create various such knowledge structures that become insight candidates. Synthetic data is particularly useful for addressing the shortcomings of real data, addressing ethical considerations such as privacy, and generating controlled scenarios to test insights and improve their robustness. However, even with the ability to generate appropriate synthetic data, which generative AI systems can do well, the importance of clean proprietary data should never be underestimated. Both high-value proprietary data and appropriately generated synthetic data positively impact the quality of the generated tokens and, therefore, the extracted knowledge. 
2. Expanded Access to Analytical Capabilities: The framework’s Insight Generator capabilities can significantly improve by combining its planning component and the domain ontologies and domain-specific insight/action plans it has access to with the reasoning abilities of frontier models and domain-specific LLMs. This combination enables the Insight Generator to filter out irrelevant or weak insight candidates, tease out causal relations among entities, generate an appropriate action plan for each, and even simulate the stability, reproducibility, and measurability of each plan before ultimately associating it with an insight.  
3. Augmented Decision-Making: The Decisioning System that is used during the Insight Evaluation and Selection step of the overall process can employ generative AI agents that act as “collaborative thinkers” to analyze alternative viewpoints and refine the generated insights, consider counterfactuals, simulate various scenarios by accessing appropriate digital twins, and propose potential actions in addition to those created by the Insight Generator. 

Challenges to a Generated Insight’s Characteristics

Even though generative AI enhances the process first presented ten years ago, because of how it responds to a prompt, reasons, and hallucinates, it can also present challenges to a candidate insight’s defining characteristics. 

• Actionability: Candidate insights may incorporate actions that may not be possible to perform under real-world constraints.
• Measurability: Generative AI systems may make inferences that make it difficult to ensure each candidate insight remains stable and to measure its effectiveness consistently.
• Reproducibility: The output of any generative AI system is probabilistic, meaning that different runs on the same dataset may result in different outputs, resulting in so-called hallucinations. Hallucinated insight candidates and/or action plan candidates will lead to a big user trust problem.
• Robustness and Endurance: Candidate insights may not endure across different contexts and over time because they are generated in a way that makes them susceptible to change.

Where Do We Go From Here?

Looking ahead, the convergence of several key trends will shape the future of insight generation:

• The continued evolution of generative AI: As generative AI models become more sophisticated, their ability to produce increasingly nuanced, context-aware, domain-specific, and creative insight candidates will grow. Future research should explore how to best leverage these capabilities while ensuring the reliability and validity of the generated insights.
• The rise of multimodal AI: The integration of multiple data modalities (text, images, audio, etc.) will enable a more holistic understanding of complex phenomena, leading to richer and more comprehensive insight candidates.
• The development of more sophisticated human-AI collaboration: The focus will shift from simply using AI to augment human capabilities to creating truly collaborative systems where humans and AI work together synergistically to generate insight/action plan pairs, with each contributing their unique strengths.
• Increased emphasis on ethical considerations: As AI plays a larger role in generating insights that inform decision-making, it will become increasingly important to address the ethical implications of these insights, including issues of bias, fairness, and transparency.
• The development of new business models: The expanded ability to generate insights of greater variety in different domains will drive new business models to monetize the insights that are generated rather than just the tools that lead to their generation.

The need to generate insights and effectively apply them is more important for organizations than mere pattern generation, which almost all are now able to do. The challenge lies in establishing the right insight generation process that ensures that those insights are reliable, ethical, and ultimately drive better outcomes.