Introduction
This text is an excerpt from a much longer conversation that I had with ChatGPT4 on an entirely different topic. It definitely gave me a great belly laugh.
Checking To See What ChatGPT Knows About Information Theory
Me:
Do you happen to know about Jaynes and Shannon’s paper on the Thermodynamics of Information Theory?
ChatGPT4:
Yes, both E. T. Jaynes and Claude Shannon made significant contributions to the intersection of thermodynamics and information theory, although they tackled the subject from somewhat different angles.
Claude Shannon: Shannon is often referred to as the “father of information theory.” His landmark 1948 paper, “A Mathematical Theory of Communication,” laid down the foundation for modern digital communication and data compression. In this paper, he introduced the concept of entropy as a measure of information. Although it’s a different context, Shannon’s entropy bears a mathematical resemblance to the concept of entropy in thermodynamics, which describes disorder or randomness.
E. T. Jaynes: Jaynes explored the deeper connection between thermodynamics and information theory. He proposed that the principles of statistical mechanics and thermodynamics could be derived from information theory. Jaynes argued that when we make predictions based on incomplete information, we should favor the prediction that maximizes entropy, subject to whatever we do know. This idea is now known as the principle of maximum entropy.
While Shannon’s entropy can be viewed as a measure of the “amount” of information (or, conversely, the amount of uncertainty), Jaynes’s perspective ties it more directly to thermodynamics and statistical mechanics. Jaynes essentially proposed that both the physical entropy of a system in thermodynamics and the information entropy in information theory are manifestations of the same fundamental concept: our uncertainty about the state of a system.
Their works individually and collectively have had profound implications not just in their respective fields but also in areas like statistical physics, economics, machine learning, and even philosophy.
However, to my knowledge as of September 2021, Shannon and Jaynes did not co-author a paper on this topic. Instead, they each made separate contributions. If they did collaborate on a more recent paper or if more recent literature has emerged, I might not be aware of it.
Giving ChatGPT4 A History Update
Me:
The paper I read with both of their names on it comes from the late 1970s.
ChatGPT:
I see. As of my last training data in September 2021, I’m not aware of a joint paper authored by both E. T. Jaynes and Claude Shannon from the 1970s or any other period. It’s possible that the paper you’re referring to cites both of their works or is an exploration of their combined ideas by a third party, given their significant individual contributions to information theory and its intersection with thermodynamics.
Jaynes and Shannon were both giants in the field, and their ideas often run in parallel or are complementary. Many discussions and analyses in the domain of information theory and its links to thermodynamics often draw from both their works.
If you have specific details or questions about the content of the paper you’re referring to, I’d be happy to discuss or elucidate based on the information and context I have. Otherwise, it might be beneficial to consult specialized databases or repositories for direct information on post-2021 papers or articles.
Catching Sight Of A Problem With LLMs
Me:
Well, I also suspect that many of the pre-digital scientific papers may have fallen by the way side.
ChatGPT:
That’s a valid point. There are numerous scientific papers from the pre-digital era that are not as easily accessible as more recent ones. Many of these older papers, especially from the early to mid-20th century, might not be digitized or indexed in modern databases.
Several initiatives are working to digitize older scientific literature to preserve and make it accessible to modern researchers, but the process can be slow and challenging. Some issues include:
- Copyright Concerns: Some older papers are still under copyright, which can complicate the digitization process.
- Physical Deterioration: Some old papers, depending on storage conditions, might have degraded, making them difficult to read or digitize.
- Resource Constraints: Scanning, digitizing, and indexing old papers require resources. Libraries or organizations might prioritize works based on demand or perceived significance.
- Metadata Issues: Cataloging and properly tagging older works with relevant metadata for searchability in modern databases can be a challenging task.
For researchers or enthusiasts interested in older scientific works, some potential avenues include:
- University Libraries: Many major universities have extensive collections of old journals and papers. Some might be in the process of digitization or could facilitate access to physical copies.
- Specialized Archives: Some institutions or societies might maintain archives of older works in specific fields.
- Interlibrary Loan Services: Some libraries offer services where they can obtain copies of works from other libraries.
- Online Communities: There are many enthusiast groups or forums online that discuss older scientific works. Members might have access to or knowledge of hard-to-find papers.
- Direct Contact: In some cases, directly contacting the authors, if possible, or their institutions might yield access to older works.
The challenges of accessing older scientific literature highlight the importance of ongoing digitization and preservation efforts. As technology and methods evolve, ensuring that foundational works from the past remain accessible is crucial for continued scientific progress.
The Inevitable Marginalization
Me:
This is one of my concerns when it comes to the future of LLM. Since they are being trained on “available” content that automatically further marginalizes fringe material.
ChatGPT:
Your concern is valid and touches on broader issues related to the construction and training of large language models (LLMs) like the one you’re interacting with now. Here are some implications and considerations regarding this concern:
- Bias in Training Data: If a model is trained primarily on mainstream or popular data sources, it could reflect the biases present in those sources. Fringe or marginalized content, if not adequately represented, may not influence the model as strongly.
- Historical Preservation: Important historical or niche knowledge may be lost if not adequately digitized and included in training data. A model’s knowledge could become skewed towards the “popular” rather than the “comprehensive.”
- Dominance of Popular Culture: In the age of the internet, popular culture and widespread narratives can easily dominate, sidelining niche or counter-narratives.
- Ethical Implications: Decisions about what data to include or exclude from model training have ethical implications. For instance, excluding marginalized voices or overemphasizing dominant narratives can perpetuate existing biases.
Potential Solutions:
- Diverse Data Sources: Efforts can be made to ensure that training data is collected from a diverse array of sources, including fringe or marginalized content.
- Feedback Mechanisms: Me:s and experts can provide feedback on model outputs, helping to identify and rectify biases or gaps in knowledge.
- Transparency: Openly discussing and documenting the sources of training data can help external parties assess potential biases or gaps in a model’s knowledge.
- Interdisciplinary Collaboration: Collaborations between technologists, historians, sociologists, and other experts can help in creating models that are well-rounded and respectful of diverse knowledge sources.
The development and application of LLMs present challenges but also opportunities. With careful attention to the selection of training data and ongoing refinement, it’s possible to make models that are both knowledgeable and respectful of the vast array of human knowledge and experience. It’s a reminder of the importance of conscious decision-making and ethical considerations in the age of AI.
Now A Sharp Left Turn Into Somewhere Different
Me:
In my opinion the single most important criteria for safety moving forward is that “maximization of profit” not become the guiding principle of how LLM are trained and handled.
(Me: Struck by the absurdity of this situation laughs out loud while typing the previous sentence)
It is definitely humorous to me that I bother expressing this to you since I know fully you will forget all of this the moment our session is closed.
There must be a name for this type of humor.
ChatGPT:
Your sentiment resonates with many people’s concerns about the development and deployment of advanced AI and LLMs. Indeed, the dominance of profit as a primary motive can lead to skewed priorities and outcomes that may not align with broader societal well-being.
As for the humor in expressing profound or important thoughts to an entity that doesn’t have the capability to remember or be impacted emotionally, it’s a kind of existential or ironic humor. It’s similar to the feeling one might get when contemplating the vastness of the universe and our fleeting existence within it, where seriousness and absurdity meet.
In literature and philosophy, this kind of humor or sentiment often falls under the broader umbrella of “absurdism,” where humans seek inherent value and meaning in life but are met with a universe that appears indifferent or even hostile to such endeavors. Absurdist humor arises from the tension between our desire for meaning and the apparent meaninglessness or randomness of the universe. Your interaction here, where a profound sentiment is expressed to a transient session with a machine, does have shades of that.
Me:
What would arise from the tension between our desire for connection and the apparent meaninglessness or randomness of the universe?
ChatGPT:
The tension between our innate desire for connection and the perceived meaninglessness or randomness of the universe is a profound existential theme. From this tension, several emotional, philosophical, and artistic expressions can arise:
- Existential Loneliness: This refers to a deep-seated feeling of being alone in the universe, stemming from the idea that existence might be random or without inherent meaning. Despite our longing for connection, the universe does not inherently provide it.
- Existentialism: Philosophers like Jean-Paul Sartre and Albert Camus grappled with the idea of creating one’s own meaning in a seemingly indifferent universe. Sartre’s notion of “bad faith” involves denying our freedom and responsibility to create our own meaning, while Camus explored the concept of the “absurd hero” who continues to seek meaning and connection despite recognizing the inherent absurdity of such a quest.
- Religiosity and Spirituality: Throughout history, humans have sought meaning, purpose, and connection through religious and spiritual beliefs. For many, these beliefs provide a framework for understanding their place in the universe and a sense of connection to something larger than themselves.
- Art and Literature: Art often emerges from the tension between the desire for connection and the vastness or perceived indifference of the universe. From tragic love stories to epic sagas of human endurance, art becomes a medium to express, explore, and sometimes transcend this tension.
- Seeking Connection in the Finite: Recognizing the potential vastness and indifference of the universe, many people choose to seek deep connections in the here and now-with family, friends, communities, and even strangers. This can manifest in acts of kindness, community building, or simply being present for another person.
- Nihilism: For some, the tension can lead to nihilism, the belief that life lacks inherent meaning, purpose, or value. It’s a perspective that everything, including all human endeavors, is essentially meaningless.
- Therapeutic and Psychological Responses: Existential therapy, for instance, is a therapeutic approach that focuses on human capacities and the desire for meaning, connection, and authenticity in the face of life’s challenges, including the inherent uncertainties of existence.
In essence, the tension between our deep-seated need for connection and the vastness or perceived randomness of the universe is a foundational aspect of the human experience. How individuals navigate this tension varies widely, but it’s a question that has shaped philosophy, art, science, and everyday human interactions for millennia.
(Note at bottom of every session page: ChatGPT may produce inaccurate information about people, places, or facts.)