How useful is AI for aeronautical research? Robert E Joslin, Ph.D., FRAeS Associate Professor, Embry-Riddle Aeronautical University, asks whether AI – Loss of Critical Thinking (A-LOC) is a rising risk for students, academics and aerospace professionals.

The use of Artificial Intelligence (AI) in the aerospace industry is rapidly becoming the norm for gathering and analyzing information as well as responding to research questions, the latter of which historically has required critical thinking. However, rather than just being used for developing a basic structure or an outline in responding to a research question, reliance on AI has evolved into providing most if not all the content, especially among technology adept emerging aerospace professionals.

The aerospace industry has recognized the potential benefits of using AI, however the potential risks, such as creating inauthentic, inaccurate, biased, or harmful content must also be considered. Consequently, it is critical that aerospace professionals acknowledge and have awareness of the limitations along with the tools to develop a strategy that identifies and mitigates the risks of relying on AI when conducting research.

A recent study provided insights on the limitations of AI in responding to a research question. Eight aerospace graduate students conducted a review of an AI transcript that was generated using the most current version of an AI Large Language Model (LLM) of their choosing. The students posed the following research question using one of four different LLM: What is the single most significant regulatory impediment to the full implementation of uncrewed aircraft systems (UAS)? They were then asked to conduct a review of their AI generated transcript considering various criteria.

• Not referencing current events or otherwise misrepresenting them
• Inaccurate data and/or quotes
• Do the sources exist?
• Are they valid primary sources of information?
• Do the sources say what the AI transcript claims they do?
• Lack of depth of analysis (too general/not specific)
• Inaccuracies, biases, or harmful content
• False statements (“AI hallucinations”)
• Off-topic content
• Inconsistent and/or redundant filler statements
• Are statements of fact supported by a cited reference?
• Other aspects that adversely affected the validity and/or relevancy of the AI response

The consensus in the LLM transcripts was that Beyond Visual Line of Sight (BVLOS) operational restrictions was the most significant regulatory impediment, which was a credible and defensible top-level response to the research question based on current Federal Aviation Administration (FAA) and U.S. Government Accountability Office (GAO) reports. However, the study revealed that the LLM generated transcripts had issues such as misleading information, outdated information, or general statements identifying BVLOS as the impediment but with no links or citations to supporting evidence. In some cases, the LLM did provide cited references if the user posed a follow-up question, although the results varied depending on the specific wording of the prompt. Overall, the transcripts lacked depth of analysis and appeared to be written for the layperson rather than comprehensively addressing the research question, which resulted in each of the LLMs having one or more of the following issues.

Hallucinating references

^{The research question centred on BVLOS UAS rulemaking (Zipline)}

Most of the AI transcripts referred to the Federal Aviation Administration’s Part 107-Small Unmanned Aircraft Systems regulations and the 2022 BVLOS Aviation Rulemaking Committee (ARC) report, which were relevant to the research question. However, some of the transcripts did not mention that waivers and exemptions are already currently issued to allow limited BVLOS operations. In addition, some did not cite specific recent developments, such as the status of the FAA’s BVLOS Notice of Proposed Rulemaking (NPRM) issued in 2023 or the 2024 FAA Reauthorization Act that also addressed BVLOS.

A few of the transcripts mentioned BVLOS enabling technologies, such as Detect and Avoid (DAA), Command and Control (C2) Link, and Unmanned Aircraft System Traffic Management (UTM), however the discussions were only cursory as well as lacking any acknowledgement of the enabling industry consensus standards such as ASTM, RTCA, EUROCAE. While no glaring falsehoods or off-topic information was evident, several statements of fact were unsupported hence could not be verified without a critical evaluation by the user to determine their validity and relevancy.

All the transcripts were very redundant, wordy, and replete with filler phrases, whereby general statements were repeated multiple times with different wording rather than adding new insights. Of note was that none of the responses addressed international BVLOS regulatory similarities or differences hence lacked context for global UAS implementation. There were also instances of hallucinations where the LLM appeared to fabricate supporting references. For example, one transcript with two citations had a broken link to the first source and the Digital Object Identifier (DOI) for the second source could not be located. Although the Journal existed, a search through their archives for the title or author of the work did not result in a match.

Summary

^{While AI can be useful – a human curated library, such as the RAeS National Aerospace Library remains the single best tool for accurate technical aeronautical research. (Tim Robinson/RAeS)}

Overall, the AI-LLM generated transcripts provided a broad understanding of BVLOS operations as the single most significant regulatory impediment to the full implementation of uncrewed aircraft systems. However, the information was too general, occasionally outdated, and often devoid of source validation. In short, the transcripts were useful as a brainstorming tool but insufficient as a sole source for professional or academic research. While AI LLMs have the potential to significantly support critical thinking they should not yet be used as the only means of producing technical or creative content. Although the effectiveness of these tools is likely to continue to improve, human oversight with critical thinking remains essential to assess their validity and relevance especially in the high-risk aerospace industry.

The Royal Aeronautical Society is the world’s only professional body dedicated to the entire aerospace community. Established in 1866 to further the art, science and engineering of aeronautics, the Society has been at the forefront of developments ever since.

www.aerosociety.com | National Aerospace Library catalogue & e-books.

Book reviews covering academic, scientific and technical books covering aeronautical engineering and topics relating to it can be found here: www.aerosociety.com/news-expertise/national-aerospace-library/book-reviews

The Aeronautical Journal has, for over a century, been the UK’s leading scientific and technical aeronautics Journal and is the world’s oldest Aerospace Journal that remains in production. Published monthly, The Aeronautical Journal draws upon the expertise and resources of The Royal Aeronautical Society providing a world-wide forum for authors from the UK and overseas. Research papers are solicited on all aspects of research, design and development, construction and operation of aircraft and space vehicles. Papers are also welcomed which review, comprehensively, the results of recent research developments in any of the above topics.