Microsoft Publishes Alp’s White Paper on “Testing in Aircraft Design and Manufacturing”

Adams & Reese Aviation & Aerospace Team Co-Leader Paul Alp was asked by Microsoft to contribute as a technical subject matter expert to the company’s recent research program on lessons for AI testing and evaluation from several industry sectors. Alp’s white paper on “Testing in Aircraft Design and Manufacturing” is published on a Microsoft research blog website specifically dedicated to the initiative to gather and evaluate experiences and learnings from other domains to advance AI evaluation and testing within the company.

Alp’s white paper discusses the international regulatory framework governing aircraft design, certification, and manufacturing, and describes the lessons this regime can provide for novel industry sectors such as AI. The paper posits that, as a result of the maturity of the aviation domain, processes for establishing standards and means for finding compliance with those standards offer significant utility and flexibility. Given the broad nature of aviation applications, the industry has evolved a wide variety of approaches to standards-setting and compliance-showing, ranging from highly prescriptive regulations issued by governmental bodies to standards developed through industry consensus.

During the early stages of aviation, government regulation was limited in order to support innovation and iteration of new technology. As the industry matured and hazards became apparent, regulations evolved into a highly prescriptive framework. In the modern era, the industry has increasingly relied on performance and risk-based standards, including those set by industry consensus, to the extent possible. These standards are used both to establish performance criteria as well as acceptable means of demonstrating compliance to those criteria. The end result is an international order that supports safe transportation with a sliding scale of regulatory involvement that is fine-tuned to reflect risks posed by different types of products and operations.

The paper also argues that, although the international aviation regime appears on its face to be highly prescriptive, it actually affords significant flexibility and opportunities for innovation as a result of its reliance on performance and risk-based standards, acceptance of industry consensus means of showing compliance, and systematic, process-based approaches to data analysis, safety management, and continuous improvement. This mature paradigm for standards-setting and compliance-showing is readily adaptable to other domains, as has been demonstrated in fields such as nuclear power and medicine. Lessons from civil aviation can also be applied to a compliance framework and international order for AI.

Microsoft’s research on risk evaluation, testing, and assurance models in other domains began in December 2024, when Microsoft’s Office of Responsible AI gathered independent experts from the fields of civil aviation, cybersecurity, financial services, genome editing, medical devices, nanoscience, nuclear energy, and pharmaceuticals. In bringing this group together, Microsoft drew on its own learnings and feedback received in its e-book, Global Governance: Goals and Lessons for AI, in which Microsoft studied the higher-level goals and institutional approaches that had been leveraged for cross-border governance in the past. 

In the white papers, the contributors discussed questions such as: which opportunities, capabilities, risks, and impacts should be evaluated as generative AI becomes more capable and widely deployed? Who should conduct evaluations, and at what stages of the technology lifecycle? What tests or measurements should be used? And how can we know if the results are reliable? (Click here to view all of Microsoft’s case studies on AI Evaluation and Testing.)

Within the white papers published, experts noted that policymakers have to weigh trade-offs in designing evaluation frameworks, and these frameworks must account for both the limits of current science and the need for agility in the face of uncertainty.

At Adams & Reese, Alp is a co-leader of the Aviation & Aerospace Team within the Intersection of Business & Government Practice Group. As a Partner in DC, Alp represents clients at the intersection of aviation regulatory, legal, and technical issues. For more than two decades, Alp has represented and advised airlines, manufacturers, repair stations, on-demand operators, technology companies, insurers, commercial space companies, corporate flight departments, and uncrewed aircraft companies on international and domestic aviation and aerospace issues. He provides a full range of advice on compliance, operations, safety, risk management, licensing, certification, airworthiness, security, crisis preparation, and emergency response.