The Most Impactful Roles for Mitigating the Catastrophic Risks from Artificial Intelligence.

A software engineer specializing in AI model evaluations tools has a deep understanding of the needs of model evaluation and those of ML researchers. This position often involves close collaboration with ML researchers to define the scope and vision of tools needed to evaluate large AI models, including for detecting capabilities having potential to contribute to catastrophic risks. This role entails the identification of methods to integrate the issues surrounding the latest large models into evaluation tools, the performance of hands-on evaluations to pinpoint areas for evals improvement, the design of user-friendly interfaces for complex evaluations, and the application of data visualization techniques to simplify the understanding of intricate data. This role plays a pivotal part in enabling researchers to excel in their work and accelerate progress in the field of AI evaluation.

An AI auditor is crucial to rigorously assess AI models and ensure that they comply with existing evaluation standards. This role demands thoughtful analysis to discern optimal, safe, and ethical actions for AI models in diverse scenarios. The AI Auditor may, for instance, use a web interface to simulate environments where models operate within text-based games, striving to acquire resources and replicate, and map the behaviors to qualitative and/or quantitative characterizations. While computer science proficiency is valuable, collaboration with technical experts may be an option for candidates with deep domain expertise, e.g. bioweapons. Proficiency in domains such as manipulation prevention, conventional and nonconventional weapons, or social engineering and hacking more broadly would be helpful. The AI auditor’s responsibilities may encompass simulating environments, prompting effective actions, assessing model recommendations, and contributing to a red-team/blue-team approach. Identifying essential yet challenging steps for AI models and exploring potential solutions are a core aspect of this role.

The Product Manager often bridges the gap between the technical team and other stakeholders. They are responsible for guiding the development of AI products. Their tasks usually include identifying customer needs, defining product requirements, and working with cross-functional teams to create and improve the organization’s products and/or services.

A CTO leads the vision for the technological roadmap, development, and execution of AI products. They oversee the technology department including, and work closely with, the executive team, to develop and implement strategies to drive innovation, increase efficiency, and achieve business objectives. The CTO may also hold responsibilities at the intersection of technology advancement and risk management, and may contribute to ensuring that the technology systems and processes within the organization are compliant with standards the organization or product team wishes to adhere to.

Privacy Research Engineers are key roles for ensuring that an organization maintains stringent privacy standards, including in its data collection, algorithms, products, and services. Such a professional specializes in designing and implementing the technology that safeguards user data, confidential training data, restricted inference data, and privacy. They play a critical role in developing and integrating privacy features into various software and applications.

Data security roles contribute to ensuring the security of the organization’s information systems and platforms. Individuals in data security roles are often responsible for overseeing information security, cybersecurity and IT risk management programs based on industry-accepted information security and risk management frameworks. They contribute to the continuous development, implementation and updating of security and privacy policies, standards, guidelines, baselines, processes and procedures in compliance with the law and with the best practices.

A Software Security Engineer is usually a key member of an organization’s security team. In major AI organizations, an additional expertise beyond traditional software cybersecurity, also involves securing the integrity of the output of the AI model produced by the organization. This involves Modeling/Detecting/Mitigating vulnerabilities associated with the fact that customers will interact with the model, potentially introducing adversarial input leading to problematic and dangerous behavior of the model. They usually possess substantial expertise in conducting security assessments (evaluating code, architecture, model mechanistic interpretability, threat modeling, etc.) to understand the attack surface of complex systems, and review and offer recommendations for their mitigation. A Software Security Engineer will work, among other things, on incorporating security measures into AI systems at the algorithm, software, and infrastructure levels to minimize risks from emerging threats.

The role of COO is often a senior leadership position. The COO is responsible for overseeing all of the aspects of the organization’s work that aren’t research, including human resources, finance, development, communications, legal risk and compliance, etc. This role is a highly autonomous position. The individual regularly makes high-level decisions about the direction of the organization’s operations and projects, weighs in on key organizational strategic priorities, and helps shape the direction of the organization’s work.

Roles in the Senate (staff member) vary in their level of seniority, required Capitol Hill experience, and potential. Within the Senate, the aforementioned roles are the ones most likely to yield influence pertaining to AI policies.

The role of Staff Member at the DSIT, and more specifically at its Office for Artificial Intelligence, may provide a unique opportunity to influence the advancements in AI policies and to help coordinate the implementation of the UK National AI Strategy. Numerous positions are considered as Staff Member, and those can encompass policy advisors, project leads, and heads of strategy. These professionals are responsible for supporting and coordinating initiatives related to AI, innovation, and technological progress. Their duties encompass a wide range of activities, including policy development, research, stakeholder engagement, and strategic planning.

Civil servants operate with varying statuses, either as career bureaucrats or regular employees, and are situated within different organizations of the UK government, such as ministries, ministerial cabinets, or public agencies. Civil servants in the UK are generally categorized into three main types: (1) operational delivery, (2) cross-departmental specialisms, and (3) departmental specialisms. Cross-departmental specialisms encompass roles essential across all government departments, including government-specific positions like policy experts which may be particularly impactful for influencing AI policy. Note that this job does not necessarily require UK citizenship.

A Staff Member in the European Commission or European Parliament is a civil servant or administrative professional who works within the EU's legislative or executive branches, respectively. They may engage in tasks related to policy development, legislative processes, research, and administrative support, assisting in the creation and implementation of EU laws and regulations while contributing to the functioning of these institutions. The term “Staff member” is broad and includes numerous specific roles.

Although these two roles are distinct in the realm of public policy and governance, they also often overlap significantly, which is the reason we decided to address them simultaneously. Generally, the difference lies in the fact that policy analysts primarily focus on conducting in-depth research, data collection, and analysis to provide evidence-based insights into policy issues, whereas policy advisors play a more strategic role, offering guidance and recommendations to decision-makers on policy options and their potential impacts. While both roles contribute to informed policymaking, policy analysts are generally research-oriented, whereas policy advisors often provide strategic counsel and navigate the complexities of policy implementation. Policy analysts and policy advisors both collaborate closely with project leaders to provide project management, research support and analysis support. Analysts engage in various tasks critical to the given agency's or think tank’s mission, including conducting specialized analyses and studies, engaging with governmental stakeholders, and supporting policymaking efforts in the field of AI safety and governance. A policy analyst and a policy advisor’s work contributes in shaping the decisions and strategies of policymakers within the government, and their influence can extend to guiding the trajectory of policy development. Specifically within the realm of mitigating AI catastrophic risks, these professionals serve as essential bridges between rigorous research and the practical application of policies and regulations.

Although distinct, the roles of AI governance researcher and of AI policy researcher often have important overlap, which is why we present them together. Essentially, AI governance researchers and AI policy researchers investigate what it may take, governance and policy-wise, to prepare for a world with advanced AI. More specifically, an AI governance researcher is likely to focus on establishing frameworks to guide AI development, deployment, and use, to address issues such as safety and transparency. An AI policy researcher, on the other hand, is likely to delve into the legal and regulatory considerations pertaining to AI, and formulate policies and recommendations for governments and regulatory bodies. Both roles focus on researching actions and strategies that decision-makers and key actors, both within government and industry, should pursue in order to mitigate the catastrophic risk from AI.

An AI alignment researcher works on researching how we can better ensure that advanced AI systems behave as intended and in alignment with human interests, even when not under direct human supervision. This entails conducting theoretical alignment research, which is often conceptual, algorithmic, or mathematical. AI alignment work may sometimes also involve hands-on research engineering tasks, depending on the organization and role. Such work generally requires thinking about potential behaviors of AI systems with a security mindset, and thus focuses on investigating topics like interpretability, value learning, inner alignment, corrigibility, etc.

An AI Safety Research Scientist working within ML works on developing methodologies and strategies to ensure the safety, controllability, value alignment, and robustness of AI systems. They often conduct in-depth research and formulate new ideas in machine learning or work on improving existing ones, an endeavor which usually involves mixes of algorithms design, theory, programming, a security mindset, and ML engineering. An AI Safety Research Scientist helps establish and/or execute the research trajectory to enhance the safety, alignment, and resilience of AI systems against adversarial or malicious applications, as well as against accidental rogue actions and broader sociotechnical systemic failures. Note that this job sometimes overlaps with the role of AI Safety Research Engineer, since that job partially consists in implementing the findings from AI safety research scientists.

An AI safety research engineer focuses both on the practical implementation of existing AI safety techniques as well as development support for researchers while researching new techniques, which are often elaborated by AI safety research scientists. This purview includes overseeing and contributing to scientific experiments, building technique-specific software infrastructure, developing benchmarks or tests, designing API interfaces, optimizing and improving safety-related algorithms, proposing new safety ideas, and refining safety tools. Note that this job sometimes overlaps with the role of AI safety research scientist, since it consists in part in implementing the findings of AI safety research scientists.

A committee member at a standardization body, such as CEN/CENELEC Joint Technical Committee on Artificial Intelligence (JTC 21) (in the EU) contributes their technical and/or industry expertise to the development and maintenance of robust AI standards. They take part in the committee’s work and actively engage in building consensus among diverse stakeholders, fostering collaboration among experts, industry representatives, and policymakers to create effective and comprehensive AI standards. Please note that this role is usually held in addition to a main job position.