Over the past decade, data analysis has become crucial across all sectors. It facilitates evidence-based decision-making, enhancing the ability to address challenges and opportunities. Generative Artificial Intelligence (GAI) applications now offer the potential to automate, accelerate, and optimize various data analysis tasks with unprecedented efficiency  (Elish & Boyd, 2018; Russell & Norvig, 2020).

The proposed presentation will discuss an advanced secondary education system offering a specialization in data science, that prepares students for a world in which artificial intelligence (AI) is already integrated into all aspects of life, exposing them to how these tools can be utilized in accordance with the requirements of the job market (Data Analyst). The specialization aims to assist students in comprehending the significance of information and data in decision-making processes, developing critical and creative thinking, mastering digital applications for data retrieval, merging, processing, analysis, and presentation. Additionally, it seeks to cultivate the ability to meet the information needs of diverse consumer groups while familiarizing students with ethical data processing issues (MOEI, 2024). Yet, until now, the matriculation examination in this specialization (as well as in all disciplines) has been conducted on computers in a closed digital environment.

This year, a pilot program will be implemented wherein some schools will conduct examinations with Large Language Models (LLMs), and by 2026 all schools in this specialization will incorporate the use of LLMs in examinations. Consequently, it is imperative to re-evaluate the pedagogical model and assessment processes considering the diverse possibilities afforded by GAI applications. There is a need to examine which competencies students must acquire, which skills are automatically performed by AI, and to learn efficient ways of utilizing relevant applications and developing appropriate question design principles.

Accordingly, an experiment was conducted last year by the Institute for Applied AI Research in Education, operating under the Research and Development Division of a national education ministry. The experiment's objectives were: 1. Formulating a concept for integrating GAI within the learning framework of the Information and Data specialization; 2. Developing design principles for assessment tasks that manifest AI literacy in the field and providing task examples; 3. Formulating recommendations for learning and teaching principles that incorporate internet research and AI applications in the field; 4. Developing a framework for systemic implementation.

We will address the following research questions: A) What competencies are required from learners in the Information and Data Specialization in the AI era? B) What are the design principles for assessment tasks suitable for matriculation examinations in an environment with AI applications? 

To conduct this research-accompanied experiment, we employed the Design-Based Research (DBR) methodology, chosen as an appropriate research model for improving theoretical understanding while acquiring practical knowledge through designing and examining interventions in a natural educational environment (Tinoca et al., 2022; Eyal & Gil,2020). The research is characterized by iterative research cycles, wherein multiple cycles of planning, implementation, analysis, and conclusion-drawing were executed to enhance the educational intervention (Sandoval & Bell, 2004). Additionally, a Delphi questionnaire was administered to all experiment partners to reach a consensus (Clayton, 1997) regarding the central design principles for questions adapted to examinations allowing AI access. The experiment partners (N=31) were experienced teachers in the specialization who worked with a team of experts, including consultants from the data field and AI in education, the specialization's supervisory team, and the experiment's steering committee. This collaborative approach allowed for the collection of practical insights from educators throughout the research process.

The research explores the impact of Generative AI (GAI) on data science education and assessment methods. Key findings reveal expert consensus on new competencies required from learners in the AI era, including: Proficiency with AI applications for data analysis, Effective prompt formulation, Strategic problem-solving before AI engagement, Metric selection and interpretation using language models, Critical evaluation of AI outputs, Creating AI-assisted visualizations and recommendations. The study acknowledges that certain previously taught topics can be reduced or eliminated due to AI capabilities, such as data extraction, graph analysis, and statistical calculations, though complete consensus on specific topics wasn't reached. A significant outcome is the development of new assessment task design principles suitable for AI integration. These include: Incorporating ambiguous questions; Encouraging problem exploration; Requiring metric construction; Promoting critical thinking; Including multi-step data development; Motivating action-oriented solutions. The research produced 24 practice assessment tasks and 13 generic question models with development guidelines for teachers. These tools emphasize various literacy skills and incorporate the established design principles, achieving over 80% consensus among research partners. The study concludes that GAI integration presents new challenges in assessing student knowledge and understanding in the Information and Data specialization. It emphasizes the importance of viewing GAI as a collaborative tool in assessment and thinking processes. The findings highlight the crucial role of teacher involvement in task development and classroom implementation, supported by expert consultants in R&D teams. The research recommends updating the curriculum and implementing programs to develop AI literacy among teachers and students, ensuring the specialization remains relevant amid ongoing GAI developments.

Clayton, M. J. (1997). Delphi: a technique to harness expert opinion for critical decision‐making tasks in education. Educational psychology, 17(4), 373-386. Elish, M. C., & Boyd, D. (2018). Situating methods in the magic of Big Data and AI. Communication monographs, 85(1), 57-80. Eyal, L., & Gil, E. (2020). Design patterns for teaching in academic settings in future learning spaces. British Journal of Educational Technology, 51(4), 1061-1077. Ministry of Education. (2023). Guidelines, procedures and instructions for the leading subject in the "Data and Information" track. Ministry of Education, Technological Education Administration. Data and Information Track Curriculum. Russell, S., & Norvig, P. (2020). Artificial intelligence: a modern approach. Hoboken. Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: Introduction. Educational psychologist, 39(4), 199-201. Tinoca, L., Piedade, J., Santos, S., Pedro, A., & Gomes, S. (2022). Design-based research in the educational field: A systematic literature review. Education Sciences, 12(6), 410.