Although computer-based tests (CBT) exhibit a high content validity measuring ICT literacy (e.g. presentation of authentic situations, interactive test environments), large studies (e.g. the National Education Panel Study (NEPS) in Germany) are still conducted using paper-and-pencil tests (PPT), if mainly for technical and financial reasons (Kuhlemeier & Hemker, 2007). The construct validity of test procedures measuring ICT literacy involving PPT compared to performance-based CBT has so far only scarcely been empirically examined (Miller & Linn, 2000). To fill this desideratum, the convergent and discriminant validity of PPT and CBT are compared. An experiment was conducted using a representative part of the items used in the NEPS in which 200 adolescents (14 – 16 years old) attended to either a paper-and-pencil test form or a performance-based exercise on the computer. Cognitive measures (verbal intelligence, reading skill, problem-solving skills) and computer use characteristics were used as validation variables. First results point to a sufficient validity of the PPT. The PPT discriminates between technological and information literacy as two sub-dimensions, but correlates stronger with cognitive measures than the CBT does, which can restrict the discriminant validity. The importance of the results will be discussed for large studies in the future.