Different tools for measuring teaching and teacher quality provide different affordances and constraints. Structured observation protocols direct observers to focus on specific facets of instruction, and provide a common technical vocabulary for describing and then improving upon those facets. Consistent language for describing teaching allows for comparisons of practice across classrooms as well as targets for instructional improvement. The PLATO manual is designed to capture the quality of English/Language Arts (ELA) classroom practice across the multiple domains of that subject. So far, PLATO has ben used to measure classroom practice in fourth through ninth grade ELA classes. While studies of factors that distinguish among more and less effective teachers will need to take multiple dimensions of classroom dynamics into account, we have focused specifically on instructional interactions targeted towards teaching. In this paper, we focus on a variety of issues related to measuring classroom practice, using PLATO as the basis for this exploration. We analyze the reliability and validity evidence we have accrued over the past five years to raise broader questions about measuring teacher and teaching effectiveness. We also discuss the use of PLATO as a tool for diagnostic information about teaching and for targeted professional development.