Leonardo Salvatore
Soka University of America
Imagine being on a sports committee tasked with judging an archer training to enter an international tournament. How would we assess the archer without at least one criterion of evaluation? Imagine she never hits the target but always manages to land her arrows on the same mark. If we prized accuracy above all else (i.e., hitting bull’s eye), she would be an amateur archer with many years of training ahead. Training means time, coaching, money, and more. Yet, if we prized precision (i.e., landing every arrow on the same mark), she would have a great chance of receiving a high score and participating in the tournament.
In the same way, we need to have a clear idea of what we are trying to achieve with our educational reforms. Are we elevating test scores above all other criteria of improvement? Or are we focusing on broader, more holistic evaluations of students’ private and social lives as well as their academic achievements? Our questions and goals do not need to exclude each other; but they largely determine how we go about our missions and projects.
Success is a target we construct—achievement depends on definition.
Of course, our ideas may—should—evolve as we test them against the volatility of reality. Empirical observations or theoretical reflections can shed light on a gap or issue in our dominant approach, which might then call for revision. But if we do not establish an objective towards which we wish to move, our precious efforts may not be as effective as we need them to be. This—to establish solid objectives while retaining flexibility and open-mindedness in implementing measures to achieve them—may well be one of the most tiresome and difficult challenges for scholars and educators alike.
Yet, it is the most important.
A lucid theoretical pathway can illuminate the crucial queries CIES 2023 raises. In the short paragraphs that follow, I would thus like to raise a (far from complete) series of questions that, I hope, will spark reflection as we carry on our many endeavors.
How should we critically look at and meet desired outcomes across time and space?
We all know that the past three years have upended the world of education. The effects of the COVID-19 pandemic, as wearisome as they may have become, force us to face undesirable realities and challenges.[i] In a sense, though destabilizing before anything else, the pandemic has revealed issues we were not considering as seriously as we should have. It has helped us to see many “hidden” inequalities and injustices that need addressing, in the world of education and beyond. As external pressures from geopolitical instability and environmental precarity (to name two) keep raging, time becomes an ever more precious resource.
We need action—now.
Yet, despite the urgency at hand, it is imperative for the researcher to pause and avoid hasty decisions; to lead with as clear a mind as possible.
When we think of outcomes, we should ask ourselves what we are really thinking about. What outcomes do we desire? Could the outcomes we desire the most (say, equal access to healthy meals for high schoolers) contradict other outcomes we intend to bring about (say, fostering sensitivity to prejudice against body types that do not conform to dominant expectations)?
What does it mean to be critical? Perhaps, to be critical in the context of outcome evaluation and project implementation is to constantly question the validity and efficacy of the outcome we seek and the process we are using. Sometimes, we will have to abandon years of work; sometimes, we may simply need to tweak a few issues. Either way, if we want to lead analyses of the merits and faults of a project, idea, or reform, we will need honesty and courage. What are the factors that may make us rethink the direction we took?
A recent study titled “A Critical Investigation of English Language Teaching in Bangladesh” shows us one way in which we can be critical, after careful, evidence-based evaluations, about desired outcomes. After assessing the effectives of a long-standing English-teaching program in Bangladesh, Mosiur Rahman and Pandian concluded that English education policies in Bangladesh need to be revisited and revised without vested Western interests and influence” (2018, p. 47). They continued: “the fundamental problem in Bangladesh, like many other developing countries, lies in its misplaced faith in imported Western methodology as a means of improving its ELT curriculum. Curricular reform should be localised and based on social and classroom needs.” (p. 48). Their critical appraisal of the program’s effectiveness in bringing about the desired outcome (i.e., English proficiency) came after careful consideration of empirical data as well as a conceptual commitment to adapting the curriculum to local needs (and not vice versa). They did not dispense with the desired outcome altogether: “[English Language Teaching] in Bangladesh has a great role to play” (p. 43). But they examined it critically and changed their course of action accordingly.
Regarding meeting outcomes, we might ask, what factors are most important to consider as we take concrete steps to reach our outcome? To make it more tangible: if we are trying to equalize literacy outcomes in primary schooling within a short timeframe, can we afford to ponder issues of curriculum diversity and risk destabilizing the structure of a program?
Outcomes, unlike archery targets, are flexible aspirations we should be molding continually. Part of the critical look with which we should gauge our desired outcomes may involve discarding them and starting from scratch. That is hard work; but the times are calling for it more than ever. Adaptability is the watchword.
In what ways may micro, meso, and/or macro educational strategies, structures, and processes be improved along with their environments?
Boeren (2019) divided the micro, meso, and macro levels into “(1) individuals and their families (the micro level); (2) schools, education and training initiatives (the meso level); and (3) regulating governments (the macro level)” (p. 277). Arguably, each level is equally important; and they are certainly interdependent. For instance, families can ensure that a child’s learning is sustained outside formal academic settings. Schools can provide services (i.e. after-school programs) to assist the children of disadvantaged families in staying on track with their learning. And regulating governments can allocate financial, material, or digital resources to schools that lack support programs.
Given their inextricable and essential relationships, are there clear, effective communication pathways between the three levels? What areas might benefit from third-party intervention (i.e., NGO-led programs to bring closer families, school representatives, and government officials)?
Within the levels, challenges arise to track improvements in students, program efficacy, and government resource allocation. How can we map changes in, say, particular program structures onto desired cognitive developments? In other words, are there ways beyond oft-inexact statistical analyses to ensure that changes at the meso level translate into real changes in students’ cognitive developments (broadly defined)? And what are the precise mechanisms through which data flows to and from regulating entities with the power to unlock financial packets that guarantee the existence of a particular program?
I recognize the challenges some of these questions present to the analytical methods at our disposal. But we should not despair; we should apply creativity and collaboration to break beyond our limits and chart new avenues of research that can begin to answer the questions which today seem far-fetched.
How do we know through rigorous methods that we ARE making progress responsively?
As Hamilton and Schwartz (2019) wrote in a report on Social and Emotional Learning (SEL), “Accurate measurement is impossible if you don’t know what you’re trying to measure” (p. 3). As the report pointed out, the large volume of potential SEL measures and the lack of clear communication between curriculum and research designers and schools and school districts can either decrease the utility of the measurements or defeat their very purpose. This is not an issue of willingness or capability. Rather, it is an issue of clarity. Confusion around SEL and the many measurements that are supposed to evaluate its development among students is hindering its great potential.
This example stands for many others in which confusion or poor communication delays progress. Before establishing rigorous analytical methods, we should agree on what the methods are for. To be sure that we are indeed making progress, there needs to be clarity around our definitions, our objectives, and the language we use to communicate them. The way to clarity begins with questions.
Does knowing that we are making progress amount to statistical significance only? What role do qualitative reports from parents, students, and teachers play in determining the efficacy of a particular measure? Are qualitative reports rigorous enough to track substantial progress? Can there be a general rule of thumb for employing one or another (or more than one) method in a given circumstance? And so on.
A Word on Equity
The 2023 CIES conference rightly places equity at the center of its scope. But what is equity? Do we have clarity on the word and its implications?
The Cambridge Dictionary defines equity as “the situation in which everyone is treated fairly according to their needs and no group of people is given special treatment.” If we adopt this working definition, an equitable world is a world where every student (read person) is treated fairly according to their needs. What are, then, students’ needs? Are we well- or ill-equipped to detect them, especially since, as the pandemic highlighted, different populations can have very different needs? If we are ill-equipped, what resources are we missing, and where might we obtain them?
Sometimes we use equity and equality interchangeably, but the two hold significant differences that can alter our approaches to teaching and reform.
Equality, in the sense we all seem to imply, is what the Cambridge Dictionary describes. It is ensuring that everyone is treated fairly, given access to the same opportunities as everyone else, and, when radical differences in circumstances arise, provided with more or less support on a particular front. In this sense, equality is anticipatory. That is, it concerns preparation, so that students may have an equal footing in their academic endeavors (competitive or otherwise).
Equity, in its stricter sense, is instead about ensuring equal outcomes, such as, for instance, very similar test scores across populations.
However, equal does not guarantee equitable. Providing equal access and resources to a population does not guarantee the same results on test scores or whichever criterion we are examining. So, when we talk about improving education for a more equitable world, which of the two meanings do we imply?
Maybe it is both. Maybe equality is what we want. Maybe equity in its strict sense is not a desirable priority. Or is it? I do not mean to suggest final answers to these questions. I simply mean to bring them to the light because, if they are indeed as crucial as I deem them, we will need to keep discussing them and remain open to transforming our very conceptions of ideas that would otherwise be adopted uncritically.
Heimonen and Hebert’s 2019 study on equity and equality as central concepts in Finland’s music education curricula may help us to shed some light on the legal and curricular dimensions of these similar yet significantly different concepts and their applications. Regardless of the answers we focus on, clarity on equity’s conceptual roots is necessary.
This same clarity is essential not only as a thinking tool but also as an index for measurements and a guide for resource allocation.
Say a program manages to equalize access to school and resources for test preparation. However, after years of equal access, test scores still show a discrepancy between two populations. By a standard of equality, this would be a minor issue since all students have equal access to school and resources (with the obvious caveats since perfect equality is only an ideal, though a vital one at that). It may need to be monitored, but no special effort would go toward addressing the discrepancy. In turn, this may free important resources, material and otherwise, that can then be used to deal with other issues elsewhere.
If, however, equity and equitable outcomes are the measures we want to focus on, our conclusions will be that much different. The school’s inequitable test outcomes suggest that equity has not been fully realized. In turn, this means that special effort should be reserved to ensure it is. And sure enough, this is a viable conclusion since test scores do affect school admission, employment, future socio-economic status, health, and much more.
Are Assila Rachid and Igbida (2022, p. 162) right in suggesting that “it is no longer a question of equalizing the chances of access to school, but of guaranteeing fair academic competition”? Or is the emphasis on equal access ever more critical in the digital post-COVID-19 age? And, crucially, what are the mechanisms by which we define fairness in academic competition?
If we wish to pursue equity in its literal sense, we will see challenges and statistics differently from how we would see them if we wished to advance equality. Our actions, measurements, and strategies depend on the ideals we want to realize.
A Word on Improvement
Another question we may ask has been receiving more and more attention as we begin to unveil the full extent of the COVID-19 pandemic’s repercussions on all things in education.
What constitutes improvement? Are numerical measurements—of test scores, income, enrollment, etc.—sufficient signs of improvement? Do we have clarity across the board on what Social Emotional Learning is and what improvements in SEL look like?
Efforts made by individuals and organizations should be based on as high a warrant of evidence as possible. This is where the utility of trials and pilot programs comes in. Before scaling up efforts, we need to ensure they show at least some promise of efficacy. An initiative that is not based on reliable evidence is unlikely to bring about the improvement it is intended to induce.
But, again, a solid guiding framework is necessary to advance fruitful reform. The post-pandemic world is one where our priorities might change drastically. Indeed, they already have. Are we clear on what and how we want to measure progress?
A Word on World
Last but never least, what does “world” mean?
A simple question, is it not? As fanatical forces the world over rage political and military wars on claims of exclusion and supremacy, we are reminded that this question is anything but simple. The question of belonging and of visibility is one with which we will have to contend.
The 2023 conference title implies that our efforts should ensure that we make the world a more equitable place (for everyone). But the questions remain.
Who is the “other” in education? How is the “other” included or excluded in decision-making processes? Who are; nay, who should count as a stakeholder with decisional power in curriculum reform and resource allocation? Global citizenship—what does it really mean in practice, and how do we assess whether or not students practice it?
Park et al.’s 2013 report for the Carnegie Foundation for the Advancement of Teaching states that “the work [of stakeholders committed to improving education] should center on engaging relevant actors in co-developing testable hypotheses” for specific problems (p. 4). I deem it safe to assume that a list of relevant actors in the context of education would include parents and students. How, then, do we engage parents and students in co-developing testable hypotheses for the specific problems we are trying to solve? Creative minds will need to devise new ways of co-creating educational reform that go beyond the tried and true but perhaps limited survey-led approach.
In a research article titled “Beyond Western, Educated, Industrial, Rich, and Democratic (WEIRD) Psychology: Measuring and Mapping Scales of Cultural and Psychological Distance,” Muthukrishna et al. (2020) offered a statistical comparison of psychological data from countries other than Western, Educated, Industrial, Rich, and Democratic (WEIRD) ones. The purpose behind this study was to remedy an allegedly universally applicable scholarly enterprise “that may have instead uncovered truths about a thin slice of our species” (p. 678). In other words, Muthukrishna et al. sought to diversify the comparative scope of psychological studies.
In the education world, overreliance on the United States (and the rest of the “West”) presents a similar problem. How might we follow in Muthukrishna et al.’s steps and gather truly international data sets and observations? The “CI” in CIES stands for Comparative and International. But can we call ourselves an international movement when a significant majority of the ideas we engage stem from partial observations of non-representative populations? Who is part of our world? And why are some people not part of it?
These queries barely begin to contextualize questions of belonging, inclusion, and exclusion in the realities of Indigenous communities around the world. Speaking of transformation and reform in Indigenous education in Australia, Hogarth calls for “a revolution: a revolutionary transformation of institutional and societal constructs; a cognitive awareness of how language and discourses are used to maintain power and a need to privilege Indigenous voices and knowledges to ensure that Aboriginal and Torres Strait Islander peoples’ rights in education are achieved” (2018, p. 663). Do we have clear frameworks to respectfully engage Indigenous knowledge without judging or dismissing it by the same standards it often seeks to contrast? Are attempts at transforming educational spaces from economically-driven to “spirituality-based” (Barkathunnisha et al., 2019) truly “decolonial,” or only nominally so?
The recent surge of interest in the many forms of Indigenous education and the alternatives they (re)present, seen partly through several conference presentations at CIES 2022, necessitates continual reflection on questions of “othering,” of access and permissibility, of space, of dominance and critical thinking, and of much more.
Concluding Remarks
To the experienced researcher, many of the questions I raised might seem obsolete or unworthy of sustained attention. Yet, no matter how much time we already spent in contemplation, today’s precarious reality demands that we return to them regularly as we further our practical commitments.
I hope I made it clear that I am not proposing any concrete answers. What I intended to do in this short essay is raise as many points and questions as possible to capture the breadth of the work and research at hand. Success is a target we construct; improvement an aspiration we constantly revise. Whatever our practical commitments, a lucid theoretical pathway can give us a solid sense of direction that will in turn make our work easier, more easily evaluated, and more effective.
For many decades, CIES has been a space where such questions emerge, develop, and eventually translate into sensible action. I hope these words will reinvigorate interest in theoretical discussions; not as vacuous ruminations of abstract ideas, but as vital steps in our efforts to catalyze meaningful educational reform. Like the archery committee without guidelines, a program of educational reform that lacks clarity of purpose risks being futile or ineffective. That, we cannot afford. CIES has led the way in ensuring that our actions are grounded in solid aspirations. I sincerely hope that the 2023 conference will continue this vital practice.
[i]See Azorín (2020) for a discussion of COVID-19’s effects on the Spanish education system; Crawford and Cifuentes-Faura (2022) for a systematic review of COVID-19’s setbacks in higher education with respect to sustainability goals; and Duby et al. (2022) for a study of the effects that COVID-19 and lockdowns have had on the educational experiences of adolescent girls and young women (15–24) in South Africa.
References
Barkathunnisha, A. B., Diane, L., Price, A., & Wilson, E. (2018). Towards a spirituality-based platform in Tourism Higher Education. Current Issues in Tourism, 22(17), 2140–2156. https://doi.org/10.1080/13683500.2018.1424810
Azorín, C. (2020). Beyond covid-19 supernova. is another education coming? Journal of Professional Capital and Community, 5(3/4), 381–390. https://doi.org/10.1108/jpcc-05-2020-0019
Boeren, E. (2019). Understanding sustainable development goal (SDG) 4 on “quality education” from Micro, Meso and Macro Perspectives. International Review of Education, 65(2), 277–294. https://doi.org/10.1007/s11159-019-09772-7
Crawford, J., & Cifuentes-Faura, J. (2022). Sustainability in higher education during the COVID-19 pandemic: A systematic review. Sustainability, 14(3), 1879. https://doi.org/10.3390/su14031879
Duby, Z., Jonas, K., Bunce, B., Bergh, K., Maruping, K., Fowler, C., Reddy, T., Govindasamy, D., & Mathews, C. (2022). Navigating education in the context of covid-19 lockdowns and school closures: Challenges and resilience among adolescent girls and Young Women in South Africa. Frontiers in Education, 7. https://doi.org/10.3389/feduc.2022.856610
Hebert, D. G., & Heimonen, M. (2019). Advancing Music Education via nordic cooperation: Equity and equality. Taylor & Francis. Retrieved June 26, 2022, from https://www.taylorfrancis.com/chapters/edit/10.4324/9781351045995-8/advancing-music-education-via-nordic-cooperation-marja-heimonen-david-hebert
Hogarth, M. (2018). Talkin’ bout a revolution: The call for transformation and reform in Indigenous Education. The Australian Educational Researcher, 45(5), 663–674. https://doi.org/10.1007/s13384-018-0277-8
Laura Hamilton and Heather Schwartz. 2019. Get Smart About Social and Emotional Learning Measurement. American Enterprise Institute. Retrieved June 26, 2022, from https://files.eric.ed.gov/fulltext/ED602480.pdf
Muthukrishna, M., Bell, A. V., Henrich, J., Curtin, C. M., Gedranovich, A., McInerney, J., & Thue, B. (2020). Beyond western, educated, industrial, rich, and Democratic (weird) psychology: Measuring and mapping scales of cultural and psychological distance. Psychological Science, 31(6), 678–701. https://doi.org/10.1177/0956797620916782
Park, S., Hironaka, S., Carver, P., & Nordstrum, L. (2013). Continuous Improvement in Education. Carnegie Foundation for the Advancement of Teaching. Retrieved June 26, 2022, from https://www.carnegiefoundation.org/wp-content/uploads/2014/09/carnegie-foundation_continuous-improvement_2013.05.pdf
Rachid, A. B. A., & Igbida, I. (2022). Quality, Equality and Equity in Education: Historical Overview and Conceptual Clarifications. International Journal of Research in Education Humanities and Commerce, 3(2), 162–179.
Rahman, M. M., & Pandian, A. (2018). A Critical Investigation of English Language Teaching in Bangladesh. English Today, 34(3), 43–49. https://doi.org/10.1017/S026607841700061X
