The MaTE Tool—Enabling Engaged Scholars at a Regional University

Providing institutionally recognized evidence of community engaged scholarship has long been problematic for engaged scholars when applying for recognition through promotion or probation pathways. To combat this, the University of Wollongong [in New South Wales, Australia] developed an online tool for use by engaged scholars to track and measure their engagement activities in a consistent and institutionally recognized form. This article outlines the process that was undertaken to develop the online system for measuring and tracking engagement (the MaTE tool). It outlines the initial recognition of the key issues arising from a comprehensive review of the literature; the drafting process undertaken to develop a prototype for the tool; and the interview stage and subsequent re-drafting process and finalization of the tool. The article concludes with a consideration of future directions for the tool and its further implementation at the university.


Introduction
The modern scholarship of engagement was first espoused by Ernest L. Boyer in the mid-nineties (Boyer, 1996) however, evidencing community engaged scholarship (CES) for academic recognition and reward has proven problematic.Like all areas of scholarly achievement, measuring, tracking, and evidencing CES are integral to its reputation and academic legitimacy within the higher education sector (Holland, 2001b).Unfortunately, the unique nature of this scholarship does not always lend itself to the more traditional and accepted forms of scholarly evidence often prioritized in recognition and reward structures.This is creating institutional barriers for engaged scholars in receiving recognition and reward for their valuable work (Cuthill & Brown, 2010;Jaeger & Thornton, 2006;Macfarlane, 2007;Ward, 2003), an issue that recently became apparent at the University of Wollongong (UOW).
The evidencing and reward issues that existed for CES scholars at the UOW (a large regional Australian university) became fully apparent in 2011 during a university-wide promotions review project.For this project a series of interviews with faculty (28 in total) were undertaken, where it quickly became apparent that CES work was misunderstood, unrecognized and unrewarded in the university's promotional process (Crookes, Else, & Smith, 2015).Whilst faculty at the university appreciated CES and felt it was worthwhile, they did not believe it was recognized in promotion processes.They suggested that this was because the scholarship itself was poorly understood across the university and that CES did not provide enough recognizable, scholarly evidence, with one individual claiming that CES needs to provide "some hard evidence." Another (senior management) academic stated that CES "is not recognized or rewarded, it is appreciated, which is not the same thing." This low perception of CES appeared to have had a major impact on how Faculty chose to focus their academic activities, with not one of the 28 interviewees stating that they would risk going to the promotional board with CES as their primary focus.
In an attempt to redress this issue at a university level, the project team created an online repository tool called the Measuring and Tracking Engagement (MaTE) tool that aims to capture a wide variety of CES evidence.The goal of the tool is to go beyond traditional forms of evidence (such as journal articles and grants) and allow the scholar to log all forms of unique CES work in a personalized data repository.This repository can then automatically format this data into a personalized report for the individual, a report that can later be utilized as supporting documentation for their work in a variety of contexts, both internal and external to the university.
This article outlines the process that was undertaken to develop the MaTE tool prototype.It discusses the initial recognition of the issues, the drafting process, the interview stage and subsequent re-drafting process and finalisation.It concludes with a consideration of future directions for the tool and its further implementation at the university.

CES: Misunderstood, Unrecognized and Unrewarded
In order to come to a greater understanding of the issues that emerged from the initial promotion-review interviews, an extensive literature review was conducted on the subject of CES (Smith, et al., 2013).This literature review revealed that the issues that appeared to plague engaged scholars at the university, were not unique and had already been identified by CES scholars internationally.The difficulties that CES focussed scholars have in relation to recognition and reward are now widely recognized (Cuthill & Brown, 2010;Duke & Moss, 2009;Jaeger & Thornton, 2006;Macfarlane, 2007;Maurana, Wolff, Beck, & Simpson, 2001;McDowell, 2001;Rice, 2002;Rudd, 2007;Saltmarsh, Giles, Ward, & Buglione, 2009;Ward, 2003) as are the various problems surrounding evidencing and measuring this unique form of scholarship (Adams, Badenhorst, & Berman, 2005;Arden, Cooper, & McLachlan, 2007;Garlick & Langworthy, 2006;Hart & Northmore, 2011;B.A. Holland, 2001bB.A. Holland, , 2009;;Rudd, 2007).From the literature, it is apparent that the lack of recognition or reward of CES is interwoven with issues surrounding evidencing, measuring and assessing such work.As CES does not always produce the same recognized forms of evidence as the more traditional scholarships of research and teaching, it is often seen as outside of the "real work" of scholars (Ward, 2003, 2).
Displaying exceptional achievement in CES is also more complicated than presenting the more widely recognized academic outputs, such as high impact journal publications or large research grants.Some evidence has shown that this has led senior staff to discourage junior staff from CES work on the basis that it is not as career-enhancing (Jaeger & Thornton, 2006).Due to this culture of scepticism toward the success and legitimacy of CES, faculty are also inclined to leave their engaged work unpublished as "grey" literature (Hart & Northmore, 2011), or alternatively, to reclassify it under the scholarships of "research" or "teaching." Consequently CES becomes increasingly less visible in the published academic sphere thus creating a circular process that perpetuates the notion of CES as a side project or outside of the main work of the university (Cuthill & Brown, 2010).Such notions support a culture of disinclination, with faculty continually placing CES beneath other scholarships and pursuing career focus areas considered to be more scholarly, legitimate and rewarded.
In order to address the status and perceptions of CES, the literature makes it clear that the way forward lies in tracking and evidencing tools or processes that recognise and legitimise the variety of CES work.However, despite evidencing and measuring processes becoming more common on an institutional scale (Carnegie Foundation for the Advancement of Teaching, 2006), there is still little information available for the individual who wishes to track and evidence their CES activities for probation, promotion or other recognition purposes.Yet leading CES scholars such as Barbara Holland maintain that this process is integral to the success of promoting CES for reasons such as academic legitimacy, image, reputation and accountability (B.A. Holland, 2001b).Therefore, as no tool or process currently existed to enable scholars at the university to track or evidence their CES activities, the promotions project team decided to embark on the process of creating a prototype.

Creating A CES Enabling Tool: What Did We Want To Achieve?
Before the project team could begin the drafting phase of the tool, it had to come to some decisions about what the tool aimed to achieve.Questions were raised as to how the tool would function and what it would produce, such as: 1. What type of evidence was it going to collect? 2. What would it produce for the individual?
3. Was it going to capture impact and outcomes?4. Was it going to be useable by the university or just the individual?While many of these questions could not be answered with finality until later in the process, it remained important that they were considered at the earliest stages of development to establish clear direction for the drafting stage.
In discussing what type of evidence would be collected, it became apparent to the project team that there was still a great deal of dispute as to what constituted CES work in the first instance.Without a proper understanding of what constitutes a CES activity, it is nearly impossible to illustrate with any certainty what type of evidence can be produced by such work.The question of what actually constitutes CES is often a major issue and barrier to its promotion within institutions and due to this, it seemed important to adopt a credible definition before proceeding any further.In con-ducting the literature review it became apparent that while there is no universal definition of CES or "engagement, " there are some highly employed definitions currently circulating.The project team decided to use the most consistently adopted definition (B.Holland & Ramaley, 2008) created by the Carnegie Foundation for the Advancement of Teaching, which defines engagement as: [T]he collaboration between institutions of higher education and their larger communities for the mutually beneficial exchange of knowledge and resources in context of partnership and reciprocity (2006).
This definition was chosen because it is highly recognized and prioritises the mutually reciprocal nature of CES; acknowledging that there needs to be an exchange of knowledge that is beneficial to both partners, rather than a top-down imposition of "research findings" from the academy.This moves away from notions of volunteerism and provides for scholarly, sustainable partnerships that produce effective outcomes and impact for all parties.While the project team recognized that a definition would not solve all disputes as to the nature of CES, it would still act to exclude activities or work that did not involve external communities, collaboration, beneficial exchange and reciprocal partnerships-all integral aspects of CES work.
After establishing how CES would be defined, the process of deciding what type of evidence the tool would collect became simpler, if not final.It was obvious that the process of discovering what type of CES evidence was produced at the university could not be elaborated on with any conclusiveness until interviews had been conducted and those involved in CES work were able to explain the scope of their work.As the interviews were planned for a later stage of the development process, the literature review was heavily relied upon in the early stages as the major source of information on engagement evidence.Some of the evidence sources supported in the literature were surveys, observations and logs, interviews and journal articles (Holland, 2001a), peer review, published articles, academic presentations, exhibitions of work, letters of recommendation, awards or public recognition (Wise, Retzleff, & Reilly, 2002), annual community reports, annual engagement forums (Adams, et al., 2005), engaged societies, engaged networks, teaching material and courseware, articles in the popular press, acting as a reviewer for engaged papers, educational programs (Macfarlane, 2007), mission statements, policy documents, report studies (Winter, Wiseman, & Muirhead, 2006), and public lectures, public debates and art exhibitions (Winter, et al., 2006).While the sources in the literature were clearly not exhaustive, this snapshot gives an indication of the variety of evidence that might reasonably be taken into account and this helped create parameters for further development.
The project team was mindful from the outset that the tool needed to produce something for the individual scholar.The aim then was that after the scholar had entered their data into the system, the tool would be capable of providing reports that could be used as recognizable sources of evidence.This report would lay out the data that had been input and allow scholars to include or exclude certain entries, depending on the intended use of the report.This report could be used for promotional purposes or for other instances in which the user may wish to display their CES work, such as applying for a community grant or an external award.
Another major issue that needed to be considered was the level of specificity of data to be collected and consequently displayed.Would the tool collect data that would evidence traditionally recognized impact and outcome measures?Ideally the project team wanted to collect as much evidence of scholarly impact and outcomes as possible, in order to help establish CES as a legitimate career path in academia.However, the collection of impact and outcomes in a tool that aimed to be as simple and usable as possible appeared to be inconsistent with the very nature of engaged impact and outcomes.The idea that the tool could provide a single "metric" of tangible impact and outcomes, was abandoned in the early stages of the actual drafting as it became apparent that evidencing impact and outcomes was an extremely varied process (depending on the activity type) and that 'metrics' were antithetical to the usability of the tool.Nevertheless, capturing wider forms of impact and outcomes remains a future aim of the project and may eventually form an extended branch of the current program.Members of the team have subsequently been involved in professional development work at the university, focusing on assisting staff to write high-quality impact statements based on their activity data (Smith, Crookes, & Crookes, 2013).
The last question to be addressed prior to the major drafting process was whether the tool would provide outcomes for the university or just the individual.One of the aims of the MaTE tool initially, was to come to a greater understanding of what CES activities were actually being undertaken at the university and therefore it seemed short-sighted to create a tool that failed to provide the university feedback as to the range and extent of its CES impact.In acknowledgement of this, it was decided that (if the tool was implemented university-wide) the university would be able to collect data at least at a superficial level from the database, in order to assess the strength of its involvement in certain CES areas.

Creating the MaTE Tool
In creating and drafting the MaTE tool, there were three distinct phases of development.These were the initial drafting and prototype programming phase; the faculty trials and interview phase; and the final revision phase.

Phase One: The Initial Draft and Prototype Programming
During this phase, several data input categories were developed in which the user was to enter basic information that was then logged as a data entry and added to their personal Activities Index.These initial categories were: Each of these categories collected information through either free text boxes or scroll lists, from which the user could choose a variety of common options.The scroll lists also allowed the user to write their own answer if none of the options adequately described the data they were inputting.
Once material had been logged in these categories and automatically added to the Activities Index, the data could then be converted into a personalized report through a reporting function.This report function created a PDF of all the data that the user chose to place into it from their Activities Index.This meant the user was able to create customized reports, depending on what they were using the document for.For example, if a user was creating a report to apply for a community grant, they may choose to only select the most relevant or significant evidence for that application.
In order to test the usability of the tool at this early stage, an email was sent out within the (then) Faculty of Health and Behavioural Sciences, asking that interested scholars send their most recent promotional documentation to the project team for their CES data to be input into a new prototype database tool.The aim of proactively and manually inputting this data was that when the prototype was complete and data entered, interviews would be set up with these scholars and they would be asked about how well they felt the tool reflected their CES work; how they would like such a tool to function; and what they would like it to produce.Ultimately 17 faculty staff, from junior to very senior academics, sent their data through to be input into the tool.

Phase Two: Faculty Trials and Interviews
By June 2013, the initial prototype of the MaTE tool had been completed and the project moved to the next stage, which involved trialling the tool with the scholars who had sent their data to be populated within the tool.The Faculty interviews were conducted between the period of June and October 2013, in which 14 of the 17 Faculty who had sent their data were contacted for a follow-up interview.All 14 agreed to be interviewed, although only 12 of the interviews were completed in the timeframe.During these interviews, the interviewees were systematically taken through each of the data-input categories outlined and asked whether they understood what each section was aiming to capture and whether they felt that the categories successfully captured their CES work.During the interviews, many issues were raised about the prototype.While some concerns were singular to a particular type of work or discipline, others appeared problematic across the CES spectrum.It was these common issues that led to significant modifications of the MaTE tool.
Perhaps unsurprisingly, it soon became clear that the majority of the interviewees were unsure of what work fitted into CES.While many had some general view of what they felt CES was, individuals would often bring up certain activities and question whether the tool would recognise them as CES.An example of this was one scholar who worked for a certain amount of hours every week at a health facility.This work maintained their clinical relevancy and contributed to their CES research and teaching in a practical way as it informed them of real community issues at a local level.It also created a community partnership through which the scholar could engage in CES research and promote community engaged learning.However, a major issue was the fact that the scholar was paid by that health facility on a separate basis from the University, for this work.So did that constitute CES?While there is often a clear and apparent line between CES work and other work, in some cases there is no clear-cut answer as to whether or not an activity is CES.Therefore the project team had to consider how they could reform the tool to promote and remind scholars of what genuine CES work was, while not devaluing or limiting the scholarships unique scope.
The first widespread "technical" issue that became apparent in relation to the tool, was that the majority of interviewees found that it was not clear what evidence the categories were aiming to capture from their titles.Two titles were considered especially problematic-"Service Learning" and "Professional Association Memberships." Of the interviewees, 10 openly stated that they had no idea what "Service Learning" meant or implied just by looking at the title.When the concept was explained to the interviewees using some of the popular notions of service-learning espoused by well-known CES scholars such as Barbara Holland (Holland, 2001a(Holland, , 2004;;Holland, Driscoll, Spring, Kerrigan, & Gelmon, 2001), the interviewees maintained this was work they did engage in, yet would not have realized that is what the title meant."Professional Association Memberships" was equally problematic, but from a different angle.In this instance, nine of the interviewees felt that they did understand the title, but argued that it was not appropriate to capture the breadth of data that was expected to be input.The main assumption about the term "Professional Association Memberships" was that it only included external professional associations that the scholar paid to be a member of on an annual or other basis.To address this, the project team had to reflect on the terminology that had been used in both instances, as well as consider how the purpose of each category could be made more apparent to the users.The team also asserted that an individual could choose to insert any data they wish, if they perceive it to fit under a heading.
Another concern expressed by a number of the interviewees was whether the prototype created a "doubling up" of certain forms of evidence.The idea of 'double-dipping' was considered highly problematic by five of the interviewees who presented concerns as to the separation of "Consultancies" from "Engaged Grants and Other Funding." These interviewees felt this presented a crossover between categories as often consultancies were a key part of a scholar's engaged funding sources.As consultancies had been placed in a separate category to cater for the many types of consultancies that existed (both paid and unpaid), putting consultancies together with funding as the interviewees suggested, was not compatible.Nevertheless, when it was explained to the interviewees that the reporting function allowed them to pick and choose what evidence they presented, the concern of "double-dipping" was generally allayed as the scholars could enter the same evidence in both categories and ultimately only present one data entry in the report (usually in the form that was most relevant to the report's purpose).
While the concerns outlined above are quite broad in relation to the tool (such as titles and overlaps), it was surprising for the project team to find how problematic certain wording could be in relation to simple data collection.Within each of the categories there were sets of questions that were structured to be answered through free text, scroll lists or simple check-boxes.The aim of these questions was simply to capture a broad overview of the evidence source, yet one of the seemingly simplest check-box style questions in the "Engaged Grants and Other Funding" section, resulted in being one of the most highly contested aspects of the tool.The question was phrased as simply "Are you a lead investigator?"with a "Yes" box that could be checked or left unchecked.The reason the project team had chosen the term "lead investigator" was that it did not want to imply a certain type of grant provider through the use of provider specific jargon.Nevertheless, it remained important that the category could recognise that some scholars have a much higher or intense role in the achievement of a grant or funding than others.Of the interviewees, six identified this specific question as being highly problematic.

Crookes et al.:
The MaTE Tool-Enabling Engaged Scholars at a Regional University Published by Nighthawks Open Institutional Repository, 2017 The general concern of all the interviewees appeared to be that people would check this box if they were named on the grant or funding source, despite their level of involvement in achieving the funds.There were several suggestions made by the interviewees as to how the question could be restructured, such as: "Are you a chief investigator or associate investigator?""Are you the first named investigator?""Are you the principal investigator?""Are you the lead investigator?" It is important to note that the majority of the interviewees who suggested a change in terminology were generally in favor of the terminology used in the grant bodies that they dealt with most, such as the Australian Research Council (ARC) or the National Health and Medical Research Council (NHMRC).While the concerns raised by these scholars were legitimate, the project team remained hesitant to implement any terminology from specific grant bodies, as they felt this would tend toward discouraging or devaluing funding sources or grants that came from other bodies, such as community grants.The aim of this tool needed to remain on enabling the individual and in using terminology from sources that certain scholars may not engage with (such as the ARC or NHMRC).There was a possibility that this would promote the disinclination toward CES evidencing, one of the main issues the MaTE tool aimed to address.

Phase Three: Final Revisions
After the interviews were completed, the third phase of development was to make revisions based on the issues that had emerged.While many small and relatively minor changes were made (e.g.extra options given in scroll lists, altered wording in some of the questions) there were several significant changes made to the tool.
The first major change was the introduction of a final and compulsory check-box at the end of each category.This check-box took the form of a statement, which the user has to agree with before being able to enter and save their data input.The statement is as follows: This item meets the definition of engagement in that it involves collaboration with external communities (such as business, industry, schools, governments, non-governmental organisations, associations, indigenous and ethnic communities and the general public), responds to a community need and/or is undertaken in a context of mutually beneficial partnership and reciprocity.
The creation of this statement check-box was in direct response to the concerns of the interviewees as to what may constitute CES for the purpose of the tool.This statement is based on the Carnegie definition (Carnegie Foundation for the Advancement of Teaching, 2006) and aims to remind users before they finish inputting their data as to what CES is and make them reflect on whether a given activity meets that definition.While the tool cannot control what people may input, if users wish to submit their report for promotional or other purposes, they will now have to justify how their activities reflect that statement.
The second major issue that needed to be addressed post-interviews was the need for further clarity with regard to the categories and the type of evidence they aimed to capture.The first step was changing the problematic titles of "Service Learning" and "Professional Association Memberships." "Service Learning" therefore became "Community Engaged Learning" and "Professional Association Memberships" became "Professional Memberships (external)." But while these titles attempted to provide some further understanding, the project team was aware that this was hardly likely to solve the problem in full.Therefore it was decided that under each heading, a short description would be laid out for users so they could see (and hopefully understand)  These descriptions aim to make the input scope of each of the categories clear while reaffirming what CES activities are to the user.
While it is optimistic to assume that these changes have resolved any-and-all issues that engaged scholars may have with the MaTE tool, they do aim to address the primary concerns that emerged in prototype trials and interviews.Key to all of these changes is a push for clarity in relation to what CES is and how it can be identified by the individual.Though people will always bring their own interpretation to such areas, the MaTE tool aims to promote legitimate CES activities through continual reiteration of what this scholarship is and how it relates to the work of the university.In doing so, it is hoped greater understanding will prompt more academics to reconsider their preconceptions of this unique scholarship and come to consider it as a viable career path, or at least a way of strengthening the work they already do by encouraging greater engagement with communities.It is also important to note that while the individual MaTE reports will be available to academics for their own purposes (such as supporting promotion and probation applications; seeking external accreditation and/or funding), the tool will also be useful for university organisational purposes to verify CES activities and their impact at individual, unit and institutional levels.At the individual level this will be doubly useful because CES activities are now incorporated into the evidence accepted for promotions purposes (Crookes, et al., 2015).

Conclusion and Implications for the Future
Currently the MaTE tool is awaiting adoption and implementation approval from the university executive.In the first instance, it is intended that it be implemented across the newly formed Faculty of Science, Medicine and Health (SMAH), as well as being made available to Fellows of the UOW's new Wollongong Academy of Tertiary Teaching and Learning Excellence (WATTLE).If the tool is made available to a teaching-focused group like WATTLE and a heavily research focused faculty like SMAH, the project team aims to foster conversation on the value of CES as a scholarship that runs across many areas of academic activity.By promoting the presentation of scholarly CES evidence, the MaTE tool may ignite further interest in the benefit of engaged scholarship, both as a unique and standalone area of activity and as a way of doing and engaging in the areas of research and teaching.This will in turn work to break down the negative cultural perceptions regarding CES that have permeated staff Crookes et al.: The MaTE Tool-Enabling Engaged Scholars at a Regional University Published by Nighthawks Open Institutional Repository, 2017 career development approaches in recent years.It is important to acknowledge that the current university data collection approach (which does not include CES) should not continue, because without collecting data and evidence of CES, the university makes the tacit assertion that it is not valued.As the initial project interviews showed, the issues surrounding CES are university-wide and therefore the benefits of this tool would be highly applicable across all faculties.
Once approved, the MaTE tool is expected to shed some light on the CES work being done across the university.This project has highlighted to the project team the amount of CES work that is actually being done by university staff; however, this work is almost uniformly not being presented or promoted as CES due to the relative lack of importance afforded to this scholarship.By promoting the wide variety of CES work that is being done (initially through use of the university data collection function via MaTE), we hope that these activities can be brought to the fore, in the same way that has happened with quality research and teaching in the past.The university will also be able to document community engagement activities and highlight key partners, data which may also be useful in reporting to government about engaged activities and asserting a broader range of impact than is presently possible.
There are plans for the tool to be improved and increased in scope in the future, for example with the incorporation of a "Project Function." The aim of such a function would be to allow users to put in much wider and detailed aspects of their work such as projects that would link to various aspects of evidence they had already input.The function would promote the evidencing of scholarly impact and outcomes through greater data capacity.This more in-depth aspect of data collection could be added on to the current model and this would acknowledge the complex and interrelated nature of CES and further promote the collection and presentation of unique forms of evidence that are so often the hallmark of valuable CES work.Another option to be considered is the merit of expanding the utility of the MaTE Tool to non-academic staff of the university, many of whom face the same issue of a lack of recognition as faculty for the vital engagement work they undertake.