Introduction

Appropriating technologies for teaching and learning purposes is a fundamental element in the contemporary educational context – especially at the time of the COVID-19 pandemic and after this event indeed. Educational institutions cannot ignore the existence of applications, software and devices that, with intentional pedagogical use, can be applied in specific didactic contexts, in order to improve the construction of knowledge.

This reality does not cease to include Distance Education (DE), which, markedly, already uses technologies for its educational process. What should happen in this modality is a greater reflection and a differentiated practice based on the experiences collected during the period – since the intentional appropriation of technologies for teaching certain content is a particular analysis of each reality. “The affordances, or ways of using, digital technologies present opportunities for innovative usage in education but also remain a challenge to apply effectively” (Paskevicius, 2017, p. 125).

Especially distance education needs this reflection, since, sometimes, the application of technologies in this modality can be considered as something “natural” or “automatic”: use of forums for discussions, videos for exhibitions, tests to verify learning, texts for individual study etc. None of these tool choices, however, can be taken thoughtlessly, since each area of knowledge, each pedagogical basis, each group of technologies and each context may demand a different set of solutions to favor the learning of this specific group of students (Garbin & Oliveira, 2019).

It is this discussion that the theoretical framework of this article covers. Developed over the 2000s by Koehler and Mishra (2005; 2008), based on the sayings of Shulman (1986; 1987), the explanatory model of the teaching action Technological Pedagogical Content Knowledge (TPACK) points out that there may be a set of pedagogical, technological and content solutions for each particular context, as shown in Figure 1.

Figure 1 

Theoretical framework TPACK.

Source: reproduced by permission of the publisher, © 2012 by tpack.org.

According to Shulman (1986; 1987), there are two bodies of knowledge required in teaching practice: Pedagogical Knowledge (PK) and Content Knowledge (CK). The former includes knowledge about teaching and learning theories and assessment, for example, and the latter involves specific elements of each area of knowledge, such as arts, philosophy, biology, history, chemistry, physics, sociology, etc.

The growing adoption of technologies in education made Koehler and Mishra (2005; 2008) create the TPACK, that incorporates Technological Knowledge (TK) into the initial structure, which refers to the understanding of how technology can be applied at the work or in everyday life, when it helps and when it does not, and when it is necessary to update yourself in technologies and how to do it (Irdalisa et al., 2020).

As a result of this theoretical approach, later researchers in the field developed their main data collection instrument, the Survey of Preservice Teachers’ Knowledge of Teaching and Technology (Schmidt et al., 2009). They also advanced in studies, especially through articles and three main works, entitled Handbook of Technological Pedagogical Content Knowledge (TPCK) for Educators (AACTE, 2008), Handbook of Technological Pedagogical Content Knowledge (TPACK) for Educators (Herring et al., 2016) and Handbook of research on TPACK in the Digital Age (Niess et al., 2019). In them, it is clear the advance of research in the area, especially regarding teacher training through the integration of technologies and the pedagogical and technological aspects that are used to develop and transform teacher training from this perspective.

So, this research applies netnography in forums of the course Calculus III, of the undergraduate in Physics from Virtual University of the State of São Paulo (Univesp), in order to understand what happens with assessment, interaction and technology in a distance education course. Thus, this article contributes in a new way to reflect about the subject of assessment, interaction and technology in distance education, with students and teachers involved in their own environment. This knowledge can help to understand how technologies can be used in a meaningful way in technology-supported teaching.

Theoretical framework

The concept embedded in TPACK framework involves how to apply technologies to teach some specific content. That includes, especially in distance education, how to assess the students, how to provide a meaningful interaction and how to appropriately use the technological tools and resources available in a Learning Management System (LMS).

Assessment can have two uses in education, according to Perrenoud (1999): or it can be used in order to improve the learning or to do a selection. The best use is to improve the learning, by assessing the students according to the competences developed and providing constructive feedbacks. If the aim of this education is focused on learning, the cornerstone must be the student. That is why schools and teaching methodologies need to suit to students, not the opposite. As much as this may materialize their life stories and expectations, teachers should not work for an “ideal student”, but for this student who stands in front of them, with his potential and his points to improve. The awareness of the interdependence of the multiple factors that involve an educational problem allows that, in the face of school failure, the old trap of blaming one of the agents of the process is not incurred to justify not learning (Schermack & Sant’Ana, 2018).

The assessment can be aided qualitatively to identify failures and carry out interventions so that students build knowledge through interaction. It is necessary to think about how to make better use of the amount of data that LMS provides, regarding the use and access of tools by students, so that the assessment is in fact meaningful and helps in learning. Ogange et al. (2018, p. 32) define this kind of teaching action as formative assessment, that is “gauged based on: (1) how students perceived the difficulty level of various types of formative assessment, and (2) how students perceived feedback provided by the eLearning system, peers, and course lecturers”.

So, if there are more than one person (at least the one who assesses and the one who is assessed), it is necessary to discuss interaction. “Due to the […] nature of online learning contexts, instructors can use formative assessment to enhance interactivity between students and other students, and between students and the instructor” (Ogange et al., 2018, p. 29).

The interaction, and most specifically the dialogue, is seen as a contact carried out by two or more people in a targeted, constructive and appreciated way by them. Included in this concept, then, are the act of paying attention and contributing to what the other is communicating. In this case, points out Peters (2003), the interaction must actually take place, and not be something programmed in didactic material – that is, in dialogue, speaking becomes a social action.

Thinking about courses that have planned synchronous interactions, such as web conferences or chats, the dialogue is well defined: the teacher and the student exchange information and need to pay attention to each other and be interested in what the other has to say. But this can also materialize in asynchronous tools and resources, such as forums. However, in this case, the instructor needs to use linguistic strategies to demonstrate more clearly this openness and interest in what the student has to say. In this context, “interaction is also considered as a key factor in online distance education and it is a good predictor of learning” (Kara et al., 2019, p. 6).

And technology is a very common theme when the subject is distance education, once a LMS is often the space where the learning takes place. Peters (2003) points out that the education supported by digital tools through the internet and compatible devices provides distance education with greater flexibility and an enormous didactic potential for change.

According to Garbin and Oliveira (2019) and Oliveira and Piconez (2016), this reality provides a wide range of pedagogical possibilities for distance education, such as strengthening interaction, individualizing teaching and independent learning. Even in simpler courses it is possible to use the potentialities of the internet in order to facilitate the learning of the most diverse learning styles, through texts, bi and three-dimensional graphics, coloured illustrations, audios, videos, animations, simulations, games, immersive environments etc.

Methodology

Netnography, also called ethnography on the internet or ethnographic research online, is a procedure to approach the reality investigated with a focus on the study of subjects inserted in cultures and digital communities. As its name implies, this data collection procedure has similarities with traditional ethnography, which involves immersing the researcher in the culture to be studied, in order to become part of it, but with the required distancing needed to the scientific description and analysis of the community. There is a possibility that netnography is one of the data collection techniques of a larger research (as is the case of this investigation) or that it is the core of the research, the central methodological approach, and other techniques may be used in order to complement. In fact, it may be necessary to monitor in person these communities, which sometimes go beyond the limits of the internet and have face-to-face relationships (Kozinets, 2009; 2015; Hine, 2015).

In both possibilities it is essential that the field to be studied is in fact characterized as a culture or a virtual community, and not just a field for momentary interaction or an online place where a message is posted and there is no further follow-up. “Community”, in a way, is a group of people producing something in synergy, that is, according to Kozinets (2009; 2015), it refers to a group of people who share social interaction, social ties and a common interactive format, location or “space”, although in this case, a virtual or computer-mediated ‘cyberspace’. And what is produced or shared has to do with “culture”, which, simply put, involves the creation and production of something – both material and symbolic – in constant flux of transformations so that its making and perpetuation or alteration are strongly based on communication (Kozinets, 2009; 2015). Both Impedovo and Malik (2019) and Fahara and Castro (2015) applied this methodology in distance education courses and had good findings.

In this investigation, netnography is used as the main procedure for data collection, and not as a research center. It is understood that, within an LMS, there are elements for interaction and collective construction of knowledge that provide the possibility of forming an online community, focusing on the learning of the subjects involved there. The main means of observing the culture produced by this community is the forum, a virtual place where the researcher can capture the records, transpose them and make the analysis of these data. In the case of this research, the LMS used by the institution is Canvas.

So, the data collection instrument was netnography, by following the interactions in forums of the course for two months. All the interaction was registered in the LMS itself, and the researcher used a logbook to take notes of standards and other relevant information. This methodological path was important in order to collect relevant data from the students and professor in their own environment.

Univesp, as said, was the locus of this research. It is the only public university in São Paulo-Brazil that is exclusively dedicated to distance education. Since 2012, the institution’s missions include:

  • knowledge as a public good, that is, openness;
  • universal access to education;
  • the application of innovative methodologies; and
  • the intensive use of technologies in educational contexts.

Univesp has offered its undergraduate courses since 2014, when 3,330 students were enrolled in the following undergraduate courses: teacher training in Physics, Chemistry, Biology, and Mathematics, as well as Production and Computer Engineering. Other entrance exams took place in the following years and, currently, about 50,000 are active undergraduate students at Univesp.

In this study, the course Calculus III, of the 6th academic semester (3rd year) of the undergraduate course in Physics in 2017, was randomly selected. So, the participants were not chosen: the actual 216 students and the professor of this course participated in this research.

The professor is a PhD in Physics from the University of São Paulo (USP), the most prestigious of Brazil. He has been professor at this same university for more than 35 years and has worked for Univesp since the beginning, teaching courses like Calculus, Physics and Mechanics.

The students have the following characteristics: 60% are white, 60% are married, 63% have children, 75% have not studied for the last 10 years, 80% have a full-time job and less than 10% have parents who reached higher education. They are older than the face-to-face students from other institutions and, as consequence, they also are not digital natives.

The data collected was analysed from Bardin’s (2007) perspective, according to whom it is necessary to organize the collected data, to code them into smaller units of meaning and categorize them, before or after the analysis itself. In this paper, the categories were defined before the collection: assessment, interaction and technologies.

Findings and results

Netnographic analysis was the main data collection procedure and the approach used in order to approximate the investigated reality. It allowed monitoring the interactions in the Questions and Answers forums of a course of the undergraduate in Physics.

Students and trainer interacted through 60 posts, divided into 10 Questions and Answers forums throughout the course, distributed as follows:

  • Video lesson 1 – Scalar Fields and Vector Fields: 13 posts;
  • Video lesson 2 – Differential and Integral Calculus in Physics: 26 posts;
  • Video lesson 3 – Curved coordinates: 2 posts;
  • Video lesson 4 – Volume integrals: 3 posts;
  • Video lesson 5 – Differential geometry elements: 1 post;
  • Video lesson 6 – Curves and line integrals: no posts;
  • Video lesson 7 – Integrals on Surfaces and Flow of a Vector Field: 3 posts;
  • Video lesson 8 – Strokes and applause theorem: no posts;
  • Video lesson 9 – Gauss (or Divergence) Theorem and Applications: 7 posts;
  • Video lesson 10 – Irrotational fields and fields with zero divergence: 1 post;
  • Video lesson 11 – Ordinary Differential Equations (part 1): no posts;
  • Video lesson 12 – Ordinary Differential Equations (part 2): 1 post;
  • Video lesson 13 – Differential operators in curvilinear (or generalized) coordinates: no posts;
  • Video lesson 14 – Green’s Identities and Green’s Theorem – Review: no posts;
  • Video – Differential and integral calculation for engineering III – Lesson 11 – Integral line of vector fields – part 1 | Univesp TV: 3 posts.

These numerical data of posts are shown in Figure 2.

Figure 2 

Percentage of posts throughout forums.

Note: Total of 60 posts (100%).

The first forums generated more interactions than the offered at the end of the course. Several factors explain this fact: the initial forums were more available to students in terms of time, the introduction to the subject used to generate more doubts and the final forums were made available closer to the final tests.

The highest level of student participation was 64%. The trainer made 36% of feedback, as shown in Figure 3. In this figure, the darker grey represents students, and the lighter grey represents the professor.

Figure 3 

Proportion of participation in forums: students (64%) and trainer (36%).

Note: Total of 60 posts (100%).

These data indicate that not all student posts have been answered. They were not always individualized, and students were often left without answers to their questions. These aspects probably justified the reduction in posts during the course. Students need answers quickly and many teachers have had some difficulties concerning availability, due to contractual issues and difficulties with technological knowledge to use a forum to assess formulas and mathematical algorithms, typical of the abstract notational language of this knowledge area.

Assessment

The first post in the Questions and Answers forum of the first video lesson, under the perspective of analysis of the assessment category, is revealed by the following student:

I’m having a hard time doing the assessment activities, I couldn’t develop any of them, and I’m worried. Is there any other material I can see? (Student A).

The student demonstrates not having understood the content or not being able to apply it in the assessment exercise, proposed by the professor. It is a question similar to what another student explains in the following message, about the same video lesson:

I am having difficulty answering the activity for assessment this week, despite having attended the classes. I am unable to develop it because it has more explanations than practical guidance. Would it be suggestive to provide material with practical activities to better understand the content explained? I confess that I am having trouble developing and understanding the explained matter. I need more examples! (Student B).

There is a similar post:

After carefully watching the video lessons of that first week and evaluating the introduction to the new topics as positive, when opening the file with the assessment activity, I noticed that there are several calculation exercises to be carried out and not questions about the theory explained in the lessons.

How to perform the proposed exercises if:

  1. in lessons there was no solution for any exercise;
  2. in the Base Book, Chapter 8, indicated for reading, there is no resolution for any exercise;
  3. in the other videos there is no resolution of any exercise;

How to proceed to understand and how to perform the proposed exercises, without having had any example done in any of the didactic materials offered?

I would appreciate clarification to be able to accomplish my tasks (Student C).

The student exposed a discrepancy in the didactic strategy of the professor in relation to the difficulties of integration between the different resources he used. He cited what was taught through the video lessons, the book chapter indicated and the other videos as insufficient resources for him to be able to carry out the assessment activity. While the teaching materials were theoretical, the assessment demanded a practical application of this theory, without explaining it or exemplifying it to the students. This message was not answered by the subject professor.

According to the ADDIE instructional design model (Analysis, Design, Development, Implementation, and Evaluation), the course design process is separate from development and implementation (Bates, 2015). This division can generate discontinuity in the learning unit, because the professionals and teams that participate in one phase do not usually participate in the next. The learning unit as a whole is not often considered.

It should be noted that at this time, Univesp prioritized the design and the development of video lessons in relation to other teaching materials, which can enhance the discontinuity of the final learning unit. Professors are encouraged to record video lessons in advance and only then plan the learning unit as a whole, without prior knowledge of the students’ contextual universe (prior knowledge required by specific content, previous education, difficulties in using the selected technologies, etc.).

In the institutional project, the assessment is the last step in the process of building the learning unit and the course. The lack of connection between what is taught in the video lesson, what is shown in the texts, and what is requested in an assessment stem from this separation of processes and teams.

The gap in the production of learning units can be summarized by this comment from a student in the course:

I did not understand the resolution of the assessment. I think the subject has not been explored as well as it should. Quite different from Calculus I and II lessons, which we had in the past semesters. Not even watching other videos available on YouTube would be able to respond to this assessment. I hope to get better (Student D).

The speeches cited up to this point are from students. They usually answer to each other, and professor respond to most posts – which can give the discussion about the second category.

Interaction

Ideally, the Interaction in this course should occur in less than 48 hours from the question and be individualized. In video lesson 1, posts from February 24th, March 4th and March 5th were answered by the professor on March 14th. The answer to the three students was the same, that is, the professor copied and pasted this same text:

If you are more specific, I can help you. Anyway, all exercises involve direct applications (partial derivatives) of the formulas that are in the second video lesson (Professor A).

There are posts that were answered on time and others whose students did not get feedback. It is noteworthy that the professor’s answers are sometimes not open to replies, that is, not open to interaction. The following speech is an example of such a response from the professor:

I think the bibliography is sufficient. However, there is no need for that. All you have to do is to derive (partial derivative, of course) the function twice with respect to x (partial derivative, of course), then derive the function twice with respect to y and then with respect to z. Add up these 3 partial derivatives. Ready. What you have obtained is a new field (which can possibly be constant). That’s it (Professor A).

By using expressions such as “of course”, “ready” and “that’s it”, the professor indicates to the student that he should know this information beforehand – even if the student has not been informed of this – and that there are no more questions to ask or explanations to be given. In fact, the student no longer questioned the professor. But interaction, especially in distance education, requires openness to dialogue by both parts.

About interaction, Muzammil et al. (2020, p. 89) points out that “engagement and interaction are closely associated in online learning, Furthermore, the less interaction between teachers and students and the more detailed programming of the path that the student must follow, without considering their individual needs, the greater the distance that students feel in distance education courses (Garbin & Oliveira, 2019). This feeling of loneliness or isolation does not help the learning process of students, who feel the need for direct contact with the professor. Technology has the potential to enhance this Interaction once it is used with this pedagogical intentionality.

It can be seen from these data that there is not a single technological solution that applies to all professors and students, in all educational contexts, to all subjects, and for every teaching vision. Teacher training for the use of technologies in such a specific field as Calculus in undergraduate Physics requires even prior knowledge of Mathematics, which students are supposed to review by themselves. The ideal solution resides in the teacher’s resilience (flexibility for necessary adjustments) to navigate in the space between specific and pedagogical knowledge and between the complex interactions between them in a certain context with the support of technological resources. According to Koehler and Mishra (2008), from the perspective of TPACK, this body of knowledge must go beyond the pure and simple joining of the three parts that compose it. It comes from the interaction of pedagogy, technology and specific content through meaningful and deeply structured teaching with technologies.

Even among the students themselves, interaction through the virtual learning environment is reduced: 17 students participated in the Q&A forums, a number that represents 14% of those enrolled in the undergraduate in Physics. The students interacted with each other only in seven posts. The forums are open to all students of the course, with the intention of encouraging the participation of the largest possible number of students and optimizing the collective construction of knowledge.

One of these interactions between students, highlighted as follows, is a clarification regarding the use of the forum:

Dear colleague, if you haven’t accessed it, go to “Discussions”, you have 23 posts and 4 answers that can clarify your doubts, don’t give up. Better, access questions and answers from lesson 2 (Student D).

Where do I find these discussions? (Student E).

Access questions and answers from video lesson 2 (Student D).

It means that the students can help themselves on forums, and the participation of all of them is an opportunity for mutual support.

Technology

The Technological Resources category appears in other posts, and the professor would need to have adequate training to deal with doubts about this. One student asks:

To send the assessment of each individual class, I identified that there is no way to send images or drawings, making it difficult to send files with formulas. Wouldn’t it be possible to make a submission available as done in the other activity, with files sent from the computer or Google Drive? (Student F).

The professor’s answer was:

I have no way to help you in this matter. See if you can refer someone from Univesp to answer your question, just like the Student Service Center (Professor A).

It is demonstrated that neither student nor teacher have adequate training to deal with the technology used to carry out the proposed activities. TPACK assumes that, in the educational process, agents – especially the teacher – have mastery of the intentional use of technology, that is, of the pedagogical content of the technologies adopted (Garbin & Oliveira, 2019). The institution itself defines these technologies and does not train the teacher for their use.

The same consideration applies to video lessons. The use of this technology is a factor that does not depend on the professor, that is, it is an institutional decision. There is no room for reflection on the pedagogical content of this technology: it is used for any and all situations. Any technology selected as most suitable for certain contexts requires the students’ perspective, their previous schooling context and the work of institutional teams in such an integrated way in order to avoid difficulties such as those reported by the student:

The slides have many problems (I have already indicated to the Student Service Center) with the formulas, although they contain very useful texts and figures (Student B).

This is because, in addition to using the expository video lesson for all teaching contexts, the slides for recording are exposed on a monitor in the studio, which makes subsequent corrections in the video lessons difficult. It would be necessary to re-record the video lesson, or at least the wrong part, and take the material again for editing and uploading.

This question of technology persisted in this post of the same student:

Professor, with all due respect for your video lessons and for you, but the errors in the formulas were already to be fixed (Student B).

And the professor replied:

To correct this, we will be making available the original version of the presentations. They will be in the LMS (Professor A).

According to Koehler and Mishra (2008), teachers need to know, in addition to the content of their course itself, how themes or their representations can be modified, based on the application of technology. In practical terms, it is important that the teacher is able to realize that technologies can be useful in understanding and doing their specific subject, at the same time that he needs to know the most common errors and misusing of this same technology.

Conclusion

At the end of this paper, it is possible to realize that technologies can be used in a meaningful way in Distance Education, considering factors like students, knowledge field, technologies, etc. It was possible to see it using netnography in forums of the course Calculus III, of the undergraduate in Physics from Univesp, based on the categories: assessment, interaction and technological resources.

Assessment category is an aspect that is often cited by students in the forums. Most of the posts refer to the discrepancy between what is taught, especially in relation to video lessons, and what is charged in the assessment. The separation between the planning of video lessons and the execution of assessment is a decisive factor in this problem. Other technologies could be used to build knowledge in an appropriate way to the content to be taught.

Regarding the Interaction category, it was found that 10% of students participate in the forums, a cornerstone of online interaction in distance education. The trainer answered most of the students’ doubts, but sometimes with a certain delay and reduced openness to dialogue.

Finally, the Technological Resources category is shown to be a potentially facilitating factor for student learning, so much so that all these discussions occurred with the use of online forums. The intentional use of technology needs to be agreed with the professor, in order to enable the integration of the content and pedagogical knowledge that he has with the technological knowledge necessary for distance education.

In summary, there is not only a possible instruction solution that can be applied for all the educational cases. Each context, professor, group of students, and institutional reality requires a specific didactic design in order to perform the best learning in the specific context. And, in order to explore the reality and understand the contexts, the use of netnography is quite effective.

Other categories can be used to discuss distance education through applying netnography – which is a good option as shown in this paper. And it is possible, also, that courses that have mandatory participation in interactive tools or are carried out for a longer period can provide more data. For further research, it is indicated to use netnography in order to go deeper in assessment subjects, which is rich in information especially in distance education, as well as to use this method in other contexts.