Theme 5: Reflections

This week has been about design research but most of the theme so far has been spent on looking for relevant research papers to involve and write about within the group project and further make the study coherent with different writings and opinions. 

I have to say that neither the lecture nor the seminar turned out to be as I expected regarding what we discussed during these events. I thought that this theme would have much more focus on the physical structure of the design so to speak, how different designs actually were created in order to for instance match the participants’ opinions as good as possible from the evaluation. Instead both these events felt more like a summary of the whole course which gave me an interesting point of view of important aspects that I didn´t think about during previous themes. 

One particular example was the process of iteration which makes it hard to predict the upcoming economical and time-consuming efforts that are required for a study. Recently I read a paper for the group work which basically was about the benefits of feedback loops, which means a repetitive cycle of the method phase often useful in order the get more in depth information. I wonder if there is any certain relationship between these loops and iterations in design research? More in depth information like this would most likely improve the prototype evaluation and therefore probably clarify necessary changes which in turn should reduce the number of iterations.

During the lecture with Haibo Li, he talked a lot about ideas. More precisely, how to come up with-, how to filter- and how to evaluate an idea.  He mentioned five so called basic questions about different aspects you have to think about while trying to further develop an idea. One of these questions, “Is the timing right?” caught my interest. How do we really know if the timing is right? Nowadays it seems to be rather tricky to actually develop something relatively new. For instance, look at all these applications for phones. Basically it´s impossible to get an idea and further implement it that differ from others.  To be honest the timing seems to be right promptly after essentially huge technologies are developed like Internet or more concrete smart phones. Then we are able to develop something to a comparatively new platform that hasn´t been completely exploited yet.

Comments

Blog post with all the comments

4 comments every week:

Theme 1: Research publications/Theory of science

I believe that the main question "What drives a successful e-Learning?" may be too broad and uncertain in the first place. Perhaps that´s why both the hypothesis and the conclusion as a result of these became excessively general as well. During our bachelor thesis we were told that one of the most important things is to minimize the main question, obviously because it makes it easier for us to answer our own question. That may be the big issue here. The author can´t really answer his question in a good way because of the uncertainty.

Comment on Jonas Jönsson’s blog

Would be interesting to know more about what kind of method they used during the study? Because you said that the method part was well presented. Was it a quantitative or a qualitative method or maybe a combination of those and were the benefits respectively the disadvantages just in this particular case well explained?

Comment on Therese Nothnagel´s blog

Did they mention why the tests only were given to I suppose a relatively small group of students instead of just an increased target group? Also, did you find some particular parts of the results which were not reliable because of the tiny target group?

Comment on Bobby Falck’s blog

Did they use different kinds of variables within the test to easier find correlations between for instance driving performance and response time, which was necessary to create the conclusions? Besides what was the definition of driving performance in this particular case?

Comment on Carl Oskar Stenvall´s blog

Theme 2: Theory

In my opinion there are far more benefits than disadvantages considering if the paper isn´t generalized enough. A generalized theory may be easier to understand but I believe it is more difficult to apply it in the process later on while using qualitative/quantitative methods. Normally a specific main question, make those parts easier than a generalized one and the more specific the theory or the study as a whole is, the easier is it to create something unic that separates from other studies and therefore become successful.

Comment on Jonas Jönsson’s blog

Spontaneously, even if the environment and the program as a whole is staged, the study seems to be reasonable, because I believe a lot of us act differently while being monitored by some media, but nowadays it affects us more or less everyday in various ways. Perhaps the program will show us some hidden part of ourselves? Therefore, the theories they have used seem to be very interesting in this particular context because of the monitoring.

Comment on Therese Nothnagel´s blog

Was this TAM-theory comparatively abstract or in other words hard to understand potentially because of some missing examples? Perhaps it was necessary to put so much weight on the theory in order to make the rest of the study comprehensible? On the other hand, it was maybe an excessive amount of information which wasn´t in the end useful for the study.

Comment on Victor Bleichner´s blog

It may be good to combine explanation and design and action theories to for instance explain the background of a specific diagram or picture and clarify why it´s put in a specific context. Hence, I believe that the design and action theory could be the easiest one to merge with other theories by showing definite pictures, diagrams and so on.

Comment on Niklas Fyrvald’s blog

Theme 3: Quantitative methods

In my opinion I believe that they didn´t accomplish the most suitable method of mixed research in this particular case about emotional presence and so on. Like you mentioned, using a quantitative method to gather qualitative data isn´t unproblematic. Personally, I missed some kind of clarified link or correlation between those methods because I thought it was quite hard to clearly understand what they came up with in the result and in the conclusion, especially when they involved so many abstract terms like social, presence and emotion.

Comment on Jonas Jönsson’s blog

In my opinion one possible reason for that mixed research is going to increase in popularity could be because of queries regarding viability and reliability. In order to accomplish a trustworthy study in the long run, these kind of questions will be raised according to me. The more technical solutions we have, the more easier it is too raise doubts about those. The best way to create viable and reliable studies may be within mixed studies if they are used in a proper way, for instance by avoiding different kinds of self-selection.

Comment on Therese Nothnagel´s blog

I totally agree with you about the vague explanation of the diagrams and the numbers. Also, I expected a more clarified explanation of the method and the result especially when so many abstract terms like emotion, social and presence are involved. I read several sentences many times as well, but was still very confused in the end. In addition, I believe it´s quite difficult to accomplish reliable general studies about emotions obviously because it´s hard to make comparatively objective conclusions and they could in turn easily be questioned.

Comment on Simon Roth´s blog

I believe that the quote is somehow supposed to emphasize quires about viability and reliability, since I assume it´s rather easy to criticize mixed research regarding these two in terms of how the quantitative and qualitative methods correlate between each other. For instance, asking a participant if he/she wants to attend to an interview within a survey would most likely affect the reliability of the mixed research negatively. Like we heard from todays lecture, interview objects should instead randomly be picked.

Comment on Carl Oskar Stenvall´s blog

Theme 4: Qualitative methods

Your question is interesting as well, but I don´t think that one of the two methods is more preferred than the other one. I believe it’s very important to make methods easy to follow and therefore make the results and the conclusions more reliable. Commonly concrete, short-answered questions with many participants are more suitable for quantitative research and abstract, developed questions with few participants are more suitable for qualitative research. Qualitative research might give better answers in general but it may be unnecessary in certain smaller frontiers as well.

Comment on Jonas Jönsson’s blog

Your question is interesting because I couldn´t find any significant purpose of the first two examples mentioned in the text. The latter one had at least a cleaning purpose. Maybe they should have put clothes/decorations on objects which have a particular meaning and are not only used as toys and therefore further discuss how these affect the objects.

Comment on Bobby Falck’s blog

Like we talked about during the seminar, this method in this particular context might give a lot of unnecessary information since it lasted for a month. Interesting to know the corresponding useful time which they really used for analyzing in the result. In addition, you mentioned that they used the system freely during a month. I assumed that some participants have used the system more than others and therefore been more acquainted with the system, so it may be difficult to equally analyze the results as well.

Comment on Victor Bleichner´s blog

We had some focus groups in our method during our bachelor thesis. I believe it was harder than I expected to get equally spread widely answers from all the participants, even if the groups were rather small, like 3-4 people in each group. Although we had homogeneous groups, one person often dominated and outvoted the others and perhaps you don´t want to interrupt him or her. Therefore it´s quite hard to accomplish perfect focus groups so to speak even if you are well prepared.

Comment on Simon Roth´s blog

Theme 5: Design Research

Interesting that my research paper had a lot in common with yours like MMORPG as a method for learning. Since it seems that your study lacked some necessary information about the game it may be hard to further argue about the study. If they only mentioned that they wanted to evaluate how users could learn collaboratively while playing together, it seems very general according to me because all MMORPGs have that more or less in common. The question is did they learn more about the game or something else? I believe that design research papers have to be put in a more specific context in order to use the benefits of it. In this case to clarify the purpose of the game by carefully explain what they actually learn collaboratively.

Comment on Jonas Jönsson’s blog

Interesting that you mention that audio feedback was measured to be the most effective while HUD was perceived to be the most effective according to the users. I assume that efficiency isn´t the main priority in this case. I guess it´s easy to miss some particular sounds as a driver while a display gives the driver the opportunity to look at the display whenever he/she wants. What was the motivation according to the users?

Comment on Simon Roth´s blog

Unlike you I think that design research is comparatively easy to grasp especially when you receive a clear background of why the design is appropriate and when possible adjustments are motivated with for instance pictures. But of course, maybe I have been fortunate to have chosen some relatively "concrete" studies about games. But design research about games may be easier to grasp than for instance some unknown new technology tool mainly because we are more acquainted with games and quite often in which context they are being used.

Comment on Carl Oskar Stenvall´s blog

Observation seems to very common within design research. I wonder if the students were aware of the observation in a way that potentially could affect the result badly? In addition observation is probably often quite useful during moments when it´s hard to do corresponding interviews, but in your case they used interviews as well. Was it necessary to combine those two according to you in order to make the procedure?

Comment on Niklas Fyrvald’s blog

Theme 5: Design Research

Prototypes have undoubtedly a very important role in research since they are supposed to work and look like the final commercial product while they simultaneously rather often are made of simpler and cheaper materials. Basically you are able to do user studies broadly to the same extent with an operational prototype which probably facilitate the financial encumbrance later on. Ongoing research is still very expensive for a lot of companies even with several prototypes involved in the process. Imagine a world without prototypes. Evaluations and user studies would then be made just on the final product where a lot of economical efforts already have taken place. That scenario would more or less prevent any research to be utilized because of the uncertainness of the result in the evaluation. Therefore is it important to emphasize a proof of the concept prototype and recommend that it always should be used within the research as early as possible to minimize the unnecessary efforts. 

Since I´m very interested in football I want to make room for this vibration, ball-synchronized prototype mentioned in the study. First of all you understand early that this study is comparatively archaic when it´s mentioned that it´s unlikely that users are willing to stream entire games on phones. Nowadays that´s not the case, therefore the reliability may be queried. Besides I want to question the persistently ball focusing technology of the prototype, since it seems like a lot of significant events where the ball is not involved during a game are missing, like bookings or offsides. 

I chose the study “Exploring the behavioral patterns of learners in an educational massively multiple online role-playing game (MMORPG)” from the journal “Computers & Education” with an impact factor of 2,621. The study is about an evaluation of the design of an educational MMORPG where the purpose is to understand the characteristics and the limitations of online games as instructional tools. 

1.      A MMORPG called Talking Island was used within the study. The purpose of the game is to teach English vocabulary and conversational skills to elementary school students. A lot of theories were used to utilize different learning strategies:

-  Situated learning theory: Gamers were provided with realistic settings and scenarios by means of daily problem-solving quests, for instance through shopping.

-  The mastery-learning model: Flash cards with voice-recognition were used in situated learning quests to practice pronunciation in a repetitive way.

-  Online problem-solving and collaborative learning: Allowed gamers to team-up within group quests and tradings.

2.      A long term observation empirical study was conducted during 335 days for gamers who played Talking Island and different behaviors were coded. Some quantitative analysis methods were used to explore gamer’s behavioral patterns as well.

- Frequency analysis: The frequencies of different categories like teaming-up, trading and chatting were measured to gather long term detailed distribution of behavior-codes. 
 - Lag sequential analysis: Common in digital learning and was used to find correlations between certain behaviors.
- Cluster-analysis: Utilized to compare different results between high participation gamers and low participation gamers.

                         
"Fig. 2. Behavior transition diagram for all the students (n = 100). (FT: Fight, PT: Pets, TL: Tools/Items, TK: Talk, GP: Group-work, TR: Trading, LF: Learning in fighting, LT: Learning in tasks, OT: Others)."

50 males and 50 females were randomly selected from 983 registered players which obviously make the study more reliable. However all the players had an Asian background so the selection was rather biased. In addition the methods lacked some analysis of cognition and social interaction since too much focus was put on behaviors.

3.      The results were comparatively easy to follow with detailed tables and well explained variables. I´m not sure if it´s common within design research or just extraordinary in this particular case, but generally speaking the results are quite often hard to fully understand in terms of limited explanations.