We have now completed the first working prototype of DigiMina tool for assessing teachers’ digital competencies. This tool and initial set of assessment tasks were evaluated with a group of teachers. At the moment we have still covered only 20% of competencies with assessment tasks. Therefore we cannot open the system for a wider audience. This autumn we will continue with creating the assessment tasks and improving the usability of the system.

The online version of the article is available through SpringerLink:

Põldoja, H., Väljataga, T., Laanpere, M., & Tammets, K. (2012). Web-based self- and peer-assessment of teachers’ digital competencies. World Wide Web. doi:10.1007/s11280-012-0176-2

I am planning to include this article in my dissertation instead of an earlier paper that was presented at the ICWL 2011 conference.

I was presenting our work on assessing teachers’ educational technology competencies and the conceptual design of DigiMina environment. As a nice surprise, our paper received the best student paper award.

Põldoja, H., Väljataga, T., Tammets, K., & Laanpere, M. (2011). Web-based Self- and Peer-assessment of Teachers’ Educational Technology Competencies. In H. Leung, E. Popescu, Y. Cao, R. Lau, & W. Nejdl (eds.), Advances in Web-Based Learning – ICWL 2011: 10th International Conference, Hong Kong, China, December 2011. Proceedings (pp. 122–131). Berlin / Heidelberg: Springer.

The conference papers covered wide variety of topics. I would like to point out a few of the presentations from the sessions that I attended. Elvira Popescu was presenting eMUSE platform that aggregates learner activities from several social media media tools (Blogger, MediaWiki, Twitter, Delicious, YouTube, Picasa, SlideShare) (preprint). This is very close to our work in EduFeedr project. Adriana Berlanga was presenting a paper co-authored with Riina Vuorikari and a group of researchers in TellNet project (draft). They were studying the eTwinning community and presented some interesting statistics. It was nice to see that Estonia has the largest percentage of teachers involved in eTwinning activities (9,74%). It would be good to have a similar study about LeMill community, but unfortunately we have detailed data only about those users who have filled up their profiles. In the KMEL symposium my colleague Vladimir Tomberg presented the conceptual design of Timeliner tool that supports collaborative scientific writing (presentation). It is nice to see that they have made an in depth design process and we have something to offer also in the research 2.0 field. However, I still think that often the main problem in collaborative writing is lack of time, not lack of tools.

I would have expected a little bit more active use of the web in the conference. It seems that I was the only one using #icwl11 hashtag in Twitter and I have found only a few presentations from SlideShare so far.

I had also some time to look around, take photos and enjoy asian food. First photos are up in Flickr and more will follow during the Christmas break.

DigiMina

In DigiMina project we are developing a framework and a web-based tool for assessing teachers’ educational technology competencies. Our educational technology competency model for teachers is based on ISTE NETS for Teachers 2008 and consists of 19 competencies. Each competency can be measured on a 5-level scale. Teachers will be able to solve competency tests in DigiMina environment. These tests will contain self-tests items, peer-assessment tasks and self-reflection tasks. Finally teachers will be able to publish their competency profiles and compare their level with other teachers.

We are currently still in the design phase but first full paper about the project has been already accepted to ICWL 2011 conference and will be published in December:

Põldoja, H., Väljataga, T., Tammets, K., & Laanpere, M. (in press). Web-based Self- and Peer-assessment of Teachers’ Educational Technology Competencies. In H. Leung, E. Popescu, Y. Cao, R. Lau, & W. Nejdl (eds.), Proceedings of the 10th International Conference on Web-based Learning (ICWL 2011).

Dippler

In Dippler project we are developing a lightweight LMS with an open and distributed architecture. Learners will use WordPress blogs that have Dippler plugin. This plugin connects to a learning management system that is used to manage courses and learning activities. There are several aspects that make Dippler different from other learning management systems. One of these is blog-based architecture that supports open learning communities. Another aspect is special focus on learning outcomes. All the course assignments can be connected to learning outcomes. In the end of the course learners can compose e-portfolios where evidences are linked with learning outcomes.

We haven’t published anything about Dippler yet but in recent months we have made a fast progress with the software development. I hope that I will be able to use Dippler in my next course that is starting in February.

In both projects I have been also thinking about possible connections with my doctoral thesis. Educational technology competencies are not directly connected with open education but there are some aspects of openness that could be emphasized in the design of DigiMina (public competency profiles, etc.). In Dippler project my challenge is to find a compromise between open educational practices and institutional requirements that universities have for learning management systems.

Dippler and some other projects have taken all the time from our developers. This means that we had to postpone the next release of EduFeedr. Hopefully we can find some time for EduFeedr in the beginning of next year.

Väljataga, T., Põldoja, H., Laanpere, M. (2011). Open Online Courses: Responding to Design Challenges. In H. Ruokamo, M. Eriksson, L. Pekkala, & H. Vuojärvi (Eds.), Proceedings of the 4th International Network-Based Education 2011 Conference The Social Media in the Middle of Nowhere (pp. 68-75). Rovaniemi: University of Lapland. [Online]

One of the reasons for coming to NBE conference was to see the beautiful summer in Lapland. I came here with four of my colleagues and we had a nice car trip through the Finland. Salla is a little bit north from the arctic circle and it is currently the time of polar day. Unfortunately the weather is cloudy and we can’t see the sun during nighttime. Salla area has more reindeers than people. I have already managed to capture one of them on the photo.

My main aim was to present EduFeedr to Finnish audience. As more educators are trying to use blogs and social software instead of a learning management systems, tools like EduFeedr become handy. There have been 19 active courses and several more tryouts in EduFeedr since we launched the beta service last September. The best example in English is New interactive environments, but the largest course is actually in Spanish (Innovación abierta en la gestión de proyectos culturales with 61 participant blogs). In addition to English (2) and Spanish (2) we have courses in Estonian (11) and Finnish (4). Here are my abstract and presentation slides from ITK.

We planned to release a version 0.6 before the ITK conference but unfortunately we didn’t manage to do that. The main features that we have planned for EduFeedr 0.6 are described in EduFeedr blog.

On the second day of ITK we had a joint video conference session with Estonian e-learning conference that took place in Tartu. From the Estonian side my colleague Mart Laanpere was talking about the development of digital learning resources by vocational and higher education teachers (PDF). From the Finnish side Ville Venäläinen and Tarmo Toikkanen presented the Sometu network. Twitter hashtag #itkesitys178 was used for short comments from the participants in both sides.

Most inspiring presentation for me was the keynote speech by Pierre Dillenbourg. He presented several “modest computing” experiments where they used simple physical objects with simple sensors/displays in the learning setting (video). Although I am now able to follow presentations in Finnish quite well I was most of the time in the English session. There were two open education related presentations by Kati Clements and Anna-Kaarina Kairamo. An interesting discussion followed Jukka Huhtamäki’s presentation about the Knowledge Federation initiative.

Teemu Leinonen presented three design concepts that our research group in Media Lab Helsinki has developed in iTEC project. TeamUP, PLATES and ReFLEX that are designed for supporting study projects where students work in small groups. These applications help to divide students to small groups, follow their work, manage/share media and reflect on the learning experiences.

I hope that next year the Estonian and Finnish e-learning conferences don’t take place on the same days. Then there could be more visitors and presenters from the neighboring country in both conferences.

One of the keynote speakers Wim Westera talked about the next steps in OER strategies (presentation). On of the important issues that must be solved is the financing model. Westera argued that micro pricing model where people pay for views or downloads doesn’t work in education. Instead of that we should look at the cross-funding. He proposed 8 cross-funding schemes (see slide 30). In my opinion the most promising models are the freemium model and the rock concert model. We can see the freemium model working quite well in Flickr where the pro account costs $24.95 for a year. Some other services like SlideShare have failed to provide a reasonable price for ad-free accounts ($19 monthly). I wonder what could be additional pro services in LeMill that would attract active authors to pay a small fee that helps us to cover the hosting costs (detailed statistics about the use of their resources, management of exercise results, …).

Another keynote presenter Erik Duval was talking about removing friction in open education. His presentation made me think about large numbers. GLOBE has reached more 1 million learning objects. This is a great result, but what is behind that number? How many of these resources have a Creative Commons license that enables educators to improve and remix the resource? Are these resources somehow connected to the authors? In many repositories the resources are not published by the actual authors. I think that in case of user generated content it is important to maintain these connections. This is something that the OER community must work on. LeMill has 24 000 resources but unfortunately we have some technical problems with our OAI-PMH interface and these are not available through GLOBE yet. If we get this fixed then LeMill resources would make 2% of all OER’s available. Erik showed also an interesting comparison data from dataTEL initiative (see slide 50). We have to make LeMill data also available for dataTEL community.

From the conference presentations I really enjoyed Rory McGreal’s talk about approaches to OER course development (article). He listed 13 suggestions for developing OER course content. I find his suggestions very relevant for Estonian context. In recent years we have had several large project where lecturers develop online course modules. However, the quality of these modules varies a lot and often there are technical limitations for reuse and re-editing (source files not available, etc).

A lot of interesting things are going on in University of British Columbia. Brian Lamb, Scott Leslie and Novak Rogic presented how they use open platforms in UBC. One wiki installation is used for the whole university and integrated with university login system. This has allowed them to reach critical mass of people who collaborate online. They have even hired a wiki gardener to manage the site. Instead of developing their own platforms they rely on major open source platforms (MediaWiki and WordPress) and add/develop the needed plugins.

Another wiki-related presentation from UBC took place in Mozilla Drumbeat Festival. Jon Beasley-Murray talked about a course project where his students were writing Wikipedia articles. Three of the articles were finally featured on the Wikipedia main page. I should start improving the Estonian Wikipedia in my courses.

Brian Panulla and Megan Kohler presented an ontological model for exchange of assessment rubrics (article, presentation). I have experienced it in my own teaching that coming up with a detailed assessment criteria is more difficult than creating/finding/adapting the learning content. Many educators would benefit from a sharing platform of educational rubrics. Actually in Estonia we have a platform for creating and sharing rubrics for primary and secondary education. Estonian teachers have created more than 250 rubrics that are published under Creative Commons BY-NC-SA license.

Nathan Yergler presented a new search engine prototype that is developed by Creative Commons (article, slidecast). DiscoverEd search engine provides only open educational resources as results. Currently the results come from OER Commons, Connexions, Open Courseware Consortium and several other repositories. The number of indexed content cannot be compared with GLOBE but the results should include only OER’s. I hope that the user experience of this search engine will be in a same level with other Creative Commons tools when it will come out of the prototype phase.

Two interesting projects come also from the Open University UK. Anna De Liddo presented qualitative data analysis tool Cohere (article, presentation). Cohere is a web-based tool and it is specially designed to annotate and analyze web data (blog and forum posts, etc.). In the paper they analyzed one course that took place in P2PU. Cohere had very nice network and cluster visualizations of memos and participants. This work is very interesting for me because we have developed simple course visualizations in EduFeedr. This should be also interesting for my Finnish colleagues who are planning to do a visual knowledge building tool for the next version of FLE.

Another really promising project from OU UK was presented by Simon Buckingham Shum (article, presentation). SocialLearn is a learning community that is designed by the principles of Web 2.0 and social software. Every member has a dashboard with gadgets and a user profile. It is possible to form communities, make connections and follow other people in SocialLearn. The site will be launched later this year.

All together I am really happy that I had a chance to participate the Open Ed conference. There were quite a lot of interesting presentations and probably I missed some good presentations in parallel sessions. Unfortunately there were only five participants from Eastern Europe. If I look at the use of LeMill I think that open educational resources have a big potential in these countries. It would be good if the Open Ed conference would take place in Europe every second year.

Proceedings of the conference are available in UOC repository and can be downloaded also as a complete PDF book.

The second presentation was at the Open Ed 2010 conference. This was a more general presentation of EduFeedr and its use on open online courses. I will embed here the slides from this presentation. The paper will be hopefully also online soon.

Both presentations received feedback from the audience, especially the presentation at the Open Ed conference. I will summarize here answers to some of the questions.

To what extent are we going to extend EduFeedr? We have agreed on two limitations that won’t allow EduFeedr to grow into a bloatware LMS. At first only the facilitators have user accounts in EduFeedr. This means that we cannot authenticate the participants after they have enrolled to the course. Still, there will be some activities that they are able to do anonymously. The second limitation is that EduFeedr works without any plugins on the course blog or participant blogs. Most of the learners use hosted blogs that do not allow plugins. This eliminates more advanced assignment workflows. My colleagues are also working on a little bit similar project where they have developed a WordPress plugin LePress for giving and submitting assignments.

Why havent we used PubSubHuubbub or FeedBack? As far as I know, both technologies require plugins on blogs. Therefore we cannot use these because of the decided limitations.

Why do we follow main comments feed instead of separate comment feeds for each blog post that belongs to the course? This is a good question. By following separate feeds we would not aggregate comments that are not related with the course. The downside of this approach is the increasing number of feeds that we have to follow. In a typical course every participant will make 10…15 blog posts. A possible solution is to aggregate comments for older blog posts less often.

How well does EduFeedr scale for large courses? Our current courses have a moderate number of participants (max: 32, total: 152). We have also tested with data from a course with 70 blogs. I would say that current user interface solutions (for example progress diagram) would become a limiting factor sooner than the performance of the system. However, not all people who sign up for massive open online courses are using a blog. For example on PLENK 2010 there are about 1200 participants and 200+ blogs.

Is it possible to install your own instance of EduFeedr? Yes, of course. You can download the source code from the development site. However, the installation procedure is not really straightforward. In addition to EduFeedr you have to set up EduSuckr web service that is actually aggregating the blogs. We would be very happy, if you would use EduFeedr on edufeedr.net. Then we have a better overview of ongoing courses and can identify problems more easily.

EduFeedr could aggregate information about open courses from other EduFeedr instances. This is a great idea! Thanks! However, I’m afraid that it will take time until we get that far.

Why we don’t host EduFeedr development site on some open community such as GitHub? This is a good suggestion. Currently we are using our own Trac and SVN that are connected with user accounts in our department. Therefore it is not easy to involve other interested people to the project. I will discuss this with our developers.

Why are we planning to evaluate the usability with think aloud testing? I do not have a comprehensive experience in think aloud testing but in few cases this method has worked well for us. We give a rather open task for the test users and use Silverback to record the sessions.

We should try to differentiate between the usability and usefulness when doing the evaluation. This is a good comment. I will take it into account.

On the development side it has been quiet in recent weeks. We released a bug fix release two weeks ago and now the developers are busy with another project. I hope to release EduFeedr 0.6 before the Christmas. This release should have a new method for connecting blog posts with the assignments and comments with the authors. Also we are planning to have participant profiles that contain all the blog posts and comments written by the person.

I received quite useful feedback for my work in the seminar. I had a feeling that my plan about combining the articles/projects together as the “open education ecosystem” made sense for most of the people.

Some suggestions:

• “Designing the Open Education Ecosystem” is too ambitious title. I have designed only some tools to the ecosystem. Also, digital ecosystems have a certain degree of self-organization and cannot be completely designed. Therefore “Desining for the Open Education Ecosystem” would be more suitable title.
• The 6th article should be written as the introduction chapter of the dissertation. Five articles would be enough for the dissertation. Later the introduction chapter could be also summarized as an article.
• The main research question “What are the pedagogical, technical and social considerations for designing educational tools for the open education ecosystem?” is too wide. However, finding the balance between the pedagogical, technical and social constraints seems to be the challenge that I have had in each project.
• I received a good hint that I can use “layers” in my digital ecosystem terminology. Etienne Wenger at al have written a book “Digital Habitats” where they have some ecosystem diagrams with layers. My colleagues in Tallinn have been critical about using layers, because it is not part of the natural ecosystems. I am not sure how far I should go with using the natural ecosystem analogy.
• I should look at the ‘practice-led research’ method to strengthen the design methodology part.
• People are missing from my layer images. This is something that must be there when the the three layers are merged into one diagram.
• Some people were not sure if “social” is the best name for the third layer. At the moment most of the items in that layer are connected to openness. There was a joke to call it the “leftover layer”.

My colleage Teemu Leinonen has proposed a research-based design methodology (only presentation, full article is not available online) where software prototype is seen as hypothesis. In the beginning of the design process the hypothesis is quite fuzzy and it will become more clear during the stages of the design process. For example in LeMill project our hypothesis was that “an online service with learning resources that can be edited and improved by others with tools for social networking and matching of interests among the participating teachers could enhance sharing and collaboration around learning materials“. I have an hypothesis for each case but the main hypothesis can be formulated when my understanding about the open education ecosystem is more clear.

Later we discussed with Teemu about the target audience of my dissertation. I think that the main target will be the open education community. I have learned a lot from the design community but my dissertation will not contribute back so much to the design community. My second target group would be the technology-enhanced learning community, especially now when they are starting to see the importance of design and openness.

Designing the Open Education Ecosystem

I am planning to do an article based dissertation that contains six papers. Four papers will focus on the projects and concepts that I have developed during my doctoral research. Last two papers will focus on the idea of learning environment as a digital ecosystem. The projects discussed in the dissertation have been developed together with my colleagues from the Learning Environments research group at the Aalto University Media Lab and Centre for Educational Technology at the Tallinn University. However, I have had a leading role in the design process.

Article 1: Progressive Inquiry Learning Object Templates

In most cases digital learning resources are used for individual learning (reading, looking, playing, quizzes) or by teachers in their class-room or online teaching (presentations). In PILOT project we are arguing that learning resources should be designed and presented in a special way in order to promote truly social constructivist learning. The project is based on the concept of progressive inquiry learning object templates (PILOT’s). These learning objects support progressive inquiry knowledge building process in computer and database supported Knowledge Building environments, found for instance in Fle3 virtual learning environment. Design research methods such as participatory design and scenario-based design are used in the project to generate distributable and reusable PILOT’s.

PILOT is the first concept that I developed when I started my doctoral studies. This was still before the rise of Web 2.0 and at time I was not thinking about learning environments as digital ecosystems. However, now I see PILOT learning resources and Fle3 learning environment as part of the ecosystem.

An article about the concept is published as a journal paper:

Põldoja, H., Leinonen, T., Väljataga, T., Ellonen, A., Priha, M. (2006). Progressive Inquiry Learning Object Templates (PILOT). International Journal on E-Learning. 5 (1), 103-111. Chesapeake, VA: AACE. [Online]

Article 2: LeMill

My most important project is the open educational resources authoring community LeMill, which has over 14 500 members and over 13 000 reusable learning resources, all created by the community members. The design of LeMill has tackled numerous challenges that hinder the authoring and sharing of educational resources by communities of teachers. The information architecture of LeMill scaffolds authors toward collaboration and sharing. The licensing scheme encourages reusing and remixing of educational content. LeMill is also a multicultural and multilingual community with teachers from more than 60 countries and learning resources in more than 40 languages.

Ecosystem idea is strongly present in the design of LeMill. Learning resources can be created by reusing and remixing content from other Web 2.0 environments. PILOT concept presented in article 1 is developed as one learning resource template in LeMill environment.

In LeMill project I was responsible for the conceptual design and interaction design. An article about our design solutions behind LeMill is published as a journal paper:

Leinonen, T., Purma, J., Põldoja, H., Toikkanen, T. (2010). Information Architecture and Design Solutions Scaffolding Authoring of Open Educational Resources. IEEE Transactions on Learning Technologies. 3 (2), 116-128. IEEE Computer Society. [Online]

Article 3: EduFeedr

In order to promote LeMill environment and authoring of open educational resources we have organized several open online courses for teachers. In these courses every participant creates their personal learning environment. Typically learners use a blog to reflect on the course assignments and various Web 2.0 environments to create content and to communicate with co-learners. One of the challenges in this kind of courses is to follow all the learning activities that take place in blogs, wikis and various Web 2.0 environments.

EduFeedr is an educationally enhanced online feed reader that is specifically designed for following open online courses that take place in blogs. Currently I have published workshop papers about the conceptual design (PDF) and technical implementation (PDF) of EduFeedr. The public beta of EduFeedr was launced in September 2010 and currently we are evaluating EduFeedr in several courses.

For the dissertation I am planning to write a paper that contains also the results from the evaluation of EduFeedr.

Article 4: LeContract

Another issue in open online courses is related to learner motivation. A lot of people who start open courses do not complete the course. Setting up personal learning objectives and strategy to reach the goals could keep learners on track. One of the ways to achieve this is to follow a personal learning contract procedure. In our current courses the students are using generic tools such as blogs to write learning contracts.

We propose that a special tool would scaffold and support the personal learning contract procedure. LeContract provides structural templates that define important parts of the learning contract. During the learning project the contracts can be reviewed and the achievement of individual learning objectives can be evaluated. Learning contracts created in LeContract could be embedded to learner blogs.

LeContract is currently in the phase of conceptual design. This work has been presented as a conference paper:

Põldoja, H., Väljataga, T. (2010). Externalization of a PLE: Conceptual Design of LeContract. In: The PLE 2010 Conference Proceedings. [PDF]

More information about LeContract concept can be found in the LeContract blog.

Article 5

The fifth article will analyzes multiple cases where teachers use LeMill together with other Web 2.0 tools to create and remix open educational resources. In this paper I’m planning to introduce the concept of digital ecosystems and describe various patterns that occur in the learning resource collections created by the teachers. The digital ecosystems approach enables teachers to become designers who create mashups with various online tools.

Article 6: The Open Education Ecosystem

The final paper will present a theoretical framework of the open education ecosystem. In that paper I am planning to draw connections between ecosystems in biology, open systems and digital ecosystems. In the seminar I will explain my current understanding about the three layers of the open education ecosystem and how the tools developed in my research fit into this ecosystem.

In my dissertation I will argue that the design of learning environments would benefit from the digital ecosystems approach. In this kind of open and personal learning environment every teacher and learner takes the role of a designer.

(Sorry about hyperlinks instead of proper references. This is a blog post.)

• New interactive environments (18 participants)
• Õpikeskkonnad ja õpivõrgustikud (23 participants)
• Intelligentne arvutikasutus (31 participants)
• E-portfoolio professionaalse arengu toena

The basic functionality of setting up a course and enrolling to the course is working fine. The facilitator must copy the blogroll code manually to the text widget in the course blog. In my courses I have updated the blogroll once a day if there are some new participants.

Since the participants do not have user accounts in EduFeedr they cannot change their details after enrolling to the course. Our reason behind this decision was to keep the system simple by having less users with accounts. We assumed that the minimal data that is asked during the enrollment (name, e-mail, blog address) does not change during the course. However, we have noticed that in few cases people have enrolled to the course twice to change their blog address. In that case the facilitator should remove one of the entries.

First weeks of testing have pointed out one important limitation. Currently it is not possible to aggregate blog posts only from a certain category (WordPress) or label (Blogger). Our first blog-based courses were inservice teacher trainings. For most of the participants in these courses created a new blog. However, with master students it is common that they will use one blog as their learning diary and have a separate category for each course. We hope to remove this limitation in version 0.6. So far the facilitator can hide posts from other courses manually.

Course feed page is quite useful for following all the blogs and discussions. However, for large courses it is not enough to display only 10 recent blog posts and comments. The facilitator should have an option to specify the number of blog posts and comments that are displayed. In the current version all pingbacks are also displayed in the comments column. In some of the courses there are actually more pingbacks than real comments, because we ask our participants to link to the weekly assignment in the course blog when they are posting their response. The facilitator can hide these pingbacks manually.

It is still to early to demonstrate good examples of progress chart and social network diagram, but the testing has indicated a few defects and limitations with these two visualizations.

This coming week we will be presenting EduFeedr in the MUPPLE workshop at the EC-TEL 2010 conference. After that we are planning to release version 0.5.1 with some fixes to small defects and user interface design issues. The next major release 0.6 requires more work and we are not sure when it will be ready. If everything goes fine we might release it before the Open Education Conference.