Using ARIS MashZone to analyse ARIS Community usage

I am a strong believer that software vendors should eat their own dog food. This also applies to IDS Scheer. Therefore, I already started some months ago to visualise the daily usage of ARIS Community using the now released ARIS MashZone product. Today, I want to share the experiences I gained while creating this community performance dashboard.

Let me start with some background information. If you are operating a website such as our corporate IDS Scheer homepage, you usually use some kind of software package to gather web traffic statistics. Well-known examples of such tools are Webalizer and Google Analytics. The tools answer how many people visit your site, what are the favourite landing pages, how many pages a visitor usually views on a visit, where people are coming from, which browsers they use, etc. This kind of information is also important for a community site such as ARIS Community, but it is only part of the whole story, because it can’t capture the level of interaction.

Communities are all about interaction and collaboration. People meet to discuss topics of their interest. They might share pieces of their work, such as ARIS Express models. You need to measure the number of interactions to get an idea if your community is an attractive social place. Typical web traffic measurement tools can’t give you this kind of information, because they usually can’t tell you how many comments and articles are posted each day. They don’t show you if your community is growing (content, members).

When we launched ARIS Community, I first installed a web traffic measurement tool. I knew that it was only a partial solution, but at least we got some nice charts right from the start. Also, my bosses were distracted enough by the nice visuals so that they were not asking for more ;-) Some months later just before the official release of ARIS Express, I also added a daily reporting functionality to our content management system (CMS). Since then, each day a file is generated containing various data points like total number of comments posted so far, number of community members, etc. Still, I had no way to visualise the data.

We are using an external provider for hosting and delivering the ARIS Express download. We also use this service for other larger downloads such as the ARIS Express video tutorials or presentations from our various events all around the globe. Again, this service is another useful source of information to measure the performance of ARIS Community, because we want to know how many people are watching our video tutorials or reading our presentations.

I think it gets clear that the whole situation is a total mess, because I ended up with at least 3 different sources of statistical data:

• ARIS Community content management system
• ARIS Express download server
• web traffic measurement tool

When the first wave of new ARIS Express users calmed down a little bit, I started looking for a way to combine those different sources of information. Lucky point in time: My employer IDS Scheer was just preparing for the release of their own dashboard solution – ARIS MashZone ;-)

ARIS MashZone is the perfect solution for such problems, because it aims at mixing (or “mashing”) together different data sources and creating an integrated personal dashboard. In addition, it allows me to share this dashboard (e.g. my boss) so that I get rid of the daily phone calls asking for the latest member count or download statistics.

I also wanted a solution, which updates automatically each day so that I don’t have to invest any time in updating the statistics, because that would probably never happen. I came up with the following solution architecture.

All 3 systems generate their data log files each day. In fact, the download server is continuously generating log files. Each night, a set of UNIX Bash scripts download those different log files and do some initial processing. For example, I’m not interested in download statistics of all files on our download server, but just on a selected set of files like our video tutorials. Therefore, I remove all “uninteresting” log entries. Also, I don’t care about the exact time a file is downloaded, but just on which day it occurred how often. Therefore, the scripts aggregate the log entries to daily stats. I could do that in the feed composer of ARIS MashZone, but it would require storing all log files where ARIS MashZone can access them. That is not really convenient for me, because the size of the initial log files is enormous. Just to give you an idea, there are more than 18,000 log files created by the download service of ARIS Express, not including any other downloads like our video tutorials. I created several data feeds in ARIS MashZone’s feed composer to read in the aggregated data files. It was very easy to create some neat visualisations. I grouped the different statistics on tabs in ARIS MashZone’s composer. For example, one tab is showing all statistics about ARIS Express and another one about ARIS Community members. I also added a time filter to the tabs so that the figures can be easily filtered by time periods (e.g. member count during first quarter of 2010). The best thing is that I’m able to allow others accessing my community mashapp. Now, if someone needs the latest figures, he can open his browser, navigate to our internal ARIS MashZone server and extract the data needed.

Unfortunately, I can’t share the mashapp with you nor post any detailed screenshots, because the usage data of ARIS Community is confidential. Still, there are some interesting lessons I learnt and which I can share with you.

• First, when you start a project, which you want to measure later using some kind of dashboard or business intelligence solution, make sure you generate and store daily data points right from the start. While developing ARIS Community, there was no time to come up with a good performance dashboard solution at the beginning, but at least we got the raw data stored each day. Later, it was not a big deal to convert the old statistic files.
• Second, you will need an automated solution. It is really great to start up your machine in the morning and see the performance statistics of the previous day. I would hate to manually update the statistics each day. Well, I think I wouldn’t do it at all :D
• Third, you can read everywhere that it is essential to measure what you are doing, otherwise you don’t know if you are moving in the right direction. This is not just true for community management, but probably for most projects. For example, you could use ARIS MashZone to visualise number of new tickets in your help desk system or how many features are already completed in the software development project you are coordinating. Also in process improvement projects, quantitative analysis comes in handy.
• Fourth, if possible do not rely on manual data entry, but generate the daily data points automatically. Of course I could look up the number of users each day and add it to a daily reporting sheet. But how do I make sure that this is also done while I’m on vacation? Instead, I use automated scripts to generate those data points automatically each day at a predefined time.
• Fifth, generate only those statistics you really need. Having available such extensive log files is very tempting, because you start generating statistics for everything. But, if nobody is asking for the statistics, there is no point in generating them. If you need those stats at a later point, you can still generate them, because you are not throwing away your log files.
• Sixth, don’t get addicted to statistics ;-) Of course I can now measure how ARIS Community is used on a daily basis, but many of those insights are also available by just looking at the page. For example, I see how often people comment and post and if they come back or just download ARIS Express. Statistics are nice, but usually they just confirm your gut feeling :-)
• Seventh, ARIS MashZone is just a tool, but you will still need your creativity and analytical thinking. You have to think about what KPIs you want to visualise and which charts might be useful. But if you are clear about those points, ARIS MashZone is a great help to get your statistic things done.

Besides those lessons learnt, there are also some interesting insights how you use ARIS Community. For example, I’m very happy to see that there was a significant slow-down of new members during Christmas and New Year. Of course, I’m not happy because fewer people signed-up, but it shows me that you all got families and didn’t had to sit alone in front of your computer during Christmas season ;-)

We also noticed that many members don’t care about privacy settings. For example, during registration we ask you what kind of newsletters you want to receive. Most people just tick all boxes and fewer remove all boxes. But the minority is really selecting what information they are interested in. In case you are not aware of those settings, please take a look at your user profile and edit it accordingly.

Finally, I can confirm an old online community rule: Many people are reading but only few are contributing. Several thousand people visit ARIS Community each day, but only a small percentage is commenting on posts or publishing their own ideas. I really hope that we can motivate you to contribute more, because in the end everyone will benefit from it!