Analysing My Twitter Network

I have been on twitter for more then 10 years, but normally use it as a source of information, just reading tweets rather than for conversations. As part of @hjarche's PKM workshop  I thought it would be interesting to look at the connections between those that I follow on twitter and see what networks exist.

There seem to be many tools for analysing twitter followers demographics  and for analysing tweets but not so many for looking at networks of those you follow (at least that I could find with a Google search).

However, I found a series of Python scripts that extracts connections between followers of followees. This process is very slow because twitter rate-limits the access to its API to 15 calls in 15 minutes. So extracting the information for the 600 people I follow took 10 hours.   It also requires access tokens which means signing up for a Twitter developer account.

The output from the scripts is a CSV file suitable for use with Gephi. I had not come across Gephi before, so this process also introduced me to a new graph analysis tool which might help with another of my problems, that of looking at the dependencies between a lot of software modules in a large program.

The process with Gephi is to load the graph file from the Python scripts and then runs some analysis and use the automatic layout algorithms to layout the graph. I found a tutorial that explains the basic details.


This produced a graph for those I follow on twitter like this:

The nodes on the graph are the people I follow on Twitter. The edges in the graph are the followship links between these people. The size of the nodes depends on how many connections there are to that node.

The "modularity analysis" found 6 "communities" based on the connection information, this is represented by the different colours. However looking at the graph, 8 different main groups seem to exist.

1. This is the main group I follow mostly software development related, especially agile software.
2. This is also technical but seems to form a different group from 1.
3. This group consists of twitter personalities, or more business related topics.
4. These are Japanese software developers.
5. This small group is a technical niche for 3D CAD
6. This group is related to Japan so is mostly separate from other groups
7. Photography related
8. Japanese Jazz artists

The lack of connection between groups 4 and 8 and the rest does not surprise me due to the language difference. These groups tweet in Japanese whereas the rest are English. Even though the Japanese developers have a strong interest in "agile software development" there are few links to the non-Japanese developers I follow:

It is clear that the main group I follow is the "software development" group. I it is interesting to me that the analysis algorithm split this into two groups.

Since this exercise was prompted by the PKM workshop I looked at the people that @hjarche and myself both follow. I was surprised that there are only a few that be both follow:

I am not sure where to go from here. My list of follows needs some pruning. Also, some areas need expanding.

This analysis just looks at connections between people, not at how often they tweet. The timeline tends to be dominated by a small number of prolific tweeters / re-tweeters so some of the conversations in the main groups probably get drowned out.

Links

• PKM Workshop: https://jarche.com/pkm/pkm-workshop/
• Gephi: https://gephi.org/
• Egonet Scripts: https://github.com/alumbreras/twitter-followers-graph

Note, the README description for the scripts is out of date.
The name of the script has changed and the commands are now:

 python egonet.py -u username -f followees
 python egonet.py -u username -f followees_graph


Using CppUnit with QtCreator.

We have been using CppUnit for many years and have a large set of tests in CppUnit and so wanted to continue using the CppUnit framework while using QtCreator as the IDE. QtCreator has a plugin for using the Qt test framework but changing test framework would be a lot of work.

The basic goals were

• automatically run tests when the code is built
• Have test failures show up in the Issues window of QtCreator 
• Easily add new tests

Running the tests

This example is based on the Money example code from CppUnit. The MoneyTest file was renamed MoneyTestCase to match the conventions we use.

To use QtCreator with the money example, the first task was to create a simple qmake .pro file for the project

 TEMPLATE = app  
 TARGET = cppunit-test  
 INCLUDEPATH += .  
 HEADERS += Money.h \  
   MoneyTestCase.h  
 SOURCES += MoneyApp.cpp \  
   MoneyTestCase.cpp  
 LIBS += -lcppunit  

This compiles and the tests can be run:

Starting /home/dibben/Develop/QtCreator/cppunit-test/cppunit-test...
....
OK (4)

But we want to run the tests automatically when they are built. So we add a custom process step.

Now when we type "Ctrl+B" to build the project the tests are automatically run.

Getting the test failures

When using the default outputter, if a test fails, we get a message such as:

..F..
MoneyTestCase.cpp:52:Assertion
Test name: MoneyTestCase::testEqual
assertion failed
- Expression: money123USD != money123FF
Failures !!!
Run: 4   Failure total: 1   Failures: 1   Errors: 0

But we want this to appear in the Issues list of QtCreator with a link to the test that failed.

To do this we need to change the output format of CppUnit. This requires changing the MoneyApp.cpp code to use the basic test runner and adding a new outputter. The main function now looks like this:

  CPPUNIT_NS::TestResult controller;  
  // Add a listener that colllects test result  
  CPPUNIT_NS::TestResultCollector result;  
  controller.addListener( &result );  
  // Add a listener that print dots as test run.  
  CPPUNIT_NS::BriefTestProgressListener progress;  
  controller.addListener( &progress );  
  // Add the top suite to the test runner  
  CPPUNIT_NS::TestRunner runner;  
  runner.run( controller );  
  runner.addTest( suite );  
  // Run the test.  
  runner.run(controller);  
  bool wasSucessful = result.wasSuccessful();  
  // Print test in a compiler compatible format.  
  #ifndef _WIN32  
      GccOutputter outputter( &result, std::cerr );  
  #else  
      MSVCOutputter outputter( &result, std::cout );  
  #endif  
  outputter.write();  

This will collect the results in the result object and then print them at the end of the run using either the GccOutputter or MSVCOutputter. These are modified versions of the Compiler outputter that produce output that QtCreator can interpret. Building this will now produce the following output in the Compile window

 18:57:46: Starting: "/home/dibben/Develop/QtCreator/cppunit-test/cppunit-test"  
 MoneyTestCase::testConstructor : OK  
 MoneyTestCase::testEqual : assertion  
 MoneyTestCase::testAdd : OK  
 MoneyTestCase::testAddThrow : OK  
 MoneyTestCase.cpp:52: error: - Expression: money123USD != money123FF  
 Failures !!!  
 Run: 4  Failure total: 1  Failures: 1  Errors: 0  
 18:57:46: The process "/home/dibben/Develop/QtCreator/cppunit-test/cppunit-test" exited with code 1.  

and QtCreator will pick up the error and display it in the Issues output pane:

So now when the code is built the tests will be run automatically and any errors will show up in the Issues pane linked to the source line for the failed test.

The code GccOutputter and MSVCOutputter is available on Github here.

Adding a new test

The money example uses a separate header and source file for the test cases. This means that to add a new test we need to do 3 things

1. Add a declaration in the header
2. Add a definition in the source file
3. Add the test to the test suite using the CPPUNIT_TEST macro.

Some of the newer frameworks make this easier but we are using CppUnit. To make this simple in QtCreator I created a plugin that adds some support functions for Cppunit. The plugin is available on Github

With the plugin creating a new test is as simple as Ctrl+Shift+T, then a dialog appears for the name of the new test. Multiple tests can be entered separated by spaces. On clicking OK this does the 3 steps above to add the test to both the header and the source file.

The plugin has some other functions as well

• Create new tests
• Create a new test case class (custom class wizard)
• Switch between source and test case

References

• CppUnit Outputter classes on Github: cppunit-utilities
• QtCreator plugin for Cppunit test creation on Github: qt-creator-cppunit-plugin
• Source code for the example: cppunit-test.zip

Evaluation based on results is not the answer to long working hours

Shinzo Abe, Japan’s Prime Minister, has announced that Japan needs a more flexible working style without the ridiculously long hours that prevent many women from working. This is a very laudable goal and something that Japan will need in the face of a shrinking population and many in the younger generation that seem reluctant to devote their lives to the company. (Typically, the photo accompanying the article about improving working conditions for women shows a committee that consists almost entirely of old men.) In fact, changes are already happening, albeit very slowly. When I started working in Japan 14 years ago many colleagues came into work on Saturdays and nearly all stayed very late. Many still stay fairly late but the company has prohibited Saturday working and is putting limits on overtime.

What really worried me in the Yomiuri article though was the quote:

“I would like you to investigate a framework for a new working-hours system that befits work styles evaluated based on results, not time,” Abe said at the meeting.

This sounds like Abe wants to replace one dysfunctional system with a different but equally dysfunctional one. A shift from the bureaucratic nightmare described by Northcote C. Parkinson to one based on industrial age thinking.

The Yomiuri article hints at the problem by describing the “entrenched system of long working hours suppressing productivity” and suggests limiting working hours as the solution. However, the problem is in the working culture that equates dedication to hours worked and working harder as the means to achieve more. Changing this will be harder than just limiting overtime. Which is probably where the focus on results comes from, companies worried about their revenue probably want that to ensure nothing actually changes.

The suggestion in the article is that employees could work whatever hours they like so long as they achive the results. This sounds remarkably like management by objectives which is strange from a country that so admired W. Edwards Demming who had “Eliminate managemement by objective” as one of his 14 points for running a company. The problems with this approach are legion. It discourages teamwork and it puts the focus on the individual instead of the system as the source of productivity. It also assumes that it is actually meaningful to measure “results” or productivity at an individual level.

To improve productivity it is necessary focus on the system. Ironically it is Japanese companies such as Toyota that have shown the world what can be achieved by doing this. Yet Abe’s statement demonstrates that this is an exception, even in Japan rather than the norm. I see this problem daily at my own workplace. Long working hours are considered normal, this gives no incentive to actually improve the working practice since you are going to be in the office anyway. And even when there is a desire to improve the system everyone is so busy dealing with what John Seddon calls ‘failure demand’ that there is no time to introduce improvements.

From the perspective of software development this means that developers are so busy debugging the code that they have no time to consider how those bugs got there in the first place and what changes they could make to prevent them. “We don’t have time to do unit testing or code review”. It is very hard for them to change the system, so simultaneously making them responsible for their results while limiting the only thing that they control, their working hours, is both demotivating and unfair.

Update: A nice article from Vangard Consulting puts the case against "management by results" in the health care sector very clearly:

Wherever management by results exists so too does the failure to accurately understand and sustainably solve systemic problems. This is in the nature of management by results because:

1. It wrongly assumes that performance can be adequately described by a system of measurement; and
2. It holds people and organisations to account for the fictions that this creates.

Another book from Packt

Packt asked me to review another book for them; this time on Review Board.  "Getting Started With Review Board" by Sandeep Rawat. Maybe they didn't actually read my previous review of the QtCreator book. After I replied that I would be happy to write an actual review, that would probably be bad,  rather than just an endorsement I didn't hear back from them

Review Board is a great tool. I have been using at work now for several years. It is a free open source tool for code review. In the very early days the setup required installing several packages but for several versions now the setup/upgrade process has been made trivially easy. It uses the python easy_install tools so normally the install, including all the pre-requisites can be done with a single command. It also has great online help and installation instructions. It is also very easy to use. I find that new users in my team just need a 5 minute explanation of the main features and they are good to go.

Which brings me to the Pack book. It is described on their web page as

This book is a concise, to-the-point guide with a practical walkthrough of the code review workflow using the features present in Reviewboard. 

Based on the table of contents and the sample chapter what this means is that the book is simply a very short description of what reviewboard is. The sample chapter on "Reviewing Code Review Requests" was simply a description of the Review Board features that are not only easy to figure out for yourself they are also described fully in the free help with the program. The book will also be out of date very soon since Review Board 2.0 is about to be released.

So I don't really understand the point of this book. Why would anyone want to spend $12  for the eBook or $22 for a paper book for something that is not really needed and available for free online.

What would actually be useful is a book on how to do code reviews. Karl Wieigers "Peer Reviews in Software: A Practical Guide" is one of the best books on the subject that I have found. Smart Bear also have a free eBook on code review available (requires registration) which while obviously meant to advertise their code review tool does discuss code review in general. However, there are not so many books that cover code reviews.

The publisher of the book is often a good indicator of the quality of the book. Some publishers, like The Pragmatic Programmers are always reliable.  I think I will be avoiding Packt books in the future.

Application Development with Qt-Creator

Packt sent me a copy of their new book on using Qt-Creator "Application Development with Qt-Creator"  by Ray Rischpater.  It is very clearly targeted at beginners and anyone with experience of other IDEs would probably find it too light. However, if you are completely new to programming it covers some of the basics of using Qt-Creator, but not really enough to create an actual application.

This is not a book for learning C++ or Qt. The code samples are very simple. There is section on how to use Qt's signals and slots but many of the topics needed for non-trivial applications are not even mentioned (for example the more advanced widgets such as lists, tables, combo-boxes or event handling.) Fortunately, Qt has comprehensive documentation that covers most of this.

The biggest problem that I had with this book was what was missing. Many of the functions that I use the most while programming were not mentioned at all. One of Qt-Creator's strengths is the C++ code model that allows fast navigation. You can find all the places a function is used (Ctrl+U), follow a symbol to where it is declared/defined (F2) and switch between header/source files (F4). This makes navigation in a large code base fast and easy. But these functions did not seem to be covered. Also, the following basic topics are not covered:

•  Search and replace
•  The different views in the left hand pane (class view, open documents etc)
•  Splitting the main editor window to view two files at the same time
•  Editor functions such as commenting out a block of code
•  Shortcut keys
•  Refactoring functions for renaming symbols / extracting functions
•  Code snippets

For anyone familiar with an IDE these are all straight forward and easy to use. However, if you are new to programming (and are therefore in the target audience for this book) then you might want a bit more explanation and description of the available functions.

The really big omission is a description of the location function (Ctrl+K). This function is central to using Qt-Creator and, for me, what makes Qt-Creator stand out as an IDE. The location field in the bottom left of the screen has many functions. Just typing a filename allows quick opening of any file in the project. It also gives access to the help system, the source control functions, lists of classes and member functions.  Since this is something that sets Qt-Creator apart it almost deserves a whole chapter by itself, but it is not even mentioned.

The lack of coverage of major functions actually suggests that the author has not actually used Qt-Creator for any serious development and has not used Qt significantly either. Slots can be private (or protected) contrary to what is claimed in the book.

The overall impression is one of a professional writer who has not actually used Qt-Creator for any real development. So the writing is clear and easy to follow but the coverage of the topic is very superficial.

If you just want to quickly learn how to setup Qt-Creator, for example to complete a class assignment, then this book covers what you need.  For professional developers on larger projects this book is missing coverage of too many functions.

Installing ReviewBoard on RHEL-5

ReviewBoard is a great free code review tool which we have been using for several years now. We run it on a Redhat server and it is connected to our Subversion repository. With the 1.7 release of ReviewBoard support for Python 2.4 was dropped and unfortunately that is the standard Python on RHEL-5. So I have been putting off upgrading for some time.

Finally decided to upgrade and switch from using the SQLite database to MySQL. Since we have a small team we have not had any performance problems with SQLite, except where the source code in a review is split across multiple pages when changing the page with an open review causes the DB to get locked and requiring a restart of Apache. The ReviewBoard docs do warn about using SQLite but we have been using ReviewBoard since before version 1 when the install process and DB support was not nearly as good as it is now.

So first task was to get a newer Python. Fortunately this turned out to be much easier than expected since the Extra Packages for Enterprise Linux (EPEL) contain the rpms for Python 2.6 so it was as easy as

    yum install python26

Now installing ReviewBoard should be extremely easy. On my Fedora box at home, I could just type

    easy_install ReviewBoard

and ReviewBoard and all its dependencies were installed. Trivially easy. Unfortunately the server is behind a proxy server. Setting the http_proxy environment variable and the values in yum.comf were enough to get yum to work, but not easy_install. That kept on giving errors.

After several frustrating hours trying to different proxy server settings I eventually stumbled on the solution: Our proxy server seems to require that https_proxy is set in addition to  http_proxy. So setting both:

   export http_proxy=http://192.168.1.1:80
   export https_proxy=http://192.168.1.1:80

and it installed!  The Google search for easy_install proxy problems never mentioned this. So I don't know whether this is an easy_install problem or our peculiar proxy server.

Of course, it still wasn't that easy. While setting up the site MySQL was not available because the install of the pymysql had failed because the MySQL devel package was not installed and there was no error message visible during the easy_install.

However, finally it works. Unfortunately, there is no easy way to migrate the old SQLite data to MySQL.

Developer Summit 2012 in Kansai

The DeveloperSummit was a free one day conference (in Japanese) for developers held in Kobe, Japan.  It is nice to see more of these events taking place in Kansai rather than just in Tokyo. This event was very popular with several of the sessions being completely full.

The keynote presentation was from Oikawa Takuya, one of the Japanese developers on the Google Chrome team. The main theme was the changes in project management style required when developing for “the cloud.” Out is the old style product lifecycle with releases every few years and in is the “versionless” development with very frequent or “continuous” releases.  Also, the use of open source and the challenges that brings to development when contributions from many people must be integrated. This was an interesting presentation giving some insight into the how Google develops some of its software and Chrome in particular.

The first presentation on the “Agile” track was from a Microsoft evangelist entitled “Continuous value delivery to the next decade.”  This presentation also discussed the differences between waterfall and agile development and how development is done at Microsoft. However, the whole presentation seemed like an advert for TFS. The main point seemed to be the advantages of using a single tool for project management, bug tracking version control etc rather than several different tools. And Microsoft just happens to have such a tool. This seemed to me like agile for those who want to say they are doing agile but don’t want to change too much. The process seemed to be more like a series of mini-waterfalls backed up by comprehensive tools.

Following the Google and Microsoft promotions we had an advertising spot for JIRA (the bug tracking system from Atlassian) given by Suzuki Yusuke, an engineer from Growth xPartners which seems to be an Atlassian partner in Japan. This was less a less subtle promotion since the presenter even included a price list for JIRA in one of the slides. After yet another description of waterfall and agile processes (the third so far) the presenter stressed the importance of collaboration and communication in Agile development.  I would not argue with that, but I am sure that there is more to communication than a bug tracking system. The middle part of the presentation described how “Agile is implemented in the bug tracking system.” The idea seems to be to use JIRA to hold the list of tasks to be done and to act as the communication medium for the team.

We use JIRA as a bug tracking system and it seems to be one of the best bug trackers available. However, I often see it as blocking rather than enabling communication. Instead of direct communication between individuals, communication is via comments on an issue. This leads to misunderstandings and delays. Also because it is so easy to create an issue the “backlog” quickly becomes an unmanageable black hole into which requests and improvement suggestions are sent to be forgotten.

I suppose that not surprisingly, from a company selling the bug tracker there was no suggestion that the use of a bug tracker is itself actually a symptom of a bigger problem.  A truly agile team would probably have no use for a bug tracker.

The presentation that I was most looking forward to was that by Wachi Yukei on “The evolution of test driven development.” He is one of the translators of “GrowingObject Oriented Software Guided by Tests”, the Japanese version of which was just released on the same day as the developer summit. This presentation gave an overview of GOOS. The description of Mocks was fairly hard to follow and my Japanese colleagues who were not familiar with mocks did not seem to understand it at all.  Actually the biggest problem seemed to be the direct transliteration of English expressions into Japanese, such as “walking skeleton.” This is not a problem for me as an English speaker – in fact it makes it easier – but makes it hard for non-English speakers to understand. However, the presentation did include a lot of good stuff from GOOS such as the need for end-to-end tests and thin slices of functionality. I am not quite sure on the relevance of the photo of David Cameron.

Many of the presentations are on SlideShare (in Japanese)

Continuous value delivery to the next decade:

http://www.slideshare.net/tomohn/a1-continuous-value-delivery-to-the-next-decade

Collaboration in enterprise development – how to connect the development team with customers using JIRA

http://www.slideshare.net/yusuke/2012a2

The evolution of test driven development

http://www.slideshare.net/digitalsoul0124/ss-14351848

Visual Studio 2012 breaks VS2010

StackOverflow just rescued me again from yet another Microsoft problem. It seems if you try to install Visual Studio 2012 when you already have a Visual Studio 2010 installation then the 2010 installation gets broken.

Fortunately MS have released a SP to 2010 which fixes the problem. But it should not happen in the first place. Unless you do everything on a virtual machine when you try a new program like VS2012 you will want to keep your existing environment working. Did they not test installing VS2012 onto a machine with VS2010? I can't believe that they did not, which means that they must not care about breaking existing environments.  There was no warning message that it would happen and the Update for 2010 was only available after installing 2012.

The knowledge jigsaw

Most of my colleagues don’t  read books.  They are software developers in desparate need of improving their skills yet they try to get by with muddling through with what they already know or relying on the boss to provide the information they need. This has puzzled and frustrated me for some time. If they were using some other means to learn – blogs, personal contacts, seminars then I could understand but they don’t do that either. Yet when I talk to them they all profess to wanting to improve their skills and knowledge.

One common aspect that came up in conversations is that they seem to expect a book to be both directly relevant to them and to answer whatever specific problem they are having at the moment.  Of course, this is rarely the case. When they read one book, do not understand the context and can’t apply whatever the book describes directly they dismiss it as not relevant and a waste of time. This then puts them off technical books in general.

To try and explain why they should read I now use the metaphor of a jigsaw puzzle. Reading an article or book is like picking up a single piece of the puzzle. Sometimes you  can see something recognizable on the piece and see how it fits. More often the picture on the piece only makes sense when it is seen in the context of the surrounding pieces.

To understand most technical books you need to understand the context. You get the context through experience and through reading widely. 

Windows SDK Woes

Finally got a Windows 7 PC at work. Installing the Microsoft Platform SDK kept on failing with a not too helpful error message.


Installation of the "Microsoft Windows SDK for Windows 7" product has reported the following error: Please refer to Samples\Setup\HTML\ConfigDetails.htm document for further information.

The log file is not too helpful as it contains many errors, but most of them seem to be spurious.

It seems that if a more recent version of the "Visual C++ 2010 x86 Redistributable" (vcredist_x64.exe) is installed then the SDK install fails. The solution is to first unistall the redistributable package then install the SDK.

Since the redistributables are updated with windows update I expect that this is a common problem. MS could at least put a note on the download page for the SDK about this.

Keyboard Cleaning

I really like my Kinesis Ergo Elan keyboard which I bought about 12 years ago. It is old enough to have a 5-pin DIN connector with PS-2 adapter, which is now plugged into a PS-2 to USB  converter. I use it at work, so it has had more than 10 years of continuous use without any real problem. However, it was starting to look a bit grotty and the escape key was a bit sticky. Then the other day some keys stopped working so I thought it was time for a good clean.

The only way to get all the muck out was a complete disassembly. Fortunately, the Kinesis keys come off fairly easily. The cause of the sticky escape key was very obvious once it was dissasembled - some sticky mess all around the contacts.  Most of the keyboard is screwed together but the function keys were held in place by plastic lugs which had been melted over the backing plate. Not a good design for easy disassembly. So it was necessary to cut of the top of the lug to take the function key assembly apart and then glue it back together.

The keys before cleaning

Something disgusting on the function keys 

Once all the keys were removed, they could be cleaned and all the grot removed from underneath them.

Key switches with keys removed. 

The keyboard cleaned up pretty nicely. The plastic is slightly yellowed and the legend on the "enter" key is slightly faded, but considering the use that this keyboard has had it is in remarkably good repair. I have just ordered the newer model (the Kinesis Advantage) for use at home, to replace the Microsoft Natural keyboard that I have been using.

The cleaned keyboard

Agile Japan 2011

Despite the fact that the study of Japanese companies by Nonaka and Takeuchi in their seminal paper “The new new product development game”  is credited with giving the name to Scrum  I don’t see a widespread adoption of agile methods for software development in Japan. Certainly at the large IT company where I work, process heavy waterfall style development seems to be the norm. A couple of years ago, the “quality division” tried to introduce a waterfall process in order to unify the process across all projects in the company to improve “quality.” This effort demonstrated everything that was wrong about imposing heavy defined processes and fortunately our division successfully opted out.



Introducing change anywhere is hard and I struggle to improve our development process. Two of the problems that I face are that agile methods do not seem well known, at least among my colleagues, and a belief that such techniques may work in America but would not work in Japan. This made me especially happy to be able to attend Agile Japan 2011  because the keynote speaker was Linda Rising, talking about introducing change, and it was a chance for my colleagues to see that Agile is used in Japan.

The main conference was held in Tokyo but there were 10 satellite conferences held around Japan, from Okinawa to Hokkaido, including one in Sendai. Because there was a satellite conference in Osaka it was easy for me to take the whole team. Initially I was expecting just a video link to Tokyo, and this was the case for the morning session. But for the afternoon they had a local program – with parallel sessions! There seemed to be about 60 attendees in Osaka.

In the keynote from Linda Rising she discussed her favourite patterns from “Fearless Change.” I had read this book some time ago and it was a nice refresher of the patterns. I will definitely have to re-read the book again. The comment that seemed to resonate most was “Just do it!”. This presentation is now on YouTube


The first presentation in Osaka in the afternoon was by Kawabata Mitsuyoshi of Agileware  talking about the 5 eXtreme Programing values. The most interesting part of this presentation for me was the specific difficulties of doing agile in Japan. XP values face to face communication but often when you want to discuss an issue the response is “I will send an email” or “let’s discuss it later.” This is something I have noticed myself with developers using email to “communicate” with someone a few meters away. I doubt that this problem is unique to Japan though.

The second session I attended in the afternoon was a workshop by Tarumoto Tetsuya  on  Usability Testing and more specifically DIY usability testing when you can’t afford or wait for a professionally run usability test. This included a very interesting description of the usability testing for the Suica  contactless smart card used for JR railways in Tokyo.

To demonstrate how to run a test, asking the users what they are thinking rather than asking them their opinion of the software being tested the workshop included a Kanji game. Unfortunately this was completely impossible for me, lacking the many years of hard study of Kanji which all Japanese have at school.

Overall the conference was very well run, had some interesting presentations and I definitely hope that there will be an Agile Japan 2012 and that I will be able to go.

3D rotation about an arbitrary axis

I often need to do some geometric calculations involving rotation of a vector about an axis. However, not often enough to have the matrices for the rotation to hand. The calculation is not particulary difficult and is described in many places, but last time I needed this I found a very good write up by Glenn Murray:

http://inside.mines.edu/~gmurray/ArbitraryAxisRotation/

Not only is this an easy to follow description, it also has the final rotation matrices which makes it very easy to code.

Printing from QtCreator

Why are the printing functions from most IDEs so bad? OK so I have not really compared the printing function in many IDEs, but the ones that I have used hardly offer any control over the printing at all and print in a large font.  I do like to read through listings on paper and to make notes on the listing  when trying to understand code or planning some modifications. But, I don't like priniting out pages and pages just to get a single function.

For whole files, I have been using a2ps. It can print multiple pages per sheet, supports two sided printing and produces readable listings with a small font. However, prinitng selected text was more difficult.

Now, QtCreator has support for external tools so I can print both whole files and selected text from within QtCreator but using a2ps. To print a file I just configured an external tool with the command line:
/usr/bin/a2ps --tabsize=4 --sides=tumble --medium=a4 -f8 -Ec++  --center-title=${CurrentProject:FilePath} -g -d "%{CurrentDocument:FilePath}"

To print a selection is similar, but instead of specifying the file as an argument, %{CurrentDocument:Selection} is specified as the input to the tool as shown below.

Hard Choices

The Hard Choices board game from the Software Engineering Institute is a game designed to teach the effect of technical debt in software development. Players choose between taking a shortcut or taking the "long way" and gathering more tools. Shortcuts incur a -1 movement penalty on subsequent turns unless one turn is spent "refactoring."

An interesting suggestion from the Facilitator's guide is to play multiple rounds, keeping the shortcut and tools cards from the previous round. This simulates a second release where shortcuts from the first round will continue to slow development.

I want to play this with my development team but I am not sure how well the will take to playing a dice game in the office. The discussion following the game is likely to be the most important part.

The game itself is provided as a PDF download to print out. Just printing it out seemed a bit too simple so I printed it on some ink jet films and stuck the board on a piece of poystyrene board and the tool cards onto some thick card.

A monitor for Cruise Control

I have wanted to build a monitor for our Cruise Control build server ever since I heard about the use of lava lamps to report the build status. The “easy” way seems to be to use an X10 automation system but unfortunately I have not been able to find that for sale in Japan. (It probably conflicts with local radio licensing laws. )

Our builds have been staying broken for tool long. The email notification  does not seem to encourage the fixing of the build. We definately needed something more obvious to show the build status. So when I recently found that there was a 4-way USB relay board available at a fairly reasonable price from Elefine I decided to build my own monitor. The hard part is controlling the power from the computer and the relay board takes care of that. Once I got the board all I needed to do was put it in a case and create some driver software to control it.

Fitting the board into a box and wiring it up was fairly simple. Unfortunately the USB port on the relay board is flush with the edge of the circuit board. This means that a fairly large hole is needed in the case to take the USB plug which looks a bit messy. 

For the lamps I decided not to use lava lamps because they will have to sit on the desk in the office and they tend to run very hot.  Instead I have adapted a small globe light, taking out the normal bulb and replacing it with two separate lights, one red and one green.

For the software I had already made a cruise control monitor plugin for QtCreator so I had most of the code already. All that was needed was to add code to switch the relays on and off depending on the build status. I tried to use the DLL which is provided by emx for the relay board but that did not work well – causing the program to crash on exit. So instead the code uses their command line utility for controlling the board.

Using the 3DConnexion mouse with a Qt Application

This note describes my experiences getting a 3D mouse to work in a Qt application on Windows.

The 3DConnexion 3D mouse is a 6-DOF controller often used for CAD and engineering applications. 3DConnexion provide an SDK for windows which provides integration with MFC applications but not for Qt.


RAWINPUT support for Qt

The recommended method of supporting the 3DConnexion 3D mouse is to use Windows RAWINPUT message to receive the data from the mouse and then process those message to get the motion data.

There are 2 problems here

• Qt has no native support for RAWINPUT - it is a windows only system and Qt has no corresponding Qt events for the 3D mouse
• The "out-of-the-box" version of Qt does not forward RAWINPUT messages as they are received - at least up to Qt 4.6.2. On windows 7 Qt4.7.1 seemed to work with no modification.

The Raw Input API provides a "stable and robust way for applications to accept raw input from any HID, including the keyboard and mouse." [msdn] This enables the application to get the actual data sent by the HID device without any processing being done by windows first.

The application first registers to receive Raw Input from a particular device type, then when that device has data the application receives a WM_INPUT message in its event queue and the queue status flag QS_RAWINPUT is set. The application can then retrieve the data using the Raw Input API.

Raw Input was added in Windows XP, so is not available in Windows 2000
(Which is important -- see below.)

Raw Input and Qt

The problem with Qt (at least in 4.6.2) is that the version from the installer does not process Raw Input messages correctly. The way of collecting these the Raw Input messages is to set an event filter on the QApplication which gets the native windows messages. These can then be processed by the application using the Windows Raw Input API. However, the WM_INPUT messages are not being received by window/event filter the until some other event such as a mouse or timer event occurrs. Moving the 3D mouse causes no events to be received by the event filter but then if the normal mouse is moved (or key pressed etc)
all(?) the 3D mouse events arrive.

The problem can be traced to QEventDispatcherWin32?, which maps the windows messages onto Qt Events. Specifically, the line

   MsgWaitForMultipleObjectsEX(nCount, pHandles, INFINITE, QS_ALLINPUT,MWMO_ALERTABLE);

in the processEvents function of QEventDispatcherWin32. This is blocking and not waking on WM_INPUT events.

The windows documentation for this notes that QS_ALLINPUT, which includes QS_INPUT, “does not include QS_RAWINPUT in Windows2000.” Therefore, the problem is that the binary Qt installer is built to work on all windows platforms, so during the compile _WIN32_WINNT is defined to something less
that WinXP? (0×501) so that raw-input messages are not handled.

The fix is to re-compile Qt passing -D _WIN32_WINNT=0×501 to configure so that the WM_INPUT messages are correctly passed to the event filter when the 3D mouse is moved.

So this is not really a Qt bug, but rather that Qt is compiled for an "earlier" Windows platform which does not support Raw Input.

Sample Code for Qt and the 3D Mouse

I updated the 3DConennexion code for MFC to work with Qt. It uses an event filter to pick up the raw windows events then extracts the 3D Mouse information. This is then available as a Qt signal.

The zip file below contains a very simple application which displays the motion data in a window. This was tested with Qt 4.7.1.

3DMouse.zip

A script for adding test cases to a CppUnit test class

One of the problems with using CppUnit as the unit test framwork in C++ is that adding a single test requires updating the code in two files and in total 3 places. A declaration in the header file, a macro in the header file to regsiter the test and the test itself. This makes adding a test too hard, which means the tests themselves become too big because it is easier to add extra asserts to an existing test than to add a new test. Some of the newer C++ frameworks overcome this, but we have been using CppUnit for a long time and have a lot of tests already written so I do not really want to change frameworks.

To fix this I created a small Python script to add a new test, so working from the command line, I just need to write

newtest TestCase testMyNewTest

and the declaration and prototype are created. I don't need to even look at the header file I can just go straight to the definition of the test function and add the test.

The header file for a CppUnit test case looks like this:

class TestCase : public CppUnit::TestFixture

{

CPPUNIT_TEST_SUITE( TestCase );

    CPPUNIT_TEST( testSomething );

    CPPUNIT_TEST( testNextBit );

  CPPUNIT_TEST_SUITE_END();

public:

    void    setUp();
    void    tearDown();

    void    testSomething();

    void    testNextBit();

};

And the source file

CPPUNIT_TEST_SUITE_REGISTRATION( TestCase );

void TestCase::testSomething()

{

 // test code here

}

void TestCase::testNextBit()

{

 // test code here

}

The script is very simple, it searches for the last CPPUNIT_TEST macro in the header and adds a macro for the new test case. Searchs for the last test function, and adds a test after that. Then it adds a defintion at the end of the source file.

The script makes two assumptions

• The name of the class matches the name of the file. So the above test class would be in TestCase.h and TestCase.cpp
• The test class already has at least one test function when the script is run. Since I use a separate script to generate the boilerplate test class with a single test this is always true in my case.

The script is available here in case it is of use to anyone else: newtest.py

A CruiseControl monitor for QtCreator

We use CruiseControl to run automatic builds after every commit and I thought it would be nice to monitor the build status from within QtCreator. I also wanted to try creating a simple QtCreator plugin to see how easy it would be to extend it. I already use the Firefox add-on to monitor CruiseControl which does all that I need for a monitor so I just wanted to port that to work in QtCreator.  The plugin adds a red/green icon in the left hand bar to indicate the build status and a tooltip to show the status of individual builds.

The tutorial from VCreateLogic was very helpful setting up a basic plugin.  This tutorial describes how to set up a project, how to add menu items or new settings. So the basic configuration was straightforward.

The Welcome plugin which is part of QtCreator has a class for getting rss feeds. This formed the base for getting the data from the CruiseControl server. The main change required was an update to how the xml is parsed. Also, my first version worked fine when getting data from a test file but would not get data from the server. The original does not set the port, since I assume all normal rss feeds use the default port 80, but our CruiseControl server does not use port 80.  Once discovered this was trivial to fix.

The hardest part was finding out how to add the display widgets to the left hand bar in QtCreator.  A bit of poking around found a modemanager which manages the widgets. That seems to work but I am worried about API changes in future versions since there seems to be no official documentation for the API.

The source for the plugin is available on http://code.google.com/p/qtcreator-cc-plugin/

At the moment only the monitoring function is implemented.  Hopefully I will be able to find time to add functions to restart or stop builds from a context menu.