Analysing My Twitter Network

I have been on twitter for more then 10 years, but normally use it as a source of information, just reading tweets rather than for conversations. As part of @hjarche's PKM workshop  I thought it would be interesting to look at the connections between those that I follow on twitter and see what networks exist.

There seem to be many tools for analysing twitter followers demographics  and for analysing tweets but not so many for looking at networks of those you follow (at least that I could find with a Google search).

However, I found a series of Python scripts that extracts connections between followers of followees. This process is very slow because twitter rate-limits the access to its API to 15 calls in 15 minutes. So extracting the information for the 600 people I follow took 10 hours.   It also requires access tokens which means signing up for a Twitter developer account.

The output from the scripts is a CSV file suitable for use with Gephi. I had not come across Gephi before, so this process also introduced me to a new graph analysis tool which might help with another of my problems, that of looking at the dependencies between a lot of software modules in a large program.

The process with Gephi is to load the graph file from the Python scripts and then runs some analysis and use the automatic layout algorithms to layout the graph. I found a tutorial that explains the basic details.


This produced a graph for those I follow on twitter like this:

The nodes on the graph are the people I follow on Twitter. The edges in the graph are the followship links between these people. The size of the nodes depends on how many connections there are to that node.

The "modularity analysis" found 6 "communities" based on the connection information, this is represented by the different colours. However looking at the graph, 8 different main groups seem to exist.

1. This is the main group I follow mostly software development related, especially agile software.
2. This is also technical but seems to form a different group from 1.
3. This group consists of twitter personalities, or more business related topics.
4. These are Japanese software developers.
5. This small group is a technical niche for 3D CAD
6. This group is related to Japan so is mostly separate from other groups
7. Photography related
8. Japanese Jazz artists

The lack of connection between groups 4 and 8 and the rest does not surprise me due to the language difference. These groups tweet in Japanese whereas the rest are English. Even though the Japanese developers have a strong interest in "agile software development" there are few links to the non-Japanese developers I follow:

It is clear that the main group I follow is the "software development" group. I it is interesting to me that the analysis algorithm split this into two groups.

Since this exercise was prompted by the PKM workshop I looked at the people that @hjarche and myself both follow. I was surprised that there are only a few that be both follow:

I am not sure where to go from here. My list of follows needs some pruning. Also, some areas need expanding.

This analysis just looks at connections between people, not at how often they tweet. The timeline tends to be dominated by a small number of prolific tweeters / re-tweeters so some of the conversations in the main groups probably get drowned out.

Links

• PKM Workshop: https://jarche.com/pkm/pkm-workshop/
• Gephi: https://gephi.org/
• Egonet Scripts: https://github.com/alumbreras/twitter-followers-graph

Note, the README description for the scripts is out of date.
The name of the script has changed and the commands are now:

 python egonet.py -u username -f followees
 python egonet.py -u username -f followees_graph


Using CppUnit with QtCreator.

We have been using CppUnit for many years and have a large set of tests in CppUnit and so wanted to continue using the CppUnit framework while using QtCreator as the IDE. QtCreator has a plugin for using the Qt test framework but changing test framework would be a lot of work.

The basic goals were

• automatically run tests when the code is built
• Have test failures show up in the Issues window of QtCreator 
• Easily add new tests

Running the tests

This example is based on the Money example code from CppUnit. The MoneyTest file was renamed MoneyTestCase to match the conventions we use.

To use QtCreator with the money example, the first task was to create a simple qmake .pro file for the project

 TEMPLATE = app  
 TARGET = cppunit-test  
 INCLUDEPATH += .  
 HEADERS += Money.h \  
   MoneyTestCase.h  
 SOURCES += MoneyApp.cpp \  
   MoneyTestCase.cpp  
 LIBS += -lcppunit  

This compiles and the tests can be run:

Starting /home/dibben/Develop/QtCreator/cppunit-test/cppunit-test...
....
OK (4)

But we want to run the tests automatically when they are built. So we add a custom process step.

Now when we type "Ctrl+B" to build the project the tests are automatically run.

Getting the test failures

When using the default outputter, if a test fails, we get a message such as:

..F..
MoneyTestCase.cpp:52:Assertion
Test name: MoneyTestCase::testEqual
assertion failed
- Expression: money123USD != money123FF
Failures !!!
Run: 4   Failure total: 1   Failures: 1   Errors: 0

But we want this to appear in the Issues list of QtCreator with a link to the test that failed.

To do this we need to change the output format of CppUnit. This requires changing the MoneyApp.cpp code to use the basic test runner and adding a new outputter. The main function now looks like this:

  CPPUNIT_NS::TestResult controller;  
  // Add a listener that colllects test result  
  CPPUNIT_NS::TestResultCollector result;  
  controller.addListener( &result );  
  // Add a listener that print dots as test run.  
  CPPUNIT_NS::BriefTestProgressListener progress;  
  controller.addListener( &progress );  
  // Add the top suite to the test runner  
  CPPUNIT_NS::TestRunner runner;  
  runner.run( controller );  
  runner.addTest( suite );  
  // Run the test.  
  runner.run(controller);  
  bool wasSucessful = result.wasSuccessful();  
  // Print test in a compiler compatible format.  
  #ifndef _WIN32  
      GccOutputter outputter( &result, std::cerr );  
  #else  
      MSVCOutputter outputter( &result, std::cout );  
  #endif  
  outputter.write();  

This will collect the results in the result object and then print them at the end of the run using either the GccOutputter or MSVCOutputter. These are modified versions of the Compiler outputter that produce output that QtCreator can interpret. Building this will now produce the following output in the Compile window

 18:57:46: Starting: "/home/dibben/Develop/QtCreator/cppunit-test/cppunit-test"  
 MoneyTestCase::testConstructor : OK  
 MoneyTestCase::testEqual : assertion  
 MoneyTestCase::testAdd : OK  
 MoneyTestCase::testAddThrow : OK  
 MoneyTestCase.cpp:52: error: - Expression: money123USD != money123FF  
 Failures !!!  
 Run: 4  Failure total: 1  Failures: 1  Errors: 0  
 18:57:46: The process "/home/dibben/Develop/QtCreator/cppunit-test/cppunit-test" exited with code 1.  

and QtCreator will pick up the error and display it in the Issues output pane:

So now when the code is built the tests will be run automatically and any errors will show up in the Issues pane linked to the source line for the failed test.

The code GccOutputter and MSVCOutputter is available on Github here.

Adding a new test

The money example uses a separate header and source file for the test cases. This means that to add a new test we need to do 3 things

1. Add a declaration in the header
2. Add a definition in the source file
3. Add the test to the test suite using the CPPUNIT_TEST macro.

Some of the newer frameworks make this easier but we are using CppUnit. To make this simple in QtCreator I created a plugin that adds some support functions for Cppunit. The plugin is available on Github

With the plugin creating a new test is as simple as Ctrl+Shift+T, then a dialog appears for the name of the new test. Multiple tests can be entered separated by spaces. On clicking OK this does the 3 steps above to add the test to both the header and the source file.

The plugin has some other functions as well

• Create new tests
• Create a new test case class (custom class wizard)
• Switch between source and test case

References

• CppUnit Outputter classes on Github: cppunit-utilities
• QtCreator plugin for Cppunit test creation on Github: qt-creator-cppunit-plugin
• Source code for the example: cppunit-test.zip

Evaluation based on results is not the answer to long working hours

Shinzo Abe, Japan’s Prime Minister, has announced that Japan needs a more flexible working style without the ridiculously long hours that prevent many women from working. This is a very laudable goal and something that Japan will need in the face of a shrinking population and many in the younger generation that seem reluctant to devote their lives to the company. (Typically, the photo accompanying the article about improving working conditions for women shows a committee that consists almost entirely of old men.) In fact, changes are already happening, albeit very slowly. When I started working in Japan 14 years ago many colleagues came into work on Saturdays and nearly all stayed very late. Many still stay fairly late but the company has prohibited Saturday working and is putting limits on overtime.

What really worried me in the Yomiuri article though was the quote:

“I would like you to investigate a framework for a new working-hours system that befits work styles evaluated based on results, not time,” Abe said at the meeting.

This sounds like Abe wants to replace one dysfunctional system with a different but equally dysfunctional one. A shift from the bureaucratic nightmare described by Northcote C. Parkinson to one based on industrial age thinking.

The Yomiuri article hints at the problem by describing the “entrenched system of long working hours suppressing productivity” and suggests limiting working hours as the solution. However, the problem is in the working culture that equates dedication to hours worked and working harder as the means to achieve more. Changing this will be harder than just limiting overtime. Which is probably where the focus on results comes from, companies worried about their revenue probably want that to ensure nothing actually changes.

The suggestion in the article is that employees could work whatever hours they like so long as they achive the results. This sounds remarkably like management by objectives which is strange from a country that so admired W. Edwards Demming who had “Eliminate managemement by objective” as one of his 14 points for running a company. The problems with this approach are legion. It discourages teamwork and it puts the focus on the individual instead of the system as the source of productivity. It also assumes that it is actually meaningful to measure “results” or productivity at an individual level.

To improve productivity it is necessary focus on the system. Ironically it is Japanese companies such as Toyota that have shown the world what can be achieved by doing this. Yet Abe’s statement demonstrates that this is an exception, even in Japan rather than the norm. I see this problem daily at my own workplace. Long working hours are considered normal, this gives no incentive to actually improve the working practice since you are going to be in the office anyway. And even when there is a desire to improve the system everyone is so busy dealing with what John Seddon calls ‘failure demand’ that there is no time to introduce improvements.

From the perspective of software development this means that developers are so busy debugging the code that they have no time to consider how those bugs got there in the first place and what changes they could make to prevent them. “We don’t have time to do unit testing or code review”. It is very hard for them to change the system, so simultaneously making them responsible for their results while limiting the only thing that they control, their working hours, is both demotivating and unfair.

Update: A nice article from Vangard Consulting puts the case against "management by results" in the health care sector very clearly:

Wherever management by results exists so too does the failure to accurately understand and sustainably solve systemic problems. This is in the nature of management by results because:

1. It wrongly assumes that performance can be adequately described by a system of measurement; and
2. It holds people and organisations to account for the fictions that this creates.

Another book from Packt

Packt asked me to review another book for them; this time on Review Board.  "Getting Started With Review Board" by Sandeep Rawat. Maybe they didn't actually read my previous review of the QtCreator book. After I replied that I would be happy to write an actual review, that would probably be bad,  rather than just an endorsement I didn't hear back from them

Review Board is a great tool. I have been using at work now for several years. It is a free open source tool for code review. In the very early days the setup required installing several packages but for several versions now the setup/upgrade process has been made trivially easy. It uses the python easy_install tools so normally the install, including all the pre-requisites can be done with a single command. It also has great online help and installation instructions. It is also very easy to use. I find that new users in my team just need a 5 minute explanation of the main features and they are good to go.

Which brings me to the Pack book. It is described on their web page as

This book is a concise, to-the-point guide with a practical walkthrough of the code review workflow using the features present in Reviewboard. 

Based on the table of contents and the sample chapter what this means is that the book is simply a very short description of what reviewboard is. The sample chapter on "Reviewing Code Review Requests" was simply a description of the Review Board features that are not only easy to figure out for yourself they are also described fully in the free help with the program. The book will also be out of date very soon since Review Board 2.0 is about to be released.

So I don't really understand the point of this book. Why would anyone want to spend $12  for the eBook or $22 for a paper book for something that is not really needed and available for free online.

What would actually be useful is a book on how to do code reviews. Karl Wieigers "Peer Reviews in Software: A Practical Guide" is one of the best books on the subject that I have found. Smart Bear also have a free eBook on code review available (requires registration) which while obviously meant to advertise their code review tool does discuss code review in general. However, there are not so many books that cover code reviews.

The publisher of the book is often a good indicator of the quality of the book. Some publishers, like The Pragmatic Programmers are always reliable.  I think I will be avoiding Packt books in the future.

Application Development with Qt-Creator

Packt sent me a copy of their new book on using Qt-Creator "Application Development with Qt-Creator"  by Ray Rischpater.  It is very clearly targeted at beginners and anyone with experience of other IDEs would probably find it too light. However, if you are completely new to programming it covers some of the basics of using Qt-Creator, but not really enough to create an actual application.

This is not a book for learning C++ or Qt. The code samples are very simple. There is section on how to use Qt's signals and slots but many of the topics needed for non-trivial applications are not even mentioned (for example the more advanced widgets such as lists, tables, combo-boxes or event handling.) Fortunately, Qt has comprehensive documentation that covers most of this.

The biggest problem that I had with this book was what was missing. Many of the functions that I use the most while programming were not mentioned at all. One of Qt-Creator's strengths is the C++ code model that allows fast navigation. You can find all the places a function is used (Ctrl+U), follow a symbol to where it is declared/defined (F2) and switch between header/source files (F4). This makes navigation in a large code base fast and easy. But these functions did not seem to be covered. Also, the following basic topics are not covered:

•  Search and replace
•  The different views in the left hand pane (class view, open documents etc)
•  Splitting the main editor window to view two files at the same time
•  Editor functions such as commenting out a block of code
•  Shortcut keys
•  Refactoring functions for renaming symbols / extracting functions
•  Code snippets

For anyone familiar with an IDE these are all straight forward and easy to use. However, if you are new to programming (and are therefore in the target audience for this book) then you might want a bit more explanation and description of the available functions.

The really big omission is a description of the location function (Ctrl+K). This function is central to using Qt-Creator and, for me, what makes Qt-Creator stand out as an IDE. The location field in the bottom left of the screen has many functions. Just typing a filename allows quick opening of any file in the project. It also gives access to the help system, the source control functions, lists of classes and member functions.  Since this is something that sets Qt-Creator apart it almost deserves a whole chapter by itself, but it is not even mentioned.

The lack of coverage of major functions actually suggests that the author has not actually used Qt-Creator for any serious development and has not used Qt significantly either. Slots can be private (or protected) contrary to what is claimed in the book.

The overall impression is one of a professional writer who has not actually used Qt-Creator for any real development. So the writing is clear and easy to follow but the coverage of the topic is very superficial.

If you just want to quickly learn how to setup Qt-Creator, for example to complete a class assignment, then this book covers what you need.  For professional developers on larger projects this book is missing coverage of too many functions.

Installing ReviewBoard on RHEL-5

ReviewBoard is a great free code review tool which we have been using for several years now. We run it on a Redhat server and it is connected to our Subversion repository. With the 1.7 release of ReviewBoard support for Python 2.4 was dropped and unfortunately that is the standard Python on RHEL-5. So I have been putting off upgrading for some time.

Finally decided to upgrade and switch from using the SQLite database to MySQL. Since we have a small team we have not had any performance problems with SQLite, except where the source code in a review is split across multiple pages when changing the page with an open review causes the DB to get locked and requiring a restart of Apache. The ReviewBoard docs do warn about using SQLite but we have been using ReviewBoard since before version 1 when the install process and DB support was not nearly as good as it is now.

So first task was to get a newer Python. Fortunately this turned out to be much easier than expected since the Extra Packages for Enterprise Linux (EPEL) contain the rpms for Python 2.6 so it was as easy as

    yum install python26

Now installing ReviewBoard should be extremely easy. On my Fedora box at home, I could just type

    easy_install ReviewBoard

and ReviewBoard and all its dependencies were installed. Trivially easy. Unfortunately the server is behind a proxy server. Setting the http_proxy environment variable and the values in yum.comf were enough to get yum to work, but not easy_install. That kept on giving errors.

After several frustrating hours trying to different proxy server settings I eventually stumbled on the solution: Our proxy server seems to require that https_proxy is set in addition to  http_proxy. So setting both:

   export http_proxy=http://192.168.1.1:80
   export https_proxy=http://192.168.1.1:80

and it installed!  The Google search for easy_install proxy problems never mentioned this. So I don't know whether this is an easy_install problem or our peculiar proxy server.

Of course, it still wasn't that easy. While setting up the site MySQL was not available because the install of the pymysql had failed because the MySQL devel package was not installed and there was no error message visible during the easy_install.

However, finally it works. Unfortunately, there is no easy way to migrate the old SQLite data to MySQL.

Developer Summit 2012 in Kansai

The DeveloperSummit was a free one day conference (in Japanese) for developers held in Kobe, Japan.  It is nice to see more of these events taking place in Kansai rather than just in Tokyo. This event was very popular with several of the sessions being completely full.

The keynote presentation was from Oikawa Takuya, one of the Japanese developers on the Google Chrome team. The main theme was the changes in project management style required when developing for “the cloud.” Out is the old style product lifecycle with releases every few years and in is the “versionless” development with very frequent or “continuous” releases.  Also, the use of open source and the challenges that brings to development when contributions from many people must be integrated. This was an interesting presentation giving some insight into the how Google develops some of its software and Chrome in particular.

The first presentation on the “Agile” track was from a Microsoft evangelist entitled “Continuous value delivery to the next decade.”  This presentation also discussed the differences between waterfall and agile development and how development is done at Microsoft. However, the whole presentation seemed like an advert for TFS. The main point seemed to be the advantages of using a single tool for project management, bug tracking version control etc rather than several different tools. And Microsoft just happens to have such a tool. This seemed to me like agile for those who want to say they are doing agile but don’t want to change too much. The process seemed to be more like a series of mini-waterfalls backed up by comprehensive tools.

Following the Google and Microsoft promotions we had an advertising spot for JIRA (the bug tracking system from Atlassian) given by Suzuki Yusuke, an engineer from Growth xPartners which seems to be an Atlassian partner in Japan. This was less a less subtle promotion since the presenter even included a price list for JIRA in one of the slides. After yet another description of waterfall and agile processes (the third so far) the presenter stressed the importance of collaboration and communication in Agile development.  I would not argue with that, but I am sure that there is more to communication than a bug tracking system. The middle part of the presentation described how “Agile is implemented in the bug tracking system.” The idea seems to be to use JIRA to hold the list of tasks to be done and to act as the communication medium for the team.

We use JIRA as a bug tracking system and it seems to be one of the best bug trackers available. However, I often see it as blocking rather than enabling communication. Instead of direct communication between individuals, communication is via comments on an issue. This leads to misunderstandings and delays. Also because it is so easy to create an issue the “backlog” quickly becomes an unmanageable black hole into which requests and improvement suggestions are sent to be forgotten.

I suppose that not surprisingly, from a company selling the bug tracker there was no suggestion that the use of a bug tracker is itself actually a symptom of a bigger problem.  A truly agile team would probably have no use for a bug tracker.

The presentation that I was most looking forward to was that by Wachi Yukei on “The evolution of test driven development.” He is one of the translators of “GrowingObject Oriented Software Guided by Tests”, the Japanese version of which was just released on the same day as the developer summit. This presentation gave an overview of GOOS. The description of Mocks was fairly hard to follow and my Japanese colleagues who were not familiar with mocks did not seem to understand it at all.  Actually the biggest problem seemed to be the direct transliteration of English expressions into Japanese, such as “walking skeleton.” This is not a problem for me as an English speaker – in fact it makes it easier – but makes it hard for non-English speakers to understand. However, the presentation did include a lot of good stuff from GOOS such as the need for end-to-end tests and thin slices of functionality. I am not quite sure on the relevance of the photo of David Cameron.

Many of the presentations are on SlideShare (in Japanese)

Continuous value delivery to the next decade:

http://www.slideshare.net/tomohn/a1-continuous-value-delivery-to-the-next-decade

Collaboration in enterprise development – how to connect the development team with customers using JIRA

http://www.slideshare.net/yusuke/2012a2

The evolution of test driven development

http://www.slideshare.net/digitalsoul0124/ss-14351848