Re: Advertising the Git User's Survey 2011
On Sat, 2 Oct 2010, Ramkumar Ramachandra wrote:
Show 15 quoted lines
> Ævar Arnfjörð Bjarmason writes:
> > On Mon, Sep 20, 2010 at 13:38, Jakub Narebski <jnareb@gmail.com> wrote:
> >
> > > Currenly the survey has more than 5000 responses (in a not whole month)
> >
> > That seems somewhat low, but maybe people just aren't that interested
> > in taking surveys.
>
> Nice work on the survey! This is our best year by far. Some general
> observations:
>
> Interesting statistic: 24508 people viewed it, 7821 people completed
> it, but 0 people started filling out information and later decided not
> to submit it. It could mean that many people clicked through and found
> the survey, but probably left because it looked too long at a glance?
Without the knowledge how those two numbers are calculated we can only speculate what do they mean.
I think that '0' in 'Incomplete' statistics is here because this survey doesn't have compulsory questions: answering all questions are optional, so it might mean that even if one question in the survey is answered, then the survey is considered complete by Survs.com statistics.
I personally do not like the "wizard" formatting of surveys, i.e. dividing survey into page so you are not presented with very long page, but are presented withc chunks of survey at glance. Even if you see how much survey did you fill in (how many pages there are in total), and even if you can go back to previous page.
I'd prefer to create a better information about survey upfront. We say that all questions are optional ("Note that you may skip questions as you like"). It is also stated that you can fill only a part of survey, and later go back to finish it... hmmm, I wonder if those cases where one edited his/her survey responses multiple times are counted as one finished survey, but multiple views.
We could also write how much time it takes on average to fil the survey.
> The average time spent on the survery is 34 minutes
It would be interesting to have more detailed statistics of time spent on the survey that only average time, a single number. When one is filling open-form essay-length question, it would obviously take much more time than for one who doesn't.
But Survs.com currently doesn't provide it. I can try to ask for it, though (via feedback).
Show 7 quoted lines
> - I think we can bring that down to 10~15 minutes if we design
> questions to extract more information. Also, there's little incentive
> for taking the survey: while many companies actually give out
> discounts/ coupons for taking surveys, the least we can do is present
> real-time results in the most interesting manner possible ie. survey
> takers should see the "results so far" immediately after taking the
> survey; some visualizations such as pie charts?
What we can do is after finishing the survey to redirect to the survey analysis page:
https://www.survs.com/results/33Q0OZZE/MV653KSPI2
http://tinyurl.com/GitSurvey2010Analysis
instead of IIRC currently used redirect to
https://git.wiki.kernel.org/index.php/GitSurvey2010
As far as I know Survs.com doesn't provide any API for extracting data or survey statistics required for creating such visualization. Neither we have a place where such app could be created, I think.
Show 5 quoted lines
>
> In questions 5, 10, 12, 13, 16, cut down out the options that have
> very few respondents and let them all go into "other". It probably
> doesn't actually save the survey taker any time, but I think seeing a
> long page with many options can be scary.
The "5. Which Git version(s) are you using?" is not that long. We could create a cut off a bit earlier, perhaps on 1.4 (i.e. have "pre 1.4") or even earlier, we could remove alternate implementations answers (git-bigfiles, JGit, other implementations), or even concatenate 'master', 'next', 'pu' into single response... but would it buy us much?
The other side of removing choices, relying instead on "other, please specify" response is that it makes it harder to analyze results of survey: different people use different words for the same thing (and there are also spelling mistakes), and make results less reliable: people do not fill "other" if there is at least partial match, or do not know how to specify their version.
In "10. What Git interfaces, implementations and frontends do you use?" we can remove tools marked as deprecated... if not for the fact that it is actively interesting to know how many people use such deprecated tools. Besides, the list of answers to this question is not overly long, I don't think.
We can remove those choices in "12. What Git GUIs (graphical user interfaces) do you use?" that got less than 1% rounded, or less than 10 responses. On the other hand some people stated earlier that the list of possible choices in the survey (not necessarily about this question in specific) serve as reminder / information about possible choices.
The other side of removing options from "13. Which git hosting site(s) do you use for your project(s)?" is that when sending requests to announce the survey to those git hosting sites that are not on this list, some of them requested to be added (which is impossible after starting the survey; and before survey begins it is little sense to send announcements).
Besides all of those below 1% rounded (Codesion, GitFarm, The Chaw,
CipherHive) are also those that I didn't get response to request for
announcing Git User's Survey 2010...
We could make it more organized though, e.g. by sorting list of options
alphabetically, or something like that.
Removing options from "16. Which of the following features do you use?" would make it harder to analyze and less reliable. Especially in this question different people consider different features important enough to mention, and describe feature in many different ways.
Besides, each option except of "git cvsserver" got more than 1% rounded, and having "git cvsserver" is interesting on its own (perhaps in other question?).
> 1. Country of residence: we can probably make this a nice click-on-map
> interface as opposed to freeform text. It'll be more useful to us,
> and more interesting to users when we advertise the results.
It would be nice to have click-on-map (Google Maps or Bing Maps based), something like Ohloh provides, resulting in map of survey responders similar to the map of git users and git contributors on Ohloh
http://www.ohloh.net/p/git/map
it isn't something that Survs.com offers currently. I can only ask for it to be provided...
Another solution would be to have pre-filled combo box (<select> field) with the list of countries to choose from, with GeoIP ised to pre-select the country. I can generate list of all countries myself
$ perl -MLocale::Country \
-wle 'print join("\n", sort (all_country_names()))'as far as I know Survs.com doesn't offer GeoIP nor any API to hook it to survey questions.
> 2. Age: Maybe we restrict the input to 2-digit integers and draw a
> graph with all these integers to show a mean, median etc?
Restricting input doesn't give us much.
There is nice histogram of responder's age for Git User's Survey 2009
https://git.wiki.kernel.org/index.php/GitSurvey2009#02._How_old_are_you_.28in_years.29.3F
and tabularization of responses. We can calculate mean, median, mode (aka modal score, i.e. most common response), perhaps after eliminating outliers, but would it give us much information?
> 11. Just change this to an optional sometimes/ often? Why should users
> spend time clicking on "never"?
"Never" is here because you can't un-click response in given row. I'm also not sure how not answering row is represented in the export of survey data (which we use for further analysis).
> 17, 18: Merge perhaps?
Those questions are split because of limitation of Survs.com; the "other, please specify" results in limited width _text field_, while it is much easier to write in-depth response in large _textarea_ field that wuestion 18 provides.
> 24, 25: Merge into single question with options: "Yes, and my problems
> are solved more often than not", "Yes, but my problems often
> remain unresolved", "No, I don't go to others for help".
> 26, 27: Merge into "How do you talk to other people using Git, either
> for technical help or otherwise?"
I don't think it is good idea. Those two issues, namely requesting help and being (perhaps silent) part of git community are two unrelated issues.
> Ofcourse, I understand that there must be some technical constraints
> due to which some things are not implementable (eg. survs doesn't
> provide the feature?), but I've not taken that into consideration.
Note that as it currently stands we can use Survs.com account only for 2011 survey, provided that it is done earlier than this year (perhaps 1 June -- 31 July?), as our Premium account which we got thanks to generosity of Survs.com admins (after Survs.com got out of beta) will downgrade to the Free plan (which is offersn much too low limits) on Sep 22, 2011.
--
Jakub Narebski
Poland
Ævar Arnfjörð Bjarmason
