From: Ramkumar Ramachandra <artagnon@gmail.com>
Date: Sun, 03 Oct 2010 12:48:22 GMT
Subject: Re: Advertising the Git User's Survey 2011
Message-ID: <20101003124818.GA328@kytes>
In-Reply-To: <201010031121.03795.jnareb@gmail.com>

Hi Jakub,

Thanks for taking the time to go through my long (somewhat
unstructured) note.

Jakub Narebski writes:
> On Sat, 2 Oct 2010, Ramkumar Ramachandra wrote:
> > Interesting statistic: 24508 people viewed it, 7821 people completed
> > it, but 0 people started filling out information and later decided not
> > to submit it. It could mean that many people clicked through and found
> > the survey, but probably left because it looked too long at a glance?
> 
> Without the knowledge how those two numbers are calculated we can only
> speculate what do they mean.
> 
> I think that '0' in 'Incomplete' statistics is here because this survey
> doesn't have compulsory questions: answering all questions are optional,
> so it might mean that even if one question in the survey is answered,
> then the survey is considered complete by Survs.com statistics.

I just opened up survs.com, signed up for a free account and created a
survey with two compulsory questions- I couldn't submit the results
without filling up both of them. Then I looked at the statistics and I
saw a '0' in incomplete- I'm confused. Perhaps we can email survs.com
and ask them?

> I personally do not like the "wizard" formatting of surveys, i.e. 
> dividing survey into page so you are not presented with very long page,
> but are presented withc chunks of survey at glance.  Even if you see
> how much survey did you fill in (how many pages there are in total),
> and even if you can go back to previous page.

Same. I also think "wizard" is a bad idea for this- it would make it
look like some sort of online test :p

> I'd prefer to create a better information about survey upfront.  We say
> that all questions are optional ("Note that you may skip questions as
> you like").  It is also stated that you can fill only a part of survey,
> and later go back to finish it... hmmm, I wonder if those cases where
> one edited his/her survey responses multiple times are counted as one
> finished survey, but multiple views.

Sure. Again, we should perhaps email survs.com for such
clarifications.

> We could also write how much time it takes on average to fil the survey.

Sounds good.

> > The average time spent on the survery is 34 minutes
> 
> It would be interesting to have more detailed statistics of time spent
> on the survey that only average time, a single number.  When one is
> filling open-form essay-length question, it would obviously take much
> more time than for one who doesn't.

Hm, yes. I didn't think about this.

> > - I think we can bring that down to 10~15 minutes if we design
> > questions to extract more information. Also, there's little incentive
> > for taking the survey: while many companies actually give out
> > discounts/ coupons for taking surveys, the least we can do is present
> > real-time results in the most interesting manner possible ie. survey
> > takers should see the "results so far" immediately after taking the
> > survey; some visualizations such as pie charts?
> 
> What we can do is after finishing the survey to redirect to the
> survey analysis page:
> 
>   https://www.survs.com/results/33Q0OZZE/MV653KSPI2
>   http://tinyurl.com/GitSurvey2010Analysis
> 
> instead of IIRC currently used redirect to
> 
>   https://git.wiki.kernel.org/index.php/GitSurvey2010
> 
> 
> As far as I know Survs.com doesn't provide any API for extracting data
> or survey statistics required for creating such visualization.  Neither
> we have a place where such app could be created, I think.

Oh, ok- we should take too much trouble though; we should keep
cost-to-benefit ratio in mind.

> > In questions 5, 10, 12, 13, 16, cut down out the options that have
> > very few respondents and let them all go into "other". It probably
> > doesn't actually save the survey taker any time, but I think seeing a
> > long page with many options can be scary.
> 
> The "5. Which Git version(s) are you using?" is not that long.  We could
> create a cut off a bit earlier, perhaps on 1.4 (i.e. have "pre 1.4") or
> even earlier, we could remove alternate implementations answers 
> (git-bigfiles, JGit, other implementations), or even concatenate 'master',
> 'next', 'pu' into single response... but would it buy us much?
> 
> The other side of removing choices, relying instead on "other, please
> specify" response is that it makes it harder to analyze results of survey:
> different people use different words for the same thing (and there are
> also spelling mistakes), and make results less reliable: people do not
> fill "other" if there is at least partial match, or do not know how to
> specify their version.

Ofcourse. We should only remove options that don't necessarily tell us
very much.

> We can remove those choices in "12. What Git GUIs (graphical user
> interfaces) do you use?" that got less than 1% rounded, or less than
> 10 responses.  On the other hand some people stated earlier that the
> list of possible choices in the survey (not necessarily about this
> question in specific) serve as reminder / information about possible
> choices.

Hm, I hadn't thought of that. I suppose it can be used to advertise
various applications.

> The other side of removing options from "13. Which git hosting site(s)
> do you use for your project(s)?" is that when sending requests to 
> announce the survey to those git hosting sites that are not on this
> list, some of them requested to be added (which is impossible after
> starting the survey; and before survey begins it is little sense to
> send announcements).

I see. How are we going to tackle this in future?

> Besides all of those below 1% rounded (Codesion, GitFarm, The Chaw,
> CipherHive) are also those that I didn't get response to request for
> announcing Git User's Survey 2010...

Interesting.

> We could make it more organized though, e.g. by sorting list of options
> alphabetically, or something like that.

Sounds good.

> > 1. Country of residence: we can probably make this a nice click-on-map
> >    interface as opposed to freeform text. It'll be more useful to us,
> >    and more interesting to users when we advertise the results.
> 
> It would be nice to have click-on-map (Google Maps or Bing Maps based),
> something like Ohloh provides, resulting in map of survey responders
> similar to the map of git users and git contributors on Ohloh
> 
>   http://www.ohloh.net/p/git/map
> 
> it isn't something that Survs.com offers currently.  I can only ask for
> it to be provided...
> 
> 
> Another solution would be to have pre-filled combo box (<select> field)
> with the list of countries to choose from, with GeoIP ised to pre-select
> the country.  I can generate list of all countries myself
> 
>   $ perl -MLocale::Country \
>     -wle 'print join("\n", sort (all_country_names()))'
> 
> as far as I know Survs.com doesn't offer GeoIP nor any API to hook it
> to survey questions.

I suppose we could always work out a way to display the results from
the information Survs.com gives us.

> > 2. Age: Maybe we restrict the input to 2-digit integers and draw a
> >    graph with all these integers to show a mean, median etc?
> 
> Restricting input doesn't give us much.

I just meant it as a sanity check in case people enter "34 years old"
and the like.

> There is nice histogram of responder's age for Git User's Survey 2009
> 
>   https://git.wiki.kernel.org/index.php/GitSurvey2009#02._How_old_are_you_.28in_years.29.3F
> 
> and tabularization of responses.  We can calculate mean, median, mode
> (aka modal score, i.e. most common response), perhaps after eliminating
> outliers, but would it give us much information?

Nice histogram! How did we manage to do this in 2009? Did we use a
custom-made application to do the survey?

> > 11. Just change this to an optional sometimes/ often? Why should users
> >     spend time clicking on "never"?
> 
> "Never" is here because you can't un-click response in given row.
> I'm also not sure how not answering row is represented in the export
> of survey data (which we use for further analysis).

Oh, right. I didn't think about this.

> > 17, 18: Merge perhaps?
> 
> Those questions are split because of limitation of Survs.com; the 
> "other, please specify" results in limited width _text field_, while
> it is much easier to write in-depth response in large _textarea_ field
> that wuestion 18 provides.

Ah, I thought so.

> > Ofcourse, I understand that there must be some technical constraints
> > due to which some things are not implementable (eg. survs doesn't
> > provide the feature?), but I've not taken that into consideration.
> 
> Note that as it currently stands we can use Survs.com account only for
> 2011 survey, provided that it is done earlier than this year (perhaps
> 1 June -- 31 July?), as our Premium account which we got thanks to 
> generosity of Survs.com admins (after Survs.com got out of beta) will
> downgrade to the Free plan (which is offersn much too low limits)
> on Sep 22, 2011.

I see. Any thoughts on long-term plans? Do we pay for the premium
account or do we build a custom application?

-- Ram