Thank you to those who donated to Piano Society in 2017.

problems with statistics

Discussion in 'General' started by Francois de Larrard, May 9, 2010.

  1. hanysz

    hanysz Member

    Joined:
    Dec 27, 2010
    Messages:
    244
    Likes Received:
    0
    Occupation:
    Pianist
    Location:
    Adelaide, Australia
    Home Page:
    I'd be willing to look at the source code and see whether I'm able to help. (The answer might well be "no", since it's about 10 years since I last programmed in C. But I'm kind of curious as to how much I can remember, and it costs nothing to look.) Message me if you want to follow up on this; it's probably not a good idea to post the source code in a public forum.
     
  2. techneut

    techneut Active Member Piano Society Artist

    Joined:
    Jun 12, 2006
    Messages:
    9,927
    Likes Received:
    2
    Location:
    Netherlands
    Last Name:
    Breemer
    First Name:
    Chris
    Thanks for the offer, but even if C coding was your daily bread today this would not help so much. It's not that the C code is bad or buggy, it's just that the external conditions have changed. We now have a different way of handling the access logs, and the program needs to be adapted to that. As well as simplified so that I can still understand my own logic after a couple of months :D

    But if you're just curious I can mail you the source code.
     
  3. hanysz

    hanysz Member

    Joined:
    Dec 27, 2010
    Messages:
    244
    Likes Received:
    0
    Occupation:
    Pianist
    Location:
    Adelaide, Australia
    Home Page:
    Indeed I am curious :)
     
  4. techneut

    techneut Active Member Piano Society Artist

    Joined:
    Jun 12, 2006
    Messages:
    9,927
    Likes Received:
    2
    Location:
    Netherlands
    Last Name:
    Breemer
    First Name:
    Chris
    I am now under steam with rewriting the code as needed. It's starting to do something again. I'll send you the code when it's ready.
     
  5. techneut

    techneut Active Member Piano Society Artist

    Joined:
    Jun 12, 2006
    Messages:
    9,927
    Likes Received:
    2
    Location:
    Netherlands
    Last Name:
    Breemer
    First Name:
    Chris
    The stats program is shaping up again, I think I'm almost ready to regenerate the full stats from february 2010 until now. I more or less needed to overhaul the entire C code. The cumulative stats has to process ~7 million downloads and sort them in various ways. This was eating up memory like crazy and I had to re-think the way this enormous wad of data was stored in memory. Had some terrible time today with untraceable heap corruptions but luckily they disappeared after I simplified a certain part of the code :D
    All in all quite a project but the end seems near now. Would be great to get back to the piano again :roll:
     
  6. Francois de Larrard

    Francois de Larrard Administrator Staff Member Piano Society Artist Trusted Member

    Joined:
    Nov 30, 2008
    Messages:
    743
    Likes Received:
    3
    Occupation:
    Researcher
    Location:
    Lyon, France
    Home Page:
    Last Name:
    Larrard
    First Name:
    Francois de
    Good to read that, Chris. Don't fall in the last turn !
     
  7. hanysz

    hanysz Member

    Joined:
    Dec 27, 2010
    Messages:
    244
    Likes Received:
    0
    Occupation:
    Pianist
    Location:
    Adelaide, Australia
    Home Page:
    Thanks Chris for doing this! I hope it hasn't been too unpleasant a task.
     
  8. techneut

    techneut Active Member Piano Society Artist

    Joined:
    Jun 12, 2006
    Messages:
    9,927
    Likes Received:
    2
    Location:
    Netherlands
    Last Name:
    Breemer
    First Name:
    Chris
    At long last, the download statistics have risen from the ashes. It took me a long time to get all the debris cleaned up and start with a clean slate. We now have statistics dating back to feb. 2010. I have once more added the 'Most active downloaders' column, from which we can see (by clicking the links) that many of the power downloaders are located in China.

    A few words about interpreting these numbers.

    First of all I make no guarantee that these numbers are 100% accurate. The amount of data trawled is enormous and I have to make educated guesses on what to process and what to ignore. These numbers should be seen as more than an indication.

    Next, I strongly suspect that there are many sites who regularly scrape ALL recordings from our site. Because of that, it stands to reason that whoever has the most recordings is always the most downloaded artist. In fact I think these numbers are directly related to the number of tracks on the site. For example my download figure number is roughly 3 times higher than that of Monica, and I have roughly 3 times as many recordings on the site. So really these figures do not prove a lot other than the law of large numbers.

    Also, I suspect there are certain sites that have a 'recording of the day/week/month' and play it over and over, directly from the site, thus skyrocketing the numbers for that recording and artist. For example during may, june, and july 2010 someone was really fond of one little item from my Mikrokosmos cycle. In december, an old recording of Chopin's op.66 tops the charts. There seems to be no rule to it, except it seems to happen more for people with lots of recordings. It would be interesting to correlate this to the power downloaders to see who's doing this. An exercise for another day.

    Currently, the nightly (midnight European time) process of generating the full stats takes about 15 minutes, and a lot of memory, during which time I could imagine some disruption of services. This will of course slowly increase over time. We'll have to see if and when that becomes a problem. I really need to think about a way to update the statistics each day instead of completely regenerate them. That's for another day also.

    If anybody finds any mistakes in the stats (missing/wrong links, duplicates, stuff that should not be there, etc.....) please let me know. I noticed in one page that Bach is listed twice, but with all the Bach transcriptions that doesn't surprise me and I can't be bothered to fix it. Things like double names make maintaining the stats a PITA.

    Hm, I see that now the New Recordings list is a bit lame... the artist name is missing. I think I know why and will fix that later today.
     
  9. Francois de Larrard

    Francois de Larrard Administrator Staff Member Piano Society Artist Trusted Member

    Joined:
    Nov 30, 2008
    Messages:
    743
    Likes Received:
    3
    Occupation:
    Researcher
    Location:
    Lyon, France
    Home Page:
    Last Name:
    Larrard
    First Name:
    Francois de
    Hi Chris,
    Thanks for the news about the progress of this big site. However, to date the stats are still not accessible, are they ?
     
  10. techneut

    techneut Active Member Piano Society Artist

    Joined:
    Jun 12, 2006
    Messages:
    9,927
    Likes Received:
    2
    Location:
    Netherlands
    Last Name:
    Breemer
    First Name:
    Chris
    Yes they are !
     
  11. hanysz

    hanysz Member

    Joined:
    Dec 27, 2010
    Messages:
    244
    Likes Received:
    0
    Occupation:
    Pianist
    Location:
    Adelaide, Australia
    Home Page:
    Statistics look great now, thanks Chris. Also looks like the new recordings list is OK again, complete with artists' names.
     
  12. Francois de Larrard

    Francois de Larrard Administrator Staff Member Piano Society Artist Trusted Member

    Joined:
    Nov 30, 2008
    Messages:
    743
    Likes Received:
    3
    Occupation:
    Researcher
    Location:
    Lyon, France
    Home Page:
    Last Name:
    Larrard
    First Name:
    Francois de
    OK, I've found how to access. Thanks for this important work, Chris.

    As you mention, some pieces seem to have an unexpected success (something I have long noticed). Also, I find that during december there were about 600,000 downloads, but only 221 downloaders, which makes about 3,000 downloads per downloader ! According to my calculation, if I count 4 minutes as a mean per recording, each one must spend 8x24h00 days to listen those 3,000 tunes... Hence there are only 11 downloaders who downloads more than 100 tunes. As you wrote, there seems to be some robots who are pumping continuously the site. The most efficient one (65.49.37.163) downloaded 121,598 tunes, while there are only about 5,000 recordings. When you love, you don't count :lol: !

    Well, finally we have to keep modest about the disemination of the site :? . How many real downloaders, that are people who brose the site, choose some tunes, donwload and listen them ? Apparently less than 200, so more or less as many as people registered as 'artists'. This is in line with the public we find in classical music concerts (at least in Europe): elderly people and younger people, most of them being musicians...

    But this does not decrease the value of this site and the people who maintain and feed it ! Viva la musica !
     
  13. techneut

    techneut Active Member Piano Society Artist

    Joined:
    Jun 12, 2006
    Messages:
    9,927
    Likes Received:
    2
    Location:
    Netherlands
    Last Name:
    Breemer
    First Name:
    Chris
    Actually I currently process only the first 300 or so IP addresses I encounter. I did not want that right column to go on forever, and wanted to conserve some memory. It could mean that some power downloaders, and maybe many 'normal' listeners are not shown, and it certainly means that the numbers don't add up as they should. I'll see if I can display them all, or otherwise refine this by showing only the 300 or so BIGGEST downloaders.

    Also, we should not assume that all these guys actually listen to the music. Mostly I think just steal them, for whatever reason best understood by themselves.
     
  14. hanysz

    hanysz Member

    Joined:
    Dec 27, 2010
    Messages:
    244
    Likes Received:
    0
    Occupation:
    Pianist
    Location:
    Adelaide, Australia
    Home Page:
    Indeed. Weird things happen on the internet. I've seen the Chopin Berceuse from my web site advertised as a "free ringtone" (at least it was free, they weren't trying to sell it for money). Not just a phrase or two, but the entire five minutes of it. I doubt that anyone is actually using it for that purpose.
     
  15. techneut

    techneut Active Member Piano Society Artist

    Joined:
    Jun 12, 2006
    Messages:
    9,927
    Likes Received:
    2
    Location:
    Netherlands
    Last Name:
    Breemer
    First Name:
    Chris
    Oh, most PS recordings are being misused as ringtones. These guys just don't care and grab everything they can get. My Iberia Evocacion seems to be very popular despite the nasty slip in the climax. But you never know that anyone isn't indeed using it. Myself, I have Mahler 8th for a ringtone. That will teach people trying to ring me :p
     
  16. Rachfan

    Rachfan Active Member Piano Society Artist

    Joined:
    Aug 5, 2008
    Messages:
    2,153
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Chief Operating Officer, retired
    Location:
    U.S.A.
    Last Name:
    April
    First Name:
    David
    This topic reminds me of a remark made by the famous American humorist Mark Twain. He said: "There are three kinds of lies--lies, damned lies, and statistics." :lol:

    In another related matter, I did see someplace on the Internet that my Catoire recordings were available for ring tones. What will they think of next? Maybe they could play the "Etude-fantastique" while attempting to decipher the intuitive icons, the settings, and "user friendly" features of the cell phone?

    David
     
  17. pianolady

    pianolady Monica Hart, Administrator Staff Member Piano Society Artist Trusted Member

    Joined:
    Jun 14, 2006
    Messages:
    8,712
    Likes Received:
    1
    Last Name:
    Hart
    First Name:
    Monica
    None of you ever said anything about the nice, new look of our site and forum. Chris put a lot of work into it. Looks great, doesn't it? :)
     
  18. musical-md

    musical-md Active Member Piano Society Artist Trusted Member

    Joined:
    Nov 29, 2010
    Messages:
    1,251
    Likes Received:
    1
    Gender:
    Male
    Occupation:
    Physician, Chief Medical Officer
    Location:
    Biloxi, MS, USA
    Last Name:
    Del Rio
    First Name:
    Eddy
    Yes Monica, I did notice (and liked very much) the new look. Thanks Chris!
     
  19. timmyab

    timmyab New Member

    Joined:
    Mar 9, 2009
    Messages:
    65
    Likes Received:
    0
    Location:
    Bristol, UK
    The site's looking very smart.Good to hear the stats are back again.
    Just one question.How do you find them?I'm sure they used to be at the bottom of the front page.
    Something else I've always wondered about.Is there a link on the board index page to the front page?I haven't found one in the time I've been here and I'm thinking that people who find their way to the forums via a search engine might not realize that there's more here than just the forums.
     
  20. pianolady

    pianolady Monica Hart, Administrator Staff Member Piano Society Artist Trusted Member

    Joined:
    Jun 14, 2006
    Messages:
    8,712
    Likes Received:
    1
    Last Name:
    Hart
    First Name:
    Monica
    We've (okay, Chris...) tried making a link to the home page, but couldn't do it.

    To get to the stats:
    1. Home page
    2. About Piano Society
    3. Recording Download Statistics
     

Share This Page