CS Rarity List [WIP] - status update on page 75

[Law]

Hi everyone, glad to see this thread moving! I'm on kinda shared/limited internet right now, so I'm going to look into amazon aws cloud to run the script (also need to remember how to run it, its been a while LOL). It might take a bit to set up, and I've been busy with work/life. I'll try to get this done as soon as I can :)

[nickjr]

Hi everyone, glad to see this thread moving! I'm on kinda shared/limited internet right now, so I'm going to look into amazon aws cloud to run the script (also need to remember how to run it, its been a while LOL). It might take a bit to set up, and I've been busy with work/life. I'll try to get this done as soon as I can

Good to see you

[Solloby]

I super appreciate that Law <3
If you ned me to help run the script I'm happy to do so, would just need instructions c:

[nickjr]

I am officially taking it lighter this semester because of my health, which means MORE CHICKEN SMOOTHIE TIME MUAHAHA--coughs I mean. More time to relax and not be stressed about school, of course

(it wasn't official until today ahaha)

I can also run the script if you give instructions! Sol and I (and whoever else wants to) can probably split up the IDs we'd do. (Since we're basically taking censuses of the rereleases, that's... a LOT of pets, so splitting up the IDs among 2+ people seems like a good idea if we can.)

[Solloby]

If you've looked at my Master Data with Stats excel file, that is how I am planning to build what is essentially a rarity list.

So if anyone's interested, here is the plan:
1. Collect Rerelease Data and add to Master Data
2. Sort pets by rarity (simple Excel sort)
3. Use Chi-square tests to determine which pets can be considered to have the same rarity
4. Use step 3 to build a rarity list draft containing July, Aug, Sept and UR pets only
5. Identify non-UR pets released from Oct onwards that need adding to the list and determine how we want to handle them (preferably through rarity changes archive, I am hoping we don't have to go back and count them but we may have to).

Does that sound logical?

[nickjr]

Can you elaborate on step 3? Otherwise it looks good. For step 5 maybe consult with DD and look at the draft that DD and Lya made because I thiiiiink they finished combing through the Rarity Changes thread page by page? So unless I'm wrong about that, their draft is basically a complete Rarity Changes Archive lol

[Solloby]

I started Step 3 in the Master Data with Stats excel file I uploaded on the last page or so, then stopped once I realised we had a VR at the top of the list above the OMGSRs. I'm guessing the major thing the rereleases changed was the rats, as they had less adoptions originally compared to the dogs, but would have more rerelease adoptions since they were not intended to be so rare.

Yes DD and Lya's draft is probably best for that, along with the Rares List guide.

[nickjr]

Ohhh I see now.

kjadhkdjahHAHAHAHA I found a rerelease named "ratherlargebox"

ACCURATE

Anyway I grabbed the rough ID ranges for the rereleases (mostly from Law's post on page 38); since we're only looking at VR+, I figure rough ID ranges are fine for 2009 and beyond since we can just ignore the regular monthly/advent/etc pets on the 17th and 19th for 2009 and beyond

Ranges should encompass all rereleases and then some regular advent pets from the 17th and 19th. (so bigger than necessary)

Pet counts done with mental math estimates because my hands are tired

2008:
808451 - 822788 (exact ID range)
14k-ish pets

2009:
7268000 - 7326000??
Huh, I didn't realize the rerelease box was up into the 19th...
I RAN INTO SO MANY DECEMBER 2009 MONTHLY PETS AND EXTENDED RELEASE PETS AND HANUKKAH 2009 PETS WHILE HUNTING FOR THE LAST 2009 RERELEASE -FLIPS TABLES-
in fact I'm still not sure if I ever hit the last 2009 rerelease (7324203?)
58k-ish pets

2010:
25432000 - 25633000
201k pets

2011-12-18:
54673200 - 54831930
158k-ish pets

2011-12-24:
55494770 - 55648460
Went a little into the 25th... -groan-
154k-ish pets

2012:
85357870 - 85544870
187k-ish pets

2013:
114896000 - 115146000
250k-ish pets

2014:
152503100 - 152734100
231k-ish pets

2015:
190186300 - 190395300
209k-ish pets

2016:
224153000 - 224425000
272k-ish pets

2017:
255265000 - 255505000
240k-ish pets

I'm trying to convince myself that it's okay to take censuses of these even though they have off-account pets because we're not comparing these pets to the pets that aren't even rare enough to be rereleased. Something is really bugging me about including the off-account pets and I can't figure out what OTL it's probably nothing significant...

EDIT: Note that we won't be getting full counts of the zebras since they're not limited to December 18th releases. But I don't think we need to get counts of those lol they're supposed to be the rarest pets on the site

Also no full counts of the white Malk dragon (but... maybe closer? since they haven't been available in monthly adopts since they were available with the Malk adopts... idk)

And just to be comprehensive in this edit... no full counts of any store pets

Did I miss anything?

[Solloby]

Off account releases overemphasise the impact of the rereleases on the rarity list. Nothing we can really do about it though, afaik.

[nickjr]

Yeah, that's roughly what I was thinking @Sol

I'm looking at Law's script and I'm wondering if there's a way to check each page for one of the things that differentiates an off-account pet from an on-account pet? The only ones I can see for sure are the "Home ‹ Adopted pet" breadcrumb trail at the top of the page (on-account ones include the owner's name and the group the pet is in) and "Don't have a pet yet? You can adopt a pet of your own!" (off-account) vs "Don't have any pets yet? You can adopt your own!" (on-account, not yours) vs ... no equivalent for a pet you own haha

I know there's a way to use Python to scrape webpages but idk if it can be added to the script?

If it can't, we'll just roll with this

edit: I edited my post above a bit.

I think we're looking at running the script on 1.8 million pets.

We should definitely spread that out over multiple people and over a period of time LOL

Even if I reduced the ranges to be exact, I don't think I can chop off more than like 100k pets because I came pretty close to the exact ranges (usually only 1k or so off each end)