CS Rarity List [WIP] - status update on page 75

[SabrinaB]

Ah i see, thank you
All this coding stuff is sounding like japanese to me (which i dont speak) so i was just curious why theh would be needed.
All clear now, thanks!
I hope he can figure it out

[Lya]

Why do we need off account pets?
I remember seeing tess (i believe) say they dont count dor the rarity tags

I need to show my boyfriend what an off account page looks like so that he can find something to help filter out the off account pets in the code. He doesn't play CS. ^^;

I guess you probably already found a way, but since I can't be sure, here's a possibility:

Make the program access the pets' pages. Then it can a) search for the string "Is this your pet?" or b) check if the pet's owner name is a hyperlink.
Since both the string and the hyperlink are only on pet pages of pets on accounts, everything without it could be sorted out as off-account.

[Erythro]

UPDATE TIMEEEEE

Good News
Boyfriend and I were able to sit down and look more into this code so here's the checklist of what we accomplished.
How to read on-account versus off-account ✓
How to read deleted pets and omit them ✓
How to read itemless pets ✓
How to count everything and the output ✓
How to turn that output into easy to read data for you guys to look at/analyze ✓

Bad News
Two major issues with what we were looking at: How to actually read the pages (downloading all the html files or opening 270 million pages, etc) but we can eventually figure that out, and the one that is going to screw us up.
When you add items to your pet, CS creates a new custom link for your image and only your image. We can't use the links of these images to determine what pet we are looking because they are unique to your pet. However, item-less pets can be counted using their static images (the images posted in the archive). It would take too much time/effort to create a code that compare image to image so we are going to have to go the unfortunate route. Since we are counting each and every pet (thanks to the random releases of zebras), we are having the code give us a text document of all of the urls of pets with items. We are going to have to manually count those but this does significantly decrease the number of pets we need to count. I'm sorry guys, I wish we could do more with the knowledge we have.

TL;DR Gotta figure out how to ergonomically count 270 million pets. All pets with items have to be manually counted.

[kee;]

Im glad there’s at least some good news!
If any of us can be of any help with the pets that have to be manually counted, please ask!

-kee

[EmberzTale]

I'd also be up for helping count pets manually!

[nickjr]

Glad to see you're alive and kicking! And that progress sounds great 8D

Law had some preprocessing thing and hard-coded thing that went something like this:
> get noitems=1 image for a pet ID
> chop off bottom rows of pixels (the owner tag)
> compare the rest of the pixels to hard-coded reference list of pets

Hard-coded reference list of pets was done like the above for each pet e.g. take any noitems=1 image of a sunjewel, chop off the owner tag, label remaining image as "Sunjewel"

This is a terrible explanation uuugh on phone OTL hope it helps?

[ElementalInsanity]

Glad to see some progress even if I can’t understand anything being said lol. =‘)
If I have time when the counting begins I would definitely be open to help.

[daryll]

Hello, boyfriend here.

The program can do the following:

• Identify ON-ACCOUNT/OFF-ACCOUNT
• Identify deleted pets
• Identify item-less pets
• Go through every HTML link downloaded and saved in the directory (I was told there are 270M+ pets - storing 270M pages is 4.5TB)

The way I'm approaching this is taking the HTML link for each pet (ID 1, 2 ...) and checking for specific unique strings.

The above mentioned specific unique strings in each HTML file:

Code: Select all

static final String ONLINE = "your account to get HTML";
static final String OFFLINE = "Don't have a pet yet";
static final String STATICIMG = "<img src=\"https://static.chickensmoothie";
static final String ITEMSIMG = "<img src=\"/pet/";

If you know of a better string identifier, please let me know as I can change these constant variables easily.

This is fine and dandy but identifying the type of pet from the image is extremely difficult without a key/legend.
Pets with the static, hard-coded reference links are easy to identify because I can compare it to known strings.

Output
I have not started the output yet as I have only (partially?) completed reading from the HTML file.

My girlfriend has told me that I should:

• Identify the type of pet
• Identify ID of the pet

ID is easy; for both item & item-less pets, the IDs are within the URL links.

Not easy: identifying type of pet (if it has items or a custom URL)

I can separate item & item-less pets with the current code.

Output can be formatted to your liking - current plan for this:

Code: Select all

ID: ##
Type: ____


NOTE: Type will only work for hard-coded reference links --- I cannot figure out a way to identify images that do not have a hard-coded reference link



Also: the work around for 270 million pages is to maybe do 1000 links at a time


Problem with this: each HTML link NEEDS to be re-named as the program is reading the file name and is incrementing by 1 after each URL

Code: Select all

Scanner input = new Scanner(new File("ID"+n+".html"));


Where "ID"+n is the file name

What I could do is save every file the same as Windows will concatenate (1), (2), (3) ... at the end of the file name
I will try this later this afternoon, that way there is no need to re-name links.
02:13 PM: Each URL is saved as the name of the pet and owner; I will have to manually update every name when downloading

Also #2: I assume that once the pet is logged and saved into the output, you do not need to re-visit the pet (so I can delete the read-through HTML links) - Let me know if this is true

[nickjr]

Hi, daryll!

I think those strings look good. We could get false positives if someone decided to put those strings into their username or their pet's name, but... given what those strings are, I find that really unlikely lol (worth noting that pet names can have valid HTML without problems, although it'll just show up as plaintext)

... I think we might also get false positives during events, which we should take into consideration considering we're gonna be looking at 270 million pages spread out over time -gulps-. Every time we load a page during an event, there's a chance that the page will load with a banner (just a regular image) at the bottom of the page, and banner images are also part of static.chickensmoothie.com (examples:
Summer 2017 - https://static.chickensmoothie.com/pic. ... 680B578640
Summer 2016 - https://static.chickensmoothie.com/tran ... O6Ac8_tGqg
I'm pretty sure I got these links from banners that I was about to click). I don't know if there's other stuff in the <img> tags for banners.

The banners don't show up outside of events, though. So maybe the program can just be run outside of events if you don't want to add additional checks to the program. Or we could hope that the next event has trans in the url like it did for Summer 2016 haha
edit: or you could just refresh the page if it has a banner until there's no banner (or just claim the banner lol) before you save the page, but I feel like that might get mentally tiring to have to constantly be on alert for a banner

I assume that the deleted pets are identified by process of elimination (having none of those string constants on the page)? Maybe another workaround for the banner-staticimg conflict is to identify staticimg pets by elimination instead, since deleted pets show this message:
"Sorry, that pet could not be found. Maybe you can adopt a new pet instead :)."

I'm not sure how much this suggestion will help, but maybe you can narrow down the key/legend of pet types of compare against based on a pet's adoption date*? For example, if the pet was adopted in March 2009, then you don't have to compare it against July 2008 pet types like the Sunjewel, Moonswirl, etc. right?** (Exact boundaries might be difficult to determine, especially for 2008 pets, and you'll have to be comparing to basically the full list for any December 18 pet as well as some December 19, 2009 and December 24, 2011 and December 25, 2011 pets, but still)

* we shouldn't do pet ID because of recreated pets (we can't count the actual recreated pets unless they were recreated before 2012, but oh well, I guess?)

** this means we'll miss that October 2008 Sunjewel that does have a rarity as well as... some banana floating around from a non-rerelease date with a rarity I think?, but eh, that's so infrequent that I can probably count how often it happened on just my fingers (whereas recreated pets, while still super rare and probably still negligible, are a little more common than the real oddballs I just mentioned)

This is a mess and not nearly as substantial as the quantity of text suggests oh god I'm sorry

edit: Re: file name
Is there a way to automate the file names or something? Like maybe have something go through the files in chronological order of when they were saved, and rename every file to be either the same (to get Windows to add the (1) (2) etc) or directly rename to what you want?

and yeah, I don't think we'll need to look at the pet pages again once we know what type of pet an ID is

[Erythro]

Nick, I think you have a brilliant idea.

We could have a code to count up to the end of July release date, a code to count up to August release date, and so on as well as codes to count rereleases?

Instead of all of the threads looking for UR/OMGSR/etc, we can have a thread dedicated to zebras so that people can post the zebras and we can seperate them by rerelease/special release since the special releases are the ones released during horse litters. We can manually add those into our count.

This would save us so much time when it comes to this.

My only concern is pets that were recreated like the March 2011 pony here that sit in 2012 IDs which if were a list pet, we would miss.