Archive Pagination Get
Several thousand 88x31 buttons ago I understood I needed to divide the archive into pages, and I did not do that because It Felt Like Work. Well, I am once again dragging myself into a depraved nest of code to slay the dragons I myself birthed into this world. (Translation: finally gonna update it.) Fact is, the archive is already over 50MB and NeoCities isn't getting any smaller. Hopefully this update will make it less overwhelming.
It's best to divide the pages by letter, rather than # of buttons. That means 27 pages. Yay. We'll keep things simple, and name them consistently and logically: 88x31-a, 88x31-b, etc.
Extremely tempting to use frames for this, just to drive people on Hacker News insane, but no. These need to be individual pages. I'll keep the 88x31 explanation header for context, and plorp the buttons below it.
The current code works by taking a big ol .json of neocitizen data and tossing it into a single template:
rendered_template = template.render(citizen_data=citizen_data,population=population,today_date=today_date,alphanumeric=alphanumeric)
The fix is to iterate over the alphabet, and render a page for each letter. Which is actually pretty easy (pardon the 🍝):
for letter in alphanumeric:
page = template.render(citizen_data=citizen_data,population=population,today_date=today_date,alphanumeric=alphanumeric,letter=letter)
if letter[0] == "#":
page_filename = '88x31.html'
else:
page_filename = '88x31-' + str(letter) + ".html"
with open(filename, 'w') as f:
f.write(page)
Now I'm not sure why I dragged my feet for so long, and I feel kinda silly about that.
💀 💀 💀DEATH BECOMES THEM 💀💀 💀
The other thing I needed to do is a deadlink check. Why does it matter? Because people actually cruise the archive looking for sites to follow, and there's no point wasting anyone's time with dead links when we have an easy way to check. Right? Right?? Right.
Currently, Crawlie only verifies if a site exists when the site button is added to the archive. Just clicking around, I landed on multiple dead sites (RIP) please stop deleting your sites I beg you.
Anyway I didn't feel... super great... about doing like 7,500 API calls to verify all these links. But according to 44Nifty's blog post on special sauce, 500k API calls gives a sauce penalty. So it seems 500k is what Kyle Drake deems excessive.
Therefore, the plan was to do this quickly under the cover of darkness, while Kyle was trapped in chatbot hell, and hope the site overlords didn't perceive me.
The tasks: pull a list of living neocitizens, check the url to see if they're still alive, then update the corresponding .json entry. This info is stored under a status key.
Active site:
DED site:
Maaaan this feels like work 😭 but ☝️☝️☝️☝️ we've heard that excuse before so onward!
Digging for Graves...
Once again, this really was not that much work. It took far longer to actually check howevermany thousands of urls.
graves = list()
with open(JSON_FILE) as f:
citizen_data = json.load(f)
for subdomain in citizen_data.keys():
if citizen_data[subdomain]["status"]:
url = "https://{0}.neocities.org".format(subdomain)
try:
site_data = nc.info(subdomain)
except neocities.neocities.NeoCities.InvalidRequestError as e:
citizen_data[subdomain]["status"] = False
print(("{} is dead").format(url))
graves.append(url)
with open(JSON_FILE, "w") as outfile:
json.dump(citizen_data, outfile)
This resulted in 392 corpses, assuming none of the names are typos. I am saddened by some of the names I recognize from my web wanderings. Let us have a moment of silence, to pay respect to our fallen neighbors:
🪦 4ivilo, 🪦 angelgirl134123, 🪦 artists-of, 🪦 atohacya, 🪦 avertedvision, 🪦 ayu-mi-x, 🪦 betah, 🪦 bojanthelibrarian, 🪦 cafemika, 🪦 caors, 🪦 cappyy, 🪦 conduit-7, 🪦 cosmopopcomics, 🪦 deathslingxr, 🪦 drhelvetica, 🪦 edn, 🪦 eightbriitt, 🪦 everlastingcontrast, 🪦 freesha2002, 🪦 fretnoize, 🪦 fundamentallyunloveable, 🪦 geniphony, 🪦 geno7, 🪦 gh0stprince, 🪦 goodgoobgooby, 🪦 greyasashe, 🪦 halssite, 🪦 hatto, 🪦 heart-soda, 🪦 helianthuspetal, 🪦 hermione-art, 🪦 herocore, 🪦 hydratriangle, 🪦 icg, 🪦 ihearttheundead, 🪦 incelperspective, 🪦 isfim, 🪦 jaysmicezone, 🪦 jonniecore, 🪦 junothesilly, 🪦 katzentrys, 🪦 kirby677, 🪦 kittydoll, 🪦 konbie, 🪦 konfetti, 🪦 lambdafun, 🪦 leanneu, 🪦 lightninglove, 🪦 limineonal, 🪦 line-space, 🪦 lisathepainful, 🪦 logicalwillow, 🪦 lukuak, 🪦 manjaruntu, 🪦 medulladeath, 🪦 megalomaniac, 🪦 milk2008, 🪦 miserabledolly, 🪦 mollysdndjournal, 🪦 mordzine, 🪦 mortalboy, 🪦 natolmo, 🪦 naxdot, 🪦 ninjabou, 🪦 omarlego, 🪦 otamadachi, 🪦 owlcollective, 🪦 peachpum, 🪦 perish-x, 🪦 pleurodelinae, 🪦 prosequitur, 🪦 quoderatdemonstrandum, 🪦 rebdotexe, 🪦 rh0mbus0fruin, 🪦 rozariosanguinem, 🪦 scheherazades-niche, 🪦 shitlist, 🪦 sightseer, 🪦 sillylittledog, 🪦 skeletalcomrade, 🪦 stefwithanf, 🪦 strawbearie, 🪦 superstarmations, 🪦 t0by-toxxik, 🪦 takkyuudou2003fan, 🪦 techrakatt, 🪦 techramancer, 🪦 terminalrot, 🪦 thatpupi, 🪦 theglittersalamango, 🪦 theisadoragroove, 🪦 theran, 🪦 they-walk-among-us, 🪦 thoughtcrimes, 🪦 timesmice, 🪦 toothman, 🪦 treefingerfilms, 🪦 untitledblog, 🪦 vendetta2525, 🪦 vivus, 🪦 vodkabinereb, 🪦 vomitboyz, 🪦 wardof, 🪦 wastelandimperiatrix, 🪦 wavecave, 🪦 webpage1990colourised, 🪦 weedpizza, 🪦 wicked-forest, 🪦 widepop, 🪦 wtrclover, 🪦 wyrmwhisper, 🪦 xxbunnyratxx, 🪦 xxxemobunnyxxx, 🪦 ziggycore, 🪦 1roomsurvival, 🪦 chuyas, 🪦 cloutsandrec, 🪦 coffin-corner, 🪦 cubicsimulation, 🪦 dfbw, 🪦 headcaze, 🪦 niconiconapster, 🪦 xenonical, 🪦 16-no-solitude, 🪦 angelnite, 🪦 br2k5, 🪦 eigenvoid, 🪦 kitz, 🪦 marzka, 🪦 schooltrenchcoat, 🪦 trench, 🪦 zaks-slaughterhouse, 🪦 cyberblank, 🪦 softmoon, 🪦 stardustdreamz, 🪦 tai7kmusic, 🪦 erythronium, 🪦 cyberangeldust, 🪦 dc-blog, 🪦 dunkingtruth, 🪦 fartmother, 🪦 galerexia, 🪦 goat-online, 🪦 h0pey0ng, 🪦 i-me-and-myself, 🪦 lauragrave, 🪦 luv-ghoul, 🪦 vasterror, 🪦 vampireautopsy, 🪦 barneysmind, 🪦 elsenn, 🪦 francess, 🪦 its11pmwhatamidoing, 🪦 kmartparkinglot, 🪦 sentimentality, 🪦 sunnysystem, 🪦 vilxdryad, 🪦 andrymeda, 🪦 chmuryverse, 🪦 fille-de-pierrot, 🪦 rindustrial, 🪦 angelpuppyclub, 🪦 angelscake, 🪦 awhe, 🪦 babybellcheese, 🪦 blooperblog, 🪦 cardcaptors, 🪦 cherrychocochan, 🪦 codecrusader, 🪦 deathbycats, 🪦 evemarie, 🪦 glittertown, 🪦 gnomescourt, 🪦 janpan, 🪦 jt802t, 🪦 juriettoo, 🪦 kojou, 🪦 ladytron, 🪦 lorekeeping, 🪦 lyricaltokarev, 🪦 majickss, 🪦 momolover, 🪦 much-ado-about-everything, 🪦 n64bug, 🪦 nyatchi, 🪦 oswalds, 🪦 pinocchiop, 🪦 princesslunagameblog, 🪦 qso404, 🪦 radiationcat, 🪦 radpage, 🪦 raspergine, 🪦 realitv, 🪦 sealparty, 🪦 snowenti, 🪦 sockspace, 🪦 sodagirl, 🪦 stickmanishere, 🪦 strangled, 🪦 teshief, 🪦 theariaeliot, 🪦 thenoxwitch, 🪦 tlaylor, 🪦 uglygirlswag, 🪦 uwuboa, 🪦 vashti, 🪦 vocaland, 🪦 waxynwane, 🪦 wiverscuomo, 🪦 xn--0s9h, 🪦 ygg, 🪦 andrewsstuff, 🪦 aomuroguna, 🪦 artiostudio, 🪦 axetrax, 🪦 berry-playground, 🪦 bluenight, 🪦 cineparaidiotas, 🪦 divsel, 🪦 federiefederi, 🪦 ghoulish-cinnamon-site, 🪦 glytch, 🪦 gone-girl, 🪦 heirofslime, 🪦 hypercomplex, 🪦 kengo, 🪦 ketraline, 🪦 ketsuu, 🪦 lovelttr, 🪦 lunatic-writing-projects, 🪦 nekatina, 🪦 nekhnona, 🪦 nicolasperez, 🪦 omnis-unfair-mind, 🪦 pawzbutton, 🪦 primaballerina, 🪦 randomzigzag, 🪦 sillivis, 🪦 skellsparty, 🪦 somecaninething, 🪦 spacetroll, 🪦 squareroot, 🪦 sunnyapples, 🪦 tipheret, 🪦 typoism, 🪦 uglypsyche, 🪦 vampirology, 🪦 webcatz, 🪦 yuahouse, 🪦 2ainnet, 🪦 beezlebabe, 🪦 chocov4mp, 🪦 circustiger, 🪦 cornpop397, 🪦 dansdungeon, 🪦 doompy, 🪦 emoalien, 🪦 exodusfleet, 🪦 extensioncord, 🪦 funkydealer, 🪦 funwplushtrap, 🪦 g00nsgarbage, 🪦 goofysillygoober, 🪦 izabellaofcastile, 🪦 jackalopes, 🪦 jlon, 🪦 jubiland, 🪦 koiitann, 🪦 linksfem, 🪦 madewithrealsugar, 🪦 mikumyung, 🪦 nancer, 🪦 neogeist, 🪦 rosie-eclairs, 🪦 rymin, 🪦 schooltown, 🪦 silversquid, 🪦 soapfriendo, 🪦 sodiecake, 🪦 thecohort, 🪦 trenty-brenty, 🪦 vinylwolf, 🪦 walpurgizt, 🪦 xxxiwudnvrstopuxxx, 🪦 07151129, 🪦 aetherie-99, 🪦 angelmorningstar, 🪦 baileylockheart, 🪦 bibliohound, 🪦 biocatzard, 🪦 bisexualism, 🪦 chaoticdreamz, 🪦 cybertomboy, 🪦 dariusaur, 🪦 duskspirals, 🪦 epicvgmusic, 🪦 f-t-p, 🪦 fishstew, 🪦 frogobongo, 🪦 futen, 🪦 g0shiki, 🪦 gammagoop, 🪦 ghostnoire, 🪦 liliyoa, 🪦 mayfl0wer, 🪦 mazetthew, 🪦 menheraaudino, 🪦 minnowpond, 🪦 mrsaturn, 🪦 obsession-central, 🪦 pankines-zone, 🪦 parhelictriangle, 🪦 podminton, 🪦 polybiuzz, 🪦 savior-god, 🪦 swoesight, 🪦 techrahouse, 🪦 terminationevent, 🪦 thesolarstudio, 🪦 vetulicolia, 🪦 vomitdistrict, 🪦 waloeders, 🪦 whitepeach, 🪦 3dsangel, 🪦 captain-goofball-kandi, 🪦 christinee, 🪦 dannkestreet, 🪦 dykeism, 🪦 exhibitd, 🪦 germpills, 🪦 hgari, 🪦 jayzzztv, 🪦 jonathn, 🪦 juneish, 🪦 kaiserpug, 🪦 kekkon2puro, 🪦 loosesocks, 🪦 luciuscorvus, 🪦 mehlancholia, 🪦 metallicscorner, 🪦 networkneighbourhood, 🪦 rotten-egghead, 🪦 slank1800, 🪦 snowibunni, 🪦 sparklewiziz, 🪦 sunkissed-feathers, 🪦 thegportal, 🪦 thejesusof, 🪦 toxic-revolution, 🪦 traversetown, 🪦 trendgender, 🪦 xxcrazyrainbow02xx, 🪦 zanefolder, 🪦 alysrealm, 🪦 bloodthirstymasquerade, 🪦 cupidspostalservice, 🪦 everybodysleeps, 🪦 nethazard, 🪦 sacred-gate, 🪦 thatwitecat, 🪦 drabunwolfcat, 🪦 winteryear, 🪦 applejar, 🪦 happylittlecomputerfag, 🪦 kiittypawbz, 🪦 kwason, 🪦 meowlimit, 🪦 polyphylactery, 🪦 sacredchoral, 🪦 tranceclickheart, 🪦 anarkrist, 🪦 griferssiters, 🪦 fairy-hermit, 🪦 ratteta, 🪦 webforever, 🪦 aspdestos, 🪦 keatongamer1248, 🪦 sweet-pea, 🪦 game-boy, 🪦 inwealorwoe, 🪦 bughug, 🪦 farikz, 🪦 greytext, 🪦 lovealone, 🪦 mshellfire, 🪦 tauruspawz, 🪦 bananaslurpee, 🪦 hellojoy, 🪦 nachowinnie, 🪦 netmagicalgirl, 🪦 hohi, and 🪦 olistormarts.
Rest in peace, beloved citizens.
Now. The weirdness. You can't go digging up a bunch of dead bodies without finding a little weirdness.
WISE FROM YOUR GWAVE
1979 appears to be a dead site, but when I check the site info, it does not throw the expected Invalid Request Error. The profile throws a 404 error. So what gives? 1979, are you dead or just sleeping? Or is this NeoCities weirdness under the hood?
What now
God, I hope nothing.
We've got a scraper. We've got pagination. We've got dead link finder thing. We've got a scraper. Did I already say that?
No more updates until next year!!! ✊
