Register - Login
Views: 96133379
Main - Memberlist - Active users - Calendar - Wiki - IRC Chat - Online users
Ranks - Rules/FAQ - Stats - Latest Posts - Color Chart - Smilies
12-10-18 07:59:13 PM

Jul - Meta - Search indexing New poll - New thread - New reply
Pages: 1 2Next newer thread | Next older thread
Xkeeper

Level: 251


Posts: 23358/24710
EXP: 251089660
For next: 2950216

Since: 07-03-07

Pronouns: they/them, she/her, etc.

Since last post: 20 hours
Last activity: 59 min.

Posted on 09-11-18 02:15:11 PM Link | Quote

robots.txt and the wayback machine

I heard they were ignoring it for government-related websites, but were not actually ignoring it on a general scale at this time. As far as I know, that's been the status for two years at this point.
Sanqui
1760
πŸ¦‰
Level: 78


Posts: 1747/1761
EXP: 4230757
For next: 151469

Since: 12-20-09

Pronouns: any
From: Czechia (NEW!)

Since last post: 28 days
Last activity: 7 hours

Posted on 09-12-18 04:26:45 PM Link | Quote
My reflex was to be opposed to this change. Then I thought "well maybe threads made in the past year could be indexed?". But ultimately, I think I end up being in favor. It's definitely something worth thinking about. What an interesting time.

As for IA, I can always archive the whole of jul with archivebot which ignores robots.txt, ifneedbe
Sanqui
1760
πŸ¦‰
Level: 78


Posts: 1748/1761
EXP: 4230757
For next: 151469

Since: 12-20-09

Pronouns: any
From: Czechia (NEW!)

Since last post: 28 days
Last activity: 7 hours

Posted on 09-12-18 04:42:30 PM Link | Quote
I am proud to be a part of a forum that is now unplugged. Delisted. Unindexed. Our own little hidden island in the terrible ocean of Google.
Xkeeper

Level: 251


Posts: 23358/24710
EXP: 251089660
For next: 2950216

Since: 07-03-07

Pronouns: they/them, she/her, etc.

Since last post: 20 hours
Last activity: 59 min.

Posted on 09-12-18 04:48:20 PM Link | Quote
Originally posted by Sanqui
My reflex was to be opposed to this change. Then I thought "well maybe threads made in the past year could be indexed?". But ultimately, I think I end up being in favor. It's definitely something worth thinking about. What an interesting time.

As for IA, I can always archive the whole of jul with archivebot which ignores robots.txt, ifneedbe

At that point I'd sooner just provide an archive of the database, but with certain things scrubbed. Private messages, passwords, restricted forums -- basically stuff that isn't public anyway. (But I'd only want it in trusted hands, not publicly searchable, otherwise it kind of defeats the point.)
Sanqui
1760
πŸ¦‰
Level: 78


Posts: 1749/1761
EXP: 4230757
For next: 151469

Since: 12-20-09

Pronouns: any
From: Czechia (NEW!)

Since last post: 28 days
Last activity: 7 hours

Posted on 09-12-18 05:02:44 PM Link | Quote
Who joins after finding this board from Google, anyway?! I joined through a forum signature and I'm sure most people join by the word of mouth -- most recent influxes being from Twitter and Mastodon advertising. I say out with the late search engine
sofi

🌠
Level: 106


Posts: 3814/3821
EXP: 12614409
For next: 57534

Since: 02-18-11

Pronouns: she/her
From: γŸγΎγ”γ£γ‘ζ˜Ÿ

Since last post: 5 days
Last activity: 1 day

Posted on 09-12-18 08:14:31 PM Link | Quote
i think you did the right thing tbh

i had to go through and delete or edit several old posts because of sensitive information they contained. i used to openly talk about some incredibly personal things here and people could find it if they searched
Rambly

Level: 87


Posts: 1987/2110
EXP: 6250549
For next: 142225

Since: 07-22-07

Pronouns: she/her

Since last post: 2 hours
Last activity: 2 hours

Posted on 09-12-18 08:52:59 PM Link | Quote
are we deep web now



honestly i never felt comfortable being fully open about myself here because i knew it was all indexed on google, so i wholeheartedly approve this change. probably won't change my habits, but it's still a good measure

...i didn't use google to search very often anyway; usually i'll literally just go back on thread listings to around the time i think something was posted and ctrl+f for the thread i'm looking for

not exactly the most, uh. efficient means of searching. but it works well enough for the times i need something specific from long past, which isn't often
Xkeeper

Level: 251


Posts: 23358/24710
EXP: 251089660
For next: 2950216

Since: 07-03-07

Pronouns: they/them, she/her, etc.

Since last post: 20 hours
Last activity: 59 min.

Posted on 09-12-18 11:21:13 PM Link | Quote
Goog and friends are still crawling (because in a twist of utter irony, you have to let them crawl pages to know that they aren't supposed to crawl pages, what the hell) but they should no longer save, index, or cache anything. Probably.

I'm blocking other bots as I run across them, because there's an awful lot of misbehaving ones out there.
Pages: 1 2Next newer thread | Next older thread
Jul - Meta - Search indexing New poll - New thread - New reply




Rusted Logic

Acmlmboard - commit 220d144 [2018-11-04]
©2000-2018 Acmlm, Xkeeper, Inuyasha, et al.

25 database queries.
Query execution time: 0.199709 seconds
Script execution time: 0.012360 seconds
Total render time: 0.212069 seconds