Register - Login
Views: 95454251
Main - Memberlist - Active users - Calendar - Wiki - IRC Chat - Online users
Ranks - Rules/FAQ - Stats - Latest Posts - Color Chart - Smilies
10-19-18 01:40:40 PM

Jul - Meta - Two suggestions both regarding useragents New poll - New thread - New reply
Next newer thread | Next older thread
Jamie

Level: 15


Posts: 23/53
EXP: 15428
For next: 956

Since: 06-03-14

Pronouns: he/him
From: England

Since last post: 4 hours
Last activity: 1 hour

Posted on 09-21-18 03:15:19 AM Link | Quote
This was discussed earlier awhile ago in the IRC, I remember pinging XK and kak about it: could "Mobile" and "Phone" become a catch-all for triggering the mobile view? I think it appears in most UAs regarding mobile devices (iOS and Windows Phone respectively).

Also, wouldn't an effective way of blocking spiders be to ban the useragent matching "bot" or "spider" at .htaccess level (or whatever this server is using)?

Xkeeper

Level: 244


Posts: 23358/23358
EXP: 229317926
For next: 869340

Since: 07-03-07

Pronouns: they/them, she/her, etc.

Since last post: 16 hours
Last activity: 12 hours

Posted on 09-21-18 10:19:51 AM Link | Quote
In an unamusing ironic twist, bots have to be able to crawl pages to know that they aren't allowed to crawl pages.

I wish I was fucking kidding.
Kak
heh
Level: 73


Posts: 1808/1810
EXP: 3331963
For next: 153905

Since: 09-03-13

From: ???

Since last post: 18 hours
Last activity: 17 hours

Posted on 09-21-18 10:48:05 AM Link | Quote
Bots also won't necessarily report themselves as such.

(Though well known spiders like GoogleBot or BingBot do, soooo....)
StapleButter
Member
Level: 42


Posts: 491/500
EXP: 507795
For next: 13567

Since: 02-24-13

From: your dreams

Since last post: 3 days
Last activity: 2 days

Posted on 09-21-18 10:56:59 AM Link | Quote
although completely denying them access yields the same result, ie they aren't able to crawl and index shit
Rena

Star Mario
Fennel
Level: 129


Posts: 5256/5257
EXP: 24427273
For next: 622381

Since: 07-22-07

Pronouns: he/him/whatever
From: RSP Segment 6

Since last post: 27 days
Last activity: 6 days

Posted on 09-21-18 01:45:11 PM Link | Quote
Well good bots will obey robots.txt and have a proper user agent string, and bad bots... are bad.

Anyway it occurs to me that there doesn't appear to be a way to see a user's title (or ban message) on mobile.
Xkeeper

Level: 244


Posts: 23358/23358
EXP: 229317926
For next: 869340

Since: 07-03-07

Pronouns: they/them, she/her, etc.

Since last post: 16 hours
Last activity: 12 hours

Posted on 09-21-18 03:48:50 PM Link | Quote
Originally posted by StapleButter
although completely denying them access yields the same result, ie they aren't able to crawl and index shit

In the case of Google and friends, they still have earlier non-blocked versions of your website and will continue to use that in search results, even if they are blocked in the future.

The eventual plan is to outright deny everything, but due to this bullshit they need to crawl to see the "no index, no cache, no follow" tags.
Rena

Star Mario
Fennel
Level: 129


Posts: 5257/5257
EXP: 24427273
For next: 622381

Since: 07-22-07

Pronouns: he/him/whatever
From: RSP Segment 6

Since last post: 27 days
Last activity: 6 days

Posted on 09-21-18 04:43:53 PM Link | Quote
Wait, why do we want to block Google?
Xkeeper

Level: 244


Posts: 23358/23358
EXP: 229317926
For next: 869340

Since: 07-03-07

Pronouns: they/them, she/her, etc.

Since last post: 16 hours
Last activity: 12 hours

Posted on 09-21-18 06:01:26 PM Link | Quote
Thankfully I can just direct you to the answer
Jamie

Level: 15


Posts: 26/53
EXP: 15428
For next: 956

Since: 06-03-14

Pronouns: he/him
From: England

Since last post: 4 hours
Last activity: 1 hour

Posted on 09-24-18 09:23:19 AM Link | Quote
Originally posted by Xkeeper
Originally posted by StapleButter
although completely denying them access yields the same result, ie they aren't able to crawl and index shit

In the case of Google and friends, they still have earlier non-blocked versions of your website and will continue to use that in search results, even if they are blocked in the future.

The eventual plan is to outright deny everything, but due to this bullshit they need to crawl to see the "no index, no cache, no follow" tags.

I thought eventually they just disappeared, after a while, but you're more than likely right on this.

Also a lot of abuse bots (ie ones that ignore robots.txt) still use bot at the useragent level - AhrefsBot and Semrush being two examples
Xkeeper

Level: 244


Posts: 23358/23358
EXP: 229317926
For next: 869340

Since: 07-03-07

Pronouns: they/them, she/her, etc.

Since last post: 16 hours
Last activity: 12 hours

Posted on 09-24-18 03:59:27 PM Link | Quote
Eventually I plan on updating it to outright block bots, but for right now you could say this is something of a transitional period.
Next newer thread | Next older thread
Jul - Meta - Two suggestions both regarding useragents New poll - New thread - New reply




Rusted Logic

Acmlmboard - commit 5d36857 [2018-03-03]
©2000-2018 Acmlm, Xkeeper, Inuyasha, et al.

29 database queries, 1 query cache hits.
Query execution time: 0.174947 seconds
Script execution time: 0.019686 seconds
Total render time: 0.194633 seconds
Memory used: 1048576