Register - Login
Views: 99393307
 Main - Memberlist - Active users - Calendar - Wiki - IRC Chat - Online users
Ranks - Rules/FAQ - Stats - Latest Posts - Color Chart - Smilies		 04-24-22 10:23:49 AM
Jul - Meta - Two suggestions both regarding useragents New poll - New thread - Thread closed
Next newer thread | Next older thread
Jamie
Requested (also Termingamer2-JD rereg)
Level: 29


Posts: 23/193
EXP: 143932
For next: 3953

Since: 06-03-14


Since last post: 2.8 years
Last activity: 2.8 years
Posted on 09-21-18 06:15:19 AM Link
 
This was discussed earlier awhile ago in the IRC, I remember pinging XK and kak about it: could "Mobile" and "Phone" become a catch-all for triggering the mobile view? I think it appears in most UAs regarding mobile devices (iOS and Windows Phone respectively).

Also, wouldn't an effective way of blocking spiders be to ban the useragent matching "bot" or "spider" at .htaccess level (or whatever this server is using)?



____________________
Layout by Jamie (Cirnozie).
My blog | My (derp)board
Xkeeper

Level: 263


Posts: 23358/25343
EXP: 296722316
For next: 2238137

Since: 07-03-07

Pronouns: they/them/????????

Since last post: 9 days
Last activity: 3 days
Posted on 09-21-18 01:19:51 PM Link
In an unamusing ironic twist, bots have to be able to crawl pages to know that they aren't allowed to crawl pages.

I wish I was fucking kidding.

____________________
(Lv 244 with 228534683 EXP)
Kak

...
Level: 80


Posts: 1808/1928
EXP: 4755100
For next: 27869

Since: 09-03-13

From: ???

Since last post: 60 days
Last activity: 57 days
Posted on 09-21-18 01:48:05 PM Link
Post #1808
Bots also won't necessarily report themselves as such.

(Though well known spiders like GoogleBot or BingBot do, soooo....)

____________________
--=[!]=--
Arisotura
Member
Level: 49


Posts: 491/614
EXP: 880012
For next: 3871

Since: 02-24-13

From: your dreams

Since last post: 93 days
Last activity: 51 days
Posted on 09-21-18 01:56:59 PM Link
although completely denying them access yields the same result, ie they aren't able to crawl and index shit

____________________
Kuribo64 -- NSMB2 hacking and other crap
Rena
I had one (1) message in Discord deleted and proceeded to make a huge, huge mess about how it was a violation of free speech and how moderators are supposed to be spam janitors and nobody should have the right to tell me not to talk about school shootings
Level: 135


Posts: 5256/5390
EXP: 29051684
For next: 283321

Since: 07-22-07

Pronouns: he/him/whatever
From: RSP Segment 6

Since last post: 333 days
Last activity: 333 days
Posted on 09-21-18 04:45:11 PM Link
Post #5256 · Fri, 2018 Sep 21, 13:45:11
Well good bots will obey robots.txt and have a proper user agent string, and bad bots... are bad.

Anyway it occurs to me that there doesn't appear to be a way to see a user's title (or ban message) on mobile.

____________________
Programs
stuffgoeshere
Log outShut down
Post #5256...[blag][Twitter][My Arts]
Xkeeper

Level: 263


Posts: 23358/25343
EXP: 296722316
For next: 2238137

Since: 07-03-07

Pronouns: they/them/????????

Since last post: 9 days
Last activity: 3 days
Posted on 09-21-18 06:48:50 PM Link
Originally posted by StapleButter
although completely denying them access yields the same result, ie they aren't able to crawl and index shit

In the case of Google and friends, they still have earlier non-blocked versions of your website and will continue to use that in search results, even if they are blocked in the future.

The eventual plan is to outright deny everything, but due to this bullshit they need to crawl to see the "no index, no cache, no follow" tags.

____________________
(Lv 244 with 228541052 EXP)
Rena
I had one (1) message in Discord deleted and proceeded to make a huge, huge mess about how it was a violation of free speech and how moderators are supposed to be spam janitors and nobody should have the right to tell me not to talk about school shootings
Level: 135


Posts: 5257/5390
EXP: 29051684
For next: 283321

Since: 07-22-07

Pronouns: he/him/whatever
From: RSP Segment 6

Since last post: 333 days
Last activity: 333 days
Posted on 09-21-18 07:43:53 PM Link
Post #5257 · Fri, 2018 Sep 21, 16:43:53
Wait, why do we want to block Google?

____________________
Programs
stuffgoeshere
Log outShut down
Post #5257...[blag][Twitter][My Arts]
Xkeeper

Level: 263


Posts: 23358/25343
EXP: 296722316
For next: 2238137

Since: 07-03-07

Pronouns: they/them/????????

Since last post: 9 days
Last activity: 3 days
Posted on 09-21-18 09:01:26 PM Link
Thankfully I can just direct you to the answer

____________________
(Lv 244 with 228543620 EXP)
Jamie
Requested (also Termingamer2-JD rereg)
Level: 29


Posts: 26/193
EXP: 143932
For next: 3953

Since: 06-03-14


Since last post: 2.8 years
Last activity: 2.8 years
Posted on 09-24-18 12:23:19 PM Link
 
Originally posted by Xkeeper
Originally posted by StapleButter
although completely denying them access yields the same result, ie they aren't able to crawl and index shit

In the case of Google and friends, they still have earlier non-blocked versions of your website and will continue to use that in search results, even if they are blocked in the future.

The eventual plan is to outright deny everything, but due to this bullshit they need to crawl to see the "no index, no cache, no follow" tags.

I thought eventually they just disappeared, after a while, but you're more than likely right on this.

Also a lot of abuse bots (ie ones that ignore robots.txt) still use bot at the useragent level - AhrefsBot and Semrush being two examples

____________________
Layout by Jamie (Cirnozie).
My blog | My (derp)board
Xkeeper

Level: 263


Posts: 23358/25343
EXP: 296722316
For next: 2238137

Since: 07-03-07

Pronouns: they/them/????????

Since last post: 9 days
Last activity: 3 days
Posted on 09-24-18 06:59:27 PM Link
Eventually I plan on updating it to outright block bots, but for right now you could say this is something of a transitional period.

____________________
(Lv 244 with 228624887 EXP)
Next newer thread | Next older thread
Jul - Meta - Two suggestions both regarding useragents New poll - New thread - Thread closed


Rusted Logic
Acmlmboard - commit 47be4dc [2021-08-23]
©2000-2022 Acmlm, Xkeeper, Kaito Sinclaire, et al.

29 database queries.
Query execution time:  0.099535 seconds
Script execution time:  0.024875 seconds
Total render time:  0.124410 seconds


TidyHTML vomit below
line 1 column 1 - Warning: missing <!DOCTYPE> declaration
line 2 column 273 - Warning: unescaped & or unknown entity "&page"
line 119 column 11 - Warning: <form> isn't allowed in <table> elements
line 118 column 10 - Info: <table> previously mentioned
line 120 column 11 - Warning: missing <tr>
line 120 column 119 - Warning: missing </font> before </td>
line 124 column 16 - Warning: plain text isn't allowed in <tr> elements
line 120 column 11 - Info: <tr> previously mentioned
line 125 column 68 - Warning: missing </nobr> before </td>
line 141 column 68 - Warning: missing </nobr> before <tr>
line 147 column 35 - Warning: missing <tr>
line 147 column 50 - Warning: missing </font> before </td>
line 148 column 37 - Warning: unescaped & or unknown entity "&id"
line 147 column 194 - Warning: missing </font> before </table>
line 149 column 35 - Warning: missing <tr>
line 149 column 50 - Warning: missing </font> before </td>
line 149 column 91 - Warning: missing </font> before </table>
line 156 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 158 column 9 - Warning: missing <tr>
line 176 column 13 - Warning: missing <tr>
line 179 column 74 - Warning: <style> isn't allowed in <td> elements
line 179 column 9 - Info: <td> previously mentioned
line 187 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 189 column 9 - Warning: missing <tr>
line 207 column 13 - Warning: missing <tr>
line 210 column 74 - Warning: <style> isn't allowed in <td> elements
line 210 column 9 - Info: <td> previously mentioned
line 215 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 217 column 9 - Warning: missing <tr>
line 235 column 13 - Warning: missing <tr>
line 238 column 74 - Warning: <style> isn't allowed in <td> elements
line 238 column 9 - Info: <td> previously mentioned
line 243 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 245 column 9 - Warning: missing <tr>
line 263 column 13 - Warning: missing <tr>
line 266 column 74 - Warning: <style> isn't allowed in <td> elements
line 266 column 9 - Info: <td> previously mentioned
line 269 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 271 column 9 - Warning: missing <tr>
line 289 column 13 - Warning: missing <tr>
line 294 column 4703 - Warning: replacing unexpected input with </input>
line 294 column 5017 - Warning: discarding unexpected </span>
line 297 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 299 column 9 - Warning: missing <tr>
line 317 column 13 - Warning: missing <tr>
line 320 column 74 - Warning: <style> isn't allowed in <td> elements
line 320 column 9 - Info: <td> previously mentioned
line 326 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 328 column 9 - Warning: missing <tr>
line 346 column 13 - Warning: missing <tr>
line 349 column 4473 - Warning: replacing unexpected input with </input>
line 349 column 4787 - Warning: discarding unexpected </span>
line 352 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 354 column 9 - Warning: missing <tr>
line 372 column 13 - Warning: missing <tr>
line 375 column 74 - Warning: <style> isn't allowed in <td> elements
line 375 column 9 - Info: <td> previously mentioned
line 378 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 380 column 9 - Warning: missing <tr>
line 398 column 13 - Warning: missing <tr>
line 401 column 74 - Warning: <style> isn't allowed in <td> elements
line 401 column 9 - Info: <td> previously mentioned
line 411 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 413 column 9 - Warning: missing <tr>
line 431 column 13 - Warning: missing <tr>
line 434 column 74 - Warning: <style> isn't allowed in <td> elements
line 434 column 9 - Info: <td> previously mentioned
line 437 column 17 - Warning: missing <tr>
line 437 column 17 - Warning: discarding unexpected <table>
line 440 column 35 - Warning: missing <tr>
line 440 column 50 - Warning: missing </font> before </td>
line 440 column 91 - Warning: missing </font> before </table>
line 442 column 35 - Warning: missing <tr>
line 442 column 50 - Warning: missing </font> before </td>
line 443 column 37 - Warning: unescaped & or unknown entity "&id"
line 442 column 194 - Warning: missing </font> before </table>
line 444 column 17 - Warning: discarding unexpected </textarea>
line 444 column 28 - Warning: discarding unexpected </form>
line 444 column 35 - Warning: discarding unexpected </embed>
line 444 column 43 - Warning: discarding unexpected </noembed>
line 444 column 53 - Warning: discarding unexpected </noscript>
line 444 column 64 - Warning: discarding unexpected </noembed>
line 444 column 74 - Warning: discarding unexpected </embed>
line 444 column 82 - Warning: discarding unexpected </table>
line 444 column 90 - Warning: discarding unexpected </table>
line 446 column 9 - Warning: missing </font> before <table>
line 458 column 25 - Warning: discarding unexpected </font>
line 467 column 37 - Warning: discarding unexpected </font>
line 445 column 1 - Warning: missing </center>
line 120 column 63 - Warning: <img> lacks "alt" attribute
line 125 column 19 - Warning: <td> attribute "width" has invalid value "120px"
line 125 column 93 - Warning: <img> lacks "alt" attribute
line 141 column 19 - Warning: <td> attribute "width" has invalid value "120px"
line 141 column 98 - Warning: <img> lacks "alt" attribute
line 148 column 44 - Warning: <img> proprietary attribute value "absmiddle"
line 148 column 142 - Warning: <img> proprietary attribute value "absmiddle"
line 148 column 216 - Warning: <img> proprietary attribute value "absmiddle"
line 161 column 22 - Warning: <img> lacks "alt" attribute
line 161 column 63 - Warning: <img> lacks "alt" attribute
line 161 column 112 - Warning: <img> lacks "alt" attribute
line 161 column 162 - Warning: <img> lacks "alt" attribute
line 172 column 15 - Warning: <img> lacks "alt" attribute
line 183 column 890 - Warning: <img> proprietary attribute value "absmiddle"
line 183 column 890 - Warning: <img> lacks "alt" attribute
line 192 column 23 - Warning: <img> lacks "alt" attribute
line 192 column 64 - Warning: <img> lacks "alt" attribute
line 192 column 113 - Warning: <img> lacks "alt" attribute
line 192 column 163 - Warning: <img> lacks "alt" attribute
line 193 column 11 - Warning: <img> lacks "alt" attribute
line 203 column 15 - Warning: <img> lacks "alt" attribute
line 219 column 11 - Warning: <img> lacks "alt" attribute
line 220 column 22 - Warning: <img> lacks "alt" attribute
line 220 column 63 - Warning: <img> lacks "alt" attribute
line 220 column 112 - Warning: <img> lacks "alt" attribute
line 220 column 162 - Warning: <img> lacks "alt" attribute
line 221 column 11 - Warning: <img> lacks "alt" attribute
line 231 column 15 - Warning: <img> lacks "alt" attribute
line 248 column 22 - Warning: <img> lacks "alt" attribute
line 248 column 63 - Warning: <img> lacks "alt" attribute
line 248 column 112 - Warning: <img> lacks "alt" attribute
line 248 column 161 - Warning: <img> lacks "alt" attribute
line 249 column 11 - Warning: <img> lacks "alt" attribute
line 259 column 15 - Warning: <img> lacks "alt" attribute
line 274 column 23 - Warning: <img> lacks "alt" attribute
line 274 column 64 - Warning: <img> lacks "alt" attribute
line 274 column 113 - Warning: <img> lacks "alt" attribute
line 274 column 163 - Warning: <img> lacks "alt" attribute
line 285 column 15 - Warning: <img> lacks "alt" attribute
line 292 column 4465 - Warning: <img> proprietary attribute value "absmiddle"
line 292 column 4465 - Warning: <img> lacks "alt" attribute
line 302 column 23 - Warning: <img> lacks "alt" attribute
line 302 column 64 - Warning: <img> lacks "alt" attribute
line 302 column 113 - Warning: <img> lacks "alt" attribute
line 302 column 163 - Warning: <img> lacks "alt" attribute
line 303 column 11 - Warning: <img> lacks "alt" attribute
line 313 column 15 - Warning: <img> lacks "alt" attribute
line 320 column 901 - Warning: <div> anchor "xklayout" already defined
line 323 column 1526 - Warning: <img> proprietary attribute value "absmiddle"
line 323 column 1526 - Warning: <img> lacks "alt" attribute
line 331 column 23 - Warning: <img> lacks "alt" attribute
line 331 column 64 - Warning: <img> lacks "alt" attribute
line 331 column 113 - Warning: <img> lacks "alt" attribute
line 331 column 163 - Warning: <img> lacks "alt" attribute
line 342 column 15 - Warning: <img> lacks "alt" attribute
line 357 column 23 - Warning: <img> lacks "alt" attribute
line 357 column 64 - Warning: <img> lacks "alt" attribute
line 357 column 113 - Warning: <img> lacks "alt" attribute
line 357 column 163 - Warning: <img> lacks "alt" attribute
line 358 column 11 - Warning: <img> lacks "alt" attribute
line 368 column 15 - Warning: <img> lacks "alt" attribute
line 375 column 901 - Warning: <div> anchor "xklayout" already defined
line 383 column 22 - Warning: <img> lacks "alt" attribute
line 383 column 63 - Warning: <img> lacks "alt" attribute
line 383 column 112 - Warning: <img> lacks "alt" attribute
line 383 column 162 - Warning: <img> lacks "alt" attribute
line 394 column 15 - Warning: <img> lacks "alt" attribute
line 404 column 1076 - Warning: <img> proprietary attribute value "absmiddle"
line 404 column 1076 - Warning: <img> lacks "alt" attribute
line 416 column 23 - Warning: <img> lacks "alt" attribute
line 416 column 64 - Warning: <img> lacks "alt" attribute
line 416 column 113 - Warning: <img> lacks "alt" attribute
line 416 column 163 - Warning: <img> lacks "alt" attribute
line 417 column 11 - Warning: <img> lacks "alt" attribute
line 427 column 15 - Warning: <img> lacks "alt" attribute
line 434 column 901 - Warning: <div> anchor "xklayout" already defined
line 443 column 44 - Warning: <img> proprietary attribute value "absmiddle"
line 443 column 142 - Warning: <img> proprietary attribute value "absmiddle"
line 443 column 216 - Warning: <img> proprietary attribute value "absmiddle"
line 452 column 25 - Warning: <img> lacks "alt" attribute
line 457 column 267 - Warning: <img> lacks "alt" attribute
line 149 column 50 - Warning: trimming empty <font>
line 294 column 4770 - Warning: trimming empty <label>
line 349 column 4540 - Warning: trimming empty <label>
line 437 column 17 - Warning: trimming empty <tr>
line 440 column 50 - Warning: trimming empty <font>
line 125 column 68 - Warning: <nobr> is not approved by W3C
line 141 column 68 - Warning: <nobr> is not approved by W3C
line 177 column 27 - Warning: <nobr> is not approved by W3C
line 208 column 27 - Warning: <nobr> is not approved by W3C
line 236 column 27 - Warning: <nobr> is not approved by W3C
line 264 column 27 - Warning: <nobr> is not approved by W3C
line 290 column 27 - Warning: <nobr> is not approved by W3C
line 318 column 27 - Warning: <nobr> is not approved by W3C
line 347 column 27 - Warning: <nobr> is not approved by W3C
line 373 column 27 - Warning: <nobr> is not approved by W3C
line 399 column 27 - Warning: <nobr> is not approved by W3C
line 432 column 27 - Warning: <nobr> is not approved by W3C
Info: Document content looks like HTML5
Info: No system identifier in emitted doctype
Tidy found 177 warnings and 0 errors!


The alt attribute should be used to give a short description
of an image; longer descriptions should be given with the
longdesc attribute which takes a URL linked to the description.
These measures are needed for people using non-graphical browsers.

For further advice on how to make your pages accessible
see http://www.w3.org/WAI/GL.
You are recommended to use CSS to specify the font and
properties such as its size and color. This will reduce
the size of HTML files and make them easier to maintain
compared with using <FONT> elements.

You are recommended to use CSS to control line wrapping.
Use "white-space: nowrap" to inhibit wrapping in place
of inserting <NOBR>...</NOBR> into the markup.

About HTML Tidy: https://github.com/htacg/tidy-html5
Bug reports and comments: https://github.com/htacg/tidy-html5/issues
Official mailing list: https://lists.w3.org/Archives/Public/public-htacg/
Latest HTML specification: http://dev.w3.org/html5/spec-author-view/
Validate your HTML documents: http://validator.w3.org/nu/
Lobby your company to join the W3C: http://www.w3.org/Consortium

Do you speak a language other than English, or a different variant of
English? Consider helping us to localize HTML Tidy. For details please see
https://github.com/htacg/tidy-html5/blob/master/README/LOCALIZE.md