NSFW, RSS and Login Verification

[Arne_Jacob]

Hey, I have a bug report (?) about QQ and RSS. I'm trying to add RSS links from the NSFW section of QQ to a discord bot for update alerts on a server of mine, but it only returns a 403 - Forbidden error, because it needs you to be logged in even for the RSS feed. I'm wondering if this is intended behavior or an error, as it doesn't seem particularly useful to have a feed that doesn't work unless used from specific clients.

 

[Biigoh]

I didn't even know we had RSS on QQ....

You might wanna ask alethiophile or ultima333 about this

 

[Arne_Jacob]

I didn't even know we had RSS on QQ....

You might wanna ask alethiophile or ultima333 about this

Click to shrink...

Thanks, will do!

 

[alethiophile]

If RSS worked from a client that wasn't logged in, it would defeat the purpose of keeping the NSFW subforums invisible to non-logged-in users. This is in fact the intended behavior.

I'm not sure what the easiest way is to make an RSS client log in to the site.

 

[Arne_Jacob]

If RSS worked from a client that wasn't logged in, it would defeat the purpose of keeping the NSFW subforums invisible to non-logged-in users. This is in fact the intended behavior.

I'm not sure what the easiest way is to make an RSS client log in to the site.

Click to shrink...

Yeah, that's what I assumed. Using a local browser extension lets me log in to the site, but that doesn't work for the purpose I want to use the RSS feed for - by the simple fact that it's local only.


Additionally, the only information the RSS feed delivers is:

• There has been a new threadmark made
• The author of the story
• The date of the update
• A short description of the update (Usually an author's note.)
• A link to the post

Of these, the only thing that personally I think is justified being hidden away from non-logged-in clients is the description part, but maybe that's just me. If there was a way to get a stripped down feed or something that would be awesome, but if not then whatever... I suppose I could experiment with desktop RSS clients or something, maybe one of them can pass to webhooks.

Either way, thanks for the answer.

 

[KnotMyProblem]

Might be thread necromancy but I was having a similar issue and thought I'd comment on what I found. XenForo, the software this site uses, doesn't support inline authentication and only authenticates via Cookies and active sessions. This isn't an issue on some sites like SB where everything is public, but since QQ requires authentication for most read access this means most RSS readers won't work.

 

[Omniscopia]

I used FeedBro browser plugin, but it stopped working after XenForo update. I still could add feeds, but thereafter they cannot be updated.

I tried RSS Guard, but it cannot add feeds at all, even with try of authentication: "Status: network error (access to content was denied)"

Does anyone still use Q.Q. RSS successfully?


P.S. FeedBro stopped working with SB and SV as well. RSS Guard work with both.

 

[PDV]

I've been following With This Ring (not NSFW) by RSS for years, and it has entirely broken this month. I tried reconnecting to the threadmark RSS, and that got a new set of 10 entries but immediately stopped working. Both subscriptions were to https://forum.questionablequesting....en.8938/threadmarks.rss?threadmark_category=1. First failure August 9th, failure discovered, reconnected, and re-failed August 20th.

 

[dariosin]

RSS feed is the way i follows all the stories in various forums and platforms all these years, is there any solution or workaround for this?

 

[TheMidnightRook]

RSS feed is the way i follows all the stories in various forums and platforms all these years, is there any solution or workaround for this?

Click to shrink...

Click the "watch thread" button.

 

[KnotMyProblem]

RSS feed is the way i follows all the stories in various forums and platforms all these years, is there any solution or workaround for this?

Click to shrink...

It's been really annoying me. SB, RR, SV, AO3, and most other sites having working RSS. Usually I'll read a chapter right from the RSS reader rather than opening the main site (it makes reading on mobile way easier). QQ is the only site I regularly use where it doesn't work and it's pretty annoying.

 

[PDV]

Click the "watch thread" button.

Click to shrink...

I did, long ago; it doesn't help. Only works if you remember to check the forum without prompting. I don't; RSS is the way I keep track of everything that periodically has something new. (Except chatrooms, and I don't remember to check most of those all that often.) There's a lot of people here mostly to follow stories/quests, I'm pretty sure, and I'd bet most of them are in a similar boat. People who post more actively I'm sure don't have that problem.

@alethiophile , @ultima333 , have you looked into this at all? Much appreciated.

 

[AztecCroc]

I did, long ago; it doesn't help. Only works if you remember to check the forum without prompting. I don't; RSS is the way I keep track of everything that periodically has something new. (Except chatrooms, and I don't remember to check most of those all that often.) There's a lot of people here mostly to follow stories/quests, I'm pretty sure, and I'd bet most of them are in a similar boat. People who post more actively I'm sure don't have that problem.

@alethiophile , @ultima333 , have you looked into this at all? Much appreciated.

Click to shrink...

Turn on email notifications then.

 

[SnarkLord]

I..don't want to sound like an asshole, even accidentally, it....email notifs exist for a reason.
Mainly so people *don't* have to constantly manually check the website if anything they follow has updated.
And I'm pretty sure that a supermajority of users use this option.
And as alethiopile said, part of QQ being locked away from non-users is intentional design, which is likely never ever being changed.
If part of that design happens to break your RSS, sorry but you'll have to deal with it...there's little reason why admins/mods would be interested in figuring out a stable working RSS when email notification already works without an issue

 

[UserNameRandom?]

I'm not sure what the easiest way is to make an RSS client log in to the site.

Click to shrink...

Alright, cards on the table. I just spent a couple of minutes looking up documentation and managed to put together a working python script for RSS guard. It uses cookies to authenticate, so users will need to grab those themselves.

I don't really use RSS, so I'm not entirely sure what's normal here, but it's basically a python script that RSS Guard calls.

I would rather not post it without explicit permission from the admins, though. As it curls the site with authentication cookies-- 'xf_user', 'xf_csrf', and 'xf_session'--, I want to make sure they're completely fine with it first.

It seems to work, It seems to list everything right, but once again in fear of repetition that I do not use RSS in my day to day life and I have no idea what the real use case of rss looks like.

 

[PDV]

Email notifications don't really work. Not, like, QQ is doing them wrong, just, using email for notifications mostly does not work to notify me. I don't check my email enough, there's too much in my inbox to catch things, and anything that shows up frequently (especially, like in this case, every day) gets automatically filtered as not important (primarily by Gmail, though also, a bit, by my eyes). I have them enabled; I remember to check QQ for unrelated reasons more often than I ever see a QQ email. "Inbox fatigue" is talked about often enough that I believe this is a general problem that affects most people, or at least most people with office jobs.

People who use RSS have an app and/or website (I use TheOldReader) which collects various feeds (I have about a hundred), collects updates for them regularly, and categorizes them into user-created groups labeled by whether you've read them. It's one place to look for all the things you follow; what's new, how large the backlog is of each source, and the history of everything else you've read from that source ever.

Also, I really doubt this is actually about the login, since creating the feed works just fine, and that doesn't need a login. I just double-checked; the link I've used ~four times now is https://forum.questionablequesting....en.8938/threadmarks.rss?threadmark_category=1 It's accessible even while logged out, and generates a properly-formed threadmarks.rss file with the last ten entries, which matches the RSS spec. It's something else in the process, probably with the polling, that's broken. My guess is an overactive spam guard or something else that's trying to keep out excessive form submissions or something.

 

[KnotMyProblem]

Email notifications don't really work. Not, like, QQ is doing them wrong, just, using email for notifications mostly does not work to notify me. I don't check my email enough, there's too much in my inbox to catch things, and anything that shows up frequently (especially, like in this case, every day) gets automatically filtered as not important (primarily by Gmail, though also, a bit, by my eyes).

Click to shrink...

I don't use my primary email for anything NSFW related. I keep a separate alt email for that, which I almost never check. The only time I check it is when looking for 2fa codes or signing up for a new service. I suspect most users of doing the same. I think for SFW sites emails might make sense, but not when the majoirty of users are using alt emails.

 

[Zone04]

Hey, I have a bug report (?) about QQ and RSS. I'm trying to add RSS links from the NSFW section of QQ to a discord bot for update alerts on a server of mine, but it only returns a 403 - Forbidden error, because it needs you to be logged in even for the RSS feed. I'm wondering if this is intended behavior or an error, as it doesn't seem particularly useful to have a feed that doesn't work unless used from specific clients.

Click to shrink...

I've had a similar issue a few years ago for SB, SV, and QQ. SV was the only one that consistently worked. It's to do with verification and authentication primarily, along with Zenforo Node permissions. All three sites are similar enough to where they use similar external methods of verification and protection against bots. A primary cause of problems that allowed my RSS to work is primarily going through about 9 different RSS programs on Windows 10 (at the time), of which only 2 worked, which means your bot also has to use the correct RSS version (there are multiple methods).

Try as many different bots that you can find. If you know how to program, I would recommend making your own in different ways.

Secondly, IP, User-agent, device information, and many more. See RSS is typically completely unencrypted and broadcast out, however modern website protections against bots, ddos attacks, hackers, intercepting, and any more attack vectors, actually unintentionally block and curtail simple RSS feeds. These services see a different user agent that's abnormal, or an IP that's commonly used to attack (VPN), or go through a server hop, and they automatically send the request through a 403 message, or ask for 'prove you're human' bullshit. Except rss can't prove shit.

The user agents for RSS feeds is typically script related, sometimes python, sometimes others, and this gets flagged.

IP wise, discord bots tend to use data center IP's. Yo will not successfully access this any of these sites from data center IP's because services like cloud flare heavily scrutinizer and usually actively block those IP's. Discord bots use Amazon and Google data centers as host. Only solution for you would be privately hosting it on your local machine and IP.

Furthermore, RSS doesn't store tokens. It's an open service that's extremely simple. Check info, get info, fuck off. Issue is, most services require tokens for authentication, or cookies. Nsfw sections are a no go, sfw are allowed. The issue then? The fact that because of the user agent and ip, if you were in the RSS processes place you'd get challenged. With RSS being so smart, it won't be able to verify or confirm or prove anything. Even more so with the lack of tokens.

There are services that allow you to skip Cloud flare by keeping a token. This wouldn't work.

Basically, the short story is the following: unless your bot is hosted on a small server with a dedicated IP that has a good history with cloud flare (you can request for an IP to be unblocked manually), or hosted locally on your home IP, most modern services with WAF & Bot Protection will be your enemy.

You can bypass these, mind you it's hit or miss. Browsers these days collect a lot more than just browser type and version and ip and that's it, they usually collection device information. Peripheral devices plugged in, do you have a camera, how many extensions do you have, is your cursor active (it's why completing captchas successfully sometimes still failes, mouse movement robotic or top simple, browser too fresh with nothing identifiable), how many tabs are open, etc.

You can try injecting authentication cookies into the bot for when it sends requests, you can try spoofing user-agent's, and you can even run a vm on a small server you bought somewhere and host the boy through that so it looks like a normal browser and computer. You can also spoof Hardware ID and simulate peripheral devices when a browser does a cursory surface check. You can use a 2nd QQ account specifically for RSS feeds on that server, and simply leave it untouched. The only issue is that you'll be paying for it. At that point just make your own scraper using your account.

Not sure if it's bannable but I've got one that checks all the 'new threadmarks' section that are watched, running as a service in the background of my machine, so whenever my computer is on, which is always, then it'll just keep inform me. I've found it's far better than the inbuilt 'send notification' shit. Because a solid fourth or fifth of all watched posts don't actually send notifications, and the bell up top stops updating or misses new threadmarks. My bell up top currently numbers 705 unread, and I know it should be far more.

Edit: just came to mind, but you can also use proxies. I've got a VPN on, but it's got a static IP that gets refreshed every couple of days, so I have to complete a challenge every so often.

2nd edit: just read through the whole thread and saw post from UNR? and would like to clarify that for for previous reasons it more than likely won't work unless hosted on or running from your IP, if you do use discords services it will not work. Because, again, it uses AWS and Google servers, which means IP's that constantly get flagged for review, which means your bot will get challenged.

Cookies and Auth tokens will work to let you access all content. Which is great. Also, RSS guards was one of the few I tested that actually consistently worked. Good and reliable usually. Nice choice.

 

[SnarkLord]

Ivadmit this question/post is a bit off topic, but i genuinely need to know

I don't use my primary email for anything NSFW related. I keep a separate alt email for that

Click to shrink...

Unless the og account is shared by the entire family, or ties to a business, why do you even waste Time for creating a newx separate account?
You're still using your email, it's a hassle to switch accounts if you have 2fa login enabled, and its more of a hassle if you need to use it for something in that case...

 

[LurkingInTheDeceit]

Because a solid fourth or fifth of all watched posts don't actually send notifications, and the bell up top stops updating or misses new threadmarks. My bell up top currently numbers 705 unread, and I know it should be far more.

Click to shrink...

This may or may not account for the discrepancy, but when people delete their posts on QQ, any alerts for that post also disappear. This is also the case for thread creation notifications (for Watched subforums), in the very limited occasions when we delete them. Undeleting (which users can't do, but staff can) doesn't bring the notification back.

So anyone posting their chapter too early, or making a mistake they want to quickly fix, or double-posting due to a connection issue or whatever, etc, may have deleted a threadmarked post, removed the alert, but your non-native monitoring probably keeps a record.

Not sure if it's bannable but

Click to shrink...

I'm not very tech-minded, but my understanding is that as long as you're rate-limiting, so as to not cause undue stress to the servers, that should be fine. Having more than one account is bannable, though, so advise against that.

 

[Spudr]

You're still using your email, it's a hassle to switch accounts if you have 2fa login enabled, and its more of a hassle if you need to use it for something in that case...

Click to shrink...

It's not really a hassle if you're using two different providers. For instance, I use yahoo and gmail for different purposes.

 

[SnarkLord]

It's not really a hassle if you're using two different providers. For instance, I use yahoo and gmail for different purposes.

Click to shrink...

...i honestly forgot that other web-based mail providers beaide google even exist.
..still don't see why have a 2nd mail account

 

[AztecCroc]

...i honestly forgot that other web-based mail providers beaide google even exist.
..still don't see why have a 2nd mail account

Click to shrink...

So you can check your mail in public without worry? So you don't have to deal with the NSFW spam on your main?

 

[SnarkLord]

So you can check your mail in public without worry? So you don't have to deal with the NSFW spam on your main?

Click to shrink...

...filters and autoblock/autodelete mail options are a thing?
And if it's something actually nsfw I'm not going to open the mail until i come home/wouldn't care and still open it, depending on the source...
If someone's being nosy what I'm doing on my phone and looking over my shoulder they can eff off