looks like X/twitter(?) broke something again #983
Comments
broke again zedeus/nitter#983
|
Also, the syndication api for |
|
Yes, it is not working now. I hope the Nitter people fix this soon. |
|
Is there a online/CLI tool converting |
Not really. showReplies=false shows years-old content when not logged in. |
|
Down again... |
That's because in that specific example, those tweets were years ago. Look again at the like count, notice anything? |
We can just search for the first |
Indeed #!/usr/bin/python3
import requests
import re
import urllib
url = "https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk"
with urllib.request.urlopen(url) as response:
encoding = response.info().get_param('charset', 'utf8')
html = response.read().decode(encoding)
result = re.search('script id="__NEXT_DATA__" type="application\/json">([^>]*)<\/script>', html)[1]
print(result) |
Interesting, but this doesn't return RSS with 'item', 'pubDate' etc. tags. Maybe a script using https://github.com/lkiesow/python-feedgen would do the job? |
Not sure I understand ? It expose far more informations than needed and it does expose the date and all Here's an example for one tweet only : {
"type": "tweet",
"entry_id": "tweet-1519480761749016577",
"sort_index": "1691455400412446720",
"content": {
"tweet": {
"id": 0,
"location": "",
"conversation_id_str": "1519480761749016577",
"created_at": "Thu Apr 28 00:56:58 +0000 2022",
"display_text_range": [
0,
52
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": [],
"media": []
},
"favorite_count": 4600599,
"favorited": false,
"full_text": "Next I’m buying Coca-Cola to put the cocaine back in",
"id_str": "1519480761749016577",
"lang": "en",
"permalink": "/elonmusk/status/1519480761749016577",
"possibly_sensitive": false,
"quote_count": 171975,
"reply_count": 187438,
"retweet_count": 649833,
"retweeted": false,
"text": "Next I’m buying Coca-Cola to put the cocaine back in",
"user": {
"blocking": false,
"created_at": "Tue Jun 02 20:12:29 +0000 2009",
"default_profile": false,
"default_profile_image": false,
"description": "Blades of Glory",
"entities": {
"description": {
"urls": []
},
"url": {}
},
"fast_followers_count": 0,
"favourites_count": 30569,
"follow_request_sent": false,
"followed_by": false,
"followers_count": 153112066,
"following": false,
"friends_count": 410,
"has_custom_timelines": false,
"highlightedLabel": {
"badge": {
"url": "https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_bigger.jpg"
},
"description": "X",
"userLabelType": "BusinessLabel",
"userLabelDisplayType": "Badge"
},
"id": 0,
"id_str": "44196397",
"is_translator": false,
"listed_count": 126597,
"location": "𝕏Ð",
"media_count": 1659,
"name": "Elon Musk",
"normal_followers_count": 153112066,
"notifications": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1690621312",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1683325380441128960/yRsRRjGO_normal.jpg",
"protected": false,
"screen_name": "elonmusk",
"show_all_inline_media": false,
"statuses_count": 29441,
"time_zone": "",
"translator_type": "none",
"url": "",
"utc_offset": 0,
"verified": false,
"withheld_in_countries": [],
"withheld_scope": "",
"is_blue_verified": true
}
}
}
},EDIT : Maybe you meant a directly usable solution for an end user, and of course it's not, the snippet need to be adapted by a dev. |
Ok, thank you, I'll try this. |
calling the syndication URL without being logged in twitter doesn't retrieve the most recent tweets. If I call this url in postman, I retrieve 100 tweets from 10/19/2018 to 07/31/2023; no tweets from august... |
It retrieves the tweets with the highest like count from that user, which doesnt sound good if your goal is retrieving the most recent tweets, as there's no guarantee new tweets will make it to the top 100 tweets from that user. And even if they did, it might take a considerable amount of time |
|
I've noticed that for smaller accounts that have less than 100 tweets, that syndication URL does not load any tweets. |
No. That is the case for all big accounts. I am interested in the most recent Tweets and this approach will lead to nothing. |
im using this for my bot and it working fine with cookies and headers. |
Yes. And there is at least one tweet from August with more likes (>807K) than some older tweets which are included (e.g. <680K). |
|
is there any forecast for solving this problem? |
|
Looks like https://nitter.privacydev.net/ is working |
|
That one is a fork which uses account credentials. See #830 |
|
I am aware but couldn't nitter implement a system that aurora uses with lots of accounts that rotate per user? |
That's hard to maintain and simple for twitter to ban by just filtering "if number of accounts per IP > SOME_CONSTANT: ban all of them" |
User feeds not working on this |
|
I switched to the privacydevel fork, credentials in but its still 404ing the same endpoint upstream is having problems with |
|
Strange. privacydev (without credentials) works more or less for @ElonMusk, but not for other users like for instance @BarackObama. |
|
Sorry about this @animegrafmays. Stopped |
|
Now that the moronic-talk is over. Questions about Nitter, specifically with regards to the regular accounts version. For those that don't know, the guest_accounts version used 2 curl commands to generate temporary accounts, however there was recently a fork (which I can't find right now) that used real accounts instead.
Any info helpful, thank you! |
|
@RunlevelConsulting literally the 7th result after typing "nitter" in the github search: https://github.com/sekai-soft/guide-nitter-self-hosting |
|
out of nowhere my instance started working again, anyone else? |
|
Mine never stopped working (except for video, that has been broken for a long time) but I'm using it with a real account, would you happen to have it configured with some of those as well? |
|
Yes, I'm working with some real accounts, but I remember that stopped working too, I used my own instance for scraping, now I migrated everything to a new system, I'm glad that nitter is still working. |
Replying to myself for those who also have problems playing video: it does work when you disable Enable HLS video streaming and proxy video streaming through the server while keeping Enable mp3 video playback enabled in Settings (the cog wheel in the top bar, top right). In this case you get mp4 video files which can be streamed directly in the interface. If you're using Firefox you may need to disable 'Enhanced Tracking Protection' since it (rightfully) considers |
|
It is also possible in some cases when twimg/twitter JS is disabled to click on
a link to the video itself that will show, then download the file. For some folks
enhanced tracking protection is mandatory and cannot be turned off. Fixing that
would require a way to create an exception in firefox itself to allow twimg.com
from a nitter instance.
|
|
You can add per-site exceptions for ETP (Enhanced Tracking Protection) in Firefox. These exceptions are all-or-nothing so it is not possible to only let |
|
That might be enough so long as nobody ever deploys an ad-supported Nitter instance
|
|
Whatever it might be worth here is a working set of settings when using https://xcancel.com/ Display Media Link replacements (blank to disable) Only drawback so far is that the replacement of youtube doesnt work in a post, only for quoted posts. And that loading too many pages at once will throw you into some kind of temporary blacklist at xcancel.com (so have at least a few seconds between each load/refresh if you visit more than one account). Also Im not sure on how multiloading usernames affects getting onto the tempory blacklist. That is when you load multiple accounts in one go such as https://xcancel.com/elonmusk,cb_doge,joerogan,nsagov (just picked some random accounts as example :-) |
|
tonight during the voting in the US presidential election we topped 1.21 million unique visitors on nitter.poast.org. right now, during the counting we are averaging 21 thousand requests per second across 4 ryzen 7 3800x load balanced. this is insane. and the 400 accounts we are using isn't even close to enough. i offloaded a lot of requests via proxy directly to twitter via a separate proxy_pass. helped massively. never been more proud to put so much effort into running it 🙏 |
|
I have been using https://nitter.privacydev.net/MarkDavis/ to keep an eye on the one twitter user I follow as I much prefer the Nitter presentation. For the past three days I have been getting nothing but "Instance has been rate limited. Use another instance or try again later." messages when I attempt to check. Did they break something new, or am I the only one seeing this? |
|
works fine for us -- https://nitter.poast.org/MarkDavis/ might be that the instance you are using is genuinely rate limited. not sure how many accounts he is using, but on our instance we run 4 dedicated servers in round robin with 100s of accounts, so usually reloading on ours fixes that issue if it ever appears for you. this isn't Xitter changing anything or nitter being broken, it's related to rate limits of accounts used to display the tweets to you |
|
Thank you for your feedback. It does indeed appear to be a problem with that instance in particular. I have changed over to xcancel.com and it is working well. I had though the instance I was using the only functional one and was pleased to discover I was wrong. |
|
you're welcome friend. im glad you found a solution that fits your needs |
|
@dhdurgee here is all the instances that works: https://status.d420.de |
Im experiencing the same issue as you since few days now. However im using Tor and none of the other current working instances including xcancel.com allow Tor connections which is very sad. |
I made the decision after several days of heavy abuse from TOR exits to block them. I will not be removing the block and have automated updating it with any new exits. prior to blocking TOR exits, we were receiving on average over 30k requests per second. after it's down to 10-11k. all used for scraping. it's unfortunate that people abuse TOR in this way, but it's not something I will consider removing. |
Odd xcancel.com works for me on Tor with the default settings. Can you send an email at antibot [at] xcancel.com? With some explanation and a screenshot. I'll try to see what's going on. |
|
My guess is the few surviving Nitter instances have to take extreme precautions against
bulk scraping expecially by bots or AI to avoid being blocked.
|
|
If someone wanted to set up a dedicated (maybe .onion?) nitter instance just for Tor,
they'd need a way to rate limit each connection to human possible numbers of
connections. This would have to be for each connection separately, and at a very
low rate like one request per second. DDOS scripts might be possible to break by
having the whole instance respond to a wave of bot hits by going offline for first
30 seconds, then five minutes, then an hour and escalating as needed if attacks
continue.
|
|
I'm sure this has been answered but as it's now an enormous thread, it's hard to find the details: What is the canonical way to configure a nitter instance using multiple accounts for use, ideally with tokens being refreshed if needed? Thank you. |
|
If you can just whitelist QuiteRSS reader, that would be great, because it throws an error saying "RSS reader not yet whitelist!" - I think thats the only issue preventing from using Tor in the reader. The actual onion links on the wiki dont really work either. |
Please follow the instructions and send an email at rss [at] xcancel.com |
|
This should work for people wanting to add accounts by hand https://gist.github.com/cmj/998f59680e3549e7f181057074eccaa3 |
and others
The text was updated successfully, but these errors were encountered: