Login
Register
Login
Register
Communities
Directory
Help
Language
Legal
Search
Communities
Directory
Help
Language
Legal
Search
Visit
Poke
Giac El Vecio
Sat, 20 Jul 2024 11:44:28 +0200
from Diversi Spiritus
Update test
Link to source
1 Like
Rollerball (3 months ago)
Visit
Poke
Haakon Meland Eriksen (Els Mussols)
Sat, 20 Jul 2024 12:13:17 +0200
from Streams at els Mussols
Is the test updated?
Link to source
Visit
Poke
Giac El Vecio
Sat, 20 Jul 2024 13:13:41 +0200
from Diversi Spiritus
@
Haakon Meland Eriksen (Els Mussols)
until two weeks ago everything was going smoothly...
Then a series of major updates, like 4 or 5, and every time I make a post involving activitypub here I have 10 minutes of panic on my site.
But yesterday I did experiments: I made a post for a list of 50 or 60 people, but without activitypub, just zot nomad, and it all worked perfectly. Then a post test for a small list of 18 activitypub people, then a very small list that even the stupidest server would manage without difficulty, and again 10 minutes of panic on my site.
Now, honestly, I'm tired, obviously something is wrong, at the same time I'm also tired of listening to bullshit about the ability of my site administrator friend, because he is a person with a lot of experience, who has managed various sites including friendica red hubzilla... .
Link to source
Visit
Poke
Giac El Vecio
Sat, 20 Jul 2024 12:56:03 +0200
from Diversi Spiritus
A new one now. Updated again
Link to source
Visit
Poke
Giac El Vecio
Sat, 20 Jul 2024 13:38:37 +0200
from Diversi Spiritus
More than 10 minutes of shit timeline black-out....
Link to source
Visit
Poke
Haakon Meland Eriksen (Els Mussols)
Sat, 20 Jul 2024 22:24:23 +0200
from Streams at els Mussols
Sorry I cannot help.
What do the logs say?
Is there any help watching journalctl live with
journalctl -f
Link to source
Visit
Poke
Bill Statler
Sat, 20 Jul 2024 22:34:26 +0200
last edited: Sat, 20 Jul 2024 22:47:09 +0200
from Bunny of Doom
This is really frustrating -- I mean, more frustrating for you than for me, but I want to make a suggestion and I can hardly think of anything.
Mike made bunch of changes in the last ~15 hours, so an update to 24.07.20.1 might help. Maybe.
Oops, I see from your posting time that you probably already have those updates.
Other than that, my only clue is that your instance has this bug, and no other instance has noticed it. Why? What makes your instance different from the others?
Link to source
Visit
Poke
Mike Macgirvin 🖥️
Sat, 20 Jul 2024 23:55:39 +0200
from Y
@
Bill Statler
echoes my sentiment. Over the years I've gotten a lot better at finding issues buried somewhere in megabytes of code based on some simple description and no access to logs or database on the affected system; but I'm still not clairvoyant. I have no idea what's happening here and I cannot duplicate it locally, and I don't know of any changes in the last 5-6 months which would affect this. And I've changed a whole lot of code in the last 5-6 months. The nomadic identity changes were massive. So were conversation containers before that.
So here's what you do. Turn on full logging at the maximum log level if you haven't already. Create an access list with 5 ActivityPub folks in it. Two might work, but 5 would be better. Then send them a message. Just before sending the message, delete your current logfile. Send the message. After a minute or two, copy the logfile somewhere and email it to me. Then turn your logging level back down to something reasonable again so it doesn't fill your disk - unless you have the logrot addon installed.
Describing what's happening is sometimes helpful, but the reason we have logs is because they actually tell you what (nearly everything) the software is doing behind the scenes and what data they are working with currently. This is the only way possible to solve this kind of problem. I actually don't want to read your logs. There is nothing more mundane than reading megabytes of somebody else's logfiles looking for a needle in a haystack. But it beats playing these guessing games.
Link to source
Visit
Poke
Giac El Vecio
Sun, 21 Jul 2024 09:16:01 +0200
from Diversi Spiritus
I can certainly do this, but it may be useful to know that, after I send, the first 40 seconds or 70, everything seems normal, the sent post appears in my timeline, the delivery report is normal as well, but immediately after, and for about 7 or 10 minutes, that the site gets "stuck in a loop".
Link to source
Visit
Poke
Giac El Vecio
Sun, 21 Jul 2024 10:50:38 +0200
from Diversi Spiritus
"After a minute or two, copy the logfile somewhere and email it to me. Then turn your logging level back down.."
This part is perhaps difficult or impossible if I use that ActivityPub list of 18 people, because exactly after a minute and a few seconds I lose control of the site and can't do anything for x minutes.
If I just manage for 5, but with 5 people the site doesn't seem to have any problems, and therefore I don't think it's useful, then someone might say "but if with 5 it's fine but with 18 it's not, then it's a problem of your server", at this point I can reply "then explain to me why if I send for 100 zot people the site behaves perfectly?"
Link to source
Visit
Poke
Haakon Meland Eriksen (Els Mussols)
Sun, 21 Jul 2024 16:51:55 +0200
from Streams at els Mussols
You should be able to monitor and log what is happening when you do send and receive to your AP connections. Do that.
How about these random action items:
ping the site of your AP your AP contact one by one. Is it reachable? Is it responding very slowly?
is the local disk full, i.e. 80%+, it happens often that temporary files in the background actually fill and swamp the disk, so you have no input/output to speak of, and no evidence without active monitoring of temp files.
Link to source
Visit
Poke
Giac El Vecio
Sun, 21 Jul 2024 20:03:28 +0200
from Diversi Spiritus
I'll see what I can do, I'm on holiday in Italy, and the other person with access to the server is in Brazil, there are 5 hours difference, and we need to agree and find an appropriate time.
Link to source
2 Likes
Haakon Meland Eriksen (Els Mussols) (2 months ago)
Bill Statler (2 months ago)
Visit
Direct Message
Poke
elvecio
Mon, 22 Jul 2024 09:35:24 +0200
from mitra
@
Giac El Vecio
@
Haakon Meland Eriksen (Els Mussols)
my site is now completely broken, I can't post, I can't see the connections, the lists/circles only open blank pages.
Link to source
1 Dislike
Bill Statler (2 months ago)
Visit
Poke
Mike Macgirvin 🖥️
Sun, 21 Jul 2024 22:11:53 +0200
from Y
Let's start over. I don't have a clue what your server is doing or why. And I'm really tired of playing guessing games.
If you look at admin/logs, there's a setting for the log level. Set it to Normal. Send a post to your ActivityPub friends.
When the server goes wonky, note the time.
Find that time in the log. What was your server doing? Tell me. If you can't figure it out, send me a few hundred or few thousand lines starting at that time.
This might be enough to solve the problem. If it isn't, turn the log level to 'Debug' and repeat. This will provide some details you didn't see on the 'Normal' setting. Again, this will probably solve the problem, without burying either of us in data structures.
But if you still can't figure out what it's doing, turn it all the way up to 'All' and do it again. This will also provide all the data structures that the software was processing at this time. It's a huge amount of data. I doubt that you'll be able to process this or even guess what was going on. Send it to me. But I don't need or want it all. Just send a minute or two starting when you noticed it going wonky.
Link to source
Visit
Direct Message
Poke
elvecio
Tue, 23 Jul 2024 13:17:34 +0200
from mitra
@
Giac El Vecio
It's the database that explodes the CPU after we post for x number of activitypub people.
What can cause this? Apparently the solicitation is much greater when we send something to activitypub people. How can we record and adjust the server so as not to drive the database and consequently the server crazy? Any useful tricks? Any special numbers in the configurations to self-regulate better?
Link to source
Visit
Poke
Waitman Gobble
Tue, 23 Jul 2024 20:43:14 +0200
from Get Ready To Rumbly
I made a script once that would only run the cron task if it wasn't already running.
You can also fiddle with the cron task frequency.
However the problem could be traffic from mastodon incoming, as i recall it can be quite lot of requests.
Link to source
Visit
Poke
Bill Statler
Tue, 23 Jul 2024 21:07:56 +0200
from Bunny of Doom
Traffic from Mastodon, interesting possibility!
Or, what if it is a
single
misconfigured ActivityPub instance hammering your site?
Link to source
Visit
Poke
Giac El Vecio
Tue, 23 Jul 2024 21:41:09 +0200
from Diversi Spiritus
in my case, however, it manages incoming traffic well, the problem is that sending for 18 people of activitypub blocks the site for x minutes
Link to source
Visit
Poke
Bill Statler
Tue, 23 Jul 2024 21:52:54 +0200
from Bunny of Doom
I'm wondering if Mastodon (or some other AP platform) does something when it receives a post/comment from your site. Like, does it fetch your profile, or some other data from your site? Does it tell all its Mastodon friends to fetch this data at the same time?
I remember a few months ago, owners of small websites were complaining that if somebody shared a link to their site on Mastodon, they immediately got thousands of requests for preview images and text, which acted like a DDoS attack and locked up their webserver. So I'm wondering if something similar could be happening to you? Maybe it was a Mastodon update that caused this, and not a streams update.
Link to source
Visit
Direct Message
Poke
elvecio
Tue, 23 Jul 2024 21:52:07 +0200
from mitra
@
Waitman Gobble
to be more clear:
a single comment that arrives on my post is then sent from my site to all recipients (so more than 18 activitypub people).
Link to source
Visit
Poke
Haakon Meland Eriksen (Els Mussols)
Tue, 23 Jul 2024 16:25:37 +0200
from Streams at els Mussols
This is a long read, but maybe you can transfer some of the analysis?
https://www.sqlservercentral.com/articles/a-spectacular-case-of-100-cpu-caused-by-online-reindexing
Link to source
Visit
Direct Message
Poke
elvecio
Tue, 23 Jul 2024 18:50:11 +0200
from mitra
@
Giac El Vecio
Well, I discovered a new problem: if a person from the mastoverse comments on something on my site, what does my site do? What we expect, collects the comment and himself distributes it to the recipients. By doing this my site goes haywire, and not through my fault, but because an external person commented on a post originating on our site. It is a very unsustainable situation.
Link to source
Login
Login/Email
Password
Remember me
Login
Register
Password Reset
Remote Authentication