A bit more context there since you might wonder why customers can cause Sev1's.
Well, I work for a Database Technology company and we provide a managed service offering. This managed service offering has SLA's that essentially enforce a 5 minute response time for any "urgent" issue.
Well, a common urgent issue is that the customer suddenly wants to load in a bunch of new data without informing us which causes the cluster to stop accepting write loads.
It's to the point where _most_ if not all urgent pages result in some form of scaling of the cluster.
Since this is a customer driven behavior, there is no real ability to plan for it - and since these particular customers have special requirements (and thus, less ability to automate scaling operations), I'm unsure if there is any recourse here.
It's to the point that it doesn't even feel like an SRE team anymore - we should just instead be called "On-demand scaling agents". Since we're constantly trying to scale ahead of our customers.
All in all, I'm starting to feel like this is a management/sales level issue that I cannot possibly address. If we're selling this managed service offering as essentially "magic" that can be scaled whenever they need then it seems like we're being setup for failure at the organizational level. Not to mention, not being smart about costs behind scaling and factoring that into these contracts.
So, fellow SRE's have you had to have this conversation with a larger org? What works for something like this? What doesn't? Should I just seek greener pastures at this point?
P.S. - Posted c/Programming due to lack of a c/SRE
Look, we can debate the proper and private way to do Captchas all day, but if we remove the existing implementation we will be plunged into a world of hurt.
I run tucson.social - a tiny instance with barely any users and I find myself really ticked off at other Admin's abdication of duty when it comes to engaging with the developers.
For all the Fediverse discussion on this, where are the github issue comments? Where is our attempt to convince the devs in this.
No, seriously WHERE ARE THEY?
Oh, you think that just because an "Issue" exists to bring back Captchas is the best you can do?
NO it is not the best we can do, we need to be applying some pressure to the developers here and that requires EVERYONE to do their part.
The Devs can't make Lemmy an awesome place for us if us admins refuse to meaningfully engage with the project and provide feedback on crucial things like this.
So are you an admin? If so, we need more comments here: https://github.com/LemmyNet/lemmy/issues/3200
We need to make it VERY clear that Captcha is required before v0.18's release. Not after when we'll all be scrambling...
EDIT: To be clear I'm talking to all instance admins, not just Beehaw's.
UPDATE: Our voices were heard! https://github.com/LemmyNet/lemmy/issues/3200#issuecomment-1600505757
The important part was that this was a decision to re-implement the old (if imperfect) solution in time for the upcoming release. mCaptcha and better techs are indeed the better solution, but at least we won't make ourselves more vulnerable at this critical juncture.
The problem to federation is that there is at least some complexity to subscribing to other communities - at least right now. I love it here, but I have to be honest, it is a bit difficult to navigate for most.
But perhaps we're missing the point? What if we should be trying to sell the local aspect harder?
There is a demand for hyper-local networks as evidenced by Nextdoor. Couple this with an increase in people wanting to have a better digital commons - one not controlled by a single corporation.
For my instance tucson.social, I'm going to get some signs printed and do some local advertising. Pretty sure if I sneak some signs around the U of A campus, but mostly in public spaces where it's legal and proper.
I think that the local nature of all of this makes advertising a bit more effective and locally relevant. I don't want to sell it as a "reddit" but as another "place" that will exist whether or not reddit does (due to the non-profit or whatever other org we construct) to talk about our city.
At the same time I do this, I want to reach out to important community members to see if they might be interested in donating once I have a formal non-profit. I'd make sure to emphasize the utility such a site might have to local businesses once advertising is possible.
I have no idea if this will all work, but it's something I'm trying to do anyways. I just believe communities should be local, and that the online representation of them is as close to a mirror of the local one as possible. I also want to foster conversations that help people grow and connect.
So maybe, one of the admins here sees this, because I'd really like to join forces in a more meaningful way and hopefully gain the ability to deliver a meaningful experience to all sorts of communities. My immediate skills are technical, and perhaps if I follow along and learn your tips along the way, I can have the best possible chance of making this happen - and constructing the blueprints for others to follow.
I'm also open to other wisdom from the beehaw community! Have you done local marketing and advertising? I could use some tips. Have you formed a non-profit before? I'd definitely like to hear from you! Are you in Tucson and want to get more deeply involved? - dm me!
The only thing I'd prefer not to hear is how difficult it is. I'm fully aware that I've chosen to go all in on terrible odds. I don't really care anymore. lol