Lemmy World SysOp - eviltoast

Lemmy.World is looking for 4 new Systems operators to help with our growing community.

Volunteers will assist our existing systems team with monitoring and maintenance.

We’re ideally looking for chill folks that want to give back to their community and work on our back-end infrastructure. Must have 4+ years of professional experience working in systems administration. We are not looking for junior admins at this time. Please keep in mind that, while this is a volunteer gig, we would ask you to be able to help at least 5-10 hours a week. We also understand this is a hobby and that family and work comes first.

Applicants must be okay with providing their CV and/or LinkedIn profile AND sitting for a video interview. This is due to the sensitivity of the infrastructure you will have access to.

We are an international team that works from both North America EST time (-4) and Europe CEST (+2) so we would ask that candidates be flexible with their availability.

If you are in AEST (+10) or JST (+9) please let us know, as we are looking for at least one Sysadmin to help out during our overnight.

You may be asked to participate in an on-call pool. Please keep in mind that this is a round-robin style pool, so it’s alright if you’re busy as it will just move along the chain.

If you’re interested and want to apply, click here.

  • quantum_mechanic@lemmy.world
    link
    fedilink
    arrow-up
    317
    arrow-down
    6
    ·
    1 year ago

    I have the experience, but not the energy nor passion as I am almost burned out already. I hope you find some awesome people.

    • indierockspockears@lemmy.world
      link
      fedilink
      arrow-up
      73
      arrow-down
      1
      ·
      1 year ago

      If it was a paying gig would you consider it? 5 to 10 hours a week, let’s say 10. What kind of salary would you expect?

      Just curious.

      • JJROKCZ@lemmy.world
        link
        fedilink
        English
        arrow-up
        103
        arrow-down
        1
        ·
        1 year ago

        I also have the desired skill set and experience far surpassing what they’re asking for but not the time or energy to do this since my work already demands 60+ hours a week and on-call from me. Yes I’m American.

        To answer your pay question; around 4-500 would be the average pay for 10 hours this position in the working world. Since the fediverse instances have next to zero reliable income (donations can’t be counted as reliable) I understand this is a difficult if not impossible bill to pay. This is why they’re asking for volunteers whose work schedule is more sane and therefore have the energy and time to commit. I wish I was available to do so, maybe if my current job search is successful at finding something more chill.

      • quantum_mechanic@lemmy.world
        link
        fedilink
        arrow-up
        46
        arrow-down
        3
        ·
        edit-2
        1 year ago

        Knowing this is a volunteer project, I’d never request renumeration. If I were contracting with a large company, I guess I’d charge 300-500 per day. That’s just based on quotes I get on LinkedIn, as I’ve never worked as a contractor. Also I couldn’t have it interfere with my main job, where I’m also on call, so it would be lower priority.

          • Chunk@lemmy.world
            link
            fedilink
            arrow-up
            11
            ·
            1 year ago

            I can’t tell if this career path is worth the stress but you’re describing a lot of money. And it makes me feel like this is worth it.

            • Smokeydope@lemmy.world
              link
              fedilink
              arrow-up
              36
              ·
              edit-2
              1 year ago

              Survivorship bias. the IT market is a lot more saturated now than it was 20 years ago. This person got in somewhere good early on and rode that career train. These opportunities rarely exist today unless you arre a charismatic super talented genius. I would not bet on most people being able to ever pull those kinds of numbers before burning out. Being a money chaser is not worth it in today’s world, value your health mental wellbeing and personal life equally if not more than your bank account. Live well below your means and learn how to save/invest, and find a life/career path you can feel good and satisfied in even if its not paying a whole lot. You r quality of life will skyrocket and you will have enough $ to feel secure. I worked years in industrial trade, made good cash, it wasnt worth it in the end. Just my 2 cents.

              • Unforeseen@lemmy.world
                link
                fedilink
                arrow-up
                10
                ·
                1 year ago

                Yeah it’s hard to gain the high value skillsets these days, and I think it’s one of the reason those of us that have been in it for decades are able to do fine now. I got lucky where early on in my career I got in with a company that was small and had enormous growth over a 10 year span.

                I’m similar paid to the person in question and am also an independant contractor. I make similar due to my experience level and rather unique combination of skills so I just cut my hours way back and typically work a few hours a day. So burnout is a non issue. I take no work where I’m on call or “first responder” and I make sure it’s always written into the contract that way.

                It took 25+ years of busting ass to get here, although I have no regrets, and I recognize I am incredibly lucky to have the circumstances play out this way.

              • tomkatt@lemmy.world
                link
                fedilink
                English
                arrow-up
                8
                ·
                edit-2
                1 year ago

                This person got in somewhere good early on and rode that career train. These opportunities rarely exist today unless you arre a charismatic super talented genius.

                Not really true. It takes a bit of knowing your worth, advocating for yourself in interviews, and job hopping as needed for pay raises every year or two while continuing to build your skills both on the job and outside of it. IT isn’t an industry that lends itself to job stability and high pay if you stay in a role long term, and stagnation can certainly be a factor if you decide to stop learning things.

                Also. it’s still very possible to get in, but focus these days is DevOps, automation, virtualization, and more recently, AI. You won’t make bank in some shitty low tier helpdesk role.


                A good start would be certification path to pick up some straightforward “guaranteed to get you work” kind of certs like:

                • Linux+
                • Network+
                • VMware VCP-DCV (and later with experience maybe VCP-CMA)
                • Any Redhat cert
                • Security+ if you’re interested in cybersecurity and/or federal work (USA, not sure about other places)

                Alternately, getting a few programming languages under your belt is totally doable for free with Youtube and other online courses and then doing your own projects with public repositories on Github for prospective employers to see. Getting a foot in the door with dev is gonna be very luck of the draw though.

                You definitely wont’ start out making a wage that high on the Ops side, but finding a foot in the door at between $25 and $30 an hour shouldn’t be hard once you get some bare minimum experience under your belt.

                College grads may have an easier time, but I wouldn’t know, I dropped out and went the certification/experience route some 15+ years ago.

                • corsicanguppy@lemmy.ca
                  link
                  fedilink
                  arrow-up
                  3
                  ·
                  1 year ago

                  Any Redhat cert

                  Pick up the Oracle cert. It looks like Oracle is gonna be more valuable than RedHat for a while.

              • nijave@lemmy.world
                link
                fedilink
                English
                arrow-up
                5
                arrow-down
                1
                ·
                1 year ago

                $300/day @ 8 hours = $37.50/hr / $78k/yr $500/day @ 8 hours = $62.50/hr / $130k/yr

                Which are imo on the low end of U.S. tech salary. Considering a contractor also has overhead like healthcare, business expenses, any time spent in administration take home would potentially be lower.

                The market is softer right now but will probably rebound in a couple years.

              • banneryear1868@lemmy.world
                link
                fedilink
                arrow-up
                4
                ·
                1 year ago

                I’d say IT sysadmin work sucks unless the company runs proper shit and has a handle on it, which is more than often beyond your influence in the position. A lot of times the company doesn’t want to hire enough people, and overworking yourself will never be enough to get everything done. So you spend your days trying to handle the highest impact tasks coming from management and above, constantly being pulled in different directions, and never really feeling like you got anything meaningful accomplished. IT priorities also often conflict with business priorities, and if not handled right you will be caught in the middle of this and all your decisions will be scrutinized by the interested parties.

            • corsicanguppy@lemmy.ca
              link
              fedilink
              arrow-up
              11
              ·
              1 year ago

              describing a lot of money

              If you’re in IT for the money, you’re gonna hate it or yourself before long. Lagom.

            • agent_flounder@lemmy.world
              link
              fedilink
              arrow-up
              9
              ·
              1 year ago

              Infosec is a good choice if you have the ability to think analytically and to learn new things faster than average. Demand for such jobs is probably going to be growing for a while still.

              • DokPsy@lemmy.world
                link
                fedilink
                arrow-up
                4
                ·
                1 year ago

                That barrier to entry is no joke tho. I’m still trying to get myself to a point where I can confidently put my hat in the ring for infosec roles

      • rolaulten@lemmy.world
        link
        fedilink
        arrow-up
        14
        arrow-down
        37
        ·
        1 year ago

        So I’m a systems engineer in the real world for an (almost) unicorn (current valuation might even have tossed us over that magic number). My salary is on the lower end of the spectrum but I’m happy with it because normally the work life balances is dandy. My total comp is well into 6 figures USD. Oh and I’m fully remote.

        Now, this is not something you can get out of highschool. I’ve been working with Linux for 10+ years, built (and maintained) entire AD forests, have a fairly deep understanding of networking and containerization, etc.

        Again. You don’t start like me. You start getting a gig in front line help desk and answer questions. In your free time at work you learn (that’s never going to stop). Eventually your outgrow help desk and move into some other role (and keep learning). The people who are successful in this field A) can always be learning, B) have a means to destress/avoid burnout and C) have customer service skills.

    • _bug0ut@lemmy.world
      link
      fedilink
      arrow-up
      12
      arrow-down
      1
      ·
      1 year ago

      oof its like you’re either me or i’m you. hope you find your way past the burn out or out of it if you end up sinking into it. i’m going on like 3.5 years of battling it and there are better days and worse days, but i have no idea what else to even do. managing infra and writing code have been my entire career up to now.

      • quantum_mechanic@lemmy.world
        link
        fedilink
        arrow-up
        11
        arrow-down
        1
        ·
        1 year ago

        I’m about 1.5 years into it. Lost most passion for the job, but there are flashes of motivation here and there. Considering trying to move into full time development, but that would take a lot of effort. Tired of keeping up with the kubernetes ecosystem too.

        • _bug0ut@lemmy.world
          link
          fedilink
          arrow-up
          10
          ·
          edit-2
          1 year ago

          Ah, alright, not quite me - I’ll be 14 years deep in November. Honestly, one of the things that kept me motivated over the years was moving around - I stayed at the same company, but I started out doing QA (by hand, no automation), then got moved to handle release management, then moved to IT as a general Linux admin and spent a few years doing that, made friends with an infosec manager and he offered me a spot on his team working remote and doing container/docker security which morphed into a cloud security thing after he left the company (I hated the cloud). A couple years back I moved back to non-cloud/non-infosec work doing automation stuff with Ansible mainly, and for the time being only for our on-prem infrastructure (this may change in the future and I’m not really looking forward to it all that much).

          At this point, nothing is really helping get my head back into the game 100% but I can still put out work and I’m just trying to find the joy in small victories and chasing the high you get when the code you wrote works flawlessly. I’m blessed to have a solid management structure above me who a) know me, b) like me (and the feeling is mutual, they’re all great people), and c) are happy with my output.

          I don’t envy you working with kubernetes - my time in container security came during the early days of large companies trying to move to turning everything into microservices. It was a wild west kind of vibe and I basically had free reign… nowadays, I don’t think I’d enjoy any of that in its current form.

          I have great soft skills and I write pretty well, but outside of that my skillset is basically a degraded/decayed technology one because I’ve been treading water for a while now and not actively keeping up with all the shit in our sector that changes on a constant basis.

          I’ve also seriously weighed moving into development, but I’m not sure if that’s just going to fix anything for me. I like writing Python, but I don’t know how that would feel full-time. Sucks, man.

          • corsicanguppy@lemmy.ca
            link
            fedilink
            arrow-up
            4
            arrow-down
            1
            ·
            1 year ago

            14 years deep in November.

            Since July 1996 here, amateur/volunteer since ~93.

            not actively keeping up with all the shit in our sector that changes on a constant basis

            Learn the stuff that doesn’t change as fast for a foundation, and go from there into stuff that tickles the coding fancy. Let the mayfly tech be handled by caffeine-addled thrill-chasers, while you just build stuff and sleep well.

            • _bug0ut@lemmy.world
              link
              fedilink
              arrow-up
              3
              ·
              1 year ago

              That’s basically where I’m wedging myself in now. Ansible and Python, higher value but lower stress projects. Bigger wins, but ones that are able take the time needed to put them together, test, and refine.

              It’s almost a back-to-my-roots kind of thing for me, but with a fresh twist in terms of approach. I’m basically writing automations that make life easier for ops guys, to boil it down to it’s essence.

        • corsicanguppy@lemmy.ca
          link
          fedilink
          arrow-up
          3
          ·
          1 year ago

          Tired of keeping up with the kubernetes ecosystem

          Yep. You can either do the work, or spend many times that on people to do the work in kubes.

        • nijave@lemmy.world
          link
          fedilink
          arrow-up
          2
          ·
          1 year ago

          Oof Kubernetes. Awesome community and powerful tech but the rate of change is insane. Like 6 months of CNCF is like 10 years of Red Hat + Java.

  • ekZepp@lemmy.world
    link
    fedilink
    English
    arrow-up
    230
    arrow-down
    1
    ·
    edit-2
    1 year ago

    In my opinion Lemmy.world should start selling a bit of merchandising (t-shirt and so on), just to add a little on the donation side.

    BTW. the donation links are in the group info.

  • edric@lemm.ee
    link
    fedilink
    English
    arrow-up
    146
    arrow-down
    5
    ·
    edit-2
    1 year ago

    I read it as Lemmy World PsyOp at first and thought there’s some conspiracy bullshit happening on the instance. lol. Good luck on your search!

  • Gallardo994@lemmy.world
    link
    fedilink
    arrow-up
    145
    arrow-down
    11
    ·
    1 year ago

    A CTO of 5 years with many more years of experience here. I would be really glad to help, but not in scenario where I have to prepare a CV for international readers and have no pay at all as this looks to me like a job application with no job.

    Considering you are running on Digitalocean infrastructure, I am completely unsure why you would ever need Ansible and Terraform as it just adds complexity without certain benefit, especially if you mention Kubernetes which DO already provides with two clicks.

    I’d personally suggest trying out ArgoCD for declarative clusters. With this thing, I’ve seen 2 companies maintained by a single DevOps engineer with no problems. Huge timesaver and makes everything transparent.

    In case this process changes and becomes less corporate-y and more transparent, I’ll be ready to apply. Hope you’re going to find the right people! Long live Lemmy World!

    • Mulch5516@lemmy.world
      link
      fedilink
      arrow-up
      48
      arrow-down
      3
      ·
      1 year ago

      Ansible/Teraform are portable. I don’t see it’s usage as a failing, rather as avoiding DO lock in.

      Agreed with the rest though. This is quite the ask.

    • marmarama@lemmy.world
      link
      fedilink
      arrow-up
      18
      arrow-down
      1
      ·
      1 year ago

      I’m a big fan of Kubernetes, and for larger projects the flexibility and power it brings is unrivalled. But for smaller projects, assuming equal levels of competence, delivery teams using managed Kubernetes are almost universally later and have more issues than teams that use simpler solutions. Container-as-a-service solutions like GCP CloudRun or AWS FarGate help somewhat, but are not cheap for a given amount of compute time.

      Terraform (or IaC in general) absolutely has a place, because even if you use Kubernetes, most projects have more infrastructure to manage than just the cluster - at the very least, lemmy.world has a CloudFlare proxy to manage - and clicking buttons in a management portal is not a repeatable way of deploying that, or deploying the Kubernetes clusters themselves.

      Ansible also has a place, particularly if you’re deploying onto bare metal. I wouldn’t use it for new deployments unless I had bare metal to configure and maintain, but lemmy.world is deployed onto a bare metal server as I understand it. Plus, the most effective tooling is generally the one your team understands.

    • douglasg14b@lemmy.world
      link
      fedilink
      arrow-up
      14
      ·
      1 year ago

      IaC is the right move. It transfers complexity, it necessarilt doesn’t add it. It makes your deployments reproduceable and automated.

      Which is a baseline to having highly available infrastructure. Not everyone will be familiar, or have the right mindset for that sort of DevOps.

  • cyberpunk007@lemmy.world
    link
    fedilink
    arrow-up
    88
    ·
    1 year ago

    Senior Network engineer with lots of experience in the field (servers + network), 15+ years if you need help let me know I’m happy to lend a hand.

  • cbarrick@lemmy.world
    link
    fedilink
    English
    arrow-up
    76
    ·
    1 year ago

    I’m qualified, but 5-10 hours can mean a lot of different things.

    Are you looking just for oncall/incident response, or are there more active reliability projects that you need help on?

  • Mulch5516@lemmy.world
    link
    fedilink
    arrow-up
    100
    arrow-down
    25
    ·
    1 year ago

    4 applicants x 5-10 hours is .5 to 1 full time employees. Very generously speaking the ask here is for 100k/yr in free labor. The stringent interview process is going to be very limiting on potential candidates.

    The experience isn’t going to be a learning experience since you’re looking for people that already know it all and I wouldn’t even put it on a resume, it just advertises to employers you’re ok being lowballed.

    Perhaps this is a necessity for an instance of this size, but to me that seems to indicate that lemmy.world has reached the upper end of reasonable scalability, which given the workings of the fediverse would be fine.

      • UFO64@lemmy.world
        link
        fedilink
        arrow-up
        79
        arrow-down
        4
        ·
        1 year ago

        I don’t think that user was personally complaining, more pointing out that LW might be facing it’s first crisis of resources. Good sys-ops people don’t come cheap, and bad one often can do far more harm than help.

        It’s tricky trying to handle something like this when LW is foundationally not an enterprise or anything close to a business.

  • PineapplePartisan@lemmy.world
    link
    fedilink
    English
    arrow-up
    65
    arrow-down
    1
    ·
    1 year ago

    Out of curiosity, will you be able to weed out bad faith volunteers? I am sure there are a variety of interests that would be more than willing to pay a junior admin to be a Lemmy Sysop and it’s not like the candidate will volunteer that information.

  • narp@feddit.de
    link
    fedilink
    arrow-up
    78
    arrow-down
    27
    ·
    1 year ago

    How about another approach?

    There is no good reason for Lemmyworld to keep on growing to an extent that this kind of overhead is necessary. The idea of Lemmy is decentralization and not creating a new reddit instance. Close your registration, limit your amount of communities and let Lemmy grow in other directions.

      • narp@feddit.de
        link
        fedilink
        arrow-up
        21
        arrow-down
        6
        ·
        1 year ago

        Understandable, but aren’t growth and instability related in this case? There are many instances with capacity that are already run by capable people. Just spread the load (ahem) across the Lemmy verse and only handle as much as you can. But maybe I’m missing a point, I just think that this would be the best for Lemmy in the long run.

        • rambaroo@lemmy.world
          link
          fedilink
          arrow-up
          17
          arrow-down
          3
          ·
          edit-2
          1 year ago

          I mean is that even up to them? People gravitate to the places with the most content, and right now that’s lemmy world. I think the only way they could combat that is to make lemmy world private, but it might lead to people not using lemmy at all instead of spreading out to other instances.

          The other option I see is to make the instances more specialized and basically do away with generalist instances like lemmy world. So you have an instance focused on news with its own subcommunities, one for gaming, one for politics etc. But that could hurt usability. It’s not an easy problem to solve.

          • narp@feddit.de
            link
            fedilink
            arrow-up
            20
            arrow-down
            4
            ·
            edit-2
            1 year ago

            I agree with your points and like the idea of more specialized instances and also country related instances. I think it’s solvable if the different admins work together.

            Lemmy.world doesn’t have to go private, they could just not accept more users and communities for a while. It wouldn’t change much since everyone will still be able to post and comment on Lemmy.world from all instances. New users would just have to choose a different instance that’s all.

            For me that’s the whole point, I don’t see any benefit of a big instance, the Lemmyverse doesn’t need one.

  • corsicanguppy@lemmy.ca
    link
    fedilink
    arrow-up
    51
    arrow-down
    2
    ·
    1 year ago

    You may be asked to participate in an on-call pool

    Unfortunately, this is where I noped out. But I ditch most paid positions where I can’t avoid standby-time.

  • Obsession@lemmy.world
    link
    fedilink
    English
    arrow-up
    31
    ·
    1 year ago

    Threw my hat in the ring, I’m a senior devops engineer.

    Don’t have any Lemmy experience though. I have no desire to self host it, but I wouldn’t mind being part of the team to maintain a large instance.

  • zeppo@lemmy.world
    link
    fedilink
    English
    arrow-up
    25
    ·
    1 year ago

    I spent 4-5 years running a high traffic server using Linux, nginx, apache, php and whatever we did with Python, and would be glad to help. This was in 2010 though, so….

    • hemmes@lemmy.world
      link
      fedilink
      English
      arrow-up
      13
      ·
      1 year ago

      I know, I meet all of the Soft skills, like four of the Systems knowledge, and maybe 0.5 of the Ideal Devops skills. But I have certifications, love 90s cartoons, and hate oatmeal raisin cookies - so I’m thinking I’m perfect.