[Help] New self-hosted instance not crawling communities - eviltoast

Just got my own instance up and running finally, but it doesn’t seem to be federating as expected.

I can search for and subscribe to communities, but unless I specifically search for the URL (e.g. selfhost@lemmy.ml) nothing shows up. I’m also making this post from my instance as well (to see if anything happens…)

Is this something I just need to wait a few hours to happen or is something incorrect.

Thanks in advance


Edit: Oh, and when I do subscribe to a community, I can only see new posts. It doesn’t retrieve older posts

  • redcalcium@c.calciumlabs.com
    link
    fedilink
    arrow-up
    6
    ·
    edit-2
    1 year ago

    I can only see new posts. It doesn’t retrieve older posts

    This is normal. You’ll only receive new posts going forward from ActivityPub. Think of it like joining a mailing list, you’ll only receive new mails after subscribing. If you want to pull older post, you’ll have to do it manually by pasting the post (or comment) url in the search field.

    • NotNotMike@notnotlemmy.comOP
      link
      fedilink
      arrow-up
      2
      ·
      1 year ago

      Yep you’re right. And I’ve discovered that, in order to see posts from a community, at least one user in your instance needs to be subscribed to a community to view it.

      Would be nice if someday the browse functionality provided by some sites was integrated into Lemmy but for now I had to go through and manually subscribe to a bunch of things.

      I’d use a scraper but there’s a lot of content I don’t care to see right now, and I’m assuming subscribing comes with a small storage cost, so may as well avoid it for now

  • fireshaper@social.belowland.com
    link
    fedilink
    arrow-up
    5
    ·
    1 year ago

    You don’t get federated automatically. On your own instance you will only see posts in communities that you subscribe to. What I did was set up a “bot” user that I used to subscribe to a bunch of communities that I might find interesting when just browsing the All feed but I didn’t want in my personal Subscribed feed. There’s a few tools that will automatically subscribe a bot user on your instance to all the top communities but I found just manually going through https://lemmyverse.net/communities and clicking subscribe as that bot user worked for me.

  • datallboy@lemmy.world
    link
    fedilink
    arrow-up
    3
    ·
    1 year ago

    What version are you using? There were issues federating with v 0.18.0. I updated my instance to 0.18.0-rc.10 and that resolved the issues.

  • terribleplan@lemmy.nrd.li
    link
    fedilink
    English
    arrow-up
    3
    ·
    edit-2
    1 year ago

    Things don’t get backfilled, so until a new action happens on an old post/comment/etc they won’t show up on your instance. New things should make their way in eventually though.

    Taking the link of a specific post/comment from the community instance and searching for it from your instance should populate it on your instance, just like you probably had to do to get this community to show up so you could subscribe/post at all.

    There are backfill tools/scripts, but unless you really want old posts I wouldn’t use them. It unnecessarily increases the load on already struggling popular/overloaded instances like lemmy.world.

    • NotNotMike@notnotlemmy.comOP
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      Do you know how much data is put onto the local device when subscribing to all these communities? Ballpark is it in the MBs? GBs?

      Trying to determine whether discovering things I don’t want is worth it

      • maybe_its_flare@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        edit-2
        1 year ago

        My personal instance of about:

         communities | persons | posts | comments | post_likes | comment_likes
        -------------+---------+-------+----------+------------+---------------
                 568 |  103616 | 72621 |   383006 |    1906668 |       2152368
        

        takes up about 17GB of storage thus far!