Comment on Discussion: Long-term need for automation tools for moderation

• asudox@lemmy.asudox.dev ⁨1⁩ ⁨day⁩ ago

  Well, my initial idea was to build this only for lemmy.

  However, the API was not good enough for my use case. Polling new posts and comments was my main issue with it. So mostly scaling issues. You could miss some posts and comments and the amount of API requests will get bigger with the amount of communities the bot will moderate. There are also some problems with rate limiting.

  They can be solved by directly querying the database, but who’s going to give you database access? So you’d have to selfhost lemmy yourself just for the bot. And I’d imagine the database would grow pretty fast with the number of communities. I explicitly do not want to store any posts or comments. And even then, maintaining the required SQL query for Lemmy posts is the last thing I want to do.

  Another solution would be using Lemmy’s new webhook system, but I don’t know how reliable it will be.

  So I stopped halfway through and started a new project with new goals:

  • Make a new federated platform

  With federation, the problems above would be solved. This also allows it to be hosted without having to find an instance for it or even self host it yourself.

  • Stronger integration with platforms via a modular federation system

  If I made it depend on Lemmy, a strong integration with other platforms wouldn’t be possible. Piefed has features that Lemmy doesn’t, for example. People can maintain a set of platform specific activitypu bstructs and enable the bot to federate with that platform.

  ---

  Not really answering your question, but I’d like to make a clarification: The bots will only be able to operate within the community (or I guess group) boundaries. They cannot manage any instances. Furthermore, my main intention is for them to be used primarily as moderation bots, but they can also be used as general purpose bots within the community.

  source

  • nutomic@lemmy.ml ⁨9⁩ ⁨hours⁩ ago

    To get new posts and comments for all known communities you only need to make regular requests to /api/v3/post/list?limit=50&sort=New&type_=All and /api/v3/comment/list?limit=50&sort=New&type_=All. Its not necessary to make separate requests for each community. The default rate limit allows 180 read requests per minute so you can comfortably poll this every second (in practice every 30s or so should be enough). If you miss an item (ie post or comment id was skipped) just load the following page.

    The plugin system in 1.0 would be another option. It will still take some time until that is released, but there shouldnt be any reliability issues.

    Youre right that federation solves these problems, but instead you get another problem of writing all this federation code and making sure it is compatible with different platforms. Lemmy’s federation code has around 12k lines so that is a lot. It seems much simpler to use the API for Lemmy, Piefed etc and write abstractions for common functionality.

    Anyway this is my opinion. Its your project so in the end its your decision how to implement it.

    source

    • asudox@lemmy.asudox.dev ⁨35⁩ ⁨minutes⁩ ago

      Yep. I did use those endpoints with the ModeratorView (The bot doesn’t need posts or comments from communities it doesn’t moderate) on my first attempt. I went with the federation approach at last because of future scalability issues. Though that is probably an exaggeration. If the default rate limit is 180 requests per minute, that would be more than enough, honestly. The scale at which the scalability issues I mentioned appear at about more than 4500 comments/posts per minute. If I am being realistic here, we’ll probably never reach that in the near future. So actually the rate limiting issue is practically not an issue in the foreseeable future.

      The plugin system would work. The fetching problem would also disappear.

      Though I don’t think the federation code would be huge. I am not trying to make it compatible with all platforms. For example I’ll write the required ActivityPub structs to send moderation related activities and actors. The Group’s instance would handle distributing the activities, so even though this project might not federate with Piefed for example, it would still receive the activities the bot sends to the Group’s inbox through the instance’s software, Lemmy.

      If someone wanted to get the bot to work on another Ap platform that supports groups, they would have to write the necessary Ap actors, activities, and a bit of glue code, and that would be it… or at least that’s how I’m planning it.

      I guess I’ll try to work with the plugin system if I can’t achieve what I want and keep it simple. It would at least be a learning experience, if nothing else. Thank you for the info.

      source