Chenyo's org-static-blog

Posts tagged "telegram":

25 Sep 2024

Build a free Telegram Mensa bot

1. Previously on my Telegram bot

In the post Build a free Telegram sticker tag bot, I detailed the process of integrating various online services to create a sticker tag bot. After using it for a month, I encountered an intriguing production issue: on one occasion, the inline query result was incomplete. Interestingly, restarting the Render deployment resolved the problem, though I never fully understood the root cause.

Despite this minor hiccup, the sticker tag bot has proven to be reliable most of the time. However, I found myself not utilizing it as frequently as anticipated. This was primarily because sending a recent sticker directly is often more convenient than invoking the bot by name.

At the conclusion of that post, I hinted at a second functionality I had in mind for the bot: a Mensa bot designed to inform me about the daily offerings at each Mensa (university cafeteria).

2. Why do I need a Mensa bot

One mobile app I frequently use on weekdays is Mensa, which lists daily menus for each Zürich Mensa to help people make their most important decision of the day. However, it was merely an inconvenience for myself that the app lacked images of the meals. I found it difficult to imagine the dishes based solely on the menu descriptions. To fix this, I decide to add the meal images myself.

3. How to get the data

I couldn’t find an official API for this, so I decided to scrape the webpage myself. Here’s where things got tricky: the Mensa web pages use JavaScript to render content. This meant I couldn’t just grab the page - I needed a browser to run the JavaScript first.

3.1. Scrape menus locally

On a local machine, it’s pretty straightforward. You just grab a scraper library like go-rod and figure out the right API calls. After you’ve snagged the page, you can use an HTML parser like goquery to pull out all the menus.

3.2. Scrape menus on Render

The only snag with go-rod is it’s too bulky for a Render free account. It needs to install Chromium first, but Render’s 512M RAM can’t handle that. I didn’t want to hunt for another free host, so go-rod had to go.

Claude then pitched the idea of online scraping services. Most I found were expensive, aimed at heavy-duty scraping. But I lucked out with AbstractAPI, offering 1000 total requests for free. If I’m smart, that could last me about half a year. ScraperAPI seemed promising with 1000 monthly free requests. But it choked on Javascript rendering for my targeted pages even with render=true.

AbstractAPI has its quirks too. The scraped result comes out messy, full of \n and \&#34. So I had to clean it up before goquery can make sense of it.

4. Performance

AbstractAPI’s free plan only lets you make one request per second. That’s kinda slow when you factor in page rendering and parsing time. The full HTML page is a whopping 3M, so I’ve gotta wait a bit for each bot request.

I thought about caching results to cut down on requests. But here’s the thing: menu images usually update right before meal times. If I didn’t catch the image last time, I need the latest scoop. So, I end up scraping fresh data for each Mensa every single time.

5. Conclusion

Here’s a quick look at what the bot can do now: it tags stickers and scrapes Mensa menus. Keep in mind the GIF is sped up to twice the normal speed.

telegram.gif
Figure 1: The current Telegram bot

As in the previous post, I aim to demonstrate that while Internet services can be costly, there often remain free solutions for building hobby projects. This may require more extensive research and additional processing, but it’s still feasible. I hope this continues to be the case in the years to come.

The repository is also public now.

Tags: tool telegram
08 Sep 2024

Build a free Telegram sticker tag bot

1. What happened

When I started to use Telegram, I really missed being able to search stickers by text, like I can in WeChat. Sadly, Telegram only lets you search emojis by text, or find stickers using emojis.

After digging around, I discovered the easiest way to add this cool feature was through Telegram bots. The idea? Store a tag-to-sticker map in the bot, then fetch all matching stickers when given a tag in the bot’s inline mode. There are a few sticker tag bots on Telegram already, but they’re either dead or can’t handle Unicode input like Chinese characters. Plus, I’m not super keen on trusting my data to someone else’s database. Moreover, I might want to use a bot for other personal stuff later.

So, I decided to build my own Telegram bot.

2. What do I need

My goal was to create a bot using only free services, including cloud storage for key-value pairs and a hosting platform to keep the bot running.

I stumbled upon Render from an X recommendation which offers 750 hours per month for free deployment (which equals 31 days), so I deployed my bot there once I got the bot running locally. But then I found out Render’s free tier doesn’t offer permanent storage and shuts down services after 15 minutes of inactivity.

A sticker tag bot without memory isn’t much use to anyone, so I went hunting for free cloud storage. With some help from Claude and Perplexity, I discovered Firebase Realtime database, which offers 1GB storage and 10GB throughput per month on its free tier.

Even with cloud storage, a bot that konks out every 15 minutes just won’t cut it - I need my stickers now! So my next quest was finding a way to keep the bot awake, which led me to UptimeRobot. It’s actually a web monitoring tool, but I can use it to ping the bot regularly, tricking it into staying “active” instead of dozing off.

So, to sum it up, building this sticker tag bot required:

  • a Telegram bot from BotFather,
  • a free Render deployment,
  • a Firebase’s free key-value storage, and
  • an UptimeRobot’s free monitoring service.

However, these services do not work together automatically. Gluing them together required additional effort.

3. How to build a bot

The first step in building any bot is asking BotFather for a new bot and keeping the bot token secure. Telegram offers a helpful tutorial that explains the process using Java. Examples in other languages can be found in their Gitlab repo. In my opinion, the most challenging part here is creating a unique bot username that is still available.

The next step involves working with the Telegram bot API in the chosen programming language. This includes learning how to handle messages effectively. For example, I used tgbotapi(Golang).

3.1. How to build a sticker tag bot

A sticker tag bot needs two main functionalities:

  • Handle direct messages to add new sticker tags.
  • Handle inline queries to search for stickers using a given tag.

To implement the first functionality, I created a simple state machine with two states:

  1. The initial state waits for a sticker input and then moves to the tag state.
  2. The tag state waits for any text input to use as the tag for the previous sticker.

To implement the second functionality, one needs to use the InlineQueryResultCachedSticker method.

For local testing, one can use a lightweight local key-value storage to store and search sticker tags. I used BadgerDB(Golang) for example.

I noticed that the generated file ID for the same sticker is different each time, making it hard to check for duplicates when adding new tags. To address this, I added a /delete method to remove tags when needed.

3.2. How to make the bot private

I couldn’t find an official way to make a bot visible only to me. Suggested by Claude, I predefined a list of authorized users. Then I performed a sanity check before handling any messages.

4. How to deploy the bot on Render

Deploying a service on Render with a free account is challenging due to the lack of shell access, disk access, and non-so-live logs. The process of making everything work at this stage was time-consuming and I even contacted Render’s technical support although they only responded “Ok I will close the ticket” after the issue was self-resolved.

Three main steps are required here:

  1. start an HTTP server with several endpoints at the bot, and
  2. configure the web service and environment variables on Render’s dashboard,
  3. configure the Telegram webhook at the bot.

In step 1, starting an HTTP server at 0.0.0.0:<some-port> is necessary. One should also enable GET methods for the root and a health check endpoint to allow Render to scan the service regularly.

In step 2, one needs to fill in the service configuration and environment variables in different boxes. This includes settings such as port, build command, and health check endpoint. The issue I encountered was Render could not scan any port even if I have triple-checked that everything worked fine locally. In the end, I solved this issue by adding the Golang build tag -tags netgo in the build command. Actually this flag was configured by default, but I initially replaced it with a simpler build command.

In step 3, one needs to configure the webhook with the Bot API and to enable the POST method for the webhook at the HTTP server (this can also be handled by the Bot API). The webhook can be https: //<your project>.onrender.com/<your-token> (or another unique URL). This URL informs Telegram where to send and receive all messages for the bot.

5. How to connect to the Firebase

The Firebase Realtime database stores key-value pairs in the JSON format. Connecting the bot with the database requires the following steps:

  1. Create the app and the database on Firebase’s dashboard. Specifically, one needs to store the following 3 values for interaction:
    • The database URL, which looks like https: //<your-app>-default-rtdb.*.firebasedatabase.app.
    • The credential file, which can be downloaded at Project settings->Service accounts->Firebase Admin SDK (and should also be added to Render).
  2. Import the language-specific Firebase API to configure the database in the bot. For example, I use firebase(Golang).
  3. Update the database rules in Firebase dashboard to only allow authorized writes for specific tags, e.g., one name/path to refer to those key-value pairs.

It’s worth noting that connecting to the database on Render may take some time after a fresh start. During this initialization period, the log may display a 502 Bad Gateway error to the database.

6. How to configure UptimeRobot

Before configuring UptimeRobot, an attempt was made to ping the bot from within itself, but this approach did not function for Render.

Using UptimeRobot to maintain the bot’s active status involves two primary steps:

  1. Enable the HEAD method (the sole method available for a free account) for any endpoint on the HTTP server.
  2. Configure an HTTP(S) monitor for that endpoint, which appears as <you-project>.onrender.com/<endpoint>, and establish the monitoring interval to less than 15 minutes.

7. Conclusion

This post isn’t meant to be a step-by-step guide for building a Telegram bot. It skips some steps and doesn’t include screenshots. But don’t worry, most of the missing bits can be figured out using AI language models these days. The rest really depends on each specific situation. The main point here is to show how to set up a free small web service, even when there’s no single platform that does it all.

When I first wrote this, my bot had been up and running for 10 days. It only had 30 minutes of downtime, which I think happened because UptimeRobot couldn’t reach Render’s IP address during that time.

Right now, the repository is private since I plan to add a second functionality to the bot soon.

Tags: tool telegram
Other posts