[tutorial] Gather Bread Data

Bread is seen as an important part of German culture. There even is a bread institute, that gathers information about bakeries, does quality control and research and has a database about registered kinds of bread, too.

What better way is there to check how well German culture is doing, than check for this number of registered breads. I am very very shure, that that there is a direct relationship between numbers of different breads and the blossoming of German culture.

And also, a friend gave me an ESP32 and I needed to figure out a project to run on this thing.

Firstly, I checked how to extract the value for number of breads from the homepage of the Bread Institute and save it to a text file. This was done with a very short Python script:

import urllib.request
import re
from datetime import datetime
from anyio import fail_after

page = urllib.request.urlopen('https://www.brotinstitut.de/brotkultur')
xml = page.read()
page.close()
text = xml.decode("utf-8")

result = re.search('>\d+<', text).group()
number = result[1:5]

now = datetime.now()
current_time = now.strftime("%Y-%m-%d %H:%M:%S")
print("\n", current_time, " ", number)

s = '\n' + current_time + ', ' + number

f = open('XXX/brottimer.csv', 'a', encoding="utf-8")
f.write(s)

Next, I ported the code to the ESP32 in C++. Since I never worked in this language before, it took me quite a while and I needed to import a few libraries.

Programming was done in VS Code and the PlatformIO framework.

This code you will find for download at the end of this blog post, because we are not done yet!

The friend who gave me the ESP now told me to make this project into a Telegram Bot. So I tried this next. In Python, this was easily done, as I only needed to change a few lines of code and import a Telegram Bot Library.

First though, you need to create a new Chat Bot in Telegram. For this you have to have a little chat with the Telegram Botfather (@BotFather). You can follow the first few steps of this tutorial to get your API Key.

Then you can add it to the Python source code:

import os
import telebot
import urllib.request
import re
from datetime import datetime

BOT_TOKEN="XXX"

bot = telebot.TeleBot(BOT_TOKEN)

@bot.message_handler(commands=['info', 'help', 'hallo', 'hello'])
def send_welcome(message):
    bot.reply_to(message, "Howdy! I am BobBrotBot, the nice Brot Bot\n I am here to give you the current number of breads registered at the German Bread Institute (Brotinstitut.de)\n you can access this data with the command brot\n\n\n I am currently in version 0.0.1 and i am being developed by Moronaut (moronaut.de)\n ")

@bot.message_handler(commands=['brot'])
def brotinfo(message):
    page = urllib.request.urlopen('https://www.brotinstitut.de/brotkultur')
    xml = page.read()
    page.close()
    text = xml.decode("utf-8")
    result = re.search('>\d+<', text).group()
    number = result[1:5]
    now = datetime.now()
    current_time = now.strftime("%Y-%m-%d %H:%M:%S")
    s = 'at ' + current_time + ', there are ' + number + ' breads registered.'
    bot.reply_to(message, s)

bot.infinity_polling()

And there you have a Telgram Bot, that runs on a regular PC. The problem is though that it has to be constantly running. So that’s a good project to run on an ESP32, which will need much less energy and is powerful enough for such a little task.

For this we firstly need to import all the necessary libraries. I used the Universal Telegram Bot Library from, Brian Lough, which you can download here.

#include "WiFi.h"
#include "WiFiClientSecure.h"
#include "UniversalTelegramBot.h"
#include "time.h"
#include "HTTPClient.h"
#include "string.h"
#include "regex"

Then define some constants and variables, that we will need later:

// network settings
#define WIFI_SSID "XXX"
#define WIFI_PASSWORD "XXX"
#define BOT_TOKEN "XXX"
WiFiClientSecure secured_client;
UniversalTelegramBot bot(BOT_TOKEN, secured_client);
char* BrotURL = "https://www.brotinstitut.de/brotkultur";

//time settings
const unsigned long BOT_MTBS = 1000; 
unsigned long bot_lasttime;
const char* ntpServer = "pool.ntp.org";
const long  gmtOffset_sec = 3600;
const int   daylightOffset_sec = 3600;
int counts;
std::string now;

And then implement the functionality. The first function will gather the current time:

std::string getTime () {
  //get current time 
  configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);
  struct tm timeinfo;
  if(!getLocalTime(&timeinfo)){
    Serial.println("Failed to obtain time");
      std::string timePrint = "NA";
    return(timePrint);
  } else {
    char buffer[80];
    strftime(buffer,sizeof(buffer),"%F %H:%M:%S",&timeinfo);
    std::string timePrint(buffer);
    //timePrint = (&timeinfo, "%Y-%B-%d %H:%M:%S");
    return(timePrint;
  }
}

The next function does the actual regular expression search on the source code of the homepage:

std::string regexChar(char* bufferIn, int c) {
  std::string regIn;
  regIn.assign(bufferIn, c);
  std::regex aSearch("<div class=\"counter\"><p>([0-9]{1,4})<");
  std::smatch am;
  if (std::regex_search(regIn, am, aSearch)) {
    std::smatch bm;
    std::regex bSearch("([0-9]{1,4})");
    std::string search2 = am.str();
    if (std::regex_search(search2, bm, bSearch)) {
      std::string out = bm.str();
      return(out);
    }
    return("NA");
  }
  return("NA");
}

Since the memory of the ESP is too small to read the whole source code at once (there is a long table of bakeries at the end of the stream), we need a function that feeds the source code piece by piece to the regular expression function. This function gets cancelled, once the regular expression was found, so that we don’t have to read in the whole source code.

std::string grabHTML(char* URL) {
  bool regHit = false;
  HTTPClient http;
  http.begin(URL);
  int httpCode = http.GET();
  if(httpCode == HTTP_CODE_OK) {
    int len = http.getSize();
    char buff[2048] = { 0 };
    WiFiClient * stream = http.getStreamPtr();
    // read all data from server
    while(http.connected() && (len > 0 || len == -1)) {
      size_t size = stream->available();
      if(size) {
        int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));
        //do regex things
        std::string regExResult = regexChar(buff, c);
        if(regExResult != "NA"){
          regHit = true;
          http.end();
          return(regExResult);
        }
        if(len > 0) {
          len -= c;
        }
      }
      delay(1);
    }
  } 
  http.end();
  std::string noRegExResult = "NA";
  return(noRegExResult);
}

The last bit of functionality that we need is to handle the messages of the chatbot itself. This function checks for new messages and replies to them one by one. It outputs the number of breads, a little help text and some tiny statistical information.

void handleNewMessages(int numNewMessages) {
  Serial.println("handleNewMessages");
  Serial.println(String(numNewMessages));

  for (int i = 0; i < numNewMessages; i++) {
    String chat_id = bot.messages[i].chat_id;
    String text = bot.messages[i].text;

    String from_name = bot.messages[i].from_name;
    if (from_name == "")
      from_name = "Guest";

    if (text == "/start" || text == "/help") {
      String welcome = "Howdy, " + from_name + "!\n";
      welcome += "I am BobBrotBot, the friendly Brot Bot.\n";
      welcome += "I am here to give you the current number of breads registered at the German Bread Institute (Brotinstitut.de).\n";
      welcome += "You can access this data with the command /brot\n";
      welcome += "Access statistics with /info\n\n ";
      welcome += "I am currently in version 0.0.3 and I am being developed by Moronaut (moronaut.de)";
      bot.sendMessage(chat_id, welcome);
    } 

    if (text == "/brot") {
      std::string brotZeit= getTime();
      std::string brotCount  = grabHTML(BrotURL);
      std::string brotmsg = "Hey!\nAt the moment there are " + brotCount + " breads registered!\nTime checked was: " + brotZeit + ".";
      String outmsg = brotmsg.c_str();
      bot.sendMessage(chat_id, outmsg);
      counts += 1;
    }

    if (text == "/info") {
      std::string countStr = std::to_string(counts);
      std::string infoStr = "I was booted at " + now + ". And since then I was asked about bread for " + countStr + " times.";
      String outStr = infoStr.c_str();
      bot.sendMessage(chat_id, outStr);
    }
  }
}

Finally, we have to get internet connection and the current time in the setup:

void setup(){
  Serial.begin(115200);
  Serial.println();

  // attempt to connect to Wifi network:
  Serial.print("Connecting to Wifi SSID ");
  Serial.print(WIFI_SSID);
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  secured_client.setCACert(TELEGRAM_CERTIFICATE_ROOT); // Add root certificate for api.telegram.org
  while (WiFi.status() != WL_CONNECTED){
    Serial.print(".");
    delay(500);
  }
  Serial.print("\nWiFi connected. IP address: ");
  Serial.println(WiFi.localIP());

  now = getTime();
}

And in the loop, we ask the program to check for new messages each second and reply to them:

void loop(){
  if (millis() - bot_lasttime > BOT_MTBS){
    int numNewMessages = bot.getUpdates(bot.last_message_received + 1);

    while (numNewMessages){
      Serial.println("got response");
      handleNewMessages(numNewMessages);
      numNewMessages = bot.getUpdates(bot.last_message_received + 1);
    }

    bot_lasttime = millis();
  }
}

The code is done with this. Now we also need a nice profile picture. This one was AI generated, to fit the theme of the project. I needed to install stable diffusion on my computer for this and let my old 1060 ti work really hard.

The results were all semi-nice, so in the end I still had to do a bit of illustration work myself, to combine parts of the 4 generated images.

From now on you can talk to the Brotboter on Telegram via his username (@BobBrotBob).

All source files and the profile picture can be downloaded HERE.


Other Tutorials


Beitrag veröffentlicht

in

,

von