Postman self documenting tests and runner import

I’ve been working on the last days doing extensive API testing and needed to find an easy way to document my tests. Postman offers already something very useful that uploads your tests documentation to their website. But sometimes we need just a simple HTML that can be privately delivered to the client.

That’s where this project was born:

https://github.com/martinberlin/postman-reporter

The intention of this simple PHP script is to generate a Standalone HTML for your Postman tests that you can send to the client without the need to upload all the tests in the open internet.

It serves to achieve two things:

  1. Make a standalone HTML document from your Postman Collections
  2. Import the test run-results into a mysql database

With the second one only the importing is done. It’s then up to you how to present this data. It populates two tables, resume and detail, first one with the performance result and the detailed with a line per test. Much more information can be extracted from the runner json this is just a kickstart idea. Have fun with it!

If it calls your interest then please read more details in the github repository.

3D Printer upgrade

Since last year, when I started 3D-printing, I bought my first printer just to see if I find my way into it.

Anycubic Delta "Kossel Plus"
Anycubic Delta “Kossel Plus”

Since then I found out that I could remember my days in the university studing graphic design and that I can actually design pretty well. But from that to product design there is million-light years. Anyways I like a lot Blender as a 3D Modeling software and I’m starting to get quite advanced doing my own models.

That combined with my passion to soldier electronic stuff and to create new devices has found it’s way. So it’s time for a more professional update.

Last month I purchased a new Prusa MK3.

https://www.flickr.com/photos/mjtmail/
Photo by the real Tiggy Flickr: https://www.flickr.com/photos/mjtmail/

I would be really happy to posting my results with the new machine and sharing with all of you the experience of working and creating new stuff with it.

Reading an image bitmap file from the web using ESP8266 and C++

There are a couple of different ways to do it, but I wanted to do it after a simple image example, to understand a bit better how reading a stream from the web to get as far as the pixel information and send it to a display. As a reference then I started with /GxEPD Library :

There are a couple of basic things to understand when dealing with streams of information. One is that the information comes on chunks specially if there is a large file, then buffering whatever is coming is essential if you want to read from it. The first examples I did without buffering just filled the 7.5 E-ink display of separated lines, that only resembled part of the web screenshot I was going to send it.

So how comes an image you request from the web then ?

First of all like any other web content there is a request made to an endpoint to whatever script or API that delivers the image. This part of the code is well reflected here:

String request;
  request  = "GET " + image + " HTTP/1.1\r\n";
  request += "Accept: */*\r\n";
  request += "Host: " + host + "\r\n";
  request += "Connection: close\r\n";
  request += "\r\n";
  Serial.println(request);

  if (! client.connect(host, 80)) {
    Serial.println("connection failed");
    client.stop();
    return;
  }
  client.print(request); //send the http request to the server
  client.flush();

In this case is a get Request. Then in the case of reading a Windows BMP image that is one of the easiest formats to read, the first thing is to check for the starting bits, that for a .bmp image file are represented by 2 bytes represented by HEX 0x4D42

But before that, when you send a Request and the server replies with a Response, it comes with the headers. For example it looks something like this:

HTTP/1.1 200 OK
Host:display.local
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0)
Accept: text/html

(And some more that I will spare here ending at the end with an empty line only with “\r” known as Carriage return)

Then after this the image should start. So there are two choices:

1 To make something that loops reading the first lines discarding the headers and then attempts to read the 2 starting bytes of the image

2 To read from the start including the headers and scan this 2 bytes until we find 4D42 that represent the start of the image

Between the two I prefer the first since it looks cleaner. If we where to take the second one for this image it will look like this, note is a 4-bit bmp:

5448 5054 312F 312E 3220 3030 4F20 D4B 440A 7461 3A65 5720 6465 202C 3130 4120
6775 3220 3130 2038 3131 343A 3A38 3334 4720 544D A0D 6553 7672 7265 203A 7041
6361 6568 322F 342E 312E 2036 4128 616D 6F7A 296E 4F20 6570 536E 4C53 312F 302E
312E 2D65 6966 7370 5020 5048 372F 302E 332E D30 580A 502D 776F 7265 6465 422D
3A79 5020 5048 372F 302E 332E D30 430A 6E6F 656E 7463 6F69 3A6E 6320 6F6C 6573
A0D 7254 6E61 6673 7265 452D 636E 646F 6E69 3A67 6320 7568 6B6E 6465 A0D 6F43
746E 6E65 2D74 7954 6570 203A 6D69 6761 2F65 6D62 D70 D0A 310A 3065 3637 A0D
4D42 ->BMP starts here. File size: 122998
Image Offset: 118
Header size: 40
Width * Height: 640 x 384 / Bit Depth: 4
Planes: 1
Format: 0
Bytes read:122912

Then as we can see in this example they come as the starting bits the image headers itself that are readed with this part of code:

// BMP signature
if (bmp == 0x4D42)
{
    uint32_t fileSize = read32();
    uint32_t creatorBytes = read32();
    uint32_t imageOffset = read32(); // Start of image data
    uint32_t headerSize = read32();
    uint32_t width  = read32();
    uint32_t height = read32();
    uint16_t planes = read16();
    uint16_t depth = read16(); // bits per pixel
    uint32_t format = read32();
}
uint16_t read16()
{
  // Reads 2 bytes and returns then
  uint16_t result;
  ((uint8_t *)&result)[0] = client.read(); // LSB
  ((uint8_t *)&result)[1] = client.read(); // MSB
  return result;
}

uint32_t read32()there
{
  // Reads 4 fucking bytes
  uint32_t result;
  ((uint8_t *)&result)[0] = client.read(); // LSB
  ((uint8_t *)&result)[1] = client.read();
  ((uint8_t *)&result)[2] = client.read();
  ((uint8_t *)&result)[3] = client.read(); // MSB
  return result;
}

In there comes a very important 2 bytes of information and without it is impossible or I just couldn’t find out how to read the pixels, and that’s Image Offset: 118 which means at byte 118 the image information starts. Also Depth that represents how many bits represents one single pixel. So in 1 bit, we can store a black and white image, and if we want full RGB then we need 24 bits per pixel, also 1 byte for each color (Red, Green and Blue)
Our dear Wikipedia says about this:

For an uncompressed, packed within rows, bitmap, such as is stored in Microsoft BMP file format, a lower bound on storage size for a n-bit-per-pixel (2n colors) bitmap, in bytes, can be calculated as:

size = width • height • n/8, where height and width are given in pixels.

So there we have then the Image Offset: 118, but to get to read this headers, we already got from the client 32 bytes. Then we need to make the difference and start reading the image:

// Attempt to move pointer where image starts
client.readBytes(buffer, imageOffset-bytesRead);

That should be it, then we need to read every row up to the reported width in our example 640, inside of a height loop of 384 pixels. And then read each pixel taking in account the pixel depth. In the code example this looks a bit rough around the corners:

    if ((planes == 1) && (format == 0 || format == 3)) { // uncompressed is handled
      // Attempt to move pointer where image starts
      client.readBytes(buffer, imageOffset-bytesRead);
      size_t buffidx = sizeof(buffer); // force buffer load

      for (uint16_t row = 0; row < height; row++) // for each line
      {
        uint8_t bits;
        for (uint16_t col = 0; col = sizeof(buffer))
          {
            client.readBytes(buffer, sizeof(buffer));
            buffidx = 0; // Set index to beginning
          }
          switch (depth)
          {
            case 1: // one bit per pixel b/w format
              {
                if (0 == col % 8)
                {
                  bits = buffer[buffidx++];
                  bytesRead++;
                }
                uint16_t bw_color = bits & 0x80 ? GxEPD_BLACK : GxEPD_WHITE;
                display.drawPixel(col, displayHeight-row, bw_color);
                bits <<= 1;
              }
              break;

            case 4: // was a hard word to get here
              {
                if (0 == col % 2) {
                  bits = buffer[buffidx++];
                  bytesRead++;
                }
                bits <<= 1;
                bits < 0x80 ? GxEPD_WHITE : GxEPD_BLACK;
                display.drawPixel(col, displayHeight-row, bw_color);
                bits <<= 1;
                bits < 0xFF  / 2) ? GxEPD_WHITE : GxEPD_BLACK;
                display.drawPixel(col, displayHeight-row, bw_color);
                bytesRead = bytesRead +3;
              }
          }
        } // end pixel
      } // end line

And I still have an issue that still didn't found why it does not work. This code works good and I can see images in 4-bits and 24-bits but it hangs on 1-bit image.
It's something about the headers, using the point 1 described before, also discarding headers the 1-bit image works. But not the other depths (4/24)
It's maybe some basic thing about how the byte stream comes that I'm not getting or I'm simply missing something stupid enough not to get around it.
There are other better examples on ZinggJM Repositories that deal much better with the buffering and other aspects, where the BMP reading truly works. But sometimes I like to understand the stuff and fight with it, before implementing something, since it's only way to learn how stuff works.
Have you ever though how far we are that every OS and every Browser have the resources to read almost any existing Image or Video format ? How many Megabytes of software is that ? ;)
That's what I love about coding simple examples in C++ on the Espressif chips. That you need to go deep, there is no such a thing of ready made json_decode or do-whatever libraries as in PHP. You need to read it from the bits. But the cool thing is that if you get around it, then you have a grasp of what is need to be done to read in this case a very simple Bitmap Format image. I cannot imagine how to read a compressed JPG or a PNG, I think for that yes, I will put my head down and use some library.
UPDATE: I found out after about 4 hours fight why it is. And it's the fact that I'm reading the bytes in chunks of 2. Reading them one by one and adding lastByte in the comparison to check them then it works for both 1 and 4 bits images. I can post the solution here if someone is interested, but if not, I will keep it as is to avoid making it a boring long read.

E-Paper driver to make your electronics smaller

Got from Waveshare some of this E-Paper ESP8266 driver boards:
e-paper esp8266 driver-board

I’m trying to make the electronics of my displays as small as possible and to avoid the wire-chaos inside the case with the goal of having more space for the battery. This will basically spare the need of the driver HAT and additional cables, something that is still present in my first prototype:

A bit messy uh ?

The only issue I see and I’m still sorting out with Waveshare is that the cable is not coming with the RAW display. But I think it will be not a show-stopper.

E-Paper ESP8266 driver board example developed by Waveshare

  • Supports Floyd-Steinberg dithering algorithm, more color combinations, better shadow rendering for the original image
  • Supports popular image formats: BMP, JPEG, GIF, PNG, etc.

Specifications

  • WiFi protocol: 802.11b/g/n
  • Interface: 3-wire SPI, 4-wire SPI (default)
  • Operating voltage: 5V
  • Operating current: 50mA ~ 100mA
  • Outline dimension: 29.57mm x 48.26mm
  • Mounting holes size: 2.9mm

When I find the time will post about the first tests and specially about consumption. Currently I’m using a 7.5 B/w only display with a 1200mA battery for our office. My only wish is that Waveshare electronics makes bigger E-paper displays so we can make more visible charts for our clients (10.3″ or 13 inches)

UPDATE from Waveshare

The sales people replied me, the cable is included when you buy the driver, it’s on Package contents Tab on Waveshare website:

  1. e-Paper ESP8266 Driver Board x1
  2. e-Paper Adapter x1
  3. 24PIN FFC x1
    1
    2+3-> 24 pins cable is included.
    So that’s it, you just need this driver, and to buy one RAW display. So it will run using a core of only 3 elements: Driver, Cable and E-paper display

E-Paper driver board specifications

 

Wemos D1 consumption and VCC pins

So this 6 min boring video was just an experiment to make a small consumption table comparing how the Wemos D1 mini behaves when receiving power through the 5v and 3v3  pins.

5v:
75.5 avg mA only Wifi
126.5 mA with relay on
2.5 mA on deep sleep mode

3v3:
75.4 avg mA only Wifi
119.5 mA with relay on
2.4 mA on deep sleep mode

All code examples will be published in this repository:
https://github.com/martinberlin/esp

Playing with a Wemos D1 (ESP8266)

After learning some Arduino and building some demos I decided to give it a try to another Chips. And this one is the ESP8266, you can find it in eBay or alixpress searching for “wemos D1”

Basically this little guy has kind of the same capabilities of an Atmel chip with less Digital outputs(9) but comes with WiFi built it. So that means you can use it as a station (STA) or a soft access point. That enables to do a lot of stuff, for example deploying this with sensors and being able to read from another device connected with the same Wifi all of them. In turn, you could control the digital outputs, and turn on / off stuff that are connected to the D1.

This is just a small example built in 5 minutes. I just connected a led with a proper resistance to D4. I still do not understand well with pin 2 turns on D4 but that is important now since it’s just a small demo.

UPDATE: Now I understand. You need to use the constants to get the right pins. So using D4 will automatically point it to the right pin that is 2 :) But it’s just much simpler to use the constants unless you want to go all the time to the datasheet. After building this simple example I build a bigger one that can send a message from the web to a simple display and show some weather measurements. Will share it later on a new post if there is any interest about it. You can comment here if it would be interesting for you to read it.

NOTE: I made a mistake here and connected the 3.7 V. to the 3.3V pin. That is incorrect! It should be connected to the 5V pin that in turn goes to the power regulator
/*
* Based on ESP8266 LLMNR responder sample
* Copyright (C) 2017 Stephen Warren 
*
* Based on:
* ESP8266 Multicast DNS (port of CC3000 Multicast DNS library)
* Version 1.1
* Copyright (c) 2013 Tony DiCola (tony@tonydicola.com)
* ESP8266 port (c) 2015 Ivan Grokhotkov (ivan@esp8266.com)

* Instructions:
* - Update WiFi SSID and password as necessary.
* - Flash the sketch to the ESP8266 board.
* - Point your browser to http://ip-you-get-in-serial-monitor-arduino/
*
*/

#include 
#include 
#include 
#include 

const char* ssid = "yourRouterSSID";
const char* password = "yourPassword";

ESP8266WebServer web_server(80);

void handle_http_not_found() {
  web_server.send(404, "text/plain", "Not Found");
}

void handle_http_root() {
  // NOTE: Including bootstrap here just to make fancy buttons
  String headers = "	";
  String html = "Server options:
";
  html += "<a class='btn btn-primary' href='4/1' target='frame'>D4 on</a>
";
  html += "<a class='btn btn-primary' href='4/0' target='frame'>D4 off</a>
";
  html += "";
  web_server.send(200, "text/html", headers + html);
}

void handle4off() {
digitalWrite(2, 0);
web_server.send(200, "text/html");
}
void handle4on() {
digitalWrite(2, 1);
web_server.send(200, "text/html");
}

void setup(void) {
Serial.begin(115200);
pinMode(2, OUTPUT);
digitalWrite(2, 0);

// Connect to WiFi network
WiFi.mode(WIFI_STA);
WiFi.begin(ssid, password);
Serial.println("");

// Wait for connection
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.print("Connected to ");
Serial.println(ssid);
Serial.print("IP address: ");
Serial.println(WiFi.localIP());

// Start LLMNR responder
LLMNR.begin("esp8266");
Serial.println("LLMNR responder started");
// Start HTTP server
web_server.onNotFound(handle_http_not_found);
web_server.on("/", handle_http_root);
web_server.on("/4/1", handle4on);
web_server.on("/4/0", handle4off);
web_server.begin();

Serial.println("HTTP server started");
}

void loop(void) {
web_server.handleClient();
}

So after uploading this little script to your Wemos and of course updating your SSID and password to your router it should act like a server. Notice that I’m using a hidden Iframe to keep things simple, since I see other examples that submit a Form to itself and analyze the arguments. That is cool maybe for a bigger applicaion, but for simple apps, is enough to just point to an URL and reply with a 200 Http status.

Starting again with Arduino and 3D design

About a month ago I joined the amazing community of Thingiverse and started uploading my first designs made on Blender.

One of the first projects is to control automatically the curtain blinds to ensure we don’t have to get up to turn them manually. Actually it’s just a great excuse to learn more about arduino and stepper motor controlling.

This is the basic schema to control light detection:

Then I just connected some leds to the digital output:


int ledPinMin = 2;
void setup() {
Serial.begin(9600);
pinMode(ledPinMin, OUTPUT); // declare the ledPin as an OUTPUT
pinMode(ledPinMin+1, OUTPUT);
pinMode(ledPinMin+2, OUTPUT);
pinMode(ledPinMin+3, OUTPUT);
pinMode(ledPinMin+4, OUTPUT);
pinMode(ledPinMin+5, OUTPUT);
}
void loop() {
int sensorValue = analogRead(A5);
int ledPin = sensorValue/50 +2;
digitalWrite(ledPin, 1);
delay(sensorValue);
digitalWrite(ledPin, LOW);
delay(100);
}

I really like the projects like arduino and the power that brings 3D printing to the table. It’s a great combination. Anything can be done you just need to have an idea and be creative enough to make it happen.

How routing internally works in Symfony 4 framework

At the moment of writing this I’m testing the Symfony 4 Framework after reading the following article and also Fabian’s latest posts on Medium.  I’m testing weather the not yet officially released version, will be a future candidate for some new project or not. But the interesting part of this research is that I already realized what I won’t do in new projects. But let’s focus first on the mission of describing how routing works internally in the framework.
So we will go from the Frontend Controller up to the internal guts that say from the route /test let’s execute Controller:test()

http:// bestpractices.local/test

The front-end controller located in /web/index.php bootstraps the framework:

$kernel = new Kernel(getenv('APP_ENV'), getenv('APP_DEBUG'));
$request = Request::createFromGlobals();
$response = $kernel->handle($request);
$response->send();

As we already know all this framework is based on Request -> Response concept from the beginning to the latest controller action and the Kernel is no exception.  If we open this Kernel we can see that it’s mission is to setup Cache and Logs directories, register the bundles, configure the Container and Routing.

To go deeper in the chain we need to open HttpKernel with the mission described as:
HttpKernel notifies events to convert a Request object to a Response one.
There on the handleRaw method “Handles a request to convert it to a response” is where the whole framework workflow takes place:

namespace Symfony\Component\HttpKernel;

class HttpKernel implements HttpKernelInterface, TerminableInterface {
private function handleRaw(Request $request, $type = self::MASTER_REQUEST)
    {
        $this->requestStack->push($request);
        // request
        $event = new GetResponseEvent($this, $request, $type);
        $this->dispatcher->dispatch(KernelEvents::REQUEST, $event);

  // At this point the routing is already resolved
  // dump($request->attributes);exit();

        if ($event->hasResponse()) {
            return $this->filterResponse($event->getResponse(), $request, $type);
        }

        // load controller
        if (false === $controller = $this->resolver->getController($request)) {
            throw new NotFoundHttpException(sprintf('Unable to find the controller for path "%s". The route is wrongly configured.', $request->getPathInfo()));
        }

// ... Return a response

As we can see on the first lines, the Routing part, is resolved dispatching an Event called KernelEvents::REQUEST

namespace Symfony\Component\HttpKernel\EventListener;

class RouterListener implements EventSubscriberInterface {

public function onKernelRequest(GetResponseEvent $event)
    {
        $request = $event->getRequest();

        $this->setCurrentRequest($request);

        if ($request->attributes->has('_controller')) {
            // routing is already done
            return;
        }

        // add attributes based on the request (routing)
        try {
            // matching a request is more powerful than matching a URL path + context, so try that first
            if ($this->matcher instanceof RequestMatcherInterface) {
                $parameters = $this->matcher->matchRequest($request);
                // dump($parameters); // here to see it matched

            } else {
                $parameters = $this->matcher->match($request->getPathInfo());

            }

And in this part of the code we can see that dumping the $parameters the Request attributes are already assigned

$parameters = array(
  "_controller" => "App\Controller\DefaultController::testAction"
  "_route" => "test"
);

Note: Request->attributes is used to store information about the current Request such as the matched route, the controller, etc
But more interesting is to replace this dump for a : dump($this->matcher);
Then we can see that the real matcher of the routing is done by srcDevDebugProjectContainerUrlMatcher since we are currently using DEV as an environment.

This file, that we can find in this path /var/cache/dev/srcDevDebugProjectContainerUrlMatcher.php , is the responsible of returning the request parameters with the right controller to be executed for this route.
Since doing all the routing on the fly will be a real show stopper and we are concerned with performance: reading the configuration from the filesystem may slow down the application.

That’s why there’s a PhpMatcherDumper class which can generate an implementation of UrlMatcherInterface with all configuration in an optimized way.

Even in DEV, this UrlMatcher file, is only rendered again if we update any part of the routing. That’s one of the reasons clearing all cache will make a slow first request.

Lessons learnt from this research are:

  • Keep your routes clean
    Not used / deprecated routes that are still there will make this UrlMatcher unnecessary long and hence slower to resolve.
  • Avoid at all costs using Annotations for the routing
    It’s generally considered a bad practice in OOP and contains much more Cons than Pros
  • Avoid using Assetic
    This generates fully unnecessary routes for every assset in your HTML templates. New versions of Symfony will not come with Assetic so it’s better to get along without it.

I never care to check this routing to the core before and I highly recommend from time to time to check the internals of Symfony. You will learn a lot of the framework essentials and also will be more cautious of how to take the best of it. In my personal case, I learnt much more doing this small research taking your own conclusions, than reading some other developer top 5 list on framework optimization.

The code to reproduce this can be downloaded from public repository:
https://github.com/martinberlin/symfony-flex

Integrating SSO in a project using Shibboleth authentication

Shibboleth is an open-source identity federation solution that’s used in research and education communities worldwide.

Graphic by ncsu.edu

Basically is a federating agent that consists of 2 components:

1- Service Provider (SP)
 which request attributes, represented in this implementation als storage.luckycloud.de using Apache2-mod-shib 
2- Identity Provider(IDP)
 which broadcasts attributes, and will be implemented in the Login-logic of the site

Now the SP is actually out-of-the-box with apache2 module mod-shib. But my interest here is to do the Identity Provider part als Symfony module.

 

Is someone else out there interested on following the implementation ?

Friction powered Longboard

This is a redesign of the Rotor Flettner project. The effect works after testing it mounted on a windsurf board but it has a potential flaw: Magnus effect has a smaller force than the wind pulling you to the side. The result is that you go more in the wind direction than in the direction that you really want to go. The equivalent to having a very small sail.

But anyways it was a lot of fun to make it and having already the engine and the structure to mount it I thought, why not to try doing something with it :)

And it works, went about 2 kilometers back and forth after discovering I made some design errors and that friction is not my friend. Engineering lessons learnt:
1.- Control cables of the engine should not be touching the floor
2.- Engine should transmit power using some similar material as the wheel
3.- Friction generates heat and 50% of the battery or more is wasted (maybe good for the winter)

Skate and position detail

Wheel coupling with engine