PlatformIO: An alternative to Arduino IDE and a complete ecosystem for IoT

PlatformIO Logo

As an introduction I would like to make clear that I’m not a C++ advanced coder or IoT professional, I do this just because it’s a challenge, and because it’s a lot of fun compared to my 9 hrs/5 days a week web developer doing PHP and Admin panels for clients in Germany. Two months ago I started tinkering with SPI Cameras and the Espressif systems boards and that’s how this open source project was born. So far I was using Arduino since I do not need a super IDE, sometimes I also edit the code with vi or gedit instead of open a monster IDE that will eat your CPU alive. But gladly this is not the case. I was since long looking for a competitive alternative to Arduino when this github issue on the FS2 Camera Project called my interest:

tablatronix commented 8 days ago

I had to manually install this button library as it is not in platformio , is it in arduinos?

And that’s how after a few clicks I discovered Plaftormio.org that according to their home page it’s an open source IoT ecosystem. But what interested me more than this is the subtitle:  “Cross-platform IDE and unified debugger. Remote unit testing and firmware updates

So instead of answering tablatronix issue, I started installing this new IDE, that happened in a breeze.
1. First thing is to install the VS CODE Version. I choosed the Visual Studio code option since it has more features. Microsoft’s Visual Studio Code is the base and PlatformIO is built on the top of this.

2. Go to “Extensions” and install PlatformIO IDE “Development environment for IoT, Arduino, Espressif (ESP8266/ESP32)”

platformio IDE extensions is marked in RED just before PlatformIO logo

3. In my case since I used Arduino IDE before, go to PlatformIO alien face logo and select “Import project from Arduino”. Then select the folder where your Arduino project is and PlatformIO will create a Workspace for you with all the necessary structure.

4. Adjust serial monitor baud rate.  If you are using Espressif chips edit platformio.ini and add the following line at the end:

monitor_baud = 115200

This will make your serial work in the right baud rate for ESP8266 /ESP32 (Serial is the plug icon in the bottom just before the Terminal > icon)

5. FS Data. If you are using SPIFFS that in Arduino is the /data folder inside your sketch you will notice that in PlatformIO this folder needs to be at the root level. So you can simply move it one directory above. To upload SPIFFS data use the following command in the terminal:

pio run --target uploadfs
IMPORTANT: As difference with Arduino if you try to save a file in /1.JPG here you will get it saved on:
/spiffs/1.JPG so create this folder on /data or you will get; VFSFileImpl(): fopen(/spiffs/1.jpg) failed

6. Libraries in PlatformIO live inside a folder on your project called .piolibdeps Thanks god, they are not distributed on 3 different parts, like it happens with Arduino (Some in Arduino/libraries, some in Programs/Arduino/libraries and so on) but are there available for you in just one place :)
To install a new library is very easy, just go to PlatformIO.org/lib and type the name of the library to search. If found copy the resultant line in the Terminal just like point 5 and the IDE will look this up for you and download it using github to the  .piolibdeps folder.

pio lib install "WifiManager"

Sometimes you will need a library in a special branch and there is also an easy way to do this. Just use :

pio lib install <repository#tag> 

// Install development version of WifiManager library
pio lib install https://github.com/tzapu/WiFiManager.git#development

7. If you use Arduino before it’s possible that you used some of it’s constants. So if your sketch compiles, but it does not runs like expected try to add this include line at the beginning

#include <Arduino.h>
Including Arduino.h at the beginning of the sketch

As a footer note I must make clear that it takes some time to get accustomed to the new IDE. But it comes with a great benefit as it shows you much better C++ warnings, it’s prettier to use, and has the advantage that you can Ctrl+Click and navigate between different classes. Something that it was not possible to do with my old IDE. And the most beautiful update is that all libraries are in a single place per Project, like it should be, leaving you a clear idea of how much code and dependencies you are adding into your sketch.

In FS2 project porting it to the ESP8266 Wemos board this happened like a breeze. And it was just the fact of upgrading my code to use OneButton library. adding the include Arduino.h and that is. After uploading SPIFFS data and the new code, the camera was ready to be used. Now it’s not always that easy. trying to compile the same code to an Heltec ESP32 board, I got SPI communication issues with the camera and I’m still fighting with it.

UPDATE: I found what is going on here, and it’s that there are some problems with my sketch that uses I2C and wire library to comunicate with the Camera on the Heltec ESP32 environment. Error message is:
i2cCheckLineState(): Bus Invalid State, TwoWire() Can’t init. I solved it updating the platformio.ini configuration file to use:
board = lolin_d32
So using this board it compiles and works correctly. No idea why it does not using heltec_wifi_lora_32 I will have to research more.

Additional links of interest:

Source filter: src_filter
This option allows to specify which source files should be included/excluded from build process. It could be useful to keep the same core for ESP8266 and ESP32 but including different files to support both. So far I’ve been using 2 different branches something that is hard to maintain for obvious reasons.

Library dependencies: lib_deps declaration in platformio.ini
Like composer for PHP, this IDE offers a way to declarate your libs in the ini file. To check how a professional software for 3D-printing does take a look in Marlin configuration file. This is a very important point to keep in mind since after the lib_deps list is correctly added and tested, people testing your code just need to hit “build” to download them, saving a huge amount of time. This are FS2 library dependencies.

FS2 Camera project

If you are interested to build your own WiFi Camera for around 40€ of components and maybe 5€ of 3D-printed case, then check this project on github:

https://github.com/martinberlin/FS2

I’m giving out in december ten free 3D-printed cases, if you are interested on getting one and putting together a WIFi Camera please choose between Black PLA, black PETG or RED transparent and comment in this post. You can send me the address later per private message or in twitter. Thanks for your interest!

Chunk uploading files via WiFi with Espressif boards

Derived from this FS2 Camera project I would like to make an sequential uploader that matches this use-case.
Any file major than 50 Kilobytes will be uploaded in chunks of 50 Kb, using the following workflow:

Picture.jpg (150 Kb) -> Read fifo from Arducam
                                              -> Send first 50 Kb to upload API along with an md5 hash of this first 50 Kb (Called API from now on)

API uploads this first 50 Kb chunk and returns a true if md5(File)==hash received if not returns false

-> ESP board receives this signal and repeats the upload if it’s false up to N times (3 or 4 times at least)

When sending the last part to API, we will add some parameter like EOF = 1 or something in this direction so the API understand this will be the last part to receive. And in successful upload of this last part the server side API will put this chunks together and return the full URL to the image.

This can be the basis to upload big images with consistency.

As you can see here uploading any File that is larger than 40 / 50 Kb is already a big challenge.  And I still didn’t saw this implemented anywhere so it will be a great challenge to build something in this direction


ESP-32 a new standard of speed and connectivity

Developed by Espressif the ESP32 brings a new standard in low-cost WiFi boards. After porting one project to this new board I must say that it’s very similar to the ESP8266 and most code except some updated libraries, should work with minimal effort. 

ESP32 is a series of low-cost, low-power system on a chip microcontrollers with integrated Wi-Fi and dual-mode Bluetooth. The ESP32 series employs a TensilicaXtensa LX6 microprocessor in both dual-core and single-core variations and includes in-built antenna switches, RF balun, power amplifier, low-noise receive amplifier, filters, and power-management modules.” – Wikipedia description

What is still pending on my side is to compare power consumption and other details as WiFI range (With and without antenna) between ESP8266 and this new ESP32. But overall I’m really excited to build things on the top of this.

Combined with @tablatronix amazing WiFi Manager library it gives creators the possibility to have an independant IoT device with custom configuration, autoconnect and full info of the device.

It’s still pretty much on development so they are many things that are normally worst at the beginning (SPI Flash File System for example, a hell lot slower and at the moment only usable for small configuration files for me) But that’s normal, things are going to be fixed over the time, and hell it’s open source so let’s give these guys a hand and report new findings so they can fix it as soon as possible.

Yesterday I found the time to refactor my digital Camera project to use ESP32 (Makerfocus Heltec board). If you are interested in seeing the code updates please my commits in the board/esp32-oled branch.


So far still on my tests I won’t expand myself too long on this one. Giving some pointers and links for further research below.

Arduino Core
https://github.com/espressif/arduino-esp32

SPIFFS (ESP8266 code should be refactored to reflect library update)
https://github.com/espressif/arduino-esp32/blob/master/libraries/SPIFFS/examples/SPIFFS_Test/SPIFFS_Test.ino

PINS
https://github.com/makeitlabs/ratt/wiki/ESP32-Pin-Mapping

Using the OLED built in the Makerfocus ESP32

The ESP32 directory when installed using Arduino in Ubuntu is:
/home/martin/.arduino15/packages/esp32 -> I find weird that Arduino  keeps libraries in 3 different Folders making it very confusing to see library source code.


Uploading your pictures directly into the cloud

Yesterday I found some time to make an example of image push to the Cloud using Seafile storage. The idea is that instead of making a regular file upload that needs some kind of Backend gallery to preview the pictures, we can have a different take, like pushing the picture directly to your Cloud-storage.

WiFi Camera (C++) > API Endpoint (PHP) > Seafile cloud (Python/C)

So the first thing I tried is to browse the Seafile Web API manual and repeat in command line the curl commands. And then I found in packagist this great library that is a wrapper for all commands to implement them easily on your script.

For sure this would be also possible to do it without the middleware PHP, directly in C++ on the Espressif  SoC,  but I would like to have proper error handling and also to save a copy of the image in the case that the Cloud->push fails for any reason. It can be also a very complicated task considering they are at least 3 API calls using the bearer Auth token.

SeaUP

Short for Seafile upload this a code-example to make a PHP File upload using Seafile API and a simple JSON configuration

Repository lives here: https://github.com/martinberlin/seaup

Configuration is easy:

1– You need to obtain a Token

curl -d “username=username@example.com&password=123456” https://yourseafileserver.com/api2/auth-token

{“token”: “14fd3c026886e3121b2ca630805ed425c272Xxxx”}

2- Edit the sea-config.json and add it along with your seafile server and repository ID settings

{
  "seafile_host": "https://storage.luckycloud.de",
  "token": "14fd3c026886e3121b2ca630805ed425c272Xxxx",
  "repository_id": "Is on the last part of Url when opening a library: #my-libs/lib/REPOSITORY_ID"
}

3- That’s it ! You are ready to open the test provided in the github repository and see if the image appears on the cloud

FS2 camera from Blender to production

In this entry, I wanted to document what is the process of making one of this cameras, starting from the 3D – model to the end product where you turn it on and connects to WiFi ready to take pictures.

After removing the support and sanding the round columns the first thing is to connect the front and back case together and see that they fit correctly. Usually, they do but PET is a tricky plastic to print and the end termination is a bit rougher than with PLA so it requires some post-production work. As an advantage, this plastic is stronger than PLA, and will stand a crash much better since it’s more elastic and resistant. I would say the best termination and strength balance would be to print this in ABS but I dislike the smokes and the fact that is also very difficult to print at home.

When this step is ready then it’s the time to heat up the soldier and prepare the ON/off switch and the shutter button. Then there are 8 cables more that go from the Arducam (2 or 5 mega version) to the Wemos D1, that is the responsible of uploading the picture to the cloud. This is a prototype for myself so it looks a bit messy but shows how it is at this stage:

3 pair of cables from the left to the right: Battery, ON/off and shutter button

Then comes the reality shock moment that is to connect the Wemos ESP8266 through USB to the computer and upload the program that will do the magic of receiving the JPEG image from Arducam and upload it to a php API endpoint. Usually, at this point there is something that needs to be corrected, but either nothing works or all is fine and dandy. I open the mobile hotspot and turn on the camera. See if connects, try to take a picture, preview it on the PHP-gallery.  Try to test timelapse mode, see it works, and that’s pretty much it. A new camera is ready to be delivered.

FS2 WiFi camera

Connecting to SSL from Espressif 8266 SoC

There are 2 ways that I know about validating an SSL certificate:

1 –  Generate a root certificate in DER format (somehow tricky)

2 –  Copying the SHA1 hash from the browser certificate details (easier)

… and as we are mostly lazy developers we will go for the number 2

DER format into hexa

In this example, we will generate the code to validate https:://api.github.com

In the address bar, to the left of the URL, click on the circled ‘i’ icon for more information.

Click on the ‘>’ icon.

Click on “More Information” button at the bottom of the window.

In the new window titled Page Info – https://api.github.com, click on the “View Certificate” button. In the “Certificate Viewer” window, click on the “Details” tab.

In the “Certificate Hierarchy” top window pane click on “DigiCert High Assurance EV Root CA” so it is highlighted.

Click on the “Export…” button at the bottom of the page.

At the bottom of the page select “X.509 Certificate (DER)” format then click on Save.

Use your favorite program to convert the binary DER format to ASCII. Here is what I do in Ubuntu:

$ xxd -i DigiCertHighAssuranceEVRootCA.crt.der >cacert.h

Edit cacert.h to add PROGMEM and const keywords like in this ESP8266 example

Using a SHA1 hash

Well this one is the most easier and the one I use. Repeating the first steps of the first option, in the first window that you see the certificate information there is the SHA1 hash at the bottom.

So the trick is to simply copy it and for api.github.com is:
5F:F1:60:31:09:04:3E:F2:90:D2:B0:8A:50:38:04:E8:37:9F:BC:76

Open it in any text editor and replace the “:” for ” ” spaces to paste it in the firgerprint variable declaration:

const char* fingerprint = “5F F1 60 31 09 04 3E F2 90 D2 B0 8A 50 38 04 E8 37 9F BC 76”;

Here you can see an example that is ready to compile and test

And since I’m really lazy to delete this Gutenberg demo of WordPress new editor I will leave the rest here. In resume, in this new editor pressing (+) you can add anything you want in the world. And it would be even cooler if they add a magical (-) button so you can remove the content with the same ease as adding it.

The rest comes from the new editor bells & whistles and it’s not of my writing

Imagine everything that WordPress can do is available to you quickly and in the same place on the interface. No need to figure out HTML tags, classes, or remember complicated shortcode syntax. That’s the spirit behind the inserter—the (+) button you’ll see around the editor—which allows you to browse all available content blocks and add them into your post. Plugins and themes are able to register their own, opening up all sort of possibilities for rich editing and publishing.

Go give it a try, you may discover things WordPress can already add into your posts that you didn’t know about. Here’s a short list of what you can currently find there:

  • Text & Headings
  • Images & Videos
  • Galleries
  • Embeds, like YouTube, Tweets, or other WordPress posts.
  • Layout blocks, like Buttons, Hero Images, Separators, etc.
  • And Lists like this one of course :)

Visual Editing

A huge benefit of blocks is that you can edit them in place and manipulate your content directly. Instead of having fields for editing things like the source of a quote, or the text of a button, you can directly change the content. Try editing the following quote:

The editor will endeavor to create a new page and post building experience that makes writing rich posts effortless, and has “blocks” to make it easy what today might take shortcodes, custom HTML, or “mystery meat” embed discovery.

Matt Mullenweg, 2017

The information corresponding to the source of the quote is a separate text field, similar to captions under images, so the structure of the quote is protected even if you select, modify, or remove the source. It’s always easy to add it back.

Blocks can be anything you need. For instance, you may want to add a subdued quote as part of the composition of your text, or you may prefer to display a giant stylized one. All of these options are available in the inserter.

You can change the amount of columns in your galleries by dragging a slider in the block inspector in the sidebar.

Media Rich

If you combine the new wide and full-wide alignments with galleries, you can create a very media rich layout, very quickly:

Accessibility is important — don’t forget image alt attribute

Sure, the full-wide image can be pretty big. But sometimes the image is worth it.

The above is a gallery with just two images. It’s an easier way to create visually appealing layouts, without having to deal with floats. You can also easily convert the gallery back to individual images again, by using the block switcher.

Any block can opt into these alignments. The embed block has them also, and is responsive out of the box:

You can build any block you like, static or dynamic, decorative or plain. Here’s a pullquote block:

Code is Poetry

The WordPress community

If you want to learn more about how to build additional blocks, or if you are interested in helping with the project, head over to the GitHub repository.


Thanks for testing Gutenberg!

👋

Digital Camera FS2

Days after publishing this post about ArduCam and ESP8266 I got some good feedback and 2 friends asked me to get one Camera. At the same time my new 3D Printer “Prusa MK3” has arrived so I decided to make a case remake and release a new small Low-Resolution, instant WiFi upload Camera. What I’m trying to achieve here is a digital Polaroid. Press the shooter buttton and the JPEG will be uploaded to a digital gallery in the next 4 seconds. So it’s a pure WiFi camera, without memory card, and you need to be online to use it.
And that’s nowadays very easy right ? You just need to make a mobile hotspot in the phone if you are outside home. And if the camera does not detect a WiFi then creates an Access point called:
CAM-autoconnect

Then you have to connect to it through the phone and browse 162.168.4.1 there will greet you a “WiFi manager” so you can select a WiFi and write the credentials to make a connection. After that you are all set, you just need to enable the hotspot and the camera will reset and connect to it automatically.

It can take a picture both with the shutter button or have a Video stream or Photo shoot via the Web UI. ( cam.local )

 

3D Renderings made with Blender

I decided to make a small release of 5 FS2 digital WiFi instant upload cameras at the price of 70 € each.

Materials and costs if anyone is interested to make one are the following:

  • 1S 3.7V Li-Polymer  (Got one in eBay)
    Battery size: 6 x 41 x 68mm  2000mAh /hr
    8.50 €
  • Charger: Adafruit Micro Lipo w/MicroUSB Jack  (eBay Not in the picture since this is just my personal prototype)
    8 €
  • Arducam Multi Camera Adapter Board 2MP (eBay)
    25 €
  • Wemos D1 mini
    6 € but can be purchased for less
  • Various connectors and cables (8 Pin SPI white)
    2.5 €

That gives a total of 50 € as total hardware costs, printing the case and testing that all works together I’m summing up and additional 20. And you can get 1 year of API use for free. Then you can move it to your own server or you will get part of my Amazon AWS invoice ;)

With the battery full loaded should be online for about 18 hours. It has an On/Off switch that is still not in the picture.

The photos now when it has all cables soldered are better than before although I liked some of the strange effects when the cables where loose. The idea is to make a camera that you shoot blindly without looking at the frame. That gives for me interesting results and I had a lot of fun with it. That combined that in about 4 seconds the picture is already online is really cool. If you are interested in getting one just contact me through this website. Shipping costs are not included.

Picture previews:

1280x960

Photo shooter has 3 options:

  1. ONE CLICK Shuts only one picture
  2. LONG CLICK enters time-lapse mode (a picture every 5 min. but can be configured to your request, should be minimun time 5 seconds though)
  3. DOUBLE CLICK disables time-lapse mode

The picture is uploaded in my API but the code is called PHP-gallery and it’s in a public repository on github so it can be hosted in any location that supports PHP 5+ and image-magick (Thumbs generator)

ArduCAM plus Espressif equals WiFiCAM

I’ve been having some fun taking pictures last weekend with a self built camera. It costs about 15 € in Ebay and summing up the 6 of the ESP8266 Wemos D1 it makes a total hardware cost of 21 euros. Let’s say 35 if you add a Lion battery and an USB charger to it.

The most basic layout looks like this.

Using here a 2 megapixels ArduCam. There is also a 5 megapixels version. The Pin definition of the camera is like any other serial parallel interface device:

Wiring with Wemos D1
D0 CS
D7 Mosi
D6 Miso
D5 Clk
D2 Sda
D1 Scl

Arducam put together some Instructions Kit where you can find the previous table plus a demo example here.

I went a bit further and made a simple PHP Gallery plus upload-receiver hook and packed it together in a git repository here:

Php bootstrap4 image gallery

I got some critics saying that this is of course invented and I’m not creating anything new here. But of course the idea is not to make a professional WiFi camera. If so I will simply get one ! The idea was to built one from the scratch.

And the beauty of it is that it does not take perfect pictures. Just because now it’s not soldered sometimes it makes an interesting noise in the pictures and imperfections. The idea is to take this and make it whatever you want. Like for example a security cam that takes a serie of pictures or even records a stream when it detects movement. Or get a 5MB pixels more advanced camera and pack it together in a 3D printed case putting a shutter button and some eink display to preview the picture and change settings.

This is an example of a picture with noise:

And this another one when the right amount of light is there and the cables are good connected

For more pictures and to see how the Php gallery script works check this link.

Postman self documenting tests and runner import

I’ve been working on the last days doing extensive API testing and needed to find an easy way to document my tests. Postman offers already something very useful that uploads your tests documentation to their website. But sometimes we need just a simple HTML that can be privately delivered to the client.

That’s where this project was born:

https://github.com/martinberlin/postman-reporter

The intention of this simple PHP script is to generate a Standalone HTML for your Postman tests that you can send to the client without the need to upload all the tests in the open internet.

It serves to achieve two things:

  1. Make a standalone HTML document from your Postman Collections
  2. Import the test run-results into a mysql database

With the second one only the importing is done. It’s then up to you how to present this data. It populates two tables, resume and detail, first one with the performance result and the detailed with a line per test. Much more information can be extracted from the runner json this is just a kickstart idea. Have fun with it!

If it calls your interest then please read more details in the github repository.