Category: Dev

Returning documents as one file back from Azure CosmosDB – Part II

In my previous post, I showed, how to get all documents of one collection into one single JSON-file. The best reason to do it this way, is you can use a browser, being independent from the Hardware. But that has a drawback in some situation. If your documents content is to big/large, than the Data Explorer cannot concatenate all together, wherefore it results in paging. ..Not what we want, bad!

There is another option on getting all documents from CosmosDB into one JSON file. The “Data migration tool” for Azure CosmosDB. It’s pretty easy to install and use. Just hold your connection string ready. I will not duplicate content, that is covers more details on, how to use and what to fill in, to get the right output. So, take this post a hint to this tool. I used it very often for various tasks, why I like to recommend, to have a look at this. Visit here

If you have experiences with Data migration tool, let me know and leave a comment.

Accessing VSTS REST API – Get Release definition

As of the need to automate some tasks in VSTS, I was looking for a solution to access VSTS without any UI. To get this done,
I first used the VSTS CLI [
CLI Link] to do things right. But unfortunately CLI is not “complete” (missing e.g. Release mgt) and also I cannot install
VSTS CLI everywhere. With that in mind, I decided to use the full REST api [
Documentation], to get things done. Here are the steps to take, to get at least one release definition by use of REST
calls to VSTS:

Create a PAT (Personal Access Token) in VSTS, to gain Access…

  1. …by clicking on your profile Icon and choosing “Security”
  2. Next, click “Add” to generate a token
  3. Fill in the Name of your PAT, expiration days and so on. Under “Authorized Scopes” you should select “Selected Scopes”
    and check all relevant scopes, that you like to authorize.

With that, open Postman (or another tool, that you like or use)

what we will do here is first getting a list of all projects along with their ID

  1. Add the calling REST URI of following Format:
  2. select Authorization tab and type “Basic Auth”
  3. leave the “User” field empty
  4. add the new created PAT into “Password” field
  5. Make sure you have a GET Operation and click “Send” -> below in Postman-Window the resulting response of the REST
    call will be displayed.

the release definitions of a project, which we can use for further documentational Tasks

      1. Now, for getting the list of Release definitions, you have to read carefully the documentation on
        Visual Studio Doc . There is a documentation lag! There is the Definition of REST Api for retrieving
        the definitions as follows:
        GET https://{instance}/{project}/_apis/release/definitions?api-version={version}
        But it should be as in the examples (like so):
        GEThttps://fabfiber-inc.vsrm.visualstudio.com/MyFirstProject/_apis/Release/definitions
        notice the red vsrm
    2. with that, create a new REST call as a GET Operation by typeing URL in that Format:

(you can also append the api version)

  1. With Authorization set, as before, you can send the call and get back a list of Release defionition in JSON Format
  2. For retrieving a single one with details, you have to pass a specific ID, from the list before, that you can pass as a parameter to the REST Api

    ! as a  hint, don’t use any GUIDs. 🙂

That’s all! Now, you can analyse that resulting JSON or use it for further documentation. I will soon post an article, where I reuse this JSON to generate a readme.md for documentational reasons …

Getting from CSV to JSON is pretty easy

This is a short reminder for myself, but could be helpfull for someone else out there. Nothing new!

I often look for easy solution, that resolve “easy” problems. This is one: “Convert something to something”.

After looking arround in the outer spheres of Internet, I realize, that OoB tools can do the job better or at least without any dependencies or such things.

So, how to convert a regular CSV to JSON? Take PowerShell!

It’s easy, isnt’it 🙂

Getting started with C# on Intel Edison (Yocto Linux)

Intro

Hi there,

after Intel rejected Edison and Co. I was not sure, what to do with the neat piece of hardware, I am owning. Despite of not getting any Support and updates and so forth it is still Hardware, that has WiFi, Bluetooth, SD Card,… onboard. So it has enough capabilities to build cool things. So I decided, not to throw away 😉 and see, what I can do with it from another perspective.

With that in mind, I asked myself, if it isn’t possible, to get my favorite dev setup running on Intel Edison: .Net Framework with C#

For half a year, I gave it already a try, but it was a little bit tricky, by downloading the whole package and compiling it on the Edison (Install Mono by hand [“David’s Random Projects and Documents Web Page”]). In my case, there were problems with storage and compile errors. But then, there was a package update for use with opkg, so that I was able, to get Mono Environment installed and usable. Read here, how things work…

Install Mono

Please make sure, that you added these unofficial packages to your
/etc/opkg/base-feeds.conf

src/gz all http://repo.opkg.net/edison/repo/all
src/gz edison http://repo.opkg.net/edison/repo/edison
src/gz core2-32 http://repo.opkg.net/edison/repo/core2-32

After that, type

(it upgrades all packages, it can eat up your free space!)
in to your bash.

Having that done, only

is needed and everything is ready to use (takes a moment, to download and configure).

 

Testing Mono installation

You should have a look to your mono version, that is now installed.

The output should display something similar to this, where Mono compiler version is 4.2.2. that maps to .NetFramework 4.6 (I thought, please correct me, if I am wrong).

Writing code

After that, you could write your first test app in C#

Create a test folder (like in screenshot above)

 (typing ‚I’ for Insert | for Save: ESC ‚:’ after that ‚x’)


For compiling your first App, type:

This builds the code to tests.exe

shows up tests.exe in your folder

running your app is easy; just type:

and this is the result:

Adding Hardware access (GPIO,…)

But writing C# Apps only on Intel Edison is not that, what the device is made for, so I needed some access to the underlying hardware. At first I tried to access the mraa libs from Intel by P/Invoke, but thankfully that guy here Mayuki Sawatari wrote an assembly, that has everything in it (ありがとうございます, すごいです。).

I tried to get everything compiled at my Edison under Yocto Linux, but that was not possible, therefore I cloned it to my Windows machine and opened the solution file with Visual Studio 2017. The build was successful and I could copy over the resulting Dll to my home directory on Edison, where I now have this available for further development.

Again, create a file (e.g. pinTest.cs) and copy this code here (It’s slightly modified version from what you can find inside that Git Repository – I adapted it to my Intel Edison Arduino Breakout Board):

This Code is a working Blink example, which blinks onboard LED every second.

Finally

Although Intel Edison is discontinued, it is working and with fresh development tools, it can be sweet to hack some usefull things with it.

The upside with this is, you Can also Code on your Windows machine and test your Code by mocking mraa before releasing it to the device. Welcome to easy DevOps 😉

So, if you own one and like to give this here a try, leave a comment or share your Projects.

 

cheers

How to bootstrap an ESP8266 with Azure Services

One of the things I played around with ESP8266 and Azure IoTHub was, how I can get the whole infrastructure deployable and also to get the code working for other devs, without sharing my Azure environment and credentials.

The main problem was, to keep all modules decoupled from each other, so that the IoT device (here my ESP8266) can reach my Azure Endpoints all the time, either endpoints have changed by redeployments or new devices are added.

So I started developing the following architecture:

Bootstrap architecture
Bootstrap architecture

As you can see, the device first tries by connecting over WiFi to reach the Azure backend, that is a function. That functions responsibility is to create a device identity. If it does not exists, it will be created and then the function sends back the device’s identity together with a new endpoint. That endpoint directs to a storage account containing the up to date firmware as a blob.

So, on receiving the identity and the storage endpoint, the device can now connect to the storage, downloads the firmware and starts flashing. After the flash process is done, the device tries to connect to Azure IoT Hub. If connection has been successful established, it starts sending telemetry data (here it is temperature and fake battery level) to IoT Hub.

When now a new firmware is ready for flash on productive devices, an administrator or so is able, to send an update command, with what all connected devices can set their self to firmware update mode and start downloading/ flashing process. That’s all!

With this approach my devices are decoupled from the backend. The only one thing I need is a little piece of code, that enables my device to find the first endpoint. But with that, I can start deleting my Azure resource group and redeploy it, as long I have fun doing it. And fortunately, I can use this, to also share my code and deployment scripts, without sharing any secrets 🙂

This is, what DevOps is for. Making life easier and safer. If you like to, take part on my project and contribute. This version of code and deployment is a draft. There is a lot of things to do, to get this smooth and fluent. So, everyone is welcome to adjust and optimize the code and get things right. https://github.com/totosan/DevOpsIoT

 

DevOps in der IoT angewendet – Ein Bericht von der buildingIoT

Ein Bericht über meinen Aufenthalt auf der Konferenz buildingIoT.

Vom 3.05.17 bis 5.05.2017 fand die buildingIoT Konferenz in Heidelberg statt. Es war ein Format, bei dem “Entwickler” und Gleichgesinnte sich zum Austausch über alle erdenklichen Themen im Scope des IoT treffen. Es wurde über Technologien, Prozesse, Erfahrungen und die Themen der Digitalisierung gesprochen. Dabei entstanden auch die einen oder anderen guten Kontakte bei den Get-Together Events.

Für mich war der Besuch hier ein besonderes Erlebnis, da ich das erste Mal vor einem Publikum mit so einer hohen Bandbreite an Erfahrungen und Kenntnissen ein Referat abhielt. Mein Thema war “Wie DevOps in der IoT-Entwicklung aussehen kann”. Meine Nachricht war aber eher “Wieso DevOps die einzige Wahl bei IoT ist…”.

Der erste Tag der Konferenz begann entspannt mit einer ganzen Anzahl von Workshops rund um das Thema IoT. Vom Beginners-Guide bis hin zum MQTT-Deepdive gab es eine interessante Auswahl über den ganzen Tag.
Am Abend wurden dann alle Referenten, Sponsoren und Organisatoren zum Essen geladen. Mir kam dies nur gelegen, da ich so auch andere Referenten kennen lernen konnte. – An dieser Stelle möchte ich gern noch einen Gruß an Steffen und Niko entrichten 😉  war toll euch zu treffen; unsere Gespräche haben mich bereichert. (Das Restaurant Tati kann ich übrigens sehr empfehlen – dort fand unser Treffen statt).

Am Donnerstag war es dann auch für mich soweit. Nach der Keynote begann meine Session mit 70min DevOps in der IoT.

Es war super… der Raum war voll, niemand hat den Talk verlassen und am Ende belagerten mich noch soviel Interessenten und Fragende, dass ich die Mittagspause auch glatt nicht mehr geschafft hatte (Das ehrt mich).
Was mich aber am Meisten gefreut hatte, war, dass selbst meine Demo, die aus Coding, Builds, Deployments, Backend in Azure und selbst meine IoT-Hardware mit WLAN-Anbindung einfach funktionierte. Ich kann mich also wohl zufrieden schätzen. 🙂

Im weiteren Tagesverlauf habe ich dann noch die unterschiedlichsten Themen verfolgt. Zum Beispiel, wie Spracherkennung in Geräten umgesetzt werden kann; oder Digitalisierungs-Stories mit Opitz Consulting.
Zum Abend hin gab es dann noch reichlich Austausch über allerhand Themen und Erfahrungen der Teilnehmer untereinander beim üblichen Networking im “Get-Together” mit kleinen Häppchen und nem Bier 🙂

Leider musste ich am Freitag auf eine firmeninterne Schulung, weshalb ich dann die Konferenz leider frühzeitig verlassen musste. Dennoch würde ich gern nächstes Jahr wieder vorbei kommen.

An dieser Stelle auch noch einmal von mir einen herzlichsten Dank an das Orga-Team und die Sponsoren. Das habt ihr klasse gemacht.

Beste Grüße
Thomas

“Run Login-AzureRmAccount to login.” in AzureRM when already logged in with PowerShell

I worked on a Release pipeline in VSTS for some month ago. Because I experimented with AzureCLI2.0 in my Release-Template, I switch from Hosted-BuildAgent to onPremise-BuildAgent. With setting things up and working out the details on how my release can run on my local BuildAgent, I had a successfull Release-Pipline.

Today I decided to switch the BuildAgent from my local one to a remote server, that suited my needs. I installed and configured my BuildAgent like I did on the other server and checked my release pipe with a deployment of already  working Bits.
But what I didn’t expect, was the following error message:

It’s curious, because from the log you can see, that there is already a login existing. So what could that error be.

After some time consuming investigation, I found, that my server installation, regarding PowerShell and the desired modules like AzureRM and so, are installed all to different modules folder, wherefor the agent and the release tasks are – let’s call it – irritated.

You can see the list of installed modules:

The Fix: I uninstalled all Azure PowerShell modules and reinstalled them with Web Platform Installer.

An alternative maybe, is also something, that I found here (it didn’t worked for me): Blog post from Darren Robinson
Here is a solution about updating all modules, but read yourself 🙂

 

 

My experiences with ESP8266 development

Reasons for ESP8266

For some weeks ago, I bought some ESP8266-12E modules with developer boards and started developing IoT-test solutions. I also have some other dev-boards like Intel Edison or Arduino, so why to change to another?
The reason is, the Edison is much far away from being “atomic” (I mean, it has a complete yocto linux installed and therefor more features running, than I need). My Arduino Uno comes without WiFi and buying a Shield is expensive compared to ESP. For simple prototyping or home automation tasks for my own, it is sufficient. So, a neat little device, that can “concentrate” on its tasks is, what I want.
Often I am loosing time on fighting around with Edisons “complexity” and OS. With ESP I can go forward by only coding and HW development. There are no service, that can interrupt work.

Starting development

At first, I tried to find a way to start development. So, I looked around to find some useful sites, that can bring me up. I realized, that there are thousands of How-Tos and “Getting started” Guides. My starting point was a development platform called PlatformIO (http://platformio.org/). It is really easy, to write code and “deploy” it to the borad. Because I love developing software with C#, I am a little bit lazy, to start coding with C/C++. After installing PlatformIO and getting started, I found some simple example codes written in LUA. And because I was not experienced in writing LUA-Code, I decided to start with it. LUA-Script is pretty simple and easy to learn, but after a while, I found, that it consumes much resources of the ESP and I was not right convinced about the language features, therefor I changed to Javascript.

Esprino is a firmware, that enables Javascript development for the ESP. It brings a Web interface, where you can write your code and start it immediately. Also you have the possibility, to Upload the whole prepared firmware. For my purposes, it was not what I want and like to deal with. So, I switched to the next development experiences. Arduino IDE!

I am familiar with Arduino IDE and developing with C/C++. Also the reasons I wrote before, lead me to the conclusion, that this way of writing code for ESP is an adequate way, although it is not that comfortable to write code with Arduino IDE. The IDE has no OOB support for writing for ESP8266, but setup of that IDE is pretty simple. You have to add the ESP-Package-URL to the preferences for the board manager (for example: https://adafruit.github.io/arduino-board-index/package_adafruit_index.json,http://arduino.esp8266.com/stable/package_esp8266com_index.json). After setting up, I could start coding as I would normally write code for my Arduino. But after some hours of fiddling and try-error procedures, I switched to VSCode.

Working with VSCode and Arduino

For the first few days I could go with that solution – Coding in VSCode (with additional extensions like clang, C/Cpp,…) and building/uploading than with Arduino IDE. But as one can imagin, this smells and therefor cries for a better solution.
I found a nice solution for setting VSCode up to support Arduino; you can find this great article here. But I also created a PS script, that automates the setup process a little bit.


It checks, if a local folder “VisualStudioCodeArduino” exists. In case it doesn’t its clones the files from Fabien’s Git repo and copies all necessary files to their destinations. After running that script I can start calling code . in PS. With that I can start coding in VSCode and also run a build and deploy from the tasks. It really runs fine and it feels like a charm. So with that script and base code, that could be used as a starting point I put all in my GitHub repo here. With that Setup, I am able to get quickly ready for dev.


Please feel free, to modify the script or give comments of youre experiences.

Accessing sqlserver instance with CommandLine

Working with local SQLServer can sometimes be challenging, if you don’t have any tools, to access a database. For administrational reasons it could be helpful, to gain access to SQL Server you can simply use the commandline cmd.exe or powershell tools. This is nothing new, but I think, it is not so common.

So, to start open up cmd.exe and type for example

This command opens (-S) a trusted (-E) connection to you local instance of a SQLLocalDB 2012. Note: this command ist case sensitive.

Than cmd prompt for further commands “1>”. Here you can type T-SQL statements like

This prompts for a Terminator for example (GO + <Enter-Key>).

After this your SQL Server instance runs this command and results with a number of databases, that are attached.

If you like to know more about read here: https://msdn.microsoft.com/en-us/library/ms162773.aspx

Calling linux commandline functions from NODE.JS

Currently I’am working on a project, where device management (dm) is needed. So a part of dm is to control devices remotely. My setup is a linux based device, that is connected to Azure Clouds IoT Hub with the new DeviceManagement-Feature. That device is controled by a NodeJs-App. One task of that is, to handle a reboot signal from the cloud. The problem I had, was to break through the node.js environment and to convince the device, to reboot. After some try-outs, I came up with following simple solution – maybe someone has the same problem, than this is a good start:

1) npm install sys child_process

2) …and use it as follows:

cheers!