Monitoring Open Source Success in Arduino | Silvano Cerza (Arduino)
We have found it vitally important to be aware of the quality and performance of our software. In this talk ill be telling you a story on how we, the tooling team, started using datalog in an unusual way to understand how successful are some of our operand source projects, including the arduino cli. The arduino cli is one of the most important projects of the company, as the name implies its a command line tool that can be used to compile code upload it to our boards and lots of other things, its also the backbone of the next version of our New ide, the ide 2.0 anyway, i wont be talking about software, neither collecting metrics from users or servers and the like ill be talking about something easier to grasp that everybody can easily understand information that we can share with developers and the community alike. That is the total number of downloads of our open source projects, a simple but meaningful metric, many failures too more interesting to developers, but also to members of the community that are using nightly wheels and want to be on the bleeding edge. Our stories start without usual questions that every developer ask themselves at least once how do i know people are using my software. The first thing that comes to mind is adding metrics to it. Thats useful, since you can also track other meaningful informations, but that puts a burden on the users they might not like being tracked and we fully respect their privacy.
It can also be disabled, easily client side, so we might lose some precious information. So you keep thinking what else can i track? What information is valuable and at least you get it? The download counts, and this is exactly the problem we had with the arduino cli. We didnt know if people were actually using it or not, and neither we could add invasive monitoring systems to it. So we decided to go with the simplest solution, monitor downloads, but we had another problem. We have multiple distribution channels, the official arduino website and the github releases page for each of our different projects. We couldnt simply read the number and call it a day. Neither we wanted to what we really wanted was collect. This information and show other number of downloads to the success of the project grew over time. At the time of the themes were already using data to monitor our backend infrastructure. So this was settled with nowhere to gather the information were still missing, an important piece, though: how do we get the data from our distribution channels? And this is when the data of github action commits comes into play? It has been created by massimiano, pp, plus his art, and thanks so much since we were using github workflows for running tests, building new releases and whatnot. This is the best solution for us. As a small team. We could leverage knowledge we already had. We didnt need to learn new tools, so we come back, but we could come up to speed and see ourselves right away less than a day.
Let me show you how we did it. I avoid showing the entire data workflows and focus on the important parts. If you want to look at the workflow flow, its fine ill leave some links in the slide, but workflows that gather downloads from github and arduino are extremely similar. Apart from the logic that gets the numbers, we wont delve into how we requested information since thats, not the scope of this talk. In both cases, we have three steps: one to request the number of knowledge one to send the metrics to that run and one to send an, even together in case of failures, notice how we must set a secret with our api key to make the action work Correctly, we store that keying the repository secrets as usual. The matrix data instead is returned by the fetch step of the workflows. As i said, i wont go in details of those steps. Just know that the output of the step is a string containing the json array, with the download counts for each released asset and thats. It youre done you youre getting download. Now, if you want to know how we get, the downloads count feel free to take a look at the workflows that are linked in this slide, its all open source after all. But what if there have been issues, fetching the information or the workflow? Failing someone for silly way, you must handle that so im using the same action.
We send an event to the dot to that all in case of failures, as you can see, this step is identical. Apart from the different tags, again arduino city and uh gita, we could avoid sending failures. You really and use a custom step, some other action to notify the developers on slack email or whatever you want. I think its much better leveraging datadog in this case to collect failures, since we can create a historical view, but also because we can easily use multiple channels to notify developers and it wont require adding the editing the workflows to add new ways of communication. Now that we push all the information we want it to data, we can start building our dashboards to show some graphs. Recently, we started gathering this information also for the new arduino ide 2.0, even if still in beta, we already had a good amount of downloads. As you can see from the screenshots of our dashboards, we can already get some interesting insights. Most of the users of the arduino cli are on wheels while most of the users of the arduino ide 2.0 around windows. Monitoring that nightly build failures is also quite easy. The only different thing are the tags pushed and here are both monitors for nightly and stuff collection failures right now, theyre, all green so were more than happy, and this is uh the end of our story. Our now, a small team like us solved our need for information or history.
In reality, i think this is a nervous story, were still evolving. This system and theres lots of space for improvement, more information together and understanding. I nightly build failures for the ide 2.0 cli download counts from hombrew or some other place. Understanding, after all, is an endless journey. In any case, im glad you listened to this story and nobody gave you some ideas on how to better understand the urban suits projects. You can find all my contacts on this last slide feel free to ask me anything.