Continuous Integration with @travis-ci + @PlatformIO

[ivankravets]

More details http://docs.platformio.org/en/latest/ci/travis.html

@PaulStoffregen Please enable https://travis-ci.org for this project. See step 1, 2, and step 4 here: http://docs.travis-ci.com/user/getting-started/

It should look like https://travis-ci.org/ivankravets/Time

Comments are welcome!

[jamesmyatt]

This would be great. But there's a conflict.

[PaulStoffregen]

Looks, we tried this on the audio library. It did not work well. Almost every week it generated false alarms. The server side stopped working several times.

I'm not interested to use tools that are so unreliable that they cost more time & effort than they save.

[ivankravets]

@PaulStoffregen did you have a problem with PlatformIO or Travis?

[PaulStoffregen]

I don't know what was the cause of so many problems. All I know is the system as a whole provided little value but cost a lot of time & attention.

[ivankravets]

It was 2015. Would you like to enable it again? Would be glad to understand if the problem still persists.

[PaulStoffregen]

To be perhaps a bit blunt, I need to be convinced a tool is going to provide real value AND that it's not going to cost more time & trouble than it saves.

For a very mature code base that is getting very few changes, I have a hard time imagining CI will being any value at all, even if it requires zero effort to set up, even it if has absolutely zero maintenance cost.

[ivankravets]

I got you. If you will need CI later, please try PIO and ping me for any problems.

[PaulStoffregen]

For this Time repo, probably never.

Perhaps OneWire would make sense. But I'm not interested in just checking if the code compiles. That's such a low bar, just not worthwhile. The test would need to upload code to a real board, perhaps compare the reading from a DS18B20 against temperature read from some known-good commercial temperature sensing product.

But above all else, the system must be highly resilient against false alarms. That is the huge problem with CI setups, in my opinion. They tend to be far too noisy. I only want a message asking for my attention when there is a very high degree of certainty that the code has broken. If the code is ok but the test system breaks (especially intermittent or transient failures), the test system needs to NOT bother me.

[ivankravets]

That is the huge problem with CI setups, in my opinion

This feature is built-in PlatformIO, a lot of companies build own Board Farms and run tests on physical devices via PIO Unit Testing engine. It can work in remote mode too.

You can even have a board connected to your local machine and ask Travis to deploy tests on remote board via PIO Remote.

Here is a simple example: https://github.com/platformio/platformio-remote-unit-testing-example

[PaulStoffregen]

You can even have a board connected to your local machine and ask Travis to deploy tests on remote board via PIO Remote.

Does this system have safegaurds against false alarms?

[ivankravets]

This is not a problem of PlatformIO. Sometimes these free services, such as Travis, etc, may cancel/stop builds... We don't depend on these services, you can use PIO with other CI services or even install own( free, Jenkins, GitLab). See a full list with a known CI where we provided examples:

• http://docs.platformio.org/en/latest/ci/index.html

[PaulStoffregen]

I understand you're working hard to make a great tool. That's why I'm taking a little time to reply...

You can make CI more valuable, more useful, more attractive for people to use PlatformIO, if you can build in safegaurds against false alarms. You need to accept the brittle nature of these systems and design things to fail gracefully.

The huge problem is these systems are designed from a technical perspective, rather than a human value perspective. They give errors, they report problems, and generally work in the ordinary way you'd expect from (not so reliable) software. Anything that's not a pass is treated as a fail. That's bad. It's distracting. It wastes human time & attention. Plenty of "not pass" cases are not necessarily the code under test failing.

If you want to make a great CI tool, one that stands out and actually provides more value than all the others, it needs to be designed from a philosophy that every second of the user's time is precious. If you manage to make a tool that does, even to only some degree better than others, I'm sure it'll become extremely popular.