Testing

Very interesting ideas.

• With command query separation and OO, there are some tests that you shouldn't write. https://medium.com/@smeriwether/the-magic-tricks-of-unit-testing-28ce0b300cee
• Don't do Faux-O. Either do simple or once the project gets large, then do real OO.
  • “I can't tell where the implementation actually is, as it's spread across files and isn't on one screen! That's the point!”

Really helpful. I had notes somewhere, but basically it's the difference between 3D printing a bicycle once and saying “let's call this a wheel, this a fork, this a frame, this a handlebar, etc”, and test them all separately. Then integrate them together. Swap parts out easily, etc. All that standard stuff.

Makes it a lot easier to use the code when it's not a 500-line long monstrosity

Interesting…

Doing TDD feels slower because you're constantly context switching between testing and dev, but you end up getting done faster because you spend less time debugging because you're always green.

Don't want fragile tests.

https://wiki.ith.intel.com/display/swpc/SWPC-X%3A+Introduction+to+Test+Driven+Development%3A+A+discussion+and+practical+walk+through

Also TDD doesn't miraculously make better code. Somehow mocks in ideal unit tests aren't necessary because coupling is super separate? Interesting. https://news.ycombinator.com/item?id=14661285

AFL with cmake: https://foxglovesecurity.com/2016/03/15/fuzzing-workflows-a-fuzz-job-from-start-to-finish/

https://nullprogram.com/blog/2019/01/25/

Linux talk recommends http://lcamtuf.coredump.cx/afl/ (google fuzz toolkit) as well as some classes: http://events.linuxfoundation.org/sites/events/files/slides/ELCdeck-final.pdf#page=78

Another google toolkit http://honggfuzz.com/

Choose item by sku 12345
Item price should be $7.00
Set quantity to 6
Shopping cart total should be $42.00 –TestObsessed

Should be easy to read for a non-programmer and written early on. Not sure if the code needs to read the text directly or not, but it'd be nice. Cucumber, Fitnesse.

In order to test at this level of abstraction for say, ipmctl, you have to think about it.

• Should let the entire (SW) part run, all the way sideways and down. There's dependencies all over the place that while you'll test them individually, it sucks to “expect” them every single time you want to run a different high level test!
• Will need to stub out the hardware calls, but hopefully at the right level of abstraction for the test you want to run.
• CMake should probably then compile all the C files that the main ipmctl build does, but then remove from that the files/functions you want to stub.

Unit tests are

100% is a bad metric. Good tests that eventually reach 100% coverage is much better.

Apparently they can instrument the code similar to bullseye to measure coverage.

Competitor is Simics, which will do coverage measurement, requires tracker for kernel.

• Not really hardware, but as long as it boots, that's good enough / better for me.

Does it handle kernels? (virtual address space)

The way our VectorCast is set up, we can only write tests for a specific file. No interaction with any other files, so all of our integration tests are worthless.

Currently getting VectorCast working, I am saying “the expected output is the length of the message type”. However, if the length of the message changes according to the spec, then things down the line will break!

• Testing intermediate variables that aren't passed externally. Need to stub an internal function that uses them and assert a value that's from it.

• Doesn't support TEST.EXPECTED:foo:SOME_VARIABLE+1 ←- only supports one variable at the end
• So, you put it in expected_user_code block. However, that only says T/F, it doesn't tell you the actual value of foo, which C asserts can do
• No code highlighting template, so gets really taxing to read.
• Requires repeating a lot of the same thing
• Doesn't look like C
• Can't validate that individual lines are covered as a result of a test.
• Checking return codes. Don't hard-code them
• We currently need to manually duplicate all the cmake file changes into VectorCast (includes, libraries?!? not sure and whatnot). However, because people don't update VectorCast right away, it's sometimes hard to know what is needed to fix the build in VectorCast. This would be fixed by using a graph correlated with commits
• The Oregon VectorCast license server is down…probably fixed with a call to VectorCast and another system setup.

As transition, OR the lines covered from Vc and bullseye Still need a testing framework. Validation black box / Glenn's tool is one good option for quick iteration + custom diag cmds. Copy what kernel does for coverage output from Bullseye? Currently duplicate cmake includes in VC so that VC can build correctly

You can check code coverage from tests on hardware?? But most programmers try and run on emulators…Best practice for unit testing (stackexchange)

• Excellent gcov structure overview
• IBM guy
• Intel person made it work with Bullseye (apparently Intel has site license), code at \fw\tools\CodeCov, CSE Code Coverage.docx inside of doc folder
• Less impressive gcov on an embedded system paper