Re: deviations tagging

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Thu, Jun 16, 2022 at 02:05:24PM +0200, Roberto Bagnara wrote:
> On 6/15/22 09:19, Roger Pau Monné wrote:
> > On Tue, Jun 14, 2022 at 03:47:12PM -0700, Stefano Stabellini wrote:
> > > Hi all,
> > > 
> > > Roberto was suggesting to use the following different categories for
> > > tagging deviations. We could pick any "TAG" we like for the in-code
> > > comments (or other tagging systems).
> > > 
> > > I am also CCing the MISRA C team to give them early visibility on this.
> > > Feel free to provide early feedback if you have any. The plan is to
> > > discuss it further during the next fusa-sig call and come up with a more
> > > detailed proposal (including the actual tags, how to use them and more)
> > > for xen-devel next.
> > > 
> > > Cheers,
> > > 
> > > Stefano
> > > 
> > > 
> > > 
> > > adopted
> > > 
> > >     The report should be considered originated by adopted code without any
> > >     contribution of native code to the report.
> > > 
> > > safe
> > > 
> > >     The report is correct but the specific behavior is safe under every
> > >     aspect assumed to be covered by the guideline.
> > > 
> > > relied
> > > 
> > >     The report is correct but the rule concerns exclusively "developer
> > >     confusion" or readability matters that are not relevant for adopted 
> > > code,
> > >     which is assumed to work as is and it is not meant to be read, 
> > > reviewed
> > >     or modified by human programmers.  To be used for adopted code only.
> > > 
> > > false-positive
> > > 
> > >     In the opinion of the developer the violation report is not correct
> > >     and the problem has been notified to the tool provider.
> > >     To be used only for violation reports.
> > 
> > Do we want to tag false positives?  There's no benefit at all from our
> > code base tagging false positives, I think those should get fixed in
> > the checker tool, or otherwise marked as false positives somewhere
> > else (ie: in the tool itself).
> 
> A false positive is a "definite violation" report issued by a tool which
> turns out not to be real.  (Note that a "possible violation" report,
> sometimes called a "caution" or an "orange", cannot be a false positive
> because it is not a positive: not all tools make this distinction.)
> 
> False positives will be reported, by all tools.  It is not a simple matter
> of the tool being sound or defective: the MISRA coding standards are human
> artifacts with ambiguities and defects.  Issues  of the form "is this a 
> violation?"
> can be in the agenda of the MISRA working groups for years before a final
> decision is made.  During this period, some tools will implement one
> interpretation and other tools will implement a different interpretation.
> (If this surprises you, there are known defects of the C standard that,
> after 20+ years, still nobody is sure how to solve.)
> 
> So, a false positive might sit there for ages, and the benefit for your
> code base in tagging it is that you need not reanalyze it over and
> over again.

Right, we already carry bodges for compilers, so carrying those for
checkers seems inevitable if we plan to use them.

I guess it also depends on how obvious a false-positive is from a
human PoV, and how many of them do we need to tag.  I would really
like to avoid having files plagued with tags to placate
false-positives from the possibly different set of checker tools that
we support, specially if those false-positives are trivial to triage
for a human.

> > > compliant
> > > 
> > >     The developer can prove that the possible non-compliance shown by
> > >     caution report cannot happen in any situation and can motivate such
> > >     claim.  To be used only for caution reports.
> > > 
> > > false-negative
> > > 
> > >     The developer has found a non-compliance not shown by the tool and the
> > >     problem has been notified to the tool provider.
> > 
> > I'm also not sure tagging false-negatives is helpful either, specially
> > if we want to consider this tag system is not bound to any specific
> > tool.  What could be a false negative to a specific checker tool might
> > not be to another, and hence the tag would cause confusion.  False
> > negatives should be tagged like any other violation, ignoring the fact
> > a specific checker tool hasn't been able to spot it.
> 
> > Also I think the usage of 'report' in the descriptions is confusing.
> > AFAICT this is supposed to mean tags are added in reaction to reports
> > by checker tools, but what about deviations that are find by humans,
> > there's no 'report' in that case likely to refer to.  The language
> > seem to be focused against a tags being a reaction to a report from
> > checker tools.
> 
> I am not sure what you mean by "deviation that are found by humans".
> I guess you mean "violations that are found by humans."
> This is indeed rare, because humans are not very good at spotting
> MISRA violations, but once they do, tagging the occurrence is helpful
> during the period in which you and the tool vendor (and maybe the
> MISRA C working group if there is no consensus on whether there
> is a violation there or not) decide what to do about it.

My main concern is why non-compliances not found by a checker tool
needs to be tagged in a specific way (ie: false-negative), IMO they
should just be tagged using the same tags as if the non-compliance was
found by a checker tool.

Anyone interested in false-negatives from a specific checker tool
should be able to obtain them by cross-checking the tags in the code
with the report from the checker tool, and any tags in the source not
matching a checker report would be a false-negative.

Thanks, Roger.