Re: [Xen-devel] [HACKERS] [OSSTEST PATCH 0/1] PostgreSQL db: Retry on constraint violation [and 2 more messages]

[Ian Jackson <ian.jackson@xxxxxxxxxxxxx>]

Kevin Grittner writes ("Re: [HACKERS] [OSSTEST PATCH 0/1] PostgreSQL db: Retry 
on constraint violation [and 2 more messages]"):
> [various things]

I get the feeling from your message that I have irritated you.  I'm
sorry if I have.  In particular, I wanted to say that I certainly
don't expect bugs to be fixed immediately.  I appreciate your
attention to detail.  I also appreciate you taking the time to deal on
your -hackers list with someone who really isn't a database hacker!

I still hope to be able to convince you that the definition of
SERIALIZABLE (in the pgsql docs) ought to be a stronger version, which
covers even non-committed transactions.


> [Ian Jackson:]
> > But I have just demonstrated a completely general technique which can
> > be used to convert any "spurious" constraint failure into a nonsense
> > transaction actually committed to the database.
> 
> Really?  I see a lot of daylight between "you must discard any
> results from a transaction which later throws a serialization
> failure" and "if you catch the exception from a constraint
> violation within a subtransaction it is possible to create a
> serialization anomaly".  I don't actually see how one relates to
> the other.  What am I missing?

I don't think it can be useful to state (in the pgsql docs) the caveat
you propose, to wit:
  "you must discard any results from a transaction which later throws
   a serialization failure"

I think any scenario where this caveat declares a behaviour to be "not
a bug", can be converted into a scenario where there is undoubtedly a
bug.  So this caveat does not actually reduce the difficulty of
implementing the promises made by the documentation.

The conversion is as follows: if a scenario is affected by the caveat,
in there must be at least one transaction T which firstly produces
"impossible" results I, and in which some later statement S produces a
serialisation failure.

To exhibit the corresponding unavoidable bug: Execute an identical
scenario, with exactly the same sequence of steps in the same order,
up to S.  However, instead of S, execute ROLLBACK.

Now this anomalous transaction T' is not "a transaction which later
throws a serialization failure".  There is no requirement to discard
the "impossible" results I.  The results I are supposed to be correct,
but they are not.

(I am making the hopefully-warranted assumption that ROLLBACK cannot
itself cause a serialisation failure.)

In theory this argument does not prove that every bug which the caveat
is trying to address, is as important as the same bug without the
caveat.  This is because perhaps T' is an unrealistic transaction
somehow.

But I don't really think that the primary documentation should be
complicated, and should give weaker statements about the system
behaviour, simply as an aid to bug prioritisation.


> > 1. Declare that all spurious failures, in SERIALIZABLE transactions,
> > are bugs.
> 
> It's not clear to me what you mean here.

I mean that the pgsql documentation should simply say that for
SERIALIZABLE transactions, there is always a coherent serialisation.
Even if the transaction failed due to a constraint violation, or
serialisation failure, or database disconnection, or whatever.

I think there is no "useful" weaker guarantee.

By which I mean that for any reasonable weaker guarantee: any scenario
in which pgsql violates the stronger guarantee, can be converted into
a scenario where pgsql violates the weaker guarantee.

Conversely, any correct implementation of the weaker guarantee
necessarily implements the stronger guarantee too.  So the weaker
guarantee does not make implementation easier.  All it does is
complicate application code.


> > 2. State that the SERIALIZABLE guarantee in Postgresql only applies to
> > transactions which (a) complete successsfully and (b) contain only
> > very simple pgsql constructs.
> 
> About 24 hours ago you reported a bug which was hard enough to hit
> that it took 5 years for anyone to find it.

Well, of course I have a slightly different point of view.

The fact that pgsql might generate "spurious" constraint violations,
even in SERIALIZABLE transactions, has been known for many years.  See
the URLs etc. in my patch, which include a reference from 2009.  Many
people have always regarded this as a bug.  The pgsql authors have not
agreed.

From my point of view, I have developed a trick which allows me to
convince you that it has been a bug all along :-).


> The behavior we have been providing, modulus the bug you just found
> and any related bugs of the same ilk, is "we don't allow
> serialization anomalies into the database or in results from
> serializable transactions which commit."  That is enough to be very
> useful to many.

As I explain, I think that if you make that promise, you will have to
do all the work necessary to make the stronger promise anyway.

So you may as well make the stronger (and more comprehensible!)
promise.


> > Of course I understand that these problems may be hard to solve.  But,
> > I think it needs to be recognised that these problems are all bugs.
> 
> Yes, when they are shown to allow serialization anomalies, they
> certainly are bugs.  That has now happened.

I think all scenarios of this kind can be shown to allow serialisation
anomalies.


> > "If you do not use features X or Y, these bugs will not affect you" is
> > a useful thing to perhaps say, even if X or Y is as general as "INSERT
> > ... ON CONFLICT" or "... EXCEPTION WHEN ...".
> 
> Well, if they are not fixed soon, it might be worth adding
> something to the docs; but as far as I can see, the specifics would
> be more like "If you catch errors generated by declarative
> constraints or unique index insertion within a subtransaction, and
> then discard the work of the subtransaction without throwing any
> error, you might create a serialization anomaly, depending on what
> else you do within the transaction."  Have you seen anything that
> suggests that this is not a complete description of the scope of
> the issue?  I don't see any point in a statement that there might
> be undiscovered bugs in the software, either in general or relative
> to any particular feature.

I'm afraid I have no idea.  But if that is a complete description of
the bug (if it doesn't affect "INSERT ... ON CONFLICT" for example)
then that is good.

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel