[Xen-devel] Re: [PATCH] libxl: reimplement buffer for bootloading and drop data if buffer is full

[Roger Pau Monnà <roger.pau@xxxxxxxxxxxxx>]

2011/10/12 Ian Campbell <Ian.Campbell@xxxxxxxxxx>:
> On Tue, 2011-10-11 at 13:26 +0100, Roger Pau Monne wrote:
>> # HG changeset patch
>> # User Roger Pau Monne <roger.pau@xxxxxxxxxxxxx>
>> # Date 1318335991 -7200
>> # Node ID 2fb4bf8c16cd35ddc0bf7ddc7ff8fda4b9678211
>> # Parent Â64f17c7e6c33e5f1c22711ae9cbdcbe191c20062
>> libxl: reimplement buffer for bootloading and drop data if buffer is full.
>>
>> Implement a buffer for the bootloading process that appends data to the end 
>> until it's full. Drop output from bootloader if a timeout has occurred and 
>> the buffer is full. Prevents the bootloader from getting stuck when using 
>> ptys with small buffers.
>>
>> Signed-off-by: Roger Pau Monne <roger.pau@xxxxxxxxxxxxx>
>>
>> diff -r 64f17c7e6c33 -r 2fb4bf8c16cd tools/libxl/libxl_bootloader.c
>> --- a/tools/libxl/libxl_bootloader.c ÂTue Oct 11 10:26:32 2011 +0200
>> +++ b/tools/libxl/libxl_bootloader.c ÂTue Oct 11 14:26:31 2011 +0200
>> @@ -21,6 +21,7 @@
>>
>> Â#include <sys/stat.h>
>> Â#include <sys/types.h>
>> +#include <sys/ioctl.h>
>>
>> Â#include "libxl.h"
>> Â#include "libxl_internal.h"
>> @@ -28,7 +29,8 @@
>> Â#include "flexarray.h"
>>
>> Â#define XENCONSOLED_BUF_SIZE 16
>> -#define BOOTLOADER_BUF_SIZE 1024
>> +#define BOOTLOADER_BUF_SIZE 4096
>> +#define BOOTLOADER_TIMEOUT 1
>>
>> Âstatic char **make_bootloader_args(libxl__gc *gc,
>> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â libxl_domain_build_info *info,
>> @@ -165,10 +167,11 @@ static pid_t fork_exec_bootloader(int *m
>> Â */
>> Âstatic char * bootloader_interact(libxl__gc *gc, int xenconsoled_fd, int 
>> bootloader_fd, int fifo_fd)
>> Â{
>> - Â Âint ret;
>> + Â Âint ret, read_ahead, timeout = 0;
>>
>> Â Â Âsize_t nr_out = 0, size_out = 0;
>> Â Â Âchar *output = NULL;
>> + Â Âstruct timeval wait;
>>
>> Â Â Â/* input from xenconsole. read on xenconsoled_fd write to bootloader_fd 
>> */
>> Â Â Âint xenconsoled_prod = 0, xenconsoled_cons = 0;
>> @@ -177,6 +180,10 @@ static char * bootloader_interact(libxl_
>> Â Â Âint bootloader_prod = 0, bootloader_cons = 0;
>> Â Â Âchar bootloader_buf[BOOTLOADER_BUF_SIZE];
>>
>> + Â Â/* Set timeout to 1s before starting to discard data */
>> + Â Âwait.tv_sec = BOOTLOADER_TIMEOUT;
>> + Â Âwait.tv_usec = 0;
>
> There are some portability snaggles with the timeout argument to a
> select (IIRC Linux can modify it). I think it would be wise to
> reinitialise this right before each select call.

Yes, I didn't remember Linux can modify the timeout struct when
returning from select, fixed it.

>> +
>> Â Â Âwhile(1) {
>> Â Â Â Â Âfd_set wsel, rsel;
>> Â Â Â Â Âint nfds;
>> @@ -185,15 +192,26 @@ static char * bootloader_interact(libxl_
>> Â Â Â Â Â Â Âxenconsoled_prod = xenconsoled_cons = 0;
>> Â Â Â Â Âif (bootloader_prod == bootloader_cons)
>> Â Â Â Â Â Â Âbootloader_prod = bootloader_cons = 0;
>> + Â Â Â Â/* Move buffers around to drop already consumed data */
>> + Â Â Â Âif (xenconsoled_prod > 0) {
>> + Â Â Â Â Â Âxenconsoled_prod -= xenconsoled_cons;
>> + Â Â Â Â Â Âmemmove(xenconsoled_buf, &xenconsoled_buf[xenconsoled_cons], 
>> xenconsoled_prod);
>> + Â Â Â Â Â Âxenconsoled_cons = 0;
>> + Â Â Â Â}
>> + Â Â Â Âif (bootloader_prod > 0) {
>> + Â Â Â Â Â Âbootloader_prod -= bootloader_cons;
>> + Â Â Â Â Â Âmemmove(bootloader_buf, &bootloader_buf[bootloader_cons], 
>> bootloader_prod);
>> + Â Â Â Â Â Âbootloader_cons = 0;
>> + Â Â Â Â}
>
> If the timeout occurs then we will drop through and each of the FD_ISSET
> will fail and we will loop back round to the top and do this processing
> before trying again, which will ensure that xenconsoled_cons == 0.
>
> I think this removes the need for the timeout var you added, as well as
> the associated continue statement and (I think) makes the control flow
> simpler.
>
>> Â Â Â Â ÂFD_ZERO(&rsel);
>> Â Â Â Â ÂFD_SET(fifo_fd, &rsel);
>> Â Â Â Â Ânfds = fifo_fd + 1;
>> - Â Â Â Âif (xenconsoled_prod == 0 || (xenconsoled_prod < 
>> BOOTLOADER_BUF_SIZE && xenconsoled_cons == 0)) {
>> + Â Â Â Âif (xenconsoled_prod == 0 || (xenconsoled_prod < 
>> BOOTLOADER_BUF_SIZE && xenconsoled_cons == 0) || timeout) {
>
> If you've had a timeout there will be data pending on this FD won't
> there, so this change essentially means that after the first timeout the
> timeout on the select becomes effectively zero?

Well, I cannot know this for sure, but it's the most likely situation.

> Can you just make this:
> Â Â Â Â Â if (xenconsoled_prod == 0 || xenconsoled_cons == 0) {

I don't think there's any need for this pair of "ifs", since they will
always be hit, because xenconsoled_cons and bootloader_cons is always
0 at this point, I agree with you that this might be dropping data to
aggressively, since the timeout will not be needed. Something like
this might be more suitable:

if (bootloader_prod == 0 || ret == 0) {

The "timeout" variable is not really needed, I've just added it to
make the code cleaner and easier to understand, but ret can be used in
the same way. From my point of view, we should only add the file
descriptors with a full buffer to the select set after a timeout has
occurred, since checking for a timeout (ret == 0) in the FD_ISSET may
make us enter an infinite loop (because there's data pending on the
file descriptor, and the timeout is not hit).

>
> The previous memmove will ensure we hit this case. Then inside the if
> FD_ISSET(xenconsoled_fd) you can check for prod == BOOTLOADER_BUF_SIZE
> and discard from the buffer (e.g. we can discard the tail just by
> modifying prod?).

We are discarding from the tail? I'm not sure it's the best way to do
it, pygrub uses curses control sequences, and I think it's best to try
to keep the stream as clean as possible, dropping the tail would
really mess up with screen printing (maybe just mess up a frame, but
the output will be rubbish).

> Hmm. that might discard a bit too aggressively, perhaps leaving this
> check as it was and doing the discard if (ret == 0 && xenconsoled_prod
> == BOOTLOADER_BUF_SIZE) in the else clause if the FD_ISSET works?

This might make the select loop too fast, since we add all the
descriptors to the set, and it's a strong possibility that the timeout
will never be hit, so ret will never be 0 and we will be looping
forever.

>> Â Â Â Â Â Â ÂFD_SET(xenconsoled_fd, &rsel);
>> Â Â Â Â Â Â Ânfds = xenconsoled_fd + 1 > nfds ? xenconsoled_fd + 1 : nfds;
>> Â Â Â Â Â}
>> - Â Â Â Âif (bootloader_prod == 0 || (bootloader_prod < BOOTLOADER_BUF_SIZE 
>> && bootloader_cons == 0)) {
>> + Â Â Â Âif (bootloader_prod == 0 || (bootloader_prod < BOOTLOADER_BUF_SIZE 
>> && bootloader_cons == 0) || timeout) {
>> Â Â Â Â Â Â ÂFD_SET(bootloader_fd, &rsel);
>> Â Â Â Â Â Â Ânfds = bootloader_fd + 1 > nfds ? bootloader_fd + 1 : nfds;
>> Â Â Â Â Â}
>> @@ -208,12 +226,23 @@ static char * bootloader_interact(libxl_
>> Â Â Â Â Â Â Ânfds = bootloader_fd + 1 > nfds ? bootloader_fd + 1 : nfds;
>> Â Â Â Â Â}
>>
>> - Â Â Â Âret = select(nfds, &rsel, &wsel, NULL, NULL);
>> + Â Â Â Âret = select(nfds, &rsel, &wsel, NULL, &wait);
>> Â Â Â Â Âif (ret < 0)
>> Â Â Â Â Â Â Âgoto out_err;
>> + Â Â Â Âif (ret == 0) {
>> + Â Â Â Â Â Âtimeout = 1;
>> + Â Â Â Â Â Âcontinue;
>> + Â Â Â Â}
>> + Â Â Â Âtimeout = 0;
>>
>> Â Â Â Â Â/* Input from xenconsole, read xenconsoled_fd, write bootloader_fd 
>> */
>> Â Â Â Â Âif (FD_ISSET(xenconsoled_fd, &rsel)) {
>> + Â Â Â Â Â Âif (xenconsoled_prod == XENCONSOLED_BUF_SIZE) {
>> + Â Â Â Â Â Â Â Âif (ioctl(xenconsoled_fd, FIONREAD, &read_ahead) < 0)
>
> I think we can avoid this as described above, but if not then you need
> to be wary of the case where read_ahead > XENCONSOLED_BUF_SIZE.

Yes, I've also missed that one, this should be checked to avoid
passing a negative integer to read or moving memory regions outside of
the buffer.

>> + Â Â Â Â Â Â Â Â Â Âgoto out_err;
>> + Â Â Â Â Â Â Â Âmemmove(xenconsoled_buf, &xenconsoled_buf[read_ahead], 
>> XENCONSOLED_BUF_SIZE - read_ahead);
>> + Â Â Â Â Â Â Â Âxenconsoled_prod -= read_ahead;
>> + Â Â Â Â Â Â}
>> Â Â Â Â Â Â Âret = read(xenconsoled_fd, &xenconsoled_buf[xenconsoled_prod], 
>> XENCONSOLED_BUF_SIZE - xenconsoled_prod);
>> Â Â Â Â Â Â Âif (ret < 0 && errno != EIO && errno != EAGAIN)
>> Â Â Â Â Â Â Â Â Âgoto out_err;
>> @@ -230,6 +259,12 @@ static char * bootloader_interact(libxl_
>>
>> Â Â Â Â Â/* Input from bootloader, read bootloader_fd, write xenconsoled_fd 
>> */
>> Â Â Â Â Âif (FD_ISSET(bootloader_fd, &rsel)) {
>> + Â Â Â Â Â Âif (bootloader_prod == BOOTLOADER_BUF_SIZE) {
>> + Â Â Â Â Â Â Â Âif (ioctl(bootloader_fd, FIONREAD, &read_ahead) < 0)
>> + Â Â Â Â Â Â Â Â Â Âgoto out_err;
>> + Â Â Â Â Â Â Â Âmemmove(bootloader_buf, &bootloader_buf[read_ahead], 
>> BOOTLOADER_BUF_SIZE - read_ahead);
>> + Â Â Â Â Â Â Â Âbootloader_prod -= read_ahead;
>> + Â Â Â Â Â Â}
>> Â Â Â Â Â Â Âret = read(bootloader_fd, &bootloader_buf[bootloader_prod], 
>> BOOTLOADER_BUF_SIZE - bootloader_prod);
>> Â Â Â Â Â Â Âif (ret < 0 && errno != EIO && errno != EAGAIN)
>> Â Â Â Â Â Â Â Â Âgoto out_err;
>
>
>

I'm wrapping another patch which I will send right now.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel