[coyotos-dev] Interrupt management

[Jonathan S. Shapiro]

Soliciting input on a minor issue in interrupt management.

The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call. To work
around this, we tend to have "helper processes" that sit in a loop of
the form

for(;;) {
wait for interrupt
post notice to driver
}

For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

Reactions?


shap

[Sam Mason]

Post by Jonathan S. Shapiro
The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call.

Why can't it work in a similar fashion to the sleep() interface? or is
the purpose of the helper process to cause the "correct" process to
block?

Post by Jonathan S. Shapiro
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

Why do interrupts need to be reserved (especially in the kernel)? I'd
expect that the ability to wait on a hardware interrupt to be pretty
closely guarded and hence not an issue.


I've just noticed that the kernel docs refer to a IrqRange (maybe
introduced in [1]) interface that doesn't appear to be defined anywhere.


Sam

[1] http://dev.eros-os.com/hg/coyotos/trunk/rev/8c368493cb5c

[Jonathan S. Shapiro]

Post by Sam Mason

Post by Jonathan S. Shapiro
The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call.

Why can't it work in a similar fashion to the sleep() interface? or is
the purpose of the helper process to cause the "correct" process to
block?

Because the process waiting for the interrupt needs to do other things.
What we want here is a notification upcall. The role of the helper
process is to convert the wait into an upcall.

Post by Sam Mason

Post by Jonathan S. Shapiro
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

Why do interrupts need to be reserved (especially in the kernel)? I'd
expect that the ability to wait on a hardware interrupt to be pretty
closely guarded and hence not an issue.

Guarded by whom?

They need to be reserved because devices need to sort out who owns what
interrupt at device startup time. The process of reservation is how this
is done.


shap

[Sam Mason]

Post by Jonathan S. Shapiro

Post by Sam Mason

Post by Jonathan S. Shapiro
The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call.

Why can't it work in a similar fashion to the sleep() interface? or is
the purpose of the helper process to cause the "correct" process to
block?

Because the process waiting for the interrupt needs to do other things.
What we want here is a notification upcall. The role of the helper
process is to convert the wait into an upcall.

Of course; I was only thinking of things like disk drivers when you know
when you're waiting for a response (with SCSI or IDE command queuing
even this isn't true). The general case is that of being interrupted,
hence the name.

Post by Jonathan S. Shapiro

Post by Sam Mason

Post by Jonathan S. Shapiro
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

Why do interrupts need to be reserved (especially in the kernel)? I'd
expect that the ability to wait on a hardware interrupt to be pretty
closely guarded and hence not an issue.

Guarded by whom?

I think maybe I used the wrong term, maybe "closely held"? I meant a
capability that wouldn't make its way out to normal untrusted processes,
like the one to drive a checkpoint.

Post by Jonathan S. Shapiro
They need to be reserved because devices need to sort out who owns what
interrupt at device startup time. The process of reservation is how this
is done.

I thought the general case with modern PCI busses was that of interrupt
sharing, which implies to me that interrupts can't be reserved. Or
rather, it can't be knowing a priori which interrupts can be shared.
Drivers could register whether they support sharing, but this doesn't
seem to be very helpful when you have two pieces of hardware on the same
interrupt line.


Sam

[Jonathan S. Shapiro]

Post by Sam Mason

Post by Jonathan S. Shapiro

Post by Sam Mason

Post by Jonathan S. Shapiro
The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call.

Why can't it work in a similar fashion to the sleep() interface? or is
the purpose of the helper process to cause the "correct" process to
block?

Because the process waiting for the interrupt needs to do other things.
What we want here is a notification upcall. The role of the helper
process is to convert the wait into an upcall.

Of course; I was only thinking of things like disk drivers when you know
when you're waiting for a response (with SCSI or IDE command queuing
even this isn't true). The general case is that of being interrupted,
hence the name.

This is the solicited vs. unsolicited distinction. For better or worse,
solicited interrupts are less and less useful this way. As the devices
become progressively more asynchronous (SATA) and start to behave more
like general busses, treating all interrupts as unsolicited becomes
progressively more attractive. There are still differences from a
scheduling and rate planning perspective, but less and less from an
interrupt demultiplexing perspective.

Post by Sam Mason

Post by Jonathan S. Shapiro

Post by Sam Mason

Post by Jonathan S. Shapiro
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

Why do interrupts need to be reserved (especially in the kernel)? I'd
expect that the ability to wait on a hardware interrupt to be pretty
closely guarded and hence not an issue.

Guarded by whom?

I think maybe I used the wrong term, maybe "closely held"? I meant a
capability that wouldn't make its way out to normal untrusted processes,
like the one to drive a checkpoint.

It doesn't. But that doesn't alter the need to negotiate assignment of
interrupts to hardware devices.

Post by Sam Mason

Post by Jonathan S. Shapiro
They need to be reserved because devices need to sort out who owns what
interrupt at device startup time. The process of reservation is how this
is done.

I thought the general case with modern PCI busses was that of interrupt
sharing, which implies to me that interrupts can't be reserved. Or
rather, it can't be knowing a priori which interrupts can be shared.
Drivers could register whether they support sharing, but this doesn't
seem to be very helpful when you have two pieces of hardware on the same
interrupt line.

Interrupt sharing is the problem that one would would *desperately* like
to prevent. The single worst decision in the design of the PCI bus was
the fact that the specification only supports four distinct physical
interrupt messages at the PCI layer and compliant implementations are
permitted to implement only one of those (and many laptop
implementations actually do that).

IMHO, current hardware interrupt architectures present a depressingly
strong argument *against* microkernels.


shap

[Sam Mason]

Post by Jonathan S. Shapiro

Post by Sam Mason
I was only thinking of things like disk drivers when you know
when you're waiting for a response (with SCSI or IDE command queuing
even this isn't true). The general case is that of being interrupted,
hence the name.

This is the solicited vs. unsolicited distinction. For better or worse,
solicited interrupts are less and less useful this way. As the devices
become progressively more asynchronous (SATA) and start to behave more
like general busses, treating all interrupts as unsolicited becomes
progressively more attractive. There are still differences from a
scheduling and rate planning perspective, but less and less from an
interrupt demultiplexing perspective.

Again I'm showing my naivety, but I'd never heard of a real distinction
between the two. You're going to livelock under (say) a gigabit network
adapter's packet storm, but a harddrive will only issue interrupts when
you've got time to ask for more data.

Post by Jonathan S. Shapiro
Interrupt sharing is the problem that one would would *desperately* like
to prevent.

But is it possible to prevent it on current hardware?

Post by Jonathan S. Shapiro
The single worst decision in the design of the PCI bus was
the fact that the specification only supports four distinct physical
interrupt messages at the PCI layer and compliant implementations are
permitted to implement only one of those (and many laptop
implementations actually do that).

I still don't understand how you go about reserving interrupts when any
form of sharing is going to exist.

Post by Jonathan S. Shapiro
IMHO, current hardware interrupt architectures present a depressingly
strong argument *against* microkernels.

I remember being told a nice old saying about the job of a hardware
engineer being to make software engineer's life difficult, and the
converse. Somehow that seems relevant!


I think I should have stayed quiet though, I'm not sure if any of this
applies to your original question.


Sam

[Jonathan S. Shapiro]

Post by Sam Mason
Again I'm showing my naivety, but I'd never heard of a real distinction
between the two. You're going to livelock under (say) a gigabit network
adapter's packet storm, but a harddrive will only issue interrupts when
you've got time to ask for more data.

Right. This is why a good gigabit network driver will do regime shifts
between interrupt driven and polled interrupts according to load.

Post by Sam Mason

Post by Jonathan S. Shapiro
Interrupt sharing is the problem that one would would *desperately* like
to prevent.

But is it possible to prevent it on current hardware?

No. But there are better and worse assignments of interrupts that are
possible at system bringup.

Post by Sam Mason

Post by Jonathan S. Shapiro
The single worst decision in the design of the PCI bus was
the fact that the specification only supports four distinct physical
interrupt messages at the PCI layer and compliant implementations are
permitted to implement only one of those (and many laptop
implementations actually do that).

I still don't understand how you go about reserving interrupts when any
form of sharing is going to exist.

What you find on a desktop system is that you have four independent
hardware lines that you can allocate, and you rarely have four devices
that are simultaneously active on the bus.


shap

[Sam Mason]

Post by Jonathan S. Shapiro

Post by Sam Mason
Again I'm showing my naivety, but I'd never heard of a real distinction
between the two. You're going to livelock under (say) a gigabit network
adapter's packet storm, but a harddrive will only issue interrupts when
you've got time to ask for more data.

Right. This is why a good gigabit network driver will do regime shifts
between interrupt driven and polled interrupts according to load.

Yes, IP networks are also nice in that if things get too bad you can
drop packets and normal retry and backoff mechanisms (of well behaved
peers) should help to reduce load. Not sure how much this applies to
other types of hardware though.

Post by Jonathan S. Shapiro
there are better and worse assignments of interrupts that are
possible at system bringup.
What you find on a desktop system is that you have four independent
hardware lines that you can allocate, and you rarely have four devices
that are simultaneously active on the bus.

And, as I understand, bigger boxes have multiple independent PCI busses
each with their own four lines making sharing even less common.

Though I'm still not sure what's being reserved! It sounds as though
shared interrupt lines is the general case, though hopefully not the
common one, so it seems to make sense to reserve device specific
interrupt lines.


Sam

[Christopher Nelson]

Post by Jonathan S. Shapiro
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

I like it. One ring to rule them all. Less total overhead for
servicing them, and potential multithreadedness could allow servicing
these requests to scale on multi-core machines.

[Jonathan S. Shapiro]

Post by Christopher Nelson

Post by Jonathan S. Shapiro
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

I like it. One ring to rule them all. Less total overhead for
servicing them, and potential multithreadedness could allow servicing
these requests to scale on multi-core machines.

Actually, the scaling is better with the "one irq, one helper" approach.

[Christopher Nelson]

Post by Jonathan S. Shapiro

Post by Christopher Nelson

Post by Jonathan S. Shapiro
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

I like it. One ring to rule them all. Less total overhead for
servicing them, and potential multithreadedness could allow servicing
these requests to scale on multi-core machines.

Actually, the scaling is better with the "one irq, one helper" approach.

I don't think so. If you have one device that emits a tremendous
amount of interrupts, like say a loaded network card, someone is going
to get starved if you only have one thread handling them.

[Jonathan S. Shapiro]

Post by Christopher Nelson

Post by Jonathan S. Shapiro
Actually, the scaling is better with the "one irq, one helper" approach.

I don't think so. If you have one device that emits a tremendous
amount of interrupts, like say a loaded network card, someone is going
to get starved if you only have one thread handling them.

You misread me. I was saying that one thread per irq is more scalable.
Actually, one thread per CPU is as good as it gets.

[Jonathan S. Shapiro]

Post by Jonathan S. Shapiro
You misread me. I was saying that one thread per irq is more scalable.
Actually, one thread per CPU is as good as it gets.

Sorry. I should have said "I wasn't clear."

[Christopher Nelson]

Yes, I misunderstood you. :-)

Post by Jonathan S. Shapiro

Post by Christopher Nelson

Post by Jonathan S. Shapiro
Actually, the scaling is better with the "one irq, one helper" approach.

I don't think so. If you have one device that emits a tremendous
amount of interrupts, like say a loaded network card, someone is going
to get starved if you only have one thread handling them.

You misread me. I was saying that one thread per irq is more scalable.
Actually, one thread per CPU is as good as it gets.
_______________________________________________
coyotos-dev mailing list
http://www.coyotos.org/mailman/listinfo/coyotos-dev

[Charles Landau]

Post by Jonathan S. Shapiro
The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call. To work
around this, we tend to have "helper processes" that sit in a loop of
the form
for(;;) {
wait for interrupt
post notice to driver
}
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

I'm not sure what terminology you're using. In the past you've argued
that the term "process" should be used for a single thread of execution,
and several of those might happen to share an address space.

In CapROS interrupts are fielded by a thread/process executing:

for(;;) {
wait for interrupt
call interrupt service procedure
}

The interrupt service procedure is a parameter to the Linux procedure
request_irq(). It executes in the same address space as the other
threads/processes of the driver, with which it shares variables.

Interrupt reservation is done using the allocIRQ operation on the
DevPrivs capability, as inherited from EROS. (No link, because I can no
longer find the EROS code online.)

[Jonathan S. Shapiro]

Post by Charles Landau

Post by Jonathan S. Shapiro
The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call. To work
around this, we tend to have "helper processes" that sit in a loop of
the form
for(;;) {
wait for interrupt
post notice to driver
}
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

I'm not sure what terminology you're using. In the past you've argued
that the term "process" should be used for a single thread of execution,
and several of those might happen to share an address space.

Yes. And that it was I mean here by "multithreaded". Multiple kernel
processes sharing an address space.

Post by Charles Landau
for(;;) {
wait for interrupt
call interrupt service procedure
}

This doesn't quite answer my question, though. Is this single process
waiting for *any* interrupt, or is it waiting for some particular
interrupt?

Either is feasible, but there are scalability problems with a "one
waiter" design.

We're not using Linux drivers at this point.

Post by Charles Landau
Interrupt reservation is done using the allocIRQ operation on the
DevPrivs capability, as inherited from EROS. (No link, because I can no
longer find the EROS code online.

Urk. I'll make a note to figure out what happened to the online code.
What I should probably do is just check the final state of the EROS tree
into a Mercurial repository.


shap

[Charles Landau]

Post by Jonathan S. Shapiro

Post by Jonathan S. Shapiro
for(;;) {
wait for interrupt
call interrupt service procedure
}

This doesn't quite answer my question, though. Is this single process
waiting for *any* interrupt, or is it waiting for some particular
interrupt?

I wasn't clear. It waits for a particular interrupt.
A different driver will have a different thread/process waiting for its
interrupt, and sharing address space with only that driver.

[David-Sarah Hopwood]

Post by Jonathan S. Shapiro
Soliciting input on a minor issue in interrupt management.
The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call. To work
around this, we tend to have "helper processes" that sit in a loop of
the form
for(;;) {
wait for interrupt
post notice to driver
}
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

I don't see why this really matters very much. It's a kernel
implementation decision that doesn't affect the kernel <-> driver
interface, AFAICS, and so can be changed at any time without
compatibility issues.

For scalability, you probably want the number of such threads to be
min(number of CPUs, number of physical hardware IRQ lines). How many
address spaces? Well, this code has to be completely trusted since
it may be dispatching interrupts for any device, so there's not really
any security benefit to be had from using separate address spaces.
Then again, I can't see any strong functional requirement for these
threads to share an address space, either. So do whatever's
convenient.

[Jonathan S. Shapiro]

Post by David-Sarah Hopwood

Post by Jonathan S. Shapiro
Soliciting input on a minor issue in interrupt management.
The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call. To work
around this, we tend to have "helper processes" that sit in a loop of
the form
for(;;) {
wait for interrupt
post notice to driver
}
For a variety of reasons, mainly having to do with interrupt
reservation, I'm wondering whether this function shouldn't be
centralized to a single (possibly multi-threaded) process.

I don't see why this really matters very much. It's a kernel
implementation decision that doesn't affect the kernel <-> driver
interface, AFAICS, and so can be changed at any time without
compatibility issues.

1. This has absolutely nothing to do with the kernel implementation.

2. The fact that it can be changed in theory does not mean that we will
ever have time to revisit it in practice. I posed this question because
I was implementing urgently and knew that I did not have time to do it
twice.


shap

[Nathaniel W Filardo]

Post by Jonathan S. Shapiro
Soliciting input on a minor issue in interrupt management.
The Coyotos kernel cannot perform general up-calls, mainly because it
has no place to store the capabilities that it might call.
[snip]
Reactions?

The last time you and I talked about interrupts (so probably a little under
a year ago in the JHU ACM office), I was arguing aginst the IrqWait
mechanism and suggested using kernel-held AppNotice caps as an alternative.
The upshot, as I recall, was that AppNotice caps worked so long as there was
some mechanism to prevent reentrancy; I don't recall how we settled that,
though a few ideas jump to mind. Similarly for IRQ sharing.

I think we agreed that the registration call "granting" the AppNotice cap to
the kernel (it had it already, thus the scarequotes) and binding the IRQ
channel also had to provide the spacebank from which the kernel could
allocate enough space for whatever metadata it needed.

Is there something that has subsequently invalidated this design or that I
am misremembering from the conversation? Is my ignorance showing? :)

Be well.
--nwf;

[Jonathan S. Shapiro]

The last time you and I talked about interrupts...
I think we agreed that the registration call "granting" the AppNotice cap to
the kernel (it had it already, thus the scarequotes) and binding the IRQ
channel also had to provide the spacebank from which the kernel could
allocate enough space for whatever metadata it needed.

Doesn't work.

Generating an upcall is MUCH more complex than moving a waiting process
to the ready queue. Further, there is no guarantee that the target of
the AppNotice is ready to receive.


shap