mCoroWaitList was introduced to prevent an assertion failure crash:
LLCoprocedureManager never expects to fill LLCoprocedurePool::mPendingCoprocs
queue. The queue limit was arbitrarily set to 4096 some years ago, but in
practice LLViewerAssetStorage can post way more requests than that.
LLViewerAssetStorage checked whether the target LLCoprocedureManager pool's
queue looked close to full, and if so posted the pending request to its
mCoroWaitList instead. But then it had to override the base LLAssetStorage
method checkForTimeouts() to continually check whether pending tasks could be
moved from mCoroWaitList to LLCoprocedureManager.
A simpler solution is to enlarge LLCorpocedureManager::DEFAULT_QUEUE_SIZE, the
upper limit on mPendingCoprocs. Since mCoroWaitList was an unlimited queue,
making DEFAULT_QUEUE_SIZE "very large" does not increase the risk of runaway
memory consumption.
Making coproc scoped to the for loop will make sure the destructor gets
called every loop iteration. Keeping it's scope outside the for loop
means the pointer keeps valid till the next assigment that happens
inside pop_wait_for when it gets assigned a new value.
Triggering the dtor inside pop_wait_for can lead to deadlock when inside
the dtor a coroutine tries to call enqueueCoprocedure (this happens).
enqueueCoprocedure then will try to grab the lock for try_push but this
lock is still held by pop_wait_for.
The tactic of pushing an empty QueuedCoproc::ptr_t to signal coprocedure close
only works for LLCoprocedurePools with a single coprocedure (e.g. "Upload" and
"AIS"). Only one coprocedureInvokerCoro() coroutine will pop that empty
pointer and shut down properly -- the rest will continue waiting indefinitely.
Rather than pushing some number of empty pointers, hopefully enough to notify
all consumer coroutines, close() the queue. That will notify as many consumers
as there may be.
That means catching LLThreadSafeQueueInterrupt from popBack(), instead of
detecting empty pointer.
Also, if a queued coprocedure throws an exception, coprocedureInvokerCoro()
logs it as before -- but instead of rethrowing it, the coroutine now loops
back to wait for more work. Otherwise, the number of coroutines servicing the
queue dwindles.
The new close(void) method simply acquires the logic from
~LLCoprocedureManager() (which now calls close()). It's useful, even if only
in test programs, to be able to shut down all existing LLCoprocedurePools
without having to name them individually -- and without having to destroy the
LLCoprocedureManager singleton instance. Deleting an LLSingleton should be
done only once per process, whereas test programs want to reset the
LLCoprocedureManager after each test.
Reinstate LLCoprocedureManager::countPending() and count() methods. These were
removed because boost::fibers::buffered_channel has no size() method, but
since all users run within a single thread, it works to increment and
decrement a simple counter.
Add count information and max queue size to log messages.
Since the consuming coroutine LLCoprocedurePool::coprocedureInvokerCoro() has
been observed to outlive the LLCoprocedurePool instance that owns the
CoprocQueue_t, closing that queue isn't enough to keep the coroutine from
crashing at shutdown: accessing a deleted CoprocQueue_t is fatal whether or
not it's been closed.
Make LLCoprocedurePool store a shared_ptr to a heap CoprocQueue_t instance,
and pass that shared_ptr by value to consuming coroutines. That way the
CoprocQueue_t instance is guaranteed to live as long as the last interested
party.
By the time "LLApp" listeners are notified that the app is quitting, the
mainloop is no longer running. Even though those listeners do things like
close work queues and inject exceptions into pending promises, any coroutines
waiting on those resources must regain control before they can notice and shut
down properly. Add a final "LLApp" listener that resumes ready coroutines a
few more times.
Make sure every other "LLApp" listener is positioned before that new one.
Add LLCoros::TempStatus instances around known suspension points so
printActiveCoroutines() can report what each suspended coroutine is waiting
for.
Similarly, sprinkle checkStop() calls at known suspension points.
Make LLApp::setStatus() post an event to a new LLEventPump "LLApp" with a
string corresponding to the status value being set, but only until
~LLEventPumps() -- since setStatus() also gets called very late in the
application's lifetime.
Make postAndSuspendSetup() (used by postAndSuspend(), suspendUntilEventOn(),
postAndSuspendWithTimeout(), suspendUntilEventOnWithTimeout()) add a listener
on the new "LLApp" LLEventPump that pushes the new LLCoros::Stopping exception
to the coroutine waiting on the LLCoros::Promise. Make it return the new
LLBoundListener along with the previous one.
Accordingly, make postAndSuspend() and postAndSuspendWithTimeout() store the
new LLBoundListener returned by postAndSuspendSetup() in a LLTempBoundListener
(as with the previous one) so it will automatically disconnect once the wait
is over.
Make each LLCoprocedurePool instance listen on "LLApp" with a listener that
closes the queue on which new work items are dispatched. Closing the queue
causes the waiting dispatch coroutine to terminate. Store the connection in an
LLTempBoundListener on the LLCoprocedurePool so it will disconnect
automatically on destruction.
Refactor the loop in coprocedureInvokerCoro() to instantiate TempStatus around
the suspending call.
Change a couple spammy LL_INFOS() calls to LL_DEBUGS(). Give all logging calls
in that module a "CoProcMgr" tag to make it straightforward to re-enable the
LL_DEBUGS() calls as desired.
Ansariel