Restore original deleteRequest/removeRequest implementation removing
a small race. Remove a short-lived additional timeout scheme on requests
which really isn't appropriate as originally implemented as we can have
very long-lived requests on big regions.
This was yet another refresh from v-d because of significant changes
to lltexturefetch that would not have been resolvable by casual
application of any merge tool. There are still a few questions
outstanding but this is the initial, optimistic merge.
This doesn't really address 3325 directly but it is the result of research
done while hunting it down. First, this is a thread safety improvement for
canceled requests that have gone into resource wait state. I don't think
we've seen a failure there but there was a window. It also cleans the
resource wait queue earlier which lets us do less work and get requests
more quickly into llcorehttp by bypassing the resource wait state. With
this, I finally feel comfortable about rundown of requests.
A/B comparison with original code showed the newer issuing lower-priority
requests of the cache reader and some other minor changes. Brought them
into agreement (this is cargo-cult programming). Made the HTTP resource
semaphore an atomic int for rigorous correctness across threads. I
swear I'm going to tear down this code someday.
Isolate llcorehttp initialization into a utility class (LLAppCoreHttp)
that provides glue between app and library (sets up policies, handles
notifications). Introduce 'TextureFetchConcurrency' debug setting to
provide some field control when absolutely necessary.
Move releaseHttpWaiters() to commonUpdate from doWork.
More appropriate home for it. Have deleteOK() defer deletion
of anything in WAIT_HTTP_RESOURCE2 state to keep pointers valid
for the releaseHttpWaiters() method. It will then transition
canceled operations to SEND_HTTP_REQ where they can be deleted.
When releasing HTTP waiters, avoid unnecessary sort activity.
For Content-Type in responses, let libcurl do the work and removed
my parsing of headers. Drop Content-Encoding as libcurl will deal
with that. If anyone is interested, they can parse.
Well, achieved that by doing work in bulk when needed. But
turned into some additional things. Change timebase from
mS to uS as, well, things are headed that way. Implement
an HttpReplyQueue::fetchAll method (advertised one, hadn't
implemented it).
Doesn't use sets or maps and so there's no ordering assumption to
be violated when priorities are changed. Should also be faster.
Still want to get rid of the ancillary list, however...
First, try to issue ranged GETs that are always at least partially
satisfiable. This will keep Varnish-type caches from simply sending
back 200/full asset responses to unsatisfiable requests. Implement
awareness of Content-Range headers as well. Currently they're not
coming back but they will be someday.
The fetch state machine received a new timeout during the WAIT_HTTP_REQ
state. For the integration, rather than jump the state to done, we issue
a request cancel and let the notification plumbing do the rest without
any race conditions or special-case logic.
Only thing interesting in this changeset is the discovery that a sleep
in the fake HTTP server ties up tests. Need to thread that or fail on
client disconnect or something to speed that up and make it usable for
bigger test scenarios. But good enough for now...
With much trial-and-error, cleaned up the banner on the texture console and made everything
mostly fit. Added global cache read, cache write and resource wait count events to the
console display to show if cache is working. On clean exit, emit a log line to report
stats to log file (intended for automated tests, maybe):
LLTextureFetch::endThread: CacheReads: 2618, CacheWrites: 117, ResWaits: 0, TotalHTTPReq: 117
Non-LLThread-based threads need to do some registration or LLMutex locks taken out in these
threads will not work as expected (SH-3154). We'll get a better solution later, this fixes
some things for now. Tracing of operations now supported. Global and per-request (via
HttpOptions) tracing levels of [0..3]. The 2 and 3 levels use libcurl's VERBOSE mode
combined with CURLOPT_DEBUGFUNCTION to stream high levels of detail into the log. *Very*
laggy but useful. Simple GET request supported (no Range: header). Really just a
degenrate case of a ranged get but supplied an API anyway. Global option to use the
LLProxy interface to setup CURL handles for either socks5 or http proxy usage. This
isn't really the most encapsulated way to do this but a better solution will have to
come later. The wantHeaders and tracing options are now supported in HttpOptions giving
per-request controls. Big refactoring of the HTTP resource waiter in lltexturefetch.
What I was doing before wasn't correct. Instead, I'm implementing the resource wait
after the Semaphore model (though not using system semaphores). So instead of having
a sequence like: SEND_HTTP_REQ -> WAIT_HTTP_RESOURCE -> SEND_HTTP_REQ, we now
do WAIT_HTTP_RESOURCE -> WAIT_HTTP_RESOURCE2 (actual wait) -> SEND_HTTP_REQ. Works
well but the prioritized filling of the corehttp library needs some performance
work later.
Retry/response handling is decided in policy so moved that there. Removed special case
206-without-content-range response in transport. Have this sitation recognizable in the
API and let callers deal with it as needed.
surprised me. Added a retry queue similar to ready queue to the
policy object which is sorted by retry time. Currently do five
retries (after the initial try) delayed by .25, .5, 1, 2 and 5
seconds. Removed the retry logic from the lltexturefetch module.
Upped the waiting time in the unit test for the retries. People
won't like this but tough, need tests.
Went through all the code and tried to document lock and thread usage
in the module. There's a huge comment block introducing all of this
at the beginning and I believe it's correct (though not quite complete).
Keep it updated, people. Added a new state, WAIT_HTTP_RESOURCE, that's
sort of a side-state of SEND_HTTP_REQ. If we hit a high-water mark
for HTTP requests, the extra are shunted to the new state once. Once
levels fall to a low-water mark, we run through a wait list of UUIDs,
sort the valid ones by priority and release them for service. This
keeps the HTTP layer busy while leaving the active queue shallow enough
that requests can still be re-prioritzed cheaply. Priority model
changed. The new state uses the PRIORITY_LOW mask, the old users
of _LOW are now at PRIORITY_NORMAL and sleepers woken up after an
external event are kicked off at PRIORITY_HIGH. This combination
along with the new state should avoid priority inversion and keep
things running without resorting to an infinite pipeline. New
state displays as "HTW" with green text in the texture console.
Request cancelation and worker run-down should now be more
correct but this edge case may need more attention.
Implemented first global policy definitions to support SSL CA certificate configuration
to support https: operations. Fixed HTTP 206 status handling to match what is currently
being done by grid services and to lay a foundation for fixes that will be a response
to ER-1824. More libcurl CURLOPT options set on easy handles to do peer verification
in the traditional way. HTTP POST working and now reporting asset metrics back to
grid for the viewer's asset system. This uses LLSD so that is also showing as compatible
with the new library.
William Todd Stinson