accept headers in mesh and textures. For texture metrics
reporting, use the AP_INVENTORY policy class which is
non-pipelined and pointing (usually) in the right direction.
Use a do-while(false) structure to manage common exit path
code in onCompleted() methods. Identical to a 'goto' but
might amuse the pedantic. Tuning on background fetch to
have it cycle faster. This is experimental. I suspect
with HTTP balancing in llcorehttp, we can do away with the
timers here.
as transfers can appear delayed with deep pipelining and more
requests in the pool. Added bad HTTP status error (typically
getting a 0 back as HTTP status from libcurl) to the list of
retryable errors. There's a response stream problem with libcurl
and pipelining that induces this problem. Retrying helps but
may not be entirely safe. Watch bug 1420 on the libcurl sourceforge
bug tracker. Extend options of test/example program to include
un-ranged requests. Document the excessive data transfer induced
when ranged requests are disabled. This is an abnormal mode for
very rare users so we'll just eat that for now.
Intended for users with bad networking gear or twitchy ISPs, if set to
True, forces plain GET requests to asset servers for textures and meshes.
This change kicked off a slight refactor in the mesh repository code which
made it resilient against unexpected 200's and responses not covering
the requested start range. There's still too much data copying in the
Mesh code (always has been). Would love to fix that and get rid of the
monolithic temp buffer. Cleaned up white space damage caused by unnamed
linden who likes to drag his magical editor through code.
data so their isn't an opportunity for gaps over overruns (init_data).
Start some preliminary tweaking of policy class numbers. It looks
like I can easily drop the default connection count to '4' and
still hit the throttles. Did some experiments running pipeline
deeper which was mostly fine for textures but tended to slow
meshes. Reason uncertain but a depth of '5' seems generally healthy
for mesh. I had one run of 52.6S with a theoretical minimum of 51.2S.
That's as good as I've ever seen.
GetTexture and GetMesh2 at a pipeline depth of 5. Create
global debug option, HttpPipelining, to enable and disable
HTTP pipelining (defaults to true). Tweak texture and
mesh low- and high-water request levels based on pipelining
status and depth. Fixup texture console which was damaged
in a recent release. Split logging of the no-request
HTTP error case into two cases: one for missing URL in
HTTP request, one for HTTP request not created. A refactor
in llcorehttp is coming: I will be moving all libcurl-
using code into libcurl-specific modules.
Tried to add consistent mesh upload retries in the HTTP work but a
combination of bad status choices in the upload service and the
one-shot nature of the upload capabilities means that status
information can be lost. For now, retain the wonderful manual
retry logic. At some future point, we might fix the services or
add application-level retry.
Problem involved a 3-way livelock between the main, upload
and decomposition threads. Viewer is shutting down but an
upload is in the 'generate hulls' state. Main thread asks
upload request to discard and spins waiting for it to finish.
Upload thread is in generateHulls spinning waiting for the
decomposition thread to process a mesh request. Decomposition
thread is sleeping waiting for main thread to deliver work
that upload thread has asked the decomposition thread to do.
Start using DNS cache in legacy LLCurl code. Go to 15 seconds
particularly as we're using threaded resolver at this point.
Documentation cleanup. Add libcurl status checking and logging
for curl_easy_setopt() operations that fail. Shouldn't happen
and we'll just continue anyway but there's info in the logs to
track these down now. Cleaned up logic around FASTTIMER enable
defines used to evaluate pipeline stalls in main thread.
Removed long-standing thread race around caps strings and
URL construction. Not a significant risk but refactoring the
code to get rid of them removed one huge eyesore. It can be
made even slicker if desired (see notes).
Much earlier identified some thread races including two on mskininfoq
and mdecompositionq. I actually hit one in testing the other day so
I'm fixing them now. Put them under the mMutex lock and use the
mutex in such a way that main thread stalls are not added.
While giving myself a full project code review, found a bug in
the MeshUseGetMesh1 setting. Mostly defaulted to old configuration
but used the GetMesh2 caps which would have been a huge DoS
resource sink. Did some documentation maintenance as well while
I was in there. More for the to-do list, etc.
Added toTerseString() conversion on HttpStatus to generate a string
that's more descriptive than the hex value of the HttpStatus value
but still forms a short, searchable token (e.g. "Http_503" or
"Core_7"). Using this throughout the viewer now, no live cases
of toHex(), I believe.
Added 'MeshUseGetMesh1' and 'MeshUseHttpRetryAfter' debug settings
to control mesh transport behavior. First forces the use of the
legacy mesh fetch style with high concurrency and connection churn.
The second, on by default, honors Retry-After values if they are
reasonable. If off or unreasonable, internal delay times are used.
In case of HTTP errors or parsing/processing errors, fail the
fetch request rather than do a retry spin. Add logging for all
such failure paths. Added a development/debug flag to create
probabilistic failures to test these modes and general error
recovery by higher-level layers.
lists to reflect current. Describe the functional flow of things
for a single LOD request. Put together to-do list for follow on
work. Knock down the low/high water limits for GetMesh a bit,
100/200 too high, 75/150 should be better, vis-a-vis pathological
failures.
Have the ::notifyLoadedMeshes() method doing correct locking
and stall avoidance at the same time. This method now does
lazy mutex lock acquisition (trylock()) and if it fails on
either, it gives up and comes back later. Capture the maximum
number of sequential failures and report this at the end of
the run in the log. (So far, with big mesh regions, I've
only seen 1s and 2s.) Locking/mutex requirements sorted in
other locations as well. LLMutex gets trylock() method as
well as new LLMutexTrylock scoped locking class. Clean up
some documentation, more to do.
added a Mesh status line to the texture fetch console. Mesh is
often in competition with textures and so the mesh information
seems appropriate there. Do get a nice feel for progress and
you definitely see when the throttles kick in.
While linking GetMesh2 to the old setting was simpler from a user
point-of-view, they really shouldn't be linked and the old one will
go away. This one may be renamed to AssetMaxConcurrentRequests or
something similar if we get to the mesh/texture unification step.
This really extended into the client-side request throttling.
Moved this from llmeshrepository (which doesn't really want
to do connection management) into llcorehttp. It's now a
class option with configurable rate. This still isn't the
right thing to do as it creates coupling between viewer
and services. When we get to pipelining, this notion becomes
invalid.
Generally sorted the mesh timeout parameters for maximum
transport time (staying with default 30 for connect). 60S
for normal meshes, 600S for large. Also documented default
option values in httpoptions.h. Useful to have these. In
the future, the timeouts might go into standard llsd options
where they can be tracked a bit more.
With this checkin, legacy LLCurl is out of the mesh code.
Uploaders and responders are converted and functioning. Logging
has been cleaned up throughout the code to use the macro form
with tag/subtag capability. DEBUGS-level logging added for some
upload path milestones. Better error information flow from
failed responses to viewer alert boxes but I'd really, really
like to do better here. Mesh upload problems are completely
opaque as a user. Minor cleanups (removed dead members,
method signatures tidied, less data conversion). Could almost
call this complete, will likely have platform cleanups, however.
Viewer modified for preference for GetMesh2 caps. When found,
uses this cap and uses 1/4 the connection concurrency specified
by MeshMaxConcurrentRequests. Also uses a modified calculation
for high/low water feeding into the llcorehttp library.
Taught llappcorehttp to register signals on the settings values
that chagne behavior. Have initialization and settings changes
sweep through settings and change them. Dynamic changes are tried
but have no effect (produce a warning message) as dynamic settings
still aren't supported but the plumbing is now connected. Just
need to change llcorehttp. Bounced the 'teleport started' signal
around and it ended up back where it started with some cleanup.
This is making me less angry...
Monty Brandenberg