Incorporate the new libcurl 7.38.0 build with curl bug 1420
workaround. Add developer-centric testing code to evaluate
the workaround or a future fix for 1420.
some problems disabling pipelining on a multi handle with outstanding
requests so build a more conservative system that allows requests
to drain before setting curl multi options. Would rather not have
this but it is significantly safer. "HttpPipelining" debug setting
is now fully dynamic. Connection limits can also be made dynamic
in the near future. Upped the default connection count back to 8 for
now but will revisit this in the tuning phase. It might be time to
combine mesh and textures into a single asset class. For normal
server operations that would be a clear path, but for server under
load, the current scheme may be better. Minor cleanup in logging
to elminate some redundant strings. Might add some more tracing to the
stall logic 'just in case'.
GetTexture and GetMesh2 at a pipeline depth of 5. Create
global debug option, HttpPipelining, to enable and disable
HTTP pipelining (defaults to true). Tweak texture and
mesh low- and high-water request levels based on pipelining
status and depth. Fixup texture console which was damaged
in a recent release. Split logging of the no-request
HTTP error case into two cases: one for missing URL in
HTTP request, one for HTTP request not created. A refactor
in llcorehttp is coming: I will be moving all libcurl-
using code into libcurl-specific modules.
Added toTerseString() conversion on HttpStatus to generate a string
that's more descriptive than the hex value of the HttpStatus value
but still forms a short, searchable token (e.g. "Http_503" or
"Core_7"). Using this throughout the viewer now, no live cases
of toHex(), I believe.
This really extended into the client-side request throttling.
Moved this from llmeshrepository (which doesn't really want
to do connection management) into llcorehttp. It's now a
class option with configurable rate. This still isn't the
right thing to do as it creates coupling between viewer
and services. When we get to pipelining, this notion becomes
invalid.
earlier maintenance branch but the code never showed up. I'll do
it again. Spam is still available by bumping 'CoreHttp' tag up
to DEBUGS level logging. Needed for QA. Can also get this data
from tracing.
Much improved. Unified the global and class options into a single
option list. Implemented static and dynamic setting paths as much
as possible. Dynamic path does require packet/RPC but otherwise
there's near unification. Dynamic modes can't get values back yet
due to the response/notifier scheme but this doesn't bother me.
Flatten global and class options into simpler struct-like entities.
Setter/getter available on these when needed (external APIs) but code
can otherwise fiddle directly when it knows what to do. Much duplicated
options/state removed from HttpPolicy. Comments cleaned up. Threads
better described and consistently mentioned in API docs. Integration
test extended for 503 responses with Reply-After headers.
Mesh repo is using three policy classes now: one for
large objects, one for GetMesh2 regions, one for
GetMesh regions. It's also detecting the presence
of the cap and using the correct class. Class
initialization cleaned up significantly in llappcorehttp
using data-directed code. Pulled in the changes to
HttpHeader done for sunshine-internal then did a
refactoring pass on the header callback which now
uses a unified approach to clean up and deliver
header information to all interested parties. Added
support for using Retry-After header information on
503 retries.
Not certain what the source of the short data is with one resident but I'm
going to make these problems retryable as they are transport-related. Lift
the retry detection into a method that should be reusable by others interested
in determining what is retryable. Trace output handling on the libcurl debug
callback was attrocious. Some unsafe length handling on my part was protected
by a second layer of defense. Made that correct and more useful by logging
actual data sizes during trace.
Added second mesh class as well as an asset upload class.
Refactored initialization to use less code and more data to
cleanly get http started. Modified mesh to use the new
http class for large requests (>2MB for now). Added additional
timeout setting to llcorehttp to distinguish connection timeout
from transport timeout and are now using transport timeout
values for large asset downloads that may need more time.
Reformatted messages around request retry. Successfully retried requests
also message so you can see the cycle closed. Added additional retryable
error codes (timeout, other libcurl failures). Commenting and removed some
unnecessary std::min logic.
Data problems after connections are established should be retried as well.
Extend to appropriate libcurl codes. Also allow our connectivity to drop
to as low as a single connection when trying to recover.
The fetch state machine received a new timeout during the WAIT_HTTP_REQ
state. For the integration, rather than jump the state to done, we issue
a request cancel and let the notification plumbing do the rest without
any race conditions or special-case logic.
Some additional work let me enable a memory check for the clean shutdown case and
generally do a better job on other interfaces. Request queue waiters now awake
on shutdown and don't sleep once the queue is turned off. Much better semantically
for how this will be used.
With this commit, the cleanup paths should be production quality. Unit tests have been
expanded to include cases requiring thread termination and cleanup by the worker thread.
Special operation/request added to support the unit tests. Thread interface expanded
to include a very aggressive cancel() method that does not do cleanup but prevents the
thread from accessing objects that will be destroyed.
Groundwork is used for the default class which currently represents
texture fetching. Class options implemented from API user into
HttpLibcurl. Policy layer is going to start doing some traffic
shaping like work to solve problems with consumer-grade gear.
Need to have dynamic aspects to policies and that starts now...
Non-LLThread-based threads need to do some registration or LLMutex locks taken out in these
threads will not work as expected (SH-3154). We'll get a better solution later, this fixes
some things for now. Tracing of operations now supported. Global and per-request (via
HttpOptions) tracing levels of [0..3]. The 2 and 3 levels use libcurl's VERBOSE mode
combined with CURLOPT_DEBUGFUNCTION to stream high levels of detail into the log. *Very*
laggy but useful. Simple GET request supported (no Range: header). Really just a
degenrate case of a ranged get but supplied an API anyway. Global option to use the
LLProxy interface to setup CURL handles for either socks5 or http proxy usage. This
isn't really the most encapsulated way to do this but a better solution will have to
come later. The wantHeaders and tracing options are now supported in HttpOptions giving
per-request controls. Big refactoring of the HTTP resource waiter in lltexturefetch.
What I was doing before wasn't correct. Instead, I'm implementing the resource wait
after the Semaphore model (though not using system semaphores). So instead of having
a sequence like: SEND_HTTP_REQ -> WAIT_HTTP_RESOURCE -> SEND_HTTP_REQ, we now
do WAIT_HTTP_RESOURCE -> WAIT_HTTP_RESOURCE2 (actual wait) -> SEND_HTTP_REQ. Works
well but the prioritized filling of the corehttp library needs some performance
work later.
Implemented/modified PUT & POST to not used chunked encoding for the request.
Made the unit test much happier and probably a better thing for the pipeline.
Have a cheesy static & dynamic proxy capability using both local options and
a way to wire into LLProxy in llmessages. Not a clean thing but it will get
the proxy path working with both socks5 & http proxies. Refactoring to get
rid of unneeded library handler and unified an HttpStatus return for all
requests. Big batch of code removed as a result of that and more is possible
as well as some syscall avoidance with a bit more work. Boosted the unit
tests for simple PUT & POST test which revealed the test harness does *not*
like chunked encoding so we'll avoid it for now (and don't really need it
in any of our schemes).
Retry/response handling is decided in policy so moved that there. Removed special case
206-without-content-range response in transport. Have this sitation recognizable in the
API and let callers deal with it as needed.
surprised me. Added a retry queue similar to ready queue to the
policy object which is sorted by retry time. Currently do five
retries (after the initial try) delayed by .25, .5, 1, 2 and 5
seconds. Removed the retry logic from the lltexturefetch module.
Upped the waiting time in the unit test for the retries. People
won't like this but tough, need tests.
Implemented first global policy definitions to support SSL CA certificate configuration
to support https: operations. Fixed HTTP 206 status handling to match what is currently
being done by grid services and to lay a foundation for fixes that will be a response
to ER-1824. More libcurl CURLOPT options set on easy handles to do peer verification
in the traditional way. HTTP POST working and now reporting asset metrics back to
grid for the viewer's asset system. This uses LLSD so that is also showing as compatible
with the new library.
Identified and reacted to the priority inversion problem we
have in texturefetch. Includes the introduction of a priority_queue
for the requests that are ready. Start some parameterization in
anticipation of having policy_class everywhere. Removed _assert.h
which isn't really needed in indra codebase. Implemented async
setPriority request (which I hope I can get rid of eventually along
with all priorities in this library). Converted to using unsigned
int for priority rather than float. Implemented POST and did
groundwork for PUT.
Rider Linden