The new helper functions check_curl_easy_setopt() and
check_curl_multi_setopt() encapsulate the pervasive idiom:
code = curl_{easy,multi}_setopt(handle, option, arg);
check_curl_{easy,multi}_code(code, option);
But since each of these helper functions contains its own local CURL{,M}code
variable 'code', having a caller-scope variable reused for every such call is
no longer necessary -- in fact is no longer used at all. That produces a fatal
warning with MSVC. Get rid of those now-unused variables.
Under pipelining, requests were given a 5x timeout factor due to the
way that the timeout clock works in libcurl. Under CDN load,
connections were not being torn down quickly and it was only
this timer that led to disconnect and retry. So, we want to
break a connection that isn't making progress but that isn't
immediately possible. We'll compromise with a 60S timeout that
(we hope) will be neither too long for stalled connections nor
too short for large asset transfer requests.
Incorporate the new libcurl 7.38.0 build with curl bug 1420
workaround. Add developer-centric testing code to evaluate
the workaround or a future fix for 1420.
as transfers can appear delayed with deep pipelining and more
requests in the pool. Added bad HTTP status error (typically
getting a 0 back as HTTP status from libcurl) to the list of
retryable errors. There's a response stream problem with libcurl
and pipelining that induces this problem. Retrying helps but
may not be entirely safe. Watch bug 1420 on the libcurl sourceforge
bug tracker. Extend options of test/example program to include
un-ranged requests. Document the excessive data transfer induced
when ranged requests are disabled. This is an abnormal mode for
very rare users so we'll just eat that for now.
some problems disabling pipelining on a multi handle with outstanding
requests so build a more conservative system that allows requests
to drain before setting curl multi options. Would rather not have
this but it is significantly safer. "HttpPipelining" debug setting
is now fully dynamic. Connection limits can also be made dynamic
in the near future. Upped the default connection count back to 8 for
now but will revisit this in the tuning phase. It might be time to
combine mesh and textures into a single asset class. For normal
server operations that would be a clear path, but for server under
load, the current scheme may be better. Minor cleanup in logging
to elminate some redundant strings. Might add some more tracing to the
stall logic 'just in case'.
Start using DNS cache in legacy LLCurl code. Go to 15 seconds
particularly as we're using threaded resolver at this point.
Documentation cleanup. Add libcurl status checking and logging
for curl_easy_setopt() operations that fail. Shouldn't happen
and we'll just continue anyway but there's info in the logs to
track these down now. Cleaned up logic around FASTTIMER enable
defines used to evaluate pipeline stalls in main thread.
Removed long-standing thread race around caps strings and
URL construction. Not a significant risk but refactoring the
code to get rid of them removed one huge eyesore. It can be
made even slicker if desired (see notes).
Much improved. Unified the global and class options into a single
option list. Implemented static and dynamic setting paths as much
as possible. Dynamic path does require packet/RPC but otherwise
there's near unification. Dynamic modes can't get values back yet
due to the response/notifier scheme but this doesn't bother me.
Flatten global and class options into simpler struct-like entities.
Setter/getter available on these when needed (external APIs) but code
can otherwise fiddle directly when it knows what to do. Much duplicated
options/state removed from HttpPolicy. Comments cleaned up. Threads
better described and consistently mentioned in API docs. Integration
test extended for 503 responses with Reply-After headers.
termination test. Sheesh. Also get some more numbers out of the
example/load test program which drives traffic. This should
give some useful insights into how the current http throttle
works (or doesn't) with varying client demands.
Mesh repo is using three policy classes now: one for
large objects, one for GetMesh2 regions, one for
GetMesh regions. It's also detecting the presence
of the cap and using the correct class. Class
initialization cleaned up significantly in llappcorehttp
using data-directed code. Pulled in the changes to
HttpHeader done for sunshine-internal then did a
refactoring pass on the header callback which now
uses a unified approach to clean up and deliver
header information to all interested parties. Added
support for using Retry-After header information on
503 retries.
Not certain what the source of the short data is with one resident but I'm
going to make these problems retryable as they are transport-related. Lift
the retry detection into a method that should be reusable by others interested
in determining what is retryable. Trace output handling on the libcurl debug
callback was attrocious. Some unsafe length handling on my part was protected
by a second layer of defense. Made that correct and more useful by logging
actual data sizes during trace.
Header container moves from a vector of raw lines to a vector
of string pairs representing name/value pairs in headers. For
incoming headers, we normalize the name to lowercase and trim
it. Values are only left-trimmed. Outgoing headers are left
as-is. Simple find() method for the common case, forward and
reverse iterators for those few who need to do it themselves.
The HTTP status line (e.g. 'HTTP/1.1 200 Ok') is no longer treated
as a header to be returned to caller. Unit tests, as usual,
were a bear but they absolutely ensured outgoing HTTP header
conformance after the change. Grunt work paid off.
LLTextureFetch was also given a second options structure
for texture fetches. Same as the original but with header return
to caller requested. Baked textures should use this, the other
20,000 texture fetch requests should continue to use the original.
Added second mesh class as well as an asset upload class.
Refactored initialization to use less code and more data to
cleanly get http started. Modified mesh to use the new
http class for large requests (>2MB for now). Added additional
timeout setting to llcorehttp to distinguish connection timeout
from transport timeout and are now using transport timeout
values for large asset downloads that may need more time.
Header container moves from a vector of raw lines to a vector
of string pairs representing name/value pairs in headers. For
incoming headers, we normalize the name to lowercase and trim
it. Values are only left-trimmed. Outgoing headers are left
as-is. Simple find() method for the common case, forward and
reverse iterators for those few who need to do it themselves.
The HTTP status line (e.g. 'HTTP/1.1 200 Ok') is no longer treated
as a header to be returned to caller. Unit tests, as usual,
were a bear but they absolutely ensured outgoing HTTP header
conformance after the change. Grunt work paid off.
LLTextureFetch was also given a second options structure
for texture fetches. Same as the original but with header return
to caller requested. Baked textures should use this, the other
20,000 texture fetch requests should continue to use the original.
Nat Goodspeed