Apparently, in previous Boost versions, boost::noncopyable was sneaking into
the namespace via other headers. Now the compiler complains about its absence
without an explicit #include.
For reasons not yet diagnosed, specifically in Mac Release builds, the tests
in test_httprequest.hpp consistently crash with a backtrace suggesting that
the worker thread is calling LLCore::HttpLibcurl::completeRequest() after the
foreground thread calls HttpRequest::destroyService().
Weirdly, even executing a tut::skip() call in every test<n>() function up to
the point of the crash does not eliminate the crash.
NickyD discovered that the substitute default allocator used for llcorehttp
tests was returning badly-aligned storage, which caused access violations on
alignment-sensitive data such as std::atomic. Thanks Nicky!!
Moreover, the llcorehttp test assertions regarding memory usage, well-
intentioned though they are, have been causing us trouble for years. Many have
already been disabled.
The problem is that use of test_allocator.h affected *everything* defined with
that header file's declarations visible. That inevitably included specific
functions in other subsystems. Those functions then (unintentionally) consumed
the special allocator, throwing off the memory tracking and making certain
memory-related assertions consistently fail.
This is a particular, observable bad effect of One Definition Rule violations.
Within a given program, C++ allows multiple definitions for the same entity,
but requires that all such definitions be the same. Partial visibility of the
global operator new() and operator delete() overrides meant that some
definitions of certain entities used the default global allocator, some used
llcorehttp's. There may have been other, more subtle bad effects of these ODR
violations.
If one wanted to reimplement verification of the memory consumption of
llcorehttp classes:
* Each llcorehttp class (for which memory tracking was desired) should declare
class-specific operator new() and operator delete() methods. Naturally,
these would all consume a central llcorehttp-specific allocator, but that
allocator should *not* be named global operator new().
* Presumably that would require runtime indirection to allow using the default
allocator in production while substituting the special allocator for tests.
* Recording and verifying the memory consumption in each test should be
performed in the test-object constructor and destructor, rather than being
sprinkled throughout the test<n>() methods.
* With that mechanism in place, the test object should provide methods to
adjust (or entirely disable) memory verification for a particular test.
* The test object should also provide a "yes, we're still consuming llcorehttp
memory" method to be used for spot checks in the middle of tests -- instead
of sprinkling in explicit comparisons as before.
* In fact, the llcorehttp test object in each test_*.hpp file should be
derived from a central llcorehttp test-object base class providing those
methods.
llcorehttp's test_allocator.{h,cpp} overrides global operator new(), operator
new[](), operator delete() and operator delete[](). The two operator new()
functions used to be declared with throw(std::bad_alloc). Worse, for VS 2013
and previous, we needed _THROW0() and _THROW1(std::bad_alloc) instead,
requiring #if logic.
But with dynamic throw declarations deprecated, we must actually remove those.
That obviates the THROW_BAD_ALLOC() / THROW_NOTHING() workarounds in
test_allocator.cpp.
In three different places we use the same pattern: an ssl_thread_id_callback()
function (a static member of LLCrashLogger, in that case) that used to be
passed to CRYPTO_set_id_callback() and therefore returned an unsigned long
representing the ID of the current thread.
But GetCurrentThread() is a HANDLE, an alias for a pointer, and you can't
uniquely cram a 64-bit pointer into an unsigned long.
Fortunately OpenSSL has a more modern API for retrieving thread ID. Pass
each ssl_thread_id_callback() function to CRYPTO_THREADID_set_callback()
instead, converting it to accept CRYPTO_THREADID* and call
CRYPTO_THREADID_set_pointer() or CRYPTO_THREADID_set_numeric() as appropriate().
LLThread::currentID() used to return a U32, a distinct unsigned value
incremented by explicitly constructing LLThread or by calling LLThread::
registerThreadID() early in a thread launched by other means. The latter
imposed an unobvious requirement on new code based on std::thread. Using
std:🧵:id instead delegates to the compiler/library the problem of
distinguishing threads launched by any means.
Change lots of explicit U32 declarations. Introduce LLThread::id_t typedef to
avoid having to run around fixing uses again if we later revisit this decision.
LLMutex, which stores an LLThread::id_t, wants a distinguished value meaning
NO_THREAD, and had an enum with that name. But as std:🧵:id promises
that the default-constructed value is distinct from every valid value,
NO_THREAD becomes unnecessary and goes away.
Because LLMutex now stores LLThread::id_t instead of U32, make llmutex.h
#include "llthread.h" instead of the other way around. This makes LLMutex an
incomplete type within llthread.h, so move LLThread::lockData() and
unlockData() to the .cpp file. Similarly, remove llrefcount.h's #include
"llmutex.h" to break circularity; instead forward-declare LLMutex.
It turns out that a number of source files assumed that #include "llthread.h"
would get the definition for LLMutex. Sprinkle #include "llmutex.h" as needed.
In the SAFE_SSL code in llcorehttp/httpcommon.cpp, there's an ssl_thread_id()
callback that returns an unsigned long to the SSL library. When LLThread::
currentID() was U32, we could simply return that. But std:🧵:id is very
deliberately opaque, and can't be reinterpret_cast to unsigned long.
Fortunately it can be hashed because std::hash is specialized with that type.
LLThread::currentID() used to return a U32, a distinct unsigned value
incremented by explicitly constructing LLThread or by calling LLThread::
registerThreadID() early in a thread launched by other means. The latter
imposed an unobvious requirement on new code based on std::thread. Using
std:🧵:id instead delegates to the compiler/library the problem of
distinguishing threads launched by any means.
Change lots of explicit U32 declarations. Introduce LLThread::id_t typedef to
avoid having to run around fixing uses again if we later revisit this decision.
LLMutex, which stores an LLThread::id_t, wants a distinguished value meaning
NO_THREAD, and had an enum with that name. But as std:🧵:id promises
that the default-constructed value is distinct from every valid value,
NO_THREAD becomes unnecessary and goes away.
Because LLMutex now stores LLThread::id_t instead of U32, make llmutex.h
#include "llthread.h" instead of the other way around. This makes LLMutex an
incomplete type within llthread.h, so move LLThread::lockData() and
unlockData() to the .cpp file. Similarly, remove llrefcount.h's #include
"llmutex.h" to break circularity; instead forward-declare LLMutex.
It turns out that a number of source files assumed that #include "llthread.h"
would get the definition for LLMutex. Sprinkle #include "llmutex.h" as needed.
In the SAFE_SSL code in llcorehttp/httpcommon.cpp, there's an ssl_thread_id()
callback that returns an unsigned long to the SSL library. When LLThread::
currentID() was U32, we could simply return that. But std:🧵:id is very
deliberately opaque, and can't be reinterpret_cast to unsigned long.
Fortunately it can be hashed because std::hash is specialized with that type.
For reasons not yet diagnosed, specifically in Mac Release builds, the tests
in test_httprequest.hpp consistently crash with a backtrace suggesting that
the worker thread is calling LLCore::HttpLibcurl::completeRequest() after the
foreground thread calls HttpRequest::destroyService().
Weirdly, even executing a tut::skip() call in every test<n>() function up to
the point of the crash does not eliminate the crash.
NickyD discovered that the substitute default allocator used for llcorehttp
tests was returning badly-aligned storage, which caused access violations on
alignment-sensitive data such as std::atomic. Thanks Nicky!!
Moreover, the llcorehttp test assertions regarding memory usage, well-
intentioned though they are, have been causing us trouble for years. Many have
already been disabled.
The problem is that use of test_allocator.h affected *everything* defined with
that header file's declarations visible. That inevitably included specific
functions in other subsystems. Those functions then (unintentionally) consumed
the special allocator, throwing off the memory tracking and making certain
memory-related assertions consistently fail.
This is a particular, observable bad effect of One Definition Rule violations.
Within a given program, C++ allows multiple definitions for the same entity,
but requires that all such definitions be the same. Partial visibility of the
global operator new() and operator delete() overrides meant that some
definitions of certain entities used the default global allocator, some used
llcorehttp's. There may have been other, more subtle bad effects of these ODR
violations.
If one wanted to reimplement verification of the memory consumption of
llcorehttp classes:
* Each llcorehttp class (for which memory tracking was desired) should declare
class-specific operator new() and operator delete() methods. Naturally,
these would all consume a central llcorehttp-specific allocator, but that
allocator should *not* be named global operator new().
* Presumably that would require runtime indirection to allow using the default
allocator in production while substituting the special allocator for tests.
* Recording and verifying the memory consumption in each test should be
performed in the test-object constructor and destructor, rather than being
sprinkled throughout the test<n>() methods.
* With that mechanism in place, the test object should provide methods to
adjust (or entirely disable) memory verification for a particular test.
* The test object should also provide a "yes, we're still consuming llcorehttp
memory" method to be used for spot checks in the middle of tests -- instead
of sprinkling in explicit comparisons as before.
* In fact, the llcorehttp test object in each test_*.hpp file should be
derived from a central llcorehttp test-object base class providing those
methods.
The crash can appear on some non-windows platforms (any LP64 model platforms).
Depending on alignment this can overwrite one word of the pointer `op` declared
above. Subsequently it will crash when later writing to memory through that
pointer
llcorehttp's test_allocator.{h,cpp} overrides global operator new(), operator
new[](), operator delete() and operator delete[](). The two operator new()
functions used to be declared with throw(std::bad_alloc). Worse, for VS 2013
and previous, we needed _THROW0() and _THROW1(std::bad_alloc) instead,
requiring #if logic.
But with dynamic throw declarations deprecated, we must actually remove those.
That obviates the THROW_BAD_ALLOC() / THROW_NOTHING() workarounds in
test_allocator.cpp.
In three different places we use the same pattern: an ssl_thread_id_callback()
function (a static member of LLCrashLogger, in that case) that used to be
passed to CRYPTO_set_id_callback() and therefore returned an unsigned long
representing the ID of the current thread.
But GetCurrentThread() is a HANDLE, an alias for a pointer, and you can't
uniquely cram a 64-bit pointer into an unsigned long.
Fortunately OpenSSL has a more modern API for retrieving thread ID. Pass
each ssl_thread_id_callback() function to CRYPTO_THREADID_set_callback()
instead, converting it to accept CRYPTO_THREADID* and call
CRYPTO_THREADID_set_pointer() or CRYPTO_THREADID_set_numeric() as appropriate().
cmake -E copy does only one file at a time, and older CMake versions don't
handle wildcards. But cmake -E copy is specifically for portability. When the
copy operation itself is Darwin-only, we can count on having 'cp' available.
The new helper functions check_curl_easy_setopt() and
check_curl_multi_setopt() encapsulate the pervasive idiom:
code = curl_{easy,multi}_setopt(handle, option, arg);
check_curl_{easy,multi}_code(code, option);
But since each of these helper functions contains its own local CURL{,M}code
variable 'code', having a caller-scope variable reused for every such call is
no longer necessary -- in fact is no longer used at all. That produces a fatal
warning with MSVC. Get rid of those now-unused variables.
Id rather than sha1 hash, since that is rarely used in modern
certs. The previous form was storing trusted certs using an empty sha1
hash value as the key, which meant most certificates matched... not good.
Modify the LLCertException to pass certificate information back as
LLSD rather than an LLPointer<LLCertificate>, because when the
exception is being thown from the certificate constructor that results
in one of a couple of other exceptions (even refcounting won't save
you when the problem is that the thing you're pointing to never
finished coming into being properly).
Update the certificates in the llsechandler_basic_test to modern
conventions, and extend the classes to allow for an optional
validation date so that the test can use a fixed date. Also make all
the certificates include the plain text form for ease of reference.
Turns out that Monty didn't intend for the int-flavored representation of
HttpStatus to expand to 64 bits even when unsigned long is that wide. So
change the implicit conversion operator, and its uses, to U32 instead. That
produces a consistent toHex() result for both 32-bit and 64-bit builds.
since we no longer support 32-bit Mac builds.
The old build-darwin-i386 directory name appeared in a shocking number of
files. Change CMake paths to use ${CMAKE_BINARY_DIR} -- or, when trying to
find the packages subdirectory, ${AUTOBUILD_INSTALL_DIR}. Change the rest to
at least look for build-darwin-*.
This is the function in indra/llmessage/tests/testrunner.py that iterates
through ports in a specified range, looking for an available one. Other
platforms understand a specification of port 0 to mean: "You pick one. I'll
just use whichever one you picked."
Instead of having testrunner.run()'s caller pass a Thread object on which to
run the caller's server instance's serve_forever() method, just pass the
server instance. testrunner.run() now constructs the Thread. This API change
allows run() to also call shutdown() on the server instance when done, and
then join() the Thread.
The hope is that this will avoid the Python runtime forcing the process
termination code to 1 due to forcibly killing the daemon thread still running
serve_forever().
While at it, eliminate calls to testrunner.freeport() -- just make the runtime
pick a suitable port instead.
Loath though I am to skip testing, this consistent failure is not a failure in
the software being tested (llcorehttp) but rather in the dummy server with
which we're testing it.
It's possible that raising an exception in a worker thread -- even though
we're TRYING to suppress it -- is what's causing the process to terminate with
nonzero rc.
llcorehttp/tests had a clone of llmessage/tests/testrunner.py that was almost
identical save for recognizing an extra optional parameter. Migrate those few
lines into llmessage/tests/testrunner.py; eliminate the copy in llcorehttp;
help test_llcorehttp_peer.py find the testrunner.py in llmessage/tests.
In recent versions of Boost, BOOST_THREAD_LIBRARY depends on
BOOST_SYSTEM_LIBRARY. In llcorehttp/CMakeLists.txt, these were
incorrectly ordered for Linux. Somewhat oddly, that appears to have
caused Linux link errors even in llmath. Fix at least this problem.
Sometimes it can be useful to have http_proxy set in the environment, but if
we leave it set while INTEGRATION_TEST_llcorehttp is running, the test hangs.
Suppress that variable for that integration test.
Andrey Lihatskiy